Support for js-ipfs as a replacement for HTTP gateway

[daviddias]

Right now this extension relays on the HTTP Gateway to serve the pages through IPFS. Ideally, we should be able to capture the requests and reply back without having to make a request to a gateway, this way we could serve pages from js-ipfs too.

Thoughts @lidel ?

[victorb]

I've been trying to do this, by intercepting the requests with the API that the web extensions provide and there are ways of canceling or redirecting requests, but I haven't found a way of injecting content into the request's body or overwriting it completely

[daviddias]

Would https://flyweb.github.io/ be a way to achieve this? Seems that the project has matured a lot since I last checked.

[lidel]

Can't do much with current WebExtension APIs related to WebRequest. One can cancel request, redirect it and modify headers, but we can't create artificial response with arbitrary body.

I glanced at FlyWeb and it does not seem to be feasible: my understanding is that "FlyWeb node" can't be run from within WebExtension, and if we expect user to run additional process in her system that could be "discovered" by Firefox.. it is just as good as running go-ipfs in the first place. Or am I missing something?

The only(??) "stable" thing that could work right now is HTTP gateway, we could use it the same way as go-ipfs one.

I am not sure if it is possible for browserified js-ipfs to provide local IPFS2HTTP Gateway (exposing tcp port with HTTP service just like go-ipfs does), but there is an open issue for this very topic at ipfs/js-ipfs#693 (and PR at ipfs/js-ipfs#698).

Any ideas / hacks would be appreciated. I am shooting blanks at this one.

[daviddias]

I am not sure if it is possible for browserified js-ipfs to provide local IPFS2HTTP Gateway (exposing tcp port with HTTP service just like go-ipfs does),

It wouldn't be browserified/webpacked as that step would remove any native Node.js modules dependencies. To confirm, what you are saying is that we can open a TCP server or an HTTP server from a WebExtension? If yes, could you provide me with an example? I'm pretty sure we can get the full gateway working for js-ipfs as well pretty quickly.

[victorb]

I don't we will be able to open TCP sockets in web extensions, there is no API for that. However, we might be able to create a Websocket API that you can use from a webpage, and that way load the content. That would be pretty nifty actually.

…

On Tue, May 9, 2017 at 10:07 PM, David Dias ***@***.***> wrote: I am not sure if it is possible for browserified js-ipfs to provide local IPFS2HTTP Gateway (exposing tcp port with HTTP service just like go-ipfs does), It wouldn't be browserified/webpacked as that step would remove any native Node.js modules dependencies. To confirm, what you are saying is that we can open a TCP server or an HTTP server from a WebExtension? If yes, could you provide me with an example? I'm pretty sure we can get the full gateway working for js-ipfs as well pretty quickly. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#248 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAcD9I3SokDUyd-9VKkbq5MiDr0L5zswks5r4MeQgaJpZM4NVxl9> .

[lidel]

@victorbjelkholm even if we hack around and pass resource data over Websocket API, I am not sure how we could construct artificial HTTP response with it.

Example: addon intercepts request to resource at https://ipfs.io/ipfs/<mhash>, we read data at <mhash> from js-ipfs instance via Websocket API. And.. what then?

@diasdavid there is an open Bug 1247628 - Enable "TCP and UDP Socket API" for WebExtensions, which would be game-changer for us. Without it, we can't really have real "http gateway" provided by js-ipfs.

[victorb]

Yeah, not fully developed thought yet but if we inject a tiny script to read from websocket endpoint on `onHeadersReceived` and use that to read the content, it might be possible. But as I said, not fully sure this is even possible.

…

On Tue, May 9, 2017 at 10:23 PM, Marcin Rataj ***@***.***> wrote: @victorbjelkholm <https://github.com/VictorBjelkholm> even if we hack around and pass resource data over Websocket API, I am not sure how we could construct artificial HTTP response with it. Example: addon intercepts request to resource at https://ipfs.io/ipfs/ <mhash>, we read data at <mhash> from js-ipfs instance via Websocket API. And.. what then? PS. FYI there is an open Bug 1247628 - Enable "TCP and UDP Socket API" for WebExtensions <https://bugzilla.mozilla.org/show_bug.cgi?id=1247628>, which would be game-changer for us. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#248 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAcD9G0StJDqgekBZehgzbAOw66EGCDKks5r4MtNgaJpZM4NVxl9> .

[lidel]

onHeadersReceived is too late, it is fired when the HTTP response headers sent by public gateway already have been received[1]:

onBeforeRequest is what addon uses right now for redirect, but its API is very limited and you can either cancel or redirect request[2].

There is no obvious way to "manually build response" with WX APIs, but I wonder if thinking outside the WX box and using something like Service Worker API could be our silver bullet[3]:

Your service worker can respond to requests using the FetchEvent event. You can modify the response to these requests in any way you want, using the FetchEvent.respondWith method.

---

[1]: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/webRequest
[2]: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/webRequest/onBeforeRequest
[3]: https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API

[daviddias]

There is no obvious way to "manually build response" with WX APIs, but I wonder if thinking outside the WX box and using something like Service Worker API could be our silver bullet[3]:

That would require Service Workers to be spawned from a WebExtension and we would have to spawn one for every domain so that the requests can be captured.

That gives me an idea of having a service-worker.ipfs.io that will install a service worker on your browser with js-ipfs on it, and doing the requests through js-ipfs instead of loading hashes from http. This is actually pretty cool. It would only work for ipfs.io urls, it is an interesting start :)

[lidel]

That would require Service Workers to be spawned from a WebExtension and we would have to spawn one for every domain so that the requests can be captured.

Idea: if we keep already existing redirect in onBeforeRequest, then all IPFS requests have the same hostname and we only need Service Worker for 127.0.0.1:8080 (or service-worker.ipfs.io).

[daviddias]

Yeah, that can work two

• We should not use 127.0.0.1:8080 and use redirect requests to service-worker.ipfs.io, this way we keep the browser happy with https
• We lead the user to open the service-worker.ipfs.io page once the extension is installed, so it gets installed on their browser
• At that moment, we don't need the extension to do all the redirects because the service-worker.ipfs.io can do it. We will still want the extension to be a nice UI though that can configure the node that is running on that Service Worker.

[daviddias]

So, just to make sure we are all on the same page (this is indeed very interesting). This is what I'm proposing:

I'm convinced that this actually works!

Then, we can juice this up by running another js-ipfs node in the extension so that we can also have WebRTC enabled, since WebExtensions are closer to get RTC DataChannel than Service Workers.

[ghost]

This means offline use cases will be impossible eh?

What about @kyledrake's search bar hack?

[daviddias]

@lgierth it doesn't mean that at all. You just need to load it once and you are golden.
I believe your confusion is because you are seeing a domain there, but that is how service workers work, they get downloaded/installed and then they are capable of capturing requests to that domain, it is kind of an app cache with the ability to touch network requests. Read more at https://github.com/w3c/ServiceWorker/blob/master/explainer.md

What search bar hack? Could you provide url?

[ghost]

You just need to load it once and you are golden.

Okay that's much better :) Still not ideal but something one could work with :)

I just checked about the search bar hack, and it was purely to capture ipfs:// in lack of programmable protocol handlers.

[ghost]

So, service-worker.ipfs.io would have to expose:

• / (installs the service worker)
• /ipfs (usual gateway)
• /ipns (dito)

right?

[daviddias]

Yes :)

[ghost]

Ideally the serviceworker installation could happen semi-automatically when installing the webextension. So that's it's just really hard to miss it.

[ghost]

Does service-worker.ipfs.io have to be a js-ipfs node? I figure it can just be a regular gateway, with / being a dnslink page.

[dryajov]

@diasdavid would the service worker run in an iframe as @kumavis suggested? That gives you the ability of sharing the IPFS node across all the tabs that load the iframe.

[daviddias]

@lgierth the point of it being a js-ipfs node is so that you don't have to touch the Gateway through HTTP and you route all the content requests through http. I might be missing your point if it happens to be a gateway node, how would that be better than just making requests to the gateway?

@dryajov no need to do it for what we are trying to achieve with the WebExtension, but yeah we could. The point is not to share the IPFS node so that apps can use it programmatically, it is to fetch content through IPFS protocol.

[ghost]

@lgierth the point of it being a js-ipfs node is so that you don't have to touch the Gateway through HTTP and you route all the content requests through http. I might be missing your point if it happens to be a gateway node, how would that be better than just making requests to the gateway?

Oh yeah the actual service-worker is of course a js-ipfs node -- what I mean is the thing that serves the service-worker.js in the first place. That can be a simple dnslink page from what I understand.

[daviddias]

Ah! Yeah, sure, the service-worker is not any endpoint on the Web, it just needs to be a place to fetch a blob of JS, so yes, even the way the service-worker is loaded can be done through IPFS :)

[ghost]

Cool cool 👍

[kumavis]

do requests across domains hit the service worker?

coolstuff.io -> service-worker.ipfs.io

seems likes no, but they are considering it
https://developers.google.com/web/updates/2016/09/foreign-fetch

because of this, we need to redirect the request to a service worker or iframe

• by overriding via the extension
• or developer buy-in

Oh yeah the actual service-worker is of course a js-ipfs node

cant use tcp/udp or webrtc in ServiceWorker atm tho 😿

[ghost]

cant use tcp/udp or webrtc in ServiceWorker atm tho 😿

Can you use FlyWeb in a ServiceWorker though? FlyWeb will take care of local discovery, and the upcoming merge of the js-ipfs and go-ipfs networks takes care of global discovery. With this, the lack of webrtc discovery is less of a pressing issue imho.

[kumavis]

@lgierth the assumption is you're running a node locally?

[kumavis]

metamask (ethereum browser integration) deals with these same limitations:
wanting to expose a new api to webpages without break the web
wanting to connect to a tcp/udp p2p network
wanting the damn thing to work without the user to install a bunch of stuff first
considering running a bridged network via webrtc + hybrid clients
exploring ServiceWorkers and other new things that might help

happy to share all i've learned and hope we can come up with something else clever

[daviddias]

by overriding via the extension

Yes, that is what is being proposed here: #248 (comment) :)

Just to be clear, what we are trying to solve first with this approach is to make any /ipfs/Qm... content through the IPFS the network.

Can you use FlyWeb in a ServiceWorker though?

Unfortunately no. FlyWeb won't appear on Chrome anytime soon too.

js-ipfs and go-ipfs networks takes care of global discovery. With this, the lack of webrtc discovery is less of a pressing issue imho.

Yeah, with relay in it will work just fine :)

@kumavis, @lgierth the assumption is you're running a node locally?

Locally as in "inside the Service Worker", no daemon needed

[dryajov]

@kumavis

considering running a bridged network via webrtc + hybrid clients

I believe this is what circuit will bring.

[kumavis]

@dryajov i was talking more about the ethereum network, but yes ipfs has a nice solution on the roadmap there

[kumavis]

Locally as in "inside the Service Worker", no daemon needed

@diasdavid no tcp/udp or webrtc - just going to hit a local node via http and fallback to the gateway?

[dryajov]

@kumavis Not necesarily, with circuit you can dial to a browser node, albeit over a third relay node, but all content in browser should be reachable without the need of HTTP or the gateway.

[daviddias]

quick update: making progress here -> https://github.com/ipfs/ipfs-service-worker/tree/v0.1.0

Now I need to complete this the Awesome WebWorkers endeavor because WebWorkers and Service Workers share the same limitations (i.e: Window is not defined)

[dryajov]

:+1

[daviddias]

Update: ipfs-shipyard/ipfs-service-worker-demos#1 (comment)

Any service worker experts out there? I've a couple of questions that I'm not finding good answers on the interwebs

[Stebalien]

We may also be able to redirect to an extension URL. On firefox, this would be moz-extension://webext-id/gateway.html#/ipfs/.... (which could be hidden behind a custom protocol?).

[lidel]

I feel a quick update is due, so here are some news from "js-ipfs in webextension" land:

Toggling ipfs node type works in Brave!
– #320 (comment)

Implementation is experimental (and Brave-only),
but if a curious soul wants to play with it, here are the steps:

• Enable Embedded IPFS Support via js-ipfs
• Brave build/install

Due to missing apis, we can't use js-ipfs in Firefox and Chrome yet, but we are playing with idea of using js-ipfs as a fallback for uploading content when go-ipfs is down.

[lidel]

Closing this with Q3 2018 status update :)

Service Worker endeavor

• meta issue for tracking work related to js-ipfs in service worker: IPFS runs as a Service Worker! in-web-browsers#55
  • the service worker gateway now has a dedicated repo:
    https://github.com/ipfs-shipyard/service-worker-gateway
• Known Issue: js-ipfs-api does not work in service worker in Firefox (yet): https://github.com/ipfs/js-ipfs-api/issues/848

Follow above links for future SW updates.

Injecting payload from js-ipfs

As for "js-ipfs fetching responses and returning them to user without hitting HTTP" we have a prototype of mozilla/libdweb native protocol handler capable of doing just that (works with both js-ipfs and js-ipfs-api). Protocol handler work continues in #533

We also have working prototype of local discovery and TCP transport, which reuses FlyWeb's internals via some high level APIs: #553