Add delegated routing to js-ipfs browser

[jacobheun]

This issue is a child of Connectivity Magic #1459, and is intended to track the completion of the Delegated Routing portion of that.

Until browsers have the support needed to handle DHT queries quickly (direct dialing capabilities), delegated peer and content routing will be an important part of ensuring nodes are able to find peers and content.

js-libp2p already has support for custom content and peer routing modules. There are currently two modules that can leverage the IPFS api to achieve both. An example of this is at https://github.com/libp2p/js-libp2p/tree/v0.25.3/examples/delegated-routing.

TODOs

These are the necessary steps to adding delegated routing support to js-ipfs in the browser:

• Deploy go-ipfs nodes that have support for the needed delegated routing endpoints. Context at gateway: make API commands configurable kubo#5565
• Add delegated content routing module to js-ipfs, libp2p browser config
• Add delegated peer routing module to js-ipfs, libp2p browser config

Refs:

• Delegated Routing libp2p/js-libp2p#120

[lidel]

Some additional context:

Enabling preload nodes as delegated routers

IIUC if we want to reuse preload nodes as delegated routers, we need to prepare them first:

• expose swarm/connect, dht/findprovs, and dht/findpeer in addition to refs (which is already being used by existing preload logic)
  • Based on Add Gateway.APICommands for /api allowlists go-ipfs-config#10 I think list of exposed commands can be tweaked via Gateway.APICommands config
• ensure preload nodes stay connected to gateways

@mburns is working on it

Wiring up *-delegated*-routing modules to use preloads

I don't see libp2p-delegated-content-routing and libp2p-delegated-routing being wired up in js-ipfs to use existing preload servers yet.
(Note that by default both modules will try ipfs.io/api/v0/ and fail, as some of required commands are not exposed on ipfs.io gateways)

@alanshaw are you planning to PR and release this before IPFS Camp?
cc @hugomrdias as delegated DHT may be useful for IPNS as a fallback

Tweak the way js-ipfs picks preload server

Right now js-ipfs tries preload nodes in order, so the first one gets hit by default and second one may not be fully utilized unless first one is under too much load and starts returning errors. We should pick one at random to distribute the load and make it scale nicely when someone defines more than two preload nodes.

[jacobheun]

Added delegate routing config in #2195.

For the go-ipfs preload node config, it would be fantastic if we could specify that configuration somewhere to make it easier for the community to run their own delegate routing nodes. Do we have the setup process for doing that @mburns ? I'd be happy to add it to the delegated routing module README's, and the libp2p example.

[mburns]

For just IPFS, I used the notes from ipfs/kubo#5565, specifically:
$ ipfs config Gateway.APICommands --json '["dht/findprovs", "dht/findpeer", "refs", "swarm/connect"]'

For our Production setup--with Nginx fronting for IPFS--we had to allow that URL path to be passed to the API endpoint, which is not the default backend (port 8080 or 4001):

http {
  upstream api {
    server 127.0.0.1:5001;
  }

  server {
    ...

    location ~ /api/v0/(dht\/findpeer|dht\/findprovs|swarm\/connect|refs) {
      proxy_set_header Host $host:443;
      proxy_set_header X-Ipfs-Gateway-Prefix "";
      proxy_pass http://api;
    }
  }
}

[lidel]

For the record, I was unsuccessful with Gateway.APICommands and go-ipfs v0.4.21-rc3. Seems that neither ipfs/kubo#5565 nor ipfs/kubo#5565 were merged, and undocumented APICommands setting added in ipfs/go-ipfs-config#10 is simply ignored.

Instead, some commands are hardcoded to just work on gateway port, some are not:

$ ipfs config Gateway.APICommands --json '[]'

$ curl -I 'http://127.0.0.1:8080/api/v0/refs?r=true&arg=QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR'
HTTP/1.1 200 OK

$ curl -I -X POST 'http://127.0.0.1:8080/api/v0/dht/findpeer?arg=QmNsW4Dxk3Bcjc1ofu45wATsyK3HLGC38KZr5nN2NjNo6e'
HTTP/1.1 404 Not Found

Update: Gateway.APICommands seems to be broken: libp2p/js-libp2p#371 (review)

[jacobheun]

I was able to use the API commands against IPFS master locally to get the delegate example working. I created a PR at js-libp2p to include the setup instructions. libp2p/js-libp2p#371