feat: preload content #1464

alanshaw · 2018-07-24T13:53:57Z

This PR adds a new config property preload:

new IPFS({
  preload: {
    enabled: false,
    addresses: ['/multiaddr/api/address']
  }
})

preload.enabled (default false) enables/disabled preloading - should the default be false?
preload.addresses array of node API addresses to preload content on. This are the addresses we make a /api/v0/refs?arg=QmHash request to, to initiate the preload

This PR upgrades the following APIs to preload content. After adding content with ipfs.files.add (for example), we make a request to the first preload gateway addresses (providing preload.enabled is true), and will fall back to the second etc.

MFS preloading is slightly different - we periodically submit your MFS root to the preload nodes when it changes.

NOTE: this PR adds an option to dag, block and object APIs allowing users to opt out of preloading by specifying preload: false in their options object.

TODO

This PR will make a head request to the root hash of uploaded content, I need to double check that this will preload that hash as well as all of it's descendants or if we need to send a preload request for EVERY hash we add Now uses /api/v0/refs?arg=QmHash instead
Add the new bootstrap and gateway addresses for the preload nodes when provided by @lgierth

lidel

preload.enabled (default false) enables/disabled preloading - should the default be false?

IIUC the reason of endeavor is to have it enabled by default in browser context to restore magic of uploaded content being immediately seen by public gateways.

So it should be enabled by default in browsers, can be disabled by default in node.

lidel · 2018-07-25T16:07:32Z

src/core/preload.js

+// Tools like IPFS Companion redirect requests to IPFS gateways to your local
+// gateway. This is a hint to those tools that they shouldn't redirect these
+// requests as they will effectively disable the preloading.
+const redirectOptOutHint = 'x-ipfs-preload'


Something to note is that browser Companion expects x-ipfs-companion-no-redirect.
If it does not see it in URL, it will redirect to local gateway, which we don't want (local node on a laptop may go offline).

We can add another URL hint to Companion, but is it necessary?
To limit the number of moving pieces before Dweb Summit I would just use x-ipfs-companion-no-redirect for now.

daviddias · 2018-07-25T17:21:03Z

enables/disabled preloading - should the default be false?

I propose that we default to true. It improve user experience a ton.

lidel · 2018-07-25T17:47:29Z

This PR will make a head request to the root hash of uploaded content, I need to double check that this will preload that hash as well as all of it's descendants or if we need to send a preload request for EVERY hash we add

@alanshaw check Lars' comment at #1459 (comment)

IIUC with /api/v0/refs?r=true&arg=<cid> when adding N files you only need 1 preload request (for root) instead of N+1 (one per file + wrapping dir).

alanshaw · 2018-07-26T08:46:25Z

Preload on MFS - do we? It’ll mean that every cp, mv, write etc. will need a preload. It's potentially a lot of churn and a lot of preload requests.

Half of me feels like MFS is your own storage and you don’t really deal with CIDs. Right now if I wanted to share something with someone via IPFS I'd use ipfs.add and share the hash, not copy to MFS, stat path, and then share the hash.

If we don't actively preload MFS content, it'll still be accessible (provided the sharer stays online) due to adding the preload nodes to our bootstrap list, but it won't be as readily available.

On the other hand, I think it's confusing to have some content preloaded and other content not. Data added to IPFS should always be readily available and preloading everything will ensure that.

I am leaning towards preload MFS also.

@diasdavid @achingbrain @olizilla can have opinions?

daviddias · 2018-07-26T09:46:00Z

@alanshaw you can set a rule that when MFS gets used, you set a flag that gets checked every 30s (or so) and if it is true, it will preload the current root. This way we don't bombard the preload gateways for every node update.

alanshaw · 2018-07-26T09:49:48Z

I think that could work but we'd have to switch to requesting /api/v0/refs?r=true&arg=QmFoo (#1459 (comment)) instead of the HEAD request otherwise we'd have to recurse over everything in MFS & send (potentially) hundreds of requests.

achingbrain · 2018-07-26T09:55:48Z

it's confusing to have some content preloaded and other content not

This is probably why it's worth doing. There will be a lot of network activity and extra storage taken up on the preload peers, and the MFS abstraction becomes really leaky when you have to share new CIDs for a given path every time you update something but it's probably better than not being able to get the content someone else has shared with you.

set a flag that gets checked every 30s (or so) and if it is true, preload the current root.

If you wrap ipfs.dag.put with something that does the preloading, every MFS operation will get preloaded automagically as that's what it uses internally. No setInterval required.

alanshaw · 2018-07-26T14:14:43Z

Ok this should be good for another look now - preload is enabled by default and it now uses the /api/v0/refs API for preloading. It will preload at the first address configured and fallback to the subsequent ones on failure.

Preload API requests are now done asynchronously so they don't effect the time it takes to add content. The preload module keeps track of in flight requests and cancels any that are still flying when stop is called on the node.

daviddias · 2018-07-27T10:05:39Z

README.md

@@ -231,6 +231,9 @@ Creates and returns an instance of an IPFS node. Use the `options` argument to s
        - `enabled` (boolean): Make this node a relay (other nodes can connect *through* it). (Default: `false`)
        - `active` (boolean): Make this an *active* relay node. Active relay nodes will attempt to dial a destination peer even if that peer is not yet connected to the relay. (Default: `false`)

+- `preload` (object): Configure external nodes that will preload content added to this node
+    - `enabled` (boolean): Enable content preloading (Default: `true`)
+    - `addresses` (array): Multiaddr API addresses of nodes that should preload content. NOTE: nodes specified here should also be added to your node's bootstrap address list at `config.Boostrap`


We will later need to over all the config values to the Config object like we did for js-libp2p.

daviddias · 2018-07-27T10:05:54Z

src/core/components/index.js

@@ -26,4 +26,4 @@ exports.dht = require('./dht')
 exports.dns = require('./dns')
 exports.key = require('./key')
 exports.stats = require('./stats')
-exports.mfs = require('ipfs-mfs/core')
+exports.mfs = require('./mfs')


daviddias · 2018-07-27T10:06:18Z

src/core/components/mfs.js

+  })
+
+  return mfs(mfsSelf, mfsSelf._options)
+}


daviddias

This work is solid. Great job with covering all the cases and adding tests for them.

achingbrain · 2018-07-27T10:53:05Z

src/core/config.js

@@ -8,6 +8,10 @@ const schema = Joi.object().keys({
    Joi.string()
  ).allow(null),
  repoOwner: Joi.boolean().default(true),
+  preload: Joi.object().keys({
+    enabled: Joi.boolean().default(true),
+    addresses: Joi.array().items(Joi.multiaddr().options({ convert: false }))


Should we require at least one entry here?

achingbrain · 2018-07-27T10:54:15Z

src/core/mfs-preload.js

+        return log.error('failed to stat MFS root for preload', err)
+      }
+
+      if (rootCid !== stats.hash) {


Unless I'm missing something, if the root cid has not changed, it looks like we stop checking for changes.

Yes good catch! If only there was a test for this...

License: MIT Signed-off-by: Alan Shaw <[email protected]>

olizilla · 2018-07-27T14:18:26Z

🚀

ghost assigned alanshaw Jul 24, 2018

ghost added the status/in-progress In progress label Jul 24, 2018

alanshaw mentioned this pull request Jul 24, 2018

⚡️ v0.31.0 RELEASE 🚀 #1458

Closed

23 tasks

alanshaw changed the title ~~[WIP] feat: preload content~~ feat: preload content Jul 25, 2018

alanshaw requested review from jacobheun, achingbrain, daviddias, wraithgar, olizilla, lidel and vasco-santos July 25, 2018 15:37

alanshaw force-pushed the feat/preload-content branch from 91c5999 to d040f26 Compare July 25, 2018 15:51

lidel suggested changes Jul 25, 2018

View reviewed changes

alanshaw mentioned this pull request Jul 26, 2018

Achieving Connectivity Magic #1459

Closed

alanshaw changed the title ~~feat: preload content~~ [WIP] feat: preload content Jul 26, 2018

alanshaw changed the title ~~[WIP] feat: preload content~~ feat: preload content Jul 26, 2018

This was referenced Jul 26, 2018

feat: preload mfs and dag.put #1468

Closed

feat: preload object.new object.put object.patch.* #1471

Closed

daviddias reviewed Jul 27, 2018

View reviewed changes

src/core/components/mfs.js

})

return mfs(mfsSelf, mfsSelf._options)

}

Copy link

Member

daviddias Jul 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

daviddias approved these changes Jul 27, 2018

View reviewed changes

achingbrain suggested changes Jul 27, 2018

View reviewed changes

alanshaw added 18 commits July 27, 2018 13:59

chore: appease the linters

e6d22d1

License: MIT Signed-off-by: Alan Shaw <[email protected]>

chore: more linter changes

2f958b9

License: MIT Signed-off-by: Alan Shaw <[email protected]>

refactor: switch to /api/v0/refs

557c8da

License: MIT Signed-off-by: Alan Shaw <[email protected]>

fix: many fixes

1cdd724

License: MIT Signed-off-by: Alan Shaw <[email protected]>

fix: remove .only

f62bae1

License: MIT Signed-off-by: Alan Shaw <[email protected]>

fix: fallback in order

ac9d246

License: MIT Signed-off-by: Alan Shaw <[email protected]>

fix: start/stop fix and add preload request time logging

89502aa

License: MIT Signed-off-by: Alan Shaw <[email protected]>

fix: bootstrap default list in tests

7c73142

License: MIT Signed-off-by: Alan Shaw <[email protected]>

fix: more fix bootstrap tests

3aa2a1d

License: MIT Signed-off-by: Alan Shaw <[email protected]>

fix: more config fix

ecf1cc3

License: MIT Signed-off-by: Alan Shaw <[email protected]>

feat: preload mfs root preiodically and preload dag.put

977a074

License: MIT Signed-off-by: Alan Shaw <[email protected]>

fix: do not monitor if stopped while starting

ef4616a

License: MIT Signed-off-by: Alan Shaw <[email protected]>

feat: async mfs-preload start and stop

9a4b5e0

License: MIT Signed-off-by: Alan Shaw <[email protected]>

feat: preload object.* API

eb7e01f

License: MIT Signed-off-by: Alan Shaw <[email protected]>

feat: preload block.put

fa1c9f3

License: MIT Signed-off-by: Alan Shaw <[email protected]>

chore: remove .only

787d1dd

License: MIT Signed-off-by: Alan Shaw <[email protected]>

test: add dag.put preload test

d49f17e

License: MIT Signed-off-by: Alan Shaw <[email protected]>

fix: ensure MFS is continually preloaded

9df1012

License: MIT Signed-off-by: Alan Shaw <[email protected]>

alanshaw force-pushed the feat/preload-content branch from 74a79f3 to 9df1012 Compare July 27, 2018 12:59

alanshaw merged commit bffe080 into master Jul 27, 2018

ghost removed the status/in-progress In progress label Jul 27, 2018

alanshaw deleted the feat/preload-content branch July 27, 2018 14:04

ghost mentioned this pull request Jul 28, 2018

Remove bootstrap peers wss0 and wss1 #1474

Closed

lidel mentioned this pull request Nov 27, 2018

CORS issues on preload nodes ipfs/infra#447

Closed

This was referenced Sep 15, 2021

[Snyk] Fix for 1 vulnerabilities rtpro/js-ipfs#45

Open

[Snyk] Fix for 1 vulnerabilities awaisab172/js-ipfs#33

Open

baby636 mentioned this pull request Jul 31, 2022

[Snyk] Security upgrade node-fetch from 2.6.7 to 3.2.10 baby636/js-ipfs#313

Open

awaisab172 mentioned this pull request Aug 1, 2022

[Snyk] Fix for 1 vulnerabilities awaisab172/js-ipfs#49

Open

thexdesk mentioned this pull request Aug 1, 2022

[Snyk] Fix for 1 vulnerabilities thexdesk/js-ipfs#22

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: preload content #1464

feat: preload content #1464

alanshaw commented Jul 24, 2018 •

edited

Loading

lidel left a comment

lidel Jul 25, 2018

daviddias commented Jul 25, 2018

lidel commented Jul 25, 2018 •

edited

Loading

alanshaw commented Jul 26, 2018

daviddias commented Jul 26, 2018

alanshaw commented Jul 26, 2018 •

edited

Loading

achingbrain commented Jul 26, 2018 •

edited

Loading

alanshaw commented Jul 26, 2018

daviddias Jul 27, 2018

daviddias Jul 27, 2018

daviddias Jul 27, 2018

daviddias left a comment

achingbrain Jul 27, 2018

achingbrain Jul 27, 2018

alanshaw Jul 27, 2018

olizilla commented Jul 27, 2018

feat: preload content #1464

feat: preload content #1464

Conversation

alanshaw commented Jul 24, 2018 • edited Loading

lidel left a comment

Choose a reason for hiding this comment

lidel Jul 25, 2018

Choose a reason for hiding this comment

daviddias commented Jul 25, 2018

lidel commented Jul 25, 2018 • edited Loading

alanshaw commented Jul 26, 2018

daviddias commented Jul 26, 2018

alanshaw commented Jul 26, 2018 • edited Loading

achingbrain commented Jul 26, 2018 • edited Loading

alanshaw commented Jul 26, 2018

daviddias Jul 27, 2018

Choose a reason for hiding this comment

daviddias Jul 27, 2018

Choose a reason for hiding this comment

daviddias Jul 27, 2018

Choose a reason for hiding this comment

daviddias left a comment

Choose a reason for hiding this comment

achingbrain Jul 27, 2018

Choose a reason for hiding this comment

achingbrain Jul 27, 2018

Choose a reason for hiding this comment

alanshaw Jul 27, 2018

Choose a reason for hiding this comment

olizilla commented Jul 27, 2018

alanshaw commented Jul 24, 2018 •

edited

Loading

lidel commented Jul 25, 2018 •

edited

Loading

alanshaw commented Jul 26, 2018 •

edited

Loading

achingbrain commented Jul 26, 2018 •

edited

Loading