Compatibility with webpack 5

[Gozala]

I have been trying to write an example that uses webpack 5 and sadly things don't work out of the box. I would like to use this thread to document problems, ways to workaround and track the issues that need to be addressed to get make things compatible.

[Gozala]

Attempt to build code that uses js-ipfs produces lot of errors like these:

ERROR in ./node_modules/cbor/lib/commented.js 3:15-32
Module not found: Error: Can't resolve 'stream' in '/Users/gozala/Projects/js-ipfs-3/examples/browser-service-worker/node_modules/cbor/lib'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "stream": require.resolve("stream-browserify") }'
        - install 'stream-browserify'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "stream": false }
 @ ./node_modules/cbor/lib/cbor.js 3:0-42
 @ ./node_modules/ipfs-core/src/components/pin/pin-manager.js 11:13-28
 @ ./node_modules/ipfs-core/src/components/init.js 34:19-47
 @ ./node_modules/ipfs-core/src/components/index.js 95:0-32
 @ ./node_modules/ipfs-core/src/index.js 20:19-42
 @ ./node_modules/ipfs/src/index.js 5:13-37
 @ ./src/worker.js 4:0-24 16:21-32
 @ ./src/main.js 3:15-72 25:23-80

All of them seem to come from cbor that depends on streams (although it could be that fixing that would uncover more sites

js-ipfs/packages/ipfs-core/src/components/pin/pin-manager.js

Line 11 in 308baa4

const cbor = require('cbor')

Adding resolve.fallback us suggested by the error message, gets build go through, however it does not actually address the problem but rather defers it to the runtime:

Uncaught (in promise) ReferenceError: Buffer is not defined
    at Object.../../node_modules/core-util-is/lib/util.js (util.js:103)
    at __webpack_require__ (bootstrap:18)
    at Object.../../node_modules/stream-browserify/node_modules/readable-stream/lib/_stream_readable.js (_stream_readable.js:67)
    at __webpack_require__ (bootstrap:18)
    at Object.../../node_modules/stream-browserify/node_modules/readable-stream/readable-browser.js (readable-browser.js:1)
    at __webpack_require__ (bootstrap:18)
    at Object.../../node_modules/stream-browserify/index.js (index.js:28)
    at __webpack_require__ (bootstrap:18)
    at Object../node_modules/cbor/lib/commented.js (commented.js:3)
    at __webpack_require__ (bootstrap:18)

There is also relevant issue isaacs/core-util-is#29 on file which should address that if fixed.

[Gozala]

It is possible to workaround that using webpack.ProvidePlugin as follows:

plugins: [
    new webpack.ProvidePlugin({
      Buffer: ['buffer', 'Buffer'],
    })
  ]

However that just uncovers another runtime problem:

_stream_writable.js:57 Uncaught (in promise) ReferenceError: process is not defined
    at Object.../../node_modules/stream-browserify/node_modules/readable-stream/lib/_stream_writable.js (_stream_writable.js:57)
    at __webpack_require__ (bootstrap:21)
    at fn (hot module replacement:60)
    at Object.../../node_modules/stream-browserify/node_modules/readable-stream/readable-browser.js (readable-browser.js:4)
    at __webpack_require__ (bootstrap:21)
    at fn (hot module replacement:60)
    at Object.../../node_modules/stream-browserify/index.js (index.js:28)
    at __webpack_require__ (bootstrap:21)
    at fn (hot module replacement:60)
    at Object../node_modules/cbor/lib/commented.js (commented.js:3)

[Gozala]

That last issue could be overcome via

plugins: [
    new webpack.ProvidePlugin({
      Buffer: ['buffer', 'Buffer'],
      process: 'process/browser'
    })
 ]

[achingbrain]

This should be handled by aegir and not in this repo.

Uncaught (in promise) ReferenceError: Buffer is not defined

This is usually solved by adding Buffer: true to the .aegir.js file for a given module. If it's not working please open an issue against aegir.

[Gozala]

This is for IPFS examples (or use of js-ipfs in general) so I don't think telling users to use aegir to use js-ipfs is right approach. That said I think we can close it as #3374 end up demonstrating this.

[achingbrain]

Don't get me wrong, I'm not suggesting we should tell people to use aegir, but the best way for us to be compatible with webpack 5 is to upgrade the version of webpack aegir uses so we are bundling with webpack 5 ourselves. That way we dogfood it and (ideally) the user will not have to do any special config.