🐛 BUG: workerd module resolution issues ("Error: No such module")

[stevezhu]

Which Cloudflare product(s) does this pertain to?

Workers Vitest Integration

What version(s) of the tool(s) are you using?

3.65.0 [Wrangler], 1.20240614.0 [workerd], 0.4.12 [@cloudflare/vitest-pool-workers]

What version of Node are you using?

22.5.0

What operating system and version are you using?

Windows 11
(I also tested on MacOS, but not too thoroughly. Fairly certain this also happens there.)

Describe the Bug

Observed behavior

It seems there are issues with module resolution for dependencies with workerd.

The two issues I've seen are:

1. Can't resolve directory imports like below. If index.js is added explicitly, then it works. Error logs below under "Issue 1". Update: Seems to be fixed by fix: ensure that directory imports in commonjs modules get resolved to their index file #6455

- const encoders = require('./encoders');
+ const encoders = require('./encoders/index.js');

2. Can't resolve json files. Error logs below under "Issue 2".

Expected behavior

Should be able to resolve both of these imports.

Steps to reproduce

Reproduction for issue 1: https://github.com/stevezhu/cloudflare-vitest-integration-resolution-bug
Reproduction for issue 2: https://github.com/stevezhu/cloudflare-vitest-integration-resolution-bug/tree/json-resolution-error

Steps to run

1. pnpm i
2. pnpm run test

Please provide a link to a minimal reproduction

https://github.com/stevezhu/cloudflare-vitest-integration-resolution-bug

Please provide any relevant error logs

Issue 1:

➜ pnpm run test

> vitest-integration-resolution-bug@ test D:\Development\cloudflare-vitest-integration-resolution-bug
> vitest run


 RUN  v1.5.0 D:/Development/cloudflare-vitest-integration-resolution-bug

[vpw:inf] Starting isolated runtimes for vitest.config.ts...
workerd/server/server.c++:2821: error: Fallback service failed to fetch module; payload = ; spec = /?specifier=%2FD%3A%2FDevelopment%2Fcloudflare-vitest-integration-resolution-bug%2Fnode_modules%2F.pnpm%2Fasn1.js%405.4.1%2Fnode_modules%2Fasn1.js%2Flib%2Fasn1%2Fencoders&referrer=%2FD%3A%2FDevelopment%2Fcloudflare-vitest-integration-resolution-bug%2Fnode_modules%2F.pnpm%2Fasn1.js%405.4.1%2Fnode_modules%2Fasn1.js%2Flib%2Fasn1%2Fapi.js%3Fmf_vitest_no_cjs_esm_shim&rawSpecifier=.%2Fencoders
 ❯ src/index.test.ts (0)

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Failed Suites 1 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 FAIL  src/index.test.ts [ src/index.test.ts ]
Error: No such module "D:/Development/cloudflare-vitest-integration-resolution-bug/node_modules/.pnpm/[email protected]/node_modules/asn1.js/lib/asn1/encoders".
 ❯ node_modules/.pnpm/[email protected]/node_modules/asn1.js/lib/asn1/api.js?mf_vitest_no_cjs_esm_shim:3:18
 ❯ node_modules/.pnpm/[email protected]/node_modules/asn1.js/lib/asn1.js?mf_vitest_no_cjs_esm_shim:7:15
 ❯ node_modules/.pnpm/[email protected]/node_modules/jwk-to-pem/src/ec.js?mf_vitest_no_cjs_esm_shim:3:12
 ❯ node_modules/.pnpm/[email protected]/node_modules/jwk-to-pem/src/jwk-to-pem.js?mf_vitest_no_cjs_esm_shim:3:10
 ❯ node_modules/.pnpm/[email protected]/node_modules/get-jwks/src/get-jwks.js?mf_vitest_no_cjs_esm_shim:5:18

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯

 Test Files  1 failed (1)
      Tests  no tests
   Start at  16:50:06
   Duration  1.04s (transform 21ms, setup 0ms, collect 0ms, tests 0ms, environment 0ms, prepare 189ms)

[vpw:dbg] Shutting down runtimes...
 ELIFECYCLE  Test failed. See above for more details.

Issue 2:

➜ pnpm run test

> vitest-integration-resolution-bug@ test D:\Development\cloudflare-vitest-integration-resolution-bug
> vitest run


 RUN  v1.5.0 D:/Development/cloudflare-vitest-integration-resolution-bug

[vpw:inf] Starting isolated runtimes for vitest.config.ts...
workerd/server/server.c++:2874: error: Fallback service failed to fetch module; exception = (unknown):-1: failed: std::exception: Uncaught JsExceptionThrown
stack: 7ff6a2c9f623 7ff6a31afb98 7ff6a306f0b0 7ff6a2d5c61e 7ff6a2ae199d 7ff6a341ff21 7ff6a3067edd 7ff6a341f09e 7ff6a3054dc7 7ff6a3054cd1 7ff6a4202868 7ff6a4200b2f 7ff6a41fe89b 7ff6a41fe3f2 7ff6a2941bdf 7ff6a2940f77; spec = /?specifier=%2FD%3A%2FDevelopment%2Fcloudflare-vitest-integration-resolution-bug%2Fnode_modules%2F.pnpm%2Felliptic%406.5.6%2Fnode_modules%2Felliptic%2Fpackage.json&referrer=%2FD%3A%2FDevelopment%2Fcloudflare-vitest-integration-resolution-bug%2Fnode_modules%2F.pnpm%2Felliptic%406.5.6%2Fnode_modules%2Felliptic%2Flib%2Felliptic.js%3Fmf_vitest_no_cjs_esm_shim&rawSpecifier=..%2Fpackage.json 
 ❯ src/index.test.ts (0)

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Failed Suites 1 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 FAIL  src/index.test.ts [ src/index.test.ts ]
Error: No such module "D:/Development/cloudflare-vitest-integration-resolution-bug/node_modules/.pnpm/[email protected]/node_modules/elliptic/package.json".
 ❯ node_modules/.pnpm/[email protected]/node_modules/elliptic/lib/elliptic.js?mf_vitest_no_cjs_esm_shim:5:20
 ❯ node_modules/.pnpm/[email protected]/node_modules/jwk-to-pem/src/ec.js?mf_vitest_no_cjs_esm_shim:5:7
 ❯ node_modules/.pnpm/[email protected]/node_modules/jwk-to-pem/src/jwk-to-pem.js?mf_vitest_no_cjs_esm_shim:3:10
 ❯ node_modules/.pnpm/[email protected]/node_modules/get-jwks/src/get-jwks.js?mf_vitest_no_cjs_esm_shim:5:18

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯

 Test Files  1 failed (1)
      Tests  no tests
   Start at  16:52:24
   Duration  1.17s (transform 21ms, setup 0ms, collect 0ms, tests 0ms, environment 0ms, prepare 191ms)

[vpw:dbg] Shutting down runtimes...
 ELIFECYCLE  Test failed. See above for more details.
 ```

[davidbarratt]

From my testing, this seems to be a bug only when using pnpm, does that seem right to you?

[davidbarratt]

it's almost like it's not following the symlinks correctly or something like that.

[stevezhu]

From my testing, this seems to be a bug only when using pnpm, does that seem right to you?

It doesn't seem to be a pnpm issue for me. I just tested with npm and got the same results.

It shouldn't be a symlink issue because this is happening within the dependency and it only happens when the index.js for a directory import isn't explicitly defined. If you go into the dependency and manually add index.js, the error goes away.

[MasterKale]

1. Can't resolve directory imports like below. If index.js is added explicitly, then it works.

Jumping in here to +1 this problem in particular. In my Cloudflare Worker project (using npm) the ES2015 build of @peculiar/asn1-schema has a build/es2015/errors/index.js file...

...but unfortunately the import paths within @peculiar/asn1-schema/build/es2015/ that only specify "./errors", without a reference to index.js, error out with No such module:

 FAIL  test/index.spec.ts [ test/index.spec.ts ]
Error: No such module "<snip>/node_modules/@peculiar/asn1-schema/build/es2015/errors".
  imported from "<snip>/node_modules/@peculiar/asn1-schema/build/es2015/index.js"

For sake of curiosity I found out I can go into node_modules/ and manually change these import paths to "./errors/index" to see the error go away. This isn't a fix, of course, but does support @stevezhu's observations above.

[MasterKale]

@stevezhu @davidbarratt Try updating to the latest @cloudflare/[email protected], which includes #6455 a.k.a. "fix: ensure that directory imports in commonjs modules get resolved to their index file"

My index.js-related problem above has been fixed 🎉

[stevezhu]

@stevezhu @davidbarratt Try updating to the latest @cloudflare/[email protected], which includes #6455 a.k.a. "fix: ensure that directory imports in commonjs modules get resolved to their index file"

My index.js-related problem above has been fixed 🎉

Thanks! That does fix the index.js issue but it seems like the json resolution issue is still there.

[davidbarratt]

@MasterKale I have done that, but still facing the same error: #5367 (comment)

[capaj]

I get the same on ubuntu:

workerd/server/server.c++:3059: error: Fallback service failed to fetch module; payload = ; spec = /?specifier=node%3Achild_process&referrer=%2Fhome%2Fcapaj%2Fwork-repos%2Fteleprompt.lol%2Fcallnotes%2Fnode_modules%2F.pnpm%2Fdetect-libc%402.0.2%2Fnode_modules%2Fdetect-libc%2Flib%2Fdetect-libc.js%3Fmf_vitest_no_cjs_esm_shim&rawSpecifier=node%3Achild_process

[dario-piotrowicz]

I'm not sure, but json the issue might be a duplicate of #6591

(I'm just mentioning it so that when we solve one we can cross check the other as well)