ava test framework with worker threads - Module did not self-register

[simplenotezy]

Possible bug

I've been noticing this is quite a normal error message to receive, but most of the suggestions recommend deleting node_modules, or similar - which is not really relevant in my case, since the error only happens during a Github Actions deployment.

My github actions step looks something like this:

        run: |
          npm ci
          npm run build --if-present
          npm run test

• I have tried adding npm install --platform=linux --ignore-scripts=false --arch=x64 sharp after npm ci but it will give me the same error.
• I have even tried pushing a build without the package-lock.json to see if that would make a difference during deployment, but to no avail.
• I verified node and npm version are same on my local machine as well as on the Github actions runner
• Output of npm install --platform=linux --arch=x64 --verbose sharp in the github actions runner: https://gist.github.com/simplenotezy/bb1dc5fa506c8629c03d80703247969e

I am not really sure how to proceed here.

❯ npm ls sharp
[email protected] /Users/mf/Projects/project/backend
└── [email protected]

❯ npm why sharp
[email protected]
node_modules/sharp
  sharp@"^0.30.3" from the root project

Not sure if it is related to worker threads (https://sharp.pixelplumbing.com/install#worker-threads). The error does occur when running my tests, which I believe could be multi-threaded (Using Ava 4)

Is this a possible bug in a feature of sharp, unrelated to installation?

• Running npm install sharp completes without error.
• Running node -e "require('sharp')" completes without error.

Are you using the latest version of sharp?

• I am using the latest version of sharp as reported by npm view sharp dist-tags.latest.

What is the output of running npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp?

System:
    OS: Linux 5.11 Ubuntu 20.04.4 LTS (Focal Fossa)
    CPU: (2) x64 Intel(R) Xeon(R) Platinum 8171M CPU @ 2.[60](https://github.com/DoubbleDK/backend/runs/5741770630?check_suite_focus=true#step:7:60)GHz
    Memory: 5.74 GB / 6.78 GB
    Container: Yes
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 16.14.0 - /opt/hostedtoolcache/node/16.14.0/x[64](https://github.com/DoubbleDK/backend/runs/5741770630?check_suite_focus=true#step:7:64)/bin/node
    Yarn: 1.22.17 - /usr/local/bin/yarn
    npm: 8.3.1 - /opt/hostedtoolcache/node/16.14.0/x64/bin/npm
  npmPackages:
    sharp: ^0.30.3 => 0.30.3

What is the expected behaviour?

Should install without errors

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this problem

This happens during my Github Actions deployment, of a Nest.js framework that uses Sharp.

    steps:
      - uses: actions/checkout@v2
      - name: Use Node.js 16.x
        uses: actions/setup-node@v1
        with:
          node-version: 16.x
      - name: Run tests
        run: |
          npm ci
          npm run build --if-present
          npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp
          npm run test

Please provide sample image(s) that help explain this problem

  Error: 
  Something went wrong installing the "sharp" module

  Module did not self-register: '/home/runner/work/backend/backend/node_modules/sharp/build/Release/sharp-linux-x64.node'.

  Possible solutions:
  - Install with the --verbose flag and look for errors: "npm install --ignore-scripts=false --verbose sharp"
  - Install for the current linux-x64 runtime: "npm install --platform=linux --arch=x64 sharp"
  - Consult the installation documentation: https://sharp.pixelplumbing.com/install

  › - Consult the installation documentation: https://sharp.pixelplumbing.com/install
  › Object.<anonymous> (node_modules/sharp/lib/sharp.js:31:9)
  › Object.<anonymous> (node_modules/sharp/lib/constructor.js:8:1)

[lovell]

Not sure if it is related to worker threads (https://sharp.pixelplumbing.com/install#worker-threads). The error does occur when running my tests, which I believe could be multi-threaded (Using Ava 4)

Did you try ava --no-worker-threads ?

[simplenotezy]

@lovell Thanks for prompt reply. Just realised it might be due to that, so I am about to issue a deploy without worker threads. If this is the case, my next question would be: how can I have ava with multiple worker threads to work with Sharp?

[simplenotezy]

Yes, it actually fixed it.

[lovell]

https://sharp.pixelplumbing.com/install#worker-threads

The main thread must call require('sharp') before worker threads are created to ensure shared libraries remain loaded in memory until after all threads are complete.

You'll need to work out how to inject your own logic into ava's main thread.

Alternatively, use a musl-based Linux such as Alpine, which sensibly ignores Node.js' attempt to call dlclose - see https://wiki.musl-libc.org/functional-differences-from-glibc.html#Unloading-libraries

[lovell]

I've slightly expanded the existing docs about this via commit d0c8e95

[simplenotezy]

Awesome, thanks!

[titanism]

You can also do this as a better workaround (save this as ava.config.js and remove ava config from your package.json):

const isCI = require('is-ci');

module.exports = {
  workerThreads: !isCI
};

[lovell]

Thanks @titanism that's a neat solution.

Given this is primarily due to a feature of glibc-based Linux, a possible alternative might be something like:

const { familySync, GLIBC } = require('detect-libc');

module.exports = {
  workerThreads: familySync() !== GLIBC
};

[codetheweb]

In my case I needed worker threads to use AVA's shared workers.

Just requiring the module in the config file for AVA:

// ava.config.js
require("sharp");
// ...

worked for me.

[lovell]

The forthcoming libvips v8.13.0, which will ship with the forthcoming sharp v0.31.0, should remove the need for all these workarounds - see libvips/libvips#2934

[titanism]

Hi there @lovell per forwardemail/forwardemail.net@5460f9a we updated to latest version v0.32.1 and the error still persists with worker threads. This issue is still not resolved. Please re-open or update the worker threads docs at https://sharp.pixelplumbing.com/install#worker-threads to suggest the following workaround:

npm install detect-libc

// ava.config.js
const { familySync, GLIBC } = require('detect-libc');

module.exports = {
  // <https://github.com/lovell/sharp/issues/3164>
  workerThreads: familySync() !== GLIBC
};

[titanism]

@lovell we updated the docs for you, please see PR at #3692

[lovell]

@titanism As of sharp v0.31.0 this should no longer be a problem. Are you able to verify if the removal of this line still allows your tests to pass? https://github.com/forwardemail/forwardemail.net/blob/748b21304ec5224d14ab5b98e7dc0e51ad96b72f/ava.config.js#L10

[luixo]

Hello,
I got a similar problem migrating from a single config of vitest to a workspace setup.
I'm not sure what exactly happened. Before migration I used vmThreads pooling method, afterwards I had multiple projects, each using the exact same method.
Unfortunately I'm not that into what's happening in vitest to tell exactly what's changed between non-workspace and workspace environments.

Adding import("sharp") into general vitest.config.ts (which is in an outer nodejs context, I presume) resolved the problem.

[titanism]

@lovell I've verified and confirmed this fixes the issue in forwardemail/forwardemail.net@69b8401. Thank you for your follow up 🙏 We love using sharp in https://forwardemail.net.

[titanism]

Actually @lovell I stand corrected, this is a bug and is still not working.

[titanism]

Same error:

2024-07-19T17:27:17.7673481Z   Uncaught exception in test/caldav/index.js
2024-07-19T17:27:17.7674145Z 
2024-07-19T17:27:17.7674430Z   test/caldav/index.js:20
2024-07-19T17:27:17.7674874Z 
2024-07-19T17:27:17.7680687Z    19:                                               
2024-07-19T17:27:17.7682003Z    20: const utils = require('../utils');            
2024-07-19T17:27:17.7683032Z    21: const CalDAV = require('../../caldav-server');
2024-07-19T17:27:17.7683608Z 
2024-07-19T17:27:17.7684830Z   Error: 
2024-07-19T17:27:17.7685536Z   Something went wrong installing the "sharp" module
2024-07-19T17:27:17.7686116Z 
2024-07-19T17:27:17.7687860Z   Module did not self-register: '/home/runner/work/forwardemail.net/forwardemail.net/node_modules/.pnpm/[email protected]/node_modules/sharp/build/Release/sharp-linux-x64.node'.
2024-07-19T17:27:17.7689444Z 
2024-07-19T17:27:17.7689765Z   Possible solutions:
2024-07-19T17:27:17.7690928Z   - Using worker threads? See https://sharp.pixelplumbing.com/install#worker-threads
2024-07-19T17:27:17.7692922Z   - Install with verbose logging and look for errors: "npm install --ignore-scripts=false --foreground-scripts --verbose sharp"
2024-07-19T17:27:17.7695422Z   - Install for the current linux-x64 runtime: "npm install --platform=linux --arch=x64 sharp"
2024-07-19T17:27:17.7697253Z   - Consult the installation documentation: https://sharp.pixelplumbing.com/install
2024-07-19T17:27:17.7698100Z 

[titanism]

@lovell not working in [email protected]

[titanism]

No big deal though btw, we have no problem using workerThreads: false in CI. It doesn't really affect us 🙏

[lovell]

If possible please can you try v0.33.0+ as the final piece of the puzzle was added via commit 4a37a27

[titanism]

Done per forwardemail/forwardemail.net@22d5c04, waiting on build to finish.

[titanism]

Ah, it requires a Node engine change:

Error: Could not load the "sharp" module using the darwin-arm64 runtime
Possible solutions:
- Please upgrade Node.js:
    Found 18.16.0
    Requires ^18.17.0 || ^20.3.0 || >=21.0.0

Going to try to get this sorted out and report back.

[titanism]

It works @lovell !!!! ✅

https://github.com/forwardemail/forwardemail.net/actions/runs/10013293960/job/27680579340 (passing)