[BUG] overrides cause lock file to be out of sync, since 8.6

[jklingen]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

npm overrides that work fine with npm 8.5.5, are causing an error during npm ci as of npm 8.6:

npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! Missing: [email protected] from lock file

After running npm install again (as suggested by the error message) npm ci is working again, but the transient trim-newlines dependency version is no longer overridden in package-lock.json.

Expected Behavior

npm install with overrides configuration should produce a package-lock.json that can be processed by npm ci and contains overridden versions.

Steps To Reproduce

On a local development machine, create a blank project with defaults and add a dependency with vulnerable transient dependency:

mkdir npm-overrides-issue
cd npm-overrides-issue
npm init
npm i svg2sprite-cli

-> 3 high severity vulnerabilities

add overrides configuration to mitigate transient vulnerability

  "overrides": {
    "trim-newlines": "^3.0.1"
  }

then

npm i
npm ci

-> still 3 high severity vulnerabilities, ci still works

force update of package-lock.json

rm -rf package-lock.json node_modules
npm i
npm ci

-> found 0 vulnerabilities, ci fails with the following error

npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! Missing: [email protected] from lock file

in order to check that it actually works with previous npm versions:

npm i -g [email protected]
rm -rf package-lock.json node_modules
npm i
npm ci

-> found 0 vulnerabilities, ci works as expected

Environment

• npm: 8.6.0
• Node.js: v16.14.0
• OS Name: MacOS
• System Model Name: MacBook Pro 2021
• npm config:

; "user" config from /Users/redacted/.npmrc

; redacted: custom registry configuration for internal dependencies, not related

; node bin location = /Users/redacted/.nvm/versions/node/v16.14.0/bin/node
; cwd = /Users/redacted/dev/npm-overrides-issue
; HOME = /Users/redacted
; Run `npm config ls -l` to show all defaults.

• package.json

{
  "name": "npm-overrides-issue",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "svg2sprite-cli": "^2.0.1"
  },
  "overrides": {
    "trim-newlines": "^3.0.1"
  }
}

[jklingen]

After some more investigation, the new error seems to be caused by this change, which was released with 8.6.0:
#4599

The question remains: how can the lock file be out of sync if we just ran npm i before 🤔

[nlf]

i fixed this as part of #4709 and did a bunch of manual testing to make sure the package-lock.json stays correctly in sync. it does now, and the fix will be released next week. it was the same root cause as your other issue #4687 so i'm going to go ahead and close this one as a duplicate

[jchiem]

Hi, I believe this bug is still applicable but I'm unsure if it's dependent on the complexity of the dependencies.

I've set up a public repository which provides more details about still being able to reproduce this in versions above [email protected].

(I can also open a new issue if that is the preferred method of doing things, when I was reading the CONTRIBUTING.md it wasnt clear if you prefer to have people comment in the "duplicate issue", or if "once an issue is closed" people should instead start a new issue and just list the duplicate issue.)

Let me know if you'd rather have me open a new issue, or if you prefer re-opening this one.