Inconsistent coverage levels between runs with "all"

[jameswilddev]

Link to bug demonstration repository

• npm install
• npm run-script oneOff
  https://github.com/jameswilddev/nyc-repro

I've seen a few other issues reporting similar symptoms, but their fixes are things I'm already doing or are not applicable (different tools): #981

Expected Behavior

Repeated runs produce the same result (which is expected to be 100% here).

Observed Behavior

Repeated runs produce 92% or 100% coverage.

Troubleshooting steps

• still occurrring when I put cache: false in my nyc config

Environment Information

System:
    OS: Linux 4.18 Ubuntu 18.04.2 LTS (Bionic Beaver)
    CPU: (4) x64 Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
    Memory: 557.58 MB / 7.64 GB
  Binaries:
    Node: 11.15.0 - /usr/bin/node
    npm: 6.7.0 - /usr/bin/npm
  npmPackages:
    nyc: ^14.1.1 => 14.1.1 
    ts-node: ^8.1.0 => 8.1.0 
    typescript: ^3.4.5 => 3.4.5 

[jameswilddev]

Just compared a successful and failed run's json reporter output; the files are identical, but the coverage values in the terminal output still differ.

[jameswilddev]

Also compared the build output (/dist) of a successful and unsuccessful build; they are again identical.

[jameswilddev]

I can get 10 consistent passes by changing all to false (NYC config at the bottom of package.json). It seems whatever the bug is, it can be disabled that way. However, I then don't get coverage failures for modules with no coverage at all (as expected).

[elliot-nelson]

Very interesting!

The issue here appears to be that istanbul is attempting to instrument both your original source typescript file, and your output dist/ javascript version. Whichever "wins" determines the file coverage object being populated. That's a problem because if the wrong one wins, the coverage lines don't match up (one file has ~33 lines, the other has ~35 lines).

For a reporter like text, this ends up just showing incorrect coverage %; for a reporter like html, it actually blows up when it tries to show the report because the annotator can't read the lines that should exist.

I think this might be a configuration conflict between your tsconfig and jasmine.json config, though. Either your nyc+jasmine runtime should be in ts-node, and target only .ts files (no pre-build required); or, your nyc+jasmine should target only .js files, and have no idea that typescript exists at all (not ideal, as you would be losing the TS->JS source mapping). Something is not right if istanbul knows about both the .ts and .js files.

[jameswilddev]

That makes sense, and explains why all would trigger it if it's actively looking for files to include and picking up ones it shouldn't. Going to have to play and see if I can fix it my end but doubtful.

[bcoe]

@elliot-nelson great digging. I've seen this issue for several years, and you're correct it seems to be a race condition between which JSON file is loaded from .nyc_output first.

I would love to better document how to avoid this problem, as it's a frequent complaint for TypeScript users. If TypeScript is being built before running tests, It might also be worth looking at c8 which I've been finding works a bit better for my TypeScript libraries (the disclaimer being it doesn't work if TypeScript is being dynamically built in tests).