Upstream file not correctly recognized as output of other project when using path mappings

[TimUnderhay]

🔎 Search Terms

"typescript not compiling single file unless in file"

🕗 Version & Regression Information

• This is the behavior in every version I tried, and I reviewed the FAQ for entries about building

Discovered / tested with 5.4.5.

⏯ Playground Link

https://github.com/TimUnderhay/ts-build-references-bug

💻 Code

This is a snippet from the minimal repro repo I've created (https://github.com/TimUnderhay/ts-build-references-bug). The readme contains relevant code / reproduction details. This really can't be condensed into a snippet or playground link, as far as I can tell.

This is a reproduction of issue #44845. When using references within a monorepo to build an app that depends on a shared library reference, certain files get omitted from the tsc output for no evident reason.

The repo

Under projects/, there are two TS 'references', one a shared lib -- 'shared', and an app called 'server'. Server contains a reference to the shared lib, and also a path alias called :shared, pointing to the same place.

projects/server/tsconfig.json:

{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "baseUrl": "./src",
    "rootDir": "..",
    "lib": ["ES2022"],
    "outDir": "./dist",
    "paths": {
      ":shared/*": [
        "../../shared/src/*"
      ]
    },
    "types": [
      "node"
    ]
  },
  "include": [
    "src/**/*.ts",
    "../shared/src/**/*.ts"
  ],
  "references": [
    {
      "path": "../shared"
    }
  ]
}

Steps to reproduce:

• From repo root, run npm install.
• cd projects/server
• npm run build (is just rm -rf dist/ && tsc --build -v)
• Check projects/server/dist/shared/src output. Notice that only the shared logging module was built. 'errors.ts' and 'global-type-overrides.ts' should be there based on the include pattern, but they aren't.

🙁 Actual behavior

Check projects/server/dist/shared/src output. Notice that only the shared logging module was built. 'errors.ts' and 'global-type-overrides.ts' should be there based on the include pattern, but they aren't.

🙂 Expected behavior

All modules derived from globs specified in include should be built.

Additional information about the issue

If one builds projects/shared on its own, all output files are present and accounted for in projects/shared/dist. The issue appears to be related to references. Building projects/server with eithertsc or tsc --build -v appear to yield the same end result -- missing files under projects/server/dist/shared.

It Gets Even Weirder

In projects/server/src/server.ts, if you uncomment the line // import { log } from ':shared/logging.js';, the entire shared subdir gets omitted from projects/server/dist!!

Just. Shouldn't. Happen.

[TimUnderhay]

This very much appears to be a references bug. If one comments out references in the server tsconfig.json, the issue goes away, and all expected files are compiled, but obviously the benefits of references are then negated.

{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "baseUrl": "./src",
    "rootDir": "..",
    "lib": ["ES2022"],
    "outDir": "./dist",
    "paths": {
      ":shared/*": [
        "../../shared/src/*"
      ]
    },
    "types": [
      "node"
    ]
  },
  "include": [
    "src/**/*.ts",
    "../shared/src/**/*.ts"
  ],
  // "references": [
  //   {
  //     "path": "../shared"
  //   }
  // ]
}

[RyanCavanaugh]

This is exactly what project references is for - treating another config's files as "not part of your project" and thus not built. If the non-reference behavior is the behavior you want, then simply remove the reference.

[TimUnderhay]

@RyanCavanaugh the documentation at https://www.typescriptlang.org/docs/handbook/project-references.html says:

“Build mode (see below) will automatically build the referenced project if needed

By separating into multiple projects, you can greatly improve the speed of typechecking and compiling, reduce memory usage when using an editor, and improve enforcement of the logical groupings of your program.“

“Additionally, to preserve compatibility with existing build workflows, tsc will not automatically build dependencies unless invoked with the --build switch”

“Running tsc --build (tsc -b for short) will do the following:

Find all referenced projects
Detect if they are up-to-date
Build out-of-date projects in the correct order”

This all seems to suggest that building is very much an expected part of references. Why would it build some files but not others?

[RyanCavanaugh]

The expected setup for project references is that each input file corresponds to exactly one output file

[TimUnderhay]

Respectfully, I don’t think that jives with what I pasted from the documentation. It also doesn’t answer the question of why it sometimes builds shared files but other times it doesn’t.

What then is the point of references if it doesn’t actually build the references? Everything written seems to point to references as building referenced projects, but it seems you’re saying that isn’t the case.

[RyanCavanaugh]

I don't know what to tell you; this feature is widely adopted, has worked this way since it shipped six years ago, and you're the first person to raise this scenario or interpret the docs that way. The scenario where the same input file ends up as multiple output files isn't supported, and having the same input file not appear in multiple outputs was a motivating scenario for why references were added in the first place.

[TimUnderhay]

I think you’ve misunderstood what I’ve written. I’m saying that files specifically included in compilation by virtue of the includes field aren’t being compiled. I’m not saying that a single input file should produce multiple output files.

Help me to understand.

[RyanCavanaugh]

Sure.

shared builds shared/src/errors.ts to shared/dist/src/errors.js. The is the first output

server also includes this file, but it's recognized as an input of another project, so it's not built. This would have been the second output of shared/src/errors.ts

This is on purpose; tsc --explainFiles talks about it:

../shared/dist/src/global-type-overrides.d.ts
  Imported via ':shared/global-type-overrides.js' from file 'src/server.ts'
  Matched by include pattern '../shared/src/**/*.ts' in 'tsconfig.json'
  File is output of project reference source '../shared/src/global-type-overrides.ts'
  File is ECMAScript module because '../shared/package.json' has field "type" with value "module"
../shared/dist/src/errors.d.ts
  Imported via ':shared/errors.js' from file 'src/server.ts'
  Matched by include pattern '../shared/src/**/*.ts' in 'tsconfig.json'
  File is output of project reference source '../shared/src/errors.ts'
  File is ECMAScript module because '../shared/package.json' has field "type" with value "module"
src/server.ts
  Matched by include pattern 'src/**/*.ts' in 'tsconfig.json'
  File is ECMAScript module because 'package.json' has field "type" with value "module"
../shared/src/logging.ts
  Matched by include pattern '../shared/src/**/*.ts' in 'tsconfig.json'
  File is ECMAScript module because '../shared/package.json' has field "type" with value "module"

There actually is a bug here, which is that logging.ts should have been identified as an output of the referenced project, but wasn't. The emit of server/dist/shared/src/logging.js is a bug; it shouldn't have been produced

[TimUnderhay]

Thank you for helping me understand, @RyanCavanaugh.

[runspired]

I don't know what to tell you; this feature is widely adopted, has worked this way since it shipped six years ago, and you're the first person to raise this scenario or interpret the docs that way.

I've also interpreted the docs to mean that tsc -b will walk the graph and build referenced projects if they are not yet built already. It is very clearly worded as doing so in the docs, and until recently it seemed it was doing so. Now however, I'm finding it often silently does not, even with `--force.

Moreover, --trace-resolution shows that even with references, tsc is attempting to resolve files not via the reference paths but via node_modules ... which doesn't work for projects that use hardlinks.

I feel as though the solution here is to use both paths and references and pray for the best on the build ... the validity of which is unclear from the docs.

[sheetalkamat]

I am going to assign this issue to myself since this is related to work i am doing in #56254 as well as the fix is more involved from build info and --build perspective than a single line change