Discussion: Setting up ts_project with custom path mapping in a monorepo

[lencioni]

I've been working to set up Bazel with ts_project in a large, custom monorepo (i.e. not using something like Lerna or yarn workspaces). We recently resolved a challenge and I'd like to publicly document it here in case it helps someone else (or if someone sees a better solution for us). Feel free to close this issue.

Our monorepo is set up so that we have a "frontend" directory with a lot of projects as subdirectories under "frontend". The structure is roughly like this:

frontend/
frontend/project-a/
frontend/project-b/
frontend/project-c/
...

Imports inside the same project use relative paths:

import foo from './foo';

And imports from other projects use custom colon-import paths:

import someOtherProject from ':some-other-project';
import anotherThing from ':a-different-project/anotherThing';

We've been running TypeScript without Bazel for a while, and had these custom imports wired up via the following path mapping in our base tsconfig:

{
  "compilerOptions": {
    "paths": {
      ":*": [
        "./frontend/*",
      ],
    }
  }
}

When setting up Bazel, we initially expanded this path mapping to look more like this:

{
  "compilerOptions": {
    "paths": {
      ":*": [
        "./frontend/*",

        // Bazel generates the .d.ts files in the bazel-out directory, so we
        // need to tell TypeScript that it might be able to find them there.
        //
        // This hack was inspired by formatjs:
        // https://github.com/formatjs/formatjs/blob/268d6aef6fc08180a0be691b631eb1cdfb68ca31/tsconfig.json#L19-L41
        // And the rootDirs note in the docs here:
        // https://bazelbuild.github.io/rules_nodejs/TypeScript.html#ts_project
        "./bazel-out/darwin-fastbuild/bin/frontend/*",
        "./bazel-out/k8-fastbuild/bin/frontend/*",
        "./bazel-out/x64_windows-fastbuild/bin/frontend/*",
        "./bazel-out/darwin-dbg/bin/frontend/*",
        "./bazel-out/k8-dbg/bin/frontend/*",
        "./bazel-out/x64_windows-dbg/bin/frontend/*",
      ],
    }
  }
}

This seemed to work okay for some of our projects, however, once we started getting into more complex projects that depended on types that were built from a different project, we started seeing errors in our Bazel ts_project output like this:

frontend/complex-project/foo/bar.ts(27,16): error TS2571: Object is of type 'unknown'.

It seemed that these types ended up as unknown instead of the actual types we expected.

We looked at the .d.ts files that ts_project built for these types, and found code that started like this:

import Foo from ':some-dependency/foo/bar';
// ...

Was built into this:

// In a file in frontend/complex-project
// ...
export declare const FooType: import("../../bazel-out/darwin-fastbuild/bin/frontend/some-dependency/foo/bar").Foo<Bar>;
// ...

We expected the import path to look something like "../../some-dependency/foo/bar" but instead got "../../bazel-out/darwin-fastbuild/bin/frontend/some-dependency/foo/bar", which doesn't really exist. As a result, TypeScript infers it as any which probably gets converted to an unknown somewhere along the line via the generics used here.

After a bit of messing around, we believe that we were able to arrive at an okay solution: add rootDirs and an extra paths mapping like this:

{
  "compilerOptions": {
    "rootDirs": [
      "../..",
      // Ensure that the relative paths in types produced by bazel get turned into
      // "frontend/" paths.
      "../../bazel-out/darwin-fastbuild/bin",
      "../../bazel-out/k8-fastbuild/bin",
      "../../bazel-out/x64_windows-fastbuild/bin",
      "../../bazel-out/darwin-dbg/bin",
      "../../bazel-out/k8-dbg/bin",
      "../../bazel-out/x64_windows-dbg/bin",
    ],

    "paths": {
      ":*": [
        "./frontend/*",

        // Bazel generates the .d.ts files in the bazel-out directory, so we
        // need to tell TypeScript that it might be able to find them there.
        //
        // This hack was inspired by formatjs:
        // https://github.com/formatjs/formatjs/blob/268d6aef6fc08180a0be691b631eb1cdfb68ca31/tsconfig.json#L19-L41
        // And the rootDirs note in the docs here:
        // https://bazelbuild.github.io/rules_nodejs/TypeScript.html#ts_project
        "./bazel-out/darwin-fastbuild/bin/frontend/*",
        "./bazel-out/k8-fastbuild/bin/frontend/*",
        "./bazel-out/x64_windows-fastbuild/bin/frontend/*",
        "./bazel-out/darwin-dbg/bin/frontend/*",
        "./bazel-out/k8-dbg/bin/frontend/*",
        "./bazel-out/x64_windows-dbg/bin/frontend/*",
      ],

      // WARNING: Don't use frontend/ imports, use colon instead. This is for bazel build
      // type resolution: The rootDirs above produce "frontend/" paths in the .d.ts files
      // produced by bazel, and then we remap those paths back into bazel-out here.
      // Note this does mean you can never install a package named "frontend", but that's
      // probably fine since this package is unlikely to be something we'd install:
      // https://www.npmjs.com/package/frontend
      "frontend/*": [
        "./bazel-out/darwin-fastbuild/bin/frontend/*",
        "./bazel-out/k8-fastbuild/bin/frontend/*",
        "./bazel-out/x64_windows-fastbuild/bin/frontend/*",
        "./bazel-out/darwin-dbg/bin/frontend/*",
        "./bazel-out/k8-dbg/bin/frontend/*",
        "./bazel-out/x64_windows-dbg/bin/frontend/*",
      ],
    }
  }
}

This configuration causes our special colon imports to be converted from ":some-project" to "frontend/some-project", which then works because of the second "frontend/*" path mapping.

The main caveat that we've thought of with this approach is that it means we can't just install and use a package named "frontend", but that seems unlikely to cause us any problems.

[mgenov]

@lencioni could you provide more details where tsconfig.json files are defined?

[lencioni]

Sure thing @mgenov!

For the most part, we currently have a tsconfig.json file in each of our frontend projects. So with the structure I mentioned above, that would be:

frontend/project-a/tsconfig.json
frontend/project-b/tsconfig.json
frontend/project-c/tsconfig.json

We are using project references, and each of these tsconfig.json files extend from a more centralized config in one of our projects that has our global settings in it like the rootDirs and paths stuff I mentioned above. Here's an example of one of these project tsconfig.json files:

{
  "extends": "../typescript/tsconfig.dts.json",
  "include": [".", "./**/*.json"],

  "compilerOptions": {
    "types": ["jest", "node"]
  },

  "references": [
    {
      "path": "../project-b"
    }
  ]
}

We have created some internal tooling that scans the code in our repo for dependencies (currently using jest-haste-map) and keeps parts of these files like the references and compilerOptions.types up to date automatically.

In a few cases today we have some tsconfig.json files more deeply nested in projects. This is mostly to support things in our repo like cypress, which have some different global types that would conflict with other things.

frontend/project-a/tsconfig.json
frontend/project-b/tsconfig.json
frontend/project-b/cypress/tsconfig.json
frontend/project-c/tsconfig.json

We plan to add a lot more tsconfig.json files in our repo, such as in all of our test directories, so that we can improve TypeScript performance and avoid dragging in test-specific global types into production code, but first we have some things we need to solve, such as upgrading our tooling to manage the exclude field for these deeply nested tsconfig.json files. As we roll this out, I believe we will also have some cycles between our test directories that we will need to resolve as well.

[mgenov]

Thanks for your feedback @lencioni

I've made a showcase project to try to illustrate this usage but I encountered the following compilation error:

frontend/project-a/index.ts(1,24): error TS2307: Cannot find module ':project-b' or its corresponding type declarations.

when executing

bazel build //frontend/project-a:tsconfig

Could you take a look what could cause the error?

Code showcase is located at: https://github.com/mgenov/bazel-typescript-showcase

[lencioni]

I think the baseUrl needs to be a bit different in your repo. I opened a PR that I think fixes it: mgenov/bazel-typescript-showcase#1

[mgenov]

Yes, with this change everything is working as expected. Thanks!

[Zemnmez]

Despite reading all the available documentation I'm still unable to discern what exactly the correct way to do this is.

The documentation only refers to this '../..' stuff:

"compilerOptions": {
    "rootDirs": [
        ".",
        "../../bazel-out/host/bin/path/to",
        "../../bazel-out/darwin-fastbuild/bin/path/to",
        "../../bazel-out/k8-fastbuild/bin/path/to",
        "../../bazel-out/x64_windows-fastbuild/bin/path/to",
        "../../bazel-out/darwin-dbg/bin/path/to",
        "../../bazel-out/k8-dbg/bin/path/to",
        "../../bazel-out/x64_windows-dbg/bin/path/to",
    ]
}

But there's some reference here to a baseUrl above in this thread?

All I've managed to do is get relative imports like ../gen/core_en to work, and only during compilation, not in VSCode, which is left in the dark about the types here.

If the documentation is all to recommend using ts_project, it seems like it should be easier to use than ts_library and ts_binary, which, based on what I've read online it doesn't seem to be.

Can I get clarity on whether 'ts_project' is meant to usable at this point, and how exactly it can be made usable?

Edit: sorry for the tone here, I think I'm hitting a strange edge-case with tsconfig.json where VSCode is not picking it up for some reason.

[Zemnmez]

With my tsconfig.json and my WORKSPACE in the same folder, I can get the code building with this tsconfig.json:

{
      "extends": "@tsconfig/node14/tsconfig.json",
    // If building without sandboxing, we need to prevent TypeScript from descending into
    // Bazel's external folder which contains third-party Bazel dependencies.
    "exclude": ["external/*"],

    "compilerOptions": {
        "baseUrl": ".",
	"declaration": true,
        "preserveSymlinks": true,
        "paths": {
            "*": [
                "*",
                "bazel-out/x64_windows-fastbuild/bin/*",
                "bazel-out/x64_windows-dbg/bin/*",
                "bazel-out/k8-fastbuild/bin/*",
                "bazel-out/k8-dbg/bin/*",
                "bazel-out/host/bin/*",
                "bazel-out/darwin-fastbuild/bin/*",
                "bazel-out/darwin-dbg/bin/*",
            ]
        },
    }
}

-- with the caveat that for whatever reason this doesn't work for VSCode.

OK -- it looks like though it's compiling successfully, that doesn't mean you can actually run it -- it's failing at runtime, because the require() is failing

[Zemnmez]

Sorry for the spam. I've pushed the repo to https://github.com/Zemnmez/quickcult/tree/broken-monorepo as far as I've managed to get it. yarn bazel_run //solve:main should work but fails at runtime, and VSCode is unable to do code completions.

[bzitzow]

What was the solution for this?

[Nick-Mazuk]

@Zemnmez Did you ever manage to figure out VS Code completions?

[bhavitsharma]

@Nick-Mazuk, here's what I did to get auto-completions:
Added the path/root references as mentioned in the docs. I also symlinked node_modules in WORKSPACE root directory like this, so vscode can find all the respective npm packages and things like eslint, prettier can work:

ln -s /actual/path/to/node_modules node_modules

You can find the actual path to node_modules by doing - $(bazel info output_base)/external/npm/node_modules

[willjschmitt]

The solutions here worked well before transitions were introduced, but when a rule includes a transition, a generated suffix is added to the output path, such as bazel-out/darwin-fastbuild-ST-4a519fd6d3e4 instead of simply bazel-out/darwin-fastbuild.

We encountered this with rules_docker introducing transitions on container_image in v0.23.0. We aren't benefiting from the transition introduced there, so we are solving this by noop-ing the transition to not introduce an output directory change, for now, but I can imagine where the wider use of platforms and transitions could cause the workaround with hardcoded output directories to become more fragile

[willjschmitt]

Maybe not super relevant to this discussion, but if others face this type of issue, the way we noop'ed the transition for container_image specifically is:

load("@io_bazel_rules_docker//container:container.bzl", _container = "container")

def _noop_image_transition_impl(settings, attr):
    return settings

_noop_image_transition = transition(
    implementation = _noop_image_transition_impl,
    inputs = [],
    outputs = [],
)

noop_image_transition_container_image = rule(
    attrs = _container.image.attrs,
    executable = True,
    outputs = _container.image.outputs,
    toolchains = ["@io_bazel_rules_docker//toolchains/docker:toolchain_type"],
    implementation = _container.image.implementation,
    cfg = _noop_image_transition,
)

I imagine this could be simpler, but I'm not super familiar with transitions yet.