Skip to content

Commit

Permalink
fix(dep-graph): set the number of workers with environment variables
Browse files Browse the repository at this point in the history
In a docker container, the result of os.cpus() can be wrong (nodejs/node#28762).
  • Loading branch information
danielpayetdev authored and vsavkin committed Jan 28, 2022
1 parent 6dde019 commit b86feb1
Show file tree
Hide file tree
Showing 2 changed files with 30 additions and 23 deletions.
45 changes: 24 additions & 21 deletions nx-dev/nx-dev/public/documentation/shared/using-nx/nx-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
The Nx CLI isn't like most command lines that accomplishes a predefined task. Nx can be configured to work with
different tools and even different languages.

Nx allows you to break up your codebase into different **projects**. The Nx CLI provides commands to operate and manage the different parts of the codebase. These commands fall into three categories:
Nx allows you to break up your codebase into different **projects**. The Nx CLI provides commands to operate and manage
the different parts of the codebase. These commands fall into three categories:

- Acting on code (Build, Test, Serve)
- Modifying code
Expand All @@ -27,9 +28,9 @@ nx build my-js-app
However, `nx build` is only an abstraction over what it means to "build" projects rather than tied to a certain
implementation. For instance, if you have a `project.json` file defining `build` using
an **[executor](/executors/using-builders)**, that executor will be invoked. If you don't specify an
executor for the build target, `nx build my-react-app` will invoke the `build` npm script in the project's folder. Every
argument you pass into `run` will be forwarded to the executor or the npm script.
an **[executor](/executors/using-builders)**, that executor will be invoked. If you don't specify an executor for the
build target, `nx build my-react-app` will invoke the `build` npm script in the project's folder. Every argument you
pass into `run` will be forwarded to the executor or the npm script.

Along with running a target on a single project, Nx provides some commands to run the same target across multiple
projects.
Expand All @@ -41,8 +42,8 @@ nx run-many --target=build --projects=app1,app2
nx run-many --target=test --all # Runs all projects that have a test target, use this sparingly.
```

The [`nx affected` command](/cli/affected) isolates the set projects that may have changed in behavior and
runs a target across them. This is more efficient than running all projects every time.
The [`nx affected` command](/cli/affected) isolates the set projects that may have changed in behavior and runs a target
across them. This is more efficient than running all projects every time.

```bash
nx affected --target=build
Expand All @@ -59,22 +60,21 @@ nx generate @nrwl/react:storybook-configuration shared-button # Configures story
```

Again, like `nx run`, `nx generate` is only an abstraction over generating code. `nx generate` can generate anything you
want via **generators**. **[Generators](/generators/using-schematics)** can be installed as part of a
plugin or developed locally within an Nx workspace to fit the needs of your organization.
want via **generators**. **[Generators](/generators/using-schematics)** can be installed as part of a plugin or
developed locally within an Nx workspace to fit the needs of your organization.

A [workspace generator](/generators/workspace-generators) is a custom generator for your
workspace. `nx generate workspace-generator my-generator` generates a workspace generator which can be run with
the [`nx workspace-generator` command](/cli/workspace-generator). This can be useful to allow your
organization to consistently generate projects according to your own standards.
the [`nx workspace-generator` command](/cli/workspace-generator). This can be useful to allow your organization to
consistently generate projects according to your own standards.

```bash
nx workspace-generator my-generator
```

Upgrading a package is not always as simple as bumping the version in `package.json`.
The [`nx migrate` command](/cli/migrate) facilitates not only managing package versions but also runs
migrations specified by package maintainers. See
the [full guide to updating Nx](/using-nx/updating-nx).
The [`nx migrate` command](/cli/migrate) facilitates not only managing package versions but also runs migrations
specified by package maintainers. See the [full guide to updating Nx](/using-nx/updating-nx).

```bash
nx migrate latest # Updates the version of Nx in `package.json` and schedules migrations to be run
Expand All @@ -84,23 +84,23 @@ nx migrate --run-migrations # Runs the migrations scheduled by the previous comm
## Understanding the codebase

Nx creates and maintains a project graph between projects based on import statements in your code and uses that
information to run executors only on the [affected](/cli/affected) projects in a codebase. A visual
version of the [project project graph](/structure/dependency-graph) is also available to help developers
understand the architecture of the codebase.
information to run executors only on the [affected](/cli/affected) projects in a codebase. A visual version of
the [project project graph](/structure/dependency-graph) is also available to help developers understand the
architecture of the codebase.

The [`nx graph` command](/cli/dep-graph) displays this project graph in a web browser for you to
explore.
The [`nx graph` command](/cli/dep-graph) displays this project graph in a web browser for you to explore.

Note: In older versions of Nx, the project graph was launched with `nx dep-graph`. For backward compatibility, that command is aliased to `nx graph`.
Note: In older versions of Nx, the project graph was launched with `nx dep-graph`. For backward compatibility, that
command is aliased to `nx graph`.

```bash
nx graph
nx graph --watch # Updates the browser as code is changed
nx affected:graph # Highlights projects which may have changed in behavior
```

The [`nx list` command](/cli/list) lists the currently installed Nx plugins and other available plugins.
The `nx list` command can list the generators and executors that are available for a plugin.
The [`nx list` command](/cli/list) lists the currently installed Nx plugins and other available plugins. The `nx list`
command can list the generators and executors that are available for a plugin.

**List installed plugins:**

Expand All @@ -115,5 +115,8 @@ There are some environment variables that you can set to log additional informat

- Setting **NX_VERBOSE_LOGGING=true** will print debug information useful for troubleshooting.
- Setting **NX_PERF_LOGGING=true** will print debug information useful for profiling executors and Nx itself.
- Setting **NX_PROFILE=myfile.json** will output a profile that you can view in Chrome Dev Tools. Read more [here](https://nx.dev/guides/performance-profiling).
- Setting **NX_PROJECT_GRAPH_MAX_WORKERS=3** will tell Nx to use at most 3 workers to compute its project graph (in some
CI setups the default value can be too high and can result in a SIGKILL).
- Setting **NX_TASKS_RUNNER_DYNAMIC_OUTPUT=false** will use non-dynamic terminal output strategy (what you see in CI),
even when you terminal can support the dynamic version.
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ function buildExplicitDependencies(
// files we need to process is >= 100 and there are more than 2 CPUs
// to be able to use at least 2 workers (1 worker per CPU and
// 1 CPU for the main thread)
if (totalNumOfFilesToProcess < 100 || os.cpus().length < 3) {
if (totalNumOfFilesToProcess < 100 || getNumberOfWorkers() <= 2) {
return buildExplicitDependenciesWithoutWorkers(
jsPluginConfig,
ctx,
Expand Down Expand Up @@ -294,7 +294,7 @@ function buildExplicitDependenciesUsingWorkers(
totalNumOfFilesToProcess: number,
builder: ProjectGraphBuilder
) {
const numberOfWorkers = os.cpus().length - 1;
const numberOfWorkers = getNumberOfWorkers();
const bins = splitFilesIntoBins(
ctx,
totalNumOfFilesToProcess,
Expand Down Expand Up @@ -344,6 +344,10 @@ function buildExplicitDependenciesUsingWorkers(
});
}

function getNumberOfWorkers(): number {
return +process.env.NX_PROJECT_GRAPH_MAX_WORKERS ?? os.cpus().length - 1;
}

function createContext(
workspaceJson: WorkspaceJsonConfiguration,
nxJson: NxJsonConfiguration,
Expand Down

1 comment on commit b86feb1

@vercel
Copy link

@vercel vercel bot commented on b86feb1 Jan 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.