Support a "default" option for build.jobs

[do-not-use-parker-timmerman]

Problem

There is no way to "unset"/reset to "default, the number of build jobs specified to Cargo.

Consider your team has a .cargo/config.toml located in the root of your repository. You want to set the following:

[build]
jobs = -1

So that on all developer's machines, one core will be left available when running cargo for your computer to do other tasks.

But your CI infrastructure also runs cargo, and you want CI to use all possible cores to makes builds as fast as possible. You can set the CARGO_BUILD_JOBS environment variable, but via the env variable (or --jobs cli option) but there is no way to override the number of jobs back to "default"/num cpus.

Proposed Solution

A solution is to define a new value that represents the default/num_cpus state. Two proposals would be:

1. Reinterpret 0 jobs as meaning "default"
   a. Today specifying 0 as the number of jobs returns an error, so changing this shouldn't(?) break any flows
2. Add a new supported value like "d", "default", or "num_cpus" that when specified we use the default value

Personally I'd prefer option 1 as it seems like the easiest, but I'm not sure if there are any flows that rely on cargo returning an error.

Notes

No response

[weihanglo]

Some previous works:

• Specifying jobs = 0 in .cargo/config hangs cargo forever without output #9219
• Allow a negative number of jobs to count from maximum number of cores #9217 (comment)
• Support for negative --jobs parameter, counting backwards from max CPUs #10844

I am okay with interpreting 0 as default (all possible parallelism based on environment and jobserver). I wonder what others' opinions are.

By the way, .cargo/config.toml is considered as a set of settings for environment. If a config is not intended to be shared, probably just remove it from the file.

[charmitro]

Is this still relevant? If yes, I’d like to give it a try.

[weihanglo]

Thank you for asking! Personally, I am happy with the idea, but the team has yet discussed it. Once it is generally accepted you will see it labeled as S-accepted Status: Issue or feature is accepted, and has a team member available to help mentor or review .

BTW, I'd also like to learn prior arts from other tools. How does make handle --jobs 0? What about other build tools like bazel, buck or else? It would be helpful if someone can collect the data.

[charmitro]

BTW, I'd also like to learn prior arts from other tools. How does make handle --jobs 0? What about other build tools like bazel, buck or else? It would be helpful if someone can collect the data.

Ack.
I can try to collect data from other tools, especially the famous ones.

[charmitro]

I collected information from make, bazel & cmake.

In all three, we observe the same logic, i.e. they require a positive integer argument.

For reference,

• make: the '-j' option requires a positive integer argument
• Basel: ERROR: While parsing option --jobs=0: Value '(0)' must be at least 1.
• cmake: The value requires a positive integer argument.

[ehuss]

I have a slight concern that 0 might be somewhat ambiguous. For example, ninja treats it as "infinity".

Some other data:

• msbuild: maxCpuCount without a value is number of processors.
• buck2: 0 is number of cpus.
• buck: has 5 different options, like thread_core_ratio_reserved_cores
• meson: Less than 1 means to leave the decision up to the build tool (which is often number of cpus).
• gradle: Looks like org.gradle.workers.max does not support it.
• go: -p must be a positive integer
• swift: -j must be a positive integer

Would there be any objections to using an english string to represent the "number of cpus" or some other meaning? Using 0 probably isn't so much of a leap from our use of negative numbers (and has some precedence), but a short string might be a little clearer.

[weihanglo]

Second it. A simple English word is probably a clearer choice. This also enables some future extensions like simple math expressions (see #9221 (comment)).

I propose the short word to be CPUS. However, I see Bazel also has a --local_cpu_resources which chooses HOST_CPUS and accepts expressions. Not sure if Cargo will support remote build or distributed build in the future, but HOST_CPUS does sound reasonable in terms of future compatibility.

[weihanglo]

Let's put it this way. We could choose CPUS. The semantics is “all available parallelism in the system”, equivalent to std::thread::available_parallelism.

If we do have distributed build support in the future, I don't think we will need a breaking change. CPUS can still mean “all available parallelism in the system”. Whether the system refers to host or the entire distributed system is something we can define later.

[charmitro]

I also agree, having something like CPUS or ALL that is interpreted into std::thread::available_parallelism can make this configuration variable look meaningful.

[weihanglo]

I'd like to accept this feature request. Let's do an FCP.

@rfcbot FCP merge

Currently there is no way to unset a config value, and this proposes using a simple English word to unset previously set build.job back to the default: use all available parallelism on the system.

Without this feature, it is impossible to unset build.job unless you edit the config file where it is defined.

For bike shedding the short word, we can decide later. The implementation side can start earlier once this feature is accepted.

[weihanglo]

all CAPS doesn't work?

@rfcbot fcp merge

[rfcbot]

Team member @weihanglo has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Eh2406
• @Muscraft
• @arlosi
• @ehuss
• @epage
• @joshtriplett
• @weihanglo

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[epage]

I lean towards "default"

0 seems reasonable with us having -1 but that can be added later

[weihanglo]

The Cargo team discussed this in the meeting today. We choose default as the magic word to unset to the default parallelism. @charmitro if you are still interested in helping with this, feel free to send a pull request :)

[charmitro]

The Cargo team discussed this in the meeting today. We choose default as the magic word to unset to the default parallelism. @charmitro if you are still interested in helping with this, feel free to send a pull request :)

Still interested, thanks for the update!

@rustbot claim

[charmitro]

Should this be implemented only on both CLI(--jobs) and config([build]) or only on the CLI?

[weihanglo]

I think the answer is yes. The two values are merged here. You might like to change i32 to an enum here.

[ParkMyCar]

Wooo! Glad to see this making progress, thanks @charmitro for putting up a PR! :)

Side note: I opened this ticket originally with a different GitHub account, but have since merged everything into this one. To dissuade folks from tagging the old one, I renamed it, hence the 'do-not-use-parker-timmerman', sorry for any confusion!

[charmitro]

Thanks for the heads up, noted!