Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow a negative number of jobs to count from maximum number of cores #9217

Closed
orlp opened this issue Mar 1, 2021 · 4 comments · Fixed by #10844
Closed

Allow a negative number of jobs to count from maximum number of cores #9217

orlp opened this issue Mar 1, 2021 · 4 comments · Fixed by #10844
Labels
A-documenting-cargo-itself Area: Cargo's documentation A-jobserver Area: jobserver, concurrency, parallelism C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` Performance Gotta go fast!

Comments

@orlp
Copy link
Contributor

orlp commented Mar 1, 2021

If you want your machine to remain usable during a compile session usually you want to set the number of jobs
lower than your maximum number of cores. But this varies heavily from machine to machine, and makes it
annoying to automate in bigger build scripts.

I would suggest extending the --jobs (or -j) parameter to negative numbers, where -j -1 would be 'all but one core',
-j -2 would be 'all but two cores', etc. This makes it really easy for scripts to maintain a usable machine and scale
to the number of cores available without having to look up the actual number of cores.

For sanity and convenience of automatic build scripts I would suggest that if -j was sufficiently negative to specify no or a negative amount of cores, to clamp the number of cores to at least 1, instead of giving an error.

So to be precise I would suggest changing jobs to i32 instead of u32 in BuildConfig::new (and wherever necessary to make that work) and replace this line with

let jobs = match jobs.or(cfg.jobs) {
    None => ::num_cpus::get() as u32,
    Some(j) if j < 0 => (::num_cpus::get() as i32 + j).max(1) as u32,
    Some(j) => j as u32,
};

I'm neutral on whether 0 should still not be allowed or whether that should be mapped to ::num_cpus::get().

@orlp orlp added the C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` label Mar 1, 2021
@orlp
Copy link
Contributor Author

orlp commented Mar 1, 2021

Testing my pull request for this made me uncover this issue: #9219.

@orlp
Copy link
Contributor Author

orlp commented Mar 1, 2021

Not covered in my pull request:

  • A fix for the aforementioned issue for jobs = 0 in the .cargo/config.
  • Documentation for the new parameter behavior - I already felt the commandline docs were quite long. I don't know the best way to document this.
  • A test such that if you specify e.g. --jobs -2 that cargo does in fact use N-2 parallel jobs, where N is the maximum number of CPUs. I feel this test would be quite involved and not worth the effort.

@ehuss ehuss added A-jobserver Area: jobserver, concurrency, parallelism Performance Gotta go fast! labels Mar 2, 2021
@Eh2406 Eh2406 added the A-documenting-cargo-itself Area: Cargo's documentation label May 27, 2021
@epage
Copy link
Contributor

epage commented Jul 11, 2022

Acceptance summary: this was discussed in the now-closed #9221

  • While there is precedence for 0 being "all`, there isn't precedence for this
  • alexcrichton was fine being a trailblazer with this and a FCP was started with a documentation concern being raised
  • dtolnay did express interest in a more complex expression language for this. This hasn't been discussed further
  • The original PR was closed due to inactivity

@the8472
Copy link
Member

the8472 commented Jul 26, 2022

If you want your machine to remain usable during a compile session usually you want to set the number of jobs
lower than your maximum number of cores.

Wouldn't running it under nice -n 19 cargo [...] or schedtool -B -e cargo [...] be a better approach? It would lead to better CPU utilization while still leaving foreground processes responsive.

@bors bors closed this as completed in 223e84a Aug 1, 2022
sunshowers pushed a commit to nextest-rs/nextest that referenced this issue Nov 15, 2024
cargo --jobs supports negative values:
rust-lang/cargo#9217

currently this fails:

```
cargo nextest run --build-jobs -1                                                                                                                                                                                                   
error: unexpected argument '-1' found

  tip: to pass '-1' as a value, use '-- -1'
```

because clap doesn't support this by default and requires
`allow_hyphen_values = true`

clap-rs/clap#1245
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-documenting-cargo-itself Area: Cargo's documentation A-jobserver Area: jobserver, concurrency, parallelism C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` Performance Gotta go fast!
Projects
None yet
5 participants