Skip to content

Commit

Permalink
Auto merge of #10737 - ehuss:revert-num-cpus, r=weihanglo
Browse files Browse the repository at this point in the history
[beta] Revert #10427: switch from num_cpus

This temporarily reverts #10427 (Use available_parallelism instead of num_cpus) per the discussion at rust-lang/rust#97549. `available_parallelism` does not handle cgroups v1 on Linux unlike num_cpus. I am concerned that this potentially affects a significant percentage of users. For example, Docker just added cgroups v2 support last year. Various Linux distributions have only recently switched to it as the default. The following is what I can find on the web:

* Fedora (since 31)
* Arch Linux (since April 2021)
* openSUSE Tumbleweed (since c. 2021)
* Debian GNU/Linux (since 11)
* Ubuntu (since 21.10)
* RHEL and RHEL-like distributions (since 9)

This also appears to affect CircleCI.

The consequence is that Cargo ends up using too much parallelism and can run out of memory.

I'm not sure what to do about 1.63.  If std adds support for cgroups v1, then I don't think there is anything to do there. Otherwise I think we should revert similarly if that doesn't happen.
  • Loading branch information
bors committed Jun 8, 2022
2 parents 4751950 + 0cfdbc0 commit a748cf5
Show file tree
Hide file tree
Showing 3 changed files with 4 additions and 13 deletions.
1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ libc = "0.2"
log = "0.4.6"
libgit2-sys = "0.13.2"
memchr = "2.1.3"
num_cpus = "1.0"
opener = "0.5"
os_info = "3.0.7"
pathdiff = "0.2"
Expand Down
10 changes: 2 additions & 8 deletions src/cargo/core/compiler/build_config.rs
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
use crate::core::compiler::CompileKind;
use crate::util::interning::InternedString;
use crate::util::{CargoResult, Config, RustfixDiagnosticServer};
use anyhow::{bail, Context as _};
use anyhow::bail;
use cargo_util::ProcessBuilder;
use serde::ser;
use std::cell::RefCell;
use std::path::PathBuf;
use std::thread::available_parallelism;

/// Configuration information for a rustc build.
#[derive(Debug)]
Expand Down Expand Up @@ -74,12 +73,7 @@ impl BuildConfig {
its environment, ignoring the `-j` parameter",
)?;
}
let jobs = match jobs.or(cfg.jobs) {
Some(j) => j,
None => available_parallelism()
.context("failed to determine the amount of parallelism available")?
.get() as u32,
};
let jobs = jobs.or(cfg.jobs).unwrap_or(::num_cpus::get() as u32);
if jobs == 0 {
anyhow::bail!("jobs may not be 0");
}
Expand Down
6 changes: 1 addition & 5 deletions src/cargo/core/compiler/timings.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ use anyhow::Context as _;
use cargo_util::paths;
use std::collections::HashMap;
use std::io::{BufWriter, Write};
use std::thread::available_parallelism;
use std::time::{Duration, Instant, SystemTime};

pub struct Timings<'cfg> {
Expand Down Expand Up @@ -381,9 +380,6 @@ impl<'cfg> Timings<'cfg> {
};
let total_time = format!("{:.1}s{}", duration, time_human);
let max_concurrency = self.concurrency.iter().map(|c| c.active).max().unwrap();
let num_cpus = available_parallelism()
.map(|x| x.get().to_string())
.unwrap_or_else(|_| "n/a".into());
let max_rustc_concurrency = self
.concurrency
.iter()
Expand Down Expand Up @@ -446,7 +442,7 @@ impl<'cfg> Timings<'cfg> {
self.total_fresh + self.total_dirty,
max_concurrency,
bcx.build_config.jobs,
num_cpus,
num_cpus::get(),
self.start_str,
total_time,
rustc_info,
Expand Down

0 comments on commit a748cf5

Please sign in to comment.