Use cgroup quotas for calculating available_parallelism

[the8472]

Automated tests for this are possible but would require a bunch of assumptions. It requires root + a recent kernel, systemd and maybe docker. And even then it would need a helper binary since the test has to run in a separate process.

Limitations

• only supports cgroup v2 and assumes it's mounted under /sys/fs/cgroup
• procfs must be available
• the quota gets mixed into sched_getaffinity, so if the latter doesn't work then quota information gets ignored too

Manually tested via

// spawn a new cgroup scope for the current user
$ sudo systemd-run -p CPUQuota="300%" --uid=$(id -u) -tdS


// quota.rs
#![feature(available_parallelism)]
fn main() {
    println!("{:?}", std::thread::available_parallelism()); // prints Ok(3)
}

strace:

sched_getaffinity(3041643, 32, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]) = 32
openat(AT_FDCWD, "/proc/self/cgroup", O_RDONLY|O_CLOEXEC) = 3
statx(0, NULL, AT_STATX_SYNC_AS_STAT, STATX_ALL, NULL) = -1 EFAULT (Bad address)
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0444, stx_size=0, ...}) = 0
lseek(3, 0, SEEK_CUR)                   = 0
read(3, "0::/system.slice/run-u31477.serv"..., 128) = 36
read(3, "", 92)                         = 0
close(3)                                = 0
statx(AT_FDCWD, "/sys/fs/cgroup/system.slice/run-u31477.service/cgroup.controllers", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0444, stx_size=0, ...}) = 0
openat(AT_FDCWD, "/sys/fs/cgroup/system.slice/run-u31477.service/cpu.max", O_RDONLY|O_CLOEXEC) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=0, ...}) = 0
lseek(3, 0, SEEK_CUR)                   = 0
read(3, "300000 100000\n", 20)          = 14
read(3, "", 6)                          = 0
close(3)                                = 0
openat(AT_FDCWD, "/sys/fs/cgroup/system.slice/cpu.max", O_RDONLY|O_CLOEXEC) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=0, ...}) = 0
lseek(3, 0, SEEK_CUR)                   = 0
read(3, "max 100000\n", 20)             = 11
read(3, "", 9)                          = 0
close(3)                                = 0
openat(AT_FDCWD, "/sys/fs/cgroup/cpu.max", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
sched_getaffinity(0, 128, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]) = 40

r? @joshtriplett
cc @yoshuawuyts

Tracking issue and previous discussion: #74479

[the8472]

ping @joshtriplett

[joshtriplett]

Sorry for the delay in reviewing this.

I don't think we should be parsing mountinfo to find a cgroup2 mount point, for a few reasons. There's a standard location to mount it, and we already assume the standard location for things like /proc. I think we should do the same for the cgroup2 filesystem. In addition, it doesn't seem particularly uncommon for container setups to mount things into the container via unusual means, such as bind-mounting a subset of things into the container; that may well mean we have the requisite files but don't see a mountpoint for cgroup2. (For similar reasons, we shouldn't use something like fstatfs to check if the file we open comes from cgroup2.)

I think we should just open the file in the expected path (/sys/fs/cgroup), and if it doesn't exist, ignore it.

Separately from that, and this is a more minor implementation detail, I'd also like to reduce the number of joins. Rather than repeatedly joining onto the cgroup mount point, you could join the group to the cgroup mount point once, get the ancestors, and walk up them until you hit the mount point.

Finally, I think it might be worth doing the minor optimization here of repeatedly reading file data into the same buffer, rather than using read_to_string and allocating a fresh string each time.

[the8472]

I don't think we should be parsing mountinfo to find a cgroup2 mount point, for a few reasons. There's a standard location to mount it, and we already assume the standard location for things like /proc. I think we should do the same for the cgroup2 filesystem. In addition, it doesn't seem particularly uncommon for container setups to mount things into the container via unusual means, such as bind-mounting a subset of things into the container; that may well mean we have the requisite files but don't see a mountpoint for cgroup2. (For similar reasons, we shouldn't use something like fstatfs to check if the file we open comes from cgroup2.)

Bind mounts of a cgroup2 fs would still show up as cgroup2 in mountinfo. Googling also suggests that some systemd configurations mount v2 under /sys/fs/cgroup/unified. We could try /sys/fs/cgroup/ as hardcoded fallback if parsing mountinfo doesn't work.

ACK on the other points.

[joshtriplett]

Bind mounts of a cgroup2 fs would still show up as cgroup2 in mountinfo.

Standard bind mounts would, but some specialized setups seem to use special-purpose filesystems to selectively mirror bits and pieces of special filesystems while masking other pieces. In any case, I'd also love to avoid the overhead of parsing mountinfo.

One more bit of rationale: reading the cgroup files is just a few directories, proportional to the depth of cgroups the process is in (typically small). mountinfo, on some large systems, can be huge.

Googling also suggests that some systemd configurations mount v2 under /sys/fs/cgroup/unified.

True, though that's only with older systems/configurations. But we could, for instance, try /sys/fs/cgroup (the canonical location) and fall back to /sys/fs/cgroup/unified (the location on some older systems). I still think that'd be preferable to parsing mountinfo. There's a canonical location for this, and we don't need to do a discovery process to find it, any more than we should do a discovery process for /sys or /dev.

[the8472]

It would be nice if this convention were as cemented in file-hierarchy(7). But we can start like that and if someone has a more exotic setup they can file an issue and then we can introduce mountinfo scanning.

mountinfo, on some large systems, can be huge.

Yeah, I have read about that somewhere, which is why I preemptively added the warning about the function being costly.

[joshtriplett]

It would be nice if this convention were as cemented in file-hierarchy(7).

Sent a PR: systemd/systemd#22624

[joshtriplett]

Update: Per discussion on the systemd PR, it turns out that if cgroup2 is mounted on /sys/fs/cgroup/unified/, it won't actually have resource controllers attached. So there's no point in falling back to that mount point and looking for resource quotas.

Given that, I think it'd be appropriate to just look at /sys/fs/cgroup.

[the8472]

@rustbot ready

[the8472]

Updated the PR comment, it now has an strace too.

[joshtriplett]

@bors r+

[bors]

📌 Commit e18abbf has been approved by joshtriplett