Unable to create containers when cgroup subsystems are mounted in multiple places #1367
Comments
|
Yeah, this is related to #798 and several other issues we've had in the past. IMO we need to rework how mountpoint checking is done (the code is basically unreadable and is [in my mind] fragile). In particular this code also had similar issues with mounting different cgroup versions (which has now been fixed but I don't like the fix 😉). In my mind, the nice way for this to be handled (so that it can also accomodate #774) is this:
The The above idea could be extended to check (with |
|
Ok so iiuc this a known issue and it looks like it may require a fairly large change in order to address it. Has anyone started to look at this? If not we could maybe try to get a PR organised. Is this something that would be likely to be accepted? |
|
I had started to look at it, but I've been swamped by other things recently. IMO if you put together a patch that managed to clean up the mountpoint-finding magic we're currently using to be more resistant to different setups I'd be okay with merging it (though I can't speak for the other maintainers). I wouldn't recommend going straight for the entire rewrite that I proposed above -- simply because it would touch a lot of code that would make people nervous ( Though I would like to point out that having multiple mountpoints of the same subsystem is a bit of an odd thing to do (they track the same information) -- however as I mentioned above this is a symptom of a bigger issue we have in the current cgroup manager implementation in runC. |
|
Ok cool, that sounds reasonable to me. Yeah I agree it is a bit strange... The background here is that a user was attempting to deploy Garden alongside some other software that was doing its own independent cgroup setup, which then caused runc to start erroring with the above message. |
|
@cyphar Opened a PR for your review. |
|
Fixed by #1372 |
In cgroup v1, the parent of a cgroup mount is on tmpfs (or maybe any other fs but definitely not cgroupfs). Use this to traverse up the tree until we reach non-cgroupfs. This should work in any setups (nested container, non-standard cgroup mounts) so issue [1] won't happen. Theoretically, the only problematic case is when someone mounts cpuset cgroupfs into a directory on another cgroupfs. Let's assume people don't do that -- if they do, they will get other error (e.g. inability to read cpuset.cpus file). [1] opencontainers#1367 Signed-off-by: Kir Kolyshkin <[email protected]>
In cgroup v1, the parent of a cgroup mount is on tmpfs (or maybe any other fs but definitely not cgroupfs). Use this to traverse up the tree until we reach non-cgroupfs. This should work in any setups (nested container, non-standard cgroup mounts) so issue [1] won't happen. Theoretically, the only problematic case is when someone mounts cpuset cgroupfs into a directory on another cgroupfs. Let's assume people don't do that -- if they do, they will get other error (e.g. inability to read cpuset.cpus file). [1] opencontainers#1367 Signed-off-by: Kir Kolyshkin <[email protected]>
In cgroup v1, the parent of a cgroup mount is on tmpfs (or maybe any other fs but definitely not cgroupfs). Use this to traverse up the tree until we reach non-cgroupfs. This should work in any setups (nested container, non-standard cgroup mounts) so issue [1] won't happen. Theoretically, the only problematic case is when someone mounts cpuset cgroupfs into a directory on another cgroupfs. Let's assume people don't do that -- if they do, they will get other error (e.g. inability to read cpuset.cpus file). [1] opencontainers#1367 Signed-off-by: Kir Kolyshkin <[email protected]>
In cgroup v1, the parent of a cgroup mount is on tmpfs (or maybe any other fs but definitely not cgroupfs). Use this to traverse up the tree until we reach non-cgroupfs. This should work in any setups (nested container, non-standard cgroup mounts) so issue [1] won't happen. Theoretically, the only problematic case is when someone mounts cpuset cgroupfs into a directory on another cgroupfs. Let's assume people don't do that -- if they do, they will get other error (e.g. inability to read cpuset.cpus file). [1] opencontainers#1367 Signed-off-by: Kir Kolyshkin <[email protected]>
In cgroup v1, the parent of a cgroup mount is on tmpfs (or maybe any other fs but definitely not cgroupfs). Use this to traverse up the tree until we reach non-cgroupfs. This should work in any setups (nested container, non-standard cgroup mounts) so issue [1] won't happen. Theoretically, the only problematic case is when someone mounts cpuset cgroupfs into a directory on another cgroupfs. Let's assume people don't do that -- if they do, they will get other error (e.g. inability to read cpuset.cpus file). [1] opencontainers#1367 Signed-off-by: Kir Kolyshkin <[email protected]>
In cgroup v1, the parent of a cgroup mount is on tmpfs (or maybe any other fs but definitely not cgroupfs). Use this to traverse up the tree until we reach non-cgroupfs. This should work in any setups (nested container, non-standard cgroup mounts) so issue [1] won't happen. Theoretically, the only problematic case is when someone mounts cpuset cgroupfs into a directory on another cgroupfs. Let's assume people don't do that -- if they do, they will get other error (e.g. inability to read cpuset.cpus file). [1] opencontainers#1367 Signed-off-by: Kir Kolyshkin <[email protected]>
In cgroup v1, the parent of a cgroup mount is on tmpfs (or maybe any other fs but definitely not cgroupfs). Use this to traverse up the tree until we reach non-cgroupfs. This should work in any setups (nested container, non-standard cgroup mounts) so issue [1] won't happen. Theoretically, the only problematic case is when someone mounts cpuset cgroupfs into a directory on another cgroupfs. Let's assume people don't do that -- if they do, they will get other error (e.g. inability to read cpuset.cpus file). [1] opencontainers#1367 Signed-off-by: Kir Kolyshkin <[email protected]>
In cgroup v1, the parent of a cgroup mount is on tmpfs (or maybe any other fs but definitely not cgroupfs). Use this to traverse up the tree until we reach non-cgroupfs. This should work in any setups (nested container, non-standard cgroup mounts) so issue [1] won't happen. Theoretically, the only problematic case is when someone mounts cpuset cgroupfs into a directory on another cgroupfs. Let's assume people don't do that -- if they do, they will get other error (e.g. inability to read cpuset.cpus file). [1] opencontainers/runc#1367 Signed-off-by: Kir Kolyshkin <[email protected]>
In cgroup v1, the parent of a cgroup mount is on tmpfs (or maybe any other fs but definitely not cgroupfs). Use this to traverse up the tree until we reach non-cgroupfs. This should work in any setups (nested container, non-standard cgroup mounts) so issue [1] won't happen. Theoretically, the only problematic case is when someone mounts cpuset cgroupfs into a directory on another cgroupfs. Let's assume people don't do that -- if they do, they will get other error (e.g. inability to read cpuset.cpus file). [1] opencontainers/runc#1367 Signed-off-by: Kir Kolyshkin <[email protected]>
In cgroup v1, the parent of a cgroup mount is on tmpfs (or maybe any other fs but definitely not cgroupfs). Use this to traverse up the tree until we reach non-cgroupfs. This should work in any setups (nested container, non-standard cgroup mounts) so issue [1] won't happen. Theoretically, the only problematic case is when someone mounts cpuset cgroupfs into a directory on another cgroupfs. Let's assume people don't do that -- if they do, they will get other error (e.g. inability to read cpuset.cpus file). [1] opencontainers/runc#1367 Signed-off-by: Kir Kolyshkin <[email protected]>
In cgroup v1, the parent of a cgroup mount is on tmpfs (or maybe any other fs but definitely not cgroupfs). Use this to traverse up the tree until we reach non-cgroupfs. This should work in any setups (nested container, non-standard cgroup mounts) so issue [1] won't happen. Theoretically, the only problematic case is when someone mounts cpuset cgroupfs into a directory on another cgroupfs. Let's assume people don't do that -- if they do, they will get other error (e.g. inability to read cpuset.cpus file). [1] opencontainers/runc#1367 Signed-off-by: Kir Kolyshkin <[email protected]>
We recently ran into an issue whereby runc was not able to create containers and was failing with the following error:
This was happening due to an additional cpu,memory cgroup subsystem mounted somewhere on the system (at
/cgroupsin this case):It seems that this is only a problem when two mounts of the same subsystem are performed in a certain order (i.e in the example above, the
/cgroupsmount appears first in/proc/self/mountinfo).Is runc able/supposed to support this sort of setup?
Cheers,
Ed & @craigfurman
The text was updated successfully, but these errors were encountered: