kata not supporting cpuset/cpus via cri? #1079
Comments
|
similar failure on $ docker run -d --cpuset-cpus="0-1" ubuntu sleep 30000
347be183369a3a73d0a1be40dc64a418401f219e8aa1f34a6a9efd58c0b4e331
docker: Error response from daemon: OCI runtime create failed: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"failed to write 0-1 to cpuset.cpus: write /sys/fs/cgroup/cpuset/docker/347be183369a3a73d0a1be40dc64a418401f219e8aa1f34a6a9efd58c0b4e331/cpuset.cpus: invalid argument\"": unknown.while no error with runc: # docker run -d --cpuset-cpus="0-1" --runtime runc ubuntu sleep 30000
7852efd248e08125a91b703db545913785d257db00d1a560717e85cb8dba0fac |
peter-wangxu
changed the title
|
Thanks for the issue, @peter-wangxu To clarify, your original comment / issue description just suggestes --cpus 2 -- this should work with Kata today. Your comment does make use of --cpuset-cpus, though, which is not supported today. I believe this covers the same issue. WDYT? I'm hoping to get this feature added once more of us are back from holidays. |
|
@jcvenegas - FYI. |
|
hi @egernst, thanks for point out my flaw in the original issue. What I expected is the
sudo docker run --name cpus_runc --runtime runc -d --cpus 2 ubuntu:latest sleep 1000
peter@ubuntu:/sys/fs/cgroup/cpu/docker/1824e5251ce38ac1efbdaf78845e405319fcf7b505fca881ade888b1fe2c47bd$ cat cpu.cfs_period_us
100000
peter@ubuntu:/sys/fs/cgroup/cpu/docker/1824e5251ce38ac1efbdaf78845e405319fcf7b505fca881ade888b1fe2c47bd$ cat cpu.cfs_quota_us
200000
$ sudo docker run --name cpus_kata --runtime kata-runtime -d --cpus 2 ubuntu:latest sleep 1000
5ef1ae1bc8a062d765fea2a1f034144e3968df06f4b5b5882ebcad16b575b05c
peter@ubuntu:/sys/fs/cgroup/cpu/kata/5ef1ae1bc8a062d765fea2a1f034144e3968df06f4b5b5882ebcad16b575b05c$ cat cpu.cfs_period_us
100000
peter@ubuntu:/sys/fs/cgroup/cpu/kata/5ef1ae1bc8a062d765fea2a1f034144e3968df06f4b5b5882ebcad16b575b05c$ cat cpu.cfs_quota_us
-1My doubt is how kata throttles the qemu-based container? Did I miss something here? |
|
@egernst, i have the same question for this cpus resources seeting. |
|
@peter-wangxu @zhiminghufighting
a POD is a group of containers sharing something (network, etc), when kata is the runtime, those containers run inside a virtual machine (qemu), cgroups and namespaces are applied in the guest not in the host, it's no possible for a process (like qemu) be part of two or more cgroups at the same time. For example if the cpu constrain for container A is 2 and 3 for the container B, what should be the cpu constrain for qemu 2, 3 or 5? also this number must be updated when a container is created or finished. |
|
this issue is fixed in #1189 |
|
fixed #1189 |
Description of problem
sudo docker run --name cpuset -d --cpus 2 ubuntu:latest sleep 1000
$ docker inspect a3c6c718df1e [ { "Id": "a3c6c718df1e4f5ec595ea13894d5446ff5c504b0d4b346288926f2bb32dd329", "Created": "2018-12-28T08:57:51.536880929Z", "Path": "sleep", "Args": [ "1000" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 3107, "ExitCode": 0, "Error": "", "StartedAt": "2018-12-28T08:57:54.048865781Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:ea4c82dcd15a33e3e9c4c37050def20476856a08e59526fbe533cc4e98387e39", "ResolvConfPath": "/var/lib/docker/containers/a3c6c718df1e4f5ec595ea13894d5446ff5c504b0d4b346288926f2bb32dd329/resolv.conf", "HostnamePath": "/var/lib/docker/containers/a3c6c718df1e4f5ec595ea13894d5446ff5c504b0d4b346288926f2bb32dd329/hostname", "HostsPath": "/var/lib/docker/containers/a3c6c718df1e4f5ec595ea13894d5446ff5c504b0d4b346288926f2bb32dd329/hosts", "LogPath": "/var/lib/docker/containers/a3c6c718df1e4f5ec595ea13894d5446ff5c504b0d4b346288926f2bb32dd329/a3c6c718df1e4f5ec595ea13894d5446ff5c504b0d4b346288926f2bb32dd329-json.log", "Name": "/cpuset", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "docker-default", "ExecIDs": null, "HostConfig": { "Binds": null, "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "default", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "shareable", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "kata-runtime", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 2000000000, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": [], "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DiskQuota": 0, "KernelMemory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": 0, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": [ "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "ReadonlyPaths": [ "/proc/asound", "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/b594a2fe867fe9ffc09374a20cd7c3dbec958043159dbc781d3298412771dc4e-init/diff:/var/lib/docker/overlay2/f960b3b362e0e00fb5d0b490ae166a1457fd71d954384a492bff115cc26a017a/diff:/var/lib/docker/overlay2/8f14df32c43aafc75f2ed28cd34f29ec64e65536fe2679df7f12e248321d77e0/diff:/var/lib/docker/overlay2/c9cea40e5eb480d51721c4a323cf58c7bcc0b3861fbd6212ab0a0e771987c160/diff:/var/lib/docker/overlay2/d8ca8d8b58b68a60a0a9f68d9ac542968caf49952f15d255eb2c272d48229d77/diff", "MergedDir": "/var/lib/docker/overlay2/b594a2fe867fe9ffc09374a20cd7c3dbec958043159dbc781d3298412771dc4e/merged", "UpperDir": "/var/lib/docker/overlay2/b594a2fe867fe9ffc09374a20cd7c3dbec958043159dbc781d3298412771dc4e/diff", "WorkDir": "/var/lib/docker/overlay2/b594a2fe867fe9ffc09374a20cd7c3dbec958043159dbc781d3298412771dc4e/work" }, "Name": "overlay2" }, "Mounts": [], "Config": { "Hostname": "a3c6c718df1e", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": [ "sleep", "1000" ], "Image": "ubuntu:latest", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": {} }, "NetworkSettings": { "Bridge": "", "SandboxID": "1f15e59ae26be28242d21cfeaae218812ee77c6d23ae9c7a6dd7eb4d5222fa47", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "/var/run/docker/netns/1f15e59ae26b", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "16231dd45f18ebe3a4c09db5d9c42f8d492b3183838034cb2baccf7c77751e03", "Gateway": "172.17.0.1", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "172.17.0.3", "IPPrefixLen": 16, "IPv6Gateway": "", "MacAddress": "02:42:ac:11:00:03", "Networks": { "bridge": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "8e8972f0131f530c622fe4ea158523d84a90db69b5e33bf1ab30918ce56b7ce4", "EndpointID": "16231dd45f18ebe3a4c09db5d9c42f8d492b3183838034cb2baccf7c77751e03", "Gateway": "172.17.0.1", "IPAddress": "172.17.0.3", "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "02:42:ac:11:00:03", "DriverOpts": null } } } } ]cat /sys/fs/cgroup/cpuset/kata/a3c6c718df1e4f5ec595ea13894d5446ff5c504b0d4b346288926f2bb32dd329/cpuset.cpus
0-2
obviously, both the docker inspect and cgroup parameter did not respect my input
--cpus 2Expected result
we should see
0-1from the output of cpuset.cpusActual result
kata-collect-data.sh
Meta details
Running
kata-collect-data.shversion1.4.1 (commit 45ea48e)at2018-12-28.01:19:16.260745953-0800.Runtime is
/usr/bin/kata-runtime.kata-envOutput of "
/usr/bin/kata-runtime kata-env":Runtime config files
Runtime default config files
Runtime config file contents
Config file
/etc/kata-containers/configuration.tomlnot foundOutput of "
cat "/usr/share/defaults/kata-containers/configuration.toml"":KSM throttler
version
Output of "
/usr/libexec/kata-ksm-throttler/kata-ksm-throttler --version":systemd service
Image details
Initrd details
No initrd
Logfiles
Runtime logs
Recent runtime problems found in system journal:
Proxy logs
Recent proxy problems found in system journal:
Shim logs
No recent shim problems found in system journal.
Throttler logs
No recent throttler problems found in system journal.
Container manager details
Have
dockerDocker
Output of "
docker version":Output of "
docker info":Output of "
systemctl show docker":Have
kubectlKubernetes
Output of "
kubectl version":Output of "
kubectl config view":Output of "
systemctl show kubelet":No
crioPackages
Have
dpkgOutput of "
dpkg -l|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"":No
rpmThe text was updated successfully, but these errors were encountered: