Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

podman system service consuming 100% CPU #8271

Closed
dmesser opened this issue Nov 8, 2020 · 3 comments
Closed

podman system service consuming 100% CPU #8271

dmesser opened this issue Nov 8, 2020 · 3 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@dmesser
Copy link

dmesser commented Nov 8, 2020

/kind bug

Description

On my Fedora 33 machine, running podman 2.1.1, I have 5 containers running, controlled via systemd user service which are autostarted at boot (loginctl enable-linger). I noticed since some time now that two podman system services consume all available CPU time on my 2 core system.

Steps to reproduce the issue:

  1. Run podman user containers, start via generated systemd units

  2. observe system utilization via top

Describe the results you received:

top - 12:08:28 up 1 day, 11:26,  2 users,  load average: 4.57, 4.41, 4.33
Tasks: 148 total,   1 running, 147 sleeping,   0 stopped,   0 zombie
%Cpu(s): 98.8 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.7 hi,  0.2 si,  0.0 st
MiB Mem :   3832.2 total,    686.9 free,   1246.6 used,   1898.7 buff/cache
MiB Swap:  12155.0 total,  11891.7 free,    263.2 used.   2949.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   3399 root      20   0 1616032   9268     84 S  99.0   0.2   2069:47 /usr/bin/podman system service
   3405 dmesser   20   0 1690020  12664   2308 S  98.7   0.3   2069:41 /usr/bin/podman system service
   1921 dmesser   20   0  755456  26900  18316 S   0.3   0.7   0:33.02 grafana-server --homepath=/usr/share/grafana --config=/etc/grafana/grafa+
   1975 dmesser   20   0  972604 216612  22764 S   0.3   5.5  12:06.23 influxd
      1 root      20   0  108532   7860   4392 S   0.0   0.2   0:05.71 /usr/lib/systemd/systemd --switched-root --system --deserialize 29
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.08 [kthreadd]
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [rcu_gp]
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [rcu_par_gp]
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/0:0H-kblockd]
      9 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [mm_percpu_wq]
     10 root      20   0       0      0      0 S   0.0   0.0   0:00.34 [ksoftirqd/0]
     11 root      20   0       0      0      0 I   0.0   0.0   0:01.91 [rcu_sched]
     12 root      rt   0       0      0      0 S   0.0   0.0   0:00.04 [migration/0]
     13 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/0]
     14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/1]
     15 root      rt   0       0      0      0 S   0.0   0.0   0:00.14 [migration/1]
     16 root      20   0       0      0      0 S   0.0   0.0   0:01.62 [ksoftirqd/1]
     18 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/1:0H-kblockd]
     19 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [kdevtmpfs]
     20 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [netns]
     21 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [rcu_tasks_kthre]
     22 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [rcu_tasks_rude_]

Describe the results you expected:

Low to none CPU utilization from podman system components.

Additional information you deem important (e.g. issue happens only occasionally):

The issue already appeared on Fedora 32, and continues to happen after upgrading to Fedora 33.

Output of podman version:

$ podman version
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Wed Oct  7 18:21:20 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 2
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: <redacted>
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.18-300.fc33.x86_64
  linkmode: dynamic
  memFree: 696098816
  memTotal: 4018405376
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-4.dev.giteecccdb.fc33.x86_64
    version: |-
      slirp4netns version 1.1.4+dev
      commit: eecccdb96f587b11d7764556ffacfeaffe4b6e11
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 12469395456
  swapTotal: 12745433088
  uptime: 35h 27m 45.72s (Approximately 1.46 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/dmesser/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 5
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/dmesser/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  volumePath: /home/dmesser/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1602087680
  BuiltTime: Wed Oct  7 18:21:20 2020
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

Package info (output of rpm -q podman):

podman-2.1.1-12.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

No

Additional environment details (AWS, VirtualBox, physical, etc.):

Hetzner VM

@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Nov 8, 2020
@xomachine
Copy link

It might be a duplicate of #7946

Could you test against the latest master?

@Venefilyn
Copy link

I've got the same issue on v2.1.1. @xomachine could you or anyone point me on how to install from master? Are there builds we can use directly or do we have to build it ourselves?

@rhatdan
Copy link
Member

rhatdan commented Nov 14, 2020

This is fixed in master. Closing. We should have a new release next week, BTW.

@rhatdan rhatdan closed this as completed Nov 14, 2020
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests

5 participants