/dev cannot be mounted when rootless podman-run started with -t option.

[bbros-dev]

/kind bug

Description

We are in the process of refactoring the Firecracker build scripts to use Podman (or Docker)....
Unfortunately, it looks like we've hit this prior reported issue in the container run code here, obviously with podman in place of docker (wrapped):

Error: container_linux.go:349: 
  starting container process caused "process_linux.go:449: 
    container init caused \"open /dev/console: permission denied\"": 
      OCI runtime permission denied error

The OpenSuse repo is not yet updated to 2.0.0.
Any insight into whether 2.0.0 would fix this issue?
Update: Issue still present with v2.0.0

One selling point to get the changed merged is the code path stays the same regardless of which OCI engine a user has installed for them.
Obviously we don't want to then have to justify introducing a bug workaround when trying to make Docker optional.

Steps to reproduce the issue:

1. Install Podman or Podman-rootless v1.9.3 (update: or v2.0.0):

   /usr/bin/podman run -i -t --rm \
     --volume /dev:/dev \
     --volume /usr/local/src/firecracker:/firecracker:z \
     --env OPT_LOCAL_IMAGES_PATH=/firecracker/build \
     --env PYTHONDONTWRITEBYTECODE=1 \
     fcuvm/dev:v16 chown -R 1000:1000 /firecracker/build

Describe the results you received:

Error: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"open /dev/console: permiss
ion denied\"": OCI runtime permission denied error
+ return 126

Describe the results you expected:

No error.

Additional information you deem important (e.g. issue happens only occasionally):

Reproducible

Output of podman version:

$ podman version
Version:            1.9.3
RemoteAPI Version:  1
Go Version:         go1.10.1
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.10.1
  podmanVersion: 1.9.3
host:
  arch: amd64
  buildahVersion: 1.14.9
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.16, commit: '
  cpus: 2
  distribution:
    distribution: ubuntu
    version: "18.04"
  eventLogger: file
  hostname: desktop.local.lan
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.3.0-40-generic
  memFree: 1819754496
  memTotal: 16652148736
  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 0.4.3
      commit: unknown
  swapFree: 3934515200
  swapTotal: 4156551168
  uptime: 13h 53m 44.48s (Approximately 0.54 days)
registries:
  search:
  - docker.io
store:
  configFile: /home/<user>/.config/containers/storage.conf
  containerStore:
    number: 13
    paused: 0
    running: 0
    stopped: 13
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/<user>/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 64
  runRoot: /run/user/1000/containers
  volumePath: /home/<user>/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt-cache policy podman
podman:
  Installed: (none)
  Candidate: 1.9.3~1
  Version table:
     1.9.3~1 500
        500 http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04  Packages

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical

[bbros-dev]

Still happens with podman 2.0.0:

11:15 $ podman version
Version:      2.0.0
API Version:  1
Go Version:   go1.14.3
Built:        Thu Jan  1 10:00:00 1970
OS/Arch:      linux/amd64

✔ /usr/local/src/firecracker [issue-1973 L|✚ 1…1] 
11:15 $ tools/devtool test -- --ignore integration_tests/performance
[Firecracker devtool] 2020-06-26 01:15:29 UTC
[Firecracker devtool] Starting test run ...
Error: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"open /dev/con
sole: permission denied\"": OCI runtime permission denied error
Error: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"open /dev/con
sole: permission denied\"": OCI runtime permission denied error

[rhatdan]

Why do you need all of /dev mounted into the container? Do you need a specific device?

The issue is /dev/console is

ls -l /dev/console
crw--w----. 1 root tty 5, 1 Jun 18 15:50 /dev/console

Which means that the inside of a non root process is not allowed to write to it.

As a non root user if I do:

$ echo hello > /dev/console
bash: /dev/console: Permission denied

So if I do this in the container I will get the same thing.

Without the -t option podman and the oci (crun, crun) do not create /dev/console

$ podman run fedora ls -l /dev/console
ls: cannot access '/dev/console': No such file or directory

If you use the -t option, they create the console

$ podman run -t fedora ls -l /dev/console
crw--w----. 1 root tty 136, 0 Jun 26 09:25 /dev/console

If you volume mount in /dev of /dev/console it is the unwritable one
from the host

$ podman run -v /dev:/dev fedora ls -l /dev/console
crw--w----. 1 nobody nobody 5, 1 Jun 18 19:50 /dev/console

When podman attempts to create the /dev/console on top of a unwriteable object
it blows up.

$ podman run -v /dev:/dev -t fedora ls -l /dev/console
Error: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused "open /dev/console: permission denied"": OCI runtime permission denied error

[bbros-dev]

Thanks for that breakdown @rhatdan.

Why do you need all of /dev mounted into the container?

I'm still coming to grips with the build infrastructure, so can't offer more than "To be compatible with the parameters/options passed to Docker".
Sorry, I know that is not very satisfying, I'll add more as I learn more.
Right now we can workaround with sudo podman ...., but that kind of defeats the point of Podman so it won't be what we push upstream - it will allow us to continue working in the interim.

From your description the first thing to try is removing the -t option, then dig further into the fcuvm container.

Standby...

[bbros-dev]

Updated

For those following at home, we're still on Podman 2.0.0...

Removing -t (--tty) results in a different error.

$ /usr/bin/podman run --user 1000:1000 --workdir /firecracker -i --rm --volume /dev:/dev --volume /usr/local/src/firecracker
:/firecracker:z --env OPT_LOCAL_IMAGES_PATH=/firecracker/build --env PYTHONDONTWRITEBYTECODE=1 fcuvm/dev:v16 cargo build --target-dir /firecracker/build/cargo_target --target x86_64-unknown-linux-musl

Error: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linu
x.go:378: setting rlimits for ready process caused \\\"error setting rlimit type 7: invalid argument\\\"\"": OCI runtime err
or

Removing -i (--interactive) results in the error as above.

So interestingly, dropping all the cruft and just trying to run this image we get the "error setting rlimit type 7: invalid argument" error:

$ /usr/bin/podman run --rm -it fcuvm/dev:v16 bash

Error: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linu
x.go:378: setting rlimits for ready process caused \\\"error setting rlimit type 7: invalid argument\\\"\"": OCI runtime err
or

In turn the "error setting rlimit type 7: invalid argument" led to issue #5526 and this workaround (?) that gets us started:

$ /usr/bin/podman run --ulimit nofile=1048576:1048576 --rm -it fcuvm/dev:v16 bash
root@e8ca9f3e8e17:/firecracker# 

Looking at the Dockerfile there does not seem anything too funky.

So is --ulimit nofile=1048576:1048576:

1. A workaround for an issue?
2. The right thing to do. No issue?

[bbros-dev]

@rhatdan we believe we can now answer you question:

Why do you need all of /dev mounted into the container?

The dev sharing appears to allow the group ID to be 'found'.
Remove ---volume dev:dev and we see this error:

$ /usr/bin/podman run \
    --userns=keep-id \
    --user $(id -u):$(id -g) \
    --workdir /firecracker --interactive --tty \
    --ulimit nofile=1048576:1048576 \
    --rm \
    --volume /usr/local/src/firecracker:/firecracker:z \
    --env OPT_LOCAL_IMAGES_PATH=/firecracker/build \
    --env PYTHONDONTWRITEBYTECODE=1 fcuvm/dev:v16 bash

groups: cannot find name for group ID 1000
1000@82d566fd503c:~$

[bbros-dev]

In case anyone else encounters this issue:

We got a rootless and privilege-free run working by:

• Removing --tty
• Removing --volume dev:dev
• Adding: --userns=keep-id

Because we are retro fitting Podman to a workflow (script) using Docker we introduced a oci_engine_arg_init function that inserts --userns=keep-id when grep podman <<<${oci_engine} is true.
This seems as small price to pay for Podman name-spacing compared to the Docker daemon setup for user name-spacing- if I'm reading the Docker docs correctly.

I believe the instructions below will work with Docker (minus --userns=keep-id), so I'll leave to @rhatdan to reopen this if you believe there is an issue to be addressed.

Otherwise it appears this issue is closed by:

     /usr/bin/podman run \
       --userns=keep-id \
       --user $(id -u):$(id -g) \
       --workdir /firecracker \
       --interactive \
       --ulimit nofile=1048576:1048576 \
       --rm \
       --volume /usr/local/src/firecracker:/firecracker:z \
       --env OPT_LOCAL_IMAGES_PATH=/firecracker/build \
       --env PYTHONDONTWRITEBYTECODE=1 \
       fcuvm/dev:v16 \
       cargo build --target-dir /firecracker/build/cargo_target --target x86_64-unknown-linux-musl

[rhatdan]

Great @bbros-dev I am glad you found a workaround. There are issues to be handled because of rootless support.
If you would like to open a PR to update the troublshooting.md file it would be welcome. But I am not sure there is common
enough problems in your issue.