/var/run/secrets/kubernetes.io/serviceaccount unexpected in build container

[patrikbeno]

Actual behavior

Kubernetes: /var/run/secrets/kubernetes.io/serviceaccount is unexpectedly mounted in build container.

Expected behavior

Dockerfile build is isolated from host.
Only a few very specific system mounts are provided: /dev, /sys, /proc, /etc/{hostname,resolv.conf,hosts}

Arbitrary /var/run/* mount points do not seem to belong to this category.

k8s secrets (/var/run/secrets/kubernetes.io/*) may be needed in a k8s pod, and this can be configured using automountServiceAccountToken.

But whether or not k8s pod needs/uses this feature, files in /var/run shouldn't be automatically mounted into the build container.

It may be useful to do so, but only on-demand (explicit configuration option), never by default, and especially not if there is no opt-out available.

To Reproduce

kubectl -it --rm kaniko --image gcr.io/kaniko-project/executor:debug --command sh

cat <<EOF >Dockerfile
FROM alpine
RUN df -ha
RUN rm -rfv /var/run/* # this is completely valid operation in container
EOF

/kaniko/executor -c . -f Dockerfile --no-push

INFO[0000] Retrieving image manifest alpine
INFO[0000] Retrieving image alpine from registry index.docker.io
E0923 16:56:34.836595      10 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors
INFO[0008] Built cross stage deps: map[]
INFO[0008] Retrieving image manifest alpine
INFO[0008] Returning cached image manifest
INFO[0008] Executing 0 build triggers
INFO[0008] Unpacking rootfs as cmd RUN df -ha requires it.
INFO[0009] RUN df -ha
INFO[0009] Taking snapshot of full filesystem...
INFO[0010] cmd: /bin/sh
INFO[0010] args: [-c df -ha]
INFO[0010] Running: [/bin/sh -c df -ha]
Filesystem                Size      Used Available Use% Mounted on
[...]
tmpfs                     8.6G     12.0K      8.6G   0% /var/run/secrets/kubernetes.io/serviceaccount
[...]
INFO[0010] Taking snapshot of full filesystem...
INFO[0010] No files were changed, appending empty layer to config. No layer added to image.
INFO[0010] RUN rm -rfv /var/run/*
INFO[0010] cmd: /bin/sh
INFO[0010] args: [-c rm -rfv /var/run/*]
INFO[0010] Running: [/bin/sh -c rm -rfv /var/run/*]
rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/..data': Read-only file system
rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/ca.crt': Read-only file system
rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/token': Read-only file system
rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/namespace': Read-only file system
rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/..2021_09_23_16_56_10.269422877/ca.crt': Read-only file system
rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/..2021_09_23_16_56_10.269422877/token': Read-only file system
rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/..2021_09_23_16_56_10.269422877/namespace': Read-only file system
removed directory: '/var/run/secrets/kubernetes.io/serviceaccount/..2021_09_23_16_56_10.269422877'
removed directory: '/var/run/secrets/kubernetes.io/serviceaccount'
removed directory: '/var/run/secrets/kubernetes.io'
removed directory: '/var/run/secrets'
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

Additional Information

Kaniko Image (fully qualified with digest):
gcr.io/kaniko-project/executor@sha256:fcccd2ab9f3892e33fc7f2e950c8e4fc665e7a4c66f6a9d70b300d7a2103592f

There is more meaningful use case where this is real life problem:

FROM alpine:edge
RUN echo "https://dl-cdn.alpinelinux.org/alpine/edge/testing" | tee -a /etc/apk/repositories &&\
    apk add ca-certificates &&\
    apk upgrade

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[patrikbeno]

Related: #1297

[itz-Jana]

Bump, similar issue here. Need to delete /var/* for my build. Works in normal Docker, but not with Kaniko.