Kaniko builds fail on k8s GitLab CI runner when /etc/ssh is mounted

[JacobHenner]

Actual behavior
When kaniko is used to build an image within a GitLab CI Kubernetes executor, the build fails with the following error if /etc/ssh is mounted within the executor (the Kubernetes pod created by GitLab for the build).

�[37mDEBU�[0m[0049] Not adding /etc/ssh because it is ignored    
�[37mDEBU�[0m[0049] Whiting out /etc/ssh/.wh..wh..opq            
error building image: error building stage: failed to get filesystem from image: removing whiteout etc/ssh/.wh..wh..opq: unlinkat /etc/ssh/.wh..opq: read-only file system

However, if I change the mount path to something else (e.g. /etc/ssh.bak/), this error does not occur.

I wonder if this is due to the presence of /etc/ssh within the FROM image itself. Perhaps if /etc/ssh.bak existed in the image, it'd fail too? I'm not sure why the mounts on the system where kaniko is executing would interact with the paths in the container image.

Expected behavior
I expected the kaniko build to work successfully, without being affected by the mounts within the Kubernetes pod where kaniko is running.

To Reproduce

1. Run kaniko within a GitLab CI Kubernetes executor pod, with the GitLab runner configured to mount the contents of a ConfigMap as /etc/ssh
2. Wait for kaniko to fail with the above error.

Additional Information

• Dockerfile

FROM <internal image>

RUN ls -la

CMD bash

• Build Context - No additional build context
• Kaniko Image (fully qualified with digest) - gcr.io/kaniko-project/executor@sha256:d60705cb55460f32cee586570d7b14a0e8a5f23030a0532230aaf707ad05cecd

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[JacobHenner]

I just tested using a FROM image without /etc/ssh, which leads me to believe that somehow there is a conflict when a path exists within both the image and the kaniko host, if the path on the kaniko host is read-only (as it would be for a read only Kubernetes ConfigMap volume)?

[tejal29]

@JacobHenner Can you please specify your mount definition in the pod spec and the debug logs?

[tejal29]

Looks like, kaniko is missing the fact that /etc/ssh is mounted.

[JacobHenner]

    volumeMounts:
    - mountPath: /builds
      name: repo
    - mountPath: /volume/obscuredA
      name: obscuredA
    - mountPath: /volume/obscuredB
      name: obscuredB
    - mountPath: /volume/obscuredC
      name: obscuredC
    - mountPath: /etc/ssh
      name: ssh
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-vnb9h
      readOnly: true

  volumes:
  - configMap:
      defaultMode: 420
      name: ssh
    name: ssh

[JacobHenner]

This issue is still occurring in the latest image. I am trying to track it down. Kaniko's logs indicate that /etc/ssh is detected as a mounted directory (mentioned in the list of mounts at the start of execution), doesn't add /etc/ssh because it's ignored, and then it encounters the white out error.

I am not sure what the expected behavior is here, since I'm having some difficulty following how kaniko is manipulating these whiteout files. As far as I know, they are created by Kaniko or hidden in some layer of the Dockerfile's parent image, as they are not present in the running parent image or the running GitLab executor that's calling kaniko.

What is the expected behavior (at a technical level, beyond it working)?