Gitlab runner k8s: The build failed when setting the flag --ignore-var-run=false

[quangthe]

Actual behavior

Use gitlab runner (run on k8s) to build Docker image using kaniko.

The build failed when setting the flag --ignore-var-run=true.

error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..data: read-only file system

The log with debug enabled

INFO[0001] Retrieving image manifest debian:buster      
INFO[0001] Retrieving image debian:buster from registry index.docker.io 
INFO[0004] Built cross stage deps: map[]                
INFO[0004] Retrieving image manifest debian:buster      
INFO[0004] Returning cached image manifest              
INFO[0004] Executing 0 build triggers                   
INFO[0004] Building stage 'debian:buster' [idx: '0', base-idx: '-1'] 
INFO[0004] Unpacking rootfs as cmd RUN groupadd -g $GID $UNAME requires it. 
DEBU[0004] Ignore list: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/freezer false} {/dev/mqueue false} {/scripts-282-64633 false} {/logs-282-64633 false} {/builds false} {/busybox false} {/certs/client false} {/dev/termination-log false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/var/run/secrets/kubernetes.io/serviceaccount false}] 
DEBU[0004] Not adding /dev because it is ignored        
DEBU[0004] Not adding /etc/hostname because it is ignored 
DEBU[0004] Not adding /etc/resolv.conf because it is ignored 
DEBU[0004] Not adding /proc because it is ignored       
DEBU[0004] Not adding /sys because it is ignored        
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..data: read-only file system

Expected behavior

Suggestion: Should provide the option to explicitly include some specific paths under /var/run instead of including all /var/run as we don't need all stuff under /var/run. There are cases in which we only need to keep state of some directories, such as /var/run/postgresql.

To Reproduce

• Setup k8s gitlab runner for the project
• Run the pipeline with provided Dockerfile and .gitlab-ci.yml in this ticket.

Additional Information

• Dockerfile

FROM debian:buster

ARG UNAME=postgres
ARG UID=70
ARG GID=70

ENV LANG en_US.UTF-8 
ENV LANGUAGE en_US:en 
ENV LC_ALL en_US.UTF-8

RUN groupadd -g $GID $UNAME
RUN useradd -m -u $UID -g $GID -s /bin/bash $UNAME
RUN usermod -aG root $UNAME

USER postgres

• Build Context

stages:
  - build

build-push-docker-image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.12.1-debug
    entrypoint: [""]
  before_script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}"
      --ignore-var-run=false
      --verbosity debug

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	No
Please check if the build works in docker but not in kaniko	Yes
Please check if this error is seen when you use --cache flag	No
Please check if your dockerfile is a multistage dockerfile	No

[ghost]

I was bitten by this today as well. Seems like kaniko makes incorrect assumptions.

Here is a simple example that demonstrates the problem and does not require GitLab, Kubernetes and --ignore-var-run:

% docker run -it --entrypoint /bin/sh gcr.io/kaniko-project/executor:debug
/workspace # cat <<EOF > Dockerfile
> FROM alpine
> RUN mkdir /var/bla
> EOF
/workspace # mkdir -p /var/bla
/workspace # /kaniko/executor --destination=image --no-push --tar-path=/tmp/image.tar
INFO[0000] Retrieving image manifest alpine
INFO[0000] Retrieving image alpine from registry index.docker.io
INFO[0001] Built cross stage deps: map[]
INFO[0001] Retrieving image manifest alpine
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Building stage 'alpine' [idx: '0', base-idx: '-1']
INFO[0001] Unpacking rootfs as cmd RUN mkdir /var/bla requires it.
INFO[0001] RUN mkdir /var/bla
INFO[0001] Initializing snapshotter ...
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Cmd: /bin/sh
INFO[0001] Args: [-c mkdir /var/bla]
INFO[0001] Running: [/bin/sh -c mkdir /var/bla]
mkdir: can't create directory '/var/bla': File exists
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1
/workspace # rmdir /var/bla
/workspace # /kaniko/executor --destination=image --no-push --tar-path=/tmp/image.tar
INFO[0000] Retrieving image manifest alpine
INFO[0000] Retrieving image alpine from registry index.docker.io
INFO[0001] Built cross stage deps: map[]
INFO[0001] Retrieving image manifest alpine
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Building stage 'alpine' [idx: '0', base-idx: '-1']
INFO[0001] Unpacking rootfs as cmd RUN mkdir /var/bla requires it.
INFO[0001] RUN mkdir /var/bla
INFO[0001] Initializing snapshotter ...
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Cmd: /bin/sh
INFO[0001] Args: [-c mkdir /var/bla]
INFO[0001] Running: [/bin/sh -c mkdir /var/bla]
INFO[0001] Taking snapshot of full filesystem...
INFO[0002] Skipping push to container registry due to --no-push flag

Kaniko extracts the source image contents in / instead of an empty directory:

kaniko/pkg/executor/build.go

Line 329 in d7f7bc1

_, err := getFSFromImage(config.RootDir, s.image, util.ExtractFile)

kaniko/pkg/config/init.go

Line 50 in d7f7bc1

RootDir = constants.RootDir

kaniko/pkg/constants/constants.go

Line 21 in d7f7bc1

RootDir = "/"

And that is the problem.

Note that you might get bitten by this in case you execute kaniko multiple times to build multiple images as well. The kaniko argument --cleanup might save you here, though.

A simple GitLab Kubernetes Runner test case.

Dockerfile:

FROM alpine
RUN touch /var/bla

.gitlab-ci.yml:

test:
  image: "gcr.io/kaniko-project/executor:debug"
  script:
    - >-
      /kaniko/executor
      --destination image
      --no-push
      --tar-path=/image.tar
      --ignore-var-run=false

Running this on a GitLab Kubernetes Runner fails because /var/run/secrets/kubernetes.io/serviceaccount is populated in the k8s container where you run kaniko.

Setting --ignore-var-run=false makes kaniko want to try to delete /var/run in case it is present in the source image and the Dockerfile contains some commands the require the image to be unpacked:

kaniko/pkg/executor/build.go

Lines 309 to 340 in d7f7bc1

	// Unpack file system to root if we need to.
	shouldUnpack := false
	for _, cmd := range s.cmds {
	if cmd.RequiresUnpackedFS() {
	logrus.Infof("Unpacking rootfs as cmd %s requires it.", cmd.String())
	shouldUnpack = true
	break
	}
	}
	if len(s.crossStageDeps[s.stage.Index]) > 0 {
	shouldUnpack = true
	}
	if s.stage.Index == 0 && s.opts.InitialFSUnpacked {
	shouldUnpack = false
	}
	if shouldUnpack {
	t := timing.Start("FS Unpacking")
	retryFunc := func() error {
	_, err := getFSFromImage(config.RootDir, s.image, util.ExtractFile)
	return err
	}
	if err := util.Retry(retryFunc, s.opts.ImageFSExtractRetry, 1000); err != nil {
	return errors.Wrap(err, "failed to get filesystem from image")
	}
	timing.DefaultRun.Stop(t)
	} else {
	logrus.Info("Skipping unpacking as no commands require it.")
	}

This part here seems to fail:

kaniko/pkg/util/fs_util.go

Lines 395 to 411 in d7f7bc1

	case tar.TypeSymlink:
	logrus.Tracef("Symlink from %s to %s", hdr.Linkname, path)
	// The base directory for a symlink may not exist before it is created.
	if err := os.MkdirAll(dir, 0755); err != nil {
	return err
	}
	// Check if something already exists at path
	// If so, delete it
	if FilepathExists(path) {
	if err := os.RemoveAll(path); err != nil {
	return errors.Wrapf(err, "error removing %s to make way for new symlink", hdr.Name)
	}
	}
	if err := os.Symlink(hdr.Linkname, path); err != nil {
	return err
	}
	}

as it assumes that it can just delete stuff.

[ghost]

These issues seem to relate in one way or another, some are rather old:
#1750
#2793
#2776
#2701
#1297
#1353
#2764