-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't use kaniko with alpine v3.12+ due to /var/run behavior #1297
Comments
@bobcatfish I don't think using I tried building your image like this to inspect what the FS looks like.
it failed as expected. The files in the
Another question I had is, from the I removed the read-only secret mounted to
Can the read only secret be mounted in another dir? Another option is to look into |
This wouldn't work on Kubernetes that mounts the service account secret automatically under /var/run right? |
So one ugly hack would be to install your package in a different root and then copy it over / in a final scratch image. I made it work from this minimal Dockerfile directly inside a Kubernetes container: FROM alpine:3.12 AS SRC
RUN set -x; \
# Actually make the installation in a different root dir
mkdir -p /proot/etc; \
\
apk -p /proot add --initdb && \
\
cp -r /etc/apk /proot/etc; \
\
apk -p /proot update && \
apk -p /proot fix && \
apk -p /proot add curl ca-certificates tzdata zip unzip openssl && \
\
<whatever needs to be done> \
\
# Clean up
rm -rf /proot/dev; \
rm -rf /proot/sys; \
rm -rf /proot/proc; \
unlink /proot/var/run; \
rm -rf /proot/var/cache/apk/*
FROM scratch
COPY --from=SRC /proot/ /
RUN <all the commands you'd have run after your pkg install> Indeed this is only a basic workaround as this will come to bite you back anytime you need to install more packages in an image dependant of this one.... If you don't have much specific needs at least it builds an alpine:3.12 :D |
Another hack would be to not fail if Say, you create an upgrade script
|
Wouldn't it be easier for kaniko to extract images and run commands in a separate root context ? How difficult would it be to implement? |
After looking at the output of
However it is still installed.
|
@olivier-mauras That would involve some major design changes. The way kaniko executed run command is, it actually calls a |
Yeah like using chroot so that you don't have any mixups... Would that work? https://golang.org/pkg/syscall/#Chroot EDIT: https://github.com/GoogleContainerTools/kaniko/blob/master/pkg/commands/run.go#L210 Am I understanding correctly that |
If we choose this approach for every command, i,e map "/" to another directory in "tmp", then i see 2 issues.
Another approach would be to map "/" to "/tmp/kanikoRootXXX" at the beginning of the build. (which is probably what you are suggesting in the edit) I don't think its not feasible or ugly. It could be a little hard to wire up. I would not mind pursuing this direction. |
Exactly
This would probably solve quite a bunch of |
The only caveat is, currently i am the only one working actively on this project in 20% capacity. |
I'm running into this too. Upgrading from a Ruby image that uses 3.10 to 3.12 and I'm hitting this in my Gitlab CI. Unsure what the best path forward is there. |
One particularly quick fix is : |
@bobcatfish can you share the workaround that is working for you? We are running into the same issues using Kaniko in our Tekton pipelines. |
Hey @jpower432 - our workaround is just to pin to alpine 3.11 tektoncd/pipeline#2757 This works for us because we didn't have any particular need to use 3.12 but won't work for you if you actually need 3.12 :O |
same here |
i have other issues continually w/ kaniko when any operations (like |
Seeing this issue as well, using kaniko in a gitlab runner. I suppose the solution is to pin all alpine builds we have at 3.11? |
@mattsurge yes. |
@tejal29 , Do we have any timeline to fix this issue for the latest alpine builds, we are kinda blocked to use kaniko to build alpine images. we can't ignore alpine-baselayout since it has core package updates. |
Is this problem solved now? |
Another solution is to not mount the service account token automatically: GitLab has a feature request to add this as an option: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4786 |
It seems like there is a setting for this? https://github.com/GoogleContainerTools/kaniko#flag---ignore-var-run |
@tobiasmcnulty
|
Boo. Okay. Thanks for testing it. Just saw this in the docs looking for something else and didn't see it mentioned elsewhere on this issue. |
Note that if it did work, and you are building on K8s, you'd probably end up with secrets in your image. |
* refactor(ci): Use Todie spec * feat: bump to 4.16.4 * fix: disable alpine package upgrade Due to this bug GoogleContainerTools/kaniko#1297
Hey folks, any news about this error? I'm getting this when I try to run Log:
|
Bump :) |
Bump. I'm trying something else, where I need to delete the content of /var/* and it's failing due to this. |
Bump |
Thanks bro, it's worked for me. |
Actual behavior
Tekton is using Kaniko to build a Docker image from alpine and recently the builds started failing.
TL;DR
The alpine:3.12 image has /var/run aliased to /run. When running kaniko in a kubernetes pod with service accounts, the serviceaccounts often seem to end up mounted to /var/run.
Kaniko is ignoring the contents and state of /var/run in the base image (alpine:3.12) but unfortunately some details of alpine seem to depend on /var/run being a symlink to /run, and so not preserving that is causing upgrading alpine packages to fail.
Details
We discovered this in tektoncd/pipeline#2738.
It seems the problem is caused by recent versions of alpine-baselayout in alpine3.12. When we build from alpine 3.12 and upgrade all alpine packages, the alpine-baselayout upgrade fails:
Expected behavior
Kaniko should detect that /var/run is a symlink in the base image and preserve that. (I think! I'm not sure if it's that simple.)
To Reproduce
Using this dockerfile and mounting a file into /var/run, I can build with docker but not with Kaniko.
Trying to build with kaniko:
The error above about not being able to remove the file seems to come from https://git.alpinelinux.org/aports/tree/main/alpine-baselayout/alpine-baselayout.pre-upgrade which works just fine if /var/run is a symlink to /run, which I discovered by trying to do the same thing by using the alpine image directly without kaniko:
That works just fine!
I tried not whitelisting /var/run and that didn't work either:
Finally, using docker to build the image (from the pipelines repo checkout) worked just fine:
Additional Information
Triage Notes for the Maintainers
--cache
flagThe text was updated successfully, but these errors were encountered: