-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pipeline nightly build is broken #2738
Comments
The https://gitlab.alpinelinux.org/alpine/aports/-/tags/v3.12.0 was released 4dd ago, which is when the last successful nightly happened. |
|
I can reproduce this on my cluster running Tekton v0.12.0
|
It looks like something is trying to remove a mounted secret:
|
btw looks like this might be a duplicate of #2726 looks like pinning didnt fix it :S |
Here's a log from a recent successful run, where this "pre-upgrade" thing doesnt seem to be getting involved:
It's interesting that the successful log references these versions:
In the failed log we have these versions:
|
It seems like the error might be coming from:
maybe that's not actually something we want to upgrade? we're trying to upgrade all packages: Lines 3 to 5 in 4f670ce
|
https://git.alpinelinux.org/aports/tree/main/alpine-baselayout/alpine-baselayout.pre-upgrade
wut |
I'm not sure what's going on but I recommend building from older checkouts of pipelines and seeing if this was caused by a change that was introduced in the pipelines repo; if so we can use a binary search to find the problem. |
Just realized that to make 0.13 ill need to fix this - and I'm build cop tomorrow anyway, so no time like the present :D |
Okay so I was able to run kaniko locally and reproduce this more or less by introducing the slightly contrived step of mounting a read only file into /var/run:
I got this error:
I then pinned to 3.11 and it built just fine. It seems like pinning to 3.12 isn't working b/c 3.12 is a moving target; evne since my comment above (#2738 (comment)) I'm seeing a different version being used when repro-ing:
I'm gonna pin to 3.11 and put a bit more time in to see if I can figure out why this has only started happening and if I should report it somewhere. |
We're not quite at the bottom of tektoncd#2738 but it seems like alpine 3.12 is having this problem and 3.11 is not. 3.12 seems to be still being updated which may be why we are seeing different behavior across runs using 3.12; for now pinning to 3.11 should be a way to be able to build our releases while we try to understand what's going on.
We're not quite at the bottom of tektoncd#2738 but it seems like alpine 3.12 is having this problem and 3.11 is not. 3.12 seems to be still being updated which may be why we are seeing different behavior across runs using 3.12; for now pinning to 3.11 should be a way to be able to build our releases while we try to understand what's going on
I think this is a bizarre collision of kaniko behaviour and alpine relying on I think our options are:
|
We're not quite at the bottom of #2738 but it seems like alpine 3.12 is having this problem and 3.11 is not. 3.12 seems to be still being updated which may be why we are seeing different behavior across runs using 3.12; for now pinning to 3.11 should be a way to be able to build our releases while we try to understand what's going on
I think it's just a perfect storm of conditions that could've happened in any prior alpine release, but by chance didn't. the base images for alpine 3.12 don't have the latest as soon as the alpine base images include that package upgrade, this issue will mostly disappear until the next perfect storm. 😆 |
ahh makes sense @joshsleeper ! thanks for explaining :D do you happen to know how one could track this kind of thing (e.g. are there release notes somewhere that mention this?) np if not, thanks anyway for the info |
for all the things people, including myself, love about Alpine Linux, I think it's a fairly small crew running that ship so their announcement processes aren't extensive. They have a "Latest Development" feed on their homepage that tracks package updates (which is really just a feed of commits), but I think that's about it? I think it's mostly a side-effect of the fact that by design alpine doesn't maintain package history really, so generally the only correct version of alpine packages to be using is the latest. if there are upgrades, you're supposed to have them full stop. when a major release is being cut (e.g. 3.11, 3.12, etc.) they commit to pin to specific package versions (let's say something like hopefully that's more helpful than man-splain-y. I've just had to dig into this before at my own company to understand why we had various odd issues with alpine that we never had with other distros. It's worth noting that I think this probably isn't a I'm still undecided on if this should be fixed by alpine or k8s, but I'm guessing it'll end up being alpine since other distros aren't having similar issues... that I know of. |
I think we've successfully working around this and it seems like GoogleContainerTools/kaniko#1297 probably won't have a solution for a while. Considering this resolved! |
Expected Behavior
Pipeline nightly build works
Actual Behavior
Pipeline nightly build is broken.
Building the base image fails with:
Steps to Reproduce the Problem
Additional Info
The image is base on alpine.
It used to be latest, and now it's pinned on 3.12 which is the version that was used in the last working run: https://dashboard.dogfooding.tekton.dev/#/namespaces/default/pipelineruns/pipeline-release-nightly-w5xcr
The only visible difference in the run log is the following. In the successful run:
while in the failing run:
The text was updated successfully, but these errors were encountered: