New docker-container builders fail first bake using local cache #1325
Comments
|
I got a stacktrace to better understand this issue So the case is that first builds loads cache but it remains only a lazy ref https://github.com/moby/buildkit/blob/v0.10.4/cache/remote.go#L336 created with provider from session. Then a second build comes in when the first session is already dropped and matches against the previous lazy ref. Then unlazy gets called and fails because of the session is already gone. I guess the simplest fix is to try to disable lazy behavior for local cache imports from session because it seems fragile. More proper fixes would be to make sure lazy ref is not matched if it is a different session or add the current session to the group(not sure if this is quite safe actually). On bake we might need a fix as well to keep the original session alive until all builds have completed. I'm thinking of the case where a "local source" would need to be pulled in by a subsequent build(not sure how practical). But I think this cache issue could appear by just doing two individual builds with same cache source from two different terminals. |
|
Not sure, if it's related to this issue. But since the upgrade to Docker version 23.0.0 which embeds buildx 0.10.2 as its default builder, some people are encountering issues during building a devcontainer (a feature from VSCode). The error |
|
@jaudiger I don't think this is related. Can you show the output of |
|
@crazy-max While working on a small repro which can be done with this Dockerfile (Dockerfile) and this command: I found the culprit. If I remove the option: |
I can confirm this new error. Happens on gitlab CI/CD with docker:dind service when building docker container. Started happening today, with no changes to any docker/CI or related files. |
|
We're also seeing the same error suddenly appear despite not changing anything CI/CD or Docker related on our end for months. GitLab CI/CD running dind with Docker 20.10.13 with |
|
Confirmed same error here. |
|
Same error here, works nice yesterday and today launching devcontainer I got the error: "ERROR: failed to solve: Unavailable: error reading from server: EOF" and "ERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF"... I scoured the internet looking for a solution, but so far I haven't found anything that would help me, I even formatted my ubuntu but the error keep error logs: on this repo: launch just when trying to launch devcontainer, the microsoft extension for vscode (using |
|
@jaudiger @bwenzel2 @m-melis @antonioconselheiro Same as moby/buildkit#3576, will be fixed with moby/moby#44920. Closing this issue since it has been fixed in BuildKit 0.11.2 (moby/buildkit#3493) |
|
@crazy-max has there been a new docker image release with the fix? |
|
@denibertovic For this issue, it's already fixed with BuildKit 0.11.2 which is already released. For moby/moby#44920 it will be on next Moby patch release (23.0.1). |
|
@crazy-max I understand. I'm using the official docker images from docker hub in my CI/CD pipeline. And from what I can tell 23.0.1 was not released yet. The latest one that's pushed seems to be 23.0.0. |
|
This issue completely breaks my ability to build and run any docker devcontainer in VS Code on Ubuntu 22.04. I've been able to block it in the This was a new install. Firs time in a while. This will probably seriously disrupt a lot of people who use devcontainers in Docker. |
Damn where were you when this issue first started and I was editing VSC code LOL |
|
Is this still expected to be a problem? I'm encountering this using:
When I do the exact same build on a Linux VM, I don't hit this issue. |
I am still getting the error when the Buildkit in my case is v0.13.2 |
Behavior
Using a fresh
buildxdocker-container builder, abakeusing a (populated) local cache and a build context (i.e.COPY,RUN --mount, etc.) will fail with one ofERROR: failed to solve: Canceled: grpc: the client connection is closingorERROR: failed to solve: Unavailable: error reading from server: EOFDesired behavior
The first build with a fresh builder must succeed against a local cache for practical use of the local cache in CI applications. With a builder that has already
baked the images, this issue become intermittent. That case should also succeed consistently.Environment
docker info:Steps to reproduce
Prepare the files (unzip this to skip):
base/Dockerfile:layer/Dockerfile:images.json:{ "target": { "common": { "platforms": [ "linux/amd64" ] }, "base": { "context": "base", "cache-from": [ "type=local,src=../cache/base" ], "cache-to": [ "type=local,mode=max,dest=../cache/base" ], "inherits": ["common"], "tags": [ "base" ] }, "layer": { "context": "layer", "cache-from": [ "type=local,src=../cache/layer" ], "cache-to": [ "type=local,mode=max,dest=../cache/layer" ], "contexts": { "base_target": "target:base" }, "inherits": ["common"], "tags": [ "layer" ] } } }Create the builder:
Populate the cache:
For each subsequent test, remove the builder, recreate it, and rebuild the
baketargets:Each such test fails with
ERROR: failed to solve: Canceled: grpc: the client connection is closingorERROR: failed to solve: Unavailable: error reading from server: EOF.image_defs.zip
The text was updated successfully, but these errors were encountered: