unable to push images to harbor registry

[arun-vc]

I'm currently running tektoncd pipelines that contains task: https://github.com/tektoncd/catalog/blob/main/task/buildkit-daemonless/0.1/buildkit-daemonless.yaml which builds images inside the container with buildkit. When pushing to harbor i've only been able to push larger images (~1.2Gb) in around 2/20 attempts. Images around 25Mb are built and pushed almost instantly.

I run into the same issue when using the deployment and service example: https://github.com/moby/buildkit/blob/master/examples/kubernetes/deployment%2Bservice.rootless.yaml

time="2022-11-30T10:07:35Z" level=debug msg="stopping session" spanID=xxxxxxxxxx traceID=xxxxxxxx

#17 ERROR: failed to push harbor.xxx.xxx.xxxx/scipy:4bd9b68: failed to copy: io: read/write on closed pipe

or

#10 exporting to image
#10 1008.1 error: failed commit on ref
read tcp  xxx.xxxx->x.xxxx:443: read: connection reset by peer

Point to note:

I had no issues building and pushing similar size images to the same harbor registry when using docker in docker

[GusPrice]

My team has been running into a similar issue starting a week or so ago, but on Google Artifact Registry. I was about to write up my own issue, but figured I'd add here instead.

  time="2022-12-01T19:47:06Z" level=debug msg="do request" digest="sha256:XXXXXXXXXXXXXXXX" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.content-type=application/octet-stream request.header.user-agent=buildkit/0.0.0+unknown request.method=PUT size=21216758 url="https://us-docker.pkg.dev/artifacts-uploads/namespaces/example/uploads/XXXX=?digest=sha256%XXXXXXXXXXXXXXXXXXXX"
  time="2022-12-01T19:47:06Z" level=debug msg=push digest="sha256:XXXXXXXXXXXXXXXXXXXXX" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=31412630
  time="2022-12-01T19:47:06Z" level=debug msg="checking and pushing to" digest="sha256:XXXXXXXXX" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=31412630 url="https://us-docker.pkg.dev/v2/example/blobs/sha256:XXXXXXXXXXXXXXXX"
  time="2022-12-01T19:47:06Z" level=debug msg="do request" digest="sha256:XXXXXXXXXXXXXXXX" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=buildkit/0.0.0+unknown request.method=HEAD size=31412630 url="https://us-docker.pkg.dev/v2/example/blobs/sha256:XXXXXXXXXXXXXXXX"
  time="2022-12-01T19:47:06Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to push us-docker.pkg.dev/example:XXXXXXXXXXXXXXXX: failed to copy: io: read/write on closed pipe"
  failed to push us-docker.pkg.dev/example:XXXXXXXXXXXXXXXX: failed to copy: io: read/write on closed pipe
    1 18879bc buildkitd --debug
  github.com/moby/buildkit/exporter/containerimage.(*imageExporterInstance).Export
  	/src/exporter/containerimage/export.go:304
  github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func6
  	/src/solver/llbsolver/solver.go:273
  github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1
  	/src/solver/llbsolver/solver.go:615
  github.com/moby/buildkit/solver.(*Job).InContext
  	/src/solver/jobs.go:593
  github.com/moby/buildkit/solver/llbsolver.inBuilderContext
  	/src/solver/llbsolver/solver.go:611
  github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve
  	/src/solver/llbsolver/solver.go:272
  github.com/moby/buildkit/control.(*Controller).Solve
  	/src/control/control.go:382
  github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1
  	/src/api/services/control/control.pb.go:2242
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
  go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
  	/src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:325
  main.unaryInterceptor.func1
  	/src/cmd/buildkitd/main.go:569
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
  github.com/moby/buildkit/api/services/control._Control_Solve_Handler
  	/src/api/services/control/control.pb.go:2244
  google.golang.org/grpc.(*Server).processUnaryRPC
  	/src/vendor/google.golang.org/grpc/server.go:1340
  google.golang.org/grpc.(*Server).handleStream
  	/src/vendor/google.golang.org/grpc/server.go:1713
  google.golang.org/grpc.(*Server).serveStreams.func1.2
  	/src/vendor/google.golang.org/grpc/server.go:965
  runtime.goexit
  	/usr/local/go/src/runtime/asm_amd64.s:1594
1 18879bc buildkitd --debug
  main.unaryInterceptor.func1
  	/src/cmd/buildkitd/main.go:573
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
  github.com/moby/buildkit/api/services/control._Control_Solve_Handler
  	/src/api/services/control/control.pb.go:[224](https://github.com/example/actions/runs/xxxxxx/jobs/xxxxxxxxxx#step:6:225)4
  google.golang.org/grpc.(*Server).processUnaryRPC
  	/src/vendor/google.golang.org/grpc/server.go:1340
  google.golang.org/grpc.(*Server).handleStream
  	/src/vendor/google.golang.org/grpc/server.go:1713
  google.golang.org/grpc.(*Server).serveStreams.func1.2
  	/src/vendor/google.golang.org/grpc/server.go:965
  runtime.goexit
  	/usr/local/go/src/runtime/asm_amd64.s:1594
  
  time="2022-12-01T19:47:06Z" level=debug msg="session finished: <nil>"

[mqasimsarfraz]

We are also recently seeing something similar in our workflows:

ERROR: failed to solve: failed to push ghcr.io/inspektor-gadget/inspektor-gadget: failed to copy: io: read/write on closed pipe
Error: buildx failed with: ERROR: failed to solve: failed to push ghcr.io/inspektor-gadget/inspektor-gadget: failed to copy: io: read/write on closed pipe

I can't think of anything that changed recently except maybe we started using buildkit v0.11.0 in CI.

[eiffel-fl]

Hi.

My team has been running into a similar issue starting a week or so ago, but on Google Artifact Registry. I was about to write up my own issue, but figured I'd add here instead.

Are you able to reproduce this behavior all the time or does it occur sporadically?
In the first case, could you please provide a reproducer?

Best regards.

[jonathangreen]

I'm also seeing this issue when pushing to github container registry (ghcr.io). It seems to happen sporadically, but reasonably frequently. Causing 1/4 of our pushes to fail. We're only pushing to ghcr.io, so I can't say if its isolated there or not.

https://github.com/ThePalaceProject/circulation/actions/runs/3931836023/jobs/6724193000#step:8:851

[akhilerm]

Is there some publicly image available with which I can reproduce this issue easily. For eg: pushing the said image to ghcr.io has a high chance of hitting this bug. The current image that I use for testing is an oracle database image which is ~ 3.3GiB and I am hitting the issue only 2/10 tries.

What I plan to do :

1. Fetch the image from dockerhub (which I assume users are not hitting the issue)
2. Push the image to ghcr.io (instead of using buildx, will use containerd directly)

[corinz]

Same issue, root cause still unknown?

[almereyda]

It is said this is caused by

• "failed to copy: io: read/write on closed pipe" on ctr images push while pushing large images containerd/containerd#7972

In docker/build-push-action#761 (comment) also the GitHub Packages team suggests to workaround by downgrading buildkit to v1.10.6, until upstream is resolved.

[jedevc]

Closing this issue, since we're tracking this in docker/build-push-action#761 🎉