v6.4.0 breaks GitHub Actions cache

[TastyPi]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

We've been seeing errors whenever our workflow tries to export layers to the GitHub cache, e.g.:

#37 exporting to GitHub Actions Cache
#37 preparing build cache for export
#37 writing layer sha256:02e1e64c0260323cc3d99a8927d5ecf7bff2f32154352f1822ff9990ee9c0b98
#37 writing layer sha256:02e1e64c0260323cc3d99a8927d5ecf7bff2f32154352f1822ff9990ee9c0b98 0.2s done
#37 writing layer sha256:046ed2589a65eca532f9472df4530273d6b0abd0c9b5985659d6e90b28e28ee5
ERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

------
 > exporting to GitHub Actions Cache:
------
ERROR: failed to solve: error writing layer blob: Patch "https://acghubeus1.actions.githubusercontent.com/Odv8msvJgMS5iWxgU43K2EnWLcwj1p7E8K5nLVIfXGs2O9tj1t/_apis/artifactcache/caches/205053": read tcp 172.17.0.2:57224->20.60.63.33:443: use of closed network connection

Fixing the action to v6.3 instead of v6 fixed the issue.

Expected behaviour

Exporting to GitHub action cache should work

Actual behaviour

Exporting to GitHub action cache always fails

Repository URL

No response

Workflow run URL

No response

YAML workflow

We actually use a composite action

runs:
  using: composite
  steps:
    - uses: docker/setup-buildx-action@v3
    - name: Docker build
      uses: docker/build-push-action@v6
      with:
        context: .
        tags: ${{ inputs.tag }}
        push: ${{ inputs.push }}
        provenance: false
        build-args: REDACTED
        cache-from: type=gha
        cache-to: type=gha,mode=max

Workflow logs

This is just the logs for this action with bits redacted.

3_Run ..github_actions_docker_build.txt

BuildKit logs

No response

Additional info

No response

[norrisjeremy]

We too are seeing similar issues for one repo of ours.
Yet seemingly other repos we have do not experience this issue after updating to v6.4.0.

[crazy-max]

Thanks for reporting, can you enable debug for the BuildKit container in docker/setup-buildx-action@v3 step and give us logs output please?

[norrisjeremy]

Hi @crazy-max,

Here is a redacted build log from one of our private repos exhibiting this issue with debug enabled.

docker_build-push-action.txt

Thanks,
Jeremy

[crazy-max]

Here is a redacted build log from one of our private repos exhibiting this issue with debug enabled.

docker_build-push-action.txt

Would need BuildKit logs from post step, see https://docs.docker.com/build/ci/github-actions/configure-builder/#buildkit-container-logs

[norrisjeremy]

Hi @crazy-max,

Do you mean the logs for the post docker/setup-buildx-action step or the logs for the post docker/build-push-action step?

Thanks,
Jeremy

[crazy-max]

docker/setup-buildx-action yes

[norrisjeremy]

Hi @crazy-max,

Here is redacted log for the post docker/setup-buildx-action step.

Post docker_setup-buildx-action.txt

Thanks,
Jeremy

[crazy-max]

Thanks, here are the relevant part of the logs:

2024-07-16T17:00:58.9862132Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A25749db20348e6351d5f6f0377fd90683b7c9c38cb8c5e8828b64418414f9d2d&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9868519Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A3713021b02770a720dea9b54c03d0ed83e03a2ef5dce2898c56a327fee9a8bca&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9875706Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Ac205e3cd6d4fe77e75ac773d2095d5b077da4e409208e91a437736ac3516bf8a&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9881773Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A9d979625e03710120862618df652a44147c39cea63f81fc3317150576d13970d&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9887767Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A5e1d8a175912f23735362c719758739f9820779a4c3e65183d14a78a62c4d385&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9894218Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Ac1e08517414cd778c7f615a79a994f56eb3cd786e6a1d18327cfc33033d37fb7&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9900522Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Afd5a4f3a56d5413c8dac531df69282453b580858b02db963336b39e7e13fb2fc&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9907061Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A9674d23d14435baed80cb7b4d2f79eed4fd8a7b170e1cd9c904cdabc14c478d5&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0027843Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A740a1bf374b050206a2e7dbebe3fc39f28b07ad5a7d3ff01ca94e381633f93c1&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0034235Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A334a3bdc2a7bbe4ddc90029152da641bbba2e051aa401859fb9ec6ee84589279&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0040924Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A4ce000a43472e4a2527834764b5044674760f1e2a766480798d03a93b51a0b39&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0047214Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Ab4ea33a8f66c0b2df9875ba27c31925f1981f302f57fc9dfee62449cb5f23b3c&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0053463Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A34ce04bd839bdd14be00b3ecc06ab94cd6a74063954fe7007d30f1b3607efead&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0059767Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A09ac36fdd82d2b54719ef1b6d4b31b34e8e4a1000cb4ffad9dae86c8a0b17db6&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0066136Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Af836a8b5317b9de84ae0e8852420f7067e99c33a94281958bc1be35e106d18de&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0072403Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Aa4d2191ac6530b4133830d8ca3369ea70c6382652778b846329993ed82ffd279&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0086340Z time="2024-07-16T17:00:50Z" level=debug msg="save cache req https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches body={\"key\":\"buildkit-blob-1-sha256:09ac36fdd82d2b54719ef1b6d4b31b34e8e4a1000cb4ffad9dae86c8a0b17db6\",\"version\":\"693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2\"}"
2024-07-16T17:00:59.0090243Z time="2024-07-16T17:00:50Z" level=debug msg="save cache resp: {\"cacheId\":1421}"
2024-07-16T17:00:59.0097314Z time="2024-07-16T17:00:50Z" level=debug msg="upload cache chunk https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/1421, range 0-237"
2024-07-16T17:00:59.0135751Z time="2024-07-16T17:00:50Z" level=debug msg="commit cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/1421, size 238"
2024-07-16T17:00:59.0146114Z time="2024-07-16T17:00:51Z" level=debug msg="save cache req https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches body={\"key\":\"buildkit-blob-1-sha256:18ebc6ff070748ff10785729f9b5b7a953c708f6c25b5ff759cd83136eab965b\",\"version\":\"693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2\"}"
2024-07-16T17:00:59.0149916Z time="2024-07-16T17:00:51Z" level=debug msg="save cache resp: {\"cacheId\":1422}"
2024-07-16T17:00:59.0152425Z time="2024-07-16T17:00:51Z" level=debug msg="upload cache chunk https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/1422, range 0-1536"
2024-07-16T17:00:59.0166115Z time="2024-07-16T17:00:51Z" level=debug msg="commit cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/1422, size 1537"
2024-07-16T17:00:59.0174606Z time="2024-07-16T17:00:51Z" level=debug msg="save cache req https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches body={\"key\":\"buildkit-blob-1-sha256:25749db20348e6351d5f6f0377fd90683b7c9c38cb8c5e8828b64418414f9d2d\",\"version\":\"693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2\"}"
2024-07-16T17:00:59.0180806Z time="2024-07-16T17:00:51Z" level=debug msg="save cache req https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches body={\"key\":\"buildkit-blob-1-sha256:277e32004e30b487b4bd089b32d8b35c1698a9379a81a4bb5bab75b16554457e\",\"version\":\"693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2\"}"
2024-07-16T17:00:59.0184251Z time="2024-07-16T17:00:51Z" level=debug msg="save cache resp: {\"cacheId\":***}"
2024-07-16T17:00:59.0186849Z time="2024-07-16T17:00:51Z" level=debug msg="upload cache chunk https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/***, range 0-33554431"
2024-07-16T17:00:59.0197410Z time="2024-07-16T17:00:51Z" level=debug msg="upload cache chunk https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/***, range 33554432-51895362"
2024-07-16T17:00:59.0203898Z time="2024-07-16T17:00:53Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = error writing layer blob: Patch \"https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/***\": http: read on closed response body"
2024-07-16T17:00:59.0213290Z error writing layer blob: Patch "https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/***": http: read on closed response body
2024-07-16T17:00:59.0216172Z 7 v0.14.1 buildkitd --debug --allow-insecure-entitlement=network.host
2024-07-16T17:00:59.0217227Z github.com/tonistiigi/go-actions-cache.(*Cache).doWithRetries
2024-07-16T17:00:59.0218231Z 	/src/vendor/github.com/tonistiigi/go-actions-cache/cache.go:455
2024-07-16T17:00:59.0219446Z github.com/tonistiigi/go-actions-cache.(*Cache).uploadChunk
2024-07-16T17:00:59.0220405Z 	/src/vendor/github.com/tonistiigi/go-actions-cache/cache.go:410
2024-07-16T17:00:59.0221346Z github.com/tonistiigi/go-actions-cache.(*Cache).upload.func1
2024-07-16T17:00:59.0222352Z 	/src/vendor/github.com/tonistiigi/go-actions-cache/cache.go:314
2024-07-16T17:00:59.0223168Z golang.org/x/sync/errgroup.(*Group).Go.func1
2024-07-16T17:00:59.0223919Z 	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
2024-07-16T17:00:59.0224582Z runtime.goexit
2024-07-16T17:00:59.0225059Z 	/usr/local/go/src/runtime/asm_amd64.s:1650
2024-07-16T17:00:59.0225495Z 
2024-07-16T17:00:59.0226022Z 7 v0.14.1 buildkitd --debug --allow-insecure-entitlement=network.host
2024-07-16T17:00:59.0227023Z github.com/tonistiigi/go-actions-cache.(*Cache).uploadChunk
2024-07-16T17:00:59.0228035Z 	/src/vendor/github.com/tonistiigi/go-actions-cache/cache.go:412
2024-07-16T17:00:59.0229033Z github.com/tonistiigi/go-actions-cache.(*Cache).upload.func1
2024-07-16T17:00:59.0230129Z 	/src/vendor/github.com/tonistiigi/go-actions-cache/cache.go:314
2024-07-16T17:00:59.0231020Z golang.org/x/sync/errgroup.(*Group).Go.func1
2024-07-16T17:00:59.0231800Z 	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
2024-07-16T17:00:59.0232506Z runtime.goexit
2024-07-16T17:00:59.0232985Z 	/usr/local/go/src/runtime/asm_amd64.s:1650
2024-07-16T17:00:59.0233436Z 
2024-07-16T17:00:59.0233984Z 7 v0.14.1 buildkitd --debug --allow-insecure-entitlement=network.host
2024-07-16T17:00:59.0235383Z github.com/moby/buildkit/cache/remotecache/gha.(*exporter).Finalize
2024-07-16T17:00:59.0236295Z 	/src/cache/remotecache/gha/gha.go:230
2024-07-16T17:00:59.0237170Z github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1.1
2024-07-16T17:00:59.0238135Z 	/src/solver/llbsolver/solver.go:693
2024-07-16T17:00:59.0239006Z github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1
2024-07-16T17:00:59.0239929Z 	/src/solver/llbsolver/solver.go:1064
2024-07-16T17:00:59.0240634Z github.com/moby/buildkit/solver.(*Job).InContext
2024-07-16T17:00:59.0241322Z 	/src/solver/jobs.go:808
2024-07-16T17:00:59.0242035Z github.com/moby/buildkit/solver/llbsolver.inBuilderContext
2024-07-16T17:00:59.0242874Z 	/src/solver/llbsolver/solver.go:1060
2024-07-16T17:00:59.0243741Z github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1
2024-07-16T17:00:59.0244655Z 	/src/solver/llbsolver/solver.go:674
2024-07-16T17:00:59.0245304Z golang.org/x/sync/errgroup.(*Group).Go.func1
2024-07-16T17:00:59.0246047Z 	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
2024-07-16T17:00:59.0246717Z runtime.goexit
2024-07-16T17:00:59.0247160Z 	/usr/local/go/src/runtime/asm_amd64.s:1650
2024-07-16T17:00:59.0247594Z 
2024-07-16T17:00:59.0248146Z 7 v0.14.1 buildkitd --debug --allow-insecure-entitlement=network.host
2024-07-16T17:00:59.0248944Z main.unaryInterceptor
2024-07-16T17:00:59.0249401Z 	/src/cmd/buildkitd/main.go:716
2024-07-16T17:00:59.0250413Z google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
2024-07-16T17:00:59.0251587Z 	/src/vendor/google.golang.org/grpc/server.go:1154
2024-07-16T17:00:59.0252524Z github.com/moby/buildkit/api/services/control._Control_Solve_Handler
2024-07-16T17:00:59.0253429Z 	/src/api/services/control/control.pb.go:2466
2024-07-16T17:00:59.0254130Z google.golang.org/grpc.(*Server).processUnaryRPC
2024-07-16T17:00:59.0254895Z 	/src/vendor/google.golang.org/grpc/server.go:1343
2024-07-16T17:00:59.0255615Z google.golang.org/grpc.(*Server).handleStream
2024-07-16T17:00:59.0256337Z 	/src/vendor/google.golang.org/grpc/server.go:1737
2024-07-16T17:00:59.0257120Z google.golang.org/grpc.(*Server).serveStreams.func1.1
2024-07-16T17:00:59.0258229Z 	/src/vendor/google.golang.org/grpc/server.go:986
2024-07-16T17:00:59.0258918Z runtime.goexit
2024-07-16T17:00:59.0259416Z 	/usr/local/go/src/runtime/asm_amd64.s:1650

@tonistiigi Looking at the error http: read on closed response body, it seems to be a resource handling issue which might be transient, so it could make sense to retry?

I will revert #1133 in the meantime while this is being investigating.

[crazy-max]

Change has been reverted in latest 6.4.1 https://github.com/docker/build-push-action/releases/tag/v6.4.1

[crazy-max]

tracked in moby/buildkit#5162