[BUG] build --push does not push when building for a single platform using a docker-container builder instance

[MatTheCat]

Description

I have a fpm service in a compose.yaml file which configures both build and image. The idea is that someone can build the image and push it to GHCR and other people just have to pull it.

In the build section, I had linux/amd64 and linux/arm64 under platforms. Then docker compose build --push fpm ends with

=> [fpm] exporting to image                                                                                                                                                                                                   5.2s
 => => exporting layers                                                                                                                                                                                                        0.0s
 => => exporting manifest sha256:fbe36cfeec0a7b39fa582e8deafe7c23f8008b411d6b585494fd6f1b93115697                                                                                                                              0.0s
 => => exporting config sha256:f76708d8ac763f0f9a128f92b2150322630fe71f03e985d2ad35775eb5f738c4                                                                                                                                0.0s
 => => exporting attestation manifest sha256:1499cc8f26c8078dca5a670f6370f4a18776be47b34c06d06f9610b8da31fd2f                                                                                                                  0.0s
 => => exporting manifest sha256:a2f4f5c5825bcb633e2743bdb3e9e06db74051c2ad394f8d9ddfd64d9798d8c6                                                                                                                              0.0s
 => => exporting config sha256:0f88783b5159dca5d8e705917d6258c7d5c027151e426b4a7ea067192108daa1                                                                                                                                0.0s
 => => exporting attestation manifest sha256:35f8a1b44db2c00fdcd0cf327e6dbcb4e55d50f4f113a79bfa525c8791057026                                                                                                                  0.0s
 => => exporting manifest list sha256:d8fc9235699fe74e2f88e333cc808fa75f07508ea1418f59b87fd925bed8451c                                                                                                                         0.0s
 => => pushing layers                                                                                                                                                                                                          2.4s
 => => pushing manifest for ghcr.io/…/local/fpm:latest@sha256:d8fc9235699fe74e2f88e333cc808fa75f07508ea1418f59b87fd925bed8451c                                                                            2.7s
 => [fpm auth] …/local/fpm:pull,push token for ghcr.io

As this took some time I dropped platforms, but then docker compose build --push fpm ended with

=> [fpm] exporting to docker image format                                                                                                                                                                                     1.6s
 => => exporting layers                                                                                                                                                                                                        0.0s
 => => exporting manifest sha256:acea847c73c12a7640eb942dfebffcdf5b39d8369805e08f9e42e58527343d16                                                                                                                              0.0s
 => => exporting config sha256:0f88783b5159dca5d8e705917d6258c7d5c027151e426b4a7ea067192108daa1                                                                                                                                0.0s
 => => sending tarball                                                                                                                                                                                                         1.6s
 => [fpm fpm] importing to docker

The behavior is the same if I configure a single platform, be it linux/amd64 (mine) or linux/arm64.

docker compose push fpm always work though.

Steps To Reproduce

I’m not sure what is the cause, but in my case having a single platform under a service’s build section is enough to prevent docker compose build --push to push the image.

Compose Version

$ docker compose version
Docker Compose version 2.20.3

$ docker-compose version
Docker Compose version 2.20.3

Docker Environment

$ docker info
Client:
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.11.2
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.20.3
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 22
  Running: 1
  Paused: 0
  Stopped: 21
 Images: 17
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 091922f03c2762540fd057fba91260237ff86acb.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.5-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 14.97GiB
 Name: tekyn-hp-pavilion
 ID: 2acdbff5-9fc9-4ad4-9dc3-154527fe7756
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[MatTheCat]

Same thing is happening with Compose 2.22.0

[MatTheCat]

No change with Compose 2.23.0 (Docker 24.0.7, Buildx 0.11.2).

[ndeloof]

Tested with latest codebase, can't reproduce:

$ cat compose.yaml 
services:
  test:
    build: .
    image: ndeloof/test:platform
 
$ docker compose build --push
[+] Building 7.6s (7/7) FINISHED                                                                                docker:desktop-linux
 => [test internal] load build definition from Dockerfile                                                                       0.0s
...
 => [test] pushing ndeloof/test:platform with docker                                                                            3.0s
 => => pushing layer 98beb4629e3a                                                                                               2.5s

I wonder this was fixed as we updated buildx version bundled by Docker Compose.
Can you please give https://github.com/docker/compose/releases/tag/v2.23.3 a try ?
(download binary and move it to ~/.docker/cli-plugins/docker-compose)

[MatTheCat]

Still no luck 😞

$ docker compose version
Docker Compose version 2.23.3

$ cat compose.yaml
services:
    test:
        build: .
        image: ghcr.io/[…]/test

$ docker compose build test --push
[+] Building 0.3s (4/4) FINISHED                                                                                                                        docker-container:lucid_easley
 => [test internal] load .dockerignore                                                                                                                                           0.1s
 => => transferring context: 2B                                                                                                                                                  0.0s
 => [test internal] load build definition from Dockerfile                                                                                                                        0.1s
 => => transferring dockerfile: 135B                                                                                                                                             0.0s
 => [test] exporting to docker image format                                                                                                                                      0.1s
 => => exporting layers                                                                                                                                                          0.0s
 => => exporting manifest sha256:94cb73ac9f02afbbb8bef54381135f6e7d219ca38b6ef5a02f79d7c567ad9b93                                                                                0.0s
 => => exporting config sha256:734400a254924dbeafa357043ff90435990700d710c4ba011b180bea2d6a674e                                                                                  0.0s
 => => sending tarball                                                                                                                                                           0.0s
 => [test] importing to docker

Only difference seems to be the container registry 🤔

[ndeloof]

weird. Using ghcr (while I'm not logged in) I still get a push attempt:

$ docker compose version
Docker Compose version v2.23.3
$ docker compose build --push
[+] Building 7.1s (6/6) FINISHED                                                                                docker:desktop-linux
 => [test internal] load .dockerignore                                                                                          0.0s
...
 => ERROR [test] pushing ghcr.io/ndeloof/test:platform with docker                                                              1.5s
 => => pushing layer 98beb4629e3a                                                                                               1.1s
------
 > [test] pushing ghcr.io/ndeloof/test:platform with docker:
------
unauthorized: unauthenticated: User cannot be authenticated with the token provided.

Can you please check the active builder by running docker buildx ls ?

[MatTheCat]

Sure, thanks for your time 🙏

$ docker buildx ls
NAME/NODE       DRIVER/ENDPOINT             STATUS  BUILDKIT             PLATFORMS
lucid_easley *  docker-container                                         
  lucid_easley0 unix:///var/run/docker.sock stopped                      
default         docker                                                   
  default       default                     running v0.11.7+d3e6c1360f6e linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

I recall docker compose not wanting to use default so I had to create another one.

Edit: after trying to build --push I get

NAME/NODE       DRIVER/ENDPOINT             STATUS  BUILDKIT             PLATFORMS
lucid_easley *  docker-container                                         
  lucid_easley0 unix:///var/run/docker.sock running v0.11.5              linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default         docker                                                   
  default       default                     running v0.11.7+d3e6c1360f6e linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

docker buildx version outputs github.com/docker/buildx 0.12.0 542e5d810e4a1a155684f5f3c5bd7e797632a12f though.

[ndeloof]

I was able to reproduce this issue when using a docker-container builder instance, while docker build -t ndeloof/test:platform --push . succeed. I'll investigate what's wrong with compose way to configure buildx

[MatTheCat]

I confirm that switching back to the default builder, build --push behaves as expected for me 👌

[ndeloof]

the root cause for this issue is that buildkit doesn't allow to configure multiple exporters. When using the default builder (implemented by docker engine) the load attribute is just ignored (as the local image store is used to store the built image) and the push attribute is respected.

Also see moby/buildkit#1555

[MatTheCat]

Why would Compose not respect push using a docker-container builder? 🤔

[ndeloof]

this is not compose, but buildkit API. Only one "exporter" can be configured to move the built result to out from builder. Can be either a registry ("push") or Docker engine local image cache ("load"). When using the "default" builder (implemented by docker engine) "load" is not required as image is built inside local image cache, so "push" export image to registry in addition to the image being present in local cache. When using an alternative builder (docker-container, or cloud-based) we have no way to configure both exporter, until moby/buildkit#1555 allows to do

[MatTheCat]

Thanks for the explanation, I understand now!

Should I close this issue as it provides a workaround (using the default docker builder) and will be fixed in another repo?

[ndeloof]

let's indeed close this issue, as there's anyway nothing we can do until upstream offers a way to workaround this limitation

[guyarad]

@ndeloof thanks for providing explanations! However, I still don't really understand. In my compose file I have few services, each has a build section, with a platforms section field. The build.tags is a list of remote tags. If multiple platforms are configured compose-build will push to the remote registry. If a single platform is configured, it will attempt loading the image. Not sure I understand why would compose-build select the default exporter instead of the customer builder.

[ndeloof]

@guyarad it uses the same buildkit builder in both case, but the build parameters differ as multi-platform is enabled. In such circumstances the builder won't handle the push flag in the same way as docker engine does not support multi-platform images in his image store.

[guyarad]

@ndeloof thanks for the quick response. I'm still a little confused though. Why would build parameters be different? who is passing those build parameters?
I'm asking because it seems like docker-compose-build command should generate those build parameters. And in that case, the bug is indeed in docker-compose and not buildkit.

[ndeloof]

yes indeed, docker compose build actually just creates build parameters sent to buildkit. But we have no control on how buildkit do manage those. The way compose expect the build to complete is by image to be available to be ran (this is docker compose purpose), and as such load is required. But when using multi-platform this requires an addition exporter step for buildkit to manage this specific build combination, as docker engine can't store multi-platform images.
If you want to get more control on the build process, you should look at buildx and bake.

[guyarad]

Is it possible to see the actual parameters being passed to buildkit via docker compose build?

[MatTheCat]

@ndeloof I’m not sure, but did moby/buildkit#4134 unlock this issue?