Support dependencies when using bake

[howardjohn]

It would be useful for bake to automatically determine dependencies during the build phase. For example:

==> docker-bake.hcl <==
target "sleep" {
    tags = ["localhost:5000/sleep"]
    dockerfile = "Dockerfile.sleep"
    output = ["type=registry"]
}
target "main" {
    tags = ["localhost:5000/main"]
    dockerfile = "Dockerfile.main"
    output = ["type=registry"]
}
group "all" {
    targets = ["sleep","main",]
}

==> Dockerfile.main <==
FROM localhost:5000/sleep

==> Dockerfile.sleep <==
FROM alpine

RUN sleep 10

On a fresh build, this will fail with failed to solve: rpc error: code = Unknown desc = failed to load cache key: localhost:5000/sleep:latest: not found

On a build where sleep already exists, we may still want this ordering, to ensure we are getting the newly built sleep rather than the existing remote one.

[tonistiigi]

We can't locally scan the contents of the dockerfile as they may have contents we don't understand. Also even with delaying the main target this will only work if the builder implements a local imagestore(currently only true for docker daemon driver).

What we could do is to allow to set inputs to main target that are outputs of another target.

target "main" {
  inputs {
    sleep = "sleep" // or "localhost:5000/sleep" = "sleep"
  }
}

FROM localhost:5000/sleep AS sleep

// ^ sleep is now either this image or input passed by name to the build

There are also plans to allow linking Dockerfiles together that also would solve this issue https://docs.google.com/document/d/1IafaByYR_Ao5ENw8bvIh8rbSbyh-mCAieNRrUOD3tAA/edit

[howardjohn]

Also even with delaying the main target this will only work if the builder implements a local imagestore(currently only true for docker daemon driver)

Couldn't it pull them from remote registry as well? Maybe its a bit more disconnected, but it seems like it would work from a technical standpoint. Rather than "wait for this image to appear in my local store, then use it" its "wait for this image which I know is going to be pushed to the remote store, then use it"

Thanks for the link to the doc!

[tonistiigi]

Yes, if the target that is requested pushes to the registry and nothing else accesses that registry location before then your approach would work as well (except the auto-detection) and they don't replace each other. For your example of linking stages this looks like a bad solution though. More useful for a case like "only push after tests complete".

[paulbsch]

Unless I'm missing something, I would think a simple "depends-on" field for targets would suffice:

target "sleep" {
    tags = ["localhost:5000/sleep"]
    dockerfile = "Dockerfile.sleep"
    output = ["type=registry"]
}
target "main" {
    tags = ["localhost:5000/main"]
    dockerfile = "Dockerfile.main"
    output = ["type=registry"]
    depends-on = ["sleep"]
}
group "all" {
    targets = ["sleep","main",]
}

Then the build of the "main" target would not start until all of the targets listed in "depends-on" have completed building. In this case, that would obviously just be "sleep".

However it would be implemented, this would be very useful for one of the projects I am working on.

[felipecrs]

This is related to #141.

[eitsupi]

It would be useful to be able to define things in multiple Dockerfiles and a single docker-bake.hcl that can be done in a multi-stage build of a single Dockerfile.

[eitsupi]

It seems be resolved by v0.8.0-rc1?
https://github.com/docker/buildx/blob/v0.8.0-rc1/docs/reference/buildx_bake.md#using-a-result-of-one-target-as-a-base-image-in-another-target