Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changing the structure of the section #15164

Merged
merged 9 commits into from
Jul 27, 2022
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions _data/toc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1392,11 +1392,14 @@ manuals:
title: Hello Build
- sectiontitle: Buildx
section:
- path: /build/buildx/
title: Working with Buildx
- path: /build/buildx/install/
title: Install Buildx

- path: /build/buildx/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about the order, looks confusing to me to have the parent page after a subpage.

As I can see with compose it's the first page: https://docs.docker.com/compose/ so we should be consistent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we discussed I wasn't aware you saw this whole page as a "Build Overview", sure let's rename that page and reorder the pages, then, so 1) Buildx Overview, 2) Install Buildx for consistency.

Copy link
Member

@crazy-max crazy-max Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe Overview and Install is enough as it's already subpages of Buildx section. Don't think we need to repeat ourselves?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean, that there's hierarchy providing context. But when we need to derive it from the folder context and there's multiple in the doc set, I think it's best for any "just" 'overview' or 'install' to be the ones for the main section (so doc set root index or install) for clear disambiguation in a glance.

title: Working with Buildx
- path: /build/buildx/multiple-builders/
title: Using multiple builders
- path: /build/buildx/multiarch-images/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be consistent:

Suggested change
- path: /build/buildx/multiarch-images/
- path: /build/buildx/multi-platform-images/

Needs to rename multiarch-images.md to multi-platform-images.md

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's true. It's called "multi-arch" in Desktop docs, and in conversation we do say multi-arch all the time.
I think most users would equate multi-platform with macOs/Linux/Windows.
And architectures is clear that it refers to HW such as arm, mips, ppc64le, and even s390x.

Also, taking for example this sentence in the original content:
"When you invoke a build, you can set the --platform flag to specify the target platform for the build output, (for example, linux/amd64, linux/arm64, or darwin/amd64)" or this command example "docker buildx build --platform linux/amd64,linux/arm64 "?
We also say "An experimental package is available for arch-based distributions." in Engine doc set.

Just trying to point out that it seems to be a combination of both. Platform is not enough to determine the arch (following the above interpretation): imho multi-architecture being more specific is a better candidate.

Copy link
Member

@crazy-max crazy-max Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum right, WDYT @tonistiigi @jedevc @thaJeztah?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've noticed this inconsistency as well, agreed it would be good to clarify this, but maybe out of scope for this PR, there's a lot of build docs that mix-and-match between the two, so I'd propose to pick something here that works, and then discuss separately later.

Personally, I think multi-platform is better when talking about building images. A couple reasons:

  • This is what the OCI spec refers to - the CPU architecture is part of the platform (the OS is the other main component).
  • Buildkit will eventually (hopefully) support building windows images. Say we have an amd64 linux image and a amd64 windows image - in this case, it's not as accurate to talk about multi-arch images, even though the underlying mechanism is the same. Multi-platform is a nice catch-all.
  • The buildx/buildkit flags are --platform - calling it multi-arch seems like it can be confusing, multi-platform seems more inline with existing behavior.

Maybe there are still use-cases for talking about multi-arch though, for example, maybe around QEMU, when we're using linux throughout, but emulating different CPU architectures.

An experimental package is available for arch-based distributions

I think this is probably unrelated, I think this refers to ArchLinux, not multi-architecture/platform support.

Copy link
Contributor Author

@dockertopia dockertopia Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on the Engine example. I revisited it and @jedevc is right, but a simple docs search renders this
image
image
( some are community hits)
And searching multi-platform results in a very different search results list.

It seems for the discussion there's the case for both in content, when platform is the same and we do want to distinguish architectures, but that we'd like to introduce the feature as multi-platform as it's seen as capturing both.

But it leaves the case of harmonizing the remaining docs content.

title: Building multi-platform images
- sectiontitle: Docker Compose
section:
- path: /compose/
Expand Down
113 changes: 2 additions & 111 deletions build/buildx/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Buildx
title: Working with Buildx
description: Working with Docker Buildx
keywords: build, buildx, buildkit
keywords: build, buildx, BuildKit
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keywords should be lowercased to me (cc @usha-mandya @thaJeztah)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, this was a suggestion by the IDE. Makes sense to have a standard. Just out of curiosity, does it need to be lowercase, i.e., otherwise something is broken?

Copy link
Member

@thaJeztah thaJeztah Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question; I think either way may work, as long as we're consistently using the same.

Looks like the search auto-complete uses them case-insensitive, but it may be that they're used different elsewhere;

Screenshot 2022-07-26 at 11 04 03

"elsewhere" could be as part of an URL and/or anchor, in which case it's possible that they're treated as "different tags". I guess "lowercase" would be the safest option for these 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah indeed there is the auto-complete search but I was thinking about search engines and the meta name="keywords" that is case-insensitive. See also #14642 (comment) about keywords meta tag abuse. So might only be relevant for our auto-complete.

Copy link
Contributor Author

@dockertopia dockertopia Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, in your previous example all of them are like Docker, BuildKit, etc.
image

Comment: I agree that making them all lowercase allows less room for creating multiple like (docker, Docker, dockerfile, Dockerfile) in terms of metadata, in terms of the presentation (they are visible) they make more sense with case. It would be great to decouple that. As in, write them in suitable case (rendering) and them being always interpretated in a case-insensitive way(metadata).

Question for @usha-mandya do we then change all keywords to be lowercase in our docs sets?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dockertopia Agree on having a standard for keywords. Suggest we use lowercase going forward.

We can create a ticket in the backlog to retrospectively update all keywords to be lowercase. It's not a high priority as the search results are case-insensitive.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @usha-mandya. Sounds good.

redirect_from:
- /buildx/working-with-buildx/
---
Expand Down Expand Up @@ -47,115 +47,6 @@ automatically added to the "docker images" view by default, whereas when using
other drivers, the method for outputting an image needs to be selected
with `--output`.

## Work with builder instances

By default, Buildx uses the `docker` driver if it is supported, providing a user
experience very similar to the native `docker build`. Note that you must use a
local shared daemon to build your applications.

Buildx allows you to create new instances of isolated builders. You can use this
to get a scoped environment for your CI builds that does not change the state of
the shared daemon, or for isolating builds for different projects. You can create
a new instance for a set of remote nodes, forming a build farm, and quickly
switch between them.

You can create new instances using the [`docker buildx create`](../../engine/reference/commandline/buildx_create.md)
command. This creates a new builder instance with a single node based on your
current configuration.

To use a remote node you can specify the `DOCKER_HOST` or the remote context name
while creating the new builder. After creating a new instance, you can manage its
lifecycle using the [`docker buildx inspect`](../../engine/reference/commandline/buildx_inspect.md),
[`docker buildx stop`](../../engine/reference/commandline/buildx_stop.md), and
[`docker buildx rm`](../../engine/reference/commandline/buildx_rm.md) commands.
To list all available builders, use [`docker buildx ls`](../../engine/reference/commandline/buildx_ls.md).
After creating a new builder you can also append new nodes to it.

To switch between different builders, use [`docker buildx use <name>`](../../engine/reference/commandline/buildx_use.md).
After running this command, the build commands will automatically use this
builder.

Docker also features a [`docker context`](../../engine/reference/commandline/context.md)
command that you can use to provide names for remote Docker API endpoints. Buildx
integrates with `docker context` to ensure all the contexts automatically get a
default builder instance. You can also set the context name as the target when
you create a new builder instance or when you add a node to it.

## Build multi-platform images

BuildKit is designed to work well for building for multiple platforms and not
only for the architecture and operating system that the user invoking the build
happens to run.

When you invoke a build, you can set the `--platform` flag to specify the target
platform for the build output, (for example, `linux/amd64`, `linux/arm64`, or
`darwin/amd64`).

When the current builder instance is backed by the `docker-container` driver,
you can specify multiple platforms together. In this case, it builds a manifest
list which contains images for all specified architectures. When you use this
image in [`docker run`](../../engine/reference/commandline/run.md) or
[`docker service`](../../engine/reference/commandline/service.md), Docker picks
the correct image based on the node's platform.

You can build multi-platform images using three different strategies that are
supported by Buildx and Dockerfiles:

1. Using the QEMU emulation support in the kernel
2. Building on multiple native nodes using the same builder instance
3. Using a stage in Dockerfile to cross-compile to different architectures

QEMU is the easiest way to get started if your node already supports it (for
example. if you are using Docker Desktop). It requires no changes to your
Dockerfile and BuildKit automatically detects the secondary architectures that
are available. When BuildKit needs to run a binary for a different architecture,
it automatically loads it through a binary registered in the `binfmt_misc`
handler.

For QEMU binaries registered with `binfmt_misc` on the host OS to work
transparently inside containers, they must be statically compiled and registered
with the `fix_binary` flag. This requires a kernel >= 4.8 and
binfmt-support >= 2.1.7. You can check for proper registration by checking if
`F` is among the flags in `/proc/sys/fs/binfmt_misc/qemu-*`. While Docker
Desktop comes preconfigured with `binfmt_misc` support for additional platforms,
for other installations it likely needs to be installed using
[`tonistiigi/binfmt`](https://github.com/tonistiigi/binfmt){:target="_blank" rel="noopener" class="_"}
image.

```console
$ docker run --privileged --rm tonistiigi/binfmt --install all
```

Using multiple native nodes provide better support for more complicated cases
that are not handled by QEMU and generally have better performance. You can
add additional nodes to the builder instance using the `--append` flag.

Assuming contexts `node-amd64` and `node-arm64` exist in `docker context ls`;

```console
$ docker buildx create --use --name mybuild node-amd64
mybuild
$ docker buildx create --append --name mybuild node-arm64
$ docker buildx build --platform linux/amd64,linux/arm64 .
```

Finally, depending on your project, the language that you use may have good
support for cross-compilation. In that case, multi-stage builds in Dockerfiles
can be effectively used to build binaries for the platform specified with
`--platform` using the native architecture of the build node. A list of build
arguments like `BUILDPLATFORM` and `TARGETPLATFORM` is available automatically
inside your Dockerfile and can be leveraged by the processes running as part
of your build.

```dockerfile
# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM golang:alpine AS build
ARG TARGETPLATFORM
ARG BUILDPLATFORM
RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM" > /log
FROM alpine
COPY --from=build /log /log
```

## High-level build options

Expand Down
4 changes: 2 additions & 2 deletions build/buildx/install.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Install Docker Buildx
description: How to install Docker Buildx
keywords: build, buildx, buildkit
keywords: build, buildx, BuildKit
---

## Docker Desktop
Expand Down Expand Up @@ -75,4 +75,4 @@ sets up docker builder command as an alias to `docker buildx`. This results in
the ability to have [`docker build`](../../engine/reference/commandline/build.md)
use the current buildx builder.

To remove this alias, run [`docker buildx uninstall`](../../engine/reference/commandline/buildx_uninstall.md).
To remove this alias, run [`docker buildx uninstall`](../../engine/reference/commandline/buildx_uninstall.md).
78 changes: 78 additions & 0 deletions build/buildx/multiarch-images.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
---
title: Building multi-platform images
description: Different strategies for building multiplatform images
crazy-max marked this conversation as resolved.
Show resolved Hide resolved
keywords: build, buildx, BuildKit, multiplatform images
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
keywords: build, buildx, BuildKit, multiplatform images
keywords: build, buildx, buildkit, multi-platform images

---
BuildKit is designed to work well for building for multiple platforms and not
only for the architecture and operating system that the user invoking the build
happens to run.

When you invoke a build, you can set the `--platform` flag to specify the target
platform for the build output, (for example, `linux/amd64`, `linux/arm64`, or
`darwin/amd64`).

When the current builder instance is backed by the `docker-container` driver,
you can specify multiple platforms together. In this case, it builds a manifest
list which contains images for all specified architectures. When you use this
image in [`docker run`](../../engine/reference/commandline/run.md) or
[`docker service`](../../engine/reference/commandline/service.md), Docker picks
the correct image based on the node's platform.

You can build multi-platform images using three different strategies that are
supported by Buildx and Dockerfiles:

1. Using the QEMU emulation support in the kernel
2. Building on multiple native nodes using the same builder instance
3. Using a stage in Dockerfile to cross-compile to different architectures

QEMU is the easiest way to get started if your node already supports it (for
example. if you are using Docker Desktop). It requires no changes to your
Dockerfile and BuildKit automatically detects the secondary architectures that
are available. When BuildKit needs to run a binary for a different architecture,
it automatically loads it through a binary registered in the `binfmt_misc`
handler.

For QEMU binaries registered with `binfmt_misc` on the host OS to work
transparently inside containers, they must be statically compiled and registered
with the `fix_binary` flag. This requires a kernel >= 4.8 and
binfmt-support >= 2.1.7. You can check for proper registration by checking if
`F` is among the flags in `/proc/sys/fs/binfmt_misc/qemu-*`. While Docker
Desktop comes preconfigured with `binfmt_misc` support for additional platforms,
for other installations it likely needs to be installed using
[`tonistiigi/binfmt`](https://github.com/tonistiigi/binfmt){:target="_blank" rel="noopener" class="_"}
image.

```console
$ docker run --privileged --rm tonistiigi/binfmt --install all
```

Using multiple native nodes provide better support for more complicated cases
that are not handled by QEMU and generally have better performance. You can
add additional nodes to the builder instance using the `--append` flag.

Assuming contexts `node-amd64` and `node-arm64` exist in `docker context ls`;

```console
$ docker buildx create --use --name mybuild node-amd64
mybuild
$ docker buildx create --append --name mybuild node-arm64
$ docker buildx build --platform linux/amd64,linux/arm64 .
```

Finally, depending on your project, the language that you use may have good
support for cross-compilation. In that case, multi-stage builds in Dockerfiles
can be effectively used to build binaries for the platform specified with
`--platform` using the native architecture of the build node. A list of build
arguments like `BUILDPLATFORM` and `TARGETPLATFORM` is available automatically
inside your Dockerfile and can be leveraged by the processes running as part
of your build.

```dockerfile
# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM golang:alpine AS build
ARG TARGETPLATFORM
ARG BUILDPLATFORM
RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM" > /log
FROM alpine
COPY --from=build /log /log
```
36 changes: 36 additions & 0 deletions build/buildx/multiple-builders.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
title: Using multiple builders
description: How to instantiate and work with multiple builders
keywords: build, buildx, BuildKit, builders, build drivers
---
By default, Buildx uses the `docker` driver if it is supported, providing a user
experience very similar to the native `docker build`. Note that you must use a
local shared daemon to build your applications.

Buildx allows you to create new instances of isolated builders. You can use this
to get a scoped environment for your CI builds that does not change the state of
the shared daemon, or for isolating builds for different projects. You can create
a new instance for a set of remote nodes, forming a build farm, and quickly
switch between them.

You can create new instances using the [`docker buildx create`](../../engine/reference/commandline/buildx_create.md)
command. This creates a new builder instance with a single node based on your
current configuration.

To use a remote node you can specify the `DOCKER_HOST` or the remote context name
while creating the new builder. After creating a new instance, you can manage its
lifecycle using the [`docker buildx inspect`](../../engine/reference/commandline/buildx_inspect.md),
[`docker buildx stop`](../../engine/reference/commandline/buildx_stop.md), and
[`docker buildx rm`](../../engine/reference/commandline/buildx_rm.md) commands.
To list all available builders, use [`docker buildx ls`](../../engine/reference/commandline/buildx_ls.md).
After creating a new builder you can also append new nodes to it.

To switch between different builders, use [`docker buildx use <name>`](../../engine/reference/commandline/buildx_use.md).
After running this command, the build commands will automatically use this
builder.

Docker also features a [`docker context`](../../engine/reference/commandline/context.md)
command that you can use to provide names for remote Docker API endpoints. Buildx
integrates with `docker context` to ensure all the contexts automatically get a
default builder instance. You can also set the context name as the target when
you create a new builder instance or when you add a node to it.
57 changes: 57 additions & 0 deletions build/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,60 @@ BuildKit is the backend evolution from the Legacy Builder, it comes with new
and much improved functionality that can be powerful tools for improving your
builds' performance or reusability of your Dockerfiles, and it also introduces
support for complex scenarios.

## Docker Build features

Docker Build is way more than your `docker build` command and is not only about packaging your code, it’s a whole ecosystem of tools and features that support you not only with common workflow tasks but also provides you with support for more complex and advanced scenarios.
Here’s an overview of all the use cases with which Build can support you:

### Building your images

* **Packaging your software**
Bundle and package your code to run anywhere, from your local Docker Desktop, to Docker Engine and Kubernetes on the cloud.
To get started with Build, see the [Hello Build](hellobuild.md) page.

* **Choosing a build driver**
Run Buildx with different configurations depending on the scenario you are working on, regardless of whether you are using your local machine or a remote compute cluster, all from the comfort of your local working environment.
Here’s a quick overview of the drivers and the use cases they support:
* `docker`– use the default built-in builder to get started quickly.
* `docker-container` – spin up a dedicated builder in a docker container to unlock more advanced features like advanced caching, and multi-arch images.
* `kubernetes`– connect to your Kubernetes cluster to unlock more features and to run your builds at scale.
* `remote`– manually manage your own BuildKit builder and connect directly to it.

* **Optimizing builds with cache management**
Improve build performance by using a persistent shared build cache to avoid repeating costly operations such as package installations, downloading files from the internet, or code build steps.

* **Creating build-once, run-anywhere with multi-architecture builds**
Collaborate across platforms with one build artifact.
See [Build multi platform images](buildx/multiarch-images.md).

### Automating your builds

* **Integrating with GitHub**
Automate your image builds to run in GitHub actions using the official docker build actions. See:
* [GitHub Action to build and push Docker images with Buildx](https://github.com/docker/build-push-action).
* [GitHub Action to extract metadata from Git reference and GitHub events](https://github.com/docker/metadata-action/).

* **Orchestrating builds across complex projects together**
Connect your builds together and easily parameterize your images using buildx bake.
<!--replace when Bake content is onboarded -->
See [High-level build options](buildx/index.md/#high-level-build-options)

### Customizing your Builds

* **Select your build output format**
Choose from a variety of available output formats, to export any artifact you like from BuildKit, not just docker images.
See [Set the export action for the build result](../engine/reference/commandline/buildx_build.md/#output).

* **Managing build secrets**
Securely access protected repositories and resources at build time without leaking data into the final build or the cache.

### Extending BuildKit

* **Custom syntax on Dockerfile**
Use experimental versions of the Dockerfile frontend, or even just bring your own to BuildKit using the power of custom frontends.
See also the [Syntax directive](../engine/reference/builder/#syntax).

* **Configure BuildKit**
Take a deep dive into the internal BuildKit configuration to get the most out of your builds.
See also [`buildkitd.toml`](https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md), the configuration file for `buildkitd`.