ci: publish multi-arch image manifest lists

[ngraef]

This change adds linux/arm64 binaries to the release. It also publishes an arm64 container image for all variants (standard, debug, rootless, static) and releases (dev, edge, latest).

The build and push process uses buildx to push the individual images by digest (i.e. untagged) and reference them in a single, tagged manifest list. This avoids cluttering Docker Hub's tag list with <tag>-<arch> tags.

Fixes #2233

[srenatus]

Thank you for looking into this!

I've got a few questions, please bear with me, I've never used buildx before.

1. With the manifest list, can you still pull images by tag as before? I.e. openpolicyagent/opa:0.36.1, openpolicyagent/opa:latest, ``openpolicyagent/opa:edge`?
2. What does it look like on dockerhub? I'm sure you've got some example of the desired end state...
3. Our docker image smoke tests fail on the PR, it seems like we're missing something there: https://github.com/open-policy-agent/opa/runs/4879113973?check_suite_focus=true#step:4:59

Finally, do you have an idea how we could (smoke) test the produced binaries and images, preferable from github actions...? Are there any handy tools to emulate linux/arm64...?

[ngraef]

1. With the manifest list, can you still pull images by tag as before? I.e. openpolicyagent/opa:0.36.1, openpolicyagent/opa:latest, openpolicyagent/opa:edge?

Yes! Clients continue to pull images the same way. docker pull openpolicyagent/opa:0.36.1 defaults to pulling the image from the manifest list that matches the client architecture (linux/amd64 in most cases). You can force pulling a specific architecture using the --platform option, like docker pull --platform linux/arm64 openpolicyagent/opa:0.36.1.

If you're curious to see what a manifest list contains, run docker manifest inspect golang:1.17 or docker buildx imagetools inspect golang:1.17.

2. What does it look like on dockerhub? I'm sure you've got some example of the desired end state...

You've probably seen it on most of the "official" images, like the example from golang below. In our case, it will just be two architectures rather than the 10 shown.

3. Our docker image smoke tests fail on the PR, it seems like we're missing something there: https://github.com/open-policy-agent/opa/runs/4879113973?check_suite_focus=true#step:4:59

Already fixed it before seeing your review 😀

Finally, do you have an idea how we could (smoke) test the produced binaries and images, preferable from github actions...? Are there any handy tools to emulate linux/arm64...?

Docker has setup-quemu-action that is helpful. It allows you to build and run images for different architectures. We don't need it to build in this case because we're cross-compiling, but we could use it to test the built arm64 images.

[srenatus]

Thanks for your replies there, it's very helpful to me. 😃

but we could use it to test the built arm64 images.

Would you mind looking into that, too? It would be great if we had some assurance in our pipeline before publishing arm64 images.

[ngraef]

but we could use it to test the built arm64 images.

Would you mind looking into that, too? It would be great if we had some assurance in our pipeline before publishing arm64 images.

I added smoke tests for arm64 images. I also added the platform string to the version command output.

[anderseknert]

Wow, this is an awesome contribution @ngraef 👏

Just tried it and for the first time I can run OPA containerized on my M1 without the dreaded:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Nice! 👍

[srenatus]

🎉 This looks great and I'm thankful for you both taking on this decent amount of work on our build processes, and taking the time to explain the changes.

I haven't played with it locally yet, I'm about to do that next.

[srenatus]

Could you please rebase and squash this, so we can get it merged?

[bencooper222]

I don't think this is happening in practice? This image was released 25 days ago and is x86-only.

https://hub.docker.com/layers/opa/openpolicyagent/opa/0.42.0/images/sha256-83ee526b78c6204ccc1fce0021d6c2eacfc2393e402a88260a994875448195ad?context=explore

[anderseknert]

@bencooper222 you’ll need to use the static image https://hub.docker.com/layers/opa/openpolicyagent/opa/latest-static/images/sha256-b14bf57a46c2d73f117f8bab45788581cf2fddd87f16a888ba4e1d05d347b34d?context=explore

[ngraef]

@bencooper222 This was partially reverted. See #4282 for more context.

[timlaroche]

Hi all, just wondering if #4282 will be reversed as it seems that the issue in #4280 regarding bytecodealliance/wasmtime#3183 has been closed as fixed? It would be nice to have the manifests for the non-static images also.

[srenatus]

We still have problems testing it. There are not arm64 runners in Github, and we don't have the bandwidth to maintain our own arm-based runners on any other hardware. So we'd either have to ship something untested; or don't ship it at all

It could be worth retrying to test the images in Qemu. Last time I tried it, I ran into SIGILL exceptions.