Move from OSTree to OCI for updates #1823

jlebon · 2024-10-30T17:44:11Z

Currently, FCOS pushes updates via an OSTree repo. To better align with the bootable containers initiative, let's move to updating FCOS via a container image. Container images are already being published in https://quay.io/repository/fedora/fedora-coreos.

Note this ticket is separate from #1263, which covers also fleshing out the story of layering. In this ticket, we're strictly scoping the effort to changing the transport used.

Take the digest pullspec for the base OS bootable container and put it in the new `oci-image` field in the release metadata. Part of coreos/fedora-coreos-tracker#1823.

When updating the release index, gather the OCI pullspecs across all arches for a given release into a single list and inject it into the new `oci-images` key, the same way we do for OSTree commits and `commits`. Part of coreos/fedora-coreos-tracker#1823.

Currently, we're starting a total of N x M scrapers, where N is the number of streams, and M the number of arches. So right now, this would be 3 x 4 = 12 scrapers. This is very wasteful because each scraper individually downloads the release index and update metadata every 30 seconds, even though that metadata is not different per architecture. I think the reason it was set up this way is in case we wanted to host separate e.g. release index or update files _per_ architecture in S3 instead of all togother. This can be seen by the fact the code supports templating those URLs with `basearch`. However, it's unlikely we'll be changing that design decision, so let's just do the saner thing and rework the scraping to be stream-based. This is done by changing the scraper to host not one single `Graph` object, but instead a `HashMap<String, Graph>` which maps architectures to graphs. Then, when a request for a graph comes in, we lookup in our cache keying off of the requested architecture. This is prep for adding another dimension to the matrix, which is whether the OCI version of the graph was reported. If we didn't do this cleanup first, it would have blown up the number of scrapers to 24. Part of coreos/fedora-coreos-tracker#1823.

When parsing the release index, check for the new `oci-images` key. If present, also build up a separate graph with only nodes containing OCI information. In that case, the node payload is the pullspec and the scheme declared in the node metadata is `oci`. When a client requests a graph, check if the `oci=` URL parameter was set. If so, return back the OCI graph instead of the OSTree one. Part of coreos/fedora-coreos-tracker#1823.

Add support in both release metadata and the release index for specifying OCI image pullspecs for a given release. In both cases, the new fields are located at the same level as the existing `commits` key holding OSTree checksums. The key is called `oci-image` in release metadata and `oci-images` in the release index. Part of coreos/fedora-coreos-tracker#1823.

jlebon · 2024-11-01T20:04:13Z

OK, some progress on this:

release: support OCI image pullspecs stream-metadata-go#75 adds support for a new oci-image key in the release metadata and a new oci-images key in the release index
Inject oci-image in release metadata and oci-images in release index coreos-assembler#3919 populates those newly supported keys from the base-oscontainer key in the build metadata
Add support for querying OCI graph fedora-coreos-cincinnati#99 teaches Cincinnati to read those keys and serve an OCI-based graph when the request includes oci=true
daemon/transaction-types: Support custom origins for digested pullspecs rpm-ostree#5120 teaches rpm-ostree to allow custom origin information when passing a digest pullspec to rpm-ostree rebase

So the next step on this is support in Zincati for a config knob that tells it to request the OCI version of the graph and deploy using rpm-ostree rebase with appropriate custom origin info.

Add support in both release metadata and the release index for specifying OCI image pullspecs for a given release. In both cases, the new fields are located at the same level as the existing `commits` key holding OSTree checksums. The key is called `oci-image` in release metadata and `oci-images` in the release index. Part of coreos/fedora-coreos-tracker#1823.

cgwalters · 2024-11-01T20:40:38Z

Awesome!

Currently, we're starting a total of N x M scrapers, where N is the number of streams, and M the number of arches. So right now, this would be 3 x 4 = 12 scrapers. This is very wasteful because each scraper individually downloads the release index and update metadata every 30 seconds, even though that metadata is not different per architecture. I think the reason it was set up this way is in case we wanted to host separate e.g. release index or update files _per_ architecture in S3 instead of all together. This can be seen by the fact the code supports templating those URLs with `basearch`. However, it's unlikely we'll be changing that design decision, so let's just do the saner thing and rework the scraping to be stream-based. This is done by changing the scraper to host not one single `Graph` object, but instead a `HashMap<String, Graph>` which maps architectures to graphs. Then, when a request for a graph comes in, we lookup in our cache keying off of the requested architecture. This is prep for adding another dimension to the matrix, which is whether the OCI version of the graph was requested. If we didn't do this cleanup first, it would have blown up the number of scrapers to 24. Part of coreos/fedora-coreos-tracker#1823.

When parsing the release index, check for the new `oci-images` key. If present, also build up a separate graph with only nodes containing OCI information. In that case, the node payload is the pullspec and the scheme declared in the node metadata is `oci`. When a client requests a graph, check if the `oci=` URL parameter was set. If so, return back the OCI graph instead of the OSTree one. Part of coreos/fedora-coreos-tracker#1823.

Add support in both release metadata and the release index for specifying OCI image pullspecs for a given release. In both cases, the new fields are located at the same level as the existing `commits` key holding OSTree checksums. The key is called `oci-image` in release metadata and `oci-images` in the release index. Part of coreos/fedora-coreos-tracker#1823.

Take the digest pullspec for the base OS bootable container and put it in the new `oci-image` field in the release metadata. Part of coreos/fedora-coreos-tracker#1823.

When updating the release index, gather the OCI pullspecs across all arches for a given release into a single list and inject it into the new `oci-images` key, the same way we do for OSTree commits and `commits`. Part of coreos/fedora-coreos-tracker#1823.

When parsing the release index, check for the new `oci-images` key. If present, also build up a separate graph with only nodes containing OCI information. In that case, the node payload is the pullspec and the scheme declared in the node metadata is `oci`. When a client requests a graph, check if the `oci=` URL parameter was set. If so, return back the OCI graph instead of the OSTree one. Part of coreos/fedora-coreos-tracker#1823.

jlebon · 2024-11-08T17:47:54Z

So the next step on this is support in Zincati for a config knob that tells it to request the OCI version of the graph

Instead of a config knob, Zincati can just cue off of the origin. If the OS is on OSTree, query the OSTree graph. If it's on OCI, query the OCI graph. So then the barrier release is just about switching the origin over from OSTree to OCI.

Discussed the rollout plan for this with @travier and @dustymabe:

Switch next bootimages to deploy-via-container whenever all the code needed has landed in FCOS.
- This means that only new nodes will be updating via containers to start, giving us a way to bake it before switching updating nodes.
After X next releases (e.g. 2 or 3), do barrier release to switch origin on existing nodes.
Switch the testing bootimages to deploy-via-container. If we're still before Beta freeze, just do it. If we're after Beta freeze, do it at GA.
- Stable follows 2 weeks after.
After X testing releases, do barrier release to switch origin on existing nodes.
- Stable follows 2 weeks after.

Take the digest pullspec for the base OS bootable container and put it in the new `oci-image` field in the release metadata. Part of coreos/fedora-coreos-tracker#1823.

When updating the release index, gather the OCI pullspecs across all arches for a given release into a single list and inject it into the new `oci-images` key, the same way we do for OSTree commits and `commits`. Part of coreos/fedora-coreos-tracker#1823.

jlebon · 2024-11-15T17:20:14Z

One thing we discussed related to this was signing. Currently, the FCOS OCI artifacts are not signed (really, AFAICT none of the Fedora container images are signed either). Ideally, we close that gap before we fully switch over.

At least the OSTree commit inside of the OCI is still signed, but we'd still need to download and extract without any verification. But even then it's not really used in any meaningful way client-side currently because when importing into the OSTree repo, it's the OCI "merge commit" that gets deployed. We'd have to unencapsulate instead and in the process point at the remote OSTree config containing the GPG settings. So yeah... all around much better if we just fix this at the OCI level.

Related Robosignatory issue: https://pagure.io/robosignatory/issue/22

jlebon · 2024-11-20T19:42:56Z

One thing we discussed related to this was signing.

Was chatting with @cgwalters about this. One interesting note is that the Konflux pipelines building rhel-bootc and centos-bootc do sign container images today. See e.g. the shield icons in https://quay.io/repository/centos-bootc/centos-bootc?tab=tags. Ideally, we also have that setup for fedora-bootc as we bring it up in the Fedora Konflux, and we can piggy-back on it for FCOS too (even if we're not building the whole thing in Konflux just yet).

Otherwise, it should also be possible to make the rpm-ostree consumption of the embedded signature better.

Parse rpm-ostree status to detect if the current booted deployment is an OCI image. If so, query the OCI graph for cincinnati and rebase to the correct OCI image. Requires coreos/fedora-coreos-cincinnati#99 and coreos/rpm-ostree#5120 Part of: coreos/fedora-coreos-tracker#1823

Cincinnati now serves a separate graph for OCI images, since coreos/fedora-coreos-cincinnati#99 Allow displaying the oci graph by passing `oci=true` as a query parameter. See coreos/fedora-coreos-tracker#1823

travier · 2025-01-10T14:46:14Z

The Change Checkpoint: Proposal submission deadline (Self Contained Changes) for F42 is Tue 2025-01-14.

jlebon · 2025-01-10T16:00:00Z

Re. signing, @travier and I were playing with

$ rpm-ostree rebase ostree-remote-image:fedora:registry:quay.io/fedora/fedora-coreos:testing-devel

and confirmed that it does verify the OSTree signature using the fedora remote. And I now realize this is also how it's set up in Rawhide already. Ideally though, the signing state would also show up in rpm-ostree status or bootc status as it does for the non-OCI case.

Anyway, we still of course want to instead sign the OCI itself but we don't expect that to happen before we move to Konflux.

cgwalters · 2025-01-10T16:10:55Z

Yes, there's a lot of custom stuff in the ostree-container stack to do this for exactly this reason.

Anyway, we still of course want to instead sign the OCI itself

Yes.

Cincinnati now serves a separate graph for OCI images, since coreos/fedora-coreos-cincinnati#99 Allow displaying the oci graph by passing `oci=true` as a query parameter. See coreos/fedora-coreos-tracker#1823

jlebon · 2025-01-14T22:07:01Z

Ideally though, the signing state would also show up in rpm-ostree status or bootc status as it does for the non-OCI case.

I didn't do exactly this, but I consider it addressed by containers/bootc#1028 and coreos/rpm-ostree#5223. See notably sample output in coreos/rpm-ostree#5223 (comment).

travier · 2025-01-15T09:40:46Z

We might also have to fix coreos/rpm-ostree#4951 as well.

jbtrystram · 2025-01-15T15:37:38Z

Related Fedora change request : https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates

jbtrystram · 2025-01-16T09:31:33Z

Documenting the next steps (after coreos/zincati#1241 is merged):

Release Zincati
Document steps to take for people that have disabled Zincati
Prepare a barrier release doing the rebase to OCI
Write a MOTD explaining what to do if Zincati is disabled (we don't want to force a reboot on people that have disabled Zincati)
Add an entry to the major changes documation page

jbtrystram · 2025-01-20T10:27:08Z

So we discussed how to make the switch to OCI through Zincati : running a service to manually rebase through a barrier release would cause two reboots (one to get the update through Zincati, one to rebase to the OCI image), and double the download, causing the second reboot to be outside the Zincati window potentially.

An alternative is to fake out the origin file, causing Zincati to think the deployment is now OCI, so the next update will be fetched through OCI and rebased.
This works but create a cosmetic issue :

[core@cosa-devsh ~]$ rpm-ostree status
State: idle
Deployments:
● ostree-remote-image:fedora:registry:quay.io/fedora/fedora-coreos:testing
             CustomOrigin: Fedora CoreOS testing stream
                  Version: 41.20250105.2.0 (2025-01-06T19:46:25Z)

  (error fetching image metadata)
             CustomOrigin: Fedora CoreOS testing stream
                  Version: 41.20241215.2.0 (2024-12-17T00:07:38Z)

See how rpm-ostree status cannot find the "faked deployment". Though it's cosmetic only, I was able to overlay packages just fine.

travier · 2025-01-20T14:13:44Z

Document how to setup/update proxy configuration for the OCI case. Likely based on https://docs.fedoraproject.org/en-US/bootc/proxy/

jbtrystram · 2025-01-21T14:14:33Z

Document how to setup/update proxy configuration for the OCI case. Likely based on docs.fedoraproject.org/en-US/bootc/proxy

Seems those docs are based on FCOS docs already https://docs.fedoraproject.org/en-US/fedora-coreos/proxy/

dustymabe · 2025-01-22T19:43:52Z

We discussed this during the community meeting today:

ACTION: @jbtrystram to update the change to make clear that bootimages will migrate first, then existing notes after. Also mention clearly that deriving the images won't be supported for now (@jbtrystram:matrix.org, 16:53:17)
INFO: FCOS will move to default to pulling updates from OCI registry in the F42 time frame. (@dustymabe:matrix.org, 16:56:20)

Switch boot images to use OCI for updates. This is a step towards bootable containers support and bootc rebase. See https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates Requires coreos/zincati#1241 See coreos/fedora-coreos-tracker#1823

With the f42 rebase we will switch disk images to use OCI for updates. While this is transparent for most users, it's still a big technical change so it's worth mentionning it, at least so users using can look into it. See https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates coreos/fedora-coreos-tracker#1823

jlebon added kind/enhancement area/updates component/zincati component/rpm-ostree area/bootable-containers Related to the bootable containers effort. F42 labels Oct 30, 2024

jlebon mentioned this issue Oct 30, 2024

Roadmap to Fedora Bootable Containers #1726

Open

7 tasks

This was referenced Nov 1, 2024

release: support OCI image pullspecs coreos/stream-metadata-go#75

Merged

Inject oci-image in release metadata and oci-images in release index coreos/coreos-assembler#3919

Merged

Add support for querying OCI graph coreos/fedora-coreos-cincinnati#99

Merged

This comment was marked as off-topic.

Sign in to view

jlebon mentioned this issue Nov 7, 2024

Use bootc install to-filesystem to build our bootimages #1827

Open

This comment was marked as off-topic.

Sign in to view

jlebon mentioned this issue Dec 10, 2024

update_agent: support rebase to OCI pullspec coreos/zincati#1241

Open

jbtrystram mentioned this issue Dec 18, 2024

graph: display the oci graph coreos/fedora-coreos-browser#49

Merged

travier added the F42-Changes label Jan 10, 2025

jbtrystram added the meeting topics for meetings label Jan 15, 2025

dustymabe removed the meeting topics for meetings label Jan 22, 2025

jbtrystram mentioned this issue Jan 23, 2025

[testing] deploy container images by default coreos/fedora-coreos-config#3333

Open

jbtrystram mentioned this issue Jan 23, 2025

Major-changes: Document the switch to OCI for updates coreos/fedora-coreos-docs#698

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move from OSTree to OCI for updates #1823

Move from OSTree to OCI for updates #1823

jlebon commented Oct 30, 2024

jlebon commented Nov 1, 2024

cgwalters commented Nov 1, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

jlebon commented Nov 8, 2024

jlebon commented Nov 15, 2024

jlebon commented Nov 20, 2024

travier commented Jan 10, 2025

jlebon commented Jan 10, 2025 •

edited

Loading

cgwalters commented Jan 10, 2025

jlebon commented Jan 14, 2025

travier commented Jan 15, 2025

jbtrystram commented Jan 15, 2025

jbtrystram commented Jan 16, 2025 •

edited by travier

Loading

jbtrystram commented Jan 20, 2025

travier commented Jan 20, 2025 •

edited

Loading

jbtrystram commented Jan 21, 2025

dustymabe commented Jan 22, 2025

Move from OSTree to OCI for updates #1823

Move from OSTree to OCI for updates #1823

Comments

jlebon commented Oct 30, 2024

jlebon commented Nov 1, 2024

cgwalters commented Nov 1, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

jlebon commented Nov 8, 2024

jlebon commented Nov 15, 2024

jlebon commented Nov 20, 2024

travier commented Jan 10, 2025

jlebon commented Jan 10, 2025 • edited Loading

cgwalters commented Jan 10, 2025

jlebon commented Jan 14, 2025

travier commented Jan 15, 2025

jbtrystram commented Jan 15, 2025

jbtrystram commented Jan 16, 2025 • edited by travier Loading

jbtrystram commented Jan 20, 2025

travier commented Jan 20, 2025 • edited Loading

jbtrystram commented Jan 21, 2025

dustymabe commented Jan 22, 2025

jlebon commented Jan 10, 2025 •

edited

Loading

jbtrystram commented Jan 16, 2025 •

edited by travier

Loading

travier commented Jan 20, 2025 •

edited

Loading