firstboot support on read-only root file systems

[laenion]

From what I understand the current approach for firstboot detection is

• Check whether file /boot/ignition.firstboot exists
• Check whether command line parameter ignition.firstboot exists (will always enforce firstboot)

The first option however does not work if the root file system including /boot is a read-only file system (as it is the case on openSUSE Kubic systems) - the file cannot be deleted any more.

One option I was thinking about is to additionally use a GRUB2 environment variable instead, e.g. storing something like ignition.firstboot=1 if the system is supposed to execute Ignition (or the other way around: skip execution if ignition.firstboot=0).

Any thoughts on this approach?

[dustymabe]

From what I understand the current approach for firstboot detection is

• Check whether file /boot/ignition.firstboot exists

this is mostly an implementation detail of how we've implemented it in Fedora CoreOS right now. This only works because we've created that file and we have a grub snippet that checks for that file and sets a variable, which later gets used in the kernel command line line.

• Check whether command line parameter ignition.firstboot exists (will always enforce firstboot)

This is the way ignition gets triggered for us, we just implemented the weird hacky answer (mentioned above) to actually get the kernel command line arg populated. It doesn't matter how you get the kernel CLI arg populated as long as you get it populated :)

The first option however does not work if the root file system including /boot is a read-only file system (as it is the case on openSUSE Kubic systems) - the file cannot be deleted any more.

One option I was thinking about is to additionally use a GRUB2 environment variable instead, e.g. storing something like ignition.firstboot=1 if the system is supposed to execute Ignition (or the other way around: skip execution if ignition.firstboot=0).

That sounds interesting. Maybe we could also just set/use a grub env var in Fedora CoreOS and Red Hat CoreOS to achieve this goal. Not sure if we have thought about this before and disqualified it for some reason.

[laenion]

Thanks for the details!

The major disadvantage of saving variables into the GRUB2 environment block is that not all setups are supported: See https://www.gnu.org/software/grub/manual/grub/grub.html#Environment-block for the limitations (i.e. it doesn't work with LVM, software RAIDs and a few file systems - openSUSE contains a patched GRUB2 for Btrfs support) - that's why I wouldn't use this as the only method, but only as an additional option.

In case of doubt I'll just implement it to see if this would be viable.

[ajeddeloh]

That sounds interesting. Maybe we could also just set/use a grub env var in Fedora CoreOS and Red Hat CoreOS to achieve this goal. Not sure if we have thought about this before and disqualified it for some reason.

Hadn't occurred to me either. @laenion does bring up some good points against it though. I wish grub would have just allocated 1kb out of it's core for the grubenv instead of putting it in a file. Use the grubenv has the disadvantage (or maybe advantage, depending on how you look at it) of being less obvious and harder to change.

[cgwalters]

The first option however does not work if the root file system including /boot is a read-only file system (as it is the case on openSUSE Kubic systems) - the file cannot be deleted any more.

Is this like the design for ostreedev/ostree#1265 where it's a ro bind mount and the update system knows how to remount it?

Or if not, how do you update kernels? Or to rephrase, where does the kernel/initramfs live and wouldn't it make sense to have the firstboot flag there too?

[laenion]

That depends a bit on the perspective, but I'd say the designs are quite different (though the linked approach is closing the gap a bit) - mainly because we are not using anything like ostree, but btrfs snapshots as the core technology.

In our case the complete root file system is read-only (as described in ostreedev/ostree#1265 if I understood that correctly), including the kernel and the initrd. The only necessary exceptions are /var and /boot/grub2/<arch>, typically created and mounted as a separate subvolume. A special case is /etc, which is mounted as an overlay file system over the original /etc contents and saving the differences into /var.

For updates a snapshot of the root file system will be created. The update is then temporarily setting that snapshot to read-write, performing the update in that snapshot (i.e. not influencing the currently running system) and then setting that snapshot as the new default snapshot. On reboot GRUB will just use the default snapshot to boot.

/boot being part of the root file system / snapshots has the huge advantage that it is easy to do a rollback in case a kernel version or initrd should break the system, but it also means that it is not writeable (with /boot/grub2/<arch> being the exception as mentioned).

[cgwalters]

I guess if Ignition ends up mounting /var we could without much loss of generality use /var/ignition.firstboot or so?

[dustymabe]

@laenion
The only necessary exceptions are /var and /boot/grub2/<arch>

In that case we could just make it easier for distros to patch in the location they plan to use for the file. i.e. right now for FCOS we are using /boot/ignition.firstboot and you could use /boot/grub2/ignition.firstboot

@cgwalters
I guess if Ignition ends up mounting /var we could without much loss of generality use /var/ignition.firstboot or so?

I don't think this will work for the same reason we are having issues with UEFI today. If we keep the same approach of using a file AND a grub snippet then we'll need to use the same filesystem as the grub.cfg. If we switch the grub env vars (if we can) then we shouldn't have to worry about files and cleaning them up.

[laenion]

Using /var to store the flag file sounds tempting, but that directory is not part of the root file system and thus not directly visible to GRUB (where currently the decision to trigger firstboot is made by either appending the kernel parameter or not).
To get this working the firstboot flag would have to be evaluated by some Ignition systemd service in dracut instead of GRUB.

Another file-based option would be to use /boot/grub2/<arch>/ - as far as I`m aware that directory is always there, the exact name of the directory can be resolved by using GRUB variables and the directory has to be writeable. It just feels wrong for me to put flag files in there...

In any case I agree making the path configurable would be a good idea.

Regarding using a GRUB2 environment variable I've added support in my own fork (still untested, as I'm currently trying to get rid of the other /boot dependencies) for setting up a POC.

[laenion]

I implemented and tested storing the firstboot flag in a GRUB environment variable; this could look something like laenion/ignition-dracut@cb70bb2.

I haven't removed support for the flag file yet. To activate firstboot, instead of writing /ignition.firstboot one would call grub2-editenv - set ignition_firstboot="ignition.firstboot rd.neednet=1 ip=dhcp" now.

Do you have any idea on how you want to proceed with CoreOS yet?

[ajeddeloh]

Where are you storing the grub env? Usually it's in the grub root (i.e. /boot) which is also where we're storing the ignition flag file.

[laenion]

Good point, I hadn't realized this only works for btrfs on openSUSE (as it contains a patched grub-editenv to write the variables into the btrfs header). On other file systems only GRUB itself could unset the flag, as GRUB doesn't know about read-only mounts.

I guess this rules out GRUB environment variables.

[laenion]

Unfortunately I couldn't come up with a better idea, so it seems flag files are the only reliable way working in all setups.

In that case we could just make it easier for distros to patch in the location they plan to use for the file. i.e. right now for FCOS we are using /boot/ignition.firstboot and you could use /boot/grub2/ignition.firstboot

Which configuration mechanism would you suggest? There isn't one for this project currently...

[dustymabe]

@laenion - my suggestion would be that we extend https://github.com/coreos/ignition-dracut/blob/master/grub/02_ignition_firstboot so that it can be parameterized maybe? Right now we moved to detecting where the boot partition is using a FS label (so that it can work for both UEFI and non-UEFI, see coreos/ignition-dracut#52). Does your /boot/grub2/<arch>/ filesystem have a label that could be used?

[laenion]

To answer the last question (sorry for the late reply): No it doesn't. But 02_ignition_firstboot has been moved to https://github.com/coreos/fedora-coreos-config anyway, so the implementation is distribution specific now for quite some time.

From my point of view this ticket could be closed, except when you want to keep it open to find a more general approach in the future.

[jlebon]

Yeah, the /boot/ignition.firstboot mechanism now purely lives at the distro integration layer. Other distros can implement their own mechanism in its place (including just hardcoding the karg and removing it from the boot config if that makes the most sense).

Tentatively closing.

[jlebon]

(including just hardcoding the karg and removing it from the boot config if that makes the most sense).

Which... actually is something we could easily switch to for CoreOS now since we always modify kargs anyway for the rootmap stuff and rdcore makes it safe and trivial to do. But meh, probably not worth the churn if there are no additional benefits.