Use ephemeral disks to store container images

[arnaldo2792]

I’d like the ability to use the ephemeral disks provided in a host as storage for container images.

[arnaldo2792]

It is possible to use a bootstrap container to change the root directory of either docker or containerd so that container images can be stored in ephemeral disks.

Given a bootstrap container called setup-runtime-storage, with the following definition:

[settings.bootstrap-containers.setup-runtime-storage]
source = "<SOURCE>"
mode = "always"
essential = true # You may change this depending on your requirements

Where <SOURCE> is the URL of an image with the following definition:

FROM alpine
RUN apk add e2fsprogs bash parted
ADD setup-runtime-storage ./
RUN chmod +x ./setup-runtime-storage
ENTRYPOINT ["sh", "setup-runtime-storage"]

And setup-runtime-storage as:

#!/usr/bin/env bash
set -ex

# Symlinks to ephemeral disks are created here by udev
EPHEMERAL_DISKS=/.bottlerocket/rootfs/dev/disk/ephemeral

# Exit early if there aren't ephemeral disks
if [ ! -d ${EPHEMERAL_DISKS} ]; then
  echo "${EPHEMERAL_DISKS} doesn't exist"
  exit 1
fi

# Use the first ephemeral disk
DISK="$(cd ${EPHEMERAL_DISKS} && readlink -f $(ls | head -n 1))"
# There will be only one partition
PARTITION="${DISK}p1"
# For k8s, use containerd, for ECS use docker
CONTAINER_RUNTIME=<containerd/docker>
# Path where the disk will be mounted
MOUNT_POINT=/.bottlerocket/rootfs/mnt/${CONTAINER_RUNTIME}
# The path to the container's runtime root
RUNTIME_ROOT_PATH=/.bottlerocket/rootfs/var/lib/${CONTAINER_RUNTIME}

# If the partition doesn't exist, create it
if [ ! -e ${PARTITION} ]; then
  parted -s ${DISK} mklabel gpt 1>/dev/null
  parted -s ${DISK} mkpart primary ext4 0% 100% 1>/dev/null
  mkfs.ext4 -F ${PARTITION}
fi

mkdir -p ${MOUNT_POINT}
mount ${PARTITION} ${MOUNT_POINT}

# If the container's runtime root directory exists, you have to manually delete it
if [ ! -d ${RUNTIME_ROOT_PATH} ]; then
  # Use /mnt since the symlink will be seen from the host's perspective
  ln -snf /mnt/${CONTAINER_RUNTIME} ${RUNTIME_ROOT_PATH}
 fi

With the provided configurations the bootstrap container will be executed at every boot, and the boot will fail if the bootstrap container fails. From the admin container, you can see that containerd/docker are using the new mounted disk by listing the files in the partition:

bash-5.0# ls -la /mnt/docker/
total 68
drwx--x---. 14 root root  4096 Feb 23 01:36 .
drwxr-xr-x.  3 root root  4096 Feb 23 01:36 ..
drwx--x--x.  4 root root  4096 Feb 23 01:36 buildkit
drwx--x---.  3 root root  4096 Feb 23 01:37 containers
drwx------.  3 root root  4096 Feb 23 01:36 image
drwx------.  2 root root 16384 Feb 23 01:36 lost+found
drwxr-x---.  3 root root  4096 Feb 23 01:36 network
drwx--x---. 17 root root  4096 Feb 23 01:37 overlay2
drwx------.  4 root root  4096 Feb 23 01:36 plugins
drwx------.  2 root root  4096 Feb 23 01:36 runtimes
drwx------.  2 root root  4096 Feb 23 01:36 swarm
drwx------.  2 root root  4096 Feb 23 01:37 tmp
drwx------.  2 root root  4096 Feb 23 01:36 trust
drwx-----x.  2 root root  4096 Feb 23 01:36 volumes

The script provided is just an example and should only be used as a reference.

[cyrus-mc]

The issue with the solution shown here is that it assumes the secondary disk is an instance store, as the udev rules only add ephemeral disks if the model attribute identifies it as an instance store. Using an instance store for container images probably isn't ideal as you lose all that data on restart and thus would be forced to download all container images again, slowing the restart process.

Issue I am running into now is that the order of disks is constantly changing from nvme2n1 and nvme1n1 that I can't just hard code the disk name.

[bcressey]

Using an instance store for container images probably isn't ideal as you lose all that data on restart and thus would be forced to download all container images again, slowing the restart process.

Whether or not this matters depends on the use case, but instance store data is only lost when the instance is stopped and started again. Container images will still be there after a normal reboot. So it could be fine if the instances are managed by automation like ASGs, Karpenter, or MNG, but it might not be a great fit for instances that might be long-lived but are turned off when not in use.

Issue I am running into now is that the order of disks is constantly changing from nvme2n1 and nvme1n1 that I can't just hard code the disk name.

For ephemerals it can be simpler to group them all into the same RAID array or LVM volume, if you don't need to distinguish them. If you do, then you could give them specific filesystem labels, and treat the absence of links in /dev/disk/by-label as a sign that they haven't been formatted yet.

[bcressey]

The recommended approach is changing with Bottlerocket 1.9.0 because of the issue with CSI plugins reported in #2218 (thanks, @diranged) and the fix in #2240. The example from @arnaldo2792 above shows using a symlink, but for best results I'd recommend using a bind mount instead in newer versions.

The right approach depends on whether the bootstrap container needs to support older and newer versions of Bottlerocket. If older versions aren't a concern, the logic can be simplified by quite a bit.

Here's the Dockerfile for the container:

FROM amazonlinux:2
RUN yum -y install e2fsprogs bash mdadm util-linux
ADD setup-runtime-storage ./
RUN chmod +x ./setup-runtime-storage
ENTRYPOINT ["sh", "setup-runtime-storage"]

And here's the setup-runtime-storage script with the logic for Bottlerocket before and after 1.9.0, showing how to use one or more ephemerals in a RAID-0 array:

#!/usr/bin/env bash
set -ex

ROOT_PATH="/.bottlerocket/rootfs"

# Symlinks to ephemeral disks are created here by udev
declare -a EPHEMERAL_DISKS
EPHEMERAL_DISKS=("${ROOT_PATH}"/dev/disk/ephemeral/*)

# Exit early if there aren't ephemeral disks
if [ "${#EPHEMERAL_DISKS[@]}" -eq 0 ]; then
  echo "no ephemeral disks found"
  exit 1
fi

MD_NAME="scratch"
MD_DEVICE="/dev/md/${MD_NAME}"
MD_CONFIG="/.bottlerocket/bootstrap-containers/current/mdadm.conf"

# Create or assemble the array.
if [ ! -s "${MD_CONFIG}" ] ; then
  mdadm --create --force --verbose \
    "${MD_DEVICE}" \
      --level=0 \
      --name="${MD_NAME}" \
      --raid-devices="${#EPHEMERAL_DISKS[@]}" \
      "${EPHEMERAL_DISKS[@]}"
  mdadm --detail --scan > "${MD_CONFIG}"
else
  mdadm --assemble --config="${MD_CONFIG}" "${MD_DEVICE}"
fi

# Format the array if not already formatted.
if ! blkid --match-token TYPE=ext4 "${MD_DEVICE}" ; then
  mkfs.ext4 "${MD_DEVICE}"
fi

MOUNT_POINT="${ROOT_PATH}/mnt/${MD_NAME}"

# Mount the array in the host's /mnt.
mkdir -p "${MOUNT_POINT}"
mount "${MD_DEVICE}" "${MOUNT_POINT}"

# Keep track of whether we can unmount the array later. This depends on the
# version of Bottlerocket.
should_umount="no"

# Bind state directories to the array, if they exist.
for state_dir in containerd docker kubelet ; do
  # The correct next step depends on the version of Bottlerocket, which can be
  # inferred by inspecting the mounts available to the bootstrap container.
  if findmnt "${ROOT_PATH}/var/lib/${state_dir}" ; then
    # For Bottlerocket >= 1.9.0, the state directory can be bind-mounted over
    # the host directory and the mount will propagate back to the host.
    mkdir -p "${MOUNT_POINT}/${state_dir}"
    mount --rbind "${MOUNT_POINT}/${state_dir}" "${ROOT_PATH}/var/lib/${state_dir}"
    mount --make-rshared "${ROOT_PATH}/var/lib/${state_dir}"
    should_umount="yes"
  elif [ ! -L "${ROOT_PATH}/var/lib/${state_dir}" ] ; then
    # For Bottlerocket < 1.9.0, the host directory needs to be replaced with a
    # symlink to the state directory on the array. This works but can lead to
    # unexpected behavior or incompatibilities, for example with CSI drivers.
    if [ -d  "${ROOT_PATH}/var/lib/${state_dir}" ] ; then
      # The host directory exists but is not a symlink, and might need to be
      # relocated to the storage array. This depends on whether the host has
      # been downgraded from a newer version of Bottlerocket, or whether it's
      # the first boot of an older version.
      if [ -d "${MOUNT_POINT}/${state_dir}" ] ; then
        # If downgrading from a version of Bottlerocket that supported bind
        # mounts, the directory will exist but should be empty, except for
        # subdirectories that may have been created by tmpfiles.d before an
        # upgrade to that version. Keep a copy of the directory just in case.
        rm -rf "${ROOT_PATH}/var/lib/${state_dir}.bak"
        mv "${ROOT_PATH}/var/lib/${state_dir}"{,.bak}
      else
        # Otherwise, treat it as the first boot of an older version, and move
        # the directory to the array.
        mv "${ROOT_PATH}/var/lib/${state_dir}" "${MOUNT_POINT}/${state_dir}"
      fi
    else
      # The host directory does not exist, so the target directory likely needs
      # to be created.
      mkdir -p "${MOUNT_POINT}/${state_dir}"
    fi
    # Any host directory has been dealt with and the symlink can be created.
    ln -snfT "/mnt/${MD_NAME}/${state_dir}" "${ROOT_PATH}/var/lib/${state_dir}"
  fi
done

# When using bind mounts, the parent directory where the array is mounted can
# be unmounted. This avoids a second, redundant mount entry under `/mnt` for
# every new mount in one of the state directories.
if [ "${should_umount}" == "yes" ] ; then
  umount "${MOUNT_POINT}"
fi

[yaroslav-nakonechnikov]

For everyone who will try that script - it won't work from scratch. You'll need to adapt it.

for what i noticed:

• blkid doesn't return anything in my case. But as it runs only at boot time - i removed that check.
• operations with containerd directory are not working. failing with:
  mv: can't remove '/.bottlerocket/rootfs/var/lib/containerd': Resource busy.
  i removed whole last part, as for my way of usage it would work to use ephemeral only for pod's "persistent" data.
• also, it may be usefull to do chmod 777 /.bottlerocket/rootfs/mnt/scratch if you plan to use FS with hostpath(https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) volumes.

[hgambarian]

@bcressey Hey, do you have a new script for bottlerocket 1.16x ?
Seems this one is not working, even after connecting to the instance and running that script locally, it gives me many errors

[runningman84]

@hgambarian how can you even run these scripts locally? In my tests I wasn't even able to access the block devices from the admin container...

[zepellin]

In the case anyone coming across this, the script above works for us with bottlerocket v1.18.0 (EKS) ran from bootstrap container. Thanks @bcressey !

[ei-grad]

Fails on instances without ephemeral disks, although the code was expected to exit early in such cases:

[   12.258083] bootstrap-containers@setup-runtime-storage[1553]: + ROOT_PATH=/.bottlerocket/rootfs
[FAILED] Failed to start bootstrap container setup-runtime-storage.
[   12.320404] bootstrap-containers@setup-runtime-storage[1553]: + declare -a EPHEMERAL_DISKS
See 'systemctl status bootstrap-contain…p-runtime-storage.service' for details.
[DEPEND] Dependency failed for Bottlerocket final configuration complete.
[   12.460644] bootstrap-containers@setup-runtime-storage[1553]: + EPHEMERAL_DISKS=("${ROOT_PATH}"/dev/disk/ephemeral/*)
[DEPEND] Dependency failed for Isolates multi-user.target.
[   12.580163] bootstrap-containers@setup-runtime-storage[1553]: + '[' 1 -eq 0 ']'
         Starting Reboot system if needed fo…guration changes to take effect...
[   12.634281] bootstrap-containers@setup-runtime-storage[1553]: + MD_NAME=scratch
[  OK  ] Finished Reboot system if needed fo…figuration changes to take effect.
[   12.740343] bootstrap-containers@setup-runtime-storage[1553]: + MD_DEVICE=/dev/md/scratch
[   12.840213] bootstrap-containers@setup-runtime-storage[1553]: + MD_CONFIG=/.bottlerocket/bootstrap-containers/current/mdadm.conf
[   12.910163] bootstrap-containers@setup-runtime-storage[1553]: + '[' '!' -s /.bottlerocket/bootstrap-containers/current/mdadm.conf ']'
[   12.980143] bootstrap-containers@setup-runtime-storage[1553]: + mdadm --create --force --verbose /dev/md/scratch --level=0 --name=scratch --raid-devices=1 '/.bottlerocket/rootfs/dev/disk/ephemeral/*'
[   13.080190] bootstrap-containers@setup-runtime-storage[1553]: mdadm: cannot open /.bottlerocket/rootfs/dev/disk/ephemeral/*: No such file or directory

It might also be more appropriate to exit early with code 0 instead of 1.

Corrected version: https://gist.github.com/ei-grad/7e24b6964988790694c30834c94f1571