multi-kernel, cross-compiling, bash based Hook & (default+foreign) ke…

…rnels build (incl GHA matrix) (#205) ### multi-kernel, cross-compiling, bash based Hook & (default+external) kernels build (incl GHA matrix) Original RFC below; please check [commits](https://github.com/tinkerbell/hook/pull/205/commits) for the (many) updates done after the first drop. Original RFC kept for reference, below the line. -------------- ### RFC: multi-kernel, cross-compiling, bash based Hook & (default+external) kernels build (incl GHA matrix) > _very early stage RFC_ This is a rewrite of the build system. The _produced_ default artifacts (aarch64/x86_64) should be equivalent, save for an updated 5.10.213+ kernel and arm64 fixes. It's missing, at least, documentation and linters, possibly more, that I removed and intend to rewrite. But since it's a large-ish change, I'd like to collect some feedback before continuing. ### Main topics - Makefile build replaced with bash. I'm not too proud of the bash, I will definitely clean it up a lot -- it's the concepts I'd need confirmation on. - Replaces the emulated Alpine kernel build with a Debian based cross-compiling build - Much faster building. Emulating x86_64 on arm64 is very slow and vice-versa. - Replaces kernel .config's with the `defconfig` versions, via Kbuild's `make savedefconfig` - Replaces Git-SHA1-based image versioning ("current revision") with content-based hashing. - This way, there's much higher cache reuse, and new versions are pushed only when components actually changed (caveat emptor) - Should allow people to develop Hook without having to build a kernel, depending on CI frequency and luck. - Introduces multiple "flavors" of hook. Instead of restricted to 2 hardcoded flavors (x86_64 and aarch64, built from source), we can now define multiple flavors, each with an ID and version/configure/build methods. - the `hook-default-amd64` and `hook-default-arm64` kernels are equivalent to the two original. - the `armbian-` prefixed kernels are actually Armbian kernels for more exotic arm64 SBCs, or Armbian's generic UEFI kernels for both arches. Those are very fast to "build" since Armbian publishes their .deb packages in OCI images, and here we just download and massage them into the format required by Linuxkit. - `hook.yaml` is replaced with `hook.template.yaml` which is templated via a limited-var invocation of `envsubst`; only the kernel image and the arch is actually different per-flavor. - Introduced a distributed GitHub Actions build workflow. The bash build system produces JSON objects that drive the matrix stages: - One matrix is per-arch, and builds all the containers whose source is hosted in this repo (bootkit, docker, mdev) - Second matrix is per-flavor(/kernel), and builds the kernel - Third matrix, depending on the other two, is per-flavor(/kernel), and builds Hook itself (via LinuxKit) and prepares a .tar.gz into GH artifacts - Auto-updating of the kernel via kernel.org's JSON endpoint (ofc only works for LTS or recent-enough stable kernels). Could opt-out/use a fixed version. - Auto updating of Armbian kernels via OCI tag listing via `skopeo`. Can opt-out/use a fixed version. - DTB-producing Kernel builds (aarch64) produce a `dtbs.tar.gz` artifact together with the initrd and vmlinuz. #### Flavors (/kernels) ##### Hook's own kernels | ID | Current version | Description | |----------------------|-----------------|---------------------------| | `hook-default-arm64` | 5.10.213 | Hook's own aarch64 kernel | | `hook-default-amd64` | 5.10.213 | Hook's own x86_64 kernel | ##### Armbian kernels - External kernels, taken from Armbian's OCI repos. Those are "exotic" kernels for certain SoC's. - `edge`: release candidates or stable but rarely LTS, more aggressive patching - `current`: LTS kernels, stable-ish patching | ID | Current version | Description | |---------------------------|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------| | `armbian-bcm2711-current` | 6.6.22 | bcm2711 (Broadcom) current, from RaspberryPi Foundation with many Armbian fixes for CNCF-landscape projects; for the RaspberryPi 3b+/4b/5 | | `armbian-meson64-edge` | 6.7.10 | meson64 (Amlogic) edge Khadas VIM3/3L, Radxa Zero/2, LibreComputer Potatos, and many more | | `armbian-rockchip64-edge` | 6.7.10 | rockchip64 (Rockchip) edge, for many rk356x/3399 SoCs. Not for rk3588! | | `armbian-uefi-arm64-edge` | 6.8.1 | Armbian generic edge UEFI kernel | | `armbian-uefi-x86-edge` | 6.8.1 | Armbian generic edge UEFI kernel | #### Proof of working-ness? In my fork: ![image](https://github.com/tinkerbell/hook/assets/639959/7fd7db57-9642-4b8a-8d6f-8539e9fc5fd9) - https://github.com/rpardini/tinkerbell-hook/actions/runs/8396946361 A full build workflow, with all misses: - Default kernels build (in GHA default runners) in 15-21 minutes each - Armbian kernels are done in less than 2 minutes each - LK containers in 1 minute for x86_64, 4 minutes for arm64 (done under qemu) - Most Hook builds in around 1 minute - Full build done in <24m. - https://github.com/rpardini/tinkerbell-hook/actions/runs/8396643610/job/22998447592 A full build run with all cache hits - Everything done in <4 minutes - https://github.com/rpardini/tinkerbell-hook/releases/tag/20240322-2128 artifacts for testing (initramfs+vmlinuz+dtbs artifact for each flavor) - https://github.com/rpardini?tab=packages&repo_name=tinkerbell-hook OCI packages in ghcr.io ([quay.io had a meltdown today](https://status.redhat.com/incidents/qh68rjfg6xs6)) #### Future possibilities: - it would be fairly simple to add Debian/Ubuntu kernels as well as Armbian firmware. - Many, many more Armbian kernels could be added, but save for Allwinner and the Rockchip `-rkr` vendor kernel, I think they might be too niche. Users should have an easy time adding it themselves if they need, though. - Better support for u-boot's "pxelinux" booting requires changes outside of Hook (namely in Smee/ipxedust) which I'll PR eventually. - Certain arm64 SoCs require changes in iPXE (nap.h) -- same, I'll PR those to ipxedust repo. - All these Hook flavors are used in a "showcase" Helm chart based on stack that I will also PR to the charts repo. #### TO-DO - Find a better name for "flavor". Naming things is hard. Hats? Swords? - Update README.md and CONTRIBUTING.md; For now all I have is really - `bash build.sh config-kernel <flavor>` & follow instructions to configure kernel; only works for default flavors - `bash build.sh build-kernel <flavor>` builds the kernel - `bash build.sh build <flavor>` builds Hook with that kernel - Restore golang & shellcheck linting - Update to Linuxkit 1.2.0 and new linuxkit pkgs. I tried, but there's some incompatible changes that we need to figure out. - Consider using `actuated` for native arm64 building? -- https://actuated.dev/blog/arm-ci-cncf-ampere --- Thanks for reading this far. I'm looking forward to your feedback!
tinkerbell · Apr 30, 2024 · df70921 · df70921
2 parents 1ab4a76 + ab6b609
commit df70921
Show file tree

Hide file tree

Showing 48 changed files with 6,689 additions and 14,419 deletions.
diff --git a/.envrc b/.envrc
diff --git a/.github/workflows/build-all-matrix.yaml b/.github/workflows/build-all-matrix.yaml
@@ -0,0 +1,188 @@
+name: All Kernels and Hooks
+on:
+  #schedule:
+  #  # every day at 5am UTC
+  #  - cron: '0 5 * * *'
+  workflow_dispatch:
+
+env: # Global environment, passed to all jobs & all steps
+  # Default to quay.io, which is also the default for the CLI. 
+  # Allow to use ghcr.io as an alternative, change & uncomment below:
+  REGISTRY: "quay.io" # or ghcr.io, determines which will be logged-in to
+  #HOOK_KERNEL_OCI_BASE: "ghcr.io/${{ github.repository_owner }}/tinkerbell/kernel-"
+  #HOOK_LK_CONTAINERS_OCI_BASE: "ghcr.io/${{ github.repository_owner }}/tinkerbell/linuxkit-"
+
+  # Apart from the quay/ghcr coordinates above (used for both pulling & pushing), we might also want to
+  # log in to DockerHub (with a read-only token) so we aren't hit by rate limits when pulling the linuxkit pkgs.
+  # To do so, set the secret DOCKER_USERNAME and DOCKER_PASSWORD in the repo secrets, and set the below to yes.
+  LOGIN_TO_DOCKERHUB: "yes"
+
+  HOOK_VERSION: "0.9.0-alpha1-build-${{github.run_number}}" # Use a forced Hook version
+
+  # Which flavors to build? space separated list, must match one of the TAG='s in flavors (this is used by matrix_prep job in gha-matrix command)
+  CI_TAGS: "standard armbian-sbc armbian-uefi lts" # 'dev' is not included
+
+  # GHA runner configuration. See bash/json-matrix.sh for more details.
+  CI_RUNNER_LK_CONTAINERS_ARM64: "ARM64" # Use a self-hosted runner with the "ARM64" tag for the ARM64 builds of LK containers
+
+jobs:
+
+  matrix_prep:
+    name: "Prepare matrix JSON"
+    runs-on: ubuntu-latest
+    outputs:
+      created: ${{ steps.date_prep.outputs.created }} # refer to as ${{needs.prepare.outputs.created}}
+      kernels_json: ${{ steps.prepare-matrix.outputs.kernels_json }}
+      lkcontainers_json: ${{ steps.prepare-matrix.outputs.lkcontainers_json }}
+      lk_hooks_json: ${{ steps.prepare-matrix.outputs.lk_hooks_json }}
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v4
+
+      - name: Prepare release ID (current date) # This only used for the GitHub Release; not included in any way in the build process.
+        id: date_prep
+        run: echo "created=$(date -u +'%Y%m%d-%H%M')" >> "${GITHUB_OUTPUT}"
+
+      - name: Run the matrix JSON preparation bash script
+        id: prepare-matrix
+        run: bash build.sh gha-matrix # This sets the output "kernels_json" & "lkcontainers_json" & "lk_hooks_json" internally
+
+  build-linuxkit-containers:
+    needs: [ matrix_prep ]
+    runs-on: "${{ matrix.runner }}" # the runner to use is determined by the 'gha-matrix' code
+    strategy:
+      fail-fast: true
+      matrix:
+        include: ${{ fromJSON(needs.matrix_prep.outputs.lkcontainers_json) }}
+    name: "LinuxKit containers for ${{ matrix.docker_arch }}"
+    steps:
+      - name: Checkout build repo
+        uses: actions/checkout@v4
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Docker Login to quay.io
+        if: ${{ env.REGISTRY == 'quay.io' }}
+        uses: docker/login-action@v3
+        with: { registry: "quay.io", username: "${{ secrets.QUAY_USERNAME }}", password: "${{ secrets.QUAY_PASSWORD }}" }
+
+      - name: Docker Login to GitHub Container Registry
+        if: ${{ env.REGISTRY == 'ghcr.io' }}
+        uses: docker/login-action@v3
+        with: { registry: "ghcr.io", username: "${{ github.repository_owner }}", password: "${{ secrets.GITHUB_TOKEN }}" }
+
+      - name: Build and Push LinuxKit containers for ${{matrix.docker_arch}}
+        env:
+          DOCKER_ARCH: "${{ matrix.docker_arch }}"
+          DO_PUSH: "yes"
+        run: bash build.sh linuxkit-containers
+
+  build-kernels:
+    needs: [ matrix_prep ] # depend on the previous job...
+    runs-on: "${{ matrix.runner }}" # the runner to use is determined by the 'gha-matrix' code
+    strategy:
+      fail-fast: false # let other jobs try to complete if one fails, kernels might take long, and they'd be skipped on the next run
+      matrix:
+        include: ${{ fromJSON(needs.matrix_prep.outputs.kernels_json) }}
+    name: "Kernel ${{ matrix.kernel }}"
+    steps:
+      - name: Checkout build repo
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx # nb: no need for qemu here, kernels are cross-compiled, instead of the compilation being emulated
+        uses: docker/setup-buildx-action@v3
+
+      - name: Docker Login to quay.io
+        if: ${{ env.REGISTRY == 'quay.io' }}
+        uses: docker/login-action@v3
+        with: { registry: "quay.io", username: "${{ secrets.QUAY_USERNAME }}", password: "${{ secrets.QUAY_PASSWORD }}" }
+
+      - name: Docker Login to GitHub Container Registry
+        if: ${{ env.REGISTRY == 'ghcr.io' }}
+        uses: docker/login-action@v3
+        with: { registry: "ghcr.io", username: "${{ github.repository_owner }}", password: "${{ secrets.GITHUB_TOKEN }}" }
+
+      - name: Build and push Kernel ${{matrix.kernel}} (${{ matrix.arch }})
+        env:
+          DO_PUSH: "yes"
+        run: bash build.sh build-kernel "${{ matrix.kernel }}"
+
+  build-hook-ensemble:
+    needs: [ matrix_prep, build-linuxkit-containers, build-kernels ] # depend on the previous job...
+    runs-on: "${{ matrix.runner }}" # the runner to use is determined by the 'gha-matrix' code
+    strategy:
+      fail-fast: false # let other jobs try to complete if one fails
+      matrix:
+        include: ${{ fromJSON(needs.matrix_prep.outputs.lk_hooks_json) }}
+    name: "Hook ${{ matrix.kernel }}"
+    steps:
+      - name: Checkout build repo
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx # nb: no need for qemu here, kernels are cross-compiled, instead of the compilation being emulated
+        uses: docker/setup-buildx-action@v3
+
+      - name: Docker Login to DockerHub # read-only token, required to be able to pull all the linuxkit pkgs without getting rate limited.
+        if: ${{ env.LOGIN_TO_DOCKERHUB == 'yes' }}
+        uses: docker/login-action@v3
+        with: { registry: "docker.io", username: "${{ secrets.DOCKER_USERNAME }}", password: "${{ secrets.DOCKER_PASSWORD }}" }
+
+      - name: Docker Login to quay.io
+        if: ${{ env.REGISTRY == 'quay.io' }}
+        uses: docker/login-action@v3
+        with: { registry: "quay.io", username: "${{ secrets.QUAY_USERNAME }}", password: "${{ secrets.QUAY_PASSWORD }}" }
+
+      - name: Docker Login to GitHub Container Registry
+        if: ${{ env.REGISTRY == 'ghcr.io' }}
+        uses: docker/login-action@v3
+        with: { registry: "ghcr.io", username: "${{ github.repository_owner }}", password: "${{ secrets.GITHUB_TOKEN }}" }
+
+      - name: GitHub Actions Cache for 'cache' dir
+        uses: actions/cache@v4
+        if: ${{ matrix.gha_cache == 'yes' }} # only set to yes for GH-hosted runners; see gha-matrix bash impl
+        with:
+          path: cache
+          key: "lk-cache-${{ matrix.docker_arch }}-${{ matrix.kernel }}-${{ hashFiles('linuxkit-templates/*') }}-${{ hashFiles('bash/**/*.sh') }}"
+          restore-keys: |
+            lk-cache-${{ matrix.docker_arch }}-${{ matrix.kernel }}
+            lk-cache-${{ matrix.docker_arch }}
+          save-always: true # always save the cache, even if build fails
+
+      - name: "Build Hook with Kernel ${{matrix.kernel}} (${{ matrix.arch }}) - cache: ${{matrix.gha_cache}}"
+        env:
+          DO_BUILD_LK_CONTAINERS: "no" # already built them; this is only for hook/linuxkit.
+        run: bash build.sh build "${{ matrix.kernel }}"
+
+      - name: Upload deb as artifact ${{ matrix.arch.name }} ${{ matrix.distro }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: "hook-tarball-${{ matrix.kernel }}"
+          path: out/*.tar.gz
+
+  release:
+    name: Publish all Hooks to GitHub Releases
+    needs: [ matrix_prep, build-hook-ensemble ]
+    runs-on: ubuntu-latest
+    steps:
+
+      - name: Download built Hook artifacts
+        uses: actions/download-artifact@v4
+        with:
+          pattern: "hook-tarball-*"
+          merge-multiple: true
+          path: out
+
+      # Release the artifacts into GitHub Releases. @TODO this GHA Action is not ideal, uses old nodejs, but I can't find a better one.
+      - name: "GH Release"
+        uses: "marvinpinto/action-automatic-releases@latest"
+        with:
+          repo_token: "${{ secrets.GITHUB_TOKEN }}"
+          automatic_release_tag: "${{needs.matrix_prep.outputs.created}}"
+          prerelease: false
+          title: "${{needs.matrix_prep.outputs.created}}"
+          files: |
+            out/*.tar.gz
diff --git a/.gitignore b/.gitignore
@@ -2,10 +2,13 @@ bin/
 dbg/
 dist/
 .env
-hook-*
 !/hook-bootkit/
 !/hook-docker/
 !/hook-mdev/
 hook*.*.yaml
+!linuxkit-templates
 out/
+cache/
 *.swp
+.idea
+kernel/Dockerfile.autogen.*
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -17,20 +17,7 @@ Please read and understand the DCO found [here](docs/DCO.md).
 
 ### Environment Details
 
-Building is handled by `make`, please see the [Makefile](Makefile) for available targets.
-
-#### Nix
-
-This repo's build environment can be reproduced using `nix`.
-
-##### Install Nix
-
-Follow the [Nix installation](https://nixos.org/download.html) guide to setup Nix on your box.
-
-##### Load Dependencies
-
-Loading build dependencies is as simple as running `nix-shell` or using [lorri](https://github.com/nix-community/lorri).
-If you have `direnv` installed the included `.envrc` will make that step automatic.
+Building is handled by a bash script, please see the [build.sh](build.sh) for details.
 
 ### How to Submit Change Requests
 

diff --git a/Makefile b/Makefile