Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(*): merge build action #548

Merged
merged 10 commits into from
Apr 15, 2024
Merged

ci(*): merge build action #548

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d7e4624
ci(*): merge build action
Mossaka Apr 3, 2024
8007d8b
ci(*): add arch to slug
Mossaka Apr 3, 2024
8741d1e
disable image signing on windows
Mossaka Apr 3, 2024
b205f19
add id-token permission to ci
Mossaka Apr 3, 2024
50df349
use the right artifact name for downloading
Mossaka Apr 3, 2024
2c0eaa5
remove permission for write contents in build action
Mossaka Apr 3, 2024
19cedc5
feat(ci): disable signing in build ci and add schemas
Mossaka Apr 7, 2024
61dc1ee
feat(ci): modify the runtime_sub step
Mossaka Apr 7, 2024
df5f1f1
feat(ci): use annotations for logs
Mossaka Apr 11, 2024
9599c79
feat(ci): extract signing to it's own action
Mossaka Apr 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 56 additions & 2 deletions .github/workflows/action-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@

# yaml-language-server: $schema=https://json.schemastore.org/github-action.json

# this action will require write permission to id-token
name: Build shims

on:
Expand All @@ -15,13 +19,24 @@ on:
slug:
required: true
type: string
arch:
required: false
type: string
sign:
default: false
type: boolean

jobs:
build:
build-sign-upload:
permissions:
id-token: write
name: build for ${{ inputs.slug }}
runs-on: ${{ inputs.os }}
steps:
- uses: actions/checkout@v3
- name: describe runner
run: |
echo "Running job with os: '${{ inputs.os }}', arch: '${{ inputs.arch }}', slug: '${{ inputs.slug }}', runtime: '${{ inputs.runtime }}', target: '${{ inputs.target }}'"
Mossaka marked this conversation as resolved.
Show resolved Hide resolved
- uses: actions/checkout@v4
- name: Setup build env
run: |
os=$(echo "$RUNNER_OS" | tr '[:upper:]' '[:lower:]')
Expand All @@ -35,12 +50,51 @@ jobs:
- name: Setup cross-rs
if: runner.os == 'Linux'
run: ./scripts/setup-cross.sh ${{ inputs.target }}
- name: Setup cosign for signing
if: ${{ inputs.runtime != 'common' && inputs.sign }}
uses: sigstore/[email protected]
with:
cosign-release: 'v2.2.2'
- name: Setup build profile
shell: bash
run: echo "OPT_PROFILE=release" >> ${GITHUB_ENV}
- name: Build
run: make build-${{ inputs.runtime }}
- name: Run tests
timeout-minutes: 5
run: |
make test-${{ inputs.runtime }}
if: ${{ inputs.arch == 'x86_64' }}
- name: Sign the binary
Mossaka marked this conversation as resolved.
Show resolved Hide resolved
if: ${{ inputs.runtime != 'common' && inputs.slug != 'windows' && inputs.sign }}
run: |
make dist-${{ inputs.runtime }}
# Check if there's any files to archive as tar fails otherwise
if stat dist/bin/* >/dev/null 2>&1; then
cosign sign-blob --yes \
--output-signature containerd-shim-${{ inputs.runtime }}-v1.sig \
--output-certificate containerd-shim-${{ inputs.runtime }}-v1.pem \
--bundle containerd-shim-${{ inputs.runtime }}-v1.bundle \
dist/bin/containerd-shim-${{ inputs.runtime }}-v1

cosign sign-blob --yes \
--output-signature containerd-shim-${{ inputs.runtime }}d-v1.sig \
--output-certificate containerd-shim-${{ inputs.runtime }}d-v1.pem \
--bundle containerd-shim-${{ inputs.runtime }}d-v1.bundle \
dist/bin/containerd-shim-${{ inputs.runtime }}d-v1

cosign sign-blob --yes \
--output-signature containerd-${{ inputs.runtime }}d.sig \
--output-certificate containerd-${{ inputs.runtime }}d.pem \
--bundle containerd-${{ inputs.runtime }}d.bundle \
dist/bin/containerd-${{ inputs.runtime }}d

# Copy the certs to the dist/bin folder
cp *.sig dist/bin/
cp *.pem dist/bin/
else
echo "No files to sign"
fi
- name: Package artifacts
if: ${{ inputs.runtime != 'common' }}
shell: bash
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/action-fmt.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# yaml-language-server: $schema=https://json.schemastore.org/github-action.json

name: Run lint

on:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/action-test-image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# yaml-language-server: $schema=https://json.schemastore.org/github-action.json

name: Run end to end tests on kind

on:
Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/action-test-k3s.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# yaml-language-server: $schema=https://json.schemastore.org/github-action.json

name: Run end to end tests on k3s

on:
Expand All @@ -22,13 +24,13 @@ jobs:
- name: Download artifacts
uses: actions/download-artifact@master
with:
name: containerd-shim-${{ inputs.runtime }}-linux-musl
name: containerd-shim-${{ inputs.runtime }}-x86_64-linux-musl
path: dist
- name: Unpack artifats
shell: bash
run: |
mkdir -p dist/bin
tar -xzf dist/containerd-shim-${{ inputs.runtime }}-linux-musl.tar.gz -C dist/bin
tar -xzf dist/containerd-shim-${{ inputs.runtime }}-x86_64-linux-musl.tar.gz -C dist/bin
- name: Download test image
uses: actions/download-artifact@master
with:
Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/action-test-kind.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# yaml-language-server: $schema=https://json.schemastore.org/github-action.json

name: Run end to end tests on kind

on:
Expand Down Expand Up @@ -28,13 +30,13 @@ jobs:
- name: Download artifacts
uses: actions/download-artifact@master
with:
name: containerd-shim-${{ inputs.runtime }}-linux-musl
name: containerd-shim-${{ inputs.runtime }}-x86_64-linux-musl
path: dist
- name: Unpack artifats
shell: bash
run: |
mkdir -p dist/bin
tar -xzf dist/containerd-shim-${{ inputs.runtime }}-linux-musl.tar.gz -C dist/bin
tar -xzf dist/containerd-shim-${{ inputs.runtime }}-x86_64-linux-musl.tar.gz -C dist/bin
- name: Download test image
uses: actions/download-artifact@master
with:
Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/action-test-smoke.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# yaml-language-server: $schema=https://json.schemastore.org/github-action.json

name: Run smoke tests

on:
Expand All @@ -22,13 +24,13 @@ jobs:
- name: Download artifacts
uses: actions/download-artifact@master
with:
name: containerd-shim-${{ inputs.runtime }}-linux-musl
name: containerd-shim-${{ inputs.runtime }}-x86_64-linux-musl
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting, I don't see where this arch was being included in the inputs.runtime before. Or is this extra field new?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this PR, only the release pipeline runs both x86_64 and aarch64. The ci does not have a matrix for different archs. I added that to make the build job on par with the one in the release CI.

path: dist
- name: Unpack artifats
shell: bash
run: |
mkdir -p dist/bin
tar -xzf dist/containerd-shim-${{ inputs.runtime }}-linux-musl.tar.gz -C dist/bin
tar -xzf dist/containerd-shim-${{ inputs.runtime }}-x86_64-linux-musl.tar.gz -C dist/bin
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Maybe make this a makefile target since the tar command is repeated in multiple workflows?

- name: Download test image
uses: actions/download-artifact@master
with:
Expand Down
12 changes: 10 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# yaml-language-server: $schema=https://json.schemastore.org/github-action.json

name: CI

on:
Expand Down Expand Up @@ -39,20 +41,26 @@ jobs:

build-ubuntu:
name: ${{ matrix.runtime }}
permissions:
id-token: write
strategy:
matrix:
os: ["ubuntu-22.04"]
runtime: ["common", "wasmtime", "wasmedge", "wasmer"]
libc: ["musl", "gnu"]
arch: ["x86_64", "aarch64"]
uses: ./.github/workflows/action-build.yml
with:
os: ${{ matrix.os }}
runtime: ${{ matrix.runtime }}
target: "x86_64-unknown-linux-${{ matrix.libc }}"
slug: "linux-${{ matrix.libc }}"
target: "${{ matrix.arch }}-unknown-linux-${{ matrix.libc }}"
slug: "${{ matrix.arch }}-linux-${{ matrix.libc }}"
arch: ${{ matrix.arch }}

build-windows:
name: ${{ matrix.runtime }}
permissions:
id-token: write
strategy:
matrix:
os: ["windows-latest"]
Expand Down
99 changes: 18 additions & 81 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# yaml-language-server: $schema=https://json.schemastore.org/github-action.json

name: Release
run-name: ${{ inputs.crate }}@${{ inputs.version }} (DryRun:${{ inputs.dry_run }})

Expand Down Expand Up @@ -64,10 +66,15 @@ jobs:
with:
script: |
const crate = '${{ inputs.crate }}';
let runtime = crate.replace(/^containerd-shim-/, '');
core.setOutput('runtime', runtime);
const non_shim_crates = ['wasm', 'wasm-test-modules', 'oci-tar-builder'];
core.setOutput('is_shim', !non_shim_crates.includes(runtime));
if non_shim_crates.includes(runtime) {
core.setOutput('runtime', 'common');
core.setOutput('is_shim', false)
} else {
const runtime = crate.replace(/^containerd-shim-/, '');
core.setOutput('runtime', runtime);
core.setOutput('is_shim', true);
}
### If we are releasing a crate rather than producing a bin, check for crates.io access
- name: Check crates.io ownership
if: ${{ steps.runtime_sub.outputs.is_shim != 'true' }}
Expand All @@ -85,7 +92,6 @@ jobs:

build-and-sign:
permissions:
# cosign uses the GitHub OIDC token
id-token: write
needs:
- pre-release
Expand All @@ -94,83 +100,14 @@ jobs:
arch: ["x86_64", "aarch64"]
include:
- ${{ needs.pre-release.outputs }}
runs-on: "ubuntu-22.04"
steps:
- name: Matrix description
run: |
echo "Running job with dry_run: '${{ inputs.dry_run }}' crate: '${{ matrix.crate }}', version: '${{ matrix.version }}', runtime: '${{ matrix.runtime }}', and is_shim: '${{ matrix.is_shim }}'."
- uses: actions/checkout@v4
- name: Setup build env
run: ./scripts/setup-linux.sh
- name: Setup rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
env:
RUST_CACHE_KEY_OS: rust-release-cache-${{ matrix.crate }}-${{ matrix.arch }}
with:
rustflags: '' #Disable. By default this action sets environment variable is set to -D warnings. We manage this in the Makefile
- name: Setup cross-rs
run: ./scripts/setup-cross.sh ${{ matrix.arch }}-unknown-linux-musl
- name: Setup build profile
shell: bash
run: echo "OPT_PROFILE=release" >> ${GITHUB_ENV}
- name: Setup cosign for signing
if: ${{ matrix.is_shim == 'true' }}
uses: sigstore/[email protected]
with:
cosign-release: 'v2.2.2'
- name: Build
timeout-minutes: 20
run: make build-${{ matrix.runtime }}
- name: Test
if: ${{ matrix.arch == 'x86_64' }}
timeout-minutes: 10
run: make test-${{ matrix.runtime }}
- name: Sign the binary
if: ${{ matrix.is_shim == 'true' }}
run: |
make dist-${{ matrix.runtime }}
# Check if there's any files to archive as tar fails otherwise
if stat dist/bin/* >/dev/null 2>&1; then
cosign sign-blob --yes \
--output-signature containerd-shim-${{ matrix.runtime }}-v1.sig \
--output-certificate containerd-shim-${{ matrix.runtime }}-v1.pem \
--bundle containerd-shim-${{ matrix.runtime }}-v1.bundle \
dist/bin/containerd-shim-${{ matrix.runtime }}-v1

cosign sign-blob --yes \
--output-signature containerd-shim-${{ matrix.runtime }}d-v1.sig \
--output-certificate containerd-shim-${{ matrix.runtime }}d-v1.pem \
--bundle containerd-shim-${{ matrix.runtime }}d-v1.bundle \
dist/bin/containerd-shim-${{ matrix.runtime }}d-v1

cosign sign-blob --yes \
--output-signature containerd-${{ matrix.runtime }}d.sig \
--output-certificate containerd-${{ matrix.runtime }}d.pem \
--bundle containerd-${{ matrix.runtime }}d.bundle \
dist/bin/containerd-${{ matrix.runtime }}d

# Copy the certs to the dist/bin folder
cp *.sig dist/bin/
cp *.pem dist/bin/
else
echo "No files to sign"
fi
- name: Package artifacts
if: ${{ matrix.is_shim == 'true' }}
shell: bash
run: |
# Check if there's any files to archive as tar fails otherwise
if stat dist/bin/* >/dev/null 2>&1; then
tar -czf dist/containerd-shim-${{ matrix.runtime }}-${{ matrix.arch }}.tar.gz -C dist/bin .
else
tar -czf dist/containerd-shim-${{ matrix.runtime }}-${{ matrix.arch }}.tar.gz -T /dev/null
fi
- name: Upload artifacts
if: ${{ matrix.is_shim == 'true' }}
uses: actions/upload-artifact@master
with:
name: containerd-shim-${{ matrix.runtime }}-${{ matrix.arch }}
path: dist/containerd-shim-${{ matrix.runtime }}-${{ matrix.arch }}.tar.gz
uses: ./.github/workflows/action-build.yml
with:
os: "ubuntu-22.04"
runtime: ${{ matrix.runtime }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there will be times when this value won't be common as the check in the action is expecting

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, can you elaborate your concerns here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This value in the release.yml will be something like oci-tar-builder

runwasi/.github/workflows/release.yml

Lines 66 to 69 in 8c98100

const crate = '${{ inputs.crate }}';
let runtime = crate.replace(/^containerd-shim-/, '');
core.setOutput('runtime', runtime);
const non_shim_crates = ['wasm', 'wasm-test-modules', 'oci-tar-builder'];

and I think signing will fail in the new signing step at https://github.com/containerd/runwasi/pull/548/files#diff-43035769cde349b680ebf72f18e67b049e730d0d0ec91174e6455a685b3c2accR61-R64

Copy link
Member Author

@Mossaka Mossaka Apr 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! I have modified the logic of runtime_sub step to replace the name of non-shim crate to common in this commit: 61dc1ee

target: "${{ matrix.arch }}-unknown-linux-musl"
slug: "${{ matrix.arch }}-linux-musl"
arch: ${{ matrix.arch }}
sign: true

release:
permissions:
Expand Down
Loading