Deduplicate github actions #892

petrutlucian94 · 2024-12-11T13:45:26Z

This PR introduces the following CI workflow improvements:

We have multiple github workflows that run e2e tests and share a significant amount of logic. We'll use reusable workflows and composite actions, making the workflows much easier to maintain.
Merge the following workflows:
- integration
- integration-informing
- python (we want the lint jobs to pass before running e2e tests)
The nightly and "cron" jobs are merged into a single workflow.
- To easily test cron jobs, we will configure them to run whenever the workflow definition changes.
Addresses a few flaky integration tests (test_ingress and test_microk8s_installed).

github-advanced-security · 2024-12-12T10:22:04Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

.github/workflows/nightly-test.yaml

petrutlucian94 · 2024-12-15T10:38:54Z

tests/test_ingress.py::test_ingress is constantly failing on Moonray, need to figure out why.

We have multiple github actions that run e2e tests and share a significant amount of logic. We'll add reusable actions, making the workflows much easier to maintain.

As part of the test cleanup, we're removing the k8s snap, ensuring that its services and mounts go away. One of the tests installs microk8s, which interferes with the k8s snap cleanup assertions. We'll fix this flaky test by removing the microk8s snap.

get_external_service_ip returns an empty string, however the test asserts that the ip is not None and proceeds with the curl: 2024-12-12 11:28:46 DEBUG Execute command ['curl', '', '-H', 'Host: foo.bar.com'] in instance k8s-integration-530bc4-37 We'll update the assertion and catch empty strings as well. At the same time, we'll increase the timeouts to reduce test flakiness.

The nightly job is also a cron job that executes daily, so it makes sense to merge those two workflows.

The moonray job is failing, however we only have logs from the "default" and "kube-system" namespaces. This change will collect logs from all k8s namespaces.

We'll need to apply the strict/moonray patches not only when building the snap, but also when running the tests.

test_containerd_path_cleanup_on_failed_init holds an open port and expects the bootstrap to fail, however that won't be the case when using the lxd harness. We'll skip this test for now.

berkayoz

Did a first pass, great work

tests/integration/tests/test_ingress.py

k8s/scripts/inspect.sh

.github/workflows/nightly-test.yaml

louiseschmidtgen

Nice work @petrutlucian94, I love the move of shared bits into actions!
Did a first pass and left some suggestions.

.github/actions/download-k8s-snap/action.yaml

.github/workflows/build-snap.yaml

.github/workflows/e2e-tests.yaml

.github/workflows/nightly-test.yaml

tests/integration/tests/test_cleanup.py

louiseschmidtgen · 2024-12-17T08:30:09Z

Could you please also update the integration test in k8s-dqlite to use your new actions? https://github.com/canonical/k8s-dqlite/blob/master/.github/workflows/k8s-snap-integration.yaml

petrutlucian94 · 2024-12-17T09:32:38Z

Could you please also update the integration test in k8s-dqlite to use your new actions? https://github.com/canonical/k8s-dqlite/blob/master/.github/workflows/k8s-snap-integration.yaml

Sure, we can handle the k8s-dqlite actions after this PR merges. Let's add a separate card though.

This reverts commit 5020f39.

bschimke95

Great work @petrutlucian94
Did a first pass

.github/actions/download-k8s-snap/action.yaml

.github/actions/install-lxd/action.yaml

.github/workflows/nightly-test.yaml

.github/workflows/security-scan.yaml

k8s/scripts/inspect.sh

* cover 1.32 as part of the nightly tests * get go version from go.mod * update step names * add some TODOs * make lxd channel configurable * bump ubuntu versions * add get-e2e-tags dependencies

petrutlucian94 · 2024-12-17T12:30:57Z

Follow-up items that will be handled in subsequent PRs:

move Trivy and TICS checks to separate scripts
add a job to automatically detect snap releases
- should also be used by microk8s
add "verbose" inspect.sh flag
update k8s-dqlite job: https://github.com/canonical/k8s-dqlite/blob/master/.github/workflows/k8s-snap-integration.yaml

bschimke95

LGTM

petrutlucian94 requested a review from a team as a code owner December 11, 2024 13:45

petrutlucian94 marked this pull request as draft December 11, 2024 13:45

petrutlucian94 force-pushed the lpetrut/ci_cleanup branch 27 times, most recently from b08d093 to 4c5c98e Compare December 12, 2024 10:21

petrutlucian94 marked this pull request as ready for review December 12, 2024 13:36

petrutlucian94 commented Dec 12, 2024

View reviewed changes

.github/workflows/nightly-test.yaml Outdated Show resolved Hide resolved

petrutlucian94 force-pushed the lpetrut/ci_cleanup branch from a621aed to 413bc2a Compare December 13, 2024 07:08

petrutlucian94 added 7 commits December 16, 2024 16:12

Deduplicate github actions

b7aff35

We have multiple github actions that run e2e tests and share a significant amount of logic. We'll add reusable actions, making the workflows much easier to maintain.

Fix flaky microk8s test

86ee937

As part of the test cleanup, we're removing the k8s snap, ensuring that its services and mounts go away. One of the tests installs microk8s, which interferes with the k8s snap cleanup assertions. We'll fix this flaky test by removing the microk8s snap.

Merge nightly test and cron job

1324c7e

The nightly job is also a cron job that executes daily, so it makes sense to merge those two workflows.

Fix nightly job tag

3582e0c

Pass test flavor

7db793a

Include all namespaces in inspection reports

5020f39

The moonray job is failing, however we only have logs from the "default" and "kube-system" namespaces. This change will collect logs from all k8s namespaces.

petrutlucian94 force-pushed the lpetrut/ci_cleanup branch from 6909408 to 5020f39 Compare December 16, 2024 14:12

petrutlucian94 added 2 commits December 16, 2024 18:11

Apply flavor patches before running the tests

b99b334

We'll need to apply the strict/moonray patches not only when building the snap, but also when running the tests.

Skip broken test

25e9901

test_containerd_path_cleanup_on_failed_init holds an open port and expects the bootstrap to fail, however that won't be the case when using the lxd harness. We'll skip this test for now.

petrutlucian94 force-pushed the lpetrut/ci_cleanup branch from e21789e to 25e9901 Compare December 16, 2024 16:27

berkayoz reviewed Dec 17, 2024

View reviewed changes

tests/integration/tests/test_ingress.py Show resolved Hide resolved

k8s/scripts/inspect.sh Outdated Show resolved Hide resolved

.github/workflows/nightly-test.yaml Show resolved Hide resolved

.github/workflows/nightly-test.yaml Outdated Show resolved Hide resolved

louiseschmidtgen reviewed Dec 17, 2024

View reviewed changes

Revert "Include all namespaces in inspection reports"

256baf4

This reverts commit 5020f39.

bschimke95 reviewed Dec 17, 2024

View reviewed changes

petrutlucian94 force-pushed the lpetrut/ci_cleanup branch from b5251bd to 2057dbf Compare December 17, 2024 12:19

Address PR feedback

7267858

* cover 1.32 as part of the nightly tests * get go version from go.mod * update step names * add some TODOs * make lxd channel configurable * bump ubuntu versions * add get-e2e-tags dependencies

petrutlucian94 force-pushed the lpetrut/ci_cleanup branch from 2057dbf to 7267858 Compare December 17, 2024 12:29

bschimke95 approved these changes Dec 17, 2024

View reviewed changes

petrutlucian94 merged commit 5c3e5e0 into main Dec 18, 2024
30 checks passed

petrutlucian94 deleted the lpetrut/ci_cleanup branch December 18, 2024 09:41

berkayoz mentioned this pull request Dec 18, 2024

Fix containerd-related path cleanup on failed bootstrap/join and associated test. #910

Merged

petrutlucian94 mentioned this pull request Jan 8, 2025

CI improvements #927

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deduplicate github actions #892

Deduplicate github actions #892

petrutlucian94 commented Dec 11, 2024 •

edited

Loading

github-advanced-security bot commented Dec 12, 2024

petrutlucian94 commented Dec 15, 2024

berkayoz left a comment

louiseschmidtgen left a comment

louiseschmidtgen commented Dec 17, 2024

petrutlucian94 commented Dec 17, 2024 •

edited

Loading

bschimke95 left a comment

petrutlucian94 commented Dec 17, 2024

bschimke95 left a comment

Deduplicate github actions #892

Deduplicate github actions #892

Conversation

petrutlucian94 commented Dec 11, 2024 • edited Loading

github-advanced-security bot commented Dec 12, 2024

petrutlucian94 commented Dec 15, 2024

berkayoz left a comment

Choose a reason for hiding this comment

louiseschmidtgen left a comment

Choose a reason for hiding this comment

louiseschmidtgen commented Dec 17, 2024

petrutlucian94 commented Dec 17, 2024 • edited Loading

bschimke95 left a comment

Choose a reason for hiding this comment

petrutlucian94 commented Dec 17, 2024

bschimke95 left a comment

Choose a reason for hiding this comment

petrutlucian94 commented Dec 11, 2024 •

edited

Loading

petrutlucian94 commented Dec 17, 2024 •

edited

Loading