Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement CI bootloader script #86

Merged
merged 9 commits into from
Nov 11, 2024

Conversation

SimeonEhrig
Copy link
Member

At the moment, sometimes if I changes something on the CI, I also need to modify the content of .gitlab-ci.yaml in the projects, which uses the CI scripts. At the moment, there are 4 projects, where I need to do the same changes. To remove the need of this redundant work, I want to implement a CI bootloader.

The basic idea of the bootloader is, there are only two jobs defined in the .gitlab-ci.yaml. The first job downloads the CI bootloader and execute it. The bootloader, scans it's environment (environment variables) and creates depending on it a GitLab CI yaml file, which contains the whole test pipeline such as unit and integration tests. The second job simply executes the CI pipeline.

In the best cases, we don't need to change the .gitlab-ci.yaml anymore, if we change the CI scripts. Maybe it is required to set optional variables for single projects. For example we change the default Julia version for integration tests from 1.10 to 1.11 but project X does not support it yet. Therefore it still needs to run with version 1.10 until the problem is solved.

Possible .gitlab-ci.yaml:

stages:
- generate
- run

generate_pipeline:
  image: julia:1.10
  stage: generate
  script:
    - apt update && apt install -y git
    - git clone --depth 1 -b dev https://github.com/QEDjl-project/QuantumElectrodynamics.jl.git /generator
    - julia --project=/generator/.ci/CI -e 'import Pkg; Pkg.instantiate()'
    - julia --project=/generator/.ci/CI bootloader.jl > $CI_PROJECT_DIR/pipeline.yaml
  artifacts:
    paths:
      - pipeline.yaml
    expire_in: 1 week
  interruptible: true
  tags:
    - cpuonly

run_pipeline:
  stage: run
  trigger:
    include:
      - artifact: pipeline.yaml
        job: generate_pipeline
    strategy: depend

@SimeonEhrig SimeonEhrig force-pushed the ciBootloader branch 3 times, most recently from daf0f99 to 7153e37 Compare November 5, 2024 12:14
Copy link
Member

@AntonReinhard AntonReinhard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely a good idea in my opinion and should make maintenance a lot more straightforward and adding new packages simpler.

.ci/CI/src/Bootloader.jl Outdated Show resolved Hide resolved
.ci/CI/src/Bootloader.jl Outdated Show resolved Hide resolved
.ci/CI/src/Bootloader.jl Show resolved Hide resolved
.ci/CI/src/Bootloader.jl Outdated Show resolved Hide resolved
.ci/CI/src/Bootloader.jl Outdated Show resolved Hide resolved
.ci/CI/src/Bootloader.jl Outdated Show resolved Hide resolved
.ci/CI/src/Bootloader.jl Outdated Show resolved Hide resolved
.ci/CI/src/integTestGen.jl Outdated Show resolved Hide resolved
.ci/CI/src/integTestGen.jl Outdated Show resolved Hide resolved
- The already share functions. which are copied between both packages.
- Preparation for the CI bootloader script.
- set CI_DEV_PKG_ environment in unit tests
@SimeonEhrig SimeonEhrig force-pushed the ciBootloader branch 4 times, most recently from 6423e6b to 5b2551b Compare November 6, 2024 09:23
- remove a lot of submodule definitions to make code reusable
- remove redundant code from SetupDevEnv.jl
- fix bug in `add_integration_test_job_yaml!` if the target branch is `main`
@SimeonEhrig SimeonEhrig marked this pull request as ready for review November 11, 2024 14:16
@SimeonEhrig
Copy link
Member Author

The current implementation contains a lot of code, which needs to be refactored or unified. Nevertheless I want to merge this unclean version to enable easier bug fixes and extensions of the CI.
Refactoring and tests will be implemented in following up PR's.

@AntonReinhard
Copy link
Member

Looks good to me.

Test PRs are:

* [Use CI bootloader QEDbase.jl#135](https://github.com/QEDjl-project/QEDbase.jl/pull/135)

* [Use ci bootloader QEDcore.jl#87](https://github.com/QEDjl-project/QEDcore.jl/pull/87)

* [Use CI bootloader QEDprocesses.jl#112](https://github.com/QEDjl-project/QEDprocesses.jl/pull/112)

* [Use ci bootloader QEDevents.jl#54](https://github.com/QEDjl-project/QEDevents.jl/pull/54)

* [Use CI bootloader QEDfields.jl#56](https://github.com/QEDjl-project/QEDfields.jl/pull/56)

I take it that we can update these to the new dev branch and merge them after this is through?

@SimeonEhrig
Copy link
Member Author

Looks good to me.

Test PRs are:

* [Use CI bootloader QEDbase.jl#135](https://github.com/QEDjl-project/QEDbase.jl/pull/135)

* [Use ci bootloader QEDcore.jl#87](https://github.com/QEDjl-project/QEDcore.jl/pull/87)

* [Use CI bootloader QEDprocesses.jl#112](https://github.com/QEDjl-project/QEDprocesses.jl/pull/112)

* [Use ci bootloader QEDevents.jl#54](https://github.com/QEDjl-project/QEDevents.jl/pull/54)

* [Use CI bootloader QEDfields.jl#56](https://github.com/QEDjl-project/QEDfields.jl/pull/56)

I take it that we can update these to the new dev branch and merge them after this is through?

That's correct. I need only remove the last commit. Than they are ready for merge.

@SimeonEhrig SimeonEhrig merged commit e4f53b3 into QEDjl-project:dev Nov 11, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants