Skip to content

Commit

Permalink
[sdlf-cicd] 2.8.0 doc update
Browse files Browse the repository at this point in the history
  • Loading branch information
cnfait committed Jan 14, 2025
1 parent 6b56294 commit 35eacec
Showing 1 changed file with 41 additions and 0 deletions.
41 changes: 41 additions & 0 deletions docs/constructs/cicd.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,49 @@
!!! note
`sdlf-cicd` is defined in the [sdlf-cicd](https://github.com/awslabs/aws-serverless-data-lake-framework/tree/main/sdlf-cicd) folder of the [SDLF repository](https://github.com/awslabs/aws-serverless-data-lake-framework).

## Simplified CICD with SDLF > 2.8.0

**SDLF > 2.8.0** has a simplified CICD setup that can be used with any [source provider](https://docs.aws.amazon.com/codebuild/latest/userguide/access-tokens.html) supported by AWS CodeBuild. SDLF creates two CodeBuild projects able to deploy SDLF constructs (a third is created during the deployment process, but the user is never expected to interact directly with it), leaving the configuration of the source step to the user:

* `sdlf-cicd-bootstrap`, publishing SDLF constructs in the private CloudFormation registry,
* `sdlf-cicd-{name provided by the user}`, deploying the actual infrastructure,
* and the third appearing during deployment, that should not be modified: `codeseeder-sdlf`.

These CodeBuild projects are required in every account SDLF is used, and can be deployed with the `./deploy-generic.sh` script in the `sdlf-cicd` module:

```
./deploy-generic.sh --help # provides details on the available options
./deploy-generic.sh -p aws_profile sdlf-main
```

Replace `aws_profile` with the name of the AWS profile giving access to the account SDLF will be used in. `sdlf-main` is a user-provided name - feel free to use a different one, within constraints.

Once the CodeBuild projects are created, the user can edit them manually to configure the source provider. Please refer to the [AWS CodeBuild documentation](https://docs.aws.amazon.com/codebuild/latest/userguide/access-tokens.html) for details on how to do this. `sdlf-cicd-bootstrap` should have a git repository containing a copy of the entire SDLF repository as source. `sdlf-cicd-{name provided by the user}` should use an empty git repository as source - this is the repository end users are going to work in, pushing CloudFormation templates referencing SDLF constructs, as seen in the [public workshop](https://catalog.us-east-1.prod.workshops.aws/workshops/501cb14c-91b3-455c-a2a9-d0a21ce68114/en-US/10-demo/200-foundations).

There is no SDLF-specific DevOps account in this setup.

`./deploy-generic.sh` can be run multiple times in the same account if required, with different names. This can be useful if teams with different scopes are responsible for the data architectures. For example:

```
./deploy-generic.sh -p aws_profile sdlf-main
./deploy-generic.sh -p aws_profile sdlf-engineering
```

This would create the following:
* `sdlf-cicd-bootstrap`
** this is created only once, no matter how many times `./deploy-generic.sh` is used in a given account.
* `sdlf-cicd-sdlf-main`
** this CodeBuild project could be linked to a git repository owned by a data platform team, creating centralized storage layers with `sdlf-foundations`.
* `sdlf-cicd-sdlf-engineering`
** this CodeBuild project could be linked to a git repository owned by a data engineers team, creating technical catalogs and data processing pipelines with `sdlf-dataset` and the various `sdlf-stage-*`.


## Infrastructure

!!! note
This section and the rest of the page does not apply to the setup described above.

![SDLF CICD](../_static/sdlf-cicd.png)

`sdlf-cicd` is a special construct that can be used to deploy other SDLF constructs. An end-to-end example is available in the [official SDLF workshop](https://sdlf.workshop.aws/).
Expand Down

0 comments on commit 35eacec

Please sign in to comment.