Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: custom runtimes, optional VPC, python 3.11 #74

Merged
merged 12 commits into from
Oct 31, 2023
Merged

Conversation

emileten
Copy link
Contributor

@emileten emileten commented Oct 5, 2023

A large PR with several breaking changes.

  • ⚠️ breaking change. bootstrapper runtime made configurable. This necessitated to 'merge' the database and bootstrapper into a single construct to avoid JSII issues. With this, all apps have their runtime configurable.

  • ⚠️ breaking change. switched to Function and fromDockerBuild for lambda functions, because the state of PythonFunction makes it very hard to add flexible configurability. Now, users can override any of the lambda Function parameters they want ( code, but also memory, time out...).

  • made VPC optional for lambdas

  • harmonized pgstac to 0.7.10 in default runtimes.

  • harmonized python to 3.11 in default runtimes.

@emileten emileten requested a review from vincentsarago October 5, 2023 09:54
…rfile for all apps

BREAKING CHANGE: clients need to provide aws_lambda.AssetCode to configure their apps. Solely the python application and the requirements.txt file is not supported anymore.
@emileten emileten force-pushed the feat/custom-runtimes branch from 8950777 to eb20644 Compare October 5, 2023 09:58
@emileten emileten changed the title custom runtimes from Dockerfile for all apps feat!: custom runtime for bootstrapper and custom runtimes from Dockerfile for all apps Oct 5, 2023
@emileten emileten self-assigned this Oct 5, 2023
@vincentsarago vincentsarago self-requested a review October 5, 2023 10:48
Copy link
Member

@vincentsarago vincentsarago left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me.

We should make sure to document this part to explain how basic dockerfile should look

# IMPORTANT: Use the same version of python for the asset building and lambda deployment
FROM public.ecr.aws/lambda/python:3.10

WORKDIR /tmp
RUN python -m pip install pip -U

# Install application dependency 
COPY runtime/requirements.txt requirements.txt
RUN python -m pip install -r requirements.txt "mangum>=0.14,<0.15" -t /asset

# Copy handler and custom code
COPY runtime/src/*.py /asset/

@sharkinsspatial
Copy link
Member

@emileten We should also consider adding dockerignore file with cdk.out included at a minimum to reduce overall context size now that you are using Docker builds for all Lambdas.

lib/ingestor-api/index.ts Outdated Show resolved Hide resolved
lib/bootstrapper/index.ts Outdated Show resolved Hide resolved
@emileten emileten linked an issue Oct 10, 2023 that may be closed by this pull request
@emileten emileten force-pushed the feat/custom-runtimes branch from a79fd3c to a53adef Compare October 11, 2023 08:24
@emileten emileten force-pushed the feat/custom-runtimes branch from a53adef to 385ab1b Compare October 11, 2023 09:01
…s. Allow the user to provide anything and let the CDK method raise error and overwrite values defined within our construct. Make this clear in the documentation
@emileten
Copy link
Contributor Author

emileten commented Oct 12, 2023

I addressed comments ; I'll see if I can fit that into eoAPI and after confirming it works I'll fix conflicts and merge.

@emileten emileten changed the title feat!: custom runtime for bootstrapper and custom runtimes from Dockerfile for all apps feat!: custom bootstrapper runtime, switch to fromDockerBuild for all lambda functions, add STAC browser and STAC ingestor, allow for low-security deployment, pgstac 0.7.10, python 3.11 Oct 31, 2023
@emileten emileten changed the title feat!: custom bootstrapper runtime, switch to fromDockerBuild for all lambda functions, add STAC browser and STAC ingestor, allow for low-security deployment, pgstac 0.7.10, python 3.11 feat!: custom bootstrapper runtime, switch to fromDockerBuild for all lambda functions, allow for low-security deployment, pgstac 0.7.10, python 3.11 Oct 31, 2023
@emileten emileten changed the title feat!: custom bootstrapper runtime, switch to fromDockerBuild for all lambda functions, allow for low-security deployment, pgstac 0.7.10, python 3.11 feat!: custom bootstrapper runtime, switch to fromDockerBuild for all lambda functions, optional lambda VPC, pgstac 0.7.10, python 3.11 Oct 31, 2023
@emileten emileten changed the title feat!: custom bootstrapper runtime, switch to fromDockerBuild for all lambda functions, optional lambda VPC, pgstac 0.7.10, python 3.11 feat!: custom runtimes, optional VPC, python 3.11 Oct 31, 2023
@emileten
Copy link
Contributor Author

emileten commented Oct 31, 2023

@vincentsarago FYI in the meanwhile (this took me forever 🙄) I did these changes (I updated the description):

  • -> python 3.11 (so I'll close your other PR)
  • made VPC optional for lambdas (this allows for a cheap, low-security deployment without a nat gateway)
  • to make the bootstrapper runtime customizable I had to merge the database and the boostrapper construct together. This is a breaking change but unfortunately necessary: the way things were designed, with the two constructs being interdependent, but separated, was preventing me from making the bootstrapper options configurable (was encountering some JSII issues).

This is tested in deployment and good to go, about to merge!

@emileten emileten changed the title feat!: custom runtimes, optional VPC, python 3.11 feat: custom runtimes, optional VPC, python 3.11 Oct 31, 2023
@emileten emileten changed the title feat: custom runtimes, optional VPC, python 3.11 feat: custom runtimes, optional VPC, python 3.11 Oct 31, 2023
@emileten emileten force-pushed the feat/custom-runtimes branch from 95676ef to d59c936 Compare October 31, 2023 07:24
BREAKING CHANGE:

* database and bootstrapper constructs merged to make the bootstrapper runtime configurable

* switched to `Function` for all Lambda deployments to allow for flexibility
@emileten emileten force-pushed the feat/custom-runtimes branch from d59c936 to d4859f6 Compare October 31, 2023 07:25
@emileten emileten merged commit ba6bf09 into main Oct 31, 2023
3 checks passed
@emileten emileten deleted the feat/custom-runtimes branch October 31, 2023 07:33
github-actions bot pushed a commit that referenced this pull request Oct 31, 2023
# [6.0.0](v5.4.1...v6.0.0) (2023-10-31)

### Features

* custom runtimes, optional VPC, python 3.11 ([#74](#74)) ([ba6bf09](ba6bf09))

### BREAKING CHANGES

* the `bootstrapper` construct was deleted and is no longer available. In addition, we switched from `PythonFunction` to `Function` for all lambdas.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make runtimes for database bootstrapper configurable
4 participants