Kubernetes on host #8265

potiuk · 2020-04-12T18:05:50Z

Make sure to mark the boxes below before creating PR: [x]

Description above provides context of the change
Unit tests coverage for changes (not needed for documentation changes)
Commits follow "How to write a good git commit message"
Relevant documentation is updated including usage instructions.
I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

potiuk · 2020-04-12T18:13:25Z

Hey @dimberman @kaxil @turbaszek -> I have almost working change where Kind Kubernetes cluster is created on the host side and it can be used to run the tests with Kubernetes. I nicely integrated it with breeze so that you can run:

breeze kind-cluster start -> will start the kind cluster

breeze kind-cluster deploy --force-build -> will deploy airflow to the cluster

and then you should be able to run tests. I just wanted to share it with you so that you can take a look. This can be very easily reused for the helm chart deployments (@dimbermann) and I think it's a better way of handling Kind Setup than relying in Github Action specific configuration from #8222 - because it can be run for local testing as well and it can be run on any CI (Travis/GA/Cloud Build. ....) . I updated the documentation including explanation how I imagine running the tests locally with breeze.

I am planning to complete it tomorrow - but please take a look in the meantime :).

It is based on the #8261 so only look at the latest commit.

dimberman · 2020-04-13T16:36:21Z

@potiuk is there a reason we're still defaulting to such an old k8s? We should set it to at least 1.17 (unless that should be a separate PR).

mik-laj · 2020-04-13T17:02:39Z

@dimberman Old release? Version 1.14 is the latest version of the stable release channel on GKE. Version 1.15 is the latest version of the regular release channel. I don't think we need to use the latest possible version, but the most popular version.
https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels#finding_out_whats_new

mik-laj · 2020-04-13T17:09:31Z

The following Kubernetes versions are currently available for new clusters in Amazon EKS:

1.15.11
1.14.9
1.13.12
1.12.10

https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

You can use the newer version if you use GKE on AWS. However, this is a new service that was released yesterday.
https://cloud.google.com/anthos/gke/docs/aws/concepts/architecture

potiuk · 2020-04-13T19:31:31Z

No reason whatsoever. Just carried from the past. Right now it's as easy as modifying a few constants :)

scripts/ci/kubernetes/app/templates/airflow.template.yaml

kubernetes_tests/test_kubernetes_pod_operator.py

scripts/ci/_utils.sh

scripts/ci/in_container/entrypoint_ci.sh

scripts/ci/kubernetes/app/templates/airflow.template.yaml

.github/workflows/ci.yml

kubernetes_tests/test_kubernetes_executor.py

potiuk · 2020-06-03T11:14:48Z

The following Kubernetes versions are currently available for new clusters in Amazon EKS:

@mik-laj - for now the deployment we have is kinda "abomination" - we will replace it with helm charts shortly - for now I know the Deployments we have won't work for higher versions of Kubernetes but I do not want to lose time trying to fix them since we are going to swap them with the helm charts.

kaxil · 2020-06-03T22:40:45Z

Nice work @potiuk

jhtimmins · 2020-06-04T00:46:41Z

@potiuk I think this may be causing the generate-requirements command to fail. When i run ./breeze generate-requirements --python 3.7, I get the following error:

./breeze: line 426: INTEGRATIONS[*]: unbound variable

I've also experienced it causing the command to hang indefinitely.

feluelle · 2020-06-04T08:07:18Z

scripts/ci/ci_run_airflow_testing.sh

+#TODO FIXME
+if [[ "${INTEGRATIONS[*]}" == *"kubernetes"* ]]; then


hmm @potiuk ? I get ./breeze: line 382: INTEGRATIONS[*]: unbound variable when starting ./breeze

Yep, Mac OS thingie. I used to catch it early before I switched to Linux. Fixed in #9135

Thanks @potiuk !

feluelle · 2020-06-04T08:08:03Z

Oh did not read that @jhtimmins - same for me.

Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782 (cherry picked from commit ff5dccc)

)

We upgraded to v2 in apache#8265 for other places in `.github/workflows/ci.yml`. This was left, so fixing it.

We upgraded to v2 in #8265 for other places in `.github/workflows/ci.yml`. This was left, so fixing it.

We upgraded to v2 in apache#8265 for other places in `.github/workflows/ci.yml`. This was left, so fixing it.

Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782 (cherry picked from commit ff5dccc)

Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes apache#8782 (cherry picked from commit ff5dccc)

We upgraded to v2 in apache/airflow#8265 for other places in `.github/workflows/ci.yml`. This was left, so fixing it. GitOrigin-RevId: a38c35a47f542bb480e331bf78d19ee612aa5041

boring-cyborg bot added the area:dev-tools label Apr 12, 2020

potiuk requested review from dimberman, turbaszek and kaxil April 12, 2020 18:05

potiuk force-pushed the kubernetes-on-host branch 2 times, most recently from cf2f550 to c18a829 Compare April 13, 2020 07:44

potiuk force-pushed the kubernetes-on-host branch from c18a829 to 65d597c Compare April 15, 2020 11:13

potiuk added this to the Airflow 1.10.11 milestone Apr 26, 2020

potiuk force-pushed the kubernetes-on-host branch from 65d597c to da8a050 Compare April 26, 2020 10:00

potiuk force-pushed the kubernetes-on-host branch 2 times, most recently from 7d30b1b to 9ca8192 Compare May 27, 2020 18:33

potiuk mentioned this pull request May 27, 2020

Use production image for k8s tests #9038

Merged

6 tasks

msumit reviewed May 28, 2020

View reviewed changes

scripts/ci/kubernetes/app/templates/airflow.template.yaml Show resolved Hide resolved

potiuk force-pushed the kubernetes-on-host branch 7 times, most recently from e9f88b4 to 2088d81 Compare June 3, 2020 00:28

dimberman requested changes Jun 3, 2020

View reviewed changes

potiuk force-pushed the kubernetes-on-host branch from 2088d81 to f960abb Compare June 3, 2020 06:59

turbaszek reviewed Jun 3, 2020

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

kubernetes_tests/test_kubernetes_executor.py Show resolved Hide resolved

potiuk mentioned this pull request Jun 3, 2020

Add Production Helm chart support #8777

Merged

6 tasks

feluelle reviewed Jun 4, 2020

View reviewed changes

potiuk added a commit that referenced this pull request Jun 9, 2020

fixup! Kubernetes Cluster is started on host not in the container (#8265

782d916

)

kaxil added a commit to astronomer/airflow that referenced this pull request Jun 14, 2020

Use actions/cache@v2

1bd8574

We upgraded to v2 in apache#8265 for other places in `.github/workflows/ci.yml`. This was left, so fixing it.

kaxil mentioned this pull request Jun 14, 2020

Use actions/cache@v2 for all static checks #9293

Merged

6 tasks

potiuk pushed a commit that referenced this pull request Jun 14, 2020

Use actions/cache@v2 (#9293)

a38c35a

We upgraded to v2 in #8265 for other places in `.github/workflows/ci.yml`. This was left, so fixing it.

kaxil added a commit to kaxil/airflow that referenced this pull request Jun 27, 2020

Use actions/cache@v2 (apache#9293)

d935d75

We upgraded to v2 in apache#8265 for other places in `.github/workflows/ci.yml`. This was left, so fixing it.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubernetes on host #8265

Kubernetes on host #8265

potiuk commented Apr 12, 2020

potiuk commented Apr 12, 2020

dimberman commented Apr 13, 2020

mik-laj commented Apr 13, 2020

mik-laj commented Apr 13, 2020 •

edited

Loading

potiuk commented Apr 13, 2020

potiuk commented Jun 3, 2020 •

edited

Loading

kaxil commented Jun 3, 2020

jhtimmins commented Jun 4, 2020

feluelle Jun 4, 2020

potiuk Jun 4, 2020

jhtimmins Jun 4, 2020

feluelle commented Jun 4, 2020

		#TODO FIXME
		if [[ "${INTEGRATIONS[]}" == "kubernetes"* ]]; then

Kubernetes on host #8265

Kubernetes on host #8265

Conversation

potiuk commented Apr 12, 2020

potiuk commented Apr 12, 2020

dimberman commented Apr 13, 2020

mik-laj commented Apr 13, 2020

mik-laj commented Apr 13, 2020 • edited Loading

potiuk commented Apr 13, 2020

potiuk commented Jun 3, 2020 • edited Loading

kaxil commented Jun 3, 2020

jhtimmins commented Jun 4, 2020

feluelle Jun 4, 2020

Choose a reason for hiding this comment

potiuk Jun 4, 2020

Choose a reason for hiding this comment

jhtimmins Jun 4, 2020

Choose a reason for hiding this comment

feluelle commented Jun 4, 2020

mik-laj commented Apr 13, 2020 •

edited

Loading

potiuk commented Jun 3, 2020 •

edited

Loading