Add integration-test phase

[daddz]

I am trying to make use of skaffold for the whole CI/CD pipeline and a major thing that is still missing for me are integration tests.

Since I was already digging into the skaffold source due to another issue I thought why not try to get something running on this front.

The projects I mainly work on are python/django focused so my approach is coming from that perspective. I don't know how much sense it makes for other languages/frameworks so it would be good to get some feedback.

A pipeline could consist of the following steps:

1. Build image(s) (with test dependencies)
   • skaffold build --file-output build_test.json
2. Deploy to new integration test namespace and run integration tests
   • skaffold integrationtest -n my-project-$(pipeline_id) -a build_test.json
3. Build image(s) (wihtout test dependencies)
   • skaffold build -p production --file-output build_prod.json
4. Deploy to stage/production
   • skaffold deploy -p production -a build_prod.json

For the configuration I thought of a new integrationtest top-level key:

# skaffold.yaml
integrationtest:
    podSelector: 'app=myapp'
    testScript: 'pytest tests/'

The pod that is used to execute the tests is found via the selector and the test script is executed via kubectl exec in that pod.
I'm not sure if it makes sense to use an array here so different tests can be run in different pods.

I implemented my ideas in my fork/branch over at: https://github.com/daddz/skaffold/commit/c0c095b65a05ec4f09a8552ce392821d2ae1524d

I haven't had any experience with Go before this so the code might be pretty lackluster but I just wanted to get some kind of PoC running and collect some feedback for the general idea before I spend more time on it.

Let me know your thoughts!

Related issue: #992

[nkubala]

hey @daddz thanks for the idea! integration testing in k8s is one of the biggest unsolved problem in cloud native CI/CD right now, and something that our team has thought quite a bit about in the past, but in practice it seems to always be harder than in principle. I think something like this would make sense to further the CI/CD story with skaffold, but I'm worried that it's difficult to extend the design to be something generic for k8s.

for example, testing pods with your design makes sense, but should we try and allow for testing entire services (or any other k8s resource) as well? how do we parse test results and incorporate them in a meaningful way in the skaffold output? what if your app is state dependent - how do we handle setup, teardown, etc.?

these are all questions that I would love to see solved, but I don't think skaffold is the right place to solve them. if a kubernetes-native integration testing framework existed I think we'd love to support calling out to that in skaffold, but it's a little out of scope of our own codebase.

that said, if you decide to build this out somewhere else as a standalone tool, we'd be willing to consider it as a first implementation of a new verification phase of the Skaffold pipeline :)

[daddz]

hey @nkubala thanks for the feedback! If you don't mind I'd like to go into a little more detail to get a better understanding! :)

(...) and something that our team has thought quite a bit about in the past, but in practice it seems to always be harder than in principle (...)

Do you have any notes/discussions somewhere I could read up on the issues that come up in practice? My use case is pretty small in scope and basic so it would be cool to see the bigger picture.

but I'm worried that it's difficult to extend the design to be something generic for k8s.

That's a good point, so far my approach is not extensible and not generic. How about building it with different strategies in mind - like for example the builders.
So for the start there's only the "simple/basic" strategy that uses the approach I suggested and maybe a "custom" strategy where user's can define it themselves (like custom builders).
That way it could be easily extended if someone comes around with "the" integration test framework for k8s.

how do we parse test results and incorporate them in a meaningful way in the skaffold output?

I'd argue that this is not something skaffold should be concerned about. It's up to the user to process this further. Maybe adding a file output for the logs makes sense so users can process it further in their pipeline.

what if your app is state dependent - how do we handle setup, teardown, etc.?

That should be handled by the deployment scripts the user provides via kubectl/helm/kustomize, right? Or do you think there should be support for some explicit pre/post handling?

that said, if you decide to build this out somewhere else as a standalone tool, we'd be willing to consider it as a first implementation of a new verification phase of the Skaffold pipeline :)

I think my approach is way too simple to warrant a standalone tool for now :D

But that's also why I'd like to get more insights and better understand other use cases.

Thanks a lot!

[daddz]

I've put my ideas into some examples:

Design variant 1 (old proposal)

See: Design proposal for integrationtest command #2671

Configuration:

integrationtest:
    podSelector: 'app=skaffold-integration'
    testCommand: 'pytest tests/pass.py

Design variant 2

Configuration:

integrationtest:
    kubectl:
        before: 'setup.sh' # optional
        after: 'teardown.sh' # optional
        podSelector: 'app=skaffold-integration'
        testCommand: 'pytest tests/pass.py'

Comments

Unlike variant 1 this offers the possibility to solve the problem with different approaches like many other commands (e.g. build, deploy).

In case a new tool emerges that solves this problem in a better way it could be integrated into skaffold without breaking existing implementations based on this 'simple kubectl' integration test execution.

Optional scripts in before and after can take care if the tests need some specific measures before/after the tests. These are executed in the pod that will also run the tests.

Design variant 3

Configuration:

integrationtest:
    kubectl:
        - podSelector: 'app=skaffold-integration-svc1'
          testCommand: 'pytest tests/svc1.py'
          before: 'svc1/setup.sh'
        - podSelector: 'app=skaffold-integration-svc2'
          testCommand: 'pytest tests/svc2.py'
          before: 'svc2/setup.sh'
          after: 'svc2/teardown.sh'

Comments

Like variant 2 but supports multiple tests for different pods. Not sure if this makes sense/is a real scenario.

Common usage within CI/CD pipeline

• build (with test dependencies): skaffold build -p test --file-output build_test.json
• deploy test env: skaffold deploy -n $(pipeline_id) -a build_test.json
• test: skaffold integrationtest
• cleanup test env: skaffold delete -n $(pipeline_id)
• build (without test dependencies): skaffold build -p prod --file-output build_prod.json
• deploy stage/prod env: skaffold deploy -a build_prod.json

[briandealwis]

So the approach I've been using in the Container Debugging Support images is to push the integration's tests into separate testing containers that are deployed alongside against the cluster. This approach has a few advantages:

• the real containers are built once (no separate production vs test builds): we're testing what will be deployed, and avoids possible interference from test frameworks
• these tests are run in-cluster and reference other containers through k8s facilities instead of accessing services via backdoors

The disadvantage:

• it's a one-shot deal — I think you'd need to do multiple skaffold run with different profiles to test with different DBs or different mocked services

This specific example isn't perfect as I'm actually using busybox to run the tests, and so Skaffold doesn't show the test logs as there is no built image. I plan to turn this into an built image and enabled via a profile so that they aren't deployed by my prod.

[balopat]

Thanks for writing these ideas up @daddz - as we discussed on the Skaffold Office Hours - I think we should find/tease out interesting workflows where we can leverage skaffold's knowledge about the project. The current shape of the proposals here doesn't add much value yet as it is only a "verb" for scripts.

Maybe we could also document possible integration testing patterns:

• use skaffold run -p <profile> as a deployer within your tests
• use skaffold run -p <profile> as a separate CI step and run integration tests as another...
• etc..

[daddz]

the real containers are built once (no separate production vs test builds): we're testing what will be deployed, and avoids possible interference from test frameworks

I really like that approach. I might dabble with this for my approach.

I think we should find/tease out interesting workflows where we can leverage skaffold's knowledge about the project. The current shape of the proposals here doesn't add much value yet as it is only a "verb" for scripts.

I agree, my idea so far doesn't leverage skaffolds knowledge. As I get more familiar with skaffolds codebase I'm sure I'll find better possibilities.

Maybe we could also document possible integration testing patterns:
* use skaffold run -p as a deployer within your tests
* use skaffold run -p as a separate CI step and run integration tests as another...

You mean making use of skaffold run to do what I had in mind?
I didn't think about it yet but makes sense. If it's possible to reuse it that would be great of course.
Maybe that way it would just need some more flags to provide some basic support instead of a whole new command. I'll play around with this idea!

[tstromberg]

Labelling for future discussion.

[balopat]

Thinking about this more here is a distilled recommendation that will justify closing this issue:

Principle: for quick dev loop use cases Skaffold is the orchestrator locally, for CI/CD pipelines Skaffold provides building blocks, not an orchestration.

In this particular case we can see that integration tests typically ran in CI/CD pipelines. Hence we would recommend using skaffold build / run / delete to manage the application. And in between we recommend the use of whatever flexible scripts are going to execute integration tests.