chore: Make e2e tests runnable against remote cluster

[jannfis]

This allows for running the end-to-end test suite against a real workload, deployed to a Kubernetes cluster.

Refer to the README.md for a more detailed explanation.

Signed-off-by: jannfis [email protected]

Note on DCO:

If the DCO action in the integration test fails, one or more of your commits are not signed off. Please click on the Details link next to the DCO action for instructions on how to resolve this.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• Optional. My organization is added to USERS.md.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).

[codecov]

Codecov Report

Merging #5895 (48da4ec) into master (1bc3348) will increase coverage by 0.04%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5895      +/-   ##
==========================================
+ Coverage   40.91%   40.95%   +0.04%     
==========================================
  Files         147      147              
  Lines       19635    19656      +21     
==========================================
+ Hits         8033     8051      +18     
- Misses      10497    10499       +2     
- Partials     1105     1106       +1     

Impacted Files	Coverage Δ
cmd/argocd-util/commands/app.go	31.57% <0.00%> (-0.14%)	⬇️
util/kube/portforwarder.go	0.00% <0.00%> (ø)
cmd/argocd-util/commands/cluster.go	0.00% <0.00%> (ø)
util/settings/settings.go	41.46% <0.00%> (+1.52%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1bc3348...48da4ec. Read the comment docs.

[sbose78]

Thank you!

[sbose78]

Will review further, but this work looks very sophisticated :)

[alexmt]

Awesome work @jannfis . Added couple comments about using personal docker repo name.

[alexmt]

I'm sure this PR required a lot of effort. I think we should try to use it a much as we can to take advantage of it.

May be run every night agains https://cd.apps.argoproj.io/

@sbose78 , @jannfis do you plan to run it internally as well?

[sbose78]

May be run every night agains https://cd.apps.argoproj.io/

+1 we should do this. Does this get deployed nightly ?

do you plan to run it internally as well?

@alexmt Not internally per se, rather use existing automation ( OpenShift CI ) for spinning up multi-node clusters on AWS with ease to test https://github.com/argoproj/argo-cd with public test logs ( example ).

Generally speaking, the goal of running e2e tests against a deployed instance of Argo CD is :

• Remove the need for manual testing prior to an Argo CD release
• Run this on merge-to-master ( and eventually make it a PR ) in an openshift-agnostic way.
• In the mid-term, the aim is to organize tests in such a way that their successful execution gives us a high-level of confidence 'as a user'.

Initially, this is what we plan on doing:

1. Build https://github.com/argoproj/argo-cd/ master ( or use an existing built image )
2. Deploy a multi-node OpenShift on AWS
3. Deploy the freshly baked ArgoCD on OpenShift
4. Run e2e tests against the live instance.

All logs would be public therefore we should be able to catch failures and log bugs against them on this repository early on and potentially fix them.

Note,

I think we should try to use it a much as we can to take advantage of it.

+1

• Once this process is stable enough, I would propose (1) to (4) to be done on every merge-to-master on https://github.com/argoproj/argo-cd/. Given that this would be an additional webhook to Argo CD repository, this would have to go through the enhancement proposal process.
• No downstream-specific concepts would leak into the tests.

[jannfis]

Thanks for review @sbose78 and @alexmt

I have addressed all your comments, PTAL.

[alexmt]

LGTM.

Just one more comment #5895 (comment) Please feel free to merge after resolving it.