CI refactor

[abraverm]

Current CI issues:

• Spread across two Jenkins masters (internal\external) which makes it hard on community contributres
• Jenkins jobs are nested - for one to find the failing step or root cause, it requires digging through job consoles and log files.
• Hard to reproduce locally as CI tests against provider environments and setups which the developer doesn't have access
• Local testing is limited (mostly) to unit testing, and its hard to setup a local integration testing environment
• There is no staging process for CI workflows, for example chaning release process
• One has to be familiar with Jenkins pipelines and Openshift and Docker to have some level of understanding of CI workflow

Assumptions:

1. Developer (Red Hatter or community member) is the number one citizen
2. If it doesn't test locally, then it shouldn't be tested anyway
3. Testing shouldn't change system environment and requires little as possible

Changes:

1. OS testing is defined on the CI system - if one wants to test Linchpin on some OS then clone the repo with the relevant changeset and run it on that OS. For local testing that could mean run the tests inside a container or a VM.
2. Any open source providers (Openshift, Openstack, Beaker, Duffy, libvirt, Docker, etc) should be deployed with a Ansible playbook and for the developer to decide where to deploy them (VM, private cloud, public cloud, etc). The playbook should return credential and other connection settings for the test framework. CI system should deploy those providers too, but optimized (meeting some conditions).
3. Testing public providers is done by providing credentials, up to developer to come up with.
4. The testing framework input is connection settings and target workspace
5. Move to a public CI system so anyone cad develop new CI workflows and test them, such as Travis, Circle-CI, etc. (No Openshift, Jenkins and Dockerfiles)
6. Move to TOX as a common python test automation to trigger linting, unit tests and even integration tests.

[samvarankashyap]

@abraverm These are very informative.
I appreciate your effort in putting them together.

[14rcole]

Hard to reproduce locally as CI tests against provider environments and setups which the developer doesn't have access

@mansikulkarni96 How did you plan on managing credentials in restylinchpin? If there's a way to store credentials in restylinchpin instead of passing them up each time, that could resolve this issue

[samvarankashyap]

Related to #1229

[samvarankashyap]

Though I believe this is necessary With all the permissions involved It takes time to implement the ci
This issue needs to be broken down into multiple ci issues.
Removing milestone on this issue. Since it cannot happen on a deadline

[abraverm]

1. Why complicate running simple unit tests with an interoperability test framework? This doesn't help making tests simpler or easier to understand. There are much better and well known tools for that, such as Tox.
2. Github doesn't allow managing sub issues. This issue encapsulates the whole problem of CI. We can create more issues and link them back to this one. However I do believe this is a major change needed and milestone v2.0 is a perfect target for it. Release date can be pushed.

[14rcole]

From what I understand, Tox only solves the problem of unit tests, Carbon can streamline both our unit and integration tests. Also, I believe we would no longer be using Jenkins

[samvarankashyap]

I believe with everything in place with Github's actions.
Majority of the ci work has been refactored with few exceptions
(Release process, Libvirt testing, downstream testing)
Since with have tickets to resolve the above I believe it is safe to close this ticket.
If there are any concerns please feel free to re-open the ticket.