Refactor processes to make it easier to specify which tests to run

[pihme]

We want to refactor the processes. In particular, we thought about getting rid of Run All Tests in Camunda Cloud.

We want to replace it by a process that takes a list of testProcessId and then uses a multi instance to run all tests in that list.
Then the hierarchy of process would be:
Run List of Tests in Camunda Cloud > Run Test per Cluster Plan > Run Test in Camunda Cloud > Test Process

This would allow callers of the processes to specify which tests to run. We believe this will be a more flexible approach.

work items

• Remove process run-all-tests-in-camunda-cloud-per-region #612
• Reduce TestReport interface #614
• Add new processes #648
• Move region check for chaos tests into chaos tests zeebe-chaos#126
• Deprecate obsolete processes

[deepthidevaki]

@pihme I guess you are probably busy to work on this issue. Would it make sense if I take up this task? I either need this to integrate e2e-tests or should find a temporary way to run e2e-tests until this is done.

[pihme]

Sure, let's talk about it first, though.

[pihme]

Summary of discussions with @deepthidevaki today:

• The interface (input/output) of a test will be reduced
  • Input stays as it is today
  • Output will be reduced to (see below)
  • Additionally, there will be a new module api that has Java interfaces for this API
  • And we aim to publish a release of testbench to Maven, so that tests can import the API as dependency

public interface TestReport {

  TestResult getTestResult();

  List<String> getFailureMessages();

  public enum TestResult {
    PASSED,
    FAILED,
    SKIPPED;    
  }
}

• The test workflows will be simplified
  • Eliminate layers that are confusing
  • Make the test type to run (chaos test, sequential test, e2e) the outmost loop
  • This way, we have more flexibility to specify which tests to run, and with which configuration
  • Make the cluster plan the inner loop; which cluster plans to run in becomes part of the configuration for a test type
  • One side effect of this refactoring would be that the check whether we are in a chaos region would need to be moved and implemented in the chaos test
  • Visually this could look like this:

• We also thought about a nice to have feature: each test type could deploy a workflow to start this test manually and provide a human task form to configure the test run. We think this will make it easier to test the processes during development
• To be discussed later, how we configure tests to either be short for a nightly run or longer (e.g. the week of QA prior to a release)

In terms of implementation steps:

• Deepthi will use what is already there to integrate and test her e2e test in isolation
• in the meantime, I will proceed with the refactoring. Most likely I will introduce new BPMN models alongside the old one
• Deepthi will join when she is done with her e2e tests
• Later, we will switch the QA run to use the new processes
• After Q2, when we implemented deletion of workflows, then we can delete the old ones, yippie!

cc and request for comments @Zelldon

[ChrisKujawa]

Thanks for the update @pihme. I think it looks fine.

One side effect of this refactoring would be that the check whether we are in a chaos region would need to be moved and implemented in the chaos test

Not 100% sure what this means and how it would look like and why 😁 Maybe you can elaborate a bit on this.

After Q2, when we implemented deletion of workflows, then we can delete the old ones, yippie!

😆 lets see whether we really can do this in one quarter :D

[pihme]

Thanks for the update @pihme. I think it looks fine.

One side effect of this refactoring would be that the check whether we are in a chaos region would need to be moved and implemented in the chaos test

Not 100% sure what this means and how it would look like and why grin Maybe you can elaborate a bit on this.

I hope this helps: zeebe-io/zeebe-chaos#126