Implement a realistic BlockchainTests test run strategy

[holgerd77]

The blockchain tests have been somewhat forgotten, nevertheless it's important that these tests run on a regular basis (problem is that tests take a long time).

I just ran them over-night with npm run testBlockchain, a lot of tests (66!) are failing atm:

1..956
# tests 956
# pass  890
# fail  66

We urgently need at least some strategy on how we can realistically come back to a situation where we run these tests on a regular basis. It's not realistic though probably to have these tests run on every PR, even if the switch to CircleCI now comes with shorter build times.

A start would be to create a setup which at least eases triggering a test run. One idea would be to have a separate circle config config-blockchain-tests.yml or similar and create a non-to-be-merged run-blockchain-tests PR with a renamed/replaced config file to have these tests run on CircleCI every 2-4 weeks.

I will subsequently add a test PR with this to get a first impression how long tests on CircleCI will run.

[jwasinger]

I think for something like this, it would be a good idea to look at creating a solution for running these tests in parallel on cloud infrastructure. We could look at using a service like ECS (Docker container orchestration PaaS provided by AWS).

An alternative is to look into doing this via CircleCI and pay for more container instances to increase test throughput. This route could be a pain if CircleCI isn't designed to easily handle this use case.

@vpulim do you have any input/suggestions regarding this issue?

[jwasinger]

@holgerd77 @vpulim if my suggestion to use ECS (or some equivalent service) is something that you are interested in exploring, I'd be happy to start looking into it. I'm already involved with devops over on ewasm.

[vpulim]

@jwasinger unfortunately, my experience with container orchestration is fairly limited but ECS seems like a perfectly fine solution if you already have experience with it, and I would love to be able to run the full test suite in under an hour ;) @holgerd77 should make the call though...

[jwasinger]

@holgerd77 should make the call though...

Read that as, "we should come to consensus on a solution" ;) . Anyways, I'm fine with whatever you guys propose (you guys deal with this codebase/CI more than I do atm). Would like to add that I think the integration for this ECS solution would be as simple as adding a single step to CircleCI.

[vpulim]

@jwasinger yes :) 👍

[holgerd77]

Actually the situation on CircleCI is much better then I expected, a complete blockchain test run here takes around 30 minutes. And we DO have unlimited build times here since we apply as an OSS project (see this comment).

So not sure if there is the need for an extra setup here. We SHOULD fix those tests though. 😄

[holgerd77]

Will close this for now since there is a low-level solution in place and we are at the same time pretty low on resources.