Match unit tests on start of test name, and split some test classes

[ximinez]

High Level Overview of Change

Changes the matching of the parameters to the --unittest command line option to match the patterns against the beginning of the test name.

• For example, without this change, to run the TxQ tests, one must specify --unittest=TxQ1,TxQ2 on the command line. With this change, one can use --unittest=TxQ, and both will be run.
• An exact match will prevent any further partial matching.
• This could have some side effects for different tests with a common start of name. For example, NFToken, NFTokenBurn, NFTokenDir. This might be useful. If not, the shorter-named test(s) can be renamed. For example, NFToken to NFTokenBase.

Speaking of NFToken, the second commit splits it, along with the NFTokenBurn, and Offer test classes into 4 or 5 sub-classes each, based on the different FeatureBitsets that are currently run sequentially. In Windows, this speeds up a Debug build unit test run by about 5x! (Details below.)

There are two commits. The don't necessarily need to be squashed when merging, though it would be fine if they are.

Context of Change

Long test run times have been a thorn in all of our sides for a while, but are even worse in Windows Debug builds, because there is a ton of overhead. At the recent team summit, someone threw out the idea of adding regex parsing to the matching. This doesn't do full regex matching, but that keeps it pretty simple.

Note that these changes only affect unit tests.

Type of Change

• [X ] New feature (non-breaking change which adds functionality)
• [X ] Tests (You added tests for code that already exists, or your new feature included in this PR)

Before / After

Before:

These examples are with the tip of develop, commit aded4a7 as of this writing

--unittest=TxQ (fails)

$ rippled --unittest=TxQ
Failed: No tests run
119ms, 0 suites, 0 cases, 0 tests total, 1 failure

--unittest=TxQ1, TxQ2 (works)

$ rippled --unittest=TxQ1,TxQ2
ripple.app.TxQ1 queue sequence
[...]
ripple.app.TxQ2 acct in queue but empty
[...]
Longest suite times:
   24.4s ripple.app.TxQ2
   21.4s ripple.app.TxQ1
45.8s, 2 suites, 37 cases, 4787 tests total, 0 failures

Full test run with multiple runners --unittest --unittest-jobs=16 (2228.9 seconds, bound by the longest running test)

$ rippled --unittest --unittest-jobs=16
[...]
Longest suite times:
 2227.4s ripple.tx.NFToken
 1192.5s ripple.tx.NFTokenBurn
  288.9s ripple.tx.Offer
  259.0s ripple.app.TheoreticalQuality
  224.8s ripple.app.ShardArchiveHandler
  224.1s ripple.tx.NFTokenDir
  199.7s ripple.tx.ReducedOffer
  196.5s ripple.app.AMMExtended
  166.9s ripple.app.AMM
  163.4s ripple.app.ReportingETL
2228.9s, 211 suites, 1792 cases, 633774 tests total, 0 failures

After:

--unittest=TxQ (succeeds!)

$ rippled --unittest=TxQ
ripple.app.TxQ1 queue sequence
[...]
ripple.app.TxQ2 acct in queue but empty
[...]
Longest suite times:
   25.2s ripple.app.TxQ2
   22.1s ripple.app.TxQ1
47.4s, 2 suites, 37 cases, 4787 tests total, 0 failures

--unittest=TxQ1, TxQ2 (results unchanged)

$  ./build/cmake/msvc19.ON//Debug/rippled --unittest=TxQ1,TxQ2
ripple.app.TxQ1 queue sequence
[...]
ripple.app.TxQ2 acct in queue but empty
[...]
Longest suite times:
   25.4s ripple.app.TxQ2
   22.1s ripple.app.TxQ1
47.6s, 2 suites, 37 cases, 4787 tests total, 0 failures

Full test run with multiple runners --unittest --unittest-jobs=16 (438.7 seconds, not bound by the longest running test)

$ rippled --unittest --unittest-jobs=16
[...]
Longest suite times:
  314.6s ripple.app.TheoreticalQuality
  293.4s ripple.tx.NFToken0
  281.7s ripple.tx.NFToken4
  279.3s ripple.tx.NFToken1
  273.5s ripple.tx.NFToken2
  271.7s ripple.tx.NFToken3
  261.2s ripple.app.ShardArchiveHandler
  220.9s ripple.tx.NFTokenDir
  216.4s ripple.tx.ReducedOffer
  215.8s ripple.tx.NFTokenBurn4
438.7s, 222 suites, 1806 cases, 633788 tests total, 0 failures

The fact that the overall test run takes longer than the longest suite (TheoreticalQuality in this case) means that that longest running test suite finished before all the other tests were finished, which allowed the child to be reused to run more tests. I haven't seen that happen in on a full run in any configuration in years.

Test Plan

Because only unit test code has changed, as long as exiting tests pass as they normally do, there's no additional functional testing needed.

Future Tasks

Full regex parsing?

[HowardHinnant]

Nice!

[intelliot]

I guess ximinez:unittest-prefix has a branch protection rule requiring linear history? Anyway, GitHub can't automatically update this branch.

@ximinez , at your convenience, feel free to bring this up-to-date with develop. Also, if you'd like the commit message to differ from what you've already used in the commits here, feel free to provide a suggested commit message as well.

[ximinez]

I guess ximinez:unittest-prefix has a branch protection rule requiring linear history? Anyway, GitHub can't automatically update this branch.

Turns out I did. Totally forgot about that. I turned it off.

at your convenience, feel free to bring this up-to-date with develop. Also, if you'd like the commit message to differ from what you've already used in the commits here, feel free to provide a suggested commit message as well.

Merge is done. Assuming this gets squashed down to one commit, I think the following would be a fine commit message:

Match unit tests on start of test name

* For example, without this change, to run the TxQ tests, must specify
  `--unittest=TxQ1,TxQ2` on the command line. With this change, can use
  `--unittest=TxQ`, and both will be run.
* An exact match will prevent any further partial matching.
* This could have some side effects for different tests with a common
  name beginning. For example, NFToken, NFTokenBurn, NFTokenDir. This
  might be useful. If not, the shorter-named test(s) can be renamed. For
  example, NFToken to NFTokens.
* Split the NFToken, NFTokenBurn, and Offer test classes. Potentially speeds
  up parallel tests by a factor of 5.

[intelliot]

@manojsdoshi - are we ok to merge this to develop now?