Adding mutation testing to build gradle with PiTest

[MitchellGale]

Description

Adds PiTest Gradle to build.gradle in core. This adds support for Mutation testng/Analysis to our testing ability. It should help us make more effective tests for past/future features.

Why does this help?

Mutation testing is used to find missing parts of your project not covered by existing tests. It modifies (mutates) one part of code and checks if a test is killed (fails) because of the mutation. Sometimes it will make return statements return null, other times it will switch == to !=, or < with >, or || with && and lots more mutations. More mutations can be found in the default mutations column here https://pitest.org/quickstart/mutators/. Often mutation testing provides a more clear picture of the quality of tests compared to line/branch coverage alone.

If you notice mutations that "survive" your test suite, consider adding additional tests to kill the mutations.
Short presentation PDF on mutation testing MutationTesting_Presentation.pdf.

Docs for Gradle plugin for PiTest.

Currently to be manually run, but could be added to GitHub actions later. It generates an html page indicating mutations that survived and were killed by our test suite.

Only adding support for running mutation testing in core module.

Steps to run

./gradlew pitest

Report will be placed in core/build/reports/pitest/index.html.

Sample Report

PiTest_Results.zip

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #170 (4a59426) into integ-AddPiTestMutationTesting (662a938) will decrease coverage by 35.50%.
The diff coverage is n/a.

@@                          Coverage Diff                          @@
##             integ-AddPiTestMutationTesting     #170       +/-   ##
=====================================================================
- Coverage                             98.27%   62.76%   -35.51%     
=====================================================================
  Files                                   327       10      -317     
  Lines                                  8457      658     -7799     
  Branches                                553      119      -434     
=====================================================================
- Hits                                   8311      413     -7898     
- Misses                                  142      192       +50     
- Partials                                  4       53       +49     

Flag	Coverage Δ
query-workbench	62.76% <ø> (?)
sql-engine	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...nsearch/sql/sql/parser/context/ParsingContext.java
...ava/org/opensearch/sql/sql/parser/ParserUtils.java
...org/opensearch/sql/data/model/ExprStringValue.java
...ch/planner/logical/rule/OptimizationRuleUtils.java
...protocol/response/format/CsvResponseFormatter.java
...nsearch/data/value/OpenSearchExprValueFactory.java
...rch/sql/planner/physical/catalog/CatalogTable.java
...ion/operator/predicate/UnaryPredicateOperator.java
...earch/response/agg/CompositeAggregationParser.java
...l/expression/window/ranking/DenseRankFunction.java
... and 327 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[Yury-Fridlyand]

Do you have a report example? How can I run the task?

[MitchellGale]

Do you have a report example? How can I run the task?

@Yury-Fridlyand I updated the description to include that info.

PiTest_Results.zip

[Yury-Fridlyand]

When you will publish to upstream, please consider to add

• pitest doc links
• A failure analysis, which describes what
  • is killed
  • is survived
  • is missing to make test pass
• You can also include your PiTest presentation slides if they don't include confidential information

[MaxKsyunz]

Looks really cool @MitchellGale-BitQuill!

Can I run pittest in other modules besides :core?

As a developer, why would I used it? After I do, what would my next steps be?

[acarbonetto]

Would it be worth opening a discussion with the amazon guys for this PR and demo it to them?

[MitchellGale]

@MaxKsyunz I just edited the description to answer your question :). But currently it is only added to core, I can certainly add it to additional (or all, but aI think all would be overkill) modules.
Also added more information about rational for adding it.

@acarbonetto Perhaps, could be good to get feedback. Also could be done if they have any questions after seeing the upstream PR.

@Yury-Fridlyand Added link to docs for PiTest. Added presentation slides :). Added more details on features of PiTest, would be happy to discuss adding more if there is anything else I can improve!

Thanks all for feedback.