[FEATURE][EXPERIMENTAL] Dependency graph based testing strategy and related pipeline #3738

cdkini · 2021-11-23T19:55:39Z

THIS IS A PROOF OF CONCEPT

Changes proposed in this pull request:

Programmatically determine which test files to run in Azure through a script (we follow a similar approach in our docs integration). This brings down the coverage of each individual run but increases performance greatly. The number of tests selected are dependent on how many files have changed and how important those files are to the codebase (a change to data_context.py will cause 99% of the suite to run while a change to rule.py will be quick since it is less impactful).

CI/CD currently

CI/CD with script (changed rule.py)

Note that this script is NOT running the same number of tests (that's where the performance gain comes from)
Also note that this is a best case scenario; on average, tests will be faster

If we supplement this dynamic test runner with daily runs of the entire suite, I think we can maintain strong coverage while dramatically improving CI/CD time. The question is, do we think the complexity of this script is worth the proposed benefit and does it provide enough coverage to be reliable?

Definition of Done

Please delete options that are not relevant.

My code follows the Great Expectations style guide
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added unit tests where applicable and made sure that new and existing tests are passing.
I have run any local integration tests and made sure that nothing is broken.

Thank you for submitting!

netlify · 2021-11-23T19:55:44Z

✔️ Deploy Preview for niobium-lead-7998 ready!

🔨 Explore the source changes: 300d1c9

🔍 Inspect the deploy log: https://app.netlify.com/sites/niobium-lead-7998/deploys/61b0d3cf116cd7000780222f

😎 Browse the preview: https://deploy-preview-3738--niobium-lead-7998.netlify.app

github-actions · 2021-11-23T19:55:59Z

HOWDY! This is your friendly 🤖 CHANGELOG bot 🤖

Please don't forget to add a clear and succinct description of your change under the Develop header in docs_rtd/changelog.rst, if applicable. This will help us with the release process. See the Contribution checklist in the Great Expectations documentation for the type of labels to use!

✨ Thank you! ✨

cdkini · 2021-11-23T23:04:47Z

azure-pipelines.yml

@@ -155,7 +155,7 @@ stages:

          - script: |
              pip install pytest pytest-cov pytest-azurepipelines
-              pytest $(GE_pytest_opts) --napoleon-docstrings --junitxml=junit/test-results.xml --cov=. --cov-report=xml --cov-report=html --ignore=tests/cli --ignore=tests/integration/usage_statistics
+              python scripts/determine_test_files_to_run.py | xargs pytest $(GE_pytest_opts) --napoleon-docstrings --junitxml=junit/test-results.xml --cov=. --cov-report=xml --cov-report=html --ignore=tests/cli --ignore=tests/integration/usage_statistics


We should take the output of the script and store it in a var so it's only used once.

cdkini · 2021-11-23T23:05:06Z

great_expectations/rule_based_profiler/rule/rule.py

+# This is a test change
 import copy
 from typing import Dict, List, Optional


A small change to trigger an example run.

…tions into feature/script-for-streamlined-testing-in-azure-pipelines

cdkini · 2021-12-08T00:58:09Z

azure-pipelines-dependency-graph-testing.yml

+# This pipeline is meant to run the GE test suite with an experimental test runner strategy.
+# The significant changes between this YAML and the primary azure-pipelines.yml file is
+# that this only tests Python 3.8 (for performance considerations) and utilizes a
+# custom script to filter the test files selected and passed on to pytest.
+trigger:
+  branches:
+    include:
+    - pre_pr-*
+    - develop
+    - main


Let me know if this should be streamlined further (or conversely, more closely follow the main YAML).

An alternative approach would be to add this change in our current pipeline but as an additional stage in our compatability or comprehensive matrices. This would introduce the change to production but reduce the overall burden of the experiment.

…tions into feature/script-for-streamlined-testing-in-azure-pipelines

alexsherstinsky · 2021-12-08T15:42:13Z

scripts/determine_tests_to_run.py

@@ -0,0 +1,265 @@
+"""


alexsherstinsky · 2021-12-08T15:44:53Z

azure-pipelines-dependency-graph-testing.yml

@@ -0,0 +1,333 @@
+# This pipeline is meant to run the GE test suite with an experimental test runner strategy.


alexsherstinsky

LGTM! ❤️ Beautiful code -- so excited to start using it! Thank you so much, @cdkini !

…tions into feature/script-for-streamlined-testing-in-azure-pipelines

…NT-29/instrument-expectation-suite-for-usage-stats * feature/ANT-29/datacontext-is-singleton: update tests after review and fantastic suggestsions [FEATURE][EXPERIMENTAL] Dependency graph based testing strategy and related pipeline (#3738) Fix issue where configuration store didn't allow nesting (#3811) [FEATURE] Add suite creation type field to CLI SUITE "new" and "edit" Usage Statistics events (#3810)

feat: Add script to scripts/ dir

62dfac7

cdkini changed the title ~~[EXPERIMENTAL][WIP][FEATURE] Script to streamline Azure pipeline testing runs~~ [WIP][FEATURE] Script to streamline Azure pipeline testing runs Nov 23, 2021

cdkini self-assigned this Nov 23, 2021

cdkini added 11 commits November 23, 2021 15:17

feat: Use script in pipeline

b811e4f

test: Add test change

5415422

feat: Add git integration

1a40916

test: Add another test change

ab92796

chore: Revert data context

1328d14

test: Another test change

50292cc

test: Another test change

47a9236

docs: Add docstr to script

9ffd9ae

chore: Bring down depth to 2

2917fcd

docs: Additional docstr updates

e69d070

docs: Additional docstr updates

0f97f51

cdkini changed the title ~~[WIP][FEATURE] Script to streamline Azure pipeline testing runs~~ [Proof of Concept] Script to streamline Azure pipeline testing runs Nov 23, 2021

cdkini commented Nov 23, 2021

View reviewed changes

cdkini added 2 commits November 23, 2021 18:15

docs: Add comment in graph traversal func

8486139

Merge branch 'develop' of github.com:great-expectations/great_expecta…

1d32194

…tions into feature/script-for-streamlined-testing-in-azure-pipelines

cdkini force-pushed the feature/script-for-streamlined-testing-in-azure-pipelines branch from d205a37 to 1d32194 Compare November 23, 2021 23:15

cdkini closed this Nov 24, 2021

cdkini added 6 commits November 24, 2021 10:13

feat: Misc tweaks to algo

5da95b5

feat: Misc changes

8b99f40

Merge branch 'develop' of github.com:great-expectations/great_expecta…

1d3fb19

…tions into feature/script-for-streamlined-testing-in-azure-pipelines

chore: Revert test changes

872c930

docs: Update all docstrs

371244f

docs: Comment on script output

bb8af24

cdkini reopened this Nov 24, 2021

cdkini added 8 commits December 7, 2021 11:02

chore: Revert test changes

cc15d0e

Merge branch 'develop' of github.com:great-expectations/great_expecta…

7715aab

…tions into feature/script-for-streamlined-testing-in-azure-pipelines

chore: Move changes to their own YAML

ed133f7

chore: Test change

f74b682

feat: Limit Azure YAML to 3.8

30d4b26

feat: Clean up pipeline YAML

e71dcf9

chore: Restore test change

2215050

feat: Misc changes

ab2b438

cdkini changed the title ~~[Proof of Concept] Script to streamline Azure pipeline testing runs~~ [FEATURE][EXPERIMENTAL] Dependency graph based testing strategy and related pipeline Dec 8, 2021

cdkini marked this pull request as ready for review December 8, 2021 00:45

cdkini commented Dec 8, 2021

View reviewed changes

cdkini added 5 commits December 8, 2021 08:25

Merge branch 'develop' of github.com:great-expectations/great_expecta…

8385768

…tions into feature/script-for-streamlined-testing-in-azure-pipelines

feat: Further clean up pipeline YAML

366eddc

test: See what final pipeline looks like

b07eed5

chore: Revert test changes

cf48f35

chore: Add back remainder of matrices

a82a4f7

cdkini requested review from alexsherstinsky, donaldheppner, NathanFarmer, Shinnnyshinshin and anthonyburdi December 8, 2021 13:48

alexsherstinsky reviewed Dec 8, 2021

View reviewed changes

scripts/determine_tests_to_run.py

@@ -0,0 +1,265 @@

"""

Copy link

Contributor

alexsherstinsky Dec 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

alexsherstinsky reviewed Dec 8, 2021

View reviewed changes

alexsherstinsky approved these changes Dec 8, 2021

View reviewed changes

cdkini added 2 commits December 8, 2021 10:47

Merge branch 'develop' of github.com:great-expectations/great_expecta…

1e49934

…tions into feature/script-for-streamlined-testing-in-azure-pipelines

fix: Fix edge case paths

300d1c9

cdkini merged commit 1f150ff into develop Dec 8, 2021

cdkini deleted the feature/script-for-streamlined-testing-in-azure-pipelines branch December 8, 2021 15:54

fjork3 mentioned this pull request Dec 9, 2021

release candidate for 0.13.46 #3831

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE][EXPERIMENTAL] Dependency graph based testing strategy and related pipeline #3738

[FEATURE][EXPERIMENTAL] Dependency graph based testing strategy and related pipeline #3738

cdkini commented Nov 23, 2021 •

edited

Loading

netlify bot commented Nov 23, 2021 •

edited

Loading

github-actions bot commented Nov 23, 2021

cdkini Nov 23, 2021

cdkini Nov 23, 2021

cdkini Dec 8, 2021

alexsherstinsky Dec 8, 2021

alexsherstinsky Dec 8, 2021

alexsherstinsky left a comment

		@@ -0,0 +1,333 @@
		# This pipeline is meant to run the GE test suite with an experimental test runner strategy.

[FEATURE][EXPERIMENTAL] Dependency graph based testing strategy and related pipeline #3738

[FEATURE][EXPERIMENTAL] Dependency graph based testing strategy and related pipeline #3738

Conversation

cdkini commented Nov 23, 2021 • edited Loading

THIS IS A PROOF OF CONCEPT

Definition of Done

netlify bot commented Nov 23, 2021 • edited Loading

github-actions bot commented Nov 23, 2021

HOWDY! This is your friendly 🤖 CHANGELOG bot 🤖

cdkini Nov 23, 2021

Choose a reason for hiding this comment

cdkini Nov 23, 2021

Choose a reason for hiding this comment

cdkini Dec 8, 2021

Choose a reason for hiding this comment

alexsherstinsky Dec 8, 2021

Choose a reason for hiding this comment

alexsherstinsky Dec 8, 2021

Choose a reason for hiding this comment

alexsherstinsky left a comment

Choose a reason for hiding this comment

cdkini commented Nov 23, 2021 •

edited

Loading

netlify bot commented Nov 23, 2021 •

edited

Loading