Run real worlds projects tests with cmd line tool #147
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
instead of using Analysis object Refactor toml schema of r-w projects to account for different experiments that may be conducted with FawltyDeps
In verify analysis
mknorps
changed the title
The if-condition that checks if a section should be verified must be compared against None. Consider: if self.imports: will fail if the imports field is missing from the TOML (as intended), but it will _also_ fail when the imports field is assigned an empty list (imports = []). The correct if-condition should be: if self.imports is not None: Same for the three other sections. Prefix relevant prints with the new experiment name, and also print when we are _skipping_ the verification of a section in the analysis.
mknorps
changed the title
mknorps
requested review from
vreuter and
Nour-Mws
and removed request for
vreuter
Wow, AWESOME!! |
vreuter
suggested changes
Feb 15, 2023
Co-authored-by: Vince <[email protected]>
mknorps
force-pushed
the
run-real-worlds-projects-tests-with-cmd-line-tool
branch
from
8926568 to
dde8f36
Compare
mknorps
force-pushed
the
run-real-worlds-projects-tests-with-cmd-line-tool
branch
from
d181989 to
0cb67de
Compare
Nour-Mws
approved these changes
Feb 15, 2023
vreuter
approved these changes
Feb 16, 2023
mknorps
deleted the
run-real-worlds-projects-tests-with-cmd-line-tool
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change in testing real projects that prepare the ground for testing dependency to import mapping.
This is a result of a very productive pair programming session of @jherland and mine.
The major change is that we no longer test using
Analysisobject, but runningfawltydepscommand with json output. (As we can see machines are already happy to be able to read FD output 🤖 ). Simplified version ofrun_fawltydeps(first added totest_cmdline.py) creates a subprocess that runsfawltydepscommand and collects the output.The term "experiment" used later, refers to one run of
fawltydepswith given command line arguments on a real-world project.Additional changes:
*.tomlfiles of real projects is changed to contain more than one experiment. All experiments are underexperimentskeyword, and for exampleall.Experiment, we have command-line arguments (args) and results of the experiment matching theAnalysisresults.Experimentobject contains averify_analysis_json methodthat does all comparison of the current experiment (self) and a given analysis.