Performance tests: Create wpCLI() utility #50868
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WunderBart
added
[Type] Performance
youknowriad
reviewed
May 23, 2023
There was a problem hiding this comment.
Personally, I'm not sure this PR is a good idea. Right now, the only dependency to running e2e tests is a "URL". This PR adds an extra dependency: to run the e2e tests, you actually need to have wp cli available.
- For instance, today, we're able to run the e2e tests over any custom local environment (wp-env is not enforced)
- We're also able to run e2e tests over WP-core using its own environemnt without enabling the Gutenberg Plugin.
All these use-cases are still possible after this PR but it becomes too complex as you need to setup wp-cli properly and provide the right env variable.
On the other hand, we're creating a discrepancy between running the performance tests via the performance.js runner and directly via |
|
Size Change: 0 B Total Size: 1.4 MB ℹ️ View Unchanged
|
Good point, thought maybe the best way to deal with that is to actually bundle the theme to test with in the repo? (like we do for the empty them). That said, I can see us using wp-cli for performance tests (at least in a preparation step, not within the tests themselves) but creating a util for e2e tests is a bit too much as it adds this dependency. It seems important to me that e2e tests remain independent to be able to run the tests over different kind of environments to check for regressions when back porting to core ... |
That should work well! 🙌 I can see there even is a dedicated |
|
cc @oandregal just want to make sure to include you in the discussion here about the themes used in performance tests. |
The PR is here: #50905 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
performance.jsrunner and directly vianpm run test:performance <suite-name>.Why?
For more consistent testing via a controlled environment.
Testing Instructions
The following should install the necessary themes in the before hook and revert by updating them in the after hook:
npm run test:performance specs/performance/front-end-classic-theme.test.jsnpm run test:performance specs/performance/front-end-block-theme.test.jsExample run: