VizReg e2e tests: programmatically test all combinations of a given list of props/values #48260
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ciampo
added
[Type] Enhancement
ciampo
changed the title
|
Size Change: +316 B (0%) Total Size: 1.33 MB
ℹ️ View Unchanged
|
|
Flaky tests detected in 735d9d2. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/4261441013
|
|
Edit: pushing a new, simpler approach. |
ciampo
changed the title
ciampo
changed the title
chad1008
reviewed
Feb 22, 2023
ciampo
force-pushed
the
feat/storybook-e2e-tests-custom-controls
branch
from
3a6c8a5 to
4ca2d32
Compare
|
@ciampo I don't fully understand the code base, but does it make sense to run the following in parallel? From what I have tested, it seems to be about 10 seconds faster on average. diff --git a/test/storybook-playwright/specs/hstack.spec.ts b/test/storybook-playwright/specs/hstack.spec.ts
index eee5de11dd..164d68d27b 100644
--- a/test/storybook-playwright/specs/hstack.spec.ts
+++ b/test/storybook-playwright/specs/hstack.spec.ts
@@ -36,7 +36,7 @@ const PROP_VALUES_TO_TEST = [
},
];
-test.describe( 'HStack', () => {
+test.describe.parallel( 'HStack', () => {
test.beforeEach( async ( { page } ) => {
await gotoStoryId( page, 'components-experimental-hstack--default', {
decorators: { marginChecker: 'show', customE2EControls: 'show' },
diff --git a/test/storybook-playwright/specs/vstack.spec.ts b/test/storybook-playwright/specs/vstack.spec.ts
index 18eaeab70b..560e6e62aa 100644
--- a/test/storybook-playwright/specs/vstack.spec.ts
+++ b/test/storybook-playwright/specs/vstack.spec.ts
@@ -36,7 +36,7 @@ const PROP_VALUES_TO_TEST = [
},
];
-test.describe( 'VStack', () => {
+test.describe.parallel( 'VStack', () => {
test.beforeEach( async ( { page } ) => {
await gotoStoryId( page, 'components-experimental-vstack--default', {
decorators: { marginChecker: 'show', customE2EControls: 'show' },I have also tried to explore the way to dynamically pass props to the component for each test, but so far I have not been able to find any approach other than using textarea and state 😅 @kevin940726 |
|
Thank you for the suggestion re. parallel execution! I'm quite inexperienced about Playwright, and therefore I wasn't aware of the
Yeah, I also didn't find a better approach so far. I also decided not to spend too long on it, since my main goal was correctness (and not necessarily getting to the most refined solution 😅 ) |
…ions via snapshots
1. expect prop config as input when computing permutations 2. use new form inputs to submit prop config instead of individual prop controls
ciampo
force-pushed
the
feat/storybook-e2e-tests-custom-controls
branch
from
4ca2d32 to
af44091
Compare
t-hamano
approved these changes
Feb 24, 2023
There was a problem hiding this comment.
LGTM!
Yeah, I also didn't find a better approach so far. I also decided not to spend too long on it, since my main goal was correctness (and not necessarily getting to the most refined solution 😅 )
Yes, if we find a better approach in the future, let's incorporate it 👍
|
I'm not familiar with the storybook visual testing suites, but have you tried the experimental components testing? It uses Regarding parallelism though, you can configure the entire project to be fully parallel. Or use |
Thank you, I've enabled the
That is very cool, and something that I haven't personally looked into yet! Probably worth exploring separately, so that we can merge these tests and proceed with reviewing #47914 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
This PR adds utils for testing all combinations of a a given set of props during visual regression testing.
Why?
This is a way to speed up the creation of VizReg tests, useful for programmatically testing a component in series of configurations that depend on the props passed.
I initially considered using the storybook controls add-on and pass the prop values via URL params, but I found two issues with this approach:
That's why I decided to implement a "custom" way to set props directly from e2e tests.
How?
<textarea />to the page where it's possible to input a JSON object with the prop values that needs to be tested. Submitting the form will apply those prop values to the component being tested via Story args.VStackandHStackcomponents (in preparation to Fix HStack and VStack alignment prop #47914)Testing Instructions
Generate the snapshots
npm run distclean && npm cinpx playwright installnpm run storybook:e2e:devnpm run test:e2e:storybook -- --update-snapshotsTest VizReg with on
HStackandVStackMake an amend to `VStack` or `HStack` that would cause a change in how the component visually render, for examplem(click to expand)