[Mobile] - E2E Tests - Fix block insertion flakiness #46619
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
geriux
added
Mobile App - i.e. Android or iOS
|
Size Change: +183 B (0%) Total Size: 1.32 MB
ℹ️ View Unchanged
|
added 2 commits
December 16, 2022 18:46
…fic cases like "Adding a block before" option
SiobhyB
reviewed
Dec 19, 2022
There was a problem hiding this comment.
Thanks for working on these, @geriux! 🙌
I can confirm both suites pass when I run them locally for iOS and I see the iOS tests pass on the Mobile Gutenberg PR too. I can also confirm the gutenberg-editor-block-insertion-2.test.js tests pass for me on Android.
I'm having a bit of trouble getting the gutenberg-editor-block-insertion.test.js tests to pass locally on Android. I keep running into the following error, with the tests going on to fail:
Gutenberg Editor tests for Block insertion › should be able to insert multi-paragraph text, and text to another paragraph block in between
TypeError: blockActionsMenuButton.click is not a function
586 |
587 | const listBlockTextLocatorIOS = options.isEmptyBlock
> 588 | ? `(//XCUIElementTypeStaticText[contains(@name, "List")])`
| ^
589 | : `//XCUIElementTypeButton[contains(@name, "List")]//XCUIElementTypeTextView`;
590 |
591 | const listBlockTextLocator = isAndroid()
at EditorPage.<anonymous> (__device-tests__/pages/editor-page.js:588:38)
at asyncGeneratorStep (../../node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
at _next (../../node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
I wondered if that may be related to my emulator not supporting Appium's .click function for some reason, but am also seeing the suite failed to run on the Gutenberg Mobile PR:
FAIL gutenberg/packages/react-native-editor/__device-tests__/gutenberg-editor-block-insertion.test.js
Test suite failed to run
[quit()] Unexpected data in simpleCallback.
at exports.newError (gutenberg/node_modules/wd/lib/utils.js:152:13)
at PromiseWebdriver.<anonymous> (gutenberg/node_modules/wd/lib/callbacks.js:61:12)
at cb (gutenberg/node_modules/wd/lib/webdriver.js:188:22)
at gutenberg/node_modules/wd/lib/webdriver.js:202:5
at Request._callback (gutenberg/node_modules/wd/lib/http-utils.js:89:7)
at Request.self.callback (gutenberg/node_modules/request/request.js:185:22)
at Request.<anonymous> (gutenberg/node_modules/request/request.js:1161:10)
at IncomingMessage.<anonymous> (gutenberg/node_modules/request/request.js:1083:12)
Any ideas?
That's interesting, I've tried to make it fail locally but it passes correctly 🤔
Have you tried using the "Wipe data" option in Android Studio under the Device Manager window? Sometimes I've had to do that when my emulator gets stuck and doesn't get any Thanks for linking to that test workflow, I think something went wrong in that one and the environment because a few tests failed randomly 😅 |
SiobhyB
approved these changes
Dec 20, 2022
There was a problem hiding this comment.
Thanks, Gerardo! I see the most recent tests on the Gutenberg Mobile PR have passed. I've also been able to get them to pass locally for Android after some changes to my local environment :D
All looks good to me, thanks again for improving the E2E tests! 🙇♀️
geriux
removed
the
Mobile App - Automation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related PRs:
What?
This PR addresses an issue with two E2E tests that fail constantly, related to Block insertion.
Why?
To avoid constant failures of these flaky tests, we are updating them so they're more stable.
How?
These tests check that block insertion works correctly by adding different blocks and then checking the output HTML to have the right content added, this is a problem if there are no other checks along the way while blocks are being added and just go directly to check the HTML content, due to delays or content taking more time to be added.
To avoid unstable test results, we are asserting that each block that is added exists before continuing to add the rest.
This PR updates the following E2E tests:
Block Insertion
For this test we check some elements exist after being created, we also added a check for the Paragraph's TextInput to be there before typing content.
Block Insertion 2
We are also checking each block exists after they're added before adding more blocks.
It now waits for the media library picker to be opened
waitForMediaLibraryafter adding theImageblock.The
longPressMiddleOfElementhelper, now supports passing the element's size, since I saw on Android for theAdd blockbutton, it was not returning the rightwidthsize of the element, causing the test to break on Android.Testing Instructions
CI check on Gutenberg Mobile should pass from the latest commit.
Testing Instructions for Keyboard
N/A
Screenshots or screencast
N/A