Basic playwright electron support #12207

xai · 2023-02-22T14:11:12Z

What it does

Allows to write tests against an Electron app by starting the Electron app first and then attaching Playwright to it. Also, it solves the workspace problem by creating it upfront and pointing to it on app start.

Because playwright cannot access native UI elements, we introduce an optional environment variable THEIA_ELECTRON_DISABLE_NATIVE_ELEMENTS that, when set to 1, enforces the rendering of HTML-based menus and file chooser dialogs, which can be accessed by playwright.

Playwright also requires us to disable early rendering of the window, which we also do via an environment variable (THEIA_ELECTRON_NO_EARLY_WINDOW).

A unified TheiaAppLoader is introduced that can be used in both browser and electron tests.
All existing tests are adjusted to use the new AppLoader and can be run in Electron if the environment variable USE_ELECTRON is set to true.
For convenience, a yarn target ui-tests-electron has been added to the playwright package.json.
Furthermore, the target ui-tests-ci has been added (runs all playwright tests in the browser and in Electron) and the Github workflow has been adjusted.

Contributed on behalf of STMicroelectronics

How to test

Build the electron example as usual
yarn electron build
Verify that electron is working
yarn electron start
Run the electron playwright tests:
```
cd examples/playwright
yarn build
yarn ui-tests-electron
```
The testsuite automatically sets all required environment variables, so there is no need to do so manually.

Note that, #12405 still prevents running the playwright tests on Windows or MacOS, which means testing this PR currently requires Linux.

I noticed some flakiness in both browser and electron playwright tests (especially the ones for text-editor, terminal, and outline-view), but I could reproduce these issues on master, so in my opinion, they are not specific to changes introduced in this PR.

Review checklist

As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

As a reviewer, I agree to behave in accordance with the review guidelines

tsmaeder · 2023-03-02T09:26:41Z

@xai could you rebase your changes on latest master to make sure the ubuntu test failures are not due to the old flakiness of the test suite?

tsmaeder · 2023-03-02T09:39:03Z

examples/playwright/configs/playwright.config.ts

-        toMatchSnapshot: { threshold: 0.15 }
+        viewport: { width: 1920, height: 1080 },
+        trace: {
+            mode: 'retain-on-failure'


If we're running on CI: where will the videos be available and are we sure we have enough space to store the videos for all PR's?

Good point. I will just disable this.

tsmaeder · 2023-03-02T10:59:06Z

examples/playwright/src/theia-app-loader.ts

+    ) { }
+
+    playwrightOptions(workspace?: TheiaWorkspace): object {
+        const executablePath = this.electronAppPath + '/node_modules/.bin/electron';


This does not work on Windows: I'm getting the message electron.launch: Failed to launch: Error: spawn ../electron/node_modules/.bin/electron ENOENT and the electron app is not launched.

doing this in ElectronLaunchOptions.playwrightOptions(...) works for me, but probably, we need to dispatch on the platform here:

const executablePath = path.resolve(this.electronAppPath, 'node_modules/.bin/electron.cmd');

Thank you! There were several issues with paths on Windows. Following your suggestion on using path instead of modifying strings, it should now work on Windows.

Ecxept that now nothing works on Windows anymore, due to #12405 😬

examples/playwright/src/tests/theia-explorer-view.test.ts

examples/playwright/src/tests/theia-status-bar.test.ts

tsmaeder · 2023-03-02T12:45:10Z

packages/core/src/electron-main/electron-main-application.ts

@@ -197,7 +197,8 @@ export class ElectronMainApplication {
    }

    async start(config: FrontendApplicationConfig): Promise<void> {
-        this.useNativeWindowFrame = this.getTitleBarStyle(config) === 'native';
+        const args = this.processArgv.getProcessArgvWithoutBin(process.argv);
+        this.useNativeWindowFrame = this.getTitleBarStyle(config) === 'native' && !args.includes('--no-native-window-frame');


Would it make sense to make this a theia CliContribution that we can include into apps to be tested?

tsmaeder · 2023-03-02T12:46:14Z

packages/core/src/electron-main/electron-main-application.ts

@@ -305,13 +306,19 @@ export class ElectronMainApplication {

    async openDefaultWindow(): Promise<BrowserWindow> {
        const [uri, electronWindow] = await Promise.all([this.createWindowUri(), this.createWindow()]);
+        if (!this.useNativeWindowFrame) {


Was this just a bug before?

On Linux, I ended up with two menubars, so I got rid of the native one using this workaround. Maybe I should try and figure out why that happens instead of fixing the symptom 😉

inlann · 2023-03-03T08:59:52Z

Hello:

Because playwright cannot access native UI elements, we introduce an optional command line switch --no-native-window-frame that enforces the rendering of HTML-based menus, which can be accessed by playwright.

HTML-based menus don't seem to work on MacOS:

So, the test failed 😢 :

mac@Titans-iMac playwright % yarn ui-tests -g "electron"
yarn run v1.22.19
$ yarn && playwright test --config=./configs/playwright.config.ts -g electron
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
success Already up-to-date.
$ yarn clean && yarn build
$ rimraf lib *.tsbuildinfo
$ tsc --incremental && npx playwright install chromium

Running 7 tests using 1 worker

  ✓  1 …s:34:9 › Theia Electron Application › should load and show main content panel (15ms)
  ✘  2 ../../src/tests/theia-electron-app.test.ts:38:9 › Theia Electron Application › open about dialog using menu (30.0s)
  ✘  3 ../../src/tests/theia-electron-app.test.ts:47:9 › Theia Electron Application › toggle explorer view using menu (30.0s)
  ✓  4 ../../src/tests/theia-electron-app.test.ts:55:9 › Theia Electron Application › open quick command palette (5.1s)
  ✓  5 ../../src/tests/theia-electron-app.test.ts:62:9 › Theia Electron Application › open about dialog using command (5.8s)
  ✓  6 ../../src/tests/theia-electron-app.test.ts:73:9 › Theia Electron Application › select all using command (11.4s)
  ✓  7 …/../src/tests/theia-electron-app.test.ts:80:9 › Theia Electron Application › toggle explorer view using command (7.1s)


  1) ../../src/tests/theia-electron-app.test.ts:38:9 › Theia Electron Application › open about dialog using menu 

    page.waitForSelector: Timeout 30000ms exceeded.
    =========================== logs ===========================
    waiting for locator('#theia\\:menubar .p-MenuBar-itemLabel').locator('text=Help') to be visible
    ============================================================

       at ../../src/theia-main-menu.ts:47

      45 |
      46 |     protected menuBarItem(label = ''): Promise<ElementHandle<SVGElement | HTMLElement> | null> {
    > 47 |         return this.page.waitForSelector(this.menuBarItemSelector(label));
         |                          ^
      48 |     }
      49 |
      50 |     protected menuBarItemSelector(label = ''): string {

        at TheiaMenuBar.menuBarItem (/Users/mac/Work/eclipsesource_theia/examples/playwright/src/theia-main-menu.ts:47:26)
        at TheiaMenuBar.openMenu (/Users/mac/Work/eclipsesource_theia/examples/playwright/src/theia-main-menu.ts:30:40)
        at /Users/mac/Work/eclipsesource_theia/examples/playwright/src/tests/theia-electron-app.test.ts:40:34

  2) ../../src/tests/theia-electron-app.test.ts:47:9 › Theia Electron Application › toggle explorer view using menu 

    page.waitForSelector: Timeout 30000ms exceeded.
    =========================== logs ===========================
    waiting for locator('#theia\\:menubar .p-MenuBar-itemLabel').locator('text=View') to be visible
    ============================================================

       at ../../src/theia-main-menu.ts:47

      45 |
      46 |     protected menuBarItem(label = ''): Promise<ElementHandle<SVGElement | HTMLElement> | null> {
    > 47 |         return this.page.waitForSelector(this.menuBarItemSelector(label));
         |                          ^
      48 |     }
      49 |
      50 |     protected menuBarItemSelector(label = ''): string {

        at TheiaMenuBar.menuBarItem (/Users/mac/Work/eclipsesource_theia/examples/playwright/src/theia-main-menu.ts:47:26)
        at TheiaMenuBar.openMenu (/Users/mac/Work/eclipsesource_theia/examples/playwright/src/theia-main-menu.ts:30:40)
        at /Users/mac/Work/eclipsesource_theia/examples/playwright/src/tests/theia-electron-app.test.ts:48:34

  Slow test file: ../../src/tests/theia-electron-app.test.ts (1.5m)
  Consider splitting slow test files to speed up parallel execution

  2 failed
    ../../src/tests/theia-electron-app.test.ts:38:9 › Theia Electron Application › open about dialog using menu 
    ../../src/tests/theia-electron-app.test.ts:47:9 › Theia Electron Application › toggle explorer view using menu 
  5 passed (1.9m)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

inlann · 2023-03-03T09:16:21Z

Does "playwright cannot access native UI elements" mean native file/folder selection dialogs such as 'Add folder to workspace' is not available?

xai · 2023-03-03T09:48:06Z

Does "playwright cannot access native UI elements" mean native file/folder selection dialogs such as 'Add folder to workspace' is not available?

Yes, correct. This is the second thing that I am currently working on. The plan is to use the plain FileDialogService instead of ElectronFileDialogservice for the Playwright tests.

xai · 2023-03-03T10:11:03Z

HTML-based menus don't seem to work on MacOS

Thank you very much for trying this on MacOS!

So using the command-line switch, you end up with no menus at all? Not good. On Linux, I suddenly had two menubars, so I set the visibility to false in https://github.com/eclipsesource/theia/blob/54798d2dc3a81b8a985c318b63cb678fd0309bfa/packages/core/src/electron-main/electron-main-application.ts#L310 as well as in line 320. What happens on MacOS if that is removed again? Also, to have the complete information here, can you please look at whether a "window.titleBarStyle" preference is set in your $HOME/.theia/settings.json and, if so, what its value is?

(Edit: typo)

xai · 2023-03-31T09:32:52Z

Does "playwright cannot access native UI elements" mean native file/folder selection dialogs such as 'Add folder to workspace' is not available?

Yes, correct. This is the second thing that I am currently working on. The plan is to use the plain FileDialogService instead of ElectronFileDialogservice for the Playwright tests.

Now the DefaultFileDialogService is used when the --no-native-window-frame command-line switch is provided. I also added a test case for opening a file using the File menu. While it works on Linux, there is a remaining issue on Windows with file system access when Electron is launched via Playwright. I am still investigating this.

inlann · 2023-04-24T10:25:37Z

Hello
I ran this e2e test successfully in Jenkins. If anyone is interested in this, here is my pipeline:

pipeline {
    agent { label "linux-x64" }

    stages {
        stage('The Matrix') {
            matrix {
                agent { label "${AGENT}" }
                axes {
                    axis {
                        name 'AGENT'
                        values 'linux-x64'  // TODO: add 'win64'
                    }
                }
                stages {
                    stage('Install & Build') {
                        steps {
                            nodejs(nodeJSInstallationName: 'Node_v16') {
                                sh 'yarn all'
                                sh 'yarn electron build'
                                sh 'yarn electron rebuild'
                            }
                        }
                    }

                    stage('Test') {
                        steps {
                            nodejs(nodeJSInstallationName: 'Node_v16') {
                                sh 'xvfb-run -s "-screen 0 1024x768x24" yarn --cwd examples/playwright ui-tests -g "electron"'
                            }

                            nodejs(nodeJSInstallationName: 'Node_v16') {
                                sh 'xvfb-run -s "-screen 1 1280x720x24" yarn --cwd examples/playwright ui-tests -g "electron"'
                            }
                        }
                    }
                }
            }
        }
    }
}

This pipeline requires Xvfb and NodeJS extension.

xai · 2023-05-30T12:31:12Z

Using a CLI switch to disable native elements was a bad idea by me, so I now replaced this with an environment variable THEIA_ELECTRON_DISABLE_NATIVE_ELEMENTS. As playwright electron is the only use case we are aware of so far, it is not justified to clutter the command line arguments with things not relevant to users.

xai · 2023-09-07T14:55:32Z

Current status of this PR (I also updated the PR description):

Integrated the previously explicit electron playwright tests into the existing tests
Ensured that all existing tests can also be run in electron (yarn ui-tests-electron)
Adjusted Github workflow to additionally test in electron

While testing, I noticed two remaining problems that in my opinion are not specific to the changes introduced by this PR:

Nothing works on Windows and MacOS (Playwright tests fail in Windows and Mac #12405)
Stability issues: some of the tests are flaky. Looks like race conditions, even when only one worker is used. I could reproduce this with the current master branch and ran some experiments. On my machine, some of the existing browser tests fail in 50 to 70% of all runs. I am not sure why the Github CI seems less affected by this so far.

xai · 2023-11-24T15:33:05Z

Just rebased again because of a conflict.

Current state:

Implemented suggested changes from @tsmaeder
Reverted unnecessary changes
Removed electron playwright workflow in the CI for now.
This will be re-added once we are sure that the stability is sufficient.

As discussed with @planger, we currently consider our support for electron playwright experimental (Playwright itself also considers its support for Electron automation experimental). This also means that Windows is not supported at this time, as we still observe plenty of stability issues there.

tsmaeder · 2023-11-30T09:53:39Z

I'm O.K. to merge this PR, since it won't break the existing PR checks, however, there are a couple of problems on Windows, which should be addressed in the near future:

Computing the workspace path is broken: it produces a path like \\C:\\Users...", which is not a valid windows path: the leading \leads to Theia being opened on theexamples\electron` folder.
We rely on cwd for copying testing data to the workspace folder in temp. But we're really trying to compute a folder relative to the current javascript file. We should use __dirname for this.
When passing a valid path for the workspace, instead of just opening that workspace, Theia ends up with a multi-root workspace including the test workspace plus the examples\electron folder as its roots.
Closing tabs is broken in test on Windows: the close box is only visible if you hover over it (at least on Windows). This never happens, so checking for closable or closing a tab always times out.

I doubt most of these problems are specific to the work in this PR, though, so following the principle of "don't make it worse", I think we can merge this one.

planger · 2023-11-30T10:41:35Z

Thanks for your review @tsmaeder! We should certainly look into the Windows support in near future.
If you are ok to merge, can you please approve this PR?
Thank you!

Contributed on behalf of STMicroelectronics. Change-Id: I7c817b00262229ab096ecbbacca5c465b9fa6302

This patch adds the possibility to disable natively rendered elements by setting the environment variable THEIA_ELECTRON_DISABLE_NATIVE_ELEMENTS to 1. We need this functionality for testing menu actions or use cases that involve file choosers using electron playwright. When the environment variable is set, the app loader enforces the rendering of HTML menus that playwright can access. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

* Use unified AppLoader for both electron and browser tests * Control early window loading of electron via environment variable THEIA_ELECTRON_NO_EARLY_WINDOW * Introduce environment variable USE_ELECTRON to run playwright tests in electron * Add ui-tests-electron target to package.json * Adjust workflow to use the new ui-tests-ci target (only one worker + both electron and browser tests) * Merge electron playwright tests into the existing tests * Skip tests that are currently too flaky in electron: - theia-output-view.test.ts - theia-quick-command.test.ts Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

Ensures that the workspace is defined before electron launches. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

Instead of using useNativeDialogs() to change bindings, we use to decide whether to call the extended FileDialogService or the parent implementation. This should simplify debugging and help moving away from the environment variable approach towards preferences or something else. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

This generalized flag applies to all electron-based UI and should therefore go to core. Also, the name is more general now. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

As suggested in the code review, we remove the explicit class and convert the options within `TheiaElectronAppLoader` if necessary. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

Replaced with factory methods within TheiaApp class. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

path.normalize() does not transform drive letters to lower case, so we revert back to the original version. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

xai · 2023-11-30T11:30:10Z

Rebased because of failed license check.

xai force-pushed the playwright-electron-support branch from e1d7afe to 54798d2 Compare February 22, 2023 14:16

JonasHelming requested a review from tsmaeder February 27, 2023 15:00

tsmaeder reviewed Mar 2, 2023

View reviewed changes

xai force-pushed the playwright-electron-support branch 4 times, most recently from 253ef88 to 1dca53e Compare March 29, 2023 15:41

vince-fugnitto added electron issues related to the electron target playwright issues related to playwright tests labels Mar 29, 2023

xai force-pushed the playwright-electron-support branch 2 times, most recently from a70cf99 to 032e71e Compare March 31, 2023 09:26

vince-fugnitto modified the milestone: 1.37.0 Apr 27, 2023

xai force-pushed the playwright-electron-support branch from 291b749 to f59e791 Compare May 30, 2023 12:22

xai force-pushed the playwright-electron-support branch from f59e791 to 2b018b0 Compare July 28, 2023 10:17

xai force-pushed the playwright-electron-support branch 2 times, most recently from ad81bb8 to 87dcd72 Compare August 29, 2023 12:20

xai force-pushed the playwright-electron-support branch 2 times, most recently from 548e3de to d421ec9 Compare September 7, 2023 14:38

planger mentioned this pull request Sep 19, 2023

Improve start-up performance #12924

Open

11 tasks

planger requested a review from tsmaeder September 28, 2023 12:05

xai requested a review from tsmaeder November 24, 2023 15:33

planger and others added 19 commits November 30, 2023 12:26

First working version

f10a58b

Contributed on behalf of STMicroelectronics. Change-Id: I7c817b00262229ab096ecbbacca5c465b9fa6302

fix: workspace loading in theia-electron-app-loader

eead583

Ensures that the workspace is defined before electron launches. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

fix: Remove duplicated bindings

199f814

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

chore: move useNativeDialogs to core + rename to useNativeElements

a2d7297

This generalized flag applies to all electron-based UI and should therefore go to core. Also, the name is more general now. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

chore: revert corrupted warning message

1536956

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

chore: refactor TheiaAppLoader

625d699

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

chore: simplify initialization of TheiaApploader

0063a43

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

chore: remove duplicated mode 'serial' annotations

b82f413

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

chore: remove unnecessary open() calls in test

8a43294

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

chore: remove class ElectronLaunchOptions

0d19547

As suggested in the code review, we remove the explicit class and convert the options within `TheiaElectronAppLoader` if necessary. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

fix: always set env variables for playwright-electron

c09bdce

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

chore: use namespaces instead of static classes in theia-app-loader

c3e9272

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

chore: remove TheiaAppMainPageObjects class

e5aebaa

Replaced with factory methods within TheiaApp class. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

fix: revert path transformation for windows/macos

fad7d23

path.normalize() does not transform drive letters to lower case, so we revert back to the original version. Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

fix: re-enable all playwright tests in the test suite

2f78ce1

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

fix: disable playwright-electron workflow

f8f01fe

Contributed on behalf of STMicroelectronics Signed-off-by: Olaf Lessenich <[email protected]>

xai force-pushed the playwright-electron-support branch from 423c681 to f8f01fe Compare November 30, 2023 11:29

tsmaeder approved these changes Nov 30, 2023

View reviewed changes

tsmaeder merged commit 487e92b into eclipse-theia:master Nov 30, 2023
14 checks passed

vince-fugnitto added this to the 1.44.0 milestone Nov 30, 2023

Maddobun mentioned this pull request Jan 26, 2024

[Electron Playwright] Theia electron app loader opens the app with multi-root workspace #13325

Open

marcdumais-work mentioned this pull request Mar 13, 2024

[playwright] Update documentation #13489

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Basic playwright electron support #12207

Basic playwright electron support #12207

xai commented Feb 22, 2023 •

edited

Loading

tsmaeder commented Mar 2, 2023

tsmaeder Mar 2, 2023

xai Mar 3, 2023

tsmaeder Mar 2, 2023

xai Mar 31, 2023

xai Sep 7, 2023

tsmaeder Mar 2, 2023

tsmaeder Mar 2, 2023

xai Mar 3, 2023

inlann commented Mar 3, 2023

inlann commented Mar 3, 2023

xai commented Mar 3, 2023

xai commented Mar 3, 2023 •

edited

Loading

xai commented Mar 31, 2023

inlann commented Apr 24, 2023 •

edited

Loading

xai commented May 30, 2023

xai commented Sep 7, 2023

xai commented Nov 24, 2023

tsmaeder commented Nov 30, 2023

planger commented Nov 30, 2023

xai commented Nov 30, 2023

Basic playwright electron support #12207

Basic playwright electron support #12207

Conversation

xai commented Feb 22, 2023 • edited Loading

What it does

How to test

Review checklist

Reminder for reviewers

tsmaeder commented Mar 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

inlann commented Mar 3, 2023

inlann commented Mar 3, 2023

xai commented Mar 3, 2023

xai commented Mar 3, 2023 • edited Loading

xai commented Mar 31, 2023

inlann commented Apr 24, 2023 • edited Loading

xai commented May 30, 2023

xai commented Sep 7, 2023

xai commented Nov 24, 2023

tsmaeder commented Nov 30, 2023

planger commented Nov 30, 2023

xai commented Nov 30, 2023

xai commented Feb 22, 2023 •

edited

Loading

xai commented Mar 3, 2023 •

edited

Loading

inlann commented Apr 24, 2023 •

edited

Loading