Cypress Hangs Running Tests With Electron (Headless or Headed)

[nbcarey]

Current behavior:

Our CI/CI pipeline runs our Cypress tests on a schedule. This has been working as expected. However, yesterday (12 September 2019), the behavior changed. Instead of running to completion (with or without errors), the test suite now hangs on one particular test case. The CI/CD pipeline eventually times out after 60 minutes.

Running the tests manually on my machine with Chrome via npx cypress run --browser chrome or npx cypress open --browser chrome works just fine.

Running the tests manually with Electron, headless or headless, does not. It exhibits the same hang that our CI/CD pipeline exhibits.

Digging into it I see that it is hanging on the afterEach() action, trying to revisit the same page, via cy.visit() on which the test started. Playing around with things, It seems to fail the same way, any time cy.visit() is called twice in the context of the same test of describe() block.

Even odder, turning on debug logging via

env DEBUG=cypress:* npx cypress run --browser electron 2>&1 | tee cypress-log.txt

completed successfully.

Desired behavior:

Tests should run to completion and not hang.

Steps to reproduce: (app code and test code)

1. Clone this repo: https://github.com/nbcarey/cypress-io-hangs-with-electron
2. Running the command npm run succeeds-with-chrome gives you the happy path.
3. Running the command npm run hangs-with-electron gives you the unhappy path.

Versions

• MacOS Mojave, v10.14.6

• Chrome, v77.0.3865.75

• Node.js, v10.16.3

• Npm, v6.11.3

• Cypress, v3.4.1 (and Electron 61?)

[hiqqs]

headless electron for us stopped working for also. Noticed that electron is not able to connect to localhost as part of our test setup which is whats causing ours to hang. They are working fine in chrome but will be broken for CI

Can also resolve other domains no problem but cannot use our local web server, noticed after updating to latest webpack

[nbcarey]

I finally got almost all of our tests that were hanging working again.

Adding cy.wait() calls in strategic places seemed to fix some, but not all of the hangs, and breaking up single *.spec.js files into multiple files.

It's weird, because an individual test run by itself would work, but running the entire *.spec.js test file wouldn't. And once I got the test file to work in toto, running it as part of the entire test suite would fail.

My suspicion is some sort of race condition or timing issue in Electron. I did notice that when a test would hang, a process called Cypress Helper on my Mac would spin up to 100% CPU, consuming an entire core. Took a snapshot of it and seemed to be buried deep, deep somewhere within Electron.

Also noticed an Node.js error message getting written to stderr:

(node:16167) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exitEarlyWithErr listeners added. Use emitter.setMaxListeners() to increase limit

Wonder if that is tied to it.

[jennifer-shehane]

@nbcarey Thanks so much for providing a repo. I actually couldn't get the first case in chrome or electron from your repo to pass. Here's a screenshot and description of the failure I'm getting.

  1) product-module Any product "before each" hook for "Has a product title":
     CypressError: Timed out retrying: Expected to find element: 'div[data-element~=product-results-result-set-desktop] article[data-element~=product-results-product-module-desktop]', but never found it.

Because this error occurred during a 'before each' hook we are skipping the remaining tests in the current suite: 'Any product'

The mention of it just beginning to fail on a specific date would never be a failure caused from Cypress. Our releases our packaged complete applications, so nothing will change underneath. I would definitely encourage doing some debugging on your side to determine what changed in your application that could have triggered a change (even minor dep updates that seem innocuous could have created a condition that will fail in Cypress Electron).

[rudolfs]

Also just bumped into the same issue. Test suite passes in chrome 76, but hangs on electron 61 on CI, locally head-less and with a GUI. I could narrow it down to this:

• it only happens if there is more than one testcase in one file, this would be enough to trigger it:

context('session', () => {
  it('first', () => {
    cy.visit('/')
  })

  it('second', () => {
    cy.visit('/')
  })
})

• the app that we're testing makes GraphQL (apollo) requests to a backend; this backend is offline and in chrome we get an exception: OPTIONS http://localhost:8080/graphql net::ERR_CONNECTION_REFUSED

Will try to provide a repo with a test case soon.

[jagregory]

I'm also seeing this issue with Electron 61 in any tests where I need to switch users for a multi-user interaction.

[rudolfs]

Didn't manage to create a test repo yet, but here's the debug output of the test run of the small test-case from #5139 (comment):
https://gist.github.com/rudolfs/25bc14986d4d1451d98cee3c4307fb30

It seems to hang on trying to clean up a request from the first test, which has a timeout: null.

[hiqqs]

@rudolfs are you still having this issue?

[rudolfs]

@johnhiggs yes. I had to pause my investigation because of work priorities. I was trying to create a minimal test case but it's a little tricky because we use ReasonML.

[hiqqs]

We were finally able to figure out our issue today related to electron not working. It was related to our webpack upgrade and failing on a form try catch error block not cooperating with electron. IMO an electron bug

[julian-sf]

Yeah, so our team's been running into this issue as well. A LOT of the problems I've seen relating to this are just coming from old Electron bugs. Electron 61 was released TWO entire years ago on September 5, 2017. So yeah, this is going to be an issue until we can get a newer version of Electron running with Cypress.

In a way this IS a Cypress issue, because we need to be able to run on a more recent version of electron. The Cypress team is currently working on an upgrade, so it looks like we probably just have to wait until then.

• Roadmap
• Upgrade to the latest version of Electron #3568
• Electron v5.0.10 #4720

[flotwig]

Hey @julian-sf, Electron 5.0.10 upgrade (Chrome 73 - PR: #4720) is releasing with 3.5.0, which should come out later this week. So hopefully you won't need to wait much longer.

We're also working on upgrading to Electron 6 and keeping it up to date so this big gap doesn't happen again.

[julian-sf]

@flotwig Awesome! I know you guys have been working hard on it. I appreciate the update and I'm very much looking forward to the new version!

[fr0]

Hey @julian-sf, Electron 5.0.10 upgrade (Chrome 73 - PR: #4720) is releasing with 3.5.0, which should come out later this week. So hopefully you won't need to wait much longer.

I assume this got delayed for some reason?

[flotwig]

@fr0 Indeed, we have had some hiccups. But... soon. We aren't adding any new features for 3.5.0, just making sure that what we're releasing is quality.

[rudolfs]

This issue was resolved for me in 3.5.0. 🎉

[ngbrown]

Headless electron has started hanging only on our CI build with both 3.5.0 and 3.6.0, while it worked in 3.4.1. It hangs after finishing the first spec file and before starting the second spec file.

I do have multiple cy.visit() calls in my spec.

In addition, there is also #5558 and #5603 that seem very similar. Maybe they are more applicable to the 3.5.0 and 3.6.0 versions?

[DaveLHX]

Note to others. We had a similar issue but turns out it was caused by something we had not considered. Confirm dialogs were blocking were the cy.visit() action in subsequent tests.
It does not happen with chrome and I'm pretty sure it did not with electron 6 months ago(last I worked on this specific project) .

Its not a problem with chrome because of this

The following comment fixed our issue while we look to better handle these cases.
#2118 (comment)

[johnknoop]

It hangs in our CI environment as well (headless Electron on Windows) since upgrading from 3.4.1 to 3.7 3.5.0.

Edit: trying to see if it works in 3.7.0 right now...

Edit 2: still hangs in v 3.7.0.

[aceleghin]

I have the same issue, It hangs in my machine (headless Electron on Windows, cypress 3.7), I have 18 files to test and it hangs always at 16 of 18.

Now I am trying with --browser chrome

[stephane-dereppe]

My workaround is indeed to use the --browser chrome flag with the docker image: cypress/browsers:node12.8.1-chrome78-ff70

[DaveLHX]

I have the same issue, It hangs in my machine (headless Electron on Windows, cypress 3.7), I have 18 files to test and it hangs always at 16 of 18.

Now I am trying with --browser chrome

does the last test that runs before it hangs have a confirm dialog as was the case for us?

[stephane-dereppe]

Not sure. We tried to reorder our tests to identify the problem… but the stuck is always after the same number of tests...

…

On 12 Dec 2019, at 04:02, DaveLHX ***@***.***> wrote: ialog as was the ca

[aceleghin]

I have the same issue, It hangs in my machine (headless Electron on Windows, cypress 3.7), I have 18 files to test and it hangs always at 16 of 18.
Now I am trying with --browser chrome

does the last test that runs before it hangs have a confirm dialog as was the case for us?

No, I don't see any dialog, if I check the records in GitHub I see all passed tests until 16 of 18 and the others are orange, I also try to wait more than 1 hour but nothing happened.
With --browser chrome works.

[simplenotezy]

I too have issues with tests hanging. This test have been hanging here for 17 minutes:

Only thing that works currently is cancelling the build, running without tests and crossing my fingers.

Any possible workaround? Perhaps running tests with the --browser chrome would work?

[jennifer-shehane]

@simplenotezy Your issue is likely something not related to this issue. Please open a new issue with a fully reproducible example. The tests should not hang ever.

[jennifer-shehane]

Since this issue hasn't had activity in a while - and we've updated Electron version quite a lot since this was opened, we'll close the issue until we can confirm this is still happening. Please comment if there is new information to provide concerning the original issue and we'd be happy to reopen.