Add FAQ for why Cypress passes locally and not in CI.

[jennifer-shehane]

I'm submitting a...

[ ] Bug report
[x] Content update
[ ] Process update (build, deployment, ... )

Type of bug / changes

It is pretty common question / complain from users and deserves an answer.

Some saved responses in the past:

1

There are a lot of reasons tests could fail in CI and pass locally. On top of that - running in CI is a bit of a black box. So, the combination of these 2 facts make it especially frustrating to debug failing tests that solely fail in CI.

Some reasons this could happen:

1. There is a problem isolated to the Chrome version of our Electron browser. We're working on updating it every day. Electron v5.0.10 cypress#4720 Installing a later Chrome version in Travis would highlight if this is your problem.
2. If there is a build process in between running tests locally and running in CI - sometimes Cypress test failures are highlighting a bug in the build process.
3. Variability in timings when running your application in CI. If you have a test that will only pass when a network request finishes in 10ms (which happens every time on your local machine) and then that request takes 20ms on your CI machine (for whatever reason from that machine - location, CPU, etc slows it down) - this can cause some unexpected failures. You'll want to make sure to write assertions around the important steps that need to be reached before moving on to the next action in your tests. for example - ensuring a network request has finished before looking for the DOM element that relies on the data from that network request. I would suggest reviewing the video of the test failures, looking at the timings of commands and actions in the Command Log, and comparing them to your local run's Command Log to highlight if this is the issue.

2

Tests run in CI run within the Electron browser. So, the first suggestion is to run the tests locally within the Electron browser with DevTools open to see if there are any issues there.

Since you already mentioned running the tests locally with other browser combinations, next I would make sure that video recording and/or screenshots are turned on during the CI run. Review the video to see if anything can be detected there that is causing the failure.

If it's still not obvious, compare the Command Log from within the video to the Command Log of the same test passing locally when run in Cypress. Take a screenshot of each at the same moment and compare each command side by side. Seeing differences of the execution time or missing commands, etc can sometimes illuminate differences in the timings of requests or other methods that may have caused an issue.

3 (more about flaky tests)

Most often in cases of flaky tests, we see that there is not enough assertions surrounding the actions or XHR requests/responses necessary before moving on to the next assertion. So if there is any variation in the speed of the XHR requests response in interactive mode versus run mode, then there will start to be failures in one over the other - or flaky tests.

So, for example if you had a test to check that when you click a 'Save' button that the users page refreshes with the correct content, you may write a test like so:

cy.get('.name-input').type('Jennifer')
cy.get('form').contains('Save').click()
cy.get('#user- name).should('contain', 'Jennifer')

Each command by default will wait 4 seconds to meet it's requirements, but if the POST for the form request sometimes takeds longer to respond than it takes for the "cy.get('#user- name).should('contain', 'Jennifer')" to run, then it will fail sometimes. Because of this, we recommend asserting on as many required steps as possible - this also helps later to isolate where the exact failure is when debugging on an actual bug.

So, the example above would be less flaky written like this:

cy.server()
cy.route('POST', 'users/*').as('updateUser')
cy.route('GET', 'users').as('getUsers')

cy.get('.name-input').type('Jennifer')
cy.get('form').contains('Save').click()
cy.wait('@UpdateUser').then((xhr) => {
expect(xhr.requestBody).to.have.property('name', 'Jennifer')
}
cy.url('contains', 'users/123')
cy.wait('@getusers')
cy.get('#user- name).should('contain', 'Jennifer')

Now when the last line runs, we don't have to worry about the POST or that the POST sent the right information, that the url changed and the GET's or whatever requests needed to respond - we know it's in the right state to find the content here.

[sainthkh]

1. I guess the title should be "Add FAQ for why Cypress passes locally and not in CI." Right?
2. There is one more case: not using return when you use promise. If return is missing, it fails.

[anveshpuli]

The page does not load while testing in headless chrome mode but works fine with headed chrome mode

[jennifer-shehane]

@anveshpuli Issues in our Cypress documentation repo are reserved for documentation changes.

We recommend questions relating to how to use Cypress be asked in our community chat. Also try searching our existing GitHub issues, reading through our documentation, or searching Stack Overflow for relevant answers.

[DaniloSI]

When testing something that includes DatePickers, it may fail because of the time zone. The time zone from which the pipeline will run can be different from the time zone of the local machine. When creating the test, the expected date value may be different from the real one, since the expected one is fixed and the real one can vary according to the time zone. This problem happened to me and I solved it by treating the date in ISO format without the offset.

[RabinMallick]

Facing same issue in 2022.
Tests are passing locally but fails in Bitbucket pipeline.
Any promising solution?

[gethari]

@RabinMallick Did you get any solution

[vipingoyal7]

@RabinMallick: have u got any solution?

[adina-ahmad]

Did anybody get solution to this one?
Tests randomly fails in Azure CI pipelines.

[dariarozhko]

Experiencing the same issue: tests always passed locally but failing in circleci. Did anybody find a solution for this?

[barrynorman]

Same same ;)
Why is this happening? There must be a reason?

[longmauriceproj]

I'm experiencing the same issue after upgrading to React v18. All e2e tests passed locally and in CI prior to the upgrade. The tests still pass locally but many tests now fail with unhelpful error messages.

For context, I'm running cypress v12.9.0, react v18.0.0, and node v18.16.0

[MikeMcC399]

@longmauriceproj

You may be able to get some help from the Cypress technical community

If you take advantage of this you should say which CI provider you are using and give some details of your workflow together with the error messages you are seeing.

This repository is for making changes to the documentation. It is not the right place to report a technical issue or to get help.