Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate test targets for unit and integration tests #831

Merged
merged 1 commit into from
Oct 2, 2023
Merged

Separate test targets for unit and integration tests #831

merged 1 commit into from
Oct 2, 2023

Conversation

matus-tomlein
Copy link
Contributor

This PR aims to improve the flakiness of the integration tests in the following ways:

  • Separate out a new testing target for the integration tests and only run it on iOS. It is not necessary to run the integration tests on all platforms and this makes the impact of flaky tests smaller.
  • The integration tests are run separately from unit tests. Even in case they fail, it will be contained to that one test run and less tests have to be re-run.
  • Don't use the fail-fast strategy on the tests in CI because that cancels any other jobs in case one of them fails. It makes the impact of flaky tests larger.

While these changes don't solve the flakiness, I think they can significantly improve the impact of the flakiness of the tests so that it happens less often and is faster to re-run failed tests.

@matus-tomlein matus-tomlein marked this pull request as ready for review September 28, 2023 16:10
@matus-tomlein matus-tomlein changed the base branch from master to release/5.5.0 September 28, 2023 16:10
mscwilson
mscwilson approved these changes Sep 29, 2023
View reviewed changes
Copy link
Contributor

@mscwilson mscwilson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, very much needed improvement

.github/workflows/build.yml Show resolved Hide resolved
@matus-tomlein matus-tomlein changed the base branch from release/5.6.0 to release/5.5.0 October 2, 2023 12:04
@matus-tomlein matus-tomlein merged commit f02d952 into release/5.5.0 Oct 2, 2023
15 checks passed
@matus-tomlein matus-tomlein deleted the issue/micro_test_updates branch October 2, 2023 12:05
@matus-tomlein matus-tomlein mentioned this pull request Oct 2, 2023
Merged
greg-el pushed a commit that referenced this pull request Oct 9, 2023
@matus-tomlein @greg-el
Separate targets for unit and integration tests (#831)
33464c3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mscwilson mscwilson mscwilson approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@matus-tomlein @mscwilson