Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: adds weekly scheduled smoke tests #28

Merged
merged 2 commits into from
Dec 24, 2022

Conversation

AriPerkkio
Copy link
Member

@AriPerkkio AriPerkkio commented Dec 23, 2022

This PR fixes/implements the following bugs/features

  • Bug
  • Feature
  • Breaking changes
  • Test improvements

What

Adds weekly scheduled smoke tests to be run on every Sunday at 00:00. Tests can also be trigger manually (workflow_dispatch). These tests will be looking for rule crashes.

Some example ESLint community plugins that are using this tool already:

  • eslint-plugin-react
  • eslint-plugin-jest
  • eslint-plugin-jest-extended
  • eslint-plugin-jest-dom
  • eslint-plugin-testing-library
  • eslint-plugin-unicorn

Helps finding issues like below even before releasing:

Here is the first run: AriPerkkio#1

Detected 26770 ESLint reports and/or crashes.
Scanned 5288 repositories.

Rules:

  • vitest/expect-expect
  • vitest/no-conditional-tests
  • vitest/lower-case-title

How

Plugin maintainer making sure all existing rules do not crash

Utilizes https://github.com/marketplace/actions/eslint-remote-tester-runner. Each test run will check as many repositories as it can in 5 hours 30 minutes (default value of timeLimit). Utilizes over 10K repositories from eslint-remote-tester-repositories.

Available configuration options:

When test fails the @github-actions bot will open new issue with title Results of weekly scheduled smoke test. It will reuse the existing issue if previous one hasn't yet been closed. It's recommended to close the issues once their findings are fixed.

Below are two examples of test results:

Copy link
Member

@veritem veritem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@veritem veritem merged commit 1096d44 into vitest-dev:main Dec 24, 2022
@AriPerkkio
Copy link
Member Author

@veritem the ts-node is actually required, it was now removed by 978dad5.

https://github.com/AriPerkkio/eslint-remote-tester#eslint-remote-testerconfigjs

Configuration file can also be written in TypeScript if ts-node is installed.

@veritem
Copy link
Member

veritem commented Dec 24, 2022

I haven't read about that, let me add it in another commit.

@AriPerkkio AriPerkkio deleted the test/smoke-tests branch December 25, 2022 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants