Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some tests are incorrectly skipped for macOS CI #117721

Closed
shepmaster opened this issue Nov 8, 2023 · 6 comments · Fixed by #132979
Closed

Some tests are incorrectly skipped for macOS CI #117721

shepmaster opened this issue Nov 8, 2023 · 6 comments · Fixed by #132979
Labels
A-CI Area: Our Github Actions CI O-macos Operating system: macOS T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Comments

@shepmaster
Copy link
Member

CI specifies these two jobs for x86_64-apple-darwin:

rust/src/ci/github-actions/ci.yml

Lines 528 to 544 in 341efb1

- name: x86_64-apple-1
env: &env-x86_64-apple-tests
SCRIPT: ./x.py --stage 2 test --skip tests/ui --skip tests/rustdoc --skip tests/run-make-fulldeps
RUST_CONFIGURE_ARGS: --build=x86_64-apple-darwin --enable-sanitizers --enable-profiler --set rust.jemalloc --set llvm.ninja=false
RUSTC_RETRY_LINKER_ON_SEGFAULT: 1
MACOSX_DEPLOYMENT_TARGET: 10.12
MACOSX_STD_DEPLOYMENT_TARGET: 10.12
NO_LLVM_ASSERTIONS: 1
NO_DEBUG_ASSERTIONS: 1
NO_OVERFLOW_CHECKS: 1
<<: *job-macos-xl
- name: x86_64-apple-2
env:
SCRIPT: ./x.py --stage 2 test tests/ui tests/rustdoc tests/run-make-fulldeps
<<: *env-x86_64-apple-tests
<<: *job-macos-xl

Note these two lines (with padding I've added):

./x.py --stage 2 test --skip tests/ui --skip tests/rustdoc --skip tests/run-make-fulldeps 
./x.py --stage 2 test        tests/ui        tests/rustdoc        tests/run-make-fulldeps

It appears that the intent was for the first job to run a subset of the tests and the second job to run the complementary subset. However, it seems that --skip performs a substring match while the list of directories is an exact match. This means that the tests in rustdoc-js, rustdoc-json, and rustdoc-ui are never run on macOS CI — they are excluded by --skip and never explicitly added back.

Surprisingly, it seems that tests in rustdoc-js-std are run. That seems like a secondary bug as it isn't behaving the same.
@shepmaster shepmaster added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) C-bug Category: This is a bug. A-CI Area: Our Github Actions CI labels Nov 8, 2023
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Nov 8, 2023
@shepmaster
Copy link
Member Author

My suggestion would be to have another flag that performs the same substring match. For example, using --only as the option:

./x.py --stage 2 test --skip tests/ui --skip tests/rustdoc --skip tests/run-make-fulldeps 
./x.py --stage 2 test --only tests/ui --only tests/rustdoc --only tests/run-make-fulldeps

@saethlin saethlin removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Nov 9, 2023
@onur-ozkan onur-ozkan self-assigned this Nov 9, 2023
@onur-ozkan
Copy link
Member

If you don't want --skip to match as a SUBSTRING, you can use the --exact flag with x.

e.g., x --stage 2 tests/rustdoc-js --skip tests/rustdoc --force-rerun -- --exact

Does this help?

@onur-ozkan onur-ozkan removed their assignment Apr 12, 2024
@onur-ozkan onur-ozkan removed the C-bug Category: This is a bug. label Apr 12, 2024
@madsmtm
Copy link
Contributor

madsmtm commented Nov 12, 2024

@rustbot label O-macos

@rustbot rustbot added the O-macos Operating system: macOS label Nov 12, 2024
@onur-ozkan
Copy link
Member

If you don't want --skip to match as a SUBSTRING, you can use the --exact flag with x.

e.g., x --stage 2 tests/rustdoc-js --skip tests/rustdoc --force-rerun -- --exact

Does this help?

This should work fine, feel free to re-open the issue if it doesn't help.

@onur-ozkan onur-ozkan closed this as not planned Won't fix, can't repro, duplicate, stale Nov 13, 2024
@madsmtm
Copy link
Contributor

madsmtm commented Nov 13, 2024

We need to actually do this in our CI IMO before the issue should be closed.

@onur-ozkan onur-ozkan reopened this Nov 13, 2024
@onur-ozkan
Copy link
Member

We need to actually do this in our CI IMO before the issue should be closed.

Yeah, makes sense.

@onur-ozkan onur-ozkan mentioned this issue Nov 13, 2024
Merged
GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this issue Nov 13, 2024
@GuillaumeGomez
Rollup merge of rust-lang#132979 - onur-ozkan:skip-exact, r=jieyouxu
ea16a5e 
use `--exact` on `--skip` to avoid unintended substring matches

Without the `--exact` flag, using `--skip tests/rustdoc` can unintentionally skip other tests that match as substrings such as `rustdoc-gui`, `rustdoc-js`, etc.

For debugging, run: `./x.py --stage 2 test rustdoc-ui --skip tests/rustdoc` and `./x.py --stage 2 test rustdoc-ui --skip tests/rustdoc -- --exact`

Resolves rust-lang#117721
bors added a commit to rust-lang-ci/rust that referenced this issue Nov 13, 2024
@bors
Auto merge of rust-lang#132979 - onur-ozkan:skip-exact, r=<try>
8675613 
use `--exact` on `--skip` to avoid unintended substring matches

Without the `--exact` flag, using `--skip tests/rustdoc` can unintentionally skip other tests that match as substrings such as `rustdoc-gui`, `rustdoc-js`, etc.

For debugging, run: `./x.py --stage 2 test rustdoc-ui --skip tests/rustdoc` and `./x.py --stage 2 test rustdoc-ui --skip tests/rustdoc -- --exact`

Resolves rust-lang#117721

try-job: x86_64-apple-1
bors added a commit to rust-lang-ci/rust that referenced this issue Nov 13, 2024
@bors
Auto merge of rust-lang#132979 - onur-ozkan:skip-exact, r=<try>
505cd65 
use `--exact` on `--skip` to avoid unintended substring matches

Without the `--exact` flag, using `--skip tests/rustdoc` can unintentionally skip other tests that match as substrings such as `rustdoc-gui`, `rustdoc-js`, etc.

For debugging, run: `./x.py --stage 2 test rustdoc-ui --skip tests/rustdoc` and `./x.py --stage 2 test rustdoc-ui --skip tests/rustdoc -- --exact`

Resolves rust-lang#117721

try-job: x86_64-apple-1
@bors bors closed this as completed in ee2d862 Nov 27, 2024
rust-timer added a commit to rust-lang-ci/rust that referenced this issue Nov 27, 2024
@rust-timer
Unrolled build for rust-lang#132979
10c6af1 
Rollup merge of rust-lang#132979 - onur-ozkan:skip-exact, r=jieyouxu,tgross35

use `--exact` on `--skip` to avoid unintended substring matches

Without the `--exact` flag, using `--skip tests/rustdoc` can unintentionally skip other tests that match as substrings such as `rustdoc-gui`, `rustdoc-js`, etc.

For debugging, run: `./x.py --stage 2 test rustdoc-ui --skip tests/rustdoc` and `./x.py --stage 2 test rustdoc-ui --skip tests/rustdoc -- --exact`

Resolves rust-lang#117721

try-job: x86_64-apple-1
github-actions bot pushed a commit to rust-lang/miri that referenced this issue Nov 28, 2024
@matthiaskrgr
Rollup merge of #132979 - onur-ozkan:skip-exact, r=jieyouxu,tgross35
487c54a 
use `--exact` on `--skip` to avoid unintended substring matches

Without the `--exact` flag, using `--skip tests/rustdoc` can unintentionally skip other tests that match as substrings such as `rustdoc-gui`, `rustdoc-js`, etc.

For debugging, run: `./x.py --stage 2 test rustdoc-ui --skip tests/rustdoc` and `./x.py --stage 2 test rustdoc-ui --skip tests/rustdoc -- --exact`

Resolves rust-lang/rust#117721

try-job: x86_64-apple-1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-CI Area: Our Github Actions CI O-macos Operating system: macOS T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

use --exact on --skip to avoid unintended substring matches onur-ozkan/rust
5 participants
@shepmaster @madsmtm @saethlin @onur-ozkan @rustbot