Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update rand in the stdlib tests, and remove the getrandom feature from it. #104658

Merged
merged 1 commit into from
Jan 8, 2023

Conversation

thomcc
Copy link
Member

@thomcc thomcc commented Nov 21, 2022

The main goal is actually removing getrandom, so that eventually we can allow running the stdlib test suite on tier3 targets which don't have getrandom support. Currently those targets can only run the subset of stdlib tests that exist in uitests, and (generally speaking), we prefer not to test libstd functionality in uitests, which came up recently in #104095 and #104185. Additionally, the fact that we can't update rand/getrandom means we're stuck with the old set of tier3 targets, so can't test new ones.

Anyway, I haven't checked that this actually does allow use on tier3 targets (I think it does not, as some work is needed in stdlib submodules) but it moves us slightly closer to this, and seems to allow at least finally updating our rand dep, which definitely improves the status quo. Checked and works now.

For the most part, our tests and benchmarks are fine using hard-coded seeds. A couple tests seem to fail with this (stuff manipulating the environment expecting no collisions, for example), or become pointless (all inputs to a function become equivalent). In these cases I've done a (gross) dance (ab)using RandomState and Location::caller() for some extra "entropy".

Trying to share that code seems way more painful than it's worth given that the duplication is a 7-line function, even if the lines are quite gross. (Keeping in mind that sharing it would require adding rand as a non-dev dep to std, and exposing a type from it publicly, all of which sounds truly awful, even if done behind a perma-unstable feature).

See also some previous attempts:

r? @Mark-Simulacrum

@rustbot rustbot added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Nov 21, 2022
@rustbot
Copy link
Collaborator

rustbot commented Nov 21, 2022

Hey! It looks like you've submitted a new PR for the library teams!

If this PR contains changes to any rust-lang/rust public library APIs then please comment with @rustbot label +T-libs-api -T-libs to tag it appropriately. If this PR contains changes to any unstable APIs please edit the PR description to add a link to the relevant API Change Proposal or create one if you haven't already. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary.

Examples of T-libs-api changes:

  • Stabilizing library features
  • Introducing insta-stable changes such as new implementations of existing stable traits on existing stable types
  • Introducing new or changing existing unstable library APIs (excluding permanently unstable features / features without a tracking issue)
  • Changing public documentation in ways that create new stability guarantees
  • Changing observable runtime behavior of library APIs

@thomcc
Copy link
Member Author

thomcc commented Nov 21, 2022

Is the SmallRng seed's size really target-dependent? That's awful.

@thomcc
Copy link
Member Author

thomcc commented Nov 21, 2022

I'll look into that and finish this up tomorrow.

@rustbot author

@rustbot rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 21, 2022
@rust-log-analyzer

This comment has been minimized.

@rustbot rustbot added the T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. label Nov 21, 2022
@thomcc thomcc force-pushed the rand-update-and-usable-no_std branch 2 times, most recently from e495774 to 392b168 Compare November 21, 2022 06:48
@rust-log-analyzer

This comment has been minimized.

@thomcc thomcc force-pushed the rand-update-and-usable-no_std branch from 392b168 to e8fb90b Compare November 21, 2022 07:38
@thomcc

This comment was marked as outdated.

@thomcc thomcc force-pushed the rand-update-and-usable-no_std branch from e8fb90b to 2a0b7f6 Compare November 21, 2022 09:19
@thomcc
Copy link
Member Author

thomcc commented Nov 21, 2022

This seems to pass on all the targets I care to test on that would normally fail (most notably wasm32), which implies it really has excised getrandom from the stdlib's dev-dep tree (this seems to be confirmed by cargo tree, but that can be misleading).

So I don't think anything else is needed here, and it's ready for review.

@rustbot ready

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 21, 2022
library/std/tests/env.rs Outdated Show resolved Hide resolved
library/alloc/benches/slice.rs Outdated Show resolved Hide resolved
@klensy
Copy link
Contributor

klensy commented Nov 21, 2022

If all goes well, maybe bump getrandom (https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md#028---2022-10-20) too?

@thomcc
Copy link
Member Author

thomcc commented Nov 21, 2022

I'll look into bumping getrandom in a later PR when I can examine the changelog more closely (which is something we try to do for non-test-only deps, even well-vetted ones like getrandom). Part of my short-term motivation is unblocking my testing of #103503, which does require rust-random/getrandom#317, and thus a newish getrandom (although I'm unsure if that's been in a release yet).

@the8472
Copy link
Member

the8472 commented Nov 21, 2022

For the most part, our tests and benchmarks are fine using hard-coded seeds. A couple tests seem to fail with this (stuff manipulating the environment expecting no collisions, for example), or become pointless (all inputs to a function become equivalent). In these cases I've done a (gross) dance (ab)using RandomState and Location::caller() for some extra "entropy".

It feels iffy to run randomized tests with fixed seeds. Can we conditionally import and use the getrandom feature on some tier-1 targets?

@thomcc
Copy link
Member Author

thomcc commented Nov 21, 2022

It feels iffy to run randomized tests with fixed seeds. Can we conditionally import and use the getrandom feature on some tier-1 targets?

Doing this conditionally would be more painful since I'd have to duplicate a body of a #[cfg(...)] and a [target.'cfg(...)'.dev-dependencies] in more places. It also would add more conditional compliation that only fails on obscure targets, which is annoying for maintenance. I'm not sure it's really worth it given that most of those cases that actually benefit from randomized testing, I've tried to use a RNG method that isn't a fixed seed (e.g. the hacky test_rng() thing should genuinely give a different seed at least each test run).

I guess I'll take a look at what it looks like though.

@thomcc
Copy link
Member Author

thomcc commented Nov 21, 2022

I guess I'll take a look at what it looks like though.

As per #104658 (comment) I didn't end up going this route, but I have made it so that the places that were using non-determinism to improve testing are still using it. Avoiding copying the cursed function twice required moving a few of the tests inside liballoc's unit tests as a result.

rand = "0.7"
rand_xorshift = "0.2"
rand = { version = "0.8.5", default-features = false }
rand_xorshift = { version = "0.3.0", default-features = false }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Mark-Simulacrum Mark-Simulacrum added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 4, 2023
@thomcc thomcc removed the I-libs-nominated Nominated for discussion during a libs team meeting. label Jan 4, 2023
@thomcc thomcc force-pushed the rand-update-and-usable-no_std branch from 25df61d to a4bf36e Compare January 4, 2023 22:56
@thomcc
Copy link
Member Author

thomcc commented Jan 4, 2023

This was discussed in the most recent t-libs meeting. The consensus was that this implementation is acceptable.

I've rebased and squashed (to simplify the rebase) so this should be good to go.

@rustbot ready

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 4, 2023
@Mark-Simulacrum
Copy link
Member

@bors r+ rollup=iffy

@bors
Copy link
Contributor

bors commented Jan 7, 2023

📌 Commit a4bf36e has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 7, 2023
@bors
Copy link
Contributor

bors commented Jan 8, 2023

⌛ Testing commit a4bf36e with merge 2afe585...

@bors
Copy link
Contributor

bors commented Jan 8, 2023

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 2afe585 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Jan 8, 2023
@bors bors merged commit 2afe585 into rust-lang:master Jan 8, 2023
@rustbot rustbot added this to the 1.68.0 milestone Jan 8, 2023
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (2afe585): comparison URL.

Overall result: ❌✅ regressions and improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
0.7% [0.5%, 0.8%] 7
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
-0.1% [-0.1%, -0.1%] 1
All ❌✅ (primary) - - 0

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

This benchmark run did not return any relevant results for this metric.

@thomcc thomcc deleted the rand-update-and-usable-no_std branch January 8, 2023 19:18
@RalfJung RalfJung mentioned this pull request Jan 9, 2023
bors added a commit to rust-lang/miri that referenced this pull request Jan 9, 2023
RalfJung pushed a commit to RalfJung/rust that referenced this pull request Jan 9, 2023
spikespaz pushed a commit to spikespaz/dotwalk-rs that referenced this pull request Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. T-libs Relevant to the library team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.