Allow building rustdoc without first building rustc (MVP) #79540
Conversation
jyn514
added
T-rustdoc
rust-highfive
added
the
S-waiting-on-review
|
Once you'be built with |
I discussed this in 'Unresolved Questions'.
Hmm, you mean have |
jyn514
added
the
C-discussion
|
I think rustup management is the wrong approach; we should be able to use similar logic to LLVM downloads and get a rustc from previous CI commit. That'll always work if you don't modify rustc (which we can check via git). This would still require using x.py but I think that's actually not a huge burden in practice - it should work with any python version. I think we can do more to ease the introduction to x.py, but I would rather do so for all contributors rather than piecemeal. |
|
Ok, I like that idea - then most of the concerns here go away because you're still going through x.py, and |
|
That sounds like a great idea indeed! |
Mark-Simulacrum
added
S-waiting-on-author
|
@Mark-Simulacrum ideally this would only download the rustc if you were going to use it for something (and not for, say |
|
I'm only adding remarks. Iknow that my understanding of the big picture is flawed, so feel definitively free to ignore any of those. And this is definitively not a rant, but pure feedback. I am certain that least when those where introduced they made total sense. I am not sure if this PR is the right place to discuss those points either. If you want to, it may be better to continue on zulip. And I really think that this PR goes in the right direction.
Regarding this very specific sub issue, I think that adding a
As an external contributor, I 100% understand the need for a tool like I think that having a dedicated Finally, rustdoc doesn't feel any more special than any other rust project from an exteral point of view. Since, the documentation for |
|
I recently found out that the downloads shell out to curl or whatever on platforms anyway, so I'm less concerned now about keeping the logic there, but I would like to avoid duplicating it if possible. I would not worry about extra downloads to start though, I expect almost everyone to need that download anyway. Note that it shouldn't be a problem to use that compiler for bootstrap as well, so you'd just be downloading a different set of libraries, not too much more (well rustc-dev would be new, but that's I think roughly 60MB overhead these days, so not horrible). |
|
@robinmoussu we can discuss more on Zulip if you like, but the tl;dr is that
I also want to caution against trying to fix every problem at once; that's a good way to fix none of them (as I found in rust-lang/rustc-dev-guide#843). |
|
Ok, this now downloads stage1 rustc from commit build artifacts if |
jyn514
added
S-waiting-on-review
|
Oh and CI is failing because it uses |
|
Annoyingly, GitHub actions makes the .git directory read-only, so I can't run
|
|
📌 Commit 4aec8a5 has been approved by |
bors
added
S-waiting-on-bors
|
☀️ Test successful - checks-actions |
Use format string in bootstrap panic instead of a string directly
This fixes the following warning when compiling with nightly:
```
warning: panic message is not a string literal
--> src/bootstrap/builder.rs:1515:24
|
1515 | panic!(out);
| ^^^
|
= note: `#[warn(non_fmt_panic)]` on by default
= note: this is no longer accepted in Rust 2021
help: add a "{}" format string to Display the message
|
1515 | panic!("{}", out);
| ^^^^^
help: or use std::panic::panic_any instead
|
1515 | std::panic::panic_any(out);
| ^^^^^^^^^^^^^^^^^^^^^^
```
Found while working on rust-lang#79540. cc rust-lang#81645, which landed in 1.51.
This was introduced as part of the MVP for `download-rustc`. Unfortunately, it doesn't work very well: - Steps are ignored by default, which makes it easy to leave out a step that should be built. For example, the MVP forgot to enable any tests, so it was *only* possible to build locally. - It didn't work correctly even when it was enabled: calling `builder.ensure()` would completely ignore the constant and rebuild the step anyway. This has no obvious fix since `ensure()` has to return a `Step::Output`. Instead, this handles `download-rustc` in `impl Step for Rustc` and `impl Step for Std`, which to my knowledge are the only build steps that don't first go through `impl Step for Sysroot` (`Rustc` is used for the `rustc-dev` component). See rust-lang#79540 (comment) and rust-lang#81930 for further context. Here are some example runs with these changes and `download-rustc` enabled: ``` $ x.py build src/tools/clippy Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 1m 09s Building stage1 tool cargo-clippy (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.11s $ x.py test src/tools/clippy Updating only changed submodules Submodules updated in 0.01 seconds Finished dev [unoptimized + debuginfo] target(s) in 0.09s Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.09s Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.28s Finished release [optimized] target(s) in 15.26s Running build/x86_64-unknown-linux-gnu/stage1-tools/x86_64-unknown-linux-gnu/release/deps/clippy_driver-8b407b140e0aa91c test result: ok. 592 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out $ x.py build src/tools/rustdoc Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 41.28s Build completed successfully in 0:00:41 $ x.py test src/test/rustdoc-ui Building stage0 tool compiletest (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.12s Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.10s test result: ok. 105 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.15s $ x.py build compiler/rustc Finished dev [unoptimized + debuginfo] target(s) in 0.09s Build completed successfully in 0:00:00 ``` Note a few things: - Clippy depends on stage1 rustc-dev artifacts, but rustc didn't have to be recompiled. Instead, the artifacts were copied automatically. - All steps are always enabled. There is no danger of forgetting a step, since only the entrypoints have to handle `download-rustc`. - Building the compiler (`compiler/rustc`) automatically does no work.
…imulacrum Remove `ENABLE_DOWNLOAD_RUSTC` constant `ENABLE_DOWNLOAD_RUSTC` was introduced as part of the MVP for `download-rustc` as a way not to rebuild artifacts that have already been downloaded. Unfortunately, it doesn't work very well: - Steps are ignored by default, which makes it easy to leave out a step that should be built. For example, the MVP forgot to enable any tests, so it was only possible to *build* locally. - It didn't work correctly even when it was enabled: calling `builder.ensure()` would completely ignore the constant and rebuild the step anyway. This has no obvious fix since `ensure()` has to return a `Step::Output`. Instead, this handles `download-rustc` in `impl Step for Rustc` and `impl Step for Std`, which to my knowledge are the only build steps that don't first go through `impl Step for Sysroot` (`Rustc` is used for the `rustc-dev` component). See rust-lang#79540 (comment) and rust-lang#81930 for further context. Here are some example runs with these changes and `download-rustc` enabled: ``` $ x.py build src/tools/clippy Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 1m 09s Building stage1 tool cargo-clippy (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.11s $ x.py test src/tools/clippy Finished dev [unoptimized + debuginfo] target(s) in 0.09s Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.09s Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.28s Finished release [optimized] target(s) in 15.26s Running build/x86_64-unknown-linux-gnu/stage1-tools/x86_64-unknown-linux-gnu/release/deps/clippy_driver-8b407b140e0aa91c test result: ok. 592 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out $ x.py build src/tools/rustdoc Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 41.28s Build completed successfully in 0:00:41 $ x.py test src/test/rustdoc-ui Building stage0 tool compiletest (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.12s Building rustdoc for stage1 (x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.10s test result: ok. 105 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.15s $ x.py build compiler/rustc Finished dev [unoptimized + debuginfo] target(s) in 0.09s Build completed successfully in 0:00:00 ``` Note a few things: - Clippy depends on stage1 rustc-dev artifacts, but rustc didn't have to be recompiled. Instead, the artifacts were copied automatically. - All steps are always enabled. There is no danger of forgetting a step, since only the entrypoints have to handle `download-rustc`. - Building the compiler (`compiler/rustc`) automatically does no work. Helps with rust-lang#81930. r? `@Mark-Simulacrum`
…lacrum Always compile rustdoc with debug logging enabled when `download-rustc` is set Previously, logging at DEBUG or below would always be silenced, because rustc compiles tracing with the `static_max_level_info` feature. That makes sense for release artifacts, but not for developing rustdoc. Instead, this compiles two different versions of tracing: one in the release artifacts, distributed in the sysroot, and a new version compiled by rustdoc. Since `rustc_driver` is always linked to the version of sysroot, this copy/pastes `init_env_logging` into rustdoc. To avoid compiling an unnecessary version of tracing when `download-rustc` isn't set, this adds a new `using-ci-artifacts` feature for rustdoc and passes that feature in bootstrap. Addresses rust-lang#81930. This builds on rust-lang#79540. r? `@Mark-Simulacrum`
On reflection on the issue in rust-lang#79540 (comment), I think the bug was actually using the `compiler/` filter, not using `--author=bors`. rust-lang@9a1d617 has no CI artifacts because it was merged as part of a rollup: ``` $ curl -I https://ci-artifacts.rust-lang.org/rustc-builds/96e843ce6ae42e0aa519ba45e148269de347fd84/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz HTTP/2 404 ``` So 9a1d617 is the correct commit to download, and that's what `--author=bors` does: $ git log --author=bors 4aec8a5 commit 9a1d617 Ideally it would look for "the most recent bors commit not followed by a change to `compiler/`", which would exclude things like documentation changes and avoid redownloading more than necessary, but - Redownloading isn't the end of the world, - That metric is hard to implement, and - Documentation-only or library-only changes are very rare anyway since they're usually rolled up with changes to the compiler.
…acrum Fix commit detected when using `download-rustc` On reflection on the issue in rust-lang#79540 (comment), I think the bug was actually using the `compiler/` filter, not using `--author=bors`. rust-lang@9a1d617 has no CI artifacts because it was merged as part of a rollup: ``` $ curl -I https://ci-artifacts.rust-lang.org/rustc-builds/96e843ce6ae42e0aa519ba45e148269de347fd84/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz HTTP/2 404 ``` So 9a1d617 is the correct commit to download, and that's what `--author=bors` does: $ git log --author=bors 4aec8a5 commit 9a1d617 Ideally it would look for "the most recent bors commit not followed by a change to `compiler/`", which would exclude things like documentation changes and avoid redownloading more than necessary, but - Redownloading isn't the end of the world, - That metric is hard to implement, and - Documentation-only or library-only changes are very rare anyway since they're usually rolled up with changes to the compiler. Helps with rust-lang#81930. r? `@Mark-Simulacrum`
…acrum Fix commit detected when using `download-rustc` On reflection on the issue in rust-lang#79540 (comment), I think the bug was actually using the `compiler/` filter, not using `--author=bors`. rust-lang@9a1d617 has no CI artifacts because it was merged as part of a rollup: ``` $ curl -I https://ci-artifacts.rust-lang.org/rustc-builds/96e843ce6ae42e0aa519ba45e148269de347fd84/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz HTTP/2 404 ``` So 9a1d617 is the correct commit to download, and that's what `--author=bors` does: $ git log --author=bors 4aec8a5 commit 9a1d617 Ideally it would look for "the most recent bors commit not followed by a change to `compiler/`", which would exclude things like documentation changes and avoid redownloading more than necessary, but - Redownloading isn't the end of the world, - That metric is hard to implement, and - Documentation-only or library-only changes are very rare anyway since they're usually rolled up with changes to the compiler. Helps with rust-lang#81930. r? `@Mark-Simulacrum`
Motivation
The compile times for rustc are extremely long and a major issue for
recruiting new contributors to rustdoc. People interested in joining
often give up after running into issues with submodules or python
versions. stage1 rustdoc fundamentally doesn't care about bootstrapping
or stages, it just needs
rustc_privateavailable.Summary of Changes
[rust] download_rustcoptionlog --author=borsRustdocfor any stage. Instead, copy the CI artifacts from the downloaded sysroot stage0/ to stage0-sysroot/ or stage1/ inSysroot. This is done with anENABLE_DOWNLOAD_STAGE1constant which is off by default.The vast majority of work is done in bootstrap.py, which downloads the artifacts and extracts them to stage0/ in place of the beta compiler. Rustbuild just takes care of copying the artifacts to stage1 if necessary.
Future work
x.py setup toolswill compile rustbuild twice. Instead, this should download a separate beta compiler for stage0 and only use CI artifacts for stage1 onward. Allow building rustdoc without first building rustc (MVP) #79540 (comment)x.py setup toolsto enable this conveniently (it doesn't make sense to use this for compiler developers). jyn514@cb5d8c8debugandtracelogging). Allow building rustdoc without first building rustc (MVP) #79540 (comment), jyn514@6a5d512git log --author=borssometimes breaks. This should usegit merge-baseinstead. Allow building rustdoc without first building rustc (MVP) #79540 (comment)Some of this work has already been done in (the history for) jyn514@673476c.