Unable to build 1.53.0 from official tarball due to miri

[liushuyu]

Hi there, I tried to build Rust 1.53.0 from the source tarball downloaded from https://static.rust-lang.org/dist/rustc-1.53.0-src.tar.gz and it failed to compile due to cargo-miri could not locate the Git repository.

After a quick search, the issue supposedly should have been fixed in rust-lang/miri@e5b3400 but for some reason, the fix didn't land in the final distribution despite in #84182 it was stated that the fix would be included in the next stable release.

Here is the backtrace:

Backtrace

  thread 'main' panicked at 'Unable to generate vergen keys!: could not find repository from '/build/i0szw9jw/rustc-1.53.0-src/src/tools/miri/cargo-miri'; class=Repository (6); code=NotFound (-3)

  Stack backtrace:
     0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
               at /build/i0szw9jw/rustc-1.53.0-src/vendor/anyhow-1.0.34/src/error.rs:464:25
     1: vergen::feature::git::configure_git
               at /build/i0szw9jw/rustc-1.53.0-src/vendor/vergen/src/feature/git.rs:175:55
     2: vergen::config::Instructions::config
               at /build/i0szw9jw/rustc-1.53.0-src/vendor/vergen/src/config.rs:139:9
     3: vergen::gen::config_from_instructions
               at /build/i0szw9jw/rustc-1.53.0-src/vendor/vergen/src/gen.rs:81:32
     4: vergen::gen::vergen
               at /build/i0szw9jw/rustc-1.53.0-src/vendor/vergen/src/gen.rs:69:5
     5: build_script_build::main
               at ./build.rs:10:5
     6: core::ops::function::FnOnce::call_once
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/ops/function.rs:227:5
     7: std::sys_common::backtrace::__rust_begin_short_backtrace
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/sys_common/backtrace.rs:125:18
     8: std::rt::lang_start::{{closure}}
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/rt.rs:49:18
     9: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/ops/function.rs:259:13
        std::panicking::try::do_call
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/panicking.rs:379:40
        std::panicking::try
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/panicking.rs:343:19
        std::panic::catch_unwind
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/panic.rs:431:14
        std::rt::lang_start_internal
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/rt.rs:34:21
    10: std::rt::lang_start
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/rt.rs:48:5
    11: main
    12: __libc_start_main
               at /build/okrh64bn/glibc/csu/../csu/libc-start.c:314:16
    13: _start
               at /build/okrh64bn/glibc/csu/../sysdeps/x86_64/start.S:120', src/tools/miri/cargo-miri/build.rs:10:24
  stack backtrace:
     0: rust_begin_unwind
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/panicking.rs:493:5
     1: core::panicking::panic_fmt
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/panicking.rs:92:14
     2: core::result::unwrap_failed
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/result.rs:1355:5
     3: core::result::Result<T,E>::expect
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/result.rs:997:23
     4: build_script_build::main
               at ./build.rs:10:5
     5: core::ops::function::FnOnce::call_once
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/ops/function.rs:227:5
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
command did not execute successfully: "/build/i0szw9jw/rustc-1.53.0-src/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "32" "-v" "--release" "--frozen" "--manifest-path" "/build/i0szw9jw/rustc-1.53.0-src/src/tools/miri/cargo-miri/Cargo.toml" "--message-format" "json-render-diagnostics"
expected success, got: exit code: 101

[Urgau]

cc #86286

[RalfJung]

Possibly the fix landed after 1.53 was branched?

Anyway, the stable release tarballs do not contain Miri any more (they were never intended to), so I'd say this is fixed by #86568.

[RalfJung]

Yeah, the git fix landed with #85402 which reached stable with 1.54.0.

[RalfJung]

I should also mention that Miri is simply not supported on stable. I don't know how exactly you are building rustc from that tarball, but you might have enabled too many "extra tools"?

[mati865]

Just have hit this error with 1.53 tarball with extended = true but tools commented-out.
Turns out when you do ./x.py install it will try to install miri and cargo-miri as well (remember we are using stable tarball).
This fails when building not in a git repository. You can see log from MSYS2 CI when it's building inside git repository: https://github.com/msys2/msys2-autobuild/runs/2945722561?check_suite_focus=true

You can see miri is present in the end:

$ ls pkg/mingw-w64-clang-x86_64-rust/clang64/bin/
cargo-clippy.exe  cargo.exe          rls.exe            rust-gdbgui  rustc_driver-02356bf24940476b.dll  std-40fc5b9b2ad404cb.dll
cargo-fmt.exe     clippy-driver.exe  rust-analyzer.exe  rust-lldb    rustdoc.exe                        test-38b9f97bb94ce93c.dll
cargo-miri.exe    miri.exe           rust-gdb           rustc.exe    rustfmt.exe

I believe this bug should be reopened and retitled because x.py install should not build miri from such tarballs.

[RalfJung]

The bug will, I think, be fixed in the next release since Miri is simply not part of the tarball any more.

I think the bug you are asking is already open: #74709

[mati865]

I have just found #86568 which should solve the issue on nightly but ideally it'd be backported to beta and stable.

[RalfJung]

Backporting to stable makes little sense (unless there's a point release, which I wouldn't expect).

Backporting to beta already happened in #86413.

[paulmenzel]

Reading the comments, I missed a solution for building Rust 1.53.0 with extended = true. Building Rust 1.50.0 worked with:

python3 ./x.py build --exclude src/tools/miri

but with 1.53.0 the --exclude switch seems to be ignored. How can Rust 1.53.0 be built including Cargo?

[mati865]

How can Rust 1.53.0 be built including Cargo?

One workaround is to create and initialize git repository in the directory where you want to build Rust (preferably before unpacking Rust to save time).

[RalfJung]

@paulmenzel I haven't tried it but shouldn't setting the tools work?

# Installs chosen set of extended tools if `extended = true`. By default builds
# all extended tools except `rust-demangler`, unless the target is also being
# built with `profiler = true`. If chosen tool failed to build the installation
# fails. If `extended = false`, this option is ignored.
#tools = ["cargo", "rls", "clippy", "rustfmt", "analysis", "src"] # + "rust-demangler" if `profiler`

So if you just want cargo, I'd try tools = ["cargo"].

[paulmenzel]

@paulmenzel I haven't tried it but shouldn't setting the tools work?

# Installs chosen set of extended tools if `extended = true`. By default builds
# all extended tools except `rust-demangler`, unless the target is also being
# built with `profiler = true`. If chosen tool failed to build the installation
# fails. If `extended = false`, this option is ignored.
#tools = ["cargo", "rls", "clippy", "rustfmt", "analysis", "src"] # + "rust-demangler" if `profiler`

So if you just want cargo, I'd try tools = ["cargo"].

Thank you. Sorry, I didn’t mention it. I had tried it, and it did not work.

[RalfJung]

@Mark-Simulacrum is that expected that Miri is built even when tools is explicitly set to a list that does not include Miri?

[Mark-Simulacrum]

In the sense that I'm not surprised, yes -- tools is pretty poorly supported around rustbuild, it's not something I have in cache to think about when we introduce new code so its easy to miss a dependency there.

It should likely get fixed to skip building miri when tools doesn't include it though.

[paulmenzel]

It should likely get fixed to skip building miri when tools doesn't include it though.

Sorry, I do not understand. It’d be great if you rephrased it.

[paulmenzel]

Can I patch the source some how (delete the miri files?) to fix the build?

[Mark-Simulacrum]

I'm not sure why the exclude switch is being ignored; you'll need to provide a full verbose log to try and dig in on what specifically is causing miri to be invoked despite the exclude.

FWIW we don't generally expect people to use x.py build in dist tarballs -- x.py dist/install are more likely to work.

[paulmenzel]

I'm not sure why the exclude switch is being ignored; you'll need to provide a full verbose log to try and dig in on what specifically is causing miri to be invoked despite the exclude.

How can I increase the verbosity?

FWIW we don't generally expect people to use x.py build in dist tarballs -- x.py dist/install are more likely to work.

src/bootstrap/README.md does not mention that.

[Mark-Simulacrum]

x.py build -vvv, for example.

I'd be happy to accept a PR (though not sure that's quite the place to update, seems OK though) that adds some text about preferring x.py dist/install when using the tarballs.

[paulmenzel]

Log from /pkg/rustc-1.53.0-0/build/rustc-1.53.0-src/build/bootstrap/debug/bootstrap -vvv dist --exclude src/tools/miri:
rustc-miri.log

(Sorry, the screen buffer was too small, but hopefully the messages still help.)

[Mark-Simulacrum]

Looks like this will be fixed by #87282, it's getting invoked through extended = true (which directly runs the dist step for miri, ignoring --exclude). I have not reviewed that PR yet, but expect to do so soon; going to add this issue to the fixes list on it though.

[Mark-Simulacrum]

I think you can fix this on 1.53 by patching the source here https://github.com/rust-lang/rust/blob/1.53.0/src/bootstrap/dist.rs#L1176 to return None from that step. You may need to set deny-warnings = false or fix some other compiler warnings in order to build with that function stubbed out like that.

[paulmenzel]

As the Rust 1.52.1 release builds and installs just fine, did anybody find the commit causing the regression?

[RalfJung]

Possibly in 1.52.1 Miri happened to build on stable even outside a git repo? That would have been a happy accident, we never tested this.

This commit landed with #83922 which first shipped in 1.53. Maybe older versions of vergen failed more gracefully when there is no git repo.

[mati865]

FWIW Miri was present in version 1.43 http://web.archive.org/web/20200930154630/https://packages.msys2.org/package/mingw-w64-x86_64-rust
Probably since it's introduction but I don't have artifacts from back then.