Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to build 1.53.0 from official tarball due to miri #86436

Closed
liushuyu opened this issue Jun 18, 2021 · 25 comments · Fixed by #87282
Closed

Unable to build 1.53.0 from official tarball due to miri #86436

liushuyu opened this issue Jun 18, 2021 · 25 comments · Fixed by #87282
Labels
A-miri Area: The miri tool C-bug Category: This is a bug. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Comments

@liushuyu
Copy link
Contributor

Hi there, I tried to build Rust 1.53.0 from the source tarball downloaded from https://static.rust-lang.org/dist/rustc-1.53.0-src.tar.gz and it failed to compile due to cargo-miri could not locate the Git repository.

After a quick search, the issue supposedly should have been fixed in rust-lang/miri@e5b3400 but for some reason, the fix didn't land in the final distribution despite in #84182 it was stated that the fix would be included in the next stable release.

Here is the backtrace:

Backtrace

  thread 'main' panicked at 'Unable to generate vergen keys!: could not find repository from '/build/i0szw9jw/rustc-1.53.0-src/src/tools/miri/cargo-miri'; class=Repository (6); code=NotFound (-3)

  Stack backtrace:
     0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
               at /build/i0szw9jw/rustc-1.53.0-src/vendor/anyhow-1.0.34/src/error.rs:464:25
     1: vergen::feature::git::configure_git
               at /build/i0szw9jw/rustc-1.53.0-src/vendor/vergen/src/feature/git.rs:175:55
     2: vergen::config::Instructions::config
               at /build/i0szw9jw/rustc-1.53.0-src/vendor/vergen/src/config.rs:139:9
     3: vergen::gen::config_from_instructions
               at /build/i0szw9jw/rustc-1.53.0-src/vendor/vergen/src/gen.rs:81:32
     4: vergen::gen::vergen
               at /build/i0szw9jw/rustc-1.53.0-src/vendor/vergen/src/gen.rs:69:5
     5: build_script_build::main
               at ./build.rs:10:5
     6: core::ops::function::FnOnce::call_once
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/ops/function.rs:227:5
     7: std::sys_common::backtrace::__rust_begin_short_backtrace
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/sys_common/backtrace.rs:125:18
     8: std::rt::lang_start::{{closure}}
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/rt.rs:49:18
     9: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/ops/function.rs:259:13
        std::panicking::try::do_call
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/panicking.rs:379:40
        std::panicking::try
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/panicking.rs:343:19
        std::panic::catch_unwind
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/panic.rs:431:14
        std::rt::lang_start_internal
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/rt.rs:34:21
    10: std::rt::lang_start
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/rt.rs:48:5
    11: main
    12: __libc_start_main
               at /build/okrh64bn/glibc/csu/../csu/libc-start.c:314:16
    13: _start
               at /build/okrh64bn/glibc/csu/../sysdeps/x86_64/start.S:120', src/tools/miri/cargo-miri/build.rs:10:24
  stack backtrace:
     0: rust_begin_unwind
               at /build/i0szw9jw/rustc-1.53.0-src/library/std/src/panicking.rs:493:5
     1: core::panicking::panic_fmt
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/panicking.rs:92:14
     2: core::result::unwrap_failed
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/result.rs:1355:5
     3: core::result::Result<T,E>::expect
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/result.rs:997:23
     4: build_script_build::main
               at ./build.rs:10:5
     5: core::ops::function::FnOnce::call_once
               at /build/i0szw9jw/rustc-1.53.0-src/library/core/src/ops/function.rs:227:5
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
command did not execute successfully: "/build/i0szw9jw/rustc-1.53.0-src/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "32" "-v" "--release" "--frozen" "--manifest-path" "/build/i0szw9jw/rustc-1.53.0-src/src/tools/miri/cargo-miri/Cargo.toml" "--message-format" "json-render-diagnostics"
expected success, got: exit code: 101

@liushuyu liushuyu added the C-bug Category: This is a bug. label Jun 18, 2021
@jonas-schievink jonas-schievink added A-miri Area: The miri tool T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jun 18, 2021
@Urgau
Copy link
Member

Urgau commented Jun 18, 2021

cc #86286

wip-sync pushed a commit to NetBSD/pkgsrc-wip that referenced this issue Jun 22, 2021
@RalfJung
Copy link
Member

RalfJung commented Jul 3, 2021

Possibly the fix landed after 1.53 was branched?

Anyway, the stable release tarballs do not contain Miri any more (they were never intended to), so I'd say this is fixed by #86568.

@RalfJung RalfJung closed this as completed Jul 3, 2021
@RalfJung
Copy link
Member

RalfJung commented Jul 3, 2021

Yeah, the git fix landed with #85402 which reached stable with 1.54.0.

@RalfJung
Copy link
Member

RalfJung commented Jul 3, 2021

I should also mention that Miri is simply not supported on stable. I don't know how exactly you are building rustc from that tarball, but you might have enabled too many "extra tools"?

@mati865
Copy link
Contributor

mati865 commented Jul 3, 2021

Just have hit this error with 1.53 tarball with extended = true but tools commented-out.
Turns out when you do ./x.py install it will try to install miri and cargo-miri as well (remember we are using stable tarball).
This fails when building not in a git repository. You can see log from MSYS2 CI when it's building inside git repository: https://github.com/msys2/msys2-autobuild/runs/2945722561?check_suite_focus=true

You can see miri is present in the end:

$ ls pkg/mingw-w64-clang-x86_64-rust/clang64/bin/
cargo-clippy.exe  cargo.exe          rls.exe            rust-gdbgui  rustc_driver-02356bf24940476b.dll  std-40fc5b9b2ad404cb.dll
cargo-fmt.exe     clippy-driver.exe  rust-analyzer.exe  rust-lldb    rustdoc.exe                        test-38b9f97bb94ce93c.dll
cargo-miri.exe    miri.exe           rust-gdb           rustc.exe    rustfmt.exe

I believe this bug should be reopened and retitled because x.py install should not build miri from such tarballs.

@RalfJung
Copy link
Member

RalfJung commented Jul 3, 2021

The bug will, I think, be fixed in the next release since Miri is simply not part of the tarball any more.

I think the bug you are asking is already open: #74709

@mati865
Copy link
Contributor

mati865 commented Jul 3, 2021

I have just found #86568 which should solve the issue on nightly but ideally it'd be backported to beta and stable.

@RalfJung
Copy link
Member

RalfJung commented Jul 3, 2021

Backporting to stable makes little sense (unless there's a point release, which I wouldn't expect).

Backporting to beta already happened in #86413.

@paulmenzel
Copy link

Reading the comments, I missed a solution for building Rust 1.53.0 with extended = true. Building Rust 1.50.0 worked with:

python3 ./x.py build --exclude src/tools/miri

but with 1.53.0 the --exclude switch seems to be ignored. How can Rust 1.53.0 be built including Cargo?

@mati865
Copy link
Contributor

mati865 commented Jul 15, 2021

How can Rust 1.53.0 be built including Cargo?

One workaround is to create and initialize git repository in the directory where you want to build Rust (preferably before unpacking Rust to save time).

@RalfJung
Copy link
Member

@paulmenzel I haven't tried it but shouldn't setting the tools work?

# Installs chosen set of extended tools if `extended = true`. By default builds
# all extended tools except `rust-demangler`, unless the target is also being
# built with `profiler = true`. If chosen tool failed to build the installation
# fails. If `extended = false`, this option is ignored.
#tools = ["cargo", "rls", "clippy", "rustfmt", "analysis", "src"] # + "rust-demangler" if `profiler`

So if you just want cargo, I'd try tools = ["cargo"].

@paulmenzel
Copy link

paulmenzel commented Jul 15, 2021

@paulmenzel I haven't tried it but shouldn't setting the tools work?

# Installs chosen set of extended tools if `extended = true`. By default builds
# all extended tools except `rust-demangler`, unless the target is also being
# built with `profiler = true`. If chosen tool failed to build the installation
# fails. If `extended = false`, this option is ignored.
#tools = ["cargo", "rls", "clippy", "rustfmt", "analysis", "src"] # + "rust-demangler" if `profiler`

So if you just want cargo, I'd try tools = ["cargo"].

Thank you. Sorry, I didn’t mention it. I had tried it, and it did not work.

@RalfJung
Copy link
Member

@Mark-Simulacrum is that expected that Miri is built even when tools is explicitly set to a list that does not include Miri?

@RalfJung RalfJung reopened this Jul 15, 2021
@Mark-Simulacrum
Copy link
Member

In the sense that I'm not surprised, yes -- tools is pretty poorly supported around rustbuild, it's not something I have in cache to think about when we introduce new code so its easy to miss a dependency there.

It should likely get fixed to skip building miri when tools doesn't include it though.

@paulmenzel
Copy link

It should likely get fixed to skip building miri when tools doesn't include it though.

Sorry, I do not understand. It’d be great if you rephrased it.

@paulmenzel
Copy link

Can I patch the source some how (delete the miri files?) to fix the build?

@Mark-Simulacrum
Copy link
Member

I'm not sure why the exclude switch is being ignored; you'll need to provide a full verbose log to try and dig in on what specifically is causing miri to be invoked despite the exclude.

FWIW we don't generally expect people to use x.py build in dist tarballs -- x.py dist/install are more likely to work.

@paulmenzel
Copy link

paulmenzel commented Jul 22, 2021

I'm not sure why the exclude switch is being ignored; you'll need to provide a full verbose log to try and dig in on what specifically is causing miri to be invoked despite the exclude.

How can I increase the verbosity?

FWIW we don't generally expect people to use x.py build in dist tarballs -- x.py dist/install are more likely to work.

src/bootstrap/README.md does not mention that.

@Mark-Simulacrum
Copy link
Member

x.py build -vvv, for example.

I'd be happy to accept a PR (though not sure that's quite the place to update, seems OK though) that adds some text about preferring x.py dist/install when using the tarballs.

@paulmenzel
Copy link

Log from /pkg/rustc-1.53.0-0/build/rustc-1.53.0-src/build/bootstrap/debug/bootstrap -vvv dist --exclude src/tools/miri:
rustc-miri.log

(Sorry, the screen buffer was too small, but hopefully the messages still help.)

@Mark-Simulacrum
Copy link
Member

Looks like this will be fixed by #87282, it's getting invoked through extended = true (which directly runs the dist step for miri, ignoring --exclude). I have not reviewed that PR yet, but expect to do so soon; going to add this issue to the fixes list on it though.

@Mark-Simulacrum
Copy link
Member

I think you can fix this on 1.53 by patching the source here https://github.com/rust-lang/rust/blob/1.53.0/src/bootstrap/dist.rs#L1176 to return None from that step. You may need to set deny-warnings = false or fix some other compiler warnings in order to build with that function stubbed out like that.

@paulmenzel
Copy link

As the Rust 1.52.1 release builds and installs just fine, did anybody find the commit causing the regression?

@RalfJung
Copy link
Member

RalfJung commented Jul 23, 2021

Possibly in 1.52.1 Miri happened to build on stable even outside a git repo? That would have been a happy accident, we never tested this.

This commit landed with #83922 which first shipped in 1.53. Maybe older versions of vergen failed more gracefully when there is no git repo.

@mati865
Copy link
Contributor

mati865 commented Jul 23, 2021

FWIW Miri was present in version 1.43 http://web.archive.org/web/20200930154630/https://packages.msys2.org/package/mingw-w64-x86_64-rust
Probably since it's introduction but I don't have artifacts from back then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-miri Area: The miri tool C-bug Category: This is a bug. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants