rust-lld doesn't work on Linux unless executed with rustup wrapper

[alexcrichton]

I noticed that some CI builds started breaking over winter break with the 1.49.0 release of rustc. It looks like this is the behavior I'm running into:

$ echo 'fn main() {}' > foo.rs
$ rustc foo.rs --target wasm32-unknown-unknown
$ `rustup which rustc` foo.rs --target wasm32-unknown-unknown
error: linking with `rust-lld` failed: exit code: 127
  |
  = note: "rust-lld" "-flavor" "wasm" "--rsp-quoting=posix" "-z" "stack-size=1048576" "--stack-first" "--allow-undefined" "--fatal-warnings" "--no-demangle" "--export-dynamic" "--no-entry" "-L" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib" "-L" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/self-contained" "foo.foo.7rcbfp3g-cgu.0.rcgu.o" "foo.foo.7rcbfp3g-cgu.1.rcgu.o" "foo.foo.7rcbfp3g-cgu.2.rcgu.o" "foo.foo.7rcbfp3g-cgu.3.rcgu.o" "foo.foo.7rcbfp3g-cgu.4.rcgu.o" "foo.foo.7rcbfp3g-cgu.5.rcgu.o" "-o" "foo.wasm" "--export" "main" "--export=__heap_base" "--export=__data_end" "foo.belfx9afw9cmv8.rcgu.o" "--gc-sections" "-O0" "-L" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/libstd-57520f873e4386ea.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/libpanic_abort-2bb76cd91836cd74.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/libdlmalloc-2362cd2b752fe8cf.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/librustc_demangle-4e35388d2cf809d9.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/libhashbrown-7528bb1d1a92d729.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/librustc_std_workspace_alloc-46e1c6d1138ee9b3.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/libunwind-ffe09f2236ae74f5.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/libcfg_if-f26ee661ae55669e.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/liblibc-0476900897b6039b.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/liballoc-b0c04b53f6ac3156.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/librustc_std_workspace_core-50344da360c5e489.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/libcore-0b67ff3e00a8bf99.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib/libcompiler_builtins-5ca68120625f03c4.rlib"
  = note: rust-lld: error while loading shared libraries: libLLVM-11-rust-1.49.0-stable.so: cannot open shared object file: No such file or directory


error: aborting due to previous error

This appears to work with 1.48.0. It looks like in Rust 1.48.0 the rust-lld executable had libLLVM statically linked and now it has it dynamically linked. Unless the rustup wrapper is executed though (which munges LD_LIBRARY_PATH I believe) the executable no longer works (e.g. if you execute the rustc binary directly).

I think this is a case where the -rpath business may need to get involved? Ideally rust-lld would know that its libraries, when installed, are a few layers up.

[JohnTitor]

Assigning P-high as discussed as part of the Prioritization Working Group procedure and removing I-prioritize.

[bkaestner]

This issue was probably introduced between 9310e3b and 215f2d3. Unfortunately, I lack the experience to bisect Rust's source further, or rather I encounter unrelated issues. Hopefully, that range can help someone else.

For completeness (and to make sure that I actually did the correct thing), here's how I found the range: with issue-80703.sh

#!/bin/bash
echo 'fn main() {}' > rust.rs

rustc                      foo.rs --target wasm32-unknown-unknown
`rustup ${TC} which rustc` foo.rs --target wasm32-unknown-unknown

and RUST_SRC_REPO set to a checked out git repository, the command

cargo-bisect-rustc -v --preserve --target wasm32-unknown-unknown --start 2020-04-08 --end 2020-12-31 --script `pwd`/issue-80703.sh

yielded

<snip individual tests>

searched toolchains nightly-2020-04-08 through nightly-2020-12-31

********************************************************************************
Regression in nightly-2020-05-23
********************************************************************************

fetching https://static.rust-lang.org/dist/2020-05-22/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2020-05-22: 40 B / 40 B [=================] 100.00 % 8.71 KB/s converted 2020-05-22 to 9310e3bd4f425f84fc27878ebf2bda1f30935a63
fetching https://static.rust-lang.org/dist/2020-05-23/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2020-05-23: 40 B / 40 B [==============] 100.00 % 1011.98 KB/s converted 2020-05-23 to 215f2d3294b08dbdcf8f7d40de21ef1e7eae0a2d
looking for regression commit between 2020-05-22 and 2020-05-23
opening existing repository at "../repos/rust/"
refreshing repository
fetching (via local git) commits from 9310e3bd4f425f84fc27878ebf2bda1f30935a63 to 215f2d3294b08dbdcf8f7d40de21ef1e7eae0a2d
opening existing repository at "../repos/rust/"
refreshing repository
looking up first commit
looking up second commit
checking that commits are by bors and thus have ci artifacts...
finding bors merge commits
found 7 bors merge commits in the specified range
  commit[0] 2020-05-21UTC: Auto merge of #71930 - Nadrieril:exhaustiveness-remove-tyerr, r=varkor
  commit[1] 2020-05-21UTC: Auto merge of #72433 - RalfJung:rollup-srft8nx, r=RalfJung
  commit[2] 2020-05-22UTC: Auto merge of #71956 - ecstatic-morse:remove-requires-storage-analysis, r=tmandry
  commit[3] 2020-05-22UTC: Auto merge of #72000 - cuviper:dist-llvm, r=Mark-Simulacrum
  commit[4] 2020-05-22UTC: Auto merge of #72458 - RalfJung:rollup-g1w1vws, r=RalfJung
  commit[5] 2020-05-22UTC: Auto merge of #72460 - RalfJung:rollup-28fs06y, r=RalfJung
  commit[6] 2020-05-22UTC: Auto merge of #72464 - RalfJung:rollup-xhm7w7u, r=RalfJung
ERROR: no commits between 9310e3bd4f425f84fc27878ebf2bda1f30935a63 and 215f2d3294b08dbdcf8f7d40de21ef1e7eae0a2d within last 167 days

The --start date was chosen by trail-and-error, to be honest. I expected the regression not to be in 1.48's original nightly and thus had --start 2020-08-08 at first, and then just subtracted another four months. The affected 2020-05-23 was fortunately in that second range already.

Hope that helps!

[pnkfelix]

visited for P-high review. assigned to @lqd as the rust-lld expert/stakeholder that I'm familiar with. :)

[lqd]

In case anyone is blocked by this issue until it's fixed, there's also this workaround: rustup component add llvm-tools on your toolchain of choice, and the OP will work.

[lqd]

Until the PR lands on nightly, I've verified this issue as fixed on master commit 101fa903bb9209d270086da279247625a2869211.