Segfault compiling libc on armv7-unknown-linux-gnueabihf

[fenhl]

Originally reported as rust-lang/libc#1441 but I was asked to report this here.

Steps to reproduce

cargo new --bin compile-fail
cd compile-fail
rustup override set nightly
echo 'libc = "=0.2.60"' >> Cargo.toml
cargo check --release

Output:

     Created binary (application) `compile-fail` package
info: using existing install for 'nightly-armv7-unknown-linux-gnueabihf'
info: override toolchain for '/home/fenhl/compile-fail' set to 'nightly-armv7-unknown-linux-gnueabihf'

  nightly-armv7-unknown-linux-gnueabihf unchanged - rustc 1.38.0-nightly (273f42b59 2019-07-21)

    Updating crates.io index
   Compiling libc v0.2.60
error: Could not compile `libc`.

Caused by:
  process didn't exit successfully: `rustc --crate-name build_script_build /home/fenhl/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.60/build.rs --color always --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=aaad41c440293f27 -C extra-filename=-aaad41c440293f27 --out-dir /home/fenhl/compile-fail/target/release/build/libc-aaad41c440293f27 -L dependency=/home/fenhl/compile-fail/target/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)

Workaround

Run the cargo check command with RUSTFLAGS='-Ccodegen-units=1' or add the following to the Cargo.toml:

[profile.release]
codegen-units = 1

[gnzlbg]

@jonas-schievink this is also a regression with respect to stable Rust, so we probably need to avoid shipping a beta with this issue.

[mati865]

Can you obtain backtrace using gdb or lldb?

[jonas-schievink]

cc @nikic, this might also be fallout from the LLVM update

[fenhl]

@mati865 is there a guide I can use for this?

[mati865]

@fenhl

$ gdb --args cargo check --release

r (enter)

bt (enter)

q (enter)

and paste the output.

[fenhl]

GNU gdb (Raspbian 8.2.1-2) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from cargo...done.
(gdb) r
Starting program: /home/fenhl/.cargo/bin/cargo check --release
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
process 16341 is executing new program: /home/fenhl/.rustup/toolchains/nightly-armv7-unknown-linux-gnueabihf/bin/cargo
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[New Thread 0x76d512f0 (LWP 16344)]
   Compiling libc v0.2.60
[New Thread 0x769fe2f0 (LWP 16345)]
[Detaching after fork from child process 16346]                          ] 0/4: libc(build.rs)                                                                                              
[Thread 0x769fe2f0 (LWP 16345) exited]

Thread 2 "cargo" received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x76d512f0 (LWP 16344)]
futex_wait_cancelable (private=0, expected=0, futex_word=0xe77938) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
(gdb) bt
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xe77938) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x0, cond=0xe77910) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xe77910, mutex=0x0) at pthread_cond_wait.c:655
#3  0x00bd0900 in std::sys::unix::condvar::Condvar::wait () at src/libstd/sys/unix/condvar.rs:69
#4  std::sys_common::condvar::Condvar::wait () at src/libstd/sys_common/condvar.rs:41
#5  std::sync::condvar::Condvar::wait () at src/libstd/sync/condvar.rs:204
#6  std::thread::park () at src/libstd/thread/mod.rs:911
#7  0x00bde4b4 in std::sync::mpsc::blocking::WaitToken::wait () at src/libstd/sync/mpsc/blocking.rs:71
#8  0x00baf508 in std::sync::mpsc::oneshot::Packet<T>::recv ()
#9  0x00baddb8 in std::sync::mpsc::Receiver<T>::recv ()
#10 0x00bb2d38 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#11 0x00bb1b44 in std::panicking::try::do_call ()
#12 0x00bebf88 in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:82
#13 0x00bb2110 in core::ops::function::FnOnce::call_once{{vtable-shim}} ()
#14 0x00bcfb74 in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/a7f28678bbf4e16893bb6a718e427504167a9494/src/liballoc/boxed.rs:766
#15 0x00beb48c in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/a7f28678bbf4e16893bb6a718e427504167a9494/src/liballoc/boxed.rs:766
#16 std::sys_common::thread::start_thread () at src/libstd/sys_common/thread.rs:13
#17 std::sys::unix::thread::Thread::new::thread_start () at src/libstd/sys/unix/thread.rs:79
#18 0x76f56494 in start_thread (arg=0x76d512f0) at pthread_create.c:486
Backtrace stopped: Cannot access memory at address 0x1ec9a
(gdb) q
A debugging session is active.

	Inferior 1 [process 16341] will be killed.

Quit anyway? (y or n) y

[gnzlbg]

cc @nikic - suspicious that this was reported right after the llvm 9 upgrade

[mati865]

@fenhl gdb haven't detected failed thread. Could you try again with this?

$ gdb --args cargo check --release

r (enter)

set pagination off (enter)

thread apply all bt (enter)

[fenhl]

Thread 2 (Thread 0x76d512f0 (LWP 19622)):
#0  __GI___tls_get_addr (ti=0xdc38b4) at dl-tls.c:828
#1  0x00bd0a64 in core::option::Option<T>::as_ref () at /rustc/a7f28678bbf4e16893bb6a718e427504167a9494/src/libcore/option.rs:266
#2  std::thread::local::lazy::LazyKeyInner<T>::get () at src/libstd/thread/local.rs:279
#3  std::thread::local::fast::Key<T>::get () at src/libstd/thread/local.rs:409
#4  std::panicking::update_panic_count::PANIC_COUNT::__getit () at src/libstd/thread/local.rs:176
#5  std::thread::local::LocalKey<T>::try_with () at src/libstd/thread/local.rs:254
#6  std::thread::local::LocalKey<T>::with () at src/libstd/thread/local.rs:234
#7  std::panicking::update_panic_count () at src/libstd/panicking.rs:226
#8  std::panicking::panicking () at src/libstd/panicking.rs:303
#9  std::thread::panicking () at src/libstd/thread/mod.rs:724
#10 std::sys_common::poison::Flag::done () at src/libstd/sys_common/poison.rs:36
#11 <std::sync::mutex::MutexGuard<T> as core::ops::drop::Drop>::drop () at src/libstd/sync/mutex.rs:448
#12 core::ptr::real_drop_in_place () at /rustc/a7f28678bbf4e16893bb6a718e427504167a9494/src/libcore/ptr/mod.rs:175
#13 std::thread::park () at src/libstd/thread/mod.rs:917
#14 0x00bde4b4 in std::sync::mpsc::blocking::WaitToken::wait () at src/libstd/sync/mpsc/blocking.rs:71
#15 0x00baf508 in std::sync::mpsc::oneshot::Packet<T>::recv ()
#16 0x00baddb8 in std::sync::mpsc::Receiver<T>::recv ()
#17 0x00bb2d38 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#18 0x00bb1b44 in std::panicking::try::do_call ()
#19 0x00bebf88 in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:82
#20 0x00bb2110 in core::ops::function::FnOnce::call_once{{vtable-shim}} ()
#21 0x00bcfb74 in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/a7f28678bbf4e16893bb6a718e427504167a9494/src/liballoc/boxed.rs:766
#22 0x00beb48c in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/a7f28678bbf4e16893bb6a718e427504167a9494/src/liballoc/boxed.rs:766
#23 std::sys_common::thread::start_thread () at src/libstd/sys_common/thread.rs:13
#24 std::sys::unix::thread::Thread::new::thread_start () at src/libstd/sys/unix/thread.rs:79
#25 0x76f56494 in start_thread (arg=0x76d512f0) at pthread_create.c:486
Backtrace stopped: Cannot access memory at address 0x21046

Thread 1 (Thread 0x76ff4010 (LWP 19619)):
#0  futex_reltimed_wait_cancelable (private=0, reltime=0x7effb8a0, expected=0, futex_word=0xdcd3b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0x7effb910, mutex=0x2, cond=0xdcd390) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xdcd390, mutex=0x2, abstime=0x7effb910) at pthread_cond_wait.c:667
#3  0x00be44bc in std::sys::unix::condvar::Condvar::wait_timeout () at src/libstd/sys/unix/condvar.rs:100
#4  0x00bd0df8 in std::sys_common::condvar::Condvar::wait_timeout () at src/libstd/sys_common/condvar.rs:51
#5  std::sync::condvar::Condvar::wait_timeout () at src/libstd/sync/condvar.rs:405
#6  std::thread::park_timeout () at src/libstd/thread/mod.rs:1006
#7  0x00bde5f4 in std::sync::mpsc::blocking::WaitToken::wait_max_until () at src/libstd/sync/mpsc/blocking.rs:82
#8  0x00baf4d8 in std::sync::mpsc::oneshot::Packet<T>::recv ()
#9  0x00badb40 in std::sync::mpsc::Receiver<T>::recv_timeout ()
#10 0x00bad3d0 in jobserver::imp::Helper::join ()
#11 0x00bb0d70 in <jobserver::HelperThread as core::ops::drop::Drop>::drop ()
#12 0x006f5218 in core::ptr::real_drop_in_place ()
#13 0x00719b20 in cargo::core::compiler::job_queue::JobQueue::execute ()
#14 0x007d9050 in cargo::core::compiler::context::Context::compile ()
#15 0x00550a20 in cargo::ops::cargo_compile::compile_ws ()
#16 0x0054f878 in cargo::ops::cargo_compile::compile ()
#17 0x00490a44 in cargo::commands::check::exec ()
#18 0x0045a184 in cargo::cli::main ()
#19 0x0049b270 in cargo::main ()
#20 0x004787cc in std::rt::lang_start::{{closure}} ()
#21 0x00be2350 in std::rt::lang_start_internal::{{closure}} () at src/libstd/rt.rs:49
#22 std::panicking::try::do_call () at src/libstd/panicking.rs:296
#23 0x00bebf88 in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:82
#24 0x00be2e7c in std::panicking::try () at src/libstd/panicking.rs:275
#25 std::panic::catch_unwind () at src/libstd/panic.rs:394
#26 std::rt::lang_start_internal () at src/libstd/rt.rs:48
#27 0x0049cff8 in main ()

[nikomatsakis]

Report from compiler-team pre-triage:

It seems like a good next step would be to bisect the regression to a specific nightly or (better) PR. The cargo-bisect-rustc tool may be of use here!

[fenhl]

I'm going to need some help with that because it just said “regression in nightly-2019-07-25” which is obviously wrong. Doing some further testing, it seems the segfault doesn't occur every time cargo check --release is run:

$ while command sleep 5; do cargo clean; if cargo check --release |& grep SIGSEGV; then echo segfault; else echo no segfault; fi; done
segfault
no segfault
segfault
segfault
segfault
segfault
segfault
segfault
segfault
segfault
segfault
no segfault

[gnzlbg]

@fenhl can you manually test if the nightlies before the one from 2019-07-21 also have this issue ? Nailing the first nightly with this problem would be very helpful.

[fenhl]

Looks like the issue first appears on nightly-2019-07-18 (bc2e84c 2019-07-17). On nightly-2019-07-17 (07e0c36 2019-07-16), the command ran successfully 100 times.

[mati865]

So we have the winner: #62592

[gnzlbg]

The most suspicious winner. We should give a “bravest dev of the quarter” award to whoever updates the LLVM version. Alex would often get this award, but this time it would be @nikic and i’d Bet Alex would love to not have to do this kind of thing and instead only work on the wasm stuff that interest them. Or maybe we should have an LLVM working group whose job is to regularly keep rustc following the latest LLVM trunk so that bugs in LLVM upstream are discovered quickly as they happen. Idk.

[nagisa]

Typically SIGSEGV indicates an LLVM assertion firing. Would be great if a reproduction was attempted with rustc that has LLVM assertions enabled.

Furthermore the backtraces posted above are for cargo, not for the rustc process it spawns. Would be nice to see a backtrace from rustc with LLVM assertions enabled.

[nikic]

@nagisa The stack trace in #62896 (comment) indicates a segfault during __GI___tls_get_addr, so I'd assume this is a TLS miscompile.

[fenhl]

@nagisa I don't really know how to use plain rustc, but I'd be happy to help if instructions are provided.

[nagisa]

@nikic from what I can tell that backtrace is taken after the process broke on a signal other than SIGSEGV.

[msizanoen1]

@stefson The LLVM fork of rustc at https://github.com/rust-lang/llvm-project. The same as the one used in CI.

[msizanoen1]

Confirmed the PR fixes the issue:

sirius@localhost:~$ rustup-toolchain-install-master e66a6282275802fcb0a29ba58ddc445fc64ac8ef
detecting the channel of the `e66a6282275802fcb0a29ba58ddc445fc64ac8ef` toolchain...
downloading <https://rust-lang-ci2.s3-us-west-1.amazonaws.com/rustc-builds/e66a6282275802fcb0a29ba58ddc445fc64ac8ef/rustc-nightly-armv7-unknown-linux-gnueabihf.tar.xz>...
completed                                         
downloading <https://rust-lang-ci2.s3-us-west-1.amazonaws.com/rustc-builds/e66a6282275802fcb0a29ba58ddc445fc64ac8ef/rust-std-nightly-armv7-unknown-linux-gnueabihf.tar.xz>...
completed
toolchain `e66a6282275802fcb0a29ba58ddc445fc64ac8ef` is successfully installed!
sirius@localhost:~$ rustc +e66a6282275802fcb0a29ba58ddc445fc64ac8ef -C opt-level=3 hello.rs
sirius@localhost:~$ ./hello
Hello, world
sirius@localhost:~$

[Lokathor]

are the side tools like clippy and rustfmt likely to also build in the next nightly now?

[msizanoen1]

It is another issue.

[repnop]

Just had this pop up for me on both stable (1.40.0) and nightly (1.42.0 f6449ba23 2020-01-21) with my Raspberry Pi Zero W (arm-unknown-linux-gnueabihf) and was able to reproduce with the steps described. setting codegen-units = 1 fixes the problem

[thebigredgeek]

Also running into this on Raspberry Pi 4 via Balena.io builds. Can confirm that setting codegen-units = 1 fixes the problem there

[faern]

I can also still reproduce this on latest stable (1.43.1) with the same SIGSEGV. codegen-units=1 indeed does not trigger the bug.

[haslersn]

I just reproduced this with

rustc 1.43.1 (8d69840ab 2020-05-04)
cargo 1.43.0 (2cbe9048e 2020-05-03)

[cztomsik]

@pnkfelix I think this should be either reopened or codegen-units should be set to 1 by default

[stefson]

I tried to reproduce this with rustc-1.44.1 on armv7-unknown-linux-gnueabihf, and it did not fail, neither does it with rustc-1.41.1

I used reproducer from the initial posting. If you need people to confirm the breakage, please offer a reproducer

[sabirmgd]

I'm still getting this on x86_64-unknown-linux-musl.

It particularly fails when the code tries to create an http client of any sort, for example (s3 rusoto client, elasticsearch client)

I created a small project to actually re-produce this issue:
I'm using docker on alpine, here is my environment:

FROM alpine:3.13.4

# <channel>[-<date>][-<host>]
# <channel>       = stable|beta|nightly|<major.minor>|<major.minor.patch>
# <date>          = YYYY-MM-DD
# <host>          = <target-triple>
ARG TOOLCHAIN_CHANNEL=nightly
ARG TOOLCHAIN_DATE=2021-03-25
ARG TOOLCHAIN_HOST=x86_64-unknown-linux-musl

ENV RUSTUP_HOME=/usr/local/rustup \
    CARGO_HOME=/usr/local/cargo \
    PATH=/usr/local/cargo/bin:$PATH

RUN apk upgrade; \
    apk update; \ 
    apk add util-linux pciutils usbutils coreutils binutils findutils grep \
            curl bash openssl libressl-dev gcc build-base ca-certificates; \
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y -v \
      --no-modify-path \
      --default-toolchain ${TOOLCHAIN_CHANNEL}-${TOOLCHAIN_DATE}-${TOOLCHAIN_HOST} \
      --profile minimal; \
    chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
    rustup --version; \
    cargo --version; \
    rustc --version; \
    rustup component add llvm-tools-preview; \
    rustup component add clippy; \
    cargo install grcov --vers 0.6.1;

as you can see, I locked it down to a specific nightly version ( but you can feel free to change it to any, I tried many versions.

main.rs:

static DEFAULT_ELASTIC_SEARCH_ADDRESS: &str =
    "https://myaddress.com";

use elasticsearch::http::transport::SingleNodeConnectionPool;
use elasticsearch::http::transport::TransportBuilder;
use elasticsearch::http::Url;
use elasticsearch::Error;
use elasticsearch::{auth::Credentials, Elasticsearch};

fn main() {
    let _client = create_client();
}


#[allow(dead_code)]
pub fn create_client() -> Result<Elasticsearch, Error> {
    let username = "user".to_string();
    let password = "password".to_string();
    let credentials = Credentials::Basic(username, password);
    let endpoint = DEFAULT_ELASTIC_SEARCH_ADDRESS.to_string();
    let url = Url::parse(&endpoint)?;
    let conn_pool = SingleNodeConnectionPool::new(url);
    let transport = TransportBuilder::new(conn_pool).auth(credentials).build()?;
    Ok(Elasticsearch::new(transport))
}


#[test]
fn query_studies_within_files() {
    let _client = create_client();
}

and the cargo.toml file

[package]
name = "test_http"
version = "0.1.0"
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
dotenv = "0.15.0"
elasticsearch = "7.12.0-alpha.1"
rusoto_s3 = "0.46.0"
rusoto_core = "0.46.0"
rusoto_credential = "0.46.0"
serde = { version = "1.0.125", features = ["derive"] }
serde_yaml = "0.8.17"
serde_json = "1.0.64"
json = "0.12"

I'm not really sure what could possibly be the reason is, it something to do with https because if I remove the https from the url static DEFAULT_ELASTIC_SEARCH_ADDRESS: &str = "https://myaddress.com"; to be myaddress.com, the test succeeds. however, of course, it would not get a proper client.

[bjorn3]

@sabirmgd This is unlikely to have the same cause given that this issue is an architecture specific miscompilation and you are on a different architecture. Can you open a new issue?