Line numbers in backtrace still do not appear on macOS for Cargo projects

[kennytm]

Repro steps:

1. Create a cargo project.

   cargo new --bin a && cd a && echo 'fn main() { panic!(); }' > src/main.rs 

2. Run it. Note that the line numbers are missing (unexpected).

   RUST_BACKTRACE=1 cargo run

   Output

   thread 'main' panicked at 'explicit panic', src/main.rs:1:12
   stack backtrace:
      0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
      1: std::sys_common::backtrace::_print
      2: std::panicking::default_hook::{{closure}}
      3: std::panicking::default_hook
      4: std::panicking::rust_panic_with_hook
      5: std::panicking::begin_panic
      6: a::main
      7: __rust_maybe_catch_panic
      8: std::rt::lang_start
      9: main
   

3. Run the actual hard-linked program. The line numbers now appear (expected).

   RUST_BACKTRACE=1 target/debug/deps/a-*

   Output

   thread 'main' panicked at 'explicit panic', src/main.rs:1:12
   stack backtrace:
      0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
                at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
      1: std::sys_common::backtrace::_print
                at src/libstd/sys_common/backtrace.rs:71
      2: std::panicking::default_hook::{{closure}}
                at src/libstd/sys_common/backtrace.rs:60
                at src/libstd/panicking.rs:381
      3: std::panicking::default_hook
                at src/libstd/panicking.rs:397
      4: std::panicking::begin_panic
                at src/libstd/panicking.rs:577
      5: std::panicking::begin_panic
                at /Users/travis/build/rust-lang/rust/src/libstd/panicking.rs:538
      6: a::main
                at src/main.rs:1
      7: panic_unwind::dwarf::eh::read_encoded_pointer
                at src/libpanic_unwind/lib.rs:99
      8: rust_panic
                at src/libstd/panicking.rs:459
                at src/libstd/panic.rs:361
                at src/libstd/rt.rs:59
      9: a::main
   

The problem is that in rust-lang/rust#44251 we did not search for the deps/ folder. The fix is either,

1. Symlink target/debug/deps/a-«hash».dSYM to target/debug/a-«hash».dSYM (actual name irrelevant), don't touch libstd. This may also fix No deterministic/easily findable name for debug symbol directory on macOS #4056; or
2. Amend Add libbacktrace support for Apple platforms (resubmitted) rust#44251 to look for deps/ as well, don't touch cargo.

cc @JohnColanduoni

---

Meta

$ cargo -vV
cargo 0.23.0-nightly (33250c48b 2017-09-09)
release: 0.23.0
commit-hash: 33250c48b4763b01478d780e76206484a1d5b207
commit-date: 2017-09-09

$ rustc -vV
rustc 1.22.0-nightly (539f2083d 2017-09-13)
binary: rustc
commit-hash: 539f2083de809b5c8304fe7426655cfeb0e66d5e
commit-date: 2017-09-13
host: x86_64-apple-darwin
release: 1.22.0-nightly
LLVM version: 4.0

[alexcrichton]

I think Cargo can definitely manage lifting this directory up a level, but currently nothing in Rust knows about it (neither Cargo nor rustc itself). This is the same way for a number of debug files on Windows where rustc/cargo only know about a subset of the artifacts, not everything together. I'm under the impression, though, that there's an API that's typically used on OSX to find this information and just doesn't work when spotlight is turned off (who knew?) and maybe that should be called?

[JohnColanduoni]

@alexcrichton

There is such an API (which lldb and gdb use) in the DebugSymbols framework but like CoreSymbolication it's a private API. If we're willing to lean on that I think it might be better to just use CoreSymbolication from start to finish (weakly linked or via dlsym).

[kennytm]

The risk of using private APIs are:

1. It is unstable. But AFAIK the CoreSymbolication API don't change a lot. Not sure about DebugSymbols.
2. Depending on how smart Apple's auto checker is, you may not be able to distribute the program to App Store (both Mac and iOS, see Mac app store build uses non-public APIs  electron/electron#8555)

Unless there's a way to entirely strip off the CoreSymbolication/DebugSymbols part in release build (leaving only dladdr-based backtrace), I don't recommend using the private APIs.

[alexcrichton]

Ah that's a bummer :( But yeah I think we should avoid these sorts of APIs in libstd. Maybe a "hack" that's specific to Cargo then?

[gilescope]

We're so close to something that works 'out-of-the-box' for OSX...

[gilescope]

For intelliRust people reading this, you can add an external tool that's run before launch. Getting it to kick off cp works:

cp
-R $ProjectFileDir$/target/debug/deps/ $ProjectFileDir$/target/debug/

(using the arg -n as well is faster but gives a non-zero error code second time around so you might want to put this in a script as intellirust won't continue otherwise.)
(Also you migth want to break the compile and run into two seperate steps as otherwise you'll always get the line numbers of the prior compile..)

[kennytm]

I think we could do the lift-up *.dSYM thing after #4570 is merged.

@gilescope Symlink should be enough, no need to copy unless symlink fails.

[kennytm]

Just to note that I'm currently working on fixing this (uplift the dSYM).

[therealprof]

Is this supposed to work now? I'm still not getting any line numbers out-of-the-box.

[kennytm]

@therealprof It is supported since 1.22.1.

[therealprof]

Doesn't work for me and it's driving me nuts.

This is all I get, even when calling the debug binary directly from the target/debug/deps folder

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:335:20
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::print
   2: _ZN3std9panicking12default_hook28_$u7b$$u7b$closure$u7d$$u7d$17h87a5c1996ce480feE.llvm.A7B5D2F1
   3: _ZN3std9panicking12default_hook17h99b58f5ba7dd5ae4E.llvm.A7B5D2F1
   4: std::panicking::rust_panic_with_hook
   5: _ZN3std9panicking11begin_panic17h61c0da3081a85bacE.llvm.A7B5D2F1
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::panicking::panic
  10: <core::option::Option<T>>::unwrap
  11: parser::datamodel::Datamodel::_create_intermediate_objects
  12: parser::datamodel::Datamodel::insert_object
  13: parser::parser::parse_data_model
  14: objdumper::main
  15: __rust_maybe_catch_panic
  16: std::rt::lang_start
  17: main

Im running on: rustc 1.24.0-nightly (73bca2b9f 2017-11-28)

[kennytm]

@therealprof Probably a rustc issue rust-lang/rust#46346.

[therealprof]

Could be, but interestingly, I've never seen it work and rust-lldb craps out on the debug info, too.

[raultang]

I did not see line number with rust version 1.37.0 in OSX as well, this info is useful when debugging.

rustc 1.37.0 (eae3437df 2019-08-13)

localhost:bserver $ cargo run --release
    Finished release [optimized] target(s) in 0.09s
     Running `target/release/bserver`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Log4rs(Os { code: 2, kind: NotFound, message: "No such file or directory" })', src/libcore/result.rs:999:5
stack backtrace:
   0: std::panicking::default_hook::{{closure}}
   1: std::panicking::default_hook
   2: std::panicking::rust_panic_with_hook
   3: std::panicking::continue_panic_fmt
   4: rust_begin_unwind
   5: core::panicking::panic_fmt
   6: core::result::unwrap_failed
   7: bserver::main
   8: std::rt::lang_start::{{closure}}
   9: std::panicking::try::do_call
  10: __rust_maybe_catch_panic
  11: std::rt::lang_start_internal
  12: main

While I try to run in Ubuntu:

root@midware-01:/tmp/bserver/target/release# ./bserver
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Log4rs(Os { code: 2, kind: NotFound, message: "No such file or directory" })', src/libcore/result.rs:1084:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47
   3: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:36
   4: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:200
   5: std::panicking::default_hook
             at src/libstd/panicking.rs:214
   6: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:477
   7: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:384
   8: rust_begin_unwind
             at src/libstd/panicking.rs:311
   9: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  10: core::result::unwrap_failed
             at src/libcore/result.rs:1084
  11: bserver::main
  12: std::rt::lang_start::{{closure}}
  13: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:49
  14: std::panicking::try::do_call
             at src/libstd/panicking.rs:296
  15: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  16: std::panicking::try
             at src/libstd/panicking.rs:275
  17: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  18: std::rt::lang_start_internal
             at src/libstd/rt.rs:48
  19: main
  20: __libc_start_main
  21: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

But still can not locate where the issue happened, actually this is the line have problem 11: bserver::main, below code is what it failed with:

fn main() {
log4rs::init_file("resources/log4rs.yml", Default::default()).unwrap();
}

I am new to Rust, but as other do, I am thinking whether we can print the failed line number in bserver::main, then it is quick to track down the issue where it happened.

[sfackler]

You need to enable debuginfo in release builds if you want line numbers.

[raultang]

@sfackler Great and Thanks I can see the line numbers after enable debug info.

For further I am thinking we may can keep the the info sync.
Because in Ubuntu the release mode I see the line numbers for other library, but not seeing for my code; and in OSX release mode there is no line number at all.

[Timmmm]

I still get this on Mac. Here's an example - I made it panic immediately.

$ ./backend_mac
thread 'main' panicked at 'foo', src/lib.rs:1531:3
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

It got a line number for the panic, so clearly there are some line numbers in there (this is a debug build). But the actual backtrace has no line numbers:

$ RUST_BACKTRACE=1 ./backend_mac
thread 'main' panicked at 'foo', src/lib.rs:1531:3
stack backtrace:
   0: std::panicking::begin_panic
   1: backend::run
   2: backend::main
   3: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

And full gives hashes of some kind rather than line numbers.

$ RUST_BACKTRACE=full ./backend_mac
thread 'main' panicked at 'foo', src/lib.rs:1531:3
stack backtrace:
   0:        0x10a90aab4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd4b962ed89f71a03
   1:        0x10a920d10 - core::fmt::write::h94ae1e793baa7a00
   2:        0x10a908e7b - std::io::Write::write_fmt::h5c716758fdc3057f
   3:        0x10a90c385 - std::panicking::default_hook::{{closure}}::hc6119c7d16548caf
   4:        0x10a90c0c7 - std::panicking::default_hook::heae8b62897b351dc
   5:        0x10a90c925 - std::panicking::rust_panic_with_hook::hc36596b4257bea99
   6:        0x10a8f3944 - std::panicking::begin_panic::{{closure}}::hb5bc6ba99c1e394b
   7:        0x10a8f4388 - std::sys_common::backtrace::__rust_end_short_backtrace::h8e6f5ddefb57809c
   8:        0x10a9248e2 - std::panicking::begin_panic::h01bedb9db4e539d7
   9:        0x10a8f31b0 - backend::run::h5e545650eb6cf81d
  10:        0x10a8f2ddd - backend::main::h67bc835cb48f2e30
  11:        0x10a8f311e - core::ops::function::FnOnce::call_once::hb5d60e8083894aa0
  12:        0x10a8f2f01 - std::sys_common::backtrace::__rust_begin_short_backtrace::h561760d368f03a48
  13:        0x10a8f2e84 - std::rt::lang_start::{{closure}}::h44c8c841cbc64bfa
  14:        0x10a90cc00 - std::rt::lang_start_internal::hd0c760d47f593c0a
  15:        0x10a8f2e61 - std::rt::lang_start::h213c53a46f07f64c
  16:        0x10a8f2e12 - _main

This isn't expected surely?

[Timmmm]

Ah I figured it out. I had moved the executable - you need to move the your_executable.dSYM folder with it to get line numbers.

[bombless]

I'm on Mac and using cargo 1.49.0 with rustc 1.49.0
I can see line numbers on backtrace with executable built by cargo but couldn't using bare rustc.
It seems that the executable need files from ./target/debug/deps to print line numbers, because once you remove this folder line numbers disappears, and if you rename it back line numbers show up again.

I tested for a while and found that you can build with option -C debuginfo=1 to solve this problem.

E.g. rustc -C debuginfo=1 foo.rs

[ThePerfectComputer]

I'm actually still having an issue with this. If I don't build in release mode on MacOS(M1 Monterey), things are fine. But when I do build with --release with the following enabled Cargo.toml:

[profile.release]
debug = 1

I don't get line numbers.

Here's the code I'm working on and if you run cargo run --release test-vcd-files/VCD_file_with_errors.vcd

You'll get something like the following:

../scopes.rs:(Couldn't determine line number) No more words left in vcd file."', src/main.rs:24:31

I'm using backtrack by the way, as seen here

Testing on my x86 Linux machine however, things seem to work fine.