DWARF info for `static` vars in lib crates has stopped being produced reliably in LTO builds #90357

cbiffle · 2021-10-28T00:42:48Z

As of the 2021-09-23 nightly, rustc has stopped producing debug output for variables in a specific combination of circumstances:

The variable is static
It is defined in a lib crate.
The lib crate is included in a bin crate.
The bin crate is built with lto = true.

We noticed this after bumping past 09-23 to the 2021-10-27 nightly, when our debugger broke. The static is still being included in the ELF output (both allocated space, and represented correctly in the symtab), but is missing from DWARF. Bisecting indicates 2021-09-23 as the culprit (09-22 works).

Here is a repro case that I've reduced that works on linux x86-64, at least, and probably other DWARF platforms.

https://github.com/cbiffle/rustc-dwarf-regression-repro

@rustbot modify labels: +regression-from-stable-to-nightly -regression-untriaged

The text was updated successfully, but these errors were encountered:

This is the last toolchain that does not suffer from this bug: rust-lang/rust#90357

Mark-Simulacrum · 2021-10-28T02:50:56Z

Bumping to regression-stable-to-beta, because 2021-09-23 is a 1.57 nightly. But would be good to confirm this reproduces on beta.

luqmana · 2021-10-28T03:03:23Z

cargo-bisect-rustc points at 308dffd (#89179). That being a rollup, #89041 looks the most relevant.

apiraino · 2021-10-28T09:11:51Z

Assigning priority as discussed in the Zulip thread of the Prioritization Working Group.

@rustbot label -I-prioritize +P-high

cuviper · 2021-11-19T18:31:37Z

I think this may be a duplicate of #90301. Please test on a current nightly, and when 1.57.0-beta.4 goes out it should have the backport via #90938.

It's still possible that #89041 introduced its own problem, but maybe it just exacerbated the LLVM issue.

pnkfelix · 2021-11-22T19:54:49Z

It looks to me like the problem still reoccurs on the current nightly:

% rustup override set nightly-2021-11-22
info: using existing install for 'nightly-2021-11-22-x86_64-unknown-linux-gnu'
info: override toolchain for '/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro' set to 'nightly-2021-11-22-x86_64-unknown-linux-gnu'

  nightly-2021-11-22-x86_64-unknown-linux-gnu unchanged - rustc 1.58.0-nightly (65f3f8b22 2021-11-21)

% rustup show active-toolchain
nightly-2021-11-22-x86_64-unknown-linux-gnu (directory override for '/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro')
% rustc --version
rustc 1.58.0-nightly (65f3f8b22 2021-11-21)
% cargo clean
% cargo build --release --verbose
   Compiling nodwarfrepro v0.1.0 (/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro)
     Running `rustc --crate-name nodwarfrepro --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C debuginfo=2 -C metadata=f2ccefc4246296b8 -C extra-filename=-f2ccefc4246296b8 --out-dir /media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro/target/release/deps -L dependency=/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro/target/release/deps`
     Running `rustc --crate-name nodwarfrepro --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto -C debuginfo=2 -C metadata=f4b2592c51b39d90 -C extra-filename=-f4b2592c51b39d90 --out-dir /media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro/target/release/deps -L dependency=/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro/target/release/deps --extern nodwarfrepro=/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro/target/release/deps/libnodwarfrepro-f2ccefc4246296b8.rlib`
    Finished release [optimized + debuginfo] target(s) in 2.55s
% dwarfdump target/release/nodwarfrepro | grep THE_STATIC
%

For completeness, here is a transcript showing the behavior that is expected, from nightly-2021-09-22

% rustup override set nightly-2021-09-22
info: using existing install for 'nightly-2021-09-22-x86_64-unknown-linux-gnu'
info: override toolchain for '/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro' set to 'nightly-2021-09-22-x86_64-unknown-linux-gnu'

  nightly-2021-09-22-x86_64-unknown-linux-gnu unchanged - rustc 1.57.0-nightly (ac2d9fc50 2021-09-21)

% rustup show active-toolchain
nightly-2021-09-22-x86_64-unknown-linux-gnu (directory override for '/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro')
% rustc --version
rustc 1.57.0-nightly (ac2d9fc50 2021-09-21)
% cargo clean
% cargo build --release --verbose
   Compiling nodwarfrepro v0.1.0 (/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro)
     Running `rustc --crate-name nodwarfrepro --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C debuginfo=2 -C metadata=f2ccefc4246296b8 -C extra-filename=-f2ccefc4246296b8 --out-dir /media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro/target/release/deps -L dependency=/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro/target/release/deps`
     Running `rustc --crate-name nodwarfrepro --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto -C debuginfo=2 -C metadata=f4b2592c51b39d90 -C extra-filename=-f4b2592c51b39d90 --out-dir /media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro/target/release/deps -L dependency=/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro/target/release/deps --extern nodwarfrepro=/media/pnkfelix/Rust/rust-90357/objdir-dbgopt/rustc-dwarf-regression-repro/target/release/deps/libnodwarfrepro-f2ccefc4246296b8.rlib`
    Finished release [optimized + debuginfo] target(s) in 2.78s
% dwarfdump target/release/nodwarfrepro | grep THE_STATIC
                        DW_AT_name                  THE_STATIC
                        DW_AT_linkage_name          _ZN12nodwarfrepro10THE_STATIC17he3077d0fbb05bed3E
global die-in-sect 0x00000089, cu-in-sect 0x00000071, die-in-cu 0x00000023, cu-header-in-sect 0x00000066 'THE_STATIC'
name at offset 0x000000d1, length   10 is 'THE_STATIC'
name at offset 0x00000102, length   49 is '_ZN12nodwarfrepro10THE_STATIC17he3077d0fbb05bed3E'
%

to work around rust-lang/rust#90357

cbiffle · 2022-04-04T17:31:31Z

This appears to still be reproducing as of rustc 1.61.0-nightly (9c06e1ba4 2022-03-29); we're stuck on 2021-09-22 because basically anything newer breaks our debugger.

cbiffle · 2022-04-04T20:18:31Z

So, after working out how to build a toolchain that includes our cross target and rust-lld and whatnot locally, I've tested today's mainline d5139f4 with a revert for PR #89041, commit d5de680 -- which @luqmana had previously identified as the likely culprit.

I can confirm that DWARF output for our statics on thumbv7em and kin is restored after that commit is reverted. Shall I prepare a PR?

- Re-enabled basic-types-globals which has been disabled since 2018 - Updated its now-rotted assertions about GDB's output to pass - Rewrote header comment describing some previous state of GDB behavior that didn't match either the checked-in assertions _or_ the current behavior, and so I assume has just been wrong for some time. - Copy-pasta'd the test into a version that uses LTO to check for regression on rust-lang#90357, because I don't see a way to matrix the same test into several build configurations.

These tests were disabled four years ago; it appears that it was intended to be temporary. In that time their assertions have rotted. I've gone through and fixed the assertions where necessary and rehabilitated them. In several cases I've had to leave lldb disabled, because the lldb output has changed in ways that I don't understand. I figured gdb coverage was better than no coverage. It's worth noting that these tests being enabled would probably have caught the regression observed in rust-lang#90357.

… r=pnkfelix Revert "Work around invalid DWARF bugs for fat LTO" Since September, the toolchain has not been generating reliable DWARF information for static variables when LTO is on. This has affected projects in the embedded space where the use of LTO is typical. In our case, it has kept us from bumping past the 2021-09-22 nightly toolchain lest our debugger break. This has been a pretty dramatic regression for people using debuggers and static variables. See rust-lang#90357 for more info and a repro case. This commit is a mechanical revert of d5de680 from PR rust-lang#89041, which caused the issue. (Note on that PR that the commit's author has requested it be reverted.) I have locally verified that this fixes rust-lang#90357 by restoring the functionality of both the repro case I posted on that bug, and debugger behavior on real programs. There do not appear to be test cases for this in the toolchain; if I've missed them, point me at 'em and I'll update them.

…=pnkfelix Revert "Work around invalid DWARF bugs for fat LTO" Since September, the toolchain has not been generating reliable DWARF information for static variables when LTO is on. This has affected projects in the embedded space where the use of LTO is typical. In our case, it has kept us from bumping past the 2021-09-22 nightly toolchain lest our debugger break. This has been a pretty dramatic regression for people using debuggers and static variables. See rust-lang#90357 for more info and a repro case. This commit is a mechanical revert of d5de680 from PR rust-lang#89041, which caused the issue. (Note on that PR that the commit's author has requested it be reverted.) I have locally verified that this fixes rust-lang#90357 by restoring the functionality of both the repro case I posted on that bug, and debugger behavior on real programs. There do not appear to be test cases for this in the toolchain; if I've missed them, point me at 'em and I'll update them.

cbiffle added C-bug Category: This is a bug. regression-untriaged Untriaged performance or correctness regression. labels Oct 28, 2021

rustbot added I-prioritize Issue: Indicates that prioritization has been requested for this issue. regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. and removed regression-untriaged Untriaged performance or correctness regression. labels Oct 28, 2021

cbiffle changed the title ~~DWARF info for static vars in lib crates has stopped being produced reliably~~ DWARF info for static vars in lib crates has stopped being produced reliably in LTO builds Oct 28, 2021

camelid added A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) E-needs-bisection Call for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc labels Oct 28, 2021

cbiffle added a commit to oxidecomputer/hubris that referenced this issue Oct 28, 2021

Revert toolchain to 2021-09-22.

61806e5

This is the last toolchain that does not suffer from this bug: rust-lang/rust#90357

cbiffle mentioned this issue Oct 28, 2021

Revert toolchain to 2021-09-22. oxidecomputer/hubris#231

Merged

cbiffle added a commit to oxidecomputer/hubris that referenced this issue Oct 28, 2021

Revert toolchain to 2021-09-22.

a8e07b3

This is the last toolchain that does not suffer from this bug: rust-lang/rust#90357

Mark-Simulacrum added this to the 1.57.0 milestone Oct 28, 2021

Mark-Simulacrum added regression-from-stable-to-beta Performance or correctness regression from stable to beta. and removed regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. labels Oct 28, 2021

rustbot added P-high High priority and removed I-prioritize Issue: Indicates that prioritization has been requested for this issue. labels Oct 28, 2021

apiraino added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Oct 28, 2021

pnkfelix self-assigned this Oct 28, 2021

apiraino removed the E-needs-bisection Call for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc label Nov 4, 2021

pnkfelix added regression-from-stable-to-stable Performance or correctness regression from one stable version to another. and removed regression-from-stable-to-beta Performance or correctness regression from stable to beta. labels Dec 2, 2021

japaric mentioned this issue Jan 4, 2022

enabling LTO triggers "...location info is incomplete..." knurling-rs/defmt#649

Open

Urhengulas pushed a commit to knurling-rs/app-template that referenced this issue Jan 5, 2022

Disable LTO (#61)

b4327d0

to work around rust-lang/rust#90357

japaric mentioned this issue Mar 11, 2022

re-enable LTO knurling-rs/app-template#63

Closed

steveklabnik mentioned this issue Apr 4, 2022

bump toolchain to last night's nightly oxidecomputer/hubris#438

Merged

cbiffle mentioned this issue Apr 4, 2022

Work around invalid DWARF bugs for fat LTO #89041

Merged

cbiffle mentioned this issue Apr 5, 2022

Revert "Work around invalid DWARF bugs for fat LTO" #95685

Merged

cbiffle mentioned this issue Apr 5, 2022

Reenable debuginfo tests #95696

Closed

timblakely mentioned this issue May 1, 2022

Notification/interrupt latency: resetting IRQ notification mask takes hundreds of cycles oxidecomputer/hubris#517

Closed

bors closed this as completed in 98190b7 Jul 16, 2022

jieyouxu added A-reproducibility Area: Reproducible / deterministic builds A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. labels Aug 14, 2024

jieyouxu mentioned this issue Aug 14, 2024

Tracking Issue for Reproducible Build bugs and challenges #129080

Open

81 tasks

jieyouxu added the A-LTO Area: Link-time optimization (LTO) label Aug 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DWARF info for `static` vars in lib crates has stopped being produced reliably in LTO builds #90357

DWARF info for `static` vars in lib crates has stopped being produced reliably in LTO builds #90357

cbiffle commented Oct 28, 2021

Mark-Simulacrum commented Oct 28, 2021

luqmana commented Oct 28, 2021

apiraino commented Oct 28, 2021

cuviper commented Nov 19, 2021

pnkfelix commented Nov 22, 2021

cbiffle commented Apr 4, 2022 •

edited

Loading

cbiffle commented Apr 4, 2022

DWARF info for static vars in lib crates has stopped being produced reliably in LTO builds #90357

DWARF info for static vars in lib crates has stopped being produced reliably in LTO builds #90357

Comments

cbiffle commented Oct 28, 2021

Mark-Simulacrum commented Oct 28, 2021

luqmana commented Oct 28, 2021

apiraino commented Oct 28, 2021

cuviper commented Nov 19, 2021

pnkfelix commented Nov 22, 2021

cbiffle commented Apr 4, 2022 • edited Loading

cbiffle commented Apr 4, 2022

DWARF info for `static` vars in lib crates has stopped being produced reliably in LTO builds #90357

DWARF info for `static` vars in lib crates has stopped being produced reliably in LTO builds #90357

cbiffle commented Apr 4, 2022 •

edited

Loading