Tracking Issue for Reproducible Build bugs and challenges

[jieyouxu]

This is a tracking issue for collecting and triaging bugs and challenges that hinder our ability to produce reproducible and deterministic builds.

This tracking issue is used as a hub for connecting to other relevant issues, e.g., bugs or open design questions. This tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature. Instead, open a dedicated issue for the specific matter and add the A-reproducibility Area: Reproducible / deterministic builds label, and tag with OS/architecture/target labels as suitable.

This tracking issue is unlikely to be exhaustive. Please add suitable entries and edit as new issues and PRs pop up or if old issues and PRs are rediscovered.

Copy pastas for the inline labels

A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.)
A-diagnostics Area: Messages for errors, warnings, and lints
A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.
A-reproducibility Area: Reproducible / deterministic builds
A-run-make Area: port run-make Makefiles to rmake.rs
A-testsuite Area: The testsuite used to check the correctness of rustc
C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC
F-trim-paths Feature: trim-paths
T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
T-cargo Relevant to the cargo team, which will review and decide on the PR/issue.
T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
T-lang Relevant to the language team, which will review and decide on the PR/issue.
T-libs Relevant to the library team, which will review and decide on the PR/issue.
T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue.
O-wasm Target: WASM (WebAssembly), http://webassembly.org/
O-windows Operating system: Windows
O-windows-msvc Toolchain: MSVC, Operating system: Windows
O-windows-gnu Toolchain: GNU, Operating system: Windows
O-macos Operating system: macOS
O-apple Operating system: Apple (macOS, iOS, tvOS, visionOS, watchOS)
O-AArch64 Armv8-A or later processors in AArch64 mode
O-x86_64 Target: x86-64 processors (like x86_64-*)

Bugs and issues

• tests/run-make/reproducible-builds ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-testsuite Area: The testsuite used to check the correctness of rustc , A-run-make Area: port run-make Makefiles to rmake.rs ; several skipped combinations due to unreproducibility)
• [Inquiry] Inconsistent binary size with 'cargo +stable build' vs 'cargo +1.50-x86_64-pc-windows-msvc' ? #82188 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , O-windows-msvc Toolchain: MSVC, Operating system: Windows ; needs further investigation)
• rustdoc output directory structure leaks home directory structure #18370 ( T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. ; status unclear)
• Randomize the ordering of struct fields when compiling in debug mode #38550 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. ; related to -Z randomize-layout, status unclear)
• --remap-path-prefix does not apply to secondary files in diagnostics #66251 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-diagnostics Area: Messages for errors, warnings, and lints , --remap-path-prefix; sysroot handling in diagnostics)
• Non-deterministic perf.rust-lang.org runs (e.g. syn-opt). #69060 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. , perf; unclear what's to do here)
• Occasional non-reproducibility when building rustc #69352 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. ; possible non-determinism in LLVM)
• User type annotations can result in different mir-opt test results in CI vs locally #73501 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-testsuite Area: The testsuite used to check the correctness of rustc , A-mir-opt Area: MIR optimizations ; unsure how to repro/debug)
• Formalize SVH stability for a specific release and target #73932 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-linkage Area: linking into static, shared libraries and binaries ; Stable Version Hashes)
• sysroot spans still not printed, when remap-path-prefix is set with a custom path #74786 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-testsuite Area: The testsuite used to check the correctness of rustc ; --remap-path-prefix)
• Accepted convention for incurring dependencies on external files in macro expansion hurts reproducibility #75887 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-macros Area: All kinds of macros (custom derive, macro_rules!, proc macros, ..) ; macro expansions with deps on external files)
• Reproducible build run-make tests fails if -C debuginfo=2 (on Linux this time) #89911 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) ; -Cdebuginfo=2)
• Windows builds are not deterministic with --crate-type=bin and path-prefix remapping #88982 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) , A-linkage Area: linking into static, shared libraries and binaries , O-windows Operating system: Windows ; affects tests/run-make/reproducible-builds, bin, --remap-path-prefix, -Z remap-cwd-prefix)
• --crate-type=rlib + -Cdebuginfo=2 + -Zremap-cwd-prefix=. is not reproducible on Windows #129117 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-testsuite Area: The testsuite used to check the correctness of rustc , A-run-make Area: port run-make Makefiles to rmake.rs ; -Cdebuginfo=2, -Z remap-cwd-prefix=., rlib)

Tracking/meta issues:

• CI for deterministic / reproducible builds #75362 ( T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. )
• Add an unstable 'extra randomization' compile-time flag for hashers #65042 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. )
• Audit all UNTRACKED options #84232 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. )
• Tracking Issue for trim-paths RFC 3127 #111540 ( C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC )
• Tracking Issue for -Z remap-cwd-prefix #89434 ( C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC )

Resolved:

• reproducible-build-2 fails if debuginfo-level > 0 #66568 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) ; -Cdebuginfo=2)
• OSX compilation with debuginfo isn't deterministic #47086 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) ; -Cdebuginfo=2)
• Rustdoc output slightly nondeterministic #24473 ( T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. )
• Bit-for-bit deterministic / reproducible builds #34902 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. ; current status unclear)
• Detect and report usage of environment variables at build time and run time. #44074 (
  T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. ; env var info)
• debug_loc info is (slightly) non-deterministic #45397 ( A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. , A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) ; looks LLVM related, seems to have resolved)
• reproducible builds broken in rustc 1.56.0 due to LLVM 13 update #90301 ( A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. , A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) )
  • Upstream fix: https://reviews.llvm.org/D113468
  • DWARF info for static vars in lib crates has stopped being produced reliably in LTO builds #90357
• OSX compilation with debuginfo isn't deterministic #47086 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , O-macos Operating system: macOS , A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) )
• rustc produces indeterministic asm #57041 (insufficiently specific)
• Cargo should provide --remap-path-prefix or not hash it from RUSTFLAGS. cargo#6914 ( T-cargo Relevant to the cargo team, which will review and decide on the PR/issue. ; resolved separately)
• rustdoc generation of search-index.js depends on the order crates are documented #61216 ( T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. ; resolved in a separate improvement PR but can't find which)
• Create a deterministic FxHashMap wrapper #63713 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. )
  • Related: Intermittent failure in src/test/ui/reify-intrinsic.stderr #65036
• Intermittent failure in src/test/ui/reify-intrinsic.stderr #65036 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-testsuite Area: The testsuite used to check the correctness of rustc )
• Semi-spurious ui test fluctuation #81296 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-testsuite Area: The testsuite used to check the correctness of rustc )
• reproducible-build-2 fails if debuginfo-level > 0 #66568 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , O-macos Operating system: macOS , A-run-make Area: port run-make Makefiles to rmake.rs , A-testsuite Area: The testsuite used to check the correctness of rustc )
• remap-path-prefix will not re-compile properly in dev build #66955 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. ; --remap-path-prefix)
• Source file names are included into a release binary even if abort upon panic is enabled #75263 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. ; --remap-path-prefix involved)
• Binaries end up containing path to the rust-src component despite --remap-path-prefix #73167 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. ; --remap-path-prefix involved)
• Enable --remap-path-prefix for absolute paths by default #40552 (closed in favor of more specific issues and RFC-based approaches)
• --remap-path-prefix doesn't work in debug mode? #84125 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. ; --remap-path-prefix involved)
• registry paths hard coded in binary cargo#9311 ( T-cargo Relevant to the cargo team, which will review and decide on the PR/issue. )
• Tiny difference of the diagnostics for generator on some environment #71222 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-diagnostics Area: Messages for errors, warnings, and lints )
• Error reporting order on riscv64gc-unknown-linux-gnu #72913 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-diagnostics Area: Messages for errors, warnings, and lints )
• Reproducibility regression caused by 57edf88 #76496 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-diagnostics Area: Messages for errors, warnings, and lints )
• --remap-path-prefix not remapping out_dir in debug info on Wasm #80776 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-diagnostics Area: Messages for errors, warnings, and lints , O-wasm Target: WASM (WebAssembly), http://webassembly.org/ )
• --remap-path-prefix behavior regression resulting in bad paths in 1.50.0 #82074 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , --remap-path-prefix)
• remap-path-prefix appears to be applied in a reverse order #82108 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , --remap-path-prefix; paths reverse order from clang/gcc but was intentional)
• The build prefix is part of the compiled compiler #82392 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , -Crpath)
• Reproducible outputs without build-environment-specific command lines #87325 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , -Z remap-cwd-prefix; closed in favor of tracking issue Tracking Issue for -Z remap-cwd-prefix #89434 for -Z remap-cwd-prefix)
• --remap-path-prefix doesn't map paths to .pdb files, even in release mode #87825 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) , O-windows-msvc Toolchain: MSVC, Operating system: Windows , --remap-path-prefix)
• DWARF info for static vars in lib crates has stopped being produced reliably in LTO builds #90357 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) , https://github.com/rust-lang/rust/labels/A-llvm, https://github.com/rust-lang/rust/labels/A-lto)
• nondeterministic ordering in rmeta section #113584 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. ; issue disappeared after stable version update but unclear what fixed it)
• building the std for aarch64_be-unknown-linux-gnu_ilp32 twice in a row, the binary libcompiler_builtins.rlib is inconsistent #119372 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , compiler_builtins)
• Nondeterminism encountered in metadata for doc_link_resolutions #124357 (fuchsia build system specific, unclear)
• Separate --remap-path-prefix for diagnostics and debug info #73740 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) , A-diagnostics Area: Messages for errors, warnings, and lints , --remap-path-prefix)
• Add flag to modify the working directory assumed by the diagnostics #88754 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. , A-diagnostics Area: Messages for errors, warnings, and lints )

Related issues, RFCs, MCPs and discussions

• Do not implement HashStable for HashSet (MCP 533) #108312 ( T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. )
• New rustc and Cargo options to allow path sanitisation by default rfcs#3127 ( F-trim-paths Feature: trim-paths )
• Reproducible command line + determinism compiler-team#450

Related PRs

• Print environment variables accessed by rustc as special comments into depinfo files #71858
• Work around invalid DWARF bugs for fat LTO #89041
• Export ZERO_AR_DATE for macos linker invocations #71931
• data_structures: Add deterministic FxHashMap and FxHashSet wrappers #64131
• Make re-export collection deterministic #65043
• Make weak item traversal deterministic #81393
• Use FxHashSet instead of Vec for well formed tys #88771
• Use UnordMap and UnordSet for id collections (DefIdMap, LocalDefIdMap, etc) #106977
• Add TRACKED_NO_CRATE_HASH and use it for --remap-path-prefix #84233
• Implement rustc part of RFC 3127 trim-paths #115214
• Simplify trim-paths feature by merging all debuginfo options together #122450
• feat: implement RFC 3127 -Ztrim-paths cargo#12625
• Fix rustdoc missing handling of remap-path-prefix option #128736
• Ignore remap-path-prefix in metadata hash. cargo#6966
• Do not show DefId in diagnostics #71310
• resolve: Sort E0408 errors by Symbol str #72982
• SessionDiagnostic: Fix non-determinism in generated format string. #76515
• Set path of the compile unit to the source directory #82102
• Apply path remapping to DW_AT_GNU_dwo_name when producing split DWARF #91566
• Remove the install prefix from the rpath set when using -Crpath #86025
• Introduce -Z remap-cwd-prefix switch #87320
• link.exe: Don't embed full path to PDB file in binary. #121297
• Revert "Work around invalid DWARF bugs for fat LTO" #95685
• Generate outlined_atomics.rs in a deterministic order. compiler-builtins#549
• rustdoc: Add support for --remap-path-prefix #107099