Windows builds are not deterministic with --crate-type=bin and path-prefix remapping

[danakj]

The existing reproducible-build tests only verify that building with --crate-type=rlib produces the same output when building from different locations when using --remap-path-prefix to strip out the location differences.

#87320 adds another way to do this with --remap-cwd-prefix. It added a test that verifies the same thing with rlibs, but also added a test for --crate-type=bin. This test for bin passes on Linux but fails with a difference in the output on Windows: #87320 (comment)

2021-09-15T00:11:03.6840467Z /d/a/rust/rust/build/x86_64-pc-windows-msvc/test/run-make-fulldeps/reproducible-
build/reproducible-build/first /d/a/rust/rust/build/x86_64-pc-windows-msvc/test/run-make-fulldeps/reproducible-build/
reproducible-build/reproducible-build differ: char 140561, line 276  <<<<====
2021-09-15T00:11:03.6843153Z make[1]: Leaving directory '/d/a/rust/rust/src/test/run-make-fulldeps/reproducible-build'
2021-09-15T00:11:03.6843848Z 
2021-09-15T00:11:03.6844310Z ------------------------------------------
2021-09-15T00:11:03.6844776Z stderr:
2021-09-15T00:11:03.6845251Z ------------------------------------------
2021-09-15T00:11:03.6845850Z make[1]: *** [Makefile:78: remap_cwd_bin] Error 1

To verify, I added another test of --remap-path-prefix with --crate-type=bin to see if that produces non-deterministic build outputs on Windows as well, and it does: #87320 (comment)

2021-09-15T16:43:43.1281275Z /d/a/rust/rust/build/x86_64-pc-windows-msvc/test/run-make-fulldeps/reproducible-
build/reproducible-build/reproducible-build /d/a/rust/rust/build/x86_64-pc-windows-msvc/test/run-make-fulldeps/reproducible-
build/reproducible-build/foo differ: char 139921, line 272  <<<<====
2021-09-15T16:43:43.1282876Z make[1]: Leaving directory '/d/a/rust/rust/src/test/run-make-fulldeps/reproducible-build'
2021-09-15T16:43:43.1283397Z 
2021-09-15T16:43:43.1283769Z ------------------------------------------
2021-09-15T16:43:43.1284164Z stderr:
2021-09-15T16:43:43.1284554Z ------------------------------------------
2021-09-15T16:43:43.1285077Z make[1]: *** [Makefile:72: different_source_dirs_bin] Error 1

Thus we see that the output of --crate-type=bin is not deterministic on Windows, based on running rustc from different directories.

Meta

This is with rustc at tip of tree, as the commit queue found it in merge e63d3d7 .
HEAD was c3c0f80d608 at that time, I believe.

No root cause determined yet, and I don't have the Windows binary outputs to see what is at the differing location yet.

[danakj]

2 tests that demonstrate this failure are being added in #87320, with them disabled at the moment: https://github.com/rust-lang/rust/pull/87320/files#diff-ef6ac253a54908edf6e7bc0670cd3e2756ba7cea7d203016b7e4ca1520928701R17

• different_source_dirs_bin
• remap_cwd_bin

[ChrisDenton]

Is #87825 perhaps related?

[danakj]

Likely, thanks. I'm working on getting a Windows environment set up and can confirm then. Also attempting to sort out what clang does here, since Chromium has no such C++ problems.

[nellshamrell]

I created a Docker development environment and sample application which replicates this issue https://github.com/nellshamrell/reproducible_build_basic_exp (it does still require a Windows workstation with Docker installed)

[nellshamrell]

UPDATE - passing the Clink-arg=/Brepro to rustc appears to make .exe binaries (at least with the one project I tested) build reproducibly! Going to try this in that test and see if it helps.

[danakj]

We've gotten as far as trying to build Rust things on Windows in Chromium and our determinism bot quickly found rustc-link-driven outputs to be non-deterministic.

Debugging happened here: https://bugs.chromium.org/p/chromium/issues/detail?id=1453509

The problem is that rustc is passing absolute paths for object files to lld-link, which then encodes them into the .EXE and .PDB file.

1. We give a relative path to the sysroot, like --sysroot=local_rustc_sysroot and we also -Zremap-cwd-prefix=. but then rustc generates paths to the sysroot as absolute, which includes the cwd like $PWD/local_rustc_sysroot/lib/rustlib/x86_64-pc-windows-msvc/lib/libstd.rlib which then appears in the .PDB making the output non-deterministic. In theory -Zremap-cwd-prefix should be rewriting these paths, if they are really supposed to turn the --sysroot value into an absolute path, but currently that does not happen.
2. Rustc produces a .o file in /tmp for the bin crate which is necessarily an absolute path, and which differs from execution to execution, and can't possibly be rewritten by -Zremap-cwd-prefix either.

The .EXE itself has bits that differ and which are not the path strings themselves, but when these paths are all made relative (including by moving the /tmp object to a relative path) then the .EXE differences go away.