Occasional non-reproducibility when building rustc

[jgalenson]

Over the past few days I have occasionally seen my builds of rustc be non-reproducible. The differences most often seem to be in stage 1's libparking_log, librustc_parse, or librustc_driver.

A git bisect suggests that this was caused by 138c50f, although due to the fact that this issue is itself difficult to reproduce this could potentially be incorrect. But after many builds with and without that patch, I have only seen the error with it. Assuming that is correct, I believe this issue might well be due to non-determinism in LLVM's -O3, but I'm not sure how to run just the LLVM portion on the compile so I'm not sure.

Note that this is a part of #34902.

[Mark-Simulacrum]

It seems plausible that bumping the optimization level would cause instability in llvm, cc @eddyb

[eddyb]

cc @nagisa @nikic @hanna-kruppe Is there any easy way to diagnose this sort of thing, short of getting -C llvm-args=-print-after-all dumps for one of the affected crates and diffing them?

@Mark-Simulacrum If we do have to do this the hard way, maybe exposing cargo rustc via ./x.py rustc would be a good idea? It also would've come in handy when debugging bugs while compiling stage1 libcore (being able to add flags easily just to one crate's rustc's invocation).

[eddyb]

@jgalenson I forget if/how we expose this, but if you can change codegen-units to 1, can you still reproduce? I think you'd need that for -print-after-all (otherwise you'd get threads stomping on eachother), but also it might tell us what role ThinLTO plays in this.

[jgalenson]

@eddyb Doesn't codegen-units default to 1? I did not change anything for it in config.toml.

[nikic]

@eddyb You can use -Z no-parallel-llvm to avoid interleaving without touching optimization settings, it's not necessary to disable codegen units.

I don't have a great suggestion on how to debug this beyond diffing the transform log.

There is some open non-determinism in LVI (will usually manifest in jump threading), though no idea if that is hit or something else.

[Mark-Simulacrum]

If we do have to do this the hard way, maybe exposing cargo rustc via ./x.py rustc would be a good idea? It also would've come in handy when debugging bugs while compiling stage1 libcore (being able to add flags easily just to one crate's rustc's invocation).

This wold be pretty painful to do today I suspect do to how rustbuild works internally, and I haven't had the energy to try and investigate changes there for the past couple months.

[eddyb]

@Mark-Simulacrum How so, can't we "just" turn one of the cargo build commands into a cargo rustc command?

[Mark-Simulacrum]

Hm, perhaps. Though I'm not sure :) Certainly one would likely expect some way to get the arguments passed in a stage-specific way... I would need to look into it, and I don't have the bandwidth or energy right now unfortunately.

[eddyb]

Certainly one would likely expect some way to get the arguments passed in a stage-specific way...

I mean, everything would still be go through the src/bootstrap/bin/rustc.rs wrapper (i.e. we wouldn't change anything below invoking Cargo), we'd just have a way to pass extra arguments and building one specific crate.

For now it's probably easier to locally change src/bootstrap/bin/rustc.rs to check for a specific --crate-name argument and add the right LLVM args.