ci: Enable opt-dist for dist-aarch64-linux builds

[mrkajetanp]

Move the CI dist-aarch64-linux job to an aarch64 runner and enable optimised dist builds with the opt-dist pipeline.

For the time being, disable bolt on aarch64 due to upstream bolt bugs.

r? @Kobzol
cc @lqd

try-job: dist-aarch64-linux

[rustbot]

Some changes occurred in src/tools/opt-dist

cc @Kobzol

[Kobzol]

Hi! Could you please split the part that moves the job to the aarch64 runner and the PGO/LTO part? So that we can evaluate the CI cost of these two actions separately. Thanks!

[mrkajetanp]

Either way this is a completely new dockerfile, so do you mean just replace this with a simple ./x dist call and then wrap it with opt-dist separately? Just in separate commits or separate PRs altogether?

[Kobzol]

I meant a separate PR, so that we can land these two changes (move to aarch64 host first, and then enable optimizations) separately :)

[mrkajetanp]

Makes sense, just wanted to make sure - no problem :)

[lqd]

What improvements are you seeing with this PR, over the current artifacts?

[mrkajetanp]

I've not yet benchmarked the changes, and I'm not sure how they compare to the artifacts from cross-compilation because I was only doing aarch64 runs but specifically adding opt-dist with LTO and PGO seems to increase the binary sizes of the main artifacts as follows:

librustc_driver-8c547d00f2ea16d5.so - 84M -> 94M
libLLVM.so.19.1-rust-1.85.0-nightly - 106M -> 108M
rustc -> 2.7M for both, +100 bytes

[Kobzol]

@bors try

Let's also see how long it takes with the optimizations.

[bors]

⌛ Trying commit 02b958d with merge b828608...

[bors]

⌛ Trying commit 02b958d with merge 945b9be...

[lqd]

That’s not going to be the good try job Jakub :3

[bors]

💔 Test failed - checks-actions

[Kobzol]

Ah, crap. Thanks!

@bors try

[bors]

⌛ Trying commit 02b958d with merge d156b32...

[bors]

☀️ Try build successful - checks-actions
Build commit: d156b32 (d156b32c73dd32916f0ca83fb0104e600fbad49d)

[mrkajetanp]

So that's an extra hour for LTO+PGO without the cache. 2h22 vs 3h22.

[lqd]

@bors try

[bors]

⌛ Trying commit 02b958d with merge ef33f24...

[bors]

☀️ Try build successful - checks-actions
Build commit: ef33f24 (ef33f249830d94b7afdb529458aae4052f14ca98)

[mrkajetanp]

1h54 cached, not so bad. Back to roughly the same time as the x86 cross build then.
Completely outside of the CI costs conversations, these changes not making the overall turnaround time for bors longer than it is already is probably a nice bonus.

[lqd]

I assume good benchmark results can also help with the cost discussion.

[Kobzol]

Indeed! You can download the CI artifacts e.g. using rustup-toolchain-install-master and benchmark it locally using rustc-perf. It would be nice to see the perf. diff. Let me know on Zulip if you want help with that.

[mrkajetanp]

rustup-toolchain-install-master

Huh TIL that this is a thing - very neat, thanks for the suggestion!

[mrkajetanp]

Between these two builds, downloaded from the CI with rustup-toolchain-install-master, rustc-perf gives me:


All metrics with a mean improvement of 10-20%, the only thing that's worse is the artifact sizes, I already put the numbers for those in a previous comment.

[mrkajetanp]

Just for completeness I ran a build with LTO+PGO+BOLT on AArch64. While the bolt hacks and workarounds can get it to work it gets very mixed results for the time being. Some improvements in compile times and on average it's beneficial but the artifact size really explodes. At the moment it doesn't work without out-of-tree patches to llvm anyway.

[Kobzol]

Yeah BOLT is quite bad in reusing the old .text segment currently. For BOLT, don't look at instruction counts, these will be artificially inflated by the larger artifact size. For BOLT cycles and wall-time should be improved, and it looks like it is (mean -2% walltime improvement, although wall-time measurements are quite noisy, this looks genuine).

[mrkajetanp]

don't look at instruction counts, these will be artificially inflated by the larger artifact size

Is the "instructions" metric in rustc-perf not just "number of instructions executed"?

Apart from that yeah, the improvements are there so once BOLT is fixed we can still enable it.

[Kobzol]

Is the "instructions" metric in rustc-perf not just "number of instructions executed"?

Yeah, but somewhat counter-intuitively, that includes the load of the binary itself into the address space, the work of the dynamic linker etc., and this can be inflated a lot if the binary suddenly gets a lot larger :) When we enabled BOLT for the Rust compiler (#116352 (comment)), the icounts were horrible, but cycles had really nice improvements.

[Kobzol]

Although previously, the icount regressions were mostly for tiny crates, which is what I would expect given the larger binary. In your screenshot, the icount regressions were for large crates, where the binary size of the compiler shouldn't have such an effect. Anyway, we'll deal with BOLT later, no need to dig into that now :)

[mrkajetanp]

Ahh right yeah that makes sense - thank you for the explanation! Exactly, let's cross that bridge when we get to it :)

[mrkajetanp]

@Kobzol wanna try opt-dist on centos as well?

[Kobzol]

@bors try

[bors]

⌛ Trying commit 0f4f465 with merge 3d52e24...

[bors]

☀️ Try build successful - checks-actions
Build commit: 3d52e24 (3d52e243cf55ef270d2d201a3bf28ec172a63c18)