Change LLVM BOLT flags #114141

Kobzol · 2023-07-27T19:32:59Z

I talked to the BOLT maintainers about the binary size effect of BOLT. Currently, BOLTing LLVM increases its binary size from ~120 MiB to ~170 MiB, which is not ideal. Now we can track both runtime performance and (rustc, LLVM, ...) artifact sizes in perf.RLO, so I'd like to try experimenting with changing the flags to reduce libLLVM.so size without regressing the performance gains of BOLT too much.

r? @ghost

Kobzol · 2023-07-27T19:33:17Z

@bors try @rust-timer queue

bors · 2023-07-27T19:33:26Z

⌛ Trying commit bc6b5fe8160438c4cff1d58ea5e9f57783b25dd3 with merge b734abfa73b23b4b0db768de4dba2901bca0ead5...

bors · 2023-07-27T20:50:50Z

☀️ Try build successful - checks-actions
Build commit: b734abfa73b23b4b0db768de4dba2901bca0ead5 (b734abfa73b23b4b0db768de4dba2901bca0ead5)

the8472 · 2023-07-27T21:10:24Z

src/bootstrap/bolt.rs

+        // Try to reuse old text segments to reduce binary size
+        .arg("--use-old-text")
+        .arg("--lite=0")
+        .arg("--no-huge-pages")


I assume the huge page alignment just adds padding, so it should compress fine and not affect download sizes and also shouldn't affect the amount of stuff that gets loaded into memory.

I'm not actually sure what this does, as it seems to be undocumented. I'll try without it.

rust-timer · 2023-07-28T09:55:47Z

Finished benchmarking commit (b734abfa73b23b4b0db768de4dba2901bca0ead5): comparison URL.

Overall result: ❌✅ regressions and improvements - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.7%	[0.3%, 1.1%]	4
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.3%	[-0.3%, -0.3%]	1
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.6%	[1.6%, 3.6%]	6
Improvements ✅ (primary)	-1.6%	[-1.8%, -1.4%]	2
Improvements ✅ (secondary)	-2.2%	[-2.9%, -0.9%]	5
All ❌✅ (primary)	-1.6%	[-1.8%, -1.4%]	2

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 654.227s -> 655.834s (0.25%)

Kobzol · 2023-07-28T11:49:13Z

Okay, pretty much no regressions and -50 MiB of LLVM size. Pretty nice. I'll try next what happens with just --use-old-text.

Kobzol · 2023-07-28T11:50:07Z

@bors try @rust-timer queue

Trying just --use-old-text.

bors · 2023-07-28T11:50:16Z

⌛ Trying commit b996e5e with merge b5ad0060faf5ff61111a1a191fe848f4955ac46f...

bors · 2023-07-28T13:03:49Z

☀️ Try build successful - checks-actions
Build commit: b5ad0060faf5ff61111a1a191fe848f4955ac46f (b5ad0060faf5ff61111a1a191fe848f4955ac46f)

rust-timer · 2023-07-29T08:21:41Z

Finished benchmarking commit (b5ad0060faf5ff61111a1a191fe848f4955ac46f): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.4%, -0.4%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.4%	[-0.4%, -0.4%]	1

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.0%	[-1.2%, -0.9%]	2
All ❌✅ (primary)	-	-	0

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 651.183s -> 650.167s (-0.16%)

Kobzol · 2023-07-29T08:52:57Z

Ok, looks like adding this flag doesn't regress perf and saves 50 MiB of libLLVM size. That looks nice.

r? bootstrap

lqd · 2023-07-29T09:04:06Z

@bors r+

bors · 2023-07-29T09:04:07Z

📌 Commit b996e5e has been approved by lqd

It is now in the queue for this repository.

Kobzol · 2023-07-29T09:05:29Z

BOLT can be tricky, so let's make sure that this is easier to revert if some issuea crop up.

@bors rollup=never

bors · 2023-07-29T09:37:52Z

⌛ Testing commit b996e5e with merge f45961b...

bors · 2023-07-29T11:24:20Z

☀️ Test successful - checks-actions
Approved by: lqd
Pushing f45961b to master...

rust-timer · 2023-07-29T20:50:30Z

Finished benchmarking commit (f45961b): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.2%	[1.7%, 3.8%]	4
Improvements ✅ (primary)	-1.9%	[-2.1%, -1.6%]	3
Improvements ✅ (secondary)	-2.8%	[-2.8%, -2.8%]	1
All ❌✅ (primary)	-1.9%	[-2.1%, -1.6%]	3

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.1%	[3.1%, 3.1%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	3.1%	[3.1%, 3.1%]	1

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 652.022s -> 653.409s (0.21%)

Strip unexpected debuginfo from `libLLVM.so` and `librustc_driver.so` when not requesting any debuginfo As seen in rust-lang#114175 and in [this zulip discussion](https://rust-lang.zulipchat.com/#narrow/stream/247081-t-compiler.2Fperformance/topic/Artifact.20sizes/near/379302655), there's still some small amount of debuginfo in LLVM's shared library on linux, even when not requesting it (nightly CI), coming from `libstdc++`. ``` $ readelf --debug-dump=info ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-16-rust-1.73.0-nightly.so | grep DW_TAG_compile_unit -A5 | grep DW_AT_comp_dir | cut -d ":" -f 2- | counts 101 counts ( 1) 39 (38.6%, 38.6%): (indirect string, offset: 0x7): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/libsupc++ ( 2) 38 (37.6%, 76.2%): (indirect string, offset: 0x43fb2): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11 ( 3) 23 (22.8%, 99.0%): (indirect string, offset: 0x18ed8): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++98 ( 4) 1 ( 1.0%,100.0%): (indirect string, offset: 0x53f04): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src ``` Similarly, here's `librustc_driver.so` when not requesting debuginfo from either rustc or the tools (nightly CI), coming e.g. from our LLVM wrapper: ``` $ readelf --debug-dump=info ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-e534b3a316089f5f.so | grep DW_TAG_compile_unit -A5 | grep DW_AT_comp_dir | cut -d ":" -f 2- | counts 116 counts ( 1) 34 (29.3%, 29.3%): (indirect string, offset: 0x3c11): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/libsupc++ ( 2) 32 (27.6%, 56.9%): (indirect string, offset: 0x9753c): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11 ( 3) 25 (21.6%, 78.4%): (indirect string, offset: 0x393bd): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++98 ( 4) 23 (19.8%, 98.3%): (indirect string, offset: 0x33ed3): /cargo/registry/src/index.crates.io-6f17d22bba15001f/compiler_builtins-0.1.98 ( 5) 1 ( 0.9%, 99.1%): (indirect string, offset: 0xaffff): /rustc/0d95f9132909ae7c5f2456748d0ffd1c3ba4a8e8 ( 6) 1 ( 0.9%,100.0%): (indirect string, offset: 0xb604a): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src ``` To reduce the size of distributed artifacts, this PR strips debuginfo from the LLVM and `rustc_driver` shared libraries, when: - no debuginfo is requested when building LLVM: `link-shared` is true, `optimize` is true and `release-debuginfo` is false - no debuginfo is requested when building the rustc driver: - `debuginfo-level-rustc` and `debuginfo-level-tools` are off. - when building with a stage != 0 compiler: since this is about the distributed artifacts, there's no need to do this at other stages. - for both: on a x64 linux host and target where `strip -g` is available and fixes the issue (I don't know how to strip debuginfo from a `.dylib` on mac). The LLVM BOLTed .so, and `librustc_driver.so` are big there, and this will help a little. Other targets/hosts can be added in the future if we want to. rust-lang#114175 did the same thing unconditionally in `opt-dist`, prior to BOLTing LLVM. But this should only be used in conjunction with the other config options mentioned above, and which `opt-dist` doesn't know about. Therefore, it makes more sense as in bootstrap when building LLVM and rustc when applicable and no debuginfo is requested. This shouldn't interact badly with CI caching builds and artifacts, right? --- From the other PR, `libLLVM-16-rust-1.73.0-nightly.so` prior to rust-lang#114141: - master: 173.13 MiB - stripped debuginfo: 165.12 MiB (-8 MiB, -4.6%) `libLLVM-16-rust-1.73.0-nightly.so` after rust-lang#114141: - master: 121.13 MiB - stripped debuginfo: 113.12 MiB (still -8 MiB, -6.6%) `librustc_driver.so`: - master: 118.58 MiB - stripped debuginfo: 106.46 MiB (-12 MiB, -10.2%) (Results are also available in this most recent [perf run's artifact sizes](https://perf.rust-lang.org/compare.html?start=b321edd1b2d4bd00c7b4611e8f20a03ee7b77023&end=810ab570d5d27facb91806e5d9847815d9dac22a&stat=instructions%3Au&tab=artifact-size))

…ikic Remove usage of `--use-old-text` for BOLT This flag has [reduced](rust-lang#114141) the size of `libLLVM.so` by ~50 MiB, but sadly it is quite non-deterministic and the size savings frequently fail, thus causing large artifact size [swings](rust-lang#114297 (comment)). To avoid the swings, it would be better to just disable the flag for now. r? `@nikic`

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jul 27, 2023

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 27, 2023

This comment has been minimized.

Sign in to view

the8472 reviewed Jul 27, 2023

View reviewed changes

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 28, 2023

Change LLVM BOLT flags

b996e5e

Kobzol force-pushed the llvm-bolt-flags branch from bc6b5fe to b996e5e Compare July 28, 2023 11:49

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 28, 2023

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 29, 2023

Kobzol marked this pull request as ready for review July 29, 2023 08:52

rustbot assigned clubby789 Jul 29, 2023

bors removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 29, 2023

bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Jul 29, 2023

bors added the merged-by-bors This PR was explicitly merged by bors. label Jul 29, 2023

bors merged commit f45961b into rust-lang:master Jul 29, 2023

rustbot added this to the 1.73.0 milestone Jul 29, 2023

Kobzol deleted the llvm-bolt-flags branch July 29, 2023 11:29

bors mentioned this pull request Jul 29, 2023

Move BOLT from bootstrap to opt-dist #113592

Merged

This was referenced Jul 30, 2023

Check if we can strip debuginfo prior to bolting libLLVM.so #114175

Closed

Allow stripping debuginfo from LLVM's shared library #114259

Closed

lqd mentioned this pull request Jul 31, 2023

Strip unexpected debuginfo from libLLVM.so and librustc_driver.so when not requesting any debuginfo #114305

Merged

klensy mentioned this pull request Aug 8, 2023

Update to LLVM 17 #114048

Merged

1 task

Kobzol mentioned this pull request Aug 9, 2023

Remove usage of --use-old-text for BOLT #114649

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change LLVM BOLT flags #114141

Change LLVM BOLT flags #114141

Kobzol commented Jul 27, 2023

Kobzol commented Jul 27, 2023

This comment has been minimized.

bors commented Jul 27, 2023

bors commented Jul 27, 2023

This comment has been minimized.

the8472 Jul 27, 2023

Kobzol Jul 28, 2023

rust-timer commented Jul 28, 2023

Kobzol commented Jul 28, 2023

Kobzol commented Jul 28, 2023

This comment has been minimized.

bors commented Jul 28, 2023

bors commented Jul 28, 2023

This comment has been minimized.

rust-timer commented Jul 29, 2023

Kobzol commented Jul 29, 2023

lqd commented Jul 29, 2023

bors commented Jul 29, 2023

Kobzol commented Jul 29, 2023

bors commented Jul 29, 2023

bors commented Jul 29, 2023

rust-timer commented Jul 29, 2023

Change LLVM BOLT flags #114141

Change LLVM BOLT flags #114141

Conversation

Kobzol commented Jul 27, 2023

Kobzol commented Jul 27, 2023

This comment has been minimized.

bors commented Jul 27, 2023

bors commented Jul 27, 2023

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rust-timer commented Jul 28, 2023

Overall result: ❌✅ regressions and improvements - no action needed

Kobzol commented Jul 28, 2023

Kobzol commented Jul 28, 2023

This comment has been minimized.

bors commented Jul 28, 2023

bors commented Jul 28, 2023

This comment has been minimized.

rust-timer commented Jul 29, 2023

Overall result: ✅ improvements - no action needed

Kobzol commented Jul 29, 2023

lqd commented Jul 29, 2023

bors commented Jul 29, 2023

Kobzol commented Jul 29, 2023

bors commented Jul 29, 2023

bors commented Jul 29, 2023

rust-timer commented Jul 29, 2023

Overall result: no relevant changes - no action needed