Large-scale correctness and performance tests with crater

[petr-tik]

Thanks for creating mold and making it so easy to use with rust and cargo.

I have been running it on my personal machine and can report anecdata about the speedup.

Now that you have AWS resources
https://twitter.com/rui314/status/1517780094215127040
and want to run comprehensive tests of linking as many packages with mold

I ask you to consider allocating some of those resources to crater runs to perform performance and correctness tests
on a large set of rust packages and test executables.

https://github.com/rust-lang/crater

It's widely used by rustc developers to check if upcoming rust compiler changes break
a subset of existing crates from crates.io by building and running the test suite.

the consistency of the cargo test interface will enable you to get thorough feedback about mold's performance and correctness when comparing link times and test results between mold-linked and standard-linked test executables on Linux.

Relevant documentation

Though I haven't used it myself though.

Setting up an agent on an AWS instance
https://github.com/rust-lang/crater/blob/master/docs/agent-machine-setup.md

Running the agent on the command line including generating an HTML report
https://github.com/rust-lang/crater/blob/master/docs/cli-usage.md

Example output

Example report here:
https://crater-reports.s3.amazonaws.com/pr-94775/index.html
as seen here
rust-lang/rust#94775 (comment)
with files available for download and local analysis
https://crater-reports.s3.amazonaws.com/pr-94775/downloads.html

Suggested experiments

Build all executables and all tests with mold-enabled and standard linux toolchains.

• Check all tests results match
• Extract link timings from the reports and do statistical analysis

[rui314]

Currently I'm using Gentoo as a test suite. We build all Gentoo packages from build and run their tests before making a release. Gentoo includes various Rust packages, I'm not too worried about a possible breakage that can only be found in Rust.

It looks to me that Crater is focused on finding compatibility issues and regressions in newer Rust compiler themselves. I'm not sure if we can find new issues that we couldn't find with Gentoo.

As to the performance measurement, it would be very hard to isolate the performance of the linker from the entire builds. If you build a lot of crates from scratch, I believe rustc dominates, and the linker consumes a relatively small amount of time. Besides that, since builds are running in parallel, if a single invocation of the linker seems too slow, it may be caused by other processes running in background. For benchmarking, we need a silent machine and run only the linker on that machine.