Add a cargo option to generate coverage reports using -Z instrument-coverage

[richkadel]

The Rust unstable book includes some examples of the workflow for generating LLVM coverage reports using -Z instrument-coverage.

Given the flexibility of this feature, having these detailed examples is useful, but there should be an "easy button" for generating coverage reports for the more common use cases. (I have some personal bash scripts to simplify the process during development, for example.) cargo can provide a great foundation for cross platform automation of Rust coverage analysis.

Some loose requirements, off the top of my head, include:

• Enable coverage analysis of both cargo run and cargo test options. (Enabling the coverage report workflow will, therefore, probably be a flag option to these commands, versus a separate command, I think.)
• Automatically configure LLVM_PROFILE_FILE, with the right filename patterns for filename deconfliction, and a path to a coverage-specific subdirectory of the target directory.
• Automatically delete old profraw, profdata, and report data, before starting another coverage analysis (unless the user provides a flag option to retain old data).
• Enable all of the features shown in the unstable book documentation, including test coverage (for example, if doc tests are included, the workflow should automatically enable --persist-doctests, and the workflow should automatically include the right test binaries when running the llvm-cov command).
• There should be a way to include llvm-cov options in the cargo command.
• There should be a default llvm-cov option configuration that generates a coverage report with minimal cargo flags, and perhaps a few easy flags for other common configurations (for example, generating results as text, with ANSI terminal color highlights or not, or generating HTML, with files saved in the coverage target subdirectory with associated profraw/profdata files).
• There should be a way to re-run the coverage reports with different options without re-running the binaries/tests.
  • If possible, use timestamps and dependencies, and automatically re-use old results if the program/tests were not re-built by cargo

Longer term:

• There should be a way to generate reports from multiple binary/test runs, by merging the profraw data from multiple executions (being careful not to automatically delete data from prior runs when the user requests this feature).

[richkadel]

Added to the tracking issue (#79121) for -Z instrument-coverage stabilization.

[chadbrewbaker]

cargo test -Ztimings is also problematic. It would be nice to shim in a prefix script like /usr/bin/time -v to get memory usage; I'm working on a Rust port of /usr/bin/time -v that emits JSON/CSV/PythonDict for easy parsing.

Debug symbols for production call stack profiling is another sore point. https://pyroscope.io/ uses https://github.com/iovisor/bcc/blob/master/tools/profile.py under the hood. This script needs modified to properly annotate Rust call stacks. Is cargo build able to emit debug symbol files as part of the production build? https://wiki.ubuntu.com/Debug%20Symbol%20Packages

Ideally one could get a coverage report from the production call stack traces too.

[inquisitivecrystal]

@rustbot label A-code-coverage C-feature-request T-cargo

[jhpratt]

Highly related: cargo llvm-cov