Ignore all #[test]-related code automatically

[Swatinem]

UPDATE2(taiki-e):

See #123 (comment)

---

UPDATE(taiki-e):

If you want to ignore all #[test]-related code, consider using coverage-helper crate version 0.2+.

cargo-llvm-cov excludes code contained in the directory named tests from the report by default, so you can also use it instead of coverage-helper crate.

https://github.com/taiki-e/cargo-llvm-cov#exclude-function-from-coverage

---

AFAIK this is not currently possible (though I would love to be proven wrong), so just filing this as a nice-to-have tracking issue.

What I would like is that only library code counts towards coverage. Having all of the #[test] and #[cfg(test)] code (fn and whole utility modules) count towards coverage metrics right now skews those numbers up quite a bit.

Right now cargo-llvm-cov automatically ignores all the examples and tests code via -ignore-filename-regex but unit test functions are still counted (and possibly doctests once those are being fixed [tentative fix in #122]).

[taiki-e]

The idea that probably works here is a helper attribute that generates #[cfg_attr(coverage, no_coverage)].

#[cfg(coverage)]
use coverage_helper::test;

#[test]
fn test() {
}

generated:

#[cfg(coverage)]
use coverage_helper::test;

#[cfg_attr(coverage, no_coverage)]
#[::core::prelude::v1::test]
fn test() {
}

UPDATE: this idea is now implemented as https://github.com/taiki-e/coverage-helper.

[scouten-adobe]

@taiki-e I'm not an expert on attribute macros. Would you (or anyone) be kind enough to translate this into code that we could implement?

Adding the no_coverage attribute in front of every test seems unsustainable.

[taiki-e]

Above my approach is now implemented as https://github.com/taiki-e/coverage-helper.

[scouten-adobe]

I appreciate the coverage-helper crate, but since the #[no_coverage] attribute is not yet stabilized, I've decided not to use this approach in my project.

For those following along, if you want to exclude test-related code from coverage and base your CI builds on stable Rust versions, there's a more drastic approach that does seem to solve the problem. See adobe/xmp-toolkit-rs#66.

[taiki-e]

I'm surprised myself that I (probably) haven't documented this EDIT: #123 (comment), but a way that doesn't depend on the unstable features that have worked since the earliest versions is to put unit tests in a directory named tests.

My projects over the past few years have adopted a directory structure that obviously has this in mind.

• https://github.com/taiki-e/portable-atomic/tree/main/src/tests
• https://github.com/taiki-e/parse-changelog/tree/main/src/tests
• https://github.com/taiki-e/dependabot-config/tree/main/src/tests
• https://github.com/taiki-e/atomic-maybe-uninit/tree/main/src/tests

[taiki-e]

I'm surprised myself that I (probably) haven't documented this

Nah, It's already documented 10 months ago.

cargo-llvm-cov excludes code contained in the directory named tests from the report by default, so you can also use it instead of coverage-helper crate.

[wyfo]

Would it make sense to exclude tests.rs files also? If the test module is small, user may prefer having src/tests.rs instead of src/tests/mod.rs.

[taiki-e]

exclude tests.rs files

Seems reasonable to me.

[taiki-e]

In the recent rustc, it seems the module-level coverage(off) is supported:

#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))] // <---
mod tests {
   // ...
}

(Perhaps coverage-helper library should be deprecated for most cases in favor of the module-level coverage(off) ...)

[Zalathar]

Oh yes, that was rust-lang/rust#126721.