Code coverage errors on ls tests

[chadbrewbaker]

The ls tests dislike llvm source based code coverage.

The CI is currently using the grcov crate? It would be nice if default environment variables were given in comments.

coreutils/.github/workflows/CICD.yml

Line 621 in a017c1b

grcov . --output-type files --ignore build.rs --ignore "/*" --ignore "[a-zA-Z]:/*" --excl-br-line "^\s*((debug_)?assert(_eq|_ne)?!|#\[derive\()" | sort --unique

rustc 1.54.0-nightly (c1e8f3a58 2021-05-30) macOS 11.4 aarch64 on commit a017c1b

RUSTFLAGS="-Zinstrument-coverage" cargo test

thread 'test_ls::test_ls_across' panicked at 'assertion failed: `(left == right)`

Diff < left / right > :
<"default.profraw\ntest-across-1\ntest-across-2\ntest-across-3\ntest-across-4\n"
>"test-across-1\ntest-across-2\ntest-across-3\ntest-across-4\n"

Options in decreasing order of preference:

• Get the coverage written to a temp directory instead of a local directory. I think this is what the %t syntax is for? But I can't figure out how to use it. This is required as the test suite cleans up files it writes to local directories?
• Use the grcov crate just like the CI.
• Exclude ls tests under code coverage.
• Muck with ls tests to strip the profile file.

[chadbrewbaker]

I got this to generate a report, but coverage seems to be missing for many tests.

cargo clean
# https://github.com/mozilla/grcov/issues/555 didn't make the latest release
cargo install --git https://github.com/mozilla/grcov
RUSTFLAGS="-Zinstrument-coverage" LLVM_PROFILE_FILE="your_name-%p-%m.profraw" cargo test
grcov . -s . --binary-path ./target/debug/ -t html --branch --ignore-not-existing  --ignore build.rs --ignore "/*" --ignore "[a-zA-Z]:/*" -o ./target/debug/coverage/

[marco-c]

I tested source-based coverage a while ago on the project, here's the patch I used:

diff --git a/util/build-code_coverage.sh b/util/build-code_coverage.sh
index 7ad3165f..914cd0ba 100644
--- a/util/build-code_coverage.sh
+++ b/util/build-code_coverage.sh
@@ -26,31 +26,31 @@ done
 
 export CARGO_INCREMENTAL=0
 export RUSTC_WRAPPER=""     ## NOTE: RUSTC_WRAPPER=='sccache' breaks code coverage calculations (uu_*.gcno files are not created during build)
-# export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zno-landing-pads"
-export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"
-export RUSTDOCFLAGS="-Cpanic=abort"
-export RUSTUP_TOOLCHAIN="nightly-gnu"
-cargo build ${FEATURES_OPTION}
-cargo test --no-run ${FEATURES_OPTION}
-cargo test --quiet ${FEATURES_OPTION}
-cargo test --quiet ${FEATURES_OPTION} ${CARGO_INDIVIDUAL_PACKAGE_OPTIONS}
+export RUSTFLAGS="-Zinstrument-coverage"
 
 export COVERAGE_REPORT_DIR
 if [ -z "${COVERAGE_REPORT_DIR}" ]; then COVERAGE_REPORT_DIR="${REPO_main_dir}/target/debug/coverage-nix"; fi
 rm -r "${COVERAGE_REPORT_DIR}" 2>/dev/null
 mkdir -p "${COVERAGE_REPORT_DIR}"
 
+export LLVM_PROFILE_FILE="${COVERAGE_REPORT_DIR}/coreutils-%p-%m.profraw"
+
+cargo build ${FEATURES_OPTION}
+cargo test --no-run ${FEATURES_OPTION}
+cargo test --quiet ${FEATURES_OPTION}
+cargo test --quiet ${FEATURES_OPTION} ${CARGO_INDIVIDUAL_PACKAGE_OPTIONS}
+
 ## NOTE: `grcov` is not accepting environment variable contents as options for `--ignore` or `--excl_br_line`
 # export GRCOV_IGNORE_OPTION="--ignore build.rs --ignore '/*' --ignore '[A-Za-z]:/*' --ignore 'C:/Users/*'"
 # export GRCOV_EXCLUDE_OPTION="--excl-br-line '^\s*((debug_)?assert(_eq|_ne)?!|#\[derive\()'"
 # * build LCOV coverage file
-grcov . --output-type lcov --output-path "${COVERAGE_REPORT_DIR}/../lcov.info" --branch --ignore build.rs --ignore '/*' --ignore '[A-Za-z]:/*' --ignore 'C:/Users/*' --excl-br-line '^\s*((debug_)?assert(_eq|_ne)?!|#\[derive\()'
+grcov . --binary-path ./target/debug/ --output-type lcov --output-path "${COVERAGE_REPORT_DIR}/../lcov.info" --branch --ignore build.rs --ignore '/*' --ignore '[A-Za-z]:/*' --ignore 'C:/Users/*'
 # * build HTML
 # -- use `genhtml` if available for display of additional branch coverage information
 genhtml --version 2>/dev/null 1>&2
 if [ $? -eq 0 ]; then
     genhtml "${COVERAGE_REPORT_DIR}/../lcov.info" --output-directory "${COVERAGE_REPORT_DIR}" --branch-coverage --function-coverage | grep ": [0-9]"
 else
-    grcov . --output-type html --output-path "${COVERAGE_REPORT_DIR}" --branch --ignore build.rs --ignore '/*' --ignore '[A-Za-z]:/*' --ignore 'C:/Users/*' --excl-br-line '^\s*((debug_)?assert(_eq|_ne)?!|#\[derive\()'
+    grcov . --binary-path ./target/debug/ --output-type html --output-path "${COVERAGE_REPORT_DIR}" --branch --ignore build.rs --ignore '/*' --ignore '[A-Za-z]:/*' --ignore 'C:/Users/*'
 fi
 if [ $? -ne 0 ]; then exit 1 ; fi

[tertsdiepraam]

@marco-c Does that fix this issue? If so, it would be great if you could open a PR!

[marco-c]

I don't know, since I'm not familiar with the project, I don't know what the coverage is supposed to be.
Perhaps @chadbrewbaker can try it out and report back.

[miDeb]

I attempted to move CI to source-based source-based (hoping it would fix the unreproducible stack overflow in #2418), but I only managed to get it to work on ubuntu.
My attempt can be found here: https://github.com/miDeb/coreutils/blob/maint/source-based-coverage/.github/workflows/CICD.yml. I removed all other workflows from it to make it simpler.
On windows, I get

[ERROR] A panic occurred at C:\Users\runneradmin.cargo\registry\src\github.com-1ecc6299db9ec823\grcov-0.8.0\src\producer.rs:555: No input files found

(https://github.com/miDeb/coreutils/runs/2877914439?check_suite_focus=true)
On macos, I get

[ERROR] Error while executing llvm tools: Failed to execute "/Users/runner/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/bin/llvm-profdata" "merge" "-sparse" Argument list too long (os error 7)

(https://github.com/miDeb/coreutils/runs/2878039679?check_suite_focus=true)

I don't know what's going wrong.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.