Coverage html #46
Merged
Coverage html #46
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371
Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
aaronmondal
had a problem deploying
to
production
GitHub Actions
Error
aaronmondal
had a problem deploying
to
production
GitHub Actions
Error
aaronmondal
had a problem deploying
to
production
GitHub Actions
Error
aaronmondal
added a commit
that referenced
this pull request
Sep 30, 2024
* Revert "Allow nativelink flake module to upload results (TraceMachina#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371 * Introduce reproducible branch-based coverage Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
aaronmondal
added a commit
that referenced
this pull request
Sep 30, 2024
* Revert "Allow nativelink flake module to upload results (TraceMachina#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371 * Introduce reproducible branch-based coverage Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
aaronmondal
added a commit
that referenced
this pull request
Sep 30, 2024
* Revert "Allow nativelink flake module to upload results (TraceMachina#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371 * Introduce reproducible branch-based coverage Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
aaronmondal
added a commit
that referenced
this pull request
Sep 30, 2024
* Revert "Allow nativelink flake module to upload results (TraceMachina#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371 * Introduce reproducible branch-based coverage Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
aaronmondal
added a commit
that referenced
this pull request
Sep 30, 2024
* Revert "Allow nativelink flake module to upload results (TraceMachina#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371 * Introduce reproducible branch-based coverage Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.