NativeLink flake module should make --remote_upload_local_results configurable #1371
Assignees
Labels
Comments
aaronmondal
added
bug
aaronmondal
added a commit
to aaronmondal/nativelink
that referenced
this issue
Sep 27, 2024
…#1369)" This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371
aaronmondal
added a commit
to aaronmondal/nativelink
that referenced
this issue
Sep 29, 2024
* Revert "Allow nativelink flake module to upload results (TraceMachina#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371 * Introduce reproducible branch-based coverage Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
aaronmondal
added a commit
to aaronmondal/nativelink
that referenced
this issue
Sep 30, 2024
* Revert "Allow nativelink flake module to upload results (TraceMachina#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371 * Introduce reproducible branch-based coverage Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
aaronmondal
added a commit
to aaronmondal/nativelink
that referenced
this issue
Sep 30, 2024
* Revert "Allow nativelink flake module to upload results (TraceMachina#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371 * Introduce reproducible branch-based coverage Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
aaronmondal
added a commit
to aaronmondal/nativelink
that referenced
this issue
Sep 30, 2024
* Revert "Allow nativelink flake module to upload results (TraceMachina#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371 * Introduce reproducible branch-based coverage Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
aaronmondal
added a commit
to aaronmondal/nativelink
that referenced
this issue
Sep 30, 2024
* Revert "Allow nativelink flake module to upload results (TraceMachina#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371 * Introduce reproducible branch-based coverage Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
aaronmondal
added a commit
to aaronmondal/nativelink
that referenced
this issue
Sep 30, 2024
* Revert "Allow nativelink flake module to upload results (TraceMachina#1369)" (TraceMachina#1372) This partially reverts commit 9600839. The original idea was to implement a `readonly` setting to dynamically configure `--remote_upload_local_results`. While this is fairly straightforward to implement it turned out that the UX implications around the environment variables/scripts/files to configure this are nontrivial and we need to evaluate the different approaches in more depth first. For now, revert to readonly by default and also add a small modifier to the relevant workflow so that the write-access workflow retains the ability to write artifacts on pushes to main. Fixes TraceMachina#1371 * Introduce reproducible branch-based coverage Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix. As next steps we should continuously publish the generated HTML pages via the web infrastructure and add hooks to report regressions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Otherwise users running with read-only keys will get confusing errors about failed uploads.
Currently, a workaround is to set
--remote_upload_local_results=falsemanually for local builds.The text was updated successfully, but these errors were encountered: