rules_go is broken with Bazel@HEAD on Bazel CI

[sgowroji]

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2975#0187b16f-4ccc-4a47-b0b3-db74beb40640

Platform : rules_go (:ubuntu: 20.04 LTS (OpenJDK 11, gcc 9.4.0))

Buildkite Error logs:

--
  | java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=ConfiguredTargetKey{label=//tests/core/go_path:link_path, config=BuildConfigurationKey[2f0a347b1f73bf3f17deb468d4e100f97615469b3c118a69513684cc30601d99]}, actionIndex=2}' (requested by nodes 'TargetCompletionKey{topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@90904c3b, actionLookupKey=ConfiguredTargetKey{label=//tests/core/go_path:link_path, config=BuildConfigurationKey[2f0a347b1f73bf3f17deb468d4e100f97615469b3c118a69513684cc30601d99]}, willTest=false}', 'ArtifactNestedSetKey{rawChildren=[File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/cmd/bin/bin.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/pkg/lib/embed_test.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/pkg/lib/lib.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/pkg/lib/external_test.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/pkg/lib/internal_test.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/pkg/lib/embedded_src.txt, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/pkg/lib/renamed_embedded_src.txt, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/pkg/lib/template/index.html.tmpl, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/github.com/bazelbuild/rules_go/go/tools/bzltestutil/init.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/github.com/bazelbuild/rules_go/go/tools/bzltestutil/lcov.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/github.com/bazelbuild/rules_go/go/tools/bzltestutil/test2json.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/github.com/bazelbuild/rules_go/go/tools/bzltestutil/wrap.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/github.com/bazelbuild/rules_go/go/tools/bzltestutil/xml.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/pkg/lib/transitive/transitive.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/vendor/example.com/repo2/vendored.go, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/pkg/linux_amd64/example.com/repo/cmd/bin.a, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/pkg/plan9_arm/example.com/repo/cmd/bin.a, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/pkg/linux_amd64/example.com/repo/pkg/lib.a, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/pkg/linux_amd64/github.com/bazelbuild/rules_go/go/tools/bzltestutil.a, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/pkg/linux_amd64/example.com/repo/pkg/lib/transitive.a, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/pkg/linux_amd64/example.com/repo/vendor/example.com/repo2.a, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/pkg/lib/data.txt, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/src/example.com/repo/pkg/lib/testdata/testdata.txt, File:[[<execution_root>]bazel-out/k8-fastbuild/bin]tests/core/go_path/link_path/extra.txt]}', ...)
  | at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:591)
  | at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:399)
  | at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
  | at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
  | at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
  | at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
  | at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
  | at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
  | Caused by: java.lang.IllegalArgumentException: expected one element but was: <com.google.devtools.build.lib.remote.RemoteExecutionService$ActionResultMetadata$SymlinkMetadata@24e56aed, com.google.devtools.build.lib.remote.RemoteExecutionService$ActionResultMetadata$SymlinkMetadata@1cd5c588, com.google.devtools.build.lib.remote.RemoteExecutionService$ActionResultMetadata$SymlinkMetadata@7350f9b7, com.google.devtools.build.lib.remote.RemoteExecutionService$ActionResultMetadata$SymlinkMetadata@d9988a4, com.google.devtools.build.lib.remote.RemoteExecutionService$ActionResultMetadata$SymlinkMetadata@2a439f8, ...>   

We are unable to reproduce the above error as shown in buildkite

Steps followed :

1. git clone -v https://github.com/bazelbuild/rules_go.git
2. git reset 2a71311 --hard
3. export USE_BAZEL_VERSION=bbf78a2cec27bf42bcbba59d3f84c857dff3aa15
4. bazel build //...

CC @meteorcloudy

[fmeum]

I broke this in bazelbuild/bazel@ca30372, will send a fix that also needs to be cherry-picked into 6.2.0.

CC @coeuvre

[fmeum]

I submitted bazelbuild/bazel#18207.

[sgowroji]

Hi @fmeum, rules_go downstream still failing https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2982#0187bbbc-8709-466c-832b-b7c96ceab525. , I see the above attached PR is submitted. Could you please check and let us know.

[fmeum]

@sgowroji Based on the error message this does seem to be the same error, perhaps it didn't pick up the fix yet? We should check again tomorrow.

[meteorcloudy]

Hi @fmeum, rules_go downstream still failing https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2982#0187bbbc-8709-466c-832b-b7c96ceab525. , I see the above attached PR is submitted. Could you please check and let us know.

@sgowroji https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2982 built Bazel at bazelbuild/bazel@a92ce14, but the fix was submitted at bazelbuild/bazel@519eef4 which is a later commit.

[fmeum]

Today's build is green: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2995