[Bazel CI] IllegalStateException at RepositoryDirectoryValue$NoRepositoryDirectoryValue.getPath()

[sgowroji]

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4254#019309f7-4e6d-48ec-bc59-50c384edbd54

Platform: Ubuntu

Logs:

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'Key{moduleKey=rules_python@_, override=LocalPathOverride{path=..}}' (requested by nodes 'com.google.devtools.build.lib.bazel.bzlmod.BazelModuleResolutionValue$$Lambda/0x00000008005384d8@55549bc1')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:548)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.IllegalStateException
	at com.google.devtools.build.lib.rules.repository.RepositoryDirectoryValue$NoRepositoryDirectoryValue.getPath(RepositoryDirectoryValue.java:164)
	at com.google.devtools.build.lib.bazel.bzlmod.ModuleFileFunction.getModuleFile(ModuleFileFunction.java:568)
	at com.google.devtools.build.lib.bazel.bzlmod.ModuleFileFunction.compute(ModuleFileFunction.java:166)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:468)
	... 7 more
bazel build failed with exit code 37

Culprit:

Steps:

git clone https://github.com/bazelbuild/rules_python
git reset 772b1cc0a9fd8c6e36be08b0c98e2ad53673fb73  --hard
export USE_BAZEL_VERSION=1c47c9d412b29f7ba42c190e0b211a9438a81d59
bazel build --noenable_bzlmod --enable_workspace //... 

CC Greenteam @mai93

[meteorcloudy]

/cc @Wyverald

[meteorcloudy]

Hmm, I'm getting this error instead:

root@faa221425474:/workdri/rules_python/gazelle# bazel build --noenable_bzlmod --enable_workspace //...
Starting local Bazel server and connecting to it...
WARNING: WORKSPACE support will be removed in Bazel 9 (late 2025), please migrate to Bzlmod, see https://bazel.build/external/migration.
WARNING: Couldn't auto load rules or symbols, because no dependency on module/repository 'protobuf' found. This will result in a failure if there's a reference to those rules or symbols.
WARNING: Couldn't auto load rules or symbols, because no dependency on module/repository 'rules_android' found. This will result in a failure if there's a reference to those rules or symbols.
WARNING: Couldn't auto load rules or symbols, because no dependency on module/repository 'rules_java' found. This will result in a failure if there's a reference to those rules or symbols.
WARNING: Couldn't auto load rules or symbols, because no dependency on module/repository 'rules_python' found. This will result in a failure if there's a reference to those rules or symbols.
WARNING: Couldn't auto load rules or symbols, because no dependency on module/repository 'rules_shell' found. This will result in a failure if there's a reference to those rules or symbols.
ERROR: Traceback (most recent call last):
	File "/root/.cache/bazel/_bazel_root/848c7d8a2d908bfe9fa8fb3b30cacaf8/external/protobuf/bazel/cc_proto_library.bzl", line 3, column 26, in <toplevel>
		cc_proto_library = native.cc_proto_library
Error: no native function or rule 'cc_proto_library'
Available attributes: action_listener, alias, available_xcodes, cc_binary, cc_import, cc_libc_top_alias, cc_library, cc_shared_library, cc_static_library, cc_test, cc_toolchain, cc_toolchain_alias, cc_toolchain_suite, config_feature_flag, config_setting, constraint_setting, constraint_value, environment, existing_rule, existing_rules, exports_files, extra_action, fdo_prefetch_hints, fdo_profile, filegroup, genquery, genrule, glob, java_binary, java_import, java_library, java_package_configuration, java_plugin, java_plugins_flag_alias, java_runtime, java_test, java_toolchain, label_flag, label_setting, legacy_globals, memprof_profile, module_name, module_version, objc_import, objc_library, package, package_group, package_name, package_relative_label, platform, propeller_optimize, repo_name, repository_name, starlark_doc_extract, subpackages, test_suite, toolchain, toolchain_type, xcode_config, xcode_config_alias, xcode_version
WARNING: Target pattern parsing failed.
ERROR: Skipping '//...': error loading package under directory '': error loading package 'modules_mapping': at /root/.cache/bazel/_bazel_root/848c7d8a2d908bfe9fa8fb3b30cacaf8/external/rules_python/python/defs.bzl:17:6: at /root/.cache/bazel/_bazel_root/848c7d8a2d908bfe9fa8fb3b30cacaf8/external/rules_python/python/py_binary.bzl:18:6: at /root/.cache/bazel/_bazel_root/848c7d8a2d908bfe9fa8fb3b30cacaf8/external/rules_python/python/private/py_binary_macro.bzl:16:6: at /root/.cache/bazel/_bazel_root/848c7d8a2d908bfe9fa8fb3b30cacaf8/external/rules_python/python/private/common_bazel.bzl:18:6: at /root/.cache/bazel/_bazel_root/848c7d8a2d908bfe9fa8fb3b30cacaf8/external/rules_cc/cc/defs.bzl:16:6: initialization of module 'bazel/cc_proto_library.bzl' failed
ERROR: error loading package under directory '': error loading package 'modules_mapping': at /root/.cache/bazel/_bazel_root/848c7d8a2d908bfe9fa8fb3b30cacaf8/external/rules_python/python/defs.bzl:17:6: at /root/.cache/bazel/_bazel_root/848c7d8a2d908bfe9fa8fb3b30cacaf8/external/rules_python/python/py_binary.bzl:18:6: at /root/.cache/bazel/_bazel_root/848c7d8a2d908bfe9fa8fb3b30cacaf8/external/rules_python/python/private/py_binary_macro.bzl:16:6: at /root/.cache/bazel/_bazel_root/848c7d8a2d908bfe9fa8fb3b30cacaf8/external/rules_python/python/private/common_bazel.bzl:18:6: at /root/.cache/bazel/_bazel_root/848c7d8a2d908bfe9fa8fb3b30cacaf8/external/rules_cc/cc/defs.bzl:16:6: initialization of module 'bazel/cc_proto_library.bzl' failed
INFO: Elapsed time: 7.141s
INFO: 0 processes.
ERROR: Build did NOT complete successfully

[Wyverald]

I get the same error as Yun. The original exception does look very similar to another case that @ahumesky reported for rules_android, but that one is also no longer repro-able due to a separate error on Bazel HEAD...

[rickeylev]

I ran into this today, too, with rolling Bazel. We have a couple small integration tests with rolling bazel already, so I'm surprised it didn't show up sooner -- maybe just because rolling bazel got released over the weekend?

I think the issue is an older version of protobuf is being used, which doesn't have the cc_proto_library replacement. The fix will be to update the protobuf version.

#2379 should fix it for bzlmod.

I tried simply upgrading protobuf for WORKSPACE builds in that PR, too, but it caused failures. I think some additional workspace setup is necessary with the newer protobuf version, but I haven't figured out what, exactly. (I love that it Just Worked with bzlmod in comparison!)

#2293 may also fix this (that PR is blocked by 2379); we'll have to see once that is submitted.

[meteorcloudy]

The original IllegalStateException was a Bazel bug, will be fixed by cl/695680077