Can bzlmod pip.parse be inherited by other modules without loading python toolchain again?

[lucasmuna]

🐞 bug report <> help wanted

Affected Rule

The issue is caused by the rule: bzlmod pip.parse

Is this a regression?

I'm not sure, but probably related to this:

Issue: #1548
PR: #1549

Description

I'm adding bzlmod support for an internal repo that is loaded by other Bazel modules.

In this repo we have some rules, aspects, and python actions that should be executed using our own toolchain. In this regard, we're also switching from our own custom toolchain to the ones offered by rules_python.

Using the traditional WORKSPACE method we have no issues but when bzlmod is activated other Bazel modules cannot properly load and use our tools.

As additional info, our WORKSPACE.bzlmod is currently empty, we are offering only python3.9, and every python dependency comes from rules_python pip.

From what I read from the above Issue and PR this is somewhat intentional, but then I'm not sure how to proceed. Ideally I would want the end user (other module) to be able to inherit our python toolchain / pip requirements with bzlmod.

🔬 Minimal Reproduction

Not a really a minimal reproduction but a code snippet that might help:

# our_module/MODULE.bazel
...
bazel_dep(name = "rules_python", repo_name = "our_module_rules_python", version = "0.33.2")

python = use_extension("@our_module_rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    python_version = "3.9",
    configure_coverage_tool = True,
    is_default = True, # From what I understood this is only valid for this root
)

use_repo(python, "python_versions", "python_3_9") # Tested this ...

register_toolchains("@bazel_tools//tools/python:autodetecting_toolchain") # Tested this ...

pip = use_extension("@our_module_rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "our_module_pip",
    python_version = "3.9",
    requirements_lock = "//third_party/pip:requirements_lock.txt",
)
use_repo(pip, "our_module_pip")
...

# other_module/MODULE.bazel
...
bazel_dep(name = "other_module", version = "2.0.0")
local_path_override( # The real final user would actually load us through `http_archive` or `bazel_dep`
    module_name = "other_module",
    path = "../other_module",
)
...

Then a bazel build //... in the other_module is already enough to raise the error below.

🔥 Exception or Error

ERROR: /home/USER/.cache/bazel/_bazel_USER/69ebdc65b3da0709797efa3f05af60d8/external/rules_python~0.33.2~pip~our_module_pip/cvc5/BUILD.bazel:47:6: configurable attribute "actual" in @rules_python~0.33.2~pip~our_module_pip//cvc5:data doesn't match this configuration: No matching wheel for current configuration's Python version.

The current build configuration's Python version doesn't match any of the Python
wheels available for this wheel. This wheel supports the following Python
configuration settings:
    //_config:is_python_3.9

To determine the current configuration's Python version, run:
    `bazel config ` (shown further below)
and look for
    rules_python//python/config_settings:python_version

If the value is missing, then the "default" Python version is being used,
which has a "null" version value and will not match version constraints.


This instance of @rules_python~0.33.2~pip~our_module_pip//cvc5:data has configuration identifier db508b9. To inspect its configuration, run: bazel config db508b9.

For more help, see https://bazel.build/docs/configurable-attributes#faq-select-choose-condition.

🌍 Your Environment

Operating System:

  
Ubuntu 20.04.6 LTS | 5.15.0-113-generic GNU/Linux
  

Output of bazel version:

  
Bazelisk version: v1.11.0
Build label: 6.2.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue May 9 18:33:08 2023 (1683657188)
Build timestamp: 1683657188
Build timestamp as int: 1683657188
  

Rules_python version:

  
0.33.2
  

Anything else relevant?

If I register a python3.9 toolchain in the other_module it works. If I register, for example, python3.8, it raises the same error, as expected.

[aignas]

The problem is that you are not using transitions to ensure that your module is using the right python toolchain. Could you add snippets of BUILD.bazel files where the targets in the 'our_module' are consumed by the 'other_module'. 'our_module' should use the python version aware py_binary in order to ensure that the default toolchain version set by 'other_module' does not influnce the build configuration for the targets in 'our_module'.

So right now it is working as intended.

[lucasmuna]

Hey, thanks for looking into it.

Actually the use case is that our_module targets are not consumed as dependencies in other_module BUILD.bazel files but rather used from a build command with additional actions (from our_module aspects). So at maximum we create some custom configs for our aspect rules at other_module BUILD.bazel files.

---

other_module code snippets

BUILD.bazel custom aspect config:

load("@our_module//:defs.bzl", "aspect_config")

aspect_config(
    name = "custom_aspect_config",
    additional_flags = [],
    config_files = "@our_module//configs:config_1", # We usually offer a set of predefined configs as well
    default_feature = "awesome",
    visibility = ["//visibility:public"],
)

.bazelrc aspect config:

build:my_config --@our_module//:my_aspect_config=//:custom_aspect_config
build:my_config --output_groups=my_aspect_output
build:my_config --aspects=@our_module//:defs.bzl%my_aspect

And then we call it with, for example, bazel build --config=my_config //...

---

our_module code snippets

Usual python target used by our aspects:

load("@our_module_pip//:requirements.bzl", "requirement")
load("@our_module_rules_python//python:defs.bzl", "py_binary", "py_library")

py_binary(
    name = "runner", # This is then a default (private) attr of our aspects and is invoked by a `ctx.actions.run`
    srcs = ["runner.py"],
    visibility = ["//visibility:public"],
    deps = [":utils"],
)

py_library(
    name = "utils",
    srcs = ["utils.py"],
    visibility = ["//:__subpackages__"],
)

[aignas]

The problem is the second snippet, you should be declaring the py_binary in a similar way how the version specific py binaries are defined: https://github.com/bazelbuild/rules_python/blob/main/examples/bzlmod/tests/BUILD.bazel#L2

Or you should call pip.parse multiple times for each python version that your users may use to ensure that you have working dependency tree on any Python version.

[lucasmuna]

That did the trick, thanks @aignas.

If I may ask one more question, with workspace approach I was using a custom python toolchain name, but with bzlmod the name is directly derived from the python version. Is it possible to change it someway? If not, does it makes sense?

I found that it's hard coded here:
https://github.com/bazelbuild/rules_python/blob/084b877c98b580839ceab2b071b02fc6768f3de6/python/private/python.bzl#L96C13-L96C27

[aignas]

Why would you like to change that? With bzlmod the registration of the hermetic toolchains is happening in rules_python instead of the root module, so there is no need to have a custom name.

…

On 4 July 2024 23:23:43 GMT+09:00, Lucas Munaretto ***@***.***> wrote: That did the trick, thanks @aignas. If I may ask one more question, with workspace approach I was using a custom python toolchain name, but with bzlmod the name is directly derived from the python version. Is it possible to change it someway? If not, does it makes sense? I found that it's hard coded here: https://github.com/bazelbuild/rules_python/blob/084b877c98b580839ceab2b071b02fc6768f3de6/python/private/python.bzl#L96C13-L96C27 -- Reply to this email directly or view it on GitHub: #2029 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[lucasmuna]

Yes, for bzlmod itself there is no need for such thing. The idea would be to keep consistency between bzlmod and workspace. With workspace we usually have specialised names to avoid possible name conflicts inside big monorepos. Right now I basically modified the workspace toolchain name to match the bzlmod one.

[aignas]

The consistency between bzlmod IMHO is not needed, because you can enable bzlmod and still use the WORKSPACE file whilst you are migrating everything to bzlmod. We tried our best to make rules_python compatible when transitioning to bzlmod, but some things are different and toolchains are one of them.

You may have success in using:

# MODULE.bazel contents
use_repo(python, some_custom_name = "python_3_9")

But I am not sure if we can help you here anything beyond here.