Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pypi] Support pulling dependencies using direct url specifiers without using an index #2363

Open
mering opened this issue Oct 31, 2024 · 3 comments

Comments

@mering
Copy link

mering commented Oct 31, 2024

🐞 bug report

Affected Rule

Module extension:

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(...)

Is this a regression?

Not sure.

Description

The repository rule for pip dependencies doesn't use repository_ctx.download(). This breaks usages of --experimental_downloader_config for example to use a transparent dependency mirror.

🔬 Minimal Reproduction

repo_utils.execute_checked(
rctx,
# truncate the requirement value when logging it / reporting
# progress since it may contain several ' --hash=sha256:...
# --hash=sha256:...' substrings that fill up the console
op = op_tmpl.format(name = rctx.attr.name, requirement = rctx.attr.requirement.split(" ", 1)[0]),
arguments = args,
environment = environment,
quiet = rctx.attr.quiet,
timeout = rctx.attr.timeout,
logger = logger,
)

pip_args = (
[sys.executable, "-m", "pip"]
+ (["--isolated"] if args.isolated else [])
+ (["download", "--only-binary=:all:"] if args.download_only else ["wheel"])
+ ["--no-deps"]
+ deserialized_args["extra_pip_args"]
)
requirement_file = NamedTemporaryFile(mode="wb", delete=False)
try:
requirement_file.write(args.requirement.encode("utf-8"))
requirement_file.flush()
# Close the file so pip is allowed to read it when running on Windows.
# For more information, see: https://bugs.python.org/issue14243
requirement_file.close()
# Requirement specific args like --hash can only be passed in a requirements file,
# so write our single requirement into a temp file in case it has any of those flags.
pip_args.extend(["-r", requirement_file.name])
env = os.environ.copy()
env.update(deserialized_args["environment"])
# Assumes any errors are logged by pip so do nothing. This command will fail if pip fails
subprocess.run(pip_args, check=True, env=env)

🔥 Exception or Error

Stack trace prints to the code linked above.

🌍 Your Environment

Operating System:

Linux

Output of bazel version:

7.0.2

Rules_python version:

0.37.2

Anything else relevant?

@aignas
Copy link
Collaborator

aignas commented Nov 3, 2024

Please see the documentation here: https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#bazel-downloader-and-multi-platform-wheel-hub-repository.

This currently has a limitation that it needs hashes for all of the files in the requirements.txt. The stabilization of this feature is tracked by #260.

@aignas aignas closed this as not planned Won't fix, can't repro, duplicate, stale Nov 3, 2024
@mering
Copy link
Author

mering commented Nov 4, 2024

@aignas How can I use experimental_index_url in combination with dependencies without an index like the following:

proto-breaking-change-detector @ https://github.com/googleapis/proto-breaking-change-detector/archive/refs/tags/v2.4.0.tar.gz
tensorflow_graphics @ git+https://github.com/tensorflow/graphics@ca63cf788978db0bde2db4ce488e102c455da2e7

@aignas
Copy link
Collaborator

aignas commented Nov 14, 2024

The support for direct references has not been added yet, I'll reopen this issue and rename it to scope it to supporting it, but I am not sure when I can work on it.

proto-breaking-change-detector @ https://github.com/googleapis/proto-breaking-change-detector/archive/refs/tags/v2.4.0.tar.gz could be easy enough. Outline of the changes:

For the second one it may be harder - I am not sure how to handle git repos yet, however, with bazel 7.4 the extension can pass a labels within the extension more easily, so one could look into how to do that.

For now the workaround would be to build the wheels from sdists outside bazel and host them in some PyPI compatible artifact mirror and then use experimental_index_url flag to enable the usage of the bazel-downloader.

@aignas aignas reopened this Nov 14, 2024
@aignas aignas changed the title pip rules not using rctx.download() [pypi] Support pulling dependencies without using an index Nov 14, 2024
@aignas aignas changed the title [pypi] Support pulling dependencies without using an index [pypi] Support pulling dependencies using direct url specifiers without using an index Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants