Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: py_pex_binary-rule fails to include single-file module pypi-deps #391

Open
ekfeet opened this issue Sep 9, 2024 · 2 comments
Open
Labels
bug Something isn't working

Comments

@ekfeet
Copy link
Contributor

ekfeet commented Sep 9, 2024

What happened?

Hello,

I am trying to use the relatively new py_pex_binary-rule, but the generated pex fails when I try to run it. The error I get:

Examined the following interpreters:
1.) /usr/bin/python3.11 CPython==3.11.2

No interpreter compatible with the requested constraints was found:

  Failed to resolve requirements from PEX environment @ /home/[usr]/.pex/unzipped_pexes/6b1ca5c199fa0394c40fdcd89179767251885ff9.
  Needed cp311-cp311-manylinux_2_36_x86_64 compatible dependencies for:
   1: six
      Required by:
        isodate 0.6.1
      But this pex had no ProjectName(raw='six', validated=False, normalized='six') distributions.

I've seen this issue when depending on packages that depends on six and typing-extensions. The regular binary runs just fine.

The .deps/ in the unzipped pex only shows isodate:

ls ~/.pex/unzipped_pexes/6b1ca5c199fa0394c40fdcd89179767251885ff9/.deps/
isodate-0.6.1

The metadata of isodate shows that six is a dependency

rg six ~/.pex/unzipped_pexes/6b1ca5c199fa0394c40fdcd89179767251885ff9/.deps/isodate-0.6.1/isodate-0.6.1.dist-info/
/home/[usr]/.pex/unzipped_pexes/6b1ca5c199fa0394c40fdcd89179767251885ff9/.deps/isodate-0.6.1/isodate-0.6.1.dist-info/METADATA
26:Requires-Dist: six

Any help is much appreciated!

Version

Development (host) and target OS/architectures: Debian GNU/Linux 12/x86_64

Output of bazel --version: bazel 7.2.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

# https://github.com/bazelbuild/rules_python/releases/
bazel_dep(name = "rules_python", version = "0.35.0")

# https://github.com/aspect-build/rules_py/releases
bazel_dep(name = "aspect_rules_py", version = "0.8.0")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    configure_coverage_tool = True,
    python_version = "3.11",
)

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "pypi",
    python_version = "3.11",
    requirements_lock = "//:requirements_lock.txt",
)
use_repo(pip, "pypi")

Language(s) and/or frameworks involved:
rules_py and python

How to reproduce

My build file:

load("@aspect_rules_py//py:defs.bzl", "py_binary", "py_pex_binary")
load("@rules_python//python:pip.bzl", "compile_pip_requirements")

compile_pip_requirements(
    name = "requirements",
    src = "requirements.in",
    requirements_txt = "requirements_lock.txt",
)

py_binary(
    name = "main",
    srcs = ["main.py"],
    deps = ["@pypi//isodate"]
)

py_pex_binary(
    name = "main_pex",
    binary = ":main"
)

main.py:

import isodate

def main():
    print(isodate)

if __name__ == "__main__":
    main()

requirements.in:

isodate>=0.6.1

Can also see this gist for all the files and run bazel run //:requirements.update and then bazel run //:main_pex

Any other information?

To me it looks like six is not added to runfiles nor deps because of the site-packages structure:

ls bazel-reppro_pex_err/external/rules_python~~pip~pypi_311_six/site-packages/
__init__.py  six-1.16.0.dist-info  six.py

which won't match any of the conditions in _map_srcs in py/private/py_pex_binary.bzl. typing-extensions has the same site-packages structure. The gist above also contains a hack to remedy the problem to show where I think the issue is.

@ekfeet ekfeet added the bug Something isn't working label Sep 9, 2024
@ekfeet ekfeet changed the title [Bug]: py_pex_binary-target fails to resolve interpreter requirements [Bug]: py_pex_binary-rule fails to include single-module pypi-deps Sep 10, 2024
@ekfeet ekfeet changed the title [Bug]: py_pex_binary-rule fails to include single-module pypi-deps [Bug]: py_pex_binary-rule fails to include single-file module pypi-deps Sep 10, 2024
@tradergt
Copy link

tradergt commented Sep 19, 2024

I think this is similar but was going convert from zip to pex and got this:


  Failed to resolve requirements from PEX environment @ /home/tradergt/.pex/unzipped_pexes/8853a88f438bd541ae60f0f3cc84bba2b373decf.
  Needed cp312-cp312-manylinux_2_36_aarch64 compatible dependencies for:
   1: typing-extensions>=4.6; python_version < "3.13"
      Required by:
        psycopg 3.2.2
      But this pex had no ProjectName(raw='typing-extensions', validated=False, normalized='typing-extensions') distributions.
   2: typing-extensions<5,>=4.3
      Required by:
        snowflake-connector-python 3.12.2
      But this pex had no ProjectName(raw='typing-extensions', validated=False, normalized='typing-extensions') distributions.

This all works just fine outside of pex

@pat-trunk-io
Copy link

I got the samething happening when trying to import boto3

alexeagle pushed a commit that referenced this issue Oct 10, 2024
Addresses the bug in
#391.

Single-file modules such as `six` and `typing-extensions` does not work
with the current `py_pex_binary` rule, since it's assumed that there
exist a sub-directory within `site-packages` that is not dist-info.

An example of what a single-file module pypi-dep looks like: 
```
ls bazel-reppro_pex_err/external/rules_python~~pip~pypi_311_six/site-packages/
__init__.py  six-1.16.0.dist-info  six.py
```

Since `Distribution.load(..)` takes in the `site-packages` directory, we
only emit this part of the path, and let `uniquify=True` handle
deduplication after `_map_srcs` is applied.


---

### Changes are visible to end-users: no

### Test plan

<!-- Delete any which do not apply -->

- Manual testing; please provide instructions so we can reproduce:
Add `six` to requirements and `"@pypi_six//:pkg"` as a dep to the
py_pex_binary example; then import `six` in py_pex_binary's `say.py`.
Printing the module or `cowsay.cow(f"{six}")` shows that the previous
example and single-files modules now also work.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants