Periods get converted to dashes in package name

[sawatsky]

I have a requirements.txt file with the following:

My.Package~=1.0
My.Sub.Package~=1.1

When I run python -m piptools compile, the periods in the package name get converted to dashes:

my-package==1.0.0
    # via -r requirements.txt
my-sub-package==1.1.1
    # via
    #   -r requirements.txt
    #   my.package

I need the periods to stay periods. I have no control over the names of the packages. Some package names may have both periods and dashes, such as My.More-Complex.Package, and I don't want that to change to my-more-complex-package.

Yes, the output file above functions correctly, but we're doing extra parsing on it that's breaking because the periods are now dashes.

Edit:
Per comment below, if the periods are converted to dashes to be consistent with pip, then I'd prefer all periods can converted to dashes, including # my.package to # my-package in the example above.

[AndydeCleyre]

These days pip is doing more canonicalization, and so pip-tools is following to ensure consistency and accurate comparisons.

This request as I understand it is for package names to recreate their exact input names in the output. I'm personally a maybe on this.

FWIW:

In [1]: from pip._vendor.packaging.utils import canonicalize_name

In [2]: canonicalize_name??
Source:
def canonicalize_name(name: str) -> NormalizedName:
    # This is taken from PEP 503.
    value = _canonicalize_regex.sub("-", name).lower()
    return cast(NormalizedName, value)

In [3]: from pip._vendor.packaging.utils import _canonicalize_regex

In [4]: _canonicalize_regex
Out[4]: re.compile(r'[-_.]+', re.UNICODE)

[sawatsky]

Thank you for that information. Then I suppose I would prefer that all the periods get converted to dashes, including the comments under via (except # -r requirements.txt of course).

my-sub-package==1.1.1
    # via
    #   -r requirements.txt
    #   my-package

I can file a bug with my team to handle My.More-Complex.Package changing to my-more-complex-package.

[AndydeCleyre]

I believe the annotation periods are converted to dashes as of the recent 6.5.1 release. 🎆

[AndydeCleyre]

@sawatsky Do you think the current approach (converging names) is good enough to close this, or should it be kept open to discuss re-applying original names?

[aflaisler]

@AndydeCleyre I'm on python 3.8.12 and this behavior introduces the following error: pkg_resources.DistributionNotFound: The 'my-package==0.0.x' distribution was not found and is required by package
It still happens after updating pip. Do you know what would avoid this situation?

[AndydeCleyre]

I'm not at a computer for now but will try to replicate and investigate when I can. Can you provide definite reproduction steps?

[aflaisler]

Sure. This is how we face the issue:
1 - Add my-package==0.0.x to requirements.in (in our case c4.spanner==0.0.33)
2 - Run pip-compile requirements.in (I can see it has been added as c4-spanner==0.0.33)
3 - Call your main function from the cli (we use click)

This is the full trace:

Traceback (most recent call last):
  File "/Users/aflaisler/miniconda3/envs/vodcast/bin/vodcast", line 33, in <module>
    sys.exit(load_entry_point('c4.vodcast', 'console_scripts', 'vodcast')())
  File "/Users/aflaisler/miniconda3/envs/vodcast/bin/vodcast", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/aflaisler/Documents/vodcast/vodcast/cli.py", line 6, in <module>
    import vodcast.__main__
  File "/Users/aflaisler/Documents/vodcast/vodcast/__main__.py", line 8, in <module>
    from sklearn.metrics import make_scorer
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/sklearn/__init__.py", line 82, in <module>
    from .base import clone
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/sklearn/base.py", line 17, in <module>
    from .utils import _IS_32BIT
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/sklearn/utils/__init__.py", line 23, in <module>
    from .class_weight import compute_class_weight, compute_sample_weight
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/sklearn/utils/class_weight.py", line 7, in <module>
    from .validation import _deprecate_positional_args
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/sklearn/utils/validation.py", line 26, in <module>
    from .fixes import _object_dtype_isnan, parse_version
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/sklearn/utils/fixes.py", line 28, in <module>
    from pkg_resources import parse_version  # type: ignore
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3247, in <module>
    def _initialize_master_working_set():
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3221, in _call_aside
    f(*args, **kwargs)
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3259, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/pkg_resources/__init__.py", line 579, in _build_master
    ws.require(__requires__)
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/pkg_resources/__init__.py", line 897, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/aflaisler/miniconda3/envs/vodcast/lib/python3.8/site-packages/pkg_resources/__init__.py", line 783, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'c4-spanner==0.0.33' distribution was not found and is required by c4.vodcast

[AndydeCleyre]

Thanks!

This might be outside of pip-tools. Sounds like the req file is compiled alright.

• Are you using pip-sync to install them? If not, how are you installing the packages?
• Can you reproduce this with a plain venv?
• What version of click are you using?
• Can you provide reproduction steps for someone without access to your private packages?
• What's the output of pip list | grep -i c4.spanner? FWIW when I test with ruamel.yaml, everything works, but pip list does show the . rather than -.

[aflaisler]

Thanks for your help! In the meantime, doing some testing I realised we only have the issue when installing in editable mode via pip install -e . Could you try on your end?

Here are the answers to your questions:

Are you using pip-sync to install them? If not, how are you installing the packages?

• We use pip install -r requirements.txt (tested with pip-sync and same issue) and then install the package in editable mode pip install -e .

Can you reproduce this with a plain venv?

• We use conda env at the moment - I don't think the issue is coming from this but will try later on

What version of click are you using?

• click==7.1.2 although also tested with the latest one and same issue

What's the output of pip list | grep -i c4.spanner?

(vodcast) aflaisler@mac % pip list | grep -i c4.spanner
c4.spanner          0.0.33

[AndydeCleyre]

I haven't been able to reproduce yet, @aflaisler . Can you do so with https://github.com/andydeCleyre/1576 ?

$ python3 -m venv venv
$ . ./venv/bin/activate
$ pip install -U pip -r requirements.txt
$ pip install -e .
$ test-with-editable-dots
success

If not, can you modify that example project to reproduce the problem?

[laurence-kobold]

I'm seeing this issue with one of my team's builds.

@AndydeCleyre I've reproduced this in a modified version of your example project: https://github.com/laurence-kobold/1576

I've made two changes:

• Changing to using setup.py instead of pyproject.toml and setting install_requires to ["ruamel-yaml"]
• Importing pkg_resources in the main function

Running your sample code produces the following error:

Traceback (most recent call last):
  File "/Users/laurence/git/laurence-kobold/1576/venv/bin/test-with-editable-dots", line 33, in <module>
    sys.exit(load_entry_point('testing-editable-dots', 'console_scripts', 'test-with-editable-dots')())
  File "/Users/laurence/git/laurence-kobold/1576/testing_editable_dots/main.py", line 2, in main
    from pkg_resources import parse_version
  File "/Users/laurence/git/laurence-kobold/1576/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3257, in <module>
    def _initialize_master_working_set():
  File "/Users/laurence/git/laurence-kobold/1576/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3240, in _call_aside
    f(*args, **kwargs)
  File "/Users/laurence/git/laurence-kobold/1576/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3269, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/Users/laurence/git/laurence-kobold/1576/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 582, in _build_master
    ws.require(__requires__)
  File "/Users/laurence/git/laurence-kobold/1576/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 899, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/laurence/git/laurence-kobold/1576/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 785, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'ruamel-yaml' distribution was not found and is required by testing-editable-dots

[laurence-kobold]

This problem also happens in my team's build process, because we set the install_requires in our setup.py by reading the requirements.txt file generated by pip-compile. We do this to keep our lists of requirements in requirements.in file and setup.py in sync. You might suggest avoiding having the .in files altogether, but we want to use .in files so that we can have a layered requirements workflow.

[AndydeCleyre]

Thanks, that's helpful stuff!

In your particular case, do you really need to parse the .txt to keep the .in and setup.py in sync?

I'm looking into this, but my guess right now is that it's a bug in setuptools' handling of install_requires elements with dashes.

A temporary workaround is to replace dashes with dots when populating setup.py, maybe?

[AndydeCleyre]

I think this is fixed in the very latest setuptools, can you please try again with pip install -U setuptools sprinkled in?

[laurence-kobold]

In your particular case, do you really need to parse the .txt to keep the .in and setup.py in sync?

Good question. We could get around this by relying on users to keep both in sync, though that seems error prone. Potentially we might be able to get enough information by parsing the .in file

A temporary workaround is to replace dashes with dots when populating setup.py, maybe?

The issue with that is that we'd have to distinguish between dashes due to normalization and dashes that were in the original package name. I think an easier temporary fix is for us to downgrade pip-tools.

I think this is fixed in the very latest setuptools, can you please try again with pip install -U setuptools sprinkled in?

I'm still seeing this issue with setuptools-60.9.2

[AndydeCleyre]

Ah yes, my mistake.

[AndydeCleyre]

I do think this is a bug in setuptools, and can be reproduced without pip-tools, just with pip install ruamel-yaml and @laurence-kobold's example repo.

If you edit the launcher file that setuptools created and replace it with the much simpler:

#!/...keep original shebang here
import re
import sys
from testing_editable_dots.main import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

then it seems to work fine.

[desaintmartin]

Note that IDEs managing package installation through requirements.txt can complain that requirements are not satisified (Exemple with zope.interface in setup.py being written in requirements.txt as zope-interface, with PyCharm constantly complaining).

[AndydeCleyre]

Related:

• Setuptools:
  • [BUG] Generated console scripts which depend on pkgs with dots in name fail if the dep was specified in setup.py with dashes instead pypa/setuptools#3157
  • Match normalized requirement names pypa/setuptools#3153
• PyCharm:
  • https://youtrack.jetbrains.com/issue/PY-24636
  • https://youtrack.jetbrains.com/issue/PY-6438
  • https://youtrack.jetbrains.com/issue/PY-22114
  • https://youtrack.jetbrains.com/issue/PY-46906
  • https://youtrack.jetbrains.com/issue/PY-49425
  • https://youtrack.jetbrains.com/issue/PY-50964
  • https://youtrack.jetbrains.com/issue/PY-53076
  • https://youtrack.jetbrains.com/issue/PY-50653
  • https://youtrack.jetbrains.com/issue/PY-46883
  • https://youtrack.jetbrains.com/issue/PY-44106
• Misc:
  • Normalized package names dhatim/python-license-check#81

What still looks like the best idea to me is for everyone who wants to predictably compare or identify Python packages to use the normalization procedure described in PEP 503:

This PEP references the concept of a “normalized” project name. As per PEP 426 the only valid characters in a name are the ASCII alphabet, ASCII numbers, ., -, and _. The name should be lowercased with all runs of the characters ., -, or _ replaced with a single - character. This can be implemented in Python with the re module:

import re

def normalize(name):
    return re.sub(r"[-_.]+", "-", name).lower()

[laurence-kobold]

Thanks for opening that setuptools issue @AndydeCleyre

[AndydeCleyre]

Setuptools just released v62, with better matching logic, and seems to fix the issues 🎉.

@aflaisler @laurence-kobold

Can you confirm your needs are met by installing the relevant packages with setuptools 62?

[AndydeCleyre]

@atugushev

Do you think we ought to constrain our setuptools dependency to >=62.0.0?

[atugushev]

Do you think we ought to constrain our setuptools dependency to >=62.0.0?

I'd prefer not to. It might be too restrictive for users.

[gimbo]

@AndydeCleyre wrote:

What still looks like the best idea to me is for everyone who wants to predictably compare or identify Python packages to use the normalization procedure described in PEP 503:

Genuine question (because maybe I've missed something important)...

Can anyone explain to me why are we (the python community, pip maintainers, pip-tools maintainers, PyCharm maintainers, me, you, etc.) normalising package names using "the normalization procedure described in PEP 503" for any purpose other than constructing URL path fragments in python package repository implementations? Because as far as I can see that purpose is all that PEP 503 is talking about when it defines this normalisation.

Specifically, normalization is mentioned in PEP 503 in these places:

• Below the root URL is another URL for each individual project contained within a repository. The format of this URL is /<project>/ where the <project> is replaced by the normalized name for that project, so a project named “HolyGrail” would have a URL like /holygrail/.

• Repositories MAY redirect unnormalized URLs to the canonical normalized URL (e.g. /Foobar/ may redirect to /foobar/), however clients MUST NOT rely on this redirection and MUST request the normalized URL.

• In the section quoted by @AndydeCleyre above; that section refers to PEP 426 (for valid characters), and then says:

  The name should be lowercased with all runs of the characters ., -, or _ replaced with a single - character.

Now, a shallow reading of that paragraph might conclude that PEP 426 defines this normalisation and that it's a standard thing to do with package names — but PEP 426 does not (see below), and AFAICS the normalisation isn't defined anywhere relevant: it's just a PEP 503 thing defined for the purpose of constructing URL path fragments in a package repo implementation.

There's a whole rabbit hole worth diving into, but no mention of normalisation...

• PEP 426 — which PEP 503 references;
• PEP 566 — which replaces PEP 426;
• PEP 508 — which PEP 566 references about name formats;
• PEP 345 — which PEP 508 references about name formats, and which is ultimately superceded by... PEPs 426 and 566.

What you can find are regexes for package names, but those regexes treat periods and dashes (and underscores, in fact) identically.

So I ask again: why is this happening in non-repo-package-URL contexts? It's obviously causing problems all over the place (I'll add this fresh one to the mix, particularly as I hit it myself an hour ago), and I keep seeing PEP 503 referenced as if that justifies it... but it just doesn't. What's going on here?

And again, I'm asking this in genuine curiosity and with the assumption I'm missing something. I'm in general a user not a contributor when it comes to these amazing tools, my basic stance is gratitude, and my starting assumption is "smarter people than I know better" —
but sometimes an impertinent question is the right one to ask.

[AndydeCleyre]

@gimbo These are good questions.

My absolutely personal perspective:

• After Python docs and the PEPs, the next best source of "truth" is pypa/pip, which recently ramped up its application of this normalization. This was the real push for pip-tools, I think. Generally if it works for pip, it's got to work for us, too.
• In the absence of exhaustively prescribed universal package handling procedures, and with a variety of projects in the space, I appreciate changes which strengthen predictability, regularity, and smooth interoperability between those projects.
• To that end, converging on any "normalized" form for identifying packages seems like a good, simplifying idea, long term.
• While that PEP does not prescribe its defined normalization for wider purposes, if we want normalization in general, it may as well be used. We don't need the confusion of multiple "normalized" and "canonical" formats. The more we can converge these, the better.
• Please everyone stop using setup.* files 😢

[gimbo]

Thanks @AndydeCleyre, that makes a lot of sense. Can I then ask two more questions...?

• What's the story here with respect to namespace packages? I recently discovered them and think they sound very handy for some circumstances, but they seem to have a similar (maybe related?) pitfall. E.g. my first thought was "great, I'll call my package my_org.some-thing and I can do from my_org.some_thing.foo import bar, and it seemed to work and the sun was shining... But then something broke when I made some other change (sorry, I forget exactly, but possibly it was when I converted from setup.py to setup.cfg...) — and it seemed the solution was to call the package my_org-some-thing, which feels less satisfactory to me because the dashes are sort of doing double-duty, in that they're both separating my org name from the package name, and also breaking up the package name. Not sure if this makes sense/rings bells, but if it does, any thoughts?

• Please everyone stop using setup.* files 😢

  That covers setup.cfg too? I only just started. 🤦‍♂️ I thought they were a big improvement on setup.py because no longer executable — but it sounds like you think there are good reasons for avoiding them too? Or is it simply that pyproject.toml is the way forward and can we please standardise, people?

Thanks!

[AndydeCleyre]

It may be that your naming issue is resolved today with the new setuptools release, so I'd give that another try.

Yes, setup.cfg is definitely an improvement, and yes, the pyproject.toml spec has come far enough now that IMO there's little reason not to use it, and it will be the best-supported standard for its purpose going forward.

That said, there's a reason pip-tools itself uses setup.cfg, but I can't recall what it is.

@atugushev can probably remind us.

[laurence-kobold]

Yup, upgrading to v62 of setuptools fixes this issue!

[AndydeCleyre]

Excellent!

@desaintmartin, I'm inclined to close this issue here, as it seems pip, pip-tools, and setuptools are getting along on this front now.

Maybe some of the IDE issues will become resolved if they're using setuptools behind the scenes, and any issues that remain are probably best addressed by the IDE developers.

What do you (and the rest of you) think?

Feel free to express yourself with emoji reactions and I'll get my emoji interpreter on the case, pronto. Thanks!

[Amwam]

I believe we're still experiencing this issue, even when upgrading both pip and setuptools.
I've tried also with python 3.9 and 3.10

With a requirements.in of

repoze.who
zope.sqlalchemy
sqlalchemy

Using the command
pip-compile --upgrade --output-file=requirements.txt requirements.in

It always generates as

#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
#    pip-compile --output-file=requirements.txt requirements.in
#
repoze-who==2.4.1
    # via -r requirements.in
sqlalchemy==1.4.35
    # via
    #   -r requirements.in
    #   zope-sqlalchemy
transaction==3.0.1
    # via zope-sqlalchemy
webob==1.8.7
    # via repoze-who
zope-interface==5.4.0
    # via
    #   repoze-who
    #   zope-sqlalchemy
zope-sqlalchemy==1.6
    # via -r requirements.in

# The following packages are considered to be unsafe in a requirements file:
# setuptools

Converting the . in zope.sqlalchemy and repoze.who to dashes

We're using the latest pip and setuptools version

pip --version
pip 22.0.4 from ...

python -c 'import setuptools; print(setuptools.version.__version__)'
62.1.0

pip-compile --version
pip-compile, version 6.6.0

Maybe I haven't fully understood the desired behaviour, I'm assuming that the packages should remain using the ., but with the normalisation, should we be trying to get these packages working using the dash instead?

[AndydeCleyre]

Current expectation is to use the normalized form with dashes in our output, and this issue is figuring out if there are any significant problems with that (which could outweigh the simplification benefits).

@Amwam are you seeing any problems with this behavior?

[Amwam]

Looks like we're not seeing issues anymore.
With setuptools 58, we had a lot of import problems (not being able to find the packages listed in the requirements.txt file)
With a newer version (62+) the imports still work correctly, its just the output of the requirements.txt file that threw us off.

I think its safe for us to say, upgrading setuptools also solves our problem.

[alexpulver]

@AndydeCleyre I just found this thread. Please let me know if I should open an additional issue to continue the discussion. I will comment here in the meantime.

Current expectation is to use the normalized form with dashes in our output, and this issue is figuring out if there are any significant problems with that (which could outweigh the simplification benefits).

One issue is PyCharm complaining that requirements are not satisfied per @desaintmartin's comment:

Note that IDEs managing package installation through requirements.txt can complain that requirements are not satisified (Exemple with zope.interface in setup.py being written in requirements.txt as zope-interface, with PyCharm constantly complaining).

I couldn't find a fix or workaround for that currently.

Another issue is consistency with the intended use of normalization. Paul Moore (CPython core developer) says the following about using the normalized form in user output (Revisiting distribution name normalization):

I’d consider that to be a bug in the UI. Any time a package name is reported to the user, the display form of the name should be used. I will concede that there are a lot of such bugs around, but I suspect that’s mostly because most people don’t really care that much about such issues. But I would completely support anyone who wanted to work through the various tools and make them consistently report package names in the form used by the package (in its Name metadata entry). It’s probably a lot of work, but UX is important, so that isn’t an argument for not doing it. (It may be an argument for not having the resources to do it, of course…)

pip 22.2.1 lists the packages with dots:

$ pip list | grep core
aws-cdk.core                          1.166.1

[gimbo]

@alexpulver wrote:

One issue is PyCharm complaining that requirements are not satisfied per @desaintmartin's comment:

Note that IDEs managing package installation through requirements.txt can complain that requirements are not satisified (Exemple with zope.interface in setup.py being written in requirements.txt as zope-interface, with PyCharm constantly complaining).

Just to say: for PyCharm (and I guess other IDEs), this is a problem which goes beyond the situations covered in this issue: it comes up with any package where the project/distribution name differs from the package name. E.g. for wxPython, requirements.txt (say) needs to refer to wxPython (project/distribution name) but then you import wx (package name).

The PyCharm issue tracker gets a constant stream of issues of this type (e.g. here's the wxPython one, and @AndydeCleyre's earlier comment points to many more) where its internal mapping tracking this stuff needs to be updated. It's annoying.

To be clear: I'm not saying there isn't something here for pip-tools to pay attention to (and I like the quote from Paul Moore!) — just that this issue comes up even without dots, dashes, or underscores in project/package names.

(I also note that PEP 423 recommends always using the same name for project and package — but that PEP was deferred; a shame, I think, if only for this point. And if that was generally accepted, then converting periods to dashes would obviously be covered by that and maybe we wouldn't be having this conversation... or maybe we still would. 🙂)

[AndydeCleyre]

FWIW, if/when we can adapt to use pip's new resolution json report (#1654), I think proper names will come "for free."

At least one blocker exists for that: pypa/pip#11316

[ampedandwired]

This behavior just tripped us up as well. Specifically:

• pip-compile is normalizing the package "jaraco.classes" to "jaraco-classes"
• We then run another tool over the pip-compile result (License finder
• License-finder invokes some pip internal methods to resolve the package
• This fails for jaraco-classes with the error pip._vendor.pkg_resources.DistributionNotFound: The 'jaraco-classes==3.2.3' distribution was not found and is required by the application

If I replace "jaraco-classes" with "jaraco.classes" in the pip-compile output file, then license_finder works OK.

The pip calls that license_finder is using to resolve the package are here. Should license_finder be using some other way of resolving the package that knows about normalisation?