pth files containing code make PDM show error messages (1.8, install_cache enabled)

[pawamoy]

• I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

Not actually causing an error, though displaying error tracebacks.
Enable the feature.install_cache option, and install dependencies of a project which depends on zope.event or zope.interface.

Actual behavior

$ pdm install  # --> OK
$ pdm install  # --> second install OK but errors shown
Error processing line 1 of /home/user/project/__pypackages__/3.8/lib/zope.event-4.5.0-py3.6-nspkg.pth:

  Traceback (most recent call last):
    File "/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8/site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 553, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
Error processing line 1 of /home/user/project/__pypackages__/3.8/lib/zope.interface-5.4.0-py3.8-nspkg.pth:

  Traceback (most recent call last):
    File "/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8/site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 553, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored

...rest of the installation

Expected behavior

Some packages will add .pth files containing Python code to __pypackages__/<major.minor>/lib. PDM should ignore those and not print any traceback.

Example:

$ cat __pypackages__/3.8/lib/zope.event-4.5.0-py3.6-nspkg.pth 
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('zope',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('zope', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('zope', [os.path.dirname(p)])));m = m or sys.modules.setdefault('zope', types.ModuleType('zope'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)

Environment Information

Tracebacks shown with pdm info as well 😅

$ pdm info && pdm info --env
Error processing line 1 of /home/user/project/__pypackages__/3.8/lib/zope.event-4.5.0-py3.6-nspkg.pth:

  Traceback (most recent call last):
    File "/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8/site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 553, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
Error processing line 1 of /home/user/project/__pypackages__/3.8/lib/zope.interface-5.4.0-py3.8-nspkg.pth:

  Traceback (most recent call last):
    File "/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8/site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 553, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
PDM version:        1.8.0                                                                           
Python Interpreter: /home/user/.basher-packages/pyenv/pyenv/versions/3.9.6/bin/python (3.9)
Project Root:       /home/user/project                                         
Project Packages:   /home/user/project/__pypackages__/3.9                      
Error processing line 1 of /home/user/project/__pypackages__/3.8/lib/zope.event-4.5.0-py3.6-nspkg.pth:

  Traceback (most recent call last):
    File "/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8/site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 553, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
Error processing line 1 of /home/user/project/__pypackages__/3.8/lib/zope.interface-5.4.0-py3.8-nspkg.pth:

  Traceback (most recent call last):
    File "/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8/site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 553, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
{
  "implementation_name": "cpython",
  "implementation_version": "3.9.6",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "3.10.0-862.6.3.el7.x86_64",
  "platform_system": "Linux",
  "platform_version": "#1 SMP Fri Jun 15 17:57:37 EDT 2018",
  "python_full_version": "3.9.6",
  "platform_python_implementation": "CPython",
  "python_version": "3.9",
  "sys_platform": "linux"
}

[frostming]

Oops

importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('zope', [os.path.dirname(p)]))

The pth is trying to find a module spec under the site-packages (__pypacakges__/X.Y/lib) and load from it. So the pth caching just doesn't work for this case. Stick to non-cached installation for zope.event now.

[pawamoy]

Ah, you're right, sorry for not seeing it myself. I guess I can try to open an issue on their side as well. Feel free to close, thank you for your help 🙂

[pawamoy]

For reference, I was wondering where that .pth file was coming from, here is the answer: zopefoundation/zope.interface#187 (comment)

[pawamoy]

It seems the error messages are only shown when dependencies are installed for multiple Python versions.

• they appear when I installed for 3.8 and 3.9
• they continue to appear if I just keep 3.8
• they disappear if I just keep 3.9

Anyway, these two zope libraries are dependencies of gevent (which is a dependency of locust), and locust seems to work fine when I remove these *-nspkg.pth files...

[yingbo]

I got the same error when install tensorflow (the latest version so far, 2.6). Can I use cache for other libraries but not using cache only when install tensorflow?

[frostming]

@yingbo It is not easy work even if the selective cache is supported, as you don't know which package causes the failure. In fact, the pth files come from google-auth and protobuf.

A more verbose workaround would be as follows:

• Find which package brings those .pth files, in your case they are google-auth and protobuf.
• Delete the dist-info and .pth links of the above packages
• Turn off install cache temporarily: pdm config --local feature.install_cache off
• Run pdm sync to install the deleted packages without caches.
• You are free to reset the install cache settings later.

[yingbo]

Gotcha! Thanks @frostming

[frostming]

After all, the *.nspkg.pth is a hack for the legacy namespace packages mechanism. It is safe to just delete them.

[kamatsuoka]

I'm seeing this in version 2.2.1 with install.cache = False