PackageMetadataNotFound for editable package installed outside of site dirs

[onlinespending]

When installing a package using --editable, pip creates an .egg-link file and an easy-install.pth file in your site-packages dir that points to the source dir, which by defaults contains your .egg-info metadata. This worked just fine with the older pkg_resources package, as it references the .egg-link file to find your package's .egg-info. You can even see it handle .egg-link files in its init.py source. However, the newer importlib_metadata does not seem to be aware of .egg-link or easy-install.pth files at all. I don't even see it mentioned in the source. The only solution is to include the source dir that contains the .egg-info metadata into your PYTHONPATH, which defeats the purpose of having the .egg-link or easy-install.pth pointer in the first place.

Given that importlib_metadata doesn't appear to even process .egg-link files, is there some new 'pip install', 'setuptools', etc install process that creates editable links that importlib_metadata can understand? If not, this omission seems like a bug that would affect anyone using editable installations that make use of .egg-link pointers.

[jaraco]

Thanks for the report. You're correct that importlib_metadata (and importlib.metadata) does not have any awareness of egg-link files, and that's by design. Instead, the current design expects the metadata file to be present somewhere on sys.path (for typical file system and zip file loaders), but also provides extensibility for other loaders to provide the metadata through another mechanism.

However, this project does intend to support editable installs and .pth files, including easy-install.pth in order to honor the metadata for packages made available that way.

And I can confirm that it does:

draft $ git clone -q gh://jaraco/path
draft $ python -m venv env
draft $ env/bin/pip install -q -e path
draft $ env/bin/python -c "import importlib.metadata as md; print(md.version('path'))"
16.3.0

The egg-link file is something that setuptools installs, but I consider it a deprecated feature and unnecessary to supporting metadata.

There is an open issue already (the only other open issue #112) about a particular weakness in editable installs for src-layout projects.

I expect that issue to be addressed as Setuptools figures out the solution to PEP 660 (pypa/setuptools#2816), but until then, this project is open to someone to work out a solution/workaround for the issue here.

Since I suspect this report is a duplicate of that previous one, I'm going to mark it as such, but if what you're experiencing isn't related to a src-layout editable install, then please provide more detail here about how one could replicate the failure you're experiencing.

[onlinespending]

Thanks for the prompt reply. Yes, to workaround this issue, I can place the source directory that contains the .egg-info metadata into my PYTHONPATH. That's a bit problematic when you have several such projects. That's the beauty of simply using the site-packages in your path and .egg-link or easy-install.pth stubs to find the editable src directories. That's the intent of the links.

An example of this issue in one such project I'm working on has a console_script entry point that fails to execute because it can't find its own package.

Traceback (most recent call last):
  File "./mypkg", line 33, in <module>
    sys.exit(load_entry_point('mypkg', 'console_scripts', 'mypkg')())
  File "./mypkg", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/tools/conda/anaconda3/2020.07/lib/python3.8/importlib/metadata.py", line 504, in distribution
    return Distribution.from_name(distribution_name)
  File "/tools/conda/anaconda3/2020.07/lib/python3.8/importlib/metadata.py", line 177, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: mypkg

This occurs whether I have my package under a src dir or directly at the top level of my project dir. How can I get importlib.metadata to following the easy-install.pth link?

[onlinespending]

@jaraco could you please re-open this? Not sure why it was immediately closed before I could respond. Thank you.

The issue I'm experiencing occurs regardless of how the package source is laid out, as the open issue you referred to relates to the use of src.

if importlib.metadata requires that the source dir with the .egg-info metadata is in the PYTHONPATH, then it doesn't truly support the --editable option properly (either through .egg-link or easy-install.pth). These links are what make editable installs possible, so I'm not sure how importlib.metadata can ignore them all together. Unless my issue is somehow strictly related to entry points (such as the console_scripts in my example), or importlib.metadata provides some other facility I'm not aware of to process links to editable installations. Can you clarify?

[jaraco]

Thanks for the prompt reply.

You're welcome. You lucked out that you happened to report the issue shortly before I got engaged on this project for some maintenance :)

This occurs whether I have my package under a src dir or directly at the top level of my project dir.

So it's a known issue in the former case, but not so much in the latter case.

How are you installing the package? For example, when I install using pip, console scripts work fine:

draft $ python -m venv env
draft $ git clone -q gh://jaraco/pip-run
draft $ env/bin/pip install -q -U pip
draft $ env/bin/pip install -q -e pip-run
draft $ env/bin/pip-run --help | head
Usage:

Arguments to pip-run prior to `--` are used to specify the requirements
to make available, just as arguments to pip install. For example,

    pip-run -r requirements.txt "requests>=2.0"

That will launch python after installing the deps in requirements.txt
and also a late requests. Packages are always installed to a temporary
location and cleaned up when the process exits.

The way it works is that pip install -e invokes setup.py develop (Setuptools), which creates the pth files:

draft $ cat env/lib/python3.10/site-packages/easy-install.pth
/Users/jaraco/draft/pip-run

Because that .pth file is present and available in one of the site dirs, that's what makes that package importable and its metadata available:

draft $ env/bin/python
Python 3.10.1 (v3.10.1:2cd268a3a9, Dec  6 2021, 14:28:59) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path[-1]
'/Users/jaraco/draft/pip-run'
>>> import pathlib
>>> list(pathlib.Path(sys.path[-1]).glob('*.egg-info'))
[PosixPath('/Users/jaraco/draft/pip-run/pip_run.egg-info')]
>>> import importlib.metadata as md
>>> dist = md.distribution('pip-run')
>>> dist.version
'8.7.2'
>>> dist.metadata['name']
'pip-run'
>>> dist._path
PosixPath('/Users/jaraco/draft/pip-run/pip_run.egg-info')

draft $ cat env/bin/pip-run
#!/Users/jaraco/draft/env/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'pip-run','console_scripts','pip-run'
import re
import sys

# for compatibility with easy_install; see #2198
__requires__ = 'pip-run'

try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point


def importlib_load_entry_point(spec, group, name):
    dist_name, _, _ = spec.partition('==')
    matches = (
        entry_point
        for entry_point in distribution(dist_name).entry_points
        if entry_point.group == group and entry_point.name == name
    )
    return next(matches).load()


globals().setdefault('load_entry_point', importlib_load_entry_point)


if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(load_entry_point('pip-run', 'console_scripts', 'pip-run')())

So in my environment, the issue doesn't occur. Scripts execute fine even under an editable install.

What's different about your environment?

[onlinespending]

OK great thank you. Here's what I'm doing. Keep in my mind this as on my work's server, which I don't have root privileges on. But I have tried python 3.10.1 through brew with the same results.

To test this out I created a very simple package called mypkg under ~/pkg/mypkg which has a single function defined that prints Hello World. It has a console_script entry point to its main.py that calls that single function to print Hello World. That much is simple and works when it installs properly.

pyproject.toml

[build-system]
requires = [
    "setuptools>=42",
    "wheel"
]
build-backend = "setuptools.build_meta"

setup.py

import setuptools
setuptools.setup()

setup.cfg

[metadata]
name = mypkg
version = 0.3.0
description = A small example package

[options.entry_points]
console_scripts =
    mypkg = __main__:main

[options]
package_dir =
    = .
packages = find:
python_requires = >=3.6

[options.packages.find]
where = .

All pretty much boiler plate stuff.

I created a directory to install it into as a test at ~/install and run pip install --prefix ~/install -e ~/pkg/mypkg

Obtaining file:///home/me/pkg/mypkg
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Installing collected packages: mypkg
  Running setup.py develop for mypkg
Successfully installed mypkg

echo $PYTHONPATH (using python3.8, which is what the host has installed. But again, this issue occurs with 3.10.1)

/home/me/install/lib/python3.8/site-packages

ls ~/install/lib/python3.8/site-packages

easy-install.pth  mypkg.egg-link

less easy-install.pth

/home/me/pkg/mypkg

ls ~/install/bin

mypkg*

./mypkg

Traceback (most recent call last):
  File "./mypkg", line 33, in <module>
    sys.exit(load_entry_point('mypkg', 'console_scripts', 'mypkg')())
  File "./mypkg", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/tools/conda/anaconda3/2020.07/lib/python3.8/importlib/metadata.py", line 504, in distribution
    return Distribution.from_name(distribution_name)
  File "/tools/conda/anaconda3/2020.07/lib/python3.8/importlib/metadata.py", line 177, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: mypkg

[onlinespending]

@jaraco The one obvious difference is the use of --prefix. I've noticed that when using --user, it automatically adds the package source dir (which contains the .egg-info) to the PYTHONPATH. Naturally that works, but avoids ever having to dereference the link found in easy-install.pth. I'm guessing when you installed your package in the example above that its similarly creating these paths automatically for you? That of course would obscure the inability of importlib to properly follow the editable stubs.

[jaraco]

Thanks for clarifying. Yes, I do think --prefix is implicated. I was able to replicate the error message by following your process:

draft $ python -m venv env
draft $ git clone -q gh://jaraco/pip-run
draft $ env/bin/pip install -q -U pip
draft $ env/bin/pip install -q --prefix /Users/jaraco/draft/prefix -e pip-run
draft $ ls prefix/bin
pip-run
draft $ prefix/bin/pip-run
Traceback (most recent call last):
  File "/Users/jaraco/draft/prefix/bin/pip-run", line 33, in <module>
    sys.exit(load_entry_point('pip-run', 'console_scripts', 'pip-run')())
  File "/Users/jaraco/draft/prefix/bin/pip-run", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py", line 919, in distribution
    return Distribution.from_name(distribution_name)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py", line 518, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for pip-run

But it's no surprise that the metadata isn't found, because even without doing an editable install, Python doesn't find packages installed to a prefix by default:

draft $ rm -rf *
draft $ python -m venv env
draft $ env/bin/pip install -q -U pip
draft $ env/bin/pip install -q --prefix /Users/jaraco/draft/prefix pip-run
draft $ prefix/bin/pip-run
Traceback (most recent call last):
  File "/Users/jaraco/draft/prefix/bin/pip-run", line 5, in <module>
    from pip_run import run
ModuleNotFoundError: No module named 'pip_run'

The problem is, the packages in ./prefix aren't on the sys.path.

Are you saying that this prefix technique worked for you in the past and stopped working at some point?

[onlinespending]

@jaraco shouldn't that work fine by adding the prefix/lib/python*/site-packages path to PYTHONPATH? It works for me when I don't use --editable and only --prefix when setting the PYTHONPATH to that prefix's site-packages dir, as you'd expect. It's really the inability to follow the stub links in either easy-install.pth or .egg-link that prevents importlib.metadata from working with --editable. This problem seems to be masked when --prefix is not used

Yes, it worked prior to importlib.metadata becoming the standard. python 3.6 and package_resources handled the stub links to the editable source code just fine.

[jaraco]

shouldn't that work fine by adding the prefix/lib/python*/site-packages path to PYTHONPATH

I see. No, it doesn't work. That's because .pth files are only invoked if they exist in "site dirs" (ref). They're ignored in other paths, even if on PYTHONPATH. The behavior works when not installed editable because pip doesn't rely on metadata to load the entry point but also because the metadata is installed directly to the PYTHONPATH.

I encountered this same problem when authoring pip-run (coincidence that I used that project as an example above) because it also uses pip install --target and sets PYTHONPATH to add those packages to the Python path. To work around the issue, I found I had to add a sitecustomize module to ensure that the target directory is added to "site dirs" and thus that .pth files are processed.

The 'egg-link' support is a deprecated technique. It's only used by Setuptools and only works if pkg_resources is employed. Even if importlib_metadata were to support this non-standard mechanism of linking develop installs, it would still fail to honor installs from other build tools like flit. That's why importlib metadata is not planning to implement that behavior.

My best recommendation for you is to do one of the following (in decreasing order of my preference):

• Use a virtualenv instead of a --prefix. That's what most people do and that's why others haven't encountered this issue.
• Add a sitecustomize.py module to the site-packages directory of your prefix that adds that directory to site dirs.
• Ensure the directory with the metadata is added to the PYTHONPATH.
• Build and install your own metadata provider that resolves egg-link files.

Does that clarify?

[onlinespending]

@jaraco thanks this clarifies a lot. Ashame .pth stubs aren't invoked when in a path found in the $PYTHONPATH. Feel like that would be a sensible expectation.

I use venv when doing local development, but I do IC design, and for each chip we have a separate project work area that I'd like to install python packages to, using --prefix. For our own python packages these need to be installed --editable, since we could be making changes to them from their source dirs.

I like the idea of using sitecustomize.py. I noticed when testing out this contrived example with python 3.6.8, that it had installed a site.py file into my site-packages directory. This actually allowed python 3.8.3 to work by pointing to that site-packages directory instead. I'm curious why setuptools created that site.py file for me for python 3.6 but not 3.8

[jaraco]

It looks like it was an intentional change rooted in pypa/setuptools#2165.

[onlinespending]

ok thank you. I guess one last question. Is there a convenient way to generate such a sitecustomize.py file?

[jaraco]

ok thank you. I guess one last question. Is there a convenient way to generate such a sitecustomize.py file?

You're welcome to call pip_run.launch._inject_sitecustomize(target). If you plan to do that, let me know and I'll make it a public function. Otherwise, consider just copying that function or the one from setuptools. Other than that, I'm not aware of an easier way to do it.

[onlinespending]

@jaraco thank you. I guess I'm confused what is even supposed to live in sitecustomize.py. I thought adding the ~/install/lib/python3.8/site-packages would allow it to find the easy-install.pth file and get around this PackageMetadataNotFound issue.

~/install/lib/python3.8/site-packages/sitecustomize.py

import site

site.addsitedir('~/install/lib/python3.8/site-packages')

[jaraco]

Yes, I should think that would work. It may be that it doesn't resolve ~. What happens if you use os.path.expanduser('~/instal/...')?

[onlinespending]

@jaraco I've also tried using a full absolute path with the same results. I tried printing the site packages and user site packages before and after adding that sitedir in my sitecustomize.py. Nothing changed. It didn't add the directory I passed to addsitedir()

prefixes:
['/tools/conda/anaconda3/2020.07', '/tools/conda/anaconda3/2020.07']

site pacakages:
['/tools/conda/anaconda3/2020.07/lib/python3.8/site-packages']

user site pacakages:
/home/me/.local/lib/python3.8/site-packages

[onlinespending]

The site.py source code makes sense and does process the .pth files in that directory. So I need to trace the code and see why it's not working

[jaraco]

I too am having trouble implementing my recommendation, so I'm investigating too.

[jaraco]

Oh, interesting. I'm seeing that no easy-install.pth is present in prefix/lib/python3.10/site-packages. Only the egg-link is.

[jaraco]

I don't know what made me think that the easy-install.pth would get installed to that prefix, but it seems it doesn't. I'll investigate if that's expected.

[onlinespending]

@jaraco yeah for some reason I have to call python setup.py develop to get it to install the easy-install.pth file. pip install -e doesn't appear to create that file.

I traced the problem to, as it fails to find that dir.

site.py(173):                 if not dircase in known_paths and os.path.exists(dir):

starts with this section of the addsitedir function

names = [name for name in names if name.endswith(".pth")]
    for name in sorted(names):
        addpackage(sitedir, name, known_paths)

and then from this section of code in the addpackage function.

    with f:
        for n, line in enumerate(f):
            if line.startswith("#"):
                continue
            if line.strip() == "":
                continue
            try:
                if line.startswith(("import ", "import\t")):
                    exec(line)
                    continue
                line = line.rstrip()
                dir, dircase = makepath(sitedir, line)
                if not dircase in known_paths and os.path.exists(dir):
                    sys.path.append(dir)
                    known_paths.add(dircase)

It reads each line from the .pth file. That makepath function joins the paths found in the .pth file to the sitedir path. So it expects the paths located in the .pth file to be relative to the sitedir, which is not the case. The paths written to easy-install.pth via setup.py develop are absolute. I think I'll just have my sitecustomize.py file iterate over all .pth files and add those to sys.path myself. That would be easy.

[jaraco]

Okay. I learned some things. Most importantly:

• pip's build isolation breaks the creation of the easy-install.pth file. That's why the setuptools test disables build isolation.

After learning that, I was able to successfully perform an editable install into a prefix and have the metadata be honored:

draft $ python -m venv env
draft $ env/bin/pip install -q -U setuptools wheel pip setuptools-scm  # need wheel and setuptools-scm to build path
draft $ git clone -q gh://jaraco/path
draft $ $PYTHONPATH='/Users/jaraco/draft/prefix/lib/python3.10/site-packages' env/bin/pip install --prefix /Users/jaraco/draft/prefix --no-build-isolation --editable path
Obtaining file:///Users/jaraco/draft/path
  Checking if build backend supports build_editable ... done
  Preparing metadata (pyproject.toml) ... done
Installing collected packages: path
  Running setup.py develop for path
Successfully installed path
draft $ $PYTHONPATH='/Users/jaraco/draft/prefix/lib/python3.10/site-packages' env/bin/python -c "import importlib.metadata as md; print(md.version('path'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py", line 946, in version
    return distribution(distribution_name).version
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py", line 919, in distribution
    return Distribution.from_name(distribution_name)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/__init__.py", line 518, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for path
draft $ echo "import site; site.addsitedir('/Users/jaraco/draft/prefix/lib/python3.10/site-packages')" > prefix/lib/python3.10/site-packages/sitecustomize.py
draft $ $PYTHONPATH='/Users/jaraco/draft/prefix/lib/python3.10/site-packages' env/bin/python -c "import importlib.metadata as md; print(md.version('path'))"
16.3.0

I think that means if you pass --no-build-isolation to your pip build and create a sitecustomize that calls addsitedir, you should get a working install.

I've got to break for the night, but I wish you luck.

[onlinespending]

@jaraco Thank you very much! This got me to a workable solution. The --no-build-isolation is certainly a better option than setup.py develop. I still have the absolute path issue I just mentioned with easy-install.pth and addsitedir. But I'm content with using a simple sitecustomize.py script that adds the paths from the .pth files to sys.path myself.

[onlinespending]

Here's a sitecustomize.py solution that borrows from the relevant sections of the site.py code, but uses relative paths.

import os
import io
import sys

try:
    names = os.listdir('.')
except OSError:
    pass

names = [name for name in names if name.endswith(".pth")]
for name in sorted(names):
    try:
        f = io.TextIOWrapper(io.open_code(name), encoding="utf-8")
    except OSError:
        pass
    with f:
        for n, line in enumerate(f):
            if line.startswith("#"):
                continue
            if line.strip() == "":
                continue
            try:
                if line.startswith(("import ", "import\t")):
                    exec(line)
                    continue
                line = line.rstrip()
                if os.path.exists(line):
                    sys.path.append(line)
            except Exception:
                print("Error processing line {:d} of {}:\n".format(n+1, name),
                      file=sys.stderr)
                import traceback
                for record in traceback.format_exception(*sys.exc_info()):
                    for line in record.splitlines():
                        print('  '+line, file=sys.stderr)
                print("\nRemainder of file ignored", file=sys.stderr)
                break