Version solving failed for packages from private repository with Poetry > 1.0.10

[itssimon]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Linux

• Poetry version: 1.1.5

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/itssimon/cd2ee0d6b407cf003cad3630ccb4776d

Issue

I have two dependencies which need to be installed from private repositories (GitLab). I have configured these two repositories in the pyproject.toml and added credentials as outlined in the documentation:

[tool.poetry.dependencies]
# ...
gcds-common = {extras = ["mosaiq"], version = "^0.12.5"}
gcds-ml = {extras = ["lightgbm", "roberta"], version = "^0.4.5"}
# ...

[[tool.poetry.source]]
name = "common"
url = "https://gitlab.com/api/v4/projects/xxx1/packages/pypi/simple"
secondary = true

[[tool.poetry.source]]
name = "ml"
url = "https://gitlab.com/api/v4/projects/xxx2/packages/pypi/simple"
secondary = true

However Poetry versions > 1.0.10 are not able to resolve these dependencies. poetry update -vvv shows the following output:

PyPI: No packages found for gcds-ml >=0.4.5,<0.5.0
   1: fact: no versions of gcds-ml match >=0.4.5,<0.5.0
   1: conflict: no versions of gcds-ml match >=0.4.5,<0.5.0
   1: !  gcds-ml (^0.4.5) is satisfied by  gcds-ml (^0.4.5)
   1: ! which is caused by "gcds-clinex depends on gcds-ml (^0.4.5)"
   1: ! thus: version solving failed
   1: Version solving took 24.006 seconds.
   1: Tried 1 solutions.

...

SolverProblemError
  Because gcds-clinex depends on gcds-ml (^0.4.5) which doesn't match any versions, version solving failed.

So it seems that the private repositories are ignored and Poetry tries to resolve these dependencies with the public PyPI.

Interestingly, this all works as expected in Poetry version 1.0.10, so there must've been a regression a while ago that has not been fixed since.

[itssimon]

As indicated in #3807 this issue may be related to the fact that there's more than one secondary repository defined in my pyproject.toml.

[itssimon]

@sdispater is this something you could look into for the next release? Happy to help track it down and test where possible.
We're stuck with Poetry 1.0.10 due to this issue but would love to benefit from all the new features and improvements made since that release.

[abn]

Your pyproject.toml seems to be missing source.

gcds-common = {extras = ["mosaiq"], version = "^0.12.5", source = "common"}
gcds-ml = {extras = ["lightgbm", "roberta"], version = "^0.4.5", source = "ml"}

Can you also verify this is still an issue on master pelase? (see below for examples)

Using pipx

pipx install --force --suffix=@master 'poetry @ git+https://github.com/python-poetry/poetry.git@master'

Using a container (podman | docker)

podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -xe
python -m pip install -q git+https://github.com/python-poetry/poetry.git@master
poetry new foobar
pushd foobar
poetry source add --secondary common https://gitlab.com/api/v4/projects/xxx1/packages/pypi/simple
poetry source add --secondary ml https://gitlab.com/api/v4/projects/xxx2/packages/pypi/simple
poetry add --source common gcds-common -E mosaiq
poetry add --source ml gcds-ml -E lightgbm -E roberta
EOF

[itssimon]

Thanks for looking into this @abn.

Unfortunately, adding the source property to the dependencies doesn't make a difference and the issue persists on master :(

If the source property is indeed required in this scenario, it would also be good if that was described in this section of the documentation:
https://python-poetry.org/docs/repositories/#install-dependencies-from-a-private-repository

I'm happy to create a PR for the docs change if this is confirmed.

[itssimon]

I think this is actually an issue with pip. It seems that when multiple secondary repositories share the same hostname (e.g. gitlab.com) but have different credentials, pip will use the first credentials and then think it's already authenticated for the other repositories. GitLab returns a 404 if the credentials are wrong, so it seems to pip as if the packages don't exist, but in fact it's just an authentication error.

[JeroenDelcour]

If this was an issue with the way pip handles caching credentials, it has since been resolved in pypa/pip#10033. However, I'm still running into this issue.

I checked that I was using the latest version of pip (21.3.1) and poetry (1.1.11) and even did a fresh install of poetry to no avail. Adding the source explicitly in pyproject.toml doesn't seem to make a difference.

This issue does not appear in poetry 1.0.10. I also tried the latest preview version (1.2.0a2) and the issue is present there.

I'd be happy to help resolve this, since supporting multiple private repositories is important to us. I had a quick look to see if I could find the root cause in poetry's code, but quickly got lost in the woods. Could someone more familiar with the way poetry handles private repositories help out, or point me in the right direction?

[cquick01]

Also experiencing this issue with two GitLab Package Repository sources configured, each for a separate package.

[[tool.poetry.source]]
name = "package_1"
url = "https://gitlab.domain.tld/api/v4/projects/<PROJECT_ID_1>/packages/pypi/simple/"

[[tool.poetry.source]]
name = "package_2"
url = "https://gitlab.domain.tld/api/v4/projects/<PROJECT_ID_2>/packages/pypi/simple"

Package 1 installs fine, but package 2 is unable to be found. Removing the package 1 configuration allows package 2 to install successfully, so authentication works. Only when both sources are configured is there an issue.

Can confirm that version 1.0.10 does not have this issue.

[JeroenDelcour]

I've managed to fix this issue by using a GitLab personal access token instead of a deploy token. Using poetry 1.1.12.

I'm not sure if this is an issue with poetry, GitLab, or if this is intended behavior and I'm misunderstanding GitLab's deploy tokens.

[cquick01]

I've managed to fix this issue by using a GitLab personal access token instead of a deploy token. Using poetry 1.1.12.

I'm not sure if this is an issue with poetry, GitLab, or if this is intended behavior and I'm misunderstanding GitLab's deploy tokens.

I believe this works because you are able to use the same Personal Access Token for all the different private repos that are under the same domain name. The project-level Deploy Tokens are different for each project.

The issue here seems to be that Poetry doesn't handle multiple repository configurations under the same domain name with different credentials. Authentication will succeed for the first repo, but fail for subsequent ones.

This isn't an issue with a Personal Access Token, because Poetry will use the same token for all the repos.

Edit: Adding these print statements in LegacyRepository.__init__() shows that the same credentials are being used for all the private repos under the same domain

        self._basic_auth = None
        username, password = self._authenticator.get_credentials_for_url(self._url)
+       print(f"{self._url=}")
+       print(f"{username=}")
+       print(f"{password=}")

shows

self._url='https://gitlab.domain.tld/api/v4/projects/xxx2/packages/pypi/simple'
username='__token__'
password='glpat-<TOKEN>'
self._url='https://gitlab.domain.tld/api/v4/projects/xxx5/packages/pypi/simple'
username='__token__'
password='glpat-<TOKEN>'
self._url='https://gitlab.domain.tld/api/v4/projects/xxx0/packages/pypi/simple'
username='__token__'
password='glpat-<TOKEN>'

I'm trying to dig deeper to see if I can get it working.

[cquick01]

It seems like in src/poetry/utils/authenticator.py, the credentials are retrieved based on netloc, which is just the domain name.

I made some changes, and it seems to be working for the two private repos I've been testing with

diff --git a/src/poetry/utils/authenticator.py b/src/poetry/utils/authenticator.py
index 85f7dadc..8df4adff 100644
--- a/src/poetry/utils/authenticator.py
+++ b/src/poetry/utils/authenticator.py
@@ -109,7 +109,7 @@ class Authenticator:

         if credentials == (None, None):
             if "@" not in netloc:
-                credentials = self._get_credentials_for_netloc(netloc)
+                credentials = self._get_credentials_for_url(url)
             else:
                 # Split from the right because that's how urllib.parse.urlsplit()
                 # behaves if more than one @ is present (which can be checked using
@@ -163,18 +163,22 @@ class Authenticator:

             return auth

-    def _get_credentials_for_netloc(
-        self, netloc: str
+    def _get_credentials_for_url(
+        self, url: str
     ) -> Tuple[Optional[str], Optional[str]]:
         credentials = (None, None)

+        parsed_url = urllib.parse.urlsplit(url)
+        netloc = parsed_url.netloc
+
         for repository_name in self._config.get("repositories", []):
-            auth = self._get_http_auth(repository_name, netloc)
+            if self._config.get(f"repositories.{repository_name}.url", False) == url:
+                auth = self._get_http_auth(repository_name, netloc)

-            if auth is None:
-                continue
+                if auth is None:
+                    continue

-            return auth["username"], auth["password"]
+                return auth["username"], auth["password"]

         return credentials

But I'm not sure if it breaks any other use-cases, and now I can't get the pytests to succeed....

Maybe someone more familiar with the authenticator module could provide some insight?

[ZMarouani]

was this issue solved? I'm still experiencing this with two different work environments :

• poetry 1.2.2
  python 3.10.6
  on Wsl Ubuntu 22.04.1 LTS

• poetry 1.2.2

• python 3.8.7
  on Windows 11

I'm using a Deploy token

[cquick01]

@ZMarouani I haven't had any issues since 1.2. I'd make sure the token has the correct read_package_registry permissions set (assuming GitLab...). Can you provide a short reproducible example?