Difficulty reconciling locally available packages with dependencies living at GitHub head of tree

[yhtang]

Description

I'm trying to install a locally modified version of flax together with the paxml package, which contains in its requirements.in file flax @ git+https://github.com/google/flax.

If I do pip install -e flax/ git+https://github.com/google/paxml.git, pip will raise an error:

(venv) $ git clone https://github.com/google/flax.git
Cloning into 'flax'...

(venv) $ (cd flax/ && git log -n 1)
commit 8d097722a9e448c8bc843dc2ae0f5c360a909fc0 (HEAD -> main, origin/main, origin/HEAD)
...    

(venv) $ pip install -e flax/ git+https://github.com/google/paxml.git
ERROR: Cannot install flax 0.8.0 (from /root/test/flax) and paxml==1.2.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested flax 0.8.0 (from /root/test/flax)
    paxml 1.2.0 depends on flax 0.8.0 (from git+https://github.com/google/flax)

If I install flax first, then when I install paxml with pip install git+https://github.com/google/paxml.git, pip tries to uninstall the existing installation of flax even if it is installed from a fresh local clone using the exactly same commit as upstream:

(venv) $ pip install -e flax/
...
Building wheels for collected packages: flax
  Building editable for flax (pyproject.toml) ... done
  ...
...
Successfully <OMITTED...> flax-0.8.0 <OMITTED...>

(venv) $ pip install git+https://github.com/google/paxml.git
Collecting git+https://github.com/google/paxml.git
  ...
Collecting flax@ git+https://github.com/google/flax
  Cloning https://github.com/google/flax to /tmp/pip-install-6dttb3lh/flax_83b3146bc0904de6b75618652f5b4a09
  Running command git clone --filter=blob:none --quiet https://github.com/google/flax /tmp/pip-install-6dttb3lh/flax_83b3146bc0904de6b75618652f5b4a09
  Resolved https://github.com/google/flax to commit 8d097722a9e448c8bc843dc2ae0f5c360a909fc0
  ...
...
Building wheels for collected packages: paxml, flax, jax
  Building wheel for paxml (setup.py) ... done
  ...
  Building wheel for flax (pyproject.toml) ... done
  ...
Successfully built paxml flax ...
Installing collected packages: ...
  ...
  Attempting uninstall: flax
    Found existing installation: flax 0.8.0
    Uninstalling flax-0.8.0:
      Successfully uninstalled flax-0.8.0
...

What'll be a way to install paxml while preserving my local version of flax? Ideally, I don't want to modify the paxml requirement file. Is there a way that we can add an option to pip so that it will treat URL requirements and local/name requirements as equivalent?

Expected behavior

pip install -e flax/ git+https://github.com/google/paxml.git would install both the locally-provided flax plus flax from GitHub.

Alternatively, something like

pip install --ignore-git-head-ref=flax -e flax/ git+https://github.com/google/paxml.git

pip version

23.0.1

Python version

3.10

OS

Ubuntu 22.04

How to Reproduce

$ git clone https://github.com/google/flax.git
$ pip install -e flax/ git+https://github.com/google/paxml.git

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[jeanas]

pip install -e flax/ git+https://github.com/google/paxml.git would install both the locally-provided flax plus flax from GitHub.

That doesn't make sense, since in Python you can only have one version of a package installed per environment. (Well, technically I believe that you can have two simultaneously installed versions if they have no file in common, but they could not both provide the same package.)

Alternatively, something like

pip install --ignore-git-head-ref=flax -e flax/ git+https://github.com/google/paxml.git

There is already #8076 about that, closing in favor of that one.