Add '--ignore-constraint' option

[stephenfin]

Constraint files are often shared across an organization. When developing a package included in such a constraint file, it is not possible to install the package with constraints since the constraint on the package prevents us installing a development version.

❯ cd my-amazing-package
❯ cat constraints.txt
my-amazing-package==1.2.3
Jinja2==3.1.2
iso8601==1.1.0
msgpack==1.0.4
❯ pip install -c constraints.txt .
Processing /dev/my-amazing-package
  Preparing metadata (setup.py) ... done
ERROR: Cannot install my-amazing-package 1.2.4.dev1 (from /dev/my-amazing-package) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested my-amazing-package 1.2.4.dev1 (from /dev/my-amazing-package)
    The user requested (constraint) my-amazing-package===1.2.4.dev1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Resolve this by allowing users to opt out of individual constraints to the install, wheel, and download subcommands. This is rather manual but it's expected that tools like tox could automatically generate a value for this option when invoking pip install command.

❯ pip install -c constraints.txt --ignore-constraint my-amazing-package .
❯ pip wheel -c constraints.txt --ignore-constraint my-amazing-package .
❯ pip download -c constraints.txt --ignore-constraint my-amazing-package .

This is only added for the 2020-resolver resolver, not the legacy-resolver resolver, given the latter is deprecated for removal.

Fixes: #7839

Open questions

• Are we happy to make this a flag that the user should pass manually? I wasn't able to think of a foolproof way to detect that the we're installing a package and wanted to ignore constraints for that specific package. I also wasn't sure we wanted to do this.
• Should we validate the package name provided by the user? Currently it's free-form, in line with most other options that I could see. We might want to validate (a) it's a valid package name and (b) it's actually going to be installed (to prevent typos). Not sure this is needed though.

[sbidoul]

Is the problem this PR solves is a subset of #8076 ?
In which case it's probably better to implement the generic "dependency/constraint override" mechanism that is discussed there.
If I correctly sense the outcome of the discussion, it now boils down to nailing a decent UX for that feature.

[stephenfin]

Is the problem this PR solves is a subset of #8076 ? In which case it's probably better to implement the generic "dependency/constraint override" mechanism that is discussed there. If I correctly sense the outcome of the discussion, it now boils down to nailing a decent UX for that feature.

I wasn't aware of #8076 but I just read through it (took me a while). I think we could conflate these issues but I don't think that would be wise. #7839 seeks to avoid conflicts caused by mismatched constraints between a local package we're installing and a constraints file, whereas #8076 seeks to avoid conflicts caused by mismatched constraints between two or more packages. I think the solutions to these should be subtly different. In the latter case, we need to pick a specific winner (since there could many conflicts). In the former case (i.e. what we're tackling here) we want to simply ignore the constraint for our package under development. They're different problems that will require different inputs from the user, IMO.

[sbidoul]

@stephenfin continuing the discussion in #7839 (comment)

[pfmoore]

I think the solutions to these should be subtly different.

I think having (potentially) two subtly different approaches to handle two subtly different problems is not a particularly good user experience. I would prefer to find a common underlying mechanism or idea that unifies the two issues and allows us to implement a single, more powerful but also more general, solution.

If that means we hold off on "quick fixes" while we find the more general approach, I'm fine with that.

[jaklan]

Are there any updates related to that MR? AWS enforces usage of constraints in MWAA:

Beginning with Apache Airflow v2.7.2, your requirements file must include a --constraint statement. If you do not provide a constraint, Amazon MWAA will specify one for you to ensure the packages listed in your requirements are compatible with the version of Apache Airflow you are using.

Due to that, we have a huge problem to install our custom libraries, which need different versions of some dependencies (although we know they won't break anything).

[notatallshaw]

Due to that, we have a huge problem to install our custom libraries, which need different versions of some dependencies (although we know they won't break anything).

Can't you just provide your own constraint file as the text you've quoted says?

Otherwise this sounds like you need to be discussing this with Amazon.

[jaklan]

Due to that, we have a huge problem to install our custom libraries, which need different versions of some dependencies (although we know they won't break anything).

Can't you just provide your own constraint file as the text you've quoted says?

Otherwise this sounds like you need to be discussing this with Amazon.

@notatallshaw Airflow constraints look like that, modifying them is a nightmare - it would be much easier to overwrite a specific constraint instead of recreating the whole list.

[notatallshaw]

@notatallshaw Airflow constraints look like that, modifying them is a nightmare - it would be much easier to overwrite a specific constraint instead of recreating the whole list.

You can get the same effect of this PR by removing the line that refers to the dependency you don't want constraints on. This PR will only let you manually specify constraints to ignore, so there's very little difference with just removing them yourself.

Further you could simply provide any empty file if you don't want any constraints.

And FYI, I regularly modify these constraint files, but I have built some simple tooling to do so.

[jaklan]

@notatallshaw yes, that's what we do as well, but it's pretty annoying process - and that's why we would like to see that feature in pip, especially when the MR is already ready. The biggest difference is the fact we wouldn't need to maintain our custom constraints.txt anymore (or play with additional tooling you mentioned).

[notatallshaw]

The biggest difference is the fact we wouldn't need to maintain our custom constraints.txt anymore (or play with additional tooling you mentioned).

But you would still need to maintain custom list of packages to pass into the pip cli, right? Just making sure I'm not missing something.

I'm not a pip maintainer, but I do have my own proposal (#7839 (comment)) to address the use case originally outlined, but I don't think it would address your use case.

[jaklan]

@notatallshaw we have to provide requirements.txt anyway to install our libraries:

--constraint /usr/local/airflow/dags/constraints.txt

--extra-index-url https://...
gtm.airflow==...

so having that flag we could simply have sth like:

--constraint /usr/local/airflow/dags/constraints.txt
--ignore-constraint pydantic

--extra-index-url https://...
gtm.airflow==...  # older / newer pydantic is a dependency

to overwrite the pydantic version.

I'm not a pip maintainer, but I do have my own proposal (#7839 (comment)) to address the use case originally outlined, but I don't think it would address your use case.

If I understand correctly the proposal - it should work as we would simply overwrite the constraint without an error, just with warning (although I wonder what's the goal of constraints then if they can be overwritten implicitly).

[pfmoore]

I wonder what's the goal of constraints then if they can be overwritten implicitly

My problem with this whole discussion is similar - what's the point of constraints if they can be ignored?

After all, in this situation, AWS have required that applications use constraints (presumably so that only versions that Amazon are willing to support are installed), and you're looking for a way to subvert that requirement. I don't know any of the details here, but I'm not at all sure that's something I want pip to make easier...

[notatallshaw]

If I understand correctly the proposal - it should work as we would simply overwrite the constraint without an error, just with warning (although I wonder what's the goal of constraints then if they can be overwritten implicitly).

No, the warning would be when a version can't be determined, therefore no constraint can be applied.

In your use case I think the version can be determined, so you would still get a constraint error.

[jaklan]

@pfmoore overwriting some dependencies is actually the solution which is officially suggested by Airflow in some cases:
https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#upgrading-and-installing-dependencies-including-providers

MWAA handles installation of requirements.txt and doesn't allow you to run any pip command after that, so there's no way to achieve the above. But the point is not about MWAA here, but about:

My problem with this whole discussion is similar - what's the point of constraints if they can be ignored?

I believe Airflow is the most-known "user" of constraints (personally I don't know any other open-source library using them as a core component) - and even they specify use-cases where constraints are problematic and you need a way to omit them.

[pfmoore]

In general, I tend to agree with @notatallshaw's statement, in the comment linked above:

That said I think user built templating solutions are a reasonable solution to this problem. Yes it requires a small script then to build a dev environment, rather than one pip install, but pip can't solve all problems related to creating dev environments anyway so some users will always need to make themselves a script to reduce it to one command.

I'm not actively blocking this proposal, but I have no interest in moving it forward either. So you're going to need to find another pip maintainer who supports this idea if you want to get it implemented in any form.