Relaxing / Ignoring constraints during dependency resolution

[stonebig]

What's the problem this feature will solve?
Puting together some packages that have by default incompatible constraints.

Indeed:

• constraints are often meant by the package maintainer as:
  . "accepting complains on this known/focus set of package",
  . you're on your own support if you deviate, but not necessarly bad.
• packages rarely focus on the same versions of complementary packages.
  ==> The new resolver may create more problems than solutions, when trying to build an environment with a large set of package.

Describe the solution you'd like
Be able to ignore voluntary some constraints

Wish:

• we can put some "relax" rule to over-rule too strict packages (for our need), because we know what we want:
  pip install Spyder --relax relaxrules.r , with relax file below meaning:
  . if you want PyQt5, it must be 5.14.x
  . if you want Jedi, it must be >=0.16

PyQt5~=5.14
Jedi>=0.16

Alternative Solutions
Today:

• I have to manually recompile from source "too strict" packages, to workaround this,
• or I would have to build one virtualenv per package,
• or I would only be able to use a specific Python distribution, with much older packages and Python version.

Additional context
Maintaining WinPython

** Current pip check **

• datasette 0.39 has requirement Jinja2~=2.10.3, but you have jinja2 2.11.2.
• astroid 2.3.3 has requirement wrapt==1.11.*, but you have wrapt 1.12.1.

** Current workaround **

• Spyder manually recompiled to accept :
  . PyQt5-5.14.2 (as pip doesn't have a "long term support" of PyQt5-5.12, so the fresher version is safer)

other wishes:

• a basic GUI on Pip (tkinter or web) would still be nice, to have a better view of all the coming version conflicts.

[uranusjr]

@stonebig Would you mind separate the other wishes part into their own issues? It would be much easier to discuss them that way.

As for relxing, I am honestly uncomfortable with having such a impactful feature handy for the general audience. I was by chance also having a similar discussion in the Pipenv tracker, and both the Pipenv one and your are exactly situations I personally think the maintainers have a valid point to restrict the versions, and a user should not be able to override them easily. It’s still useful to have this option somewhere since indeed there are packages with incorrect metadata out there, but pip is too low in the packaging management realm to implement the feature IMO.

[stonebig]

Ok, separating other whishes in a few minutes.
this is moved on another issue:

- having the beautifull "pipdeptree" features in the standard pip:
    . a beautifull description of what package needs (or is needed) by what package of what version,
    . the possibility to get that programatically as json answers.

[pradyunsg]

Thanks for filing this @stonebig! I've gone ahead and re-titled this to issue to be more clearly scoped.

We have seen multiple groups of users express interest in a feature like this. @pfmoore @uranusjr and I have had this come up in our discussions during our work on the resolver, and we are aware of this user need.

We don't know how exactly this would work and what approach we'd be taking here -- we're gonna visit this specific topic at a later date, once the new resolver implementation is at feature parity with the current resolver.

[pradyunsg]

a basic GUI on Pip (tkinter or web) would still be nice, to have a better view of all the coming version conflicts.

This is a completely separate request, and can be built outside of pip and doesn't need to be built into pip. If someone wants to build this outside of pip and later propose bringing it into pip (with clear reasoning for why it can't live outside pip), that'd be perfect. I don't think pip's maintainers are going to be developing/integrating this into pip, and I welcome others to try to build such tooling on-top-of or outside of pip.

I think there has been a "pip GUI" project undertaken as part of IDLE in the past, but I don't have the time to take a look right now. :)

[stonebig]

Building a dsitribution WinPython is quite simple:

• download in a dedicated directory all the wheels (and version) you want,
• then pip install -r requirement.txt
• then one by one, try to fix all the problems:
  • missing wheels,
    • pip download --dest,
    • or cgohlke site of wonders
    • or github/gitlab (/pip-forge one day ?)
  • non-existing wheels,
    • do-compile-yourself (often fails, like for cartopy, or for Python-recent version)
    • raise issues to package maintainer
  • wheels whose beloved version of dependancies mutualy contradicts
    • ask maintainer to relax or upgrade his/her dependancies (very slow process)
    • recompile it yourself without the annoying constraint,
    • go back in version to an older one (with potential security or known issues fixed since ages)
    • or drop the wheel.

I dream of a way to reverse the problem:

• showing package maintainer how their 'too restrictive' constraints makes them incompatible with the rest of the world, (hence a GUI or a pypi website feature ?):
  • give your requirements.txt
  • precise your "beloved" package,
  • the site/gui tells you what fits / what contradicts / what downgrade your package imposes
• or do a third kind of constraints on dependancies in wheel specification:
  • supported constraints (you can speak of a problem to the maintainers when you have this "set"),
  • a "support_requires" next to "install_requires" and "extra_requires" ?

[pfmoore]

Just to note that, while I agree that over-restrictive requirements can be an issue¹, this is a fairly specialised use case. It's not that dependencies can't clash, but that putting together a Python distribution involves including (and managing) a lot of libraries that potentially have no "natural" reason to expect to be used together. So dependency clashes that the library maintainers haven't anticipated/haven't seen before are likely to be more common.

Using --no-deps and manually managing dependencies for problem packages is one option here. It's tricky without some means of identifying where the problems lie, though - we're hoping to give good error reporting for dependency clashes in the new resolver, but how to best express the information is something we don't really know yet, so that may be something that will need to be improved over time. (It might also be possible for a 3rd party tool to help here - dependency resolution and dependency graph analysis and visualisation are somewhat different problems, and separate tools may be able to focus on the different aspects of the problem.)

It's also entirely possible that pip could have options to ignore or relax certain dependency constraints. As a general problem, it could be hard to get a good UI for this (we're currently explicitly doing user research into what users want from the new dependency resolution - @ei8fdb you may want to invite @stonebig to get involved in that, if they aren't already). And I worry that while such a feature would be invaluable for specialists like @stonebig, it could easily be abused by naive users ("Cannot install X because Y is installed" - "just say --ignore-dependency=Z") and generate more confusion than it addresses - that's a further trade-off that we need to consider.

Sorry, there's no immediate answers in this, but hopefully it adds some context to the issue and explains what we're looking at when deciding how to address it.

I should also point out that this may not be something that makes it into the initial release of the resolver. Correct behaviour while satisfying the declared dependencies has to be the first priority, as I'm sure you'll understand. So --no-deps or recompiling with altered dependencies may remain the best answer for the short term.

¹ I've made the argument myself that libraries should avoid over-restricting dependencies.

[hauntsaninja]

There are some more use cases outlined in python-poetry/poetry#697

[pfmoore]

The use cases noted in the poetry issues are good to have as examples of where strict dependency resolution can cause issues, but I'm in agreement with @sdispater that ignoring declared dependency data is very dangerous and not usually the right way to handle this issue.

If a project declares certain dependency data then there are three possibilities:

1. They are correct, and using a different version of the dependency is going to cause errors.
2. They are correct, but only certain types of usage will cause errors. It's not really an installer's job to make this judgement, but users who have reviewed the code in detail and can be sure that they will never hit the cases that cause errors may want to override this decision. This seems to me that it should be a fairly rare situation, and the users involved can be assumed to be expert (as they are willing to trust their analysis over the declared dependencies and the resolver's calculations).
3. They are wrong, and you should file a bug against the project asking them to relax the dependency. Obviously, projects may not accept such a bug report, but then we're in the same situation as any other case where a bug gets left unfixed. Users can make their own local fix, or find a workaround.

In pip's case, pip install --no-deps and manually handling the process of installing the correct dependencies is an available approach for working around such issues. It's awkward, and not for the faint hearted, but IMO we don't want to make it too easy for people to ignore declared dependencies (for the same reason that heavy machinery has safety guards...)

If there is a genuine need for dependency data overrides, that pip has to address, then I would argue that the need is not limited to a single tool, and should be standardised - maybe a "local metadata override file", whose format is standardised and can be implemented by any tool that does dependency resolution (pip, poetry, pipenv, ...). This would mean that users can declare such overrides once, and not be tied to a single tool's implementation. It also means that any edge cases and potential risks can be identified and addressed once, rather than having every project go through the same process.

[davidism]

Adding my use case from #8307 for ignoring a pinned sub-dependency when that dependency is the thing being developed locally.

In Jinja, I started pinning development dependencies with pip-compile from pip-tools. One of the development dependencies is Sphinx to build the docs. Sphinx has a dependency on (the latest release of) Jinja, so pip-compile adds a pin for Jinja. I want to provide one pip command for new contributors to set up their development environment with the the pinned dev dependencies and Jinja in editable mode. I want this command to remain simple, so that new contributors can have an easy time getting started.

$ pip install -r requirements/dev.txt -e .

However, the pinned sub-dependency on Jinja takes precedence over the direct flag to install in editable mode, so Jinja2==2.11.2 is installed instead of Jinja2==3.0.0.dev0. This causes tests to fail, because they import the old version instead of the development version that new tests are written for.

I have a similar issue with Click. It has a dev dependency on pip-tools, which has a dependency on Click. A few new contributors were confused because pip list showed that Click was indeed installed, but tests were insisting that new things were not importable and failing.

---

I see -e . as a direct command to use the local version rather than any pinned version. I see -e . written out on the command line as more direct than a pinned sub-dependency pulled from a file. I don't see a legitimate case where the user asks for a local editable install but pip refuses because there's also a pinned dependency for that library. -e . is a direct request to develop at the local version, regardless of the fact that something might depend on a released version.

[pfmoore]

If you don't mind, I'm going to leave the question of how you view -e . for now. I see your point, and it has some merits, but I want to explore your underlying use case a bit further before tackling solutions, so that I'm sure I understand it.

You say you have the latest production version of Jinja pinned in your requirements/dev.txt. But that says to me "in order to have a correct development environment set up, you must have Jinja2==2.11.2. That's clearly not the case, as it appears that the in-development version of Jinja works just as well (as otherwise your preferred outcome, that the local copy takes precedence, will cause failures). So why not have Jinja2>=2.11.2 in your requirements file? That surely gives you the expected outcome while still allowing installation of the in-development version?

I wonder if the problem here is that your workflow, or the tools you are using, are resulting in over-strict pinning, which means that having Jinja2>=2.11.2 as a requirement is harder than it needs to be. I can understand that, but I want to confirm if that is the limitation here, or if there's some more fundamental problem that I'm not understanding yet.

[davidism]

Neither pip-tools nor Dependabot (which uses pip-tools) have the capability of doing anything but pinning exact dependencies. Both those projects are fairly common now, it's why I chose them. Plenty of other projects will be using them, I'm just in the more unique case that I develop projects that the projects I depend on depend on.

I'm not really clear what pip-tools could do here, since it's designed to pin exact versions. Jinja isn't a direct dependency of itself, all there is in the template file is Sphinx. Anything pip-tools does also needs to be understood by Dependabot, otherwise we lose automation. If you have any input about that, I opened an issue a while ago before I opened an issue here: jazzband/pip-tools#1150.

[uranusjr]

the assumption is that we'd expect the user to pass this in on every call to pip

That’s my assumption as well.

[pfmoore]

Agreed. We need to be 100% clear that this will instruct pip to create a broken environment (in terms of the declared dependency metadata) and it's the user's responsibility to manage this. We should stick to the principle that pip expects the target environment to pass pip check, and we don't guarantee anything if that's not the case (this is the same principle that applies when people use options like --no-deps or --ignore-installed, so there's nothing new here).

[notatallshaw]

I think providing an “overwrites file” (analogous to constraints and requirements file) makes sense.

Agreed. As to the format, I'd limit it extremely strictly, to one requirement per line, with the file only containing a single requirement for any given project name.

Ah yes I did not think of all the weird things you can do in a requirements file. I agree with all of that and in general think it should be a restrictive subset of requirements, on top of this I would say no "flag options" (e.g. a requirements file can have -r {some_other_file}).

I would also say no requirement that contains extras (e.g. abc[xyz]), on that note of handling extras I would say (on the assumption that it makes it simpler to implement) it should only act on individual package requirements. E.g. if the there is a requirement that has an extra like abc[xyz]==1.0 and the underlying package requirements are abc==1.0 and xyz>2 then the "override constraints" can act on those requirements individually, i.e. abc and/or xyz but not be able to specify abc[xyz].

The target use case, in general. is that the user is only limiting 1 package e.g. pyOpenSSL<20, but obviously in some cases they would be situations where more than one override might be needed, So in general it would be better to start from a highly restrictive place that's easier to implement but still allows the user to technically override anything.

I read it as pip won’t physically replace anything in the package, so the installed package still contain its original dependency metadata and does not affect e.g. pip check. Everything happens in memory, inside the resolver, and the overwrite information is lost as soon as installation ends.

Yes exactly, sorry for the poor wording. The idea is the "override constraints" are only valid, at most, during a single install command and does not affect packages in any permanent way.

That's gonna interfere with #9094, although I guess the assumption is that we'd expect the user to pass this in on every call to pip?

Yes, the idea is is that only the user can provide an "override constraints" file, they would need to pass it in every time they wanted it. A package can not automatically provide one, this is only a user level/pip option not something that is available to packages generally.

[notatallshaw]

I wanted to point out some limitations of my proposal, firstly it does not support something like: "If I have package "abc"<=1 then I want to force "xyz"<2, but if package "abc">1 then "xyz" does not need to be forced".

The proposal assumes the use case that the user will put in their requirements or constraints file "abc"<=1 or "abc">1 and then set their override file appropriately. I feel like this is sufficient for situations where users really want to install conflicting dependencies as they should have a very strong understanding of their requirements and constraints.

Also worth noting is that backtracking may produce "surprising" results but again the assumption is a use case where users have a strong understanding of their requirements and constraints and should be able to mostly avoid backtracking with a mix of requirements, constraints, overrides (and maybe one day pip will add a flag to not attempt to backtrack and just error out which could be useful).

[PiotrDabkowski]

+1
Very often the project will still work, even if the exact dependency version is different.

We already have the great constraints file, where we can manually specify version of dependencies (-c option). Can we add an option to for constraints to override the package-specified version, eg --force-constraints? So similar to @notatallshaw proposal, but reusing the existing constraint infra.

For example, assume that package A requires numpy==1.23 and package B requires numpy<1.20. Even if we find that package B works well with Numpy 1.23 (at least for the set of features that we care about), we will have a hard time installing things with pip, we would need to use workarounds like --no-deps, and manually installing deps later.

With --force-constraints option we would simply specify a constraint numpy==1.23 and we are good, a warning can still be logged during installation, that the dependency version has been overwritten by the const.

Implementation would be super simple - just override the dependency version requirement if it conflicts with the constraint, and log a warning.

[balloob]

This proposal is great. If this proposal can be finished in a way that PyPI maintainers are willing to accept, I would love to discuss how we (@NabuCasa / @home-assistant) can sponsor someone to build this feature. Prefer to sponsor a maintainer or active contributor to pip.

[mauritsvanrees]

I started on a PR to fix this issue. I think my code works for the problems described here. My use case is a bit different though, so I won't finish the PR and have just closed it. But I would be happy if someone picks up my work in their own PR.

The summary is in this comment: #11537 (comment)
Let me post the todo items from there.

1. Confirmation from others in the issue that this indeed fixes their problem. I will add a comment there pointing to this PR.
2. A decision on the format: extend the constraints syntax with -o like in this PR, or use a separate overrides file with strictly limited formatting. Looks like the discussion is leaning towards the last.
3. tests
4. documentation

The first item is something anyone can help with: try out my PR and see if that fixes your problem or not, and report it back.

[FRidh]

@mauritsvanrees I like #11537 and while I have not tested it, I think it solves our issues.

I agree with @pfmoore that adding this type of syntax adds more complexity and that a separate overrides file is preferable, at least for now.

In the future I can imagine a single new style file but that should really be out of scope here. Alternatively, I could imagine a new operator (version specifier?) for this, =o, so mypkg =o 0.1.5 to override all constraints and set a version (or set no version), but this might warrant a PEP; we definitely would not want individual packages to use this operator.

[pradyunsg]

In #9948, we're discussing adding a per-requirement --no-deps to allow users to opt-out of dependency handling for certain entries in a requirements.txt file. While that doesn't solve the "relaxing" part of the request here, it does provide ignoring functionality -- in a highly pip-specific manner.

If the proposed solution there works for you, please leave a 👍🏽 reaction on this comment or let us know on that issue why your usecase isn't something workable with that.

[pfmoore]

Here's a "bigger picture" question that I briefly mentioned above but I think should be made explicit. Are we willing to support people using pip in already-broken environments?

Suppose for example, we get a bug report "pip install foo fails with an error". No-one can reproduce the issue locally. After much investigation, it turns out that the user is installing into an existing environment that has a horribly broken mess of pre-installed packages, including a number of packages deep in the dependency tree of foo. But the user has done nothing but use supported pip features (like this one) to create that environment.

I'm fine with us saying we won't support that. And I'd be happy with having a requirement that we can only investigate bugs reported with either a reproducer that starts from an empty environment, or where the user has explicitly confirmed that pip check shows that their environment is clean. But do we want to offer tools that make it easier for users to get into that situation in the first place? There's still a maintenance cost involved, as we sometimes spend time trying to understand the user's problem before we get round to asking the "obvious questions".

I'm sure that the people with use cases who need this functionality would use it appropriately and with care (no sarcasm here, genuinely). It's the people who find advice on the internet and use it inappropriately without understanding the implication ("just use --break-system-packages, it fixes your issue"... 🙁) who could easily find something like this a big footgun.

[gst]

Are we willing to support people using pip in already-broken environments?

by broken do you mean pip check fails ? I guess..

but actually, isn't it already the case ipso-facto (that pip running/installing in already "broken" env is allowed/supported) ?

I mean it's not like, for instance, debian apt that refuses to continue installing new packages if one(or more) of them are actually "broken" and not fixed (in some sense of "broken", as far as I know).

After much investigation, it turns out that the user is installing into an existing environment that has a horribly broken mess of pre-installed packages,

I would guess/hope that someone would ask to pip list or pip check the given user installation/environment. Or simply/directly request him to try with fresh/clean python (v)environment too.

[pfmoore]

isn't it already the case ipso-facto (that pip running/installing in already "broken" env is allowed/supported) ?

No. There are many reasons why pip doesn't run pip check before allowing an install, but that doesn't mean that it's supported. Think of it like "undefined behaviour". You can do it, and it might allow you do achieve some really neat goal, but that doesn't mean you can rely on it. Unless we (the pip maintainers) decide between us that we want to support it, in the sense of applying our stability and backward compatibility policies to it, for example.

I would guess/hope that someone would ask to pip list or pip check the given user installation/environment.

It's surprisingly uncommon that we ask that. To an extent that's what I'm saying, do we want to start with "please run pip check", with the implication that if it reports problems, we'll say "that's your issue, fix that and you'll be fine"?

[mauritsvanrees]

If a pip environment is already broken, people may need tools to unbreak it. At the minimum, if pip check currently fails, pip install should still work, otherwise you have no other option but to throw away your environment.

I think it should work like this, and I have no idea if that is the case now:

• pip install good combination -> works
• pip check -> passes
• pip install bad combination: does pip check after analysing the dependencies, which fails, so the installation is stopped before anything is actually installed.
• Now somehow break the environment.
• pip check -> fails
• pip install another good combination -> This should work. Yes, pip check fails after analysing the dependencies, but it should realise that pip check already fails in the current environment, so pip install should continue, as the user may be trying a first step to unbreak the environment.

That said: I think it is fair to update the bug report issue template to ask people to include the output of pip check.

[pfmoore]

It's a while since I checked the details, but from what I recall, pip install runs a pip check when it's complete, and reports if there is an issue. At that point, pip uninstall and pip install still work fine. We don't ever fail because of a broken environment, you're just in "you broke it, you get to fix it" territory.

There are plenty of ways you can use documented pip features to break an environment. The obvious trivial one is pip install --no-deps A when A depends on B. Or pip install A; pip uninstall B. But that doesn't (IMO) mean we're under an obligation to do anything sane when working with that environment (it's hard even to be sure what "sane" means!) I think you can reasonably expect that pip uninstall and pip install --no-deps work. Beyond that, by all means give it a go, but you may hit issues. I think you'll find that plenty more than that will work reasonably well, but I'm not guaranteeing anything.

To be clear, I'm talking here about "broken" in the sense of "some dependency metadata isn't satisfied". You can break environments in other ways (manually delete files, for example) that will leave pip unusable. You're on your own in that case.

[notatallshaw]

Small update for anyone who needs this feature, uv has a pip-like install interface and supports a --override flag.

I will also make the observation it reads almost exactly as though someone implemented what I proposed here, so we may get real world data on if that was a good proposal or not.

[stonebig]

Interesting, at least for the:

• speed,
• compatibility aim with pypa,
• carbon footprint
• extra.

Adding the pipdeptree style features would be great, especially if accessible by api/not using the command line