What does a URL constraint mean?

[pradyunsg]

Carrying forward discussion from #8210

---

To discuss about a constraints.txt file with the following content:

packaging @ git+https://github.com/pypa/[email protected]

0. Should this be allowed?
1. What functionality does this provide?
2. Is this the best way to provide it?

[uranusjr]

To me, this means whenever packaging is requested, use this link I provided. Further version specifiers should still apply (and fail resolution if they are not satisfied by the link target). The behaviour would be similar to

pip install <packages-i-want> "packaging @ git+https://github.com/pypa/[email protected]"

but without packaging being marked as user_requested (unless it is listed in <packages-i-want>, of course).

[dhellmann]

I think of it as constraining the location from which distributions can be downloaded, just as a version expression like >=20.1 would constrain the version(s) that could be downloaded.

[pfmoore]

OK. I have done a more detailed investigation into the way that we'd implement NAME @ URL style constraints. The idea in principle is straightforward, but it has a number of interactions with other features of pip that make the overall behaviour far more complex.

The current implementation will have behaviours for all of these points that arise "by accident" as a result of the implementation choice to treat constraints as "just requirements that we don't install". The new resolver doesn't have that option, because the underlying model is completely different.

It's also true that there are likely reasonable choices to be made as to "how should this interaction work". It's not necessarily hard to choose a "correct" behaviour for any of the following points. But the implementation is decidedly non-trivial in the new resolver's existing model, so the cost of these features is disproportionately high, and we don't really have any idea at the moment if there's any actual use cases for these sorts of interactions (that's actual use cases as opposed to "yes I can see how this might be useful" 🙂)

• Multiple URLs - should they be allowed? What would they mean? After all we can only get a project from one place. But consider this in relation to the next points.
• Hash checking. If someone were to have a constraint file that said foo @ https://some/url and a requirements file that said foo --hash=xxx, what should happen? Should we hash-check the URL? Is it of any practical use? Should we select one of multiple URL constraints based on the hash?
• Wheel compatibility tags. Should URLs for constraints be required to match the target wheel compatibility rules? Should we allow multiple URLs and pick the one that matches the target system? Have we just implemented a package index via constraint URLs?
• Should we reject the edge cases that we don't support, or are we OK with just leaving them as "undefined behaviour - do not use"? In particular, even detecting some of these cases is costly.

There are quite likely other questions. These are basically just some that came up in the process of trying to see how to implement this and saying "ouch, that would be a problem" a few times 🙂

As I said, it's almost certainly possible to come up with a well defined spec for this feature. And maybe that spec would even match the current behaviour. But we don't have that spec right now, and no-one has really done any of the work needed to come up with one.

So I propose that we consider URLs as constraints as "deprecated pending work being done to properly define the feature"¹, and assume that they won't be supported in at least the initial version of the new resolver. People currently using URLs in constraint files will need to either stick to the old resolver, or find a workaround.

The only actual user of URLs as constraints that I am aware of is @sbidoul, who explained his use here. Maybe there are (more or less clumsy) workarounds that could be used instead? @dhellmann seemed OK here with not including this in the new resolver, at least initially.

To reiterate, this is not a blanket refusal to accept this feature. It's simply noting that we need to clarify the design in order to rewrite it for the new resolver, and that we're not blocking the release of the new resolver on this feature.

¹ This has horrible echoes of the dependency links mess. I really want to not have that happen again, so I want to be very clear that the plan this time is to drop the feature without any replacement if necessary, and not to leave it hanging around forever "until we have the replacement defined".

[sbidoul]

@pfmoore don't the edge case you mention also arise when direct URLs are used as regular install-requires?

[pfmoore]

@sbidoul I have no idea to be honest. I've not spotted any test failures that do that, but if there are any, then yes, that's something we'd need to address (and having addressed it, we may well then have a better chance of implementing URLs as constraints).

[uranusjr]

OTOH I very much doubt there are robust tests or user experience on this. The legacy resolver handles non-user-specified direct URLs very poorly, so there are likely many edge cases currently precluded by other restrictions. I’m +1 on delaying thoughts on this until we are able to make the new resolver widely available for a while, to let the users come up with crazy cool use cases that expose considerations we missed.

[pfmoore]

Agreed, this whole area is really emergent behaviour from a set of features that were implemented independently and never really considered in combination.

This is probably something that needs to be covered under #6536 (new resolver rollout) which is something that needs to be prioritised at some point. There will be areas where the new resolver works differently, that either aren't covered by tests, or where we've made a decision to keep a difference. and we need to look at how we get feedback on that.

[thenewguy]

I had opened an issue here #8757

Just wanted to add a mention here that it is very helpful to be able to provide the url for installation a package from as a constraint when waiting for a contributed patch to be applied by upstream and released.

I.e. something like https://github.com/some/repo/archive/bugfix-issue-foo.tar.gz#egg=somepackage to allow installing the bugfix for package "somepackage" until upstream can cut a new release. Otherwise all packages have to be vendored and renamed and it is very complicated and duplicates a lot of work.

[thenewguy]

Right now if you drop https://github.com/some/repo/archive/bugfix-issue-foo.tar.gz#egg=somepackage in a constraint file and then install a requirements.txt that lists somepackage, you get https://github.com/some/repo/archive/bugfix-issue-foo.tar.gz. Please do not remove that workflow if you are able to keep it in some form

[pfmoore]

@thenewguy Thanks for your input. I've seen a few people describe use cases for URLs as constraints that all seem to have the same underlying motivation, and I'm fairly clear that we should have this feature in some form.

However, the key here is defining the details. I posted above a list of questions about how URLs as constraints should interact with other features of pip, and so far, no-one has come up with any answers. I don't personally have any intuition on what the answers should be, so it's essentially impossible to implement the feature for the new resolver. To be clear, the existing implementation is tightly linked to the old resolver code - it's not a case of "keeping what's there", we absolutely have to do a complete re-implementation, and as such, we need to know what to implement.

Please do not remove that workflow if you are able to keep it in some form

As I say, we aren't able to keep it. What we can do is to re-implement it. But to do that we need to understand what to implement - and "what the old implementation does" isn't even meaningful in terms of the new resolver...

[thenewguy]

As I say, we aren't able to keep it. What we can do is to re-implement it. But to do that we need to understand what to implement - and "what the old implementation does" isn't even meaningful in terms of the new resolver...

Gotcha - well in my experience - this is typically used for temporary bugfix releases. It sounds like there is a lot more to consider than my knowledge of pip's internals allow, but imo, if a named package url is provided, it should be forced and installed regardless of any version specifier. If that is too much, perhaps failing if the version doesn't match. I've never used it in combination with strict versioning (because as a bugfix there really isn't a version to it as I am not in control of the release process for upstream). So perhaps, conceptually, it is an override rather than a constraint. I've not come across another way to do this.

However, it is also handy when working with a pre-release package that isn't published somewhere like pypi yet. Although this case is easier to work around than the previous use.

[dhellmann]

I don't use the feature, but it sounds like maybe one interpretation of how others use it is to (partially?) disable the resolver for a specific dependency and use the given distribution as the only available location and version. Maybe that implies some other behaviors about how that affects the other dependencies, I'm not sure. For example, should the resolver assume that the distribution at the given URL meets the version requirements of anything that depends on it, even if perhaps the version number in the distribution metadata says otherwise?

That would be kind of a "just do what I tell you" mode, which given the conversation elsewhere about how to reduce the scope of pip itself and encourage an ecosystem of surrounding tools, might make a lot of these sorts of features buildable outside of pip. For example, if pip could report the results of resolving a requirement set by producing a list of URLs without installing them (maybe it can do this already, or it means building another tool on top of the resolver library?), then someone could build that full list from a partial requirements.txt, edit the results to replace the URL for the dependency they are patching, and then install from that list of URLs without resolving dependencies at all (with pip, or something else) to get exactly what they want. That would give the user the convenience of pip helping to build the full dependency list, and mean that PyPA developers wouldn't have to entertain every variation of dependency management that anyone can come up with.

[uranusjr]

The “do as I tell you” usage is more difficult to implement, and should be discussed separately IMO (#8076 covers it). Constraints are currently implemented as adding to existing requirements. This makes them fit nicer with the rest of dependency resolution logic, and relatively easier to make sense. This is enough for most use cases in practice as well, since existing dependencies are correct most of the time, and a URL constraint just needs to add additional information to tell the resolver to specifically use that one particular artifact, much alike how version constraints do.

[pfmoore]

if a named package url is provided, it should be forced and installed regardless of any version specifier

Conceptually, that behaviour doesn't feel like a "constraint" to me, but more like an "override". I think it's important that constraints do what the name suggests, for both discoverability and understandability reasons. Overrides are being discussed separately in #8076 as @uranusjr says.

For me, conceptually, a URL constraint says "you can only get this project from this specific URL. If what's there doesn't satisfy the dependencies pip has calculated, you're out of luck".

To be fair, that logic mostly answers my above questions:

• Multiple URLs - not allowed, reject as an error.
• Hash checking - we should check the hash and fail if it doesn't match.
• Wheel compatibility tags - fail with an error if the URL points to an incompatible wheel.

For the last question, "should we reject the edge cases that we don't support" I would be pragmatic and say that if we end up in a code branch that the above logic doesn't give us an answer for, we fail with an error saying it's unsupported. On the other hand, we don't go out of our way to check for special cases, and we document the intention and explicitly state in the documentation that any other usage is not supported, and behaviour is undefined and may change or be removed without notice.

If that's an acceptable approach, we can go ahead on that basis. (However, note that the period of paid work on developing the new resolver has completed now, so implementing this will be done on volunteer time - personally, I'd like to look at it but I don't know when I'll next have time to do so).

[dhellmann]

The “do as I tell you” usage is more difficult to implement, and should be discussed separately IMO (#8076 covers it).

Sure, I'll read that, too.

Constraints are currently implemented as adding to existing requirements. This makes them fit nicer with the rest of dependency resolution logic, and relatively easier to make sense. This is enough for most use cases in practice as well, since existing dependencies are correct most of the time, and a URL constraint just needs to add additional information to tell the resolver to specifically use that one particular artifact, much alike how version constraints do.

Sure, that's another way to look at it. I was trying to point out that if the phases of resolving the dependencies and installing the packages were exposed explicitly, then many (most?) other use cases, such as this one, could be addressed by modifying the output of the resolver before passing the list to the installer. Those modifications could be left up to individual users or authors of other tools that integrate with pip, but pip's resolver wouldn't have to contain the additional complexity.

[pfmoore]

I was trying to point out that if the phases of resolving the dependencies and installing the packages were exposed explicitly, then many (most?) other use cases, such as this one, could be addressed by modifying the output of the resolver before passing the list to the installer.

That's an interesting idea. We could have something like pip install --resolve-only --out=some_file.txt and pip install --from-resolve-data=some_file.txt. I could see some pretty significant issues to consider here (what if the pip options used in the two phases were different, for a start?) and I can't imagine that we'd ever support editing that intermediate file (it's expected that people do this, but it would have to be on a "we won't help if you break stuff" basis, I'd have thought), but I can see it would be useful.

(Of course, taking that idea to its logical conclusion, we'd break pip up into a suite of tools doing the various "bits" of the process, much like the Unix idea of combining many small tools - and I doubt we're actually likely to go down that route in reality).

[dhellmann]

I was trying to point out that if the phases of resolving the dependencies and installing the packages were exposed explicitly, then many (most?) other use cases, such as this one, could be addressed by modifying the output of the resolver before passing the list to the installer.

That's an interesting idea. We could have something like pip install --resolve-only --out=some_file.txt and pip install --from-resolve-data=some_file.txt. I could see some pretty significant issues to consider here (what if the pip options used in the two phases were different, for a start?)

Yes, that's close to what I was thinking. It might be easier to break down which options apply to each phase by thinking about new sub-commands with names like pip resolve and pip deploy, with pip install encompassing both phases transparently. Most of the options to the existing install command would apply to the resolve command, but not to deploy. Separate sub-commands also has the benefit of separate argument parsers, and if the options aren't available for pip deploy, they can't be different than the values given to pip resolve. :-)

and I can't imagine that we'd ever support editing that intermediate file (it's expected that people do this, but it would have to be on a "we won't help if you break stuff" basis, I'd have thought), but I can see it would be useful.

Yes, exactly. The deploy step would just take the data as input and do what it needs to do so the listed packages were on the import path (downloading things, turning them into wheels, writing files to the filesystem, etc.). It would only work from the list, though, without applying any additional rules or processing. It has to assume that the list is "correct". It would be up to the user to make the list contain what they want, and if what they want is broken somehow then that's not pip's problem.

I originally said a list of URLs, but the output of the resolve phase might be easier to consume if it includes more of the data that the resolver has. For example, the original requirement, the reason for a dependency being added if it wasn't in the original requirement list, the URL to the package, an optional second URL for a local cached copy, etc. A lot of that data could be optional because deploy wouldn't need it, but a program that wanted to modify the data could use it to make decisions about the modifications.

(Of course, taking that idea to its logical conclusion, we'd break pip up into a suite of tools doing the various "bits" of the process, much like the Unix idea of combining many small tools - and I doubt we're actually likely to go down that route in reality).

Separate sub-commands may make it easier for users to understand conceptually, but I wouldn't go so far as to create separate executables or main programs. And I wouldn't necessarily say that pip install should write the intermediate data to a file before deploying, but internally it should build the same data structure with the resolver and pass it to the deployment code.

[pradyunsg]

IIUC, we're going in the direction of #53, lockfiles, #7819 here.

[uranusjr]

@mwchase I think you read the right code, and tour assessment is correct. I have not tried to figure out the details though, so maybe more edge cases would pop up once the implementation is written.

[mwchase]

I've figured out how most of the feature interactions I've thought about so far work out, but one of them is harder, because it involves code that doesn't exist yet.

Factory._make_candidate_from_link() has a TODO that says "Check already installed candidate, and use it if the link and editable flag match." Currently, the source_link field (which I think is what I want to key off of) is None for AlreadyInstalledCandidates, which I assume is what the method would return for the given case. My assumption/hope is that, in order to implement this functionality, the field would have to be populated when applicable, so checking against it would still work.

I assume there's also some subtlety around extras, though my initial thought/hope is, that since extras can't appear in constraints (modulo some proposed changes in other issues that wouldn't affect things), the constraint should apply to the base package and not touch the virtual one.

[uranusjr]

original_link is where pip should download data to install, so it should be set to None for AlreadyInstalledCandidate because pip does not need to download anything for it. The URL that an already-installed candidate was installed from is recorded in direct_url.json (PEP 610), and read with direct_url_helpers.dist_get_direct_url().

[mwchase]

I'm currently writing tests and taking notes on implementation details. Here's another question to consider: excluding URL constraints implicitly excludes editable constraints. If URL constraints are possible, should editable constraints be possible, or should they be explicitly excluded?

[pfmoore]

Editable constraints definitely should not be possible - "editable" is about how you install things, not about what you install. Constraints are entirely about what gets installed, so editable makes no sense there.

[mwchase]

Excellent, I didn't really want to plumb that through. So, editable constraints should be specifically disallowed after URLs are allowed, and whether a candidate is editable is unrelated to whether it satisfies the constraint.

EDIT: This does mean that, if an environment uses editable installations and does the pip freeze > constraints.txt from #8210 (comment), then there needs to be a processing step to remove leading -e. But at least in that situation, the installation will be possible.

[uranusjr]

EDIT: This does mean that, if an environment uses editable installations and does the pip freeze > constraints.txt […] there needs to be a processing step to remove leading -e. But at least in that situation, the installation will be possible.

Yes. The same issue is raised in #9209 as well, and I think it is not unreasonable for pip to ignore these invalid things with a warning instead of rage quit here. But this should be a separate feature request and handled on it own.

[mwchase]

I'm finally trying to implement this instead of writing tests and thinking really hard about the design, and I've run into another thing that didn't occur to me. It looks like the legacy resolver ignores fragments when comparing URLs. Is that desirable behavior? My current implementation does not, but that's not a conscious decision on my part.

[sbidoul]

The wheel cache currently only considers the subdirectory and hash fragments. So for consistency I would do the same when comparing URLs for this purpose.

[uranusjr]

The new resolver currently considers the whole Link object’s equality though, so maybe it’d be best to change the logic together.

[mwchase]

Okay, I've got an initial implementation. It's not yet ready for a PR; here's what needs to happen first:

• Change the link comparison logic, if we don't want to match the current cache behavior, and update the affected tests.
• Improve organization and style issues in my code changes.
• News fragment
• Documentation update (possibly change the documentation I linked upthread, and lay out the pip freeze > constraints.txt use case, with caveats)

[uranusjr]

Changing Link’s equality would probably be problematic (nobody really knows what implications this has), so I’d write a helper def links_equivalent(link1, link2) somewhere (probably models/link.py) and just use that instead. (Remember to put a clear docstring to explain why it is needed and when it should be used instead of ==).

[mwchase]

What I currently have is some helper functions that approximate the current cache-equality. They currently live next to the Constraint class, because that's where they're used. I'm fine moving them, I just first want to check that I'm actually implementing cache-equality, then try to cut down on some duplication in my implementation.

[mwchase]

Actually, on further inspection, I'm not sure that the _get_cache_path_for_parts function is relevant for comparing links in resolution. It looks like it's basically an optimization around the fact that multiple URLs can point to the same file, in a way that can be determined entirely from the syntax of the URLs.

I would rather have the behavior of URL constraints be consistent with URL requirements in the new resolver, than with URL constraints in the old resolver. Is that reasonable?

Making these changes will require changes to code that I needed to clean up anyway, as well as putting slightly more effort into rewriting one test.

[uranusjr]

Actually, on further inspection, I'm not sure that the _get_cache_path_for_parts function is relevant for comparing links in resolution.

You’re correct, it is not. During resolution, links are compared using the built-in ==:

pip/src/pip/_internal/resolution/resolvelib/candidates.py

Lines 155 to 159 in 0ffff03

	def __eq__(self, other):
	# type: (Any) -> bool
	if isinstance(other, self.__class__):
	return self._link == other._link
	return False

which in turn delegates logic to KeyBasedCompareMixin._compare:

pip/src/pip/_internal/utils/models.py

Lines 42 to 47 in 0ffff03

	def _compare(self, other, method):
	# type: (Any, Callable[[Any, Any], bool]) -> bool
	if not isinstance(other, self._defining_class):
	return NotImplemented
	return method(self._compare_key, other._compare_key)

which would compare KeyBasedCompareMixin._compare_key, which is identical to Link._url:

pip/src/pip/_internal/models/link.py

Lines 71 to 79 in 0ffff03

	self._url = url
	self.comes_from = comes_from
	self.requires_python = requires_python if requires_python else None
	self.yanked_reason = yanked_reason
	super().__init__(key=url, defining_class=Link)
	self.cache_link_parsing = cache_link_parsing

In other words, two links are considered identical during resolution only if their URLs match completely. This is the safest route, but is known to have caused issues to some people (#9429 (comment)) that may also apply to constraints, so I think we’ll need to revisit them together. So I would like the constraint implementation to also introduce a link comparison function to replace the == comparison used above.

Does that make sense?

[pradyunsg]

Umm... Why are we touching the legacy resolver here?

[mwchase]

So far as I know, I didn't.