Make the lock file more merge-friendly

[seansfkelley]

• I have searched the issues of this repo and believe that this is not a duplicate.

Issue

When two devs install dependencies on separate branches, it is very easy to end up merge-conflicted, in particular, the metadata.content-hash key often changes. It is very unclear how to resolve this manually, so I often delete the lockfile (or perhaps just that key) and rebuild it and, basically, hope that it comes out the same.

It seems like in some scenarios that merge conflicts could be resolved automatically based on pyproject.toml. Yarn does this, for instance.

[simonpercivall]

+1 Resolving the content-hash and trivial conflicts in the individual hashes section would be very welcome.

[nyanpasu64]

Workaround:

• remove content-hash line
• poetry add pathlib2
• poetry remove pathlib2

or some other similar innocuous package.

Will this issue be resolved soon?

Does a missing content-hash hurt security or not?

[seansfkelley]

Yarn's implementation of this is way less insane than I was expecting:

1. Extract the two versions of the conflicted file.
   https://github.com/yarnpkg/yarn/blob/a7334da31bf783af7a3efab451589fe7ac34c748/src/lockfile/parse.js#L397

2. Blindly try to parse the files, and if that's successful, shallowly merge them.
   https://github.com/yarnpkg/yarn/blob/a7334da31bf783af7a3efab451589fe7ac34c748/src/lockfile/parse.js#L399

If the merge conflict resulted in a syntax error, it fails. Yarn's lockfile structure is designed to make this easier: it's flat, it's sorted alphabetically and node is okay with duplicated dependencies in the tree.

[itajaja]

is content-hash mandatory though? the main problem I am encountering is that dependabot will have to ALWAYS rebase all the PRs, which could be avoided if content-hash was not in poetry.lock. in theory shouldn't be necessary, as it can always be computed by the other hashes, right?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[chdsbd]

I think this is still a valid issue and should not be closed.

[cjolowicz]

I have written a stand-alone script as a stopgap. It only goes half the way, resolving the merge conflict for metadata.content-hash. The remaining conflicts, if any, should be quite trivial to merge manually. Use at your own risk.

https://github.com/cjolowicz/scripts/blob/master/python/poetry-merge-lock.py

Update:

Please use poetry-merge-lock from PyPI instead (see comment below). This should allow you to merge without manual conflict resolution, in most cases.

[cjolowicz]

Here is a stand-alone tool which should handle most merge conflicts in the lock file:

• https://github.com/cjolowicz/poetry-merge-lock

Install it with pip or pipx:

pip install --user --upgrade poetry-merge-lock

This tool is in early development. If you're interested in trying it out, please let me know on its issue tracker if you encounter any problems.

@sdispater, @finswimmer : Would you be interested in a PR to add this as a Poetry command?

[finswimmer]

Hello @cjolowicz ,

your contribution is very welcome. I find this feature very useful. 👍

However, I cannot promise if and when this can be included. Including new features is decided by @sdispater .

But please go on!

fin swimmer

[donbowman]

as long as there is a line content-hash we will never be able to conveniently use dependabot. When dependabot runs, it creates 1 branch and merge request per update, w/ the idea that you let your CI run, and then auto-merge.
However, because of content-hash, with poetry, every single one of these branches has a merge conflict and must be manually dealt with, increasing the human time from a few seconds to ~10-minutes per, and, causing another run of the CI to be required.

The repository I just tried this on, 5 MR were created. If it uses tox, it takes no human time and 5 CI runs.
If I use poetry it takes 10 CI runs, and a manual clone/checkout/rebase/poetry update hash somehow/push, which took me more than an hour.

Multiple this by my 30 repos, and make it a daily thing to have to deal with and it becomes intractable.

The solution cannot be in a new poetry command to resolve the lock, that would still require all the manual work.
Instead, we need to be able to merge poetry.lock files that are not otherwise conflicting.

Beacuse all these MR are created at once, in parallel, we cannot teach dependabot to do it either, at the time of each MR they all come from the same spot on master so there is no conflict, its only when the first is accepted that we need to rebase the 2nd, and when it is accepted we need to rebase (twice) the 3rd, and so on.

Can we not make the issuance of content-hash optional?

event without dependabot its troublesome.

Ideas:

• just skip it, no fresh check
• change the output line to be content-hash(date) = X, have poetry find the most recent one, ignore dups, and, on regen, remove dups
• use date on the pyproject.toml and on poetry.lock as the fresh check, e.g. don't use content, trust the filesystem date
• allow generating poetry.lock w/o content-hash as an option, teach dependabot to call this option.

so perhaps if there was a --no-output-hash option to the poetry update --lock ?

[cjolowicz]

@donbowman Doesn't Dependabot rebase your PRs automatically if they conflict? This is what happens for me. I only need to resolve conflicts when rebasing or cherry-picking my own commits, in which case poetry-merge-lock mostly works fine. I sometimes follow it by poetry add insecure-package && poetry remove insecure-package to ensure that the lock file is up-to-date. (insecure-package is just an empty dummy package that's never part of my dependency tree.)

[donbowman]

@donbowman Doesn't Dependabot rebase your PRs automatically if they conflict? This is what happens for me. I only need to resolve conflicts when rebasing or cherry-picking my own commits, in which case poetry-merge-lock mostly works fine. I sometimes follow it by poetry add insecure-package && poetry remove insecure-package to ensure that the lock file is up-to-date. (insecure-package is just an empty dummy package that's never part of my dependency tree.)

this is on gitlab.
it creates e.g. 5 merge requests at the same instant.
as soon as i accept the first one, the other 4 are all in merge conflict.
How would it magically wake up and rebase these 4? You mean the next time i run it it would see the conflic, rebase, then i would accept 1 more, then it would rebase the next 3, and so on?
also, it would have the same merge conflict, how would it know how to resolve it?

I added #2654 to poetry as a suggested solution.

[cjolowicz]

@donbowman I assumed you were referring to GitHub Dependabot. The bot recreates the PR (not a rebase, sorry for the sloppy terminology) when the changes conflict with the target branch.

[donbowman]

i see.
but it still leaves the original issue, that if i issue many update's, each as a single MR, after the first is accepted, the rest all conflict.
If the dependabot script is taught to then delete and recreate, ... its very slow.
if I were to run it daily, and had 10 updates, it would take 10 days to get them all in w/ this technique.

i guess, inefficiently, i could create some webhook that when the first mr is accepted, it could somehow run the dependabot again to recreate all the remaining updates, and then repeat every hour or so until all are done.

[seansfkelley]

It looks like Dependabot actually does automatically rebase (well, rewrite-force-pushes) the PR by default, according to the documentation. Judging by the docs and that issue, it sounds like it does it with webhooks or another timely mechanism rather than waiting for the next run, though neither one is explicit on how quick the turnaround is.

Maybe you're seeing merge conflicts triggered by content-hash because you've disabled this behavior?

[donbowman]

It looks like Dependabot actually does automatically rebase (well, rewrite-force-pushes) the PR by default, according to the documentation. Judging by the docs and that issue, it sounds like it does it with webhooks or another timely mechanism rather than waiting for the next run, though neither one is explicit on how quick the turnaround is.

Maybe you're seeing merge conflicts triggered by content-hash because you've disabled this behavior?

content-hash conflicts even w/o dependabot. if 2 people change 2 branches, its in conflict.

the hosted dependabot of github may in fact have some faster trigger.
but the dependabot-core on a private gitlab does not. its a cron.

[lephuongbg]

My 2 cents, if the content-hash format can be structured as a list of sorted main dependency names and hashes calculated from resolved sub-dependencies, then it would reduce merge conflict chance a lot. Like

[metadata]
content-hash = [
    "astroid:03472c30eb2c53",
    "flask:bb564576db6a918",
    #...
]

[cjolowicz]

Here's an approach that has worked well for me and only uses git and (recent) Poetry:

git restore --staged --worktree poetry.lock
poetry lock --no-update

When rebasing a feature branch on main, this preserves pins from the main branch, and recomputes pins for your feature branch. You would then follow up with these commands to continue the rebase:

git add poetry.lock
git rebase --continue

[AbdulRahmanAlHamali]

Most the answers here are answering what to do on a merge conflict, but we should be addressing why we're having a merge conflict in the first place. It is not reasonable to have a merge conflict whenever two developers modify different dependencies. Having a conflict means you will need to realize that, merge master into your branch, recreate the lock file (which takes at least 3 minutes, because python...), re-push, re-run your CI (which could take a lot of time as well), etc. This is very bad UX.

@lephuongbg's comment is a possible solution. Is it possible to look into this?

Thank you a lot!

[mrcljx]

@neersighted I understand the suggestion is quite dangerous, but as more and more people are reporting that the conflict issue is turning into a major pain point, I was hoping that some kind of workaround could be made possible (+1 on giving this a very clear name).

---

Over the weekend I wanted to understand how other ecosystems (specifically Node.js) are solving the same problem (while keeping in mind that the metadata cost in Python is different). NPM stores the "self" package in the lock file as well (Yarn only does when using workspaces it seems):

// package-lock.json
{
 "name": "poetry-test",
 "version": "1.0.0",
 "lockfileVersion": 2,
 "requires": true,
 "packages": {
   "": {
     "name": "poetry-test",
     "version": "1.0.0",
     "license": "MIT",
     "dependencies": {
       "react": "^18.2.0"
     }
   },

Poetry instead stores a hash of relevant_content which more or less contains the dependencies of the package itself:

return sha256(json.dumps(relevant_content, sort_keys=True).encode()).hexdigest()

If we compare these two approaches, NPM's lockfile can be merged by git most of the time (and otherwise humans can merge it by hand) while Poetry is guaranteed to conflict when two commits modify the pyproject.toml.

While it seems quite possible to render relevant_content and not the sha256 to the lock file (which actually sounds related to the ask of #1301), various comments in this thread have suggested this to be a feature to ensure/increase correctness:

• Merging the locked dependencies doesn't necessarily results in a correct dependency tree.

• but that doesn't mean that the solution is correct (or up-to-date for your pyproject.toml).

Generally, a git-merged lock file is not guaranteed to be "correct" regarding to the pyproject.toml. I'd love if someone could provide the thread with a non-esoteric case where updating pyproject.toml from two files that cleanly merges and not running poetry lock would cause a problem. I'm fairly certain that one exists, but I failed to contrive one.

That said, the correctness argument seems to be ignoring the fact that:

1. this problem exists in most other package managers as well.
   a. the solution to that is to check whether running the resolver again is a noop:
   1. see npm ci
   2. see poetry lock --check
2. AFAICT this seems to be true for other operations as well. The input to the algorithm is not just pyproject.toml, it is also the current lock file itself. poetry update transitive-a and poetry update transitive-b might also contradict each other, but the hash will not cause a conflict there.

The thing that makes (1) a bit unique in Poetry is that poetry install works quite different from other package managers. As an NPM user, you make a habit of running node install when pulling the main branch which will "heal" the lock file (and one definitely runs that after a merge conflict on the lock file). For Poetry, poetry install will not do that, instead new users must be made aware of the lock subcommand.

That said, this should more or less also work if Poetry stored the relevant_contents instead of the hash for the case that someone updated a version in pyproject.toml but then didn't run poetry lock.

Regarding (2) to me it seems that this hash is giving a false sense of correctness. The only true way to ensure things are correct is to run poetry lock habitually (or if possible run it on CI).

To summarize, I find it quite hard to defend the existence of the content-hash. (I personally believe that Poetry's UX would benefit if install performed a lock --no-update, but I understand that the metadata cost might be prohibitive here).

[neersighted]

This option may be appropriately named, that it is dangerous. This will solve the problem here and now and allow us to work on the hash problem as a whole.

This is easy to assert from a non-maintainer standpoint, but we already pay a very high maintenance/support burden for other poorly designed/second-class features. I suspect you will not be able to find any consensus to merge another one.

Regarding (2) to me it seems that this hash is giving a false sense of correctness. The only true way to ensure things are correct is to run poetry lock habitually (or if possible run it on CI).

You're not incorrect, but this is not the intended way to use Poetry/a primary workflow. In general what is intended is poetry update, run CI, and determine if any failures result from broken versions, at which point they are excluded from the constraints in pyproject.toml.

If we drop hashing and make the lock file self-descriptive, we'll greatly increase the maintenance burden for the lock file, but that may be a price worth paying. However, we make the common case much more risky/dangerous. People are going to blindly trust Dependabot/Git to update the lock file, and will be opening bugs against Poetry when they get unexpected results. I'm not sure that "make sure you run poetry lock --no-update before you open a bug" will fly, since Poetry will (partially/rightfully) get the blame for doing incorrect/unexpected things.

I'd still like to hear what some of the SMEs in this area think, but since we're now talking about a "no-hash" approach instead of trying to change the solver, I'll also ping @abn @finswimmer @sdispater for opinions.

[neersighted]

A final thought: we can never come up with the 'canonical' (what the solver would pick, given all information was available to it) solution to lock file merges, but maybe we can at least start validating the lock file makes sense? That is to say, we store the dependency tree, so we could at least validate that it still makes sense/the solution doesn't have internal conflicts. We might already do that, but it certainly doesn't happen in the parts of the installer I am familiar with... cc @radoering @dimbleby

[radoering]

2. see poetry lock --check

This command just checks for the correct hash. (That's why it is quite fast.)

(I personally believe that Poetry's UX would benefit if install performed a lock --no-update, but I understand that the metadata cost might be prohibitive here).

I disagree. Even for large projects poetry install is quite fast if most of the packages are already installed. poetry lock --no-update takes significantly longer and it's just not necessary if you have taken precautions to make sure that the lock file checked into your repo is alright.

We might already do that, but it certainly doesn't happen in the parts of the installer I am familiar with...

More or less. Actually, each install contains a solver run for the target environment with the lockfile being the only available repository. If the lockfile is not consistent with the pyproject.toml you get some sort of a solver error many users don't understand. The only hint for the reason of these errors is the warning about an inconsistent lock file (based on the hash). Just a quick idea, but we could derive "solver error during install" -> "inconsistent lock file". (It's probably not that easy because there are probably a bunch of different exceptions we have to consider but it might work.) Anyway, it's less expensive to validate the lockfile for the current environment (during install) than to validate it in general.

[neersighted]

The only hint for the reason of these errors is the warning about an inconsistent lock file (based on the hash). Just a quick idea, but we could derive "solver error during install" -> "inconsistent lock file". (It's probably not that easy because there are probably a bunch of different exceptions we have to consider but it might work.) Anyway, it's less expensive to validate the lockfile for the current environment (during install) than to validate it in general.

Right, this matches my current recollection. If it's a full solver run (and not abbreviated like I thought), then it's more robust than I realized. The missing piece is inferring that a solver error during install is likely caused by an inconsistent lock file; I wonder if re-trying with an automatic re-lock makes sense in that case?

[dimbleby]

poetry lock --no-update is fast when the relevant part of the poetry cache is populated - but that's not usually the case in CI pipelines. Then that command has to go and retrieve metadata from pypi or other repositories (which can indeed be slow). For similar reasons poetry install is probably slow in CI pipelines.

---

In general it is not possible safely to merge two independent changes to a lockfile without re-solving. One can construct examples in which an upgrade to A is possible, an upgrade to B is possible, but the upgraded A and B are incompatible with one another. So if some bot has proposed both upgrades then it would be an error to merge both, even if they are compatible so far as git can tell. In that sense, being merge-friendly would actually be an anti-feature!

---

That being said, I still am confused as to what problem folk are actually seeing here. eg my experience with dependabot is that it updates only the lockfile, not pyproject.toml, and therefore its updates never make changes to the hash in the lockfile. Clearly there's something going on that y'all want to talk about - but it's not what I'm seeing.

Edit: I mostly withdraw this last comment. Looking back over some dependabot MRs: it is capable of making lockfile-only changes, but it also sometimes changes pyproject.toml - even when it didn't have to. Perhaps it would be a good idea to ask dependabot to behave as I had believed it behaved all along, ie have it not change pyproject.toml when the update that it is making is compatible with the existing requirements

[radoering]

I wonder if re-trying with an automatic re-lock makes sense in that case?

I wouldn't want to have an automatic re-lock. IMO, install should not change the lock file. At least not per default.

[neersighted]

In general it is not possible safely to merge two independent changes to a lockfile without re-solving.

This has always been my general objection. However, I wonder if users might find it more desirable to use a "best-guess" (Git merge-based) solution when possible, and to bail out when an incompatible merge happened (since we in theory have the metadata to detect that/already run the solver).

However, I just noticed a seeming non-sequitur from @Mogost:

So far, I'm getting frustrated with the amount of merger conflicts that are draining CI resources.

I don't understand what you mean -- a conflicted poetry.lock is actually going to use less resources to detect than a "merged" lock file that is not a usable solution. So in that sense, I don't think you'll get what you want if we "solve" this (which appears to be possible by choosing to compromise/break at a different point, at least to me).

[dimbleby]

I've been looking more carefully at my dependabot MRs to understand why sometimes dependabot updates pyproject.toml and sometimes it is content simply to update poetry.lock

I think the difference is that >= constraints don't cause it to update pyproject.toml, but ^ constraints do.

So perhaps there's some code already in dependabot that is trying to be smart about this, but which could be taught about ^. That would likely significantly reduce the amount of noise in the hash value.

Maybe someone on this thread who finds this bothersome would like to raise an issue (or merge request) at dependabot to investigate (or fix) this.

[djmattyg007]

Even if you don’t want renovate to group dependency updates, you can still do things like ask it to only create one PR at a time, for example. This can reduce the burden of wasted CI time. No matter which way you look at it, it will still make your life easier in dealing with this poetry lock file problem.

[zyv]

However, I just noticed a seeming non-sequitur from @Mogost:

So far, I'm getting frustrated with the amount of merger conflicts that are draining CI resources.

I don't understand what you mean -- a conflicted poetry.lock is actually going to use less resources to detect than a "merged" lock file that is not a usable solution.

Well, it's a non-non-sequitur.

Right now, if you want a completely automated solution to resolve merge conflicts with Poetry lockfiles, you have to tell dependency bots to "rebase" their PRs every time a PR touching the lockfile is merged. I'm putting "rebase" in quotes, because in the context of dependency bots like dependabot or Renovate it's not about git rebase, but rather about re-locking the lockfile.

So, imagine, you have 4 dependabot PRs. You merge 1 PR, and trigger 3 PRs to re-lock and re-run the CI (not only Poetry, which might take an insignificant amount of time as compared to tests, but everything). Then you merge the 2nd one and the dance repeats.

This is what drains CI-time, not a possibly failing Poetry call, which (if it indeed fails) will actually conserve CI time, because downstream jobs just won't be able to proceed.

So I think your thinking goes into the right direction in terms of saving CI time.

[neersighted]

Maybe this is a matter of differing workflows, but as when I use automated tools there is one PR or they are batched, the prospect of "random" failures requiring human intervention during the install stage seems much more likely to waste CI time.

[bmitc]

Have Poetry developers looked at other lock file implementations to help with creating a more merge friendly format? What about taking a look at Elixir's mix.lock format? I never had any merge issues with it, but I am constantly having issues with even small Poetry projects.

[mrcljx]

The content-hash remains to be a big point of frustration. When three engineers add dependencies it's a constant race who gets their merge in first. The other two will have to rebase either once or twice to fix the content-hash.

[simonpercivall]

The issue I'm most afraid over right now over how the poetry.lock file works is updating sub-dependencies that don't touch pyproject.toml. Now, in a pretty big team of mixed-seniority devs, they don't always know how to properly merge and inspect the result, and it's happened more than once that an upgraded sub-dependency gets invisibly downgraded in a merge conflict resolution (even worse, git makes it really hard to see things that change in a merge commit); that is, the dev blindly checks out --theirs or --ours, and then runs poetry lock --no-update.

If there was even an imperfect way poetry could decrease the surface of a conflict (and just having a built-in way to resolve content-hash conflicts would solve like 95% of conflicts), it would go a long way to making it easier to use poetry in a team. (Sure, we could go back to pinning every sub-dependency in pyproject.toml, but then what's the point.)

In a tool that I otherwise really enjoy using, this is a painful pain point.

[dimbleby]

The issue I'm most afraid over right now over how the poetry.lock file works is updating sub-dependencies that don't touch pyproject.toml

since there is no content-hash change when pyproject.toml is unchanged, a "built-in way to resolve content-hash conflicts" would make exactly no difference in these cases

[bmitc]

At the moment, I am the only developer managing dependencies with Poetry, and I actually don't know how to manage poetry.lock at all. On any merge conflict, I just blast away (i.e., delete) the poetry.lock file and regenerate it with poetry install after fixing any merge conflict in pyproject.toml.

I feel Poetry is absolutely essential to use, but I don't know how to manage the poetry.lock file even in very simple cases. When a hash changes, I don't know what to do with that without a version change as well, so I am not personally aware of what decision criteria to use when looking at a merge conflict in poetry.lock. The above method of deleting and regenerating it actually seems to work okay, but it seems like it defeats one of the purposes of the lock file.

[simonpercivall]

The issue I'm most afraid over right now over how the poetry.lock file works is updating sub-dependencies that don't touch pyproject.toml

since there is no content-hash change when pyproject.toml is unchanged, a "built-in way to resolve content-hash conflicts" would make exactly no difference in these cases

Ah, I explained badly:

1. In Branch A Update a sub dependency
2. In A Update the pyproject.toml
3. In branch B, update pyproject.toml
4. In B, merge A, resolve the conflict badly, lock --no-update, step (1) is now gone

But as been said earlier in the discussion: Of course, to go as far as possible, any solution would need to look at both sides of the merge, and try to solve them together (like the NPM way someone described, or like the now-outdated "poetry-merge-file" package). And this is of course much harder to maintain.

[simonpercivall]

This has always been my general objection. However, I wonder if users might find it more desirable to use a "best-guess" (Git merge-based) solution when possible, and to bail out when an incompatible merge happened (since we in theory have the metadata to detect that/already run the solver).

@neersighted Have you thought any more about this idea? To me, it sounds ideal if possible.

(Sorry if this became spammy, but there's been so much back and forth in this issue that I felt a moments suggestion/thought may have been lost.)

[king-of-poppk]

This is not just a dependabot issue: if you work on a long-lived branch where you add/remove/upgrade dependencies good luck with rebasing against external dependency upgrades (for instance, merged dependabot PRs)!

[Andrew-Chen-Wang]

This is more for myself, but having to rebase all the time, just accept the incoming rebase:

git rebase -s recursive -X origin/main

then re-lock all the lock files. This primarily assumes the development branch's PR is JUST for adding a new package.