Carry all incompatibilities during backtracking #60
Conversation
uranusjr
force-pushed
the
backtrack–carry-all-incompatibilities
branch
from
54464a4 to
a5ad9c1
Compare
uranusjr
force-pushed
the
backtrack–carry-all-incompatibilities
branch
from
a5ad9c1 to
44dd815
Compare
Consider this situation: * Candidate Ax depends on B * Candidate B1 depends on C==1, D==1, E==1 * Candidate B2 depends on C==2, D==1, E==1 * Candidate C1 depends on D==1 * Candidate C2 depends on D==1 * Candidate D1 depends on E!=1 In the previous implementation, the conflict on E is discovered after we resolved Ax-B1-C1-D1. D1 is marked as an incompatibility, we backtrack the C1 pin. But now we don't have an available C, and need to also backtrack the B1 pin. At this point, however, the previous implemen- tation would fail to "remember" that D1 is also marked as incompatible, and proceed to try B2. That would eventually fail, we backtrack to the same point, are stuck trying B1 and B2 repeatedly. This fix uses a list to remember all the candidates marked as incompatible along the whole backtrack process, and "re-mark" them in parent states. This makes the resolver aware, when it backtracks B1, that B2 is also not viable, and avoid hitting it.
uranusjr
force-pushed
the
backtrack–carry-all-incompatibilities
branch
from
44dd815 to
1521ccb
Compare
|
The solution sounds legit. Do you have a regression test case for this? |
|
I’m still trying to come up with a good, real world, test case. Both cases available in pypa/pip#9011 are quite big and I want to trim them down to a managable input that contains only PyPI wheels. |
|
TBH, a synthetic test case (as described in OP) should also be sufficient -- we'd want to check calls to make sure we're not repeating the same things multiple times + add a timeout. |
|
I tried to create a test case from the abstract example I described above, and… it passed on master. I think my diagnosis is on the right trck, and the fix could actually be correct (it does fix the real-world case?), but it’s very difficult to find the right input that hits the exact code path to trigger the issue. Maybe we need more candidates on a package? More packages hitting the conflict? A conflict deeper in the graph? All or some combination of the above? I don’t know. 😟 |
|
Per a meeting today, as developers created a substantive and working test case, they found significant problems with their original approach, and so this issue needs more investigation and discussion to figure this out. This is a blocker (as is pypa/pip#9011) to the pip 20.3 release. I believe @uranusjr is poking at this a bit in his volunteer time -- is that correct? Tzu-ping? @pradyunsg plans to finish the other remaining things on his 20.3 pypa/pip#8936 checklist first, so he can make use of TP's investigations, finish and merge a fix, then immediately cut the 20.3 release. |
|
Yes, I’ve successfully reproduced the issue in #61 (which this PR would not fix) and plan to continue investigation and work on a fix there. |
|
Closing in favour of #61. |
Consider this situation:
Axdepends onBB1depends onC==1D==1E==1B2depends onC==2D==1E==1C1depends onD==1C2depends onD==1D1depends onE!=1In the previous implementation, the conflict on
Eis discovered after we resolvedAx-B1-C1-D1.D1is marked as an incompatibility, we backtrack theC1pin. But now we don’t have an availableC, and need to also backtrack theB1pin. At this point, however, the previous implementation would fail to “remember” thatD1is also marked as incompatible, and proceed to tryB2. That would eventually fail, we backtrack to the same point, and end up tryingB1again. Now we are stuck.This fix uses a list to remember all the candidates marked as incompatible along the whole backtrack process, and “re-mark” them in parent states. This makes the resolver aware, when it backtracks
B1, thatB2is also not viable, and avoid hitting it.