cmd/go: go mod assumes a lexicographically linear ordering of versions

[rittneje]

This is basically an experience report, based on some issues we have run into with go mod.

Go mod fundamentally assumes that there is a simple ordering between all versions of a module. In essence, if version X is lexicographically less than version Y, then Y is a superset of X. However, this is not always true, and can lead to some unexpected implications when dealing with transitive dependencies.

For example, suppose my main module has a direct dependency on version 1.0.0 of A, and version 2.0.0 of B. Also suppose that A has a dependency on version 3.0.0 of C, and B has a dependency on version 3.1.0 of C. As per minimum version selection, my main module will select version 3.1.0 of C to satisfy the indirect dependency.

Now suppose some bug is discovered in module C. They release a new version 3.1.1, and also backport the fix to version 3.0.1. Then module A releases a new version 1.0.1 that depends on version 3.0.1 of C. But module B has not released a new version yet.

In my main module, I update my dependency on A to the new version (1.0.1), under the pretense of pulling in the bugfix. However, due to minimal version selection, I will still be using version 3.1.0 of C. The root cause of the issue is that it incorrectly assumes that just because 3.1.0 > 3.0.1, that means that 3.1.0 contains all of 3.0.1. The only way for this to work correctly is for me to manually specify version 3.1.1 of C in my go.mod file, which is especially weird because it is not a direct dependency.

A similar issue unfolds with unversioned libraries. There go mod just orders by commit date. But the commits in question need not be from the same branch, so again there may not be a convenient linear history between them. This is especially problematic with cherry picks, where if one module is tracking a release branch, then a cherry pick would cause its HEAD to be considered newer than an older commit from the main branch.

[ALTree]

cc @bcmills @jayconrod

[mvdan]

In your fifth paragraph, are you mixing up versions 3.x.y with 1.x.y? For example, it says version 1.1.1 of C, where I think it means version 3.1.1 of C.

I appreciate the experience report, by the way - I've run into similar versions of this subtle problem.

I think my only bit of input is that, if C's v3.1.0 is so terribly broken that v3.1.1 should always be preferred, v3.1.1 could retract v3.1.0, signaling to MVS that it should prefer v3.1.1. But I get that retraction should only be used for terribly broken versions, such as those containing security bugs, and often the bugs are not that severe - though still important.

[bcmills]

The Go module system uses the precedence rules defined by semantic versioning.

At any rate, I think the heart of the issue is this:

The only way for this to work correctly is for me to manually specify version [3].1.1 of C in my go.mod file, which is especially weird because it is not a direct dependency.

I don't think it actually is all that weird to need to upgrade an indirect dependency to pull in a bug-fix — and note that at go 1.17 and above your go.mod file already explicitly includes all “relevant” indirect dependencies.

[bcmills]

A similar issue unfolds with unversioned libraries. There go mod just orders by commit date. But the commits in question need not be from the same branch, so again there may not be a convenient linear history between them.

It does not merely order by commit date. The pseudo-versions that the go command generates for commits encode not only the commit timestamp, but also a version prefix derived from the highest semantic version tagged on any ancestor of the commit.

The pseudo-version derivation does require a bit of careful tagging when using release branches (to ensure that commits on the mainline are ordered correctly relative to commits on release branches), but I don't see how we can substantially improve that logic given how little version information is encoded in the git branch structure.

[rittneje]

In your fifth paragraph, are you mixing up versions 3.x.y with 1.x.y? For example, it says version 1.1.1 of C, where I think it means version 3.1.1 of C.

Yep, that was a typo, I have updated my original post.

It does not merely order by commit date. The pseudo-versions that the go command generates for commits encode not only the commit timestamp, but also a version prefix derived from the highest semantic version tagged on any ancestor of the commit.

That assumes that there are tags, which is not universally true. For example, the x repos are not tagged (except for x/text).

I don't think it actually is all that weird to need to upgrade an indirect dependency to pull in a bug-fix — and note that at go 1.17 and above your go.mod file already explicitly includes all “relevant” indirect dependencies.

That isn't really feasible when there are a lot of indirect dependencies to account for. It would be preferable for go mod to fail (or at least warn) and force me to resolve the issue explicitly (by asking for 3.1.1 in this case) in my go.mod file.

[mvdan]

In terms of warnings to watch out for, how about:

warning: you're upgrading module C from v3.0.1 (2021-09-01), to the older v3.1.0 (2021-06-01), when v3.1.1 exists (2021-10-01)

This does not imply that v3.1.1 should always be preferred over v3.1.0, but it's certainly a strong signal in many practical scenarios.

[bcmills]

I don't think a warning is appropriate. We have no particular reason to believe that v3.0.1 fixes a bug that was present in v3.1.1, rather than (say) merely providing a backport of a fix that landed in v3.1.0,

We already provide the -u=patch flag to apply patch releases to existing dependencies. If you want to ensure that dependencies are patched (independent of the precedence defined by Semantic Versioning), I suggest running go get -u=patch ./... on a regular basis, including after upgrading other dependencies.

[rittneje]

I don't think a warning is appropriate. We have no particular reason to believe that v3.0.1 fixes a bug that was present in v3.1.1, rather than (say) merely providing a backport of a fix that landed in v3.1.0.

I agree that as the module ecosystem currently stands, making this determination is not generally possible (or at the very least, fairly non-trivial). But, for example, if the go.mod file had a way to indicate that 3.0.1 actually came after 3.1.0, then it would be trivial.

We already provide the -u=patch flag to apply patch releases to existing dependencies.

This should work well enough for versioned libraries, although I still think it would be preferable for go mod to complain about incompatible versions rather than assuming you are actively running such a command.

In any case, my understanding is that for unversioned modules, the rough equivalent would be to update to "latest"? To detect the mismatch there would require the use of git commands (to ask if the chosen commit is a descendant of all desired commits), which probably wouldn't work as it requires a full (i.e., not shallow) clone.

[bcmills]

I still think it would be preferable for go mod to complain about incompatible versions rather than assuming you are actively running such a command.

According to the semver spec, v3.0.1 is not “incompatible” with v3.1.0. It is certainly possible that v3.1.0 contains a bug that is not present in v3.0.1, but that isn't fundamentally different from the situation in which v3.1.0 contains a bug and only v3.1.1 includes the fix.

The go command aims to make builds reproducible, and to make it straightforward for users to pull in fixes to dependencies as needed. However, it is never the case in general that updating a single dependency will pull in bug-fixes for all of the packages imported by that dependency — the special case of backported patch-releases is at least no worse than the general case. And the tool for the general case is go get -u ./....

In any case, my understanding is that for unversioned modules, the rough equivalent would be to update to "latest"?

Yes. In some sense, if a module is unversioned, then the notions of “major release”, ”minor release”, and “patch release” all collapse to “later commit”. We intentionally do not attempt to track version control branches beyond identifying the appropriate base for a pseudo-version, but compare #26964 (which is still open).

[bcmills]

I don't think there is anything we can feasibly do here, but if you have a specific proposal please do feel free to file it.
(See https://go.dev/s/proposal.)