cmd/go: duplicate lines for module in go.sum #28456

myitcv · 2018-10-29T11:29:51Z

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (`go version`)?

go version go1.11.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/tmp.UI0ocKgDAb"
GOPROXY=""
GORACE=""
GOROOT="/home/myitcv/gos"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/tmp.cftvdIn4ct/hello/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build473745155=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I don't yet have a full repro, but the following PR shows the effect:

myitcv/gobin#19

Reporting here in case others see the issue and can add any context/detail.

What did you expect to see?

No duplicate lines in go.sum

What did you see instead?

$ diff -wu <(cat go.sum | sort) <(cat go.sum | sort | uniq)
--- /dev/fd/63  2018-10-29 11:28:53.817890106 +0000
+++ /dev/fd/62  2018-10-29 11:28:53.813890308 +0000
@@ -18,8 +18,6 @@
 github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/gomega v1.4.1/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
 github.com/rogpeppe/go-internal v1.0.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
-github.com/rogpeppe/go-internal v1.0.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
-github.com/rogpeppe/go-internal v1.0.0 h1:o4VLZ5jqHE+HahLT6drNtSGTrrUA3wPBmtpgqtdbClo=
 github.com/rogpeppe/go-internal v1.0.0 h1:o4VLZ5jqHE+HahLT6drNtSGTrrUA3wPBmtpgqtdbClo=
 github.com/russross/blackfriday v1.5.1/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
 github.com/sclevine/agouti v3.0.0+incompatible/go.mod h1:b4WX9W9L1sfQKXeJf1mUTLZKJ48R1S7H23Ji7oFO5Bw=

The text was updated successfully, but these errors were encountered:

bcmills · 2018-11-16T15:23:46Z

This may go away on its own if we add go.sum merging logic for #28802.

tv42 · 2019-02-12T01:00:29Z

For what it's worth, I ended up with duplicate go.sum lines via lazy conflict resolution, and was surprised that go mod tidy did not clean it up.

myitcv · 2019-07-12T04:39:00Z

Just to confirm this remains an issue in 1.13beta1:

env GOPROXY=proxy.golang.org

go mod tidy
cmp go.sum go.sum.golden
cp go.sum.dup go.sum
go mod tidy
cmp go.sum go.sum.golden

-- go.mod --
module mod
-- go.sum.dup --
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/rogpeppe/go-internal v1.3.0 h1:RR9dF3JtopPvtkroDZuVD7qquD0bnHlKSqaQhgwt8yk=
github.com/rogpeppe/go-internal v1.3.0 h1:RR9dF3JtopPvtkroDZuVD7qquD0bnHlKSqaQhgwt8yk=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0 h1:0vLT13EuvQ0hNvakwLuFZ/jYrLp5F3kcWHXdRggjCE8=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
-- go.sum.golden --
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/rogpeppe/go-internal v1.3.0 h1:RR9dF3JtopPvtkroDZuVD7qquD0bnHlKSqaQhgwt8yk=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0 h1:0vLT13EuvQ0hNvakwLuFZ/jYrLp5F3kcWHXdRggjCE8=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
-- main.go --
package main

import _ "github.com/rogpeppe/go-internal/testscript"

func main() {}

Gives:

> env GOPROXY=proxy.golang.org
> go mod tidy
[stderr]
go: finding github.com/rogpeppe/go-internal v1.3.0
go: downloading github.com/rogpeppe/go-internal v1.3.0
go: extracting github.com/rogpeppe/go-internal v1.3.0
go: downloading gopkg.in/errgo.v2 v2.1.0
go: extracting gopkg.in/errgo.v2 v2.1.0
go: downloading gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
go: extracting gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
go: downloading github.com/kr/pretty v0.1.0
go: extracting github.com/kr/pretty v0.1.0
go: downloading github.com/kr/text v0.1.0
go: extracting github.com/kr/text v0.1.0

> cmp go.sum go.sum.golden
> cp go.sum.dup go.sum
> go mod tidy
> cmp go.sum go.sum.golden
[diff -go.sum +go.sum.golden]
 github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/rogpeppe/go-internal v1.3.0 h1:RR9dF3JtopPvtkroDZuVD7qquD0bnHlKSqaQhgwt8yk=
-github.com/rogpeppe/go-internal v1.3.0 h1:RR9dF3JtopPvtkroDZuVD7qquD0bnHlKSqaQhgwt8yk=
 github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
-github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/errgo.v2 v2.1.0 h1:0vLT13EuvQ0hNvakwLuFZ/jYrLp5F3kcWHXdRggjCE8=
 gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=

FAIL: /tmp/testscript286011127/0/script.txt:7: go.sum and go.sum.golden differ
error running go1.13_mod_tidy_dup_bug.txt in /tmp/testscript286011127/0

bcmills · 2019-08-12T18:34:58Z

In #33603, @kevinburkemeter mentions that he got into this state via a bad rebase.

honnix · 2020-01-03T11:39:15Z

The problem I experienced is a bit different.

+github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=
 github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=

~~The file tree checksum is not populated in go.sum on my computer (macOS) but added by TravisCI (Linux). I use the same version of Go for both.~~

Further more, if I manually add the file tree checksum locally to go.sum and run go mod tidy afterwards, it is not removed. In other words, go mod tidy is happy with either having it or not.

Updated:

I managed to find a difference between my local build and CI. There is an additional go get ... executed in CI and that results change of go.sum. This could boil down to #34506, although it is still weird go mod tidy does not clean up accordingly.

jayconrod · 2020-01-03T14:36:07Z

@honnix That sounds different than this issue. Would you please open a new one? go mod tidy should end up with a canonical set of hashes.

honnix · 2020-01-03T17:07:16Z

@jayconrod ~~Sure, that makes sense.~~ This is covered by #34506.

Hipska · 2022-05-17T15:51:43Z

Is there any update on this? https://github.com/influxdata/telegraf/blob/master/go.sum has 100+ duplicate lines..

bcmills · 2022-05-19T12:30:29Z

@Hipska, this issue is still on the backlog. If you'd like to contribute a fix I'd be happy to review it!

gopherbot · 2022-06-08T16:56:58Z

Change https://go.dev/cl/411154 mentions this issue: cmd/go/internal/modfetch: prevent duplicate hashes in go.sum

myitcv added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. GoCommand cmd/go modules labels Oct 29, 2018

myitcv added this to the Go1.12 milestone Oct 29, 2018

bcmills modified the milestones: Go1.12, Go1.13 Nov 16, 2018

neuromage mentioned this issue Jan 8, 2019

Add IS_SUBSTRING operator for use in API resource filtering. kubeflow/pipelines#645

Merged

andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019

bcmills mentioned this issue Aug 12, 2019

cmd/go: "go mod tidy" does not remove duplicate lines #33603

Closed

aarondl mentioned this issue Sep 2, 2019

cmd/go: The modules feature acts in surprising ways #34025

Closed

rsc modified the milestones: Go1.14, Backlog Oct 9, 2019

xueweiz mentioned this issue Nov 26, 2019

Cleanup go.sum kubernetes/node-problem-detector#364

Closed

kushthedude mentioned this issue Sep 21, 2020

chore: remove vendor modules from benchmark kubernetes/perf-tests#1463

Merged

Hipska mentioned this issue May 6, 2022

Output cloudrun addition influxdata/telegraf#10369

Closed

3 tasks

powersj mentioned this issue Jun 8, 2022

cmd/go/internal/modfetch: prevent duplicate hashes in go.sum #53291

Closed

gopherbot closed this as completed in 103cc66 Jun 17, 2022

golang locked and limited conversation to collaborators Jun 17, 2023

gopherbot added the FrozenDueToAge label Jun 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: duplicate lines for module in go.sum #28456

cmd/go: duplicate lines for module in go.sum #28456

myitcv commented Oct 29, 2018

bcmills commented Nov 16, 2018

tv42 commented Feb 12, 2019

myitcv commented Jul 12, 2019

bcmills commented Aug 12, 2019

honnix commented Jan 3, 2020 •

edited

Loading

jayconrod commented Jan 3, 2020

honnix commented Jan 3, 2020 •

edited

Loading

Hipska commented May 17, 2022

bcmills commented May 19, 2022

gopherbot commented Jun 8, 2022

cmd/go: duplicate lines for module in go.sum #28456

cmd/go: duplicate lines for module in go.sum #28456

Comments

myitcv commented Oct 29, 2018

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

bcmills commented Nov 16, 2018

tv42 commented Feb 12, 2019

myitcv commented Jul 12, 2019

bcmills commented Aug 12, 2019

honnix commented Jan 3, 2020 • edited Loading

jayconrod commented Jan 3, 2020

honnix commented Jan 3, 2020 • edited Loading

Hipska commented May 17, 2022

bcmills commented May 19, 2022

gopherbot commented Jun 8, 2022

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

honnix commented Jan 3, 2020 •

edited

Loading

honnix commented Jan 3, 2020 •

edited

Loading