Checksum mismatch: go get github.com/google/flatbuffers/go

[regeda]

Could not get the library due to checksum mismatch:

$ go get -u github.com/google/flatbuffers/go
go: downloading github.com/google/flatbuffers v1.12.0
go get github.com/google/flatbuffers/go: github.com/google/[email protected]: verifying module: checksum mismatch
        downloaded: h1:N8EguYFm2wwdpoNcpchQY0tPs85vOJkboFb2dPxmixo=
        sum.golang.org: h1:/PtAHvnBY4Kqnx/xCQ3OIV9uYcSFGScBsWI3Oogeh6w=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.

$ go version
go version go1.15.7 darwin/amd64

[aardappel]

Not familiar with Go.. so no idea what is causing this. Anyone else?

[ceejatec]

My theory: Google did something naughty with the git history, perhaps moved the v1.12.0 tag to a new SHA. This lead to the goproxy module cache and the checksum servers being out of sync, producing the above error. It seems to have been "fixed" now - if you run the above command today (and you don't have an earlier version of the module cached on your system) you don't get the error. But we run our own internal module proxy at our work, and it's still stuck in a mode where the checksums don't match, so we still get the above error. Now I need to figure out how to manually flush that part of the cache so our builds aren't broken. Very annoying that someone being sloppy out on the internet can break our builds like that. I mean, for Go in particular, Google of all people should know better...

[aardappel]

@ceejatec "Google" is not a single entity, I don't think there's ever been any communication between those involved in Go and FlatBuffers. The Go implementation has been made entirely by external contributors.

And no, the v1.12.0 tag has not been touched.

[ceejatec]

That was my theory based on a basically identical issue with the azure-storage-blob-go package less than a month ago : Azure/azure-storage-blob-go#236 (comment) where someone changed the v0.12.0 tag because it was mistakenly pointing at the wrong commit. Changing the tag broke the Go module caching proxies just as we see here.

In that case, we worked around it by telling Go to use a pseudo-tag based on v0.11.0+some number of commits, which the Go proxies see as a different version and hence re-download the code and checksum. We've actually done a similar thing for now for flatbuffers, using the following in our go.mod:

github.com/google/flatbuffers v0.0.0-20200312223339-6df40a247173

If the tag wasn't changed in this case, then "something else" caused Google's go module proxies to get out of sync with each other. As I said, that particular problem seems to be fixed "somewhere" now, but if you've got your own local caches you still might be messed up.

[regeda]

I've resolved the issue. I set the GOPROXY variable explicitly:

go mod tidy
GOPROXY=proxy.golang.org go get -u github.com/google/flatbuffers/go

Somehow, my default GOPROXY was pointed wrongly at proxy.golang.com.cn.

[amnonbc]

~Can someone on the flatbuffers project please create a new v1.12.1 tag so that users of the latest tag do not have to resort to these workarounds?

This will take 2 minutes, and save your users a lot of aggravation.

And ideally you should also redact v1.12.0~

[aardappel]

@amnonbc pointing to what commit?

So far, I am not even aware of an issue with our repo. We haven't changed these tags afaik. I'd like to understand what the problem is first.

[amnonbc]

I have dug a bit further and found that the culprit was a stray GOPROXY=https://goproxy.cn entry.
No idea why https://goproxy.cn contained garbage.

Removing that has solved the problem for us.
Thanks for responding. And apologies for the noise.

[ceejatec]

It's not just goproxy.cn. I can reproduce this locally and reliably by using the Athens go proxy, and it (hopefully?) isn't pulling information from goproxy.cn. I filed an issue against Athens with the details, but I'm not at all convinced it's actually an Athens bug - I'm hoping they will have more info to help diagnose what's really doing on.

[XC-]

For us this occurs:
github.com/google/flatbuffers/go: github.com/google/[email protected]: verifying module: checksum mismatch downloaded: h1:N8EguYFm2wwdpoNcpchQY0tPs85vOJkboFb2dPxmixo= sum.golang.org: h1:/PtAHvnBY4Kqnx/xCQ3OIV9uYcSFGScBsWI3Oogeh6w=

with the go.sum in place, the error tends to be different on developer machine and in CI. The downloaded checksum and the one received from sum.golang.org switch places in CI. On developer machine, the GOPROXY env variable is empty.

[Avokadoen]

Anyone who has a concrete workaround? I have the same issue as @XC-, it works fine locally but in the CI it fails with the identical error. This is a indirect dependency from badger, so I can't trivially change the flatbuffer version either ...

Doing RUN GOPROXY=proxy.golang.org go mod download in the CI Dockerfile did not resolve the issue

Any help is much appreciated :- )

[ceejatec]

@Avokadoen My workaround (posted here on February 18) does the trick for us.

[Avokadoen]

@Avokadoen My workaround (posted here on February 18) does the trick for us.

We have flatbuffers as a indirect dependency, I might be misinterpreting but the workaround you posted is to modify go.mod? In that case it would not work for us. But thanks for the tip anyways! 👍

[burningalchemist]

I'm having the same issue, when using Github-native Dependabot for security updates:

Dependabot failed to update your dependencies because there was an error resolving your Go dependency files.

Dependabot encountered the following error:
verifying github.com/google/[email protected]: checksum mismatch

Given that this dependency is indirect in my case as well, and there is no way to control the way Dependabot verifies checksums, it seems a blocking issue.

I'd appreciate any advice here, too.

[ceejatec]

Despite @aardappel 's earlier assertion, it seems clear that the v1.12.0 tag was changed at some point, and that it happened after Google had consumed the earlier commit.

Evidence:

cd /tmp && git clone git://github.com/google/flatbuffers -b v1.12.0
go mod download github.com/google/[email protected]
cd ~/co/pkg/mod/github.com/google/[email protected]
diff -r . /tmp/flatbuffers

This shows numerous differences. It is true that none of those differences are in the Go code, but the Go proxy will calculate its checksum based on the full contents of the repository, not just the Go code.

Additional evidence: if you rm -rf ~/go, set GOPROXY=direct and GOSUMDB=off, and repeat the above steps, you'll find that there are no differences. This confirms that the set of code stored by Google in its proxy is different than the tagged content here on github.

I can't find any commit in the flatbuffers repository which corresponds to the code in the module, so I don't know how this happened. But it definitely did happen, and it definitely is breaking a lot of use cases.

I also don't know if there's any way to request Google to reset its cached modules based on the actual v1.12.0 tag, but even if you could, that would quite possibly break everybody who isn't currently broken.

I think the only way forward, as was suggested back in February, is to create a "v1.12.1" tag and make sure that it doesn't get messed with. At least that way, people who are hitting this issue can add v1.12.1 to their go.mod files, which will override any dependencies they have which are pointing to the broken v1.12.0.

[aardappel]

the v1.12.0 points to 6df40a2 (pre-tag version bump for 1.12) like it always has. For the 3rd time, it hasn't moved. Whatever problem there is, it is entirely inside Go's handling of things.

This is how we release things, and it doesn't include Go: https://github.com/google/flatbuffers/wiki/FlatBuffers-release-process which means whatever is on the Go package manager (is there one?) was uploaded by someone else, not me. Would be good first to track that person down?

I have no idea how Go works, but I'd be happy to add a v1.12.1 tag pointing to whatever commit you desire to the repo if that helps.

[aardappel]

There is also a v2.0.0 release coming up, if Go is helped by any particular incantations when that tag gets added, now would be a good time to figure that out.

[ceejatec]

Go modules uses tags on GitHub which (fortunately?) conform to the standard you're using, vX.Y.Z. Details: the first time someone tries to build a Go project that declares a dependency on github.com/google/flatbuffers v1.12.0, the Go compiler will hit the Go module proxy - by default proxy.golang.org - and ask for the module. The proxy will not yet have the code for the module (FYI go modules are purely source code, not any kind of compiled artifact), so the proxy will internally fall back to "direct" mode which means it will look up that tag on GitHub and pull the source at that moment in time. It will store that source code forever as the source code for github.com/google/flatbuffers v1.12.0. It will also form a checksum of the source code, and there is a separate checksum database which is populated accordingly (by default this is sum.golang.org).

For any later builds, the Go compiler will ask the proxy for the source code of the module, and it will get it. The compiler will compute the checksum, and then request the checksum for that module from the checksum database and confirm that it matches the checksum it computed. And that is the step that is failing for a number of people today.

The fundamental cause of this problem is that the source code cached in proxy.golang.org does not match the code that is available on GitHub at that tag. This triggers problems whenever something uses a different Go proxy (or no proxy at all, ie. GOPROXY=direct) but is still referencing the default checksum database. This is, for better or worse, not at all an uncommon scenario, and normally shouldn't cause any problems because normally everybody on the planet (all Go module proxies, plus GitHub) would agree about exactly what source code constitutes a given version of a module.

I cannot explain how the world got into this state. By far the easiest explanation is that the tag on GitHub got moved sometime after the first time proxy.golang.org consumed the source code. If that didn't happen, then I'm stumped. Perhaps Google itself did something wonky because google/flatbuffers is its own code, although that seems like a really bad idea if true.

[ceejatec]

FYI, I did my same experiment above with flatbuffers v1.11.0, and there are no differences in the source code. So whatever process was done for v1.11.0 worked as expected. That makes it all the more weird that v1.12.0 is messed up, since I have to assume you did exactly the same things.

[aardappel]

Well, I am the one applying tags and doing releases, no-one else at Google is involved in that so far. And yes, I've been doing these releases exactly the same way, all 13 of them over the past 7 years.

Anyway, if I'd thus tag v1.12.1 to whatever the tag is right after that commit, it should cause something new to be cached, and everyone would be happy, right? That would be this one, which doesn't change any Go files: 697147a

[Avokadoen]

I do believe that creating a v1.12.1 release would at least help those of us that has flatbuffers as a indirect dependency to resolve our CI issues using the replace directive. Thanks you @aardappel for all the hard work you do on this project, and thank you @ceejatec for investigating! ❤️

[ceejatec]

Agreed, creating that tag would be very helpful for those with indirect (or direct!) dependencies.

@Avokadoen Using a replace directive will work, but it also introduces a small risk - later if some code you depend on uses the upcoming v1.20.0 version, your builds won't pick that up automatically. Just something to watch out for.

[aardappel]

We now have https://github.com/google/flatbuffers/tree/v1.12.1

Note I will be releasing v2.0.0 likely today, so some of you may want to update to that instead.

[burningalchemist]

@aardappel @ceejatec Updating go.mod to v.1.12.1 fixed my problem with Github/Dependabot. Thanks! 😃

[github-actions]

This issue is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days.