cmd/go: rethink "missing go.mod" mode

[rsc]

A few weeks ago, shortly after GO111MODULE=on became the default, I got to watch Rob using it. One problem he had was that when he worked in preexisting GOPATH directories, or just directories newly created for the purpose of testing one bug or another, there was no go.mod, so every go command reresolved "latest" for any imported dependencies. This made things much slower than they are intended to be (that is, much slower than they are when there is a go.mod).

I think we probably need to rethink what we do without any go.mod. There needs to be some place for the go command to write down its decisions from one command to the next. When people say "modules are slow" I wonder if this case is one of the things they mean.

/cc @bcmills @jayconrod @ianthehat

[bcmills]

Some ideas (separated for emoji-response tracking):

• When we need to implicitly resolve a package as part of a go build or go test (rather than, say, a go get), we could pull in the latest version found in the module cache instead of trying to get the absolute latest upstream version.

[bcmills]

• We could revive the “default go.mod” idea (from cmd/go: allow "go get" when outside a module in module mode #24250 (comment) and others).

The downside of that approach is that it would be really easy for users to unintentionally get stuck on some really old version of a dependency and never realize that they need to upgrade it. (That's one of the more annoying failure modes of GOPATH today.)

[bcmills]

• We could be less willing to run commands outside of a module in general; for example, we could refuse to run go get unless the user passed an explicit “global install” flag (cmd/go: offer a consistent "global install" command #30515), and we could refuse to resolve any dependencies from outside the standard library during a go build.

That would at least prompt the user to run go mod init earlier in their workflow, which would mitigate some of the sources of slowness.

[beoran]

An important use case for no go.mod file are tools that do not use any imported dependencies. For certain software, such as, say, a Go module proxy, or a simple command line tool, the programmers might want to avoid a chicken-and-egg bootstrap problem and therefore depend only on the Go standard library. Whatever the solution, I'd like this use case to be taken into consideration.

[bcmills]

@beoran, maintaining a go.mod file for a package with few (or no) external dependencies should be a negligible amount of overhead (since it never needs to change), and the go.mod file has some other important effects (such as the go directive, which declares the expected version of the Go language specification).

I think the bug-testing and legacy-GOPATH use cases are more compelling.

[rsc]

Looking at cmd/go/testdata/script/mod_outside.txt, the only "slow success" cases (where there's no go.mod so every execution has to re-resolve unknown imports) are things like 'go run x.go' (or go build, go vet etc) where x.go contains an import of a package from outside the standard library. If we make that case fail with a message about not having a go.mod, then I think we can consider this issue fixed as far as blocking setting GO111MODULE=on by default.

[gopherbot]

Change https://golang.org/cl/198778 mentions this issue: cmd/go: forbid resolving import to modules when outside of a module

[eliasnaur]

I wrote this comment on the CL:

With this CL applied, I can no longer go run my tools outside a module:

$ go run gioui.org/cmd/gogio
package gioui.org/cmd/gogio: cannot look up module for package "gioui.org/cmd/gogio": working directory is not part of a module

I can't even run (demo) programs:

$ go run gioui.org/apps/hello
package gioui.org/apps/hello: cannot look up module for package "gioui.org/apps/hello": working directory is not part of a module

This behaviour change seems to be a regression, in particular in light of https://golang.org/issue/33518.

"go get" is exempt from the new behaviour, why not "go run"?

@bcmills replied

go get is exempt because people generally only run it once: you run go get, and that resolves dependencies and installs a binary that you then invoke directly.

go run is not exempt because it is more frequently run many times in succession. That imposes a recurring cost (and non-reproducible behavior) to re-resolve the dependencies, which would become a one-time cost (and reproducible behavior) if the resolution could be recorded in a go.mod file.

Let me defend go run outside a module:

First, this CL is taking away a useful, albeit slow, feature without a replacement. I don't consider the usual workaround "use go get and run the installed command" a good enough replacement for go run. See #33518 for my arguments.

Second, this issue seems to be about running local Go packages and standalone programs. Quoting Russ:

Looking at cmd/go/testdata/script/mod_outside.txt, the only "slow success" cases (where there's no go.mod so every execution has to re-resolve unknown imports) are things like 'go run x.go' (or go build, go vet etc) where x.go contains an import of a package from outside the standard library.

I can live with not allowing go run of local files and packages outside a module. What I'm interesting in is running already published modules, such as

$ go run gioui.org/apps/hello
$ go run gioui.org/cmd/gogio -target android gioui.org/apps/hello

Ironically, both gioui.org/cmd/gogio and gioui.org/apps/hello are themselves modules, even though the go run command is run outside one.

Note that even though I personally don't care much about go running local files and packages outside a module doesn't mean it's a good idea to disallow it. Writing a short Go program and then go running it is a very useful and lightweight way to work. Forcing me to do go mod init example.com just to please go run seems like unnecessary bureaucracy (writing this argument out, I'm curious what Rob would say about this extra requirement in his work). play.golang.org doesn't force me to deal with modules, why should go run?

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default.

In short, I think submitting CL 198778 as is would be a mistake and I would be quite disappointed to see a useful feature go away for efficiency reasons.

[eliasnaur]

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default.

I ran the following experiment:

$ go version
go version devel +f923c66893 Sat Sep 14 13:08:03 2019 +0200 linux/amd64
$ export GOPATH=$(mktemp -d)
$ time go run gioui.org/cmd/gogio
go: finding gioui.org latest
go: finding gioui.org/cmd latest
go: downloading gioui.org v0.0.0-20191003172637-31e12607d739
go: downloading gioui.org/cmd v0.0.0-20191003172637-31e12607d739
go: extracting gioui.org v0.0.0-20191003172637-31e12607d739
go: extracting gioui.org/cmd v0.0.0-20191003172637-31e12607d739
go: downloading gioui.org v0.0.0-20190930145547-3784ece6dd65
go: extracting gioui.org v0.0.0-20190930145547-3784ece6dd65
go: downloading golang.org/x/image v0.0.0-20190802002840-cff245a6509b
go: downloading golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: extracting golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: extracting golang.org/x/image v0.0.0-20190802002840-cff245a6509b
go: finding golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: finding golang.org/x/image v0.0.0-20190802002840-cff245a6509b
gogio: specify a package
exit status 1

real	0m3,664s
user	0m2,028s
sys	0m0,662s
$ time go run gioui.org/cmd/gogio
go: finding gioui.org latest
go: finding gioui.org/cmd latest
gogio: specify a package
exit status 1

real	0m0,511s
user	0m0,413s
sys	0m0,158s

The first run completed in just under 5 seconds, the second in less than a second.

For comparison, I ran gogio inside the gioui.org/cmd module:

$ time go run ./gogio/
gogio: specify a package
exit status 1

real	0m0,369s
user	0m0,334s
sys	0m0,122s

That is, a difference of ~200ms between inside and outside a module.

[bcmills]

Ironically, both gioui.org/cmd/gogio and gioui.org/apps/hello are themselves modules, even though the go run command is run outside one.

That's an interesting input for #30515 / #34506. In particular, it suggests that we may want to allow go run (and possibly go install) to accept the same “global install” flag as go get in order to explicitly opt in to the slower path.

Even then, I think that only get, if that, should resolve dependencies that are not already specified by the module(s) containing the package argument(s).

[jayconrod]

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default. ... I would be quite disappointed to see a useful feature go away for efficiency reasons.

To be clear, we're not planning to remove this for efficiency reasons: we're planning to remove this because we don't feel that it's a good user experience. When GOPATH is eventually deprecated and the default value of GO111MODULE is on, a lot of users, especially those new to Go, will be stuck in this mode without realizing it. In this mode, Go essentially acts as a scripting language that goes out to the network and downloads the latest version of each imported package (and if imported packages haven't migrated to modules, their transitive dependencies as well). While this produces a correct result, it's very slow, and it's not reproducible (a key goal of modules). We don't want people to come away from this thinking "Is Go always this slow? This is horrible."

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default.

GOPROXY certainly speeds things up, but asking for the latest version of a module can still be quite slow. If it's a module that is requested infrequently, the "latest" version may not be cached, and the proxy may need to fetch the repository again.

Additionally, if any of the imported packages don't have go.mod files, we have to resolve the "latest" versions of their dependencies (and so on).

Even in the fast case, you're going out to the network, which will be slower than local builds with a warm or mostly warm cache.

First, this CL is taking away a useful, albeit slow, feature without a replacement. I don't consider the usual workaround "use go get and run the installed command" a good enough replacement for go run. See #33518 for my arguments.

I read through #33518, but I'm not directly familiar with Gio, and I'm not really sure what the workflow is. Why is go get followed by a direct call to the installed binary not usable? In that issue, it seemed like the main concerns were that linked binaries aren't cached (#33468) and that go install might not write to a directory in PATH. Those seem like issues that can be addressed separately.

Also from reading the intro text at https://git.sr.ht/~eliasnaur/gio, it sounds like this tool builds and packages programs. Won't most users always be running it in their own modules? Also, shouldn't a fixed, predictable version of gio be used instead of the latest?

[eliasnaur]

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default. ... I would be quite disappointed to see a useful feature go away for efficiency reasons.

To be clear, we're not planning to remove this for efficiency reasons: we're planning to remove this because we don't feel that it's a good user experience. When GOPATH is eventually deprecated and the default value of GO111MODULE is on, a lot of users, especially those new to Go, will be stuck in this mode without realizing it. In this mode, Go essentially acts as a scripting language that goes out to the network and downloads the latest version of each imported package (and if imported packages haven't migrated to modules, their transitive dependencies as well). While this produces a correct result, it's very slow, and it's not reproducible (a key goal of modules). We don't want people to come away from this thinking "Is Go always this slow? This is horrible."

Let me point out that it's not always slow. With a warm cache, my experiment above measures the difference in the hundreds of milliseconds. Furthermore, the output "go: finding blah go: downloading blah go: extracting blah" is a clear hint to why a cold go run is slow.

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default.

GOPROXY certainly speeds things up, but asking for the latest version of a module can still be quite slow. If it's a module that is requested infrequently, the "latest" version may not be cached, and the proxy may need to fetch the repository again.

Ok, but for the second and subsequent runs, go run only has to do ~1 query to the proxy which is likely to have the result cached, right?

Additionally, if any of the imported packages don't have go.mod files, we have to resolve the "latest" versions of their dependencies (and so on).

As more and more people switch to modules, this problem diminishes. In particular when GO111MODULE=on becomes the default, which to many indicates that Go modules are "ready for general use".

Even in the fast case, you're going out to the network, which will be slower than local builds with a warm or mostly warm cache.

I agree.

First, this CL is taking away a useful, albeit slow, feature without a replacement. I don't consider the usual workaround "use go get and run the installed command" a good enough replacement for go run. See #33518 for my arguments.

I read through #33518, but I'm not directly familiar with Gio, and I'm not really sure what the workflow is. Why is go get followed by a direct call to the installed binary not usable? In that issue, it seemed like the main concerns were that linked binaries aren't cached (#33468) and that go install might not write to a directory in PATH. Those seem like issues that can be addressed separately.

Also from reading the intro text at https://git.sr.ht/~eliasnaur/gio, it sounds like this tool builds and packages programs. Won't most users always be running it in their own modules?

Most of the time, sure. But the critical time is when a newcomer evaluates Gio. I'd like for their experience to be as painfree as possible and being able to run a Gio program with a simple go run without the PATH hassle is invaluable.

When a Gio user finally decides to use Gio, publishes their Gio program to some Git repository, they in turn can tell their users to evaluate it with just

$ go run <github.com/user/shinyprogram>

I imagine most projects would like the simplest possible "get started" steps. go run is that.

Also, shouldn't a fixed, predictable version of gio be used instead of the latest?

Sure, and that's why I suggest go running gogio in my instructions. But that doesn't stop users go installing gogio which is not reproducible. In other words, gogio must support the "use latest version" use case (when Gio reaches 1.0), and if so, it will also support the very convenient

$ go run gioui.org/cmd/gogio -target android <github.com/yourapp/blah>

outside a module.

Responding to both @bcmills'

Ironically, both gioui.org/cmd/gogio and gioui.org/apps/hello are themselves modules, even though the go run command is run outside one.

That's an interesting input for #30515 / #34506. In particular, it suggests that we may want to allow go run (and possibly go install) to accept the same “global install” flag as go get in order to explicitly opt in to the slower path.

and @jayconrod's

I read through #33518, but I'm not directly familiar with Gio, and I'm not really sure what the workflow is. Why is go get followed by a direct call to the installed binary not usable? In that issue, it seemed like the main concerns were that linked binaries aren't cached (#33468) and that go install might not write to a directory in PATH. Those seem like issues that can be addressed separately.

Great, let's implement the replacement before killing off go run outside modules. I don't know if that means fixing #30515, fixing the go install PATH issue, or something else entirely.

P.S. In case you think I'm making too big a deal out of go run versus go install and PATH: Because of Windows' powerful syscall interface for using native DLLs without Cgo, go run builds and runs Gio programs with only Go installed! (There is a runtime dependency on the non-system ANGLE DLLs, but that can be avoided with a future Direct3D backend.)
Additionally, PATH is most alien and difficult to set for Windows users.
The point is, even setting up PATH is a non-trivial increase in "getting started" complexity.

[eliasnaur]

It didn't occur to me before, but CL 198778 also breaks the gogio tool outside a module:

$ gogio -target js gioui.org/apps/hello
gogio: go list -f {{.ImportPath}} gioui.org/apps/hello failed: can't load package: package gioui.org/apps/hello: cannot look up module for package "gioui.org/apps/hello": working directory is not part of a module

The gogio tool uses go list which is not covered by @bcmills' #30515 suggestion :(

[bcmills]

Yeah, go list almost certainly should not be allowed to resolve packages when outside of a module.

go list -f {{.ImportPath}} in particular is fairly benign, but outside of a module it only succeeds when it is the identity function — so it's also not a particularly interesting example.

But it would be pretty bad if, say, you ran go list -f '{{with .Module}}{{.Path}}{{end}} example.com/apps/hello' and got back example.com/apps, and then the next time you ran a go command the package was in a different module because the owner had published a commit splitting example.com/apps/hello out into a separate module.

[bcmills]

Ok, but for the second and subsequent runs, go run only has to do ~1 query to the proxy which is likely to have the result cached, right?

That's assuming that the proxy has the module at all. For untagged, non-public modules, that query may involve fetching the head of the default branch of the repo and checking its go.mod file to see whether the path matches.

[eliasnaur]

If you're ok with go get outside of a module, can we allow go run, go list and so on outside of a module, but using only the module cache, never the network? Since the module cache is consistent, you won't run into problems where go list will say something that is incorrect when you run the next go command. Your builds are reproducible, at least relative to your module cache. Finally, the commands are fast because they don't access the network.

Quick start instructions for Gio programs will change to

$ go get gioui.org/apps/hello
$ go run gioui.org/apps/hello

and to

$ go get gioui.org/cmd/gogio gioui.org/apps/hello
$ go run gioui.org/cmd/gogio -target android gioui.org/apps/hello

for building an Android APK from a Gio program.

[jayconrod]

Let me point out that it's not always slow. With a warm cache, my experiment above measures the difference in the hundreds of milliseconds. Furthermore, the output "go: finding blah go: downloading blah go: extracting blah" is a clear hint to why a cold go run is slow.

This experiment only measures the fast case: the proxy already has the latest version of a project that has a go.mod file, and that version is cached locally. We're most concerned about the slow path where the proxy hasn't cached @latest: it might actually have that version, but it has to partially fetch the repo in order to verify there's nothing newer. If the project does not have a go.mod file, we might have to do that many times.

But the critical time is when a newcomer evaluates Gio. I'd like for their experience to be as painfree as possible and being able to run a Gio program with a simple go run without the PATH hassle is invaluable.

I definitely sympathize with this, but I think it's important for the "first time setup" instructions to be close to the "everyday best practice". Running a build tool at the latest version makes me nervous: I'd be scared to put that in a build script or CI. As a user, I want to be able to list versions of my dependencies (both libraries and tools), and from that, I want reproducible output. So I'm not convinced that running go run outside of a module as part of a build workflow for a larger project is something we should encourage.

Running these commands within a module seems fine of course, since the version will be recorded and the result will be reproducible. We know there are a number of issues around managing tool dependencies (#25922), and we're exploring some ways to fix that (#34506).

Great, let's implement the replacement before killing off go run outside modules. I don't know if that means fixing #30515, fixing the go install PATH issue, or something else entirely.

I'm happy to hold CL 198778 back for discussion for a little while, but we'd like to get it in for this release, and it should go in before the freeze.

#30515 seems viable if it's accompanied by a better cache eviction policy, but it probably won't go into 1.14.

I don't know what can be done to fix PATH though. It's admittedly confusing for new developers, but we do expect new developers to set that up when they install Go. I'd be surprised if we have a solution for that in 1.14, other than improved documentation.

[ianthehat]

once #23439 is fixed you can instead express the quickstart instructions as:

$ go get gioui.org/apps/hello && $(go env GOBIN)/hello

or right now you could do

$ GOBIN=$PWD go get gioui.org/apps/hello && ./hello

I guess we could also warn you if go get is going to write a binary to a place that is not on your PATH, but I think that would probably cause more confusion than it saved.

[eliasnaur]

Let me point out that it's not always slow. With a warm cache, my experiment above measures the difference in the hundreds of milliseconds. Furthermore, the output "go: finding blah go: downloading blah go: extracting blah" is a clear hint to why a cold go run is slow.

This experiment only measures the fast case: the proxy already has the latest version of a project that has a go.mod file, and that version is cached locally. We're most concerned about the slow path where the proxy hasn't cached @latest: it might actually have that version, but it has to partially fetch the repo in order to verify there's nothing newer. If the project does not have a go.mod file, we might have to do that many times.

go commands outside modules could be restricted to only work with projects with go.mod files. But if you insist that even the first invocation of go run, go install etc. must be fast, I think my module-cache-only proposal above is a solution.

But the critical time is when a newcomer evaluates Gio. I'd like for their experience to be as painfree as possible and being able to run a Gio program with a simple go run without the PATH hassle is invaluable.

I definitely sympathize with this, but I think it's important for the "first time setup" instructions to be close to the "everyday best practice". Running a build tool at the latest version makes me nervous: I'd be scared to put that in a build script or CI. As a user, I want to be able to list versions of my dependencies (both libraries and tools), and from that, I want reproducible output. So I'm not convinced that running go run outside of a module as part of a build workflow for a larger project is something we should encourage.

The difference between not encourage and flat out disabling functionality is significant. And again, please see my proposal above that won't use an arbitrary latest version, but rather the version in your modcache.

Great, let's implement the replacement before killing off go run outside modules. I don't know if that means fixing #30515, fixing the go install PATH issue, or something else entirely.

I'm happy to hold CL 198778 back for discussion for a little while, but we'd like to get it in for this release, and it should go in before the freeze.

#30515 seems viable if it's accompanied by a better cache eviction policy, but it probably won't go into 1.14.

And #30515 is about go install, not (yet) go run. Furthermore, -g is proposed to

As part of #34506, I'm proposing we add a -g flag (global mode) which would cause the go command to behave as if it were outside a module

Which seems to contradict CL 198778 that disables anything else than go get outside a module.

Let's say #34506 brings back the functionality of go run and friends, as long as you add -g. Then we'll have go run working fine (but slow) in Go 1.13, disabled in Go 1.14, and then working again in Go 1.1x (with a -g).

I don't know what can be done to fix PATH though. It's admittedly confusing for new developers, but we do expect new developers to set that up when they install Go. I'd be surprised if we have a solution for that in 1.14, other than improved documentation.

If you expect a certain PATH, why isn't the Go installer doing it? And why expect developers to set up PATH when you added go env -w because "Setting environment variables for go command configuration
is too difficult and system-specific." (#30411)?

That is very frustrating. You're removing functionality in Go 1.14 while leaving the (potential) replacements for Go 1.15 or perhaps later.

[eliasnaur]

By all means feel free to go ahead with CL 198778.

I also found the cache argument compelling, and I found a workaround that works on Windows:

$ cd somewhere
$ go mod init not-used.com
$ go run gioui.org/example/hello

The first line feels like busywork, in particular the unused module name, but Bryan's argument does make sense: create somewhere for the go command to record versions into and thereby eliminate any confusion about versions.

[gopherbot]

Change https://golang.org/cl/200297 mentions this issue: cmd/go: forbid module pattern 'all' when outside a module

[bcmills]

I think we missed a case: go mod verify should probably error out when outside of a module.

We should check test coverage for go mod download too.

[gopherbot]

Change https://golang.org/cl/202697 mentions this issue: cmd/go: make 'go mod verify' report an error outside a module

[gopherbot]

Change https://golang.org/cl/205065 mentions this issue: cmd/go: make 'go mod why' require a main module

[gopherbot]

Change https://golang.org/cl/216597 mentions this issue: doc/go1.14: go command behavior with GO111MODULE=on outside modules