cmd/go: drop $GOPATH/pkg

[bradfitz]

(after discussion with Russ, trying to summarize here)

It's nice to be able to tell people to set $GOPATH == $HOME, which means their source
goes into $HOME/src and their binaries conveniently go into $HOME/bin.

The only wart with $GOPATH == $HOME is the $HOME/pkg directory, which is pretty ugly,
not useful for end users, and somewhat offensive to be littering in people's $HOME.

If issue #4443 and issue #3895 get fixed, $GOPATH/pkg might get even uglier.

Unlike $GOROOT/pkg, $GOPATH/pkg doesn't contain tool binaries.  It's pretty much just a
cache.

Can't cmd/go's *.a cache go into /tmp/go-cache-$USER 0700 or ~/Library/Caches with
appropriate occasional lazy cleaning?  (or letting the system clean).

[adg]

Comment 1:

Do you propose to change the behavior for everyone, or make this configurable?
What about people that want to ship .a binaries only? I think the go tool currently
supports that. (ie, just mkdir the src/ path and put the binaries under pkg/ and it
works)

[minux]

Comment 2:

FYI, for binary-only package to work, you will need to create a dummy "package xxx" Go
source
file in the directory with an older timestamp than the .a file.
This is issue #2775.

[rsc]

Comment 3:

There are many facets to think about. This is just an issue to remind us to
do so.

[rsc]

Comment 4:

Labels changed: added priority-later, go1.1maybe, removed priority-triage, go1.1.

Status changed to Thinking.

[robpike]

Comment 5:

Labels changed: removed go1.1maybe.

[rsc]

Comment 6:

Labels changed: added go1.3.

[robpike]

Comment 7:

Labels changed: removed go1.3.

[rsc]

Comment 8:

Labels changed: added go1.3maybe.

[rsc]

Comment 9:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 10:

Labels changed: added repo-main.

[bradfitz]

Every release (especially Go 1.5) has been getting us closer to eliminating "pkg" and instead having a build cache directory somewhere (some system-specific cache directory)

Having a cache instead of "pkg" would also eliminate the frequent problems people run into where "go build" does too much work when they could've used "go install" instead.

/cc @adg @davecheney

[griesemer]

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based on historic ideas of how a program is compiled into object files which are then linked. (In the earliest days of Go I suggested that we make this compile/link process invisible - i.e., as a programmer one just "sees" source code. But we were not ready at that time for this).

[davecheney]

@bradfitz,

I view 'pkg' as this cache, and I'm fine with it living inside GOPATH
because that makes it simple to explain where to find it if people are
determined to delete it.

The change I implemented in gb was to remove the distinction between go
build, go install, and the other odd -I variants; if gb can cache
something, even while building tests, it will and this has worked out
pretty well.

For my money removing the number of ways one can tell them go tool to cache
things and just make the caching more pervasive would be a welcome
improvement.

If that cache lives inside GOPATH, or some default location relative to
HOME feels less important, but I suspect that the windows users will have a
issues with a large number of files written to their home directories,
which may not be local to their machines.

Thanks

Dave

On Wed, 30 Sep 2015 03:17 Robert Griesemer [email protected] wrote:

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based
on historic ideas of how a program is compiled into object files which are
then linked. (In the earliest days of Go I suggested that we make this
compile/link process invisible - i.e., as a programmer one just "sees"
source code. But we were not ready at that time for this).

—
Reply to this email directly or view it on GitHub
#4719 (comment).

[bradfitz]

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and Go 1.5 is basically there already), then deleting it is unnecessary. If we really wanted, we could have a go XXX subcommand to free up disk space, if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also cached go build, that's great, but then my next annoyance is that I have to look at it. I like to set my $GOPATH to $HOME, and that means I have $HOME/pkg being a useless turd right in my home directory. So I'd rather put things in the right spots:

• For Linux etc we have stuff like http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html for the user-specific cache directory.
• OS X has $HOME/Library/Caches
• Windows has %USERPROFILE%\AppData\Local\Temp (aka $TMP aka $TEMP) (from http://technet.microsoft.com/en-us/library/cc749104(v=ws.10).aspx

[davecheney]

I cannot explain why people want to delete GOPATH/pkg, I believe it is
unnecessary, and in every case where I have asked for more details not have
been provided, yet the lore continues to propagate itself.

I'm fine with using the LSB mandates tmp directories, that feels like the
smaller argument vs making the go tool more aggressive with its caching.

On Wed, 30 Sep 2015 07:50 Brad Fitzpatrick [email protected] wrote:

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and
Go 1.5 is basically there already), then deleting it is unnecessary. If we
really wanted, we could have a go XXX subcommand to free up disk space,
if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also
cached go build, that's great, but then my next annoyance is that I have
to look at it. I like to set my $GOPATH to $HOME, and that means I have
$HOME/pkg being a useless turd right in my home directory. So I'd rather
put things in the right spots:

• For Linux etc we have stuff like
  http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
  for the user-specific cache directory.
• OS X has $HOME/Library/Caches
• Windows has %USERPROFILE%\AppData\Local\Temp (aka $TMP aka $TEMP)
  (from http://technet.microsoft.com/en-us/library/cc749104(v=ws.10).aspx

—
Reply to this email directly or view it on GitHub
#4719 (comment).

[mwhudson]

If pkg is to become a cache, and possibly more hidden, it would be good to fix the ways in which GOOS + GOARCH is not quite a valid cache key: GOARM is probably the worst offender here, but also tags and things like buildmodes and the race detector (although the go tool handles the last two via installstuffix currently).

[minux]

Yeah, GOARM/GO386 and build tags (including race) should also be part of the cache key (ideally, build tag that doesn't affect the compilation of one package shouldn't be cached under the cache with the build tag.) The toolchain version should also be part of the cache key. Esp. for those who switch Go releases frequently.

[ChrisHines]

Forgive me if I have missed subtleties that have already been discussed.

I have not seen any discussion of how a system global pkg cache would work for people who switch between multiple GOPATHs, each of which might contain different versions of the same package(s). I understand that recent versions of the go tool will hash all the file names used to build a .a file to help keep the cache accurate. But different versions of a package in different GOPATHs could easily have the same file list but different code. It seems to me either separate caches are required per GOPATH (as the pkg directory does today) or the cache key would need to include something to differentiate between the multiple GOPATHs used.

[rsc]

Yes, it would have to be a good cache. :-)

[rsc]

(That is, it would have to take all these things into consideration. It's not a trivial problem, which is why it hasn't happened.)

[rogpeppe]

+1 to this, particularly making it toolchain-sensitive too. I often find myself switching between different compiler versions so I can check behaviour differences, and switching/removing $GOPATH/pkg is a pain.

[rajender]

@abitrolly minimal vendor support without GOPATH is being added as part of new modules feature to Go.11. Note Go.1.11 is not released yet. You can try the beta version.

[gopherbot]

Change https://golang.org/cl/126755 mentions this issue: cmd/go: move module cache from $GOPATH/src/mod to $GOPATH/pkg/mod

[gopherbot]

Change https://golang.org/cl/129679 mentions this issue: doc/code: drop mentions of GOPATH/pkg directory

[perillo]

shouldn't the module cache be in os.UserCacheDir() instead of $GOPATH/pkg/mod?

[bradfitz]

@perillo, we discussed that (somewhere) earlier and decided to just use $GOPATH/pkg for now while we had it. But yes, it'd have to move later when this bug is resolved.

[rsc]

@perillo, or we might leave the module source cache in $GOPATH/pkg.

I think there are good reasons not to put cached module downloads in $HOME/.cache. Quoting myself on https://dave.cheney.net/2018/07/14/taking-go-modules-for-a-spin and https://groups.google.com/forum/#!msg/golang-dev/RjSj4bGSmsw/KMHhU8fmAwAJ, back when the module cache was $GOPATH/src/mod not $GOPATH/pkg/mod:

The build cache ($GOCACHE, defaulting to $HOME/.cache/go-build) is for storing recent compilation results, so that if you need to do that exact compilation again, you can just reuse the file. The build cache holds entries that are like “if you run this exact compiler on these exact inputs. this is the output you’d get.” If the answer is not in the cache, your build uses a little more CPU to run the compiler instead of reusing the output. But you are guaranteed to be able to run the compiler instead, since you have the exact inputs and the compiler binary (or else you couldn’t even look up the answer in the cache).

The module cache ($GOPATH/src/mod, defaulting to $HOME/go/src/mod) is for storing downloaded source code, so that every build does not redownload the same code and does not require the network or the original code to be available. The module cache holds entries that are like “if you need to download [email protected], here are the files you’d get.” If the answer is not in the cache, you have to go out to the network. Maybe you don’t have a network right now. Maybe the code has been deleted. It’s not anywhere near guaranteed that you can redownload the sources and also get the same result. Hopefully you can, but it’s not an absolute certainty like for the build cache. (The go.sum file will detect if you get a different answer on re-download, but knowing you got the wrong bits doesn’t help you make progress on actually building your code. Also these paths end up in file-line information in binaries, so they show up in stack traces, and the like and feed into tools like text editors or debuggers that don’t necessarily know how to trigger the right cache refresh.)

I think that's still basically right. If the module cache is the only thing left in $GOPATH/pkg, we could decide to move it elsewhere, or we could decide to leave it there. Moving it to $HOME/.cache seems wrong though.

Really I'm just restating what I said in b8f42d7 on Jul 30, 2018 above:

If we do eliminate GOPATH/pkg for holding .a files (#4719)
then we could still keep it around for pkg/mod.
(Or we could move the module cache again then.)

[vith]

@rsc I think I understand and agree with all those points individually, but I'm missing the link between them and being opposed to using $XDG_CACHE_HOME (default $HOME/.cache) or its equivalent on other platforms.

My motivation for wanting to use a platform's standard cache location is that these caches tend to grow and never shrink, and eventually it becomes necessary to reclaim some disk space. Having a single location where all "well behaved" software stores caches that are safe to delete makes it more convenient to do that kind of maintenance.

Without standards like the XDG base directory spec, most software tends to think it deserves its own entry in your ls -a ~ output. But the more software you use, the more of a mess that becomes.

I suspect we have a different level of concern around this point:

It’s not anywhere near guaranteed that you can redownload the sources and also get the same result.

I don't think ~/.cache is meant to be limited to things that could be recomputed from locally available non-cache data. Looking inside mine, I see multiple browser caches, yarn module cache, typescript typing cache, a system package manager cache (yay), etc. All of those are caching data from the internet.

Looking at the XDG data dir spec, if a distinction must be made between the durability of the two go caches, maybe $XDG_DATA_HOME ($HOME/.local/share) should be considered.

[tv42]

I do get rsc's point, some of Go cache is "cheap to discard", some is "expensive to discard and potentially unavailable if lost". Note that XDG says nothing about how $XDG_CACHE_HOME should or can be cleaned, or how often etc, all it says is the directory is for "non-essential (cached) data". So I guess you could argue that any automatic tool that removes ~/.cache/go/mod or whatever in way that annoys the user is just too eager, and should learn better rules about prioritization and retainment. Maybe.

For what it's worth, rust seems to plan migrating from ~/.cargo to ~/.cache/cargo for what seem to me to be similar uses: rust-lang/cargo#148 (don't trust Github claims of "Merged").

[mathstuf]

The other thing a single cache directory location is for is a simple place to exclude things from backups.

[flibustenet]

Why will we want a long cache when there will be global modules proxies ?
I think if we want to be really safe we need a cache per project (instead of vendor ?). I'd like this cache hardlinked on linux.

[rsc]

This issue was originally about the .a files in $GOPATH/pkg.

Those are no longer generated when using the build cache,
and as of Go 1.12, use of the build cache is required.
So it has been quite a long time since we wrote .a files to $GOPATH/pkg.
(The build cache also fixed issue #4443 and #3895 that Brad mentioned in the issue text.)

Since the issue was written, we added $GOPATH/pkg/mod and $GOPATH/pkg/sumdb.
I am comfortable with those staying as they are today.
They don't have any of the problems that the .a files had.
Now that GOPATH defaults to $HOME/go,
$HOME/go/pkg/mod and $HOME/go/pkg/sumdb
are completely reasonable paths.
(I still prefer to set GOPATH=$HOME, but that's a choice not a default.)

Closing this issue as fixed by the now-required build cache.