proposal: cmd/go: automatically detect GOPATH

[adg]

Setting up a workspace/GOPATH is one of the big hurdles for new Go users. And maintaining (switching between) several workspaces is a common work flow for some Go developers.

This proposal is to make the GOPATH environment variable optional, and to have the go tool detect the GOPATH root on invocation. It would to this by walking up the tree from the cwd to find a directory named src, and setting GOPATH to the parent of that directory.

For new users, the getting started experience is something like:

mkdir -p src/github.com/user/hello
cd src/github.com/user/hello
vim hello.go
go install
hello

or to install something else:

mkdir src
cd src
go get github.com/docker/docker

Thoughts?

[broady]

/cc @jbuberel who was talking about this this morning.

[davecheney]

@adg thanks for raising this proposal. I thought it was going to be more along the lines of "lets have a default GOPATH value (and it might be $HOME)"

mkdir -p src/github.com/user/hello
cd src/github.com/user/hello
vim hello.go
go install
hello

I like this, but I think there are still a number of edge case that have to be explained en rote.

q. why do I need to make a sub directory, can't go just work in my current directory ?
a. no, not really, Go works with packages, and packages are a directory

q. oh, ok, well I put my code in src/hello.go and it didn't work !
a. ahh, well, src is special, you cannot put code directly in src/

q. ok, so I put my code in gocode/hello.go and it didn't work, you told me src/ was special so I didn't use it
a. err, right, sorry, you need to use src/, but you can't put any code in that, try src/hello/hello.go

q. ok, thanks for helping, that worked, but I have a question, why do I need all these directories, shouldn't Go be able to figure this out ?
a. insert long technical explantation about package names being derived from import paths and how src/ would name no import path so technically not be importable into other package, not that that it matters 'cos its a main package

q. what's a package, i just wanted to try helloworld.go ?!?

This is taken pretty much verbatim from my experiences having to justify the $PROJECT/src directory in gb.

[mdempsky]

I'd like to suggest that any automatic strategy to compute GOPATH should work anywhere within $GOPATH, not just within $GOPATH/src. E.g., it should at least also work within $GOPATH/bin and $GOPATH/pkg.

Easy variation of the proposed scheme is instead of looking for a parent directory named "src" (and then using its parent as GOPATH), look for a parent directory with a "src" child directory (and use the parent itself as GOPATH).

Another option would be to add a new identifying file within the root directory. E.g., Bazel uses an empty file named "WORKSPACE" (see "Getting Started with Bazel") and Chromium's GN uses a file named ".gn" with config settings (see "What you need for a minimal GN build").

[minux]

I think we don't need to support $GOPATH/bin and $GOPATH/pkg, because I can't think of a reason why I'd in one of those directory and still use go build/install/test. Just handle the common case looks good to me. I also don't like introducing a new file to mark GOPATH.

[adg]

@davecheney I think at some point people need to learn about packages. I don't think we should design our entire build process around the very first new user experience. Can you suggest a way of avoiding this conversation that doesn't throw out everything we already have? (My gut reaction is "just tell them to put it in "src/hello" in the first place.")

@mdempsky I concur with @minux - I have no idea why you would care about resolving $GOPATH inside pkg or bin. I never change to those directories, ever. I also don't see the purpose in adding special files, when we already have a special directory (src).

Another consideration for this proposal is that it will theoretically break anyone that already has packages whose import paths have a src path component.

[hallas]

👍 This change makes sense. I don't see what a default $GOPATH (for example at $HOME) would solve in terms of making it easier to get started with Go. I think if a user knows about $HOME (its significance) then he is proficient enough to setup a correct $GOPATH.

[davecheney]

@mdempsky I think the idea of "GOPATH is the parent of a directory called src/" works well.

@adg

I think at some point people need to learn about packages.

Agreed.

I don't think we should design our entire build process around the very first new user experience.

I'll have to disagree with you to some degree there 😄

Can you suggest a way of avoiding this conversation that doesn't throw out everything we already have? (My gut reaction is "just tell them to put it in "src/hello" in the first place.")

I think I probably conflated a number of things in my response. From my experience there are several common pain points for newcomers when they see a path like src/github.com/user/hello

1. Why do I have to setup GOPATH, why can't I just put my stuff anywhere I want (or more likely where I put it in $INSERT_PREVIOUS_LANGUAGE). To that, I agree with you, the answers is "Because reasons".
2. The second is why do they have to put things in a subdirectory of a subdirectory src/ ? Again, it's probably better to stick with "reasons".
3. The third is the confusion that putting a github'ish url in the path. We've seen lots of users who think they have to push to github to compile.

Maybe this is just a matter of introducing the terms in order. Something like

---

There are two pieces of information you need to know to write Go code. The first is all Go code must live inside a workspace, we call that GOPATH. A GOPATH workspaces is any directory that contains the following three directories src/, bin/, and pkg/. You can create one like this

mkdir -p gocode/{src,bin,pkg}

So, gocode is your workspace, your GOPATH. gocode/src is where you will put your Go packages (we'll get to that in a second), gocode/bin is where the compiler will place any programs you compile, and gocode/pkg is the location that the compiler will cache files for later reuse to speed up compilation.

As mentioned above, all the Go code you write must be arranged into packages. What is a package ? A package is just a directory. The name of the directory must match the name of the package declaration that every file in that directory shares, that's why we call them packages.

Putting these two pieces of knowledge together, if you want to write a program called hello, it needs to be in the hello package, and knowing that packages need to live inside your workspaces' src/ directory, the path to this package must be gocode/src/hello

---

This explanation obviously trades off some degree of accuracy to avoid getting bogged down in the details, please don't beat me up for this.

[nightlyone]

What about go init [-path] where path defaults to $HOME/gopath and be done with it?

That could initialize the path, create an directory example too with a minimal main + package and fill this with some dummy content working on all major systems. It then also shows what it did (e.g. go get -v -x example/...), where it puts the stuff and what it actually is (e.g. a minimal binary, an example package, etc.)

When the $GOPATH is missing, suggest setting $GOPATH or calling go init to the user, if he doesn't know what we are talking about.

(Linux- )Distributions could then also easily integrate go init, if their policy needs defaults.

(Appropriate defaults for e.g. Windows are left out for the moment but can be easily added)

[mortdeus]

Is there a reason why we can't just build our default gopath in $GOROOT? Just add a $GOROOT/usr/$USER directory which is populated with it's own /bin, /src, /pkg and auto generated by the go tool.

Its one thing to have the developer explicitly state where they specify $GOPATH resides on their system, its another to allow go to automatically dig around for one.

[tylerdmace]

I think the original proposal is fine for those who understand package and workspace structure but I don't believe it to be helpful to new Go users as was suggested. The problem is that it's very likely that new Go users will be creating test directories within existing directory trees that contain code of different languages -- and src is likely to be a directory that exists all over within that same tree.

[mortdeus]

The go tool doesn't care if you have scripts/source written in other programming languages stored inside the same directory it found a set of *.go files in. It only cares that each declared package is confined within its own directory inside "$GOPATH/src".

I "get" what @adg is saying. Ideally when you use the command go install or go build, you have fed the tool a path (either by an explicit declaration or an implicit assumption that $PWD is your intended target). Given that the go tool already knows where the package you want to build resides, all it has to do is hop out into each subsequential parent directory until it finds itself inside a parent directory called "src". Which then the go tool can deduce that ../src is either a gopath that is already known to the go tool, or it's a base directory where the go tool can construct a new gopath if it is found that one doesn't already exist at said location.

I mean it definitely could work. It's just a really hooky programming environment convention if you ask me. It doesn't seem like it would always play nicely with Unix's file system security model either. (you develop your pkg in /usr/src, go installs executable in /usr/bin, which could bring about potential issues like system util name conflicts, or confused package managers, etc etc.

Personally, I think @adg's proposal makes the go tool far too "magic wand"-ey, for the amount of convenience the language would gain from his proposal's implementation.

[tylerdmace]

My concern wasn't so much that the code of other languages would be an issue; only that src is a common directory name and I figured there would be some potential conflicts in just assuming that you were working with a Go package's src directory when in fact it could be something entirely different. You gave a good example, @mortdeus -- /usr/src could be a conflicting directory if they happened to be working out of it. It's not likely but it could happen.

[mortdeus]

Also if we did do this we would have to adopt a policy that each gopath would be isolated from each other's view unless GOPATH has been explicitly set. The reason being the fact that the go tool can't resolve a lookup path precedence ambiguity with out a directorial cue from the user that expresses the preference of their intent.

[tylerdmace]

What if go could configure workspace configurations?

$ go create workspace /home/tyler/myworkspace

Then from anywhere you could do something like:

$ go myworkspace build

Maybe allow for a default or an active workspace:

$ go active workspace myworkspace

With an active workspace set you could simply do:

$ go build

The workspace name would be derived from the path passed at creation and could be referenced as simply.

To facilitate this, you’d need to allow for a GOCONF environment variable to point to a location where a simple config file (containing workspace names and GOPATHs for each) could be written/read by the tool and at that point perhaps the complexity of the situation is just the same; Thoughts?

[mortdeus]

I was also thinking about the idea for a go "workspace" project container object that essentially gives users one last degree of expressing package addressability. You wouldn't have to manually create, set, manage, and/or mess with them in any explicit way really.

Rather the workspace mechanism would work by walking up the tree until we find ourselves inside of a "src" directory. Once there we "cd .." into the parent base dir where we ideally want to set up the gopath we associate with the pkg were building. Since that parent base directory must have an identifier, we can use it to declare a workspace context for that GOPATH.

Syntactically the "workspace" identifier is treated just like a top level domain in all its usage contexts. For example if we wanted to call an executable that is in a specific $GOPATH/bin, we can just call go tool workspace.execname. Which is useful considering the fact that that GOPATH/bin might not be appended to the system path and would otherwise require us to call the exec using its full canonical file path identifier.

In the context of it's import path string, it would be just a top level domain.
import "workspace.gorepo.com/path/2/mypkg"

Then in $GOROOT, go get will fetch and install to $GOROOT/users/$USER/$workspace/src/path/2/mypkg

In the case that the top level domain is attachted to say github.com. We can just use the top level domain syntax to allude to a sub branch and/or tagged commit. And just create the workspace context name by appending the pkgname in front of the top level domain.

So lets say we have import "v2.github.com/mortdeus/mypkg", the go get tool will see the v2, then the github.com, then the user name, and then the pkgname. Go get will resolve this to the directory $GOROOT/users/mortdeus/mypkg.v2/src/github.com/mortdeus/mypkg

Then if we have a few executables called say mypkg/server, mypkg/client, mypkg/shell, mypkg/main
we can call the executables like 'go tool mypkg.server', 'go tool mypkg.shell', 'go tool mypkg.main' (or just 'go tool mypkg' in the special case of mypkg.main).

Idk its just an idea.

[mortdeus]

oh and in the case where we have a set of workspace ambiguity where we have say $GOROOT/users/{me | you}/gopkg/src/github.com/{my|your}/gopkg

Obviously we have a conflict here regarding which "gopkg" executables are tied to the globally resolved generic "gopkg" workspace identifier as far as the go tool is concerned. The best way I can think of resolving this is to have a $GOROOT/users/mortdeus/bin that links to the workspace's executables we prefer to be called in the case of a naming conflict.

[yiyus]

Another consideration for this proposal is that it will theoretically break anyone that already has packages whose import paths have a src path component.

A solution could be to say that the GOPATH is the parent of the top (instead of the bottom) src directory in the current directory. So, for example, when running go install from /home/user/src/github.com/project/src/package1, GOPATH would be set to /home/user.

I think it is more unlikely to break something this way and with the right wording it is not more complicated (I am assuming the fraction of users with GOPATHs inside src directories is neglegible). For new users, I think the most important point is showing a sensible error message when trying to build a package outside of a GOPATH (or with this proposal, outside of a src directory).

[XANi]

shouldn't it be something like $HOME/src/go instead ? src/ is a very generic name and there is a decent chance someone will be using it for something else than golang

[crawshaw]

I like @adg's proposal at the beginning of this issue of determining the GOPATH from the current working directory if the environment variable is not set.

There are several other interesting ideas proposed here, like a default GOPATH and a go init subcommand, that are compatible with the original proposal. They could be done instead of or as well as automatically detecting the GOPATH. Are any of them arguments against determining the GOPATH from the current working directory? If they are complementary they can be considered separately. If they are arguments against @adg's original proposal, I think it would be helpful to frame them as such.

[bictorman]

In my opinion auto-detecting GOPATH via src is too magical and can lead to unintended behaviour, specially since src is ubiquitous on developing environments.

As a multiple workspace and direnv user, I can vouch for a file .gopath generated via go init. It's unambiguous, most people are already familiar with the pattern thanks to git, and enables more extensibility as a potential source of metadata, like multiple paths, build flags, etc.

[ngrilly]

@adg's proposal would be a nice improvement to the tool chain. It doesn't change anything when GOPATH is already set, and when it's not, it simplifies the workflow. Many tools like git, hg or Node/npm already successfully implement a similar behaviour.

[rsc]

I really like the idea of auto-detecting GOPATH. I've been thinking about how to do this ever since I saw that @davecheney had done the equivalent in gb.

I'm not as sure about how. Finding src does seem to be the obvious choice, and it may work well. On the other hand, it may lead to wrong or confusing behavior if it found an unrelated src. At the least I think you'd need to say it's the highest-level src, so that if someone names their GitHub repo src and you're in /home/you/src/github.com/otherguy/src/pkg that uses GOPATH=/home/you.

Then there's the problem that any directory-based detection fails when you're outside the directory. You might be in $HOME or $HOME/bin and realize you need a command and run "go get that/command" and it fails because you're not in the src tree. That's annoying.

@davecheney's other points based on experience with gb are interesting too.

Counter-proposal, as suggested by Dave: If GOPATH is not set, it defaults to the user's home directory (aka $HOME on Unix, $USERPROFILE on Windows, $home on Plan 9).

The counter-proposal is not as flexible, but it's the suggested setting, it works outside the source tree, and it can't ever be confused by an inadvertently named parent directory.

I put the counter-proposal here mainly to play devil's advocate: if we're going to autodetect, we should make sure we are considering all the possible ways, not just one, and then pick the best.

As I go through the details, though, I'm starting to like default $HOME better than default magic search based on current directory.

[davecheney]

@rsc thanks for your comments.

re: finding the wrong src directory in a hierarchy. I think this is quite
unlikely, to the point that the advice could be 'don't do that'. There are
some Go repos on github (not naming any names, but those may be children of
the alphabet group) that include a '/src/' directory in the checkout, but
they could be said to be in the former category and should be fixed.

re: not being able to do 'go $CMD' anywhere on the filesystem is quite a
limitation. In both this and the previous case gb punts on this by making
it clear you arrange your work into projects and decide which project you
are working on by being present inside that directory -- a model well
understood by most git/svn/hg users. As a backup there is a -R flag to
override the root of the project detection logic, analogous to -C in modern
versions of git.

With that said, the place where this will bite will be go get from
scratch, that is to say, the first user experience; install go; go get
$BIGPROJECT; confusing error message as there is no /src/ to be found on
the filesystem and the user was following some potted advice from a project
README, not our getting started documentation.

In the face of this, defaulting $GOPATH to $HOME would

a. solve all of the issues above
b. mean that in the average case users have to set zero environment
variables to use Go, a moderate win for posix users, and a substantial win
for Windows users.

And, as you say, if people don't like the default, all they have to do is
set $GOPATH, just as they have for the last 5 years.

On Sat, Oct 24, 2015 at 2:54 PM, Russ Cox [email protected] wrote:

I really like the idea of auto-detecting GOPATH. I've been thinking about
how to do this ever since I saw that @davecheney
https://github.com/davecheney had done the equivalent in gb.

I'm not as sure about how. Finding src does seem to be the obvious
choice, and it may work well. On the other hand, it may lead to wrong or
confusing behavior if it found an unrelated src. At the least I think
you'd need to say it's the highest-level src, so that if someone names
their GitHub repo src and you're in /home/you/src/
github.com/otherguy/src/pkg that uses GOPATH=/home/you.

Then there's the problem that any directory-based detection fails when
you're outside the directory. You might be in $HOME or $HOME/bin and
realize you need a command and run "go get that/command" and it fails
because you're not in the src tree. That's annoying.

@davecheney https://github.com/davecheney's other points based on
experience with gb are interesting too.

Counter-proposal, as suggested by Dave: If GOPATH is not set, it defaults
to the user's home directory (aka $HOME on Unix, $USERPROFILE on Windows,
$home on Plan 9).

The counter-proposal is not as flexible, but it's the suggested setting,
it works outside the source tree, and it can't ever be confused by an
inadvertently named parent directory.

I put the counter-proposal here mainly to play devil's advocate: if we're
going to autodetect, we should make sure we are considering all the
possible ways, not just one, and then pick the best.

As I go through the details, though, I'm starting to like default $HOME
better than default magic search based on current directory.

—
Reply to this email directly or view it on GitHub
#12488 (comment).

[davecheney]

Hello,

What is the status of this proposal?

If it's stalled, I'd like to make my own, simpler, proposal along the lines of #12488 (comment).

[adg]

@davecheney I'm happy for you to make a counterproposal here, unless it is not about automatically detecting GOPATHs at all.

[davecheney]

TL;DR - I want to propose that GOPATH defaults to $HOME.

On Mon, Sep 26, 2016 at 9:35 AM, Andrew Gerrand [email protected]
wrote:

@davecheney https://github.com/davecheney I'm happy for you to make a
counterproposal here, unless it is not about automatically detecting
GOPATHs at all.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#12488 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA-3HUF6GkN3RRthEjd9jelKKlCcqks5qtwU-gaJpZM4F3oTt
.

[bradfitz]

@davecheney, not until we nuke the "pkg" directory. I'd like to add proper caching to cmd/go early in the Go 1.9 cycle, and then we could probably ditch pkg.

[davecheney]

@branfitz is that because I suggested GOPATH=$HOME, if I suggested an
alternative location like GOPATH=$HOME/gocode would that unblock this
proposal so we don't have to wait til /pkg is removed ?

On Mon, Sep 26, 2016 at 9:53 AM, Brad Fitzpatrick [email protected]
wrote:

@davecheney https://github.com/davecheney, not until we nuke the "pkg"
directory. I'd like to add proper caching to cmd/go early in the Go 1.9
cycle, and then we could probably ditch pkg.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#12488 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA7gJLMgZcip7HinQlVCSho31u-Qyks5qtwljgaJpZM4F3oTt
.

[mcluseau]

@davecheney GOPATH defaulting to home won't fix the "complex CI build" issue, but is nice. If pkg is removed at some point, all we'll need will be to have a flag to specify the position of the CWD in the go import tree.

[davecheney]

If you don't want GOPATH=$HOME then all you have to do is set $GOPATH
manually. For 100% of Go users today, nothing has changed.

On Mon, Sep 26, 2016 at 10:00 AM, Mikaël Cluseau [email protected]
wrote:

@davecheney https://github.com/davecheney GOPATH defaulting to home
won't fix the "complex CI build" issue, but is nice. If pkg is removed at
some point, all we'll need will be to have a flag to specify the position
of the CWD in the go import tree.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#12488 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA2AZGcpxM-trMf7TCv_ghstjS6lFks5qtwsvgaJpZM4F3oTt
.

[bradfitz]

@branfitz is that because I suggested GOPATH=$HOME, if I suggested an
alternative location like GOPATH=$HOME/gocode would that unblock this
proposal so we don't have to wait til /pkg is removed ?

I wouldn't object, at least. But I (and I think others) would object to Go automatically creating a $HOME/pkg dumping ground, surprising users.

[crawshaw]

@bradfitz is there a plan to do away with pkg? It does not seem easy to me, it is an important performance optimization, especially if you are importing cgo packages.

What if an unset GOPATH implied pkg being somewhere off in a temp file system?

[davecheney]

To be clear, I want GOPATH to be a default, I don't care what the default
is because it can be overridden by setting GOPATH explicitly.

On Mon, Sep 26, 2016 at 10:04 AM, Brad Fitzpatrick <[email protected]

wrote:

@branfitz is that because I suggested GOPATH=$HOME, if I suggested an
alternative location like GOPATH=$HOME/gocode would that unblock this
proposal so we don't have to wait til /pkg is removed ?

I wouldn't object, at least. But I (and I think others) would object to Go
automatically creating a $HOME/pkg dumping ground, surprising users.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#12488 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA69DmumnKzwQvmL8_4Yhlb6zOevyks5qtwvxgaJpZM4F3oTt
.

[mcluseau]

On 09/26/2016 11:03 AM, Dave Cheney wrote:

If you don't want GOPATH=$HOME then all you have to do is set $GOPATH
manually. For 100% of Go users today, nothing has changed.

I agree, I just want to underline the path that put me in this thread,
which is "how to build easily in gitlab-ci", where your git repo is in
$HOME/build/some-variable-part/repo-name. We need to be able to say "hey
go, this folder where you are is the import path for mycomp.com/my-project".

[davecheney]

The former. The latter may have to set an GOPATH environment variable, this
is no different to what they have done for Go 1.1 to Go 1.7.

On Fri, Dec 11, 2015 at 10:30 AM, Dmitri Shuralyov <[email protected]

wrote:

Is this proposal about helping people who don't know/understand GOPATH
workspaces be able to use Go? Or is it about making it more convenient for
people who know it but deal with many systems, CI, etc.?

—
Reply to this email directly or view it on GitHub
#12488 (comment).

[davecheney]

@MikaelCluseau in that case nothing changes, the CI user will have to set GOPATH, just as they do today.

[davecheney]

$HOME/build/some-variable-part/repo-name. We need to be able to say "hey
go, this folder where you are is the import path for mycomp.com/my-project".

I'd say this is out of scope. Various CI tools like travis have worked around the fact that the checkout of the go project is not the root of the workspace. My proposal to set a default GOPATH does not change this.

[mcluseau]

On 09/26/2016 11:10 AM, Dave Cheney wrote:

@MikaelCluseau https://github.com/MikaelCluseau in that case nothing
changes, the CI user will have to set GOPATH, just as they do today.

yes, that's why I said it doesn't solve the problem. But I understand it
can be seen as a different issue.

[bradfitz]

@bradfitz is there a plan to do away with pkg? It does not seem easy to me, it is an important performance optimization, especially if you are importing cgo packages.

Yes. That is #4719. It will still exist, but instead of being at $GOPATH/pkg, it'll be at a OS- and user-specific cache directory, and it'll be a cache of outputs from actions of content-addressable inputs. Which means go test -i as a necessary practice goes away, etc.

[davecheney]

This proposal only changes the experience for newcomers who have not chosen to set a GOPATH variable.

At the moment all the documentation says "you have to set a GOPATH", then people get distracted and upset because they don't understand why they have to do this.

By choosing a default GOPATH, we can make our documentatation easier because we can say things like

$ go get github.com/foo/bar

will check out the github.com/foo/bar repo into $HOME/gocode/src/github.com/foo/bar.

We don't need to distract newcomers with having to set an env var, all we need to do is put a little parenthetical note at the bottom of the page

$HOME/gocode is the default path to your go workspace. If you want to change this path, set the GOPATH variable to something of your choosing.

[LaPingvino]

I foresee problems here though when you e.g. start using a new shell and in the old situation the error message would inform you that it's not set up correctly yet. With your proposal you would silently get the code checked out and duplicated, and you end up with a mess, if you don't use the same GOPATH. in my proposal that doesn't happen, because any new shell will automatically pick up the same old directory, it works when you migrate to a new system and will gently put the new users through it.

[LaPingvino]

Also, if you add this stuff to an independent tool, you can backport it to old versions where necessary.

[davecheney]

@rsc as the proposal arbitrator would you welcome a proposal to provide a default GOPATH if one is not explicitly set in the environment?

[adg]

I think we're going with #17262.