Skip to content

Commit

Permalink
"gobin" Go module caster (#23)
Browse files Browse the repository at this point in the history 
>>> Go Executable Installation

When installing a Go executable from within a Go module [1] directory
using the `go install` command [2], it is installed into the Go
executable search path that is defined through the `GOBIN` environment
variable [3] and can also be shown and modified using the `go env`
command [4]. Even though the executable gets installed globally, the
`go.mod` file [5] will be updated to include the installed packages
since this is the default behavior of the `go get` command [6]
running in "module mode" [7].

Next to this problem, the installed executable will also overwrite any
executable of the same module/package that was installed already,
but maybe from a different version. Therefore only one version of a
executable can be installed at a time which makes it impossible to work
on different projects that use the same tool but with different
versions.

>>>> History & Future

The local installation of executables built from Go modules/packages has
always been a somewhat controversial point which unfortunately,
partly for historical reasons, does not offer an optimal and
user-friendly solution up to now. The `go` command [8] is a fantastic
toolchain that provides many great features one would expect to be
provided out-of-the-box from a modern and well designed programming
language without the requirement to use a third-party solution:
from compiling code, running unit/integration/benchmark tests, quality
and error analysis, debugging utilities and many more.
Unfortunately the way the `go install` command [2] of Go versions less
or equal to 1.15 handles the installation of an Go module/package
executable is still not optimal.

The general problem of tool dependencies is a long-time known issue/weak
point of the current Go toolchain and is a highly rated change request
from the Go community with discussions like golang/go#30515 [9],
golang/go#25922 [10] and golang/go#27653 [11] to improve this essential
feature, but they've been around for quite a long time without a
solution that works without introducing breaking changes and most users
and the Go team agree on.
Luckily, this topic was finally picked up for the next upcoming Go
release version 1.16 [12] and golang/go#40276 [13] introduces a way to
install executables in module mode outside a module. The release note
preview also already includes details about this change [14] and how
installation of executables from Go modules will be handled in the
future.

>>>> The Workaround

Beside the great news and anticipation about an official solution for
the problem the usage of a workaround is almost inevitable until Go 1.16
is finally released.

The official Go wiki [15] provides a section on "How can I track tool
dependencies for a module?" [16] that describes a workaround that tracks
tool dependencies. It allows to use the Go module logic by using a file
like `tools.go` with a dedicated `tools` build tag that prevents the
included module dependencies to be picked up included for normal
executable builds. This approach works fine for non-main packages,
but CLI tools that are only implemented in the `main` package can not be
imported in such a file.

In order to tackle this problem, a user from the community created
"gobin" [17], an experimental, module-aware command to install/run main
packages.
It allows to install or run main-package commands without "polluting"
the `go.mod` file by default. It downloads modules in version-aware mode
into a binary cache path within the systems cache directory [18].
It prevents problems due to already globally installed executables by
placing each version in its own directory. The decision to use a cache
directory instead of sub-directories within the `GOBIN` path keeps the
system clean.

"gobin" is still in an early development state, but has already received
a lot of positive feedback and is used in many projects.
There are also members of the core Go team that have contributed to the
project and the chance is high that the changes for Go 1.16 were
influenced or partially ported from it.
It is currently the best workaround to...

1. prevent the Go toolchain to pick up the `GOMOD` environment
   variable [4] (see [`go env GOMOD`][4]) that is initialized
   automatically with the path to the `go.mod` file in the current
   working directory.
2. install module/package executables globally without "polluting" the
   `go.mod` file.
3. install module/package executables globally without overriding
   already installed executables of different versions.

See gobin's FAQ page [19] in the repository wiki for more details about
the project.

>>>> The Go Module Caster

To allow to manage the tool dependency problem, wand now uses "gobin"
through a new caster [20]
`go.mod` file and allows to...

1. install `gobin` itself into `GOBIN` (`go env GOBIN` [4]).
2. cast any spell incantation [21] of kind `KindGoModule` [22] by
   installing the executable globally into the dedicated `gobin` cache.

[1]: https://golang.org/ref/mod
[10]: golang/go#25922
[11]: golang/go#27653
[12]: https://github.com/golang/go/milestone/145
[13]: golang/go#40276
[14]: https://tip.golang.org/doc/go1.16#modules
[15]: https://github.com/golang/go/wiki
[16]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
[17]: https://github.com/myitcv/gobin
[18]: https://golang.org/pkg/os/#UserCacheDir
[19]: https://github.com/myitcv/gobin/wiki/FAQ
[2]: https://golang.org/cmd/go#hdr-Compile_and_install_packages_and_dependencies
[20]: https://pkg.go.dev/github.com/svengreb/wand/pkg/cast/gobin#Caster
[21]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#Incantation
[22]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#KindGoModule
[3]: https://golang.org/cmd/go/#hdr-Environment_variables
[4]: https://golang.org/cmd/go/#hdr-Print_Go_environment_information
[5]: https://golang.org/ref/mod#go-mod-file
[6]: https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them
[7]: https://golang.org/ref/mod#mod-commands
[8]: https://golang.org/cmd/go
[9]: golang/go#30515

Closes GH-22
  • Loading branch information
@svengreb
svengreb authored Nov 22, 2020
1 parent 55e8eb4 commit 95c22a0
Show file tree
Hide file tree
Showing 5 changed files with 360 additions and 3 deletions.
6 changes: 6 additions & 0 deletions pkg/cast/error.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,12 @@ import (
)

const (
// ErrCasterCasting indicates that a caster failed to cast.
ErrCasterCasting = wErr.ErrString("failed to cast")

// ErrCasterInvalidOpts indicates invalid caster options.
ErrCasterInvalidOpts = wErr.ErrString("invalid caster options")

// ErrCasterValidation indicates that a caster validation failed.
ErrCasterValidation = wErr.ErrString("caster validation failed")

Expand Down
247 changes: 247 additions & 0 deletions pkg/cast/gobin/gobin.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,247 @@
// Copyright (c) 2019-present Sven Greb <[email protected]>
// This source code is licensed under the MIT license found in the LICENSE file.

// Package gobin provides a caster to install and run Go module executables using the "github.com/myitcv/gobin" module.
// See https://github.com/myitcv/gobin for more details about "gobin".
//
// Go Executable Installation
//
// When installing a Go executable from within a Go module (1) directory using the "go install" command (2), it is
// installed into the Go executable search path that is defined through the "GOBIN" environment variable (3) and can
// also be shown and modified using the "go env" command (4).
// Even though the executable gets installed globally, the "go.mod" file (5) will be updated to include the installed
// packages since this is the default behavior of the "go get" command (6) when running in "module" mode (7).
//
// Next to this problem, the installed executable will also overwrite any executable of the same module/package that was
// installed already, but maybe from a different version. Therefore only one version of a executable can be installed at
// a time which makes it impossible to work on different projects that use the same tool but with different versions.
//
// History and Future
//
// The local installation of executables built from Go modules/packages has always been a somewhat controversial point
// which unfortunately, partly for historical reasons, does not offer an optimal and user-friendly solution up to now.
// The "go" command (8) is a fantastic toolchain that provides many great features one would expect to be provided
// out-of-the-box from a modern and well designed programming language without the requirement to use a third-party
// solution: from compiling code, running unit/integration/benchmark tests, quality and error analysis, debugging
// utilities and many more.
// Unfortunately the way the "go install" command (9) of Go versions less or equal to 1.15 handles the installation of
// an Go module/package executable is still not optimal.
//
// The general problem of tool dependencies is a long-time known issue/weak point of the current Go toolchain and is a
// highly rated change request from the Go community with discussions like https://github.com/golang/go/issues/30515,
// https://github.com/golang/go/issues/25922 and https://github.com/golang/go/issues/27653 to improve this essential
// feature, but they've been around for quite a long time without a solution that works without introducing breaking
// changes and most users and the Go team agree on.
// Luckily, this topic was finally picked up for the next upcoming Go release version 1.16 (9) and
// https://github.com/golang/go/issues/40276 introduces a way to install executables in module mode outside a module.
// The release note preview also already includes details about this change (10) and how installation of executables
// from Go modules will be handled in the future.
//
// The Workaround
//
// Beside the great news and anticipation about an official solution for the problem the usage of a workaround is almost
// inevitable until Go 1.16 is finally released.
//
// The official Go wiki (11) provides a section on "How can I track tool dependencies for a module?" (12) that describes
// a workaround that tracks tool dependencies. It allows to use the Go module logic by using a file like "tools.go" with
// a dedicated "tools" build tag that prevents the included module dependencies to be picked up included for normal
// executable builds. This approach works fine for non-main packages, but CLI tools that are only implemented in the
// "main" package can not be imported in such a file.
//
// In order to tackle this problem, a user from the community created "gobin" (13), an experimental, module-aware
// command to install/run main packages.
// It allows to install or run main-package commands without "polluting" the "go.mod" file by default. It downloads
// modules in version-aware mode into a binary cache path within the systems cache directory (14).
// It prevents problems due to already globally installed executables by placing each version in its own directory.
// The decision to use a cache directory instead of sub-directories within the "GOBIN" path keeps the system clean.
//
// "gobin" is still in an early development state, but has already received a lot of positive feedback and is used in
// many projects. There are also members of the core Go team that have contributed to the project and the chance is high
// that the changes for Go 1.16 were influenced or partially ported from it.
// It is currently the best workaround to...
// 1. prevent the Go toolchain to pick up the "GOMOD" environment variable (15) (see "go env GOMOD" (15)) that is
// initialized automatically with the path to the "go.mod" file in the current working directory.
// 2. install module/package executables globally without "polluting" the "go.mod" file.
// 3. install module/package executables globally without overriding already installed executables of different
// versions.
//
// See gobin's FAQ page (16) in the repository wiki for more details about the project.
//
// The Go Module Caster
//
// To allow to manage the tool dependency problem, this caster uses "gobin" through to prevent the "pollution" of the
// project "go.mod" file and allows to...
// 1. install "gobin" itself into "GOBIN" (`go env GOBIN` (15)).
// 2. cast any spell incantation (17) of kind "KindGoModule" (18) by installing the executable globally into the
// dedicated "gobin" cache.
//
// References
//
// (1) https://golang.org/ref/mod
// (2) https://golang.org/cmd/go#hdr-Compile_and_install_packages_and_dependencies
// (3) https://golang.org/cmd/go/#hdr-Environment_variables
// (4) https://golang.org/cmd/go/#hdr-Print_Go_environment_information
// (5) https://golang.org/ref/mod#go-mod-file
// (6) https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them
// (7) https://golang.org/ref/mod#mod-commands
// (8) https://golang.org/cmd/go
// (9) https://github.com/golang/go/milestone/145
// (10) https://tip.golang.org/doc/go1.16#modules
// (11) https://github.com/golang/go/wiki
// (12) https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
// (13) https://github.com/myitcv/gobin
// (14) https://golang.org/pkg/os/#UserCacheDir
// (15) https://golang.org/cmd/go/#hdr-Print_Go_environment_information
// (16) https://github.com/myitcv/gobin/wiki/FAQ
// (17) https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#Incantation
// (18) https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#KindGoModule
package gobin

import (
"fmt"
"os"
"os/exec"
"path/filepath"

"github.com/magefile/mage/sh"
glFS "github.com/svengreb/golib/pkg/io/fs"

osSupport "github.com/svengreb/wand/internal/support/os"
"github.com/svengreb/wand/pkg/cast"
castGoToolchain "github.com/svengreb/wand/pkg/cast/golang/toolchain"
"github.com/svengreb/wand/pkg/project"
"github.com/svengreb/wand/pkg/spell"
)

// Caster is a "github.com/myitcv/gobin" module caster.
type Caster struct {
opts *Options
}

// GoModule returns the the partial Go module information for the "github.com/myitcv/gobin" module.
func (c *Caster) GoModule() project.GoModuleID {
return *c.opts.goModule
}

// GetExec returns the path to the installed executable of the "github.com/myitcv/gobin" module.
func (c *Caster) GetExec() string {
return c.opts.Exec
}

// Cast casts a spell incantation.
// It returns an error of type *cast.ErrCast when the spell is not a spell.KindGoModule and any other error that occurs
// during the command execution.
func (c *Caster) Cast(si spell.Incantation) error {
if si.Kind() != spell.KindGoModule {
return &cast.ErrCast{
Err: fmt.Errorf("%q", si.Kind()),
Kind: cast.ErrCasterSpellIncantationKindUnsupported,
}
}

s, ok := si.(spell.GoModule)
if !ok {
return &cast.ErrCast{
Err: fmt.Errorf("expected %q but got %q", s.Kind(), si.Kind()),
Kind: cast.ErrCasterSpellIncantationKindUnsupported,
}
}

args := append([]string{"-run", s.GoModuleID().String()}, si.Formula()...)
for k, v := range s.Env() {
c.opts.Env[k] = v
}

return sh.RunWithV(c.opts.Env, c.opts.Exec, args...)
}

// Install installs the executable of the "github.com/myitcv/gobin" module.
// It does not "pollute" the "go.mod" file of the project the installation outside of the project root directory but
// using a the systems temporary directory instead.
// See the package documentation for details: https://pkg.go.dev/github.com/svengreb/wand/pkg/cast/gobin
func (c *Caster) Install(goCaster *castGoToolchain.Caster) error {
goToolchainExec := goCaster.GetExec()
cmd := exec.Command(goToolchainExec, "get", "-v", c.opts.goModule.String())
cmd.Dir = os.TempDir()
cmd.Env = os.Environ()

// Explicitly enable "module" mode to install a pinned "github.com/myitcv/gobin" module version.
c.opts.Env[castGoToolchain.DefaultEnvVarGO111MODULE] = "on"
cmd.Env = osSupport.EnvMapToSlice(c.opts.Env)

if err := cmd.Run(); err != nil {
return &cast.ErrCast{
Err: err,
Kind: cast.ErrCasterCasting,
}
}
return nil
}

// Handles returns the supported spell.Kind.
func (c *Caster) Handles() spell.Kind {
return spell.KindGoModule
}

// Validate validates the "github.com/myitcv/gobin" module caster.
// It returns an error of type *cast.ErrCast when the binary executable does not exists at the configured path and when
// it is also not available in the executable search paths of the current environment.
func (c *Caster) Validate() error {
// Check if the "gobin" executable exists at the configured path,...
execExits, fsErr := glFS.RegularFileExists(c.opts.Exec)
if fsErr != nil {
return &cast.ErrCast{
Err: fmt.Errorf("caster %q: %w", CasterName, fsErr),
Kind: cast.ErrCasterValidation,
}
}

// ...otherwise try to look up the system-wide executable search paths of the current environment...
if !execExits {
execPath, pathErr := exec.LookPath(c.opts.Exec)

// ...and the local Go binary installation path.
if pathErr != nil {
var execDirGoEnv string

if execDirGoEnv = os.Getenv(castGoToolchain.DefaultEnvVarGOBIN); execDirGoEnv == "" {
if execDirGoEnv = os.Getenv(castGoToolchain.DefaultEnvVarGOPATH); execDirGoEnv != "" {
execDirGoEnv = filepath.Join(execDirGoEnv, castGoToolchain.DefaultGOBINSubDirName)
}
}

execPath = filepath.Join(execDirGoEnv, c.opts.Exec)
execExits, fsErr = glFS.RegularFileExists(execPath)
if fsErr != nil {
return &cast.ErrCast{
Err: fmt.Errorf("caster %q: %w", CasterName, fsErr),
Kind: cast.ErrCasterValidation,
}
}

if !execExits {
return &cast.ErrCast{
Err: fmt.Errorf("caster %q: %w", CasterName, fsErr),
Kind: cast.ErrCasterValidation,
}
}
}

c.opts.Exec = execPath
}

return nil
}

// NewCaster creates a new "github.com/myitcv/gobin" module caster.
func NewCaster(opts ...Option) (*Caster, error) {
opt, optErr := newOptions(opts...)
if optErr != nil {
return nil, &cast.ErrCast{
Err: optErr,
Kind: cast.ErrCasterInvalidOpts,
}
}

return &Caster{opts: opt}, nil
}
95 changes: 95 additions & 0 deletions pkg/cast/gobin/options.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
// Copyright (c) 2019-present Sven Greb <[email protected]>
// This source code is licensed under the MIT license found in the LICENSE file.

package gobin

import (
"fmt"

"github.com/Masterminds/semver/v3"

"github.com/svengreb/wand/pkg/cast"
"github.com/svengreb/wand/pkg/project"
)

const (
// CasterName is the name of the Go toolchain command caster.
CasterName = "gobin"

// DefaultExec is the default name of the "github.com/myitcv/gobin" module executable.
DefaultExec = "gobin"

// DefaultGoModulePath is the default "gobin" module import path.
DefaultGoModulePath = "github.com/myitcv/gobin"

// DefaultGoModuleVersion is the default "gobin" module version.
DefaultGoModuleVersion = "v0.0.14"
)

// Options stores "github.com/myitcv/gobin" module caster options.
type Options struct {
// Env are caster specific additional environment variables.
Env map[string]string

// Exec ist the name or path of the "gobin" module executable.
Exec string

goModule *project.GoModuleID
}

// Option is a "github.com/myitcv/gobin" module caster option.
type Option func(*Options)

// WithExec sets the name or path to the "github.com/myitcv/gobin" module executable.
// Defaults to DefaultExec.
func WithExec(nameOrPath string) Option {
return func(o *Options) {
if nameOrPath != "" {
o.Exec = nameOrPath
}
}
}

// WithModulePath sets the "gobin" module import path.
// Defaults to DefaultGoModulePath.
func WithModulePath(path string) Option {
return func(o *Options) {
if path != "" {
o.goModule.Path = path
}
}
}

// WithModuleVersion sets the "gobin" module version.
// Uses DefaultGoModuleVersion as default version value.
func WithModuleVersion(version *semver.Version) Option {
return func(o *Options) {
if version != nil {
o.goModule.Version = version
}
}
}

// newOptions creates new "github.com/myitcv/gobin" module caster options.
func newOptions(opts ...Option) (*Options, error) {
version, versionErr := semver.NewVersion(DefaultGoModuleVersion)
if versionErr != nil {
return nil, &cast.ErrCast{
Err: fmt.Errorf("parsing default module version %q: %w", DefaultGoModulePath, versionErr),
Kind: cast.ErrCasterInvalidOpts,
}
}
opt := &Options{
Env: make(map[string]string),
Exec: DefaultExec,
goModule: &project.GoModuleID{
Path: DefaultGoModulePath,
Version: version,
},
}
for _, o := range opts {
o(opt)
}

return opt, nil
}
11 changes: 9 additions & 2 deletions pkg/cast/golang/toolchain/options.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,20 @@ const (

// DefaultEnvVarGOPATH is the default environment variable name for the Go path.
DefaultEnvVarGOPATH = "GOPATH"

// DefaultGOBINSubDirName is the default name of the sub-directory for the Go executables within DefaultEnvVarGOBIN.
DefaultGOBINSubDirName = "bin"
)

// DefaultExec is the default path to the Go executable.
var DefaultExec = mg.GoCmd()

// Options stores Go toolchain command caster options.
type Options struct {
Env map[string]string
// Env are caster specific environment variables.
Env map[string]string

// Exec is the name or path of the Go toolchain command executable.
Exec string
}

Expand All @@ -43,7 +49,8 @@ func WithEnv(env map[string]string) Option {
}
}

// WithExec sets the path to the Go executable.
// WithExec sets the name or path to the Go executable.
// Defaults to DefaultExec.
func WithExec(nameOrPath string) Option {
return func(o *Options) {
o.Exec = nameOrPath
Expand Down
Loading

0 comments on commit 95c22a0

Please sign in to comment.