cmd/go: allow go.mod.local to contain replace/exclude lines

[maeglindeveloper]

Hi everyone,
I'm actually using a go.mod file for dependencies in my go project.

I saw that there is a way to use local packages instead of online repo, using the replace() features, which is really cool when developping multiple packages at the same time.

I was wondering if there is a way split in different files the require dependencies and the replace one. Something like that

go_replace.mod => define the replace part.
go.mod => define the module and the required dep, including the go_replace.mod.

the go_replace.mod will not be pushed on git, and each developer will create it if needed.

What version of Go are you using (go version)?

go version go1.11beta2 windows/amd64

Thanks a lot!

@rsc

[mvdan]

Could you clarify what is the purpose of separating the data into two files? So that developers don't commit temporary replace directives by mistake? I'd imagine that there could be tooling to help with that. For example, a tool to add and undo temporary directives, or having CI fail on unexpected replace directives.

I'm also worried about adding more special Go module files, even if they are in theory not to be committed into git.

[flibustenet]

It's dangerous as you can commit files that use something else than what said in go.mod.
With replace in go.mod you see immediately (just before commit) that you will commit with a dependency fork.

[bcmills]

when [developing] multiple packages at the same time

We certainly do need to address that use-case somehow, but splitting up the module definitions doesn't seem like an appropriate solution: ideally I don't want individual developers to have to edit the module definition to do that.

[cosban]

I fall into the use case of "building multiple packages concurrently" as well.
I need to be able to, without being forced to commit code that is not fully vetted, build and test our modifications.
There are multiple solutions to this problem, some more elegant than others.

• Having a release modifier or command (cmd/go: add 'go release' #26420) that ignores local copies would be a simple (from the user perspective) solution and would not require additional modifications to the go.mod files.
• The solution proposed in this issue could also work. It could even be combined with the first proposal. @bcmills, when you say you "don't want individual developers to have to edit the module definition to do that", are you referring to them manualy modifying go.mod files? If not, could you please clarify your position a little more so I may understand it better? This could be resolved by allowing the developer to specify that the library is one which will be modified locally as well using something like go get -u <package@version> -local <relative/path/to/package>
• Allowing users to install local, uncommitted versions could also work for this, but probably isn't the most elegant solution. This is, users could go install their current copy of the package they are developing, and the module system would store it in some sort of development version folder. In this method, version v0.0.0-20180526204631-c40f734b202a could be the latest commit, but v0.0.0-20180526204631-c40f734b202a-devel could be the latest local copy. If the local copy is not found, the module system would be able to revert back to the last commit when using the go get command.

I believe that the first option is the best, but I'm interested in reading what direction you guys are learning toward in the interim.

[wsc1]

Hi all,

I'm not sure that my approach will help others, but in case it does, (taken from go-nuts)

There is a bare bones workaround without local go.mod nor replacements.

Suppose you need to work on module A and module B simultaneously (hopefully you set things up to avoid this, but hey, it's inevitable)

If you only need to edit one package at a time in at least one of A and B, then you don't need
to muck with local go.mod or versions tags or replacements or anything.

Let B be the module which has only one package to edit, and p the package.

Copy p somewhere in A, rename import path to p using existing go imports/ide,
edit both, build, test, etc. A's go.mod will be a little crazy and out of whack.
So what. Copy p back to where it belongs in B, re-run go imports, build/test or go mod tidy
Now commit as you like the two.

In terms of commands and tools, it's very simple: no tools, no replacements, no go.mod.local, no replacements which shouldn't be pushed/commited, no workflow restrictions. Simple "cp -r" and use with editor that understands go imports works fine.

Now, thanks to modules, if your edits correspond to releases, it doesn't even matter in what order
you commit/push the changes after the two modules have been edited.

Yes, it's still a PITA. For me it's much less painful than thinking about workflows and replacements.
But for me it also doesn't happen a lot.

That said, I do think it would be nice to have the go command do for a set of modules, say go build ./...
with ./ containing the modules in question. that would allow updates of all version in one shot for example.

[ohir]

bcmills> "ideally I don't want individual developers to have to edit the module definition to do that."

Pondering on how to "merge" go.mod with the hypothetical go.mod.local I came to the point where only simplest solution stayed viable: add a -local knob. Then, if -local flag is present, tools will use go.mod.local instead of go.mod.

I know that knobs breeding is unpopular among core team but consider it, please.

[benhalstead]

go.mod.local or similar would also help our use case. We have developers working on service 'A' which relies on library 'B'.

Our developers will make changes to both the service and the library and so have a replace in the service's go.mod file to use their local version of the library code instead.

The problem is that each developer will have checked the library out onto their filesystem with a different path so each of them need a different path in the 'replace' statement, meaning version control of go.mod is messy.

[zshbleaker]

Local dependency is useful when importing from a self-hosted system which does not support go-get(e.g. GitHub Enterprise).

[bradleyjames]

+1 to go.mod.local or just a separate file based approach. It would provide a clean logical separation and file based will be simple to ignore via .gitignore. Also overriding the mod file location (aka opt-in) during development would provide a sanity check in mitigating the risk of pushing to production.

[bradleyjames]

Local dependency is useful when importing from a self-hosted system which does not support go-get(e.g. GitHub Enterprise).

We’re using GitHub Enterprise which works for us. What specific issues are you having?

[pwaller]

I was pointed here by @bcmills on slack. Thought it was worth mentioning my use case since I can't see it obviously stated here.

I want to be able to do a git bisect with a replace directive in effect. For this to be effective, I want to avoid having a dirty working tree, so editing the go.mod isn't great for this.

I have found myself a few times wanting a replace directive without contaminating the git status output with changes, for example whilst just doing some development whilst working on multiple modules simultaneously.

[SamWhited]

I was pointed here by @bcmills on slack. Thought it was worth mentioning my use case since I can't see it obviously stated here.

Not everyone can see that link; if it's important, please copy any context needed into this discussion.

[bcmills]

@SamWhited, the discussion on Slack was literally just a link to this issue.
(I mostly treat Slack as a way to quickly route people to the relevant public issues.)

[pipe01]

An implementation I think could come in handy would be to look up on every parent directory for a replace.mod file, which contains replace and perhaps other directives.

[niemeyer]

With GO111MODULE defaulting to on by default in 1.16, it would be nice to address this issue somehow.

For clarity, the key problem problem most developers seem to be observing is not the lack of an additional mod file, but the fact there's no way to experiment with changes in local dependencies (for debugging, for example) without hacking on files that need to be skipped on every commit. An additional overriding file that could be ignored would be one way to address it, but it has the disadvantage of being silent about it. Ideally the final solution will make it visible that there are local overrides active during operation.

While this issue isn't sorted out upstream, developers may be interested in the --assume-unchanged flag of git update-index:

--[no-]assume-unchanged
    When this flag is specified, the object names recorded for the paths are
    not updated. Instead, this option sets/unsets the "assume unchanged" bit
    for the paths. When the "assume unchanged" bit is on, the user promises not
    to change the file and allows Git to assume that the working tree file
    matches what is recorded in the index. If you want to change the working
    tree file, you need to unset the bit to tell Git. This is sometimes helpful
    when working with a big project on a filesystem that has very slow lstat(2)
    system call (e.g. cifs).

    Git will fail (gracefully) in case it needs to modify this file in the
    index e.g. when merging in a commit; thus, in case the assumed-untracked
    file is changed upstream, you will need to handle the situation manually.

Some git aliases can make this a bit more convenient to use:

ghost = update-index --assume-unchanged
unghost = update-index --no-assume-unchanged
ghosted = ! git ls-files -v | grep '^[a-z]'
noghost = ! git ls-files -v | awk '/^[a-z]/ {system(\"git update-index --no-assume-unchanged \"$2)}'

This also has the disadvantage of being silent during operation. That's relatively easy to fix by automatically decorating commit messages to remind that go.mod is dirty.

[sabutterworth]

I recently proposed #44660 with exactly the same issue in mind - I frequently need to use go.mod directives to fiddle with non-VCS packages (or newer code in local than in VCS) but NOT have them in the go.mod long-term.

Someone pointed me at this proposal which has a lot of discussion on it so I decided to close my proposal and add my suggestion here - to reduce the amount of 'duplicate' issues.

Suggestion:
When go is run without -modfile it looks for a file named go.mod.dev in the folder path resolved by GOMOD (or if -modfile is specified look for {thatmodfile}.dev at the same path). If found (and the module is the same) it uses a merge strategy to combine that with the contents of go.mod to resolve the correct directives to use - with directives from the .dev file having a higher precedence over the things in the go.mod file when conflicts are encountered.

[PeterFeicht]

@sabutterworth so you're suggesting that even when I explicitly specify a .mod file by using -modfile, then the go command would still use another file in addition to that? That doesn't sound very sensible.

[ohir]

@sabutterworth

I frequently need to use go.mod directives to fiddle with non-VCS packages (or newer code in local than in VCS) but NOT have them in the go.mod long-term.

This is common in bootstrap and debug phase.

When go is run without -modfile it looks for a file named go.mod.dev in the folder path resolved by GOMOD (or if -modfile is specified look for {thatmodfile}.dev at the same path). If found (and the module is the same) it uses a merge strategy to combine that with the contents of go.mod to resolve the correct directives to use - with directives from the .dev file having a higher precedence over the things in the go.mod file when conflicts are encountered.

For experiments go.mod should not be consulted at all. Just we need to tell compiler where our experimental code is. Then - after fiddling there - we may commit and publish new or fixed interdependent modules.

[complyue]

FYI, in Haskell ecosystem, library authors tend to like Cabal, while application authors tend to like Stack; There is cabal.project.local supported, but not with similar mechanism upon stack.yaml , see: https://www.reddit.com/r/haskell/comments/hvybba/is_there_a_stack_equivalent_of_cabalprojectlocal/

Seems Go doesn't suffer from a split like cabal/stack wrt build tooling, but I kinda feel a similarity that GOPATH leans toward library authors (tend to develop multiple inter-related or even lock-step projects concurrently), while vending/go.mod leans toward application authors (focus on a single package module for most of the time).

So I think go.mod.local seemingly will be favored by lib authors, but can harm reproducibility that app authors value a lot. I observed many design decisions stack made to emphasis reproducibility, not to support stack.local.yaml might be one among others.

Personally I like #44347 GOTINKER over go.mod.local here, that seems to have the potential to make both lib and app authors happy.

[bcmills]

I kinda feel a similarity that GOPATH leans toward library authors (tend to develop multiple inter-related or even lock-step projects concurrently), while vending/go.mod leans toward application authors (focus on a single package module for most of the time).

My impression is the opposite: I think most library authors are using modules at this point, because modules make it easier for their downstream consumers to resolve and manage transitive dependencies, and libraries already have unique import paths (generally beginning with the domain name hosting the library).

I think the majority of GOPATH users at this point are developers of private or self-contained applications. Private apps may be more difficult to migrate to modules because the existing package paths might not begin with a domain name compatible with go get.

[complyue]

Sure that's the case, private apps are not on my mind previously, I agree they'll have to mass refactoring their import paths for the migration.

For typical library authors, I assume one would split functionality toward finer grained lib projects, but he must have some motivating use cases to be prototyped and tested with all those libs, or further plus one or more apps involved.

Suppose each lib and each app goes as a separate go.mod, or this isn't the case?

When you are planning several libs altogether, or to add major, inter-related features affecting multiple libs, they need to be prototyped & tested with WIP branches. Temporarily placing local replaces in one go.mod is not a big deal, but how about maintaining ~10 intermediate go.mod files at the same time? For myself, I would feel painful in such workflows, even if CI can be configured to automate removal of those local replaces, it's very unwieldy in preparing a staging area for effective feature branches of multiple libs. And I can't see automation in tooling can bring a rival experience as smooth as the traditional GOPATH provides.

[gilgad13]

Placing this issue in context with Russ's well-written plea here: #37755 (comment)

Yes, go modules are obviously superior to GOPATH. They enable reproducible builds. They allow different components to depend on different versions of the same third-party repository. They solve all the security problems recently highlighted with pip, in triplicate. However, at engineering companies of a certain size, you’ve already had to solve these problems. And, furthermore, you’ve had to solve them for a variety of languages, so your solution is cross-language. Existing processes and tools are defined in terms of this meta-builder, so institutional inertia requires compatibility.  It also allows dependencies on specific go and protoc versions.  At $dayjob, we have a custom Make-based framework, for Google I’m guessing Bazel / Blaze fills that role, and for a project like Debian its the control files and auto builder network. If you already have such a system, integrating it with GOPATH is easy, just ask that system to form a valid GOPATH with the correct versions in place everywhere (as Debian does). With go modules, it's much, much harder. The official answer as I understand it is to call directly in to go tool compile and go tool link.  However, bypassing the high-level go build command means you may need to reimplement editor and tool integrations, which is a very high bar.  Even at a conceptual level its hard because now both the go compiler and the overarching tool want to control how dependencies are fulfilled. I haven’t seen a clear story on how to integrate go modules into such a system.

I think some participants in this thread (myself included) are attempting to recreate their previous GOPATH workflows using the replace directive.  For this use case, the -modfile flag (perhaps encoded in GOFLAGS) is nice, but is still difficult to work with because you need to create an override file for each dependency you pull in to the "workspace", and regardless of whether you pass the -modfile argument as a relative or absolute path it is difficult to select the correct modfile based on the the directory a command is being run in.  I believe this is why there is interest in having a single file that defines a "workspace", which overrides the module discovery process for all checked-out modules within it.

[complyue]

I just realized that go.mod corresponds to what's called "package" elsewhere, I feel we need a construct bigger than the "Go module" in Go tooling's concept space, to properly model the source structure of Go software. So I'd add that I don't think go.mod.local is the solution, for it be at the same level as go.mod.

Go uses the term "package" for directory of source files, while in more general software engineering context, "package" as in "Package Manager", "Pip - the Package Installer for Python" and etc. refers to something bigger than "module". So now Go has the relative size of "package" and "module" flipped, this will likely to confuse even exclusive Go developers if unaware, for they are likely, e.g. to work with "apt - Advanced Package Tool" day-to-day in using their servers, workstations or laptops.

We need to start talking about the concept for the thing bigger than "package" or "Go module" in tooling related issues, currently the data model of Go tooling wrt source project structure lacks a layer of project workspace, so maybe the term "project" or "workspace" or my personal idea "go.farm" will do, just don't stop at go.mod as the biggest unit of Go software.

Update: cross link #27542 (comment)

[thepudds]

Hi @complyue @gilgad13, You might be interested in #26640 (comment), which is a sketch from a while ago of how a go.workspace or go.space could work to allow working multiple modules simultaneously without needing to do anything manually regarding replace, and would allow people to more easily organize their code around projects. Of the alternatives sketched there, I personally prefer option 2 there.

One question that came up in a prior discussion of that sketch is how would a release happen, including to check if a given module still builds and passes tests using the published versions of the modules in the local workspace. One approach would be to use mv as an extremely minimalistic “API“ to temporarily rename the go.workspace file or move modules temporarily out of the workspace. That has the benefit of being minimalistic, tangible, and explicit, but might be too minimalistic. Alternatively, a new -mod=foo mode might be able to say “use published versions, not workspace versions”.

[complyue]

@thepudds I confess I missed your posts (as well as many others') by not thoroughly reading all the discussions, a shame but I don't have too much time/energy available on Go stuff as time being. And I'm glad to know you already sketched the idea and many have discussed about it.

With respect to the release concern, I'm no expert at it, but AFAICT, similar concerns had driven a gang of Haskellers to have created Stackage, it recruit curators to work on nightly snapshots of the central registry/repository of Haskell packages (https://hackage.haskell.org), making sure all packages are vetted (compiles, passing test suites, etc.) then regularly release such snapshots as LTS Haskell. Stackage Curators are great people and smart CI tools making all that happen.

Back to the release concern we talk about here, I do think it's okay if all modules in a local go.workspace are engineered in a lock-step fashion, but for wider collaboration among upstream/downstream projects (inevitable for open source development), I don't think that solvable by a simple tool. More intensive processes/workflows with dedication of human forces might be inevitable, and https://www.snoyman.com/blog/2018/11/why-i-believe-stackage-succeeded may provide an illustration for what it will look like after done right.

[slatermorgan]

I too have come across this problem, it would be lovely to have go.space for local development!

For the time being, couldn't a solution be some sort of bash script which could append the replace directives from another file to the go.mod before building the project in the local development environment?

[ghostsquad]

I hope this isn't considered off topic, but isn't library development supposed to make as few assumptions about how it's being used as possible, and allow as much flexibility as possible? I haven't been in a situation where I would want to do a replace to a local version of a library for app development.

If my app has a feature that seems to fit with the library, usually that starts as just native app code, and migrates to that of library code. A migration though is copy/paste/refactor for flexibility as needed, and then publish (with tag).

Following the completion of the lib work, I could rip out the functionality from the app, and point it to the lib.

I guess I'd like to understand why folks are doing this kind of "replace" so often that there's this active proposal. It seems that such a workflow is because of tight coupling, and may be a design anti-pattern, and thus should not be encouraged?

[rwxrob]

I've reach out to the Kompose project, which is currently "stuck" on 1.13 because of these questions. I certainly don't know the answer. It seems people everywhere are writing off go install completely and just turning to other methods of release, which I feel is regrettable.

[complyue]

If my app has a feature that seems to fit with the library, usually that starts as just native app code, and migrates to that of library code. A migration though is copy/paste/refactor for flexibility as needed, and then publish (with tag).

After you've published the lib, when someone find the needs to improve/fix-bugs, how can they work on it without "locally-replace" the published version of your code?

[ghostsquad]

If my app has a feature that seems to fit with the library, usually that starts as just native app code, and migrates to that of library code. A migration though is copy/paste/refactor for flexibility as needed, and then publish (with tag).

After you've published the lib, when someone find the needs to improve/fix-bugs, how can they work on it without "locally-replace" the published version of your code?

Fix bugs in the library? Pull down the library, write a test that demonstrates the bug, update the implementation, push, tag.

[anacrolix]

How is this now different to go.work? Does that solve this issue?

[complyue]

Fix bugs in the library? Pull down the library, write a test that demonstrates the bug, update the implementation, push, tag.

So every verification test would need a commit (even tag every one?) tripping around the repository, no?

[thepudds]

How is this now different to go.work? Does that solve this issue?

Personally, I think go.work (#45713) is a better solution to the problems go.mod.local is trying to solve.

FWIW, I wrote a comment above early on here in
#26640 (comment) outlining why I thought go.mod.local might be a burdensome solution and why something like a go.workspace file would be better.

Also, the write up at #45713 includes a short comparison to go.mod.local:

The issue of maintaining user-specific replaces in go.mod files was brought up in #26640. It proposes an alternative go.mod.local file so that local changes to the go.mod file could be made adding replaces without needing to risk local changes being committed in go.mod itself. The go.work file provides users a place to put many of the local changes that would be put in the proposed go.mod.local file.

[seankhliao]

I believe we can close this now that workspaces should be widely available