Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

Migration from 'go dep' to 'go mod' #1331

Closed
umarcor opened this issue Mar 6, 2019 · 10 comments
Closed

Migration from 'go dep' to 'go mod' #1331

umarcor opened this issue Mar 6, 2019 · 10 comments

Comments

@umarcor
Copy link

umarcor commented Mar 6, 2019

Coming from #1152:

[@umarcor]
@Pennyzct, @jodh-intel just because of curiosity, why are you using dep instead of go mod?

[@grahamwhaley]
@umarcor - at this point, it may be mostly due to historical reasons - the kata, and forerunner, code bases have been around a few years now... I think go mod came in with golang 1.11 ?, and we still have our baseline golang supported version as 1.10.4 :-)

Now, if you'd like to propose a move to go mod, and submit a PR ;-) But, first probably best if it was discussed. /cc @sboeuf

[@jodh-intel]
Yes - at the time we introduced it, dep was the best tool available.

[@sboeuf]
@umarcor @grahamwhaley @jodh-intel I recently discussed this with @mcastelino and I think it is time for us to move to go modules. We need to open issues on every Kata repositories for this. And then let's start submit PRs. @umarcor is it something you're interested in helping with?

Let's continue here, to avoid getting @Pennyzct's PR dirtier.

Description of problem

This is neither a problem nor a requirement, because dep is not deprecated, yet. Certainly, it is still actively maintained (golang/dep#2052). Nonetheless, as go mod will eventually cease to be experimental, it is worth considering the implications.

Expected result

  • Allow developers to use either dep or go mod, at least until the baseline supported version in kata is above 1.11. Consider removing dep support afterwards.
  • Remove vendor from the repository.

Actual result

  • Only dep is supported.
  • The vendor repository is checked in the repo and it must be explicitly updated. See #1152, community#88 and community#89.

So, regarding

[@grahamwhaley]
Now, if you'd like to propose a move to go mod, and submit a PR ;-) But, first probably best if it was discussed.

[@sboeuf]
@umarcor is it something you're interested in helping with?

I said I was asking because of curiosity since I don't really understand how go mod is expected to work. I see that it always works in clean environments (such as CI jobs), but when used in a machine with other golang packages, it freaks out easily. Precisely, the behaviour that I have seen is the following:

  • If a dependency does not exist locally, the version specified in go.mod is retrieved.
  • If the dependency exists locally, it may work or it may complain because the version does not match.
    • This is sometimes desirable, because the local dependency is in a valid dirty state, i.e. some patches have been applied.
    • In other contexts, the checked out or modified local state corresponds to any other project, and it is not compatible, so it breaks the build.

Moreover, go mod allows to generate a vendor directory, but it is not honoured. I.e., I would expect it to search for dependencies in common paths (/go, ~/go, GOPATH) and save additional clones in the vendor only when there is some conflict. Then, when go build or go run is executed, it would use the versions in vendor and, if not present, it would search in the common paths.

What is happening instead is that you can copy absolutely all the dependencies to vendor and then tell go tools to use it exclusively. This is quite nonesense, because it effectively treats each repository as a completely independent project and the same dependencies are duplicated unnecessarily. At this point, I don't know if there is any benefit over using git submodules, provided that all the go dependencies are available as git repos.

Therefore, an 'easy' workaround is to just use a golang:alpine container and build the binaries with CGO_ENABLED=0 go build -a. This allows to execute each build in a clean environment, so there is no conflict with other projects in the host. The main disadvantage of this approach is that all the dependencies are downloaded each time a container is started, unless vendor is used exclusively.

Ideally, checking vendor into the repo should not be required neither with dep nor with go mod. The vendor directory should be a local temporal resource that can be 'generated' unequivocally with go.mod, Gopkg.lock and/or Gopkg.toml.

Last, the behaviour of go mod is different if the repository is inside or outside of the GOTPATH.


Anyway, the 'clean' approach to migrate from dep to go mod should be as 'simple' as:

docker run --rm -it golang:alpine sh -c "$(cat <<-EOF
# Get 'regular' dependencies to build the runtime
apk update
apk add git
go get -u github.com/golang/dep/cmd/dep
go get -u github.com/kata-containers/runtime
cd \$GOPATH/src/github.com/kata-containers/runtime
dep ensure

# Remove 'dep' related sources
git rm -rf Gopkg.lock Gopkg.toml vendor/

# Move the repo outside of GOPATH
cd ..
mv runtime ~/
cd ~/runtime

# Initialize go modules
go mod init
git add go.mod

# Profit
git config --local user.name "dep2mod"
git config --local user.email "dep2mod@script.example"
git status
git commit -m 'migrate from dep to go mod'
git push
EOF
)"

NOTE: optionally, replace git push with sh to execute further commands before pushing the result.

NOTE: on MSYS2 (Win10), prepend the first line with winpty.


Summarizing, @grahamwhaley, @sboeuf, I am interested on the discusion, on hearing your thoughs, and on seen how a project like this handles this specific issue (provided that dependency management in golang is a flaw in the language/ecosystem). Nevertheless, I can sure open PRs applying the script above to the repos you deem appropriate.

@mcastelino
Copy link
Contributor

Therefore, an 'easy' workaround is to just use a golang:alpine container and build the binaries with CGO_ENABLED=0 go build -a. This allows to execute each build in a clean environment, so there is no conflict with other projects in the host. The main disadvantage of this approach is that all the dependencies are downloaded each time a container is started, unless vendor is used exclusively.

@umarcor you can work around this issue for quicker rebuilds using go mod download
So unless the dependencies have actually changed the download should not happen

COPY go.mod go.sum ./
RUN go mod download
COPY . ./
RUN CGO_ENABLED=0 go build 

A simple example
https://github.com/mcastelino/testapi/blob/master/Dockerfile

A more comprehensive post https://medium.com/@petomalina/using-go-mod-download-to-speed-up-golang-docker-builds-707591336888

@sboeuf
Copy link

sboeuf commented Mar 6, 2019

@umarcor thanks for starting this thread.
I haven't looked at go modules very closely yet, and I don't have the capacity to dig into too many details right now. But I think the first step is to prototype if this could be achieved easily for our Kata Containers repos.
And the second step would be to decide when to move forward with the switch from dep to go mod.

@umarcor @mcastelino go mod still considered experimental, do we know when it will be considered stable?

@sboeuf
Copy link

sboeuf commented Mar 6, 2019

Nevemind, I found go modules are targeted to be fully stable in Go 1.13, targeted to be released August 2019.

@umarcor
Copy link
Author

umarcor commented Mar 6, 2019

@mcastelino, sure. But, i) that's somehow equivalent to having vendor checked in the git repo; and ii) there is no difference compared to dep. Assuming a new developer needs to download both the repo and the image:

  1. Current solution:
    • Code base is retrieved from the repo.
    • Build dependencies/tools are retrieved in a docker image.
    • Golang dependencies are retrieved as a part of the repo.
  2. Using go mod download in a Dockerfile:
    • Code base is retrieved from the repo.
    • Build dependencies/tools are retrieved in a docker image.
    • Golang dependencies are retrieved in a docker image.

Neither of those options guarantee that the developer is using the latest compatible versions of the dependencies. It does guarantee that compatible versions are used, which were the latest when 1. the last commit was pushed to the repo; or 2. the docker image was built.

The developer needs to execute go mod or dep ensureanyway, in order to update some probably slighly outdated deps. Hence, he/she can just execute them once, and share them inside the container. This is desirable, because further updates of the docker image are both significantly more lightweight and less frequent.

In CI environments that support cache (such as Travis), the startup time is similar between downloading the dependencies in the docker images, doing so in the git repo, or retrieving the cache.


  1. Ideal solution:
    • Code base is retrieved from the repo.
    • Build dependencies/tools are retrieved in a docker image.
    • Golang dependencies are not retrieved. The developer can choose:
      • Build an additional docker image and put the dependencies inside. Same as 2.
      • Use the vendor folder:
        • go mod vendor on the host, bind it in a container and go mod vendor inside.
        • dep ensure on the host, bind it in a container and dep ensure inside.

With option 3, kata-containers would not need to host (probably multiple versions) of the dependencies. Yet, the build environment would be reproducible. Precisely, vendor would contain those dependencies and versions which are not available neither on the host nor inside the container. The point is how/when does go mod support a setup as the following:

  • In the host (GOPATH):
    • Dependency A: version A.1
    • Dependency B: version B.1
  • In the image/container (/go):
    • Dependency A: version A.2
  • In subdir vendor/:
    • Dependency A: version A.2-dirty
    • Dependency B: version B.1

Now, I want to execute four builds of an app that depends on A.2 and B.1:

  • Build on the host.
    • The version of A.2-dirty matches, but it has some local uncommited changes.
      • Is it used anyway?
      • Is A.2 retrieved to GOPATH?
      • Is A.2 retrieved to vendor/?
    • Either GOPATH/B.1 or vendor/B.1 are valid. Which is used?
  • Build on the container.
    • go/A.2 and vendor/B.1 can be used. Is it done? Or are any of them downloaded again?

When building on the host, all of these might be valid, each in it's own use case:

  • Get GOPATH/A.2 and use GOPATH/B.1.
  • Get vendor/A.2 and use GOPATH/B.1.
  • Use vendor/A.2-dirty and use GOPATH/B.1.
  • Get GOPATH/A.2 and use vendor/B.1.
  • Get vendor/A.2 and use vendor/B.1.
  • Use vendor/A.2-dirty and use vendor/B.1.

How can the user define which of the six approaches above is used? I'm not sure about this being clearly defined. Mostly because 'compatibility' of versions which are already available in the environment is not checked according to a specific commit SHA. I.e., if semantic versioning is supported, multiple commits on top of the initially defined reference can be used. This makes it difficult to decide in the example above if GOPATH/B.1 and /go/B.1 are exactly the same commit, or if each of them is checked out in a different commit/tag of the same major semver.


[@sboeuf]
But I think the first step is to prototype if this could be achieved easily for our Kata Containers repos.
And the second step would be to decide when to move forward with the switch from dep to go mod.

I think that the first step is to evaluate whether go mod provides any advantage over dep ATM. Otherwise, I would suggest revisting this issue in a few months (3? 6? 9?). Even if it is considered stable in august, complex projects might hit corner cases. See e.g. gobuffalo/buffalo#1545.

Meanwhile, it is possible to keep a branch just a single commit ahead of master (see script in the first comment above). Travis keeps the cache of each job independent. Then, it is possible to 'permanently' evaluate both approaches in parallel. Two points to consider:

  • Developers working with go mod need to keep 'THE COMMIT' at the bottom or on top of each of their feature branches, and remove it before having it merged to master.
  • Maintainers need to update 'THE COMMIT' periodically.

@umarcor
Copy link
Author

umarcor commented Mar 7, 2019

@mcastelino upon further reading, yes, the reference you provided is definitely useful to accelerate multiple consecutive executions of docker build, when no files are to be shared from the host through the context. However, I was considering a completely different approach where the docker image is provided by kata, as a portable build environment. Then, the developer is not expected to build any image, but just docker run it to have the app built.

@sboeuf
Copy link

sboeuf commented Mar 7, 2019

@umarcor I think that as long as we don't run into issues with Kata repositories, and that we can achieve proper vendoring through go mod, then we should switch since it will become the new standard soon.
Basically, if there's no blocker, let's anticipate!

@bergwolf
Copy link
Member

bergwolf commented Mar 7, 2019

+1 for migrating to go mod. It gives a lot more flexibility not depending on GOPATH.

@caoruidong
Copy link
Member

I think we can move to mod when we switch to golang 1.13. At that time, go moduls will be on by default.

amshinde added a commit to amshinde/kata-runtime that referenced this issue Jul 26, 2019
Add the go.mod file required for converting runtime to go module.
This file is generated with `go mod init` using the existing
Gopkg.lock and running a `go build` followed by `go test`.

Fixes kata-containers#1331

Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
amshinde added a commit to amshinde/kata-runtime that referenced this issue Jul 31, 2019
Add the go.mod file required for converting runtime to go module.
This file is generated with `go mod init` using the existing
Gopkg.lock and running a `go build` followed by `go test`.

Fixes kata-containers#1331

Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
@HZ89
Copy link

HZ89 commented Apr 20, 2020

is any news about this issue?

@umarcor
Copy link
Author

umarcor commented Apr 20, 2020

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
7 participants