gazelle and go.mod versioning

[miekg]

Hello,

The minio-go projects uses this in the their go.mod file:

module github.com/minio/minio-go/v6

I.e. they tag on v6, with some triggery (and using gazelle version 0.17) I generate a build file that has the correct importpath: importpath = "github.com/minio/minio-go/v6 however when buiding bazel complains:

no such package '@com_github_minio_minio_go//v6': BUILD file not found on package path and referenced by '....'

IIRC in the go modules proposal adding strings that start with v are supposed to be handled in a special way (but I'll have to check)

[miekg]

/cc @asuffield

[miekg]

If you go into the minio directory in .../external and do a mkdir v6; cp -r * v6 things will compile. Not sure if this qualifies as a hack even...

[miekg]

OK, @asuffield figured it out, this is because gazelle is not reading the deps.bzl file where this is specified, meaning it tries to figure things with heuristics. If I move the go_repository rule into the WORKSPACE file gazelle uses the data as-is and things build.

[tmhdgsn]

this can be fixed using the repository_macro directive in your workspace, introduced in 4f524f2

E.g.

# gazelle: repository_macro deps.bzl%defName

[asuffield]

The last piece of this puzzle is that the dependency on the other repo is itself in a go_repository rule, and we are running without direct internet access.

With vanilla gazelle, this doesn't currently work at all. With internet access, it can resolve the path via http query. With the usage in the root repository, the repository_macro directive works.

When it's a go_repository rule, I have a patch to gazelle that I keep meaning to turn into a PR, which passes the WORKSPACE file to the gazelle invocations so that it can use entries in that file to resolve importpaths to repo names. However, that doesn't have a good solution for repository_macro (one of the reasons why I haven't sent that PR yet): in general there's no way to tell which files gazelle might need to read.

We might need a bazel feature to do something like aspects for repository rules, in order to extract this data reliably. In the general case, at the time when go_repository is executing, the rest of the things included in WORKSPACE have not even been parsed yet, and might not be parseable without completing this gazelle invocation. This is tricksy.

[jayconrod]

About the original issue, some fixes for importing modules have been merged to master since Gazelle 0.17.0 was released. We're pretty close to a 0.18.0 release, so I'd suggest using a recent commit instead of 0.17.0. Minimal module compatibility is the main missing feature, but that's won't affect most people.

Once you upgrade, you should be able to have a repository rule like this;

go_repository(
    name = "com_github_minio_minio_go_v6",
    importpath = "github.com/minio/minio-go/v6",
    sum = "h1:UxEyhY10E9qog45XmUQyX4npPao0uwLHcoqZcF9gjOI=",
    version = "v6.0.27",
)

If you have a working go.mod file, you can import all the modules with:

bazel run //:gazelle -- update-repos -from_file=go.mod

If you go into the minio directory in .../external and do a mkdir v6; cp -r * v6 things will compile. Not sure if this qualifies as a hack even...

Don't do this. Repository rules will be re-evaluated whenever anything they depend on changes, which means their directories are deleted.

this is because gazelle is not reading the deps.bzl file where this is specified, meaning it tries to figure things with heuristics. If I move the go_repository rule into the WORKSPACE file gazelle uses the data as-is and things build.

Not sure what's happening here? Is this a manually written go.mod file? It should work, but make sure the macro inside it is loaded and called from WORKSPACE.

this can be fixed using the repository_macro directive in your workspace, introduced in 4f524f2

This tells Gazelle to update rules declared in that macro when update-repos is run. Make sure there is not space after the colon: # gazelle:repository_macro deps.bzl%defName.

The last piece of this puzzle is that the dependency on the other repo is itself in a go_repository rule, and we are running without direct internet access.

With vanilla gazelle, this doesn't currently work at all. With internet access, it can resolve the path via http query. With the usage in the root repository, the repository_macro directive work

I'm not sure how this would work at all without network access. go_repository always fetches something from the network.

When it's a go_repository rule, I have a patch to gazelle that I keep meaning to turn into a PR, which passes the WORKSPACE file to the gazelle invocations so that it can use entries in that file to resolve importpaths to repo names. However, that doesn't have a good solution for repository_macro (one of the reasons why I haven't sent that PR yet): in general there's no way to tell which files gazelle might need to read.

This should work in the main repository, maybe since #493. Sorry it's not released yet, but try a newer commit from master instead of 0.17.0.

Custom repository names aren't known within go_repository rules, which sounds like the issue you're having. That's #529.

[asuffield]

We have a contraption to fetch archives via an asset management thing - but gazelle's direct http queries for golang metadata do not (currently) work, so we completely rely on picking up the repository names from WORKSPACE to resolve them.

#529 is indeed precisely what my locally-applied patch addresses, partially. I've been sitting on it for a while, trying to figure out how to get something like repository_macro to work with it.

[jayconrod]

I'll close this issue since #529 seems to cover all missing functionality. Happy to answer questions though.

I think the solution to #529 will involve invoking Gazelle with a flag telling it that it's in a go_repository. It will need to discover the main WORKSPACE file and read configuration from it.

As a workaround, there is a -known_import command line flag that tells Gazelle a specific import path is the root of a repository (so it won't need to go out to the network). You can pass flags to Gazelle through go_repository's build_extra_args attribute. So build_extra_args = ["-known_import=example.com/repo"].