proto files in another directory best practices question

[jmhodges]

I have need to interop with some tensorflow protobufs. However, the ones in question expect to be able to import each other in their current tensorflow/core/example layout. See example.proto

I've tried various things to get a copy of these files working in my own little monorepo, but protoc-gen-go doesn't quite help me out.

I have a monolithic repository that is kept at the path foobar in my GOPATH. The Go code inside it is kept at foobar/go.

At first, I tried just including the proto files in the package I was using them, foobar/go/tfrec, since I just need a way to write TFRecord files. But that causes protoc to crash saying that the imported file tensorflow/core/example/feature.proto couldn't be found.

Okay, so, I went and tried to add a foobar/proto directory since protobufs are cross-language, anyway, so maybe they deserve their own directory. But I'd rather not compile the Go code in there means that they will be outside my foobar/go code where my repo's tools are expecting them.

That seemed to work out, but I discovered it only works because those proto files expect to be generated inside one package. If you have a proto file that depends on a proto file that generates code in another package, and attempt to generate code in a directory other than where the .proto file was, the generated code won't build because the import path to the other generated package is wrong.

So, you'd say "oh, well, use import_prefix to solve that". Well, that doesn't work, because import_prefix also adjusts the import path of the github.com/google/protobuf import! And that doesn't work because I've vendored github.com/google/protobuf and its import path needs to remain the same. Using a sed to clean this up is a fragile hack.

Ah, then there's the M config you could use. The problem there is that means you have to track your proto imports in two places: once in the actual proto files, and a second time where ever you're running protoc. This is especially annoying when working with third-party proto files.

I have one proposed solution: I have code that adds a proto_prefix setting to protoc-gen-go that only adjusts the import paths to other generated proto packages. While annoying to remember the first time, it means only having to solve this once for all of my repo's packages and not having to update my build tooling every time a new import is added to a proto file.

I'm not entirely sure how this is working out in other people's repositories, though, so perhaps I've missed something obvious.

[jmhodges]

Made some edits for typos and clarity. Thanks for reading!

[jmhodges]

Oh! Maybe the answer is #139?

[neild]

Yes, I believe the answer is #139.

It looks like tensorflow builds using bazel (https://bazel.io/), which has its own system for building proto packages. It's building a "tensorflow" package from protos in many different directories. There's no easy way to do this directly with protoc, so far as I know.

If the tensorflow source files each included an option go_package line (as detailed in #139), you could build the Go package with something like:

$ protoc -Itensorflow tensorflow/tensorflow/core/**/*.proto --go_out=$GOROOT/src

(You could also use $GOROOT/src/foobar/go/vendor to put the generated package in a vendor directory adjacent to your own source code.)

[LMMilewski]

I'm going to close this issue since neild@ answered the question. Feel free to reopen if you have further questions.