Release Linux packages

[jackietung-redpanda]

Publishing connect via deb/rpm packages is part of the new rpk connect distribution strategy. More context here.

This PR adds said Linux packaging. Packages will be created and published to Redpanda's production Linux package repo on Cloudsmith.

Solution is modeled off redpanda-console, which also publishes packages easily via goreleaser's nfpm integration.

This PR must merge after CLOUDSMITH_API_KEY access has been granted. This means after this PR gets merged and TF applied.

[jackietung-redpanda]

Linux package publishing was tested by running goreleaser release locally, after commenting out certain parts of the config:

• Disable GitHub release publishing
• Keep only connect build (for speed)
• Keep only Linux / amd64 build (for speed)

[jackietung-redpanda]

cc @r-vasquez in case interested to see.

[ivotron]

shouldn't there be a dependency with redpanda-rpk?

[jackietung-redpanda]

AFAIK, the intent is not to pull in rpk automatically when someone does apt install redpanda-connect.

For almost ALL users, they would be installing apt install redpanda-rpk as the entrypoint. rpk at runtime takes care of the rest.

For airgapped users, they would be installing apt install redpanda-rpk. They would then ALSO do apt install redpanda-connect IF they want the connect plugin. I think it keeps the mental model simple this way, without adding complexity:

• You want to run rpk? apt install redpanda-rpk.
• You want to run rpk connect? Add apt install redpanda-connect

Note this is not a one-way decision - adding the dep is much easier than removing it (if folks start depending on it, rightly or wrongly!). Mechanics wise, there will be some complications (we can figure something out):

• Depend on redpanda-rpk which version of RPK? Some 24.1.x, some 24.2.x? How might that be expressed in the deps? How might that play with the existing pins folks have in their env for redpanda itself?
• Would we need to wait till we have redpanda-rpk packages that DO NOT bundle .rpk-ac.connect? I think yes. We would need to think of some time line of a sequential update schedule. Something like:
  • have redpanda-connect (no deps) - we are here.
  • have new redpanda-rpk packages A that do not have connect inside
  • have redpanda-connect start depending on A (or newer, etc)

Don't feel very strongly. I think not adding the dep is slightly more sensible right now.

As an aside - redpanda-connect is actually fully usable as an independent binary (just as we distribute it that way on our redpanda-data/connect releases on GitHub.

[ivotron]

thanks a lot for the complete explanation! I agree that this route (not adding the dependency) is preferable. This raises a question though: what is the expectation when the following happens?

• install redpanda-rpk
• run rpk connect .. which will fetch the plugin
• install redpanda-connect

I think we discussed this in one of our meetings but I forgot what was the answer.

[jackietung-redpanda]

This will be handled by new rpk (that can do fetches). My understanding is that rpk maintains it's own set of managed plugin binaries, separated by user. So the initially fetched plugin will live in some user specific folder, and that trumps everything. I.e. even if redpanda-connect is subsequently installed by the admin (root owned), it will not have an effect as long as the user's own fetched plugin remains installed.

rpk even provides a command to list plugins, show the effective plugin binary being used, and any background ones that are being "shadowed".

[jackietung-redpanda]

Prerequisite PR https://github.com/redpanda-data/qa-infra/pull/116 has been merged and applied. Linux publishing step here should have sufficient perms to push to Cloudsmith.