Support Vault's secrets engine v2

[jbialy]

Currently the vault datasource will not work with v2 version of Vault's secret engine. Extending the datasource to support this would allow for the use of versioned secrets!

https://www.vaultproject.io/api/secret/kv/kv-v2

[McShauno]

I second this request and would be glad to help if any is needed.

[hairyhenderson]

@McShauno thanks! Any help would be appreciated 🙂

I suppose the big question is how do we determine which kv version to use? Also it's a bit unclear if the Go client is any different between v1/v2 - I suspect the path is meant to be the API path, not the CLI path (for v2, the API path is /secret/data/foo, whereas for the CLI, and for v1 it's just /secret/foo).

In fact, it could work with kv v2 right now if you just use /secret/data/foo instead of /secret/foo 🤔... But that's also clearly not the best user interface.

[petrjurasek]

@hairyhenderson I think getting exact version is only missing, as getting latest version of v2 secret can be done even now:

gomplate --version
gomplate version 3.7.0
vault --version
Vault v1.4.2

1. start vault in dev mode (with k/v "secrets" engine enabled by default)

VAULT_DEV_ROOT_TOKEN_ID="root-token" vault server -dev

2. create secret with vault binary

VAULT_DEV_ROOT_TOKEN_ID="root-token" VAULT_ADDR="http://localhost:8200" vault kv put secret/foo/bar/baz hello=world
Key              Value
---              -----
created_time     2020-06-10T20:05:10.00293551Z
deletion_time    n/a
destroyed        false
version          1

3. get secret with vault binary

VAULT_DEV_ROOT_TOKEN_ID="root-token" VAULT_ADDR="http://localhost:8200" vault kv get secret/foo/bar/baz
====== Metadata ======
Key              Value
---              -----
created_time     2020-06-10T20:05:10.00293551Z
deletion_time    n/a
destroyed        false
version          1

==== Data ====
Key      Value
---      -----
hello    world

4. get secret with gomplate

VAULT_TOKEN="root-token" VAULT_ADDR="http://localhost:8200" gomplate --datasource vault=vault:// -i '{{ (datasource "vault" "secret/data/foo/bar/baz").data.hello }}'
world

But what would be really nice, if one could select exact version of secret:

VAULT_DEV_ROOT_TOKEN_ID="root-token" VAULT_ADDR="http://localhost:8200" vault kv put secret/foo/bar/baz hello=doot
Key              Value
---              -----
created_time     2020-06-10T20:16:18.559436096Z
deletion_time    n/a
destroyed        false
version          2

VAULT_DEV_ROOT_TOKEN_ID="root-token" VAULT_ADDR="http://localhost:8200" vault kv get secret/foo/bar/baz
====== Metadata ======
Key              Value
---              -----
created_time     2020-06-10T20:16:18.559436096Z
deletion_time    n/a
destroyed        false
version          2

==== Data ====
Key      Value
---      -----
hello    doot

VAULT_TOKEN="root-token" VAULT_ADDR="http://localhost:8200" gomplate --datasource vault=vault:// -i '{{ (datasource "vault" "secret/data/foo/bar/baz").data.hello }}'                                                                                                                             
doot

Basically looking for gomplate equivalent of

VAULT_DEV_ROOT_TOKEN_ID="root-token" VAULT_ADDR="http://localhost:8200" vault kv get -version=1 secret/foo/bar/baz 
====== Metadata ======
Key              Value
---              -----
created_time     2020-06-10T20:12:37.876472509Z
deletion_time    n/a
destroyed        false
version          1

==== Data ====
Key      Value
---      -----
hello    world

With current implementation of vault datasource, query parameters will invoke PUT request

VAULT_TOKEN="root-token" VAULT_ADDR="http://localhost:8200" gomplate --datasource vault=vault:// -i '{{ (datasource "vault" "secret/data/foo/bar/baz?version=1").data.hello }}'

22:19:54 FTL  error="template: <arg>:1:4: executing \"<arg>\" at <datasource \"vault\" \"secret/data/foo/bar/baz?version=1\">: error calling datasource: Couldn't read datasource 'vault': Error making API request.\n\nURL: PUT http://localhost:8200/v1/secret/data/foo/bar/baz\nCode: 400. Errors:\n\n* no data provided"

Vault API reference for getting exact version of v2 secret: https://www.vaultproject.io/api/secret/kv/kv-v2.html#read-secret-version

[hairyhenderson]

@petrjurasek I'm not sure if this'll work, but maybe try this instead:

VAULT_TOKEN="root-token" VAULT_ADDR="http://localhost:8200" gomplate --datasource 'vault=vault://?version=1' -i '{{ (datasource "vault" "secret/data/foo/bar/baz").data.hello }}'

If neither of those work (and they should be equivalent), then I think the key is to just handle the version query parameter properly.

[petrjurasek]

@hairyhenderson - same result

VAULT_TOKEN="root-token" VAULT_ADDR="http://localhost:8200" gomplate --datasource 'vault=vault://?version=1' -i '{{ (datasource "vault" "secret/data/foo/bar/baz").data.hello }}'

06:08:13 FTL  error="template: <arg>:1:4: executing \"<arg>\" at <datasource \"vault\" \"secret/data/foo/bar/baz\">: error calling datasource: Couldn't read datasource 'vault': Error making API request.\n\nURL: PUT http://localhost:8200/v1/secret/data/foo/bar/baz\nCode: 400. Errors:\n\n* no data provided"

[hairyhenderson]

Thanks @petrjurasek, that makes sense.

Looks like adding the version param is causing the PUT API to be used which isn't quite what we want. Also I'd really rather not have to use the API path, since it's different than what would be provided to vault kv get.

I'll try and find some time this weekend to dig into this a bit more.

[carltonmason]

We have been using gomplate for several years now and it has been working great. We too have a need to use v2 of the secrets engine to be able to version secrets. Looks like this request has gone a bit stale.

[hairyhenderson]

Thanks for the bump @carltonmason - indeed this has gone stale... I'd love if someone could take this on if possible! 🙂

[petrjurasek]

@hairyhenderson any hints where to start digging in the code?

I guess the PUT method is part of gomplate(?) logic, rather than vault library itself.

[petrjurasek]

Also found this might be helpful to determine version of kv

https://github.com/hashicorp/vault/blob/v1.7.3/command/kv_helpers.go#L99

[hairyhenderson]

Probably relevant to this whole discussion: at some point I'll be updating gomplate to use my new go-fsimpl library for datasource support.

Currently go-fsimpl is missing Vault support however, and I've been working on it for a little while (still a few days from a PR for it though).

[pkovtunov]

@hairyhenderson, would be happy about KV2 support also - we have a mixed Vault setup with multiple tenant spaces in there and some have KV1, others have already KV2. The secrets are used for K8S deployments in multiple stages/branches and with Gomplate we were able to reduce the complexity dramatically :) Thank you for this nice templating engine!

[hairyhenderson]

Currently go-fsimpl is missing Vault support however, and I've been working on it for a little while (still a few days from a PR for it though).

Update: go-fsimpl just got Vault support, though it doesn't yet support KV2 - that's tracked in hairyhenderson/go-fsimpl#24.

I'm not totally opposed to KV2 support going into gomplate prior to moving to go-fsimpl, but I won't have the time to do that myself - I'll be focusing on the go-fsimpl support instead.

[dbaumgarten]

I know I am a little late to the party, but for future reference:
I found a way to use the v2 kv-engine with gomplate.

You have to add an extra "/data" between the name of the secret-engine (general) and the path of the secret (secretpath):
gomplate -d vault=vault:///general/data/secretpath instead of gomplate -d vault=vault:///general/secretpath.

You can then use {{ (datasource "vault" ).data.keyinsidesecret }} in your templates. Notice the extra ".data" there. (In addition to .data there are also some other keys in the struct that may be interesting)

It's a bit strange, but seems to work just fine.

@hairyhenderson Maybe it would be worth it to add this piece of info to the documentation?

[hairyhenderson]

Thanks @dbaumgarten - this is good information. Once kv2 is supported this will break, but it's at least worth noting with that caveat!

[dbaumgarten]

Wouldn't it be possible do design the v2-support in a way that it at least emits a warning when it detects that you are using a (now broken) workaround.
Looking for path's of the form "secretengine/data/*" should work well enough.

[hairyhenderson]

@dbaumgarten perhaps? It wouldn't be as simple though... Besides, what if someone is using the literal prefix data/ in their path?

If it's documented (I am open to PRs for this!!), I think that's enough. If someone's done this workaround and things break with a future version of gomplate, it should be relatively obvious why.

[dngray]

kv2 would be something that I'm interested in having. I haven't had a look in how to implement it though, not by any means a golang expert.

Perhaps looking at how chezmoi handles vault would be worthwhile?

[hairyhenderson]

@dngray interesting! Never heard of chezmoi, thanks for the tip!

kv2 support will come probably in gomplate 4.0, when it moves to use go-fsimpl in the backend. I would accept a PR to document the workaround, as mentioned above 😉

[github-actions]

This issue is stale because it has been open for 60 days with no activity. Remove stale label or comment or this will be automatically closed in a few days.

[AndrewSav]

@hairyhenderson are you still considering this?

[AndrewSav]

Remove stale label or comment or this will be automatically closed in a few days.

I find it interesting that this issue is still marked stale, despite me commenting on it.

[hairyhenderson]

Remove stale label or comment or this will be automatically closed in a few days.

I find it interesting that this issue is still marked stale, despite me commenting on it.

I think that's because the stale workflow only runs weekly (I can increase frequency though).

[hairyhenderson]

@hairyhenderson are you still considering this?

Yep. My latest comment in #862 (comment) still stands.

To put it more concretely, #1336 will fix this issue.

[github-actions]

This issue is stale because it has been open for 60 days with no activity. Remove stale label or comment or this will be automatically closed in a few days.

[AndrewSav]

Go team ;)

[AndrewSav]

@hairyhenderson

I think that's because the stale workflow only runs weekly (I can increase frequency though).

This is quite unusual comparing to any other repo that does this. Hourly seems like a good schedule, some run every 5 minutes.

[dngray]

I would still like to see this feature.

[zachfi]

Me too. I'm stalling on one of my projects for this feature.

[hairyhenderson]

Thanks for commenting folks.

To re-iterate, this will be implemented by #1336. Once that makes it (which could very well take a while yet - I have limited time), this issue will auto-close.

I think that's because the stale workflow only runs weekly (I can increase frequency though).
This is quite unusual comparing to any other repo that does this. Hourly seems like a good schedule, some run every 5 minutes.

I have a limited number of GitHub Actions minutes available to me per month, so I'm not really interested in running the workflow that often.

I have assigned this issue to a new v4.0.0 milestone, and configured the workflow to no longer consider milestone-assigned issues as stale.

In other words: rest assured - this one'll make it in to 4.0.

[AndrewSav]

@hairyhenderson

I have a limited number of GitHub Actions minutes available to me per month

How come? GitHub Actions usage is free for standard GitHub-hosted runners in public repositories, and for self-hosted runners

[hairyhenderson]

@AndrewSav if you read that same page further down, you'll see the limits: https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#included-storage-and-minutes

Apologies - I missed the private/public split! Given this is a public repo, it doesn't apply... I have a number of private repos where it matters, so I forgot public was tracked separately 😉.

In that case, I could run that workflow more often. However, I still don't see a need.

Either way, this particular thread is entirely off-topic for this issue - if you want to continue the discussion let's take it to https://github.com/hairyhenderson/gomplate/discussions

[AndrewSav]

This is most frustrating. You said:

I think that's because the stale workflow only runs weekly (I can increase frequency though).

Now when I'm asking you to increase the frequency you "don't see a need". Why did you offer it in the first place, if you were not going to do it? Continued.

[AndrewSav]

@hairyhenderson So... fsimpl or not, parameters for dynamic backends for the Vault data source are passed via query string, which means if there is a query string the request is converted to PUT. It means we need a new interface to provide the version, but the URL is quite limiting here. How do you envision we pass this information at the data source?

--datasource 'vault=vault://?version=1&useget=true seems clumsy

[hairyhenderson]

if there is a query string the request is converted to PUT.

according to what?

With go-fsimpl, params can be set while using KV2 - see https://pkg.go.dev/github.com/hairyhenderson/go-fsimpl/vaultfs:

When parameters are set in this way, vaultfs will send a POST request to the Vault API, except when the K/V Version 2 secret engine is in use.

[AndrewSav]

according to what?

Assuming that it was you who wrote the code and documentation, according to you ;)

With go-fsimpl, params can be set while using KV2 - see https://pkg.go.dev/github.com/hairyhenderson/go-fsimpl/vaultfs:

Oh I see how that works. First we query the backend and determine from options if it's kv2, If it's not and we have query parameters we convert to POST (used to be PUT), otherwise, for kv2 we stay with GET. Got it, thanks.

[hairyhenderson]

Assuming that it was you who wrote the code and documentation, according to you ;)

🤦‍♂️ indeed... I thought you were referring to go-fsimpl there since that's the only way kv2 will be getting supported in gomplate.

[AndrewSav]

I thought you were referring to go-fsimpl there since that's the only way kv2 will be getting supported in gomplate.

I was considering doing a quick and dirty PR for kv2 version support without go-fsimpl, that's why I asked. Now I see that it is not very practical.

[hairyhenderson]

@AndrewSav if you're willing to contribute a PR, my comment last year still stands - a documented workaround would probably be worthwhile!

[AndrewSav]

@hairyhenderson if I'm not mistaken we do not have a workaround for getting a v2 version support, which I referred to in my previous comment did I miss something? I know that we have a work around for getting the latest value, but that was not what I was talking about.