Allow workers to pass an API key to flow runs that is different from their own

[EmilRex]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar request and didn't find it.
• I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

When creating a new flow run, workers inject their own api key into the flow run's environment. This means that the flow run always has the same permissions as the worker.

Describe the proposed behavior

In cases where workers and flow runs should have different permissions, it would be great if you could specify an api key for the flow run to use that is different from the worker api key.

Example Use

No response

Additional context

I'm not sure how this would be implemented, but one consideration is how environment variables are handled - especially PREFECT_ environment variables. If there was greater control over how environment variables are populated in infrastructures, this feature would likely be easier to support.

[desertaxle]

Thanks for the issue @EmilRex! This is possible by setting PREFECT_API_KEY in the env attribute on the base job template for our supported work pools. I believe users can use a secret block to prevent this API key from being stored in plaintext, but I will need to confirm that.

[EmilRex]

That would work great if that's possible! I know we are doing some manipulation of env and wasn't 100% sure if we overwrite PREFECT_API_KEY or if a user supplied value would take precedence.

[prefectcboyd]

Discussing with the team and a few customers to get more details, I think this is a pretty strong / urgent requirement for them to be capable of delivering / executing this work in production, and would be a very hard stop / do not pass go situation if it was not available.

Ideally, I think what should be possible is the ability to specify / pass a reference to a secretKeyRef in the base job template that doesn't get overwritten. Currently, any environment variables that are set through env get replaced with the ones that the worker wants to set by default (so like the API_URL and API_KEY).

If there was an option / toggle on the prefect worker side to say "inherit the worker key" or alternatively "reference this k8s secret api key", I think that might be a solution?

cc @taylor-curran

[prefectcboyd]

@EmilRex , @desertaxle - Just to confirm and verify - I tested adding an environment reference in the job template.
My current k8s worker is setup to reference the API key through a secret.
I have passed this reference as an environment variable in the job template using secretKeyRef, but it becomes overwritten and in plain text, with no reference to the secret.
Image below for reference -

[meggers]

Chiming in from the customer side, we have a critical need to not use plain text secrets (#10716) and so if we were able to say "If there was an option / toggle on the prefect worker side to say "inherit the worker key" or alternatively "reference this k8s secret api key"" instead of including the plain text worker api key, that would be a big win.

Ideally I'd like to use a k8s secret, similar to what I am doing with the worker secret, rather than a secret block.

[desertaxle]

@prefectcboyd I have a fix for the issue you're seeing in PrefectHQ/prefect-kubernetes#94. I believe merging and releasing that PR will allow us to resolve this issue.