[BUG] ShellTask ctx.secrets accessors not working with AWS Secrets

[garretcook]

Hey Team,

I'm attempting to use the convenient ctx.secrets replacement feature described here:
flyteorg/flytekit#832

However, when using AWS Secrets, the task will not run in the local or remote mode. I would normally set it up like this:

SECRET_GROUP='arn:aws:secretsmanager:region:id:secret:'
SECRET_KEY='secret-name:random5'
updater_cmd = ShellTask(
    name="task_3",
    debug=True,
    
    inputs=kwtypes(
        tenant_id=str, 
        redirect_url=str ,scope=str , 
        list_id=str ,site_id=str ,
        calendar_user=str, calendar_id=str, 
        update_sharepoint=str, update_calendar=str, 
        send_emails=str),
    script="""
    env;
    echo "test123";
    export CLIENT_ID={ctx.secrets.get('arn:aws:secretsmanager:region:id:secret:', 'secret-name:random5')};
    """,
    secret_requests=[
        Secret(
            group=SECRET_GROUP,
            key=SECRET_KEY,
            mount_requirement=Secret.MountType.ENV_VAR
        ),
    ]
)

Attempting to execute this returns:

Last Error: SYSTEM::Traceback (most recent call last):

      File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 178, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py", line 675, in dispatch_execute
        native_outputs = self.execute(**native_inputs)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/extras/tasks/shell.py", line 314, in execute
        gen_script = self._interpolizer.interpolate(self._script, inputs=kwargs, outputs=outputs)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/extras/tasks/shell.py", line 155, in interpolate
        return self._Formatter().format(tmpl, **consolidated_args)
      File "/usr/local/lib/python3.8/string.py", line 163, in format
        return self.vformat(format_string, args, kwargs)
      File "/usr/local/lib/python3.8/string.py", line 167, in vformat
        result, _ = self._vformat(format_string, args, kwargs, used_args, 2)
      File "/usr/local/lib/python3.8/string.py", line 220, in _vformat
        result.append(self.format_field(obj, format_spec))
      File "/opt/venv/lib/python3.8/site-packages/flytekit/extras/tasks/shell.py", line 133, in format_field
        return super().format_field(value, format_spec)
      File "/usr/local/lib/python3.8/string.py", line 237, in format_field
        return format(value, format_spec)

Message:

    TypeError: unsupported format string passed to _GroupSecrets.__format__

@samhita-alla recommended I open a issue from Slack: https://flyte-org.slack.com/archives/CP2HDHKE1/p1714370085210499?thread_ts=1714162847.725119&cid=CP2HDHKE1

I can use the normal task syntax perfectly, even in the same file/workflow:

@task(secret_requests=[
        Secret(
            group=SECRET_GROUP,
            key=SECRET_KEY,
        ),
])
def build_login_cmd():
    CREDENTIALS_LIST = flytekit.current_context().secrets.get(SECRET_GROUP, SECRET_KEY)
    print(CREDENTIALS_LIST)

[welcome]

Thank you for opening your first issue here! 🛠

[runllm]

The issue you're encountering with the ShellTask in Flyte when trying to use ctx.secrets to access AWS Secrets seems to be related to the interpolation of secrets within the shell script. The error message indicates a problem with formatting the secret value into the environment variable within the script. This is likely due to the way the ShellTask handles the interpolation of context variables and secrets, which might not support complex secret keys or groups directly in the script.

To address this, you might consider simplifying the secret access or ensuring that the secret keys and groups are correctly formatted and supported by the ShellTask. If the issue persists, it would be beneficial to raise this as a bug in the Flyte community or check for updates and patches that might address this specific use case.

Would you like more detailed guidance on how to adjust your script or further investigate this issue?

Sources:

• Discussion on credentials in env file
• Discussion on setting up secret value
• GitHub issue on environment variables

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[pingsutw]

@garretcook It might relate to this issue. It will allow you to specify the env var name in the shell script for your secret. #3053

[garretcook]

@garretcook It might relate to this issue. It will allow you to specify the env var name in the shell script for your secret. #3053

I feel like that would fix my issue. Am I correct in understanding #3053 has NOT been implemented at this time?