Add task execution metadata to agent create

[noahjax]

Tracking issue

https://github.com/flyteorg/flyte/issues/

Why are the changes needed?

With the update to flytekit 1.11.0 (specifically this PR) and creation of AsyncAgentBase, output_prefix was removed from the create function for agents. This is problematic for our use case, and in general seems like it is moving things towards less flexible agents.

This PR goes a step farther and also adds task execution metadata to agent create, as this information is already readily available in the create task request.

What changes were proposed in this pull request?

Add output_prefix and task_execution_metadata to AsyncAgentBase create in a backwards compatible way that allows existing agents to continue to ignore these fields.

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

Attention: Patch coverage is 71.05263% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 83.53%. Comparing base (4522dc9) to head (ef8979e).
Report is 6 commits behind head on master.

Files	Patch %	Lines
flytekit/models/task.py	72.72%	9 Missing ⚠️
flytekit/extend/backend/agent_service.py	50.00%	1 Missing ⚠️
flytekit/extend/backend/utils.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2282      +/-   ##
==========================================
- Coverage   83.54%   83.53%   -0.01%     
==========================================
  Files         324      324              
  Lines       24592    24626      +34     
  Branches     3496     3503       +7     
==========================================
+ Hits        20545    20572      +27     
- Misses       3419     3427       +8     
+ Partials      628      627       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pingsutw]

Hey @noahjax let's discuss here

@pingsutw I'm actually not sure if there is an easy way to add TaskExecutionMetadata here...I don't think I can simply add another arg to AsyncAgentBase without updating all subclasses the way I did for output_prefix. Can we merge this smaller change and followup later on how to add this additional field?

Why do we need to update all subclasses as well for TaskExecutionMetadata? did you see any error?

[noahjax]

Yeah, with these changes I see this issue in test_dummy_agent where we test an agent that does not add additional fields to the create function

TypeError: DummyAgent.create() takes 3 positional arguments but 4 were given

[pingsutw]

@noahjax, I have a quick fix and have pushed it to your branch, which will keep it backward-compatible
8188db1.

[noahjax]

Thanks @pingsutw, I'll try to clean up the rest of this