[Proposal]: Flyte System Tags and metadata

[kumare3]

This RFC proposes a way to add execution Tags and description.
this would help the user in multple ways. Please read the rfc for more context.

Signed-off-by: Ketan Umare [email protected]

[goyalankit]

very useful feature!

[goyalankit]

Are users allowed to change the labels? If they are then overriding might be an issue since you might have already fired async events to external systems. So I think it might be useful to maintain executionID as an identifier that can't be modified once execution has been created.

Alternatively, this could be an alias to the execution ID rather than overriding?

[kumare3]

executionID cannot be changed - it is immutable and unique per project/domain.
name is just an alias. I will update the doc to reflect this

[kumare3]

but i do like the idea of immutable labels as well. once added you cannot change them

[ggydush]

So we'll support both mutable and immutable labels?

[flixr]

Nice! Mostly sounds good to me.
Just that I would go with the name tags on the flyte (cli) level.
Querying could still be done on k8s labels as well...

[flixr]

I would probably call the arg here --tag to not confuse this with kubernetes labels.
And then probably assign the tags to k8s annotations.

[kumare3]

great point, but the rpc field is sadly already called label. and these will become k8s labels

[fg91]

Suggested change

	We this this is one of the most requested features in Flyte and will solve
	This is one of the most requested features in Flyte and will solve

[fg91]

If this approach is chosen, I wonder whether it would be nicer for the user to do

pyflyte run --remote --group foo --experiment bar ...

instead of

pyflyte run --remote --labels group:foo --labels experiment:bar ...

This doesn't mean that under the hood the labels mechanism couldn't be used.

[bstadlbauer]

I'm somewhat opposed to this as it could be confusing to users as to what is a label vs what is a keyword cli argument 🤔

[bstadlbauer]

Or do we wanna create group and experiment as CLI arguments and introduce them as a concept? 🤔

[fg91]

I personally prefer option 1: treat all labels the same way. Users might not want to follow the categories we deem sensible. Experiment tracking servers like Mlflow or Wandb, which also have such a tagging mechanism, simply allow users to assign arbitrary tags. I would argue that ML engineers are used to this and we should provide the same UX without imposing special naming conventions.

Only exception: execution name
I find it really helpful to have the pod names include customizable identifiers.
We have a registration script, similar to pyflyte run with has an --execution_name arg. The user provided value is appended with a random uuid, as is currently already chosen for the execution ids, and the result is checked against the execution name regex again and then passed to FlyteRemote.execute(execution_name=...) (already supported, see here). So I wouldn't treat execution name with a pod label but the pods metadata.name.

This comment is another argument for not treating execution names with labels but instead metadata.name since I agree that tags need to be mutable.

[pingsutw]

@bstadlbauer @fg91 @elibixby @flixr @goyalankit Some questions for you

1. Do you prefer key-value pair tags or tags that only have key?
2. Should we add tags to Kubernetes label?

Currently, execution spec (with labels) is serialized to byte and is stored in the execution table. it's impossible to add / delete / update tags. if we use k8s client to filtered flyteworkflow (CRD) by labels. we cannot search a execution after CR is deleted.

I have a PR that adds tags table. it allows us easily add / update / delete tags, and even attach tags to task / workflow / project. however, it's not key-value pair tags for now. If we decide to use key-value pair tags, I just need to add a new column to the tags table and update the query. I'd like to know your thought first.

btw, the current implementation works with both Mysql and Postgres.

[tsheiner]

My $0.02 is let's keep it simple and support what you call key-only tags.

A person can 'hack' this to resemble key-value if needed (ie 'costcenter-12'), but we don't need to manage that complexity on the back end or in the UI when we get to figuring out how to let folks use tags to sort/group things.

[fg91]

• In my opinion key-only tags are perfectly fine and what ML engineers are used to from experiment tracking servers

• Should we add tags to Kubernetes label

  I think being able to add/delete/update tags after the execution has already started or ended is an important feature. User story: an experiment is training/trained really well and I want to mark it for later. This is something that is not known when starting the execution. But updating/deleting/adding tags when the execution is already running would mean that the k8s labels are not in sync with what is stored in the tags table. I'd therefore say that I wouldn't apply the tags as labels to k8s.

[bstadlbauer]

Sorry for the late response here but agreed with what's been said above. Key only tags would also solve all our usecases 👍

I think being able to add/delete/update tags after the execution has already started or ended is an important feature
+1 to this and the reasoning of not applying those to k8s

[fg91]

I like this proposal very much. Currently we use decks to link to Wandb runs corresponding to Flyte executions.
We then use tags in wandb to do the grouping by experiments, tags, ... that you describe in the RFC.
Would love to do this directly in Flyte.

[davidmirror-ops]

03-30-2023 Meeting notes:
KU: launchplan name could be the default grouping tag
Tim Sheiner: that's not very prominent right now in the UI
KU: we could add all to system tags but could be an overload
TS: make sure that this proposal is not redundant to the fact users are already naming workflows and tasks
TS: instead of treating this proposal as tags, treat it as a separate...
GG: their use case is running workflows from notebooks (using Quarto/Jupyter for reports)

[tsheiner]

Slightly perpendicular to this proposal but related to the notion of making executions easier to identify:

I note that Prefect uses a system of nonsense ids for executions which is much friendlier looking and far easier to remember than Flyte alphanumeric ids. For example ‘enigmatic-waxbill’ for ‘massive-antelope.’ Would something like this be possible for Flyte?

[davidmirror-ops]

04-13-203 notes: no updates

[bstadlbauer]

Looks good to me!

There is an ongoing slack conversation related to MySQL tag storage here - not 100% sure what that is about? cc @kumare3 @ByronHsu

[fg91]

For visibility: @kasimiraula proposed that user should be able to create notes for executions, similar to what is currently possible when aborting an execution #3646

[kumare3]

Slightly perpendicular to this proposal but related to the notion of making executions easier to identify:

I note that Prefect uses a system of nonsense ids for executions which is much friendlier looking and far easier to remember than Flyte alphanumeric ids. For example ‘enigmatic-waxbill’ for ‘massive-antelope.’ Would something like this be possible for Flyte?

I think this is possible and I have thought about it. We should consider entropy considerations and cost of doing this and then all for it 👍🏽

[fg91]

Thanks for incorporating what was discussed in the contributors' syncs. I like that now:

• tags will not be attached to k8s objects as labels but instead saved in the database so that they can be modified/deleted during/after the execution.
• all tags are equal and Flyte doesn't impose any special names.
• we have a clear distinction between tags and notes.

Please consider all comments below as nit-picks that you can just resolve in case you don't agree.

[fg91]

Could we do --tag hello --tag world instead of providing a list of tags in string format?

[pingsutw]

maybe we can support both?

[pingsutw]

pyflyte run --remote --tags '["hello", "world"]' 
# and 
pyflyte run --remote --tag hello --tag hello
# and 
pyflyte run --remote --tag hello --tags '["key1", "key2"]' 

[fg91]

My opinion about this is not so strong that I'd say we need to support two options in case others prefer --tags '["hello", "world"]' . I personally find lists or jsons in string representation as cli args a bit cumbersome.

[fg91]

LG!