[Core feature] LiteralBlob and StructuredDataset metadata #5461
Labels
backlogged
For internal use. Reserved for contributor team workflow.
enhancement
New feature or request
Motivation: Why do you think this is important?
At runtime, Flyte knows about various metadata elements for a file / underlying dataset.
It would be amazing to catalog them and show them in the UI.
This metadata should be associated with the runtime objects and should be optional
Goal: What should the final outcome look like, ideally?
Example:
Note
We could also use this metadata to document lifecycle - for example if an expiration is set.
This can be useful in determining cache hits based on data TTLs. For example, each File / dataset etc can have the max-age set and this can be valid until date.
Describe alternatives you've considered
Store is separately, but then every File will become a directory and the format is only python. UI will have to implement this again.
Propose: Link/Inline OR Additional context
Add Blob metadata to
flyte/flyteidl/protos/flyteidl/core/literals.proto
Line 39 in cd37d1b
And
Structured Dataset metadata to
flyte/flyteidl/protos/flyteidl/core/literals.proto
Line 69 in cd37d1b
Also ideally the FileAccessProvider interface returns this metadata after each upload?
https://github.com/flyteorg/flytekit/blob/070027d8781d9b9f830723a9a4cae5daa6359a33/flytekit/core/data_persistence.py#L311
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?
The text was updated successfully, but these errors were encountered: