Follow-up on the run_metadata changes

[bcdurak]

Describe changes

This PR optimizes the way that we store run metadata related to different entities. (addresses the review comment here)

In our old implementation, when someone calls log_metadata, it is possible that they attach the same metadata to different entities such as pipeline runs, step runs, and model versions. While this is still possible, this process used to create X different entries with the same key-value pair in the metadata table.

In order to optimize this process, this PR separates the previous run metadata table into two tables:

• One that holds the actual key-value pair.
• One that links this pair to different resources.

For this to work, I also implemented a new RunMetadataRequest model, that can hold more than one entity per key-value pair. All the previous calls that used this request model, all affected models and schemas are adjusted accordingly.

Other related changes

• Preview for the docs changes
• Starter Template PR
• E2E Batch Template PR
• NLP Template PR
• LLM Finetuning Template PR

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• If my change requires a change to docs, I have updated the documentation accordingly.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
• If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[bcdurak]

@htahir1 It would be great if you could also take a look at the docs changes.

[github-actions]

LLM Finetuning template updates in examples/llm_finetuning have been pushed.

[github-actions]

Classification template updates in examples/mlops_starter have been pushed.

[github-actions]

E2E template updates in examples/e2e have been pushed.

[github-actions]

NLP template updates in examples/e2e_nlp have been pushed.

[htahir1]

Well done! Just one small thing

[htahir1]

As you deletd this you might want to fix other links + add a redirect in .gitbook.yaml