feat: add dataset schema versions

[davidjgoss]

Problem

This PR is the first step towards implementing the proposal from https://github.com/MarquezProject/marquez/blob/main/proposals/2676-version-dataset-schemas-separately.md.

Solution

The idea is that, without changing what already gets written to the database, we start writing to the new dataset_schema_versions and dataset_schema_versions_field_mapping tables.

Todo

• Create new database tables
• Upsert dataset schema version when creating a new dataset version
  • OpenLineage code path
  • Legacy DatasetDao code path
  • Legacy RunDao code path not viable to tackle in this PR
• Handle in retention cleanup
  • Ensure schema versions and fields get swept up in the cascade when a dataset is deleted
• Handle input dataset where schema doesn't match current version for now will not change schema version even when input dataset schema has drifted - to be discussed in Handling of input datasets where schema different from current version #2764

One-line summary: Add dataset schema versions to model and start writing to it

Checklist

• You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

[netlify]

✅ Deploy Preview for peppy-sprite-186812 ready!

Name	Link
🔨 Latest commit	fb182cf
🔍 Latest deploy log	https://app.netlify.com/sites/peppy-sprite-186812/deploys/665fde0ff4770500084bef13
😎 Deploy Preview	https://deploy-preview-2763--peppy-sprite-186812.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.74%. Comparing base (1f00c9b) to head (fb182cf).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2763      +/-   ##
============================================
+ Coverage     84.56%   84.74%   +0.18%     
- Complexity     1441     1456      +15     
============================================
  Files           251      253       +2     
  Lines          6504     6562      +58     
  Branches        303      305       +2     
============================================
+ Hits           5500     5561      +61     
+ Misses          851      850       -1     
+ Partials        153      151       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[davidjgoss]

Next step with this is to look at the deprecated DatasetResource path for ingestion. Will timebox this and abandon if it's going to take too much effort.

[davidjgoss]

This should be suitable to merge and release, so we start writing data into the new tables now, which should ease the migration later.

Next steps in this project will be (in no particular order and can be in parallel):

• Update DbRetention so it will clean up any orphaned schema versions after it cleans up dataset versions
• Develop a migration script to backfill dataset schema versions for pre-existing dataset versions
• Start to change some code paths to read from the new tables where possible

[wslulciuc]

💯 💯 💯