Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(spark):OpenLineage 1.25.0 #12041

Merged
merged 1 commit into from
Dec 12, 2024

Conversation

Jorricks
Copy link
Contributor

@Jorricks Jorricks commented Dec 5, 2024

Context

We are trying to use the Datahub project to ingest metadata of our Spark jobs that exclusively interact with Delta tables.
Recently we upgraded to a newer Delta version and this broke the integration.
This was fixed in OpenLineage/OpenLineage#3253.
Given they just released https://github.com/OpenLineage/OpenLineage/releases/tag/1.25.0, it would be nice to use this version in Datahub as well.

Potentially, in a future PR, it would also be nice to append the new metadata on amount of data written/read to be written to Datahub as well.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added the community-contribution PR or Issue raised by member(s) of DataHub Community label Dec 5, 2024
@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Dec 5, 2024
Copy link
Contributor

@treff7es treff7es left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the plugin, and it seems to be functioning properly. Thank you! I will merge it and create a new release soon with a couple of other fixes.

@treff7es treff7es merged commit 7339848 into datahub-project:master Dec 12, 2024
67 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community-contribution PR or Issue raised by member(s) of DataHub Community needs-review Label for PRs that need review from a maintainer.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants