feat(dbt): add sibling association logic to associate dbt elements with their target systems #5190
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alexey-kravtsov
pushed a commit
to infobip/datahub
that referenced
this pull request
Jul 8, 2022
…th their target systems (datahub-project#5190)
maggiehays
pushed a commit
to maggiehays/datahub
that referenced
this pull request
Aug 1, 2022
…th their target systems (datahub-project#5190)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces a new aspect,
Siblings.pdl. Although this aspect is currently only being used to associate dbt and target systems, the idea is to use this aspect or pattern to do other types of associations in the future.The aspect gets ingested via the
SiblingAssociationHookclass. This hook listens for events that could indicate a dbt<>target pair and emits aSiblingsaspects to both siblings in the case that it finds a pair. It infers this relationship by inspectingUpstreamLineageandSubtypesaspects, because there are only a few cases in which a dbt and target pair will be siblings, and these cases can be determined by those two aspects alone. It also listens for DatasetKey aspects in an attempt to reconstruct the index should one of the siblings get deleted via delete API.The aspect comes into play in Search, Entity pages, and the Lineage API. In Search and Entity pages, Datahub will merge the metadata of the two siblings to make them appear as one. In the Lineage API, lineage of you & your sibling will be merged, and then siblings in that result set will be deduped.
Here are screenshots of those three experiences:
Checklist