Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli/data product): add support for institutional memory #10686

Merged
merged 1 commit into from
Jun 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions metadata-ingestion/examples/data_product/dataproduct.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,13 @@ tags:
terms:
- urn:li:glossaryTerm:ClientsAndAccounts.AccountBalance

institutional_memory:
elements:
- title: URL for campaign
description: |-
Go here to see the campaign.
url: https://example.com/pet_of_the_week

# Custom Properties
properties:
lifecycle: production
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
GlobalTagsClass,
GlossaryTermAssociationClass,
GlossaryTermsClass,
InstitutionalMemoryClass,
InstitutionalMemoryMetadataClass,
KafkaAuditHeaderClass,
MetadataChangeProposalClass,
OwnerClass,
Expand Down Expand Up @@ -74,13 +76,23 @@ def ownership_type_must_be_mappable_or_custom(cls, v: str) -> str:
return v


class InstitutionMemoryElement(ConfigModel):
url: str
description: str


class InstitutionMemory(ConfigModel):
elements: Optional[List[InstitutionMemoryElement]] = None


class DataProduct(ConfigModel):
"""This is a DataProduct class which represents a DataProduct

Args:
id (str): The id of the Data Product
domain (str): The domain that the Data Product belongs to. Either as a name or a fully-qualified urn.
owners (Optional[List[str, Ownership]]): A list of owners and their types.
institutional_memory (Optional[InstitutionMemory]): A list of institutional memory elements
display_name (Optional[str]): The name of the Data Product to display in the UI
description (Optional[str]): A documentation string for the Data Product
tags (Optional[List[str]]): An array of tags (either bare ids or urns) for the Data Product
Expand All @@ -94,6 +106,7 @@ class DataProduct(ConfigModel):
assets: Optional[List[str]] = None
display_name: Optional[str] = None
owners: Optional[List[Union[str, Ownership]]] = None
institutional_memory: Optional[InstitutionMemory] = None
description: Optional[str] = None
tags: Optional[List[str]] = None
terms: Optional[List[str]] = None
Expand Down Expand Up @@ -251,6 +264,22 @@ def generate_mcp(
)
yield mcp

if self.institutional_memory and self.institutional_memory.elements:
mcp = MetadataChangeProposalWrapper(
entityUrn=self.urn,
aspect=InstitutionalMemoryClass(
elements=[
InstitutionalMemoryMetadataClass(
url=element.url,
description=element.description,
createStamp=self._mint_auditstamp("yaml"),
)
for element in self.institutional_memory.elements
]
),
)
yield mcp

# Finally emit status
yield MetadataChangeProposalWrapper(
entityUrn=self.urn, aspect=StatusClass(removed=False)
Expand Down
Loading