Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ingest/great_expectations): support in-memory (Pandas) data assets #9811

Merged
merged 1 commit into from
Apr 4, 2024
Merged

feat(ingest/great_expectations): support in-memory (Pandas) data assets #9811

merged 1 commit into from
Apr 4, 2024

Conversation

bouaouda-achraf
Copy link
Contributor

@bouaouda-achraf bouaouda-achraf commented Feb 11, 2024 •
edited by hsheth2
Loading

Closes #9810

Describe the feature
Adding the ability to publish metadata related to in-memory Data Assets .

Context
Use it when the data Source is not implemented yet ( victoria metrics , ..... ) .

Expected behavior
Assertion metadata are pushed to Datahub.

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata community-contribution PR or Issue raised by member(s) of DataHub Community labels Feb 11, 2024
hsheth2
hsheth2 reviewed Feb 12, 2024
View reviewed changes
Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good

Can you add a test to tests/unit/test_great_expectations_action.py?

@bouaouda-achraf
Copy link
Contributor Author

The code looks good

Can you add a test to tests/unit/test_great_expectations_action.py?

Ok i will do it ( and also rebase with master ) .
thanks 🙏

@hsheth2 hsheth2 added the pending-submitter-response Issue/request has been reviewed but requires a response from the submitter label Feb 22, 2024
@bouaouda-achraf bouaouda-achraf force-pushed the great-expectations-pandas-support branch from 059e6ca to 0c96f40 Compare February 25, 2024 07:48
@bouaouda-achraf bouaouda-achraf force-pushed the great-expectations-pandas-support branch from 0c96f40 to d338b46 Compare February 25, 2024 10:49
@bouaouda-achraf bouaouda-achraf marked this pull request as draft February 25, 2024 11:14
@bouaouda-achraf bouaouda-achraf force-pushed the great-expectations-pandas-support branch from d338b46 to 8d45e98 Compare February 25, 2024 23:08
@bouaouda-achraf bouaouda-achraf marked this pull request as ready for review February 26, 2024 10:04
@bouaouda-achraf bouaouda-achraf force-pushed the great-expectations-pandas-support branch 2 times, most recently from 8d45e98 to 7ce4d85 Compare February 26, 2024 10:10
hsheth2
hsheth2 reviewed Feb 27, 2024
View reviewed changes
Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly looks good, just need to tighten up the testing a bit

metadata-ingestion/tests/unit/test_great_expectations_action.py Show resolved Hide resolved
@bouaouda-achraf bouaouda-achraf force-pushed the great-expectations-pandas-support branch 2 times, most recently from dde0969 to 0a43794 Compare March 9, 2024 12:28
@bouaouda-achraf bouaouda-achraf marked this pull request as draft March 9, 2024 12:29
@bouaouda-achraf bouaouda-achraf marked this pull request as ready for review March 9, 2024 12:30
@bouaouda-achraf bouaouda-achraf requested a review from hsheth2 March 9, 2024 12:30
@hsheth2 hsheth2 added merge-pending-ci A PR that has passed review and should be merged once CI is green. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Mar 11, 2024
@bouaouda-achraf bouaouda-achraf force-pushed the great-expectations-pandas-support branch from 0a43794 to 68c9976 Compare March 17, 2024 21:45
@bouaouda-achraf bouaouda-achraf force-pushed the great-expectations-pandas-support branch from 68c9976 to 620234d Compare March 18, 2024 21:25
@hsheth2
Copy link
Collaborator

hsheth2 commented Mar 21, 2024

@bouaouda-achraf looks like there's a lint issue here

@bouaouda-achraf bouaouda-achraf force-pushed the great-expectations-pandas-support branch from 620234d to 7b4afa2 Compare March 21, 2024 21:38
@bouaouda-achraf
Copy link
Contributor Author

@bouaouda-achraf looks like there's a lint issue here

fixed @hsheth2

@hsheth2 hsheth2 changed the title feat: [great_expectations] supporting in-memory (Pandas) Data Assets … feat(ingest/great_expectations): support in-memory (Pandas) data assets Mar 21, 2024
@hsheth2 hsheth2 merged commit b5615fa into datahub-project:master Apr 4, 2024
53 checks passed
sleeperdeep pushed a commit to sleeperdeep/datahub that referenced this pull request Jun 25, 2024
@bouaouda-achraf @sleeperdeep
feat(ingest/great_expectations): support in-memory (Pandas) data asse…
e08a438 
…ts (datahub-project#9811)

Co-authored-by: Achraf BOUAOUDA <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hsheth2 hsheth2 hsheth2 approved these changes

Assignees

@hsheth2 hsheth2

Labels
community-contribution PR or Issue raised by member(s) of DataHub Community ingestion PR or Issue related to the ingestion of metadata merge-pending-ci A PR that has passed review and should be merged once CI is green.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[great_expectations] supporting in-memory (Pandas) Data Assets
2 participants
@bouaouda-achraf @hsheth2