Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(s3): add s3 source #4490

Merged
merged 6 commits into from
Mar 29, 2022
Merged

feat(s3): add s3 source #4490

merged 6 commits into from
Mar 29, 2022

Conversation

MugdhaHardikar-GSLab
Copy link
Contributor

@MugdhaHardikar-GSLab MugdhaHardikar-GSLab commented Mar 24, 2022 •
edited
Loading

Added features:

  • Add containers for datasets
  • Support platform_instance
  • Support for folder level datasets
  • Increased flexibility to specify dataset paths

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable)

@github-actions
Copy link

github-actions bot commented Mar 24, 2022 •
edited
Loading

Unit Test Results (build & test)

  93 files  ±0    93 suites  ±0   19m 45s ⏱️ + 6m 58s
676 tests ±0  617 ✔️ ±0  59 💤 ±0  0 ❌ ±0 

Results for commit 8c7594c. ± Comparison against base commit 34b36c0.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Mar 24, 2022 •
edited
Loading

Unit Test Results (metadata ingestion)

       5 files         5 suites   58m 31s ⏱️
   388 tests    388 ✔️   0 💤 0 ❌
1 787 runs  1 756 ✔️ 31 💤 0 ❌

Results for commit 8c7594c.

♻️ This comment has been updated with latest results.

@maggiehays maggiehays assigned rslanka and hsheth2 and unassigned hsheth2 Mar 28, 2022
@MugdhaHardikar-GSLab MugdhaHardikar-GSLab changed the title feat(data-lake): add containers and folder level dataset support feat(s3): add s3 connector source Mar 29, 2022
@MugdhaHardikar-GSLab MugdhaHardikar-GSLab changed the title feat(s3): add s3 connector source feat(s3): add s3 source Mar 29, 2022
treff7es
treff7es approved these changes Mar 29, 2022
View reviewed changes
Copy link
Contributor

@treff7es treff7es left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@treff7es treff7es merged commit 37aedfc into datahub-project:master Mar 29, 2022
maggiehays pushed a commit to maggiehays/datahub that referenced this pull request Aug 1, 2022
@MugdhaHardikar-GSLab @mayurinehate @maggiehays
feat(s3): add s3 source (datahub-project#4490)
bf22c80 
* feat(data-lake): add containers and folder level dataset support

* docs(data-lake): Update readme for data lake

* doc(data-lake): fix examples, update doc

* lint fix

* feat(s3): add s3 source, restore old data-lake source

Co-authored-by: Mayuri N <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@treff7es treff7es treff7es approved these changes

Assignees

@rslanka rslanka

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@MugdhaHardikar-GSLab @treff7es @rslanka @hsheth2 @mayurinehate