feat: Add delta_scan function
#1345
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
```
〉select * from delta_scan('./testdata/delta/table1/') order by a
〉;
+---+-------+
| a | b |
+---+-------+
| 1 | hello |
| 2 | world |
+---+-------+
```
universalmind303
approved these changes
Jul 18, 2023
vrongmeal
approved these changes
Jul 19, 2023
Comment on lines
+17
to
+22
| # # Relative path | ||
| # query IT | ||
| # select * from delta_scan('./testdata/delta/table1/') order by a; | ||
| # ---- | ||
| # 1 hello | ||
| # 2 world |
There was a problem hiding this comment.
So it works outside the slt tests, but for whatever reason, these sorts of relative paths don't seem to be working when running in an slt.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Supports querying delta tables from local paths, s3, and gcs. I kept the implementation separate from the other object store functions since constructing the accessor is a bit different for delta. Delta tables also only exist at one path, it doesn't make sense to support passing in more than one url.
Test data uploaded to s3 and gcs as appropriate. Currently the test data for delta consists of a single table made up of two parquet files and 3 json metadata logs.