-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make table partitioning accessible in register/read APIS #1185
Comments
Hi guys, coming back to this now, since we can already find partition in paths in the object store, and looking at the API, I think what is missing is the ability to infer ListingOptions::table_partition_cols from the base path given to the TableProvider. |
@Igosuki I think the idea is to extend these public apis to allow users explicitly specify table partition columns (i.e. the I think there is value in making partition column statically typed as well instead of only treating them as strings. But this is out of scope for this issue. |
@houqp not sure I would tag this as |
I believe |
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The ability to read partitioned tables is added the the
ListingTable
in #1141. This is a very common feature that would be interesting to expose in the highest level APIs such asExecutionContext.read_xxx
andExecutionContext.register_xxx
Describe the solution you'd like
The cleanest solution would probably to have a partiotioning field in
CsvReadOptions
/AvroReadOptions
and also make read_parquet use aParquetReadOptions
read_parquet
/register_parquet
read_csv
/register_csv
read_avro
/register_avro
CREATE EXTERNAL TABLE
The text was updated successfully, but these errors were encountered: