-
-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG: fix read_parquet
with dask>=2022.12.0
#230
BUG: fix read_parquet
with dask>=2022.12.0
#230
Conversation
…`, `dask_geopandas.io.arrow.ArrowDatasetEngine._arrow_table_to_pandas` to be compatible with `dask>=2022.12.0`
…table_to_pandas` methods
There are two failing tests at the moment. One will be fixed by #229 (and is unrelated to the parquet IO), and the other is a case that should error but is giving a wrong error message. But that indeed seems to be related to the lines you link to ( |
And thanks for opening the PR! |
You're right, I was confused as to why that other test was failing, glad there's a PR for that one already. Thanks for checking it out! |
read_parquet
with dask>=2022.12.0
read_parquet
with dask>=2022.12.0
We can probably further improve this and make the integration more robust, but let's already merge this so we can have green CI and have this running with latest dask. Thanks again @the-matt-morris! |
closes #225
Updates method signature of the following so that they are compatible with all versions of
dask
, includingdask>=2022.12.0
:dask_geopandas.io.arrow.ArrowDatasetEngine._arrow_table_to_pandas
dask_geopandas.io.arrow.GeoDatasetEngine._arrow_table_to_pandas
dask_geopandas.io.parquet.GeoArrowEngine._create_dd_meta
It's been a little while since I played around with this, but I had written previously that these lines in dask are problematic, as it calls
_arrow_table_to_pandas
, where it wasn't called prior todask==2022.12.0
.At the time, I thought we may need an update to
GeoDatasetEngine._arrow_table_to_pandas
to handle this, but I feel like someone with a better grasp on this library's internals may have a better idea. Either way, it is known that this PR in its current state is an attempt to solve the problem, but it doesn't succeed entirely in doing that, as you'll see with one of the unit tests failing.