Implement xarray dataset raster return for cop30 and worldcover datasets #30
Conversation
Fixed merge conflict between remote branches local vs cryocloud on updated main.py search
|
Thanks for starting this @Jack-Hayes ! If adding a new run-time dependency (odc.stac) you'll want to add it here (with version constraints) Line 33 in f366a45 And then also lower in that file (with "*") so that pixi recognizes it: Line 211 in f366a45 I moved the 'contributing' guide from the readme to this page in the docs recently https://coincident.readthedocs.io/en/latest/user_guide/contribute.html |
src/coincident/search/main.py
Outdated
|
|
||
| # Used to access formatters | ||
| from pystac_client.item_search import ItemSearch as _ItemSearch | ||
| from shapely.validation import make_valid |
There was a problem hiding this comment.
Nice. I didn't know about this function. Currently, the code just checks for ccw ordering
coincident/src/coincident/search/main.py
Lines 80 to 82 in f366a45
Out of curiosity, were did you encounter an invalid geometry where this was needed?
There was a problem hiding this comment.
A lot of flight extent polygons from the NOAA Costal LiDAR catalog are snake-like and non-contiguous which resulted 'invalid' polygons even after a simplify() with a tolerance of 0.01 (and I believe some of the really bad flight extent polygons were still invalid after simplify(0.01).make_valid(), so I buffered those by a small amount before searching for PCD sites).
There were also issues with simplify(0.01) making holes in some of the NEON flight extent polygons which were created by merging all of the tile extents in the flight, and make_valid() fixed some of these (see one of our Slack threads on this!). I ended up using a minor buffer on these as well for initial PCD site searches rather than simplify.
There was a problem hiding this comment.
Gotcha thanks! Could leave this for a separate PR, but if including, you could add one of these polygons to test (can add as a geojson in a 'tests/data' subfolder).
src/coincident/search/main.py
Outdated
| max(y for x, y in coordinates), | ||
| ) | ||
| items = stac.to_pystac_items(gf) | ||
| ds = odc.stac.load( |
There was a problem hiding this comment.
This will be slow for large areas because by default chunks=None. chunks='auto'chunks={} requires dask, but should to be much faster.
There was a problem hiding this comment.
Ah, great point 😊 I'm assuming we'll eventually add support for dask so I might as well incorporate it here?
Related: #22
Related: #23
This PR implements returning an xarray.core.dataset.Dataset raster for cop30 and worldcover datasets rather than the respective item metadata. There are pretty big issues with efficiency in terms of loading in these rasters for large AOIs. I haven't implemented any GDAL enhancements (OSGeo/gdal#11317) or addressed odc-stac for accepting the geodataframe/stac-geoparquet directly (I'm still using to_pystac_items here).