Parallel raster writing using dask and rioxarray

[GrahamReveley]

Is there a way to write compressed rasters in parallel from rioxarray in a robust way?

Every time I've done this (example in the below code) I've ended up with GDAL throwing errors. I've also seen no errors thrown and chunks being written incorrectly (no data/fill values being added into the rasters)

^^^
#lots of delayed dask processing happening beforehand

result.rio.to_raster("output.tif", 
tiled=True, 
windowed=True, 
lock=SerializableLock(), 
compress="LZW", 
predictor=2, 
num_threads="ALL_CPUS")

If anyone has had good experience on this I'd be happy to have some pointers on this. For reference, we're dealing with very very large rasters (could be single or multi band) at 90m globally potentially. It would be great to be able to leverage the full potential of dask/xarray/rioxarray/gdal to output these things faster and more efficiently.

Thanks!

[snowman2]

Try this:

result.rio.to_raster(
    "output.tif",
    tiled=True,
    lock=<see https://corteva.github.io/rioxarray/stable/examples/dask_read_write.html to determine the best lock for your use case. SerializableLock won't work.>,
    compress="LZW",
    predictor=2,
)

Removed args:

• windowed=True (this is for writing with a single threaded application to use less memory (not for dask))
• num_threads="ALL_CPUS" (this is probably best used with a single threaded application and not with dask)

[GrahamReveley]

I've attempted this solution with the dask multi-worker lock (below)

result.rio.to_raster("output.tif", tiled=True, lock=Lock('rio',client=client) compress="LZW", predictor=2, )

and it doesn't seem to work, it just loads the entire tif into memory and kills the computer I'm working on. This lock seems to be the one I want to use as I'm running this on a dask cluster. I'll attempt the threading lock but since this is a multi-worker example I don't think it will bare any fruit.

Note: The serializable lock does in fact work when compression is not used.

[snowman2]

Note: The serializable lock does in fact work when compression is not used.

See: #220