Save larger raster with zstd compression writes dirty block

[sehHeiden]

I want to save larger rasters. I often got a problem when the raster was too large.
I use zstd compression, for this example I tried both predictor 3 and 2 both had the same issue.

When the writing process reached: 4.194.304 KB I always get:

Traceback (most recent call last):
  File "rasterio\\_io.pyx", line 1782, in rasterio._io.DatasetWriterBase.write
  File "rasterio\\_io.pyx", line 213, in rasterio._io.io_multi_band
  File "rasterio\\_err.pyx", line 195, in rasterio._err.exc_wrap_int
rasterio._err.CPLE_AppDefinedError: An error occurred while writing a dirty block from GDALRasterBand::IRasterIO
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "D:...\venv\Lib\site-packages\IPython\core\interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-11-808977ff40aa>", line 1, in <module>
    mosaic9.rio.to_raster(
  File "...\venv\Lib\site-packages\rioxarray\raster_array.py", line 1125, in to_raster
    return RasterioWriter(raster_path=raster_path).to_raster(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\venv\Lib\site-packages\rioxarray\raster_writer.py", line 292, in to_raster
    rds.write(data, window=window)
  File "rasterio\\_io.pyx", line 1784, in rasterio._io.DatasetWriterBase.write
rasterio.errors.RasterioIOError: Read or write failed. An error occurred while writing a dirty block from GDALRasterBand::IRasterIO

Code Sample

mosaic9.rio.to_raster(
    out_path,
    compress="zstd",
    zstd_level=1,
    num_threads="all_cpus",
    tiled=True,
    dtype="float32",
    predictor=3,
    windowed=True,
  )

This did not happen without the compression.

mosaic9.rio.to_raster(
    out_path,

just successfully wrote 19 GB to disk. I did not ran into this problem with smaller raster.

Problem description

Problem, because large file sizes I why I turned compression on.

Environment Information

The rioxarray version is something older. But from history can't a hint that this behaviour has already been fixed.

rioxarray (0.15.1) deps:
rasterio: 1.3.9
xarray: 2024.1.1
GDAL: 3.6.4
GEOS: 3.11.2
PROJ: 9.0.1
PROJ DATA: ...\venv\Lib\site-packages\rasterio\proj_data
GDAL DATA: None
Other python deps:
scipy: 1.13.0
pyproj: 3.6.1
System:
python: 3.11.8 (tags/v3.11.8:db85d51, Feb 6 2024, 22:03:32) [MSC v.1937 64 bit (AMD64)]
executable: ...\venv\Scripts\python.exe
machine: Windows-10-10.0.22631-SP0

Installation method

• pypi

[snowman2]

Does adding bigtiff="YES" help?
See: https://gdal.org/drivers/raster/gtiff.html

mosaic9.rio.to_raster(
    out_path,
    compress="zstd",
    zstd_level=1,
    num_threads="all_cpus",
    tiled=True,
    dtype="float32",
    predictor=3,
    windowed=True,
    bigtiff="YES",
  )

[snowman2]

Possible duplicate #709

[sehHeiden]

Possible a duplicate. I tried it with lzw and got the same error, but without a trace (silent).

Setting bigtiff to "YES" did help!