Compressed data ended before the end-of-stream marker was reached

[wxbinge]

This error is easily resolved by redownloading the radar data, but I'm running this particular script operationally, so when this error occurs, the script exits and I would miss the cycle.

I was wondering is there a way to bypass this error?

I am pulling the radar data from the TGFTP server.

Traceback (most recent call last):
File "/wfs/radar/pyscripts/national_composite_archival.py", line 186, in
radar.append(pyart.io.read_nexrad_level3(i)) # Opening radar file and appending each file to an array
File "/anaconda3/envs/WFS/lib/python3.10/site-packages/pyart/io/nexradl3_read.py", line 78, in read_nexrad_level3
nfile = NEXRADLevel3File(prepare_for_read(filename, storage_options=storage_options))
File "/anaconda3/envs/WFS/lib/python3.10/site-packages/pyart/io/nexrad_level3.py", line 146, in init
buf2 = bz2.decompress(buf[bpos:])
File "/anaconda3/envs/WFS/lib/python3.10/bz2.py", line 341, in decompress
raise ValueError("Compressed data ended before the "
ValueError: Compressed data ended before the end-of-stream marker was reached

[zssherman]

@wxbinge Hmm would maybe a try and except, catch the error and rerun the download on the file work? or are you using pyarts nexrad reader to download the file?

[wxbinge]

I don't know why 'try and except' didn't cross my mind, but you're right. That will work to bypass the error. I'm using bash to download the radar files, so I'm going to need to do something creative to see if I can invoke a download from within the python script if the error arises.

[mgrover1]

@wxbinge that error occurs when the file is not completely downloaded.

See the stack overflow discussion here

You will need to add logic within your script to check to see if the file is completely downloaded yet, or consider streaming the data from Amazon Web Services (AWS), as we did in this example:

• NEXRAD AWS Example