[8pt] Missing catchment reaches with crosswalk files

[RobHanna-NOAA]

During a full BED of 4.5.2.5, over 500 branches triggered the following error:

./logs/unit/16040203_unit.log:8074:fiona._err.CPLE_OpenFailedError: /fim_temp/fim_4_5_2_5/16040203/branches/8954000027/gw_catchments_reaches_filtered_addedAttributes_crosswalked_8954000027.gpkg: No such file or directory

Log details:

Burn in bridges 16040203 8954000027
/usr/local/lib/python3.10/dist-packages/rasterstats/io.py:335: NodataWarning: Setting nodata to -999; specify nodata explicitly
  warnings.warn(
Traceback (most recent call last):
  File "fiona/_shim.pyx", line 83, in fiona._shim.gdal_open_vector
  File "fiona/_err.pyx", line 291, in fiona._err.exc_wrap_pointer
fiona._err.CPLE_OpenFailedError: /fim_temp/fim_4_5_2_5/16040203/branches/8954000027/gw_catchments_reaches_filtered_addedAttributes_crosswalked_8954000027.gpkg: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/foss_fim/src/heal_bridges_osm.py", line 152, in <module>
    process_bridges_in_huc(**args)
  File "/foss_fim/src/heal_bridges_osm.py", line 63, in process_bridges_in_huc
    catchments_df = gpd.read_file(catchments)
  File "/usr/local/lib/python3.10/dist-packages/geopandas/io/file.py", line 297, in _read_file
    return _read_file_fiona(
  File "/usr/local/lib/python3.10/dist-packages/geopandas/io/file.py", line 338, in _read_file_fiona
    with reader(path_or_bytes, **kwargs) as features:
  File "/usr/local/lib/python3.10/dist-packages/fiona/env.py", line 408, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/fiona/__init__.py", line 264, in open
    c = Collection(path, mode, driver=driver, encoding=encoding,
  File "/usr/local/lib/python3.10/dist-packages/fiona/collection.py", line 162, in __init__
    self.session.start(self, **kwargs)
  File "fiona/ogrext.pyx", line 540, in fiona.ogrext.Session.start
  File "fiona/_shim.pyx", line 90, in fiona._shim.gdal_open_vector
fiona.errors.DriverError: /fim_temp/fim_4_5_2_5/16040203/branches/8954000027/gw_catchments_reaches_filtered_addedAttributes_crosswalked_8954000027.gpkg: No such file or directory
Command exited with non-zero status 1

---

This was detected even before running post processing on that output folder, but running the command line script grep scanning logs for errors. ie) grep -H -r -i -n "error" ./logs/ > ./all_errors_from_logs.log. That error file results are in the 4.5.2.5 output folder.

I will now run post-processing to see if other types of errors are discovered.

[CarsonPruitt-NOAA]

I was able to replicate the issue on this branch. The error being thrown is obviously due to the absence of the crosswalked catchments dataset gw_catchments_reaches_filtered_addedAttributes_crosswalked_8954000027.gpkg, but it appears that there is some sort of "quiet error" upstream. I bet it has to do with the recent changes to add_crosswalk.py in #1205.

[mluck]

Yes, it's because add_crosswalk.py is exiting here https://github.com/NOAA-OWP/inundation-mapping/blob/dev/src/add_crosswalk.py#L92. It should be handled via a fim_exit code.

[RobHanna-NOAA]

Yes, it's because add_crosswalk.py is exiting here https://github.com/NOAA-OWP/inundation-mapping/blob/dev/src/add_crosswalk.py#L92. It should be handled via a fim_exit code.

Optional... If you want it logged somewhere, you can create a new fim_exit_codes and then add that code to process_branch so it knows how to log or drop it. If you even want to log it. Just putting it to screen will never get seen. At a min, just put a note in the code about it.

[mluck]

There's some abnormal stuff happening that I'm still looking into, but ultimately none of the midpoints of the DEM-derived reaches (red) are within 100 meters of the NWM stream lines (blue) so no streams are returned in the crosswalk

[CarsonPruitt-NOAA]

How is the stream being routed uphill there??

[mluck]

That's part of the abnormality I'm looking into...it's an example of reverse flow which we hadn't seen in a while. You can see the burned in streamline exiting in the bottom of the image.

[mluck]

It looks like the flowdir grid for that branch gets corrupted in gagewatershed which affects the DEM-derived reaches (red) and the gw_catchments (purple lines in flowdir). The DEM appears to be correct. Interesting that we can replicate this problem and that it only seems to happen for select branches.

DEM

flowdir

[mluck]

The strangeness in the flowdir is being caused by a dry lake/valley bed (gray area) being raised to a constant elevation in the depression-filling process. Blue lines are NWM level paths and end at sinks (to=0).

[CarsonPruitt-NOAA]

Ah, that makes sense. I don't think this is impactful or widespread enough to try and fix the routing, but we should definitely take care of the exit 1 status for this branch. @RobHanna-NOAA Was this the only one in the whole BED that had this problem?

[mluck]

It sounds like there are lots (the original issue mentions > 500 branches). I'm not surprised this happens since if (when) the DEM reach midpoint isn't spatially aligned within 100 meters of the NWM stream (and we know that does happen) then the crosswalk will be empty and result in this error being triggered later in the pipeline.

[mluck]

I'll add NO_VALID_CROSSWALKS to FIM_exit_codes

[RobHanna-NOAA]

No. This bug, as well as the new card from this morning 1215 src optimization error for hydro ID, both showed up in a test UAT run done on UCS3 yesterday. It was based on the new 4.5.2.5 code.