[1pt] PR: Retrieve code to process NWS AHPS benchmarks for Alaska

[AliForghani-NOAA]

This PR brings back the preprocess_ahps_nws.py code to FIM4 and generates new AHPS benchmark datasets for sites SXRA2 and SKLA2 in Alaska. The new AHPS benchmark datasets are available on dev1 here: "/dev_fim_share/foss_fim/outputs/ali_ahps_alaska/AHPS_Results_Alaska/19020302/"

This PR closes issue #1130.

To process a new station, follow these steps:

1. Add the name of the new site (s) to the/data/inputs/ahps_sites/evaluated_ahps_sites.csv file.
2. Collect/Download the grid depth dataset, typically available as ESRI gdb.
3. Use arcpy (or ArcGIS pro ) to convert the grid depths (in ESRI gdb) into TIFF files
   • Make sure the TIFF files have crs
   • Store all the TIFF files in a directory called "depth_grid," which should be a sub-folder inside a folder named after the gage code (must be a 5-character code)
4. Run the script as described below. Note that sites in CONUS and Alaska cannot be mixed in a single run. Separate runs should be done for Alaska sites and CONUS sites.

Note that for the "SKLA2" site, the downloaded ESRI-GDB grid files had a datum issue. This issue was manually corrected during the conversion of GDB files into TIFF files.

Also, a manual modification of the outputs was needed for these sites in Alaska. The WRDS API had some issues for retrieving upstream/downstream feature ids of the gages for these Alaska sites (for example, for being used in "ahps_skla2_huc_19020302_flows_major.csv" file). So, this part was manually applied using QGIS.

Additions

• data/nws/preprocess_ahps_nws.py ... retrieved from previous versions of FIM and updated for shapely v2

Changes

• tools/tools_shared_functions.py ... updated for shapely v2

Testing

The new code was tested successfully for sites SXRA2 and SKLA2 in Alaska. The code was run as below:

python  foss_fim/data/nws/preprocess_ahps_nws.py 
-s inputPath    -d outputPath    -r  referenceRaster

• inputPath: path to a directory that includes the subdirectories "sxra2" and "skla2". Each of these subdirectories should contain a "depth_grid" folder with TIFF files as described above.
• referenceRaster: is the path to an arbitrary output TIFF file from a FIM run for an Alaska HUC . Note that for a site in CONUS, this referenceRaster must be from a FIM run for a CONUS HUC.

Below is a comparison for the site "sxra2" between the generated "major" (with stage =14ft) benchmark TIFF file (with EPSG:3338) and the corresponding plot available on the water.noaa.gov website.

Deployment Plan (For developer use)

How does the changes affect the product?

• Code only?
• Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
• If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
• Require new pre-clip set?
• Has new or updated python packages?
• If applicable, has a deployment plan be created with the deployment person/team?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• Passes all unit tests locally (inside interactive Docker container, at /foss_fim/, run: pytest unit_tests/)
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Reviewers requested
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations

Data Copy Checklist (For Technical Lead use only)

@RobHanna-NOAA

• EFS (/test_cases/nws_test_cases/validation_data_nws/19020302/)
• S3
• Dev1
• ESIP
• UCS2
• UCS3 - skipped (now skipping UCS3 for all input changes)

Also:

• A newer version from efs .... /inputs/ahps_sites/evaluated_ahps_sites.csv
• Done for all 5 enviros.

[AliForghani-NOAA]

@mluck Can you confirm that by having AHPS benchmark dataset for SXRA2 and SKLA2, now we can include these two sites into the list of sites in /data/inputs/ahps_sites/evaluated_ahps_sites.csv?

[AliForghani-NOAA]

@RyanSpies-NOAA
I have made new changes to the code to make sure the script creates "lid_domain.shp" files for each lid site. Also the new changes ensure having "extent" in the name of the threshold extent tiff files (e.g, "ahps_skla2_huc_19020302_extent_action.tif").

The new outputs of the script for the two site in Alaska are here:
"/dev_fim_share/foss_fim/outputs/ali_ahps_alaska/revised_AHPS_Results_Alaska/"

Note that the new outputs do not have your manual addition of upstream/downstream feature ids as you prepared on the previous version. I just created this new version to confirm the changes on the code-- if it is approved by you, we can go ahead and use your manually-revised dataset for prod.

[RyanSpies-NOAA]

Noting here that I manually performed the upstream/downstream tracing for SKLA2 & SXRA2 because the WRDS api is currently not configured to perform flowline tracing for Alaska sites.

SXRA2 feature_ids:
75004400003054
75004400010435
75004400001554
75004400001543
75004400003048
75004400000119
75004400001498
75004400006005
75004400000142
75004400007456
75004400007477
75004400005985
75004400008935
75004400003077
75004400003078
75004400010458
75004400001569
75004400007494
75004400000151
75004400005967
75004400000091
75004400005961
75004400008898
75004400000085
75004400001499
75004400000093
75004400008893
75004400005957
75004400001440
75004400004561
75004400001504
75004400000092
75004400003024
75004400001505
75004400003003
75004400003008
75004400000094
75004400007461
75004400010471
75004400005909
75004400007511
75004400008915
75004400003011
75004400008904
75004400008911
75004400008912
75004400010386

SKLA2 feature_ids:
75004400010513
75004400003154
75004400010514
75004400009023
75004400004617
75004400009016
75004400009015
75004400004615
75004400003133
75004400004590
75004400005909

[RyanSpies-NOAA]

I was able to run the new preprocess_ahps_nws.py code to generate the benchmark data and successfully ran an alpha test for the two new AK sites in 19020302.

[RyanSpies-NOAA]

Reviewed changes and completed code testing.