Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[1pt] PR: Update FIM pipeline to process in Alaska #1106

Merged
merged 19 commits into from
Apr 17, 2024

Conversation

EmilyDeardorff
Copy link
Contributor

@EmilyDeardorff EmilyDeardorff commented Mar 28, 2024

This PR updates the FIM pipeline to be able to process HUCs in southern Alaska. Running FIM in southern Alaska requires that a different CRS and a few different files be used. Additionally, some of the Alaska HUCs displayed an issue where the input stream density was too high, so this update introduces some logic to adjust the threshold of stream orders to exclude based on whether an Alaska HUC is listed as high or medium-high stream density.

Changes

data/wbd/generate_pre_clip_fim_huc8.py

  • Adjusted comment.

src/bash_variables.env

  • Changed pre-clip HUC 8 directory to be a folder with both Alaska and CONUS HUCs.

src/check_huc_inputs.py

  • Changed the included_huc_list variable to refer to a HUC list that includes Alaska.

src/derive_level_paths.py

  • Add in logic to exclude different stream orders based on whether the HUC falls into the high or medium-high density HUC lists.

src/run_by_branch.sh

  • Add in logic to check whether the HUC is in Alaska or not and to use the correct CRS accordingly.

src/run_unit_wb.sh

  • Add in logic to check whether the HUC is in Alaska or not and to use the correct CRS and DEM domain filename accordingly.

src/utils/shared_variables.py

  • Add the Alaska CRS, a list of high stream density HUCs, and a list of medium-high stream density HUCs.

Testing

  • Ran successfully on all southern Alaska HUCs (19020202, 19020301, 19020602, 19020203, 19020302, 19020800, 19020201, 19020104, 19020601, 19020401, 19020103, 19020402, 19020505, 19020503, 19020504, 19020102, 19020101, 19020502, 19020501)
  • Ran successfully on CONUS HUC 12040101 (in a combined run with 2 Alaska HUCs)

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

  • Informative and human-readable title, using the format: [_pt] PR: <description>
  • Links are provided if this PR resolves an issue, or depends on another other PR
  • If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
  • Changes are limited to a single goal (no scope creep)
  • The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
  • pre-commit hooks were run locally
  • Any change in functionality is tested
  • Passes all unit tests locally (inside interactive Docker container, at /foss_fim/, run: pytest unit_tests/)
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • CHANGELOG updated with template version number, e.g. 4.x.x.x
  • Reviewers requested
  • Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

  • Update CHANGELOG with latest version number and merge date
  • Update the Citation.cff file to reflect the latest version number in the CHANGELOG
  • If applicable, update README with major alterations

@EmilyDeardorff EmilyDeardorff changed the title WIP Update FIM pipeline to process in Alaska [1pt] PR: Update FIM pipeline to process in Alaska Apr 3, 2024
@EmilyDeardorff EmilyDeardorff added the enhancement New feature or request label Apr 3, 2024
@EmilyDeardorff EmilyDeardorff marked this pull request as ready for review April 3, 2024 23:40
@RobHanna-NOAA
Copy link
Contributor

Inputs to be copied from EFS to various locations (S3, ESIP, and the two OWP servers):

inputs/3dep_dems/10m_South_Alaska/23_11_07/
inputs/huc_lists/dev_small_test_8_Alaska_and_CONUS.lst
inputs/huc_lists/included_huc8_withAlaska.lst
inputs/huc_lists/SouthernAlaska_HUC8.lst
inputs/huc_lists/uat_and_alpha_domain_huc_list_w_Alaska.lst
inputs/landsea/water_polygons_alaska.gpkg
inputs/nld_vectors/nwm_flows_alaska_nwmv3_ID_v2.gpkg
inputs/nld_vectors/3d_nld_preprocessed_230314_3338.gpkg
inputs/nld_vectors/Leveed_Areas_NLDFS_3338_230314.gpkg
inputs/nld_vectors/System_Routes_NLDFS_3338_230314.gpkg
inputs/nwm_hydrofabric/nwm_catchments_alaska.gpkg
inputs/nwm_hydrofabric/nwm_flows_alaska_nwmV3.gpkg
inputs/nwm_hydrofabric/nwm_flows_alaska_nwmV3_ID.gpkg
inputs/nwm_hydrofabric/nwm_flows_dropz.gpkg (not sure if we need it, but copy anyways)
inputs/nwm_hydrofabric/nwm_headwaters_alaska.gpkg
inputs/nwm_hydrofabric/nwm_waterbodies_alaska.gpkg
inputs/pre_clip_huc8/24_4_3/
inputs/wbd/WBD_National_South_Alaska.gpkg
inputs/wbd/HUC8_South_Alaska/

hhs732
hhs732 previously approved these changes Apr 16, 2024
Copy link
Contributor

@hhs732 hhs732 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR was tested on one non-alaska HUC (12090301) and four Alaska HUCs (19020504, 19020202, 19020301, 19020101) successfully. inundation_nation.py was run to map the results and they look great.

@RobHanna-NOAA
Copy link
Contributor

All inputs pushed to all enviros. :)

hhs732
hhs732 previously approved these changes Apr 17, 2024
@CarsonPruitt-NOAA CarsonPruitt-NOAA merged commit 3ca0ceb into dev Apr 17, 2024
1 check passed
@CarsonPruitt-NOAA CarsonPruitt-NOAA deleted the dev-fim-pipeline-alaska branch April 17, 2024 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
4 participants