[1pt] PR: Change Post proc perms updates

[RobHanna-NOAA]

Prior to this fix, fim_post_processing.sh took just under 4 hours to reset permissions on all files and folder under the entire run. On closer inspection, it was updating permissions for all HUC folders where were already correct. A few other folders needed to have permission updates added. This will speed that up significantly.

Also, use this opportunity to added a new note to hash_compare.py and fix an annoying duration time showing milliseconds.

Changes

• fim_pipeline.sh: fix duration msgs.
• fim_post_processing.sh: permissions reset fix, a bit of output cleanup and fix duration msgs.
• src
  • bash_functions.env: update the Calc duration to allow for a msg prefix to be added to the duration calcs. Also adjusted the duration message to show hours as well, previously only min and seconds.
  • run_by_branch.sh: fix duration msgs.
  • run_unit_wb.sh: fix duration msgs.
  • src\src_adjust_ras2fim_rating.py: minor duration display msg change.
• tools\hash_compare.py: Added note

Testing

• Single HUC pipeline, so permissions duration savings not evaluated.

To validate the correct format of output for the duratoin system, you can open docker and go to foss_fim, then run some commands straight in the console:

ie)
cd foss_fim
source src/bash_functions.env

start_time=date +%s

adj_time=$(($start_time - (60 * 5)))
Calc_Duration "Duration testing : " $adj_time

adj_time=$(($start_time - (60 * 60 * 24)))
Calc_Duration "testing " $adj_time

Calc_Duration "testing " "Thu Aug 1 16:56:31 UTC 2024"

Also ran another small huc to validate all duration outputs.

Deployment Plan (For developer use)

How does the changes affect the product?

• Code only?
• If applicable, has a deployment plan be created with the deployment person/team?
• Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
• If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
• Require new pre-clip set?
• Has new or updated python packages?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• n/a New functions are documented (with a description, list of inputs, and expected output)
• n/a Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations

[mluck]

Looks good. Ran fim_pipeline.sh on 3 HUCs: 03100204, 12040101, and 12090301. The time to reset permissions was greatly reduced from 8 seconds to 2 seconds.

On dev

Resetting Permissions
Duration = 0 min(s) and 8 sec

On dev-perm-reset

Resetting Permissions
Duration = 0 min(s) and 2 sec

In addition, since there's already a fix involving time, it might be a good chance to clean up one more thing. It looks like there's an extra message line that doesn't contain any information: Total Duration is ...

======================== End of fim_pipeline for dev ==========
Thu Aug  1 14:46:19 UTC 2024
Total Duration is ...
Duration : 59 min(s) and 33 sec

[RobHanna-NOAA]

Looks good. Ran fim_pipeline.sh on 3 HUCs: 03100204, 12040101, and 12090301. The time to reset permissions was greatly reduced from 8 seconds to 2 seconds.

On dev

Resetting Permissions
Duration = 0 min(s) and 8 sec

On dev-perm-reset

Resetting Permissions
Duration = 0 min(s) and 2 sec

In addition, since there's already a fix involving time, it might be a good chance to clean up one more thing. It looks like there's an extra message line that doesn't contain any information: Total Duration is ...

======================== End of fim_pipeline for dev ==========
Thu Aug  1 14:46:19 UTC 2024
Total Duration is ...
Duration : 59 min(s) and 33 sec

Ya.. I knew that one before. It is deceiving and will fix that quickly too.

[mluck]

Tested changes, looks good.