Merge branch 'dev' into dev-pyogrio

docs/CHANGELOG.md

@@ -1,7 +1,8 @@
 All notable changes to this project will be documented in this file.
 We follow the [Semantic Versioning 2.0.0](http://semver.org/) format.
 
-## v4.4.x.x - 2024-02-13 - [PR#1077](https://github.com/NOAA-OWP/inundation-mapping/pull/1077)
+
+## v4.4.11.0 - 2024-02-16 - [PR#1077](https://github.com/NOAA-OWP/inundation-mapping/pull/1077)
 
 Replace `fiona` with `pyogrio` to improve I/O speed. `geopandas` will use `pyogrio` by default starting with version 1.0. `pyarrow` was also added as an environment variable to further speedup I/O. As a result of the changes in this PR, `fim_pipeline.sh` runs approximately 10% faster.
 
@@ -19,6 +20,24 @@ Replace `fiona` with `pyogrio` to improve I/O speed. `geopandas` will use `pyogr
 
 <br/><br/>
 
+
+## v4.4.10.1 - 2024-02-16 - [PR#1075](https://github.com/NOAA-OWP/inundation-mapping/pull/1075)
+
+We recently added code to fim_pre_processing.sh that checks the CPU count. Earlier this test was being done in post-processing and was killing a pipeline that had already been running for a while.
+
+Fix:
+- Removed the CPU test from pre-processing. This puts us back to it possibly failing in post-processing but we have to leave it for now. 
+- Exit status codes (non 0) are now returned in pre-processing and post-processing when an error has occurred.
+
+Tested that the a non zero return exit from pre-processing shuts down the AWS step functions.
+
+### Changes
+- `fim_pre_processing.sh`: added non zero exit codes when in error, plus removed CPU test
+- `fim_post_processing.sh`:  added non zero exit codes when in error
+
+<br/><br/>
+
+
 ## v4.4.10.0 - 2024-02-02 - [PR#1054](https://github.com/NOAA-OWP/inundation-mapping/pull/1054)
 
 Recent testing exposed a bug with the `acquire_and_preprocess_3dep_dems.py` script. It lost the ability to be re-run and look for files that were unsuccessful earlier attempts and try them again. It may have been lost due to confusion of the word "retry". Now "retry" means restart the entire run. A new flag called "repair"  has been added meaning fix what failed earlier.  This is a key feature it is common for communication failures when calling USGS to download DEMs.  And with some runs taking many hours, this feature becomes important.

fim_post_processing.sh

@@ -46,6 +46,7 @@ if [ "$runName" = "" ]
 then
     echo "ERROR: Missing -n run time name argument"
     usage
+    exit 22
 fi
 
 outputDestDir=$outputsDir/$runName
@@ -217,6 +218,7 @@ Tcount
 date -u
 
 find $outputDestDir -type d -exec chmod -R 777 {} +
+find $outputDestDir -type f -exec chmod -R 777 {} +
 
 echo
 echo "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"

fim_pre_processing.sh

@@ -46,7 +46,6 @@ usage()
       -skipcal          : If this param is included, the S.R.C. will be updated via the calibration points.
                             will be skipped.
     "
-    exit
 }
 
 set -e
@@ -104,16 +103,20 @@ in
     shift
 done
 
+# exit 22 means bad argument
+
 # print usage if arguments empty
 if [ "$hucList" = "" ]
 then
     echo "ERROR: Missing -u Huclist argument"
     usage
+    exit 22
 fi
 if [ "$runName" = "" ]
 then
     echo "ERROR: Missing -n run time name argument"
     usage
+    exit 22
 fi
 
 # outputsDir & workDir come from the Dockerfile
@@ -137,6 +140,7 @@ then
     # NONE is not case sensitive
     echo "Error: The -ud <unit deny file> does not exist and is not the word NONE"
     usage
+    exit 22
 fi
 
 # validate and set defaults for the deny lists
@@ -148,6 +152,7 @@ then
     # NONE is not case sensitive
     echo "Error: The -bd <branch deny file> does not exist and is not the word NONE"
     usage
+    exit 22
 fi
 
 # We do a 1st cleanup of branch zero using branchZeroDenylist (which might be none).
@@ -164,6 +169,7 @@ then
     then
         echo "Error: The -zd <branch zero deny file> does not exist and is not the word NONE"
         usage
+        exit 22
     else
         # only if the deny branch zero has been overwritten and file exists
         has_deny_branch_zero_override=1
@@ -178,17 +184,7 @@ if [ -d $outputDestDir ] && [ $overwrite -eq 0 ]; then
     echo "ERROR: Output dir $outputDestDir exists. Use overwrite -o to run."
     echo
     usage
-fi
-
-# Test to ensure we are not overuseing cores
-num_available_cores=$(echo $(grep -c processor /proc/cpuinfo))
-let total_requested_jobs=$jobHucLimit*$jobBranchLimit
-if [[ $total_requested_jobs -gt $num_available_cores ]]; then
-    echo
-    echo "ERROR: There are $num_available_cores available, but -jh (jobHucLimit) * -jb (jobBranchLimit)"\
-         "exceed the number of available cores"
-    echo
-    usage
+    exit 22
 fi
 
 ## SOURCE ENV FILE AND FUNCTIONS ##