merge with the current dev.

Merge remote-tracking branch 'origin/dev' into dev-bathymetry-datum-issue

.github/PULL_REQUEST_TEMPLATE.md

@@ -9,6 +9,16 @@
 ### Testing
 
 
+### Deployment Plan (For developer use)
+
+_How does the changes affect the product?_
+- [ ] Code only?
+- [ ] If applicable, has a deployment plan be created with the deployment person/team?
+- [ ] Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
+- [ ] If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
+- [ ] Require new pre-clip set?
+- [ ] Has new or updated python packages?
+
 ### Issuer Checklist (For developer use)
 
 _You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code._
@@ -20,11 +30,9 @@ _You may update this checklist before and/or after creating the PR. If you're un
 - [ ] The feature branch you're submitting as a PR is up to date (merged) with the latest `dev` branch
 - [ ] `pre-commit` hooks were run locally
 - [ ] Any _change_ in functionality is tested
-- [ ] Passes all unit tests locally (inside interactive Docker container, at `/foss_fim/`, run: `pytest unit_tests/`)
 - [ ] New functions are documented (with a description, list of inputs, and expected output)
 - [ ] Placeholder code is flagged / future todos are captured in comments
 - [ ] [CHANGELOG](/docs/CHANGELOG.md) updated with template version number, e.g. `4.x.x.x`
-- [ ] [Reviewers requested](https://help.github.com/articles/requesting-a-pull-request-review/)
 - [ ] Add yourself as an [assignee](https://docs.github.com/en/issues/tracking-your-work-with-issues/assigning-issues-and-pull-requests-to-other-github-users) in the PR  as well as the FIM Technical Lead
 
 ### Merge Checklist (For Technical Lead use only)

.gitignore

@@ -13,5 +13,4 @@ config/**
 config/*.swp
 .vscode/
 **/.DS_Store
-**/*_pytest.py
 .private/

CONTRIBUTING.md

@@ -25,8 +25,7 @@ guidance below.
 ## Changing the code-base
 
 Generally speaking, you should fork this repository, make changes in your
-own fork, and then submit a pull request. All new code should have associated
-unit tests (added to `/unit_tests`) that validate implemented features and the presence or lack of defects.
+own fork, and then submit a pull request. 
 Additionally, the code should follow any stylistic and architectural guidelines
 prescribed by the project. In the absence of such guidelines, mimic the styles
 and patterns in the existing code-base.
@@ -45,7 +44,13 @@ If you would like to contribute, please follow these steps:
     ```
     git checkout -b <dev-your-bugfix-or-feature>
     ```
-4. Pre-commit installation:
+
+4. Build the Docker container (if not already done in the [README.md](./README.md) page.):
+    ```
+    Docker build -f Dockerfile -t <image_name>:<tag> <path/to/repository>
+    ```
+
+5. Pre-commit installation:
 
    [pre-commit](https://pre-commit.com/) is used to run auto-formatting and enforce styling rules.
    It is a critical part of development and is enforced at the 'git commit' step. Key tools are included **inside the docker container** if you want to execute the correctly configured linting and formatting command line executables there. If you intend to execute `flake8`, `black` or `isort` from the command line **outside of the docker container**, additional configuration and installation is required, which will not be described here.
@@ -75,7 +80,7 @@ If you would like to contribute, please follow these steps:
    It should respond with the phrase *pre-commit 3.6.0* (version may not be exact).
 
 
-5. pre-commit configuration:
+6. pre-commit configuration:
 
    Now, you need to configure your local clone of the repo to honor the pre-commit hooks.
    The `pre-commit` package is used to pick up the pre-commit hooks which verify your staged changes adhere to the project's style and format requirements (configuration defined in [pyproject.toml](/pyproject.toml)).
@@ -85,37 +90,41 @@ If you would like to contribute, please follow these steps:
     $ pre-commit install
     ```
 
-6. At this point, you should be set up with `pre-commit`. When a commit is made it will run the pre-commit hooks defined in [`.pre-commit-config.yaml`](.pre-commit-config.yaml). For reference, you may run any of the pre-commit hooks manually before issuing the `git commit` command (see below). Some tools used by the pre commit git hook (`isort`, `flake8`, & `black`) are also available as command line executables **within the Docker container***, however, it is recommended to run them through `pre-commit` **outside of the container**, as it picks up the correct configuration.
+7. At this point, you should be set up with `pre-commit`. When a commit is made it will run the pre-commit hooks defined in [`.pre-commit-config.yaml`](.pre-commit-config.yaml). For reference, you may run any of the pre-commit hooks manually before issuing the `git commit` command (see below). Some tools used by the pre commit git hook (`isort`, `flake8`, & `black`) are also available as command line executables **within the Docker container***, however, it is recommended to run them through `pre-commit` **outside of the container**, as it picks up the correct configuration.
 
    ```
+   There are multiple ways to run pre-commit tests. Here are three:
+   
    # Check only the staged changes
    pre-commit run
 
    # Check all files in the repo
    pre-commit run -a
 
-   # Run only the flake8, isort, or black.
-   pre-commit run -a flake8
+   # Run only the isort, black, flake8 (in order).
    pre-commit run -a isort
-   pre-commit run -a black
+   pre-commit run -a black  (If it fails the first time, run it again and now apply a fix if required)
+   pre-commit run -a flake8
    ```
-7. Build the Docker container:
-    ```
-    Docker build -f Dockerfile -t <image_name>:<tag> <path/to/repository>
-    ```
-
-8.  [Within the container](README.md#startrun-the-docker-container), ensure sure unit tests pass ([instructions here](/unit_tests/README.md)).
-    ```
-    pytest unit_tests/
-    ```
+    **Note: The pre-commit system can be a bit tricky. If you have trouble with pre-commit steps above, you have another option.** Some environments may require add the word **sudo** in front commands .
+   ```
+   # Open up a docker container
+   cd /foss_fim
+   isort .
+   black .
+   flake8 .
+
+   # optionally close the container
+   # Back on your terminal console (outside the container), use the typical git add, git commit, git push
+
 
-9. Outside of the Docker container, commit your changes:
+8. Outside of the Docker container, commit your changes:
     ```
     git commit -m "<descriptive sentence or two of changes>"
     ```
     This will invoke pre-commit hooks mentioned in step 6 that will lint & format the code (some others as well). In many cases non-compliant code will be rectified automatically, but in some cases manual changes will be necessary. Make sure all of these checks pass. If not, make necessary changes (`git add <...>`), and re-issue `git commit -m "<...>"`.
    
-10. Push to your forked branch:
+9.  Push to your forked branch:
     ```
     git push -u origin
     ```

Dockerfile

@@ -9,6 +9,9 @@ ARG taudemVersion2=81f7a07cdd3721617a30ee4e087804fddbcffa88
 ENV taudemDir=$depDir/taudem/bin
 ENV taudemDir2=$depDir/taudem_accelerated_flowDirections/taudem/build/bin
 
+# remove reference to missing repo
+RUN rm /etc/apt/sources.list.d/apache-arrow.sources
+
 RUN apt-get update && apt-get install -y git  && rm -rf /var/lib/apt/lists/*
 
 RUN git clone https://github.com/dtarb/taudem.git
@@ -66,9 +69,20 @@ RUN mkdir -p $workDir
 RUN mkdir -p $depDir
 COPY --from=builder $depDir $depDir
 
-RUN apt-get update --fix-missing && apt-get install -y openjdk-19-jdk && rm -rf /var/lib/apt/lists/*
+# remove reference to missing repo
+RUN rm /etc/apt/sources.list.d/apache-arrow.sources
 
+RUN apt-get update --fix-missing && apt-get install -y openjdk-21-jdk && rm -rf /var/lib/apt/lists/*
 RUN apt update --fix-missing
+
+# An older version of openjdk still exists on the file system but was never cleaned up
+# After research, we realized, it just needs file cleanup. Leaving it there is triggering security warnings
+# RUN apt-get remove -y openjdk-17-jdk  (not installed, just residue left)
+RUN rm -rf ./usr/lib/jvm/*java-1.17* && \
+    rm -rf ./usr/lib/jvm/.java-1.17* && \
+    rm -rdf ./usr/lib/jvm/java-17*
+
+
 RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt install -y p7zip-full python3-pip time mpich parallel libgeos-dev expect tmux rsync tzdata
 
 RUN apt auto-remove
@@ -85,13 +99,15 @@ ENV PYTHONUNBUFFERED=TRUE
 
 ## ADD TO PATHS ##
 ENV PATH="$projectDir:${PATH}"
-ENV PYTHONPATH=${PYTHONPATH}:$srcDir:$projectDir/unit_tests:$projectDir/tools
+#ENV PATH=${PATH}:$projectDir:$projectDir/$srcDir:$projectDir/tools
+# Jul 17, 2024: Even though PYTHONPATH isn't used, it still seems to want it.
+ENV PYTHONPATH=${PATH}:$srcDir:$projectDir/tools
 
 ## install python 3 modules ##
 
 COPY Pipfile .
 COPY Pipfile.lock .
-RUN pip3 install pipenv==2022.4.8 && PIP_NO_CACHE_DIR=off pipenv install --system --deploy --ignore-pipfile
+RUN pip3 install pipenv==2023.12.1 && PIP_NO_CACHE_DIR=off pipenv install --system --deploy --ignore-pipfile
 
 # ----------------------------------
 # Mar 2023
@@ -104,11 +120,12 @@ RUN pip3 install pipenv==2022.4.8 && PIP_NO_CACHE_DIR=off pipenv install --syste
 # We download and unzip it to the same file folder that pip deployed the whitebox library.
 # Whitebox also attempts to always download a folder called testdata regardless of use.
 # We added an empty folder to fake out whitebox_tools.py so it doesn't try to download the folder
-RUN wbox_path=/usr/local/lib/python3.10/dist-packages/whitebox/ && \
-    wget -P $wbox_path https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_musl.zip && \
-    unzip -o $wbox_path/WhiteboxTools_linux_musl.zip -d $wbox_path && \
-    cp $wbox_path/WBT/whitebox_tools $wbox_path && \
-    mkdir $wbox_path/testdata
+
+# RUN wbox_path=/usr/local/lib/python3.10/dist-packages/whitebox/WBT && \
+#     wget -P $wbox_path https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_musl.zip && \
+#     unzip -o $wbox_path/WhiteboxTools_linux_musl.zip -d $wbox_path && \
+#     cp $wbox_path/whitebox_tools $wbox_path && \
+#     mkdir $wbox_path/testdata
 # ----------------------------------
 
 ## RUN UMASK TO CHANGE DEFAULT PERMISSIONS ##

Pipfile

@@ -4,23 +4,23 @@ url = "https://pypi.org/simple"
 verify_ssl = true
 
 [dev-packages]
-ipython = "==8.12.0"
+ipython = "==8.24.0"
 
 [packages]
-certifi = "==2023.7.22"
+certifi = "==2024.7.4"
 fiona = "==1.8.22"
 geopandas = "==0.14.3"
-numba = "==0.56.4"
-numpy = "==1.23.5"
+numba = "==0.60.0"
+numpy = "==1.26.4"
 pandas = "==2.0.2"
 rasterio = "==1.3.6"
 rasterstats = "==0.18.0"
 richdem = "==0.3.4"
-tqdm = "==4.65.0"
+tqdm = "==4.66.3"
 seaborn = "==0.12.2"
 python-dotenv = "==1.0.0"
 natsort = "==8.3.1"
-xarray = "==2023.1.0"
+xarray = "==2024.6.0"
 netcdf4 = "==1.6.3"
 tables = "==3.8.0"
 pyproj = "==3.5.0"
@@ -29,24 +29,31 @@ boto3 = "==1.26.109"
 jupyter = "==1.0.0"
 jupyterlab = "==3.6.7"
 ipympl = "==0.9.3"
-pytest = "==7.3.0"
-whitebox = "==2.3.1"
+whitebox = "==2.3.4"
 shapely = "==2.0.1"
 pyarrow = "==14.0.1"
 rtree = "==1.0.1"
 py7zr = "==0.20.4"
-scipy = "==1.10.1"
-gval = "==0.2.3"
+scipy = "==1.14.0"
+gval = "==0.2.7"
 flake8 = "==6.0.0"
 black = "==24.3.0"
 flake8-pyproject = "==1.2.3"
 pre-commit = "==3.3.3"
 isort = "==5.12.0"
-urllib3 = "==1.26.18"
+urllib3 = "==1.26.19"
 pyflwdir = "==0.5.8"
 pillow = "==10.3.0"
 pyogrio = "==0.7.2"
-openpyxl = "*"
+openpyxl = "==3.1.2"
+osmnx = "==1.9.3"
+lmoments3 = "==1.0.6"
+zarr = "==2.18.0"
+requests = "==2.32.3"
+aiohttp = "==3.9.5"
+distributed = "==2024.6.2"
+monaco = "==0.13.1"
+dask = "==2024.6.2"
 
 [requires]
 python_version = "3.10"