[3pt] PR: Docker, python packages and jdk updates

[RobHanna-NOAA]

This PR rolls up a bunch of other PR's and python packages requests including:

• Issue 1208 Bump OpenJDK from 17.0.8 to 17.0.10 (via updating to JDK 21.0.3)
• PR 1207 - Dependabot bump certifi from 2023.7.22 to 2024.7.4
• PR 1192 - Dependabot Bump urllib3 from 1.26.18 to 1.26.19
• Updates required from ongoing PR 1206 - Probabilistic Flood Inundation Mapping. These updates make it easier for that branch/task to continue forward and staying in sync with dev. This triggered a few other packages that needed to be updated.

Other tasks included are:

• Removing the now heavily obsolete unit_test system, including the package pytest. This included some changes to the CONTRIBUTING.md document.
• Clean of a couple of packages no longer in use: pluggy and iniconfig
• Removal of a deprecated file named config/aws_s3_put_fim3_hydrovis_whitelist.lst
• Removed duration stamps around a few parts in fim_post_processing.sh
• Fixes and updates to linting files. e.g. pyproject.toml. (line length was not working correctly)

Changes

• Dockerfile, Pipfile, Pipfile.lock: as describe above for python package changes
• .gitignore, CONTRIBUTING.md: File changes related to removing the unit_test system.
• fim_post_processing.sh: noted above.
• pyproject.toml: fixes and updates for linting

Removals

• unit_tests folder and all related files under it. Appx 25 to 30 files removed.

Testing

• A number of tests were done with various combinations using the new docker file and build testing.
• Inside the new docker container, a fim pipeline run against 12040101 was done as well as running alpha testing on it.
• The new open JDK updates (WILL BE) tested in OWP servers to ensure it now passes vulnerability tests.

Deployment Plan (For developer use)

How does the changes affect the product?

• Code only?
• Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
• If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
• Require new pre-clip set?
• Has new or updated python packages?
• If applicable, has a deployment plan be created with the deployment person/team?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• (pending) The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• n/a Passes all unit tests locally (inside interactive Docker container, at /foss_fim/, run: pytest unit_tests/)
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Reviewers requested
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations

[RobHanna-NOAA]

@mluck When you test this PR, keep this image. I have another small PR coming that I would like tested against this image so both can be merged at once. (if that is ok)

[mluck]

I built the image from Dockerfile and ran fim_pipeline.sh and synthesize_test_cases.py. All scripts completed successfully and CSI was identical to v4.5.2.6. However, there were a number of messages, copied below.

It looks like Whitebox is still trying to download the executable, unless these are just being printed to the screen.

Creating AGREE DEM using 70 meter buffer 03100204 0
Downloading WhiteboxTools pre-compiled binary for first time use ...
Downloading WhiteboxTools binary from https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_amd64.zip
Decompressing WhiteboxTools_linux_amd64.zip ...
WhiteboxTools package directory: /usr/local/lib/python3.10/dist-packages/whitebox
Downloading testdata ...

A warning:

root@4fc02f9ac92c:/# /foss_fim/tools/synthesize_test_cases.py -c DEV -jh 1 -jb 6 -vr -vg -o -b nws -v dev-depbot-urllib
/usr/local/lib/python3.10/dist-packages/dask/dataframe/__init__.py:49: FutureWarning: 
Dask dataframe query planning is disabled because dask-expr is not installed.

You can install it with `pip install dask[dataframe]` or `conda install dask`.
This will raise in a future version.

  warnings.warn(msg, FutureWarning)
================================

A lot of these warnings:

/foss_fim/tools/overlapping_inundation.py:399: RuntimeWarning: invalid value encountered in cast
  nan_tile = np.array([np.nan]).astype(dtype)[0]

A couple of these warnings:

/usr/local/lib/python3.10/dist-packages/gval/comparison/agreement.py:134: FutureWarning: It is recommended to use 'rio.write_crs()' instead. 'rio.set_crs()' will likelybe removed in a future release.
  agreement_map.rio.set_crs(crs, inplace=True)

[RobHanna-NOAA]

I built the image from Dockerfile and ran fim_pipeline.sh and synthesize_test_cases.py. All scripts completed successfully and CSI was identical to v4.5.2.6. However, there were a number of messages, copied below.

It looks like Whitebox is still trying to download the executable, unless these are just being printed to the screen.

Creating AGREE DEM using 70 meter buffer 03100204 0
Downloading WhiteboxTools pre-compiled binary for first time use ...
Downloading WhiteboxTools binary from https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_amd64.zip
Decompressing WhiteboxTools_linux_amd64.zip ...
WhiteboxTools package directory: /usr/local/lib/python3.10/dist-packages/whitebox
Downloading testdata ...

A warning:

root@4fc02f9ac92c:/# /foss_fim/tools/synthesize_test_cases.py -c DEV -jh 1 -jb 6 -vr -vg -o -b nws -v dev-depbot-urllib
/usr/local/lib/python3.10/dist-packages/dask/dataframe/__init__.py:49: FutureWarning: 
Dask dataframe query planning is disabled because dask-expr is not installed.

You can install it with `pip install dask[dataframe]` or `conda install dask`.
This will raise in a future version.

  warnings.warn(msg, FutureWarning)
================================

A lot of these warnings:

/foss_fim/tools/overlapping_inundation.py:399: RuntimeWarning: invalid value encountered in cast
  nan_tile = np.array([np.nan]).astype(dtype)[0]

A couple of these warnings:

/usr/local/lib/python3.10/dist-packages/gval/comparison/agreement.py:134: FutureWarning: It is recommended to use 'rio.write_crs()' instead. 'rio.set_crs()' will likelybe removed in a future release.
  agreement_map.rio.set_crs(crs, inplace=True)

---

1. for Whitebox: yes.. we will have to defer that one for now. There is a lot to that one. I updated the Issue card 1209 for the details so we can come back to it.

2. Dask framework: The fix is easy but requires some discussion. I would prefer to leave it for now.

3. Yes.. there are a wild amount of new warnings;
   a) Numpy: We are going to see them absolutely everywhere and a huge amount.

   b) Synthesize_test_cases.py: Issue 1228. I am going to have this fixed and merged with this PR as it creates so many warnings onscreen that we can't monitor any real issues.

   c) There are a very, very large number of other warnings mostly fitting into just a few fixes that already exist prior to this PR. All similar in nature to the numpy warning. See 1201.

   d) I am recommending taking the next step of upgrading other key packages like panda, rasterio, fiona, etc, see issue 1227. We can consider fixing more of the warnings at that point.

4. For the "/gval/comparison/agreement.py:134 / agreement_map.rio.set_crs(crs, inplace=True)" warning, gval will likely make an update on that someday via 1229.

But.. there are so many warnings, we may want to cleanup some of the current warnings (pre this release) and new warning post this release), only in the name of sheer volume,

I am open to suggestions.

The biggest driver on getting this out, is that both ITSG and depbot are wanting this merge sooner than later.

[mluck]

Seems good to me. The warnings don't affect the functionality of the scripts.