[8pt] Whitebox calling URL at runtime

[RobHanna-NOAA]

Once again, the whitebox packages is making external calls at runtime which we can not allow. Sure enough, during one of our runs, it randomly failed because of a failed external connection call.

/outputs/fim_bathy_adj_test_11/logs/unit/07120001_unit.log:59:Unexpected error: <class 'urllib.error.URLError'>
/outputs/fim_bathy_adj_test_11/logs/unit/07120001_unit.log:81:ConnectionResetError: [Errno 104] Connection reset by peer
/outputs/fim_bathy_adj_test_11/logs/unit/07120001_unit.log:107: raise URLError(err)
/outputs/fim_bathy_adj_test_11/logs/unit/07120001_unit.log:108:urllib.error.URLError: <urlopen error [Errno 104] Connection reset by peer>

(Note: HUC number not relevant)

More details;

---

Creating AGREE DEM using 70 meter buffer 07120001 0
Downloading WhiteboxTools pre-compiled binary for first time use ...
Downloading WhiteboxTools binary from https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_amd64.zip
Decompressing WhiteboxTools_linux_amd64.zip ...
WhiteboxTools package directory: /usr/local/lib/python3.10/dist-packages/whitebox
Downloading testdata ...
Unexpected error: <class 'urllib.error.URLError'>
Traceback (most recent call last):
File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/lib/python3.10/http/client.py", line 1283, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/lib/python3.10/http/client.py", line 1455, in connect
self.sock = self._context.wrap_socket(self.sock,
File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.10/ssl.py", line 1100, in _create
self.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1371, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/foss_fim/src/agreedem.py", line 321, in
agreedem(
File "/foss_fim/src/agreedem.py", line 51, in agreedem
wbt = whitebox.WhiteboxTools()
File "/usr/local/lib/python3.10/dist-packages/whitebox/whitebox_tools.py", line 307, in init
download_wbt()
File "/usr/local/lib/python3.10/dist-packages/whitebox/whitebox_tools.py", line 253, in download_wbt
urllib.request.urlretrieve(dep_url, os.path.join(work_dir, "DEM.dep"))
File "/usr/lib/python3.10/urllib/request.py", line 241, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.10/urllib/request.py", line 519, in open
response = self._open(req, data)
File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/usr/lib/python3.10/urllib/request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 104] Connection reset by peer>

[RobHanna-NOAA]

In some recent fix attempts, the notes from that are:

This fix reinstates the whitebox hack where whitebox is downloaded during the initial build of the Docker image so that external calls to download whitebox do not happen each time whitebox is run in agreedem.py during fim_pipeline.sh. Besides being unnecesary in our case, these external calls are a potential point of failure if the whitebox site is temporarily unavailable. The whitebox hack introduced in #857 was removed in #1183 due to the error #1181 which was a result of whitebox directory restructuring (see jblindsay/whitebox-tools#409 (comment)). The path in the whitebox hack has been adjusted accordingly.

[RobHanna-NOAA]

After a very large of invested time to get it working without live run-time calls to the Whitebox website, a number of things were learned and changed:

• Our earlier code attempts in Dockerfile were completely irrelevant and had no value.
• I talked to the Whitebox dev team when I discovered that they code always called their site and had no way to stop that. Whitebox make a change and released a new version, 2.3.5 which has a new feature to allow to skip the call to the whitebox site. The fix is weird though but does work. We simply need to add an environment variable named exactly "WBT_PATH". Assigning any value is fine as the system does not care about the value, just that that variable exists.
• Unfortunately, I tried a very large array of figure out exactly the correct package to download it and where to install it. All with no success. The live download file was https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_amd64.zip but I was unable to see how that installed and where. I did notice that it is not the same as using the pip whitebox install. It seems the pip install version does not. I added that note to the whitebox issue card but got no response. It makes me doubt, as irrationally as it sounds, that the pip version simply does not work. Must have missed something.
• I tried a large range of files to figure out where to unzip the zip from the URL above but no luck. However, I found where we could unzip and manually move some files and folders to overwrite pip placed files, but alot more research is required. I have those notes if needed. Our initial Docker whitebox commands used a different zip then the auto download from whitebox.
• Most if it evolves around manipulate in the /usr/local/lib/python3.10/dist-package.
• Other options are:
  • Press the whitebox folks about why it is an issue. See the whitebox issue card link above.
  • Continue testing out the manual download of the zip and moving/overwriting files overtop of the pip files installed. This can be done in the Dockerfile.
  • Look deeper at their code in their repo to figure out how their code was unzipping, and to where, on their live downloads. I

Also: Adding an docker run param of --progress=plain helps shows more details of where stuff is going.

Tidbits:

Using combinations including calls in our agree dem seem to influence it sometimes, but still had errors:

Permissions in the /usr/local/lib/python3.10/dist-packages/whitebox also had impacts.

I have a TON of more notes if someone wants them.

The riddle... keep trying using one of three options above (or more)

[RobHanna-NOAA]

Dang. Happened again on a random HUC in AWS Step functions. I have a ton of notes on possible fixes. We might have to up the priority on this a little.