Behavior of GeoWebCache When Connection Error Occurs

[KitazonoShuhei]

■Situation and Question

When a request from GeoWebCache to GeoServer fails even once, despite Tomcat used by GeoWebCache being active, GeoWebCache freezes.
This issue does not resolve until Tomcat used by GeoWebCache is restarted. At this time, accessing the GeoWebCache site results in a "504 Gateway Time-out".

For example, this issue occurs under the following conditions:
・When a request is made from GeoWebache while the table of the DB for tile acquisition is locked.
⇒ Even when the DB table lock is released, this issue does not resolve.
・When a request is made from GeoWebCache while GeoServer's Tomcat is stopped.
⇒ Even when GeoServer's Tomcat is restarted, this issue does not resolve.

Q1.Is there a solution to this problem?
Q2.What could be the cause of this issue?

■Environment

The GeoServer and GeoWebCache servers are running on separate AWS EC2 instances.

• host1
  RHEL 7.9
  GeoWebCache 1.23.1
  Tomcat 9.0.78
  JVM 11.0.15

• host2
  RHEL 7.9
  GeoServer 2.23.2
  Tomcat 9.0.78
  JVM 11.0.15

■Tomcat (GeoWebCache) catalina.out

  When GeoWebCache freezes in this issue, the following error is output in catalina.out of Tomcat used by GeoWebCache:

ERROR: [geowebcache.GeoWebCacheDispatcher] - All backends (1) failed. Reason: org.apache.http.conn.HttpHostConnectException: Connect to {path to Geoserver}: 443 {path to Geoserver} failed: connection timed out. Last request: {request path to Geoserver}
  Also, socket errors may be output in catalina.out as follows:

13:04 severe [http-nio-8080-Acceptor] org.apache.tomcat.util.net.Acceptor.run socket accept failed
  When a socket error occurs, the state has reached the upper limit set in the limits as follows. The opened files are not closed over time, and will be solved by restarting Tomcat used in GeoWebCache.

# cat /proc/(Tomcat PID)/limits
Limit              Soft Limit        Hard Limit        Units
  :
Max open files     4096              4096              4096
  :

# lsof -p (Tomcat PID) | wc -l
4132

# ls /opt/(Tomcat PID)/fd | wc -l
2119
## ■netstat command
  When GeoWebCache freezes in this issue, a large number of connections with the status "CLOSE_WAIT" remain and are not released, as follows:

# netstat -anp | grep 8080
tcp6   101   0 (IP addr):(port)  (IP addr):(port)  CLOSE_WAIT  (Tomcat PID)/java
tcp6   101   0 (IP addr):(port)  (IP addr):(port)  CLOSE_WAIT  (Tomcat PID)/java
tcp6   101   0 (IP addr):(port)  (IP addr):(port)  CLOSE_WAIT  (Tomcat PID)/java
  :
  This is also resolved by restarting Tomcat used by GeoWebCache.

■Tried

・Use a different version of GeoWebCache
　- 1.20.4, 1.20.5 No issue
　- 1.24.2, 1.23.4