Dependencytrack keeps going down every 6 days

[Gss123123]

We have dependency track war hosted on a dedicated m5.24xlarge ec2 instance. The usual CPU load is <10%, but there are periodic spikes in cpu utilization, gradually touching higher and higher marks with more days the service is live. These are the usual log errors seen at time of these spikes :-
javax.jdo.JDODataStoreException: HikariPool-4 - Connection is not available, request timed out after 30492ms.
Caused by: java.sql.SQLTransientConnectionException: HikariPool-4 - Connection is not available, request timed out after 30602ms.[Cache] >> EC.preCommit L1Cache op IS NULL!

Service details
version - 4.2.1
jdk version - 11
AWS ec2 instance type - m5.24xlarge
Vulnerability analysis is enabled
Env variables -
-e ALPINE_WORKER_THREADS=32
-e ALPINE_DATABASE_MODE='external'
-e ALPINE_DATABASE_URL='jdbc:postgresql://${db_host}:${db_port}/${db_name}'
-e ALPINE_DATABASE_DRIVER='org.postgresql.Driver'
-e ALPINE_DATABASE_DRIVER_PATH='/extlib/postgresql-42.2.18.jar'
-e ALPINE_DATABASE_USERNAME='${db_user}'
-e ALPINE_DATABASE_PASSWORD='${db_pass}'
-e ALPINE_DATABASE_POOL_ENABLED=true
-e ALPINE_DATABASE_POOL_MAX_SIZE=100
-e ALPINE_DATABASE_POOL_IDLE_TIMEOUT=600000
-e ALPINE_DATABASE_POOL_MAX_LIFETIME=600000
-e LOGGING_LEVEL=INFO
-e ALPINE_HTTP_PROXY_ADDRESS='${iboss_proxy_fqdn}'
-e ALPINE_HTTP_PROXY_PORT=443
-e ALPINE_NO_PROXY='${no_proxy_list}'
-e ALPINE_DATA_DIRECTORY='/data' \

[stevespringett]

I would recommend upgrading to the latest release. This was likely resolved in #1026 which was available in 4.3.0.

[Gss123123]

We upgraded dependency track to v4.3.6. Still seeing the same issue, Dependency track periodically peaks to 36% cpu utilisation, and then just crashes after a few days at one of the peaks. The crashing period is usually around 5 days every time.

[Gss123123]

@stevespringett any suggestions ?

[stevespringett]

Are you allocating more memory to the DT instance than what the Docker host is configured to provide? Common issue with Docker.

[Gss123123]

Its fixed at 80% of the instance. currently at 320gb memory for the service.
sudo docker run -d
--cpus="70"
-m 320g \

Can it be an issue of ALPINE_WORKER_THREADS=32 ? Though it was crashing previously too when this variable was omitted. You acn have a look at all the env variables supplied in the command given previously in thread.

[stevespringett]

32 threads should be fine. If you have a lot of CPUs, you can even increase that. The database parameters seem wrong though. 100 connections is a bit high, and the idle timeout should be half that.

[Gss123123]

Hi @stevespringett
I tried both the suggestions, the service still failed after a week. On further analysis it seems that all failures stem from the error log -- EC.preCommit L1Cache op IS NULL!

One thing I forgot to mention was, the docker container in which we are running dependency track is a homegrown container from scratch with the DT api jar being embedded in it and we are using jdk 11 in it. The last stable version of DT that was performing for us was 3.8.0 whose docker image used openjdk 1.8.

I see you have faced a similar issue earlier due to usage of jdk9 - https://datanucleus.groups.io/g/main/topic/ec_precommit_l1cache_op_is/9346765?p=

Could you share your experience of this and how it went away ?