Configuration/Server service thread leak?

[sfayer]

Hi,

It looks like the Configuration/Sever primary (in classic non-tornado mode) is slowly leaking threads on all of our DIRAC instances (versions 7r1, 7r2 & 7.3) at a rate of between 12 and 42 per day.

On a newly started instance:

$ pgrep -wcf 'Configuration/Server'
8

On a quiet development instance that has been running about a month:

$ pgrep -wcf 'Configuration/Server'
336

On our production server, we saw it hitting the default 4096 process limit after it had run for around 3 months: Clearly we can increase this limit, but I can't imagine having 3680 threads sitting about is good for performance/resource usage. Interestingly we don't seem to see this on servers that are only running secondary CS instances, only the primary appears to be affected.

Does anyone else see this happening on their instances? Does anyone see this on a tornado based CS? Would it be possible to track down where this is leaking and stop it from happening? (Although if it doesn't affect tornado CS, we'd probably consider just moving to that relatively soon, which I'm sure you'll tell me is a good idea anyway :-)

Regards,
Simon

[andresailer]

I can see that behaviour as well.
Using

ps H -eO user,%mem,%cpu,lstart,comm    | grep Configuration
[snip]
23809 dirac     2.3  0.0 Wed Nov 10 05:09:22 2021 python          S ?        00:00:00 python /opt/dirac/pro/DIRAC/Core/scripts/dirac-service.py Configuration/Server --cfg /opt/dirac/pro/etc/Configuration_Server.cfg
23809 dirac     2.3  0.0 Wed Nov 10 05:09:22 2021 python          S ?        00:00:00 python /opt/dirac/pro/DIRAC/Core/scripts/dirac-service.py Configuration/Server --cfg /opt/dirac/pro/etc/Configuration_Server.cfg
[snip]

I could correlate the creation time of the threads with
2021-11-10 04:09:22 UTC Configuration/Server INFO: Updating configuration on slave servers
One new thread for each secondary Configuration Server
(I hope I got my timezone calculations right)

[atsareg]

The ThreadPool used to force update slave servers upon configuration changes leaves threads behind. Passing to ThreadPoolExecutor looks to fix the problem, see #5592 .

[sfayer]

Thanks, looks good to me.