Try to understand the logic of multiprocessor support

[zhangxiaomei]

We are using DIRAC 8.0.24.
The CE of site is configured as:
-LocalCEType = Pool/Singularity
-NumberOfProcessors = 8
-WholeNode = True

The jobs is defined with tags:
-job.setNumberOfProcessors(8)

The jobs are running fine. In pilot log, how can I know how many cores the pilots are occuping with PoolComputingElement? For my understanding, it should have 8 cores for the jobs. But I didn't see the pilotprocessors has 8 cores:

2023-10-08 08:28:38 UTC INFO [CheckWNCapabilities] pilotProcessors = 1
2023-10-08 08:28:38 UTC INFO [CheckWNCapabilities] payloadProcessors = 1

I am not quite clear what happened here.

[fstagni]

Hello,
do not use NumberOfProcessors and WholeNode options together (to be honest, I would not use WholeNode ever...) as the 2 in a way contradict one another.

What does the following command give you?

dirac-wms-get-wn-parameters -S some.site.name -N your.ce.cn -Q the_queue.cn

?

[zhangxiaomei]

I get the same result for the command for only 1 processor even when I define NumberOfProcessors = 8 in both CE and Queue level:
(base) -bash-4.2$ dirac-wms-get-wn-parameters -S GRID.INFN-CNAF.it -N ce02-htc.cr.cnaf.infn.it -Q condor
Getting number of processors
Getting numberOfProcessors from /Resources/Computing/CEDefaults/NumberOfProcessors
Getting memory (RAM) from MJF
maxRAM could not be found in MJF, using JobParameters.getMemoryFromProc()
Getting number of GPUs
Getting GPUs from /Resources/Computing/CEDefaults/NumberOfGPUs
Looking in /Resources/Sites/GRID/GRID.INFN-CNAF.it/CEs/ce02-htc.cr.cnaf.infn.it/Queues/condor/NumberOfGPUs
Looking in /Resources/Sites/GRID/GRID.INFN-CNAF.it/CEs/ce02-htc.cr.cnaf.infn.it/NumberOfGPUs
Looking in /Resources/Sites/GRID/GRID.INFN-CNAF.it/Cloud/ce02-htc.cr.cnaf.infn.it/VMTypes/condor/NumberOfGPUs
Looking in /Resources/Sites/GRID/GRID.INFN-CNAF.it/Cloud/ce02-htc.cr.cnaf.infn.it/NumberOfGPUs
Looking in /Resources/Sites/GRID/GRID.INFN-CNAF.it/NumberOfGPUs
NumberOfGPUs could not be found in CS
1 128639 0

Only when I added NumberOfProcessors =8 in /Resource/CEDefaults, the command gets 8 processors. But I can't define all the CEs to be NumberOfProcessors=8 from the beginning:
(base) -bash-4.2$ dirac-wms-get-wn-parameters -S GRID.INFN-CNAF.it -N ce02-htc.cr.cnaf.infn.it -Q condor
Getting number of processors
Getting numberOfProcessors from /Resources/Computing/CEDefaults/NumberOfProcessors
Getting memory (RAM) from MJF
maxRAM could not be found in MJF, using JobParameters.getMemoryFromProc()
Getting number of GPUs
Getting GPUs from /Resources/Computing/CEDefaults/NumberOfGPUs
Looking in /Resources/Sites/GRID/GRID.INFN-CNAF.it/CEs/ce02-htc.cr.cnaf.infn.it/Queues/condor/NumberOfGPUs
Looking in /Resources/Sites/GRID/GRID.INFN-CNAF.it/CEs/ce02-htc.cr.cnaf.infn.it/NumberOfGPUs
Looking in /Resources/Sites/GRID/GRID.INFN-CNAF.it/Cloud/ce02-htc.cr.cnaf.infn.it/VMTypes/condor/NumberOfGPUs
Looking in /Resources/Sites/GRID/GRID.INFN-CNAF.it/Cloud/ce02-htc.cr.cnaf.infn.it/NumberOfGPUs
Looking in /Resources/Sites/GRID/GRID.INFN-CNAF.it/NumberOfGPUs
NumberOfGPUs could not be found in CS
8 128639 0

[fstagni]

There's definitely something wrong in your configuration, because this is what I get in LHCb:

 |-> dirac-wms-get-wn-parameters -S LCG.Lancaster.uk -N grendel2.hec.lancs.ac.uk -Q nordugrid-sge-grid
Getting number of processors
Getting numberOfProcessors from /Resources/Computing/CEDefaults/NumberOfProcessors
Getting numberOfProcessors from MJF
NumberOfProcessors could not be found in MJF
Looking in /Resources/Sites/LCG/LCG.Lancaster.uk/CEs/grendel2.hec.lancs.ac.uk/Queues/nordugrid-sge-grid/NumberOfProcessors
Getting memory (RAM) from MJF
maxRAM could not be found in MJF, using JobParameters.getMemoryFromProc()
Getting number of GPUs
Getting GPUs from /Resources/Computing/CEDefaults/NumberOfGPUs
Looking in /Resources/Sites/LCG/LCG.Lancaster.uk/CEs/grendel2.hec.lancs.ac.uk/Queues/nordugrid-sge-grid/NumberOfGPUs
Looking in /Resources/Sites/LCG/LCG.Lancaster.uk/CEs/grendel2.hec.lancs.ac.uk/NumberOfGPUs
Looking in /Resources/Sites/LCG/LCG.Lancaster.uk/Cloud/grendel2.hec.lancs.ac.uk/VMTypes/nordugrid-sge-grid/NumberOfGPUs
Looking in /Resources/Sites/LCG/LCG.Lancaster.uk/Cloud/grendel2.hec.lancs.ac.uk/NumberOfGPUs
Looking in /Resources/Sites/LCG/LCG.Lancaster.uk/NumberOfGPUs
NumberOfGPUs could not be found in CS
8 29284 0

Do you by chance have /Resources/Computing/CEDefaults/NumberOfProcessors=1 set in your CS? If that's the case, remove the option altogether (1 is the default anyway).

[zhangxiaomei]

Yes. I have /Resources/Computing/CEDefaults/NumberOfProcessors=1 configured. Removing it, I got the result from the jobs:
2023-10-10 09:04:32 UTC INFO [CheckWNCapabilities] pilotProcessors = 8
2023-10-10 09:04:32 UTC INFO [CheckWNCapabilities] payloadProcessors = 1
Now it looks better. I also want to understand "payloadProcessors = 1". Is the "payloadProcessors" the number of processor the payload allows to use? Here why is it "1" instead of "8" since my job is asking for "8"?
From the pilot code, it looks like I should define "WholeNode" or "maxNumberOfProcessors"?
" self.pp.payloadProcessors = 1
if "WholeNode" in self.pp.tags:
self.pp.payloadProcessors = self.pp.pilotProcessors
if self.pp.maxNumberOfProcessors > 0:
self.pp.payloadProcessors = min(self.pp.pilotProcessors, self.pp.maxNumberOfProcessors) "

[fstagni]

You really don't have to worry about "payloadProcessors", as it is only consumed by the obsolete command MultiLaunchAgent, that you are almost certainly not using it.

This also reminds me that this part can be removed now, so done in PRs:

• [devel] removed obsolete MultiLaunchAgent command Pilot#212
• [8.1] Remove obsolete cpu normalization and MJF #7237

[zhangxiaomei]

OK, thanks. It looks good now. I continue to test JUNO jobs in multi-processor mode.