Troubles in configuring multi-core jobs

[arrabito]

Dear all,

I'm having troubles to make multi-core jobs working so that I have some questions about how to configure DIRAC to run multi-core jobs. Here below my use case.

We have an application which is "almost" single core, in the sense that multiple threads are instantiated at the start of the job payload, but then they don't use CPU in parallel, so that in general only one core is used, with some exceptions.

Our sites are all configured with NumeberOfProcessors = 1 and usually these jobs just run fine, but for a particular site we had a large fraction of failures and from the batch system logs we learned that this was due to jobs exceeding the nb of requested cores, i.e. 1.

So, we have configured the problematic site with:
NumeberOfProcessors = 4

but then we get all jobs there failed with Minor Status = Received Kill signal.

In the logs of the pilots I found:

2023-01-10 18:41:54 UTC dirac-jobexec/Script INFO: Command is: /home/execute/dir_2381384/DIRAC_UDgL8Epilot/19494869/dirac_prod6_run --start_run 1101 --run 00000088 --sequential --divergent    Paranal proton South 20
Received control signal
Received Kill signal, stopping job via control signal
Sending kill signal to application PID 210
Subprocess.killChild() returned:9 
EXECUTION_RESULT[CPU] after Execution of spObject.systemCall 0.05 0.04 1.68 0.20 5.37
EXECUTION_RESULT[Thread] after Execution of spObject.systemCall {'OK': True, 'Value': (False, '', '')}

but the pilot status is Done.

I don't know yet which component sent the kill signal, but I was wondering if the configuration for multi-core jobs is fine. Is it enough to just configure the site with:

NumeberOfProcessors = 4

?

Or some additional specification should be done also in the job requirements?

For instance in the logs of the pilot I found:

2023-01-10 18:41:05 UTC INFO [CheckWNCapabilities] pilotProcessors = 4
2023-01-10 18:41:05 UTC INFO [CheckWNCapabilities] payloadProcessors = 1

but I don't know if this is correct or if we should have payloadProcessors = 4.

Thank you for your help.

[aldbr]

Hi,

Our sites are all configured with NumeberOfProcessors = 1 and usually these jobs just run fine, but for a particular site we had a large fraction of failures and from the batch system logs we learned that this was due to jobs exceeding the nb of requested cores, i.e. 1.

How do you submit your jobs? e.g. are you using an ARC or HTCondor CE? Are you submitting them via SSH specifying the underlying batch system? In this case, which batch system is it?

but then we get all jobs there failed with Minor Status = Received Kill signal.

IIUC, the Watchdog in charge of monitoring the job receives a Kill signal from the batch system, but the Pilot status is done, which is weird.
Does the pilot finish its execution before the job? It is technically possible since you have a multi-core allocation.

Or some additional specification should be done also in the job requirements?

From what I understand, it should be okay.
pilotProcessors = 4 means that you should have an allocation of 4 logical cores and payloadProcessors = 1 means that, in this 4-core allocation, 1 core is allocated to your job.
It looks like your application is seeing more than 4 cores available and tries to use more of them.

[fstagni]

Before I investigate deeper, I just want to make sure that:

• you have in your config NumberOfProcessors and not NumeberOfProcessors as you repeatedly made this typo above.
• NumberOfProcessors=1 is the default, you don't have to specify it further.
• Docs are in
  • https://dirac.readthedocs.io/en/latest/AdministratorGuide/Resources/site.html?highlight=NumberOfProcessors#configuration
  • https://dirac.readthedocs.io/en/latest/UserGuide/Tutorials/JobManagementAdvanced/index.html#submitting-multiprocessor-mp-jobs
  • https://dirac.readthedocs.io/en/latest/AdministratorGuide/Resources/computingelements.html
  • https://dirac.readthedocs.io/en/latest/CodeDocumentation/Interfaces/API/Job.html#DIRAC.Interfaces.API.Job.Job.setNumberOfProcessors

[arrabito]

Thanks a lot for these explanations.
I confirm that I used the correct syntax for NumberOfProcessors.
I didn't know the doc here:

https://dirac.readthedocs.io/en/latest/CodeDocumentation/Interfaces/API/Job.html#DIRAC.Interfaces.API.Job.Job.setNumberOfProcessors

which I think it answers to my question.

From my understanding I should indeed use in my case:

job.setNumberOfProcessors(minNumberOfProcessors=1, maxNumberOfProcessors=4)

I will let you know if it works as expected.

Thanks.

[arrabito]

Dear all,

First of all, could you please tell me if for our use case, we should use the PoolComputingElement, setting:

LocalCEType = Pool

in the CE configuration?

For the moment we haven't specified anything.

Then, I've done a few tests with:

job.setNumberOfProcessors(minNumberOfProcessors=1, maxNumberOfProcessors=4)

but jobs keep failing.

In the pilot logs I found:

2023-01-10 18:41:05 UTC INFO [CheckWNCapabilities] pilotProcessors = 4
2023-01-10 18:41:05 UTC INFO [CheckWNCapabilities] payloadProcessors = 1

Could you please tell me if this is correct? Eventually I thought that payloadProcessors should have been 4, since it can use 4 cores at max.

Anyway, I don't know if jobs failed because they tried to use more than 4 cores or because the requirement is not working as expected.

So, I've also tried a small test with 3 jobs with:

job.setNumberOfProcessors(minNumberOfProcessors=6)

and jobs succeeded. Unfortunately I don't have pilot logs.

Finally I've tried another small test of 3 jobs with:

job.setNumberOfProcessors(minNumberOfProcessors=1, maxNumberOfProcessors=6)

and 2 were Done and 1 Failed. I will try to investigate further, but I wanted to know if it's normal to have payloadProcessors = 1 for jobs that can use 1 core and at maximum 6 cores. Also I would like to know about the use of PoolComputingElement.

@aldbr These test refer to HTCondor CE, but we will also have later to run on ARC6.

Thank you.

Luisa

[fstagni]

First of all, could you please tell me if for our use case, we should use the PoolComputingElement, setting:

LocalCEType = Pool

in the CE configuration?

Yes.
TBH, you could set LocalCEType = Pool even everywhere, because if NumberOfProcess == 1 then it works exactly like InProcessCE (which today is the default).

Then, I've done a few tests with:

job.setNumberOfProcessors(minNumberOfProcessors=1, maxNumberOfProcessors=4)

but jobs keep failing.

In the pilot logs I found:

2023-01-10 18:41:05 UTC INFO [CheckWNCapabilities] pilotProcessors = 4
2023-01-10 18:41:05 UTC INFO [CheckWNCapabilities] payloadProcessors = 1

Could you please tell me if this is correct? Eventually I thought that payloadProcessors should have been 4, since it can use 4 cores at max.

You are basically declaring that your job can also run in a single processor, so payloadProcessors = 1 does not seem fully wrong to me.

[arrabito]

Thank you for your answer.

When you say that we could set LocalCEType = Pool everywhere you mean in:

/Resources/Computing/CEDefaults

?

Thanks.

[fstagni]

Yes. But of course I suggest you to start with single CEs.

[aldbr]

To check: seems like the [CheckWNCapabilities] payloadProcessors will always be 1 because the parameter is obtained from parameter maxNumberOfProcessors specified in pilotTools.
The parameter does not seem to be specified by the SiteDirector.

[fstagni]

See my answer in #7233 (reply in thread)