How are Bundled pilot proxies renewed?

[sfayer]

Hi,

We're having a lot of problems with proxy expiry in longer jobs... One of the things I'm looking at is the (now default due to tokens I guess) BundleProxy mode where the pilot proxy is included in the job wrapper: Which component is responsible for renewing that proxy once the job is running (if any)?

Regards,
Simon

[atsareg]

The payload proxy is renewed by the InProcess/SingularityComputingElement components. The method used to monitor the payload proxy and to renew it as necessary is ComputingElement._monitorProxy().
If you see expired proxy problems, most likely it is because the pilots spent too much time waiting in the local queue and their rather long proxies are still not enough and they are not renewed.

[sfayer]

Hi,

Isn't that the payload proxy rather than the pilot proxy? It's the latter we're having trouble with.
(The payload proxy renewal is also broken, but as that's dependent on the pilot proxy, I'm trying to fix things in the correct sequence!)

On the old system the pilot proxy was renewed via the grid CE, but I haven't seen/found an equivalent for that in the BundledProxy mode...

Regards,
Simon

[aldbr]

IIUC, the pilot proxy is bundled with the pilot files in:

DIRAC/src/DIRAC/WorkloadManagementSystem/Utilities/PilotWrapper.py

Line 371 in b5bc25c

def getPilotFilesCompressedEncodedDict(pilotFiles, proxy=None):

This method is called by the SiteDirector:

DIRAC/src/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py

Line 692 in b5bc25c

pilotFilesCompressedEncodedDict = getPilotFilesCompressedEncodedDict([], proxy)

The proxy used is the one attached to the CE:

DIRAC/src/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py

Line 449 in b5bc25c

executable = self._getExecutable(queue, proxy=ce.proxy, jobExecDir=jobExecDir, envVariables=envVariables)

And this proxy attached to the CE is initialized in:

DIRAC/src/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py

Lines 891 to 920 in b5bc25c

	def _setCredentials(self, ce: ComputingElement, proxyMinimumRequiredValidity: int):
	"""
	:param ce: ComputingElement instance
	:param proxyMinimumRequiredValidity: number of seconds needed to perform an operation with the proxy
	:param tokenMinimumRequiredValidity: number of seconds needed to perform an operation with the token
	"""
	getNewProxy = False
	# If the CE does not already embed a proxy, we need one
	if not ce.proxy:
	getNewProxy = True
	# If the CE embeds a proxy that is too short to perform a given operation, we need a new one
	if ce.proxy:
	result = ce.proxy.getRemainingSecs()
	if not result["OK"]:
	return result
	if result["Value"] < proxyMinimumRequiredValidity:
	getNewProxy = True
	# Generate a new proxy if needed
	if getNewProxy:
	self.log.verbose("Getting pilot proxy", f"for {self.pilotDN}/{self.vo} {proxyMinimumRequiredValidity} long")
	pilotGroup = Operations(vo=self.vo).getValue("Pilot/GenericPilotGroup")
	result = gProxyManager.getPilotProxyFromDIRACGroup(self.pilotDN, pilotGroup, proxyMinimumRequiredValidity)
	if not result["OK"]:
	return result
	ce.setProxy(result["Value"])

Where proxyMinimumRequiredValidity should be at least 1 day (unless queueCPUTime is negative, which would be unusual):

DIRAC/src/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py

Lines 318 to 319 in b5bc25c

	cpuTime = queueCPUTime + 86400
	result = self._setCredentials(ce, cpuTime)

I might be wrong, but I don't think there is any mechanism in place to renew the pilot proxy once it is submitted to the CE.
It means that if queueCpuTime is 0 and (pilot stays hours in the queue and then tries to fetch jobs after it is launched) > 86400 seconds, then the proxy expires.

Is that your problem?

Because @marianne013 also mentioned the following issue:

there's a bunch of pilot jobs [...] that basically failed immediately with "proxy expired", though at least one job sent to a different CE at the same site is still hanging in the queue.

[sfayer]

Thanks, yes the problem is that jobs can run for up to 2 days and after 1 day of running (assuming the queuing time was short) the pilot proxy expires and everything breaks.

I think there may be an additional problem with the user proxy renewal, but there are so many errors caused by the pilot proxy expiring, it isn't clear if that's independent or not yet.

Are there any plans to add a pilot proxy renewal mechanism to the pilot (say in the JobAgent or something)?

Regards,
Simon