DIRAC.Interfaces.API.Job.Job.setOutputData::outputSE - replica level specification with inclusion/exclusion

[adriansev]

Would be possible to make setOutputData of job to read a construct like:
outputSE=replica:2,SE1,SE2,!SE3
which would mean: make 2 random replicas beside the explicitly requested SE1 and SE2 (so, a total of 4 replicas) but do not use SE3
or
outputSE=replica:3,!SE4
that would mean: just make 3 random replicas but do not use SE4
This would allow flexibility in data placement from the beginning of job production
Thanks a lot!

[chaen]

In LHCb we extend the output upload of vanilla DIRAC by allowing a second replica (look for replicateUserOutputData)
https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/blob/master/src/LHCbDIRAC/Workflow/Modules/UserJobFinalization.py

We also have resolution logic for where to upload the output for production jobs.

However, I would recommend not to upload more than one replica from the job. I would suggest you upload one replicas, and then asynchronously (RMS/FTS machinery) replicate wherever you like. This is much more efficient and reliable in many aspects

[adriansev]

Thanks a lot! So, yeah, given that there is no chance to have a AUGERDIRAC, any kind of extension is not feasible, so your second suggestion would be best choice. Could you give me an advice how to do it? (i'm just slightly familiar with DIRAC). Can i do it from inside the job? How can i get the status of successful upload to trigger the replication of the lfn? Also, if the initial upload fails, can i do a retry for failover?
I found this: https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/Transformation/index.html#data-replication-based-on-catalog-query
but i'm under impression that transformations must first defined centrally (i apologize if i got it wrong) so this solution could be used later
also i found this: https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/RequestManagement/concepts.html
but also needs to be defined in central services and it's not clear to me how can i use it at the job level ..
Thanks a lot for patience and help! :)

[chaen]

No, it's not something for inside the job really. Potentially as a Module that is executed by the pilot at the end of the job, but if you can't have an extension, you would have to do it as a module generic enough for everybody.
Out of curiosity, why can't you have an extension ?

Transformations are more of a central production tool indeed, but depending how DIRAC is configured, any user could potentially create it.
That of course supposes that your central services provide all the necessary services (Transformation System, Request Management System and FTS system). The tutorial you found are exactly what you need.

[adriansev]

No, it's not something for inside the job really. Potentially as a Module that is executed by the pilot at the end of the job, but if you can't have an extension, you would have to do it as a module generic enough for everybody. Out of curiosity, why can't you have an extension ?

well, at this moment i'm trying to familiarize myself with the environment so anything that i plan to do and present as a solution i'm trying to do as minimal as possible

[chudoba]

Out of curiosity, why can't you have an extension ?

Hi Chris, we use the common DIRAC EGI instance. I suppose that our extensions would require also running our own instance of DIRAC.

[fstagni]

(here I answer for the extensions, then I will post a second answer more specific to the topic)

You can have extensions for DIRAC, WebAppDIRAC and Pilot.

Clearly, an extension of WebAppDIRAC makes sense only server side, and it does not matter for the above.

An extension of DIRAC is normally installed also on the server side, but not necessarily, as it could be installed only client side. I am not familiar with the EGI instance, but that would mean maintaining your own UI (and consider that you might need to be in sync with the server's version). Such extension could be also installed by the Pilots.

Pilots extensions only make sense for adding Pilot "Commands": https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/WorkloadManagement/Pilots/index.html#pilot-commands . Commands can be added in an extension (https://gitlab.cern.ch/lhcb-dirac/LHCbPilot) or in the main Pilot code (https://github.com/DIRACGrid/Pilot) if they are generic enough, and then can be activated/used with a configuration.

[fstagni]

Answer to your case: as of today it is not possible, this would require development. This discussion can be turned in an issue to be implemented later on.

RMS can be fed by DIRAC transformations as @adriansev pointed out above, but not only, and in fact the way your case could be treated would be by inserting directly in RMS. DIRAC's code to handle this case was started long time ago but never fully finished: https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/Workflow/Modules/UploadOutputs.py . Modifications to the Job APIs would also be necessary.

I can turn this into an issue, and I can provide guidance for the implementation (I would not have time to do this myself).

[atsareg]

Concerning the possible AugerDIRAC extension. If deemed necessary, it can be discussed how this can be installed in the DIRAC EGI service. This is not impossible, including Pilots