[v8.0] Introduce Scout Agent and Optimizer #7251

qdcampagna · 2023-10-20T02:39:31Z

This pull request introduces the Scout Agent and Optimizer. They submit a subset of primary jobs first, then only allows the primary jobs to run once a certain number of Scout jobs are successfully finished. Follow up to #7118. Closes #7083.

BEGINRELEASENOTES

*WorkloadManagement
NEW: Introduce Agent and Optimizer for scouting jobs

ENDRELEASENOTES

…RACv8 Bring up to date

fstagni

Thank you for the PR.
The main problem I see, that requires change, is that you are calling directly JobDB() to get/set JobParameters. But, JobParameters can be stored also in ElasticSearch (and will only be stored there from DIRAC v9 on). So, use JobStateUpdateClient() instead.

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

src/DIRAC/WorkloadManagementSystem/Executor/Scouting.py

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

src/DIRAC/WorkloadManagementSystem/Executor/Scouting.py

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

fstagni · 2023-10-24T07:31:05Z

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

+        """Sets defaults
+        """
+
+        self.jobDB = JobStateUpdateClient()


Maybe it was me not being clear, but this can't be a 1-to-1 substitution between JobDB and JobStateUpdateClient, as the 2 don't expose the same methods. For example, JobStateUpdateClient does not expose the selectJobs method used below.

Ah I see. Does JobStateUpdateClient access only the methods found in JobStateUpdateHandler, or are there others?

Only those.

In general, I would do this, for added clarity:

Suggested change

self.jobDB = JobStateUpdateClient()

self.jobStateUpdate = JobStateUpdateClient()

then you'll need at least also

self.jobMonitoring = JobMonitoringClient()

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

fstagni · 2023-10-24T07:33:26Z

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

+        failedjoblist = []
+        failedsitelist = []
+        stalledjoblist = []
+        scoutjobs = result['Value'].keys()


beware, this is python3 and not py2 code.

This is safer:

Suggested change

scoutjobs = result['Value'].keys()

scoutjobs = list(result['Value'])

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

src/DIRAC/WorkloadManagementSystem/Executor/Scouting.py

fstagni

And, would you add a unit test or two?

src/DIRAC/WorkloadManagementSystem/Executor/Scouting.py

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

andresailer

Could you add some documentation about this somewhere?

From the #7083

"Scout jobs" are created at the job submission -- when a user submits a set of jobs, our client tool makes a smaller set of shorter jobs as "scout jobs" and submits them (the original and the scout) altogether.

Would be interesting to know how this is implemented at the job submission level.

andresailer · 2023-11-06T10:21:18Z

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

+        self.totalScoutJobs = Operations().getValue('WorkloadManagement/Scouting/totalScoutJobs', 10)
+        self.criteriaFailedRate = Operations().getValue('WorkloadManagement/Scouting/criteriaFailedRate', 0.5)
+        self.criteriaSucceededRate = Operations().getValue('WorkloadManagement/Scouting/criteriaSucceededRate', 0.3)
+        self.criteriaStalledRate = Operations().getValue('WorkloadManagement/Scouting/criteriaStalledRate', 1.0)
+        self.criteriaFailed = Operations().getValue('WorkloadManagement/Scouting/criteriaFailed', 
+                                                    int(self.totalScoutJobs * self.criteriaFailedRate))
+        self.criteriaSucceeded = Operations().getValue('WorkloadManagement/Scouting/criteriaSucceeded', 
+                                                       int(self.totalScoutJobs * self.criteriaSucceededRate))
+        self.criteriaStalled = Operations().getValue('WorkloadManagement/Scouting/criteriaStalled', 
+                                                     int(self.totalScoutJobs * self.criteriaStalledRate))


Can this go via the Agent options, or is this used also in other places?

In any case you have to add the agent to the ConfigTemplate file?

qdcampagna · 2023-11-13T07:06:03Z

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

+    def execute(self):
+        """The ScoutingJobStatus execution method.
+        """
+        result = self.jobDB.selectJobs({'Status': 'Scouting'})


JobStateUpdateClient doesn't seem to have a method that does what we want here. However, it seems that getJobs() in JobMonitoringHandler might have this functionality. Would you recommend using getJobs, or implementing this in JobStateUpdateHandler?

Use getJobs.

fstagni · 2023-11-14T09:14:26Z

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

+        """Sets defaults
+        """
+
+        self.jobDB = JobStateUpdateClient()


In general, I would do this, for added clarity:

Suggested change

self.jobDB = JobStateUpdateClient()

self.jobStateUpdate = JobStateUpdateClient()

then you'll need at least also

self.jobMonitoring = JobMonitoringClient()

fstagni · 2023-11-14T09:17:52Z

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

+        failedjoblist = []
+        failedsitelist = []
+        stalledjoblist = []
+        scoutjobs = result['Value'].keys()


This is safer:

Suggested change

scoutjobs = result['Value'].keys()

scoutjobs = list(result['Value'])

fstagni · 2023-11-14T09:18:48Z

src/DIRAC/WorkloadManagementSystem/Agent/ScoutingJobStatusAgent.py

+
+        if len(donejoblist) >= criteriaSucceeded:
+            self.log.verbose(f'Scout (ID = {scoutid}) are done.')
+            scoutStatus = {'Status': 'Checking', 'MinorStatus': 'Scouting', 'appstatus': 'Scout Complete'}


Suggested change

scoutStatus = {'Status': 'Checking', 'MinorStatus': 'Scouting', 'appstatus': 'Scout Complete'}

scoutStatus = {'Status': JobStatus.CHECKING, 'MinorStatus': 'Scouting', 'appstatus': 'Scout Complete'}

and similar in the lines below.

fstagni · 2023-11-14T09:25:49Z

src/DIRAC/WorkloadManagementSystem/Executor/Scouting.py

+                self.jobLog.info('Skipping optimizer, since no scout \
+                                 corresponding to this job group')


This would print lots of empty spaces. Better to do

Suggested change

self.jobLog.info('Skipping optimizer, since no scout \

corresponding to this job group')

self.jobLog.info('Skipping optimizer, since no scout '

'corresponding to this job group')

fstagni · 2023-11-14T09:26:10Z

src/DIRAC/WorkloadManagementSystem/Executor/Scouting.py

+            result = jobState.getAttribute('RescheduleCounter')
+            if not result['OK']:
+                return result
+            if result['Value'] == None:


Suggested change

if result['Value'] == None:

if result['Value'] is None:

fstagni · 2024-04-23T08:33:36Z

Ping! @qdcampagna

qdcampagna · 2024-06-05T02:17:18Z

Sorry for the long delay, there were problems with the development server used to test this. We are working on setting up a new one so hopefully soon there will be progress on this.

fstagni · 2024-10-31T08:46:30Z

Hello, this PR is now 1 year old, and I do not see much movement.

qcampagn added 11 commits August 28, 2023 03:00

add Scout agent

e1baf3d

Merge remote-tracking branch 'upstream/integration' into ScoutJobs_DI…

7c96e60

…RACv8 Bring up to date

get rid of redundant directory and add back scout agent

15a765e

add Scout Optimizer

1437ffd

realign local and remote directories

0f8adab

remove __future__ imports

84090c7

Merge remote-tracking branch 'upstream/integration' into ScoutJobs_DI…

3040d20

…RACv8 Bring up to date

maintenance: fix formatting for new Scout Optimizer

370bf71

maintenance: fix indenting for Scout Optimizer

c1129d6

maintenance: more format fixing

f980634

maintenance: hopefully last format edit

be35c40

qdcampagna requested review from fstagni and atsareg as code owners October 20, 2023 02:39

fstagni requested changes Oct 20, 2023

View reviewed changes

hmiyake reviewed Oct 20, 2023

View reviewed changes

src/DIRAC/WorkloadManagementSystem/Executor/Scouting.py Show resolved Hide resolved

qcampagn added 7 commits October 21, 2023 06:43

feat: Edit unnecessary/inefficient lines

8ab1c8f

feat: forgot to replace some return result

3a8b1e1

feat: replace modulo with f-strings

e9f41ca

feat: fix bug when reschedule counter not found

ab7447b

feat: get rid of unnecessary siteClient

01706ff

feat: replace JobDB with JobStateUpdateClient

c274e6c

feat: add doc specific to Scouting Agent

c7a61d0

fstagni reviewed Oct 24, 2023

View reviewed changes

feat: easy fixes

cd8fc37

fstagni reviewed Nov 6, 2023

View reviewed changes

andresailer reviewed Nov 6, 2023

View reviewed changes

qdcampagna force-pushed the ScoutJobs_DIRACv8 branch from 36f67d0 to cd8fc37 Compare November 13, 2023 07:01

qdcampagna commented Nov 13, 2023

View reviewed changes

feat: easy fixes

bfd069a

fstagni reviewed Nov 14, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v8.0] Introduce Scout Agent and Optimizer #7251

[v8.0] Introduce Scout Agent and Optimizer #7251

qdcampagna commented Oct 20, 2023 •

edited by fstagni

Loading

fstagni left a comment

fstagni Oct 24, 2023

qdcampagna Oct 30, 2023

fstagni Oct 30, 2023

fstagni Nov 14, 2023

fstagni Oct 24, 2023

fstagni Nov 14, 2023

fstagni left a comment

andresailer left a comment

andresailer Nov 6, 2023

andresailer Nov 6, 2023

qdcampagna Nov 13, 2023

fstagni Nov 14, 2023

fstagni Nov 14, 2023

fstagni Nov 14, 2023

fstagni Nov 14, 2023

fstagni Nov 14, 2023

fstagni Nov 14, 2023

fstagni commented Apr 23, 2024

qdcampagna commented Jun 5, 2024

fstagni commented Oct 31, 2024

	self.jobDB = JobStateUpdateClient()
	self.jobStateUpdate = JobStateUpdateClient()

	scoutjobs = result['Value'].keys()
	scoutjobs = list(result['Value'])

	scoutStatus = {'Status': 'Checking', 'MinorStatus': 'Scouting', 'appstatus': 'Scout Complete'}
	scoutStatus = {'Status': JobStatus.CHECKING, 'MinorStatus': 'Scouting', 'appstatus': 'Scout Complete'}

		self.jobLog.info('Skipping optimizer, since no scout \
		corresponding to this job group')

[v8.0] Introduce Scout Agent and Optimizer #7251

Are you sure you want to change the base?

[v8.0] Introduce Scout Agent and Optimizer #7251

Conversation

qdcampagna commented Oct 20, 2023 • edited by fstagni Loading

fstagni left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fstagni left a comment

Choose a reason for hiding this comment

andresailer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fstagni commented Apr 23, 2024

qdcampagna commented Jun 5, 2024

fstagni commented Oct 31, 2024

qdcampagna commented Oct 20, 2023 •

edited by fstagni

Loading