WAIT_PERIOD_JOB_FS_SYNC might be ignored due to Node time differences

[Atticus1806]

We are currently investigating a problem, where potentially the following task was started before writing the previous tasks file was finished.
From our understanding this should be prevented by WAIT_PERIOD_JOB_FS_SYNC=30 in the global settings. But after having a look at the code we found that if one of the tasks is a minitask and the other is on a node the system time difference between both nodes might be bigger than 30 seconds, which should then "ignore" this constant.
While we do not really have a solution here, we just wanted to make aware of the problem. Do you maybe have an idea how to solve this?
cc. @christophmluscher @dthulke JanRosendahl

[christophmluscher]

@critias @curufinwe you guys have any ideas? Suggestions?

[critias]

Does synchronizing the server clock using NTP count as suggestion?
You could increase the wait period even further, but in the end if the offset between different servers is bigger than WAIT_PERIOD_JOB_FS_SYNC I don't see a simple way how to ensure the next job isn't started immediately.

Is it possible to get the time a job finished directly from the SGE server (not from the timestamp in the log file)?

In the end these are all methods to work around the basic problem that the server clocks should be synchronized...

Another way would be to write the file size of each file when a job finished and check before the next job starts if the file size matches. This would require quite a bit of code, cause more file system accesses, and it's unclear how directories and path prefixes should be handled, but should result in a stable solution independent of any filesystem sync issues for all checked files. I started some work on this a while ago with @mirkovogel but we didn't finish it since the problem went away in our case (maybe someone fixed the NTP setup 😄).
master...check-output-size