Rethink how we sleep in StageTask poll()

[ZNikke]

In order to allow for dsmc to finish setting attributes etc there is a sleep() in the StageTask poll():

dcache-endit-provider/src/main/java/org/ndgf/endit/StageTask.java

Line 121 in 71cf6b2

Thread.sleep(GRACE_PERIOD);

I suspect that this sleep() might be the reason for the somewhat unexpected behavior that the WatchingProvider is so slow, and the observation that the PollingProvider performs much better provided that we allocate a LOT of threads to it.

My reasoning is that although it's a Thread.sleep() it still suspends execution of the thread. This will wreak havoc with the watching provider performance and also increases the likelihood for event overflows. For the polling provider the GRACE_PERIOD of 1000 ms is a direct correlation to the observed performance of the 1-thread-per-Hz of staging performance.

What we really ought to do is something along the lines of:

• When we see a file with a correct size, add it to a FIFO queue together with the current time.
  • Thus allowing the thread to continue processing.
• Process the queue whenever we have a file at output that's been sitting there for more than GRACE_PERIOD
  • This can probably be done in lots of ways, the naive approach of checking the queue every 100ms will probably be good enough although there is likely a more elegant Java-ish way to do it. Since this ties in to the magic mess of Futures I'm at a loss what the best method may be.

I believe this would allow threads to do actual work instead of sleeping all the time.