Fix metadata logging even when actors crash #721
Conversation
updated. In case the ray actors crash, the key `processing_time` may not be updated to stats, so while rounding off this key, we may get an exception which stops metadata from being written. Signed-off-by: Shivdeep Singh <[email protected]>
| @@ -142,7 +142,8 @@ def orchestrate( | |||
| # Compute execution statistics | |||
| logger.debug("Computing execution stats") | |||
| stats = runtime.compute_execution_stats(ray.get(statistics.get_execution_stats.remote())) | |||
| stats["processing_time"] = round(stats["processing_time"], 3) | |||
| if "processing_time" in stats: | |||
There was a problem hiding this comment.
should we set this to 0 if not found in case downstream code depends on it?
There was a problem hiding this comment.
I think, value 0 for processing_time can be misleading.
There was a problem hiding this comment.
So a missing key downstream means what? I suspect this really means we're not properly handling the crashing of the actor and should indicate that in some other way, perhaps with a graceful shutdown.
There was a problem hiding this comment.
The only way stats["processing_time"] = round(stats["processing_time"], 3) can throw an exception is in the case where either:
- init method in file processor fails due to inability to create transform
- All of the transform invocations failed and "processing time" was never reported by file processor
In both cases, there is really no statistics of interest to be collected.
But if you really want to avoid this exception the right thing to do (in my opinion) is to do what Shivdeep is doing:
if "processing_time" in stats:
stats["processing_time"] = round(stats["processing_time"], 3)
This will return metadata with no processing time value, which is correct in this case
Note that if you want to go for this fix, please fix it in all 3 orchestrators - Python, Ray and Spark. Not just Ray
Why are these changes needed?
In case the ray actors crash, the key
processing_timemay not be updated to stats, so while rounding off this key, we may get an exception which stops metadata from being written.Related issue number (if any).
Also seen in the crash here. #719