Ideas to flesh out Durable Lifecycle functionality

[ConnorMcMahon]

As of issue creation (July 2021), Durable Functions has the ability to subscribe to various lifecycle events for orchestration and publish those events to EventGrid.

Lifecycle events are a great way to detect various anomalies within the system, and this is an area where Durable Functions customers could use a lot of help. However, there are many ways we can take our existing functionality and make it more robust. I see two separate fronts to tackle this issue.

More lifecycle events

Currently we support the following orchestration life-cycle events: Started, Pending, Failed, Terminated.

Some additional potential life-cycle events include:

• Entity creation/deletion
• Orchestration execution time (i.e. orchestration has been running for more than one hour)
• Stale orchestrations (orchestration hasn't received an update within 10 minutes)
• Orchestration custom status hit certain value
• Duplicate instance id fired
• Specific entity operation fired

More actions available on lifecycle events

Currently we only allow publishing to EventGrid. Other possible actions that would greatly expand the scope of the feature include:

• Emit data to Azure Monitor (or maybe App Insights, and then configure Azure monitor to look in app insights)?
• Send data to a webhook or some other Server Sent Event (SSE) endpoint
• Make lifecycle events an Azure Functions/WebJobs trigger!

[davidmrdavid]

Big +1 to more observability! 🌈

I think a good place to start would be to figure out how we expect users to make use of / respond to these lifecycle events beyond just monitoring. For example, it seems to me that a main use-case of these events would be to take corrective action when a DF app gets into an undesirable state. Is it accurate to say that would be the main goal of this observability layer? For the sake of discussion, the rest of my response will assume that taking corrective action is the main goal here.

I'm 100% in support of making lifecycle events into a Trigger! I would personally prioritize that over sending the data to Azure Monitor or App Insights; a Trigger seems more flexible in letting users take corrective actions.

I also think that your suggestion of events identifying stale orchestrations, duplicate instanceIds, and orchestration execution time would be great to have. To this, I'd add something like duplicate activity executions.

I would also add that we should consider if we should add more management APIs to let users respond to these events. Other than the Terminate API, what else can customers do to get themselves unstuck? This is where a "pause" and "rewind" API might come in handy :)

[olitomlinson]

I disagree with prioritising some of these things over Azure Monitor integration.

From my perspective based on experiences with Function Apps and DF, when things happen like an orchestration hasn't received an event in X minutes, or an Orchestration has is still in a running state after X minutes, this points to something that needs investigation.

1. Having this data automatically published to Azure Monitor as first class signals allows me to quickly hook up an Azure Monitor Alert and get my team to look at it.

Having a trigger (in the function App) doesn't help me one bit as I've still got to manually publish that signal to Azure Monitor via App Insights. You're making me write what I consider boilerplate code, which is just not where I want to spend dev effort.

2. Another thing, I've had issues where a Function App has become unresponsive for one reason or another, my worry here is by using a Trigger on the function App you're effectively making the App rely on itself to be able to process triggers? If the app is stuck, no processing, no triggers, no code gets executed, no App Insights gets published.