Question: What async operations are allowed in a durable function?

[dxynnez]

Hi team,

I came across the Orchestrator function code constraints doc and see the following statement:

Orchestrator code must never start any async operation except by using the IDurableOrchestrationContext API, the context.df API in JavaScript, or the context API in Python. For example, you can't use Task.Run, Task.Delay, and HttpClient.SendAsync in .NET or setTimeout and setInterval in JavaScript. The Durable Task Framework runs orchestrator code on a single thread. It can't interact with any other threads that might be called by other async APIs.

I wonder would awaiting an async method (not I/O bound) count against such constraint? We are using AsyncLocal to act as a context provider for our instrumentation, and in order to do that, we need to wrap the actual async operation invoked through the IDurableOrchestrationContext API within another async delegate, something like this:

        [FunctionName(nameof(SampleWorkflow))]
        public async Task SampleWorkflow(
            [OrchestrationTrigger] IDurableOrchestrationContext context)
        {
            await this.Delegator.ExecuteAsync(context, async () =>
            {
                await context.CallActivityWithRetryAsync(...);
                await context.CallActivityWithRetryAsync(...);
                ...
            });
        }

        And the Delegator#ExecuteAsync looks like this:
        private static readonly AsyncLocal<OperationContext> ContextLocal = new AsyncLocal<OperationContext>();

        public async Task ExecuteAsync(IDurableOrchestrationContext context, Func<Task> functionDelegate)
        {
            var correlationContext = context.GetInput<Foo>().CorrelationContext;
            ContextLocal.Value = correlationContext ;
            
            await functionDelegate();
        }
    }

In such case, do you think it would cause any problem? If so, is there any other way to achieve a similar thing (providing a context accessor per function execution scope while not relying on DI)?

[cgillum]

@dxynnez in the code you shared, it doesn't look like you're actually starting any async operations. You're just wrapping normal Durable operations in an async method, which is perfectly fine. Really, the thing that we need to protect against is introducing new threads into the orchestration's execution. The Durable Task Framework must control the single thread of execution in order to ensure determinism. APIs like Task.Delay and HttpClient.SendAsync typically have callbacks execute on worker pool threads, which is what causes problems. That's not happening in your case.

[dxynnez]

Thanks so much @cgillum . That's what I figured. I am relatively new to the .NET async world, could you elaborate a bit more on what counts against the said 'async operation'? Is it only for the APIs under the System.Threading.Tasks namespace plus some other APIs that would spin up a new thread (I know it's probably irrelevant to durable function but it would be great if you can guide me to some docs)? Thanks again!

[cgillum]

No problem @dxynnez! One way I like to think about it don't await a task if you don't know where it's coming from. If it comes from a method on IDurableOrchestrationContext, then it's safe to await. Tasks returned from methods like Task.WhenAll() and Task.WhenAny() are also safe because they don't actually create "new" tasks - they just wrap the tasks you give it. It's also safe for you to create your own method that returns Task as long as its implementation doesn't await or return an "untrusted" Task. Anything else is what I consider "untrusted" because I don't have a way to guarantee that they aren't being resumed on some new thread.

This idea of "trusted" or "safe" tasks is pretty unique to Durable Functions so I don't think there is a lot of useful documentation in the general .NET async that goes into this. There is some generic documentation which might be helpful as context.

[dxynnez]

Thanks @cgillum . That helps!