The parallel activity functions of durable function

[maonaoda]

Can I still use Task.WhenAll() when I need to execute tens of thousands activity functions of times?
Or do I need to reduce the number of parallelism by looping?

[cgillum]

It's possible to fan-out/fan-in tens of thousands of activities using a single Task.WhenAll(). We do this in our own internal performance tests using "Hello, world"-style activities. A downside to doing this are that the orchestration may become a bottleneck for overall throughput. It may also have a big memory impact on your app if the orchestration has to keep all the activity results in memory in order to complete. Using extended sessions can be helpful to mitigate some of these problems if your activity functions are short-lived.

A simple alternative that addresses these problems in a fundamental way is to break up your big orchestration into smaller orchestrations. For example, if you need to run 100K activities in parallel, you can create 100 sub-orchestrations from your main orchestration, each of which fans out to 1K parallel activities. This provides better overall distribution of memory and CPU used by orchestrations across all nodes, removing the bottleneck I mentioned earlier.

Looping is also an option, but it doesn't solve the high memory usage problem and will result in higher overall latency since you're not taking as much advantage of parallelism. However, it has the advantage of being simpler to implement and may reduce the overall load across all your VMs.

[maonaoda]

Thanks @cgillum.
Awesome, I broke my orchestration into smaller orchestrations that can run 500 times activities in parallel.

Do the smaller orchestrations need to be executed sequentially, or in parallel?
if sequentially, may be using Extended sessions will helpful to avoid the orchestrator function replay in loop?

Because the number of my activities is uncertain, so the orchestrations are also non-fixed.

the out/input of Fans out/input will be List that can be large, does this mean that this will result in more consumption of LRS Write/Read Operations?

eg:
orchestration has orchestrationA and orchestrationB.

orchestrationA is getting all data(List) from DB.
orchestrationA use smaller orchestrations that can run 500 times activities in parallel to get and merge the output(List) into all data.

orchestrationB is saving all data from orchestrationA into DB.
orchestrationB　use smaller orchestrations　that can run 500 times activities in parallel to save data from orchestrationA;

I think in the above example, when the Orchestrator function is replayed, there should cost large consumption of LRS Write/Read Operations.

I'm so sorry I asked a lot, thank you again.

[davidmrdavid]

Hi @maonada:

If you're splitting a large When.All into sub-orchestrators, then it makes sense to me that those sub-orchestrators would also be invoked in parallel. After all, if there's no dependency between the original operations (as they were parallelizable before), then it should be ok to run these sub-orchestrators in parallel as well.

On another note: I'm a bit confused about your point on LRS Write/Read Operations. Are you talking about Locally Redundant Storage when you say LRS? Just trying to clarify the terminology :-)

[maonaoda]

@davidmrdavid

If you're splitting a large When.All into sub-orchestrators, then it makes sense to me that those sub-orchestrators would also be invoked in parallel. After all, if there's no dependency between the original operations (as they were parallelizable before), then it should be ok to run these sub-orchestrators in parallel as well.

Thanks you.Since there will be reading and writing to the database in sub-orchestrators, I think it is better to synchronize.

On another note: I'm a bit confused about your point on LRS Write/Read Operations. Are you talking about Locally Redundant Storage when you say LRS? Just trying to clarify the terminology :-)

Sorry. In fact, I'm not very sure, I just found these consumptions in my last durable function, so I take it for granted that
Fans out/input with a large amount of data will increase this consumption.

[davidmrdavid]

Thanks for the response @maonaoda.
Regarding the LRS terminology, I can't comment on that as I'm not very familiar with that terminology.
Maybe @cgillum knows?