Best way to deal w/ race condition when starting "singleton" orchestrator from multiple sources

[stevebus]

Hi all,

I have a situation where I'm computing energy prices for facilities for future hours. The energy prices consist of two parts, energy costs and transport costs. For each facility and for each future hour, I'm calculating those costs in parallel and independently via orchestrators/suborchestrators/activity functions. That part works fine. When these functions are finished, they deposit the results of the calculations (with facility ID and DateTime representing the future hour) into a service bus topic.

I need to combine or coalesce these two independent prices into a single 'bundled' price for the facility/future_hour combination. I'm currently trying to do that by creating a singleton/named instanceId orchestrator per facility/hour combo (by creating an instanceId out of the facilityID and serialized datetime -- which might get reused because I do the calculations over and over for a given facility/hour combination). At any given time, there may be hundreds of these calculations occurring concurrently for a bunch of facility/future-hour combinations.

The results of the two independent calculations can come into the SB topic in any order. So my orchestrator basically looks like this...

• when launched, it deserialized the 'first' message that created it
• it creates a 'timer' because I don't want it to run forever if it doesn't get the second message in a timely fashion
• it then waits (Task.WhenAny) on either the timer to expire, or an external event to indicate that it's received the second message
• if the timer 'wins', it logs that fact and exits
• if the ExternalEvent wins, it deserializes the new message, combine the prices and other information, persists the message and exits.

That part works fine too....

The issue is in the SB topic-bound azure function that uses the DurableClient to either create the orchestrator (if it doesn't exist) or raise the external event if it does.

The logic looks basically like this

        DurableOrchestrationStatus status = await client.GetStatusAsync(instanceID);
        if (status == null || status.RuntimeStatus == OrchestrationRuntimeStatus.Completed || status.RuntimeStatus == OrchestrationRuntimeStatus.Terminated || status.RuntimeStatus == OrchestrationRuntimeStatus.Failed)

{
await client.StartNewAsync("BundledTenderOrchestrator", instanceID, input);
}
else
{
await client.RaiseEventAsync(instanceID, "TenderReceived", input);

}

99% of the time this works fine. However, I run into instances where the first message arrives and starts the orchestration, however, the second message arrives (in a parallel instance of the listening azure function) quickly enough that it also tries to start a new instance of the orchestrator because it's still in the "Pending" state (thus, it "passes" the 'if' condition above). That throws an exception stating "An Orchestration instance with the status Pending already exists."

I've tried various combination of adding status.RuntimeStatus == Pending to the 'if' case above with no success and I can't figure out the right logic here. I've also adjusted the maxQueuePollingInterval interval pretty low (2 seconds) because this process only runs once per hour and the default polling interval makes it more likely to have lots of "Pending" orchestrations.

Is there a recommended/foolproof way to handle the race condition where two separate messages/inputs/events may launch a singleton orchestrator at any time?

[vigouredelaruse]

regarding singleton instances the advice from the docs is to use a consistent instance id - that is map instance ids to specific singletons as per

are you reporting inconsistent behaviour if you call StartNewAsync() multiple times with the same instanceid?

[stevebus]

I have a consistent instanceId... And if the previous instance is 'finished' (Complete, Terminated, Failed, etc) creating a new instance with the same ID works fine. That's not really my issue, which I realize may have been a little hard to understand from my description above, but I was trying (and failing) to be succinct.

The problem is that I'm trying to use a singleton orchestrator to "converge" two separate input messages (energy price and transport price) delivered via service bus messages. So, based on some data in the individual messages (facility ID, Future hour datetime, etc), I can create the same "instance id" in both messages, thus allowing me to "point" to the same orchestrator instance. There may be dozens or hundreds of these convergence "singletons" running at any given time, for the different combinations of facility id, hour, etc.

When I receive one of the messages, I check to see if an instance has already been created for it (via client.GetStatusAsync) based on the combination of facilityID, hour, etc. (an example instance ID might look like this -> 125-040524070000 for facility ID 125 on 4/5/2024 at 7AM). If there isn't one, I create one. If there is one already running, I use client.RaiseEventAsync to raise an event to it.

The problem is that sometimes I receive a message, check for existence of an orchestrator for that instance ID, and it doesn't exist so I create it. However, I simultaneously receive the second message, and check for a running instance and doesn't find one (because the first one isn't running yet, it's in "Pending" status), so it tries to create another one with the same instanceID which causes that exception.

I just can't seem to get the logic right to deal with that race condition. It only happens occasionally, but enough to be a problem that I have to address.

[vigouredelaruse]

given

• if the ExternalEvent wins, it deserializes the new message, combine the prices and other information, persists the message and exits
• The problem is that sometimes I receive a message, check for existence of an orchestrator for that instance ID, and it doesn't exist so I create it. However, I simultaneously receive the second message, and check for a running instance

to my ears this problem description starts to sound like desired eventually consistent state acqusition failure, as in
orchestration partially complete -> new signal -> orchestration pending tries to restart on this branch -> orchestration fails
|-> orchestration waits endlessly for signal on this branch -> orchestration fails

if this is a bug and i were trying to get unstuck next i would try

1. storing continuation state messages in a durable entity as well as attempting the signalling the singletons, which you're suggesting is unreliable under load and/or
   2 posting signals back to the queue on pending state caused exceptions

so for
mitigation 1)

• when new messages arrive (if durable entities are in scope for the er technical debt) persist them (in a cache if you want ttl behavior and the cache is available to you)
• executing instances of the singleton can have a timeout that permits checking persisted state and a timeout that abandons the orchestration while still waiting for signals

that might be quicker than coming up with a loadful repro of what you're reporting which by the way i find personally alarming because i have very similar singleton orchestration startup code, hence my interest

whoops i just noticed this description of a race condition bug in the samples which really calls for de rigeur singleton startup/signalling mitigations