Non-Deterministic workflow detected

[anhhnguyen206]

Hello,

I have been running into this weird error on local (haven't seen this on Azure) and I'm wondering if someone could provide some insights.

The error that I ran into:
Unhandled exception while executing orchestration: DurableTask.Core.Exceptions.NonDeterministicOrchestrationException: Non-Deterministic workflow detected: A previous execution of this orchestration scheduled an activity task with sequence number 2 named 'GetItemByIdActivityTrigger', but the current orchestration replay instead scheduled a CreateTimerOrchestratorAction task with this sequence number. Was a change made to the orchestrator code after this instance had already started running?

I have reviewed the code against Orchestrator contraints and I don't see anything obvious. What makes me curious is the error clearly point to a CreateTimerOrchestratorAction confusion during the a replay. There is no place in the orchestrator that it tries to create a timer. Looking at the history I can see a TimerCreated event was appended to the history right before it failed.

Any insights would be much appreciated!

[cgillum]

It looks like this error message needs to be fixed. What it's trying to say is that a previous execution scheduled the "GetItemByIdActivityTrigger" activity as its 3rd action (the sequence numbers are zero-based). However, the current execution appears to be trying to schedule a timer as its 3rd action. If your orchestrator code doesn't schedule any timers explicitly, it could be that you're using retry policies, which schedule timers between retries implicitly. If you're using retry policies, check to make sure the retry count configuration is deterministic (i.e., hardcoded, based on some orchestration input, or from some activity output). If it is deterministic but you're still seeing this, then check your source code history to see if it may have been changed between versions.

[anhhnguyen206]

@cgillum Hey Chris,

Thanks for the quick response! I do have retry policies but they all have static configuration. Also the activities / sub-orchestrators executed successfully without errors and I can see the output in the history. I have also looked at our source code but I don't think we make any changes that could have cause it to become non-deterministic. I even went further and reduce all conditional logic in this particular orchestrator to only perform 4 tasks yet it still failed.

I'll be digging deeper and report back.

[Muzammil-km]

@anhhnguyen206 Hi Anh Nguyen,

I recently encountered the same issue that you reported. If you've found any insights, could you share how it was resolved and what steps were taken to address it? Your insights would be a great help.

Thank you!