Unit testing the retry logic of IDurableOrchestrationContext.CallActivityWithRetryAsync

[robledop]

Is there a way of way creating unit tests for IDurableOrchestrationContext.CallActivityWithRetryAsync to cover the scenarios where the activity fails? I need to unit test the retry logic, but I can't do something like this:

var retryOptions = new RetryOptions(
            firstRetryInterval: TimeSpan.FromSeconds(5),
            maxNumberOfAttempts: 3);
var context = new Mock<IDurableOrchestrationContext>();
context.Setup(x => x.CallActivityWithRetryAsync("Function1", retryOptions, "test"))
            .Throws<Exception>();

Because then it is CallActivityWithRetryAsync itself that will throw the exception and not Function1.

Thanks,
Robledo Pazotto

[olitomlinson]

@robledop

I'm not an authority on this, but from user-code perspective any Exception from Function1 will be marshalled back to the Orchestration, so having CallActivityWithRetryAsync throw the exception directly is semantically the same thing as Function1 throwing the exception.

@cgillum did I explain that correctly?

---

Also I've just picked up on this

"I need to unit test the retry logic"

Do you really need to unit test the retry behaviour of the framework that it actually does the retries? Or do you just need to verify that your mock is called with a correctly configured RetryOptions object ?

IMO I don't see the value in testing the framework does its job correctly. You should just focus on testing your user-code paths i.e. what happens when CallActivityWithRetryAsync throws an exception.

[cgillum]

I think one valid reason for unit testing the retry logic is to ensure that your retry options parameters are working the way you intend them to work. For example, I've seen some folks get confused about the exact meaning of some of the RetryOptions parameters. A unit test could theoretically help you verify that you configured things to work they way you thought they would work.

[olitomlinson]

Hmmm I’ll share my opinion…

Unit testing is about your user-code, you test up to any dependencies. In this case the context (IDurableOrchestrationContext) is a dependency of the unit (Assuming the SUT is a Function) - sure Verify against that dependency if you want to be certain that you’re using the correct configuration.

I think any confusion with how to use RetryOptions is a separate matter, (a matter of API surface and documentation!). I wouldn’t implement against a 3rd party dependency purely on assumption and then rely on unit tests to prove correctness of the solution.

Chris, I assume there are tests in the DF extension project that ensures the various configurations and behaviours of RetryOptions are correct? Why would I, as the consumer, want to test that too?

Thought experiment : when writing business logic, how often have you unit tested the MOD operator in C# for correctness? Or one of the built in Polly.NET retry Policies? My expectation is that you just don’t, because the maintainer ensure that behaviour works.

As you pointed out in the other answer, an integration test is what’s needed.

Unfortunately, I gave up long ago trying to find a way of doing this locally with the runtime. So instead always deploy to a real Az function cloud environment and run tests at as part of a wider acceptance E2E test suite.

[cgillum]

Unfortunately, there doesn't appear to be a good way to unit test retry behavior. You may need to instead look into integration tests for this kind of coverage. This is probably something that we should consider improving in the future.