Allow using specific message queues for each orchestrations

[Greybird]

Hi,

Context

We have a scenario, where we are using DF to orchestrate business processes and reports across a bunch of services in charge of various aspects of the data and logic.
These reports and business processes can be triggered by a user request (click on a UI button), or from system initiated events (scheduler for example).

We encounter an issue where a specific orchestration can fill the message queue to a point that another orchestration will not start running after all these messages are processed.
Example is a huge report filling the queue with a lot of tasks, while a few seconds later, a user is asking for a fairly simple process to be executed as the consequence of clicking a button, waiting for the result.

We also will have to implement processes in the future, where the same logic will be shared between massive batches and manual interactive unitary triggers.

Current mitigation of the issue

We took the decision to create two function apps, each one with its hub (and queue), one for interactive orchestrations, one for reports ones, both of them being able to run all the orchestrations.
This solved the issue for the moment, but, as we look into the future, we anticipate that this current setup will not scale well with the addition of more processes, as they will impact each others through the unique queue usage in each scenario (reports/interactive).
We are wondering if this will lead us to create a function app for each orchestration, leading to a somewhat uncomfortable situation in terms of hosting that we did not expect.

Alternatives

Previously, we were using MassTransit sagas to orchestrate these process, and as each saga had its own queue, we did not face the issue.

Idea/Proposal

It would be great to be able to separate the queues used by orchestrations.
This could be done either

• either by the DF framework internal, each orchestration having its own specific queue,
• or through a configuration in the code, stating which queue to use for each orchestration.
  The second option would probably be better to ensure retro-compatibility, while allowing more control on the sharing of queues.

A bonus feature would be to be able to direct the orchestration to a given queue at runtime, allowing the use of specific queues depending on if the process has to be executed quickly due to its interactive nature, or is a part of a massive batch that can be delayed).

Do you think this is a direction that you would be willing to take in the future ?

[olitomlinson]

@Greybird

This has been raised a few times in the past, and as you quite rightly point out, one orchestrations activities can saturate the queues and other orchestration instances must get in the queue and wait their turn.

A few thoughts, hopefully one will fit.

1. lean on Infrastructure-as-code : What has worked well for me is to invest in IaC, so that spinning-up a new Function App and Storage Account (for the Task Hub) is a frictionless process. This has allowed me and my team to light-up several Function Apps which represent different isolated processes/domains.
2. Share a storage account : Following the same principles as above, you can have many Function Apps share the same Storage Account if you wish. A Function App will create its own TaskHub if doesn't exist at runtime, which makes it significantly easier to spin up new Function Apps for each use-case. Just make sure to set an explicit TaskHub name in host.json in each Function App. However at some point this pattern may not scale as you start hitting the 20,000 transaction limit per second on a Storage Account.
3. SQL Provider for Durable Task : This may be a neat fit for your use case as work is not partitioned into queues as per the Azure Storage variant of Durable Task, so theoretically by dialling up the SQL server bandwidth you can increase breadth and process many things at the same time. It's currently in early preview, but @cgillum is very keen to get this into GA.
4. Set user expectations that things take time : This may not meet your use-case or be relevant to your situation, but having worked on a multi-tenant SaaS for many years, we have learned to set expectations for the users that operations may take some time. A problem I often come across is that users learn to expect things to happen in real-time, even if they don't necessary need them to happen in real-time - which becomes a very hard thing to maintain as you scale up with more customers and traffic. If you can, embrace async communication patterns to the user such like "We've got you're request, we will drop you an email when its done" kinda-thing. Consequently, If things do take too long, customers will tell you, and they will tell you in the tens and hundreds, which provides a remit to invest time and resource in fixing - I would call this enabling effective Product and Tech to make better decisions around what is important to customers. Apologies in advance if this is not relevant at all :)

Option 3 sounds like your best bet for a long-term solution IMO.

[Greybird]

Thanks for your very valuable feedback.

We'll look into option 3, of which we were unaware of, and seems very promising!

2. is the way we went with our separation, and 1. is difficult to put in place as this is not only a matter of automation, but also of other constraints we have on our applications hostings.
3. is indeed also something we are looking into, but we would also like to do choices not only because of the tech stack, or at least try to.

Thanks again!