Poison Message Handling on v2.8.1

[yukscck]

Hi, in v2.8.1 release, there is finally the poison message handling being introduced.

From what I understand, the handling is targeted for a very specific scenario that would result in poison messages in the Durable Functions queues. I also noticed that the queue message pop receipts are now present in a few of the events, e.g. the ReceivedMessage, DuplicateMessageDetected, etc.

As the current support for poison message handling does not cover all cases, I would like to check whether it is a good idea or not advisable to build our own System.Diagnostics.Tracing.EventListener within the Durable Functions function app to check for messages with high dequeue count and remove / move these messages somewhere else for our own handling? Thanks!

[cgillum]

It's an interesting idea, but I can't think of how you'd actually remove / move the message since your application code doesn't have access to it. We don't currently have any hooks that would allow you to inject your own logic into the message handling pipeline.

[yukscck]

With the access not being exposed through the Durable Functions' APIs, my thought was to instantiate a QueueClient and delete the message from there. We have the necessary information from the events: PartitionId (queue name), MessageId and the PopReceipt.

It feels like going against the purpose of the abstraction of the storage provider in Durable Functions, since we are explicitly accessing the storage queue. But by hooking up the EventListener outside of the actual functions' codes, we try to keep this implementation as isolated from the functions execution as much as possible.

One thing to point out is that we are bound to get such warning messages from the Durable Functions since the queue message no longer exists: Failed to update or delete message [TaskCompleted#2] with ID = 5f5cb21d-7aaa-4cea-bd82-b701d325186e because it no longer exists

Really keen to hear the thoughts from the team, whether this is advisable.

internal class DurableStorageEventListener : EventListener
{
    private readonly QueueServiceClient queueServiceClient;

    public DurableStorageEventListener(QueueServiceClient queueServiceClient)
    {
        this.queueServiceClient = queueServiceClient;
    }

    // ...

    protected override void OnEventWritten(EventWrittenEventArgs eventData)
    {
        switch (eventData.EventName)
        {
            case "ReceivedMessage":
                {
                    if ((int)eventData.Payload[8] > 10) // 8 - DequeueCount
                    {
                        this.queueServiceClient
                            .GetQueueClient((string)eventData.Payload[11])? // 11 - PartitionId (Queue Name)
                            .DeleteMessage((string)eventData.Payload[6], (string)eventData.Payload[13]); // 6 - MessageId; 13 - PopReceipt
                    }

                    break;
                }
            // ...
        }
    }
}

[cgillum]

Ah, I understand now. Yeah, this is an interesting solution. My only question would be whether the pop receipt is still considered valid by Azure Storage if your code runs after we've abandoned the message back to the queue. I can't think of a concrete reason why it wouldn't work, but it would need to be tested and may be subject to race conditions.

It feels like going against the purpose of the abstraction of the storage provider in Durable Functions, since we are explicitly accessing the storage queue

Yes, so in that sense, it's a risky strategy because it makes certain assumptions about the internal implementation details of Durable Functions which aren't guaranteed to be always true.

One thing to point out is that we are bound to get such warning messages from the Durable Functions since the queue message no longer exists

Whether you see these kinds of warnings depends on the specific timing of this delete call. For example, right now we emit these log messages before taking action on the message. If your code acts quickly, it may delete the message before we have a chance to update it, resulting in a warning. If the message is deleted after being released by the framework, then you might not see any warnings.