Durable Function (starter_function) throwing "Forbidden" Exception

[wkahseng-smrt]

We are experiencing a "Forbidden" exception when trying to run Durable Functions within our Function App.

Our Function App has VNet Integration turned on. Required resources (i.e. Blob Storage, Cosmos MongoDB and SQL Storage) have private endpoints created within the same VNet so these resources should also be reachable from within the function app.
RBAC roles has also been assigned to our Function App: Storage Blob Data Contributor, Storage Queue Data Contributor and also Storage Table Data Contributor.

We first suspected that our Function App did not have permission to access the Blob Storage, but it seems that other kind of functions are able to read and write files/blobs into the Blob Storage. So, this issue might only be seen on when trying to execute Durable Functions.

The starter_function invocation is as follows:

Result: Failure Exception: Exception: {'Message': 'Something went wrong while processing your request', 'ExceptionMessage': 'Forbidden', 'ExceptionType': 'DurableTask.AzureStorage.Storage.DurableTaskStorageException', 'StackTrace': ' at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 141\n at DurableTask.AzureStorage.Storage.Table.CreateIfNotExistsAsync() in /_/src/DurableTask.AzureStorage/Storage/Table.cs:line 50\n at DurableTask.AzureStorage.AzureStorageOrchestrationService.GetTaskHubCreatorTask() in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 337\n at DurableTask.AzureStorage.AzureStorageOrchestrationService.EnsureTaskHubAsync() in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 302\n at DurableTask.AzureStorage.AzureStorageOrchestrationService.CreateTaskOrchestrationAsync(TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1627\n at DurableTask.Core.TaskHubClient.InternalCreateOrchestrationInstanceWithRaisedEventAsync(String orchestrationName, String orchestrationVersion, String orchestrationInstanceId, Object orchestrationInput, IDictionary`2 orchestrationTags, OrchestrationStatus[] dedupeStatuses, String eventName, Object eventData, Nullable`1 startAt) in /_/src/DurableTask.Core/TaskHubClient.cs:line 619\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient.Microsoft.Azure.WebJobs.Extensions.DurableTask.IDurableOrchestrationClient.StartNewAsync[T](String orchestratorFunctionName, String instanceId, T input) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\ContextImplementations\\DurableClient.cs:line 213\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.HandleStartOrchestratorRequestAsync(HttpRequestMessage request, String functionName, String instanceId) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\HttpApiHandler.cs:line 875\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.HandleRequestAsync(HttpRequestMessage request) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\HttpApiHandler.cs:line 306'} Stack: File "/azure-functions-host/workers/python/3.11/LINUX/X64/azure_functions_worker/dispatcher.py", line 491, in _handle__invocation_request call_result = await self._run_async_func( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/azure-functions-host/workers/python/3.11/LINUX/X64/azure_functions_worker/dispatcher.py", line 774, in _run_async_func return await ExtensionManager.get_async_invocation_wrapper( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/azure-functions-host/workers/python/3.11/LINUX/X64/azure_functions_worker/extension.py", line 147, in get_async_invocation_wrapper result = await function(**args) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/site/wwwroot/df_events_starter/__init__.py", line 46, in main instance_id = await client.start_new("pms_orchestrator", None, events_json) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/durable_functions/models/DurableOrchestrationClient.py", line 89, in start_new raise Exception(ex_message)

Not sure what other type of authorization is needed as these are the insights we have from our storage account:

Any help is appreciated.

[lorenzotan]

I'm experiencing the same problem. Were you ever able to figure this out?

[lorenzotan]

For me, this ended up being a problem with permissions. I am using managed identities to access the storage account. Although my function does not use the queue or table services, the managed identity still needs permissions to access them. Once I added the appropriate roles (blob, queue and table contributor) to the managed identity, this error was fixed.

[cgillum]

Our Function App has VNet Integration turned on. Required resources (i.e. Blob Storage, Cosmos MongoDB and SQL Storage) have private endpoints created within the same VNet so these resources should also be reachable from within the function app.

You mentioned blob storage, but do you have the correct configuration for table and queue storage services? The stack trace in the error message specifically points to the Azure Tables service.

[lorenzotan]

I was focused on my own issue and overlooked that detail. My function doesn't use the queue and table services so they are set at the default configuration.