Extensions RPC lifetime mismatch between server and JobHost leads to incorrect RPC calls

[jviau]

The current design of the extension RPC server has it piggy backing off the general worker gRPC server. The lifetime mismatch between individual workers and this server causes issues during job host transitions. We will need explore one of two options:

1. Improve lifetime tracking with the existing RPC server
   • Only unregister endpoints when functions have finished draining.
   • Support multiple job hosts being registered. Will need to include an identifier for what set of endpoints an extension call needs to be routed to.
2. OR move these endpoints into a new server that lives in the job host.
   • Will need to measure the impact an extra server startup will have.

[goldmelodyvn]

Vote up. We have a lot of RPC exception when send a bulk of messages to Azure message bus.
When we did manual testing ( a few message we don't have this issue). When we did Load test with around 6000 messages. We got 9.50k Rpc exception as below:

Exception while executing function: Functions.[xxx] Result: Failure
Exception: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Azure.Functions.Worker.Extensions.Abstractions, Version=1.3.0.0, Culture=neutral, PublicKeyToken=551316b6919f366c'. The system cannot find the file specified.
File name: 'Microsoft.Azure.Functions.Worker.Extensions.Abstractions, Version=1.3.0.0, Culture=neutral, PublicKeyToken=551316b6919f366c'
   at System.ModuleHandle.ResolveType(QCallModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
   at System.ModuleHandle.ResolveTypeHandle(Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
   at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
   at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(MetadataToken caCtorToken, MetadataImport& scope, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1& derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctorWithParameters, Boolean& isVarArg)
   at System.Reflection.CustomAttribute.AddCustomAttributes(ListBuilder`1& attributes, RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1 derivedAttributes)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeParameterInfo parameter, RuntimeType caType)
   at System.Reflection.RuntimeParameterInfo.GetDefaultValueFromCustomAttributes()
   at System.Reflection.RuntimeParameterInfo.TryGetDefaultValueInternal(Boolean raw, Object& defaultValue)
   at System.Reflection.RuntimeParameterInfo.get_HasDefaultValue()
   at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition.<.ctor>b__2_7(ParameterInfo p) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 59
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.ToArray()
   at System.Collections.Immutable.ImmutableArray.CreateRange[T](IEnumerable`1 items)
   at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition..ctor(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 56
   at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToFunctionDefinition(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\RpcExtensions.cs:line 118
   at Microsoft.Azure.Functions.Worker.GrpcWorker.FunctionLoadRequestHandler(FunctionLoadRequest request, IFunctionsApplication application, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 268
Stack:    at System.ModuleHandle.ResolveType(QCallModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
   at System.ModuleHandle.ResolveTypeHandle(Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
   at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
   at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(MetadataToken caCtorToken, MetadataImport& scope, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1& derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctorWithParameters, Boolean& isVarArg)
   at System.Reflection.CustomAttribute.AddCustomAttributes(ListBuilder`1& attributes, RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1 derivedAttributes)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeParameterInfo parameter, RuntimeType caType)
   at System.Reflection.RuntimeParameterInfo.GetDefaultValueFromCustomAttributes()
   at System.Reflection.RuntimeParameterInfo.TryGetDefaultValueInternal(Boolean raw, Object& defaultValue)
   at System.Reflection.RuntimeParameterInfo.get_HasDefaultValue()
   at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition.<.ctor>b__2_7(ParameterInfo p) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 59
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.ToArray()
   at System.Collections.Immutable.ImmutableArray.CreateRange[T](IEnumerable`1 items)
   at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition..ctor(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 56
   at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToFunctionDefinition(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\RpcExtensions.cs:line 118
   at Microsoft.Azure.Functions.Worker.GrpcWorker.FunctionLoadRequestHandler(FunctionLoadRequest request, IFunctionsApplication application, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 268 

[jviau]

@goldmelodyvn that call stack is not related to this issue. Please open a separate issue.

[nusliew]

@jviau Referring to this issue (#10251), may I know if there is an ETA for rollout? My host function version is 4.36 currently. Also, I would like to know any workaround is available?