[BUG] extract fails with DefaultAzureCredential while works with AZURE_BEARER_TOKEN

[dabedin]

Release version

apiops v.4.10.3

Describe the bug

Trying to extract via command line, leveraging the default context deriving from a valid az cli session, with the parameters API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH AZURE_SUBSCRIPTION_ID, AZURE_RESOURCE_GROUP_NAME and API_MANAGEMENT_SERVICE_NAME, the command fails with

 Azure.Identity.AuthenticationFailedException: ManagedIdentityCredential authentication failed: Service request failed.
      Status: 503 (Service Unavailable)

See the full trace below.

If, instead of leveraging the default credentials, I first get a token via az account get-access-token --subscription <sub id> and pass it to extractor, it works as expected.
extractor --API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH <path> --AZURE_SUBSCRIPTION_ID <sub id> --AZURE_RESOURCE_GROUP_NAME <rg> --AZURE_BEARER_TOKEN <obtained token>

Expected behavior

Leveraging DefaultAzureCredential should work the same as explicitly passing a token obtained from the same az cli context.

Actual behavior

info: Extractor[0]
      Beginning execution...
info: Extractor[0]
      Exporting named values...
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\Users\utente\workspace\apigateway_code\azure
crit: Extractor[0]
      Azure.Identity.AuthenticationFailedException: ManagedIdentityCredential authentication failed: Service request failed.
      Status: 503 (Service Unavailable)

      Headers:
      Connection: close
      Date: Wed, 08 Nov 2023 09:34:30 GMT

See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot
       ---> Azure.RequestFailedException: Service request failed.
      Status: 503 (Service Unavailable)

      Headers:
      Connection: close
      Date: Wed, 08 Nov 2023 09:34:30 GMT

         at Azure.Identity.ManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
         at Azure.Identity.ImdsManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
         at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
         at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
         at Azure.Identity.ManagedIdentityClient.AuthenticateCoreAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
         at Azure.Identity.ManagedIdentityClient.AppTokenProviderImpl(AppTokenProviderParameters parameters)
         at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.SendTokenRequestToProviderAsync(CancellationToken cancellationToken)
         at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.FetchNewAccessTokenAsync(CancellationToken cancellationToken)
         at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
         at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
         at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenForClientParameters clientParameters, CancellationToken cancellationToken)
         at Azure.Identity.AbstractAcquireTokenParameterBuilderExtensions.ExecuteAsync[T](AbstractAcquireTokenParameterBuilder`1 builder, Boolean async, CancellationToken cancellationToken)
         at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientCoreAsync(String[] scopes, String tenantId, Boolean enableCae, Boolean async, CancellationToken cancellationToken)
         at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientAsync(String[] scopes, String tenantId, Boolean enableCae, Boolean async, CancellationToken cancellationToken)
         at Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
         at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
         at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
         at Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
         at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
         at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
         at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
         at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
         at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
         at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueFromCredentialAsync(TokenRequestContext context, Boolean async, CancellationToken cancellationToken)
         at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
         at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
         at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequestAsync(HttpMessage message, TokenRequestContext context)
         at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
         at common.HttpPipelineExtensions.GetContent(HttpPipeline pipeline, Uri uri, CancellationToken cancellationToken)
         at common.HttpPipelineExtensions.GetJsonObject(HttpPipeline pipeline, Uri uri, CancellationToken cancellationToken)
         at common.HttpPipelineExtensions.ListJsonObjects(HttpPipeline pipeline, Uri uri, CancellationToken cancellationToken)+MoveNext()
         at common.HttpPipelineExtensions.ListJsonObjects(HttpPipeline pipeline, Uri uri, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator`2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 221
         at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
         at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
         at System.Linq.AsyncEnumerable.WhereEnumerableAsyncIterator`1.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Where.cs:line 233
         at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
         at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
         at System.Threading.Tasks.Parallel.<>c__54`1.<<ForEachAsync>b__54_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at common.IAsyncEnumerableExtensions.ForEachParallel[T](IAsyncEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
         at extractor.NamedValue.ExportAll(ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, GetRestResource getRestResource, ILogger logger, IEnumerable`1 namedValueNamesToExport, CancellationToken cancellationToken)
         at extractor.Service.Export(ServiceDirectory serviceDirectory, ServiceUri serviceUri, DefaultApiSpecification defaultSpecification, IEnumerable`1 apiNamesToExport, IEnumerable`1 loggerNamesToExport, IEnumerable`1 diagnosticNamesToExport, IEnumerable`1 namedValueNamesToExport, IEnumerable`1 productNamesToExport, IEnumerable`1 backendNamesToExport, IEnumerable`1 tagNamesToExport, IEnumerable`1 subscriptionNamesToExport, IEnumerable`1 policyFragmentNamesToExport, ListRestResources listRestResources, GetRestResource getRestResource, DownloadResource downloadResource, ILogger logger, CancellationToken cancellationToken)
         at extractor.Extractor.ExportService(CancellationToken cancellationToken)
         at extractor.Extract

Reproduction Steps

apiops v.4.10.3
az cli 2.53.1

Logged in with az cli as a contributor on APIM.
Launching with extractor.win-x64.exe --API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH <path> --AZURE_SUBSCRIPTION_ID <proper id> --AZURE_RESOURCE_GROUP_NAME <proper rg> return the exception above.

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[guythetechie]

Hi @dabedin,

Our code to get the credential is located here.

If AZURE_BEARER_TOKEN exists in configuration (YAML, env var, executable argument, etc), we use it. Otherwise, we use the DefaultAzureCredential with the correct Azure environment (public cloud, US Government, etc). We don't do anything special with the DefaultAzureCredential.

As you can see in the documentation, DefaultAzureCredential tries many things prior to the Azure CLI credential. Your error message suggests it's trying failing when trying the ManagedIdentityCredential. Are you running this on a machine that has a managed identity? If so, does that managed identity have permissions on the APIM instance?