[Bug]: panic http multiple registrations for /oidc/callback

[PavGregor]

Terraform Core Version

1.9.5

Terraform Vault Provider Version

4.4.0

Vault Server Version

1.16.5+ent

Affected Resource(s)

vault_aws_secret_backend_static_role

Expected Behavior

Terraform should create vault_aws_secret_backend_static_role.role resource.

Actual Behavior

Terraform crashes with the below error.

Relevant Error/Panic Output Snippet

module.test.vault_aws_secret_backend_static_role.role: Creating...
╷
│ Error: Plugin did not respond
│ 
│   with module.test.vault_aws_secret_backend_static_role.role,
│   on ../modules/test/main.tf line 6, in resource "vault_aws_secret_backend_static_role" "role":
│    6: resource "vault_aws_secret_backend_static_role" "role" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange
│ call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-vault_v4.4.0_x5 plugin:

panic: http: multiple registrations for /oidc/callback

goroutine 28 [running]:
net/http.(*serveMux121).handle(0x2550ed0, {0x1543e7a, 0xe}, {0x1b207e0, 0xc000a4a800})
        net/http/servemux121.go:59 +0x20e
net/http.(*serveMux121).handleFunc(...)
        net/http/servemux121.go:96
net/http.HandleFunc({0x1543e7a?, 0x0?}, 0xc00087ad10?)
        net/http/server.go:2725 +0x4f
github.com/hashicorp/vault-plugin-auth-jwt.(*CLIHandler).Auth(0xc0007be120?, 0xc00087ad10, 0xc0007be8a0)
        github.com/hashicorp/[email protected]/cli.go:125 +0x594
github.com/hashicorp/terraform-provider-vault/internal/provider.(*AuthLoginOIDC).Login(0xc00087ad10?, 0xc00087ad10)
        github.com/hashicorp/terraform-provider-vault/internal/provider/auth_oidc.go:116 +0x56
github.com/hashicorp/terraform-provider-vault/internal/provider.(*ProviderMeta).setClient(0xc000514900)
        github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:293 +0xe85
github.com/hashicorp/terraform-provider-vault/internal/provider.(*ProviderMeta).getClient(...)
        github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:404
github.com/hashicorp/terraform-provider-vault/internal/provider.(*ProviderMeta).setVaultVersion(0xc000514900)
        github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:385 +0x108
github.com/hashicorp/terraform-provider-vault/internal/provider.(*ProviderMeta).GetVaultVersion(0xc000514900)
        github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:155 +0x74
github.com/hashicorp/terraform-provider-vault/internal/provider.(*ProviderMeta).IsAPISupported(0xc00088add0?, 0xc0000d3db0)
        github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:129 +0x18
github.com/hashicorp/terraform-provider-vault/internal/provider.IsAPISupported({0x14a7c40?, 0xc000514900?}, 0xc00084ade0?)
        github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:517 +0x45
github.com/hashicorp/terraform-provider-vault/vault.awsSecretBackendStaticRoleResource.MountCreateContextWrapper.func1({0x1b2c910, 0xc000430f50}, 0xc000485a80, {0x14a7c40, 0xc000514900})
        github.com/hashicorp/terraform-provider-vault/internal/provider/provider.go:269 +0xa5
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0xc0000bd0a0, {0x1b2c868, 0xc000639500}, 0xc000485a80, {0x14a7c40, 0xc000514900})
        github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:778 +0x119
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc0000bd0a0, {0x1b2c868, 0xc000639500}, 0xc00072ec30, 0xc000485900, {0x14a7c40, 0xc000514900})
        github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:909 +0xa89
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc0007fac00, {0x1b2c868?, 0xc000639440?}, 0xc0006421e0)
        github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:1074 +0xd5c
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc0006b0e60, {0x1b2c868?, 0xc000638a80?}, 0xc0004304d0)
        github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:859 +0x56f
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x14f21a0, 0xc0006b0e60}, {0x1b2c868, 0xc000638a80}, 0xc000484f00, 0x0)
        github.com/hashicorp/[email protected]/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:503 +0x1a6
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00041ec00, {0x1b2c868, 0xc0006389f0}, {0x1b33cc0, 0xc000948000}, 0xc00063ad80, 0xc0008082a0, 0x253e158, 0x0)
        google.golang.org/[email protected]/server.go:1385 +0xdd1
google.golang.org/grpc.(*Server).handleStream(0xc00041ec00, {0x1b33cc0, 0xc000948000}, 0xc00063ad80)
        google.golang.org/[email protected]/server.go:1796 +0xfb8
google.golang.org/grpc.(*Server).serveStreams.func2.1()
        google.golang.org/[email protected]/server.go:1029 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 21
        google.golang.org/[email protected]/server.go:1040 +0x125

Error: The terraform-provider-vault_v4.4.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Terraform Configuration Files

backend.tf

provider "vault" {
  address   = "https://vault_url"
  namespace = "my_namespace"
  auth_login_oidc {
    role  = "user"
    mount = "azure_ad"
  }
}

main.tf

resource "vault_aws_secret_backend_static_role" "role" {
  backend         = "aws"
  name            = "test"
  username        = "test-iam-user"
  rotation_period = "360"
}

Steps to Reproduce

Run terraform apply on the above resource definition.

Debug Output

No response

Panic Output

No response

Important Factoids

Here is the relevant auth role configuration in Vault:

vault.tf

resource "vault_jwt_auth_backend_role" "user" {
  backend        = "azure_ad"
  role_name      = "user"
  token_policies = ["user"]

  user_claim   = "email"
  groups_claim = "roles"
  role_type    = "oidc"
  oidc_scopes  = ["https://graph.microsoft.com/.default", "profile", "email"]
  allowed_redirect_uris = ["http://localhost:8250/oidc/callback",
    "https://vault_url/ui/vault/auth/azure_ad/oidc/callback",
    "http://localhost:8250/oidc/callback?namespace=my_namespace",
    "https://vault_url/ui/vault/auth/azure_ad/oidc/callback?namespace=my_namespace"
    ]
}

resource "vault_policy" "user" {
  name   = "user"
  policy = file("policies/user-policy.hcl")
}

resource "vault_identity_group" "user" {
  name     = "user"
  type     = "external"
  policies = ["user"]
}

resource "vault_identity_group_alias" "user" {
  name           = "user"
  mount_accessor = "accessor-id"
  canonical_id   = vault_identity_group.user.id
}

user-policy.hcl

path "aws/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

Creating the same resource using the Vault CLI works ok:

vault login -namespace=my_namespace -method=oidc -path=azure_ad role="user"

Complete the login via your OIDC provider. Launching browser to:

    https://login.microsoftonline.com/...

Waiting for OIDC authentication to complete...
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                ...
token_accessor       ...
token_duration       168h
token_renewable      true
token_policies       ["default" "user"]
identity_policies    []
policies             ["default" "user"]
token_meta_role      user

vault write aws/static-roles/test username=test-iam-user rotation_period=360

Key                Value
---                -----
id                 <id>
name               test
rotation_period    6m
username           test-iam-user

References

No response

Would you like to implement a fix?

None

[fairclothjm]

Duplicate of #2131

[PavGregor]

Some more details to hopefully help with identifying the root cause, compared to #2131 I'm not triggering the issue here with a login timeout due to inactivity or due to an expired token.

When I run terraform apply on the above snippet I get redirected to a browser for SSO login, that immediately succeeds and returns back to Terraform, which then proceeds to crash within a second or two.

One strange workaround that I stumbled upon just now, if I change the access policy to the following:

path "*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

then the terraform run succeeds. Changing it back to:

path "aws/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

and I get the error. I've tried this a few times and it seems to be consistent behaviour. Is there any reason why this policy change would affect the oidc login process?