hcp_vault_cluster_admin_token issues with app.terraform.io remote backend

[wilkosz]

Terraform Version and Provider Version

Terraform version: 14 or 15
HCP provider version: 0.6.0
Terraform Cloud

Affected Resource(s)

• hcp_vault_cluster_admin_token

Terraform Configuration Files

Any build using Terraform remote backend. (HVN and Vault are newly created with no configuration other than being made public)

data "hcp_vault_cluster" "main" {
  cluster_id = var.hvn.vault_id
}
resource "hcp_vault_cluster_admin_token" "main" {
  cluster_id = var.hvn.vault_id
}
provider "vault" {
  address   = "https://${data.hcp_vault_cluster.main.vault_public_endpoint_url}:8200"
  token     = hcp_vault_cluster_admin_token.main.token
  namespace = "admin"
}
resource "vault_mount" "db" {
  path = "database"
  type = "database"
}

 backend "remote" {
    hostname     = "app.terraform.io"
    organization = "dochub"

    workspaces {
      name = "dochub-testing"
    }
  }

Debug Output

Error: no vault token found

  on ../modules/hvn/providers.tf line 14, in provider "vault":
  14: provider "vault" {

1. terraform apply

Expected Behavior

Any behaviour resulting in execution plan creation

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

Important Factoids

This is only an issue with backend remote. Locally, and s3 backend both work as desired when HCP_CLIENT_ID and HCP_CLIENT_SECRET are present.
Also note that hcp_vault_cluster data resource returns the vault_public_endpoint_url.
Manually adding the environment variable VAULT_TOKEN on Terraform cloud does fix the issue, but doesn't scale.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[bcmdarroch]

Hi @wilkosz! Thanks for adding this issue. I'll do some investigation and keep you posted.

[wilkosz]

@bcmdarroch I done some discovery on this and it is not terraform version dependant it when using remote app.terraform.io

 backend "remote" {
    hostname     = "app.terraform.io"
    organization = "dochub"

    workspaces {
      name = "dochub-testing"
    }
  }

[bcmdarroch]

I think this might be an issue of the admin token timing out when using remote. The admin token is really only intended for bootstrapping and should be replaced by a more secure long-term auth solution in cases like this. We'll be working on updating our best practices in the admin token guide which will provide more in-depth guidance.

[wilkosz]

@bcmdarroch not sure if it is a timeout issue. If the token is set as an environment variable for the first run, then removed, it behaves as intended.

[Bluesboy]

@wilkosz Thanks! That worked for me.

[iniinikoski]

@wilkosz did you find out anything more about this...? In addition to this, I've bumped into a fleet of issues when working on both HCP provider and Vault provider the same time... As provider's don't understand dependencies for example, the Vault provider would be tried to be initialised with a token which does not exist yet...

Expecting Hashicorp to really build a proper suggestion on how to do the bootstrapping properly and secure way - also with Terraform Cloud.

[bcmdarroch]

Updating with the following guidance from our official Vault docs. It is recommended that the admin token only be used during initial setup of the Vault cluster. We do not have TFC-specific guidance at this time, so I will leave this issue open.

A good practice is to limit the use of the admin token and configure your preferred auth method(s) for regular use. The auth methods you choose to enable will be based on the who (person) or what (machine) will access your HCP Vault cluster. It is not uncommon to have multiple auth methods enabled.
source: https://developer.hashicorp.com/vault/tutorials/cloud-ops/hcp-vault-namespace-considerations#hcp-vault-admin-namespace

The Manage Authentication Methods guide walks through how to set up proper authentication once your cluster has been bootstrapped with the admin token.