API Token is not updated in pulumi state

[mdecalf]

What happened?

In our golang controller we have a cloudflare provider instanced like that

	providers[cloudflareProviderName], err = cloudflare.NewProvider(
		ctx,
		cloudflareProviderName,
		&cloudflare.ProviderArgs{
			ApiToken: pulumi.StringPtr(cloudflareAPIToken),
		},
	)

We use it to create/update dns record

Last week we rotate our Cloudflare API token and restarted our app with the new token

All of our stacks are in failed status with an authentication error

{...pulumi:pulumi:Stack package-stack **failed** 1 error\n\nDiagnostics:\n cloudflare:index:Record (target-web-custom-domain):\n error: refreshing urn:pulumi:stack::package::cloudflare:index/record:Record::target-web-custom-domain: 1 error occurred:\n \t* Authentication error (10000)\n\n pulumi:pulumi:Stack (package-stack):\n error: update failed\n\nResources:\n 9 unchanged\n\nDuration: 3s`...}

So after many hours we found out what was wrong, the pulumi stack.json contains the token and unfortunately the old one

  "resources": [
      ...
      {
          "urn": "urn:pulumi:stack::package::pulumi:providers:cloudflare::cloudflare",
          "custom": true,
          "id": "f0bca2d3-65de-4c2a-8ce2-68c81b903230",
          "type": "pulumi:providers:cloudflare",
          "inputs": {
              "apiClientLogging": false,
              "apiToken": "XXXXXX",
              "maxBackoff": 30,
              "minBackoff": 1,
              "retries": 3,
              "rps": 4
          },
          "outputs": {
              "apiClientLogging": false,
              "apiToken": "XXXXXX",
              "maxBackoff": 30,
              "minBackoff": 1,
              "retries": 3,
              "rps": 4
          },

After pushing the new token into the object manually, it works

After looking at the successive changelogs & issues, I didn't see any mention about that. So, two questions:

Why is the API token, which is not a resource, is in the stack?
How can existing stacks use the new token?

Example

{...pulumi:pulumi:Stack package-stack **failed** 1 error\n\nDiagnostics:\n cloudflare:index:Record (target-web-custom-domain):\n error: refreshing urn:pulumi:stack::package::cloudflare:index/record:Record::target-web-custom-domain: 1 error occurred:\n \t* Authentication error (10000)\n\n pulumi:pulumi:Stack (package-stack):\n error: update failed\n\nResources:\n 9 unchanged\n\nDuration: 3s`...}

Output of pulumi about

CLI
Version 3.70.1
Go Version go1.20.2
Go Compiler gc

Host
OS ubuntu
Version 20.04
Arch x86_64

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[mikhailshilkov]

Hi @mdecalf I'm sorry about your troubles, let's try to figure it out together.

Why is the API token, which is not a resource, is in the stack?

API token is not a resource, but cloudflare.Provider is. Providers are kind of "virtual" resources in the sense that they don't create anything external, but are tracked in state files in order for the CLI to be able to display diffs, track dependencies and so on.

Last week we rotate our Cloudflare API token and restarted our app with the new token

Could you please elaborate on that? Did you change the value in your program, then run pulumi up to apply it? That should have updated the state and all depending resources.

How can existing stacks use the new token?

You need to run pulumi up with the new token and apply all the changes.

[rpmccarter]

Hi @mikhailshilkov 👋 I'm running into this issue as well

Our existing Cloudflare API token expired, so I updated the values in my Pulumi.<stack-name>.yaml files. Running pulumi up did not resolve the issue - I continued to hit this error: 

Diagnostics:
  cloudflare:index:Record (server-record-dev):
    error: deleting urn:pulumi:dev::server::cloudflare:index/record:Record::server-record-dev: 1 error occurred:
    	* error deleting Cloudflare Record: Authentication error (10000)

Not sure I ever would have figured out this bug without finding this issue. I assumed that the providers would always use the api token from the config.

You need to run pulumi up with the new token and apply all the changes.

If this is the case, it wasn't working for me. Does the new api token only get applied at the end of pulumi up? If so, I seemed to be in a frozen state where in order to update the token, I needed to get to the end of pulumi up, but in order to get to the end of pulumi up, I needed to update the token

The solution for me was to manually delete the Cloudflare provider from state - not ideal

Any help appreciated!

[liam-auror]

Apologies for bumping an old issue, but I hit the exact same problem. The workaround was to run pulumi update with --target set to the provider's URN.

I feel this behavior is a bug: Pulumi tries to refresh/update the stack resources using the apiToken in the old stack state, which fails if that token is no longer valid. The authentication-related stack settings and/or the provider resource should be updated first before attempting any actual resource operations. Better handling of authentication-related errors & messaging would also be great to have, e.g. if reading one resource of a certain type fails, it's almost certain that any others will as well (at least in the CF context), so aborting early would make sense.

The current behavior is very frustrating to troubleshoot as there's no real indication from Pulumi about what the problem is, leading users down a rabbit hole trying to diagnose an apparent auth/API issue. It also easily results in Cloudflare rate-limiting your API requests which leads to further confusion. Changing stack settings usually has immediate effect, so the user expects any changes to the apiToken secret to at least work locally.

[VenelinMartinov]

Hey folks, sorry you've hit issues here. Are you running refresh=true for pulumi up when you get the stale credentials?

This looks like pulumi/pulumi#4981 - the issue is that the credentials are saved in the state and refresh, unfortunately, does not run the program, so can't get the new credentials - it will always use the stale ones.

The suggested workaround is not to save credentials in state - instead it is preferable to use environment variable to configure credentials for the provider.

If that is not an option the running pulumi up without --refresh=true should also update the credentials stored in the state. You could also run pulumi up and use --target to only update the provider, so that the next pulumi up --refresh=true will work.

If neither of these works could I ask for a short program which reproduces the issue, as well as the sequence of commands which triggers it?

[liam-auror]

@VenelinMartinov I've been using --refresh=false throughout for reference.

Running pulumi up targeting the provider doesn't work either, for a variety of reasons (I have multiple affected Pulumi projects, each with different states):

• Update fails due to other resources having dependencies on that provider (Error: Resource 'urn:pulumi:<REDACTED>::<REDACTED>::cloudflare:index/record:Record::<REDACTED>' will be destroyed but was not specified in --target list. Either include resource in --target list or pass --target-dependents to proceed.). I believe this case was due to the token issue occurring while running an update.
• Update fails due to 'provider not initialized, use previous provider version' (Something like that, apologies for not capturing the exact error). I believe this one was due to the token issue occurring while updating the Pulumi packages.

I believe the trigger for the issue in these projects was hitting CF API rate limits due to running too many refresh operations in parallel, then attempting to replace the token (via rolling the existing one in the state). Could also occur due to token expiry as others have reported.

[liam-auror]

For reference, I've used the following workarounds to fix the affected projects:

WARNING: Exercise extreme care when attempting these workarounds! Ensure you backup your stack state & the CF records etc. before attempting. These workarounds may result in unexpected behavior, use at your own risk.

• Add a dummy resource to the stack definition & run pulumi update --refresh=false --target
• Manually delete non-existent resources from the state (i.e. using pulumi state delete <urn>)
• (Untested) Export the stack state, manually edit & reimport.
• (Untested) Recreate stack from scratch via importing resources.

[VenelinMartinov]

Thanks for expanding on the workaround @liam-auror!