Remote cluster with Workload Identity configured with IAM Policy to the KSA, not working

[froblesmartin]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I'm not sure if this is a bug or a feature request.

I have configured ArgoCD (running in GKE) with an external GKE cluster located in a different GCP Project. Following the official documentation to use Workload Identity with ArgoCD, creating the GCP IAM Service Account and adding the annotation to the KSA, it does work.

I tried using the new approach from Google, which instead of requiring a GCP IAM Service Account and the annotation in the KSA, you can just assign GCP IAM Roles directly to the KSA, referencing it from GCP IAM policies like:

principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/KSA_NAME

But this does not work. When trying, I just get the error from the argocd-application-controller:

error synchronizing cache state : the server has asked for the client to provide credentials

Maybe the argocd-k8s-auth just needs to use a newer version of the GCP SDK, it requires a different configuration, or it is something harder.

To Reproduce

Enable GKE Workload Identity, and from a different GCP Project, assign the role for ArgoCD to manage a GKE cluster in that GCP Project:

gcloud projects add-iam-policy-binding projects/PROJECT_ID \
    --role=roles/container.admin \
    --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/argocd/sa/argocd-server \
    --condition=None

gcloud projects add-iam-policy-binding projects/PROJECT_ID \
    --role=roles/container.admin \
    --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/argocd/sa/argocd-application-controller \
    --condition=None

Then configure your remote GKE cluster with the following K8s manifest in the GKE cluster where ArgoCD is deployed:

apiVersion: v1
kind: Secret
metadata:
  name: argocd-cluster-remote1
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: remote1
  server: https://<your-remote-cluster-endpoint>
  config: |
    {
      "execProviderConfig": {
        "command": "argocd-k8s-auth",
        "args": ["gcp"],
        "apiVersion": "client.authentication.k8s.io/v1beta1"
      },
      "tlsClientConfig": {
        "insecure": false,
        "caData": "LS0tLS1...."
      }
    }

Expected behavior

I would expect ArgoCD to authenticate correctly.

Version

v2.12.4+27d1e64