Multi-cluster setup with multiple ArgoCD installations

[gnadaban]

How is Kargo supposed to be deployed or configured in multi-cluster scenarios where for geographical, data stewardship, hybrid- or multi-cloud infrastructure, resilience/blast-radius or compliance reasons there are multiple ArgoCD instances, one per Kubernetes cluster?

In such a setup, how can one configure a single Kargo instance such that multiple ArgoCD instances across clusters are checked for the respective Stages?

[krancour]

@gnadaban we built with scenarios like this in mind and have internal reasons (which we'll revisit at some point, probably soon) for why we haven't documented it extensively.

The gist of it is this...

1. You would run most Kargo components in a centralized "Kargo control plane." This would be the management controller, API server, webhooks server, and garbage collector. The k8s cluster that these components talk to (which is presumably also where they are running) would be where all Kargo resources live.

2. Your Kargo controllers would be distributed around to various clusters. They would exist 1:1 with your Argo CD instances. So, each Kargo controller talks to the Kargo control plane and one specific Argo CD instance. Each such Kargo controller needs to be assigned a unique shard name.

3. Stages (and optionally, Warehouses) need to be assigned to a particular shard, and it is the Kargo controller in that shard that will reconcile those.

What you end up with is all the Kargo data in one place, with distributed controllers "phoning home", reconciling a shard of the data and interacting with a specific Argo CD instance.

If you hale a close look at the chart's README or values.yaml file, you will get a good sense of what to enable/disable or otherwise configure for the Kargo control plane and each distributed controller.

[krancour]

I'm guessing that kubeconfigSecrets.kargo would be a shared kubeconfig secret that all sharded controllers use to talk to the control plane, can you confirm this?

Well, not quite, because the sharded controllers are all, presumably, living in different clusters, they're not going to be sharing a Secret. They would all have their own Secret. If they all happen to have the same values -- i.e. all using the same ServiceAaccount in the Kargo control plane's cluster -- that's up to you. So "shared," in that sense, is possible.

Supposing we have multiple controllers correctly deployed, the next question would be around tags and their discovery in a monorepo setup.

If Stages represent different clusters that are tracked with different tag prefixes, how would one have to define eg. Warehouse Git subscriptions? Can you perhaps provide an example?

You're going to have to be more specific, I'm afraid.

[gnadaban]

Thanks for the info about service accounts, that's what I meant, ie. that each controller has a kubeconfig to connect to the cluster (as a configured serviceaccount) the control plane is deployed. I'm guessing that these controllers will in turn maintain status of resources of the stages they correspond to, and that's all.

As for the tags, if configuration of multiple apps/clusters are tracked in a single Git monorepo what would Warehouse definitions look like? A single Warehouse with a Git subscription per cluster, eg. based on a tag prefix like this?

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: gitops
spec:
  subscriptions:
    - git:
        repoURL: https://my.repository.url.git
        commitSelectionStrategy: Lexical
        allowTags: shard1-prefix|shard2-prefix

        includePaths:
          - path/to/shard1/config
          - path/to/shard2/config          

Or multiple WH objects each strictly pertaining within the same repository to some shard?

[krancour]

So, a piece of Freight (which is generated by a Warehouse) is the (meta)artifact that moves through a pipeline. The contents of a piece of Freight wouldn't/shouldn't be Stage specific.

With respect to git repos, a Warehouse discovers a new commit, makes a piece of Freight with that commit in it, and at each Stage your promotion mechanisms will say what to do with it.

[gnadaban]

Thank you for your reply. It does make sense, though I still cannot put my finger on what our config should look like, but perhaps with time it will come.

[krancour]

@gnadaban I think for illustrative purposes, the default NewestFromBranch commit selection strategy might make this easier to conceptualize.

A typical Warehouse might look something like this:

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: example
  namespace: example
spec:
  subscriptions:
  - git:
      repoURL: https://github.com/example/repo.git
      commitSelectionStrategy: NewestFromBranch # This is actually the default; explicit here for clarity
      branch: main # This is actually the default; explicit here for clarity
  - image:
      repoURL: example/image
      imageSelectionStrategy: SemVer # This is actually the default; explicit here for clarity
      semverConstraint: ^1.0.0

This Warehouse has subscriptions to two repos where it will find artifacts -- one type of artifact being an image, and another being manifests from a Git repo. (Or more likely something like a Helm chart or Kustomize "base" and "overlay" directories from a Git repo, either of which can be rendered into plain YAML manifests by Argo CD.)

So conceptualize this Warehouse as continuously creating a stream of Freight, where each piece of Freight is a meta-artifact referencing specific versions of its two constituent artifacts. New image is found -- new Freight. New manifests found -- new Freight.

Your Stages will describe how what to do with any given piece of Freight from that stream in order to integrate its constituent artifacts into an environment.

So, a "dev" Stage may look like this:

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: dev
  namespace: example
spec:
  subscriptions:
    warehouse: example
  promotionMechanisms:
    gitRepoUpdates:
    - repoURL: https://github.com/example/repo.git
      writeBranch: stage/dev # Store Stage-specific stuff in this branch
      kustomize:
        images:
        - image: example/image
          path: stages/dev # Run kustomize edit set image on this path
    argoCDAppUpdates:
    - appName: example-dev

When a piece of Freight is (either manually or automatically) promoted to this Stage, Kargo will:

1. Check out the commit from https://github.com/example/repo.git that is indicated by the Freight
2. Run kustomize edit set image example/image:<tag> in the indicated path (stages/dev), where <tag> is the specific tag of the image that is indicated by the Freight
3. Write the results of that operation out to the stage/dev branch, which you can think of as storage
4. Forces a sync/refresh of the Argo CD Application example-dev, which is presumably syncing to the stage/dev branch mentioned in step 3

The net result is that any time you have a new manifests or a new image, Kargo "mashes them up" and leaves the results somewhere where the example-dev Argo CD Application will find them.

Details for the next Stage in your pipeline might vary just slightly. "UAT" might look like this:

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: dev
  namespace: example
spec:
  subscriptions:
    upstreamStages:
    - name: dev # Subscribe to Freight that has been verified "upstream" in dev
  promotionMechanisms:
    gitRepoUpdates:
    - repoURL: https://github.com/example/repo.git
      writeBranch: stage/uat # Store Stage-specific stuff in this branch
      kustomize:
        images:
        - image: example/image
          path: stages/uat # Run kustomize edit set image on this path
    argoCDAppUpdates:
    - appName: example-uat

Note that I have illustrated a very common scenario, but simpler and more complex scenarios are possible as well.

The main thing that changes for your scenario is that your dev and uat Stages may each be in different shards (spec.shard). Which shard they are in depends on which cluster can see the respective Argo CD Application that needs to be synced/refreshed/monitored -- which takes us back to the original question about topology and multiple Argo CD installations.