Use kargo with ArgoCD one instance per cluster

[emirot]

Hello there,

I would like to use Kargo but not sure if it is possible with my current ArgoCD deployment.
I have one Argocd per instance as described here
Does Kargo have some concept of Agent that could report back to a control plane or something ?

[krancour]

Does Kargo have some concept of Agent that could report back to a control plane or something ?

That is essentially how Kargo's topology works. One centralized control plane consisting of the api server, management controller (a distinct component from the main controller), and garbage collector. The main controller that does all the heavy lifting is the "agent" you're asking about and is intended to be distributed around your clusters, usually 1:1 with your Argo CD control planes. Each one talks back to the Kargo control plane as well as the "nearby" Argo CD control plane.

None of this is well-documented at the moment, but we are lately receiving more and more questions on this very topic, so I will rectify that documentation gap as soon as possible.

[imtiazc]

@krancour - we would like to know about how the control plane instance interacts with the "agents" running on different clusters. Specifically, what port or protocols are used.

[krancour]

@imtiazc it's pure k8s. The controllers sync with Kargo resources in the control plane's cluster. They sync with Argo CD resources in the "nearby" Argo CD control plane's cluster. i.e. It is a "phone home" model. The control plane has no permissions on your multiple clusters and does not become a high value target.

[Brightside56]

@krancour control plane has secrets for SCM what may lead to serious consequences. Controllers also have access to secrets what means controller of any shard may get an access to SCM secrets for all shards. This can be effectively mitigated only by not allowing Kargo controller to make commits directly (and allowing to make PRs instead), but it will not make people happy in many cases

kargo-api has a possibility to create freights and promotions in control plane cluster, what could allow intruder to deploy anything to anywhere in case, for example, when there is RCE vulnerability in Kargo UI/API

Yesterday I've investigated this one - https://github.com/akuity/kargo/blob/main/charts/kargo/templates/controller/cluster-roles.yaml#L33. From what I understand, permission to create promotions and freights can provide a possibility to "call home" and say "hey mom, tell this guy from another shard to deploy my backdoor to help me with cluster takeover" (by crafting malicious commit with manifests and creating freight and promotion in control plane cluster for other shard). In my case this can be mitigated by not allowing Kargo controller to create promotions (because from what I see in Kubernetes audit logs this permission is not used by controller when promotions are created using UI)

The control plane has no permissions on your multiple clusters and does not become a high value target.

With current RBAC and possibility to access an SCM not only control plane, but any shard controller may easily become high value target.

[krancour]

control plane has secrets for SCM what may lead to serious consequences

It is not the least bit uncommon that control planes manage Secrets or that agents get their Secrets from control planes. If that's a problem, I have some very bad news for you about Kubernetes itself.

I will note that there is, at least, an open issue about opting out of the control plane managing Secrets, as some users may not require that capability.

kargo-api has a possibility to create freights and promotions in control plane cluster, what could allow intruder to deploy anything to anywhere in case

The API would be pretty useless to authorized users if it lacked these capabilities. Can capabilities that the API implements for legitimate use by authorized users be abused if an intruder was able to use the API via stolen credentials? Of course. But show me a system where that isn't true.

Is it easy to "deploy anything?" Not at as easy as you may think. Deploying "anything" would require access to SCM. I know you presume that access to SCM is easily obtained via the API, but it is not. All get/list credential requests to the API serve heavily redacted Secrets. (Getting even redacted Secrets would not be possible after dealing with the other issue mentioned above.) And, in many cases, Argo CD credentials would also be required to pull this off.

So it seems then that the concern here must be less about unauthorized API use and more about obtaining and abusing the API server's own credentials. Show me a system where that doesn't have consequences, and if you do find yourself in that scenario, be glad that the control plane doesn't have credentials for other clusters.

From what I understand, permission to create promotions and freights can provide a possibility to "call home" and say "hey mom, tell this guy from another shard to deploy my backdoor to help me with cluster takeover"

Once again, these permissions have a legitimate purpose. But I will also point out that controllers are not internet-facing components and do not receive any inbound traffic from users, making them inherently less attackable than the API server. There's simply no way to coerce a controller into some arbitrary "hey mom, tell this guy..." So your concern, once again, must be more about obtaining a controller's credentials and using them differently than the controller would have. Again, show me a system where this type of compromise doesn't have serious consequences.

With current RBAC and possibility to access an SCM not only control plane, but any shard controller may easily become high value target.

Not as high value as a control plane with credentials for all the clusters it controls.

Generally speaking, your what ifs seem to have little to do with Kargo being attacked directly and revolve more around "what if the API server's or a controller's credentials are obtained and abused?" That is a concern for anything you run in Kubernetes (or anywhere else for that matter). The best that Kargo can do in the face of that is to keep its own permissions to the absolute minimum possible without impairing its ability to do the things it was designed to do, and I believe, in past conversations, we have shown a willingness to analyze and reduce permissions.

I do notice, @Brightside56 that, as with the other issue referenced above, you are raising these security concerns in response to a mention of our "phone home" architecture, which you seem to object to, with a preference for something more hub-and-spoke-ish. So while I will re-iterate our willingness to analyze and reduce permissions where possible, I will also re-iterate that radical changes to our architecture are simply not on the table.

[Brightside56]

The API would be pretty useless to authorized users if it lacked these capabilities. Can capabilities that the API implements for legitimate use by authorized users be abused if an intruder was able to use the API via stolen credentials? Of course. But show me a system where that isn't true.

Is it easy to "deploy anything?" Not at as easy as you may think. Deploying "anything" would require access to SCM. I know you presume that access to SCM is easily obtained via the API, but it is not. All get/list credential requests to the API serve heavily redacted Secrets. (Getting even redacted Secrets would not be possible after dealing with the other issue mentioned above.) And, in many cases, Argo CD credentials would also be required to pull this off.

So it seems then that the concern here must be less about unauthorized API use and more about obtaining and abusing the API server's own credentials. Show me a system where that doesn't have consequences, and if you do find yourself in that scenario, be glad that the control plane doesn't have credentials for other clusters.

Yes, this concern is mainly about possibility to compromise underlying service account or exploit RCE what can allow to manipulate with this account... Agreed that every system which is designed for centralized management of something is vulnerable at this point and it's hard to mitigate. Concentration of permissions for control plane seems to be fully legit

Once again, these permissions have a legitimate purpose. But I will also point out that controllers are not internet-facing components and do not receive any inbound traffic from users, making them inherently less attackable than the API server. There's simply no way to coerce a controller into some arbitrary "hey mom, tell this guy..." So your concern, once again, must be more about obtaining a controller's credentials and using them differently than the controller would have. Again, show me a system where this type of compromise doesn't have serious consequences.

In case of data plane I'm not agreed. I wouldn't say that permissions to touch something what may affect other shards are legitimate. In my understanding kargo-controller is data plane unit same as kubelet for Kubernetes, CI runner for distributed CI system like Jenkins/Gitlab/Github/..., ... . All of those have underlying infrastructure which can have different access policies, may be in responsibility zone of different users, teams, departments and run arbitrary workloads and may be compromised. Therefore usually data plane units are also reliably isolated by default and I believe Kargo shouldn't be an exception. CI systems have a protocols which ensures reliable isolation of data plane units, Kubernetes have node authorization, and so on, because the security of the entire business should not be decreased to the level of security of a single shard, node, runner or single anything else where intruder appears

I do notice, @Brightside56 that, as with the other issue referenced above, you are raising these security concerns in response to a mention of our "phone home" architecture, which you seem to object to, with a preference for something more hub-and-spoke-ish. So while I will re-iterate our willingness to analyze and reduce permissions where possible, I will also re-iterate that radical changes to our architecture are simply not on the table.

No, "phone home" seems to be a more interesting and promising than hub-spoke from security point of view, but not by default. Every time you write about it, I think about suffering security engineers and complex threat model that needs to be described and studied, granular and tricky permissions in many repositories need to be set, not forgotten and reviewed on regular basis... About secure defaults which may put everything at risk in case of absence. I would prefer anything what could provide clear guarantees that intruder cannot screw up whole infrastructure by exploiting single shard or its underlying infrastructure

[krancour]

I wouldn't say that permissions to touch something what may affect other shards are legitimate.

The main point of Kargo is to automate the delivery of artifacts from Stage to Stage. Since the Stages of a single Project commonly span at least two shards (e.g. at minimum, some Stages in a "non-prod" cluster and at least one in a "prod" cluster), there is a legitimate reason for a controller operating in one shard do things that "may affect other shards" -- namely there is a legitimate reason for a healthy Stage with verified Freight, reconciled by one shard's controller, to create a Promotion that will be handled by a different shard's controller.

If you want to propose incremental improvements that don't completely kneecap Kargo, we would love to hear them, but we cannot entertain restrictions that prevent Kargo from fulfilling its basic purpose.

No, "phone home" seems to be a more interesting and promising than hub-spoke from security point of view

Thank you for saying so. 😄

but not by default. Every time you write about it, I think about suffering security engineers and complex threat model that needs to be described and studied, granular and tricky permissions in many repositories need to be set, not forgotten and reviewed on regular basis...

I'm getting somewhat tired of rehashing this...

If it wasn't obvious, Kargo learned some important lessons from Argo CD, where hub-and-spoke with the control plane having broad permissions for each cluster it manages has been viewed as a weakness that leads many operators away from that topology. Security-conscious operators tend to settle for a siloed topology with a separate Argo CD control plane per cluster, and in doing so, they sacrifice the convenience of a "single pane of glass."

I try very hard not to use the Kargo OSS project as a forum for discussing Akuity's commercial offerings, but this once, I'll point out that one major innovation of Akuity's platform is its hybrid architecture that enables one Argo CD control plane to manage multiple clusters without the aforementioned weakness. It isn't a secret that this is achieved with a "phone home" model and Kargo sought nearly from the beginning to bake that innovation directly into our OSS instead of taking Argo CDs approach and then needing to "fix" it later.

If the alternative to the architecture we are using is a control plane that has credentials for multiple Argo CD control planes, wherein those Argo CD control planes have, in-turn, extensive permissions on the one or more clusters they manage, it should be obvious that compromise of that hypothetical, highly-privileged Kargo control plane is far, far worse than compromise of either the control plane we do have or any one of the sharded controllers.

The bottom line is that you're not going to convince us that hub-and-spoke is somehow better, or as you now seem to be suggesting, better as a default. You are completely justified in finding our topology more difficult to reason about and to manage than hub-and-spoke. I simply cannot disagree with you on those points, but we're not going to trade the relative security of what we have now for something we know to be weaker just because its easier to wrap one's head around.

Lest I be perceived as unjustifiably argumentative, I will re-iterate, as I always do, that we welcome incremental security improvements. It is simply the debate over the architecture that needs to be put to rest. Besides our confidence in the choice we've made, a pivot to hub-and-spoke at this stage would derail the project to an extent that no one is willing to entertain. Thank you for understanding that, and we look forward to your insight on incremental security improvements instead.

[Brightside56]

If you want to propose incremental improvements that don't completely kneecap Kargo, we would love to hear them, but we cannot entertain restrictions that prevent Kargo from fulfilling its basic purpose.

I think my main concern is supposed to be addressed with #2448, which should provide a possibility to limit shard access with specific control plane namespace and SCM secrets for particular shard. Thank you, at the moment it's more than enough