[ANSIENG-3807] | Rbac over mTLS #1804

rrbadiani · 2024-10-18T13:47:28Z

Description

This Pr aims to add support for RBAC over mTLS.
The includes changes for

MDS
Broker
Controller
SR
ERP
RP
Connect
KSQL
Replicator
C3
Using cert based token retrival
Extracting Principal from certs to assign role bindings
Molecule tests
Setting Principal Mapping Rules
Setting impersonation super users for principal propogation

Fixes # (issue)

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

zookeeper

kraft

Checklist:

Any variable/code changes have been validated to be backwards compatible (doesn't break upgrade)
I have added tests that prove my fix is effective or that my feature works
If required, I have ensured the changes can be discovered by cp-ansible discovery codebase
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
Any dependent changes have been merged and published in downstream modules

* [ANSIENG-4229] | adding new user facing variables for rbac over mtls * [try-mtls] | modified default listener and get auth token * [ANSIENG-4229] | adding 2 small scenarios for testing * [try-mtls] | [ANSIENG-4229] | modifying the default values of ssl_client_authentication and ssl_mutual_auth * [try-mtls] | fixing issues in set principal for mtls and health checks * [try-mtls] | fixing mds client auth properties * [try-mtls] | adding ldap+mtls sceanrio * [try-mtls] | changing os to rhel9 and java to 17 for mtls tests * [try-mtls] | fix listener auth issue * [try-mtls] | fix health checks for broker * [try-mtls] | adding impersonation super users in bk customer properties * [try-mtls] | unmasking secrets for better logging * [try-mtls] | fix mds health check and config validations to include auth mode mtls * [try-mtls] | fix get authorization tokens * [try-mtls] | temporary removal of ssl endpoint identification * [try-mtls] | fix verify of new molecule scenarios for mtls * [try-mtls] | chaning erp listener to oauthbearer listener for impersonation token * [try-mtls] | fixing qoutes in erp listener name * [try-mtls] | removing ssl client auth verification from controller as it is only mds server property * [try-mtls] | adding impersonation super and protected users in mds properties * [try-mtls] | modify ssl client authentication for listeners based on inventory file * [try-mtls] | principal mapping rules on listeners * [try-mtls] | overriding principal mapping rules in molecule * [try-mtls] | fix impersonation protected user and remove config override for impersonation super user * [try-mtls] | remove principal mapping rules * [try-mtls] | fixing qoutes around client auth mode in listeners * [try-mtls] | adding sr in mtls only setup * [try-mtls] | fixing sr mtls * [try-mtls] | fix ldap detection and add extra $ in molecule for escape reasons * Fix SR RBAC (#70) * [ANSIENG-4229] | adding new user facing variables for rbac over mtls * [try-mtls] | modified default listener and get auth token * [ANSIENG-4229] | adding 2 small scenarios for testing * [try-mtls] | [ANSIENG-4229] | modifying the default values of ssl_client_authentication and ssl_mutual_auth * [try-mtls] | fixing issues in set principal for mtls and health checks * [try-mtls] | fixing mds client auth properties * [try-mtls] | adding ldap+mtls sceanrio * [try-mtls] | changing os to rhel9 and java to 17 for mtls tests * [try-mtls] | fix listener auth issue * [try-mtls] | fix health checks for broker * [try-mtls] | adding impersonation super users in bk customer properties * [try-mtls] | unmasking secrets for better logging * [try-mtls] | fix mds health check and config validations to include auth mode mtls * [try-mtls] | fix get authorization tokens * [try-mtls] | temporary removal of ssl endpoint identification * [try-mtls] | fix verify of new molecule scenarios for mtls * [try-mtls] | chaning erp listener to oauthbearer listener for impersonation token * [try-mtls] | fixing qoutes in erp listener name * [try-mtls] | removing ssl client auth verification from controller as it is only mds server property * [try-mtls] | adding impersonation super and protected users in mds properties * [try-mtls] | modify ssl client authentication for listeners based on inventory file * [try-mtls] | principal mapping rules on listeners * [try-mtls] | overriding principal mapping rules in molecule * [try-mtls] | fix impersonation protected user and remove config override for impersonation super user * [try-mtls] | remove principal mapping rules * [try-mtls] | fixing qoutes around client auth mode in listeners * [try-mtls] | adding sr in mtls only setup * [try-mtls] | fixing sr mtls * [try-mtls] | fix ldap detection and add extra $ in molecule for escape reasons --------- Co-authored-by: Mansi Jain <[email protected]> * [ANSIENG-4233] | added mtls configs for connect * [ANSIENG-4233] | comment fix * [try-mtls] | fix sr changes * [ANSIENG-4233] | config fix for listener authentication * [ANSIENG-4233] | add config for connectors * [ANSIENG-4233] | code fix * [ANSIENG-4233] | code revert * [try-mtls] | modify molecule scenario to add super user and principal mapping rules * [ANSIENG-4233] | add connector mtls config * [ANSIENG-4236] | add connect replicator mtls config * [ANSIENG-4233] | delegate token fetch to broker for connector * [ANSIENG-4236] | property fix * [ANSIENG-4235] | mtls configs * [ANSIENG-4235] | mtls configs * [ANSIENG-4236] | mtls configs * [ANSIENG-4236] | mtls configs * [ANSIENG-4235] | mtls configs * [ANSIENG-4235] | property fix * [ANSIENG-4236] | add test for replicator * [pm-rules] | handle default principal mapping rules * [pm-rules] | principal mapping rules in mds, erp, listeners, sr * [pm-rules] | removing config overrides from mtls onnly scenario as principal mapping rules are added by variables * [pm-rules] | principal mapping rules fix to get proper super user * [pm-rules] | adding listener level control over principal mapping rules * [pm-rules] | fix principal mapping rules in listeners * [mtls-rp] | add mtls support in erp,rp * [mtls-rp] | fix kafka rest license issue * [ANSIENG-4235] | add eol * Connect mtls ansieng 4233 (#72) * [ANSIENG-4233] | added mtls configs for connect * [ANSIENG-4233] | comment fix * [ANSIENG-4233] | config fix for listener authentication * [ANSIENG-4233] | add config for connectors * [ANSIENG-4233] | code fix * [ANSIENG-4233] | code revert * [ANSIENG-4233] | add connector mtls config * [ANSIENG-4233] | delegate token fetch to broker for connector * Revert "Connect mtls ansieng 4233 (#72)" (#76) This reverts commit 57cc5e6. * [try-mtls] | c3 mtls support * [try-mtls] | adding c3 in mtls scenario * [try-mtls] | fix c3 bugs * [try-mtls] | adding impersonation users to molecule scenarios * [try-mtls] | fix kafka rest listener and conlfuent.license config * [try-mtls] | file based login in mtls only scenario added * [try-mtls] | fix oauth and ldap scenarios and confluent.license * [try-mtls] | send certs in kafka broker tasks for register cluster * [try-mtls] | fixing register cluster to run on internal token listener so it has sasl_ssl protocol instead of ssl for rbac over mtls * [mtls-connect] | making retries in get auth token configurable and increasing the default wait time as mds takes more time in upgrades * [mtls-connect] | adding ksql connect in molecule scenarios * [mtls-connect] | fix erp pm rules * [mtls-connect] | remove cyclic dependency in ssl_client_authentication and ssl_mutual_auth_enabled * [mtls-connect] | remove set fact for mtls old var * [mtls-connect] | add when conditions for extract principal --------- Co-authored-by: Mansi Jain <[email protected]>

… rules

CLAassistant · 2024-10-18T13:47:37Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

confluent-cla-assistant · 2024-10-18T13:47:41Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

…abled and ssl_client_authentication and askign for user confirmation for setting requested

…instead of oauth_enabled and ldap_with_oauth_enabled varaibles

…urls

…ving rolebindings to the principal for mtls

… and broker based on external mds setup or single cluster setup

…l client authentication enabled

…y certs to communicate to mds even when mds has support for oauth/ldap

rrbadiani · 2024-10-25T07:55:13Z

roles/variables/tasks/main.yml

+- name: Define the new ssl_client_authentication variable
+  set_fact:
+    to_be_ssl_client_authentication: >-
+      {%- if deployment_strategy == 'parallel' -%}


or not rbac_enabled|bool should also be there

rrbadiani · 2024-10-25T08:01:23Z

roles/variables/vars/main.yml

+        {{
+           ('ldap' in auth_mode) | ternary("User:" + mds_super_user|default('mds'), "") +
+           (auth_mode == 'ldap_with_oauth') | ternary(";", "") +
+           ('oauth' in auth_mode) | ternary("User:" + oauth_superuser_principal, "")


mtls super user should also be added.

mansisinha · 2024-10-25T10:31:48Z

roles/kafka_broker/tasks/register_cluster.yml

+    validate_certs: false
+    headers:
+      Content-Type: application/json
+      Authorization: "Bearer {{ authorization_token }}"


why do we have authorization token in mtls only ?

It got added as I copied the above task and created this one.
Although i dont think it makes any difference if we pass the token or not. If we pass it the principal will be taken from the token. and token was given for kafka broker cert. And if dont pass the token principal will be taken from the cert of kafka broker. so same principal will be used either way.

agreed, it would pass the same token and does not make any difference on the output, but since this is misleading, I think we should remove the token field

mansisinha · 2024-10-25T10:33:04Z

roles/kafka_broker/tasks/register_cluster.yml

+    client_cert: "{{ kafka_broker_cert_path if send_client_cert|bool else omit }}"
+    client_key: "{{ kafka_broker_key_path if send_client_cert|bool else omit }}"
+    body_format: json
+    # Cant use internal listener as register cluster doesnt support SSL protocol


could you share any reference/docs for this conclusion? if we are using mds endpoint, SSL should be sufficient afaik. SASL_SSL was needed for kafka rest api

3rd point in this doc https://confluentinc.atlassian.net/wiki/spaces/OAAC/pages/3653567526/mTLS+with+RBAC+Gotchas

Also to clarify the SASL_SSL protocol is needed in the body of this post request as the value of "protocol" parameter and doesnt mean the request to /registry/clusters endpoint must have some sasl credentials

mansisinha · 2024-10-25T10:36:54Z

roles/kafka_connect/tasks/health_check.yml

@@ -9,6 +9,8 @@
    oauth_password: "{{ kafka_connect_oauth_password }}"
    ldap_user: "{{ kafka_connect_ldap_user }}"
    ldap_password: "{{ kafka_connect_ldap_password }}"
+    mds_mtls_client_cert: "{{ kafka_connect_cert_path }}"


the variable name is misleading, we are not sending mds certs here, we send component wise cert for each component. Please rename it to mtls_client_cert

by mds_mtls_client_cert i meant cert of MDS's client for mtls. and not cert of MDS. But I get that this might be misleading name. mtls_client_cert makes sense. will change this

mansisinha · 2024-10-25T10:39:43Z

roles/variables/defaults/main.yml

@@ -212,9 +212,12 @@ ssl_enabled: false
 ### Set this variable to customize expiration days for certificate authority. Applies for all components of Confluent Platform.
 certificate_authority_expiration_days: 365

-### Boolean to enable mTLS Authentication on all components. Configures all components to use mTLS for authentication into Kafka
+### Boolean to enable mTLS Authentication on all components. Configures all components to use mTLS for authentication into Kafka. No need to define this anymore, use ssl_client_authentication instead.


we can instead mark this variable as deprecated

I had a little confusion there as internally at most places we do continue to use this variable. Just at the end when we need the value we use the new variable. Even in that case should we mark it deprecated ?

from customer's pov, this will still count as deprecated. It's more of an internal variable for us in that case. Also, if we do mark it as deprecated, let's updated all the molecule scenarios to replace this bool with ssl_client_authetication (even for non rbac mtls)

mansisinha · 2024-10-25T10:41:27Z

roles/variables/defaults/main.yml

@@ -700,10 +734,20 @@ kafka_broker_custom_client_properties: {}
 ### Boolean to enable the embedded rest proxy within Kafka. NOTE- Embedded Rest Proxy must be enabled if RBAC is enabled and Confluent Server must be enabled
 kafka_broker_rest_proxy_enabled: "{{confluent_server_enabled and not ccloud_kafka_enabled }}"

+### Property of ERP as MDS client. Can be set to true when ssl_client_authentication is not none. When set to true will not send oauth token or ldap creds to MDS even when MDS server has support for accepting them. Keeping false means if MDS has oauth and mtls support client will send both oauth token and cert
+kafka_broker_rest_proxy_to_mds_send_certs_only: false


this variable naming should be better, it's too long for a variable name. Please rename to something like rp_mds_certs_only and add the details in the docs

I thought all variables for ERP have the prefix kafka_broker_rest_proxy_ so I also kept the same prefix. Length wise yes its long but there are many longer variables.

mansisinha · 2024-10-25T11:00:22Z

Added some comments, other than those, Open question:

Why are we using SASL_SSL listener with MDS /register endpoint? Please share the doc reference

Suggestions:

Please rename the <component>_to_mds_send_certs_only to a better variable name for code quality. Something like rp_mds_certs_only, connect_mds_certs_only and add the description in docs.md
Please add creating connectors in the molecule tests to validate Connect server mtls and dual auth. Similarly, please add kafka produce/consumer commands in verify.yml over different listeners to validate the listeners.
Please add check to validate if mds_super_user_external_cert_path is defined in a 2 cluster setup. Adding to this, the variable should be renamed to mds_superuser_cert_path and kafka_broker_cert_path should default to this and its principal should be added to super.users. This is similar to the current flow where customer defines mds_super_user or oauth_superuser_principal. This super user principal is the principal that can provide rolebindings and which may or may not be equal to kafka principal, specially in external mds setup.

rrbadiani · 2024-10-25T13:07:18Z

Why are we using SASL_SSL listener with MDS /register endpoint? Please share the doc reference

The doc reference https://confluentinc.atlassian.net/wiki/spaces/OAAC/pages/3653567526/mTLS+with+RBAC+Gotchas

Suggestions:

Please rename the <component>_to_mds_send_certs_only to a better variable name for code quality. Something like rp_mds_certs_only, connect_mds_certs_only and add the description in docs.md

The variable name should make the purpose clear. If we shorten the name to _mds_certs_only by this name it cant convey full meaning. How about the name <component>_mds_cert_auth_only ?

Please add creating connectors in the molecule tests to validate Connect server mtls and dual auth. Similarly, please add kafka produce/consumer commands in verify.yml over different listeners to validate the listeners.

Sure will add that

Please add check to validate if mds_super_user_external_cert_path is defined in a 2 cluster setup. Adding to this, the variable should be renamed to mds_superuser_cert_path and kafka_broker_cert_path should default to this and its principal should be added to super.users. This is similar to the current flow where customer defines mds_super_user or oauth_superuser_principal. This super user principal is the principal that can provide rolebindings and which may or may not be equal to kafka principal, specially in external mds setup.

Yes agreed validation should be added and principal of this cert should be added to super.user of 2nd cluster
On default value of kafka broker cert path being same as mds super user cert. That part I don't get it. Like in case when there are 2 clusters that makes sense for custom certs. but in case when there is single cluster this variable cant be defined or even for 2nd clusters in cases of provided keystore,truststore this cant be defined.

rrbadiani and others added 14 commits October 16, 2024 10:48

[mtls-combined] | fix bugs in delegate and getting pm rules

a5a9316

[mtls-combined] | fix connect rbac send certs

376ccaa

[mtls-combined] | fix connect rbac, ksql rbac send certs

9165607

[mtls-combined] | fix ksql send certs

48af80e

[mtls-combined] | fix get token bug, missing delegate and molecule pm…

067d35c

… rules

[mtls-combined] | handle provided ks/ts

0da75e7

[mtls-combined] | fix verify of 2 scenarios

91a0970

[mtls-combined] | dont delete cert in auth mode mtls

b6430cf

[mtls-combined] | verify fixes

90a980c

[mtls-combined] | adding old var for mtls

bd1689c

[mtls-combined] | remove debug logs

5332335

[ANSIENG-3807] | revert replicator mtls changes

069ef31

[ANSIENG-3807] | adding more comments

c25dc50

rrbadiani requested a review from a team as a code owner October 18, 2024 13:47

[ANSIENG-3807] | fixing linting in filters.py file

ca89017

rrbadiani changed the base branch from 7.8.x to master October 19, 2024 09:53

rrbadiani changed the base branch from master to 7.8.x October 19, 2024 09:53

rrbadiani added 9 commits October 20, 2024 10:28

[ANSIENG-3807] | removing cyclic dependency of old ssl_mutual_auth_en…

197c02d

…abled and ssl_client_authentication and askign for user confirmation for setting requested

[ANSIENG-3807] | fixing the variables/vars.yml file to use auth_mode …

149140c

…instead of oauth_enabled and ldap_with_oauth_enabled varaibles

[ANSIENG-3807] | configure dns properly for ec2 to connect to docker …

1b2496a

…urls

[ANSIENG-3807] | remove space from cert subject to avoid errors in gi…

8c2c045

…ving rolebindings to the principal for mtls

[ANSIENG-3807] | modifying delegate to handle delegation to localhost…

479fb62

… and broker based on external mds setup or single cluster setup

[ANSIENG-3807] | adding support for replicator

593592a

[ANSIENG-3807] | give role bindings to certs user whenever mds has ss…

7a707e0

…l client authentication enabled

[ANSIENG-3807] | adding new variables to allow mds clients to use onl…

6373b54

…y certs to communicate to mds even when mds has support for oauth/ldap

[ANSIENG-3807] | adding a few molecule scenarios

1c5d258

rrbadiani commented Oct 25, 2024

View reviewed changes

mansisinha reviewed Oct 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ANSIENG-3807] | Rbac over mTLS #1804

[ANSIENG-3807] | Rbac over mTLS #1804

rrbadiani commented Oct 18, 2024 •

edited

Loading

CLAassistant commented Oct 18, 2024

confluent-cla-assistant bot commented Oct 18, 2024

rrbadiani Oct 25, 2024

rrbadiani Oct 25, 2024

mansisinha Oct 25, 2024

rrbadiani Oct 25, 2024

mansisinha Oct 25, 2024

rrbadiani Oct 25, 2024

mansisinha Oct 25, 2024

rrbadiani Oct 25, 2024

rrbadiani Oct 25, 2024

mansisinha Oct 25, 2024

rrbadiani Oct 25, 2024

mansisinha Oct 25, 2024

rrbadiani Oct 25, 2024

mansisinha Oct 25, 2024

mansisinha Oct 25, 2024

rrbadiani Oct 25, 2024

mansisinha commented Oct 25, 2024

rrbadiani commented Oct 25, 2024

[ANSIENG-3807] | Rbac over mTLS #1804

Are you sure you want to change the base?

[ANSIENG-3807] | Rbac over mTLS #1804

Conversation

rrbadiani commented Oct 18, 2024 • edited Loading

Description

Type of change

How Has This Been Tested?

Checklist:

CLAassistant commented Oct 18, 2024

confluent-cla-assistant bot commented Oct 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mansisinha commented Oct 25, 2024

rrbadiani commented Oct 25, 2024

rrbadiani commented Oct 18, 2024 •

edited

Loading