Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bitnami/kafka] Unable to set client authentication (SASL / SCRAM-SHA-512) through docker-compose #73485

Open
Shykeng opened this issue Oct 22, 2024 · 0 comments
Open

[bitnami/kafka] Unable to set client authentication (SASL / SCRAM-SHA-512) through docker-compose #73485

Shykeng opened this issue Oct 22, 2024 · 0 comments
Assignees
@jotamartos
Labels
in-progress kafka tech-issues The user has a technical issue about an application

Comments

@Shykeng
Copy link

Shykeng commented Oct 22, 2024

Name and Version

binami/kafka 3.8.0

What architecture are you using?

None

What steps will reproduce the bug?

1. JAAS set up with environment variables

kafka-broker01:
   image: bitnami/kafka:3.8.0
   container_name: kafka-broker01
   hostname: kafka-broker01
   restart: always
   environment:
     - BITNAMI_DEBUG=true
     # General settings
     - KAFKA_ENABLE_KRAFT=yes
     - KAFKA_HEAP_OPTS=-Xmx8G -Xms8G -Djavax.net.debug=ssl
     - KAFKA_KRAFT_CLUSTER_ID=ZmYxNjBiYTE0YmQ4NDFhZm
     - KAFKA_CFG_PROCESS_ROLES=broker
     - KAFKA_CFG_NODE_ID=104
     - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=false
     - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=101@kafka-controller01:9093,102@kafka-controller02:9093,103@kafka-controller03:9093   
     # TLS  
     - KAFKA_TLS_TYPE=JKS
     - KAFKA_CFG_SSL_KEYSTORE_TYPE=JKS
     - KAFKA_CFG_SSL_KEYSTORE_LOCATION=/bitnami/kafka/config/certs/kafka.keystore.jks
     - KAFKA_CFG_SSL_KEYSTORE_PASSWORD=${BROKER_KEYSTORE_PASSWORD}
     - KAFKA_CFG_SSL_TRUSTSTORE_LOCATION=/bitnami/kafka/config/certs/kafka.truststore.jks
     - KAFKA_CFG_SSL_TRUSTSTORE_PASSWORD=${BROKER_TRUSTSTORE_PASSWORD}
     # Authentication
     - KAFKA_CFG_LISTENER_NAME_BROKER_SSL_CLIENT_AUTH=required
     - KAFKA_CFG_LISTENER_NAME_INTERNAL_SSL_CLIENT_AUTH=required
     - KAFKA_CFG_LISTENER_NAME_EXTERNAL_SSL_CLIENT_AUTH=required
     - KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SCRAM-SHA-512
     - KAFKA_CFG_SASL_MECHANISM_CONTROLLER_PROTOCOL=PLAIN
     - KAFKA_CFG_SASL_MECHANISM=SCRAM-SHA-512
     - KAFKA_CFG_SASL_ENABLED_MECHANISMS=SCRAM-SHA-512
     - KAFKA_INTER_BROKER_USER=${BROKER_USERNAME}
     - KAFKA_INTER_BROKER_PASSWORD=${BROKER_PASSWORD}
     - KAFKA_CFG_LISTENER_NAME_CONTROLLER_PLAIN_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="${CONTROLLER_USERNAME}" password="${CONTROLLER_PASSWORD}" user_${CONTROLLER_USERNAME}="${CONTROLLER_PASSWORD}";
     - KAFKA_CFG_LISTENER_NAME_INTERNAL_SCRAM-SHA-512_SASL_JAAS_CONFIG=org.apache.kafka.common.security.scram.ScramLoginModule required username="${INTERNAL_USER}" password="${INTERNAL_PASSWORD}";
     - KAFKA_CFG_LISTENER_NAME_EXTERNAL_SCRAM-SHA-512_SASL_JAAS_CONFIG=org.apache.kafka.common.security.scram.ScramLoginModule required username="${EXTERNAL_USER}" password="${EXTERNAL_PASSWORD}";
     # Listeners
     - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=BROKER
     - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER # Kraft Listener
     - KAFKA_CFG_LISTENERS=BROKER://kafka-broker01:9092,INTERNAL://kafka-broker01:9093,EXTERNAL://0.0.0.0:9094
     - KAFKA_CFG_ADVERTISED_LISTENERS=BROKER://kafka-broker01:9092,INTERNAL://kafka-broker01:9093,EXTERNAL://10.210.0.170:9194
     - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:SASL_PLAINTEXT,BROKER:SASL_SSL,INTERNAL:SASL_SSL,EXTERNAL:SASL_SSL
     # SET Metrics
     - EXTRA_ARGS=-javaagent:/usr/share/java/jmx_prometheus_javaagent-0.20.0.jar=3200:/opt/bitnami/kafka/config/jmx/kafka-broker.yml
   volumes:
     - ./certs/kafka-broker01/stores:/bitnami/kafka/config/certs
     - /kafka/data/kafka-broker01:/bitnami/kafka/data
     - ./plugins/jmx_prometheus_javaagent-0.20.0.jar:/usr/share/java/jmx_prometheus_javaagent-0.20.0.jar
     - ./jmx/kafka-broker.yml:/opt/bitnami/kafka/config/jmx/kafka-broker.yml
   ports:
     - 9194:9094
     - 3204:3200
   networks:
     - kafka-network

2. JAAS set up with a configuration file :

kafka-broker01:
   image: bitnami/kafka:3.8.0
   container_name: kafka-broker01
   hostname: kafka-broker01
   restart: always
   environment:
     - BITNAMI_DEBUG=true
     # General settings
     - KAFKA_ENABLE_KRAFT=yes
     - KAFKA_HEAP_OPTS=-Xmx8G -Xms8G -Djavax.net.debug=ssl
     - KAFKA_KRAFT_CLUSTER_ID=ZmYxNjBiYTE0YmQ4NDFhZm
     - KAFKA_CFG_PROCESS_ROLES=broker
     - KAFKA_CFG_NODE_ID=104
     - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=false
     - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=101@kafka-controller01:9093,102@kafka-controller02:9093,103@kafka-controller03:9093   
     # TLS  
     - KAFKA_TLS_TYPE=JKS
     - KAFKA_CFG_SSL_KEYSTORE_TYPE=JKS
     - KAFKA_CFG_SSL_KEYSTORE_LOCATION=/bitnami/kafka/config/certs/kafka.keystore.jks
     - KAFKA_CFG_SSL_KEYSTORE_PASSWORD=${BROKER_KEYSTORE_PASSWORD}
     - KAFKA_CFG_SSL_TRUSTSTORE_LOCATION=/bitnami/kafka/config/certs/kafka.truststore.jks
     - KAFKA_CFG_SSL_TRUSTSTORE_PASSWORD=${BROKER_TRUSTSTORE_PASSWORD}
     # Authentication
     - KAFKA_CFG_LISTENER_NAME_BROKER_SSL_CLIENT_AUTH=required
     - KAFKA_CFG_LISTENER_NAME_INTERNAL_SSL_CLIENT_AUTH=required
     - KAFKA_CFG_LISTENER_NAME_EXTERNAL_SSL_CLIENT_AUTH=required
     - KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SCRAM-SHA-512
     - KAFKA_CFG_SASL_MECHANISM_CONTROLLER_PROTOCOL=PLAIN
     - KAFKA_CFG_SASL_MECHANISM=SCRAM-SHA-512
     - KAFKA_CFG_SASL_ENABLED_MECHANISMS=SCRAM-SHA-512
     - KAFKA_INTER_BROKER_USER=${BROKER_USERNAME}
     - KAFKA_INTER_BROKER_PASSWORD=${BROKER_PASSWORD}
     - KAFKA_CFG_LISTENER_NAME_CONTROLLER_PLAIN_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="${CONTROLLER_USERNAME}" password="${CONTROLLER_PASSWORD}" user_${CONTROLLER_USERNAME}="${CONTROLLER_PASSWORD}";
     # Listeners
     - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=BROKER
     - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER # Kraft Listener
     - KAFKA_CFG_LISTENERS=BROKER://kafka-broker01:9092,INTERNAL://kafka-broker01:9093,EXTERNAL://0.0.0.0:9094
     - KAFKA_CFG_ADVERTISED_LISTENERS=BROKER://kafka-broker01:9092,INTERNAL://kafka-broker01:9093,EXTERNAL://10.210.0.170:9194
     - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:SASL_PLAINTEXT,BROKER:SASL_SSL,INTERNAL:SASL_SSL,EXTERNAL:SASL_SSL
     # SET Metrics
     - EXTRA_ARGS=-javaagent:/usr/share/java/jmx_prometheus_javaagent-0.20.0.jar=3200:/opt/bitnami/kafka/config/jmx/kafka-broker.yml
     # SET JAAS
     - KAFKA_OPTS=-Djava.security.auth.login.config=/kafka/config/kafka_server_jaas.conf
   volumes:
     - ./certs/kafka-broker01/stores:/bitnami/kafka/config/certs
     - /kafka/data/kafka-broker01:/bitnami/kafka/data
     - ./plugins/jmx_prometheus_javaagent-0.20.0.jar:/usr/share/java/jmx_prometheus_javaagent-0.20.0.jar
     - ./jmx/kafka-broker.yml:/opt/bitnami/kafka/config/jmx/kafka-broker.yml
     - ./config/kafka_server_jaas.conf:/kafka/config/kafka_server_jaas.conf
   ports:
     - 9194:9094
     - 3204:3200
   networks:
     - kafka-network

3. The kafka_server_jaas.conf

internal.KafkaServer {
   org.apache.kafka.common.security.scram.ScramLoginModule required
   username="kafka-internal"
   password="changeit";
};

external.KafkaServer {
   org.apache.kafka.common.security.scram.ScramLoginModule required
   username="kafka-external"
   password="changeit";
};

4. The .env file

CONTROLLER_USERNAME="kafka-controller"
CONTROLLER_PASSWORD="changeit"

BROKER_KEYSTORE_PASSWORD="changeit"
BROKER_TRUSTSTORE_PASSWORD="changeit"
BROKER_USERNAME="kafka-broker"
BROKER_PASSWORD="changeit"

INTERNAL_USER="kafka-internal"
INTERNAL_PASSWORD="changeit"

EXTERNAL_USER="kafka-external"
EXTERNAL_PASSWORD="changeit"

What is the expected behavior?

Proper server.properties generation to enable SASL_SSL through SCRAM-SHA-512 authentication for clients.

What do you see instead?

The listener.name.{listenerName}.{saslMechanism}.sasl.jaas.config translated as KAFKA_CFG_LISTENER_NAME_<LISTENER_NAME>_<PROTOCOL>_SASL_JAAS_CONFIG in bitnami/kafka docker-compose isn't being set properly in server.properties. Also when the jaas config file is mounted in the container the same behavior is happening.

Generated server.properties:

############################# Group Coordinator Settings #############################

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
#group.initial.rebalance.delay.ms=0

auto.create.topics.enable=false
controller.listener.names=CONTROLLER
controller.quorum.voters=101@kafka-controller01:9093,102@kafka-controller02:9093,103@kafka-controller03:9093
inter.broker.listener.name=BROKER

listener.name.broker.ssl.client.auth=required
listener.name.controller.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka-controller" password="changeit" user_kafka-controller="changeit";
listener.name.external.ssl.client.auth=required
listener.name.internal.ssl.client.auth=required

node.id=104
process.roles=broker
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.controller.protocol=PLAIN
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
ssl.keystore.location=/bitnami/kafka/config/certs/kafka.keystore.jks
ssl.keystore.password=changeit
ssl.keystore.type=JKS
ssl.truststore.location=/bitnami/kafka/config/certs/kafka.truststore.jks
ssl.truststore.password=changeit
ssl.client.auth=required

ssl.truststore.type=JKS

listener.name.controller.sasl.enabled.mechanisms=PLAIN

listener.name.broker.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="kafka-broker" password="changeit";
listener.name.internal.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;
listener.name.external.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;

Both internal and external have no defined user/password with both methods:

listener.name.internal.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;
listener.name.external.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;

It's resulting on such logs on the kafka-broker01 when trying to initiate a connection with a client:

[2024-10-22 14:12:23,480] INFO [SocketServer listenerType=BROKER, nodeId=104] Failed authentication with /172.19.0.12 (channelId=172.19.0.6:9093-172.19.0.12:49798-134) (Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512) (org.apache.kafka.common.network.Selector)
javax.net.ssl|DEBUG|63|data-plane-kafka-network-thread-104-ListenerName(INTERNAL)-SASL_SSL-4|2024-10-22 14:12:23.506 UTC|SSLCipher.java:1870|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|63|data-plane-kafka-network-thread-104-ListenerName(INTERNAL)-SASL_SSL-4|2024-10-22 14:12:23.506 UTC|SSLCipher.java:2024|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|ALL|63|data-plane-kafka-network-thread-104-ListenerName(INTERNAL)-SASL_SSL-4|2024-10-22 14:12:23.506 UTC|X509Authentication.java:280|X509KeyManager class: sun.security.ssl.SunX509KeyManagerImpl
javax.net.ssl|ALL|63|data-plane-kafka-network-thread-104-ListenerName(INTERNAL)-SASL_SSL-4|2024-10-22 14:12:23.506 UTC|X509Authentication.java:299|No X.509 cert selected for EC
javax.net.ssl|ALL|63|data-plane-kafka-network-thread-104-ListenerName(INTERNAL)-SASL_SSL-4|2024-10-22 14:12:23.506 UTC|X509Authentication.java:299|No X.509 cert selected for EdDSA
javax.net.ssl|DEBUG|63|data-plane-kafka-network-thread-104-ListenerName(INTERNAL)-SASL_SSL-4|2024-10-22 14:12:23.526 UTC|SSLCipher.java:2024|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE
countdown value = 137438953472

Additional information

Additional information

I am aware of those two currents issues #27566 and #41415 that the controller doesn't support SCRAM but only PLAIN.

If you have an alternative solution or if you think there are any misconfiguration please let me know, thank you.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jotamartos jotamartos

Labels
in-progress kafka tech-issues The user has a technical issue about an application
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@javsalgar @jotamartos @Shykeng