CMAK operator

Table of Contents

• Installation
• Configuration
• Architecture
• Troubleshooting

CMAK operator is a set of tools packaged as Helm chart, that allows to install and configure CMAK (previously Kafka Manager) into Kubernetes cluster.

CMAK (previously Kafka Manager) is well-known and mature tool for monitoring and managing Apache Kafka clusters.

Installation

CMAK operator could be installed only with Helm 3. Helm chart is published to public Helm repository, hosted on GitHub itself.

It’s recommended to install CMAK operator into a dedicated namespace.

Add Helm repository

$ helm repo add cmak https://eshepelyuk.github.io/cmak-operator
$ helm repo update

Install latest version

$ helm install --create-namespace -n cmak-ns mycmak cmak/cmak-operator

Seach for available versions

$ helm search repo cmak-operator --versions
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
cmak/cmak-operator      0.2.1           3.0.0.5         CMAK operator for K8S.
cmak/cmak-operator      0.2.0           3.0.0.5         CMAK operator for K8S.

Install specific version

$ helm install --create-namespace -n cmak-ns --version 0.2.1 mycmak cmak/cmak-operator

Verify installation

❗	Exposing CMAK UI By default, CMAK operator doesn’t create neither Ingress nor any other K8S resources to expose UI via HTTP.

The simpliest test is to port forward CMAK UI HTTP port and access it from browser .

$ kubectl port-forward -n cmak-ns service/cmak 9000:9000

Then, open http://localhost:9000 in your browser.

Configuration

CMAK application

CMAK uses configuration file /cmak/conf/application.conf. Every parameter could be overriden via JVM system property, i.e. -DmyProp=myVal. Properties are passed to CMAK container via values.yaml.

For example to enable basic auth add following

values.yaml

ui:
  extraArgs:
  - "-DbasicAuthentication.enabled=true"
  - "-DbasicAuthentication.username=admin"
  - "-DbasicAuthentication.password=password"

Kafka clusters

It’s extremely easy to configure multiple clusters in CMAK, starting from cluster setup, connection settings and ending with authorization using Helm values files.

Check CMAK operator values for all available options and their description.

1. Minimal values.yaml configuration for adding a several Kafka clusters to CMAK.

cmak:
  clusters:
    - name: "cluster-stage"
      curatorConfig:
        zkConnect: "kafka01.stage:2181,kafka02.stage:2181"
    - name: "cluster-prod"
      curatorConfig:
        zkConnect: "kafka01.prod:2181,kafka02.prod:2181,kafka03.prod:2181"

Connection settings could be configured for all clusters at once or per dedicated cluster.

1. Configuring connection settings

cmak:
  clustersCommon:
    curatorConfig:
      zkMaxRetry: 100 # (1)
  clusters:
    - name: "cluster-stage"
      kafkaVersion: "2.5.0" # (2)
      curatorConfig:
        zkConnect: "kafka01.stage:2181,kafka02.stage:2181"
    - name: "cluster-prod"
      kafkaVersion: "2.1.0" # (3)
      enabled: false
      curatorConfig:
        zkConnect: "kafka01.prod:2181,kafka02.prod:2181,kafka03.prod:2181"

1. this setting is applied to both clusters.

2. applied only to cluster-stage.

3. applied only to cluster-prod.

Configuration should be passed to helm via command line during installation or upgrade.

$ helm install --create-namespace -n cmak-ns -f cmak-values.yaml mycmak cmak/cmak-operator

Architecture

CMAK operator comprises following components:

• CMAK, powered by Docker image from @hleb-albau.

• Apache ZooKeeper, powered by official Docker image.

• K8S CronJob used to configure Kafka clusters, powered by custom Docker image.

Following desing choices were made.

Dedicated Zookeeper instance.

TO BE DEFINED.

Not using REST for configuring CMAK clusters.

TO BE DEFINED.

Reconciliation with CronJob.

TO BE DEFINED.

Troubleshooting

CMAK doesn’t configure clusters from Helm values

• CMAK settings are not applied immediately, but only after reconcile.schedule period had passed.

• Check logs of cron job to see if there’s no connection failure to ZK.