Deploy a Kafka cluster in Kubernetes using the Confluent Operator


main

Confluent Operator allows you to deploy and manage Confluent Platform as a cloud-native, stateful container application on Kubernetes and OpenShift. The automation provided by Kubernetes, Operator, and Helm greatly simplifies provisioning and minimizes the burden of operating and managing Confluent Platform clusters. Operator also provides you with the portability to use Apache Kafka® in multiple provider zones and across both your private and public cloud environments.

The intention of this tutorial is to walk you through the steps to install a Kafka cluster using the Confluent Operator in Kubernetes.

Kafka Cluster Version: 5.5.2.0

  • 1 Environment check

  • 2 Installation plan execution

License key required.

There are 3 types of licenses:

Developer License

Trial (Evaluation) License

Enterprise (Subscription) License

For this tutorial I have used an Enterprise (subscription) License.

See more information about how to configure licenses with Confluent operator

Download Confluent Operator from here

Prepare your configuration file.

$VALUES_FILE refers to the configuration file you set up in Create the global configuration file.

Here you can find an example cofiguration file

Create namespace.

kubectl create ns kafka

Confluent operator installation

export VALUES_FILE=<path_to_pks_yaml>

helm upgrade --install \
  operator \
  ./confluent-operator \
  --values $VALUES_FILE \
  --namespace kafka \
  --set operator.enabled=true

Zookeeper installation

helm upgrade --install \
  zookeeper \
  ./confluent-operator \
  --values $VALUES_FILE \
  --namespace kafka \
  --set zookeeper.enabled=true

Kafka Brokers installation

At this point you can generate the SSL certificates to be used by your Kafka brokers.

helm upgrade --install kafka ./confluent-operator \
  --values $VALUES_FILE \
  --namespace kafka \
  --set kafka.enabled=true \
  --set-file kafka.tls.cacerts=/<paths-to>/ca-bundle.pem \
  --set-file kafka.tls.fullchain=/<path-to>/server-cert.pem \
  --set-file kafka.tls.privkey=/<path-to>/server-key.pem

This deployment will create the pods for the Kafka brokers and also wil create services and locad balancers in kafka Namespace.

At this point make sure that all Load balancers have host names/IPs registered with DNS - this applies for all components exposing REST endpoints too.

We can test the access to the Kafka cluster using the MDS user defined in the VALUES_FILE.

Using the Confluent tools that you can download here

Untar and cd to confluent-5.5.2/bin
curl -sL https://cnfl.io/cli | sh -s -- -b ./

Then test the access to the cluster with: confluent cluster describe --url https://broker0.mycompany.com

It should return the ID of the cluster.

If this returns a certificate related error use the following command: confluent login --save --url https://broker0.mycompany.com --ca-cert-path ca.pem

At this point, if you are using LDAP as an authentication system, you should login as MDS user in the cluster using the Confluent CLI tool and executes the roles binding for all the users used by others components (schemaregistry, restproxy, ksql, etc..).

Connect installation

Build custom image using this Dockerfile

Download the ojdbc8.jar file from the Oracle website

docker build . -t \ cp-server-connect-operator:5.5.2.0.v.1
docker push cp-server-connect-operator:5.5.2.0.v.1

Don’t forget to update the VALUES_FILE on this line with the docker image name and tag you just built and push to your Docker registry.

helm upgrade --install connect ./confluent-operator \
  --values $VALUES_FILE \
  --namespace kafka \
  --set connect.enabled=true \
  --set-file connect.tls.cacerts=/<paths-to>/ca-bundle.pem \
  --set-file connect.tls.fullchain=/<path-to>/server-cert.pem \
  --set-file connect.tls.privkey=/<path-to>/server-key.pem \
  --set global.provider.registry.fqdn=docker.io

KSQL installation

helm upgrade --install ksql ./confluent-operator \
  --values $VALUES_FILE \
  --namespace kafka \
  --set ksql.enabled=true \
  --set-file ksql.tls.cacerts=/<paths-to>/ca-bundle.pem \
  --set-file ksql.tls.fullchain=/<path-to>/server-cert.pem \
  --set-file ksql.tls.privkey=/<path-to>/server-key.pem

Schemaregistry installation

helm upgrade --install schemaregistry ./confluent-operator \
  --values $VALUES_FILE \
  --namespace <namespace> \
  --set schemaregistry.enabled=true \
  --set-file schemaregistry.tls.cacerts=/<paths-to>/ca-bundle.pem \
  --set-file schemaregistry.tls.fullchain=/<path-to>/server-cert.pem \
  --set-file schemaregistry.tls.privkey=/<path-to>/server-key.pem

Control Center installation

helm upgrade --install controlcenter ./confluent-operator \
  --values $VALUES_FILE \
  --namespace kafka> \
  --set controlcenter.enabled=true \
  --set-file controlcenter.tls.cacerts=/<paths-to>/ca-bundle.pem \
  --set-file controlcenter.tls.fullchain=/<path-to>/server-cert.pem \
  --set-file controlcenter.tls.privkey=/<path-to>/server-key.pem

  • 3 Tests and generate data

At this point you have a fully working Kafka cluster and you can start producing and consuming data. During the development process is a good practice to test producing and consuming data using the Confluent tools you downloaded in previous steps.

Some examples here below.

Create a test topic using kafka-topics CLI tool:

kafka-topics --create \
  --bootstrap-server bootstrap.mycompany.domain:9092 \
  --replication-factor 1 \
  --partitions 1 \
  --topic test-topic

For more example visit here

As usual, if you have any question, send me a message at contact@wecloudpro.com

Back to blog