How to set up multi-broker Kafka cluster?

How to set up multi-broker Kafka cluster?

In my previous post, I explained how to set up Apache Kafka locally using a single broker. In this post, I’ll tell you how to add more brokers (nodes) to the Kafka cluster. It is very easy to start several Kafka server instances and by doing that we can add broker(s) to the clusters.

If you don’t have Kafka running, please refer to my previous post as this post assumes that you have Kafka, ZooKeeper up and running. Let’s add two more brokers to the Kafka cluster but all running locally.

Step 1: Setting up a multi-broker cluster

Starting a new Kafka server is very easy by using the file. Based on the previous article, one broker is already running that listens to the request on localhost:9092 based on default configuration values.

To add more broker(s), let’s duplicate the properties file and modify some key configurations like broker-id, listener and log directory to make the new broker(s) unique in the cluster. As such these brokers will be running locally so we don’t want to overwrite logs logged by an individual broker.

Duplicate the existing properties file for each new broker:

Now edit these new files and set the following properties:

Step 2: Add 2 new Kafka brokers to the cluster

We already have a ZooKeeper instance running, we can now add two more Kafka brokers:

Step 3: Create a new topic with a replication factor of 3

As such we now have 3 brokers running in the cluster, it is good time to try replication for a topic so that the messages are copied to other brokers as well. This basically helps in case of fault tolerance when a broker goes down, we will still have the data written somewhere.

The command above will create a topic named “my-replicated-topic” with replication factor as 3 and 1 partition.

Step 4: Monitor a topic using describe flag

Here is an explanation of the output above. The first line gives a summary of all the partitions, each additional line gives information about one partition. Since we have only one partition for this topic there is only one line.

  • “leader” is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the partitions.
  • “replicas” is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
  • “isr” is the set of “in-sync” replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.

I’ve got an exercise for you: Try to monitor the previous topic where Replicas will be 0.

Step 5: Send messages to a new replicated topic

Sending messages to our new replicated topic is easy and same as shown in the previous article:

Enter messages and hit enter to send messages to the broker which will then be replicated across other registered brokers in the cluster.

Step 6: Read messages from the new replicated topic

Reading messages from the replicated topic is also the same as shown in the previous article:

As you can get from the post, adding new brokers to the cluster is very easy. The concepts of sending/reading messages to/from the topic respectively, are same no matter what the replication factor or partitions configuration is registered while creating the topic.

If you would like to learn more about Apache Kafka, please subscribe to my blog as I’ll be writing more how-to articles very soon.

Siddharth Pandey

Siddharth Pandey is a Software Engineer with thorough hands-on commercial experience & exposure to building enterprise applications using Agile methodologies. Siddharth specializes in building, managing on-premise, cloud based real-time standard, single page web applications (SPAs). He has successfully delivered applications in health-care, finance, insurance, e-commerce sectors for major brands in the UK. Other than programming, he also has experience of managing teams, trainer, actively contributing to the IT community by sharing his knowledge using Stack Overflow, personal website & video tutorials.

You may also like...

Advertisment ad adsense adlogger