Saturday, November 22, 2014

Setup Kafka in a cluster

To setup Kafka in a cluster, first we must have the zookeeper cluster setup and running (follow this link:, suppose that the zookeeper cluster consists of the zookeeper servers running at the following hostname:ports:

As I have only two computers, therefore i will use the same computers (but at different ports) to host the kafka cluster. For this case, the Kafka servers/brokers will be running at the following hostname:ports

To do this, follow this link ( to setup the kafka server. Now navigate to the kafka root folder oif each computer and modify the in "config" sub folder:

> gedit config/

In the file, search the line "zookeeper.connect" and change it to:


Then search the line "" (unique id for each broker node) and change it to "" on computer and to "" on computer

Next search the line "" and change it to "" on computer and to "" on computer

Make sure that the line "port=9092" is there and uncommented in the

Save and close the Now start the kafka server on each computer:

> bin/ config/

At this point, the kafka cluster is set up and running. We can test the cluster by creating a topic named "v-topic":

> bin/ --create --zookeeper --partitions 2 --replication-factor 1 --topic v-topic

Now run the following commands to list the topics in the kafka brokers:

> bin/ --zookeeper --list

Now run the following commands to get a description how the topic "v-topic" is partitioned in each broker:

> bin/ --describe --zookeeper --topic v-topic

To test the producer and consumer interaction, let's start a consoler producer on the computer by running the following command on that computer's terminal:

> bin/ --broker-list, --topic v-topic

Now open a terminal of the other computer and start a console consumer:

> bin/ --zookeeper --topic v-topic --from-beginning

Begin to type something in the console producer on terminal and press ENTER, you will see the output displayed in the console consumer on terminal.


It is also ok to set up multiple Kafka brokers on the same computer. For example, if we want to have two Kafka brokers running at two different ports on computer, say:

Now all that we need to do is to duplicate the after it is updated, and rename it in the same "config" folder (note that name is not important, can be anything that make sense). Now in the, modify to have the following settings:

Save and close (remember to create the folder /var/kafka1-logs with write permission), open two terminal in and run the following command in the first terminal to start a kafka broker at port 9092:

> $KAFKA_HOME/bin/ $KAFKA_HOME/config/

On the second terminal, run the following command to start a second kafka broker at port 9093:

> $KAFKA_HOME/bin/ $KAFKA_HOME/config/

Now you will have two kafka brokers running on on two different ports. To include the second broker for the console producer, change its start command to:

> $KAKFA_HOME/bin/ --broker-list,, --topic v-topic