To setup Kafka in a cluster, first we must have the zookeeper cluster setup and running (follow this link:
http://czcodezone.blogspot.sg/2014/11/setup-zookeeper-in-cluster.html), suppose that the zookeeper cluster consists of the zookeeper servers running at the following hostname:ports:
192.168.2.2:2181
192.168.2.4:2181
As I have only two computers, therefore i will use the same computers (but at different ports) to host the kafka cluster. For this case, the Kafka servers/brokers will be running at the following hostname:ports
192.168.2.2:9092
192.168.2.4:9092
To do this, follow this link (
http://czcodezone.blogspot.sg/2014/11/setup-kafka-in-single-machine-running.html) to setup the kafka server. Now navigate to the kafka root folder oif each computer and modify the server.properties in "config" sub folder:
> cd $KAFKA_HOME
> gedit config/server.properties
In the server.properties file, search the line "zookeeper.connect" and change it to:
zookeeper.connect=192.168.2.2:2181,192.168.2.4:2181
Then search the line "broker.id" (unique id for each broker node) and change it to "broke.id=1" on computer 192.168.2.2 and to "broker.id=2" on computer 192.168.2.4
Next search the line "host.name" and change it to "host.name=192.168.2.2" on computer 192.168.2.2 and to "host.name=192.168.2.4" on computer 192.168.2.4
Make sure that the line "port=9092" is there and uncommented in the server.properties
Save and close the server.properties. Now start the kafka server on each computer:
> cd $KAFKA_HOME
> bin/kafka-server-start.sh config/server.properties
At this point, the kafka cluster is set up and running. We can test the cluster by creating a topic named "v-topic":
> bin/kakfa-topics.sh --create --zookeeper 192.168.2.4:2181 --partitions 2 --replication-factor 1 --topic v-topic
Now run the following commands to list the topics in the kafka brokers:
> bin/kafka-topics.sh --zookeeper 192.168.2.4:2181 --list
Now run the following commands to get a description how the topic "v-topic" is partitioned in each broker:
> bin/kafka-topics.sh --describe --zookeeper 192.168.2.4:2181 --topic v-topic
To test the producer and consumer interaction, let's start a consoler producer on the computer 192.168.2.4 by running the following command on that computer's terminal:
> cd $KAFKA_HOME
> bin/kafka-console-producer.sh --broker-list 192.168.2.2:9092,192.168.2.4:9092 --topic v-topic
Now open a terminal of the other computer 192.168.2.4 and start a console consumer:
> cd $KAFKA_HOME
> bin/kafka-console-consumer.sh --zookeeper 192.168.2.4:2181 --topic v-topic --from-beginning
Begin to type something in the console producer on 192.168.2.2 terminal and press ENTER, you will see the output displayed in the console consumer on 192.168.2.4 terminal.
Note:
It is also ok to set up multiple Kafka brokers on the same computer. For example, if we want to have two Kafka brokers running at two different ports on computer 192.168.2.2, say:
192.168.2.2:9092
192.168.2.2:9093
Now all that we need to do is to duplicate the server.properties after it is updated, and rename it server1.properties in the same "config" folder (note that name is not important, can be anything that make sense). Now in the server1.properties, modify to have the following settings:
broker.id=3
log.dirs=/var/kafka1-logs
port=9093
Save and close server1.properties (remember to create the folder /var/kafka1-logs with write permission), open two terminal in 192.168.2.2 and run the following command in the first terminal to start a kafka broker at port 9092:
> $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
On the second terminal, run the following command to start a second kafka broker at port 9093:
> $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server1.properties
Now you will have two kafka brokers running on 192.168.2.2 on two different ports. To include the second broker for the console producer, change its start command to:
> $KAKFA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.2.2:9092,192.168.2.2:9093,192.168.2.4:9092 --topic v-topic
