Saturday, November 22, 2014

Write and test a simple Kafka producer

First we would need to start a zookeeper cluster

Now create a Maven project in Eclipse or STS (e.g. groupId=com.memeanalytics artifactId=kafka-producer), and change the pom.xml to include the following dependencies and plugins:






The dependency setting include the kafka pom and the plugin setting include the maven exec plugin.

The implementation of a kafka producer in java is very simple a straight forward, we first create a KafkaConfig object which the following initialization:
serializer.class: kafka.serializer.StringEncoder
request.required.acks: 1

The broker list we only need to specify one broker and the rest will be automatically discovered. Since we are to produce string data to the kafka broker, the StringEncoder is used as the data serializer, We also specify that we like to have acknowleges for request sent.

Next is to create a Producer object that is configured by the KafkaConfig object above:

Producer<String, String> kafkaProducer=new Producer<String, String>(config);

The actual sending of data is performed by the following line:

KeyedMessage<String, String> data=new KeyedMessage<String, String>("test-topic", "MyDataTextMessage");

The "test-topic" is the name of the topic and the "MyDataTextMessage" is the actual data sent to kafka brokers. After all the data has been sent, the kafkaProducer needs to be closed:


You can download the complete java project codes from the link below:

To run, navigate the root folder of the Maven project, and run the following command in a terminal:

> mvn compile exec:java

To test, make sure that the kafka cluster is setup and running (follow the instructions given in this link: open another terminal and run a kafka console consumer using the following command:

> $KAFKA_HOME/bin/ --zookeeper --topic test-topic --from-beginning

No comments:

Post a Comment