Apache Kafka creating a topic

posted on Nov 20th, 2016

Apache Kafka

Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. It is, in its essence, a "massively scalable pub/sub message queue architected as a distributed transaction log," making it highly valuable for enterprise infrastructures to process streaming data.

Pre Requirements

1) A machine with Ubuntu 14.04 LTS operating system

2) Apache Kakfa 2.9.2-0.8.1.1 software installed. (How to install kafka on ubuntu)

Creating a Kafka Topic

Kafka provides a command line utility named "kafka-topics.sh" to create topics on the server.

Steps to follow

Step 1 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 2 - Start the zookeeper

$ ./bin/zookeeper-server-start.sh config/zookeeper.properties

Step 3 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 4 - Start the kafka server

$ ./bin/kafka-server-start.sh config/server.properties

Step 5 - Open a new terminal (CTRL + ALT + T) and just type 'jps' (without quotes). It shows all the java processes that are running.

Apache Kafka Creating a Topic

Here Kafka is the kafka server and QuorumPeerMain is zookeeper process.

Step 6 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 7 - Create a topic.

$ ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Hello-Kafka

Step 8 - List topics. (optional)

$ ./bin/kafka-topics.sh --list --zookeeper localhost:2181

Step 9 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 10 - Producer

$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Hello-Kafka

Step 11 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 12 - Consumer

$ ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic Hello-Kafka --from-beginning

Step 13 - Stop zookeeper, kafka server, producer and consumer. Just press 'CTRL + D' in the terminals of zookeeper, kafka server, producer and consumer.

Please share this blog post and follow me for latest updates on

facebook             google+             twitter             feedburner

Previous Post                                                                                          Next Post

Labels : Apache Kafka Installation on Ubuntu   Kafka Single Node Single Broker Configuration   Kafka Single Node Multiple Brokers Configuration   Kafka Java API usage   Modifying a Kafka topic   Deleting a Kafka topic