Apache Kafka Single Node Multiple Brokers

posted on Nov 20th, 2016

Apache Kafka

Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. It is, in its essence, a "massively scalable pub/sub message queue architected as a distributed transaction log," making it highly valuable for enterprise infrastructures to process streaming data.

Pre Requirements

1) A machine with Ubuntu 14.04 LTS operating system

2) Apache Kakfa 2.9.2-0.8.1.1 software installed. (How to install kafka on ubuntu)

Kafka Single Node Multiple Brokers

In this configuration you have a single ZooKeeper and multiple broker id instances. Following are the steps to configure it.

Apache Kafka Single Node Multiple Brokers

Steps to follow

Step 1 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka/config

$ cd /usr/local/kafka/config

Step 2 - Edit server-one.properties file. Automatically it will create a new server-one.properties file if it doesn't exists and opens for editing.

$ gedit server-one.properties

Step 3 - Add these below lines to server-one.properties file. Save and close.

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
# The port the socket server listens on
port=9093
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-1
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=1000000
delete.topic.enable = true

Step 4 - Edit server-two.properties file. Automatically it will create a new server-two.properties file if it doesn't exists and opens for editing.

$ gedit server-two.properties

Step 5 - Add these below lines to server-two.properties file. Save and close.

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2
# The port the socket server listens on
port=9094
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-2
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=1000000
delete.topic.enable = true

Step 6 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 7 - Start the zookeeper

$ ./bin/zookeeper-server-start.sh config/zookeeper.properties

Step 8 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 9 - Start the kafka server

$ ./bin/kafka-server-start.sh config/server.properties

Step 10 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 11 - Start the kafka server-one

$ ./bin/kafka-server-start.sh config/server-one.properties

Step 12 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 13 - Start the kafka server-two

$ ./bin/kafka-server-start.sh config/server-two.properties

Step 14 - Open a new terminal (CTRL + ALT + T) and just type 'jps' (without quotes). It shows all the java processes that are running.

Here Kafka is the kafka server and QuorumPeerMain is zookeeper process.

Step 15 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 16 - Create a topic.

$ ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic Multibrokerapplication

Apache Kafka Single Node Multiple Brokers

Step 17 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 18 - Producer

$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Hello-Kafka

Step 19 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka

$ cd /usr/local/kafka

Step 20 - Consumer

$ ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic Hello-Kafka --from-beginning

Step 21 - Open a new terminal (CTRL + ALT + T) and just type 'jps' (without quotes). It shows all the java processes that are running.

Apache Kafka Single Node Multiple Brokers

Step 22 - Stop zookeeper, kafka server, producer and consumer. Just press 'CTRL + D' in the terminals of zookeeper, kafka server, producer and consumer.

Please share this blog post and follow me for latest updates on

facebook             google+             twitter             feedburner

Previous Post                                                                                          Next Post

Labels : Apache Kafka Installation on Ubuntu   Kafka Single Node Single Broker Configuration   Kafka Java API usage   Creating a Kafka topic   Modifying a Kafka topic   Deleting a kafka topic