Apache Kafka is an open source, distributed publish-subscribe messaging system,and its main characters include:
-Persistent Messaging
-High throughput
-Distributed: support messages partitioning over Kafka Servers and
distributing consumption over a cluster of consumer machines
while maintaining per-partition ordering machines
Producer(frontend, services, proxies, adapters, other producers)
|
Kafka ———> Zookeeper
|
Consumer(realtime, NoSQL, Hadoop, Warehouses)
Install Kafka
**Download Kafka**
[root@localhost opt] https://www.apache.org/dyn/closer.cgi/incubator/kafka/kafka-0.7.2-incubating/kafka-0.7.2-incubating-src.tgz
**Extract the Kafka downloaded package**
[root@localhost opt]# tar xzf kafka-0.8.0-beta1-src.tgz
Setup Kafka Cluster
ZooKeeper
——————————
| |
| |
Producer –> Kafka Broker –> Consumer
ZookeeperKafka provides the default and simple ZooKeeper configuration file used for lauching a single local ZooKeeper instance. Zookeeper allows distributed processes coordinating with each other through a shared hierarchical name space of data registers
**Start Zookeeper Server**
[root@localhost kafka-0.8]# bin/zookeeper-server-start.sh config/zookeeper.properties
**Data directory where the zookeeper snapshot is stored**
## dataDir=/tmp/zookeeper
**The port listening for client request**
## clientPort=2181
**Consumer group id
## groupid=test-consumer-group
**zookeeper connection string
## zookeeper.connect=localhost:2181
**Start the Kafka broker**
[root@localhost kafka-0.8]# bin/kafka-server-start.sh config/server.properties
**Create a Kafka Topic**
[root@localhost kafka-0.8]# bin/kafka-create-topic.sh –zookeeper localhost:2181 –replica 1 –partition 1 –topic kafkatopic
**Start a producer for sending messages**
[root@localhost kafka-0.8]# bin/kafka-console-producer.sh –broker-list localhost:9092 –topic kafkatopic
**Start a consumer for consuming messages**
[root@localhost kafka-0.8]# bin/kafka-console-consumer.sh –zookeeper localhost:2181 –topic kafkatopic –from-beginning
**Create a Kafka topic**
[root@localhost kafka-0.8]# bin/kafka-create-topic.sh –zookeeper localhost:2181 –replica 2 –partition 2 –topic othertopic
**Start a producer for sending messages**
[root@localhost kafka-0.8]# bin/kafka-console-producer.sh –broker-list localhost:9092,localhost:9093 –topic othertopic
**Start a consumer for consuming messages**
[root@localhost kafka-0.8]# bin/kafka-console-consumer.sh –zookeeper localhost:2181 –topic othertopic –from-beginning