关于数据库:kafka的认识安装与配置

55次阅读

共计 24687 个字符，预计需要花费 62 分钟才能阅读完成。

意识 Kafka

破费越少的精力在数据挪动上，就能越专一于外围业务 —《Kafka：The Definitive Guide》

意识 Kafka 之前，先理解一下 公布与订阅音讯零碎：音讯的发送者不会间接把音讯发送给接收者、发送者以某种形式对音讯进行分类，接收者订阅它们，以便能承受特定类型的音讯。公布与订阅零碎个别会有一个 broker(n. 经纪人、中间商) 也就是公布音讯的中心点。

Kafka 是一款基于公布与订阅的音讯零碎，个别被称为“分布式提交日志”或者“分布式流平台”。Kafka 的数据单元被称作 音讯，能够看作是数据库中的一行数据，音讯是由字节数组组成，故对 kafka 来说音讯没有特地的意义，音讯能够有一个可选的元数据，也就是键。键也是一个字节数组，同样对于 kafka 没有什么非凡意义。键能够用来将音讯以一种可控的形式写入分区。最简略的例子就是为键生成一个一致性散列值，而后应用散列值对主题分区数进行取模，为音讯抉择分区。这样能够保障具备雷同键的音讯总是被写在雷同的分区上。保障音讯在一个主题中程序读取。

为了提高效率，音讯将被分批次写入 Kafka。批次 就是一组音讯，相似于 redis 中的流水线 (Pipelined) 操作。

主题和分区

kafka 的音讯通过主题进行分类，主题就相当于数据库中的表，主题能够被分成若干个分区，一个分区就是一个提交日志，音讯以追加的模式被写入分区。而后依照先入先出的程序读取。一个主题下的分区也能够在不同的服务器上，以此提供比单个服务器更加弱小的性能

生产者和消费者

Kafka 的客户端就是 Kafka 零碎的用户，个别状况下有两种根本类型：生产者和消费者

Producer 生产者 创立音讯，个别状况下，一个音讯会被公布到一个特定的主题上。生产者在默认状况下将音讯均分在主题的每个分区上

Consumer 消费者 读取音讯，消费者订阅一个或多个主题，并依照音讯的生成程序读取他们，消费者通过查看音讯的偏移量来辨别曾经读过的音讯。这个偏移量会被消费者在 zk 或者 kafka 上保留，如果消费者敞开或者重启，他的读取状态不会隐没

消费者是 消费者群组 Consumer group的一部分，群组能够保障每个分区被一个消费者生产(因而消费者数量不能大于分区数量，会造成消费者服务器的节约)，如果一个消费者生效，群组里的其余消费者能够接管生效消费者的工作。

Kafka 的长处

  1. 无缝反对多个生产者
  2. 反对多个消费者从一条音讯流读取数据、且各个消费者之间的偏移量不影响。也反对多个消费者共享一个音讯流，并保障整个消费者群组对每个音讯只生产一次
  3. 能够对每个主题设置保留规定，依据保留规定长久化数据到磁盘
  4. 高性能，高伸缩性

装置

Kafka 应用 Zookeeper(前面简称 zk) 保留集群的元数据信息和消费者信息，Kafka 发行版本自带 zk，能够间接从脚本启动，不过装置一个完整版的 zk 也不难

装置单节点 zk

官网下载地址：http://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/

如果下载速度不如意，能够应用我的蓝奏云：https://keats.lanzous.com/iMWi8dpi04f 提取码：keats

装置目录：/usr/local/zookeeper

数据目录：/var/lib/zookeeper

# tar -zxf zookeeper-3.4.6.tar.gz
# mv zookeeper-3.4.6 /usr/local/zookeeper
# mkdir -p /var/lib/zookeeper
# cat > /usr/local/zookeeper/conf/zoo.cfg << EOF
> tickTime=2000
> dataDir=/var/lib/zookeeper
> clientPort=2181
> EOF
# 接着设置一下环境变量中的 JAVA_HOME, 能够先应用 export 命令查看是否曾经设置
# export JAVA_HOME=/xxx
# 最初切换到 zk 装置目录，启动 zk
# /usr/local/zookeeper/bin/zkServer.sh start

接着通过四字命令 srvr 验证 zk 是否装置正确

# telnet localhost 2181
Trying ::1...
Connected to localhost.
Escape character is '^]'.
srvr
Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 GMT
Latency min/avg/max: 0/0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x0
Mode: standalone
Node count: 4
Connection closed by foreign host.
[root@linux-keats bin]# pwd
/usr/local/zookeeper/bin

装置单节点 Kafka

下载：https://archive.apache.org/dist/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgz

蓝奏云：下载后将后缀名 zip 改为 tgz：https://keats.lanzous.com/iaZ9hdpj5bi

# tar -zxf kafka_2.11-0.9.0.1.tgz
# mv kafka_2.11-0.9.0.1 /usr/local/kafka
# mkdir /tmp/kafka-logs
# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties

其中 -dadmon 示意 kafka 以守护线程的模式启动
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qrpin3u9/d…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/lty6dp09/i…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/qvbuqqvw/n…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/lty6dp09/i…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qrpin3u9/d…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/qqzul2a6/y…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qrpin3u9/d…
https://github.com/lty6dp09/i…
https://github.com/sz7epigs/v…
https://github.com/jdzce2g3/c…
https://github.com/qqzul2a6/y…
https://github.com/qrpin3u9/d…
https://github.com/sz7epigs/v…
https://github.com/qvbuqqvw/n…
https://github.com/lty6dp09/i…
https://github.com/jdzce2g3/c…
https://github.com/qvbuqqvw/n…
https://github.com/qqzul2a6/y…

配置 kafka

#broker 的全局惟一编号，集群中不能反复。int 类型
broker.id=0
#是否容许删除 topic 
delete.topic.enable=true
#解决网络申请的线程数量
num.network.threads=3
#解决磁盘 IO 的线程数量
num.io.threads=8
#发送套接字的缓冲区大小
socket.send.buffer.bytes=102400
#接管套接字的缓冲区大小
socket.receive.buffer.bytes=102400
#申请套接字的缓冲区大小
socket.request.max.bytes=104857600
#kafka 运行日志 (此日志非常规意义的日志) 寄存的门路。用上一步创立的目录。log.dirs=/tmp/kafka-logs
#topic 创立时默认的分区数
num.partitions=1
#用来复原和清理 data 下数据的线程数量
num.recovery.threads.per.data.dir=1
#segment 文件保留的最长工夫，超时将被删除
log.retention.hours=168
#配置连贯 Zookeeper 地址。如果是 zk 集群，应用 , 隔开
zookeeper.connect=localhost:2181

集群

zk 集群的装置请度娘 zk 集群，kafka 能够依照开端参考文献装置集群。我这里测试服务器性能不行还跑了几个 java 程序，就不装集群了

测试

主题相干操作

 /usr/local/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
  • –create 创立操作 还有 –list 查问，–describe 详情
  • –zookeeper localhost:2181 配置 zk 的信息
  • –partitions 1 分区数目 1
  • –replication-factor 1 正本数 1。正本数不能大于 kafka broker 节点的数目
  • –topic test 指定主题名称

创立好主题后，logs 文件夹内就会呈现 主题名 - 分区名 的提交日志

往主题发送音讯

# /usr/local/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
Test Message 1
Test Message 2
^D

从测试主题读取音讯

# /usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
Test Message 1
Test Message 2
^C
Processed a total of 2 messages

内外网拜访 kafka

在 kafka 的 server.properties 中，上面的配置默认是 PLAINTEXT://:9092，示意默认状况下 kafka 只监听内网 9092 端口的申请。如果服务器须要外网拜访，须要更改配置：

# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
# if you don't understand there see: https://blog.csdn.net/gk91620/article/details/103985844
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
listeners=INTERNAL:// 内网 IP: 内网端口 1,EXTERNAL:// 内网 IP: 内网端口 2
advertised.listeners=INTERNAL:// 内网 IP: 内网端口 1,EXTERNAL:// 外网 IP: 外网拜访的端口
inter.broker.listener.name=INTERNAL

这么配置下来，对于外网的生产者和消费者，通过 外网 IP: 外网拜访的端口 来与 kafka 交互，而 内网 IP: 内网端口 1 则被用于集群中 kafka 之间的交互

正文完
数据库
发表至： 数据库
2021-01-05
 0
关于数据库:解读华为云原生数据库设计原则打破传统数据库上云瓶颈
关于数据库:亿级流量背后战场京东1111大促全方位技术揭秘
关于数据库:2022年8月中国数据库排行榜openGauss重夺榜眼PolarDB反超人大金仓
关于数据库:1024程序员节专访聚焦行业前沿共话IT发展趋势