如何使用kafka增加topic的备份数量，让业务更上一层楼

本文由云+社区发表一、困难点建立topic的时候，可以通过指定参数 –replication-factor 设置备份数量。但是，一旦完成建立topic，则无法通过kafka-topic.sh 或者命令修改replica数量。二、解决办法实际上，我们可以考虑一种 “另类” 的办法：可以利用 kafka-reassign-partitions.sh 命令对所有分区进行重新分布，在做分区重新分布的时候，通过增加每个分区的replica备份数量来达到目的。本文将介绍如何利用 kafka-reassign-partitions.sh 命令增加topic的备份数量。注意：以下命令使用到的topic名称、zookeeper的ip和port，需要读者替换成为实际集群的参数。（假设kafka集群有4个broker，id分别为：1001,1002,1003,1004）2.1、获取当前topic的所有分区分布在broker的情况[root@tbds bin]# ./kafka-topics.sh –zookeeper 172.16.32.13:2181 –topic ranger_audits –describeTopic:ranger_audits PartitionCount:10 ReplicationFactor:1 Configs: Topic: ranger_audits Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 1 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 7 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002 Isr: 1002可以看出，ranger_audits 这个topic有10个分区，每个分区只有一个feplica备份，分布在1001和1002两台broker上面。下面我们需要将ranger_audits 的每个分区数据都增加到2个replica备份，且分布到4个broker上面。2.2、创建增加replica备份数量的配置文件（注意：尽量保持topic的原有每个分区的主备份不变化。因此，配置文件的每个分区的第一个broker保持不变。）[root@tbds bin]# vim ../config/increase-replication-factor.json{“version”:1,“partitions”:[{“topic”:“ranger_audits”,“partition”:0,“replicas”:[1001,1003]},{“topic”:“ranger_audits”,“partition”:1,“replicas”:[1002,1004]},{“topic”:“ranger_audits”,“partition”:2,“replicas”:[1001,1003]},{“topic”:“ranger_audits”,“partition”:3,“replicas”:[1002,1004]},{“topic”:“ranger_audits”,“partition”:4,“replicas”:[1001,1003]},{“topic”:“ranger_audits”,“partition”:5,“replicas”:[1002,1004]},{“topic”:“ranger_audits”,“partition”:6,“replicas”:[1001,1003]},{“topic”:“ranger_audits”,“partition”:7,“replicas”:[1002,1004]},{“topic”:“ranger_audits”,“partition”:8,“replicas”:[1001,1003]},{“topic”:“ranger_audits”,“partition”:9,“replicas”:[1002,1004]}]}上面的配置文件说明，我们将topic的每个分区都增加了一个replica，且保持每个分区原有的主备份所在broker不变化，将每个分区新增的replica备份数据放到到1003和1004两个broker上面。2.3、开始执行增加分区[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 –reassignment-json-file ../config/increase-replication-factor.json –executeCurrent partition replica assignment{“version”:1,“partitions”:[{“topic”:“ranger_audits”,“partition”:3,“replicas”:[1002]},{“topic”:“ranger_audits”,“partition”:9,“replicas”:[1002]},{“topic”:“ranger_audits”,“partition”:8,“replicas”:[1001]},{“topic”:“ranger_audits”,“partition”:1,“replicas”:[1002]},{“topic”:“ranger_audits”,“partition”:4,“replicas”:[1001]},{“topic”:“ranger_audits”,“partition”:2,“replicas”:[1001]},{“topic”:“ranger_audits”,“partition”:5,“replicas”:[1002]},{“topic”:“ranger_audits”,“partition”:0,“replicas”:[1001]},{“topic”:“ranger_audits”,“partition”:6,“replicas”:[1001]},{“topic”:“ranger_audits”,“partition”:7,“replicas”:[1002]}]}Save this to use as the –reassignment-json-file option during rollbackSuccessfully started reassignment of partitions {“version”:1,“partitions”:[{“topic”:“ranger_audits”,“partition”:0,“replicas”:[1001,1003]},{“topic”:“ranger_audits”,“partition”:8,“replicas”:[1001,1003]},{“topic”:“ranger_audits”,“partition”:5,“replicas”:[1002,1004]},{“topic”:“ranger_audits”,“partition”:2,“replicas”:[1001,1003]},{“topic”:“ranger_audits”,“partition”:9,“replicas”:[1002,1004]},{“topic”:“ranger_audits”,“partition”:1,“replicas”:[1002,1004]},{“topic”:“ranger_audits”,“partition”:3,“replicas”:[1002,1004]},{“topic”:“ranger_audits”,“partition”:4,“replicas”:[1001,1003]},{“topic”:“ranger_audits”,“partition”:7,“replicas”:[1002,1004]},{“topic”:“ranger_audits”,“partition”:6,“replicas”:[1001,1003]}]}2.4、查看执行进度[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 –reassignment-json-file ../config/increase-replication-factor.json –verifyStatus of partition reassignment:Reassignment of partition [ranger_audits,0] completed successfullyReassignment of partition [ranger_audits,8] completed successfullyReassignment of partition [ranger_audits,5] completed successfullyReassignment of partition [ranger_audits,2] completed successfullyReassignment of partition [ranger_audits,9] completed successfullyReassignment of partition [ranger_audits,1] completed successfullyReassignment of partition [ranger_audits,3] completed successfullyReassignment of partition [ranger_audits,4] completed successfullyReassignment of partition [ranger_audits,7] completed successfullyReassignment of partition [ranger_audits,6] completed successfully上面显示增加分区操作成功2.5、再次查看topic的情况[root@tbds bin]# ./kafka-topics.sh –zookeeper 172.16.32.13:2181 –topic ranger_audits –describeTopic:ranger_audits PartitionCount:10 ReplicationFactor:2 Configs: Topic: ranger_audits Partition: 0 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 1 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004 Topic: ranger_audits Partition: 2 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004 Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004 Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 7 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004 Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004从上面可以看出，备份数量增加成功三、进一步思考利用上述介绍的办法，除了可以用来增加topic的备份数量之外，还能够处理以下几个场景：1、对topic的所有分区数据进行整体迁移。怎么理解呢？假如集群有N个broker，后来新扩容M个broker。由于新扩容的broker磁盘都是空的，原有的broker磁盘占用都很满。那么我们可以利用上述方法，将存储在原有N个broker的某些topic整体搬迁到新扩容的M个broker，进而实现kafka集群的整体数据均衡。具体使用方法就是：通过编写2.2章节的配置文件，将topic的所有分区都配置到新的M个broker上面去，再执行excute，即可完成topic的所有分区数据整体迁移到新扩容的M个broker节点。*2、broker坏掉的情况。*导致某些topic的某些分区的replica数量减少，可以利用kafka-reassign-partitions.sh增加replica；*3、kafka 某些broker磁盘占用很满，某些磁盘占用又很少。*可以利用kafka-reassign-partitions.sh迁移某些topic的分区数据到磁盘占用少的broker，实现数据均衡；*4、kafka集群扩容。*需要把原来broker的topic数据整体迁移到新的broker，合理利用新扩容的broker，实现负载均衡。此文已由作者授权腾讯云+社区在各渠道发布获取更多新鲜技术干货，可以关注我们腾讯云技术社区-云加社区官方号及知乎机构号