企业容器化部署成为趋势,基于k8s的容器编排成为中大企业部署服务的青眼。本文简略介绍k8s下flume程序的部署,flume个别用户大数据中数据的迁徙以及数据接入。

根底镜像构建

根底镜像构建次要包含:
1.apache-flume包
2.mongoclients包:用于flume连mongo数据库
3.lib包:程序依赖
4.run.sh:程序启动脚本文件

根底镜像Dockerfile:

FROM openjdk:11.0.12-jre# 筹备环境ENV LANGUAGE=zh_CN.UTF-8ENV TZ=Asia/ShanghaiRUN mkdir /apps/WORKDIR /apps/# 增加文件ADD apache-flume-1.9.0-bin /apps/apache-flume-1.9.0-bin# 将 mongo-java-driver-2.6.5.jar、mongo-java-driver-3.12.7.jarADD mongoclients  /apps/mongoclients# 将应用程序依赖拷贝到lib下ADD lib  /lib/# 将启动命令拷贝到apps上面ADD run.sh /apps/# 启动服务ENV AGENT_NAME=a1ENTRYPOINT ["/bin/sh","-c","bash run.sh"]

run.sh启动脚本:
启动脚本中会传递MONGO变量,用于指代咱们mongo版本

#!/bin/bashif [ $MONGO -eq 3 ]; then    ln -sf /apps/mongoclients/mongo-java-driver-3.12.7.jar /lib/elif [ $MONGO -eq 2 ]; then    ln -sf /apps/mongoclients/mongo-java-driver-3.12.7.jar /lib/fi./apache-flume-1.9.0-bin/bin/flume-ng agent \  -Xms2000m -Xmx4000m \  --conf ./apache-flume-1.9.0-bin/conf \  --conf-file ./apache-flume-1.9.0-bin/conf/flume.conf \  -n ${AGENT_NAME:-a1}

工程镜像

咱们的业务代码是依靠于根底镜像框架,所以咱们须要将咱们的工程代码打包成一个依赖拷贝到flume应用程序框架中去。
咱们将Dockerfile寄存在咱们的业务工程代码块中,而后通过mvn编译之后便可实现依赖打包镜像

Dockerfile:
xxx-1.0-SNAPSHOT.jar指代咱们的业务代码。

FROM harbor.dc.prod:8000/base/flume:1.0ADD target/xxx-1.0-SNAPSHOT.jar /lib/

构建镜像:

docker build -t harbor.dc.prod:8000/flume-access:1.0.0 .docker push harbor.dc.prod:8000/flume-access:1.0.0

docker容器下服务启动

配置文件:

a1.channels = c1a1.sources = s1a1.sinks = k1a1.sources.s1.type = 业务工程类型a1.sources.s1.access.master.url = http://data-master:30001/sourcea1.sources.s1.access.heartbeat.interval = 10a1.sources.s1.access.supplier.sourceType = 业务渠道a1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 1000a1.channels.c1.byteCapacityBufferPercentage = 20a1.channels.c1.byteCapacity = 80000000a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.kafka.topic = kafka主题a1.sinks.k1.kafka.bootstrap.servers = kafka1:9092,kafka2:9092,kafka3:9092,kafka4:9092,kafka5:9092a1.sinks.k1.kafka.producer.acks = 1a1.sinks.k1.kafka.message.max.bytes = 102400000a1.sinks.k1.kafka.consumer.max.partition.fetch.bytes = 102400000a1.sinks.k1.kafka.producer.max.request.size = 102400000a1.sinks.k1.flumeBatchSize = 20a1.sinks.k1.channel = c1a1.sources.s1.channels = c1

启动脚本:docker-run.sh

#!/bin/shversion=$2echo $versionNAME=flume-testIMAGE="harbor.dc.prod:8000/flume-access:$version"MONGO=3SELF=`readlink -f $0`D=`dirname $SELF`if [ ! -d $D/logs ]; then    mkdir -p $D/logsfidocker run -d --restart=always \  -v $D/$NAME.conf:/apps/apache-flume-1.9.0-bin/conf/flume.conf \  -v $D/logs:/apps/logs \  -e MONGO=$MONGO \  --name "flume-$NAME" "$IMAGE"

而后咱们执行:

docker-run.sh -v 1.0.0

k8s下启动

1.创立字典

咱们去k8s治理平台下的名称空间->配置核心->配置字段:创立ConfigMap

2.创立负载

去工作负载下：创立部署
根本信息:

容器信息：

存储挂载:

高级设置:

服务路由: