前言
之前我们介绍过 fluent bit 这个日志收集神器。最近我们遇到奇葩的需求，不得不利用 lua 编写 fluent bit 的 filter，来满足需求。
首先介绍一下需求：非容器的日志团队使用 filebeat, 其配置文件部分如下：
processors:
– dissect:
tokenizer: “/data/logs/%{appname}/%{filename}.log”
field: “source”
target_prefix: “”
即需要从日志 record 的 source filed 提取 appname 和 filename 两个 filed。
fluent bit 并没有如此的插件，所以不得不自己实现。
实现
lua 编写 filter 规范
官方给出的示例如下：
function cb_print(tag, timestamp, record)
return code, timestamp, record
end
Function 输入参数
Function Arguments
name
description
tag
Name of the tag associated with the incoming record.
timestamp
Unix timestamp with nanoseconds associated with the incoming record. The original format is a double (seconds.nanoseconds)
record
Lua table with the record content
Return Values
Each callback must return three values:
name
data type
description
code
integer
The code return value represents the result and further action that may follows. If code equals -1, means that filter_lua must drop the record. If _code_ equals 0 the record will not be modified, otherwise if code equals 1, means the original timestamp or record have been modified so it must be replaced by the returned values from timestamp (second return value) and record (third return value).
timestamp
double
If code equals 1, the original record timestamp will be replaced with this new value.
record
table
if code equals 1, the original record information will be replaced with this new value. Note that the format of this value must be a valid Lua table.
理解上面的规范可以结合下面的写法。注意返回值的 code。
代码实现
编写实现类似功能的 lua 文件，如下：
function dissect(tag, timestamp, record)
source = record[“source”]
if (source == nil)
then
return 0, 0, 0
else
new_record = record
local result = {}
local from = 1
local delim_from, delim_to = string.find(source, “/”, from)
while delim_from do
table.insert(result, string.sub( source, from , delim_from-1) )
from = delim_to + 1
delim_from, delim_to = string.find(source, “/”, from)
end
table.insert(result, string.sub( source, from) )
new_record[“appname”] = result[7]
new_record[“filename”] = string.sub(result[8], 1, -5 )
return 1, timestamp, new_record
end
end
备注：
在我们 k8s 环境下，业务日志挂盘路径类似于下面的格式：source = /data/logs/default/tomcat/742473c7-17dc-11e9-afc5-0a07a5c4fbe2/appname/filename.log
result[7] 之所以出现这种及其容易出现 bug 的写法，一是由于我们这边有严格的日志规范，另外，只是给大家提供一种 lua 写 filter 的思路。
制作镜像
我们是基于 fluent bit 1.0.2。所以找到官方的代码仓库，git clone 下来，稍作更改。新的 dockerfile 如下：
FROM debian:stretch as builder
# Fluent Bit version
ENV FLB_MAJOR 1
ENV FLB_MINOR 0
ENV FLB_PATCH 2
ENV FLB_VERSION 1.0.2
ENV DEBIAN_FRONTEND noninteractive
ENV FLB_TARBALL http://github.com/fluent/fluent-bit/archive/v$FLB_VERSION.zip
RUN mkdir -p /fluent-bit/bin /fluent-bit/etc /fluent-bit/log /tmp/fluent-bit-master/
RUN apt-get update && \
apt-get install -y –no-install-recommends \
build-essential \
cmake \
make \
wget \
unzip \
libssl1.0-dev \
libasl-dev \
libsasl2-dev \
pkg-config \
libsystemd-dev \
zlib1g-dev \
ca-certificates \
&& wget -O “/tmp/fluent-bit-${FLB_VERSION}.zip” ${FLB_TARBALL} \
&& cd /tmp && unzip “fluent-bit-$FLB_VERSION.zip” \
&& cd “fluent-bit-$FLB_VERSION”/build/ \
&& rm -rf /tmp/fluent-bit-$FLB_VERSION/build/*
WORKDIR /tmp/fluent-bit-$FLB_VERSION/build/
RUN cmake -DFLB_DEBUG=On \
-DFLB_TRACE=Off \
-DFLB_JEMALLOC=On \
-DFLB_TLS=On \
-DFLB_SHARED_LIB=Off \
-DFLB_EXAMPLES=Off \
-DFLB_HTTP_SERVER=On \
-DFLB_IN_SYSTEMD=On \
-DFLB_OUT_KAFKA=On ..
RUN make -j $(getconf _NPROCESSORS_ONLN)
RUN install bin/fluent-bit /fluent-bit/bin/
# Configuration files
COPY fluent-bit.conf \
parsers.conf \
parsers_java.conf \
parsers_extra.conf \
parsers_openstack.conf \
parsers_cinder.conf \
plugins.conf \
/fluent-bit/etc/
COPY dissect.lua /fluent-bit/bin/
FROM gcr.io/distroless/cc
MAINTAINER Eduardo Silva <eduardo@treasure-data.com>
LABEL Description=”Fluent Bit docker image” Vendor=”Fluent Organization” Version=”1.1″
COPY –from=builder /usr/lib/x86_64-linux-gnu/*sasl* /usr/lib/x86_64-linux-gnu/
COPY –from=builder /usr/lib/x86_64-linux-gnu/libz* /usr/lib/x86_64-linux-gnu/
COPY –from=builder /lib/x86_64-linux-gnu/libz* /lib/x86_64-linux-gnu/
COPY –from=builder /usr/lib/x86_64-linux-gnu/libssl.so* /usr/lib/x86_64-linux-gnu/
COPY –from=builder /usr/lib/x86_64-linux-gnu/libcrypto.so* /usr/lib/x86_64-linux-gnu/
# These below are all needed for systemd
COPY –from=builder /lib/x86_64-linux-gnu/libsystemd* /lib/x86_64-linux-gnu/
COPY –from=builder /lib/x86_64-linux-gnu/libselinux.so* /lib/x86_64-linux-gnu/
COPY –from=builder /lib/x86_64-linux-gnu/liblzma.so* /lib/x86_64-linux-gnu/
COPY –from=builder /usr/lib/x86_64-linux-gnu/liblz4.so* /usr/lib/x86_64-linux-gnu/
COPY –from=builder /lib/x86_64-linux-gnu/libgcrypt.so* /lib/x86_64-linux-gnu/
COPY –from=builder /lib/x86_64-linux-gnu/libpcre.so* /lib/x86_64-linux-gnu/
COPY –from=builder /lib/x86_64-linux-gnu/libgpg-error.so* /lib/x86_64-linux-gnu/
COPY –from=builder /fluent-bit /fluent-bit
#
EXPOSE 2020
# Entry point
CMD [“/fluent-bit/bin/fluent-bit”, “-c”, “/fluent-bit/etc/fluent-bit.conf”]
注意增加了 COPY dissect.lua /fluent-bit/bin/。
然后就 build 镜像即可。
使用姿势
使用比较简单的。demo 如下：
[FILTER]
Name lua
Match app.*
script /fluent-bit/bin/dissect.lua
call dissect
script lua 脚本的存放路径。
call 即为 lua 函数名。
总结
通过写这个 filter，有一下几个感悟吧。
官方的镜像基于谷歌的 distroless 镜像，没有 shell，没有包管理，调试起来很费力。平时的业务容器化场景中，明显的不合适，与阿里的富容器思维南辕北辙。当然除非你公司的业务开发能力足够强。
fluent bit 相关的资料还是有点少。遇到问题和使用一些不明白的地方，解决起来费力。除非你是 c 专家。官方文档写的也不够详细，只是描述了个大概。