关于kubernetes:kubernetes-ingressnginx-controller的指标获取方法

ingress-nginx controller 组件裸露了 /metrics 接口，prometheus 能够拉取它的指标。

本文重点关注其外部指标的获取办法。

一. 创立 ingress

1. 创立 deploy

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-deploy
spec:
  replicas: 2    ##2 正本
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.14
        imagePullPolicy: IfNotPresent
        name: nginx
      restartPolicy: Always

2. 创立 service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-svc
  name: nginx-svc
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: ClusterIP

3. 创立 ingress

apiVersion: networking.k8s.io/v1 
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
  name: nginx-ingress
spec:
  rules:
  - host: example.com
    http:
      paths: 
      - backend:
          service:
            name: nginx-svc
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific

拜访 http://example.com:20004/，controller 监听在 hostNetwork 的 20004 端口

4. 查看指标

二. 指标获取办法

重点关注以下指标：

• nginx_ingress_controller_request_size: ingress 申请 size
• nginx_ingress_controller_reseponse_size: ingress 响应 size

1. 指标定义

// ingress-nginx/internal/ingress/metric/collectors/socket.go
func NewSocketCollector(pod, namespace, class string, metricsPerHost bool) (*SocketCollector, error) {
    ...
    sc := &SocketCollector{
        ...
        responseLength: prometheus.NewHistogramVec(
            prometheus.HistogramOpts{
                Name:        "response_size",
                Help:        "The response length (including request line, header, and request body)",
                Namespace:   PrometheusNamespace,
                ConstLabels: constLabels,
            },
            requestTags,
        ),
        requestLength: prometheus.NewHistogramVec(
            prometheus.HistogramOpts{
                Name:        "request_size",
                Help:        "The request length (including request line, header, and request body)",
                Namespace:   PrometheusNamespace,
                Buckets:     prometheus.LinearBuckets(10, 10, 10), // 10 buckets, each 10 bytes wide.
                ConstLabels: constLabels,
            },
            requestTags,
        ),
        ...
    }
    ...
}

能够看到，这两个指标均是 Histogram 直方图类型。

2. 指标赋值

那这两个指标的值如何被赋值呢？

// ingress-nginx/internal/ingress/metric/collectors/socket.go
func (sc *SocketCollector) Start() {
    for {conn, err := sc.listener.Accept()
        if err != nil {continue}
        go handleMessages(conn, sc.handleMessage)
    }
}

func (sc *SocketCollector) handleMessage(msg []byte) {
    ...
    var statsBatch []socketData
    err := jsoniter.ConfigCompatibleWithStandardLibrary.Unmarshal(msg, &statsBatch)
    for _, stats := range statsBatch {
        ...
        if stats.RequestLength != -1 {requestLengthMetric, err := sc.requestLength.GetMetricWith(requestLabels)
            if err != nil {klog.Errorf("Error fetching request length metric: %v", err)
            } else {requestLengthMetric.Observe(stats.RequestLength)    // request_size
            }
        }
        if stats.ResponseLength != -1 {bytesSentMetric, err := sc.bytesSent.GetMetricWith(requestLabels)
            if err != nil {klog.Errorf("Error fetching bytes sent metric: %v", err)
            } else {bytesSentMetric.Observe(stats.ResponseLength)    // sent_bytes
            }

            responseSizeMetric, err := sc.responseLength.GetMetricWith(requestLabels)
            if err != nil {klog.Errorf("Error fetching bytes sent metric: %v", err)
            } else {responseSizeMetric.Observe(stats.ResponseLength)    // response_size
            }
        }
        ...
    }
}

从下面能够看出：

• 指标值从 conn 网络连接上读取而来，读到的数据是 []socketData 类型；
• request_size 的值来源于 socketData.RequestLength；
• respoonse_size 的值来源于 socketData.ResponseLength;
• sent_bytes 指标的值，跟 response_size 一样，均来源于 socketData.ReponseLength;

3. conn 连贯是哪里的数据

下面讲到，数据源是从 conn 网络连接上读到的 []socketData，那么 conn 是连贯的谁呢？

// ingress-nginx/internal/ingress/metric/collectors/socket.go
func NewSocketCollector(pod, namespace, class string, metricsPerHost bool) (*SocketCollector, error) {
    socket := "/tmp/prometheus-nginx.socket"
    // unix sockets must be unlink()ed before being used
    _ = syscall.Unlink(socket)

    listener, err := net.Listen("unix", socket)
    ...
}

从下面能够看到：

• listener 监听的 unix socket 文件：/tmp/prometheus-nginx.socket；
• conn 也就是跟与 socket 的连贯，数据也是通过 socket 连贯读取到的；

那么谁会向这个 socket 写入数据呢？

4.ingress-nginx POD

在 ingress-nginx 的代码中没有找到写 socket(/tmp/prometheus-nginx.socket) 的中央，到 ingress-nginx 的 POD 内看看。

# kubectl exec -it -n ingress-nginx ingress-nginx-controller-j6mmt — sh
/etc/nginx $ grep -r "prometheus-nginx.socket" ./
./lua/monitor.lua:  assert(s:connect("unix:/tmp/prometheus-nginx.socket"))
./lua/test/monitor_test.lua:      assert.stub(tcp_mock.connect).was_called_with(tcp_mock, "unix:/tmp/prometheus-nginx.socket")

能够看到，nginx 中有个 monitor 的 lua 插件，它会写 socket: /tmp/prometheus-nginx.socket，监控指标应该就是从这里来的。

/etc/nginx $ vi ./lua/monitor.lua
…
local function send(payload)
  local s = assert(socket())
  assert(s:connect("unix:/tmp/prometheus-nginx.socket"))
  assert(s:send(payload))
  assert(s:close())
end

local function metrics()
  return {
    ....
    requestLength = tonumber(ngx.var.request_length) or -1,
    responseLength = tonumber(ngx.var.bytes_sent) or -1,
    ...
  }
...

能够看到：

• monitor.lua:

  • 将 nginx 的 request_length 和 bytes_sent 变量赋值给 requestLength 和 responseLength；
  • 将数据组装成 []socketData 发送到 soket；
• ingress-nginx：从 socket 上读取 []socketData，而后给 request_size、response_size 指标赋值；

三. 总结

controller 监听到 Ingress 对象被创立，将其配置 reload 到 nignx 上，而后应用 lua 脚本监控其读写流量，通过 socket 写指标写入 controller，最初通过 /metrics 接口裸露进来：