上文讲到serviceMonitor是service监控对象的形象，本文就以kube-proxy为例，剖析如何应用serviceMonitor对象监控kube-proxy。

1. kube-proxy的部署模式

# kubectl get all -A|grep proxykube-system     pod/kube-proxy-bn64j                           1/1     Running   0          30mkube-system     pod/kube-proxy-jcl54                           1/1     Running   0          30mkube-system     pod/kube-proxy-n44bh                           1/1     Running   0          30mkube-system     daemonset.apps/kube-proxy                 3         3         3       3            3           kubernetes.io/os=linux   217d

能够看到，kube-proxy应用daemonset部署，但没有service，部署了3个Pod。

2. 减少kube-proxy的/metrics拜访端口

kube-proxy的Pod内含1个container，并且其配置文件中，metrics绑定的ip为127.0.0.1：

# kubectl edit ds kube-proxy -n kube-system......    spec:      containers:      - command:        - /usr/local/bin/kube-proxy        - --config=/var/lib/kube-proxy/config.conf        - --hostname-override=$(NODE_NAME)        env:        - name: NODE_NAME          valueFrom:            fieldRef:              apiVersion: v1              fieldPath: spec.nodeName        image: 178.104.162.39:443/dev/kubernetes/amd64/kube-proxy:v1.18.0        imagePullPolicy: IfNotPresent        name: kube-proxy

查看其配置文件/var/lib/kube-proxy/config.conf:

# kubectl exec -it kube-proxy-4vxsf /bin/sh -n kube-system# cat /var/lib/kube-proxy/config.confbindAddress: 0.0.0.0......healthzBindAddress: 0.0.0.0:10256......metricsBindAddress: 127.0.0.1:10249

能够看到，其绑定的metrics地址：127.0.0.1:10249；
要想里面能够拜访/metrics，须要将该端口转发进去，这里应用sidecar：减少1个kube-rbac-proxy container的形式，将proxy container的metrics端口转发进去：

# kubectl edit ds kube-proxy -n kube-system#在containers列表中减少- args:  - --logtostderr  - --secure-listen-address=[$(IP)]:10249  - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305  - --upstream=http://127.0.0.1:10249/  env:  - name: IP    valueFrom:      fieldRef:        apiVersion: v1        fieldPath: status.podIP  image: 178.104.162.39:443/dev/kubernetes/amd64/kube-rbac-proxy:v0.4.1  imagePullPolicy: IfNotPresent  name: kube-rbac-proxy  ports:  - containerPort: 10249    hostPort: 10249    name: https    protocol: TCP  resources:    limits:      cpu: 20m      memory: 40Mi    requests:      cpu: 10m      memory: 20Mi  terminationMessagePath: /dev/termination-log  terminationMessagePolicy: File

在daemonset的最初，还指定了serviceAccount:

serviceAccount: kube-proxyserviceAccountName: kube-proxy

daemonset批改结束后，验证10249端口是否监听：

# netstat -nalp|grep 10249|grep LISTENtcp        0      0 178.104.163.38:10249    0.0.0.0:*               LISTEN      16930/./kube-rbac-ptcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      16735/kube-proxy

3. 创立kube-proxy的service和serviceMonitor

kube-proxy没有service，须要在service的根底上，创立serviceMonitor；

kube-proxy-service.yaml定义了name=kube-proxy的service:

  • 筛选Pod: 含label, k8s-app=kube-proxy；
  • 给本人加lable: k8s-app=kube-proxy；（serviceMonitor会用）
# cat kube-proxy-service.yamlapiVersion: v1kind: Servicemetadata:  labels:    app.kubernetes.io/name: kube-proxy    app.kubernetes.io/version: v0.18.1    k8s-app: kube-proxy  name: kube-proxy  namespace: kube-systemspec:  clusterIP: None  ports:  - name: https    port: 10249    targetPort: https  selector:    k8s-app: kube-proxy

kube-proxy-serviceMonitor.yaml，它定义了name=kube-proxy的serviceMonitor：

  • 筛选service中label: k8s-app=kube-proxy的service；
# cat kube-proxy-serviceMonitor.yamlapiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  name: kube-proxy  namespace: monitoring  labels:    k8s-app: kube-proxyspec:  jobLabel: kube-proxy  endpoints:  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token    interval: 15s    port: https    relabelings:    - action: replace      regex: (.*)      replacement: $1      sourceLabels:      - __meta_kubernetes_pod_node_name      targetLabel: instance    scheme: https    tlsConfig:      insecureSkipVerify: true  selector:    matchLabels:      k8s-app: kube-proxy  namespaceSelector:    matchNames:    - kube-system

serviceMonitor定义结束，会在prometheus的dashboard看到kube-proxy的target:

同时，在prometheus-server的配置文件中，也对应减少了kube-proxy的服务发现配置：

- job_name: monitoring/kube-proxy/0  honor_labels: false  kubernetes_sd_configs:  - role: endpoints    namespaces:      names:      - kube-system  scrape_interval: 15s  scheme: https  tls_config:    insecure_skip_verify: true  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token  relabel_configs:  - action: keep        ##筛选label：k8s_app=kube-proxy的service    source_labels:    - __meta_kubernetes_service_label_k8s_app    regex: kube-proxy  - action: keep        ##筛选endpoint_port_name=https的service    source_labels:    - __meta_kubernetes_endpoint_port_name    regex: https  - source_labels:    - __meta_kubernetes_endpoint_address_target_kind    - __meta_kubernetes_endpoint_address_target_name    separator: ;    regex: Node;(.*)    replacement: ${1}    target_label: node  - source_labels:    - __meta_kubernetes_endpoint_address_target_kind    - __meta_kubernetes_endpoint_address_target_name    separator: ;    regex: Pod;(.*)    replacement: ${1}    target_label: pod .....

下面的配置次要有2个筛选项：

  • 筛选label: k8s-app=kube-proxy的service;
  • 筛选endpoint_port_name=https的service；

这跟kube-proxy的service定义统一。

4. 减少kube-proxy的rbac配置

daemonset中应用的serviceAccount: kube-proxy，须要给该sa减少clusterRole和clusterRoleBinding，否则scrape /metrics时会报401 Unauthorize；

# cat kube-proxy-clusterRole.yamlapiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  name: kube-proxyrules:- apiGroups:  - authentication.k8s.io  resources:  - tokenreviews  verbs:  - create- apiGroups:  - authorization.k8s.io  resources:  - subjectaccessreviews  verbs:  - create#kubectl apply -f kube-proxy-clusterRole.yaml
# cat kube-proxy-clusterRoleBinding.yamlapiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  name: kube-proxyroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: kube-proxysubjects:- kind: ServiceAccount  name: kube-proxy  namespace: kube-system#kubectl  apply  -f  kube-proxy-clusterRoleBinding.yaml

5. 集群内curl /metrics查看指标

# curl --header "Authorization: Bearer $TOKEN" --insecure https://178.104.163.38:10249/metrics# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.# TYPE apiserver_audit_event_total counterapiserver_audit_event_total 0# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.# TYPE apiserver_audit_requests_rejected_total counterapiserver_audit_requests_rejected_total 0# HELP go_gc_duration_seconds A summary of the GC invocation durations.# TYPE go_gc_duration_seconds summary......