关于prometheus:prometheusoperator使用五-自定义podservice自动发现配置

指标：
用户启动的 service 或 pod，在 annotation 中增加 label 后，能够主动被 prometheus 发现：

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "9121"

1. secret 保留主动发现的配置

若要特定的 annotation 被发现，须要为 prometheus 减少如下配置：

- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name

上述配置会筛选 endpoints：prometheus.io/scrape=True

将上述配置保留为 secret:

$ kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring
secret "additional-configs" created

2. 将配置增加到 prometheus 实例

批改 prometheus CRD，将下面的 secret 增加进去：

# vi /etc/kubernetes/prometheus/prometheus-prometheus.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  ......
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.5.0

# kubectl apply -f prometheus-prometheus.yaml

prometheus CRD 批改结束，能够到 prometheus dashboard 查看 config 是否被批改。

3. prometheus 实例减少 clusterrole

增加了上述配置后，prometheus-k8s- 0 的 log 会发现很多的 forbidden，这是因为其没有 service/pod 的 list 权限。老的权限：

# cat /etc/kubernetes/prometheus/prometheus-clusterRole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

须要批改其 clusterRole，减少权限，新的权限：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

执行：kubectl apply -f prometheus-clusterRole.yaml 进行更新。

参考：
1.Prometheus Operator 高级配置：https://www.qikqiak.com/post/…