指标：
用户启动的service或pod，在annotation中增加label后，能够主动被prometheus发现：

annotations:  prometheus.io/scrape: "true"  prometheus.io/port: "9121"

1. secret保留主动发现的配置

若要特定的annotation被发现，须要为prometheus减少如下配置：

- job_name: 'kubernetes-service-endpoints'  kubernetes_sd_configs:  - role: endpoints  relabel_configs:  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]    action: keep    regex: true  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]    action: replace    target_label: __scheme__    regex: (https?)  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]    action: replace    target_label: __metrics_path__    regex: (.+)  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]    action: replace    target_label: __address__    regex: ([^:]+)(?::\d+)?;(\d+)    replacement: $1:$2  - action: labelmap    regex: __meta_kubernetes_service_label_(.+)  - source_labels: [__meta_kubernetes_namespace]    action: replace    target_label: kubernetes_namespace  - source_labels: [__meta_kubernetes_service_name]    action: replace    target_label: kubernetes_name

上述配置会筛选endpoints：prometheus.io/scrape=True

将上述配置保留为secret:

$ kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoringsecret "additional-configs" created

2. 将配置增加到prometheus实例

批改prometheus CRD，将下面的secret增加进去：

# vi /etc/kubernetes/prometheus/prometheus-prometheus.yamlapiVersion: monitoring.coreos.com/v1kind: Prometheusmetadata:  labels:    prometheus: k8s  name: k8s  namespace: monitoringspec:  ......  additionalScrapeConfigs:    name: additional-configs    key: prometheus-additional.yaml  serviceAccountName: prometheus-k8s  serviceMonitorNamespaceSelector: {}  serviceMonitorSelector: {}  version: v2.5.0# kubectl apply -f prometheus-prometheus.yaml

prometheus CRD批改结束，能够到prometheus dashboard查看config是否被批改。

3. prometheus实例减少clusterrole

增加了上述配置后，prometheus-k8s-0的log会发现很多的forbidden，这是因为其没有service/pod的list权限。老的权限：

# cat /etc/kubernetes/prometheus/prometheus-clusterRole.yamlapiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  name: prometheus-k8srules:- apiGroups:  - ""  resources:  - nodes/metrics  verbs:  - get- nonResourceURLs:  - /metrics  verbs:  - get

须要批改其clusterRole，减少权限，新的权限：

apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  name: prometheus-k8srules:- apiGroups:  - ""  resources:  - nodes  - services  - endpoints  - pods  - nodes/proxy  verbs:  - get  - list  - watch- apiGroups:  - ""  resources:  - configmaps  - nodes/metrics  verbs:  - get- nonResourceURLs:  - /metrics  verbs:  - get

执行：kubectl apply -f prometheus-clusterRole.yaml进行更新。

参考：
1.Prometheus Operator高级配置：https://www.qikqiak.com/post/...