关于prometheus:Prometheus监控神器Kubernetes篇二

57次阅读

共计 13473 个字符，预计需要花费 34 分钟才能阅读完成。

在 Kubernetes 中手动形式部署 Statefulset 的 Grafana，并应用 StorageClass 来长久化数据，并且配置 ingress-nginx 拜访。

本篇应用 StorageClass 来长久化数据，搭建 Statefulset 的 Grafana，并且在 Dashboard 导入前配置后面曾经创立好的 Prometheus 的集群外部拜访地址，同时配置 ingress-nginx 内部拜访。

环境

我的本地环境应用的 sealos 一键部署，次要是为了便于测试。

OS Kubernetes HostName IP Service
Ubuntu 18.04 1.17.7 sealos-k8s-m1 192.168.1.151 node-exporter prometheus-federate-0
Ubuntu 18.04 1.17.7 sealos-k8s-m2 192.168.1.152 node-exporter grafana alertmanager-0
Ubuntu 18.04 1.17.7 sealos-k8s-m3 192.168.1.150 node-exporter alertmanager-1
Ubuntu 18.04 1.17.7 sealos-k8s-node1 192.168.1.153 node-exporter prometheus-0 kube-state-metrics
Ubuntu 18.04 1.17.7 sealos-k8s-node2 192.168.1.154 node-exporter prometheus-1
Ubuntu 18.04 1.17.7 sealos-k8s-node2 192.168.1.155 node-exporter prometheus-2

部署 Grafana

创立 Grafana 的 SA 文件 

mkdir /data/manual-deploy/grafana/
cat grafana-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: grafana
  namespace: kube-system

创立 Grafana 的 sc 配置文件 

cat grafana-data-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: grafana-lpv
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

创立 Grafana 的 pv 配置文件 

cat grafana-data-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-pv-0
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: grafana-lpv
  local:
    path: /data/grafana-data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - sealos-k8s-m2

在调度节点上创立 pv 目录与赋权 

mkdir /data/grafana-data
chown -R 65534.65534 /data/grafana-data

Dashboard 文件太大，本人下载改一下的 namespace

grafana-dashboard-configmap.yaml

# 下载到本地
cat grafana-dashboard-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: grafana-dashboards
  namespace: kube-system
  labels:
    app.kubernetes.io/name: grafana
    app.kubernetes.io/component: grafana
data:
....

创立 Grafana 的 configmap 配置文件，其中的 Prometheus 是集群外部 dns 地址，请自行调整。

cat grafana-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: kube-system
  labels:
    app.kubernetes.io/name: grafana
data:
  datasources.yaml: |
    apiVersion: 1
    datasources:
    - access: proxy
      isDefault: true
      name: prometheus
      type: prometheus
      url: http://prometheus-0.prometheus.kube-system.svc.cluster.local:9090
      version: 1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboardproviders
  namespace: kube-system
  labels:
    app.kubernetes.io/name: grafana
data:
  dashboardproviders.yaml: |
    apiVersion: 1
    providers:
    - disableDeletion: false
      editable: true
      folder: ""
      name: default
      options:
        path: /var/lib/grafana/dashboards
      orgId: 1
      type: file

我这里没有用 secret，须要的本人调整下，在 statefulset 中有调用办法，我曾经正文了。

cat grafana-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: grafana-secret
  namespace: kube-system
  labels:
    app.kubernetes.io/name: grafana
    app.kubernetes.io/component: grafana
type: Opaque
data:
  admin-user: YWRtaW4=
  admin-password: "123456"

创立 Grafana 的 statefulset 配置文件 

cat grafana-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: grafana
  namespace: kube-system
  labels: &Labels
    k8s-app: grafana
    app.kubernetes.io/name: grafana
    app.kubernetes.io/component: grafana
spec:
  serviceName: grafana
  replicas: 1
  selector:
    matchLabels: *Labels
  template:
    metadata:
      labels: *Labels
    spec:
      serviceAccountName: grafana
      initContainers:
          - name: "init-chmod-data"
            image: debian:9
            imagePullPolicy: "IfNotPresent"
            command: ["chmod", "777", "/var/lib/grafana"]
            volumeMounts:
            - name: grafana-data
              mountPath: "/var/lib/grafana"
      containers:
        - name: grafana
          image: grafana/grafana:7.1.0
          imagePullPolicy: Always
          volumeMounts:
            - name: dashboards
              mountPath: "/var/lib/grafana/dashboards"
            - name: datasources
              mountPath: "/etc/grafana/provisioning/datasources"              
            - name: grafana-dashboardproviders
              mountPath: "/etc/grafana/provisioning/dashboards"
            - name: grafana-data
              mountPath: "/var/lib/grafana"
          ports:
            - name: service
              containerPort: 80
              protocol: TCP
            - name: grafana
              containerPort: 3000
              protocol: TCP
          env:
            - name: GF_SECURITY_ADMIN_USER
              value: "admin"
              #valueFrom:
              #  secretKeyRef:
              #    name: grafana-secret
              #    key: admin-user
            - name: GF_SECURITY_ADMIN_PASSWORD
              value: "admin"
              #valueFrom:
              #  secretKeyRef:
              #    name: grafana-secret
              #    key: admin-password
          livenessProbe:
            httpGet:
              path: /api/health
              port: 3000
          readinessProbe:
            httpGet:
              path: /api/health
              port: 3000
            initialDelaySeconds: 60
            timeoutSeconds: 30
            failureThreshold: 10
            periodSeconds: 10
          resources:
            limits:
              cpu: 50m
              memory: 100Mi
            requests:
              cpu: 50m
              memory: 100Mi
      volumes:
        - name: datasources
          configMap:
            name: grafana-datasources
        - name: grafana-dashboardproviders
          configMap:
            name: grafana-dashboardproviders
        - name: dashboards
          configMap:
            name: grafana-dashboards
  volumeClaimTemplates:
  - metadata:
      name: grafana-data
    spec:
      storageClassName: "grafana-lpv"
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: "2Gi"

创立 Grafana 的 statefulset 的 svc 配置文件 

cat grafana-service-statefulset.yaml
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: kube-system
  labels:
    k8s-app: grafana
    app.kubernetes.io/name: grafana
    app.kubernetes.io/component: grafana
  annotations:
    prometheus.io/scrape: 'true'
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 3000
  selector:
    k8s-app: grafana

部署 

cd /data/manual-deploy/grafana
ls
grafana-configmap.yaml
grafana-dashboard-configmap.yaml
grafana-data-pv.yaml
grafana-data-storageclass.yaml
grafana-secret.yaml
grafana-serviceaccount.yaml
grafana-service-statefulset.yaml
grafana-statefulset.yaml
kubectl apply .

验证 

kubectl -n kube-system get sa,pod,svc,ep,sc,secret|grep grafana
serviceaccount/grafana                              1         1h
pod/grafana-0                                  1/1     Running   0          1h
service/grafana                   ClusterIP   10.101.176.62    <none>        80/TCP                         1h
endpoints/grafana                   100.73.217.86:3000                                                         1h
storageclass.storage.k8s.io/grafana-lpv               kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  33h
secret/grafana-token-lrsbd                              kubernetes.io/service-account-token   3      1h

部署 ingress-nginx

cd /data/manual-deploy/ingress-nginx
# 创立 ingress-NGINX 的 ns 与 svc
cat ngress-nginx-svc.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

# 创立 mandatory 文件
cat ngress-nginx-mandatory.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      hostNetwork: true
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

---

apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - min:
      memory: 90Mi
      cpu: 100m
    type: Container

部署 

cd /data/manual-deploy/ingress-nginx
ls
ingress-nginx-mandatory.yaml
ngress-nginx-svc.yaml
kubectl apply -f  .

验证 

kubectl -n ingress-nginx get pod,svc,ep
NAME                                            READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-controller-6ffc8fdf96-45ksg   1/1     Running   0          3d12h
pod/nginx-ingress-controller-6ffc8fdf96-76rxj   1/1     Running   0          3d13h
pod/nginx-ingress-controller-6ffc8fdf96-xrhlp   1/1     Running   0          3d13h

NAME                    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx   NodePort   10.110.106.22   <none>        80:31926/TCP,443:31805/TCP   3d13h

NAME                      ENDPOINTS                                                        AGE
endpoints/ingress-nginx   192.168.1.153:80,192.168.1.154:80,192.168.1.155:80 + 3 more...   3d13h

配置 ingress 拜访。

Prometheus ingress-NGINX 配置文件 

cat alertmanager-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prometheus-ingress
  namespace: kube-system
  annotations:
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-name: "prometheus-cookie"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: "letsencrypt-local"
    kubernetes.io/tls-acme: "false"
spec:
  rules:
  - host: prom.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: prometheus
          servicePort: 9090
  tls:
  - hosts:
      - prom.example.com

Alertmanager ingress-NGINX 配置文件 

cat alertmanager-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: alertmanager-ingress
  namespace: kube-system
  annotations:
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-name: "alert-cookie"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: "letsencrypt-local"
    kubernetes.io/tls-acme: "false"
spec:
  rules:
  - host: alert.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: alertmanager-operated
          servicePort: 9093
  tls:
  - hosts:
      - alert.example.com

Grafana ingress-NGINX 配置文件 

cat grafana-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: kube-system
  annotations:
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-name: "grafana-cookie"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: "letsencrypt-local"
    kubernetes.io/tls-acme: "false"
spec:
  rules:
  - host: grafana.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: grafana
          servicePort: http
  tls:
  - hosts:
      - grafana.example.com

部署 

cd /data/manual-deploy/ingress-nginx
alertmanager-ingress.yaml
grafana-ingress.yaml
prometheus-ingress.yaml
kubectl apply -f alertmanager-ingress.yaml
kubectl apply -f prometheus-ingress.yaml
kubectl apply -f grafana-ingress.yaml

验证 

kubectl -n kube-system get ingresses
NAME                   HOSTS                 ADDRESS         PORTS     AGE
alertmanager-ingress   alert.example.com     10.110.106.22   80, 443   15h
grafana-ingress        grafana.example.com   10.110.106.22   80, 443   30h
prometheus-ingress     prom.example.com      10.110.106.22   80, 443   15h

而后能够在在本地 dns 服务器或者 host 中绑定域名跟主机关系拜访，我这里没有配置 SSL 证书，如果你有这个需要，须要本人独自配置。

到此，针对手动部署的全副过程就此结束，本次搭建的服务是比拟新的版本，可能相互依赖中会有未知问题，尽量版本保持一致。

正文完
prometheus
发表至： prometheus
2020-09-09
 0
关于prometheus:使用VictoriaMetrics监控K8S集群
关于prometheus:为go应用添加prometheus监控指标
关于prometheus:Prometheus样本采集器汇总建议收藏-IDCF
关于prometheus:第05期使用-prometheus-监控-clickhouse-集群