最近，在对公司容器云的日志计划进行设计的时候，发现支流的 ELK 或者 EFK 比拟重，再加上现阶段对于 ES 简单的搜寻性能很多都用不上最终抉择了 Grafana 开源的 Loki 日志零碎，上面介绍下 Loki 的背景。
背景和动机
当咱们的容器云运行的利用或者某个节点呈现问题了，解决思路应该如下：
咱们的监控应用的是基于 prometheus 体系进行革新的，prometheus 中比拟重要的是 metric 和 alert,metric 是来阐明以后或者历史达到了某个值，alert 设置 metric 达到某个特定的基数触发了告警，然而这些信息显著是不够的。
咱们都晓得，k8s 的根本单位是 pod,pod 把日志输入到 stdout 和 stderr, 平时有什么问题咱们通常在界面或者通过命令查看相干的日志, 举个例子：当咱们的某个 pod 的内存变得很大，触发了咱们的 alert，这个时候管理员，去页面查问确认是哪个 pod 有问题，而后要确认 pod 内存变大的起因，咱们还须要去查问 pod 的日志，如果没有日志零碎，那么咱们就须要到页面或者应用命令进行查问了：
如果，这个时候利用忽然挂了，这个时候咱们就无奈查到相干的日志了，所以须要引入日志零碎，对立收集日志，而应用 ELK 的话，就须要在 Kibana 和 Grafana 之间切换，影响用户体验。
所以，loki 的第一目标就是最小化度量和日志的切换老本，有助于缩小异样事件的响应工夫和进步用户的体验
ELK 存在的问题
现有的很多日志采集的计划都是采纳全文检索对日志进行索引（如 ELK 计划），长处是功能丰富，容许简单的操作。然而，这些计划往往规模简单，资源占用高，操作苦难。很多性能往往用不上，大多数查问只关注肯定工夫范畴和一些简略的参数（如 host、service 等），应用这些解决方案就有点杀鸡用牛刀的感觉了。
因而，Loki 的第二个目标是，在查询语言的易操作性和复杂性之间能够达到一个衡量。
老本
全文检索的计划也带来老本问题，简略的说就是全文搜寻（如 ES）的倒排索引的切分和共享的老本较高。
起初呈现了其余不同的设计方案如：OKlog, 采纳最终统一的、基于网格的散布策略。这两个设计决策提供了大量的老本升高和非常简单的操作，然而查问不够不便。
因而，Loki 的第三个目标是，进步一个更具老本效益的解决方案。
整体架构
Loki 的架构如下：
不难看出，Loki 的架构非常简单，应用了和 prometheus 一样的标签来作为索引，也就是说，你通过这些标签既能够查问日志的内容也能够查问到监控的数据，岂但缩小了两种查问之间的切换老本，也极大地升高了日志索引的存储。
Loki 将应用与 prometheus 雷同的服务发现和标签从新标记库, 编写了 pormtail, 在 k8s 中 promtail 以 daemonset 形式运行在每个节点中，通过 kubernetes api 等到日志的正确元数据，并将它们发送到 Loki。
上面是日志的存储架构：
读写
日志数据的写次要依靠的是 Distributor 和 Ingester 两个组件, 整体的流程如下：
Distributor
一旦 promtail 收集日志并将其发送给 loki，Distributor 就是第一个接管日志的组件。因为日志的写入量可能很大，所以不能在它们传入时将它们写入数据库。这会毁掉数据库。咱们须要批处理和压缩数据。
Loki 通过构建压缩数据块来实现这一点，办法是在日志进入时对其进行 gzip 操作，组件 ingester 是一个有状态的组件，负责构建和刷新 chunck，当 chunk 达到肯定的数量或者工夫后，刷新到存储中去。每个流的日志对应一个 ingester, 当日志达到 Distributor 后，依据元数据和 hash 算法计算出应该到哪个 ingester 下面。
此外，为了冗余和弹性，咱们将其复制 n（默认状况下为 3）次。
Ingester
ingester 接管到日志并开始构建 chunk:
基本上就是将日志进行压缩并附加到 chunk 下面。一旦 chunk“填满”（数据达到肯定数量或者过了肯定期限），ingester 将其刷新到数据库。咱们对块和索引应用独自的数据库，因为它们存储的数据类型不同。
刷新一个 chunk 之后，ingester 而后创立一个新的空 chunk 并将新条目增加到该 chunk 中。
Querier
读取就非常简单了，由 Querier 负责给定一个工夫范畴和标签选择器，Querier 查看索引以确定哪些块匹配，并通过 greps 将结果显示进去。它还从 Ingester 获取尚未刷新的最新数据。
对于每个查问，一个查询器将为您显示所有相干日志。实现了查问并行化，提供分布式 grep，使即便是大型查问也是足够的。
可扩展性
Loki 的索引存储能够是 cassandra/bigtable/dynamodb，而 chuncks 能够是各种对象存储，Querier 和 Distributor 都是无状态的组件。对于 ingester 他尽管是有状态的然而，当新的节点退出或者缩小，整节点间的 chunk 会重新分配，已适应新的散列环。而 Loki 底层存储的实现 Cortex 曾经 在理论的生产中投入使用多年了。
部署
创立 namespace
oc new-project loki权限设置
oc adm policy add-scc-to-user anyuid -z default -n loki
oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:loki:default装置 loki
装置命令：
oc create -f statefulset.json -n lokistatefulset.json 如下：
{
"apiVersion": "apps/v1",
"kind": "StatefulSet",
"metadata": {"name": "loki"},
"spec": {
"podManagementPolicy": "OrderedReady",
"replicas": 1,
"revisionHistoryLimit": 10,
"selector": {
"matchLabels": {"app": "loki"}
},
"serviceName": "womping-stoat-loki-headless",
"template": {
"metadata": {
"annotations": {
"checksum/config": "da297d66ee53e0ce68b58e12be7ec5df4a91538c0b476cfe0ed79666343df72b",
"prometheus.io/port": "http-metrics",
"prometheus.io/scrape": "true"
},
"creationTimestamp": null,
"labels": {
"app": "loki",
"name": "loki"
}
},
"spec": {"affinity": {},
"containers": [
{
"args": ["-config.file=/etc/loki/local-config.yaml"],
"image": "grafana/loki:latest",
"imagePullPolicy": "IfNotPresent",
"livenessProbe": {
"failureThreshold": 3,
"httpGet": {
"path": "/ready",
"port": "http-metrics",
"scheme": "HTTP"
},
"initialDelaySeconds": 45,
"periodSeconds": 10,
"successThreshold": 1,
"timeoutSeconds": 1
},
"name": "loki",
"ports": [
{
"containerPort": 3100,
"name": "http-metrics",
"protocol": "TCP"
}
],
"readinessProbe": {
"failureThreshold": 3,
"httpGet": {
"path": "/ready",
"port": "http-metrics",
"scheme": "HTTP"
},
"initialDelaySeconds": 45,
"periodSeconds": 10,
"successThreshold": 1,
"timeoutSeconds": 1
},
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/tmp/loki",
"name": "storage"
}
]
}
],
"dnsPolicy": "ClusterFirst",
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"terminationGracePeriodSeconds": 30,
"volumes": [
{"emptyDir": {},
"name": "storage"
}
]
}
},
"updateStrategy": {"type": "RollingUpdate"}
}
}装置 promtail
装置命令：
oc create -f configmap.json -n lokiconfigmap.json 如下：
{
"apiVersion": "v1",
"data": {"promtail.yaml": "client:\n backoff_config:\n maxbackoff: 5s\n maxretries: 5\n minbackoff: 100ms\n batchsize: 102400\n batchwait: 1s\n external_labels: {}\n timeout: 10s\npositions:\n filename: /run/promtail/positions.yaml\nserver:\n http_listen_port: 3101\ntarget_config:\n sync_period: 10s\n\nscrape_configs:\n- job_name: kubernetes-pods-name\n pipeline_stages:\n - docker: {}\n \n kubernetes_sd_configs:\n - role: pod\n relabel_configs:\n - source_labels:\n - __meta_kubernetes_pod_label_name\n target_label: __service__\n - source_labels:\n - __meta_kubernetes_pod_node_name\n target_label: __host__\n - action: drop\n regex: ^$\n source_labels:\n - __service__\n - action: labelmap\n regex: __meta_kubernetes_pod_label_(.+)\n - action: replace\n replacement: $1\n separator: /\n source_labels:\n - __meta_kubernetes_namespace\n - __service__\n target_label: job\n - action: replace\n source_labels:\n - __meta_kubernetes_namespace\n target_label: namespace\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_name\n target_label: instance\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_container_name\n target_label: container_name\n - replacement: /var/log/pods/*$1/*.log\n separator: /\n source_labels:\n - __meta_kubernetes_pod_uid\n - __meta_kubernetes_pod_container_name\n target_label: __path__\n- job_name: kubernetes-pods-app\n pipeline_stages:\n - docker: {}\n \n kubernetes_sd_configs:\n - role: pod\n relabel_configs:\n - action: drop\n regex: .+\n source_labels:\n - __meta_kubernetes_pod_label_name\n - source_labels:\n - __meta_kubernetes_pod_label_app\n target_label: __service__\n - source_labels:\n - __meta_kubernetes_pod_node_name\n target_label: __host__\n - action: drop\n regex: ^$\n source_labels:\n - __service__\n - action: labelmap\n regex: __meta_kubernetes_pod_label_(.+)\n - action: replace\n replacement: $1\n separator: /\n source_labels:\n - __meta_kubernetes_namespace\n - __service__\n target_label: job\n - action: replace\n source_labels:\n - __meta_kubernetes_namespace\n target_label: namespace\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_name\n target_label: instance\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_container_name\n target_label: container_name\n - replacement: /var/log/pods/*$1/*.log\n separator: /\n source_labels:\n - __meta_kubernetes_pod_uid\n - __meta_kubernetes_pod_container_name\n target_label: __path__\n- job_name: kubernetes-pods-direct-controllers\n pipeline_stages:\n - docker: {}\n \n kubernetes_sd_configs:\n - role: pod\n relabel_configs:\n - action: drop\n regex: .+\n separator:''\n source_labels:\n - __meta_kubernetes_pod_label_name\n - __meta_kubernetes_pod_label_app\n - action: drop\n regex: ^([0-9a-z-.]+)(-[0-9a-f]{8,10})$\n source_labels:\n - __meta_kubernetes_pod_controller_name\n - source_labels:\n - __meta_kubernetes_pod_controller_name\n target_label: __service__\n - source_labels:\n - __meta_kubernetes_pod_node_name\n target_label: __host__\n - action: drop\n regex: ^$\n source_labels:\n - __service__\n - action: labelmap\n regex: __meta_kubernetes_pod_label_(.+)\n - action: replace\n replacement: $1\n separator: /\n source_labels:\n - __meta_kubernetes_namespace\n - __service__\n target_label: job\n - action: replace\n source_labels:\n - __meta_kubernetes_namespace\n target_label: namespace\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_name\n target_label: instance\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_container_name\n target_label: container_name\n - replacement: /var/log/pods/*$1/*.log\n separator: /\n source_labels:\n - __meta_kubernetes_pod_uid\n - __meta_kubernetes_pod_container_name\n target_label: __path__\n- job_name: kubernetes-pods-indirect-controller\n pipeline_stages:\n - docker: {}\n \n kubernetes_sd_configs:\n - role: pod\n relabel_configs:\n - action: drop\n regex: .+\n separator: ''\n source_labels:\n - __meta_kubernetes_pod_label_name\n - __meta_kubernetes_pod_label_app\n - action: keep\n regex: ^([0-9a-z-.]+)(-[0-9a-f]{8,10})$\n source_labels:\n - __meta_kubernetes_pod_controller_name\n - action: replace\n regex: ^([0-9a-z-.]+)(-[0-9a-f]{8,10})$\n source_labels:\n - __meta_kubernetes_pod_controller_name\n target_label: __service__\n - source_labels:\n - __meta_kubernetes_pod_node_name\n target_label: __host__\n - action: drop\n regex: ^$\n source_labels:\n - __service__\n - action: labelmap\n regex: __meta_kubernetes_pod_label_(.+)\n - action: replace\n replacement: $1\n separator: /\n source_labels:\n - __meta_kubernetes_namespace\n - __service__\n target_label: job\n - action: replace\n source_labels:\n - __meta_kubernetes_namespace\n target_label: namespace\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_name\n target_label: instance\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_container_name\n target_label: container_name\n - replacement: /var/log/pods/*$1/*.log\n separator: /\n source_labels:\n - __meta_kubernetes_pod_uid\n - __meta_kubernetes_pod_container_name\n target_label: __path__\n- job_name: kubernetes-pods-static\n pipeline_stages:\n - docker: {}\n \n kubernetes_sd_configs:\n - role: pod\n relabel_configs:\n - action: drop\n regex: ^$\n source_labels:\n - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_label_component\n target_label: __service__\n - source_labels:\n - __meta_kubernetes_pod_node_name\n target_label: __host__\n - action: drop\n regex: ^$\n source_labels:\n - __service__\n - action: labelmap\n regex: __meta_kubernetes_pod_label_(.+)\n - action: replace\n replacement: $1\n separator: /\n source_labels:\n - __meta_kubernetes_namespace\n - __service__\n target_label: job\n - action: replace\n source_labels:\n - __meta_kubernetes_namespace\n target_label: namespace\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_name\n target_label: instance\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_container_name\n target_label: container_name\n - replacement: /var/log/pods/*$1/*.log\n separator: /\n source_labels:\n - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror\n - __meta_kubernetes_pod_container_name\n target_label: __path__\n"
},
"kind": "ConfigMap",
"metadata": {
"creationTimestamp": "2019-09-05T01:05:03Z",
"labels": {
"app": "promtail",
"chart": "promtail-0.12.0",
"heritage": "Tiller",
"release": "lame-zorse"
},
"name": "lame-zorse-promtail",
"namespace": "loki",
"resourceVersion": "17921611",
"selfLink": "/api/v1/namespaces/loki/configmaps/lame-zorse-promtail",
"uid": "30fcb896-cf79-11e9-b58e-e4a8b6cc47d2"
}
}oc create -f daemonset.json -n lokidaemonset.json 如下：{
"apiVersion": "apps/v1",
"kind": "DaemonSet",
"metadata": {
"annotations": {"deployment.kubernetes.io/revision": "2"},
"creationTimestamp": "2019-09-05T01:16:37Z",
"generation": 2,
"labels": {
"app": "promtail",
"chart": "promtail-0.12.0",
"heritage": "Tiller",
"release": "lame-zorse"
},
"name": "lame-zorse-promtail",
"namespace": "loki"
},
"spec": {
"progressDeadlineSeconds": 600,
"replicas": 1,
"revisionHistoryLimit": 10,
"selector": {
"matchLabels": {
"app": "promtail",
"release": "lame-zorse"
}
},
"strategy": {
"rollingUpdate": {
"maxSurge": 1,
"maxUnavailable": 1
},
"type": "RollingUpdate"
},
"template": {
"metadata": {
"annotations": {
"checksum/config": "75a25ee4f2869f54d394bf879549a9c89c343981a648f8d878f69bad65dba809",
"prometheus.io/port": "http-metrics",
"prometheus.io/scrape": "true"
},
"creationTimestamp": null,
"labels": {
"app": "promtail",
"release": "lame-zorse"
}
},
"spec": {"affinity": {},
"containers": [
{
"args": [
"-config.file=/etc/promtail/promtail.yaml",
"-client.url=http://loki.loki.svc:3100/api/prom/push"
],
"env": [
{
"name": "HOSTNAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "spec.nodeName"
}
}
}
],
"image": "grafana/promtail:v0.3.0",
"imagePullPolicy": "IfNotPresent",
"name": "promtail",
"ports": [
{
"containerPort": 3101,
"name": "http-metrics",
"protocol": "TCP"
}
],
"readinessProbe": {
"failureThreshold": 5,
"httpGet": {
"path": "/ready",
"port": "http-metrics",
"scheme": "HTTP"
},
"initialDelaySeconds": 10,
"periodSeconds": 10,
"successThreshold": 1,
"timeoutSeconds": 1
},
"resources": {},
"securityContext": {
"readOnlyRootFilesystem": true,
"runAsUser": 0
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/etc/promtail",
"name": "config"
},
{
"mountPath": "/run/promtail",
"name": "run"
},
{
"mountPath": "/var/lib/docker/containers",
"name": "docker",
"readOnly": true
},
{
"mountPath": "/var/log/pods",
"name": "pods",
"readOnly": true
}
]
}
],
"dnsPolicy": "ClusterFirst",
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"terminationGracePeriodSeconds": 30,
"volumes": [
{
"configMap": {
"defaultMode": 420,
"name": "lame-zorse-promtail"
},
"name": "config"
},
{
"hostPath": {
"path": "/run/promtail",
"type": ""},"name":"run"
},
{
"hostPath": {
"path": "/var/lib/docker/containers",
"type": ""},"name":"docker"
},
{
"hostPath": {
"path": "/var/log/pods",
"type": ""},"name":"pods"
}
]
}
}
}
}装置服务
oc create -f service.json -n lokiservice.json 的内容如下：
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"creationTimestamp": "2019-09-04T09:37:49Z",
"name": "loki",
"namespace": "loki",
"resourceVersion": "17800188",
"selfLink": "/api/v1/namespaces/loki/services/loki",
"uid": "a87fe237-cef7-11e9-b58e-e4a8b6cc47d2"
},
"spec": {
"externalTrafficPolicy": "Cluster",
"ports": [
{
"name": "lokiport",
"port": 3100,
"protocol": "TCP",
"targetPort": 3100
}
],
"selector": {"app": "loki"},
"sessionAffinity": "None",
"type": "NodePort"
},
"status": {"loadBalancer": {}
}Loki 提供了 HTTP 接口，咱们这里就不详解了，大家能够看：
https://github.com/grafana/lo…
咱们这里说下查问的接口如何应用：
- 第一步，获取以后 Loki 的元数据类型：
curl http://192.168.25.30:30972/api/prom/label
{"values": ["alertmanager", "app", "component", "container_name", "controller_revision_hash", "deployment", "deploymentconfig", "docker_registry", "draft", "filename", "instance", "job", "logging_infra", "metrics_infra", "name", "namespace", "openshift_io_component", "pod_template_generation", "pod_template_hash", "project", "projectname", "prometheus", "provider", "release", "router", "servicename", "statefulset_kubernetes_io_pod_name", "stream", "tekton_dev_pipeline", "tekton_dev_pipelineRun", "tekton_dev_pipelineTask", "tekton_dev_task", "tekton_dev_taskRun", "type", "webconsole"]
}-
第二步，获取某个元数据类型的值：
curl http://192.168.25.30:30972/api/prom/label/namespace/values {"values":["cicd","default","gitlab","grafanaserver","jenkins","jx-staging","kube-system","loki","mysql-exporter","new2","openshift-console","openshift-infra","openshift-logging","openshift-monitoring","openshift-node","openshift-sdn","openshift-web-console","tekton-pipelines","test111"]} -
第三步，依据 label 进行查问，例如：
http://192.168.25.30:30972/api/prom/query?direction=BACKWARD&limit=1000®exp=&query={namespace="cicd"}&start=1567644457221000000&end=1567730857221000000&refId=A参数解析：
query: #一种查问语法具体见上面章节，{name=~“mysql.+”} or {namespace=“cicd”} |= "error" 示意查问，namespace 为 cicd 的日志中，有 error 字样的信息。limit: #返回日志的数量 start：# 开始工夫，Unix 工夫示意办法 默认为，一小时前工夫 end: #完结工夫，默认为以后工夫 direction: #forward 或者 backward, . 指定 limit 时候有用，默认为 backward. regexp:# 对后果进行 regex 过滤链接：blog.csdn.net/Linkthaha/