关于prometheus:开源signoz实现可观测性的分析

38次阅读

共计 13883 个字符，预计需要花费 35 分钟才能阅读完成。

Sginoz 是一个开源的 APM(Application Performance Management)，它是利用可观测性的一个实际，应用 OpenTelemetry 协定，将 traces/metrics/log 交融在一起。

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

本文重点关注 traces 和 metrics 方面的实现。

一. 整体架构

  • app 应用 opentelmetry-sdk 编写代码 (反对 java/golang/python 等)；
  • app 配置将 metrics 和 tracing 发送至 otel-collector；

  • otel-collector 定制实现了:

    • clickhouse-metrics-exporter: 将 metrics 发送至 clickhouse；
    • clickhouse-trace-exporter：将 trace 发送至 clickhouse:
  • query-server 负责从 clickhouse 中查问 metrics 和 trace 信息，并提供前端 API；

二. otel-collector

1. app 将 metrics 和 trace 发送至 otel-collector

app 中引入 opentelemetry:

go.opentelemetry.io/otel

若应用 gin 作为 http 框架：

import "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"

r := gin.Default()
r.Use(otelgin.Middleware(serviceName))

app 在运行时，指定 ServiceName 和 collector 地址：

# SERVICE_NAME=goApp INSECURE_MODE=true OTEL_EXPORTER_OTLP_ENDPOINT=192.168.0.1:4317 go run main.go

2. otel-collector 的配置

otel-collector 的配置，分为 receivers、processors、exporters，
而后由 service 通过 pipeline 组织成残缺的性能。

receivers:
  opencensus:
    endpoint: 0.0.0.0:55678
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_http:
        endpoint: 0.0.0.0:14268

processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 11000
    timeout: 10s

exporters:
  clickhousetraces:
    datasource: tcp://clickhouse:9000/?database=signoz_traces
  clickhousemetricswrite:
    endpoint: tcp://clickhouse:9000/?database=signoz_metrics
    resource_to_telemetry_conversion:
      enabled: true

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888
  extensions:
    - health_check
    - zpages
    - pprof
  pipelines:
    traces:
      receivers: [jaeger, otlp]
      processors: [signozspanmetrics/prometheus, batch]
      exporters: [clickhousetraces]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhousemetricswrite]

3. otel-collector 实现 clickhouse 的 exporter

otel-collector 将 metrics 和 traces 保留至 clickhouse，为此，signoz 在 otel-collector 中，实现了：

  • clickhousemetricsexporter;
  • clickhousetracesexporter;

存储在 clickhouse 中的 metrics 信息：

## time_series 信息
73048f54a32c :) select * from time_series_v2 limit 3;
SELECT *
FROM time_series_v2
LIMIT 3
Query id: 2540c93a-a396-42a7-b4e5-bed772ac00c5
┌─metric_name─────────────────────────────────┬──────────fingerprint─┬──timestamp_ms─┬─labels────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─labels_object──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ otelcol_exporter_enqueue_failed_log_records │   136087390202145591 │ 1659598169051 │ {"__name__":"otelcol_exporter_enqueue_failed_log_records","exporter":"clickhousemetricswrite","instance":"localhost:8888","job":"otel-collector-metrics","service_instance_id":"b3d22c64-5c09-44d7-9367-9716747842db","service_version":"latest"} │ ('otelcol_exporter_enqueue_failed_log_records','','','','','','','clickhousemetricswrite','','','localhost:8888','otel-collector-metrics','','','','','','','','','','b3d22c64-5c09-44d7-9367-9716747842db','','','latest','','','','','') │
│ otelcol_exporter_enqueue_failed_log_records │  7551020479515194203 │ 1659598139047 │ {"__name__":"otelcol_exporter_enqueue_failed_log_records","exporter":"prometheus","instance":"otel-collector:8888","job":"otel-collector","service_instance_id":"ed7abeec-d18f-4c4d-8bcb-64a9dea37976","service_version":"latest"}                │ ('otelcol_exporter_enqueue_failed_log_records','','','','','','','prometheus','','','otel-collector:8888','otel-collector','','','','','','','','','','ed7abeec-d18f-4c4d-8bcb-64a9dea37976','','','latest','','','','','')                │
│ otelcol_exporter_enqueue_failed_log_records │ 11388370004215594241 │ 1659598139047 │ {"__name__":"otelcol_exporter_enqueue_failed_log_records","exporter":"clickhousetraces","instance":"otel-collector:8888","job":"otel-collector","service_instance_id":"ed7abeec-d18f-4c4d-8bcb-64a9dea37976","service_version":"latest"}          │ ('otelcol_exporter_enqueue_failed_log_records','','','','','','','clickhousetraces','','','otel-collector:8888','otel-collector','','','','','','','','','','ed7abeec-d18f-4c4d-8bcb-64a9dea37976','','','latest','','','','','')          │
└─────────────────────────────────────────────┴──────────────────────┴───────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
## samples 信息
73048f54a32c :) select * from samples_v2 limit 3
SELECT *
FROM samples_v2
LIMIT 3
Query id: 40dfc9f5-e76e-4ecc-8b50-c791fa9e041c
┌─metric_name─────────────────────────────────┬────────fingerprint─┬──timestamp_ms─┬─value─┐
│ otelcol_exporter_enqueue_failed_log_records │ 136087390202145591 │ 1659598160222 │     0 │
│ otelcol_exporter_enqueue_failed_log_records │ 136087390202145591 │ 1659598220222 │     0 │
│ otelcol_exporter_enqueue_failed_log_records │ 136087390202145591 │ 1659598280222 │     0 │
└─────────────────────────────────────────────┴────────────────────┴───────────────┴───────┘
3 rows in set. Elapsed: 0.016 sec.

存储在 clickhouse 中的 traces 信息：

73048f54a32c :) select  * from signoz_spans limit 3;
SELECT *
FROM signoz_spans
LIMIT 3
Query id: d9d84316-c6f0-4089-87a1-187132a295b1
Connecting to database signoz_traces at localhost:9000 as user default.
Connected to ClickHouse server version 22.4.5 revision 54455.
┌─────────────────────timestamp─┬─traceID──────────────────────────┬─model──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 2022-08-04 14:42:21.920273000 │ 00000000000000000000976d1a8457bf │ {"traceId":"00000000000000000000976d1a8457bf","spanId":"0c8946ee620b3b72","name":"HTTP GET /route","durationNano":35578000,"startTimeUnixNano":1659624141920273000,"serviceName":"route","kind":2,"references":[{"traceId":"00000000000000000000976d1a8457bf","spanId":"75509681018b9e65","refType":"CHILD_OF"}],"tagMap":{"client-uuid":"1f460e28dcd19136","component":"net/http","host.name":"93a9af66f59d","http.method":"GET","http.status_code":"200","http.url":"/route?dropoff=728%2C326\u0026pickup=232%2C495","ip":"172.18.0.3","opencensus.exporterversion":"Jaeger-Go-2.30.0","service.name":"route"},"event":["{\"timeUnixNano\":1659624141920308000,\"attributeMap\":{\"event\":\"HTTP request received\",\"level\":\"info\",\"method\":\"GET\",\"url\":\"/route?dropoff=728%2C326\\u0026pickup=232%2C495\"}}"]} │
│ 2022-08-04 14:42:21.920759000 │ 00000000000000000000976d1a8457bf │ {"traceId":"00000000000000000000976d1a8457bf","spanId":"3a48059caa1e422a","name":"HTTP GET /route","durationNano":50933000,"startTimeUnixNano":1659624141920759000,"serviceName":"route","kind":2,"references":[{"traceId":"00000000000000000000976d1a8457bf","spanId":"37835dd357b1d10c","refType":"CHILD_OF"}],"tagMap":{"client-uuid":"1f460e28dcd19136","component":"net/http","host.name":"93a9af66f59d","http.method":"GET","http.status_code":"200","http.url":"/route?dropoff=728%2C326\u0026pickup=745%2C522","ip":"172.18.0.3","opencensus.exporterversion":"Jaeger-Go-2.30.0","service.name":"route"},"event":["{\"timeUnixNano\":1659624141920796000,\"attributeMap\":{\"event\":\"HTTP request received\",\"level\":\"info\",\"method\":\"GET\",\"url\":\"/route?dropoff=728%2C326\\u0026pickup=745%2C522\"}}"]} │
│ 2022-08-04 14:42:21.920505000 │ 00000000000000000000976d1a8457bf │ {"traceId":"00000000000000000000976d1a8457bf","spanId":"2754f5b55262e27b","name":"HTTP GET /route","durationNano":71846000,"startTimeUnixNano":1659624141920505000,"serviceName":"route","kind":2,"references":[{"traceId":"00000000000000000000976d1a8457bf","spanId":"44b47488d0e49c98","refType":"CHILD_OF"}],"tagMap":{"client-uuid":"1f460e28dcd19136","component":"net/http","host.name":"93a9af66f59d","http.method":"GET","http.status_code":"200","http.url":"/route?dropoff=728%2C326\u0026pickup=462%2C723","ip":"172.18.0.3","opencensus.exporterversion":"Jaeger-Go-2.30.0","service.name":"route"},"event":["{\"timeUnixNano\":1659624141920525000,\"attributeMap\":{\"event\":\"HTTP request received\",\"level\":\"info\",\"method\":\"GET\",\"url\":\"/route?dropoff=728%2C326\\u0026pickup=462%2C723\"}}"]} │
└───────────────────────────────┴──────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

三. Query Service

query-service 服务中读取 clickhouse 的 metrics 和 traces，给前端提供了 HTTP 接口。

另外，query-service 还将 prometheus 引入作为 sdk，实现了告警规定的创立及告警性能。

1. 查问 metrics

API 入口：/api/v1/query_range

// app/http_handler.go
func (aH *APIHandler) RegisterRoutes(router *mux.Router) {router.HandleFunc("/api/v1/query_range", ViewAccess(aH.queryRangeMetrics)).Methods(http.MethodGet)
    ...
}

func (aH *APIHandler) queryRangeMetrics(w http.ResponseWriter, r *http.Request) {query, apiErrorObj := parseQueryRangeRequest(r)
    ...
    ctx := r.Context()
    ...
    res, qs, apiError := (*aH.reader).GetQueryRangeResult(ctx, query)
    ...
    response_data := &model.QueryData{ResultType: res.Value.Type(),
        Result:     res.Value,
        Stats:      qs,
    }
    aH.respond(w, response_data)
}

为 clickhouse 实现了 remote-read，故查问跟 prometheus 的查问代码差不多：

  • 结构查问对象；
  • 调用 engine 执行查问；
// app/clickhouseReader/reader.go
func (r *ClickHouseReader) GetQueryRangeResult(ctx context.Context, query *model.QueryRangeParams) (*promql.Result, *stats.QueryStats, *model.ApiError) {qry, err := r.queryEngine.NewRangeQuery(r.remoteStorage, query.Query, query.Start, query.End, query.Step)
    ...
    res := qry.Exec(ctx)
    // Optional stats field in response if parameter "stats" is not empty.
    var qs *stats.QueryStats
    if query.Stats != "" {qs = stats.NewQueryStats(qry.Stats())
    }
    qry.Close()
    return res, qs, nil
}

2. 查问 traces

查问的 url: /api/v1/traces/00000000000000001348070b30cd6aec

入口代码：/api/v1/traces

// app/http_handler.go
func (aH *APIHandler) RegisterRoutes(router *mux.Router) {
    ...
    router.HandleFunc("/api/v1/traces/{traceId}", ViewAccess(aH.searchTraces)).Methods(http.MethodGet)
    ...
}
func (aH *APIHandler) searchTraces(w http.ResponseWriter, r *http.Request) {vars := mux.Vars(r)
    traceId := vars["traceId"]
    result, err := (*aH.reader).SearchTraces(r.Context(), traceId)
    if aH.handleError(w, err, http.StatusBadRequest) {return}
    aH.writeJSON(w, r, result)
}

执行查问，通过 clickhouse 的查问语法实现：

// app/clickhouseReader/reader.go
func (r *ClickHouseReader) SearchTraces(ctx context.Context, traceId string) (*[]model.SearchSpansResult, error) {var searchScanReponses []model.SearchSpanDBReponseItem
    query := fmt.Sprintf("SELECT timestamp, traceID, model FROM %s.%s WHERE traceID=$1", r.traceDB, r.spansTable)
    err := r.db.Select(ctx, &searchScanReponses, query, traceId)
    ...
    searchSpansResult := []model.SearchSpansResult{
        {Columns: []string{"__time", "SpanId", "TraceId", "ServiceName", "Name", "Kind", "DurationNano", "TagsKeys", "TagsValues", "References", "Events", "HasError"},
            Events:  make([][]interface{}, len(searchScanReponses)),
        },
    }
    for i, item := range searchScanReponses {
        var jsonItem model.SearchSpanReponseItem
        json.Unmarshal([]byte(item.Model), &jsonItem)
        jsonItem.TimeUnixNano = uint64(item.Timestamp.UnixNano() / 1000000)
        spanEvents := jsonItem.GetValues()
        searchSpansResult[0].Events[i] = spanEvents
    }
    return &searchSpansResult, nil
}

3. 创立告警规定

页面上创立告警规定后，保留在 sqlite 中：

sqlite> select * from rules;
1|2022-08-05 07:18:31.989405383+00:00|0|{"condition":{"compositeMetricQuery":{"builderQueries":{"A":{"queryName":"A","name":"A","formulaOnly":false,"metricName":"up","tagFilters":{"op":"AND","items":[]},"groupBy":["job"],"aggregateOperator":1,"expression":"A","disabled":false,"toggleDisable":false,"toggleDelete":false}},"promQueries":{"A":{"query":"up","stats":"","name":"A","legend":"","disabled":false}},"queryType":3},"op":"3","matchType":"1"},"labels":{"severity":"warning"},"annotations":{"description":"A new alert"},"evalWindow":"5m0s","alert":"service_down","source":"http://192.168.0.1:3301/alerts/new","ruleType":"promql_rule"}
sqlite>

创立告警规定的入口：

// app/http_handler.go
func (aH *APIHandler) RegisterRoutes(router *mux.Router) {
    ...
    router.HandleFunc("/api/v1/rules", EditAccess(aH.createRule)).Methods(http.MethodPost)
    ...
}
func (aH *APIHandler) createRule(w http.ResponseWriter, r *http.Request) {decoder := json.NewDecoder(r.Body)
    var postData map[string]string
    err := decoder.Decode(&postData)
    ...
    apiErrorObj := (*aH.reader).CreateRule(postData["data"])
    ...
    aH.respond(w, "rule successfully added")
}

创立规定的具体实现：

  • 首先，向 db 插入一条 rules 记录；
  • 而后，给 prometheus 减少一条 alertRule；
  • 最初，事务性的提交；
// app/clickhouseReader/reader.go
func (r *ClickHouseReader) CreateRule(rule string) *model.ApiError {tx, err := r.localDB.Begin()
    ...
    var lastInsertId int64
    {
        // 向 db 插入一条记录
        stmt, err := tx.Prepare(`INSERT into rules (updated_at, data) VALUES($1,$2);`)
        ...
        defer stmt.Close()
        result, err := stmt.Exec(time.Now(), rule)
        ...
        lastInsertId, _ = result.LastInsertId()
        groupName := fmt.Sprintf("%d-groupname", lastInsertId)
        // 让 prometheus 新增一条 AlertRule
        err = r.ruleManager.AddGroup(time.Duration(r.promConfig.GlobalConfig.EvaluationInterval), rule, groupName)
        if err != nil {tx.Rollback()
            return &model.ApiError{Typ: model.ErrorInternal, Err: err}
        }
    }
    err = tx.Commit()
    if err != nil {zap.S().Errorf("Error in committing transaction for INSERT to rules\n", err)
        return &model.ApiError{Typ: model.ErrorInternal, Err: err}
    }
    return nil
}

signoz 还部署了 alertmanager 实例，当 query-service 中的 prometheus 有告警触发时，将告警发送到 alertmanager。

signoz 前端查问告警时，通过查问 alertmanager 实现：

HTTP GET /api/alertmanager/alerts?active=true&inhibited=true&silenced=false

四. 特点

  • 对立存储和读取 metrics 和 traces，均存储在 clickhouse；

    • 解决传统上 metrics 存 prometheus，traces 存 Jager 的痛点；

  • 集成了 prometheus 的告警规定性能:

    • 通过 prometheus sdk 引入，并未引入 prometheus 过程；
  • prometheus 告警的去重、聚合、发送，通过 alertmanager 实例实现；

参考：

1.https://www.cnblogs.com/rongf…
2.signoz docs: https://signoz.io/docs/archit…
3.opentelmetry docs: https://opentelemetry.io/docs…

正文完
prometheus
发表至： prometheus
2023-02-10
 0
关于prometheus:prometheus-存储容量估算
关于prometheus:prometheusoperator源码分析-prometheus配置自动更新之rulesreloader三
关于prometheus:prometheus源码分析rules模块
关于prometheus:使用VictoriaMetrics监控K8S集群