关于prometheus:Prometheus通过httphttps拉取target的源码分析

prometheus 通过 HTTP 或者 HTTPS 拉取 target 的 metrics 信息。

alertmanager 通过 HTTP 拉取：

http://10.233.96.91:9090/metrics

// 配置
scrape_configs:
- job_name: monitoring/alertmanager/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  ......

kube-state-metrics 通过 HTTPS 拉取：

https://10.233.96.90:8443/metrics

// 配置
- job_name: monitoring/kube-state-metrics/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: https
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true

能够看到，对于 https 的 target 来说，配置：

• scheme=https;
• 减少了 bearer_token_file 的配置；
• 减少了 tls_config 的配置；

初始化 http client

结构 client 时，应用配置 cfg.HTTPClientConfig:

//scrape/scrape.go
func newScrapePool(cfg *config.ScrapeConfig, app storage.Appendable, jitterSeed uint64, logger log.Logger) (*scrapePool, error) {
    ......
    client, err := config_util.NewClientFromConfig(cfg.HTTPClientConfig, cfg.JobName, false)
    ....
    sp := &scrapePool{
        cancel:        cancel,
        appendable:    app,
        config:        cfg,
        client:        client,
        activeTargets: map[uint64]*Target{},
        loops:         map[uint64]loop{},
        logger:        logger,
    }
    .....
}

HTTPClientConfig 构造带了 bearToken 和 tlsconfig 信息：

type HTTPClientConfig struct {
    // The HTTP basic authentication credentials for the targets.
    BasicAuth *BasicAuth `yaml:"basic_auth,omitempty"`
    // The bearer token for the targets.
    BearerToken Secret `yaml:"bearer_token,omitempty"`
    // The bearer token file for the targets.
    BearerTokenFile string `yaml:"bearer_token_file,omitempty"`
    // HTTP proxy server to use to connect to the targets.
    ProxyURL URL `yaml:"proxy_url,omitempty"`
    // TLSConfig to use to connect to the targets.
    TLSConfig TLSConfig `yaml:"tls_config,omitempty"`
}

对于 https 的 client，将 tlsconfig 和 bearer token 配置到 client

// NewRoundTripperFromConfig returns a new HTTP RoundTripper configured for the
// given config.HTTPClientConfig. The name is used as go-conntrack metric label.
func NewRoundTripperFromConfig(cfg HTTPClientConfig, name string, disableKeepAlives bool) (http.RoundTripper, error) {newRT := func(tlsConfig *tls.Config) (http.RoundTripper, error) {
        // The only timeout we care about is the configured scrape timeout.
        // It is applied on request. So we leave out any timings here.
        var rt http.RoundTripper = &http.Transport{Proxy:               http.ProxyURL(cfg.ProxyURL.URL),
            MaxIdleConns:        20000,
            MaxIdleConnsPerHost: 1000, // see https://github.com/golang/go/issues/13801
            DisableKeepAlives:   disableKeepAlives,
            TLSClientConfig:     tlsConfig,
            DisableCompression:  true,
            ......
        }
        // TODO: use ForceAttemptHTTP2 when we move to Go 1.13+.
        err := http2.ConfigureTransport(rt.(*http.Transport))

        // If a bearer token is provided, create a round tripper that will set the
        // Authorization header correctly on each request.
        if len(cfg.BearerToken) > 0 {rt = NewBearerAuthRoundTripper(cfg.BearerToken, rt)
        } else if len(cfg.BearerTokenFile) > 0 {rt = NewBearerAuthFileRoundTripper(cfg.BearerTokenFile, rt)
        }
        ...
        // Return a new configured RoundTripper.
        return rt, nil
    }

    tlsConfig, err := NewTLSConfig(&cfg.TLSConfig)
    if err != nil {return nil, err}
    ...
    return newTLSRoundTripper(tlsConfig, cfg.TLSConfig.CAFile, newRT)
}

对于 bearer token，被增加到 http 的 header 中（Authorization: Bearer $token）

// NewBearerAuthRoundTripper adds the provided bearer token to a request unless the authorization
// header has already been set.
func NewBearerAuthRoundTripper(token Secret, rt http.RoundTripper) http.RoundTripper {return &bearerAuthRoundTripper{token, rt}
}

func (rt *bearerAuthRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {if len(req.Header.Get("Authorization")) == 0 {req = cloneRequest(req)
        req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", string(rt.bearerToken)))
    }
    return rt.rt.RoundTrip(req)
}

能够 curl 命令看一下：

curl -k https://172.160.230.217:8443/metrics -H "Authorization: Bearer eyJhbGciOiJSUzI1NiI...g"

应用 http client 拉取

结构好 http client，拉取的过程就很简略了：HTTP GET /metrics

// scrape/scrape.go
func (s *targetScraper) scrape(ctx context.Context, w io.Writer) (string, error) {
    if s.req == nil {req, err := http.NewRequest("GET", s.URL().String(), nil)
        if err != nil {return "", err}
        req.Header.Add("Accept", acceptHeader)
        req.Header.Add("Accept-Encoding", "gzip")
        req.Header.Set("User-Agent", userAgentHeader)
        req.Header.Set("X-Prometheus-Scrape-Timeout-Seconds", fmt.Sprintf("%f", s.timeout.Seconds()))

        s.req = req
    }

    resp, err := s.client.Do(s.req.WithContext(ctx))
    ......

    if resp.Header.Get("Content-Encoding") != "gzip" {_, err = io.Copy(w, resp.Body)
        if err != nil {return "", err}
        return resp.Header.Get("Content-Type"), nil
    }

    .....
    return resp.Header.Get("Content-Type"), nil
}