prometheus 通过 HTTP 或者 HTTPS 拉取 target 的 metrics 信息。
alertmanager 通过 HTTP 拉取:
http://10.233.96.91:9090/metrics
// 配置
scrape_configs:
- job_name: monitoring/alertmanager/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
......
kube-state-metrics 通过 HTTPS 拉取:
https://10.233.96.90:8443/metrics
// 配置
- job_name: monitoring/kube-state-metrics/0
honor_labels: true
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /metrics
scheme: https
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
insecure_skip_verify: true
能够看到,对于 https 的 target 来说,配置:
- scheme=https;
- 减少了 bearer_token_file 的配置;
- 减少了 tls_config 的配置;
初始化 http client
结构 client 时,应用配置 cfg.HTTPClientConfig:
//scrape/scrape.go
func newScrapePool(cfg *config.ScrapeConfig, app storage.Appendable, jitterSeed uint64, logger log.Logger) (*scrapePool, error) {
......
client, err := config_util.NewClientFromConfig(cfg.HTTPClientConfig, cfg.JobName, false)
....
sp := &scrapePool{
cancel: cancel,
appendable: app,
config: cfg,
client: client,
activeTargets: map[uint64]*Target{},
loops: map[uint64]loop{},
logger: logger,
}
.....
}
HTTPClientConfig 构造带了 bearToken 和 tlsconfig 信息:
type HTTPClientConfig struct {
// The HTTP basic authentication credentials for the targets.
BasicAuth *BasicAuth `yaml:"basic_auth,omitempty"`
// The bearer token for the targets.
BearerToken Secret `yaml:"bearer_token,omitempty"`
// The bearer token file for the targets.
BearerTokenFile string `yaml:"bearer_token_file,omitempty"`
// HTTP proxy server to use to connect to the targets.
ProxyURL URL `yaml:"proxy_url,omitempty"`
// TLSConfig to use to connect to the targets.
TLSConfig TLSConfig `yaml:"tls_config,omitempty"`
}
对于 https 的 client,将 tlsconfig 和 bearer token 配置到 client
// NewRoundTripperFromConfig returns a new HTTP RoundTripper configured for the
// given config.HTTPClientConfig. The name is used as go-conntrack metric label.
func NewRoundTripperFromConfig(cfg HTTPClientConfig, name string, disableKeepAlives bool) (http.RoundTripper, error) {newRT := func(tlsConfig *tls.Config) (http.RoundTripper, error) {
// The only timeout we care about is the configured scrape timeout.
// It is applied on request. So we leave out any timings here.
var rt http.RoundTripper = &http.Transport{Proxy: http.ProxyURL(cfg.ProxyURL.URL),
MaxIdleConns: 20000,
MaxIdleConnsPerHost: 1000, // see https://github.com/golang/go/issues/13801
DisableKeepAlives: disableKeepAlives,
TLSClientConfig: tlsConfig,
DisableCompression: true,
......
}
// TODO: use ForceAttemptHTTP2 when we move to Go 1.13+.
err := http2.ConfigureTransport(rt.(*http.Transport))
// If a bearer token is provided, create a round tripper that will set the
// Authorization header correctly on each request.
if len(cfg.BearerToken) > 0 {rt = NewBearerAuthRoundTripper(cfg.BearerToken, rt)
} else if len(cfg.BearerTokenFile) > 0 {rt = NewBearerAuthFileRoundTripper(cfg.BearerTokenFile, rt)
}
...
// Return a new configured RoundTripper.
return rt, nil
}
tlsConfig, err := NewTLSConfig(&cfg.TLSConfig)
if err != nil {return nil, err}
...
return newTLSRoundTripper(tlsConfig, cfg.TLSConfig.CAFile, newRT)
}
对于 bearer token,被增加到 http 的 header 中(Authorization: Bearer $token)
// NewBearerAuthRoundTripper adds the provided bearer token to a request unless the authorization
// header has already been set.
func NewBearerAuthRoundTripper(token Secret, rt http.RoundTripper) http.RoundTripper {return &bearerAuthRoundTripper{token, rt}
}
func (rt *bearerAuthRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {if len(req.Header.Get("Authorization")) == 0 {req = cloneRequest(req)
req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", string(rt.bearerToken)))
}
return rt.rt.RoundTrip(req)
}
能够 curl 命令看一下:
curl -k https://172.160.230.217:8443/metrics -H "Authorization: Bearer eyJhbGciOiJSUzI1NiI...g"
应用 http client 拉取
结构好 http client,拉取的过程就很简略了:HTTP GET /metrics
// scrape/scrape.go
func (s *targetScraper) scrape(ctx context.Context, w io.Writer) (string, error) {
if s.req == nil {req, err := http.NewRequest("GET", s.URL().String(), nil)
if err != nil {return "", err}
req.Header.Add("Accept", acceptHeader)
req.Header.Add("Accept-Encoding", "gzip")
req.Header.Set("User-Agent", userAgentHeader)
req.Header.Set("X-Prometheus-Scrape-Timeout-Seconds", fmt.Sprintf("%f", s.timeout.Seconds()))
s.req = req
}
resp, err := s.client.Do(s.req.WithContext(ctx))
......
if resp.Header.Get("Content-Encoding") != "gzip" {_, err = io.Copy(w, resp.Body)
if err != nil {return "", err}
return resp.Header.Get("Content-Type"), nil
}
.....
return resp.Header.Get("Content-Type"), nil
}