kubernetes 的 TCP 数据包可视化
介绍
k8spacket是用 Golang 编写的工具，它应用 gopacket 第三方库来嗅探工作负载（传入和传出）上的 TCP 数据包。它在运行的容器网络接口上创立 TCP 侦听器。当 Kubernetes 创立一个新容器时，CNI 插件负责提供与其余容器进行通信的可能性。最常见的办法是用 linux namespace 隔离网络并用 veth pair 连贯隔离的 namespace 与网桥。除了 bridge 类型，CNI 插件还能够应用其余类型（vlan, ipvlan,macvlan)，但都为容器创立了一个网络接口，它是k8spacket 嗅探器的次要句柄。
k8spacket有助于理解 Kubernetes 集群中的 TCP 数据包流量：
- 显示集群中工作负载之间的流量
- 告诉流量在集群外路由到哪里
- 显示无关连贯敞开套接字的信息
- 显示工作负载发送 / 接管的字节数
- 计算建设连贯的工夫
- 显示整个集群中工作负载之间的网络连接拓扑
k8spacket是一个 Kubernetes API 客户端，能够将嗅探到的工作负载解析为可视化上可见的集群资源名称（Pods和 Services）。它作为DaemonSet Pod 启动，应用 hostNetwork，并监听节点上的网络接口。
k8spacket 收集 TCP 流、解决数据，应用 Node Graph API Grafana 数据源插件（详情请查看 Node Graph API 插件），通过 API 展现在 Grafana 面板。
要装置 k8spacket，须要同时装置 Grafana。上面将在Kind 装置的 k8s 集群上做演示。
增加 k8spacket 的 helm 源
[[email protected] ~]# helm repo add k8spacket https://k8spacket.github.io/k8spacket-helm-chart
"k8spacket" has been added to your repositories
[[email protected] ~]#
[[email protected] ~]#
[[email protected] ~]#
[[email protected] ~]# helm install k8spacket --namespace k8spacket k8spacket/k8spacket --create-namespace
NAME: k8spacket
LAST DEPLOYED: Thu Oct 27 18:48:30 2022
NAMESPACE: k8spacket
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export NODE_PORT=$(kubectl get --namespace k8spacket -o jsonpath="{.spec.ports[0].nodePort}" services k8spacket)
export NODE_IP=$(kubectl get nodes --namespace k8spacket -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
[[email protected] ~]#查看 pod 信息
[email protected]:~# kubectl get pod -n k8spacket
NAME READY STATUS RESTARTS AGE
k8spacket-46587 0/1 CrashLoopBackOff 2 (23s ago) 2m24s
k8spacket-9wb5q 0/1 CrashLoopBackOff 1 (6s ago) 2m24s
k8spacket-grh7k 0/1 ImagePullBackOff 0 2m24s
k8spacket-hcgg4 0/1 CrashLoopBackOff 1 (4s ago) 2m24s
k8spacket-ng99p 0/1 CrashLoopBackOff 1 (3s ago) 2m24s
k8spacket-p7hgb 0/1 CrashLoopBackOff 1 (4s ago) 2m24s
k8spacket-pk4zt 0/1 CrashLoopBackOff 1 (4s ago) 2m24s
k8spacket-tcksl 0/1 CrashLoopBackOff 1 (6s ago) 2m24s
k8spacket-tkzcc 0/1 CrashLoopBackOff 1 (8s ago) 2m24s
k8spacket-w8r5r 0/1 CrashLoopBackOff 3 (11s ago) 2m24s
[email protected]:~#
查看报错为 tunl0 问题
[[email protected] ~]# kubectl logs -n k8spacket k8spacket-46587
2022/10/27 13:35:36 Serving requests on port 6676
2022/10/27 13:35:36 Refreshing interfaces for capturing...
2022/10/27 13:35:36 Starting capture on interface "cilium_host"
2022/10/27 13:35:36 Starting capture on interface "tunl0"
2022/10/27 13:35:36 Starting capture on interface "lxc_health"
2022/10/27 13:35:36 Starting capture on interface "cilium_net"
2022/10/27 13:35:36 Starting capture on interface "lxcaaf84592af2d"
2022/10/27 13:35:36 Starting capture on interface "lxcc06519232b44"
2022/10/27 13:35:36 reading in packets
2022/10/27 13:35:36 reading in packets
2022/10/27 13:35:36 error opening pcap handle: tunl0: That device is not up
[[email protected] ~]#批改配置
# 将 charts 包拉取到本地 在进行批改信息
[[email protected] ~]# cd /tmp/
[[email protected] tmp]# helm fetch k8spacket/k8spacket
[[email protected] tmp]# tar -zxf k8spacket-0.1.3.tgz
[[email protected] tmp]# cd k8spacket
# 设置配置为
command: "ip address | grep @ | grep -v tunl0 | sed -E's/.* (\\w+)@.*/\\1/'| tr'\\n'',' | sed 's/.$//'"
# 残缺配置如下
[[email protected] k8spacket]# vim values.yaml
[[email protected] k8spacket]# cat values.yaml
replicaCount: 1
affinity: {}
image:
repository: docker.io/k8spacket/k8spacket
pullPolicy: IfNotPresent
serviceAccount:
create: true
# Annotations to add to the service account
annotations: {}
clusterRole:
create: true
nodeSelector: {}
podAnnotations: {}
priorityClassName: ""
podSecurityContext:
runAsUser: 1000
securityContext:
allowPrivilegeEscalation: true
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
service:
type: ClusterIP
port: 8080
nodePort:
resources:
requests:
memory: "1000Mi"
cpu: "250m"
limits:
memory: "1500Mi"
cpu: "500m"
tolerations: []
k8sPacket:
metrics:
## Hide source port when 'true' (set to string value 'dynamic' instead of decimal real source port) for Prometheus metrics cardinality reasons
hideSourcePort: true
reverseLookup:
## Reverse lookup db file based on GeoLite2 Free Geolocation Data
## See: https://dev.maxmind.com/geoip/geolite2-free-geolocation-data?lang=en
geoipDBPath: "/home/k8spacket/GeoLite2-City.mmdb"
## Whois result match regexp
whoisRegexp: "(?:OrgName:|org-name:)\\s*(.*)"
tcp:
listener:
port: 6676
interfaces:
## Command to achieve containers network interfaces
command: "ip address | grep @ | grep -v tunl0 | sed -E's/.* (\\w+)@.*/\\1/'| tr'\\n'',' | sed 's/.$//'"
## How often refresh the list of network interfaces to listen
refreshPeriod: "10s"
assembler:
## See: https://pkg.go.dev/github.com/google/gopacket/tcpassembly#AssemblerOptions
maxPagesPerConnection: 50
maxPagesTotal: 50
## Every (periodDuration) seconds, flush connections that haven't seen activity in the past (closeOlderThanDuration) seconds.
flushing:
periodDuration: "10s"
closeOlderThanDuration: "20s"
[[email protected] k8spacket]#
重新安装 k8spacket
[[email protected] k8spacket]# helm uninstall k8spacket -n k8spacket
[[email protected] k8spacket]# helm install k8spacket --namespace k8spacket . --create-namespace
NAME: k8spacket
LAST DEPLOYED: Thu Oct 27 21:47:38 2022
NAMESPACE: k8spacket
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export NODE_PORT=$(kubectl get --namespace k8spacket -o jsonpath="{.spec.ports[0].nodePort}" services k8spacket)
export NODE_IP=$(kubectl get nodes --namespace k8spacket -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
[[email protected] k8spacket]#查看验证
[[email protected] ~]# kubectl get pod -n k8spacket -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
k8spacket-8kxnx 1/1 Running 0 4m27s 192.168.1.66 k8s-node-3 <none> <none>
k8spacket-cqpks 1/1 Running 0 4m27s 192.168.1.70 k8s-node-6 <none> <none>
k8spacket-h72fc 1/1 Running 0 4m27s 192.168.1.67 k8s-node-4 <none> <none>
k8spacket-jkxg9 1/1 Running 0 4m27s 192.168.1.75 k8s-node-7 <none> <none>
k8spacket-kgpql 1/1 Running 0 4m27s 192.168.1.62 k8s-master-2 <none> <none>
k8spacket-lf9br 1/1 Running 0 4m27s 192.168.1.61 k8s-master-1 <none> <none>
k8spacket-mcbv5 1/1 Running 0 4m27s 192.168.1.68 k8s-node-5 <none> <none>
k8spacket-ndlzt 1/1 Running 0 4m27s 192.168.1.64 k8s-node-1 <none> <none>
k8spacket-vfg2x 1/1 Running 0 4m27s 192.168.1.63 k8s-master-3 <none> <none>
k8spacket-vvwtr 1/1 Running 0 4m27s 192.168.1.65 k8s-node-2 <none> <none>
[[email protected] ~]#
[[email protected] ~]# kubectl get svc -n k8spacket -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
k8spacket ClusterIP 10.110.30.53 <none> 8080/TCP 4m31s app.kubernetes.io/instance=k8spacket,app.kubernetes.io/name=k8spacket
[[email protected] ~]#
拜访验证
[[email protected] ~]# curl 10.110.30.53:8080/metrics装置 dashboards 配置
[[email protected] ~]# cd /tmp/
[[email protected] tmp]#
[[email protected] tmp]# wget https://github.com/k8spacket/k8spacket/archive/refs/heads/master.zip
[[email protected] tmp]# unzip master.zip
[[email protected] tmp]#
[[email protected] tmp]# cd k8spacket-master
[[email protected] k8spacket-master]#
[[email protected] k8spacket-master]# kubectl apply -f ./dashboards/
configmap/k8spacket-logs-dashboard created
configmap/k8spacket-metrics-dashboard created
configmap/k8spacket-node-graph-dashboard created
[[email protected] k8spacket-master]#装置 Grafana
[[email protected] tmp]# helm repo add grafana https://grafana.github.io/helm-charts
"grafana" has been added to your repositories
[[email protected] tmp]# helm fetch grafana/grafana
[[email protected] tmp]#
[[email protected] tmp]# tar -zxf grafana-6.43.1.tgz 批改 Grafana 配置内容
[[email protected] tmp]# cd grafana/
[[email protected] grafana]#
[[email protected] grafana]# vim values.yaml
批改以下配置内容
persistence:
type: pvc
enabled: true
env:
GF_INSTALL_PLUGINS: hamedkarbasi93-nodegraphapi-datasource
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- name: 'default'
orgId: 1
folder: ''
type: file
disableDeletion: false
editable: true
options:
path: /var/lib/grafana/dashboards/default
dashboardsConfigMaps:
default: k8spacket-node-graph-dashboard
datasources:
nodegraphapi-plugin-datasource.yaml:
apiVersion: 1
datasources:
- name: "Node Graph API"
jsonData:
url: "http://k8spacket.k8spacket.svc.cluster.local:8080"
access: "proxy"
basicAuth: false
isDefault: false
readOnly: false
type: "hamedkarbasi93-nodegraphapi-datasource"
typeLogoUrl: "public/plugins/hamedkarbasi93-nodegraphapi-datasource/img/logo.svg"
typeName: "node-graph-plugin"
orgId: 1
version: 1
装置 Grafana
[[email protected] grafana]# helm install grafana -f values.yaml ./
NAME: grafana
LAST DEPLOYED: Thu Oct 27 22:11:27 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:
kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:
grafana.default.svc.cluster.local
Get the Grafana URL to visit by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 3000
3. Login with the password from step 1 and the username: admin
[[email protected] grafana]# 批改为 NodePort
[[email protected] grafana]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo-a ClusterIP 10.108.160.226 <none> 8080/TCP 6d9h
echo-b NodePort 10.108.200.169 <none> 8080:31414/TCP 6d9h
echo-b-headless ClusterIP None <none> 8080/TCP 6d9h
echo-b-host-headless ClusterIP None <none> <none> 6d9h
grafana ClusterIP 10.101.109.183 <none> 80/TCP 4m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d9h
[[email protected] grafana]#
[[email protected] grafana]# kubectl edit svc grafana
service/grafana edited
[[email protected] grafana]#
[[email protected] grafana]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echo-a ClusterIP 10.108.160.226 <none> 8080/TCP 6d9h
echo-b NodePort 10.108.200.169 <none> 8080:31414/TCP 6d9h
echo-b-headless ClusterIP None <none> 8080/TCP 6d9h
echo-b-host-headless ClusterIP None <none> <none> 6d9h
grafana NodePort 10.101.109.183 <none> 80:30973/TCP 4m37s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d9h
[[email protected] grafana]#
查看 Grafana 明码
[[email protected] grafana]# kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
9O1Hd9LOqJ6LKUjZTlEWAGeXRitr0CZd4p6fr00J
[[email protected] grafana]# 拜访地址
拜访
http://192.168.1.61:30973/
增加 Node Graph API 插件
http://192.168.1.61:30973/plugins
查看 Node Graph API 数据收集源
http://192.168.1.61:30973/datasources对于
https://www.oiox.cn/
https://www.oiox.cn/index.php…
CSDN、GitHub、51CTO、知乎、开源中国、思否、掘金、简书、华为云、阿里云、腾讯云、哔哩哔哩、今日头条、新浪微博、集体博客
全网可搜《小陈运维》
文章次要公布于微信公众号