共计 24975 个字符，预计需要花费 63 分钟才能阅读完成。
Calico 简介
Calico 是一种容器之间互通的网络计划。在虚拟化平台中，比方 OpenStack、Docker 等都须要实现 workloads 之间互连，但同时也须要对容器做隔离管制，就像在 Internet 中的服务仅凋谢 80 端口、私有云的多租户一样，提供隔离和管控机制。而在少数的虚拟化平台实现中，通常都应用二层隔离技术来实现容器的网络，这些二层的技术有一些弊病，比方须要依赖 VLAN、bridge 和隧道等技术，其中 bridge 带来了复杂性，vlan 隔离和 tunnel 隧道则耗费更多的资源并对物理环境有要求，随着网络规模的增大，整体会变得越加简单。咱们尝试把 Host 当作 Internet 中的路由器，同样应用 BGP 同步路由，并应用 iptables 来做平安拜访策略，最终设计出了 Calico 计划。
设计思维：Calico 不应用隧道或 NAT 来实现转发，而是奇妙的把所有二三层流量转换成三层流量，并通过 host 上路由配置实现跨 Host 转发
设计劣势：
1. 更优的资源利用
二层网络通讯须要依赖播送音讯机制，播送音讯的开销与 host 的数量呈指数级增长，Calico 应用的三层路由办法，则齐全克制了二层播送，缩小了资源开销。
2. 可扩展性
Calico 应用与 Internet 相似的计划，Internet 的网络比任何数据中心都大，Calico 同样人造具备可扩展性。
3. 简略而更容易 debug
因为没有隧道，意味着 workloads 之间门路更短更简略，配置更少，在 host 上更容易进行 debug 调试。
4. 更少的依赖
Calico 仅依赖三层路由可达。
5. 可适配性
Calico 较少的依赖性使它能适配所有 VM、Container、白盒或者混合环境场景。
Calico 网络 Node 之间通信网络
IPIP(可跨网段通信)
从字面来了解，就是把一个 IP 数据包又套在一个 IP 包里，即把 IP 层封装到 IP 层的一个 tunnel。它的作用其实基本上就相当于一个基于 IP 层的网桥！一般来说，一般的网桥是基于 mac 层的，基本不需 IP，而这个 ipip 则是通过两端的路由做一个 tunnel，把两个原本不通的网络通过点对点连接起来。
相似 vxlan 但封装开销比 vxlan 小 效率绝对更高一些，但安全性也更差
Vxlan(可跨网段通信)
与 Flannel Vxlan 原理雷同
BGP(二层网络通信)
边界网关协定（Border Gateway Protocol, BGP）是互联网上一个外围的去中心化自治路由协定。它通过保护 IP 路由表或‘前缀’表来实现自治零碎（AS）之间的可达性，属于矢量路由协定。BGP 不应用传统的外部网关协定（IGP）的指标，而应用基于门路、网络策略或规定集来决定路由。因而，它更适宜被称为矢量性协定，而不是路由协定。BGP，艰深的讲就是讲接入到机房的多条线路（如电信、联通、挪动等）交融为一体，实现多线单 IP，BGP 机房的长处：服务器只须要设置一个 IP 地址，最佳拜访路由是由网络上的骨干路由器依据路由跳数与其它技术指标来确定的，不会占用服务器的任何零碎
实际上，Calico 我的项目提供的 BGP 网络解决方案，与 Flannel 的 host-gw 模式简直一样。也就是说，Calico 也是基于路由表实现容器数据包转发，但不同于 Flannel 应用 flanneld 过程来保护路由信息的做法，而 Calico 我的项目应用 BGP 协定来主动保护整个集群的路由信息。
部署举荐计划:
BGP+Vxlan
其中 BGP 在官网的举荐计划中 以 50 个节点为界区别了不同规模应用不同的部署计划
- 小规模网络:BGP peer 一对一网络：每个节点都是有 N - 1 条路由，小型网络实用，当节点数 N 变多时，路由表更新及 AIP-SERVER 都须要接受很大的压力 相似网络拓扑构造中的 网状拓扑构造
- 大规模网络:BGP Reflector 路由反射器：抉择一到多个节点做为 Reflector，所有节点路由都汇总给 Reflector，所有节点都路由都指向 Reflector，适宜大型网络，相似网络拓扑构造中的星型网络
Calico 网络模型次要工作组件
- Felix：运行在每一台 Host 的 agent 过程，次要负责网络接口治理和监听、路由、ARP 治理、ACL 治理和同步、状态上报等。
- etcd：分布式键值存储，次要负责网络元数据一致性，确保 Calico 网络状态的准确性，能够与 kubernetes 共用；
- BGP Client（BIRD）：Calico 为每一台 Host 部署一个 BGP Client，应用 BIRD 实现，BIRD 是一个独自的继续倒退的我的项目，实现了泛滥动静路由协定比方 BGP、OSPF、RIP 等。在 Calico 的角色是监听 Host 上由 Felix 注入的路由信息，而后通过 BGP 协定播送通知残余 Host 节点，从而实现网络互通。
- BGP Route Reflector：在大型网络规模中，如果仅仅应用 BGP client 造成 mesh 全网互联的计划就会导致规模限度，因为所有节点之间俩俩互联，须要 N^2 个连贯，为了解决这个规模问题，能够采纳 BGP 的 Router Reflector 的办法，使所有 BGP Client 仅与特定 RR 节点互联并做路由同步，从而大大减少连接数。
Calico 有两种运行形式:
- 是让 calico/node 独立运行于 Kubernetes 集群之外，但 calico/kube-controllers 仍然须要以 Pod 资源运行中集群之上;
- 是以 CNI 插件形式配置 Calico 齐全托管运行于 Kubernetes 集群之上，相似于咱们后面已经部署托管 Flannel 网络插件的形式。
对于后一种形式,Calico 提供了在线的部署清单，它别离为 50 节点及以下规模和 50 节点以上规模的 Kubernetes 集群应用 Kubernetes API 作为 Dabastore 提供了不同的配置清单，也为应用独立的 etcd 集群提供了专用配置清单。但这 3 种类型的配置清单中,Calico 默认启用的是基于 IPIP 隧道的叠加网络，因此它会在所有流量上应用 IPIP 隧道而不是 BGP 路由。以下配置定义在部署清单中 DaemonSet/calico-node 资源的 Pod 模板中的 calico-node 容器之上。
配置选项
在 IPv4 类型的地址池上启用的 IPIP 及其类型，反对 3 种可用值
Always(全局流量)、Cross-SubNet(跨子网流量) 和 Never3 种可用值
- name: CALICO_IPV4POOL_IPIP
value: “Always” - 是否在 IPV4 地址池上启用 VXLAN 隧道协定，取值及意义与 Flannel 的 VXLAN 后端雷同; 但在全局流量启用 VXLAN 时将齐全不再须要 BGP 网络，倡议将相干的组件禁用
- name: CALICO_ IPV4POOL_VXLAN
value: “Never” - 须要留神的是，Calico 调配的地址池须要同 Ktbernetes 集群的 Pod 网络的定义保持一致。Pod 网络通常由 kubeadm init 初始化集群时应用 –pod-network-cidr 选项指定的网络，而 Calico 在其默认的配置清单中默认应用 192.168.0.0/16 作为 Pod 网络，因此部署 Kubernetes 集群时应该布局好要应用的网络地址，并设定此二者相匹配。对于已经应用了 flannel 的默认的 10.244.0.0/16 网络的环境而言, 咱们也能够抉择批改资源清单中的定义，从而将其批改为其余网络地址, 它定义在 DaemonSet/calico-node 资源的 Pod 模板中的 calico-node 容器之上。
官网链接：
https://docs.projectcalico.or…
示例 1: 装置 calico
wget https://docs.projectcalico.org/manifests/calico.yaml
[root@k8s-master ~]# cd /etc/kubernetes/manifests/
[root@k8s-master manifests]# cat kube-controller-manager.yaml
...
System Info:
Machine ID: 32599e2a74704b2e95443e24ea15d4f6
System UUID: 34979a62-16de-4287-b149-2d4c2d8a70fb
Boot ID: f31de60e-4f89-4553-ba7a-99a46d049936
Kernel Version: 5.4.109-1.el7.elrepo.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.7
Kubelet Version: v1.19.9
Kube-Proxy Version: v1.19.9
PodCIDR: 10.244.1.0/24 #第个节点的地址块都是由 K8S 调配
PodCIDRs: 10.244.1.0/24
Non-terminated Pods: (17 in total)
[root@k8s-master ~]# kubectl describe node k8s-node1
System Info:
Machine ID: 32599e2a74704b2e95443e24ea15d4f6
System UUID: 34979a62-16de-4287-b149-2d4c2d8a70fb
Boot ID: f31de60e-4f89-4553-ba7a-99a46d049936
Kernel Version: 5.4.109-1.el7.elrepo.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.7
Kubelet Version: v1.19.9
Kube-Proxy Version: v1.19.9
PodCIDR: 10.244.1.0/24 #每个 Node Pod 都是由 K8S 调配 IP
PodCIDRs: 10.244.1.0/24
[root@k8s-master Network]# vim calico.yaml
...
"ipam": {
"type": "host-local",
"subnet": "usePodCidr" #应用 k8s ipam 插件调配地址
},
"policy": {"type": "k8s"},
...
- name: FELIX_WIREGUARDMTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16" #为了和之前的 flannel 10.244.0.0/16 适配
- name: CALICO_IPV4POOL_BLOCK_SIZE #增加这一行批改默认块大小
value: "24"
- name: USE_POD_CIDR #应用 K8S 的调配的 IP 地址，不然 calico 和 K8S 调配的地址会不一样
value: "true"- 装置 calico
[root@k8s-master plugin]# kubectl delete -f kube-flannel.yml
podsecuritypolicy.policy "psp.flannel.unprivileged" deleted
clusterrole.rbac.authorization.k8s.io "flannel" deleted
clusterrolebinding.rbac.authorization.k8s.io "flannel" deleted
serviceaccount "flannel" deleted
configmap "kube-flannel-cfg" deleted
daemonset.apps "kube-flannel-ds" deleted
[root@k8s-master plugin]# kubectl apply -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created- calico 几个组件
[root@k8s-master ~]# ps aux|grep calico
root 10867 0.0 0.1 112816 2156 pts/1 S+ 13:51 0:00 grep --color=auto calico
root 20680 0.0 2.3 1215184 35216 ? Sl 10:27 0:06 calico-node -allocate-tunnel-addrs
root 20681 0.0 2.1 1215184 32672 ? Sl 10:27 0:06 calico-node -monitor-addresses
root 20682 2.4 3.3 1510624 51636 ? Sl 10:27 4:54 calico-node -felix
root 20683 0.0 2.3 1657832 35496 ? Sl 10:27 0:09 calico-node -confd
root 20686 0.0 2.0 1214928 31628 ? Sl 10:27 0:05 calico-node -monitor-token- 因为 calico 并没有应用 k8s 的 ipam 调配 IP, 所以节点会有 2 个 IP, 一个是 K8S 调配的 IP 一个是 calico 调配的 IP
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.54.2 0.0.0.0 UG 101 0 0 eth4
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.4.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
192.168.12.0 192.168.4.172 255.255.255.0 UG 0 0 0 tunl0 #能够看到 tunl0 的路由信息
192.168.51.0 192.168.4.173 255.255.255.0 UG 0 0 0 tunl0 #同时能够看到节点的 IP 不像之前肯定是间断的
192.168.54.0 0.0.0.0 255.255.255.0 U 101 0 0 eth4
192.168.113.0 192.168.4.171 255.255.255.0 UG 0 0 0 tunl0 #隧道接口
192.168.237.0 0.0.0.0 255.255.255.0 U 0 0 0 *
192.168.237.1 0.0.0.0 255.255.255.255 UH 0 0 0 cali7c0fb624285
192.168.237.2 0.0.0.0 255.255.255.255 UH 0 0 0 caliedaf285d4ef
192.168.237.3 0.0.0.0 255.255.255.255 UH 0 0 0 cali854da94d42a
[root@k8s-master calico]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.170 metric 100
192.168.12.0/24 via 192.168.4.172 dev tunl0 proto bird onlink #能够看到 tunl0 的路由信息
192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.170 metric 101
192.168.113.0/24 via 192.168.4.171 dev tunl0 proto bird onlink
blackhole 192.168.237.0/24 proto bird
192.168.237.1 dev cali7c0fb624285 scope link
192.168.237.2 dev caliedaf285d4ef scope link
192.168.237.3 dev cali854da94d42a scope link - 192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink 上面的路由表能够看到 calico 会为每个节点调配网络地址段 并不是应用节点的网络地址
[root@k8s-node1 ~]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.171 metric 100
192.168.12.0/24 via 192.168.4.172 dev tunl0 proto bird onlink
192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.171 metric 101
blackhole 192.168.113.0/24 proto bird #黑洞 代表本人网段
192.168.237.0/24 via 192.168.4.170 dev tunl0 proto bird onlink- 查看目前工作模式
[root@k8s-master calico]# kubectl api-resources
NAME SHORTNAMES APIGROUP NAMESPACED KIND
bgpconfigurations crd.projectcalico.org false BGPConfiguration
bgppeers crd.projectcalico.org false BGPPeer
blockaffinities crd.projectcalico.org false BlockAffinity
clusterinformations crd.projectcalico.org false ClusterInformation
felixconfigurations crd.projectcalico.org false FelixConfiguration
globalnetworkpolicies crd.projectcalico.org false GlobalNetworkPolicy
globalnetworksets crd.projectcalico.org false GlobalNetworkSet
hostendpoints crd.projectcalico.org false HostEndpoint
ipamblocks crd.projectcalico.org false IPAMBlock
ipamconfigs crd.projectcalico.org false IPAMConfig
ipamhandles crd.projectcalico.org false IPAMHandle
ippools crd.projectcalico.org false IPPool #calico 地址池
kubecontrollersconfigurations crd.projectcalico.org false KubeControllersConfiguration
networkpolicies crd.projectcalico.org true NetworkPolicy
networksets crd.projectcalico.org true NetworkSet
[root@k8s-master calico]# kubectl get ippools -o yaml
....
spec:
blockSize: 24 #掩码长度
cidr: 192.168.0.0/16 #地址池
ipipMode: Always #能够看到目前为 ipip 模式
natOutgoing: true
nodeSelector: all()- 拜访抓包
[root@k8s-master PodControl]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deployment-demo-fb544c5d8-r7pc8 1/1 Running 0 8m3s 192.168.51.1 k8s-node3 <none> <none>
deployment-demo-fb544c5d8-splfr 1/1 Running 0 8m3s 192.168.12.1 k8s-node2 <none> <none>
[root@k8s-master PodControl]# kubectl exec deployment-demo-fb544c5d8-r7pc8 -it -- /bin/sh
[root@deployment-demo-fb544c5d8-r7pc8 /]# ifconfig
eth0 Link encap:Ethernet HWaddr 16:96:97:3F:F3:C5
inet addr:192.168.51.1 Bcast:192.168.51.1 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
[root@k8s-node2 ~]# tcpdump -i eth0 -nn ip host 192.168.4.172 and host 192.168.4.173
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:48:24.421003 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [S], seq 3259804851, win 64800, options [mss 1440,sackOK,TS val 2008488248 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
11:48:24.421093 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [S.], seq 3234480084, ack 3259804852, win 64260, options [mss 1440,sackOK,TS val 1053230437 ecr 2008488248,nop,wscale 7], length 0 (ipip-proto-4) #能够看到 (ipip-proto-4) 为 IPIP 模式
11:48:24.422305 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 2008488250 ecr 1053230437], length 0 (ipip-proto-4)
11:48:24.422308 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [P.], seq 1:77, ack 1, win 507, options [nop,nop,TS val 2008488250 ecr 1053230437], length 76: HTTP: GET / HTTP/1.1 (ipip-proto-4)
11:48:24.422554 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [.], ack 77, win 502, options [nop,nop,TS val 1053230439 ecr 2008488250], length 0 (ipip-proto-4)
11:48:24.431688 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [P.], seq 1:18, ack 77, win 502, options [nop,nop,TS val 1053230447 ecr 2008488250], length 17: HTTP: HTTP/1.0 200 OK (ipip-proto-4)
11:48:24.432638 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [FP.], seq 18:276, ack 77, win 502, options [nop,nop,TS val 1053230449 ecr 2008488250], length 258: HTTP (ipip-proto-4)
11:48:24.433660 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [.], ack 18, win 507, options [nop,nop,TS val 2008488261 ecr 1053230447], length 0 (ipip-proto-4)
11:48:24.437531 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [F.], seq 77, ack 277, win 505, options [nop,nop,TS val 2008488261 ecr 1053230449], length 0 (ipip-proto-4)
11:48:24.437775 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [.], ack 78, win 502, options [nop,nop,TS val 1053230454 ecr 2008488261], length 0 (ipip-proto-4)
IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436 - 能够看到默认为 ipip 模式 也是通过封装在转发的 和 Flannel 很相似, 但绝对 Flannel 通过虚构网桥 CNI calico 间接内核 (内核的路由由 kube-proxy 或 IPVS 生成) 到在由 tunl0 传输想对 Flannel 少了一层交换机替换的过程, 性能相比 Flannel 会快一些 但这并不是 calico 最佳的模式
calicoctl 命令装置与应用
**calicoctl 装置的 2 种形式
第 1 种形式 calicoctl**
https://docs.projectcalico.or…
- 几种形式运行 calicoctl 罕用形式 1: 间接下载 2 进制 calicoctl 间接运行
[root@k8s-master ~]# curl -o calicoctl -O -L "https://github.com/projectcalico/calicoctl/releases/download/v3.20.0/calicoctl"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 615 100 615 0 0 498 0 0:00:01 0:00:01 --:--:-- 504
100 43.2M 100 43.2M 0 0 518k 0 0:01:25 0:01:25 --:--:-- 920k
[root@k8s-master ~]# mv calicoctl /usr/bin/
[root@k8s-master ~]# chmod +x /usr/bin/calicoctl
[root@k8s-master ~]# calicoctl --help
Usage:
calicoctl [options] <command> [<args>...]
create Create a resource by file, directory or stdin.
replace Replace a resource by file, directory or stdin.
apply Apply a resource by file, directory or stdin. This creates a resource
if it does not exist, and replaces a resource if it does exists.
patch Patch a pre-exisiting resource in place.
delete Delete a resource identified by file, directory, stdin or resource type and
name.
get Get a resource identified by file, directory, stdin or resource type and
name.
label Add or update labels of resources.
convert Convert config files between different API versions.
ipam IP address management.
node Calico node management.
version Display the version of this binary.
export Export the Calico datastore objects for migration
import Import the Calico datastore objects for migration
datastore Calico datastore management.
- calicoctl 命令应用
- calicoctl 默认会读取 ~/.kube/ 下文件加载认证信息, 也能够通过配置文件指定认证信息地位
[root@k8s-master calico]# mkdir /etc/calico/^C
[root@k8s-master calico]# cd /etc/calico/
[root@k8s-master calico]# cat calicoctl.cfg
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
datastoreType: "kubernetes"
kubeconfig: "/etc/kubernetes/admin.conf" #指定 conf 门路
[root@k8s-master calico]#
[root@k8s-master calico]# kubectl get ippools
NAME AGE
default-ipv4-ippool 23h
[root@k8s-master calico]# calicoctl get ippool #能够用 calicoctl 间接拜访 calico 资源
NAME CIDR SELECTOR
default-ipv4-ippool 192.168.0.0/16 all()
[root@k8s-master calico]# calicoctl get ippool default-ipv4-ippool -o yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
creationTimestamp: "2021-08-29T14:33:53Z"
name: default-ipv4-ippool
resourceVersion: "1305"
uid: c01d73f3-c0c9-4674-b27e-725a1eaa5717
spec:
blockSize: 24
cidr: 192.168.0.0/16
ipipMode: Always
natOutgoing: true
nodeSelector: all()
vxlanMode: Never
[root@k8s-master calico]# calicoctl ipam --help
Usage:
calicoctl [options] [<args>...]
Options:
-h --help Show this screen.
-c --config=<config> Path to the file containing connection
configuration in YAML or JSON format.
[default: /etc/calico/calicoctl.cfg]
--context=<context> The name of the kubeconfig context to use.
-a
-A --all-namespaces
--as=<AS_NUM>
--backend=(bird|gobgp|none)
--dryrun
--export
--felix-config=<CONFIG>
-f --filename=<FILENAME>
--force
--from-report=<REPORT>
--ignore-validation
--init-system
--ip6-autodetection-method=<IP6_AUTODETECTION_METHOD>
--ip6=<IP6>
--ip-autodetection-method=<IP_AUTODETECTION_METHOD>
...
[root@k8s-master calico]# calicoctl ipam show
+----------+----------------+-----------+------------+--------------+
| GROUPING | CIDR | IPS TOTAL | IPS IN USE | IPS FREE |
+----------+----------------+-----------+------------+--------------+
| IP Pool | 192.168.0.0/16 | 65536 | 9 (0%) | 65527 (100%) |
+----------+----------------+-----------+------------+--------------+
[root@k8s-master calico]# calicoctl ipam show --show-blocks #每个地址段应用了多少个
+----------+------------------+-----------+------------+--------------+
| GROUPING | CIDR | IPS TOTAL | IPS IN USE | IPS FREE |
+----------+------------------+-----------+------------+--------------+
| IP Pool | 192.168.0.0/16 | 65536 | 9 (0%) | 65527 (100%) |
| Block | 192.168.113.0/24 | 256 | 1 (0%) | 255 (100%) |
| Block | 192.168.12.0/24 | 256 | 2 (1%) | 254 (99%) |
| Block | 192.168.237.0/24 | 256 | 4 (2%) | 252 (98%) |
| Block | 192.168.51.0/24 | 256 | 2 (1%) | 254 (99%) |
+----------+------------------+-----------+------------+--------------+
[root@k8s-master calico]# calicoctl ipam show --show-config #查看配置信息
+--------------------+-------+
| PROPERTY | VALUE |
+--------------------+-------+
| StrictAffinity | false |
| AutoAllocateBlocks | true |
| MaxBlocksPerHost | 0 |
+--------------------+-------+
第 2 种形式 以 kubectl 插件形式运行
[root@k8s-master calico]# cp -p /usr/bin/calicoctl /usr/bin/kubectl-calico #把之前的文件改个名字就能够了
[root@k8s-master calico]# kubectl calico
Usage:
kubectl-calico [options] <command> [<args>...]
Invalid option: ''. Use flag'--help' to read about a specific subcommand
[root@k8s-master calico]# kubectl calico get nodes #和第 1 种形式相比加 kubectl
NAME
k8s-master
k8s-node1
k8s-node2
k8s-node3
[root@k8s-master calico]# kubectl calico ipam show
+----------+----------------+-----------+------------+--------------+
| GROUPING | CIDR | IPS TOTAL | IPS IN USE | IPS FREE |
+----------+----------------+-----------+------------+--------------+
| IP Pool | 192.168.0.0/16 | 65536 | 9 (0%) | 65527 (100%) |
+----------+----------------+-----------+------------+--------------+
示例 2: 批改 BGP 网络
# 获取现有配置在此基础上批改
[root@k8s-master calico]# kubectl calico get ippool -o yaml > default-ipv4-ippool.yaml
[root@k8s-master calico]# cat default-ipv4-ippool.yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: default-ipv4-ippool
spec:
blockSize: 24
cidr: 192.168.0.0/16
ipipMode: CrossSubnet #跨节点子网时应用 IPIP 没有跨子网应用 BGP
natOutgoing: true
nodeSelector: all()
vxlanMode: Never #vxlanMode 与 ipipMode 不能同时关上 必须有一个为 Never
#通过 ipipMode、vxlanMode 不同选项能够使 calico 运行在纯 GBP、ipip、vxlanMode 或混合模式下
#如:ipipMode: Never vxlanMode: Never 为纯 BGP 模式 ipipMode: Never vxlanMode: CrossSubnet 为 BGP+vxlan 模式
[root@k8s-master calico]# calicoctl apply -f default-ipv4-ippool.yaml
Successfully applied 1 'IPPool' resource(s)
#在来看路由信息 曾经没有之前的 tunl0 间接从节点网络进来
[root@k8s-master calico]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.170 metric 100
192.168.12.0/24 via 192.168.4.172 dev eth0 proto bird
192.168.51.0/24 via 192.168.4.173 dev eth0 proto bird #曾经没有之前的 tunl0 隧道
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.170 metric 101
192.168.113.0/24 via 192.168.4.171 dev eth0 proto bird
blackhole 192.168.237.0/24 proto bird
192.168.237.1 dev cali7c0fb624285 scope link
192.168.237.2 dev caliedaf285d4ef scope link
192.168.237.3 dev cali854da94d42a scope link
[root@k8s-master calico]#
抓包测试
[root@k8s-master ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deployment-demo-fb544c5d8-r7pc8 1/1 Running 0 10h 192.168.51.1 k8s-node3 <none> <none>
deployment-demo-fb544c5d8-splfr 1/1 Running 0 10h 192.168.12.1 k8s-node2 <none> <none>
#从节点 3 拜访节点 3
[root@k8s-master calico]# kubectl exec deployment-demo-fb544c5d8-r7pc8 -it -- /bin/sh
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
#间接抓 Pod IP 的包 因为没有封装 所以是 Pod IP 间接通信 没有外层 IP
[root@k8s-node2 ~]# tcpdump -i eth0 -nn ip host 192.168.51.1 and host 192.168.12.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:11:54.704770 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [S], seq 4075444778, win 64800, options [mss 1440,sackOK,TS val 2045898534 ecr 0,nop,wscale 7], length 0
22:11:54.705866 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [S.], seq 402120893, ack 4075444779, win 64260, options [mss 1440,sackOK,TS val 1090640722 ecr 2045898534,nop,wscale 7], length 0
22:11:54.706670 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 2045898537 ecr 1090640722], length 0
22:11:54.707077 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [P.], seq 1:77, ack 1, win 507, options [nop,nop,TS val 2045898537 ecr 1090640722], length 76: HTTP: GET / HTTP/1.1
22:11:54.707132 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [.], ack 77, win 502, options [nop,nop,TS val 1090640723 ecr 2045898537], length 0
22:11:54.737231 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [P.], seq 1:18, ack 77, win 502, options [nop,nop,TS val 1090640754 ecr 2045898537], length 17: HTTP: HTTP/1.0 200 OK
22:11:54.738439 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 18, win 507, options [nop,nop,TS val 2045898568 ecr 1090640754], length 0
22:11:54.739117 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [P.], seq 18:155, ack 77, win 502, options [nop,nop,TS val 1090640755 ecr 2045898568], length 137: HTTP
22:11:54.739630 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [FP.], seq 155:276, ack 77, win 502, options [nop,nop,TS val 1090640756 ecr 2045898568], length 121: HTTP
22:11:54.739810 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 155, win 506, options [nop,nop,TS val 2045898570 ecr 1090640755], length 0
[root@k8s-master calico]# calicoctl node status
Calico process is running.
IPv4 BGP status #能够看到曾经 BGP 模式了 这里看到是除去本人其它的 3 个节点
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+---------------+-------------------+-------+----------+-------------+
| 192.168.4.171 | node-to-node mesh | up | 02:27:59 | Established |
| 192.168.4.172 | node-to-node mesh | up | 02:27:58 | Established |
| 192.168.4.173 | node-to-node mesh | up | 02:27:58 | Established |
+---------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.- 到目前为止 如果是小规模的集群 比方 50 台以下 就能够间接应用了
- 如果是大规模集群 部署 reflector 路由反射器，防止过多的路由表更新 加重 AIP-SERVER 压力
# 把 maseter 配置成 reflector 节点
[root@k8s-master calico]# cat reflector-node.yaml
apiVersion: projectcalico.org/v3
kind: Node
metadata:
labels:
route-reflector: true
name: k8s-master #节点名
spec:
bgp:
ipv4Address: 192.168.4.170/24 #Master IP
ipv4IPIPTunnelAddr: 192.168.237.0 #tunl0 网络地址
routeReflectorClusterID: 1.1.1.1 #ID 信息 如果有多个 node 不能和其它反复就行
[root@k8s-master calico]# calicoctl apply -f reflector-node.yaml
Successfully applied 1 'Node' resource(s)- 配置所有节点与 reflector 节点通信
[root@k8s-master calico]# cat bgppeer-demo.yaml
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
name: bgppeer-demo
spec:
nodeSelector: all() #所有节点
peerSelector: route-reflector=="true" #与有这个标签的节点通信
[root@k8s-master calico]# calicoctl apply -f bgppeer-demo.yaml
Successfully applied 1 'BGPPeer' resource(s)
[root@k8s-master calico]# calicoctl node status
Calico process is running.
IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+---------------+-------------------+-------+----------+-------------+
| 192.168.4.171 | node-to-node mesh | up | 02:27:59 | Established |# 之前的 mesh 工作模式还在
| 192.168.4.172 | node-to-node mesh | up | 02:27:58 | Established |
| 192.168.4.173 | node-to-node mesh | up | 02:27:58 | Established |
| 192.168.4.171 | node specific | start | 14:36:40 | Idle |# 基于 reflector 工作模式
| 192.168.4.172 | node specific | start | 14:36:40 | Idle |
| 192.168.4.173 | node specific | start | 14:36:40 | Idle |
+---------------+-------------------+-------+----------+-------------+
IPv6 BGP status
#关掉 mesh 点对点的工作模式
[root@k8s-master calico]# cat default-bgpconfiguration.yaml
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Info
nodeToNodeMeshEnabled: false #是赤容许点对点通信
asNumber : 63400
[root@k8s-master calico]# calicoctl apply -f default-bgpconfiguration.yaml
Successfully applied 1 'BGPConfiguration' resource(s)
[root@k8s-master calico]# calicoctl node status
Calico process is running.
IPv4 BGP status
+---------------+---------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+---------------+---------------+-------+----------+-------------+
| 192.168.4.171 | node specific | up | 14:45:26 | Established |
| 192.168.4.172 | node specific | up | 14:45:26 | Established |
| 192.168.4.173 | node specific | up | 14:45:26 | Established |
+---------------+---------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.正文完