关于kubernetes:16kubernetes笔记-CNI网络插件二-Calico介绍

41次阅读

共计 24975 个字符，预计需要花费 63 分钟才能阅读完成。

Calico 简介

Calico 是一种容器之间互通的网络计划。在虚拟化平台中，比方 OpenStack、Docker 等都须要实现 workloads 之间互连，但同时也须要对容器做隔离管制，就像在 Internet 中的服务仅凋谢 80 端口、私有云的多租户一样，提供隔离和管控机制。而在少数的虚拟化平台实现中，通常都应用二层隔离技术来实现容器的网络，这些二层的技术有一些弊病，比方须要依赖 VLAN、bridge 和隧道等技术，其中 bridge 带来了复杂性，vlan 隔离和 tunnel 隧道则耗费更多的资源并对物理环境有要求，随着网络规模的增大，整体会变得越加简单。咱们尝试把 Host 当作 Internet 中的路由器，同样应用 BGP 同步路由，并应用 iptables 来做平安拜访策略，最终设计出了 Calico 计划。

设计思维：Calico 不应用隧道或 NAT 来实现转发，而是奇妙的把所有二三层流量转换成三层流量，并通过 host 上路由配置实现跨 Host 转发

设计劣势：

1. 更优的资源利用
二层网络通讯须要依赖播送音讯机制，播送音讯的开销与 host 的数量呈指数级增长，Calico 应用的三层路由办法，则齐全克制了二层播送，缩小了资源开销。

2. 可扩展性
Calico 应用与 Internet 相似的计划，Internet 的网络比任何数据中心都大，Calico 同样人造具备可扩展性。

3. 简略而更容易 debug
因为没有隧道，意味着 workloads 之间门路更短更简略，配置更少，在 host 上更容易进行 debug 调试。

4. 更少的依赖
Calico 仅依赖三层路由可达。

5. 可适配性
Calico 较少的依赖性使它能适配所有 VM、Container、白盒或者混合环境场景。

Calico 网络 Node 之间通信网络

IPIP(可跨网段通信)
从字面来了解，就是把一个 IP 数据包又套在一个 IP 包里，即把 IP 层封装到 IP 层的一个 tunnel。它的作用其实基本上就相当于一个基于 IP 层的网桥！一般来说，一般的网桥是基于 mac 层的，基本不需 IP，而这个 ipip 则是通过两端的路由做一个 tunnel，把两个原本不通的网络通过点对点连接起来。
相似 vxlan 但封装开销比 vxlan 小 效率绝对更高一些，但安全性也更差

Vxlan(可跨网段通信)
与 Flannel Vxlan 原理雷同

BGP(二层网络通信)
边界网关协定（Border Gateway Protocol, BGP）是互联网上一个外围的去中心化自治路由协定。它通过保护 IP 路由表或‘前缀’表来实现自治零碎（AS）之间的可达性，属于矢量路由协定。BGP 不应用传统的外部网关协定（IGP）的指标，而应用基于门路、网络策略或规定集来决定路由。因而，它更适宜被称为矢量性协定，而不是路由协定。BGP，艰深的讲就是讲接入到机房的多条线路（如电信、联通、挪动等）交融为一体，实现多线单 IP，BGP 机房的长处：服务器只须要设置一个 IP 地址，最佳拜访路由是由网络上的骨干路由器依据路由跳数与其它技术指标来确定的，不会占用服务器的任何零碎
实际上，Calico 我的项目提供的 BGP 网络解决方案，与 Flannel 的 host-gw 模式简直一样。也就是说，Calico 也是基于路由表实现容器数据包转发，但不同于 Flannel 应用 flanneld 过程来保护路由信息的做法，而 Calico 我的项目应用 BGP 协定来主动保护整个集群的路由信息。

部署举荐计划:
BGP+Vxlan

其中 BGP 在官网的举荐计划中 以 50 个节点为界区别了不同规模应用不同的部署计划

  • 小规模网络:BGP peer 一对一网络：每个节点都是有 N - 1 条路由，小型网络实用，当节点数 N 变多时，路由表更新及 AIP-SERVER 都须要接受很大的压力 相似网络拓扑构造中的 网状拓扑构造
  • 大规模网络:BGP Reflector 路由反射器：抉择一到多个节点做为 Reflector，所有节点路由都汇总给 Reflector，所有节点都路由都指向 Reflector，适宜大型网络，相似网络拓扑构造中的星型网络

Calico 网络模型次要工作组件

  1. Felix：运行在每一台 Host 的 agent 过程，次要负责网络接口治理和监听、路由、ARP 治理、ACL 治理和同步、状态上报等。
  2. etcd：分布式键值存储，次要负责网络元数据一致性，确保 Calico 网络状态的准确性，能够与 kubernetes 共用；
  3. BGP Client（BIRD）：Calico 为每一台 Host 部署一个 BGP Client，应用 BIRD 实现，BIRD 是一个独自的继续倒退的我的项目，实现了泛滥动静路由协定比方 BGP、OSPF、RIP 等。在 Calico 的角色是监听 Host 上由 Felix 注入的路由信息，而后通过 BGP 协定播送通知残余 Host 节点，从而实现网络互通。
  4. BGP Route Reflector：在大型网络规模中，如果仅仅应用 BGP client 造成 mesh 全网互联的计划就会导致规模限度，因为所有节点之间俩俩互联，须要 N^2 个连贯，为了解决这个规模问题，能够采纳 BGP 的 Router Reflector 的办法，使所有 BGP Client 仅与特定 RR 节点互联并做路由同步，从而大大减少连接数。

Calico 有两种运行形式:

  1. 是让 calico/node 独立运行于 Kubernetes 集群之外，但 calico/kube-controllers 仍然须要以 Pod 资源运行中集群之上;
  2. 是以 CNI 插件形式配置 Calico 齐全托管运行于 Kubernetes 集群之上，相似于咱们后面已经部署托管 Flannel 网络插件的形式。
    对于后一种形式,Calico 提供了在线的部署清单，它别离为 50 节点及以下规模和 50 节点以上规模的 Kubernetes 集群应用 Kubernetes API 作为 Dabastore 提供了不同的配置清单，也为应用独立的 etcd 集群提供了专用配置清单。但这 3 种类型的配置清单中,Calico 默认启用的是基于 IPIP 隧道的叠加网络，因此它会在所有流量上应用 IPIP 隧道而不是 BGP 路由。以下配置定义在部署清单中 DaemonSet/calico-node 资源的 Pod 模板中的 calico-node 容器之上。

配置选项
在 IPv4 类型的地址池上启用的 IPIP 及其类型，反对 3 种可用值
Always(全局流量)、Cross-SubNet(跨子网流量) 和 Never3 种可用值

  • name: CALICO_IPV4POOL_IPIP
    value: “Always”
  • 是否在 IPV4 地址池上启用 VXLAN 隧道协定，取值及意义与 Flannel 的 VXLAN 后端雷同; 但在全局流量启用 VXLAN 时将齐全不再须要 BGP 网络，倡议将相干的组件禁用
  • name: CALICO_ IPV4POOL_VXLAN
    value: “Never”
  • 须要留神的是，Calico 调配的地址池须要同 Ktbernetes 集群的 Pod 网络的定义保持一致。Pod 网络通常由 kubeadm init 初始化集群时应用 –pod-network-cidr 选项指定的网络，而 Calico 在其默认的配置清单中默认应用 192.168.0.0/16 作为 Pod 网络，因此部署 Kubernetes 集群时应该布局好要应用的网络地址，并设定此二者相匹配。对于已经应用了 flannel 的默认的 10.244.0.0/16 网络的环境而言, 咱们也能够抉择批改资源清单中的定义，从而将其批改为其余网络地址, 它定义在 DaemonSet/calico-node 资源的 Pod 模板中的 calico-node 容器之上。

官网链接：

https://docs.projectcalico.or…

示例 1: 装置 calico

wget https://docs.projectcalico.org/manifests/calico.yaml

[root@k8s-master ~]# cd /etc/kubernetes/manifests/
[root@k8s-master manifests]# cat kube-controller-manager.yaml 
...
System Info:
  Machine ID:                 32599e2a74704b2e95443e24ea15d4f6
  System UUID:                34979a62-16de-4287-b149-2d4c2d8a70fb
  Boot ID:                    f31de60e-4f89-4553-ba7a-99a46d049936
  Kernel Version:             5.4.109-1.el7.elrepo.x86_64
  OS Image:                   CentOS Linux 7 (Core)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.7
  Kubelet Version:            v1.19.9
  Kube-Proxy Version:         v1.19.9
PodCIDR:                      10.244.1.0/24  #第个节点的地址块都是由 K8S 调配
PodCIDRs:                     10.244.1.0/24   
Non-terminated Pods:          (17 in total)

[root@k8s-master ~]# kubectl describe node k8s-node1
System Info:
  Machine ID:                 32599e2a74704b2e95443e24ea15d4f6
  System UUID:                34979a62-16de-4287-b149-2d4c2d8a70fb
  Boot ID:                    f31de60e-4f89-4553-ba7a-99a46d049936
  Kernel Version:             5.4.109-1.el7.elrepo.x86_64
  OS Image:                   CentOS Linux 7 (Core)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.7
  Kubelet Version:            v1.19.9
  Kube-Proxy Version:         v1.19.9
PodCIDR:                      10.244.1.0/24   #每个 Node Pod 都是由 K8S 调配 IP
PodCIDRs:                     10.244.1.0/24

[root@k8s-master Network]# vim calico.yaml
...
 "ipam": {
              "type": "host-local",
              "subnet": "usePodCidr"   #应用 k8s ipam 插件调配地址 
          },
          "policy": {"type": "k8s"},

...
- name: FELIX_WIREGUARDMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            - name: CALICO_IPV4POOL_CIDR
              value: "10.244.0.0/16"  #为了和之前的 flannel 10.244.0.0/16 适配
            - name: CALICO_IPV4POOL_BLOCK_SIZE  #增加这一行批改默认块大小
              value: "24"
            - name: USE_POD_CIDR  #应用 K8S 的调配的 IP 地址，不然 calico 和 K8S 调配的地址会不一样
              value: "true"
  • 装置 calico
[root@k8s-master plugin]# kubectl delete -f kube-flannel.yml
podsecuritypolicy.policy "psp.flannel.unprivileged" deleted
clusterrole.rbac.authorization.k8s.io "flannel" deleted
clusterrolebinding.rbac.authorization.k8s.io "flannel" deleted
serviceaccount "flannel" deleted
configmap "kube-flannel-cfg" deleted
daemonset.apps "kube-flannel-ds" deleted

[root@k8s-master plugin]# kubectl apply -f calico.yaml 
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created
  • calico 几个组件
[root@k8s-master ~]# ps aux|grep calico
root     10867  0.0  0.1 112816  2156 pts/1    S+   13:51   0:00 grep --color=auto calico
root     20680  0.0  2.3 1215184 35216 ?       Sl   10:27   0:06 calico-node -allocate-tunnel-addrs
root     20681  0.0  2.1 1215184 32672 ?       Sl   10:27   0:06 calico-node -monitor-addresses
root     20682  2.4  3.3 1510624 51636 ?       Sl   10:27   4:54 calico-node -felix
root     20683  0.0  2.3 1657832 35496 ?       Sl   10:27   0:09 calico-node -confd
root     20686  0.0  2.0 1214928 31628 ?       Sl   10:27   0:05 calico-node -monitor-token
  • 因为 calico 并没有应用 k8s 的 ipam 调配 IP, 所以节点会有 2 个 IP, 一个是 K8S 调配的 IP 一个是 calico 调配的 IP
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.54.2    0.0.0.0         UG    101    0        0 eth4
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.4.0     0.0.0.0         255.255.255.0   U     100    0        0 eth0
192.168.12.0    192.168.4.172   255.255.255.0   UG    0      0        0 tunl0  #能够看到 tunl0 的路由信息
192.168.51.0    192.168.4.173   255.255.255.0   UG    0      0        0 tunl0  #同时能够看到节点的 IP 不像之前肯定是间断的
192.168.54.0    0.0.0.0         255.255.255.0   U     101    0        0 eth4
192.168.113.0   192.168.4.171   255.255.255.0   UG    0      0        0 tunl0  #隧道接口
192.168.237.0   0.0.0.0         255.255.255.0   U     0      0        0 *
192.168.237.1   0.0.0.0         255.255.255.255 UH    0      0        0 cali7c0fb624285
192.168.237.2   0.0.0.0         255.255.255.255 UH    0      0        0 caliedaf285d4ef
192.168.237.3   0.0.0.0         255.255.255.255 UH    0      0        0 cali854da94d42a


[root@k8s-master calico]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.170 metric 100 
192.168.12.0/24 via 192.168.4.172 dev tunl0 proto bird onlink   #能够看到 tunl0 的路由信息
192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink    
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.170 metric 101 
192.168.113.0/24 via 192.168.4.171 dev tunl0 proto bird onlink 
blackhole 192.168.237.0/24 proto bird 
192.168.237.1 dev cali7c0fb624285 scope link 
192.168.237.2 dev caliedaf285d4ef scope link 
192.168.237.3 dev cali854da94d42a scope link
  • 192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink 上面的路由表能够看到 calico 会为每个节点调配网络地址段 并不是应用节点的网络地址
[root@k8s-node1 ~]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.171 metric 100 
192.168.12.0/24 via 192.168.4.172 dev tunl0 proto bird onlink 
192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink 
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.171 metric 101 
blackhole 192.168.113.0/24 proto bird   #黑洞 代表本人网段 
192.168.237.0/24 via 192.168.4.170 dev tunl0 proto bird onlink
  • 查看目前工作模式
[root@k8s-master calico]# kubectl api-resources
NAME                              SHORTNAMES   APIGROUP                       NAMESPACED   KIND
bgpconfigurations                              crd.projectcalico.org          false        BGPConfiguration
bgppeers                                       crd.projectcalico.org          false        BGPPeer
blockaffinities                                crd.projectcalico.org          false        BlockAffinity
clusterinformations                            crd.projectcalico.org          false        ClusterInformation
felixconfigurations                            crd.projectcalico.org          false        FelixConfiguration
globalnetworkpolicies                          crd.projectcalico.org          false        GlobalNetworkPolicy
globalnetworksets                              crd.projectcalico.org          false        GlobalNetworkSet
hostendpoints                                  crd.projectcalico.org          false        HostEndpoint
ipamblocks                                     crd.projectcalico.org          false        IPAMBlock
ipamconfigs                                    crd.projectcalico.org          false        IPAMConfig
ipamhandles                                    crd.projectcalico.org          false        IPAMHandle
ippools                                        crd.projectcalico.org          false        IPPool  #calico 地址池
kubecontrollersconfigurations                  crd.projectcalico.org          false        KubeControllersConfiguration
networkpolicies                                crd.projectcalico.org          true         NetworkPolicy
networksets                                    crd.projectcalico.org          true         NetworkSet


[root@k8s-master calico]# kubectl get  ippools -o yaml
....
  spec:
    blockSize: 24   #掩码长度
    cidr: 192.168.0.0/16  #地址池
    ipipMode: Always      #能够看到目前为 ipip 模式
    natOutgoing: true
    nodeSelector: all()
  • 拜访抓包
[root@k8s-master PodControl]# kubectl get pod -o wide
NAME                              READY   STATUS    RESTARTS   AGE    IP             NODE        NOMINATED NODE   READINESS GATES
deployment-demo-fb544c5d8-r7pc8   1/1     Running   0          8m3s   192.168.51.1   k8s-node3   <none>           <none>
deployment-demo-fb544c5d8-splfr   1/1     Running   0          8m3s   192.168.12.1   k8s-node2   <none>           <none>
[root@k8s-master PodControl]# kubectl exec deployment-demo-fb544c5d8-r7pc8 -it -- /bin/sh
[root@deployment-demo-fb544c5d8-r7pc8 /]# ifconfig
eth0      Link encap:Ethernet  HWaddr 16:96:97:3F:F3:C5  
          inet addr:192.168.51.1  Bcast:192.168.51.1  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1



[root@k8s-node2 ~]# tcpdump -i eth0 -nn ip host 192.168.4.172  and host 192.168.4.173
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:48:24.421003 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [S], seq 3259804851, win 64800, options [mss 1440,sackOK,TS val 2008488248 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
11:48:24.421093 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [S.], seq 3234480084, ack 3259804852, win 64260, options [mss 1440,sackOK,TS val 1053230437 ecr 2008488248,nop,wscale 7], length 0 (ipip-proto-4)  #能够看到 (ipip-proto-4) 为 IPIP 模式
11:48:24.422305 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 2008488250 ecr 1053230437], length 0 (ipip-proto-4)
11:48:24.422308 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [P.], seq 1:77, ack 1, win 507, options [nop,nop,TS val 2008488250 ecr 1053230437], length 76: HTTP: GET / HTTP/1.1 (ipip-proto-4)
11:48:24.422554 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [.], ack 77, win 502, options [nop,nop,TS val 1053230439 ecr 2008488250], length 0 (ipip-proto-4)
11:48:24.431688 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [P.], seq 1:18, ack 77, win 502, options [nop,nop,TS val 1053230447 ecr 2008488250], length 17: HTTP: HTTP/1.0 200 OK (ipip-proto-4)
11:48:24.432638 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [FP.], seq 18:276, ack 77, win 502, options [nop,nop,TS val 1053230449 ecr 2008488250], length 258: HTTP (ipip-proto-4)
11:48:24.433660 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [.], ack 18, win 507, options [nop,nop,TS val 2008488261 ecr 1053230447], length 0 (ipip-proto-4)
11:48:24.437531 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [F.], seq 77, ack 277, win 505, options [nop,nop,TS val 2008488261 ecr 1053230449], length 0 (ipip-proto-4)
11:48:24.437775 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [.], ack 78, win 502, options [nop,nop,TS val 1053230454 ecr 2008488261], length 0 (ipip-proto-4)
IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436
  • 能够看到默认为 ipip 模式 也是通过封装在转发的 和 Flannel 很相似, 但绝对 Flannel 通过虚构网桥 CNI calico 间接内核 (内核的路由由 kube-proxy 或 IPVS 生成) 到在由 tunl0 传输想对 Flannel 少了一层交换机替换的过程, 性能相比 Flannel 会快一些 但这并不是 calico 最佳的模式

calicoctl 命令装置与应用

**calicoctl 装置的 2 种形式
第 1 种形式 calicoctl**

https://docs.projectcalico.or…

  • 几种形式运行 calicoctl 罕用形式 1: 间接下载 2 进制 calicoctl 间接运行
[root@k8s-master ~]# curl -o calicoctl -O -L  "https://github.com/projectcalico/calicoctl/releases/download/v3.20.0/calicoctl"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   615  100   615    0     0    498      0  0:00:01  0:00:01 --:--:--   504
100 43.2M  100 43.2M    0     0   518k      0  0:01:25  0:01:25 --:--:--  920k

[root@k8s-master ~]# mv calicoctl /usr/bin/

[root@k8s-master ~]# chmod +x /usr/bin/calicoctl 
[root@k8s-master ~]# calicoctl  --help
Usage:
  calicoctl [options] <command> [<args>...]

    create       Create a resource by file, directory or stdin.
    replace      Replace a resource by file, directory or stdin.
    apply        Apply a resource by file, directory or stdin.  This creates a resource
                 if it does not exist, and replaces a resource if it does exists.
    patch        Patch a pre-exisiting resource in place.
    delete       Delete a resource identified by file, directory, stdin or resource type and
                 name.
    get          Get a resource identified by file, directory, stdin or resource type and
                 name.
    label        Add or update labels of resources.
    convert      Convert config files between different API versions.
    ipam         IP address management.
    node         Calico node management.
    version      Display the version of this binary.
    export       Export the Calico datastore objects for migration
    import       Import the Calico datastore objects for migration
    datastore    Calico datastore management.
  • calicoctl 命令应用
- calicoctl 默认会读取 ~/.kube/ 下文件加载认证信息, 也能够通过配置文件指定认证信息地位

[root@k8s-master calico]# mkdir /etc/calico/^C
[root@k8s-master calico]# cd /etc/calico/
[root@k8s-master calico]# cat calicoctl.cfg 
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  datastoreType: "kubernetes"
  kubeconfig: "/etc/kubernetes/admin.conf" #指定 conf 门路
[root@k8s-master calico]# 

[root@k8s-master calico]# kubectl get ippools
NAME                  AGE
default-ipv4-ippool   23h
[root@k8s-master calico]# calicoctl get ippool  #能够用 calicoctl 间接拜访 calico 资源
NAME                  CIDR             SELECTOR   
default-ipv4-ippool   192.168.0.0/16   all() 


[root@k8s-master calico]# calicoctl get ippool default-ipv4-ippool -o yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  creationTimestamp: "2021-08-29T14:33:53Z"
  name: default-ipv4-ippool
  resourceVersion: "1305"
  uid: c01d73f3-c0c9-4674-b27e-725a1eaa5717
spec:
  blockSize: 24
  cidr: 192.168.0.0/16
  ipipMode: Always
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Never


[root@k8s-master calico]# calicoctl ipam --help  
Usage:
  calicoctl [options] [<args>...]

Options:
  -h --help                 Show this screen.
  -c --config=<config>      Path to the file containing connection
                            configuration in YAML or JSON format.
                            [default: /etc/calico/calicoctl.cfg]
  --context=<context>       The name of the kubeconfig context to use.
  -a
  -A --all-namespaces
     --as=<AS_NUM>
     --backend=(bird|gobgp|none)
     --dryrun
     --export
     --felix-config=<CONFIG>
  -f --filename=<FILENAME>
     --force
     --from-report=<REPORT>
     --ignore-validation
     --init-system
     --ip6-autodetection-method=<IP6_AUTODETECTION_METHOD>
     --ip6=<IP6>
     --ip-autodetection-method=<IP_AUTODETECTION_METHOD>
...

[root@k8s-master calico]# calicoctl ipam show
+----------+----------------+-----------+------------+--------------+
| GROUPING |      CIDR      | IPS TOTAL | IPS IN USE |   IPS FREE   |
+----------+----------------+-----------+------------+--------------+
| IP Pool  | 192.168.0.0/16 |     65536 | 9 (0%)     | 65527 (100%) |
+----------+----------------+-----------+------------+--------------+


[root@k8s-master calico]# calicoctl ipam show  --show-blocks  #每个地址段应用了多少个
+----------+------------------+-----------+------------+--------------+
| GROUPING |       CIDR       | IPS TOTAL | IPS IN USE |   IPS FREE   |
+----------+------------------+-----------+------------+--------------+
| IP Pool  | 192.168.0.0/16   |     65536 | 9 (0%)     | 65527 (100%) |
| Block    | 192.168.113.0/24 |       256 | 1 (0%)     | 255 (100%)   |
| Block    | 192.168.12.0/24  |       256 | 2 (1%)     | 254 (99%)    |
| Block    | 192.168.237.0/24 |       256 | 4 (2%)     | 252 (98%)    |
| Block    | 192.168.51.0/24  |       256 | 2 (1%)     | 254 (99%)    |
+----------+------------------+-----------+------------+--------------+

[root@k8s-master calico]# calicoctl ipam show  --show-config  #查看配置信息
+--------------------+-------+
|      PROPERTY      | VALUE |
+--------------------+-------+
| StrictAffinity     | false |
| AutoAllocateBlocks | true  |
| MaxBlocksPerHost   |     0 |
+--------------------+-------+

第 2 种形式 以 kubectl 插件形式运行

[root@k8s-master calico]# cp -p /usr/bin/calicoctl  /usr/bin/kubectl-calico  #把之前的文件改个名字就能够了

[root@k8s-master calico]# kubectl calico
Usage:
  kubectl-calico [options] <command> [<args>...]
Invalid option: ''. Use flag'--help' to read about a specific subcommand

[root@k8s-master calico]# kubectl calico get nodes  #和第 1 种形式相比加 kubectl
NAME         
k8s-master   
k8s-node1    
k8s-node2    
k8s-node3    

[root@k8s-master calico]# kubectl calico ipam show
+----------+----------------+-----------+------------+--------------+
| GROUPING |      CIDR      | IPS TOTAL | IPS IN USE |   IPS FREE   |
+----------+----------------+-----------+------------+--------------+
| IP Pool  | 192.168.0.0/16 |     65536 | 9 (0%)     | 65527 (100%) |
+----------+----------------+-----------+------------+--------------+

示例 2: 批改 BGP 网络

# 获取现有配置在此基础上批改

[root@k8s-master calico]# kubectl calico get ippool -o yaml > default-ipv4-ippool.yaml 

[root@k8s-master calico]# cat default-ipv4-ippool.yaml 
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  blockSize: 24
  cidr: 192.168.0.0/16
  ipipMode: CrossSubnet #跨节点子网时应用 IPIP 没有跨子网应用 BGP
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Never  #vxlanMode 与 ipipMode 不能同时关上 必须有一个为 Never
  
#通过 ipipMode、vxlanMode 不同选项能够使 calico 运行在纯 GBP、ipip、vxlanMode 或混合模式下
#如:ipipMode: Never vxlanMode: Never 为纯 BGP 模式  ipipMode: Never vxlanMode: CrossSubnet 为 BGP+vxlan 模式

[root@k8s-master calico]# calicoctl apply -f default-ipv4-ippool.yaml 
Successfully applied 1 'IPPool' resource(s)

#在来看路由信息 曾经没有之前的 tunl0 间接从节点网络进来
[root@k8s-master calico]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.170 metric 100 
192.168.12.0/24 via 192.168.4.172 dev eth0 proto bird 
192.168.51.0/24 via 192.168.4.173 dev eth0 proto bird     #曾经没有之前的 tunl0 隧道
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.170 metric 101 
192.168.113.0/24 via 192.168.4.171 dev eth0 proto bird 
blackhole 192.168.237.0/24 proto bird 
192.168.237.1 dev cali7c0fb624285 scope link 
192.168.237.2 dev caliedaf285d4ef scope link 
192.168.237.3 dev cali854da94d42a scope link 
[root@k8s-master calico]#

抓包测试

[root@k8s-master ~]# kubectl get pod -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP             NODE        NOMINATED NODE   READINESS GATES
deployment-demo-fb544c5d8-r7pc8   1/1     Running   0          10h   192.168.51.1   k8s-node3   <none>           <none>
deployment-demo-fb544c5d8-splfr   1/1     Running   0          10h   192.168.12.1   k8s-node2   <none>           <none>

#从节点 3 拜访节点 3 
[root@k8s-master calico]# kubectl exec deployment-demo-fb544c5d8-r7pc8 -it -- /bin/sh
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!

#间接抓 Pod IP 的包 因为没有封装 所以是 Pod IP 间接通信 没有外层 IP

[root@k8s-node2 ~]# tcpdump -i eth0 -nn ip host 192.168.51.1  and host 192.168.12.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:11:54.704770 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [S], seq 4075444778, win 64800, options [mss 1440,sackOK,TS val 2045898534 ecr 0,nop,wscale 7], length 0
22:11:54.705866 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [S.], seq 402120893, ack 4075444779, win 64260, options [mss 1440,sackOK,TS val 1090640722 ecr 2045898534,nop,wscale 7], length 0
22:11:54.706670 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 2045898537 ecr 1090640722], length 0
22:11:54.707077 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [P.], seq 1:77, ack 1, win 507, options [nop,nop,TS val 2045898537 ecr 1090640722], length 76: HTTP: GET / HTTP/1.1
22:11:54.707132 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [.], ack 77, win 502, options [nop,nop,TS val 1090640723 ecr 2045898537], length 0
22:11:54.737231 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [P.], seq 1:18, ack 77, win 502, options [nop,nop,TS val 1090640754 ecr 2045898537], length 17: HTTP: HTTP/1.0 200 OK
22:11:54.738439 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 18, win 507, options [nop,nop,TS val 2045898568 ecr 1090640754], length 0
22:11:54.739117 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [P.], seq 18:155, ack 77, win 502, options [nop,nop,TS val 1090640755 ecr 2045898568], length 137: HTTP
22:11:54.739630 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [FP.], seq 155:276, ack 77, win 502, options [nop,nop,TS val 1090640756 ecr 2045898568], length 121: HTTP
22:11:54.739810 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 155, win 506, options [nop,nop,TS val 2045898570 ecr 1090640755], length 0


[root@k8s-master calico]# calicoctl  node status 
Calico process is running.

IPv4 BGP status  #能够看到曾经 BGP 模式了 这里看到是除去本人其它的 3 个节点
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+---------------+-------------------+-------+----------+-------------+
| 192.168.4.171 | node-to-node mesh | up    | 02:27:59 | Established |
| 192.168.4.172 | node-to-node mesh | up    | 02:27:58 | Established |
| 192.168.4.173 | node-to-node mesh | up    | 02:27:58 | Established |
+---------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
  • 到目前为止 如果是小规模的集群 比方 50 台以下 就能够间接应用了
  • 如果是大规模集群 部署 reflector 路由反射器，防止过多的路由表更新 加重 AIP-SERVER 压力
# 把 maseter 配置成 reflector 节点

[root@k8s-master calico]# cat reflector-node.yaml 
apiVersion: projectcalico.org/v3
kind: Node
metadata:
  labels:
    route-reflector: true
  name: k8s-master   #节点名
spec:
  bgp:
    ipv4Address: 192.168.4.170/24  #Master IP
    ipv4IPIPTunnelAddr: 192.168.237.0  #tunl0 网络地址
    routeReflectorClusterID: 1.1.1.1  #ID 信息 如果有多个 node 不能和其它反复就行


[root@k8s-master calico]# calicoctl apply -f reflector-node.yaml 
Successfully applied 1 'Node' resource(s)
  • 配置所有节点与 reflector 节点通信
[root@k8s-master calico]# cat bgppeer-demo.yaml
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
  name: bgppeer-demo
spec:
  nodeSelector: all()   #所有节点
  peerSelector: route-reflector=="true" #与有这个标签的节点通信
  
[root@k8s-master calico]# calicoctl apply -f bgppeer-demo.yaml 
Successfully applied 1 'BGPPeer' resource(s)
[root@k8s-master calico]# calicoctl node status
Calico process is running.

IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+---------------+-------------------+-------+----------+-------------+
| 192.168.4.171 | node-to-node mesh | up    | 02:27:59 | Established |# 之前的 mesh 工作模式还在
| 192.168.4.172 | node-to-node mesh | up    | 02:27:58 | Established |
| 192.168.4.173 | node-to-node mesh | up    | 02:27:58 | Established |
| 192.168.4.171 | node specific     | start | 14:36:40 | Idle        |# 基于 reflector 工作模式
| 192.168.4.172 | node specific     | start | 14:36:40 | Idle        |
| 192.168.4.173 | node specific     | start | 14:36:40 | Idle        |
+---------------+-------------------+-------+----------+-------------+

IPv6 BGP status

#关掉 mesh 点对点的工作模式

[root@k8s-master calico]# cat  default-bgpconfiguration.yaml 
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false  #是赤容许点对点通信
  asNumber : 63400

[root@k8s-master calico]# calicoctl apply -f default-bgpconfiguration.yaml 
Successfully applied 1 'BGPConfiguration' resource(s)
[root@k8s-master calico]# calicoctl node status
Calico process is running.

IPv4 BGP status
+---------------+---------------+-------+----------+-------------+
| PEER ADDRESS  |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+---------------+---------------+-------+----------+-------------+
| 192.168.4.171 | node specific | up    | 14:45:26 | Established |
| 192.168.4.172 | node specific | up    | 14:45:26 | Established |
| 192.168.4.173 | node specific | up    | 14:45:26 | Established |
+---------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

正文完
kubernetes
发表至： kubernetes
2021-10-26
 0
kubernetes-installation
关于kubernetes:Kubernetes-1205-安装gitlab
关于kubernetes:kubernetes-pod-级别的-jvm-监控
关于kubernetes:使用-Rancher-进行首次金丝雀部署