SKS 是什么？
SKS 全称 SMTX Kubernetes Service，是 SmartX 公司退出的容器 IAAS 平台，如下图所示，在基于 SMTX OS 的平台上，能够通过 SKS 创立 Kubernetes Cluster 并进行治理。
SKS 的装置和治理依赖 管理工具 CloudTower。
SKS 装置
配套视频：https://www.bilibili.com/video/BV1pd4y1L77D/?vd_source=33b17c80f501bc5db46f7dcb22fc96cb
装置 CloudTower
现阶段 CloudTower 反对 SKS 的版本还是测试版，次要包含 CloudTower 安装文件、SKS 部署文件、镜像仓库部署文件和节点相干文件。其中节点相干文件即 VM-Kubernetes Node 装置模板，相似于 Tanzu 的[photon-ova](https://wp-content.vmware.com/v2/latest/ob-20611023-photon-3-amd64-vmi-k8s-v1.23.8---vmware.2-tkg.1-zshippable/photon-ova.ovf)，也是基于不同 Kubernetes 版本的。
- 系统配置
| 配置等级 | CloudTower 治理能⼒ | CloudTower 所在的虚拟机占⽤的系统资源 |
|---|---|---|
| 低配 | 10 个集群 /100 台主机 /1000 个虚拟机 | 4 核 vCPU/8 GiB 内存 /100 GiB 存储空间 |
| ⾼配 | 100 个集群 /1000 台主机 /10000 个虚拟机 | 8 核 vCPU/16 GiB 内存 /400 GiB 存储空间 |
选用高配，CloudTower 基于 CentOS，装置过程也相似，不同的是挂载装置 ISO 文件启动后，VM 会主动装置。
虚拟机 Guest OS 的默认⽤⼾名 smartx，初始密码 HC!r0cks，进入零碎后执行以下命令实现装置，在此之前确保 IP 地址配置胜利：
sudo bash
cd /usr/share/smartx/tower
sh ./preinstall.sh
cd ./installer
./binary/installer deploy实现当前：
Web 登录 CloudTower 的 IP 地址即进入治理界面。
SKS 初始化
关联 SMTX OS 集群后，在左上角库进入容器服务
- 创立镜像仓库
上传镜像仓库的文件并输出 IP 地址等参数后，零碎会主动创立镜像仓库。创立好当前，能够使 Web 形式登录，发现是 Harbor，初始用户名和明码是 admin/HC!r0cks
-
创立工作负载管控集群
- 集群的 Control-plane 只能抉择 1、3、5；Worker 节点能够任意抉择；
- 须要指定节点 IP 地址，配置 IP Pool 和 API Server 地址；
- 须要指定上传文件的 MD5 值，Win 下能够应用以下命令
C:\Users\Administrator\Downloads>certutil -hashfile sks-installer-all-bundle-v0.8.1-20230104152731.tar MD5 MD5 哈希(文件 sks-installer-all-bundle-v0.8.1-20230104152731.tar): 5510635fcf878f048b674a6427f681b5 CertUtil: -hashfile 命令胜利实现。部署实现后的管控集群如下，应用 Kubernetes 1.24.8 版本。
部署工作负载集群
在 容器服务→资源→工作负载集群，点击 + 创立工作负载集群
其中，
- Kubernetes 版本能够通过上传节点相干文件提供多个；
- CNI 临时只有 Calico；
- CSI 默认把本集群的分布式存储通过 CSI Driver 提供给 Pod 应用（Default StorageClass），如果系统配置了其余 ZBS iSCSI，能够选 SMTX ZBS iSCSI；
工作负载集群的 control-plane 和 worker 节点数量能够按需设置；
节点的拜访能够通过明码或密钥，用户名是 smartx；
配置 API Server 的接入点和节点 IP 地址；
配置 Pod IP 地址和 Service 地址。
点击创立后，零碎会耗时 10-30 分钟（视节点的多少）创立工作负载集群。
SKS 集群应用
创立实现工作负载集群后，就能够应用了。
工作负载集群的拜访
点击创立的集群，能够看到一个集群专属界面
右上角 + 创立 旁是本集群的 Kubeconfig 文件，通过它能够配置拜访该集群。
root@client:/home/zyi# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-01-controlplane-v1-25-4-bmtjms-45b52 Ready control-plane 31h v1.25.4
k8s-01-controlplane-v1-25-4-bmtjms-58h4s Ready control-plane 31h v1.25.4
k8s-01-controlplane-v1-25-4-bmtjms-sspc2 Ready control-plane 31h v1.25.4
k8s-01-workergroup1-v1-25-4-r3izcm-bcb89 Ready <none> 31h v1.25.4
k8s-01-workergroup1-v1-25-4-r3izcm-ggblg Ready <none> 31h v1.25.4
k8s-01-workergroup1-v1-25-4-r3izcm-jlfxd Ready <none> 31h v1.25.4
k8s-01-workergroup1-v1-25-4-r3izcm-p8c6x Ready <none> 31h v1.25.4- 插件设置
在每个集群，都能够按需装置插件，SKS 提供的插件有 监控、日志、LoadBalancer、Ingress 和 CSI（默认装置）
保留当前，零碎会主动装置。
应用 LoadBalancer
SKS 应用的 Load Balancer（MetaLB 的 L2 模式），须要配置 IP pool 给对外的 Service。
MetaLB 的官网：https://metallb.org/concepts/
root@client:/home/zyi# kubectl get po -n sks-system-metallb
NAME READY STATUS RESTARTS AGE
metallb-controller-8546c76b7b-8gxzd 1/1 Running 0 31h
metallb-speaker-28tb4 1/1 Running 0 31h
metallb-speaker-4lj7b 1/1 Running 0 31h
metallb-speaker-6jrmz 1/1 Running 0 31h
metallb-speaker-82gjl 1/1 Running 0 31h
metallb-speaker-jc56z 1/1 Running 0 31h
metallb-speaker-tz59m 1/1 Running 0 31h
metallb-speaker-xf48h 1/1 Running 0 31h咱们应用以下 yaml 测试 LBer：
apiVersion: v1
kind: Service
metadata:
name: hello-kubernetes
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 8080
selector:
app: hello-kubernetes
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-kubernetes
spec:
replicas: 3
selector:
matchLabels:
app: hello-kubernetes
template:
metadata:
labels:
app: hello-kubernetes
spec:
containers:
- name: hello-kubernetes
image: paulbouwer/hello-kubernetes:1.5
ports:
- containerPort: 8080
env:
- name: MESSAGE
value: I just deployed a PodVM on the SKS Cluster!!成果如下：
root@client:/home/zyi# kubectl get all |grep hello
pod/hello-kubernetes-74898bf87-5k8g4 1/1 Running 0 31h
pod/hello-kubernetes-74898bf87-kqwpb 1/1 Running 0 31h
pod/hello-kubernetes-74898bf87-w6tjq 1/1 Running 0 31h
service/hello-kubernetes LoadBalancer 10.96.3.145 192.168.81.128 80:30437/TCP 31h
deployment.apps/hello-kubernetes 3/3 3 3 31h
replicaset.apps/hello-kubernetes-74898bf87 3 3 3 31h通过 Web 拜访
CSI 测试
采纳 Kubernetes 官网 WordPress 的例子。
apiVersion: v1
kind: Service
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
ports:
- port: 3306
selector:
app: wordpress
tier: mysql
clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: wordpress
spec:
accessModes:
- ReadWriteOnce
storageClassName: smtx-elf-csi-driver #减少本环境的存储类
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
---
apiVersion: v1
kind: Service
metadata:
name: wordpress
labels:
app: wordpress
spec:
ports:
- port: 80
selector:
app: wordpress
tier: frontend
type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pv-claim
labels:
app: wordpress
spec:
accessModes:
- ReadWriteOnce
storageClassName: smtx-elf-csi-driver #减少本环境的存储类
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: wordpress
labels:
app: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: frontend
spec:
containers:
- image: wordpress:4.8-apache
name: wordpress
env:
- name: WORDPRESS_DB_HOST
value: wordpress-mysql
- name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
- containerPort: 80
name: wordpress
volumeMounts:
- name: wordpress-persistent-storage
mountPath: /var/www/html
volumes:
- name: wordpress-persistent-storage
persistentVolumeClaim:
claimName: wp-pv-claim以上 Yaml 应用的 StorageClass 能够查看
root@client:/home/zyi# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
smtx-elf-csi-driver (default) com.smartx.elf-csi-driver Delete Immediate false 32h在利用该 yaml 之前，须要创立所需的 Secret：mysql-pass
kubectl create secret generic mysql-pass --from-literal password='HC!r0cks'部署实现当前：
root@client:/home/zyi# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-b5690b99-bf61-432b-99d1-e5f1924b6acd 20Gi RWO smtx-elf-csi-driver 31h
wp-pv-claim Bound pvc-67040f10-89e1-4a86-8b5e-082786b67507 20Gi RWO smtx-elf-csi-driver 31h
root@client:/home/zyi# kubectl get svc |grep word
wordpress LoadBalancer 10.96.1.17 192.168.81.129 80:30090/TCP 31h
wordpress-mysql ClusterIP None <none> 3306/TCP 31h能够看到生产的 PVC，同样在零碎中也能够看到
查看 Wordpress 的可用性
以及能够应用了。
Contour 测试
Contour 官网：github.com/heptio/contour
实现：Go
许可证：Apache 2.0
Contour 和 Envoy 由同一个作者开发，它基于 Envoy。它最特地的性能是能够通过 CRD（IngressRoute）治理 Ingress 资源，对于多团队须要同时应用一个集群的组织来说，这有助于爱护相邻环境中的流量，使它们免受 Ingress 资源更改的影响。
它还提供了一组扩大的负载平衡算法（镜像、主动反复、限度申请率等），以及具体的流量和故障监控。
测试采纳的例子来源于 blog：
Kubernetes Ingress with Contour
-
环境：
SKS：K8S Cluster 3 contorl-plane，4 worker
Register：192.168.80.19
拓扑图
-
Go 文件
-
装置 golang
apt install golang-go -y go env -w GO111MODULE=auto go mod init basic-web-server -
Golang 文件 xx.go
package main import ( "fmt" "log" "net/http" "os" ) func main() {fmt.Println("Starting web server...") message := os.Getenv("MESSAGE") fmt.Printf("Using message: %s\n", message) http.HandleFunc("/", func(res http.ResponseWriter, req *http.Request) {fmt.Println("Request received") fmt.Fprintf(res, "Message: %s", message) }) log.Fatal(http.ListenAndServe(":8000", nil)) }
-
-
Dockerfile
-
文件
FROM golang:1.17 AS builder COPY . /var/app WORKDIR /var/app RUN ["go", "build", "-o", "app", "."] FROM debian:bullseye WORKDIR /var/app COPY --from=builder /var/app/app . CMD ["./app"] -
Build
docker build -t 192.168.80.19/sks/basic-web-server:v1 . root@client:/home/zyi/contour-test# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 192.168.80.19/sks/basic-web-server v1 eafc463bc7e4 49 minutes ago 130MB
-
-
Push to Harbor
- Download file ca.crt
-
Create folder and cp ca to it
mkdir -p /etc/docker/certs.d/192.168.80.19 cp ca.crt /etc/docker/certs.d/192.168.80.19/ -
Login and Push
docker login 192.168.80.19 docker push 192.168.80.19/sks/basic-web-server:v1
-
Create the applications in the Kubernetes cluster
-
App1
kind: Namespace apiVersion: v1 metadata: name: app1 --- kind: Deployment apiVersion: apps/v1 metadata: name: app1 namespace: app1 spec: replicas: 2 selector: matchLabels: app: app1 template: metadata: labels: app: app1 spec: containers: - name: app1 image: 192.168.80.19/sks/basic-web-server:v1 imagePullPolicy: Always ports: - containerPort: 8000 env: - name: MESSAGE value: hello from app1 --- kind: Service apiVersion: v1 metadata: name: app1 namespace: app1 spec: selector: app: app1 ports: - port: 80 targetPort: 8000 -
App2:
kind: Namespace apiVersion: v1 metadata: name: app2 --- kind: Deployment apiVersion: apps/v1 metadata: name: app2 namespace: app2 spec: replicas: 2 selector: matchLabels: app: app2 template: metadata: labels: app: app2 spec: containers: - name: app2 image: 192.168.80.19/sks/basic-web-server:v1 imagePullPolicy: Always ports: - containerPort: 8000 env: - name: MESSAGE value: hello from app2 --- kind: Service apiVersion: v1 metadata: name: app2 namespace: app2 spec: selector: app: app2 ports: - port: 80 targetPort: 8000
-
-
Proxy
Setup the routing so that our Contour ingress controller knows where to send traffic to. Do this by creating a single root proxy and include two other proxies to direct traffic to their respective application.-
App-proxy
kind: HTTPProxy apiVersion: projectcontour.io/v1 metadata: name: app1 namespace: app1 spec: routes: - services: - name: app1 port: 80 --- kind: HTTPProxy apiVersion: projectcontour.io/v1 metadata: name: app2 namespace: app2 spec: routes: - services: - name: app2 port: 80 -
There’s not much to these, they are pretty much just target proxies that can be referenced and route all traffic to their service and port. The real routing comes into play with the root proxy:
kind: HTTPProxy apiVersion: projectcontour.io/v1 metadata: name: main spec: virtualhost: fqdn: myapps includes: - name: app1 namespace: app1 conditions: - prefix: /app1 - name: app2 namespace: app2 conditions: - prefix: /app2Check the proxies
root@client:/home/zyi/contour-test# kubectl get httpproxies.projectcontour.io -A NAMESPACE NAME FQDN TLS SECRET STATUS STATUS DESCRIPTION app1 app1 valid Valid HTTPProxy app2 app2 valid Valid HTTPProxy default main myapps valid Valid HTTPProxy
-
-
Check the results
root@client:/home/zyi/contour-test# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE app1 app1 ClusterIP 10.96.2.206 <none> 80/TCP 58m app2 app2 ClusterIP 10.96.1.37 <none> 80/TCP 58m calico-apiserver calico-api ClusterIP 10.96.1.71 <none> 443/TCP 6h53m calico-system calico-kube-controllers-metrics ClusterIP 10.96.0.251 <none> 9094/TCP 6h53m calico-system calico-typha ClusterIP 10.96.2.255 <none> 5473/TCP 6h53m default hello-kubernetes LoadBalancer 10.96.3.145 192.168.81.128 80:30437/TCP 6h42m default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6h54m default wordpress LoadBalancer 10.96.1.17 192.168.81.129 80:30090/TCP 6h32m default wordpress-mysql ClusterIP None <none> 3306/TCP 6h32m kapp-controller packaging-api ClusterIP 10.96.2.99 <none> 443/TCP 6h54m kube-system kube-controller-manager ClusterIP 10.96.2.92 <none> 10257/TCP 145m kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 6h54m kube-system kube-scheduler ClusterIP 10.96.3.58 <none> 10259/TCP 145m kube-system kubelet ClusterIP None <none> 10250/TCP,10255/TCP,4194/TCP 145m sks-system-contour contour ClusterIP 10.96.3.100 <none> 8001/TCP 5h59m sks-system-contour contour-default-backend ClusterIP 10.96.0.83 <none> 80/TCP 5h59m sks-system-contour contour-envoy LoadBalancer 10.96.1.236 192.168.81.130 80:31054/TCP,443:32273/TCP 5h59mroot@client:/home/zyi/contour-test# INGRESS_IP=$(kubectl get svc -n sks-system-contour contour-envoy -o jsonpath='{.status.loadBalancer.ingress[0].ip}') root@client:/home/zyi/contour-test# echo $INGRESS_IP 192.168.81.130 -
Curl app
root@client:/home/zyi/contour-test# curl -H "host:myapps" $INGRESS_IP/app1 Message: hello from app1 root@client:/home/zyi/contour-test# curl -H "host:myapps" $INGRESS_IP/app2 Message: hello from app2
附 VMware Example:
VMware 在官网的举荐的 Ingress 工具也是 Contour，不过官网的例子没有更新，链接：
应用 Contour 的 Ingress
Error :
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-nihao
spec:
rules:
- http:
paths:
- path: /nihao
backend:
serviceName: nihao
servicePort: 80root@client:/home/zyi/contour-test# kubectl apply -f nihao.yaml
Error from server (BadRequest): error when creating "nihao.yaml": Ingress in version "v1" cannot be handled as a Ingress: strict decoding error: unknown field "spec.rules[0].http.paths[0].backend.serviceName", unknown field "spec.rules[0].http.paths[0].backend.servicePort"change new:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-nihao
spec:
rules:
- http:
paths:
- path: /nihao
pathType: Prefix
backend:
service:
name: nihao
port:
number: 80