K8s和TiDB都是目前开源社区中活跃的开源产品，TiDB
Operator项目是一个在K8s上编排管理TiDB集群的项目。本文详细记录了部署K8s及install TiDB
Operator的详细实施过程，希望能对刚"入坑"的同学有所帮助。
一、环境
Ubuntu 16.04
K8s 1.14.1
二、Kubespray安装K8s
配置免密登录
1 yum -y install expect- vi /tmp/autocopy.exp
1 #!/usr/bin/expect 2 3 set timeout 4 set user_hostname [lindex $argv ] 5 set password [lindex $argv ] 6 spawn ssh-copy-id $user_hostname 7 expect { 8 "(yes/no)?" 9 {10 send "yes\n"11 expect "*assword:" { send "$password\n"}12 }13 "*assword:"14 {15 send "$password\n"16 }17 }18 expect eof 1 ssh-keyscan addedip >> ~/.ssh/known_hosts 2 3 ssh-keygen -t rsa -P '' 4 5 for i in 10.0.0.{31,32,33,40,10,20，50}; do ssh-keyscan $i >> ~/.ssh/known_hosts ; done 6 7 /tmp/autocopy.exp root@addeip 8 ssh-copy-id addedip 910 /tmp/autocopy.exp root@10.0.0.3111 /tmp/autocopy.exp root@10.0.0.3212 /tmp/autocopy.exp root@10.0.0.3313 /tmp/autocopy.exp root@10.0.0.4014 /tmp/autocopy.exp root@10.0.0.1015 /tmp/autocopy.exp root@10.0.0.2016 /tmp/autocopy.exp root@10.0.0.50配置Kubespray
1 pip install -r requirements.txt2 cp -rfp inventory/sample inventory/mycluster- inventory/mycluster/inventory.ini
- inventory/mycluster/inventory.ini
1 # ## Configure 'ip' variable to bind kubernetes services on a 2 # ## different ip than the default iface 3 # ## We should set etcd_member_name for etcd cluster. The node that is not a etcd member do not need to set the value, or can set the empty string value. 4 [all] 5 # node1 ansible_host=95.54.0.12 # ip=10.3.0.1 etcd_member_name=etcd1 6 # node2 ansible_host=95.54.0.13 # ip=10.3.0.2 etcd_member_name=etcd2 7 # node3 ansible_host=95.54.0.14 # ip=10.3.0.3 etcd_member_name=etcd3 8 # node4 ansible_host=95.54.0.15 # ip=10.3.0.4 etcd_member_name=etcd4 9 # node5 ansible_host=95.54.0.16 # ip=10.3.0.5 etcd_member_name=etcd510 # node6 ansible_host=95.54.0.17 # ip=10.3.0.6 etcd_member_name=etcd611 etcd1 ansible_host=10.0.0.31 etcd_member_name=etcd112 etcd2 ansible_host=10.0.0.32 etcd_member_name=etcd213 etcd3 ansible_host=10.0.0.33 etcd_member_name=etcd314 master1 ansible_host=10.0.0.4015 node1 ansible_host=10.0.0.1016 node2 ansible_host=10.0.0.2017 node3 ansible_host=10.0.0.501819 # ## configure a bastion host if your nodes are not directly reachable20 # bastion ansible_host=x.x.x.x ansible_user=some_user2122 [kube-master]23 # node124 # node225 master126 [etcd]27 # node128 # node229 # node330 etcd131 etcd232 etcd33334 [kube-node]35 # node236 # node337 # node438 # node539 # node640 node141 node242 node34344 [k8s-cluster:children]45 kube-master46 kube-node节点所需镜像的文件
由于某些镜像国内无法访问需要现将镜像通过代理下载到本地然后上传到本地镜像仓库或DockerHub，同时修改配置文件，个别组件存放位置https://storage.googleapis.com，需要新建Nginx服务器分发文件。
建立Nginx服务器
- ~/distribution/docker-compose.yml
- 创建文件目录及Nginx配置文件目录
- ~/distribution/conf.d/open_distribute.conf
- 启动
- 下载并上传所需文件 具体版本号参考roles/download/defaults/main.yml文件中kubeadm_version、kube_version、image_arch参数
- 安装Docker及Docker-Compose
1 apt-get install \ 2 apt-transport-https \ 3 ca-certificates \ 4 curl \ 5 gnupg-agent \ 6 software-properties-common 7 8 curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - 910 add-apt-repository \11 "deb [arch=amd64] https://download.docker.com/linux/ubuntu \12 $(lsb_release -cs) \13 stable"1415 apt-get update1617 apt-get install docker-ce docker-ce-cli containerd.io1819 chmod +x /usr/local/bin/docker-compose20 sudo curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose- 新建Nginx docker-compose.yml
1 mkdir ~/distribution2 vi ~/distribution/docker-compose.yml 1 # distribute 2 version: '2' 3 services: 4 distribute: 5 image: nginx:1.15.12 6 volumes: 7 - ./conf.d:/etc/nginx/conf.d 8 - ./distributedfiles:/usr/share/nginx/html 9 network_mode: "host"10 container_name: nginx_distribute 1 mkdir ~/distribution/distributedfiles2 mkdir ~/distribution/3 mkdir ~/distribution/conf.d4 vi ~/distribution/conf.d/open_distribute.conf 1 #open_distribute.conf 2 3 server { 4 #server_name distribute.search.leju.com; 5 listen 8888; 6 7 root /usr/share/nginx/html; 8 9 add_header Access-Control-Allow-Origin *; 10 add_header Access-Control-Allow-Headers X-Requested-With; 11 add_header Access-Control-Allow-Methods GET,POST,OPTIONS; 1213 location / {14 # index index.html;15 autoindex on; 16 }17 expires off;18 location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|eot|ttf|woff|woff2|svg)$ {19 expires -1;20 }2122 location ~ .*\.(js|css)?$ {23 expires -1 ;24 }25 } # end of public static files domain : [ distribute.search.leju.com ]1 docker-compose up -d1 wget https://storage.googleapis.com/kubernetes-release/release/v1.14.1/bin/linux/amd64/kubeadm23 scp /tmp/kubeadm 10.0.0.60:/root/distribution/distributedfiles45 wget https://storage.googleapis.com/kubernetes-release/release/v1.14.1/bin/linux/amd64/hyperkube- 需要下载并上传到私有仓库的镜像
1 docker pull k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.4.0 2 docker tag k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.4.0 jiashiwen/cluster-proportional-autoscaler-amd64:1.4.0 3 docker push jiashiwen/cluster-proportional-autoscaler-amd64:1.4.0 4 5 docker pull k8s.gcr.io/k8s-dns-node-cache:1.15.1 6 docker tag k8s.gcr.io/k8s-dns-node-cache:1.15.1 jiashiwen/k8s-dns-node-cache:1.15.1 7 docker push jiashiwen/k8s-dns-node-cache:1.15.1 8 9 docker pull gcr.io/google_containers/pause-amd64:3.110 docker tag gcr.io/google_containers/pause-amd64:3.1 jiashiwen/pause-amd64:3.111 docker push jiashiwen/pause-amd64:3.11213 docker pull gcr.io/google_containers/kubernetes-dashboard-amd64:v1.10.114 docker tag gcr.io/google_containers/kubernetes-dashboard-amd64:v1.10.1 jiashiwen/kubernetes-dashboard-amd64:v1.10.115 docker push jiashiwen/kubernetes-dashboard-amd64:v1.10.11617 docker pull gcr.io/google_containers/kube-apiserver:v1.14.118 docker tag gcr.io/google_containers/kube-apiserver:v1.14.1 jiashiwen/kube-apiserver:v1.14.119 docker push jiashiwen/kube-apiserver:v1.14.12021 docker pull gcr.io/google_containers/kube-controller-manager:v1.14.122 docker tag gcr.io/google_containers/kube-controller-manager:v1.14.1 jiashiwen/kube-controller-manager:v1.14.123 docker push jiashiwen/kube-controller-manager:v1.14.12425 docker pull gcr.io/google_containers/kube-scheduler:v1.14.126 docker tag gcr.io/google_containers/kube-scheduler:v1.14.1 jiashiwen/kube-scheduler:v1.14.127 docker push jiashiwen/kube-scheduler:v1.14.12829 docker pull gcr.io/google_containers/kube-proxy:v1.14.130 docker tag gcr.io/google_containers/kube-proxy:v1.14.1 jiashiwen/kube-proxy:v1.14.131 docker push jiashiwen/kube-proxy:v1.14.13233 docker pull gcr.io/google_containers/pause:3.134 docker tag gcr.io/google_containers/pause:3.1 jiashiwen/pause:3.135 docker push jiashiwen/pause:3.13637 docker pull gcr.io/google_containers/coredns:1.3.138 docker tag gcr.io/google_containers/coredns:1.3.1 jiashiwen/coredns:1.3.139 docker push jiashiwen/coredns:1.3.1- 用于下载上传镜像的脚本
1 #!/bin/bash 2 3 privaterepo=jiashiwen 4 5 k8sgcrimages=( 6 cluster-proportional-autoscaler-amd64:1.4.0 7 k8s-dns-node-cache:1.15.1 8 ) 910 gcrimages=(11 pause-amd64:3.112 kubernetes-dashboard-amd64:v1.10.113 kube-apiserver:v1.14.114 kube-controller-manager:v1.14.115 kube-scheduler:v1.14.116 kube-proxy:v1.14.117 pause:3.118 coredns:1.3.119 )202122 for k8sgcrimageName in ${k8sgcrimages[@]} ; do23 echo $k8sgcrimageName24 docker pull k8s.gcr.io/$k8sgcrimageName25 docker tag k8s.gcr.io/$k8sgcrimageName $privaterepo/$k8sgcrimageName26 docker push $privaterepo/$k8sgcrimageName27 done282930 for gcrimageName in ${gcrimages[@]} ; do31 echo $gcrimageName32 docker pull gcr.io/google_containers/$gcrimageName33 docker tag gcr.io/google_containers/$gcrimageName $privaterepo/$gcrimageName34 docker push $privaterepo/$gcrimageName35 done- 修改文件inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml，修改K8s镜像仓库
1 # kube_image_repo: "gcr.io/google-containers"2 kube_image_repo: "jiashiwen"- 修改roles/download/defaults/main.yml
1 #dnsautoscaler_image_repo: "k8s.gcr.io/cluster-proportional-autoscaler-{{ image_arch }}" 2 dnsautoscaler_image_repo: "jiashiwen/cluster-proportional-autoscaler-{{ image_arch }}" 3 4 #kube_image_repo: "gcr.io/google-containers" 5 kube_image_repo: "jiashiwen" 6 7 #pod_infra_image_repo: "gcr.io/google_containers/pause-{{ image_arch }}" 8 pod_infra_image_repo: "jiashiwen/pause-{{ image_arch }}" 910 #dashboard_image_repo: "gcr.io/google_containers/kubernetes-dashboard-{{ image_arch }}"11 dashboard_image_repo: "jiashiwen/kubernetes-dashboard-{{ image_arch }}"1213 #nodelocaldns_image_repo: "k8s.gcr.io/k8s-dns-node-cache"14 nodelocaldns_image_repo: "jiashiwen/k8s-dns-node-cache"1516 #kubeadm_download_url: "https://storage.googleapis.com/kubernetes-release/ release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm"17 kubeadm_download_url: "http://10.0.0.60:8888/kubeadm"1819 #hyperkube_download_url: "https://storage.googleapis.com/ kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/ hyperkube"20 hyperkube_download_url: "http://10.0.0.60:8888/hyperkube"三、执行安装
- 安装命令
1 ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml- 重置命令
1 ansible-playbook -i inventory/mycluster/inventory.ini reset.yml四、验证K8s集群
安装Kubectl
- 本地浏览器打开https://storage.googleapis.co...
- 用上一步得到的最新版本号v1.7.1替换下载地址中的$(curl -s https://storage.googleapis.co...:// storage.googleapis.com/kubernetes-release/release/v1.14.1/bin/linux/amd64/kubectl
- 上传下载好的kubectl
1 scp /tmp/kubectl root@xxx:/root- 修改属性
1 chmod +x ./kubectl2 mv ./kubectl /usr/local/bin/kubectl- Ubuntu
1 sudo snap install kubectl --classic- CentOS
将master节点上的~/.kube/config 文件复制到你需要访问集群的客户端上即可
1 scp 10.0.0.40:/root/.kube/config ~/.kube/config执行命令验证集群
1 kubectl get nodes2 kubectl cluster-info五、TiDB-Operaor部署
安装helm
https://blog.csdn.net/bbwangj...
- 安装helm
1 curl https://raw.githubusercontent.com/helm/helm/master/scripts/get > get_helm.sh2 chmod 700 get_helm.sh3 ./get_helm.sh- 查看helm版本
1 helm version- 初始化
1 helm init --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.13.1 --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts为K8s提供 local volumes
- 参考文档https://github.com/kubernetes...
tidb-operator启动会为pd和tikv绑定pv，需要在discovery directory下创建多个目录 - 格式化并挂载磁盘
1 mkfs.ext4 /dev/vdb2 DISK_UUID=$(blkid -s UUID -o value /dev/vdb) 3 mkdir /mnt/$DISK_UUID4 mount -t ext4 /dev/vdb /mnt/$DISK_UUID- /etc/fstab持久化mount
1 echo UUID=`sudo blkid -s UUID -o value /dev/vdb` /mnt/$DISK_UUID ext4 defaults 0 2 | sudo tee -a /etc/fstab- 创建多个目录并mount到discovery directory
1 for i in $(seq 1 10); do2 sudo mkdir -p /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i}3 sudo mount --bind /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i}4 done- /etc/fstab持久化mount
1 for i in $(seq 1 10); do2 echo /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i} none bind 0 0 | sudo tee -a /etc/fstab3 done- 为tidb-operator创建local-volume-provisioner
1 $ kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/master/manifests/local-dind/local-volume-provisioner.yaml2 $ kubectl get po -n kube-system -l app=local-volume-provisioner3 $ kubectl get pv --all-namespaces | grep local-storage 六、Install TiDB Operator
- 项目中使用了gcr.io/google-containers/hyperkube，国内访问不了，简单的办法是把镜像重新push到dockerhub然后修改charts/tidb-operator/values.yaml
1 scheduler: 2 # With rbac.create=false, the user is responsible for creating this account 3 # With rbac.create=true, this service account will be created 4 # Also see rbac.create and clusterScoped 5 serviceAccount: tidb-scheduler 6 logLevel: 2 7 replicas: 1 8 schedulerName: tidb-scheduler 9 resources:10 limits:11 cpu: 250m12 memory: 150Mi13 requests:14 cpu: 80m15 memory: 50Mi16 # kubeSchedulerImageName: gcr.io/google-containers/hyperkube17 kubeSchedulerImageName: yourrepo/hyperkube18 # This will default to matching your kubernetes version19 # kubeSchedulerImageTag: latest- TiDB Operator使用CRD扩展Kubernetes，因此要使用TiDB Operator，首先应该创建TidbCluster自定义资源类型。
1 kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/master/manifests/crd.yaml2 kubectl get crd tidbclusters.pingcap.com- 安装TiDB-Operator
1 $ git clone https://github.com/pingcap/tidb-operator.git2 $ cd tidb-operator3 $ helm install charts/tidb-operator --name=tidb-operator --namespace=tidb-admin4 $ kubectl get pods --namespace tidb-admin -l app.kubernetes.io/ instance=tidb-operator七、部署TiDB
1 helm install charts/tidb-cluster --name=demo --namespace=tidb2 watch kubectl get pods --namespace tidb -l app.kubernetes.io/instance=demo -o wide八、验证
安装MySQL客户端
- 参考文档https://dev.mysql.com/doc/ref...
- CentOS安装
1 wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm2 yum localinstall mysql80-community-release-el7-3.noarch.rpm -y3 yum repolist all | grep mysql4 yum-config-manager --disable mysql80-community5 yum-config-manager --enable mysql57-community6 yum install mysql-community-client- Ubuntu安装
1 wget https://dev.mysql.com/get/mysql-apt-config_0.8.13-1_all.deb2 dpkg -i mysql-apt-config_0.8.13-1_all.deb3 apt update45 # 选择MySQL版本6 dpkg-reconfigure mysql-apt-config7 apt install mysql-client -y九、映射TiDB端口
- 查看TiDB Service
1 kubectl get svc --all-namespaces- 映射TiDB端口
1 # 仅本地访问2 kubectl port-forward svc/demo-tidb 4000:4000 --namespace=tidb34 # 其他主机访问5 kubectl port-forward --address 0.0.0.0 svc/demo-tidb 4000:4000 --namespace=tidb- 首次登录MySQL
1 mysql -h 127.0.0.1 -P 4000 -u root -D test- 修改TiDB密码
1 SET PASSWORD FOR 'root'@'%' = 'wD3cLpyO5M'; FLUSH PRIVILEGES;趟坑小记
1、K8s国内安装
K8s镜像多在gcr.io国内访问不到，基本做法是把镜像导入DockerHub或者私有镜像，这一点在K8s部署章节有详细过程就不累述了。
2、TiDB-Operator 本地存储配置
Operator在启动集群时pd和TiKV需要绑定本地存储如果挂载点不足会导致pod启动过程中找不到可已bond的pv始终处于pending或createing状态，详细配请参阅https://github.com/kubernetes...“Sharing a disk filesystem by multiple filesystem PVs”一节，同一块磁盘绑定多个挂载目录，为Operator提供足够的bond
3、MySQL客户端版本问题
目前TiDB只支持MySQL5.7版本客户端8.0会报ERROR 1105 (HY000): Unknown charset id 255
点击"K8s"了解更多详情。
文章转载自公众号"北京IT爷们儿"