前言
最近在学习如何应用jenkins联合阿里云k8s自动化集成和部署springcloud微服务和前端next.js我的项目，当初记录下来分享给大家，文章有什么不妥的中央，心愿大佬们批评指正。
DevOps CI/CD架构
面向对象
- 纯熟应用vue或者react
- 纯熟应用SpringCloud微服务
- 纯熟应用docker容器
- 纯熟应用jenkins自动化运维工具
- 纯熟应用k8s(deployment、service、ingress)
筹备工作
1、购买阿里云ACK集群(或者自行搭建)
我购买的是阿里云ACK托管版，创立集群地址
留神：创立ACK集群不免费，免费的是外面的NAT网关、SLB负载平衡、ECS服务器等
2、装置gitlab
这里我之前写了一篇阿里云ECS搭建gitlab，装置gitlab地址
3、装置jenkins
这里我之前写了一篇阿里云ECS搭建jenkins，装置jenkins地址
4、装置docker
应用官网举荐yum 装置比拟不便，装置如下：
sudo yum install -y yum-utils \ device-mapper-persistent-data \ lvm2sudo yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.reposudo yum install docker-ce docker-ce-cli containerd.io零碎架构
Next.js 前端开发
Next.js我的项目：
服务：react+next.js服务端口：3000，K8S：Deployment+Server+IngressPod名：demo-webappDockerfile
FROM node:12# 设置工作门路。所有的门路都会关联WORKDIRWORKDIR /usr/src/app# 装置依赖COPY package*.json ./RUN npm install# 拷贝源文件COPY . .# 构建利用RUN npm run build# 运行利用CMD [ "npm", "start" ]SpringCloud微服务开发
1、服务发现：
微服务：SpringCloud Eureka，微服务名：demo-eureka-server微服务端口：8761，K8S：Deployment+Service+Ingress（因为要通过网址查看服务注册状况，所以要增加ingress）Pod名：demo-eureka-server2、服务配置：
微服务：SpringCloud Config微服务名：demo-config-server微服务端口：8888，K8S：Deployment+ServicePod名：demo-config-server3、服务认证和受权：
微服务：SpringSecurity + Oauth2微服务名：demo-auth-service微服务端口：8901，K8S：Deployment+ServicePod名：demo-auth-service4、网关服务：
微服务：SpringCloud Zuul微服务名：demo-auth-service微服务端口：5555，K8S：Deployment+Service+Ingress（服务网关是所有服务对外惟一入口，所以要配置ingress）Pod名：demo-auth-service5、编写Dockerfile、K8S yaml
为了不便起见，下面的微服务名和Pod名都设置雷同名称，具体的服务开发过程这里就临时疏忽，前面有工夫再写篇搭建企业级微服务的文章吧，相干微服务的目录构造如下所示：
留神：不同服务之间Dockerfile和k8s yaml文件大同小异，咱们以Eureka为例：
Eureka Dockerfile：
FROM openjdk:8-jdk-alpineMAINTAINER "zhangwei"<zhangwei900808@126.com>RUN mkdir -p /usr/local/configsvrARG JAR_FILEADD ${JAR_FILE} /usr/local/configsvr/app.jarENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/usr/local/configsvr/app.jar"]EXPOSE 8888Eureka K8S Yaml：
apiVersion: extensions/v1beta1kind: Ingressmetadata: annotations: nginx.ingress.kubernetes.io/configuration-snippet: | rewrite ^/eureka/css/(.*)$ /eureka/eureka/css/$1 redirect; rewrite ^/eureka/js/(.*)$ /eureka/eureka/js/$1 redirect; nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/service-weight: '' generation: 4 name: <podname>-ingress namespace: defaultspec: rules: - host: baidu.com http: paths: - backend: serviceName: <podname>-svc servicePort: 8761 path: /eureka(/|$)(.*) pathType: ImplementationSpecific---apiVersion: v1kind: Servicemetadata: name: <podname>-svc namespace: defaultspec: externalTrafficPolicy: Local ports: - nodePort: 31061 port: 8761 protocol: TCP targetPort: 8761 selector: app: <podname> sessionAffinity: None type: NodePort---apiVersion: apps/v1kind: Deploymentmetadata: annotations: deployment.kubernetes.io/revision: '1' generation: 1 labels: app: <podname> name: <podname> namespace: defaultspec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: <podname> strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: <podname> spec: containers: - env: - name: LANG value: C.UTF-8 - name: JAVA_HOME value: /usr/lib/jvm/java-1.8-openjdk image: <imagename> imagePullPolicy: IfNotPresent name: <podname> ports: - containerPort: 8761 protocol: TCP resources: requests: cpu: 250m memory: 512Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30留神：不同的策服务bootstrap.yml和application.yml文件还是有所区别，比方上面这些：
demo-auth-service、demo-zuul-server 配置bootstrap.yml文件增加SpringCloud Config地址：
spring: application: name: authservice cloud: config: enabled: true # config的Server名 uri: http://demo-config-server-svc:8888demo-auth-service、demo-zuul-server、demo-config-server 配置application.yml文件增加Eureka地址：
eureka: instance: preferIpAddress: true hostname: demo-eureka-server-svc client: register-with-eureka: true fetch-registry: true service-url: # eureka的Server名 defaultZone: http://demo-eureka-server-svc:8761/eureka/Git工作流
依照Git工作流，个别咱们能够分为：开发环境(develop)、测试环境(release)、预生产环境(uat)和生产环境(prop)，对应的分支别离是：dev、test、release、master，因而不同阶段提交的代码所属分支也不雷同，且dockerfile、jenkins pipline、k8s yaml文件也不雷同，这个要留神下。
Jenkins DevOps CI/CD
1、视图标准
jenkins能够依照git工作流增加：测试视图、预生产视图、生产视图，如下所示：
2、创立流水线工作
咱们先创立流水线工作并编写pipline script脚本来创立CI/CD流水线步骤
3、编写前端的pipline script
先编写环境变量
environment { GIT_REPOSITORY="前端代码仓库地址" K8S_YAML="k8s yaml文件所在目录" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库明码" ALIYUN_DOCKER_HOST = '阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名"}步骤一：克隆代码
stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() // 测试分支，jenkins-gitlab-ssh-hash是ssh密钥，替换成本人的就好 git branch: 'test', credentialsId: 'jenkins-gitlab-ssh-hash', url: "${GIT_REPOSITORY}" script { // 获取git提交的hash值做为docker镜像tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() // 组装成残缺地址 DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } }}步骤二：代码测试
stage("Test") { steps { echo "2.Test Stage" }}步骤三：制作docker镜像
stage("Build") { steps { echo "3.Build Docker Image Stage" sh "docker build -t ${DOCKER_REPOSITORY_TAG} -f docker/Dockerfile ." }}步骤四：推送docker镜像
stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像，username 跟 password 为 阿里云容器镜像服务的账号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' }}步骤五：k8s部署docker镜像
stage("Deploy") { steps { echo "5.公布镜像" // 应用sed替换k8s yaml文件中的<imagename>和<podname> sh "sed -i 's#<imagename>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" // 执行利用k8s yaml sh "kubectl apply -f ${K8S_YAML}" }}sed 语法大家能够自行百度，这里我应用的是#分隔，而没有应用/分隔，起因是分隔的字符中蕴含/，所以不能再用。
首先，咱们来看看前端k8s yaml文件
apiVersion: extensions/v1beta1kind: Ingressmetadata: annotations: cert-manager.io/cluster-issuer: letsencrypt-prod-http01 kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' nginx.ingress.kubernetes.io/service-weight: '' generation: 3 name: <podname>-ingress namespace: defaultspec: rules: - host: baidu.com http: paths: - backend: serviceName: <podname>-svc servicePort: 3000 path: / pathType: ImplementationSpecific tls: - hosts: - baidu.com secretName: <podname>-ingress---apiVersion: v1kind: Servicemetadata: name: <podname>-svc namespace: defaultspec: ports: - port: 3000 protocol: TCP targetPort: 3000 selector: app: <podname> sessionAffinity: None type: ClusterIP---apiVersion: apps/v1kind: Deploymentmetadata: annotations: deployment.kubernetes.io/revision: '1' generation: 1 labels: app: <podname> name: <podname> namespace: defaultspec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: <podname> strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app:<podname> spec: containers: - image: <imagename> imagePullPolicy: IfNotPresent name: <podname> resources: requests: cpu: 250m memory: 512Mi下面的yaml文件中蕴含：deployment、service和ingress，ingress中配置了tls，因而能够应用https来拜访域名，并且配置了nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'所以能够主动将http 跳转到https，其中外面的<podname>和<imagename>是要被sed替换成实在名称和地址。
残缺的pipline script脚本：
pipeline { agent any environment { GIT_REPOSITORY="前端代码仓库地址" K8S_YAML="k8s yaml文件所在目录" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库明码" ALIYUN_DOCKER_HOST = '阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名" } stages { stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() // 测试分支，jenkins-gitlab-ssh-hash是ssh密钥，替换成本人的就好 git branch: 'test', credentialsId: 'jenkins-gitlab-ssh-hash', url: "${GIT_REPOSITORY}" script { // 获取git提交的hash值做为docker镜像tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() // 组装成残缺地址 DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } } } stage("Test") { steps { echo "2.Test Stage" } } stage("Build") { steps { echo "3.Build Docker Image Stage" sh "docker build -t ${DOCKER_REPOSITORY_TAG} -f docker/Dockerfile ." } } stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像，username 跟 password 为 阿里云容器镜像服务的账号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' } } stage("Deploy") { steps { echo "5.公布镜像" // 应用sed替换k8s yaml文件中的<imagename>和<podname> sh "sed -i 's#<imagename>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" // 执行利用k8s yaml sh "kubectl apply -f ${K8S_YAML}" } } }}4、编写微服务的pipline script
先编写环境变量
environment { GIT_REPOSITORY="代码仓库地址" MODULE_NAME="maven 模块名" POD_NAME="k8s pod name" K8S_YAML="${MODULE_NAME}/src/main/k8s/eurekasvr.yaml" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库明码" ALIYUN_DOCKER_HOST = 阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名"}步骤一：克隆代码
stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() // 测试分支，jenkins-gitlab-ssh-hash是ssh密钥，替换成本人的就好 git branch: 'test', credentialsId: 'jenkins-gitlab-ssh-hash', url: "${GIT_REPOSITORY}" script { // 获取git提交的hash值做为docker镜像tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() // 组装成残缺地址 DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } }}步骤二：代码测试
stage("Test") { steps { echo "2.Test Stage" }}步骤三：制作docker镜像
stage("Build") { steps { echo "3.Build Server" sh "mvn -e -U -pl ${MODULE_NAME} -am clean package -Dmaven.test.skip=true dockerfile:build -Ddockerfile.tag=${GIT_TAG} -Ddockerfile.repository=${DOCKER_REPOSITORY}" }}这里阐明一下，因为咱们应用的是maven 多模块所以打包编译的时候要分模块来打包，所以要应用-pl指定模块名，而后咱们pom.xml中应用了dockerfile-maven-plugin，如下所示：
<build> <plugins> <plugin> <groupId>com.spotify</groupId> <artifactId>dockerfile-maven-plugin</artifactId> <version>1.4.10</version> <configuration> <!-- 指定dockerfile所在目录--> <dockerfile>src/main/docker/Dockerfile</dockerfile> <buildArgs> <!--提供参数向Dockerfile传递--> <JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE> </buildArgs> </configuration> </plugin> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins></build>步骤四：推送docker镜像
stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像，username 跟 password 为 阿里云容器镜像服务的账号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' }} 步骤五：k8s部署docker镜像
stage("Deploy") { steps { echo "5.公布镜像" // 应用sed替换k8s yaml文件中的<imagename>和<podname> sh "sed -i 's#<imagename>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" // 执行利用k8s yaml sh "kubectl apply -f ${K8S_YAML}" }}sed 语法大家能够自行百度，这里我应用的是#分隔，而没有应用/分隔，起因是分隔的字符中蕴含/，所以不能再用。
首先，咱们来看看Eureka k8s yaml文件
apiVersion: extensions/v1beta1kind: Ingressmetadata: annotations: nginx.ingress.kubernetes.io/configuration-snippet: | rewrite ^/eureka/css/(.*)$ /eureka/eureka/css/$1 redirect; rewrite ^/eureka/js/(.*)$ /eureka/eureka/js/$1 redirect; nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/service-weight: '' generation: 4 name: <podname>-ingress namespace: defaultspec: rules: - host: baidu.com http: paths: - backend: serviceName: <podname>-svc servicePort: 8761 path: /eureka(/|$)(.*) pathType: ImplementationSpecific---apiVersion: v1kind: Servicemetadata: name: <podname>-svc namespace: defaultspec: externalTrafficPolicy: Local ports: - nodePort: 31061 port: 8761 protocol: TCP targetPort: 8761 selector: app: <podname> sessionAffinity: None type: NodePort---apiVersion: apps/v1kind: Deploymentmetadata: annotations: deployment.kubernetes.io/revision: '1' generation: 1 labels: app: <podname> name: <podname> namespace: defaultspec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: <podname> strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: <podname> spec: containers: - env: - name: LANG value: C.UTF-8 - name: JAVA_HOME value: /usr/lib/jvm/java-1.8-openjdk image: <imagename> imagePullPolicy: IfNotPresent name: <podname> ports: - containerPort: 8761 protocol: TCP resources: requests: cpu: 250m memory: 512Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30下面的yaml文件中蕴含：deployment、service和ingress，ingress中配置了tls，因而能够应用https来拜访域名，并且配置了nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'所以能够主动将http 跳转到https，其中外面的<podname>和<imagename>是要被sed替换成实在名称和地址。
残缺的pipline script脚本：
pipeline { agent any environment { GIT_REPOSITORY="代码仓库地址" MODULE_NAME="maven 模块名" POD_NAME="k8s pod name" K8S_YAML="${MODULE_NAME}/src/main/k8s/eurekasvr.yaml" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库明码" ALIYUN_DOCKER_HOST = 阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名" } stages { stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() git branch: 'test',credentialsId: '1297dda3-e592-4e70-8fb0-087a26c08db0', url: "${GIT_REPOSITORY}" script { // 获取git代码tag为docker仓库tag // GIT_TAG = sh(returnStdout: true,script: 'git describe --tags --always').trim() // 获取git提交hash做为docker仓库tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" // docker 阿里镜像仓库 DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } } } stage("Test") { steps { echo "2.Test Stage" } } stage("Build") { steps { echo "3.Build Server" sh "mvn -e -U -pl ${MODULE_NAME} -am clean package -Dmaven.test.skip=true dockerfile:build -Ddockerfile.tag=${GIT_TAG} -Ddockerfile.repository=${DOCKER_REPOSITORY}" } } stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像，username 跟 password 为 阿里云容器镜像服务的账号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' } } stage("Deploy") { steps { echo "5.公布镜像" sh "sed -i 's#<dockerrepository>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" sh "kubectl apply -f ${K8S_YAML}" } } }}查看k8s是否胜利
能够通过命令
kubectl get deploykubectl get podkubectl get svckubectl get ingress还能够查看pod日志，对其胜利或失败进行剖析：
kubectl logs podname拜访Eureka
postman拜访微服务接口地址
能够通过postman接口拜访对外的zuul地址，查看是否能够认证通过：
浏览器拜访Web页面地址
能够通过next.js的部署地址，查看是否部署胜利：
总结
1、jenkins pipline script脚本和Dockerfile写法网上各式各样，大家找到一个符合标准的就好
2、kubectl apply -f 的作用是：如果没有创立deployment就会创立，否则就更新
3、微服务k8s yaml文件中Service要设置type: NodePort否则，微服务间不能通信
4、jenkins 编译微服务的maven模块要用到-pl 和 dockerfile-maven-plugin
5、微服务如果要用k8s ingress要设置https，以及http主动跳转到https
援用
JenkinsPipeline部署一个Kubernetes 利用
采纳jenkins pipeline实现主动构建并部署至k8s
Configuring-CI-CD-on-Kubernetes-with-Jenkins
spring-k8s
jenkins pipeline 主动构建并部署至k8s
微服务实战（一）基于OAUTH2.0对立认证受权的微服务基础架构
应用cert-manager申请收费的HTTPS证书
SPRINGBOOT利用SPRING.PROFILES.ACTIVE=@SPRING.ACTIVE@不同环境下灵便切换配置文件
https://kuboard.cn/learning/k8s-practice/ocp/eureka-server.html#%E6%9F%A5%E7%9C%8B%E9%83%A8%E7%BD%B2%E7%BB%93%E6%9E%9C
kubernetes部署微服务spring cloud的简略例子
k8s-nginx-ingress eureka二级门路转发的问题