CI/CD+Harbor+K8S实现微服务项目持续集成和发布

著作：行癫 <盗版必究> ------ ## 一：项目描述 #### 1.环境介绍 | 服务器 | IP | 角色 | 配置 | | :--------: | :------------: | :--------------------------------: | :-----: | | jenkins | 192.168.18.210 | 持续构建项目并发布 | 1Cpu+3G | | gitlab | 192.168.18.200 | （https://www.xingdian.com）版本库 | 2Cpu+8G | | harbor | 192.168.18.230 | 镜像仓库 | 1Cpu+3G | | NFS | 192.168.18.230 | 给k8s提供持久化存储 | 1Cpu+3G | | k8s-master | 192.168.18.160 | k8s管理节点 | 2Cpu+4G | | k8s-node-1 | 192.168.18.161 | k8s项目节点 | 1Cpu+3G | | k8s-node-2 | 192.168.18.162 | k8s项目节点 | 1Cpu+3G | | k8s-node-3 | 192.168.18.163 | k8s项目节点 | 1Cpu+3G | #### 2.jenkins部署 ```shell 1.上传jdk [root@jenkins ~]# tar xzf jdk-8u191-linux-x64.tar.gz -C /usr/local/ [root@jenkins ~]# cd /usr/local/ [root@jenkins local]# mv jdk1.8.0_191/ java 2.安装tomcat [root@jenkins ~]# wget http://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.42/bin/apache-tomcat-8.5.42.tar.gz [root@jenkins ~]# tar xzf apache-tomcat-8.5.42.tar.gz -C /usr/local/ [root@jenkins ~]# cd /usr/local/ [root@jenkins local]# mv apache-tomcat-8.5.42/ tomcat 3.安装maven [root@jenkins ~]# wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz [root@jenkins ~]# tar xzf apache-maven-3.5.4-bin.tar.gz -C /usr/local/java [root@jenkins ~]# cd /usr/local/java [root@jenkins java]# mv apache-maven-3.5.4/ maven 设置变量: [root@jenkins-server ~]# vim /etc/profile JAVA_HOME=/usr/local/java MAVEN_HOME=/usr/local/java/maven PATH=$PATH:$JAVA_HOME/bin:$MAVEN_HOME/bin export PATH JAVA_HOME MAVEN_HOME [root@jenkins-server ~]# source /etc/profile 验证: [root@jenkins-server ~]# java -version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode) [root@jenkins-server ~]# mvn -v Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00) Maven home: /usr/local/java/maven Java version: 1.8.0_191, vendor: Oracle Corporation, runtime: /usr/local/java/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-693.el7.x86_64", arch: "amd64", family: "unix" 下载jenkins的安装包: 安装jenkins:2.332.3 ----通过官网直接下载war包。 官网:http://updates.jenkins-ci.org/download/war/ [root@jenkins-server ~]# wget https://get.jenkins.io/war-stable/2.332.3/jenkins.war 部署jenkins [root@jenkins-server ~]# cd /usr/local/tomcat/webapps/ [root@jenkins-server webapps]# rm -rf * [root@jenkins-server webapps]# cp /root/jenkins.war . [root@jenkins-server webapps]# ./bin/startup.sh Using CATALINA_BASE: /usr/local/tomcat Using CATALINA_HOME: /usr/local/tomcat Using CATALINA_TMPDIR: /usr/local/tomcat/temp Using JRE_HOME: /usr/local/java Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar Tomcat started. ``` ![](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612205950281.png) #### 3.gitlab部署 部署文件地址： https://docs.qq.com/doc/DQ0hScnRCbVN6QW1F ![](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612205933958.png) #### 4.harbor部署 部署链接地址：https://docs.qq.com/doc/DQ0l1bUtFdFNQSmdR ![image-20220612210023644](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612210023644.png) ## 二：项目使用 ​ 在实现部署之前在各个服务至上进行配置，以下是各个配置详情。 #### 1.jenkins配置 插件安装： ​ Maven Integration ​ Generic Webhook Trigger ​ Deploy to container ​ Git ​ Publish Over SSH ![image-20220612210702847](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612210702847.png) 配置JDK+MAVEN+GIT环境 ​ Dashboard--->Global Tool Configuration ![image-20220612210956961](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612210956961.png) 注意：需要在服务器上安装git ```shell [root@jenkins ~]# yum -y install git [root@jenkins ~]# git config --global user.email "xingdianvip@gmail.com" [root@jenkins ~]# git config --global user.name "xingdian" [root@jenkins ~]# git config --global http.sslVerify "false" ``` 例如： ![image-20220612211207274](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612211207274.png) jenkisn节点安装docker（因为需要构建镜像到harbor） ```shell [root@jenkins ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo [root@jenkins ~]# yum -y install docker-ce [root@jenkins ~]# systemctl start docker && docker enable docker ``` 配置http访问，创建daemon.json指定harbor地址 ```shell [root@jenkins ~]# cat /etc/docker/daemon.json { "insecure-registries":["192.168.18.230:80"] } [root@jenkins ~]# systemctl daemon-reload && systemctl restart docker ``` 注意：另外一种修改docker.service文件添加--insecure-registry，在这里不生效 额外配置，在企业中每个微服务项目都是独立的，但是此项目源码具有关联性，故需要执行一下操作 将所有的项目源码上传到jenkis服务器，在项目目录下执行以下命令，此过程需要耐心等待 如果想加快速度，百度搜索mvn的国内仓库地址（略） ```shell [root@jenkins tensquare_parent]# mvn install ``` 创建maven项目： ![image-20220612213835015](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612213835015.png) 注意：地址来自下面的gitlab部署 创建凭据 （再此添加kubernetes 集群master节点） ![image-20220612214908915](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612214908915.png) ![image-20220612214940777](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612214940777.png) ![image-20220612215011081](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612215011081.png) ![image-20220612215054577](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612215054577.png) 配置ssh remote hosts ![image-20220612215202282](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612215202282.png) 取消gitlab配置 ![image-20220612215240632](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612215240632.png) 配置webhook ![image-20220612213909544](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612213909544.png) ![image-20220612213933315](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612213933315.png) 脚本如下： ```shell # Jenkins机器：编译完成后，build生成一个新版本的镜像，push到远程docker仓库 # Variables # 注意下面路径里的maven-docker就是jenkins项目的名称，必须一致 JENKINS_WAR_HOME='/root/.jenkins/workspace/eureka/target' DOCKERFILE='/root/.jenkins/workspace/eureka/' # 自己创建下面目录，主要镜像构建 DOCKERFILE_HOME='/root/jenkins/docker-file/eureka_jar' HARBOR_IP='192.168.18.230' REPOSITORIES='xingdian/eureka' HARBOR_USER='admin' HARBOR_USER_PASSWD='Harbor12345' HARBOR_USER_EMAIL='xingdianvip@gmail.com' # Copy the newest war to docker-file directory. if [ -f /root/jenkins/docker-file/eureka_jar/eureka.jar ];then rm -rf eureka.jar \cp -f ${JENKINS_WAR_HOME}/tensquare_eureka_server-1.0-SNAPSHOT.jar ${DOCKERFILE_HOME}/eureka.jar else \cp -f ${JENKINS_WAR_HOME}/tensquare_eureka_server-1.0-SNAPSHOT.jar ${DOCKERFILE_HOME}/eureka.jar fi # Delete image early version. docker login ${HARBOR_IP}:80 -u ${HARBOR_USER} -p ${HARBOR_USER_PASSWD} IMAGE_ID=`sudo docker images | grep ${REPOSITORIES} | awk '{print $3}'` if [ -n "${IMAGE_ID}" ];then sudo docker rmi ${IMAGE_ID} fi # Build image. cd ${DOCKERFILE_HOME} if [ -f jdk-8u211-linux-x64.tar.gz ];then echo "jdk ok!!!!!" else # 此地址需要自己准备 wget ftp://192.168.18.234/share/jdk-8u211-linux-x64.tar.gz fi if [ -f Dockerfile ];then rm -rf Dockerfile cp -f ${DOCKERFILE}Dockerfile ${DOCKERFILE_HOME} echo "Dockerfile is ok!!" else cp -f ${DOCKERFILE}Dockerfile ${DOCKERFILE_HOME} fi TAG=`date +%Y%m%d-%H%M%S` sudo docker build -t ${HARBOR_IP}:80/${REPOSITORIES}:${TAG} . # Push to the harbor registry. sudo docker push ${HARBOR_IP}:80/${REPOSITORIES}:${TAG} ``` #### 2.gitlab配置 创建Groups和Project (Menu --- > Groups Menu --- > Project ) ![image-20220612212255399](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612212255399.png) ![image-20220612212337337](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612212337337.png) 取消main分支保护 ![image-20220612212601128](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612212601128.png) 添加本地公钥（实现项目推送） ![image-20220612212702850](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612212702850.png) 将项目空仓库下载，拷贝到本地空仓库目录下，然后推送给gitlab ```shell 上传项目的服务器需要安装以下内容： [root@xingdian ~]# yum -y install git [root@xingdian ~]# git config --global user.email "xingdianvip@gmail.com" [root@xingdian ~]# git config --global user.name "xingdian" [root@xingdian ~]# git config --global http.sslVerify "false" [root@xingdian ~]# git clone https://www.xingdian.com/diandian/diandian.git [root@xingdian ~]# cd diandian [root@xingdian diandian]# 将项目源码拷贝到此 [root@xingdian diandian]# git add . [root@xingdian diandian]# git commit -m "diandian" [root@xingdian diandian]# git push -u origin main ``` 开启允许本地网络 ![image-20220612213514193](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612213514193.png) 添加webhook gitlab： ![image-20220612213705137](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612213705137.png) #### 3.harbor配置 ​ 可以先手动构建，验证部署过程是否有问题，然后再进行自动化构建，自动化构建需要在gitlab上修改源代码，提交后，会自动触发 基础镜像配置（完成基础镜像配置后再构建） ```shell [root@harbor centos]# cat Dockerfile FROM daocloud.io/centos:7 MAINTAINER "xingdian" ENV container docker RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs RUN yum -y update; yum clean all; \ (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \ rm -f /lib/systemd/system/multi-user.target.wants/*;\ rm -f /etc/systemd/system/*.wants/*;\ rm -f /lib/systemd/system/local-fs.target.wants/*; \ rm -f /lib/systemd/system/sockets.target.wants/*udev*; \ rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \ rm -f /lib/systemd/system/basic.target.wants/*;\ rm -f /lib/systemd/system/anaconda.target.wants/*; VOLUME [ "/sys/fs/cgroup" ] CMD ["/usr/sbin/init"] [root@harbor centos]# docker build -t xingdian:latest . [root@harbor centos]# docker tag xingdian 192.168.18.230/xingdian/centos:latest [root@harbor ~]# cat /etc/docker/daemon.json { "insecure-registries":["192.168.18.230:80"] } [root@harbor ~]# systemctl daemon-reload && systemctl restart docker [root@harbor centos]# docker login 192.168.18.230 [root@harbor centos]# docker push 192.168.18.230/xingdian/centos:latest ``` 可以手动构建或者自动化构建，查看构建最终的镜像 ![image-20220612214752493](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612214752493.png) #### 4.发布到kubernetes集群 在master节点创建持续发布脚本 ``` [root@master ~]# cat eureka.sh #!/bin/bash HARBOR_IP='192.168.18.230' HARBOR_USER='admin' HARBOR_USER_PASSWD='Harbor12345' /usr/bin/yum -y install git if [ -d eureka-yaml ];then rm -rf eureka-yaml /usr/bin/git clone https://www.xingdian.com/xingdian/eureka-yaml.git else /usr/bin/git clone https://www.xingdian.com/xingdian/eureka-yaml.git fi cd eureka-yaml docker login ${HARBOR_IP}:80 -u ${HARBOR_USER} -p ${HARBOR_USER_PASSWD} tags=`curl -X GET "http://192.168.18.230/api/v2.0/projects/xingdian/repositories/eureka/artifacts?page=1&page_size=10&with_tag=true&with_label=false&with_scan_overview=false&with_signature=false&with_immutable_status=false" -H "accept: application/json" | jq | grep name ` tagss=`echo $tags | awk -F "\"" '{print $4}'` sed -i "s#eureka_image#192.168.18.230:80/xingdian/eureka:${tagss}#" eureka.yaml kubectl get pod | grep eureka if [ $? -eq 0 ];then kubectl delete -f eureka.yaml kubectl create -f eureka.yaml else kubectl create -f eureka.yaml fi ``` ![image-20220612233953314](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220612233953314.png) 所有节点docker修改http方式 ```shell [root@harbor ~]# cat /etc/docker/daemon.json { "insecure-registries":["192.168.18.230:80"] } [root@harbor ~]# systemctl daemon-reload && systemctl restart docker ``` jenkins构建发布 ![image-20220613002321701](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220613002321701.png) gitlab修改代码提交，自动触发jenkins构建 ![image-20220613002244405](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220613002244405.png) 浏览器访问构建的项目 ![image-20220613002408320](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220613002408320.png) 同理其余jar包部署，最终结果： ![image-20220622130840731](CICD+Harbor+K8S%E5%AE%9E%E7%8E%B0%E5%BE%AE%E6%9C%8D%E5%8A%A1%E9%A1%B9%E7%9B%AE%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E5%92%8C%E5%8F%91%E5%B8%83.assets/image-20220622130840731.png)