diff --git a/kubernetes-HA/基于二进制构建Kubernetes高可用集群.md b/kubernetes-HA/基于二进制构建Kubernetes高可用集群.md new file mode 100644 index 0000000..925d96b --- /dev/null +++ b/kubernetes-HA/基于二进制构建Kubernetes高可用集群.md @@ -0,0 +1,1005 @@ +

基于二进制构建Kubernetes高可用集群

+ + + +作者：行癫（盗版必究） + +------ + +## 一：环境介绍 + +#### 1.主机规划 + +| IP地址 | 主机名 | 主机配置 | 主机角色 | 软件列表 | +| :---------: | :---------------------------: | :------: | :------: | :----------------------------------------------------------: | +| 10.9.12.60 | xingdiancloud-native-master-a | 2C4G | master | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubectl、haproxy、keepalive | +| 10.9.12.64 | xingdiancloud-native-master-b | 2C4G | master | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubectl、haproxy、keepalive | +| 10.9.12.66 | xingdiancloud-native-node-a | 2C4G | worker | kubelet、kube-proxy、docker | +| 10.9.12.65 | xingdiancloud-native-node-b | 2C4G | worker | kubelet、kube-proxy、docker | +| 10.9.12.67 | xingdiancloud-native-node-c | 2C4G | worker | kubelet、kube-proxy、docker | +| 10.9.12.100 | / | / | VIP | | + +#### 2.软件版本 + +| 软件名称 | 版本 | 备注 | +| :--------: | :-----: | :-------: | +| CentOS | 7.9 | | +| kubernetes | v1.28.0 | | +| etcd | v3.5.11 | | +| calico | v3.26.4 | | +| coredns | v1.10.1 | | +| docker | 24.0.7 | | +| haproxy | 5.18 | YUM源默认 | +| keepalived | 3.5 | YUM源默认 | + +#### 3.网络分配 + +| 网络名称 | 网段 | 备注 | +| :---------: | :-----------: | :--: | +| Node网络 | 10.9.12.0/24 | | +| Service网络 | 10.96.0.0/16 | | +| Pod网络 | 10.244.0.0/16 | | + +## 二：集群准备 + +#### 1.修改主机名 + +```shell +[root@xingdiancloud-native-master-a ~]# nmcli g hostname xingdiancloud-native-master-a +``` + +备注： + +​ 所有节点按照规划一次修改 + +#### 2.地址解析 + +```shell +[root@xingdiancloud-native-master-a ~]# cat >> /etc/hosts << EOF +10.9.12.60 xingdiancloud-native-master-a +10.9.12.64 xingdiancloud-native-master-b +10.9.12.66 xingdiancloud-native-node-a +10.9.12.65 xingdiancloud-native-node-b +10.9.12.67 xingdiancloud-native-node-c +EOF +``` + +备注： + +​ 所有节点按照规划一次修改 + +#### 3.防火墙和Selinux + +​ 全部关闭及永久关闭 + +​ 此处略 + +备注： + +​ 所有节点按照规划一次修改 + +#### 4.交换分区设置 + +```shell +[root@xingdiancloud-native-master-a ~]# swapoff -a +[root@xingdiancloud-native-master-a ~]# sed -ri 's/.*swap.*/#&/' /etc/fstab +[root@xingdiancloud-native-master-a ~]# echo "vm.swappiness=0" >> /etc/sysctl.conf +[root@xingdiancloud-native-master-a ~]# sysctl -p +``` + +备注： + +​ 所有节点按照规划一次修改 + +#### 5.时间同步 + +```shell +[root@xingdiancloud-native-master-a ~]# yum -y install ntpdate +[root@xingdiancloud-native-master-a ~]# ntpdate -b ntp.aliyun.com +制定时间同步计划任务 +[root@xingdiancloud-native-master-a ~]# crontab -e +0 */5 * * * /usr/sbin/ntpdate -b ntp.aliyun.com +``` + +备注： + +​ 所有节点按照规划一次修改 + +#### 6.ipvs管理工具安装及模块加载 + +```shell +[root@xingdiancloud-native-master-a ~]# yum -y install ipvsadm ipset sysstat conntrack libseccomp + +#配置ipvasdm模块加载方式 +#添加需要加载的模块 +[root@xingdiancloud-native-master-a ~]# cat > /etc/sysconfig/modules/ipvs.modules << EOF +#!/bin/bash +modprobe -- ip_vs +modprobe -- ip_vs_rr +modprobe -- ip_vs_wrr +modprobe -- ip_vs_sh +modprobe -- nf_conntrack +EOF + +#授权，运行，检查是否加载 +[root@xingdiancloud-native-master-a ~]# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack +``` + +备注： + +​ 所有节点按照规划一次修改 + +#### 7.Linux内核优化 + +添加网桥过滤及内核转发配置文件 + +```shell +[root@xingdiancloud-native-master-a ~]# cat > /etc/sysctl.d/k8s.conf < /etc/modules-load.d/containerd.conf << EOF +overlay +br_netfilter +EOF +[root@xingdiancloud-native-master-a ~]# systemctl enable --now systemd-modules-load.service +``` + +查看是否加载 + +```shell +[root@xingdiancloud-native-master-a ~]# lsmod | grep br_netfilter +br_netfilter 28672 0 +``` + +#### 8.配置免密 + +在xingdiancloud-native-master-a上操作即可，复制公钥到其他节点 + +```shell +[root@xingdiancloud-native-master-a ~]# ssh-keygen +Generating public/private rsa key pair. +Enter file in which to save the key (/root/.ssh/id_rsa): +Created directory '/root/.ssh'. +Enter passphrase (empty for no passphrase): +Enter same passphrase again: +Your identification has been saved in /root/.ssh/id_rsa. +Your public key has been saved in /root/.ssh/id_rsa.pub. +The key fingerprint is: +SHA256:40/tHc966yq63YQ8YK84udBoZMqkCeZB5XTj8QaSOZo root@k8s-master1 +The key's randomart image is: ++---[RSA 2048]----+ +| +o= | +| +++ = | +| .o... o | +|.E . | +|.o . o So | +|+ * + o...+.. | +| + o + .o .=... | +| . .o.ooo+. +.| +| oo++.oo==+| ++----[SHA256]-----+ + +[root@xingdiancloud-native-master-a ~]# ssh-copy-id root@xingdiancloud-native-master-a +[root@xingdiancloud-native-master-a ~]# ssh-copy-id root@xingdiancloud-native-node-b +[root@xingdiancloud-native-master-a ~]# ssh-copy-id root@xingdiancloud-native-node-a +[root@xingdiancloud-native-master-a ~]# ssh-copy-id root@xingdiancloud-native-node-c +``` + +## 三：部署负载均衡高可用 + +#### 1.安装haproxy与keepalived + +在HA部署的节点上运行，本次HA部署在xingdiancloud-master-a,xingdiancloud-master-b上 + +``` +[root@xingdiancloud-native-master-a ~]# yum -y install haproxy keepalived +``` + +#### 2.HAProxy配置 + +在HA部署的节点上运行， HAProxy配置所有节点相同 + +```shell +[root@xingdiancloud-native-master-a ~]# cat >/etc/haproxy/haproxy.cfg<<"EOF" +global + maxconn 2000 + ulimit-n 16384 + log 127.0.0.1 local0 err + stats timeout 30s + +defaults + log global + mode http + option httplog + timeout connect 5000 + timeout client 50000 + timeout server 50000 + timeout http-request 15s + timeout http-keep-alive 15s + +frontend monitor-in + bind *:33305 + mode http + option httplog + monitor-uri /monitor + +frontend xingdiancloud-master + bind 0.0.0.0:6443 + bind 127.0.0.1:6443 + mode tcp + option tcplog + tcp-request inspect-delay 5s + default_backend xingdiancloud-master + +backend xingdiancloud-master + mode tcp + option tcplog + option tcp-check + balance roundrobin + default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 + server xingdiancloud-master-a 10.9.12.60:6442 check + server xingdiancloud-master-b 10.9.12.64:6442 check +EOF +``` + +#### 3.KeepAlived配置 + +主从配置不一致，需要注意 + +Master： + +```shell +[root@xingdiancloud-native-master-a ~]# cat >/etc/keepalived/keepalived.conf<<"EOF" +! Configuration File for keepalived +global_defs { + router_id LVS_DEVEL +script_user root + enable_script_security +} +vrrp_script chk_apiserver { + script "/etc/keepalived/check_apiserver.sh" + interval 5 + weight -5 + fall 2 +rise 1 +} +vrrp_instance VI_1 { + state MASTER + interface ens3 + mcast_src_ip 10.9.12.60 + virtual_router_id 51 + priority 100 + advert_int 2 + authentication { + auth_type PASS + auth_pass K8SHA_KA_AUTH + } + virtual_ipaddress { + 10.9.12.100 + } + track_script { + chk_apiserver + } +} +EOF +``` + +Backup： + +```shell +[root@xingdiancloud-native-master-b ~]# cat >/etc/keepalived/keepalived.conf<<"EOF" +! Configuration File for keepalived +global_defs { + router_id LVS_DEVEL +script_user root + enable_script_security +} +vrrp_script chk_apiserver { + script "/etc/keepalived/check_apiserver.sh" + interval 5 + weight -5 + fall 2 +rise 1 +} +vrrp_instance VI_1 { + state BACKUP + interface ens3 + mcast_src_ip 10.9.12.64 + virtual_router_id 51 + priority 99 + advert_int 2 + authentication { + auth_type PASS + auth_pass K8SHA_KA_AUTH + } + virtual_ipaddress { + 10.9.12.100 + } + track_script { + chk_apiserver + } +} +EOF +``` + +#### 4.健康检测脚本 + +Master和Backup节点均要有 + +```shell +[root@xingdiancloud-native-master-a ~]# cat > /etc/keepalived/check_apiserver.sh <<"EOF" +#!/bin/bash +err=0 +for k in $(seq 1 2) +do + check_code=$(pgrep haproxy) + if [[ $check_code == "" ]]; then + err=$(expr $err + 1) + sleep 1 + continue + else + err=0 + break + fi +done + +if [[ $err != "0" ]]; then + echo "systemctl stop keepalived" + /usr/bin/systemctl stop keepalived + exit 1 +else + exit 0 +fi +EOF + +[root@xingdiancloud-native-master-a ~]# chmod +x /etc/keepalived/check_apiserver.sh +``` + +#### 5.启动服务并验证 + +```shell +[root@xingdiancloud-native-master-a ~]# systemctl daemon-reload +[root@xingdiancloud-native-master-a ~]# systemctl enable --now haproxy +[root@xingdiancloud-native-master-a ~]# systemctl enable --now keepalived +``` + +注意： + +​ 依次启动Master节点和Backup节点 + +验证VIP： + +```shell +[root@xingdiancloud-native-master-a ~]# ip a s +1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever + inet6 ::1/128 scope host + valid_lft forever preferred_lft forever +2: ens33: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 + link/ether 00:0c:29:09:7a:32 brd ff:ff:ff:ff:ff:ff + inet 10.9.12.60/24 brd 192.168.198.255 scope global noprefixroute ens33 + valid_lft forever preferred_lft forever + inet 10.9.12.100/32 scope global ens33 + valid_lft forever preferred_lft forever + inet6 fe80::6d0d:5af:b421:6829/64 scope link noprefixroute + valid_lft forever preferred_lft forever + inet6 fe80::2dcd:beb6:b077:827d/64 scope link tentative noprefixroute dadfailed + valid_lft forever preferred_lft forever +``` + +测试网页是否正常显示： + +![image-20240616182636019](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240616182636019.png) + +## 四：ETCD集群部署 + +注意： + +​ 以下操作在xingdiancloud-native-master-a上操作 + +#### 1.创建工作目录 + +```shell +[root@xingdiancloud-native-master-a ~]# mkdir -p /data/k8s-work +``` + +#### 2.安装cfssl工具 + +​ https://github.com/cloudflare/cfssl/releases + +```shell +[root@xingdiancloud-native-master-a k8s-work]# ll +total 40232 +-rw-r--r-- 1 root root 16659824 Mar 9 2022 cfssl_1.6.1_linux_amd64 +-rw-r--r-- 1 root root 13502544 Mar 9 2022 cfssl-certinfo_1.6.1_linux_amd64 +-rw-r--r-- 1 root root 11029744 Mar 9 2022 cfssljson_1.6.1_linux_amd64 +# 授权可执行权限 +[root@xingdiancloud-native-master-a k8s-work]# chmod +x cfssl* +[root@xingdiancloud-native-master-a k8s-work]# ll +total 40232 +-rwxr-xr-x 1 root root 16659824 Mar 9 2022 cfssl_1.6.1_linux_amd64 +-rwxr-xr-x 1 root root 13502544 Mar 9 2022 cfssl-certinfo_1.6.1_linux_amd64 +-rwxr-xr-x 1 root root 11029744 Mar 9 2022 cfssljson_1.6.1_linux_amd64 +# 修改名称，放到/usr/local/bin目录下 +[root@xingdiancloud-native-master-a k8s-work]# mv cfssl_1.6.1_linux_amd64 /usr/local/bin/cfssl +[root@xingdiancloud-native-master-a k8s-work]# mv cfssl-certinfo_1.6.1_linux_amd64 /usr/local/bin/cfssl-certinfo +[root@xingdiancloud-native-master-a k8s-work]# mv cfssljson_1.6.1_linux_amd64 /usr/local/bin/cfssljson +# 安装完成，查看cfssl版本 +[root@k8s-master1 k8s-work]# cfssl version +Version: 1.6.1 +Runtime: go1.12.12 +``` + +#### 3.创建CA证书 + +注意： + +​ CA作为证书颁发机构 + +​ xingdiancloud-native-master-a 节点 + +##### 3.1 配置ca证书请求文件 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cat > ca-csr.json <<"EOF" +{ + "CN": "kubernetes", + "key": { + "algo": "rsa", + "size": 2048 + }, + "names": [ + { + "C": "CN", + "ST": "Beijing", + "L": "Beijing", + "O": "kubemsb", + "OU": "CN" + } + ], + "ca": { + "expiry": "87600h" + } +} +EOF +``` + +##### 3.2 创建ca证书 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca +2024/01/04 09:22:43 [INFO] generating a new CA key and certificate from CSR +2024/01/04 09:22:43 [INFO] generate received request +2024/01/04 09:22:43 [INFO] received CSR +2024/01/04 09:22:43 [INFO] generating key: rsa-2048 +2024/01/04 09:22:43 [INFO] encoded CSR +2024/01/04 09:22:43 [INFO] signed certificate with serial number 338731219198113317417686336532940600662573621163 + +#输出ca.csr ca-key.pem ca.pem +[root@xingdiancloud-native-master-a k8s-work]# ll +total 16 +-rw-r--r-- 1 root root 1045 Jan 4 09:22 ca.csr +-rw-r--r-- 1 root root 256 Jan 4 09:22 ca-csr.json +-rw------- 1 root root 1679 Jan 4 09:22 ca-key.pem +-rw-r--r-- 1 root root 1310 Jan 4 09:22 ca.pem +``` + +##### 3.3 配置ca证书策略 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cfssl print-defaults config > ca-config.json +cat > ca-config.json <<"EOF" +{ + "signing": { + "default": { + "expiry": "87600h" + }, + "profiles": { + "kubernetes": { + "usages": [ + "signing", + "key encipherment", + "server auth", + "client auth" + ], + "expiry": "87600h" + } + } + } +} +EOF +``` + +#### 4.创建ETCD证书 + +##### 4.1 配置etcd请求文件 + +注意： + +​ 57-59为预留IP + +``` +[root@xingdiancloud-native-master-a k8s-work]# cat > etcd-csr.json <<"EOF" +{ + "CN": "etcd", + "hosts": [ + "127.0.0.1", + "10.9.12.64", + "10.9.12.60", + "10.9.12.59", + "10.9.12.58", + "10.9.12.57" + ], + "key": { + "algo": "rsa", + "size": 2048 + }, + "names": [{ + "C": "CN", + "ST": "Beijing", + "L": "Beijing", + "O": "kubemsb", + "OU": "CN" + }] +} +EOF +``` + +##### 4.2 生成etcd证书 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd +2024/01/04 10:18:44 [INFO] generate received request +2024/01/04 10:18:44 [INFO] received CSR +2024/01/04 10:18:44 [INFO] generating key: rsa-2048 +2024/01/04 10:18:44 [INFO] encoded CSR +2024/01/04 10:18:44 [INFO] signed certificate with serial number 615580008866301102078218902811936499168508210128 +``` + +注意： + +​ 生成etcd.csr、etcd-key.pem、etcd.pem + +#### 5.部署ETCD集群 + +##### 5.1 下载etcd软件包 + +``` +https://github.com/etcd-io/etcd/releases/download/v3.5.11/etcd-v3.5.11-linux-amd64.tar.gz +``` + +##### 5.2 安装etcd软件 + +```shell +#解压etcd源码包 +[root@xingdiancloud-native-master-a k8s-work]# tar -xf etcd-v3.5.11-linux-amd64.tar.gz + +[root@xingdiancloud-native-master-a k8s-work]# cd etcd-v3.5.11-linux-amd64 +[root@xingdiancloud-native-master-a etcd-v3.5.11-linux-amd64]# ll +total 54896 +drwxr-xr-x 3 528287 89939 40 Dec 7 18:30 Documentation +-rwxr-xr-x 1 528287 89939 23535616 Dec 7 18:30 etcd +-rwxr-xr-x 1 528287 89939 17739776 Dec 7 18:30 etcdctl +-rwxr-xr-x 1 528287 89939 14864384 Dec 7 18:30 etcdutl +-rw-r--r-- 1 528287 89939 42066 Dec 7 18:30 README-etcdctl.md +-rw-r--r-- 1 528287 89939 7359 Dec 7 18:30 README-etcdutl.md +-rw-r--r-- 1 528287 89939 9394 Dec 7 18:30 README.md +-rw-r--r-- 1 528287 89939 7896 Dec 7 18:30 READMEv2-etcdctl.md + +#把etcd执行文件拷贝到/usr/local/bin目录下，后面配置文件都指定在这个文件执行命令 +[root@xingdiancloud-native-master-a etcd-v3.5.11-linux-amd64]# cp etcd* /usr/local/bin/ +#分发到其他节点 +[root@xingdiancloud-native-master-a etcd-v3.5.11-linux-amd64]# scp etcd* xingdiancloud-native-master-b:/usr/local/bin/ +etcd 100% 22MB 53.1MB/s 00:00 +etcdctl 100% 17MB 48.6MB/s 00:00 +etcdutl 100% 14MB 60.6MB/s 00:00 +``` + +##### 5.3 创建配置文件 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# mkdir /etc/etcd +``` + +xingdiancloud-native-master-a 配置： + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cat > /etc/etcd/etcd.conf < /etc/etcd/etcd.conf <<"EOF" +#[Member] +ETCD_NAME="etcd2" +ETCD_DATA_DIR="/var/lib/etcd/default.etcd" +ETCD_LISTEN_PEER_URLS="https://10.9.12.64:2380" +ETCD_LISTEN_CLIENT_URLS="https://10.9.12.64:2379,http://127.0.0.1:2379" + +#[Clustering] +ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.9.12.64:2380" +ETCD_ADVERTISE_CLIENT_URLS="https://10.9.12.64:2379" +ETCD_INITIAL_CLUSTER="etcd1=https://10.9.12.60:2380,etcd2=https://10.9.12.64:2380" +ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" +ETCD_INITIAL_CLUSTER_STATE="new" +EOF +``` + +##### 5.4 创建服务配置文件 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# mkdir -p /etc/etcd/ssl +[root@xingdiancloud-native-master-a k8s-work]# mkdir -p /var/lib/etcd/default.etcd +``` + +```shell +[root@xingdiancloud-native-master-a etcd]# cd /data/k8s-work +[root@xingdiancloud-native-master-a k8s-work]# ll +total 19896 +-rw-r--r-- 1 root root 356 Jan 4 10:03 ca-config.json +-rw-r--r-- 1 root root 1045 Jan 4 09:22 ca.csr +-rw-r--r-- 1 root root 256 Jan 4 09:22 ca-csr.json +-rw------- 1 root root 1679 Jan 4 09:22 ca-key.pem +-rw-r--r-- 1 root root 1310 Jan 4 09:22 ca.pem +-rw-r--r-- 1 root root 1078 Jan 4 10:18 etcd.csr +-rw-r--r-- 1 root root 331 Jan 4 10:16 etcd-csr.json +-rw------- 1 root root 1679 Jan 4 10:18 etcd-key.pem +-rw-r--r-- 1 root root 1452 Jan 4 10:18 etcd.pem +drwxr-xr-x 3 528287 89939 163 Dec 7 18:30 etcd-v3.5.11-linux-amd64 +-rw-r--r-- 1 root root 20334735 Dec 7 18:36 etcd-v3.5.11-linux-amd64.tar.gz +#拷贝生成的etcd,ca证书到对应ssl目录 +[root@xingdiancloud-native-master-a k8s-work]# cp ca*.pem /etc/etcd/ssl +[root@xingdiancloud-native-master-a k8s-work]# cp etcd*.pem /etc/etcd/ssl +#分发证书到其他节点 +[root@xingdiancloud-native-master-a k8s-work]# scp ca*.pem xingdiancloud-native-master-b:/etc/etcd/ssl +ca-key.pem 100% 1679 1.4MB/s 00:00 +ca.pem 100% 1310 1.0MB/s 00:00 +[root@xingdiancloud-native-master-a k8s-work]# scp etcd*.pem xingdiancloud-native-master-b:/etc/etcd/ssl +etcd-key.pem 100% 1679 1.1MB/s 00:00 +etcd.pem 100% 1452 1.2MB/s 00:00 +``` + +##### 5.5 生成etcd启动文件 + +注意： + +​ 所有etcd + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cat > /etc/systemd/system/etcd.service <<"EOF" +[Unit] +Description=Etcd Server +After=network.target +After=network-online.target +Wants=network-online.target + +[Service] +Type=notify +EnvironmentFile=-/etc/etcd/etcd.conf +WorkingDirectory=/var/lib/etcd/ +ExecStart=/usr/local/bin/etcd \ + --cert-file=/etc/etcd/ssl/etcd.pem \ + --key-file=/etc/etcd/ssl/etcd-key.pem \ + --trusted-ca-file=/etc/etcd/ssl/ca.pem \ + --peer-cert-file=/etc/etcd/ssl/etcd.pem \ + --peer-key-file=/etc/etcd/ssl/etcd-key.pem \ + --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \ + --peer-client-cert-auth \ + --client-cert-auth +Restart=on-failure +RestartSec=5 +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target +EOF +``` + +##### 5.6 启动etcd集群 + +注意： + +​ 依次启动 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# systemctl daemon-reload +[root@xingdiancloud-native-master-a k8s-work]# systemctl enable --now etcd.service +[root@xingdiancloud-native-master-a k8s-work]# systemctl status etcd +● etcd.service - Etcd Server + Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: disabled) + Active: active (running) since Thu 2024-01-04 11:21:03 CST; 1min 15s ago + Main PID: 4515 (etcd) + CGroup: /system.slice/etcd.service + └─4515 /usr/local/bin/etcd --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem --trusted-ca-file=/etc/etcd/ssl/ca.pem --peer-cert... + +Jan 04 11:21:03 xingdiancloud-native-master-a etcd[4515]: {"level":"info","ts":"2024-01-04T11:21:03.193129+0800","caller":"api/capability.go:75","msg":"enabled capabilit...on":"3.0"} +Jan 04 11:21:03 xingdiancloud-native-master-a etcd[4515]: {"level":"info","ts":"2024-01-04T11:21:03.19358+0800","caller":"etcdserver/server.go:2066","msg":"published local member ... +Jan 04 11:21:03 xingdiancloud-native-master-a etcd[4515]: {"level":"info","ts":"2024-01-04T11:21:03.1937+0800","caller":"embed/serve.go:103","msg":"ready to serve client requests"} +Jan 04 11:21:03 xingdiancloud-native-master-a etcd[4515]: {"level":"info","ts":"2024-01-04T11:21:03.194023+0800","caller":"embed/serve.go:103","msg":"ready to serve client requests"} +Jan 04 11:21:03 xingdiancloud-native-master-a etcd[4515]: {"level":"info","ts":"2024-01-04T11:21:03.194473+0800","caller":"embed/serve.go:187","msg":"serving client traf...0.1:2379"} +Jan 04 11:21:03 xingdiancloud-native-master-a systemd[1]: Started Etcd Server. +Jan 04 11:21:03 xingdiancloud-native-master-a etcd[4515]: {"level":"info","ts":"2024-01-04T11:21:03.195005+0800","caller":"etcdmain/main.go:44","msg":"notifying init daemon"} +Jan 04 11:21:03 xingdiancloud-native-master-a etcd[4515]: {"level":"info","ts":"2024-01-04T11:21:03.195052+0800","caller":"etcdmain/main.go:50","msg":"successfully notif...t daemon"} +Jan 04 11:21:03 xingdiancloud-native-master-a etcd[4515]: {"level":"info","ts":"2024-01-04T11:21:03.196251+0800","caller":"embed/serve.go:250","msg":"serving client traf...146:2379"} +Jan 04 11:21:04 xingdiancloud-native-master-a etcd[4515]: {"level":"info","ts":"2024-01-04T11:21:04.065509+0800","caller":"membership/cluster.go:576","msg":"updated clus...to":"3.5"} +Hint: Some lines were ellipsized, use -l to show in full. +``` + +##### 5.7 验证集群状态 + +![image-20240616204328332](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240616204328332.png) + +![image-20240616204354173](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240616204354173.png) + +``` +#IS LEADER 为true的为主节点 +[xingdiancloud-native-master-a etcd]# ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://10.9.12.60:2379,https://10.9.12.64:2379 endpoint status ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +| https://10.9.12.60:2379 | f79986bfdb812e09 | 3.5.11 | 20 kB | true | false | 2 | 9 | 9 | | +| https://10.9.12.64:2379 | 3ed6f5bbee8d7853 | 3.5.11 | 20 kB | false | false | 2 | 9 | 9 | | ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +``` + +## 五：Kubernetes集群部署 + +#### 1.Kubernetes软件包下载 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# wget https://dl.k8s.io/v1.28.0/kubernetes-server-linux-amd64.tar.gz +``` + +#### 2.Kubernetes软件包安装 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# tar -xvf kubernetes-server-linux-amd64.tar.gz +[root@xingdiancloud-native-master-a k8s-work]# cd kubernetes/server/bin/ +[root@xingdiancloud-native-master-a bin]# cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/ +``` + +#### 3.Kubernetes软件分发 + +```shell +[root@xingdiancloud-native-master-a bin]# scp kube-apiserver kube-controller-manager kube-scheduler kubectl xingdiancloud-native-master-b:/usr/local/bin/ +``` + +#### 4.在集群节点上创建目录 + +注意： + +​ Master节点创建 + +```shell +[root@xingdiancloud-native-master-a bin]# mkdir -p /etc/kubernetes/ +[root@xingdiancloud-native-master-a bin]# mkdir -p /etc/kubernetes/ssl +[root@xingdiancloud-native-master-a bin]# mkdir -p /var/log/kubernetes +``` + +#### 5.部署api-server + +##### 5.1 创建apiserver证书请求文件 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cat > kube-apiserver-csr.json << "EOF" +{ +"CN": "kubernetes", + "hosts": [ + "127.0.0.1", + "10.9.12.60", + "10.9.12.64", + "10.9.12.66", + "10.9.12.67", + "10.9.12.65", + "10.9.12.59", + "10.9.12.58", + "10.9.12.57", + "10.9.12.100", + "10.96.0.1", + "kubernetes", + "kubernetes.default", + "kubernetes.default.svc", + "kubernetes.default.svc.cluster", + "kubernetes.default.svc.cluster.local" + ], + "key": { + "algo": "rsa", + "size": 2048 + }, + "names": [ + { + "C": "CN", + "ST": "Beijing", + "L": "Beijing", + "O": "kubemsb", + "OU": "CN" + } + ] +} +EOF +``` + +##### 5.2 生成apiserver证书及token文件 + +注意： + +​ 生成kube-apiserver.csr、kube-apiserver-key.pem、kube-apiserver.pem + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver +``` + +生成token.csv + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cat > token.csv << EOF +$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap" +EOF +``` + +##### 5.3 创建apiserver服务配置文件 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cat > /etc/kubernetes/kube-apiserver.conf << "EOF" +KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ + --anonymous-auth=false \ + --bind-address=10.9.12.60 \ + --advertise-address=10.9.12.60 \ + --secure-port=6442 \ + --authorization-mode=Node,RBAC \ + --runtime-config=api/all=true \ + --enable-bootstrap-token-auth \ + --service-cluster-ip-range=10.96.0.0/16 \ + --token-auth-file=/etc/kubernetes/token.csv \ + --service-node-port-range=30000-32767 \ + --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \ + --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \ + --client-ca-file=/etc/kubernetes/ssl/ca.pem \ + --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \ + --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \ + --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ + --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ + --service-account-issuer=api \ + --etcd-cafile=/etc/etcd/ssl/ca.pem \ + --etcd-certfile=/etc/etcd/ssl/etcd.pem \ + --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ + --etcd-servers=https://10.9.12.60:2379,https://10.9.12.64:2379 \ + --allow-privileged=true \ + --apiserver-count=3 \ + --audit-log-maxage=30 \ + --audit-log-maxbackup=3 \ + --audit-log-maxsize=100 \ + --audit-log-path=/var/log/kube-apiserver-audit.log \ + --event-ttl=1h \ + --v=4" +EOF +``` + +##### 5.4 创建apiserver服务管理配置文件 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cat > /etc/systemd/system/kube-apiserver.service << "EOF" +[Unit] +Description=Kubernetes API Server +Documentation=https://github.com/kubernetes/kubernetes +After=etcd.service +Wants=etcd.service + +[Service] +EnvironmentFile=-/etc/kubernetes/kube-apiserver.conf +ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS +Restart=on-failure +RestartSec=5 +Type=notify +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target +EOF +``` + +##### 5.5 同步文件到集群master节点 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# cp ca*.pem /etc/kubernetes/ssl/ + +[root@xingdiancloud-native-master-a k8s-work]# cp kube-apiserver*.pem /etc/kubernetes/ssl/ + +[root@xingdiancloud-native-master-a k8s-work]# cp token.csv /etc/kubernetes/ + +[root@xingdiancloud-native-master-a k8s-work]# scp /etc/kubernetes/ssl/ca*.pem xingdiancloud-native-master-b:/etc/kubernetes/ssl + +[root@xingdiancloud-native-master-a k8s-work]# scp /etc/kubernetes/ssl/kube-apiserver*.pem xingdiancloud-native-master-b:/etc/kubernetes/ssl + +[root@xingdiancloud-native-master-a k8s-work]# scp /etc/kubernetes/token.csv xingdiancloud-native-master-b:/etc/kubernetes +``` + +需要修改为对应主机的ip地址 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# scp /etc/kubernetes/kube-apiserver.conf xingdiancloud-native-master-b:/etc/kubernetes/kube-apiserver.conf +``` + +拷贝启动文件 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# scp /etc/systemd/system/kube-apiserver.service xingdiancloud-native-master-b:/etc/systemd/system/kube-apiserver.service +``` + +##### 5.6 启动apiserver服务 + +注意： + +​ 依次启动 apiserver 服务 + +```shell +[root@xingdiancloud-native-master-a k8s-work]# systemctl daemon-reload +[root@xingdiancloud-native-master-a k8s-work]# systemctl enable --now kube-apiserver +[root@xingdiancloud-native-master-a k8s-work]# systemctl status kube-apiserver + + +# 测试 +[root@xingdiancloud-native-master-a k8s-work]# curl --insecure https://110.9.12.60:6442/ +[root@xingdiancloud-native-master-a k8s-work]# curl --insecure https://10.9.12.64:6442/ +``` + +![image-20240616205911766](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240616205911766.png) + +![image-20240616205923792](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240616205923792.png) + + + + +