K8S集群安裝部署

(資料圖片)

一、系統(tǒng)基本環(huán)境搭建1.關(guān)閉swap2.關(guān)閉SELINUX3.調(diào)整內(nèi)核參數(shù)4.調(diào)整系統(tǒng)時(shí)區(qū)5.關(guān)閉系統(tǒng)不需要的服務(wù) 二、安裝etcd1.創(chuàng)建證書(shū)文件2. 安裝etcd2.1 node1上操作2.2 node2上操作2.3 node3上操作 備注說(shuō)明 三、安裝dockernode1節(jié)點(diǎn)node2節(jié)點(diǎn)node3節(jié)點(diǎn) 四、部署api-server（Master Node）1.創(chuàng)建apiserver自簽證書(shū)2.下載安裝包3.配置api-server配置文件4.創(chuàng)建token文件5.systemd 管理 apiserver6.啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng)7.授權(quán) kubelet-bootstrap 用戶允許請(qǐng)求證書(shū) 五、部署kube-controller-manager（Master Node）1. 創(chuàng)建配置文件2. systemd 管理 controller-manager3. 啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng) 六、部署 kube-scheduler（Master Node）1. 創(chuàng)建配置文件2. systemd 管理 kube-scheduler3. 啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng) 七、查看集群狀態(tài)八、部署 kubelet(Worker Node)1. 創(chuàng)建kubelet.conf配置文件2.配置kubelet-config.yml參數(shù)文件3.生成bootstrap.kubeconfig 配置文件4. systemd 管理 kubelet5. 啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng)6.批準(zhǔn) kubelet 證書(shū)申請(qǐng)并加入集群 九、部署 kube-proxy1.創(chuàng)建kube-proxy.conf2.配置參數(shù)文件3.生成 kube-proxy.kubeconfig4.生成kubeconfig5. systemd 管理 kube-proxy6. 啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng) 十、部署 CNI 網(wǎng)絡(luò)(Master Node)1.授權(quán) apiserver 訪問(wèn) kubelet 十一、部署node3節(jié)點(diǎn)1. 拷貝已部署好的 Node 相關(guān)文件到新節(jié)點(diǎn)2. 刪除 kubelet 證書(shū)和 kubeconfig 文件3. 修改主機(jī)名4. 啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng)5. 在 Master 上批準(zhǔn)新 Node kubelet 證書(shū)申請(qǐng) 十二、驗(yàn)證測(cè)試總結(jié)說(shuō)明1.kubectl get node 不是Ready狀態(tài)

一、系統(tǒng)基本環(huán)境搭建

角色I(xiàn)P組件

node1192.168.24.136kube-apiserver, kube-controller-manager, kube-scheduler, etcd

node2192.168.24.137kubelet，kube-proxy，docker etcd

node3192.168.24.138kubelet，kube-proxy，docker etcd

1.關(guān)閉swap

[root@node1 ~]# swapoff -a[root@node1 ~]# vim /etc/fstab

2.關(guān)閉SELINUX

[root@node1 ~]# setenforce 0 && sed -i "s/^SELINUX=. */SELINUX=disabled/" /etc/selinux/config

3.調(diào)整內(nèi)核參數(shù)

[root@node1 ~]# cat > /etc/sysctl.d/k8s.conf <<EOFNET.BRIDGE.BRIDGE-NF-CALL-IPTABLES=1NET.BRIDGE.BRIDGE-NF-CALL-IP6TABLES=1EOF# 生效[root@node1="" ~]#="" sysctl="" --system

4.調(diào)整系統(tǒng)時(shí)區(qū)

#設(shè)置系統(tǒng)時(shí)區(qū)為中國(guó)/上海[root@node1 ~]# timedatectl set-timezone Asia/Shanghai#將當(dāng)前的UTC時(shí)間寫(xiě)入硬件時(shí)鐘[root@node1 ~]# timedatectl set-local-rtc 0#重啟依賴于系統(tǒng)時(shí)間的服務(wù)[root@node1 ~]# systemctl restart rsyslog[root@node1 ~]# systemctl restart crond

5.關(guān)閉系統(tǒng)不需要的服務(wù)

[root@node1 ~]# systemctl stop postfix && systemctl disable postfix

二、安裝etcd

1.創(chuàng)建證書(shū)文件

[root@localhost home]# mkdir /opt/TLS/etcd && cd /opt/TLS/etcd[root@localhost etcd]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64[root@localhost etcd]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64[root@localhost etcd]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd6[root@localhost etcd]# mkdir /usr/local/cfssl/[root@localhost etcd]# mv cfssl_linux-amd64 /usr/local/cfssl/[root@localhost etcd]# mv cfssljson_linux-amd64 /usr/local/cfssl/[root@localhost etcd]# mv cfssl-certinfo_linux-amd6 /usr/local/cfssl/[root@localhost etcd]#  vim > ca-config.json<< EOF {"signing": {"default": {"expiry": "87600h"},"profiles": {"www": {"expiry": "87600h","usages": ["signing","key encipherment","server auth","client auth"]}}}}[root@localhost etcd]#  vim ca-csr.json{"CN": "etcd CA",        "key": {"algo": "rsa",                "size": 2048        },        "names": [                {"C": "CN",                        "L": "Beijing",                        "ST": "Beijing"                }        ]}[root@localhost etcd]# vim server-csr.json{"CN": "etcd","hosts": ["192.168.24.136","192.168.24.137","192.168.24.138"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing"}]}[root@localhost etcd]# /usr/local/cfssl/cfssl_linux-amd64 gencert -initca ca-csr.json | /usr/local/cfssl/cfssljson_linux-amd64 -bare ca -2021/06/30 23:30:35 [INFO] generating a new CA key and certificate from CSR2021/06/30 23:30:35 [INFO] generate received request2021/06/30 23:30:35 [INFO] received CSR2021/06/30 23:30:35 [INFO] generating key: rsa-20482021/06/30 23:30:36 [INFO] encoded CSR2021/06/30 23:30:36 [INFO] signed certificate with serial number 725059738840226310954172302162231768550676521910[root@localhost etcd]# [root@localhost etcd]# /usr/local/cfssl/cfssl_linux-amd64 gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | /usr/local/cfssl/cfssljson_linux-amd64 -bare server2021/06/30 23:39:25 [INFO] generate received request2021/06/30 23:39:25 [INFO] received CSR2021/06/30 23:39:25 [INFO] generating key: rsa-20482021/06/30 23:39:25 [INFO] encoded CSR2021/06/30 23:39:25 [INFO] signed certificate with serial number 4113968810751077352879574991605677378957980238632021/06/30 23:39:25 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable forwebsites. For more information see the Baseline Requirements for the Issuance and Managementof Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);specifically, section 10.2.3 ("Information Requirements").[root@localhost etcd]# [root@localhost etcd]# lltotal 36-rw-r--r-- 1 root root  226 Jun 30 23:26 ca-config.json-rw-r--r-- 1 root root  956 Jun 30 23:30 ca.csr-rw-r--r-- 1 root root  143 Jun 30 23:28 ca-csr.json-rw------- 1 root root 1679 Jun 30 23:30 ca-key.pem-rw-r--r-- 1 root root 1265 Jun 30 23:30 ca.pem-rw-r--r-- 1 root root 1013 Jun 30 23:39 server.csr-rw-r--r-- 1 root root  215 Jun 30 23:35 server-csr.json-rw------- 1 root root 1679 Jun 30 23:39 server-key.pem-rw-r--r-- 1 root root 1338 Jun 30 23:39 server.pem[root@localhost etcd]#

2. 安裝etcd

2.1 node1上操作

[root@node1 opt]# wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz[root@node1 opt]# tar -zxvf etcd-v3.4.9-linux-amd64.tar.gz[root@node1 opt]# mkdir -p /opt/etcd/{bin,cfg,ssl}[root@node1 opt]# mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/[root@node1 etcd]# scp -r /opt/TLS/etcd/{ca.pem,server-key.pem,server.pem} /opt/etcd/ssl/[root@node1 opt]# vim /opt/etcd/cfg/etcd.conf#[Member] ETCD_NAME="etcd-1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd"ETCD_LISTEN_PEER_URLS="https://192.168.24.136:2380"ETCD_LISTEN_CLIENT_URLS="https://192.168.24.136:2379"#[Clustering]ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.24.136:2380"ETCD_ADVERTISE_CLIENT_URLS="https://192.168.24.136:2379" ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.24.136:2380,etcd-2=https://192.168.24.137:2380,etcd-3=https://192.168.24.138:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"#[Security]ETCD_CERT_FILE="/opt/etcd/ssl/server.pem"ETCD_KEY_FILE="/opt/etcd/ssl/server-key.pem"ETCD_TRUSTED_CA_FILE="/opt/etcd/ssl/ca.pem"ETCD_CLIENT_CERT_AUTH="true"ETCD_PEER_CERT_FILE="/opt/etcd/ssl/server.pem"ETCD_PEER_KEY_FILE="/opt/etcd/ssl/server-key.pem"ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/ssl/ca.pem"ETCD_PEER_CLIENT_CERT_AUTH="true"[root@node1 opt]# vim /usr/lib/systemd/system/etcd.service[Unit]Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service]Type=notifyEnvironmentFile=/opt/etcd/cfg/etcd.confExecStart=/opt/etcd/bin/etcdRestart=on-failureLimitNOFILE=65536[Install] WantedBy=multi-user.target[root@node1 etcd]# scp -r /opt/etcd root@node2:/opt/[root@node1 etcd]# scp -r /opt/etcd root@node3:/opt/[root@node1 etcd]# scp -r /usr/lib/systemd/system/etcd.service root@node1:/usr/lib/systemd/system/[root@node1 etcd]# scp -r /usr/lib/systemd/system/etcd.service root@node2:/usr/lib/systemd/system/[root@node1 etcd]# systemctl daemon-reload

2.2 node2上操作

[root@node2 ~]# vim /opt/etcd/cfg/etcd.conf#[Member] ETCD_NAME="etcd-2"ETCD_DATA_DIR="/var/lib/etcd/default.etcd"ETCD_LISTEN_PEER_URLS="https://192.168.24.137:2380"ETCD_LISTEN_CLIENT_URLS="https://192.168.24.137:2379"#[Clustering]ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.24.137:2380"ETCD_ADVERTISE_CLIENT_URLS="https://192.168.24.137:2379"ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.24.136:2380,etcd-2=https://192.168.24.137:2380,etcd-3=https://192.168.24.138:2380"ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"ETCD_INITIAL_CLUSTER_STATE="new"#[Security]ETCD_CERT_FILE="/opt/etcd/ssl/server.pem"ETCD_KEY_FILE="/opt/etcd/ssl/server-key.pem"ETCD_TRUSTED_CA_FILE="/opt/etcd/ssl/ca.pem"ETCD_CLIENT_CERT_AUTH="true"ETCD_PEER_CERT_FILE="/opt/etcd/ssl/server.pem"ETCD_PEER_KEY_FILE="/opt/etcd/ssl/server-key.pem"ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/ssl/ca.pem"ETCD_PEER_CLIENT_CERT_AUTH="true"[root@node2 etcd]# systemctl daemon-reload

2.3 node3上操作

[root@node3 ~]# vim /opt/etcd/cfg/etcd.conf#[Member] ETCD_NAME="etcd-3" ETCD_DATA_DIR="/var/lib/etcd/default.etcd"ETCD_LISTEN_PEER_URLS="https://192.168.24.138:2380"ETCD_LISTEN_CLIENT_URLS="https://192.168.24.138:2379"#[Clustering]ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.24.138:2380"ETCD_ADVERTISE_CLIENT_URLS="https://192.168.24.138:2379" ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.24.136:2380,etcd-2=https://192.168.24.137:2380,etcd-3=https://192.168.24.138:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"#[Security]ETCD_CERT_FILE="/opt/etcd/ssl/server.pem"ETCD_KEY_FILE="/opt/etcd/ssl/server-key.pem"ETCD_TRUSTED_CA_FILE="/opt/etcd/ssl/ca.pem"ETCD_CLIENT_CERT_AUTH="true"ETCD_PEER_CERT_FILE="/opt/etcd/ssl/server.pem"ETCD_PEER_KEY_FILE="/opt/etcd/ssl/server-key.pem"ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/ssl/ca.pem"ETCD_PEER_CLIENT_CERT_AUTH="true"[root@node3 etcd]# systemctl daemon-reload

2.4 分別啟動(dòng)etcd

[root@node1 etcd]# systemctl start etcd [root@node1 etcd]# systemctl enable etcd# 啟動(dòng)node1的etcd后需要立即啟動(dòng)node2、node3上的etcd,否則node1的etcd訪問(wèn)node2、node3不通，一段時(shí)候會(huì)報(bào)啟動(dòng)失敗

備注說(shuō)明

ETCD3.4版本ETCDCTL_API=3 etcdctl 和 etcd --enable-v2=false 成為了默認(rèn)配置，如要使用v2版本，執(zhí)行etcdctl時(shí)候需要設(shè)置ETCDCTL_API環(huán)境變量，例如：ETCDCTL_API=2 etcdctlETCD3.4版本會(huì)自動(dòng)讀取環(huán)境變量的參數(shù)，所以EnvironmentFile文件中有的參數(shù)，不需要再次在ExecStart啟動(dòng)參數(shù)中添加，二選一，如同時(shí)配置，會(huì)觸發(fā)以下類(lèi)似報(bào)錯(cuò)“etcd: conflicting environment variable - - “ETCD_NAME” is shadowed by corresponding command-line flag (either unset environment variable or disable flag)”flannel操作etcd使用的是v2的API，而kubernetes操作etcd使用的v3的API

/opt/etcd/cfg/etcd.conf 配置文件參數(shù)說(shuō)明：

參數(shù)含義

ETCD_NAME節(jié)點(diǎn)名稱，集群中唯一

ETCD_DATA_DIR數(shù)據(jù)目錄

ETCD_LISTEN_PEER_URLS集群通信監(jiān)聽(tīng)地址

ETCD_LISTEN_CLIENT_URLS客戶端訪問(wèn)監(jiān)聽(tīng)地址

ETCD_INITIAL_ADVERTISE_PEER_URLS集群通告地址

ETCD_ADVERTISE_CLIENT_URLS客戶端通告地址

ETCD_INITIAL_CLUSTER集群節(jié)點(diǎn)地址

ETCD_INITIAL_CLUSTER_TOKEN集群 Token

ETCD_INITIAL_CLUSTER_STATE加入集群的當(dāng)前狀態(tài)，new 是新集群，existing 表示加入

三、安裝docker

node1節(jié)點(diǎn)

[root@node1 opt]# wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz[root@node1 opt]# mv docker /usr/local/[root@node1 opt]# vim /usr/lib/systemd/system/docker.service[Unit]Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target[Service] Type=notify ExecStart=/usr/bin/dockerdExecReload=/bin/kill -s HUP $MAINPIDLimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TimeoutStartSec=0 Delegate=yes KillMode=process Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s[Install] WantedBy=multi-user.target[root@node1 opt]# systemctl daemon-reload[root@node1 opt]# systemctl start docker[root@node1 opt]# systemctl enable docker[root@node1 opt]# scp -r docker/* root@node2:/usr/bin/[root@node1 opt]# scp -r docker/* root@node3:/usr/bin/[root@node1 opt]# scp -r /usr/lib/systemd/system/docker.service root@node2:/usr/lib/systemd/system/ [root@node1 opt]# scp -r /usr/lib/systemd/system/docker.service root@node3:/usr/lib/systemd/system/

node2節(jié)點(diǎn)

[root@node2 opt]# systemctl daemon-reload && systemctl start docker && systemctl enable docker

node3節(jié)點(diǎn)

[root@node2 opt]# systemctl daemon-reload && systemctl start docker && systemctl enable docker

四、部署api-server（Master Node）

1.創(chuàng)建apiserver自簽證書(shū)

[root@node1 k8s]# vim /opt/TLS/k8s/ca-config.json{"signing": {"default": {"expiry": "87600h"},"profiles": {"kubernetes": {"expiry": "87600h","usages": ["signing","key encipherment","server auth","client auth"]}}}}[root@node1 k8s]# vim /opt/TLS/k8s/ca-csr.json{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Beijing","ST": "Beijing","O": "k8s","OU": "System"}]}[root@node1 k8s]# vim /opt/TLS/k8s/server-csr.json{"CN": "kubernetes","hosts": ["10.0.0.1","127.0.0.1","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local","192.168.24.136","192.168.24.137","192.168.24.138"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing","O": "k8s","OU": "System"}]}# 生成證書(shū)文件[root@node1 k8s]# /usr/local/cfssl/cfssl gencert -initca ca-csr.json | /usr/local/cfssl/cfssljson -bare ca -2021/07/03 17:50:47 [INFO] generating a new CA key and certificate from CSR2021/07/03 17:50:47 [INFO] generate received request2021/07/03 17:50:47 [INFO] received CSR2021/07/03 17:50:47 [INFO] generating key: rsa-20482021/07/03 17:50:47 [INFO] encoded CSR2021/07/03 17:50:47 [INFO] signed certificate with serial number 112636923636866601548066763582581017478124700260[root@node1 k8s]# #使用自簽 CA 簽發(fā) kube-apiserver HTTPS 證書(shū)[root@node1 k8s]# /usr/local/cfssl/cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | /usr/local/cfssl/cfssljson -bare server2021/07/03 17:53:04 [INFO] generate received request2021/07/03 17:53:04 [INFO] received CSR2021/07/03 17:53:04 [INFO] generating key: rsa-20482021/07/03 17:53:05 [INFO] encoded CSR2021/07/03 17:53:05 [INFO] signed certificate with serial number 600727103085109645844076084532530568963684939512021/07/03 17:53:05 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable forwebsites. For more information see the Baseline Requirements for the Issuance and Managementof Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);specifically, section 10.2.3 ("Information Requirements").[root@node1 k8s]# ls server*.pemserver-key.pem  server.pem[root@node1 k8s]#

2.下載安裝包

[root@node1 opt]# wget https://dl.k8s.io/v1.19.0/kubernetes-server-linux-amd64.tar.gz[root@node1 opt]# tar -zxvf kubernetes-server-linux-amd64.tar.gz[root@node1 opt]# mv kubernetes kubernetes_package[root@node1 opt]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}[root@node1 opt]# cp kubernetes_package/kubernetes/server/bin/{kube-apiserver,kube-scheduler,kube-controller-manager,kubectl} /opt/kubernetes/bin/[root@node1 opt]# ll /opt/kubernetes/bin/total 300412-rwxr-xr-x 1 root root 115245056 Jul  3 18:13 kube-apiserver-rwxr-xr-x 1 root root 107249664 Jul  3 18:13 kube-controller-manager-rwxr-xr-x 1 root root  43003904 Jul  3 18:15 kubectl-rwxr-xr-x 1 root root  42123264 Jul  3 18:13 kube-scheduler[root@node1 opt]# ln -s /opt/kubernetes/bin/kubectl /usr/bin/kubectl[root@node1 opt]# [root@node1 opt]# cp /opt/TLS/k8s/{ca-key.pem,ca.pem,server-key.pem,server.pem} /opt/kubernetes/ssl/[root@node1 opt]# ll /opt/kubernetes/ssl/total 16-rw------- 1 root root 1679 Jul  3 18:20 ca-key.pem-rw-r--r-- 1 root root 1359 Jul  3 18:20 ca.pem-rw------- 1 root root 1679 Jul  3 18:20 server-key.pem-rw-r--r-- 1 root root 1627 Jul  3 18:20 server.pem[root@node1 opt]#

3.配置api-server配置文件

[root@node1 opt]# vim /opt/kubernetes/cfg/kube-apiserver.confKUBE_APISERVER_OPTS="--logtostderr=false \--v=2 \--log-dir=/opt/kubernetes/logs \--etcd-servers=https://192.168.24.136:2379,https://192.168.24.137:2379,https://192.168.24.138:2379 \--bind-address=192.168.24.136 \--secure-port=6443 \--advertise-address=192.168.24.136 \--allow-privileged=true \--service-cluster-ip-range=10.0.0.0/24 \--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction  \--authorization-mode=RBAC,Node \--enable-bootstrap-token-auth=true \--token-auth-file=/opt/kubernetes/cfg/token.csv \--service-node-port-range=30000-32767 \--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \--tls-cert-file=/opt/kubernetes/ssl/server.pem \--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \--client-ca-file=/opt/kubernetes/ssl/ca.pem \--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \--etcd-cafile=/opt/etcd/ssl/ca.pem \--etcd-certfile=/opt/etcd/ssl/server.pem \--etcd-keyfile=/opt/etcd/ssl/server-key.pem \--audit-log-maxage=30 \--audit-log-maxbackup=3 \--audit-log-maxsize=100 \--audit-log-path=/opt/kubernetes/logs/k8s-audit.log

參數(shù)參數(shù)解釋

–logtostderr啟用日志

–v日志等級(jí)

–log-dir日志目錄

–etcd-serversetcd 集群地址

–bind-address監(jiān)聽(tīng)地址

–secure-porthttps 安全端口

–advertise-address集群通告地址

–allow-privileged啟用授權(quán)

–service-cluster-ip-rangeService 虛擬 IP 地址段

–enable-admission-plugins準(zhǔn)入控制模塊

–authorization-mode認(rèn)證授權(quán)，啟用 RBAC 授權(quán)和節(jié)點(diǎn)自管理

–enable-bootstrap-token-auth啟用 TLS bootstrap 機(jī)制

–token-auth-filebootstrap token 文件

–service-node-port-rangeService nodeport 類(lèi)型默認(rèn)分配端口范圍

–kubelet-client-xxxapiserver 訪問(wèn) kubelet 客戶端證書(shū)

–tls-xxx-fileapiserver https 證書(shū)

–etcd-xxxfile連接 Etcd 集群證書(shū)

–audit-log-xxx審計(jì)日志

啟用 TLS Bootstrapping TLS Bootstrapping機(jī)制 TLS Bootstraping：Master apiserver 啟用 TLS 認(rèn)證后，Node 節(jié)點(diǎn) kubelet 和 kube- proxy 要與 kube-apiserver 進(jìn)行通信，必須使用 CA 簽發(fā)的有效證書(shū)才可以，當(dāng) Node 節(jié)點(diǎn)很多時(shí)，這種客戶端證書(shū)頒發(fā)需要大量工作，同樣也會(huì)增加集群擴(kuò)展復(fù)雜度。為了 簡(jiǎn)化流程，Kubernetes 引入了 TLS bootstraping 機(jī)制來(lái)自動(dòng)頒發(fā)客戶端證書(shū)，kubelet 會(huì)以一個(gè)低權(quán)限用戶自動(dòng)向 apiserver 申請(qǐng)證書(shū)，kubelet 的證書(shū)由 apiserver 動(dòng)態(tài)簽署。 所以強(qiáng)烈建議在 Node 上使用這種方式，目前主要用于 kubelet，kube-proxy 還是由我 們統(tǒng)一頒發(fā)一個(gè)證書(shū)。

4.創(chuàng)建token文件

格式：token，用戶名，UID，用戶組

[root@node1 ~]# head -c 16 /dev/urandom | od -An -t x | tr -d " "ef80e12f86c4e342f449e64e3f94f3a9[root@node1 opt]# vim /opt/kubernetes/cfg/token.csvc47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node- bootstrapper"

5.systemd 管理 apiserver

[root@node1 opt]# vim /usr/lib/systemd/system/kube-apiserver.service[Unit]Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service]EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.confExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS Restart=on-failure[Install]WantedBy=multi-user.target[root@node1 opt]#

6.啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng)

[root@node1 opt]# systemctl daemon-reload[root@node1 opt]# systemctl start kube-apiserver[root@node1 opt]# systemctl enable kube-apiserver

7.授權(quán) kubelet-bootstrap 用戶允許請(qǐng)求證書(shū)

[root@node1 opt]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

五、部署kube-controller-manager（Master Node）

1. 創(chuàng)建配置文件

[root@node1 opt]# vim /opt/kubernetes/cfg/kube-controller-manager.confKUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \--v=2 \--log-dir=/opt/kubernetes/logs \--leader-elect=true \--master=127.0.0.1:8080 \--bind-address=127.0.0.1 \--allocate-node-cidrs=true \--cluster-cidr=10.244.0.0/16 \--service-cluster-ip-range=10.0.0.0/24 \--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \--root-ca-file=/opt/kubernetes/ssl/ca.pem \--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \--experimental-cluster-signing-duration=87600h0m0s"

2. systemd 管理 controller-manager

[root@node1 opt]# vim /usr/lib/systemd/system/kube-controller-manager.service[Unit]Description=Kubernetes Controller ManagerDocumentation=https://github.com/kubernetes/kubernetes[Service]EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTSRestart=on-failure[Install]WantedBy=multi-user.target

3. 啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng)

[root@node1 opt]# systemctl daemon-reload [root@node1 opt]# systemctl start kube-controller-manager [root@node1 opt]# systemctl enable kube-controller-manager

六、部署 kube-scheduler（Master Node）

1. 創(chuàng)建配置文件

[root@node1 opt]# vim /opt/kubernetes/cfg/kube-scheduler.conf

2. systemd 管理 kube-scheduler

[root@node1 opt]# vim /usr/lib/systemd/system/kube-scheduler.service[Unit]Description=Kubernetes SchedulerDocumentation=https://github.com/kubernetes/kubernetes[Service]EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.confExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTSRestart=on-failure[Install]WantedBy=multi-user.target

3. 啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng)

[root@node1 opt]# systemctl daemon-reload[root@node1 opt]# systemctl status kube-scheduler[root@node1 opt]# systemctl enable kube-scheduler

七、查看集群狀態(tài)

[root@node1 opt]# kubectl get csWarning: v1 ComponentStatus is deprecated in v1.19+NAME                 STATUS    MESSAGE             ERRORcontroller-manager   Healthy   ok                  scheduler            Healthy   ok                  etcd-1               Healthy   {"health":"true"}   etcd-0               Healthy   {"health":"true"}   etcd-2               Healthy   {"health":"true"}   [root@node1 opt]#

八、部署 kubelet(Worker Node)

1. 創(chuàng)建kubelet.conf配置文件

[root@node2 ~]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}[root@node2 ~]# vim /opt/kubernetes/cfg/kubelet.confKUBELET_OPTS="--logtostderr=false \--v=2 \--log-dir=/opt/kubernetes/logs \--hostname-override=node2 \--network-plugin=cni \--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \--config=/opt/kubernetes/cfg/kubelet-config.yml \--cert-dir=/opt/kubernetes/ssl \--pod-infra-container-image=lizhenliang/pause-amd64:3.0"

2.配置kubelet-config.yml參數(shù)文件

[root@node2 ~]# vim /opt/kubernetes/cfg/kubelet-config.ymlkind: KubeletConfigurationapiVersion: kubelet.config.k8s.io/v1beta1address: 0.0.0.0port: 10250readOnlyPort: 10255cgroupDriver: cgroupfsclusterDNS:- 10.0.0.2clusterDomain: cluster.localfailSwapOn: falseauthentication:anonymous:enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /opt/kubernetes/ssl/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0scacheUnauthorizedTTL: 30s evictionHard: imagefs.available: 15% memory.available: 100Minodefs.available: 10% nodefs.inodesFree: 5%maxOpenFiles: 1000000maxPods: 110

3.生成bootstrap.kubeconfig 配置文件

[root@node1 ~]# scp -r /opt/kubernetes_package/kubernetes/server/bin/{kubectl,kubelet,kube-proxy} root@node2:/opt/kubernetes/bin/[root@node1 ~]# scp -r /opt/kubernetes/ssl/ca.pem root@node2:/opt/kubernetes/ssl/  [root@node2 ]# cd /opt/kubernetes/cfg [root@node2 cfg]# export KUBE_APISERVER="https://192.168.24.136:6443"  145 [root@node2 cfg]# export PORT TOKEN="c47ffb939f5ca36231d9e3121a252940"[root@node2 cfg]# /opt/kubernetes/bin/kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/ssl/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=bootstrap.kubeconfig[root@node2 cfg]# /opt/kubernetes/bin/kubectl config set-credentials "kubelet-bootstrap" \--token=${TOKEN} \--kubeconfig=bootstrap.kubeconfig [root@node2 cfg]# [root@node2 cfg]# /opt/kubernetes/bin/kubectl config set-context default \--cluster=kubernetes \--user="kubelet-bootstrap" \--kubeconfig=bootstrap.kubeconfig[root@node2 cfg]# /opt/kubernetes/bin/kubectl config use-context default --kubeconfig=bootstrap.kubeconfig[root@node2 cfg]# cat bootstrap.kubeconfig apiVersion: v1clusters:- cluster:    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVRTdyUWVKS2wrNGdxOXNPMzQvclBKdDRpZ21Rd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEl4TURjd016QTVORFl3TUZvWERUSTJNRGN3TWpBNU5EWXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbGFXcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBcXdQcXFUM0RTZE1odVlGQUExVngKSFpkU1NBQ0pRVkN3Mmdvb2JHM0dTaGh3bmY3dzc2MDF0eE5NbUsraEpGNVRBWUNnS05PU2svQ3lUT0wwY0NuNgprcmQ3TEVwNkVzZzl6cXQzN2VaRzExN0MzRXZhR25lZmY0NHUyM2xJSUJQUEJMam84c2Myd2d4SFI4bGREbWlECng0bWhyY3VoVzBHdjluUllUbDdFR2Z0OHlFRFh4SXdFNDFhejgxR0dHN29jd0tZK3BoaUx4TjVkbWZEWFpCWU8KQmdmdEtBOHBjNU9QVVlrUHhXOUcyQStNdkhnTlhBa0xha2lrc0h1MGkxZUxobWQyRUhDeEttbVJLWW1ZUkYxYwpRcy9yd0ZQS1ZHRUpRUyt1NkVDbHZvTGpqZ1VPcE41Ty9aYWxIdjA0aUdkTGd5a2x3ZjJ3RlRqY1BwalYxSFJXCnBRSURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVdzIzZWN2UzJ6VVNrY0pSOEtPcG83eVRZaytzd0h3WURWUjBqQkJnd0ZvQVV3MjNlY3ZTMgp6VVNrY0pSOEtPcG83eVRZaytzd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFIUzE0ZVB5dGJ2VHZMc0E0Z3d3CmZqR09MeElRNi9iQ2hESml1Q3RxUG5XNld0VC9NdHBqbVFKN0xlMlFFejVMMitLbERsSmdWcE9HWFdqSkY2NHoKZUFDTHhyYlM0TEhhVFZZdnBHM3V0by8rdVNQRkYwTlU3QUdGZnBOVEtMK0I0TTdidUVuYUFBSnpKQnNYSXhqeQowdDdPeEljUWNlN05IRTNTUkhlaHVPbkw1Ymk0K09KbkpwaVYzK1I4ZUsvU3g3ZTRrRXFLdG9mYmY3M0dhbE9vClQzZEh3TzVkeFNkT1B4SzFCbUtHb2pZUEdtUTNrSnM2Nmw0ZmxFZGpKV2prSDd3MWdXcTZHaGx1ZXRFVFlUTzIKQzRlUVpJSTJjZnRRTkh6Q2hJcHM1cXVoNGxackYzaStITHJpbjlxR3Y2N0YwREtwRlAwMXgzVkFTUkwxK2xHcgpobUE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K    server: https://192.168.24.136:6443  name: kubernetescontexts:- context:    cluster: kubernetes    user: kubelet-bootstrap  name: defaultcurrent-context: ""kind: Configpreferences: {}users:- name: kubelet-bootstrap  user:    token: c47ffb939f5ca36231d9e3121a252940[root@node2 cfg]#

4. systemd 管理 kubelet

[root@node2 cfg]# vim /usr/lib/systemd/system/kubelet.service[Unit]Description=Kubernetes KubeletAfter=docker.service[Service]EnvironmentFile=/opt/kubernetes/cfg/kubelet.confExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTSRestart=on-failureLimitNOFILE=65536[Install]WantedBy=multi-user.target

5. 啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng)

[root@node2 cfg]# systemctl daemon-reload[root@node2 cfg]# systemctl start kubelet[root@node2 cfg]# systemctl enable kubelet

6.批準(zhǔn) kubelet 證書(shū)申請(qǐng)并加入集群

[root@node1 cfg]# kubectl get csr[root@node1 cfg]# kubectl certificate approve node-csr-i-zpjwJ6gET9K_AN0-M8Wb7XpZxrqyBcNlUKGZLv35s[root@node1 cfg]# kubectl get nodeNAME    STATUS     ROLES    AGE   VERSIONnode2   NotReady80s   v1.19.0[root@node1 cfg]# 注：由于網(wǎng)絡(luò)插件還沒(méi)有部署，節(jié)點(diǎn)會(huì)沒(méi)有準(zhǔn)備就緒 NotReady

九、部署 kube-proxy

1.創(chuàng)建kube-proxy.conf

[root@node2 cfg]# vim /opt/kubernetes/cfg/kube-proxy.confKUBE_PROXY_OPTS="--logtostderr=false \--v=2 \--log-dir=/opt/kubernetes/logs \--config=/opt/kubernetes/cfg/kube-proxy-config.yml"

2.配置參數(shù)文件

[root@node2 cfg]# vim /opt/kubernetes/cfg/kube-proxy-config.ymlkind: KubeProxyConfigurationapiVersion: kubeproxy.config.k8s.io/v1alpha1bindAddress: 0.0.0.0metricsBindAddress: 0.0.0.0:10249clientConnection:        kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfighostnameOverride: node2clusterCIDR: 10.0.0.0/24

3.生成 kube-proxy.kubeconfig

# master節(jié)點(diǎn)上生成[root@node1 k8s]# cd /opt/TLS/k8s[root@node1 k8s]# vim kube-proxy-csr.json {"CN": "system:kube-proxy","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing","O": "k8s","OU": "System"}]}[root@node1 k8s]# /usr/local/cfssl/cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | /usr/local/cfssl/cfssljson -bare kube-proxy[root@node1 k8s]# scp -r {kube-proxy-key.pem,kube-proxy.pem} root@node2:/opt/kubernetes/ssl/

4.生成kubeconfig

[root@node2 cfg]# export KUBE_APISERVER="https://192.168.24.136:6443"[root@node2 cfg]# /opt/kubernetes/bin/kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/ssl/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kube-proxy.kubeconfig [root@node2 cfg]# /opt/kubernetes/bin/kubectl config set-credentials kube-proxy \--client-certificate=/opt/kubernetes/ssl/kube-proxy.pem \--client-key=/opt/kubernetes/ssl/kube-proxy-key.pem \--embed-certs=true \--kubeconfig=kube-proxy.kubeconfig [root@node2 cfg]# /opt/kubernetes/bin/kubectl config set-context default \--cluster=kubernetes \--user=kube-proxy \--kubeconfig=kube-proxy.kubeconfig [root@node2 cfg]# /opt/kubernetes/bin/kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

5. systemd 管理 kube-proxy

[root@node2 cfg]# vim /usr/lib/systemd/system/kube-proxy.service[Unit]Description=Kubernetes ProxyAfter=network.target[Service]EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.confExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTSRestart=on-failureLimitNOFILE=65536[Install]WantedBy=multi-user.target

6. 啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng)

[root@node2 cfg]# systemctl daemon-reload[root@node2 cfg]# systemctl start kube-proxy[root@node2 cfg]# systemctl enable kube-proxy

十、部署 CNI 網(wǎng)絡(luò)(Master Node)

master下載cni包，解壓后，部署好網(wǎng)絡(luò)插件，/opt/cni需要拷貝到其他節(jié)點(diǎn)上,其他節(jié)點(diǎn)也需要

[root@node1 opt]# wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz[root@node1 opt]# tar -zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin/[root@node1 opt]# cd /opt/kubernetes/cfg [root@node1 cfg]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml#默認(rèn)鏡像地址無(wú)法訪問(wèn)，修改為 docker hub 鏡像倉(cāng)庫(kù)。[root@node1 cfg]# sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml[root@node1 cfg]# scp -r /opt/cni root@node2:/opt/[root@node1 cfg]# kubectl apply -f kube-flannel.yml[root@node1 cfg]# kubectl get pods -n kube-systemNAME                    READY   STATUS    RESTARTS   AGEkube-flannel-ds-jcp6n   1/1     Running   0          6m22s[root@node1 cfg]# kubectl get nodeNAME    STATUS   ROLES    AGE   VERSIONnode2   Ready91m   v1.19.0[root@node1 cfg]#

1.授權(quán) apiserver 訪問(wèn) kubelet

[root@node1 cfg]# vim apiserver-to-kubelet-rbac.yamlapiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:         annotations:                 rbac.authorization.kubernetes.io/autoupdate: "true"         labels:                 kubernetes.io/bootstrapping: rbac-defaults         name: system:kube-apiserver-to-kubelet rules:         - apiGroups:                - ""           resources:                 - nodes/proxy                 - nodes/stats                 - nodes/log                 - nodes/spec                 - nodes/metrics                 - pods/log           verbs:             - "*" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata:         name: system:kube-apiserver         namespace: "" roleRef:         apiGroup: rbac.authorization.k8s.io        kind: ClusterRole         name: system:kube-apiserver-to-kubelet subjects:        - apiGroup: rbac.authorization.k8s.io         kind: User         name: kubernetes

十一、部署node3節(jié)點(diǎn)

1. 拷貝已部署好的 Node 相關(guān)文件到新節(jié)點(diǎn)

[root@node2 opt]# scp -r /opt/kubernetes root@node3:/opt/[root@node2 opt]# scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@node3:/usr/lib/systemd/system/

2. 刪除 kubelet 證書(shū)和 kubeconfig 文件

[root@node3 cfg]# rm -rf /opt/kubernetes/cfg/kubelet.kubeconfig /opt/kubernetes/ssl/kubelet*[root@node3 cfg]#

注：這幾個(gè)文件是證書(shū)申請(qǐng)審批后自動(dòng)生成的，每個(gè) Node 不同，必須刪除重新生成。

3. 修改主機(jī)名

[root@node3 cfg]# vim /opt/kubernetes/cfg/kubelet.conf --hostname-override=node3[root@node3 cfg]# vim /opt/kubernetes/cfg/kube-proxy-config.yml hostnameOverride: node3[root@node3 cfg]#

4. 啟動(dòng)并設(shè)置開(kāi)機(jī)啟動(dòng)

[root@node3 cfg]# systemctl daemon-reload[root@node3 cfg]# systemctl start kubelet[root@node3 cfg]# systemctl enable kubelet[root@node3 cfg]# systemctl start kube-proxy[root@node3 cfg]# systemctl enable kube-proxy

5. 在 Master 上批準(zhǔn)新 Node kubelet 證書(shū)申請(qǐng)

[root@node1 cfg]# kubectl certificate approve node-csr-V5KygJs8u7OWvFB2QI2FyiqdLJNwVpDyHnBhVmrC4oQcertificatesigningrequest.certificates.k8s.io/node-csr-V5KygJs8u7OWvFB2QI2FyiqdLJNwVpDyHnBhVmrC4oQ approved

十二、驗(yàn)證測(cè)試

[root@node1 logs]# kubectl create deployment nginx --image=nginxdeployment.apps/nginx created[root@node1 logs]# kubectl expose deployment nginx --port=80 --type=NodePortservice/nginx exposed

總結(jié)說(shuō)明

1.kubectl get node 不是Ready狀態(tài)

我這里的問(wèn)題是，/opt/cni/ 插件從節(jié)點(diǎn)機(jī)器上沒(méi)有部署