前言：如無(wú)特殊說(shuō)明，所有操作都用root賬號(hào)在所有節(jié)點(diǎn)執(zhí)行。
說(shuō)明：kubeasz是一款國(guó)產(chǎn)開源的k8s部署軟件，采用ansible role的部署方式，部署k8s二進(jìn)制集群。熟悉ansible role的用該軟件部署k8s方便快捷。

一、機(jī)器

deploy #部署機(jī)器
master01
master02
node01

二、環(huán)境

2.1 部署機(jī)器環(huán)境

[root@localhost opt]# cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
[root@localhost opt]# uname -a
Linux localhost.localdomain 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost opt]# python -V
Python 3.9.2

2.2 master和node節(jié)點(diǎn)機(jī)器環(huán)境

[root@localhost ~]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 
[root@localhost ~]# uname -a
Linux mater01 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# python -V
Python 3.9.2

2.3 ip和主機(jī)名

deploy 192.168.30.13
master01 192.168.30.10
master02 192.168.30.11
node01 192.168.10.12

三、配置yum源

說(shuō)明：CentOS 7 系統(tǒng)的yum源不再可用，需要重新配置。

3.1 備份舊yum源文件

cd /etc/yum.repos.d/
mkdir bak/
mv * ./bak/

3.2 配置Base源

cat >> CentOS-Base.repo << EOF 
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the 
# remarked out baseurl= line instead.
#
#[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7#released updates 
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
EOF

3.3 配置CR源

cat >> CentOS-CR.repo << EOF 
# CentOS-CR.repo
#
# The Continuous Release ( CR )  repository contains rpms that are due in the next
# release for a specific CentOS Version ( eg. next release in CentOS-7 ); these rpms
# are far less tested, with no integration checking or update path testing having
# taken place. They are still built from the upstream sources, but might not map 
# to an exact upstream distro release.
#
# These packages are made available soon after they are built, for people willing 
# to test their environments, provide feedback on content for the next release, and
# for people looking for early-access to next release content.
#
# The CR repo is shipped in a disabled state by default; its important that users 
# understand the implications of turning this on. 
#
# NOTE: We do not use a mirrorlist for the CR repos, to ensure content is available
#       to everyone as soon as possible, and not need to wait for the external
#       mirror network to seed first. However, many local mirrors will carry CR repos
#       and if desired you can use one of these local mirrors by editing the baseurl
#       line in the repo config below.
#[cr]
name=CentOS-$releasever - cr
baseurl=http://mirror.centos.org/centos/$releasever/cr/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0
EOF

3.4 配置Debuginfo源

cat >> CentOS-Debuginfo.repo << EOF
# CentOS-Debug.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
## All debug packages from all the various CentOS-7 releases
# are merged into a single repo, split by BaseArch
#
# Note: packages in the debuginfo repo are currently not signed
#[base-debuginfo]
name=CentOS-7 - Debuginfo
baseurl=http://debuginfo.centos.org/7/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Debug-7
enabled=0
#
EOF

3.5 配置fasttrack源

cat >> CentOS-fasttrack.repo << EOF
#CentOS-fasttrack.repo[fasttrack]
name=CentOS-7 - fasttrack
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=fasttrack&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/fasttrack/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
EOF

3.6 配置Media源

cat >> CentOS-Media.repo << EOF
# CentOS-Media.repo
#
#  This repo can be used with mounted DVD media, verify the mount point for
#  CentOS-7.  You can use this repo and yum to install items directly off the
#  DVD ISO that we release.
#
# To use this repo, put in your DVD and use it with the other repos too:
#  yum --enablerepo=c7-media [command]
#  
# or for ONLY the media repo, do this:
#
#  yum --disablerepo=\* --enablerepo=c7-media [command][c7-media]
name=CentOS-$releasever - Media
baseurl=file:///media/CentOS/file:///media/cdrom/file:///media/cdrecorder/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
EOF

3.7 配置Sources源

cat >> CentOS-Sources.repo << EOF 
# CentOS-Sources.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the 
# remarked out baseurl= line instead.
#
#[base-source]
name=CentOS-$releasever - Base Sources
baseurl=http://vault.centos.org/centos/$releasever/os/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7#released updates 
[updates-source]
name=CentOS-$releasever - Updates Sources
baseurl=http://vault.centos.org/centos/$releasever/updates/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7#additional packages that may be useful
[extras-source]
name=CentOS-$releasever - Extras Sources
baseurl=http://vault.centos.org/centos/$releasever/extras/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7#additional packages that extend functionality of existing packages
[centosplus-source]
name=CentOS-$releasever - Plus Sources
baseurl=http://vault.centos.org/centos/$releasever/centosplus/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
EOF

3.8 配置Vault源

cat >> CentOS-Vault.repo << EOF
# CentOS Vault contains rpms from older releases in the CentOS-7 
# tree.#c7.0.1406
[C7.0.1406-base]
name=CentOS-7.0.1406 - Base
baseurl=http://vault.centos.org/7.0.1406/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.0.1406-updates]
name=CentOS-7.0.1406 - Updates
baseurl=http://vault.centos.org/7.0.1406/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.0.1406-extras]
name=CentOS-7.0.1406 - Extras
baseurl=http://vault.centos.org/7.0.1406/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.0.1406-centosplus]
name=CentOS-7.0.1406 - CentOSPlus
baseurl=http://vault.centos.org/7.0.1406/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.0.1406-fasttrack]
name=CentOS-7.0.1406 - CentOSPlus
baseurl=http://vault.centos.org/7.0.1406/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0# C7.1.1503
[C7.1.1503-base]
name=CentOS-7.1.1503 - Base
baseurl=http://vault.centos.org/7.1.1503/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.1.1503-updates]
name=CentOS-7.1.1503 - Updates
baseurl=http://vault.centos.org/7.1.1503/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.1.1503-extras]
name=CentOS-7.1.1503 - Extras
baseurl=http://vault.centos.org/7.1.1503/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.1.1503-centosplus]
name=CentOS-7.1.1503 - CentOSPlus
baseurl=http://vault.centos.org/7.1.1503/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.1.1503-fasttrack]
name=CentOS-7.1.1503 - CentOSPlus
baseurl=http://vault.centos.org/7.1.1503/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0# C7.2.1511
[C7.2.1511-base]
name=CentOS-7.2.1511 - Base
baseurl=http://vault.centos.org/7.2.1511/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.2.1511-updates]
name=CentOS-7.2.1511 - Updates
baseurl=http://vault.centos.org/7.2.1511/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.2.1511-extras]
name=CentOS-7.2.1511 - Extras
baseurl=http://vault.centos.org/7.2.1511/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.2.1511-centosplus]
name=CentOS-7.2.1511 - CentOSPlus
baseurl=http://vault.centos.org/7.2.1511/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.2.1511-fasttrack]
name=CentOS-7.2.1511 - CentOSPlus
baseurl=http://vault.centos.org/7.2.1511/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0# C7.3.1611
[C7.3.1611-base]
name=CentOS-7.3.1611 - Base
baseurl=http://vault.centos.org/7.3.1611/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.3.1611-updates]
name=CentOS-7.3.1611 - Updates
baseurl=http://vault.centos.org/7.3.1611/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.3.1611-extras]
name=CentOS-7.3.1611 - Extras
baseurl=http://vault.centos.org/7.3.1611/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.3.1611-centosplus]
name=CentOS-7.3.1611 - CentOSPlus
baseurl=http://vault.centos.org/7.3.1611/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.3.1611-fasttrack]
name=CentOS-7.3.1611 - CentOSPlus
baseurl=http://vault.centos.org/7.3.1611/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0# C7.4.1708
[C7.4.1708-base]
name=CentOS-7.4.1708 - Base
baseurl=http://vault.centos.org/7.4.1708/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.4.1708-updates]
name=CentOS-7.4.1708 - Updates
baseurl=http://vault.centos.org/7.4.1708/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.4.1708-extras]
name=CentOS-7.4.1708 - Extras
baseurl=http://vault.centos.org/7.4.1708/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.4.1708-centosplus]
name=CentOS-7.4.1708 - CentOSPlus
baseurl=http://vault.centos.org/7.4.1708/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0[C7.4.1708-fasttrack]
name=CentOS-7.4.1708 - CentOSPlus
baseurl=http://vault.centos.org/7.4.1708/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0
EOF

3.9 配置x86_64-kernel源

cat >> CentOS-x86_64-kernel.repo << EOF
[centos-kernel]
name=CentOS LTS Kernels for $basearch
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=kernel&infra=$infra
#baseurl=http://mirror.centos.org/altarch/7/kernel/$basearch/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7[centos-kernel-experimental]
name=CentOS Experimental Kernels for $basearch
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=experimental&infra=$infra
#baseurl=http://mirror.centos.org/altarch/7/experimental/$basearch/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
EOF

3.10 配置docker-ce源

cat >> docker-ce.repo << EOF
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg[docker-ce-stable-debuginfo]
name=Docker CE Stable - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/debug-$basearch/stable
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg[docker-ce-stable-source]
name=Docker CE Stable - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/source/stable
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg[docker-ce-test]
name=Docker CE Test - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg[docker-ce-test-debuginfo]
name=Docker CE Test - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/debug-$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg[docker-ce-test-source]
name=Docker CE Test - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/source/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg[docker-ce-nightly]
name=Docker CE Nightly - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg[docker-ce-nightly-debuginfo]
name=Docker CE Nightly - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/debug-$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg[docker-ce-nightly-source]
name=Docker CE Nightly - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/source/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF

3.11 配置nginx源

cat >> nginx.repo << EOF
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOF

四、關(guān)閉selinux和防火墻

4.1 關(guān)閉防火墻

systemctl disable firewalld
systemctl stop firewalld

4.2 關(guān)閉selinux

setenforce 0
sed -i 's#SELINUX=.*#SELINUX=disabled#g' /etc/selinux/config

警告：關(guān)閉selinux后一定要重啟機(jī)器，否則會(huì)被部署代碼識(shí)別為未關(guān)閉selinux而報(bào)錯(cuò)。

五、修改主機(jī)名

說(shuō)明：分別在maser01 master02 node節(jié)點(diǎn)執(zhí)行。

5.1 master01執(zhí)行

hostnamectl set-hostname master01

5.2 master02執(zhí)行

hostnamectl set-hostname master02

5.3 node01執(zhí)行

hostnamectl set-hostname node01

六、部署步驟

說(shuō)明：僅在deploy節(jié)點(diǎn)執(zhí)行。

6.1 準(zhǔn)備ssh免密登陸

配置從部署節(jié)點(diǎn)能夠ssh免密登陸所有節(jié)點(diǎn)，并且設(shè)置python軟連接

$IP為所有節(jié)點(diǎn)地址包括自身，按照提示輸入yes 和root密碼

ssh-copy-id $IP 

6.2 為每個(gè)節(jié)點(diǎn)設(shè)置python軟鏈接

ssh $IP ln -s /usr/bin/python3 /usr/bin/python

6.3 在部署節(jié)點(diǎn)編排k8s安裝

6.3.1 下載項(xiàng)目源碼、二進(jìn)制及離線鏡像

下載工具腳本ezdown，舉例使用kubeasz版本3.5.0

export release=3.5.0
wget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown
chmod +x ./ezdown

下載kubeasz代碼、二進(jìn)制、默認(rèn)容器鏡像（更多關(guān)于ezdown的參數(shù)，運(yùn)行./ezdown 查看）

#國(guó)內(nèi)環(huán)境

./ezdown -D

#海外環(huán)境

./ezdown -D -m standard

【可選】下載額外容器鏡像（cilium,flannel,prometheus等）

./ezdown -X

【可選】下載離線系統(tǒng)包 (適用于無(wú)法使用yum/apt倉(cāng)庫(kù)情形)

./ezdown -P

上述腳本運(yùn)行成功后，所有文件（kubeasz代碼、二進(jìn)制、離線鏡像）均已整理好放入目錄/etc/kubeasz

6.3.2 創(chuàng)建集群配置實(shí)例

#容器化運(yùn)行kubeasz

./ezdown -S

#創(chuàng)建新集群 k8s-01

docker exec -it kubeasz ezctl new k8s-01

2021-01-19 10:48:23 DEBUG generate custom cluster files in /etc/kubeasz/clusters/k8s-01
2021-01-19 10:48:23 DEBUG set version of common plugins
2021-01-19 10:48:23 DEBUG cluster k8s-01: files successfully created.
2021-01-19 10:48:23 INFO next steps 1: to config '/etc/kubeasz/clusters/k8s-01/hosts'
2021-01-19 10:48:23 INFO next steps 2: to config '/etc/kubeasz/clusters/k8s-01/config.yml'

然后根據(jù)提示配置’/etc/kubeasz/clusters/k8s-01/hosts’ 和 ‘/etc/kubeasz/clusters/k8s-01/config.yml’：根據(jù)前面節(jié)點(diǎn)規(guī)劃修改hosts 文件和其他集群層面的主要配置選項(xiàng)；其他集群組件等配置項(xiàng)可以在config.yml 文件中修改。
修改后的配置文件如下

cat /etc/kubeasz/clusters/k8s-01/hosts 
# 'etcd' cluster should have odd member(s) (1,3,5,...)
[etcd]
192.168.30.16
192.168.30.17
192.168.30.18# master node(s)
[kube_master]
192.168.30.16
192.168.30.17# work node(s)
[kube_node]
192.168.30.18# [optional] harbor server, a private docker registry
# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one
[harbor]
#192.168.1.8 NEW_INSTALL=false
192.168.30.16 NEW_INSTALL=true# [optional] loadbalance for accessing k8s from outside
[ex_lb]
#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.30.1750 EX_APISERVER_PORT=8443
#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.30.1750 EX_APISERVER_PORT=8443
192.168.30.16 LB_ROLE=backup EX_APISERVER_VIP=192.168.30.200 EX_APISERVER_PORT=8443
192.168.30.17 LB_ROLE=master EX_APISERVER_VIP=192.168.30.200 EX_APISERVER_PORT=8443# [optional] ntp server for the cluster
[chrony]
#192.168.30.16
192.168.30.16[all:vars]
# --------- Main Variables ---------------
# Secure port for apiservers
SECURE_PORT="6443"# Cluster container-runtime supported: docker, containerd
# if k8s version >= 1.24, docker is not supported
CONTAINER_RUNTIME="containerd"# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="calico"# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.68.0.0/16"# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="172.20.0.0/16"# NodePort Range
NODE_PORT_RANGE="30000-32767"# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="cluster.local"# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/opt/kube/bin"# Deploy Directory (kubeasz workspace)
base_dir="/etc/kubeasz"# Directory for a specific cluster
cluster_dir="{{ base_dir }}/clusters/k8s-01"# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"

cat /etc/kubeasz/clusters/k8s-01/config.yml 
############################
# prepare
############################
# 可選離線安裝系統(tǒng)軟件包 (offline|online)
INSTALL_SOURCE: "online"# 可選進(jìn)行系統(tǒng)安全加固 github.com/dev-sec/ansible-collection-hardening
OS_HARDEN: false############################
# role:deploy
############################
# default: ca will expire in 100 years
# default: certs issued by the ca will expire in 50 years
CA_EXPIRY: "876000h"
CERT_EXPIRY: "438000h"# force to recreate CA and other certs, not suggested to set 'true'
CHANGE_CA: false# kubeconfig 配置參數(shù)
CLUSTER_NAME: "cluster1"
CONTEXT_NAME: "context-{{ CLUSTER_NAME }}"# k8s version
K8S_VER: "1.26.0"############################
# role:etcd
############################
# 設(shè)置不同的wal目錄，可以避免磁盤io競(jìng)爭(zhēng)，提高性能
ETCD_DATA_DIR: "/var/lib/etcd"
ETCD_WAL_DIR: ""############################
# role:runtime [containerd,docker]
############################
# ------------------------------------------- containerd
# [.]啟用容器倉(cāng)庫(kù)鏡像
ENABLE_MIRROR_REGISTRY: true# [containerd]基礎(chǔ)容器鏡像
SANDBOX_IMAGE: "easzlab.io.local:5000/easzlab/pause:3.9"# [containerd]容器持久化存儲(chǔ)目錄
CONTAINERD_STORAGE_DIR: "/var/lib/containerd"# ------------------------------------------- docker
# [docker]容器存儲(chǔ)目錄
DOCKER_STORAGE_DIR: "/var/lib/docker"# [docker]開啟Restful API
ENABLE_REMOTE_API: false# [docker]信任的HTTP倉(cāng)庫(kù)
INSECURE_REG: '["http://easzlab.io.local:5000"]'############################
# role:kube-master
############################
# k8s 集群 master 節(jié)點(diǎn)證書配置，可以添加多個(gè)ip和域名（比如增加公網(wǎng)ip和域名）
MASTER_CERT_HOSTS:- "192.168.30.16"- "192.168.30.17"- "192.168.30.18"- "k8s.easzlab.io"- "easzlab.io.local"- "harbor.easzlab.io.local"#- "www.test.com"# node 節(jié)點(diǎn)上 pod 網(wǎng)段掩碼長(zhǎng)度（決定每個(gè)節(jié)點(diǎn)最多能分配的pod ip地址）
# 如果flannel 使用 --kube-subnet-mgr 參數(shù)，那么它將讀取該設(shè)置為每個(gè)節(jié)點(diǎn)分配pod網(wǎng)段
# https://github.com/coreos/flannel/issues/847
NODE_CIDR_LEN: 24############################
# role:kube-node
############################
# Kubelet 根目錄
KUBELET_ROOT_DIR: "/var/lib/kubelet"# node節(jié)點(diǎn)最大pod 數(shù)
MAX_PODS: 110# 配置為kube組件（kubelet,kube-proxy,dockerd等）預(yù)留的資源量
# 數(shù)值設(shè)置詳見templates/kubelet-config.yaml.j2
KUBE_RESERVED_ENABLED: "no"# k8s 官方不建議草率開啟 system-reserved, 除非你基于長(zhǎng)期監(jiān)控，了解系統(tǒng)的資源占用狀況；
# 并且隨著系統(tǒng)運(yùn)行時(shí)間，需要適當(dāng)增加資源預(yù)留，數(shù)值設(shè)置詳見templates/kubelet-config.yaml.j2
# 系統(tǒng)預(yù)留設(shè)置基于 4c/8g 虛機(jī)，最小化安裝系統(tǒng)服務(wù)，如果使用高性能物理機(jī)可以適當(dāng)增加預(yù)留
# 另外，集群安裝時(shí)候apiserver等資源占用會(huì)短時(shí)較大，建議至少預(yù)留1g內(nèi)存
SYS_RESERVED_ENABLED: "no"############################
# role:network [flannel,calico,cilium,kube-ovn,kube-router]
############################
# ------------------------------------------- flannel
# [flannel]設(shè)置flannel 后端"host-gw","vxlan"等
FLANNEL_BACKEND: "vxlan"
DIRECT_ROUTING: false# [flannel] 
flannel_ver: "v0.19.2"# ------------------------------------------- calico
# [calico] IPIP隧道模式可選項(xiàng)有: [Always, CrossSubnet, Never],跨子網(wǎng)可以配置為Always與CrossSubnet(公有云建議使用always比較省事，其他的話需要修改各自公有云的網(wǎng)絡(luò)配置，具體可以參考各個(gè)公有云說(shuō)明)
# 其次CrossSubnet為隧道+BGP路由混合模式可以提升網(wǎng)絡(luò)性能，同子網(wǎng)配置為Never即可.
CALICO_IPV4POOL_IPIP: "Always"# [calico]設(shè)置 calico-node使用的host IP，bgp鄰居通過(guò)該地址建立，可手工指定也可以自動(dòng)發(fā)現(xiàn)
IP_AUTODETECTION_METHOD: "can-reach={{ groups['kube_master'][0] }}"# [calico]設(shè)置calico 網(wǎng)絡(luò) backend: brid, vxlan, none
CALICO_NETWORKING_BACKEND: "brid"# [calico]設(shè)置calico 是否使用route reflectors
# 如果集群規(guī)模超過(guò)50個(gè)節(jié)點(diǎn)，建議啟用該特性
CALICO_RR_ENABLED: false# CALICO_RR_NODES 配置route reflectors的節(jié)點(diǎn)，如果未設(shè)置默認(rèn)使用集群master節(jié)點(diǎn) 
# CALICO_RR_NODES: ["192.168.1.1", "192.168.1.2"]
CALICO_RR_NODES: []# [calico]更新支持calico 版本: ["3.19", "3.23"]
calico_ver: "v3.23.5"# [calico]calico 主版本
calico_ver_main: "{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}"# ------------------------------------------- cilium
# [cilium]鏡像版本
cilium_ver: "1.12.4"
cilium_connectivity_check: true
cilium_hubble_enabled: false
cilium_hubble_ui_enabled: false# ------------------------------------------- kube-ovn
# [kube-ovn]選擇 OVN DB and OVN Control Plane 節(jié)點(diǎn)，默認(rèn)為第一個(gè)master節(jié)點(diǎn)
OVN_DB_NODE: "{{ groups['kube_master'][0] }}"# [kube-ovn]離線鏡像tar包
kube_ovn_ver: "v1.5.3"# ------------------------------------------- kube-router
# [kube-router]公有云上存在限制，一般需要始終開啟 ipinip；自有環(huán)境可以設(shè)置為 "subnet"
OVERLAY_TYPE: "full"# [kube-router]NetworkPolicy 支持開關(guān)
FIREWALL_ENABLE: true# [kube-router]kube-router 鏡像版本
kube_router_ver: "v0.3.1"
busybox_ver: "1.28.4"############################
# role:cluster-addon
############################
# coredns 自動(dòng)安裝
dns_install: "yes"
corednsVer: "1.9.3"
ENABLE_LOCAL_DNS_CACHE: true
dnsNodeCacheVer: "1.22.13"
# 設(shè)置 local dns cache 地址
LOCAL_DNS_CACHE: "169.254.20.10"# metric server 自動(dòng)安裝
metricsserver_install: "yes"
metricsVer: "v0.5.2"# dashboard 自動(dòng)安裝
dashboard_install: "yes"
dashboardVer: "v2.7.0"
dashboardMetricsScraperVer: "v1.0.8"# prometheus 自動(dòng)安裝
prom_install: "no"
prom_namespace: "monitor"
prom_chart_ver: "39.11.0"# nfs-provisioner 自動(dòng)安裝
nfs_provisioner_install: "no"
nfs_provisioner_namespace: "kube-system"
nfs_provisioner_ver: "v4.0.2"
nfs_storage_class: "managed-nfs-storage"
nfs_server: "192.168.30.16"
nfs_path: "/data/nfs"# network-check 自動(dòng)安裝
network_check_enabled: false 
network_check_schedule: "*/5 * * * *"############################
# role:harbor
############################
# harbor version，完整版本號(hào)
HARBOR_VER: "v2.1.5"
HARBOR_DOMAIN: "harbor.easzlab.io.local"
HARBOR_PATH: /var/data
HARBOR_TLS_PORT: 8443
HARBOR_REGISTRY: "{{ HARBOR_DOMAIN }}:{{ HARBOR_TLS_PORT }}"# if set 'false', you need to put certs named harbor.pem and harbor-key.pem in directory 'down'
HARBOR_SELF_SIGNED_CERT: true# install extra component
HARBOR_WITH_NOTARY: false
HARBOR_WITH_TRIVY: false
HARBOR_WITH_CLAIR: false
HARBOR_WITH_CHARTMUSEUM: true

6.3.3 開始安裝 如果你對(duì)集群安裝流程不熟悉，請(qǐng)閱讀項(xiàng)目首頁(yè) 安裝步驟 講解后分步安裝，并對(duì) 每步都進(jìn)行驗(yàn)證

source ~/.bashrc

dk ezctl setup k8s-01 all

dk ezctl setup k8s-01 01
dk ezctl setup k8s-01 02
dk ezctl setup k8s-01 03
......

七、坑

7.1 操作系統(tǒng)的坑

不要使用CentOS 7.9 系統(tǒng)作為master和node節(jié)點(diǎn)，該系統(tǒng)ssh有bug。
bug為宿主機(jī)設(shè)置了免密登錄，宿主機(jī)可以免密登錄到其他機(jī)器，但是kubeasz容器不可以，而kubeasz部署k8s是容器內(nèi)的ansible需要有免密登錄到其他機(jī)器的權(quán)限。