九脈堂是做網(wǎng)站的優(yōu)化大師win10
Kubernetes概述
使用kubeadm快速部署一個k8s集群
Kubernetes高可用集群二進制部署（一）主機準備和負載均衡器安裝
Kubernetes高可用集群二進制部署（二）ETCD集群部署
Kubernetes高可用集群二進制部署（三）部署api-server
Kubernetes高可用集群二進制部署（四）部署kubectl和kube-controller-manager、kube-scheduler
Kubernetes高可用集群二進制部署（五）kubelet、kube-proxy、Calico、CoreDNS
Kubernetes高可用集群二進制部署（六）Kubernetes集群節(jié)點添加
1. 工作節(jié)點（worker node）部署
1.1 docker安裝及配置
wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install docker-ce
systemctl enable docker
systemctl start docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{"exec-opts": ["native.cgroupdriver=systemd"],"registry-mirrors": ["https://8i185852.mirror.aliyuncs.com"]
}
EOF
必須配置native.cgroupdriver，不配置這個步驟會導致kubelet啟動失敗
systemctl restart docker
1.2 部署kubelet
在k8s-master1（同時作為控制平面和數(shù)據(jù)平面）上操作
1.2.1 創(chuàng)建kubelet-bootstrap.kubeconfig
BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv)#192.168.10.100 VIP(虛擬IP)
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kubelet-bootstrap.kubeconfigkubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfigkubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfigkubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig
#創(chuàng)建集群角色綁定
kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=kubelet-bootstrapkubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig
kubectl describe clusterrolebinding cluster-system-anonymouskubectl describe clusterrolebinding kubelet-bootstrap
1.2.2 創(chuàng)建kubelet配置文件
[root@k8s-master1 k8s-work]# cat > kubelet.json << "EOF"
{"kind": "KubeletConfiguration","apiVersion": "kubelet.config.k8s.io/v1beta1","authentication": {"x509": {"clientCAFile": "/etc/kubernetes/ssl/ca.pem"},"webhook": {"enabled": true,"cacheTTL": "2m0s"},"anonymous": {"enabled": false}},"authorization": {"mode": "Webhook","webhook": {"cacheAuthorizedTTL": "5m0s","cacheUnauthorizedTTL": "30s"}},"address": "192.168.10.103", #當前主機地址"port": 10250,"readOnlyPort": 10255,"cgroupDriver": "systemd", "hairpinMode": "promiscuous-bridge","serializeImagePulls": false,"clusterDomain": "cluster.local.","clusterDNS": ["10.96.0.2"]
}
EOF
1.2.3 創(chuàng)建kubelet配置文件
cat > kubelet.service << "EOF"
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \--cert-dir=/etc/kubernetes/ssl \--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \--config=/etc/kubernetes/kubelet.json \--network-plugin=cni \--rotate-certificates \--pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2 \--alsologtostderr=true \--logtostderr=false \--log-dir=/var/log/kubernetes \--v=2
Restart=on-failure
RestartSec=5[Install]
WantedBy=multi-user.target
EOF
1.2.4 同步文件到集群節(jié)點
cp kubelet-bootstrap.kubeconfig /etc/kubernetes/
cp kubelet.json /etc/kubernetes/
cp kubelet.service /usr/lib/systemd/system/
for i in k8s-master2 k8s-master3 k8s-worker1;do scp kubelet-bootstrap.kubeconfig kubelet.json $i:/etc/kubernetes/;donefor i in k8s-master2 k8s-master3 k8s-worker1;do scp ca.pem $i:/etc/kubernetes/ssl/;donefor i in k8s-master2 k8s-master3 k8s-worker1;do scp kubelet.service $i:/usr/lib/systemd/system/;done
說明：
kubelet.json中address需要修改為當前主機IP地址。vim /etc/kubernetes/kubelet.json
1.2.5 創(chuàng)建目錄及啟動服務
在所有worker節(jié)點執(zhí)行
mkdir -p /var/lib/kubelet
mkdir -p /var/log/kubernetes
systemctl daemon-reload
systemctl enable --now kubeletsystemctl status kubelet
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 NotReady <none> 12s v1.21.10
k8s-master2 NotReady <none> 19s v1.21.10
k8s-master3 NotReady <none> 19s v1.21.10
k8s-worker1 NotReady <none> 18s v1.21.10
NotReady是因為網(wǎng)絡還沒有啟動
# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-b949p 7m55s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
csr-c9hs4 3m34s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
csr-r8vhp 5m50s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
csr-zb4sr 3m40s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
說明：
確認kubelet服務啟動成功后，接著到master上Approve一下bootstrap請求。
1.3 部署kube-proxy
1.3.1 創(chuàng)建kube-proxy證書請求文件
[root@k8s-master1 k8s-work]# cat > kube-proxy-csr.json << "EOF"
{"CN": "system:kube-proxy","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "kubemsb","OU": "CN"}]
}
EOF
1.3.2 生成證書
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
# ls kube-proxy*
kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem
1.3.3 創(chuàng)建kubeconfig文件
#設置管理集群
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube-proxy.kubeconfig
#設置證書
kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig
#設置上下文
kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig
#使用上下文
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
1.3.4 創(chuàng)建服務配置文件
cat > kube-proxy.yaml << "EOF"
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.10.103 #本機地址
clientConnection:kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 10.244.0.0/103 #pod網(wǎng)絡，不用改
healthzBindAddress: 192.168.10.103:10256 #本機地址
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.10.103:10249 #本機地址
mode: "ipvs" #ipvs比iptables更適用于大型集群
EOF
1.3.5 創(chuàng)建服務啟動管理文件
cat > kube-proxy.service << "EOF"
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \--config=/etc/kubernetes/kube-proxy.yaml \--alsologtostderr=true \--logtostderr=false \--log-dir=/var/log/kubernetes \--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF
1.3.6 同步文件到集群工作節(jié)點主機
cp kube-proxy*.pem /etc/kubernetes/ssl/
cp kube-proxy.kubeconfig kube-proxy.yaml /etc/kubernetes/
cp kube-proxy.service /usr/lib/systemd/system/
for i in k8s-master2 k8s-master3 k8s-worker1;do scp kube-proxy.kubeconfig kube-proxy.yaml $i:/etc/kubernetes/;done
for i in k8s-master2 k8s-master3 k8s-worker1;do scp kube-proxy.service $i:/usr/lib/systemd/system/;done
說明：
修改kube-proxy.yaml中IP地址為當前主機IP.vim /etc/kubernetes/kube-proxy.yaml
1.3.7 服務啟動
#創(chuàng)建WorkingDirectory
mkdir -p /var/lib/kube-proxysystemctl daemon-reload
systemctl enable --now kube-proxysystemctl status kube-proxy
2. 網(wǎng)絡組件部署 Calico
2.1 下載
wget https://docs.projectcalico.org/v3.19/manifests/calico.yaml
2.2 修改文件
vim calico.yaml
#修改如下兩行，取消注釋
3683 - name: CALICO_IPV4POOL_CIDR
3684 value: "10.244.0.0/16" #pod網(wǎng)絡
2.3 應用文件
kubectl apply -f calico.yaml
2.4 驗證應用結果
[root@k8s-master1 k8s-work]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-7cc8dd57d9-dcwjv 0/1 ContainerCreating 0 94s
calico-node-2pmqz 0/1 Init:0/3 0 94s
calico-node-9ms2r 0/1 Init:0/3 0 94s
calico-node-tj5rt 0/1 Init:0/3 0 94s
calico-node-wnjcv 0/1 PodInitializing 0 94s
[root@k8s-master1 k8s-work]# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-7cc8dd57d9-dcwjv 0/1 ContainerCreating 0 2m29s <none> k8s-master2 <none> <none>
calico-node-2pmqz 0/1 Init:0/3 0 2m29s 192.168.10.103 k8s-master1 <none> <none>
calico-node-9ms2r 0/1 Init:ImagePullBackOff 0 2m29s 192.168.10.105 k8s-master3 <none> <none>
calico-node-tj5rt 0/1 Init:0/3 0 2m29s 192.168.10.106 k8s-worker1 <none> <none>
calico-node-wnjcv 0/1 PodInitializing 0 2m29s 192.168.10.104 k8s-master2 <none> <none>
[root@k8s-master1 k8s-work]#
長時間STATUS沒有發(fā)生變化，可以通過以下命令查看詳細信息
kubectl describe pod calico-node-gndtg -n kube-system
如果有pod一直處于Init:ImagePullBackOff，等待很長時間還是沒有Runing 可以嘗試下載鏡像包通過ftp上傳到服務器上。
https://github.com/projectcalico/calico/releases?page=3找到需要的版本下載，上傳images目錄下對應的鏡像到服務器
docker load -i calico-pod2daemon-flexvol.tar
docker load -i calico-kube-controllers.tar
docker load -i calico-cni.tar
docker load -i calico-node.tardocker images
我這里有四臺工作節(jié)點，其中一臺執(zhí)行命令后正常下載運行Runing，另外三臺等了很久一直處于pull狀態(tài)，最后采用了以上方法解決，總結下來還是網(wǎng)絡問題。
如果一直處于Pending，檢查一下看看node是否被打污點了
kubectl describe node k8s-master2 |grep Taint
#刪除污點
kubectl taint nodes k8s-master2 key:NoSchedule-
污點值有三個，如下：
NoSchedule：一定不被調度
PreferNoSchedule：盡量不被調度【也有被調度的幾率】
NoExecute：不會調度，并且還會驅逐Node已有Pod
最后終于Ready
# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-7cc8dd57d9-pd44j 1/1 Running 0 70m
kube-system calico-node-bpqfr 1/1 Running 0 70m
kube-system calico-node-f8c6t 1/1 Running 0 70m
kube-system calico-node-gndtg 1/1 Running 0 70m
kube-system calico-node-pptqm 1/1 Running 0 70m
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready <none> 5h v1.21.10
k8s-master2 Ready <none> 5h v1.21.10
k8s-master3 Ready <none> 5h v1.21.10
k8s-worker1 Ready <none> 5h v1.21.10
3. 部署CoreDNS
用于實現(xiàn)k8s內服務間名稱解析，例如k8s之間部署了兩個服務 想通過名稱進行訪問，或者是k8s集群內的服務想訪問互聯(lián)網(wǎng)中的一些服務。
在k8s-master1上/data/k8s-work/下執(zhí)行：
cat > coredns.yaml << "EOF"
apiVersion: v1
kind: ServiceAccount
metadata:name: corednsnamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:kubernetes.io/bootstrapping: rbac-defaultsname: system:coredns
rules:- apiGroups:- ""resources:- endpoints- services- pods- namespacesverbs:- list- watch- apiGroups:- discovery.k8s.ioresources:- endpointslicesverbs:- list- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:annotations:rbac.authorization.kubernetes.io/autoupdate: "true"labels:kubernetes.io/bootstrapping: rbac-defaultsname: system:coredns
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:coredns
subjects:
- kind: ServiceAccountname: corednsnamespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:name: corednsnamespace: kube-system
data:Corefile: |.:53 {errorshealth {lameduck 5s}readykubernetes cluster.local in-addr.arpa ip6.arpa {fallthrough in-addr.arpa ip6.arpa}prometheus :9153forward . /etc/resolv.conf {max_concurrent 1000}cache 30loopreloadloadbalance}
---
apiVersion: apps/v1
kind: Deployment
metadata:name: corednsnamespace: kube-systemlabels:k8s-app: kube-dnskubernetes.io/name: "CoreDNS"
spec:# replicas: not specified here:# 1. Default is 1.# 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.strategy:type: RollingUpdaterollingUpdate:maxUnavailable: 1selector:matchLabels:k8s-app: kube-dnstemplate:metadata:labels:k8s-app: kube-dnsspec:priorityClassName: system-cluster-criticalserviceAccountName: corednstolerations:- key: "CriticalAddonsOnly"operator: "Exists"nodeSelector:kubernetes.io/os: linuxaffinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 100podAffinityTerm:labelSelector:matchExpressions:- key: k8s-appoperator: Invalues: ["kube-dns"]topologyKey: kubernetes.io/hostnamecontainers:- name: corednsimage: coredns/coredns:1.8.4imagePullPolicy: IfNotPresentresources:limits:memory: 170Mirequests:cpu: 100mmemory: 70Miargs: [ "-conf", "/etc/coredns/Corefile" ]volumeMounts:- name: config-volumemountPath: /etc/corednsreadOnly: trueports:- containerPort: 53name: dnsprotocol: UDP- containerPort: 53name: dns-tcpprotocol: TCP- containerPort: 9153name: metricsprotocol: TCPsecurityContext:allowPrivilegeEscalation: falsecapabilities:add:- NET_BIND_SERVICEdrop:- allreadOnlyRootFilesystem: truelivenessProbe:httpGet:path: /healthport: 8080scheme: HTTPinitialDelaySeconds: 60timeoutSeconds: 5successThreshold: 1failureThreshold: 5readinessProbe:httpGet:path: /readyport: 8181scheme: HTTPdnsPolicy: Defaultvolumes:- name: config-volumeconfigMap:name: corednsitems:- key: Corefilepath: Corefile
---
apiVersion: v1
kind: Service
metadata:name: kube-dnsnamespace: kube-systemannotations:prometheus.io/port: "9153"prometheus.io/scrape: "true"labels:k8s-app: kube-dnskubernetes.io/cluster-service: "true"kubernetes.io/name: "CoreDNS"
spec:selector:k8s-app: kube-dnsclusterIP: 10.96.0.2 #需要和上邊指定的clusterDNS IP一致ports:- name: dnsport: 53protocol: UDP- name: dns-tcpport: 53protocol: TCP- name: metricsport: 9153protocol: TCPEOF
kubectl apply -f coredns.yaml
# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-7cc8dd57d9-pd44j 1/1 Running 1 24h
kube-system calico-node-bpqfr 1/1 Running 1 24h
kube-system calico-node-f8c6t 1/1 Running 1 24h
kube-system calico-node-gndtg 1/1 Running 2 24h
kube-system calico-node-pptqm 1/1 Running 1 24h
kube-system coredns-675db8b7cc-xlwsp 1/1 Running 0 3m21s
#kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-7cc8dd57d9-pd44j 1/1 Running 1 24h 10.244.224.2 k8s-master2 <none> <none>
calico-node-bpqfr 1/1 Running 1 24h 192.168.10.103 k8s-master1 <none> <none>
calico-node-f8c6t 1/1 Running 1 24h 192.168.10.104 k8s-master2 <none> <none>
calico-node-gndtg 1/1 Running 2 24h 192.168.10.106 k8s-worker1 <none> <none>
calico-node-pptqm 1/1 Running 1 24h 192.168.10.105 k8s-master3 <none> <none>
coredns-675db8b7cc-xlwsp 1/1 Running 0 3m47s 10.244.159.129 k8s-master1 <none> <none>
和Calico一樣，如果一直處于ImagePullBackOff，查看后是因為拉去鏡像的問題，可嘗試將鏡像本地下載后，上傳到服務器load
鏡像下載網(wǎng)站，去docker hub搜索要下載的鏡像和版本，下載到本地后上傳至服務器
docker load -i coredns-coredns-1.8.4-.tar
docker images
#標簽不對應的話重新打標簽
docker tag 鏡像id coredns/coredns:v1.8.4
到這步我還是沒有正常啟動，提示如下信息
kubectl describe pod coredns-675db8b7cc-q6l95 -n kube-system
嘗試刪除pod后，重新創(chuàng)建CoreDNS Pod就正常了
# 查看日志
kubectl logs -f coredns-675db8b7cc-q6l95 -n kube-system# 刪除并重新創(chuàng)建CoreDNS Pod
kubectl delete pod coredns-675db8b7cc-q6l95 -n kube-system
kubectl apply -f coredns.yaml
4. 部署應用驗證
在k8s-master1上創(chuàng)建pod
[root@k8s-master1 k8s-work]# cat > nginx.yaml << "EOF"
---
apiVersion: v1
kind: ReplicationController
metadata:name: nginx-web
spec:replicas: 2selector:name: nginxtemplate:metadata:labels:name: nginxspec:containers:- name: nginximage: nginx:1.19.6ports:- containerPort: 80
---
apiVersion: v1
kind: Service #可以通過不同的方式對k8s集群服務進行訪問
metadata:name: nginx-service-nodeport
spec:ports:- port: 80targetPort: 80nodePort: 30001 #把k8s集群中運行應用的80端口映射到30001端口protocol: TCPtype: NodePortselector:name: nginx
EOF
kubectl apply -f nginx.yaml
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-web-qzvw4 1/1 Running 0 58s 10.244.194.65 k8s-worker1 <none> <none>
nginx-web-spw5t 1/1 Running 0 58s 10.244.224.1 k8s-master2 <none> <none>
# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/nginx-web-jnbhx 1/1 Running 1 23hNAME DESIRED CURRENT READY AGE
replicationcontroller/nginx-web 1 1 1 2dNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d6h
service/nginx-service-nodeport NodePort 10.96.72.89 <none> 80:30001/TCP 2d
查看是否有30001端口
ss -anput | grep ":30001"
可以看到每臺worker節(jié)點都有
訪問：http://192.168.10.103:30001，http://192.168.10.104:30001，http://192.168.10.105:30001，http://192.168.10.106:30001
#查看組件狀態(tài)
kubectl get cs
#查看pod
kubectl get pods