使用 kubeadm 部署 kubernetes v1.25.3

基于 docker 和 cri-dockerd 部署 kubernetes v1.25.3

环境准备

主机清单

主机名 IP地址 系统版本
master01 192.168.0.11 Ubuntu 22.04.2 LTS
master02 192.168.0.12 Ubuntu 22.04.2 LTS
master03 192.168.0.13 Ubuntu 22.04.2 LTS
node01 192.168.0.31 Ubuntu 22.04.2 LTS
node02 192.168.0.32 Ubuntu 22.04.2 LTS
node03 192.168.0.33 Ubuntu 22.04.2 LTS

系统基础环境

重置 root 密码

sudo passwd root

关闭防火墙

#所有节点执行:
root@master01:~# ufw disable
root@master01:~# ufw status

时间同步

#所有节点执行:
root@master01:~# apt install -y chrony
root@master01:~# systemctl restart chrony
root@master01:~# systemctl status chrony
root@master01:~# chronyc sources

主机名互相解析

#所有节点执行:
root@master01:~# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 master01

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

192.168.50.101 master01 master01.keti.org kubeapi.keti.org kubeapi
192.168.50.102 master02 master02.keti.org
192.168.50.103 master03 master03.keti.org

192.168.50.111 node01 node01.keti.org
192.168.50.112 node02 node02.keti.org
192.168.50.113 node03 node03.keti.org
root@master01:~#

禁用swap

#所有节点执行:
root@master01:~# sed -r -i '/\\/swap/s@^@#@' /etc/fstab
root@master01:~# swapoff -a
root@master01:~# systemctl --type swap

#若不禁用Swap设备,需要在后续编辑kubelet的配置文件/etc/default/kubelet,设置其忽略Swap启用的状态错误,内容:KUBELET_EXTRA_ARGS="--fail-swap-on=false"

安装docker

https://developer.aliyun.com/mirror/docker-ce?spm=a2c6h.13651102.0.0.57e31b11Kkm0VH

#所有节点执行:
#kubelet需要让docker容器引擎使用systemd作为CGroup的驱动,其默认值为cgroupfs,因而,我们还需要编辑docker的配置文件/etc/docker/daemon.json,添加如下内容,其中的registry-mirrors用于指明使用的镜像加速服务。

root@master01:~# mkdir /etc/docker
root@master01:~# vim /etc/docker/daemon.json
{
    "registry-mirrors": [
        "<https://docker.mirrors.ustc.edu.cn>",
        "<https://hub-mirror.c.163.com>",
        "<https://reg-mirror.qiniu.com>",
        "<https://registry.docker-cn.com>"
],
    "data-root": "/new/docker/data",
    "insecure-registries": ["harbor.ketitx.com"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
    "max-size": "200m"
},
    "storage-driver": "overlay2"
}

root@master01:~# systemctl daemon-reload
root@master01:~# systemctl start docker
root@master01:~# systemctl enable docker
root@master01:~# docker version
Client: Docker Engine - Community
 Version:           20.10.21
#注:kubeadm部署Kubernetes集群的过程中,默认使用Google的Registry服务k8s.gcr.io上的镜像,由于2022年仓库已经改为registry.k8s.io,国内可以直接访问,所以现在不需要镜像加速或者绿色上网就可以拉镜像了

安装cri-dockerd

#所有节点执行:
# 下载地址:<https://github.com/Mirantis/cri-dockerd>
root@master01:~# apt install ./cri-dockerd_0.3.9.3-0.ubuntu-jammy_amd64.deb -y

#完成安装后,相应的服务cri-dockerd.service便会自动启动
root@master01:~# systemctl status cri-docker.service

安装kubeadm、kubelet和kubectl

#所有节点执行:
#在各主机上生成kubelet和kubeadm等相关程序包的仓库,可参考阿里云官网
root@master01:~# apt update
root@master01:~# apt install -y apt-transport-https curl
root@master01:~# curl <https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg> | apt-key add -
root@master01:~# cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
> deb <https://mirrors.aliyun.com/kubernetes/apt/> kubernetes-xenial main
> EOF

#更新仓库并安装
root@master01:~# apt update
root@master01:~# apt install -y kubelet kubeadm kubectl

#注意:先不要启动,只是设置开机自启动
root@master01:~# systemctl enable kubelet

#确定kubeadm等程序文件的版本
root@master01:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:55:36Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}

整合kubelet和cri-dockerd

配置cri-dockerd

#所有节点执行:

root@master01:~# vim /usr/lib/systemd/system/cri-docker.service

#ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd://
ExecStart=/usr/bin/cri-dockerd --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9 --container-runtime-endpoint fd:// --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni/cache --cni-conf-dir=/etc/cni/net.d

#说明:
需要添加的各配置参数(各参数的值要与系统部署的CNI插件的实际路径相对应):
  --network-plugin:指定网络插件规范的类型,这里要使用CNI;
  --cni-bin-dir:指定CNI插件二进制程序文件的搜索目录;
  --cni-cache-dir:CNI插件使用的缓存目录;
  --cni-conf-dir:CNI插件加载配置文件的目录;
配置完成后,重载并重启cri-docker.service服务。

root@master01:~# systemctl daemon-reload && systemctl restart cri-docker.service
root@master01:~# systemctl status cri-docker

配置kubelet

#所有节点执行:

#配置kubelet,为其指定cri-dockerd在本地打开的Unix Sock文件的路径,该路径一般默认为“/run/cri-dockerd.sock“
root@master01:~# mkdir /etc/sysconfig
root@master01:~# vim /etc/sysconfig/kubelet
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=/run/cri-dockerd.sock"
root@master01:~# cat /etc/sysconfig/kubelet
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=/run/cri-dockerd.sock"

#说明:该配置也可不进行,而是直接在后面的各kubeadm命令上使用“--cri-socket unix:///run/cri-dockerd.sock”选项

初始化第一个主节点

#第一个主节点执行:

#列出k8s所需要的镜像
root@master01:~# kubeadm config images list
registry.k8s.io/kube-apiserver:v1.25.3
registry.k8s.io/kube-controller-manager:v1.25.3
registry.k8s.io/kube-scheduler:v1.25.3
registry.k8s.io/kube-proxy:v1.25.3
registry.k8s.io/pause:3.8
registry.k8s.io/etcd:3.5.4-0
registry.k8s.io/coredns/coredns:v1.9.3

#使用阿里云拉取所需镜像
root@master01:~# kubeadm config images pull --image-repository=registry.aliyuncs.com/google_containers --cri-socket unix:///run/cri-dockerd.sock

root@master01:~# docker images
REPOSITORY                                                        TAG       IMAGE ID       CREATED
registry.aliyuncs.com/google_containers/kube-apiserver            v1.25.3   0346dbd74bcb   3 weeks ago
registry.aliyuncs.com/google_containers/kube-scheduler            v1.25.3   6d23ec0e8b87   3 weeks ago
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.25.3   603999231275   3 weeks ago
registry.aliyuncs.com/google_containers/kube-proxy                v1.25.3   beaaf00edd38   3 weeks ago
registry.aliyuncs.com/google_containers/pause                     3.8       4873874c08ef   4 months ago
registry.aliyuncs.com/google_containers/etcd                      3.5.4-0   a8a176a5d5d6   5 months ago
registry.aliyuncs.com/google_containers/coredns                   v1.9.3    5185b96f0bec   5 months ago

root@master01:~# kubeadm init --control-plane-endpoint="kubeapi.keti.org" --kubernetes-version=v1.28.3 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --token-ttl=0 --cri-socket unix:///run/cri-dockerd.sock --upload-certs --image-repository registry.aliyuncs.com/google_containers

#如提示以下信息,代表初始化完成,请记录信息,以便后续使用:
.....

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  <https://kubernetes.io/docs/concepts/cluster-administration/addons/>

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join kubeapi.keti.org:6443 --token <KUBEADM_JOIN_TOKEN> \\
        --discovery-token-ca-cert-hash sha256:f34832cc625fe84d221b29ed47ed779b045f3b96a10cda557652318e8e2736f2 \\
        --control-plane --certificate-key <KUBEADM_CERTIFICATE_KEY>

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join kubeapi.keti.org:6443 --token <KUBEADM_JOIN_TOKEN> \\
        --discovery-token-ca-cert-hash sha256:f34832cc625fe84d221b29ed47ed779b045f3b96a10cda557652318e8e2736f2

root@master01:~#  mkdir -p $HOME/.kube
root@master01:~#  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
root@master01:~#  sudo chown $(id -u):$(id -g) $HOME/.kube/config

#如果初始化报如下错误:
Error getting node" err="node \\"k8s-master01\\" not found

#1、在cri-docker.service文件指定下pause版本:
root@master01:~# vim /usr/lib/systemd/system/cri-docker.service
ExecStart=/usr/bin/cri-dockerd --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.8 --container-runtime-endpoint fd:// --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni/cache --cni-conf-dir=/etc/cni/net.d
#2、重启服务:
root@master01:~# systemctl daemon-reload
root@master01:~# systemctl restart cri-docker.service

#3、重置集群:
root@master01:~# kubeadm reset --cri-socket unix:///run/cri-dockerd.sock && rm -rf /etc/kubernetes/ /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni /etc/cni/net.d

部署网络插件

我选择了使用 Flannel,也可以自行选择 Calico、Cilium等插件。

#所有节点执行:
#下载链接:
<https://github.com/flannel-io/flannel/releases>

root@master01:~# mkdir /opt/bin
root@master01:~# cp flanneld-amd64 /opt/bin/flanneld
root@master01:~# chmod +x /opt/bin/flanneld
root@master01:~# ll /opt/bin/flanneld
-rwxr-xr-x 1 root root 39358256 11月  4 22:46 /opt/bin/flanneld*
#第一个主节点执行:
#部署kube-flannel
root@master01:~# kubectl apply -f <https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml>
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

#确认Pod的状态为“Running”
root@master01:~# kubectl get pods -n kube-flannel
NAME                    READY   STATUS    RESTARTS   AGE
kube-flannel-ds-9bkgl   1/1     Running   0          50s

#此时,k8s-master01已经就绪
root@master01:~# kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
k8s-master01   Ready    control-plane   20m   v1.25.3

添加其他节点到集群中

#k8s-master02和k8s-master03执行:
#k8s-master02和k8s-master03加入集群
root@master01:~# kubeadm join kubeapi.wang.org:6443 --token <KUBEADM_JOIN_TOKEN> --discovery-token-ca-cert-hash sha256:a13ee9d0212edbd255fe0c5929186725b217a650f0b04ba75c6a1d6e67576aea --control-plane --certificate-key <KUBEADM_CERTIFICATE_KEY> --cri-socket unix:///run/cri-dockerd.sock

#注意,命令需要加上--cri-socket unix:///run/cri-dockerd.sock
#k8s-node01、k8s-node02、k8s-node03执行
#node节点加入集群

root@master01:~# kubeadm join kubeapi.wang.org:6443 --token <KUBEADM_JOIN_TOKEN> --discovery-token-ca-cert-hash sha256:a13ee9d0212edbd255fe0c5929186725b217a650f0b04ba75c6a1d6e67576aea --cri-socket unix:///run/cri-dockerd.sock

#注意,命令需要加上--cri-socket unix:///run/cri-dockerd.sock
#第一节点验证节点添加结果

root@master01:~# kubectl get nodes
NAME           STATUS   ROLES           AGE     VERSION
k8s-master01   Ready    control-plane   25m     v1.25.3
k8s-master02   Ready    control-plane   10m     v1.25.3
k8s-master03   Ready    control-plane   8m41s   v1.25.3
k8s-node01     Ready    <none>          6m54s   v1.25.3
k8s-node02     Ready    <none>          6m31s   v1.25.3
k8s-node03     Ready    <none>          6m5s    v1.25.3

部署应用测试

部署nginx pod测试

root@master01:~# kubectl create deployment nginx --image nginx:alpine --replicas=3
deployment.apps/nginx created

root@master01:~# kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-55f494c486-4js9n   1/1     Running   0          58s
nginx-55f494c486-fsgxq   1/1     Running   0          58s
nginx-55f494c486-z2gzv   1/1     Running   0          58s

root@master01:~# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP           NODE         NOMINATED NODE   READINESS GATES
nginx-55f494c486-4js9n   1/1     Running   0          4m31s   10.244.3.2   k8s-node01   <none>           <none>
nginx-55f494c486-fsgxq   1/1     Running   0          4m31s   10.244.4.2   k8s-node02   <none>           <none>
nginx-55f494c486-z2gzv   1/1     Running   0          4m31s   10.244.5.2   k8s-node03   <none>           <none>

root@master01:~# curl 10.244.4.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="<http://nginx.org/>">nginx.org</a>.<br/>
Commercial support is available at
<a href="<http://nginx.com/>">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

#删除pod,自动拉起
root@master01:~# kubectl delete pods nginx-55f494c486-fsgxq
pod "nginx-55f494c486-fsgxq" deleted

root@master01:~# kubectl get pods -owide
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
nginx-55f494c486-4js9n   1/1     Running   0          11m   10.244.3.2   k8s-node01   <none>           <none>
nginx-55f494c486-xqcph   1/1     Running   0          16s   10.244.4.3   k8s-node02   <none>           <none>
nginx-55f494c486-z2gzv   1/1     Running   0          11m   10.244.5.2   k8s-node03   <none>           <none>
root@master01:~# kubectl create service nodeport nginx --tcp=80:80
service/nginx created

root@master01:~# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        89m
nginx        NodePort    10.103.151.47   <none>        80:31901/TCP   18s
root@master01:~# curl 10.103.151.47
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="<http://nginx.org/>">nginx.org</a>.<br/>
Commercial support is available at
<a href="<http://nginx.com/>">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

#外部访问任意node几点ip加端口(注意,随机产生)都可以访问

Q&A

kubeadm init error: CRI v1 runtime API is not implemented

Hi, I'll write this post to help anyone encountering my same issue, given that I could not find the solution in past forum discussions.

I'm following the installation PDF for the class, exactly with all required versions: Ubuntu 20.04 (on a Google Cloud VM), kubeadm 1.24.1, etc.

But when executing the kubeadm init command I got the following error:

[init] Using Kubernetes version: v1.24.1
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time="2023-01-19T15:05:35Z" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \\"unix:///var/run/containerd/containerd.sock\\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

By searching the web it appears this is an issue with the old containerd provided by Ubuntu 20.

I solved it with the following steps (as root):

  1. Set up the Docker repository as described in https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository
  2. Remove the old containerd:apt remove containerd
  3. Update repository data and install the new containerd: apt update, apt install containerd.io
  4. Remove the installed default config file: rm /etc/containerd/config.toml
  5. Restart containerd: systemctl restart containerd

The kubeadm init command worked fine afterwards.