08 | 部署网络组件 Cilium + CoreDNS

Kuberneters 集群支持多种网络插件,例如:Flannel、Cilium、Calico、Weave。企业生产环境中用的比较多的是 Cilium。所以本节课,选择 Cilium 作为集群的网络插件。

另外,部署完网络插件之后,还需要部署 DNS 服务。Kubernetes 集群标准的 DNS 组件是 CoreDNS。

所以,本节课,来介绍下如何安装:Cilium、CoreDNS。

提示:Flannel 网络插件实现简单、代码量少。如果你想学习 Kubernetes 网络插件的实现,可以学习 Flannel 的。其他网络插件的实现原理跟 Flannel 核心思想是相通的。Flannel 网络插件的安装参考附录 A。

部署 Cilium

在安装 cilium 前,要前安装好 kube-proxy。
cilium 官方提供了详细的安装文档:Cilium Quick Installation。你可以根据官方的文档来安装,也可以继续跟随本教程来安装。

安装 Cilium 命令

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm -f cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

安装 Cilium 网络插件

helm安装

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium --namespace kube-system --set image.tag=1.16.2

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "cp /opt/cni/bin/cilium-cni /opt/k8s/bin/"
  done

执行完上述命令之后,会在集群中安装 cilium 网络插件。通过以下命令可以查看安装进度:

$ kubectl -n kube-system get pods -owide|grep cilium
cilium-ckbfl                       1/1     Running   0          94m
cilium-envoy-l4gth                 1/1     Running   0          94m
cilium-envoy-tzdjw                 1/1     Running   0          94m
cilium-hhsrd                       1/1     Running   0          94m
cilium-operator-5874db7569-gn72x   1/1     Running   0          94m
cilium-operator-5874db7569-nkv8k   1/1     Running   0          94m

如果 Pod 都处在 Running 状态,说明 cilium 被正确安装。可以看到,helm install cilium cilium/cilium会在 Kubernetes 集群中安装以下组件:

官网方法安装

cilium install --version 1.16.2

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "cp /opt/cni/bin/cilium-cni /opt/k8s/bin/"
  done

也可以通过以下命令来查看 cilium 是否被正确安装:(如果使用云服务器需要提前开放安全组的10250端口)

cilium status --wait
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet              cilium-envoy             Desired: 2, Ready: 2/2, Available: 2/2
Deployment             cilium-operator          Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium                   Running: 2
                       cilium-envoy             Running: 2
                       cilium-operator          Running: 1
                       clustermesh-apiserver
                       hubble-relay
Cluster Pods:          3/3 managed by Cilium
Helm chart version:    1.16.2
Image versions         cilium             quay.io/cilium/cilium:v1.16.2@sha256:4386a8580d8d86934908eea022b0523f812e6a542f30a86a47edd8bed90d51ea: 2
                       cilium-envoy       quay.io/cilium/cilium-envoy:v1.29.9-1726784081-a90146d13b4cd7d168d573396ccf2b3db5a3b047@sha256:9762041c3760de226a8b00cc12f27dacc28b7691ea926748f9b5c18862db503f: 2
                       cilium-operator    quay.io/cilium/operator-generic:v1.16.2@sha256:cccfd3b886d52cb132c06acca8ca559f0fce91a6bd99016219b1a81fdbc4813a: 1

集群网络验证

执行以下命令创建 2 个测试 Pod:

cd /opt/k8s/work
cat > dnsutils-for-cilium-test-ds.yaml <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dnsutils-for-cilium-test
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      app: dnsutils-for-cilium-test
  template:
    metadata:
      labels:
        app: dnsutils-for-cilium-test
    spec:
      containers:
      - name: my-dnsutils
        image: tutum/dnsutils:latest
        imagePullPolicy: IfNotPresent
        command:
          - tail
          - "-f"
          - "/dev/null"
        ports:
        - containerPort: 80
EOF
kubectl create -f dnsutils-for-cilium-test-ds.yaml

执行以下命令获取 Pod 创建状态:

$ kubectl get pods -o wide -l app=dnsutils-for-cilium-test
NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
dnsutils-for-cilium-test-ngk74   1/1     Running   0          25m   10.0.1.80    k8s-01   <none>           <none>
dnsutils-for-cilium-test-rqqhc   1/1     Running   0          25m   10.0.0.133   k8s-02   <none>           <none>

待 Pod 都处在 Running状态后,测试是否可以从一个 Pod 内 ping 通另外一个 Pod 的 IP 地址:

$ kubectl exec -it dnsutils-for-cilium-test-ngk74 -- ping 10.0.0.133
PING 10.0.0.133 (10.0.0.133) 56(84) bytes of data.
64 bytes from 10.0.0.133: icmp_seq=1 ttl=63 time=0.785 ms
64 bytes from 10.0.0.133: icmp_seq=2 ttl=63 time=0.758 ms
...

注意:如果使用云主机需要提前放通 route 中提示的安全组,以免 cilium 间通信有问题导致无法访问(比如ping其他节点的Pod IP不通)。

部署 coredns

要在 Kubernetes 集群内实现服务发现,必须具备 DNS 组件,以便将服务域名解析为 Pod IP,从而通过 IP 地址进行通信。目前,CoreDNS 是 Kubernetes 环境中最常用的 DNS 解决方案。

在本小节中,我将逐步介绍如何部署 CoreDNS,确保集群内服务的 DNS 解析顺畅无阻。

下载和配置 CoreDNS

cd /opt/k8s/work
git clone https://github.com/coredns/deployment.git
mv deployment coredns-deployment

部署 coredns

cd /opt/k8s/work/coredns-deployment/kubernetes
source /opt/k8s/bin/environment.sh
./deploy.sh -i ${CLUSTER_DNS_SVC_IP} -d ${CLUSTER_DNS_DOMAIN} | kubectl apply -f -

检查 coredns 功能

我们可以通过以下 3 种方式来检查 CoreDNS 功能是否正常:

  1. 检查 CoreDNS 组件是否正常运行;
  2. 在 Pod 中访问 Kubernetes 服务;
  3. 检查 DNS 配置,并测试 DNS 解析。

具体操作如下。

检查 CoreDNS 组件是否正常运行

$ kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-6b59c98dd4-z249x   1/1     Running   0          4m57s

coredns Pod 处在 Running状态,说明 coredns 部署成功。

在 Pod 中访问 Kubernetes 服务

新建一个 Deployment:

cd /opt/k8s/work
cat > my-nginx.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      run: my-nginx
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx:1.7.9
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
EOF
kubectl apply -f my-nginx.yaml

expose 该 Deployment, 生成 my-nginx 服务:

$ kubectl expose deploy my-nginx
service "my-nginx" exposed

$ kubectl get services my-nginx -o wide
NAME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR
my-nginx   ClusterIP   10.254.4.128   <none>        80/TCP    5s    run=my-nginx

创建另一个 Pod,查看 /etc/resolv.conf 是否包含 kubelet 配置的 --cluster-dns 和 --cluster-domain,是否能够将服务 my-nginx 解析到上面显示的 Cluster IP 10.254.4.128。

检查 DNS 配置,并测试 DNS 解析

创建一个测试用的 dnsutils Pod:

cd /opt/k8s/work
cat > dnsutils-ds.yml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: dnsutils-ds
  labels:
    app: dnsutils-ds
spec:
  type: NodePort
  selector:
    app: dnsutils-ds
  ports:
  - name: http
    port: 80
    targetPort: 80
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dnsutils-ds
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      app: dnsutils-ds
  template:
    metadata:
      labels:
        app: dnsutils-ds
    spec:
      containers:
      - name: my-dnsutils
        image: tutum/dnsutils:latest
        imagePullPolicy: IfNotPresent
        command:
          - tail
          - "-f"
          - "/dev/null"
        ports:
        - containerPort: 80
EOF
kubectl create -f dnsutils-ds.yml

查看测试 Pod 是否成功创建:

$ kubectl get pods -lapp=dnsutils-ds -o wide
NAME                READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
dnsutils-ds-jc55p   1/1     Running   0          10s   10.0.1.155   k8s-01   <none>           <none>
dnsutils-ds-rnq7d   1/1     Running   0          10s   10.0.0.54    k8s-02   <none>           <none>

查看 DNS 解析配置是否正确:

$ kubectl exec -it dnsutils-ds-jc55p -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.254.0.2
options ndots:5

使用 DNS 能否成功解析 Kubernetes 内服务域名:

$ kubectl exec -it dnsutils-ds-jc55p -- nslookup kubernetes
Server:                10.254.0.2
Address:        10.254.0.2#53

Name:        kubernetes.default.svc.cluster.local
Address: 10.254.0.1
;; Got recursion not available from 10.254.0.2

$ kubectl exec -it dnsutils-ds-9zmbk -- nslookup my-nginx
Server:                10.254.0.2
Address:        10.254.0.2#53

Name:        my-nginx.default.svc.cluster.local
Address: 10.254.4.128
;; Got recursion not available from 10.254.0.2

可以看到 DNS 能够成功解析 Kubernetes 内以下 2 类服务域名:

查看使用 DNS 能否成功解析公网域名:

$ kubectl -it exec dnsutils-ds-9zmbk -- nslookup www.baidu.com
;; Got recursion not available from 10.254.0.2
;; Got recursion not available from 10.254.0.2
;; Got recursion not available from 10.254.0.2
Server:         10.254.0.2
Address:        10.254.0.2#53

Non-authoritative answer:
www.baidu.com   canonical name = www.a.shifen.com.
www.a.shifen.com        canonical name = www.wshifen.com.
Name:   www.wshifen.com
Address: 103.235.46.102
Name:   www.wshifen.com
Address: 103.235.46.115

可以看到 DNS 能够成功解析 www.baidu.com,解析到的 IP 地址为:103.235.46.102。

更全面的 Kubernetes 网络测试

通过上面的部署,我们已经成功部署了网络插件和 DNS 组件。至此,Kubernetes 的网络功能已经健全,我们可以对 Kubernetes 网络进行更全面的测试。

cilium 官网提供了测试功能,可以用来全面的测试 Kubernetes 网络。可以执行以下命令来测试:

$ cilium connectivity test
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [k8s-cluster] Creating namespace for connectivity check...

---------------------------------------------------------------------------------------------------------------------
📋 Test Report
---------------------------------------------------------------------------------------------------------------------
✅ 69/69 tests successful (0 warnings)

69/69 tests successful (0 warnings说明所有的 Kubernetes 集群网络连通性测试通过。

这里要注意:

参考

  1. https://community.infoblox.com/t5/Community-Blog/CoreDNS-for-Kubernetes-Service-Discovery/ba-p/8187
  2. https://coredns.io/2017/03/01/coredns-for-kubernetes-service-discovery-take-2/
  3. https://www.cnblogs.com/boshen-hzb/p/7511432.html
  4. https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns

补充

对 containerd 配置了镜像加速后,在 kubelet 拉取的时候发现没生效,还是拉取不到镜像。

通过 nerdctl 命令手动拉取镜像是可以正常拉取的,但是集群识别不到手工拉取的镜像。

搜索发现 ctr crictl nedrctl 三条命令的输出结果是不同的。

ctr images ls
REF                              TYPE                                                      DIGEST                                                                  SIZE     PLATFORMS                                                                                               LABELS 
docker.io/coredns/coredns:1.12.0 application/vnd.docker.distribution.manifest.list.v2+json sha256:40384aa1f5ea6bfdc77997d243aec73da05f27aed0c5e9d65bfa98933c519d97 20.0 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/riscv64,linux/s390x                            -      
docker.io/coredns/coredns:1.9.4  application/vnd.docker.distribution.manifest.list.v2+json sha256:b82e294de6be763f73ae71266c8f5466e7e03c69f3a1de96efd570284d35bb18 14.4 MiB linux/amd64,linux/arm,linux/arm64,linux/mips64le,linux/ppc64le,linux/s390x                              -      
docker.io/library/nginx:latest   application/vnd.oci.image.index.v1+json                   sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66 69.0 MiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x -      
docker.io/library/ubuntu:20.04   application/vnd.oci.image.index.v1+json                   sha256:8feb4d8ca5354def3d8fce243717141ce31e2c428701f6682bd2fafe15388214 26.2 MiB linux/amd64,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/riscv64,linux/s390x
ctr -n k8s.io images ls
REF                                                                                                                          TYPE                                                      DIGEST                                                                  SIZE      PLATFORMS                                                                                               LABELS                                                          
docker.io/library/nginx:latest                                                                                               application/vnd.oci.image.index.v1+json                   sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66 69.0 MiB  linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x io.cri-containerd.image=managed                                 
dockerproxy.net/coredns/coredns@sha256:40384aa1f5ea6bfdc77997d243aec73da05f27aed0c5e9d65bfa98933c519d97                      application/vnd.docker.distribution.manifest.list.v2+json sha256:40384aa1f5ea6bfdc77997d243aec73da05f27aed0c5e9d65bfa98933c519d97 20.0 MiB  linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/riscv64,linux/s390x                            io.cri-containerd.image=managed                                 
dockerproxy.net/library/nginx:latest                                                                                         application/vnd.oci.image.index.v1+json                   sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66 69.0 MiB  linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x io.cri-containerd.image=managed                                 
dockerproxy.net/library/nginx@sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66                        application/vnd.oci.image.index.v1+json                   sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66 69.0 MiB  linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x io.cri-containerd.image=managed                                 
quay.io/cilium/alpine-curl@sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364                           application/vnd.oci.image.index.v1+json                   sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 10.5 MiB  linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
quay.io/cilium/cilium-envoy@sha256:9762041c3760de226a8b00cc12f27dacc28b7691ea926748f9b5c18862db503f                          application/vnd.docker.distribution.manifest.list.v2+json sha256:9762041c3760de226a8b00cc12f27dacc28b7691ea926748f9b5c18862db503f 59.3 MiB  linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
quay.io/cilium/cilium-envoy@sha256:a01cadf7974409b5c5c92ace3d6afa298408468ca24cab1cb413c04f89d3d1f9                          application/vnd.docker.distribution.manifest.list.v2+json sha256:a01cadf7974409b5c5c92ace3d6afa298408468ca24cab1cb413c04f89d3d1f9 61.5 MiB  linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
quay.io/cilium/cilium@sha256:1782794aeac951af139315c10eff34050aa7579c12827ee9ec376bb719b82873                                application/vnd.docker.distribution.manifest.list.v2+json sha256:1782794aeac951af139315c10eff34050aa7579c12827ee9ec376bb719b82873 256.8 MiB linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
quay.io/cilium/cilium@sha256:4386a8580d8d86934908eea022b0523f812e6a542f30a86a47edd8bed90d51ea                                application/vnd.docker.distribution.manifest.list.v2+json sha256:4386a8580d8d86934908eea022b0523f812e6a542f30a86a47edd8bed90d51ea 212.8 MiB linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603                             application/vnd.oci.image.index.v1+json                   sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603 82.0 MiB  linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
quay.io/cilium/operator-generic@sha256:8bd38d0e97a955b2d725929d60df09d712fb62b60b930551a29abac2dd92e597                      application/vnd.docker.distribution.manifest.list.v2+json sha256:8bd38d0e97a955b2d725929d60df09d712fb62b60b930551a29abac2dd92e597 31.9 MiB  linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
quay.io/cilium/operator-generic@sha256:cccfd3b886d52cb132c06acca8ca559f0fce91a6bd99016219b1a81fdbc4813a                      application/vnd.docker.distribution.manifest.list.v2+json sha256:cccfd3b886d52cb132c06acca8ca559f0fce91a6bd99016219b1a81fdbc4813a 29.7 MiB  linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
registry.cn-beijing.aliyuncs.com/zhoujun/pause-amd64:3.1                                                                     application/vnd.docker.distribution.manifest.v2+json      sha256:784475ff64cd22bee7584c2f0edf8c6672afa691e9be6f0cb79434db4471e679 318.2 KiB linux/amd64                                                                                             io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
registry.cn-beijing.aliyuncs.com/zhoujun/pause-amd64@sha256:784475ff64cd22bee7584c2f0edf8c6672afa691e9be6f0cb79434db4471e679 application/vnd.docker.distribution.manifest.v2+json      sha256:784475ff64cd22bee7584c2f0edf8c6672afa691e9be6f0cb79434db4471e679 318.2 KiB linux/amd64                                                                                             io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
sha256:0fba21db2995c14ae9ae921eab5d2e5db3367089f274d3f586b1eef4341554d6                                                      application/vnd.docker.distribution.manifest.list.v2+json sha256:4386a8580d8d86934908eea022b0523f812e6a542f30a86a47edd8bed90d51ea 212.8 MiB linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
sha256:1596e0623f1cd6ad7f5c83ec64823d17dfcdf488ed785509b1fc38482f430b11                                                      application/vnd.docker.distribution.manifest.list.v2+json sha256:a01cadf7974409b5c5c92ace3d6afa298408468ca24cab1cb413c04f89d3d1f9 61.5 MiB  linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
sha256:1b6346e1ab671d17578169a6bc6c9a93c84b537b166920f4cee9ed1c8592ea65                                                      application/vnd.docker.distribution.manifest.list.v2+json sha256:cccfd3b886d52cb132c06acca8ca559f0fce91a6bd99016219b1a81fdbc4813a 29.7 MiB  linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
sha256:1cf5f116067c67da67f97bff78c4bbc76913f59057c18627b96facaced73ea0b                                                      application/vnd.docker.distribution.manifest.list.v2+json sha256:40384aa1f5ea6bfdc77997d243aec73da05f27aed0c5e9d65bfa98933c519d97 20.0 MiB  linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/riscv64,linux/s390x                            io.cri-containerd.image=managed                                 
sha256:2aa79125e9616dc160d117a55b11e6d48348a8e4e66da53d764f0378c5a8c893                                                      application/vnd.docker.distribution.manifest.list.v2+json sha256:8bd38d0e97a955b2d725929d60df09d712fb62b60b930551a29abac2dd92e597 31.9 MiB  linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
sha256:6f13d3371c7c4c1c4b2577b79a932853dc3f8d020c5ab656e6e0c6c941bd7d49                                                      application/vnd.docker.distribution.manifest.list.v2+json sha256:1782794aeac951af139315c10eff34050aa7579c12827ee9ec376bb719b82873 256.8 MiB linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
sha256:83f488ce64fb6e7e1df37b1631273e1e258924d38ad902b7ded1e8781d095210                                                      application/vnd.docker.distribution.manifest.v2+json      sha256:784475ff64cd22bee7584c2f0edf8c6672afa691e9be6f0cb79434db4471e679 318.2 KiB linux/amd64                                                                                             io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 
sha256:9eea940c938b4399706e0b1b7726308ae53e8bd8490cff9c12f6308fdf8af3ad                                                      application/vnd.oci.image.index.v1+json                   sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 10.5 MiB  linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
sha256:a830707172e8069c09cf6c67a04e23e5a1a332d70a90a54999b76273a928b9ce                                                      application/vnd.oci.image.index.v1+json                   sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66 69.0 MiB  linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x io.cri-containerd.image=managed                                 
sha256:adcc2d0552708b61775c71416f20abddad5fd39b52eb4ac10d692bd19a577edb                                                      application/vnd.oci.image.index.v1+json                   sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603 82.0 MiB  linux/amd64,linux/arm64                                                                                 io.cri-containerd.image=managed                                 
sha256:b6cb91c7af317bdd5522228ad63ed64e2e89470e768c591a02096c5e5edb7797                                                      application/vnd.docker.distribution.manifest.list.v2+json sha256:9762041c3760de226a8b00cc12f27dacc28b7691ea926748f9b5c18862db503f 59.3 MiB  linux/amd64,linux/arm64
crictl images ls
IMAGE                                                  TAG                 IMAGE ID            SIZE
dockerproxy.net/coredns/coredns                        <none>              1cf5f116067c6       20.9MB
docker.io/library/nginx                                latest              a830707172e80       72.4MB
dockerproxy.net/library/nginx                          latest              a830707172e80       72.4MB
quay.io/cilium/alpine-curl                             <none>              9eea940c938b4       11MB
quay.io/cilium/cilium-envoy                            <none>              b6cb91c7af317       62.2MB
quay.io/cilium/cilium-envoy                            <none>              1596e0623f1cd       64.4MB
quay.io/cilium/cilium                                  <none>              6f13d3371c7c4       269MB
quay.io/cilium/cilium                                  <none>              0fba21db2995c       223MB
quay.io/cilium/json-mock                               <none>              adcc2d0552708       85.9MB
quay.io/cilium/operator-generic                        <none>              2aa79125e9616       33.4MB
quay.io/cilium/operator-generic                        <none>              1b6346e1ab671       31.1MB
registry.cn-beijing.aliyuncs.com/zhoujun/pause-amd64   3.1                 83f488ce64fb6       326kB
nerdctl image ls
REPOSITORY         TAG       IMAGE ID        CREATED              PLATFORM       SIZE         BLOB SIZE
coredns/coredns    1.12.0    40384aa1f5ea    3 hours ago          linux/amd64    70.0 MiB     20.0 MiB
coredns/coredns    1.9.4     b82e294de6be    2 hours ago          linux/amd64    47.5 MiB     14.4 MiB
nginx              latest    c15da6c91de8    About an hour ago    linux/amd64    197.4 MiB    69.0 MiB
ubuntu             20.04     8feb4d8ca535    2 hours ago          linux/amd64    77.9 MiB     26.2 MiB

nerdctl --namespace k8s.io image ls
REPOSITORY                                              TAG       IMAGE ID        CREATED           PLATFORM       SIZE         BLOB SIZE
nginx                                                   latest    c15da6c91de8    13 minutes ago    linux/amd64    197.4 MiB    69.0 MiB
dockerproxy.net/coredns/coredns                         <none>    40384aa1f5ea    2 hours ago       linux/amd64    70.0 MiB     20.0 MiB
dockerproxy.net/library/nginx                           latest    c15da6c91de8    48 minutes ago    linux/amd64    197.4 MiB    69.0 MiB
dockerproxy.net/library/nginx                           <none>    c15da6c91de8    48 minutes ago    linux/amd64    197.4 MiB    69.0 MiB
quay.io/cilium/alpine-curl                              <none>    913e8c9f3d96    7 hours ago       linux/amd64    24.8 MiB     10.5 MiB
quay.io/cilium/cilium-envoy                             <none>    9762041c3760    5 hours ago       linux/amd64    165.2 MiB    59.3 MiB
quay.io/cilium/cilium-envoy                             <none>    a01cadf79744    7 hours ago       linux/amd64    170.8 MiB    61.5 MiB
quay.io/cilium/cilium                                   <none>    1782794aeac9    7 hours ago       linux/amd64    787.0 MiB    256.8 MiB
quay.io/cilium/cilium                                   <none>    4386a8580d8d    5 hours ago       linux/amd64    604.2 MiB    212.8 MiB
quay.io/cilium/json-mock                                <none>    5aad04835eda    4 hours ago       linux/amd64    258.6 MiB    82.0 MiB
quay.io/cilium/operator-generic                         <none>    8bd38d0e97a9    7 hours ago       linux/amd64    120.1 MiB    31.9 MiB
quay.io/cilium/operator-generic                         <none>    cccfd3b886d5    5 hours ago       linux/amd64    99.9 MiB     29.7 MiB
registry.cn-beijing.aliyuncs.com/zhoujun/pause-amd64    3.1       784475ff64cd    7 hours ago       linux/amd64    732.0 KiB    318.2 KiB
registry.cn-beijing.aliyuncs.com/zhoujun/pause-amd64    <none>    784475ff64cd    7 hours ago       linux/amd64    732.0 KiB    318.2 KiB

ctr 和 nedrctl 都有一个 Namespace 的概念,默认操作的都是 default 这个 ns。crictl 操作的是 k8s.io ,所以直接nerdctl image pull nginx:latest 的镜像是无法被 Kubernetes 加载的,需要在 pull 的时候指定 ns,那么命令就变成了 nerdctl --namespace k8s.io --debug=true image pull nginx:latest

问题处理

Coredns 报错

kubectl logs -f -n kube-system coredns-6b59c98dd4-5hr86
.:53
[INFO] plugin/reload: Running configuration SHA512 = 165c2220dade77166d5aa3508a7dd59a865dbb9a90ca5af87e487b4217222b80d5b35bedd2640b4c4895a10c969c81b9d1ff90d2592ccccc0307eebe5469b6ab
CoreDNS-1.9.4
linux/amd64, go1.19.1, 1f0a41a
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:52567->103.224.222.223:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:42164->169.254.169.240:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:57743->103.224.222.223:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:46741->103.224.222.222:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:45939->103.224.222.222:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:37908->169.254.169.240:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:51323->103.224.222.223:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:38532->103.224.222.223:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:56052->169.254.169.240:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:37307->103.224.222.222:53: i/o timeout

先ping 8.8.8.8,如果这个不通基本就是 k8s 集群网络出问题。这里我没问题。

kubectl edit cm coredns -n kube-system -o yaml , 我把forward . /etc/resove.conf换成了更可靠的域名服务器8.8.8.8,改完之后重启了delete了两个coredns的pod,等重新起来后没有报错了,好用了

data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . 8.8.8.8
        cache 30
        loop
        reload
        loadbalance
    }

已经创建的pod需要删除重新创建