08 | 部署网络组件 Cilium + CoreDNS
Kuberneters 集群支持多种网络插件,例如:Flannel、Cilium、Calico、Weave。企业生产环境中用的比较多的是 Cilium。所以本节课,选择 Cilium 作为集群的网络插件。
另外,部署完网络插件之后,还需要部署 DNS 服务。Kubernetes 集群标准的 DNS 组件是 CoreDNS。
所以,本节课,来介绍下如何安装:Cilium、CoreDNS。
提示:Flannel 网络插件实现简单、代码量少。如果你想学习 Kubernetes 网络插件的实现,可以学习 Flannel 的。其他网络插件的实现原理跟 Flannel 核心思想是相通的。Flannel 网络插件的安装参考附录 A。
部署 Cilium
在安装 cilium 前,要前安装好 kube-proxy。
cilium 官方提供了详细的安装文档:Cilium Quick Installation。你可以根据官方的文档来安装,也可以继续跟随本教程来安装。
安装 Cilium 命令
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm -f cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
安装 Cilium 网络插件
helm安装
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium --namespace kube-system --set image.tag=1.16.2
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "cp /opt/cni/bin/cilium-cni /opt/k8s/bin/"
done
执行完上述命令之后,会在集群中安装 cilium 网络插件。通过以下命令可以查看安装进度:
$ kubectl -n kube-system get pods -owide|grep cilium
cilium-ckbfl 1/1 Running 0 94m
cilium-envoy-l4gth 1/1 Running 0 94m
cilium-envoy-tzdjw 1/1 Running 0 94m
cilium-hhsrd 1/1 Running 0 94m
cilium-operator-5874db7569-gn72x 1/1 Running 0 94m
cilium-operator-5874db7569-nkv8k 1/1 Running 0 94m
如果 Pod 都处在 Running 状态,说明 cilium 被正确安装。可以看到,helm install cilium cilium/cilium会在 Kubernetes 集群中安装以下组件:
- cilium-operator(Deployment):Cilium Operator,负责协调和管理 Cilium 的高级功能;
- cilium(DaemonSet):cilium CNI 网络插件
- cilium-envoy(DaemonSet):Envoy 代理 Pods,与 Cilium 集成来提供增强的网络功能。
官网方法安装
cilium install --version 1.16.2
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "cp /opt/cni/bin/cilium-cni /opt/k8s/bin/"
done
也可以通过以下命令来查看 cilium 是否被正确安装:(如果使用云服务器需要提前开放安全组的10250端口)
cilium status --wait
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: disabled
\__/ ClusterMesh: disabled
DaemonSet cilium Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet cilium-envoy Desired: 2, Ready: 2/2, Available: 2/2
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 2
cilium-envoy Running: 2
cilium-operator Running: 1
clustermesh-apiserver
hubble-relay
Cluster Pods: 3/3 managed by Cilium
Helm chart version: 1.16.2
Image versions cilium quay.io/cilium/cilium:v1.16.2@sha256:4386a8580d8d86934908eea022b0523f812e6a542f30a86a47edd8bed90d51ea: 2
cilium-envoy quay.io/cilium/cilium-envoy:v1.29.9-1726784081-a90146d13b4cd7d168d573396ccf2b3db5a3b047@sha256:9762041c3760de226a8b00cc12f27dacc28b7691ea926748f9b5c18862db503f: 2
cilium-operator quay.io/cilium/operator-generic:v1.16.2@sha256:cccfd3b886d52cb132c06acca8ca559f0fce91a6bd99016219b1a81fdbc4813a: 1
集群网络验证
执行以下命令创建 2 个测试 Pod:
cd /opt/k8s/work
cat > dnsutils-for-cilium-test-ds.yaml <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: dnsutils-for-cilium-test
labels:
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
app: dnsutils-for-cilium-test
template:
metadata:
labels:
app: dnsutils-for-cilium-test
spec:
containers:
- name: my-dnsutils
image: tutum/dnsutils:latest
imagePullPolicy: IfNotPresent
command:
- tail
- "-f"
- "/dev/null"
ports:
- containerPort: 80
EOF
kubectl create -f dnsutils-for-cilium-test-ds.yaml
执行以下命令获取 Pod 创建状态:
$ kubectl get pods -o wide -l app=dnsutils-for-cilium-test
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dnsutils-for-cilium-test-ngk74 1/1 Running 0 25m 10.0.1.80 k8s-01 <none> <none>
dnsutils-for-cilium-test-rqqhc 1/1 Running 0 25m 10.0.0.133 k8s-02 <none> <none>
待 Pod 都处在 Running状态后,测试是否可以从一个 Pod 内 ping 通另外一个 Pod 的 IP 地址:
$ kubectl exec -it dnsutils-for-cilium-test-ngk74 -- ping 10.0.0.133
PING 10.0.0.133 (10.0.0.133) 56(84) bytes of data.
64 bytes from 10.0.0.133: icmp_seq=1 ttl=63 time=0.785 ms
64 bytes from 10.0.0.133: icmp_seq=2 ttl=63 time=0.758 ms
...
注意:如果使用云主机需要提前放通 route 中提示的安全组,以免 cilium 间通信有问题导致无法访问(比如ping其他节点的Pod IP不通)。
部署 coredns
要在 Kubernetes 集群内实现服务发现,必须具备 DNS 组件,以便将服务域名解析为 Pod IP,从而通过 IP 地址进行通信。目前,CoreDNS 是 Kubernetes 环境中最常用的 DNS 解决方案。
在本小节中,我将逐步介绍如何部署 CoreDNS,确保集群内服务的 DNS 解析顺畅无阻。
下载和配置 CoreDNS
cd /opt/k8s/work
git clone https://github.com/coredns/deployment.git
mv deployment coredns-deployment
部署 coredns
cd /opt/k8s/work/coredns-deployment/kubernetes
source /opt/k8s/bin/environment.sh
./deploy.sh -i ${CLUSTER_DNS_SVC_IP} -d ${CLUSTER_DNS_DOMAIN} | kubectl apply -f -
检查 coredns 功能
我们可以通过以下 3 种方式来检查 CoreDNS 功能是否正常:
- 检查 CoreDNS 组件是否正常运行;
- 在 Pod 中访问 Kubernetes 服务;
- 检查 DNS 配置,并测试 DNS 解析。
具体操作如下。
检查 CoreDNS 组件是否正常运行
$ kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-6b59c98dd4-z249x 1/1 Running 0 4m57s
coredns Pod 处在 Running状态,说明 coredns 部署成功。
在 Pod 中访问 Kubernetes 服务
新建一个 Deployment:
cd /opt/k8s/work
cat > my-nginx.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 2
selector:
matchLabels:
run: my-nginx
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx:1.7.9
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
EOF
kubectl apply -f my-nginx.yaml
expose 该 Deployment, 生成 my-nginx 服务:
$ kubectl expose deploy my-nginx
service "my-nginx" exposed
$ kubectl get services my-nginx -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
my-nginx ClusterIP 10.254.4.128 <none> 80/TCP 5s run=my-nginx
创建另一个 Pod,查看 /etc/resolv.conf 是否包含 kubelet 配置的 --cluster-dns 和 --cluster-domain,是否能够将服务 my-nginx 解析到上面显示的 Cluster IP 10.254.4.128。
检查 DNS 配置,并测试 DNS 解析
创建一个测试用的 dnsutils Pod:
cd /opt/k8s/work
cat > dnsutils-ds.yml <<EOF
apiVersion: v1
kind: Service
metadata:
name: dnsutils-ds
labels:
app: dnsutils-ds
spec:
type: NodePort
selector:
app: dnsutils-ds
ports:
- name: http
port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: dnsutils-ds
labels:
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
app: dnsutils-ds
template:
metadata:
labels:
app: dnsutils-ds
spec:
containers:
- name: my-dnsutils
image: tutum/dnsutils:latest
imagePullPolicy: IfNotPresent
command:
- tail
- "-f"
- "/dev/null"
ports:
- containerPort: 80
EOF
kubectl create -f dnsutils-ds.yml
查看测试 Pod 是否成功创建:
$ kubectl get pods -lapp=dnsutils-ds -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dnsutils-ds-jc55p 1/1 Running 0 10s 10.0.1.155 k8s-01 <none> <none>
dnsutils-ds-rnq7d 1/1 Running 0 10s 10.0.0.54 k8s-02 <none> <none>
查看 DNS 解析配置是否正确:
$ kubectl exec -it dnsutils-ds-jc55p -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.254.0.2
options ndots:5
使用 DNS 能否成功解析 Kubernetes 内服务域名:
$ kubectl exec -it dnsutils-ds-jc55p -- nslookup kubernetes
Server: 10.254.0.2
Address: 10.254.0.2#53
Name: kubernetes.default.svc.cluster.local
Address: 10.254.0.1
;; Got recursion not available from 10.254.0.2
$ kubectl exec -it dnsutils-ds-9zmbk -- nslookup my-nginx
Server: 10.254.0.2
Address: 10.254.0.2#53
Name: my-nginx.default.svc.cluster.local
Address: 10.254.4.128
;; Got recursion not available from 10.254.0.2
可以看到 DNS 能够成功解析 Kubernetes 内以下 2 类服务域名:
- Kubernetes 内置服务域名:
kubernetes->10.254.0.1; - 用户创建的外部服务域名:
my-nginx->10.254.4.128。
查看使用 DNS 能否成功解析公网域名:
$ kubectl -it exec dnsutils-ds-9zmbk -- nslookup www.baidu.com
;; Got recursion not available from 10.254.0.2
;; Got recursion not available from 10.254.0.2
;; Got recursion not available from 10.254.0.2
Server: 10.254.0.2
Address: 10.254.0.2#53
Non-authoritative answer:
www.baidu.com canonical name = www.a.shifen.com.
www.a.shifen.com canonical name = www.wshifen.com.
Name: www.wshifen.com
Address: 103.235.46.102
Name: www.wshifen.com
Address: 103.235.46.115
可以看到 DNS 能够成功解析 www.baidu.com,解析到的 IP 地址为:103.235.46.102。
更全面的 Kubernetes 网络测试
通过上面的部署,我们已经成功部署了网络插件和 DNS 组件。至此,Kubernetes 的网络功能已经健全,我们可以对 Kubernetes 网络进行更全面的测试。
cilium 官网提供了测试功能,可以用来全面的测试 Kubernetes 网络。可以执行以下命令来测试:
$ cilium connectivity test
ℹ️ Monitor aggregation detected, will skip some flow validation steps
✨ [k8s-cluster] Creating namespace for connectivity check...
---------------------------------------------------------------------------------------------------------------------
📋 Test Report
---------------------------------------------------------------------------------------------------------------------
✅ 69/69 tests successful (0 warnings)
69/69 tests successful (0 warnings说明所有的 Kubernetes 集群网络连通性测试通过。
这里要注意:
- Kubernetes 连接性测试可能会因为一个或多个 Pod 中打开的文件过多而无法部署。如果你遇到
too many open files这类错误,可以增加 inotify 资源限制; cilium connectivity test测试用例很多,对网络功能 测试更加全面和严格,其中有些用例可能会因为集群配置、镜像下载失败等原因失败,有些失败的测试用例,也可能不会影响正常的集群功能。所以,其测试结果仅做参考。
参考
- https://community.infoblox.com/t5/Community-Blog/CoreDNS-for-Kubernetes-Service-Discovery/ba-p/8187
- https://coredns.io/2017/03/01/coredns-for-kubernetes-service-discovery-take-2/
- https://www.cnblogs.com/boshen-hzb/p/7511432.html
- https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns
补充
对 containerd 配置了镜像加速后,在 kubelet 拉取的时候发现没生效,还是拉取不到镜像。
通过 nerdctl 命令手动拉取镜像是可以正常拉取的,但是集群识别不到手工拉取的镜像。
搜索发现 ctr crictl nedrctl 三条命令的输出结果是不同的。
ctr images ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/coredns/coredns:1.12.0 application/vnd.docker.distribution.manifest.list.v2+json sha256:40384aa1f5ea6bfdc77997d243aec73da05f27aed0c5e9d65bfa98933c519d97 20.0 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/riscv64,linux/s390x -
docker.io/coredns/coredns:1.9.4 application/vnd.docker.distribution.manifest.list.v2+json sha256:b82e294de6be763f73ae71266c8f5466e7e03c69f3a1de96efd570284d35bb18 14.4 MiB linux/amd64,linux/arm,linux/arm64,linux/mips64le,linux/ppc64le,linux/s390x -
docker.io/library/nginx:latest application/vnd.oci.image.index.v1+json sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66 69.0 MiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x -
docker.io/library/ubuntu:20.04 application/vnd.oci.image.index.v1+json sha256:8feb4d8ca5354def3d8fce243717141ce31e2c428701f6682bd2fafe15388214 26.2 MiB linux/amd64,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/riscv64,linux/s390x
ctr -n k8s.io images ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/library/nginx:latest application/vnd.oci.image.index.v1+json sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66 69.0 MiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
dockerproxy.net/coredns/coredns@sha256:40384aa1f5ea6bfdc77997d243aec73da05f27aed0c5e9d65bfa98933c519d97 application/vnd.docker.distribution.manifest.list.v2+json sha256:40384aa1f5ea6bfdc77997d243aec73da05f27aed0c5e9d65bfa98933c519d97 20.0 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/riscv64,linux/s390x io.cri-containerd.image=managed
dockerproxy.net/library/nginx:latest application/vnd.oci.image.index.v1+json sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66 69.0 MiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
dockerproxy.net/library/nginx@sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66 application/vnd.oci.image.index.v1+json sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66 69.0 MiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
quay.io/cilium/alpine-curl@sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 application/vnd.oci.image.index.v1+json sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 10.5 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/cilium-envoy@sha256:9762041c3760de226a8b00cc12f27dacc28b7691ea926748f9b5c18862db503f application/vnd.docker.distribution.manifest.list.v2+json sha256:9762041c3760de226a8b00cc12f27dacc28b7691ea926748f9b5c18862db503f 59.3 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/cilium-envoy@sha256:a01cadf7974409b5c5c92ace3d6afa298408468ca24cab1cb413c04f89d3d1f9 application/vnd.docker.distribution.manifest.list.v2+json sha256:a01cadf7974409b5c5c92ace3d6afa298408468ca24cab1cb413c04f89d3d1f9 61.5 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/cilium@sha256:1782794aeac951af139315c10eff34050aa7579c12827ee9ec376bb719b82873 application/vnd.docker.distribution.manifest.list.v2+json sha256:1782794aeac951af139315c10eff34050aa7579c12827ee9ec376bb719b82873 256.8 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/cilium@sha256:4386a8580d8d86934908eea022b0523f812e6a542f30a86a47edd8bed90d51ea application/vnd.docker.distribution.manifest.list.v2+json sha256:4386a8580d8d86934908eea022b0523f812e6a542f30a86a47edd8bed90d51ea 212.8 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603 application/vnd.oci.image.index.v1+json sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603 82.0 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/operator-generic@sha256:8bd38d0e97a955b2d725929d60df09d712fb62b60b930551a29abac2dd92e597 application/vnd.docker.distribution.manifest.list.v2+json sha256:8bd38d0e97a955b2d725929d60df09d712fb62b60b930551a29abac2dd92e597 31.9 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/operator-generic@sha256:cccfd3b886d52cb132c06acca8ca559f0fce91a6bd99016219b1a81fdbc4813a application/vnd.docker.distribution.manifest.list.v2+json sha256:cccfd3b886d52cb132c06acca8ca559f0fce91a6bd99016219b1a81fdbc4813a 29.7 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
registry.cn-beijing.aliyuncs.com/zhoujun/pause-amd64:3.1 application/vnd.docker.distribution.manifest.v2+json sha256:784475ff64cd22bee7584c2f0edf8c6672afa691e9be6f0cb79434db4471e679 318.2 KiB linux/amd64 io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
registry.cn-beijing.aliyuncs.com/zhoujun/pause-amd64@sha256:784475ff64cd22bee7584c2f0edf8c6672afa691e9be6f0cb79434db4471e679 application/vnd.docker.distribution.manifest.v2+json sha256:784475ff64cd22bee7584c2f0edf8c6672afa691e9be6f0cb79434db4471e679 318.2 KiB linux/amd64 io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
sha256:0fba21db2995c14ae9ae921eab5d2e5db3367089f274d3f586b1eef4341554d6 application/vnd.docker.distribution.manifest.list.v2+json sha256:4386a8580d8d86934908eea022b0523f812e6a542f30a86a47edd8bed90d51ea 212.8 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
sha256:1596e0623f1cd6ad7f5c83ec64823d17dfcdf488ed785509b1fc38482f430b11 application/vnd.docker.distribution.manifest.list.v2+json sha256:a01cadf7974409b5c5c92ace3d6afa298408468ca24cab1cb413c04f89d3d1f9 61.5 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
sha256:1b6346e1ab671d17578169a6bc6c9a93c84b537b166920f4cee9ed1c8592ea65 application/vnd.docker.distribution.manifest.list.v2+json sha256:cccfd3b886d52cb132c06acca8ca559f0fce91a6bd99016219b1a81fdbc4813a 29.7 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
sha256:1cf5f116067c67da67f97bff78c4bbc76913f59057c18627b96facaced73ea0b application/vnd.docker.distribution.manifest.list.v2+json sha256:40384aa1f5ea6bfdc77997d243aec73da05f27aed0c5e9d65bfa98933c519d97 20.0 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/riscv64,linux/s390x io.cri-containerd.image=managed
sha256:2aa79125e9616dc160d117a55b11e6d48348a8e4e66da53d764f0378c5a8c893 application/vnd.docker.distribution.manifest.list.v2+json sha256:8bd38d0e97a955b2d725929d60df09d712fb62b60b930551a29abac2dd92e597 31.9 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
sha256:6f13d3371c7c4c1c4b2577b79a932853dc3f8d020c5ab656e6e0c6c941bd7d49 application/vnd.docker.distribution.manifest.list.v2+json sha256:1782794aeac951af139315c10eff34050aa7579c12827ee9ec376bb719b82873 256.8 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
sha256:83f488ce64fb6e7e1df37b1631273e1e258924d38ad902b7ded1e8781d095210 application/vnd.docker.distribution.manifest.v2+json sha256:784475ff64cd22bee7584c2f0edf8c6672afa691e9be6f0cb79434db4471e679 318.2 KiB linux/amd64 io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
sha256:9eea940c938b4399706e0b1b7726308ae53e8bd8490cff9c12f6308fdf8af3ad application/vnd.oci.image.index.v1+json sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 10.5 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
sha256:a830707172e8069c09cf6c67a04e23e5a1a332d70a90a54999b76273a928b9ce application/vnd.oci.image.index.v1+json sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66 69.0 MiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
sha256:adcc2d0552708b61775c71416f20abddad5fd39b52eb4ac10d692bd19a577edb application/vnd.oci.image.index.v1+json sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603 82.0 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
sha256:b6cb91c7af317bdd5522228ad63ed64e2e89470e768c591a02096c5e5edb7797 application/vnd.docker.distribution.manifest.list.v2+json sha256:9762041c3760de226a8b00cc12f27dacc28b7691ea926748f9b5c18862db503f 59.3 MiB linux/amd64,linux/arm64
crictl images ls
IMAGE TAG IMAGE ID SIZE
dockerproxy.net/coredns/coredns <none> 1cf5f116067c6 20.9MB
docker.io/library/nginx latest a830707172e80 72.4MB
dockerproxy.net/library/nginx latest a830707172e80 72.4MB
quay.io/cilium/alpine-curl <none> 9eea940c938b4 11MB
quay.io/cilium/cilium-envoy <none> b6cb91c7af317 62.2MB
quay.io/cilium/cilium-envoy <none> 1596e0623f1cd 64.4MB
quay.io/cilium/cilium <none> 6f13d3371c7c4 269MB
quay.io/cilium/cilium <none> 0fba21db2995c 223MB
quay.io/cilium/json-mock <none> adcc2d0552708 85.9MB
quay.io/cilium/operator-generic <none> 2aa79125e9616 33.4MB
quay.io/cilium/operator-generic <none> 1b6346e1ab671 31.1MB
registry.cn-beijing.aliyuncs.com/zhoujun/pause-amd64 3.1 83f488ce64fb6 326kB
nerdctl image ls
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
coredns/coredns 1.12.0 40384aa1f5ea 3 hours ago linux/amd64 70.0 MiB 20.0 MiB
coredns/coredns 1.9.4 b82e294de6be 2 hours ago linux/amd64 47.5 MiB 14.4 MiB
nginx latest c15da6c91de8 About an hour ago linux/amd64 197.4 MiB 69.0 MiB
ubuntu 20.04 8feb4d8ca535 2 hours ago linux/amd64 77.9 MiB 26.2 MiB
nerdctl --namespace k8s.io image ls
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
nginx latest c15da6c91de8 13 minutes ago linux/amd64 197.4 MiB 69.0 MiB
dockerproxy.net/coredns/coredns <none> 40384aa1f5ea 2 hours ago linux/amd64 70.0 MiB 20.0 MiB
dockerproxy.net/library/nginx latest c15da6c91de8 48 minutes ago linux/amd64 197.4 MiB 69.0 MiB
dockerproxy.net/library/nginx <none> c15da6c91de8 48 minutes ago linux/amd64 197.4 MiB 69.0 MiB
quay.io/cilium/alpine-curl <none> 913e8c9f3d96 7 hours ago linux/amd64 24.8 MiB 10.5 MiB
quay.io/cilium/cilium-envoy <none> 9762041c3760 5 hours ago linux/amd64 165.2 MiB 59.3 MiB
quay.io/cilium/cilium-envoy <none> a01cadf79744 7 hours ago linux/amd64 170.8 MiB 61.5 MiB
quay.io/cilium/cilium <none> 1782794aeac9 7 hours ago linux/amd64 787.0 MiB 256.8 MiB
quay.io/cilium/cilium <none> 4386a8580d8d 5 hours ago linux/amd64 604.2 MiB 212.8 MiB
quay.io/cilium/json-mock <none> 5aad04835eda 4 hours ago linux/amd64 258.6 MiB 82.0 MiB
quay.io/cilium/operator-generic <none> 8bd38d0e97a9 7 hours ago linux/amd64 120.1 MiB 31.9 MiB
quay.io/cilium/operator-generic <none> cccfd3b886d5 5 hours ago linux/amd64 99.9 MiB 29.7 MiB
registry.cn-beijing.aliyuncs.com/zhoujun/pause-amd64 3.1 784475ff64cd 7 hours ago linux/amd64 732.0 KiB 318.2 KiB
registry.cn-beijing.aliyuncs.com/zhoujun/pause-amd64 <none> 784475ff64cd 7 hours ago linux/amd64 732.0 KiB 318.2 KiB
ctr 和 nedrctl 都有一个 Namespace 的概念,默认操作的都是 default 这个 ns。crictl 操作的是 k8s.io ,所以直接nerdctl image pull nginx:latest 的镜像是无法被 Kubernetes 加载的,需要在 pull 的时候指定 ns,那么命令就变成了 nerdctl --namespace k8s.io --debug=true image pull nginx:latest
问题处理
Coredns 报错
kubectl logs -f -n kube-system coredns-6b59c98dd4-5hr86
.:53
[INFO] plugin/reload: Running configuration SHA512 = 165c2220dade77166d5aa3508a7dd59a865dbb9a90ca5af87e487b4217222b80d5b35bedd2640b4c4895a10c969c81b9d1ff90d2592ccccc0307eebe5469b6ab
CoreDNS-1.9.4
linux/amd64, go1.19.1, 1f0a41a
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:52567->103.224.222.223:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:42164->169.254.169.240:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:57743->103.224.222.223:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:46741->103.224.222.222:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:45939->103.224.222.222:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:37908->169.254.169.240:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:51323->103.224.222.223:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:38532->103.224.222.223:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:56052->169.254.169.240:53: i/o timeout
[ERROR] plugin/errors: 2 1953002739233007189.2967571781216122921. HINFO: read udp 10.0.1.253:37307->103.224.222.222:53: i/o timeout
先ping 8.8.8.8,如果这个不通基本就是 k8s 集群网络出问题。这里我没问题。
kubectl edit cm coredns -n kube-system -o yaml , 我把forward . /etc/resove.conf换成了更可靠的域名服务器8.8.8.8,改完之后重启了delete了两个coredns的pod,等重新起来后没有报错了,好用了
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . 8.8.8.8
cache 30
loop
reload
loadbalance
}
已经创建的pod需要删除重新创建