高可用集群管理

Etcd 成员重要参数

--name 'default' Human-readable name for this member

--data-dir '${name}.etcd' Path to the data directory

--listen-peer-urls 'http://localhost:2380' List of URLs to listen on for peer traffic

--listen-client-urls 'http://localhost:2379' List of URLs to listen on for client traffic

Etcd 集群重要参数

--initial-advertise-peer-urls 'http://localhost:2380' List of this member's peer URLs to advertise to the rest of the cluster.

--initial-cluster'default=http://localhost:2380Initial cluster configuration for bootstrapping.

--initial-cluster-state 'new' Initial cluster state ('new' or 'existing').

--initial-cluster-token 'etcd-cluster'Initial cluster token for the etcd cluster during bootstrap.

--advertise-client-urls'http://localhost:2379'List of this member's client URLs to advertise to the public.

Etcd 安全相关参数

--cert-file Path to the client server TLs cert file.

--key-file Path to the client server TLs key file.

--client-crl-file Path to the client certificate revocation list file.

--trusted-ca-file Path to the client server TLS trusted CA cert file.

--peer-cert-file Path to the peer server TLS cert file.

--peer-key-file Path to the peer server TLs key file.

--peer-trusted-ca-file Path to the peer server TLS trusted CA file.

灾备

举个例子:Calico 会保存所有的ip分配信息,如果etcd数据丢失,可能会导致IP重复分配。

容量管理

单个对象不建议超过 1.5M

默认容量2G

不建议超过8G

Alarm & Disarm Alarm

etcd --quota-backend-bytes=$((16*1024*1024))
while [1];do dd if=/dev/urandom bs=1024count=1024 | ETCDCTL_API=3 etcdctl put key || break;done
$ ETCDCTL_API=3 etcdctl --write-out=table endpoint status
http://localhost:2379, c9ac9fc89eae9cf7, 3.4.3, 609MB, false, false, 67174, 666392422, 666392422, memberID:xxxxxxx alarm:NOSPACE
ETCDCTL_API=3 etcdctl alarm list
ETCDCTL_API=3 etcdctl defrag
$ ETCDCTL_API=3 etcdctl alarm disarm
memberID:xxxxxxx alarm:NOSPACE

碎片整理

# keep one hour of history
$ etcd --auto-compaction-retention=1

# compact up to revision 3
$ etcdctl compact 3

$ etcdctl defrag
Finished defragmenting etcd member[27.0.0.1:2379]

练习

高可用 etcd 解决方案

etcd-operator:coreos 开源的,基于 Kubernetes CRD 完成 etcd 集群配置。

Etcd statefulset Helm Chart:Bitnami(powered by vmware)

基于 Bitnami 安装 etcd 高可用集群

Kubernetes 如何使用 etcd

Etcd 是 Kubernetes 的后端存储

对于每一个 Kubernetes Object,都有对应的 storag.go 负责对象的存储操作

API Server 启动脚本中指定 ertcd servers 集群

spec: 
  containers: 
  - command: 
    - kube-apiserver
    - --advertise-address=11.13.242.28
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/etcd/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/etcd/apiserver-etcd-client.key
    - --etcd-servers=https://11.13.242.28:23799,https://11.13.242.29:23799,https://11.13.242.30:23799
    volumeMounts: 
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  volumes: 
  - hostPath: 
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-crets

早期 API Server 对 etcd 做简单的 Ping check,现在已经改为真实的 etcd api call。

Kubernetes 对象在 etcd 中的存储路径

Kubectl exec -it etcd-cadmin sh
ETCDCTL_API=3
alias ectl='etcdctl--endpoints https://127.0.0.1:2379\
  --cacert/etc/kubernetes/pki/etcd/ca.crt\
  --cert /etc/kubernetes/pki/etcd/server.crt
  --key /etc/kubernetes/pki/etcd/server.key'

etcdctl get --prefix --keys-only /
/registry/namespaces/calico-apiserver
/registry/networkpolicies/calico-apiserver/allow-apiserver
/registry/operator.tigera.io/tigerastatuses/apiserver
/registry/pods/calico-apiserver/calico-apiserver-77dffffcdf-g2tcx
/registry/pods/default/toolbox-68f79dd5f8-4664n

etcdctl get --prefix --keys-only /namespaces
/namespaces/default
/namespaces/kube-system
/namespaces/dev

Etcd 在集群中所处的位置

实践:etcd集群高可用

多少个 peer 最合适?