etcd 是基于 Raft 的分布式 key-value 存储系统,由 CoreOS 开发,常用于服务发现、共享配置以及并发控制(如 leader 选举、分布式锁等)。kubernetes 使用 etcd 存储所有运行数据。
部署准备
所有操作都是在devops机器上操作,将相关脚本编辑好,然后通过ansible 脚本模块统一执行。
定义环境变量
#################### Variable parameter setting ######################
ETCD_INSTALL_PATH="/data/apps/k8s/etcd"
ETCD_BIN_DIR="${ETCD_INSTALL_PATH}/bin"
ETCD_DATA_DIR="${ETCD_INSTALL_PATH}/data"
ETCD_WAL_DIR="${ETCD_INSTALL_PATH}/wal"
N01="master-k8s-n01=https://10.10.10.22:2380"
N02="master-k8s-n02=https://10.10.10.23:2380"
N03="master-k8s-n03=https://10.10.10.24:2380"
ETCD_ENPOIDTS="$N01,$N02,$N03"
CA_PATH="/etc/k8s/ssl"
SOFTWARE="/root/software"
VERSION=v3.3.13
PACKAGE="etcd-${VERSION}-linux-amd64.tar.gz"
HOSTNAME=`hostname`
DOWNLOAD_URL="https://github.com/devops-apps/download/raw/master/etcd/${PACKAG}"
ETH_INTERFACE=eth2
LISTEN_IP=$(ifconfig | grep -A 1 ${ETH_INTERFACE} |grep inet |awk '{print $2}')
USER=k8s
下载和分发 etcd 二进制文件
登录devops访问etcd 的 release 页面下载最新版本的发布包并通过ansible同步到三个master三个节点/root/software目录下:
wget $DOWNLOAD_URL -P $SOFTWARE
2 部署etcd集群
在devops主机上使用ansible分别对三个master节点执行以下操作:
2.1 安装etcd二进制执行命令
### 1.Check if the install directory exists.
if [ ! -d $ETCD_BIN_DIR ]; then
mkdir -p $ETCD_BIN_DIR
mkdir -p $ETCD_DATA_DIR
mkdir -p $ETCD_WAL_DIR
fi
### 2.Install etcd binary of kubernetes.
if [ ! -f "$SOFTWARE/etcd-${VERSION}-linux-amd64.tar.gz" ]; then
sudo wget $DOWNLOAD_URL -P $SOFTWARE >>/tmp/install.log 2>&1
fi
cd $SOFTWARE && tar -xzf etcd-${VERSION}-linux-amd64.tar.gz -C ./
cp -fp etcd-${VERSION}-linux-amd64/etcd* $ETCD_BIN_DIR
ln -sf $ETCD_BIN_DIR/{etcd,etcdctl} /usr/local/bin
sudo chmod -R 755 $ETCD_INSTALL_PATH
sudo chown -R $USER:$USER $ETCD_INSTALL_PATH
2.2 创建 etcd 的 systemd unit 启动服务文件
cat >/usr/lib/systemd/system/etcd.service<<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
User=${USER}
WorkingDirectory=${ETCD_DATA_DIR}
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) ${ETCD_BIN_DIR}/etcd \\
--name=${HOSTNAME} \\
--data-dir=${ETCD_DATA_DIR} \\
--wal-dir=${ETCD_WAL_DIR} \\
--cert-file=${CA_PATH}/etcd.pem \\
--key-file=${CA_PATH}/etcd-key.pem \\
--trusted-ca-file=${CA_PATH}/ca.pem \\
--peer-cert-file=${CA_PATH}/etcd.pem \\
--peer-key-file=${CA_PATH}/etcd-key.pem \\
--peer-trusted-ca-file=${CA_PATH}/ca.pem \\
--peer-client-cert-auth=true \\
--client-cert-auth=true \\
--listen-peer-urls=https://${LISTEN_IP}:2380 \\
--initial-advertise-peer-urls=https://${LISTEN_IP}:2380 \\
--listen-client-urls=https://${LISTEN_IP}:2379,https://127.0.0.1:2379 \\
--advertise-client-urls=https://${LISTEN_IP}:2379 \\
--initial-cluster-token=etcd-cluster-0 \\
--initial-cluster=${ETCD_ENPOIDTS} \\
--initial-cluster-state=new \\
--auto-tls=true \\
--auto-compaction-mode=periodic \\
--auto-compaction-retention=1 \\
--max-request-bytes=33554432 \\
--quota-backend-bytes=6442450944 \\
--heartbeat-interval=250 \\
--election-timeout=2000 \\
--peer-auto-tls=true"
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
- WorkingDirectory、--data-dir:指定工作目录和数据目录为 ${ETCD_DATA_DIR},需在启动服务前创建这个目录;
- --wal-dir:指定 wal 目录,为了提高性能,一般使用 SSD 或者和 --data-dir 不同的磁盘;
- --name:指定节点名称,当 --initial-cluster-state 值为 new 时,--name 的参数值必须位于 --initial-cluster 列表中;
- --cert-file、--key-file:etcd server 与 client 通信时使用的证书和私钥;
- --trusted-ca-file:签名 client 证书的 CA 证书,用于验证 client 证书;
- --peer-cert-file、--peer-key-file:etcd 与 peer 通信使用的证书和私钥;
- --peer-trusted-ca-file:签名 peer 证书的 CA 证书,用于验证 peer 证书;
2.3 启动 etcd 服务并检查启动状态
启动etcd服务
systemctl etcd.service start
检查etcd启动状态
systemctl status etcd|grep Active
确保状态为 active (running),否则查看日志,确认原因:
sudo journalctl -u etcd
验证etcd服务状态及查看当前leader
验证etcd服务状态
ETCDCTL_API=3 etcdctl \
--endpoints=${ETCD_ENPOIDTS} \
--cacert=/etc/k8s/ssl/ca.pem \
--cert=/etc/k8s/ssl/etcd.pem \
--key=/etc/k8s/ssl/etcd-key.pem endpoint health
当输出值为以下状态即为正常:
https://10.10.10.22:2379 is healthy: successfully committed proposal: took = 3.693346ms
https://10.10.10.23:2379 is healthy: successfully committed proposal: took = 2.80687ms
https://10.10.10.24:2379 is healthy: successfully committed proposal: took = 4.493037ms
查看当前leader
ETCDCTL_API=3 etcdctl \
--endpoints=${ETCD_ENPOIDTS} \
--cacert=/etc/k8s/ssl/ca.pem \
--cert=/etc/k8s/ssl/etcd.pem \
--key=/etc/k8s/ssl/etcd-key.pem endpoint status
输出:
https://10.10.10.22:2379, 641670f7677ceff, 3.3.13, 2.9 MB, false, 2, 6198986
https://10.10.10.23:2379, ce37122364c277fa, 3.3.13, 2.9 MB, true, 2, 6198986
https://10.10.10.24:2379, 4d1003897626ccde, 3.3.13, 2.9 MB, false, 2, 6198986
到这里,整个etcd集群就部署完成,关于更多的etcd操作可以阅读etcd官方文档,部署好etcd数据库,开部署kubernetes master节点相关组件,请参考:kubernetes集群安装指南:apiserver节点组件部署,关于etcd脚本请从此处获取;