二进制包20分钟快速部署 Kubernetes v1.14.0 集群
一 环境
| 操作系统 |
Docker版本 |
Kubernetes版本 |
Etcd版本 |
Flannel版本 |
| CentOS Linux release 7.6.1810 |
Docker version 18.09.4 |
v1.14.0 |
Version: 3.3.12 |
v0.11.0 |
二 架构
| 主机名 |
IP |
角色 |
部署应用 |
| gysl-master |
10.1.1.60 |
Msater |
Docker/Kube-apiserver/kube-scheduler/kube-controller-manager/etcd |
| gysl-node1 |
10.1.1.61 |
Node |
Docker/Kubelet/kube-proxy/flanneld/etcd |
| gysl-node2 |
10.1.1.62 |
Node |
Docker/Kubelet/kube-proxy/flanneld/etcd |
| gysl-node3 |
10.1.1.63 |
Node |
Docker/Kubelet/kube-proxy/flanneld/etcd |
三 安装过程
通过几个小时的努力，完成本次部署脚本的编写，安装脚本支持任意多个节点，主要通过三个脚本实现本次安装。
3.1 初始化脚本
#!/bin/bash
declare -A HostIP EtcdIP
HostIP=( [gysl-master]='10.1.1.60' [gysl-node1]='10.1.1.61' [gysl-node2]='10.1.1.62' [gysl-node3]='10.1.1.63' )
EtcdIP=( [etcd-master]='10.1.1.60' [etcd-01]='10.1.1.61' [etcd-02]='10.1.1.62' [etcd-03]='10.1.1.63' )
BinaryDir='/usr/local/bin'
KubeConf='/etc/kubernetes/conf.d'
KubeCA='/etc/kubernetes/ca.d'
EtcdConf='/etc/etcd/conf.d'
EtcdCA='/etc/etcd/ca.d'
FlanneldConf='/etc/flanneld'
mkdir -p {${KubeConf},${KubeCA},${EtcdConf},${EtcdCA},${FlanneldConf}}
for hostname in ${!HostIP[@]}
do
cat>>/etc/hosts<<EOF
${HostIP[${hostname}]} ${hostname}
EOF
done
# Install the Docker engine. This needs to be executed on every machine.
curl http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker-ce.repo>&/dev/null
if [ $? -eq 0 ] ;
then
yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine>&/dev/null
yum list docker-ce --showduplicates|grep "^doc"|sort -r
yum -y install docker-ce-18.09.3-3.el7
rm -f /etc/yum.repos.d/docker-ce.repo
systemctl enable docker --now && systemctl status docker
else
echo "Install failed! Please try again! ";
exit 110
fi
# Modify related kernel parameters.
cat>/etc/sysctl.d/docker.conf<<EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/docker.conf>&/dev/null
# Turn off and disable the firewalld.
systemctl stop firewalld
systemctl disable firewalld
# Disable the SELinux.
sed -i.bak 's/=enforcing/=disabled/' /etc/selinux/config
# Disable the swap.
sed -i.bak 's/^.*swap/#&/g' /etc/fstab
# Install EPEL/vim/git.
yum -y install epel-release vim git tree
yum repolist
# Alias vim.
cat>/etc/profile.d/vim.sh<<EOF
alias vi='vim'
EOF
source /etc/profile.d/vim.sh
echo "set nu">>/etc/vimrc
# Reboot the machine.
reboot
需要每个节点都执行。
3.2 安装脚本
安装脚本较长，此处省略，日志以供参考，拓展思路。此脚本在Master节点执行即可，安装过程无需连接外网，安装日志如下：
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:rJdnEzx5GyWX9YCxq77ZMc+FCabCqA+3FwmS7LnF9qo Kubernetes
The key's randomart image is:
+---[RSA 1024]----+
| .o. .|
| .. +.|
| . . o + .|
| + .. . . = |
| . + .S.= * |
| o ++o. B + o |
| .++.=.* + o .|
| .+ oo= + = . |
| Eo+o +.. o |
+----[SHA256]-----+
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '10.1.1.62 (10.1.1.62)' can't be established.
ECDSA key fingerprint is SHA256:B4e7Gq9wcgr5N6ys8U72NEhNWxIFrvng5eI7GAXLf6s.
ECDSA key fingerprint is MD5:ea:33:04:40:f8:31:a2:d0:91:c4:b4:37:48:fa:51:d6.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.1.1.62's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@10.1.1.62'"
and check to make sure that only the key(s) you wanted were added.
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '10.1.1.63 (10.1.1.63)' can't be established.
ECDSA key fingerprint is SHA256:B4e7Gq9wcgr5N6ys8U72NEhNWxIFrvng5eI7GAXLf6s.
ECDSA key fingerprint is MD5:ea:33:04:40:f8:31:a2:d0:91:c4:b4:37:48:fa:51:d6.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.1.1.63's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@10.1.1.63'"
and check to make sure that only the key(s) you wanted were added.
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '10.1.1.61 (10.1.1.61)' can't be established.
ECDSA key fingerprint is SHA256:B4e7Gq9wcgr5N6ys8U72NEhNWxIFrvng5eI7GAXLf6s.
ECDSA key fingerprint is MD5:ea:33:04:40:f8:31:a2:d0:91:c4:b4:37:48:fa:51:d6.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.1.1.61's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@10.1.1.61'"
and check to make sure that only the key(s) you wanted were added.
etcd-v3.3.12-linux-amd64/
etcd-v3.3.12-linux-amd64/README.md
etcd-v3.3.12-linux-amd64/Documentation/
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/interacting_v3.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/api_concurrency_reference_v3.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/limit.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/local_cluster.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/api_grpc_gateway.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/grpc_naming.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/rpc.swagger.json
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/v3lock.swagger.json
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/v3election.swagger.json
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/experimental_apis.md
etcd-v3.3.12-linux-amd64/Documentation/dev-guide/api_reference_v3.md
etcd-v3.3.12-linux-amd64/Documentation/integrations.md
etcd-v3.3.12-linux-amd64/Documentation/README.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-1-0-alpha-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-storage-memory-benchmark.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/README.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-2-0-rc-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-2-0-rc-memory-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-3-watch-memory-benchmark.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-2-0-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-3-demo-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/rfc/
etcd-v3.3.12-linux-amd64/Documentation/rfc/v3api.md
etcd-v3.3.12-linux-amd64/Documentation/docs.md
etcd-v3.3.12-linux-amd64/Documentation/production-users.md
etcd-v3.3.12-linux-amd64/Documentation/metrics.md
etcd-v3.3.12-linux-amd64/Documentation/v2/
etcd-v3.3.12-linux-amd64/Documentation/v2/authentication.md
etcd-v3.3.12-linux-amd64/Documentation/v2/proxy.md
etcd-v3.3.12-linux-amd64/Documentation/v2/glossary.md
etcd-v3.3.12-linux-amd64/Documentation/v2/docker_guide.md
etcd-v3.3.12-linux-amd64/Documentation/v2/configuration.md
etcd-v3.3.12-linux-amd64/Documentation/v2/README.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-1-0-alpha-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-storage-memory-benchmark.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/README.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-2-0-rc-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-2-0-rc-memory-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-3-watch-memory-benchmark.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-2-0-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-3-demo-benchmarks.md
etcd-v3.3.12-linux-amd64/Documentation/v2/rfc/
etcd-v3.3.12-linux-amd64/Documentation/v2/rfc/v3api.md
etcd-v3.3.12-linux-amd64/Documentation/v2/libraries-and-tools.md
etcd-v3.3.12-linux-amd64/Documentation/v2/discovery_protocol.md
etcd-v3.3.12-linux-amd64/Documentation/v2/runtime-configuration.md
etcd-v3.3.12-linux-amd64/Documentation/v2/upgrade_2_3.md
etcd-v3.3.12-linux-amd64/Documentation/v2/auth_api.md
etcd-v3.3.12-linux-amd64/Documentation/v2/errorcode.md
etcd-v3.3.12-linux-amd64/Documentation/v2/admin_guide.md
etcd-v3.3.12-linux-amd64/Documentation/v2/upgrade_2_2.md
etcd-v3.3.12-linux-amd64/Documentation/v2/upgrade_2_1.md
etcd-v3.3.12-linux-amd64/Documentation/v2/clustering.md
etcd-v3.3.12-linux-amd64/Documentation/v2/other_apis.md
etcd-v3.3.12-linux-amd64/Documentation/v2/production-users.md
etcd-v3.3.12-linux-amd64/Documentation/v2/metrics.md
etcd-v3.3.12-linux-amd64/Documentation/v2/runtime-reconf-design.md
etcd-v3.3.12-linux-amd64/Documentation/v2/etcd_alert.rules.yml
etcd-v3.3.12-linux-amd64/Documentation/v2/security.md
etcd-v3.3.12-linux-amd64/Documentation/v2/branch_management.md
etcd-v3.3.12-linux-amd64/Documentation/v2/internal-protocol-versioning.md
etcd-v3.3.12-linux-amd64/Documentation/v2/members_api.md
etcd-v3.3.12-linux-amd64/Documentation/v2/platforms/
etcd-v3.3.12-linux-amd64/Documentation/v2/platforms/freebsd.md
etcd-v3.3.12-linux-amd64/Documentation/v2/faq.md
etcd-v3.3.12-linux-amd64/Documentation/v2/backward_compatibility.md
etcd-v3.3.12-linux-amd64/Documentation/v2/04_to_2_snapshot_migration.md
etcd-v3.3.12-linux-amd64/Documentation/v2/etcd_alert.rules
etcd-v3.3.12-linux-amd64/Documentation/v2/api.md
etcd-v3.3.12-linux-amd64/Documentation/v2/api_v3.md
etcd-v3.3.12-linux-amd64/Documentation/v2/reporting_bugs.md
etcd-v3.3.12-linux-amd64/Documentation/v2/tuning.md
etcd-v3.3.12-linux-amd64/Documentation/v2/dev/
etcd-v3.3.12-linux-amd64/Documentation/v2/dev/release.md
etcd-v3.3.12-linux-amd64/Documentation/branch_management.md
etcd-v3.3.12-linux-amd64/Documentation/platforms/
etcd-v3.3.12-linux-amd64/Documentation/platforms/container-linux-systemd.md
etcd-v3.3.12-linux-amd64/Documentation/platforms/freebsd.md
etcd-v3.3.12-linux-amd64/Documentation/platforms/aws.md
etcd-v3.3.12-linux-amd64/Documentation/faq.md
etcd-v3.3.12-linux-amd64/Documentation/dl_build.md
etcd-v3.3.12-linux-amd64/Documentation/reporting_bugs.md
etcd-v3.3.12-linux-amd64/Documentation/tuning.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_4.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrading-etcd.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_2.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_1.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_0.md
etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_3.md
etcd-v3.3.12-linux-amd64/Documentation/dev-internal/
etcd-v3.3.12-linux-amd64/Documentation/dev-internal/logging.md
etcd-v3.3.12-linux-amd64/Documentation/dev-internal/discovery_protocol.md
etcd-v3.3.12-linux-amd64/Documentation/dev-internal/release.md
etcd-v3.3.12-linux-amd64/Documentation/learning/
etcd-v3.3.12-linux-amd64/Documentation/learning/auth_design.md
etcd-v3.3.12-linux-amd64/Documentation/learning/glossary.md
etcd-v3.3.12-linux-amd64/Documentation/learning/data_model.md
etcd-v3.3.12-linux-amd64/Documentation/learning/api_guarantees.md
etcd-v3.3.12-linux-amd64/Documentation/learning/why.md
etcd-v3.3.12-linux-amd64/Documentation/learning/api.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/
etcd-v3.3.12-linux-amd64/Documentation/op-guide/authentication.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/versioning.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/hardware.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/grafana.json
etcd-v3.3.12-linux-amd64/Documentation/op-guide/monitoring.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/configuration.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/v2-migration.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/maintenance.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/runtime-configuration.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/recovery.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/clustering.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/etcd3_alert.rules
etcd-v3.3.12-linux-amd64/Documentation/op-guide/runtime-reconf-design.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/etcd3_alert.rules.yml
etcd-v3.3.12-linux-amd64/Documentation/op-guide/security.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/performance.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/etcd-sample-grafana.png
etcd-v3.3.12-linux-amd64/Documentation/op-guide/grpc_proxy.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/gateway.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/container.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/supported-platform.md
etcd-v3.3.12-linux-amd64/Documentation/op-guide/failures.md
etcd-v3.3.12-linux-amd64/Documentation/demo.md
etcd-v3.3.12-linux-amd64/README-etcdctl.md
etcd-v3.3.12-linux-amd64/etcdctl
etcd-v3.3.12-linux-amd64/READMEv2-etcdctl.md
etcd-v3.3.12-linux-amd64/etcd
flanneld
mk-docker-opts.sh
README.md
kubernetes/
kubernetes/server/
kubernetes/server/bin/
kubernetes/server/bin/kube-controller-manager.docker_tag
kubernetes/server/bin/kube-apiserver.tar
kubernetes/server/bin/kube-proxy
kubernetes/server/bin/cloud-controller-manager.docker_tag
kubernetes/server/bin/mounter
kubernetes/server/bin/kube-proxy.docker_tag
kubernetes/server/bin/kubelet
kubernetes/server/bin/kube-scheduler.docker_tag
kubernetes/server/bin/kube-controller-manager.tar
kubernetes/server/bin/kubectl
kubernetes/server/bin/kube-apiserver
kubernetes/server/bin/kube-apiserver.docker_tag
kubernetes/server/bin/kube-controller-manager
kubernetes/server/bin/kube-proxy.tar
kubernetes/server/bin/cloud-controller-manager
kubernetes/server/bin/kube-scheduler.tar
kubernetes/server/bin/apiextensions-apiserver
kubernetes/server/bin/kubeadm
kubernetes/server/bin/hyperkube
kubernetes/server/bin/kube-scheduler
kubernetes/server/bin/cloud-controller-manager.tar
kubernetes/addons/
kubernetes/kubernetes-src.tar.gz
kubernetes/LICENSES
2019/03/31 20:34:23 [INFO] generating a new CA key and certificate from CSR
2019/03/31 20:34:23 [INFO] generate received request
2019/03/31 20:34:23 [INFO] received CSR
2019/03/31 20:34:23 [INFO] generating key: rsa-2048
2019/03/31 20:34:23 [INFO] encoded CSR
2019/03/31 20:34:23 [INFO] signed certificate with serial number 316253512009054883826466120107550244311105093255
2019/03/31 20:34:23 [INFO] generate received request
2019/03/31 20:34:23 [INFO] received CSR
2019/03/31 20:34:23 [INFO] generating key: rsa-2048
2019/03/31 20:34:23 [INFO] encoded CSR
2019/03/31 20:34:23 [INFO] signed certificate with serial number 288189004496636237074496723049170901716100041831
2019/03/31 20:34:23 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
/etc/etcd/ca.d
├── ca-config.json
├── ca.csr
├── ca-csr.json
├── ca-key.pem
├── ca.pem
├── server.csr
├── server-csr.json
├── server-key.pem
└── server.pem
0 directories, 9 files
ca-key.pem 100% 1675 27.1KB/s 00:00
ca.pem 100% 1265 2.2MB/s 00:00
server-key.pem 100% 1679 639.6KB/s 00:00
server.pem 100% 1346 1.9MB/s 00:00
etcd 100% 18MB 13.1MB/s 00:01
etcdctl 100% 15MB 41.5MB/s 00:00
etcd.service 100% 994 1.0MB/s 00:00
etcd.conf 100% 520 527.8KB/s 00:00
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
Job for etcd.service failed because a timeout was exceeded. See "systemctl status etcd.service" and "journalctl -xe" for details.
ca-key.pem 100% 1675 92.3KB/s 00:00
ca.pem 100% 1265 1.7MB/s 00:00
server-key.pem 100% 1679 328.8KB/s 00:00
server.pem 100% 1346 1.6MB/s 00:00
etcd 100% 18MB 40.7MB/s 00:00
etcdctl 100% 15MB 46.0MB/s 00:00
etcd.service 100% 994 1.0MB/s 00:00
etcd.conf 100% 520 838.6KB/s 00:00
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
Job for etcd.service failed because a timeout was exceeded. See "systemctl status etcd.service" and "journalctl -xe" for details.
ca-key.pem 100% 1675 106.9KB/s 00:00
ca.pem 100% 1265 1.0MB/s 00:00
server-key.pem 100% 1679 1.3MB/s 00:00
server.pem 100% 1346 1.5MB/s 00:00
etcd 100% 18MB 31.4MB/s 00:00
etcdctl 100% 15MB 37.7MB/s 00:00
etcd.service 100% 994 916.5KB/s 00:00
etcd.conf 100% 520 487.5KB/s 00:00
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:37:32 CST; 28ms ago
Main PID: 7373 (etcd)
Tasks: 7
Memory: 9.1M
CGroup: /system.slice/etcd.service
└─7373 /usr/local/bin/etcd --name=etcd-01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://10.1.1.61:2380 --listen-client-urls=https://10.1.1.61:2379,http://127.0.0.1:2379 --advertise-client-urls=https://10.1.1.61:2379 --initial-advertise-peer-urls=https://10.1.1.61:2380 --initial-cluster=etcd-master=https://10.1.1.60:2380,etcd-01=https://10.1.1.61:2380,etcd-02=https://10.1.1.62:2380,etcd-03=https://10.1.1.63:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/etc/etcd/ca.d/server.pem --key-file=/etc/etcd/ca.d/server-key.pem --peer-cert-file=/etc/etcd/ca.d/server.pem --peer-key-file=/etc/etcd/ca.d/server-key.pem --trusted-ca-file=/etc/etcd/ca.d/ca.pem --peer-trusted-ca-file=/etc/etcd/ca.d/ca.pem
3月 31 20:37:32 gysl-node1 etcd[7373]: 1c3555118a39401e initialzed peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)
3月 31 20:37:32 gysl-node1 etcd[7373]: raft.node: 1c3555118a39401e elected leader 63ac3c747757aa28 at term 138
3月 31 20:37:32 gysl-node1 etcd[7373]: published {Name:etcd-01 ClientURLs:[https://10.1.1.61:2379]} to cluster 575c8b9e68fd927d
3月 31 20:37:32 gysl-node1 etcd[7373]: ready to serve client requests
3月 31 20:37:32 gysl-node1 etcd[7373]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
3月 31 20:37:32 gysl-node1 etcd[7373]: ready to serve client requests
3月 31 20:37:32 gysl-node1 systemd[1]: Started Etcd Server.
3月 31 20:37:32 gysl-node1 etcd[7373]: serving client requests on 10.1.1.61:2379
3月 31 20:37:32 gysl-node1 etcd[7373]: set the initial cluster version to 3.0
3月 31 20:37:32 gysl-node1 etcd[7373]: enabled capabilities for version 3.0
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:37:34 CST; 124ms ago
Main PID: 7551 (etcd)
Tasks: 7
Memory: 10.3M
CGroup: /system.slice/etcd.service
└─7551 /usr/local/bin/etcd --name=etcd-master --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://10.1.1.60:2380 --listen-client-urls=https://10.1.1.60:2379,http://127.0.0.1:2379 --advertise-client-urls=https://10.1.1.60:2379 --initial-advertise-peer-urls=https://10.1.1.60:2380 --initial-cluster=etcd-master=https://10.1.1.60:2380,etcd-01=https://10.1.1.61:2380,etcd-02=https://10.1.1.62:2380,etcd-03=https://10.1.1.63:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/etc/etcd/ca.d/server.pem --key-file=/etc/etcd/ca.d/server-key.pem --peer-cert-file=/etc/etcd/ca.d/server.pem --peer-key-file=/etc/etcd/ca.d/server-key.pem --trusted-ca-file=/etc/etcd/ca.d/ca.pem --peer-trusted-ca-file=/etc/etcd/ca.d/ca.pem
3月 31 20:37:34 gysl-master etcd[7551]: established a TCP streaming connection with peer 63ac3c747757aa28 (stream Message reader)
3月 31 20:37:34 gysl-master etcd[7551]: established a TCP streaming connection with peer 1c3555118a39401e (stream Message reader)
3月 31 20:37:34 gysl-master etcd[7551]: published {Name:etcd-master ClientURLs:[https://10.1.1.60:2379]} to cluster 575c8b9e68fd927d
3月 31 20:37:34 gysl-master etcd[7551]: ready to serve client requests
3月 31 20:37:34 gysl-master etcd[7551]: serving client requests on 10.1.1.60:2379
3月 31 20:37:34 gysl-master etcd[7551]: ready to serve client requests
3月 31 20:37:34 gysl-master etcd[7551]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
3月 31 20:37:34 gysl-master etcd[7551]: 78df1ab24a6f1302 initialzed peer connection; fast-forwarding 8 ticks (election ticks 10) with 3 active peer(s)
3月 31 20:37:34 gysl-master systemd[1]: Started Etcd Server.
3月 31 20:37:34 gysl-master etcd[7551]: established a TCP streaming connection with peer 76bcb3b85e42210d (stream Message reader)
Please wait a moment!
2019/03/31 20:38:34 [INFO] generating a new CA key and certificate from CSR
2019/03/31 20:38:34 [INFO] generate received request
2019/03/31 20:38:34 [INFO] received CSR
2019/03/31 20:38:34 [INFO] generating key: rsa-2048
2019/03/31 20:38:34 [INFO] encoded CSR
2019/03/31 20:38:34 [INFO] signed certificate with serial number 284879897535931954074635242912207100624264127544
2019/03/31 20:38:34 [INFO] generate received request
2019/03/31 20:38:34 [INFO] received CSR
2019/03/31 20:38:34 [INFO] generating key: rsa-2048
2019/03/31 20:38:34 [INFO] encoded CSR
2019/03/31 20:38:34 [INFO] signed certificate with serial number 163588537762519336822862885460408698694735647771
2019/03/31 20:38:34 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
2019/03/31 20:38:34 [INFO] generate received request
2019/03/31 20:38:34 [INFO] received CSR
2019/03/31 20:38:34 [INFO] generating key: rsa-2048
2019/03/31 20:38:35 [INFO] encoded CSR
2019/03/31 20:38:35 [INFO] signed certificate with serial number 269430846139878968754015022650791204259891937310
2019/03/31 20:38:35 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
/etc/kubernetes/ca.d
├── ca-config.json
├── ca.csr
├── ca-csr.json
├── ca-key.pem
├── ca.pem
├── kube-proxy.csr
├── kube-proxy-csr.json
├── kube-proxy-key.pem
├── kube-proxy.pem
├── server.csr
├── server-csr.json
├── server-key.pem
└── server.pem
0 directories, 13 files
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
● kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:38:35 CST; 41ms ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 7628 (kube-apiserver)
Tasks: 1
Memory: 14.0M
CGroup: /system.slice/kube-apiserver.service
└─7628 /usr/local/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://10.1.1.60:2379,https://10.1.1.61:2379,https://10.1.1.62:2379,https://10.1.1.63:2379 --bind-address=10.1.1.60 --secure-port=6443 --advertise-address=10.1.1.60 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --enable-bootstrap-token-auth --token-auth-file=/etc/kubernetes/conf.d/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/etc/kubernetes/ca.d/server.pem --tls-private-key-file=/etc/kubernetes/ca.d/server-key.pem --client-ca-file=/etc/kubernetes/ca.d/ca.pem --service-account-key-file=/etc/kubernetes/ca.d/ca-key.pem --etcd-cafile=/etc/etcd/ca.d/ca.pem --etcd-certfile=/etc/etcd/ca.d/server.pem --etcd-keyfile=/etc/etcd/ca.d/server-key.pem
3月 31 20:38:35 gysl-master systemd[1]: Started Kubernetes API Server.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
● kube-scheduler.service - Kubernetes Scheduler
Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:38:36 CST; 20s ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 7673 (kube-scheduler)
Tasks: 7
Memory: 47.5M
CGroup: /system.slice/kube-scheduler.service
└─7673 /usr/local/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.299502 7673 shared_informer.go:123] caches populated
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.399931 7673 shared_informer.go:123] caches populated
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.500642 7673 shared_informer.go:123] caches populated
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.601146 7673 shared_informer.go:123] caches populated
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.604604 7673 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.705500 7673 shared_informer.go:123] caches populated
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.705529 7673 controller_utils.go:1034] Caches are synced for scheduler controller
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.705631 7673 leaderelection.go:217] attempting to acquire leader lease kube-system/kube-scheduler...
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.737674 7673 leaderelection.go:227] successfully acquired lease kube-system/kube-scheduler
3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.838862 7673 shared_informer.go:123] caches populated
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
● kube-controller-manager.service - Kubernetes Controller Manager
Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:38:56 CST; 20s ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 7725 (kube-controller)
Tasks: 6
Memory: 132.3M
CGroup: /system.slice/kube-controller-manager.service
└─7725 /usr/local/bin/kube-controller-manager --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect=true --address=127.0.0.1 --service-cluster-ip-range=10.0.0.0/24 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ca.d/ca.pem --cluster-signing-key-file=/etc/kubernetes/ca.d/ca-key.pem --root-ca-file=/etc/kubernetes/ca.d/ca.pem --service-account-private-key-file=/etc/kubernetes/ca.d/ca-key.pem
3月 31 20:38:59 gysl-master kube-controller-manager[7725]: I0331 20:38:59.915581 7725 request.go:530] Throttling request took 1.356935667s, request: GET:http://127.0.0.1:8080/apis/scheduling.k8s.io/v1?timeout=32s
3月 31 20:38:59 gysl-master kube-controller-manager[7725]: I0331 20:38:59.965276 7725 request.go:530] Throttling request took 1.406608026s, request: GET:http://127.0.0.1:8080/apis/scheduling.k8s.io/v1beta1?timeout=32s
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.015978 7725 request.go:530] Throttling request took 1.457255375s, request: GET:http://127.0.0.1:8080/apis/coordination.k8s.io/v1?timeout=32s
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.065993 7725 request.go:530] Throttling request took 1.507246887s, request: GET:http://127.0.0.1:8080/apis/coordination.k8s.io/v1beta1?timeout=32s
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.067050 7725 resource_quota_controller.go:427] syncing resource quota controller with updated resources from discovery: map[/v1, Resource=configmaps:{} /v1, Resource=endpoints:{} /v1, Resource=events:{} /v1, Resource=limitranges:{} /v1, Resource=persistentvolumeclaims:{} /v1, Resource=pods:{} /v1, Resource=podtemplates:{} /v1, Resource=replicationcontrollers:{} /v1, Resource=resourcequotas:{} /v1, Resource=secrets:{} /v1, Resource=serviceaccounts:{} /v1, Resource=services:{} apps/v1, Resource=controllerrevisions:{} apps/v1, Resource=daemonsets:{} apps/v1, Resource=deployments:{} apps/v1, Resource=replicasets:{} apps/v1, Resource=statefulsets:{} autoscaling/v1, Resource=horizontalpodautoscalers:{} batch/v1, Resource=jobs:{} batch/v1beta1, Resource=cronjobs:{} coordination.k8s.io/v1, Resource=leases:{} events.k8s.io/v1beta1, Resource=events:{} extensions/v1beta1, Resource=daemonsets:{} extensions/v1beta1, Resource=deployments:{} extensions/v1beta1, Resource=ingresses:{} extensions/v1beta1, Resource=networkpolicies:{} extensions/v1beta1, Resource=replicasets:{} networking.k8s.io/v1, Resource=networkpolicies:{} networking.k8s.io/v1beta1, Resource=ingresses:{} policy/v1beta1, Resource=poddisruptionbudgets:{} rbac.authorization.k8s.io/v1, Resource=rolebindings:{} rbac.authorization.k8s.io/v1, Resource=roles:{}]
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.067168 7725 resource_quota_monitor.go:180] QuotaMonitor unable to use a shared informer for resource "extensions/v1beta1, Resource=networkpolicies": no informer found for extensions/v1beta1, Resource=networkpolicies
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.067189 7725 resource_quota_monitor.go:243] quota synced monitors; added 0, kept 30, removed 0
3月 31 20:39:00 gysl-master kube-controller-manager[7725]: E0331 20:39:00.067197 7725 resource_quota_controller.go:437] failed to sync resource monitors: couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies"
3月 31 20:39:13 gysl-master kube-controller-manager[7725]: I0331 20:39:13.677245 7725 reflector.go:235] k8s.io/client-go/informers/factory.go:133: forcing resync
3月 31 20:39:14 gysl-master kube-controller-manager[7725]: I0331 20:39:14.215322 7725 pv_controller_base.go:407] resyncing PV controller
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
Switched to context "default".
member 1c3555118a39401e is healthy: got healthy result from https://10.1.1.61:2379
member 63ac3c747757aa28 is healthy: got healthy result from https://10.1.1.63:2379
member 76bcb3b85e42210d is healthy: got healthy result from https://10.1.1.62:2379
member 78df1ab24a6f1302 is healthy: got healthy result from https://10.1.1.60:2379
cluster is healthy
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}
kubelet 100% 122MB 26.1MB/s 00:04
kube-proxy 100% 35MB 21.9MB/s 00:01
flanneld 100% 34MB 20.5MB/s 00:01
mk-docker-opts.sh 100% 2139 916.8KB/s 00:00
flanneld.conf 100% 247 55.8KB/s 00:00
flanneld.service 100% 389 82.7KB/s 00:00
kubelet.yaml 100% 263 319.4KB/s 00:00
kubelet.conf 100% 365 326.0KB/s 00:00
kube-proxy.conf 100% 158 184.0KB/s 00:00
kubelet.service 100% 267 234.2KB/s 00:00
kube-proxy.service 100% 234 130.5KB/s 00:00
bootstrap.kubeconfig 100% 2163 1.5MB/s 00:00
kube-proxy.kubeconfig 100% 6265 4.4MB/s 00:00
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
● flanneld.service - Flanneld overlay address etcd agent
Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:27 CST; 430ms ago
Process: 7536 ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env (code=exited, status=0/SUCCESS)
Main PID: 7508 (flanneld)
Tasks: 7
Memory: 6.7M
CGroup: /system.slice/flanneld.service
└─7508 /usr/local/bin/flanneld --ip-masq
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.720837 7508 iptables.go:167] Deleting iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.721919 7508 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.24.0/24 -j RETURN
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.722994 7508 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.724549 7508 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.730116 7508 iptables.go:155] Adding iptables rule: -d 172.17.0.0/16 -j ACCEPT
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.737143 7508 main.go:429] Waiting for 22h69m59.914613166s to renew lease
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.737262 7508 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.744276 7508 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.24.0/24 -j RETURN
3月 31 20:39:27 gysl-node2 systemd[1]: Started Flanneld overlay address etcd agent.
3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.766442 7508 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:28 CST; 10ms ago
Docs: https://docs.docker.com
Main PID: 7579 (dockerd)
Tasks: 8
Memory: 32.1M
CGroup: /system.slice/docker.service
└─7579 /usr/bin/dockerd --bip=172.17.24.1/24 --ip-masq=false --mtu=1450 -H fd:// --containerd=/run/containerd/containerd.sock
3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.843896719+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.843917442+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154920, CONNECTING" module=grpc
3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.843973658+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154920, READY" module=grpc
3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.844332744+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.848229255+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.848828116+08:00" level=info msg="Loading containers: start."
3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.081132437+08:00" level=info msg="Loading containers: done."
3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.167227705+08:00" level=info msg="Docker daemon" commit=774a1f4 graphdriver(s)=overlay2 version=18.09.3
3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.167281411+08:00" level=info msg="Daemon has completed initialization"
3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.175538228+08:00" level=info msg="API listen on /var/run/docker.sock"
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:28 CST; 61ms ago
Main PID: 7727 (kubelet)
Tasks: 1
Memory: 2.1M
CGroup: /system.slice/kubelet.service
└─7727 /usr/local/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.1.1.62 --kubeconfig=/etc/kubernetes/conf.d/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/conf.d/bootstrap.kubeconfig --config=/etc/kubernetes/conf.d/kubelet.yaml --cert-dir=/etc/kubernetes/ca.d --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
3月 31 20:39:28 gysl-node2 systemd[1]: Started Kubernetes Kubelet.
● kube-proxy.service - Kubernetes Proxy
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:28 CST; 7ms ago
Main PID: 7728 (systemd)
Tasks: 0
Memory: 0B
CGroup: /system.slice/kube-proxy.service
└─7728 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
kubelet 100% 122MB 27.7MB/s 00:04
kube-proxy 100% 35MB 13.8MB/s 00:02
flanneld 100% 34MB 33.6MB/s 00:01
mk-docker-opts.sh 100% 2139 1.1MB/s 00:00
flanneld.conf 100% 247 225.5KB/s 00:00
flanneld.service 100% 389 357.8KB/s 00:00
kubelet.yaml 100% 263 193.7KB/s 00:00
kubelet.conf 100% 365 331.3KB/s 00:00
kube-proxy.conf 100% 158 130.4KB/s 00:00
kubelet.service 100% 267 295.5KB/s 00:00
kube-proxy.service 100% 234 198.3KB/s 00:00
bootstrap.kubeconfig 100% 2163 2.0MB/s 00:00
kube-proxy.kubeconfig 100% 6265 3.7MB/s 00:00
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
● flanneld.service - Flanneld overlay address etcd agent
Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:39 CST; 391ms ago
Process: 7534 ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env (code=exited, status=0/SUCCESS)
Main PID: 7502 (flanneld)
Tasks: 7
Memory: 9.3M
CGroup: /system.slice/flanneld.service
└─7502 /usr/local/bin/flanneld --ip-masq
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.088309 7502 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.088315 7502 iptables.go:167] Deleting iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.091984 7502 iptables.go:167] Deleting iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.095011 7502 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.100.0/24 -j RETURN
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.098419 7502 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.099751 7502 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.103532 7502 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.106520 7502 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.100.0/24 -j RETURN
3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.113480 7502 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
3月 31 20:39:39 gysl-node3 systemd[1]: Started Flanneld overlay address etcd agent.
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:39 CST; 10ms ago
Docs: https://docs.docker.com
Main PID: 7573 (dockerd)
Tasks: 8
Memory: 31.9M
CGroup: /system.slice/docker.service
└─7573 /usr/bin/dockerd --bip=172.17.100.1/24 --ip-masq=false --mtu=1450 -H fd:// --containerd=/run/containerd/containerd.sock
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.230510356+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.230556184+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154910, CONNECTING" module=grpc
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.230711652+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154910, READY" module=grpc
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.231101930+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.234478410+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.234950238+08:00" level=info msg="Loading containers: start."
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.406988224+08:00" level=info msg="Loading containers: done."
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.497837879+08:00" level=info msg="Docker daemon" commit=774a1f4 graphdriver(s)=overlay2 version=18.09.3
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.497901197+08:00" level=info msg="Daemon has completed initialization"
3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.502801194+08:00" level=info msg="API listen on /var/run/docker.sock"
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:39 CST; 58ms ago
Main PID: 7721 (kubelet)
Tasks: 1
Memory: 4.2M
CGroup: /system.slice/kubelet.service
└─7721 /usr/local/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.1.1.63 --kubeconfig=/etc/kubernetes/conf.d/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/conf.d/bootstrap.kubeconfig --config=/etc/kubernetes/conf.d/kubelet.yaml --cert-dir=/etc/kubernetes/ca.d --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
3月 31 20:39:39 gysl-node3 systemd[1]: Started Kubernetes Kubelet.
● kube-proxy.service - Kubernetes Proxy
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:39 CST; 21ms ago
Main PID: 7722 (systemd)
Tasks: 0
Memory: 0B
CGroup: /system.slice/kube-proxy.service
└─7722 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
3月 31 20:39:39 gysl-node3 systemd[1]: Started Kubernetes Proxy.
kubelet 100% 122MB 32.7MB/s 00:03
kube-proxy 100% 35MB 14.4MB/s 00:02
flanneld 100% 34MB 30.4MB/s 00:01
mk-docker-opts.sh 100% 2139 3.5MB/s 00:00
flanneld.conf 100% 247 227.9KB/s 00:00
flanneld.service 100% 389 359.0KB/s 00:00
kubelet.yaml 100% 263 197.9KB/s 00:00
kubelet.conf 100% 365 517.1KB/s 00:00
kube-proxy.conf 100% 158 244.5KB/s 00:00
kubelet.service 100% 267 379.4KB/s 00:00
kube-proxy.service 100% 234 324.5KB/s 00:00
bootstrap.kubeconfig 100% 2163 429.6KB/s 00:00
kube-proxy.kubeconfig 100% 6265 4.7MB/s 00:00
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
● flanneld.service - Flanneld overlay address etcd agent
Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:49 CST; 319ms ago
Process: 7580 ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env (code=exited, status=0/SUCCESS)
Main PID: 7550 (flanneld)
Tasks: 7
Memory: 6.7M
CGroup: /system.slice/flanneld.service
└─7550 /usr/local/bin/flanneld --ip-masq
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.414921 7550 vxlan_network.go:60] watching for new subnet leases
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.415303 7550 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.96.0/24 -j RETURN
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.416682 7550 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.418320 7550 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.438055 7550 iptables.go:155] Adding iptables rule: -d 172.17.0.0/16 -j ACCEPT
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.443066 7550 main.go:429] Waiting for 22h69m59.922013672s to renew lease
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.443213 7550 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
3月 31 20:39:49 gysl-node1 systemd[1]: Started Flanneld overlay address etcd agent.
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.459736 7550 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.96.0/24 -j RETURN
3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.469674 7550 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:49 CST; 9ms ago
Docs: https://docs.docker.com
Main PID: 7622 (dockerd)
Tasks: 8
Memory: 28.6M
CGroup: /system.slice/docker.service
└─7622 /usr/bin/dockerd --bip=172.17.96.1/24 --ip-masq=false --mtu=1450 -H fd:// --containerd=/run/containerd/containerd.sock
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549105373+08:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>}]" module=grpc
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549111902+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549148708+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154bb0, CONNECTING" module=grpc
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549210269+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154bb0, READY" module=grpc
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549578647+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.554893473+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.555866350+08:00" level=info msg="Loading containers: start."
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.695192119+08:00" level=info msg="Loading containers: done."
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.729225641+08:00" level=info msg="Docker daemon" commit=774a1f4 graphdriver(s)=overlay2 version=18.09.3
3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.729282016+08:00" level=info msg="Daemon has completed initialization"
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:50 CST; 50ms ago
Main PID: 7770 (kubelet)
Tasks: 1
Memory: 2.1M
CGroup: /system.slice/kubelet.service
└─7770 /usr/local/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.1.1.61 --kubeconfig=/etc/kubernetes/conf.d/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/conf.d/bootstrap.kubeconfig --config=/etc/kubernetes/conf.d/kubelet.yaml --cert-dir=/etc/kubernetes/ca.d --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
3月 31 20:39:50 gysl-node1 systemd[1]: Started Kubernetes Kubelet.
● kube-proxy.service - Kubernetes Proxy
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2019-03-31 20:39:50 CST; 20ms ago
Main PID: 7771 (systemd)
Tasks: 0
Memory: 0B
CGroup: /system.slice/kube-proxy.service
└─7771 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
3月 31 20:39:50 gysl-node1 systemd[1]: Started Kubernetes Proxy.
[root@gysl-master ~]# kubectl get cs,nodes
NAME STATUS MESSAGE ERROR
componentstatus/scheduler Healthy ok
componentstatus/controller-manager Healthy ok
componentstatus/etcd-0 Healthy {"health":"true"}
componentstatus/etcd-2 Healthy {"health":"true"}
componentstatus/etcd-1 Healthy {"health":"true"}
componentstatus/etcd-3 Healthy {"health":"true"}
NAME STATUS ROLES AGE VERSION
node/10.1.1.61 Ready node 4m23s v1.14.0
node/10.1.1.62 Ready node 4m22s v1.14.0
node/10.1.1.63 Ready node 4m22s v1.14.0
3.3 安装失败回滚脚本
#!/bin/bash
declare -A HostIP EtcdIP
HostIP=( [gysl-master]='10.1.1.60' [gysl-node1]='10.1.1.61' [gysl-node2]='10.1.1.62' [gysl-node3]='10.1.1.63' )
EtcdIP=( [etcd-master]='10.1.1.60' [etcd-01]='10.1.1.61' [etcd-02]='10.1.1.62' [etcd-03]='10.1.1.63' )
BinaryDir='/usr/local/bin'
KubeConf='/etc/kubernetes/conf.d'
KubeCA='/etc/kubernetes/ca.d'
EtcdConf='/etc/etcd/conf.d'
EtcdCA='/etc/etcd/ca.d'
FlanneldConf='/etc/flanneld'
for node_ip in ${HostIP[@]}
do
if [ "${node_ip}" == "${HostIP[gysl-master]}" ] ; then
ps -ef|grep -e kube -e etcd -e flanneld|grep -v grep|awk '{print $2}'|xargs kill
rm -rf {${KubeConf},${KubeCA},${EtcdConf},${EtcdCA},${FlanneldConf}}
rm -rf ${BinaryDir}/*
else
ssh root@${node_ip} "ps -ef|grep -e kube -e etcd -e flanneld|grep -v grep|awk '{print $2}'|xargs kill"
ssh root@${node_ip} "rm -rf {${KubeConf},${KubeCA},${EtcdConf},${EtcdCA},${FlanneldConf}}"
ssh root@${node_ip} "rm -rf ${BinaryDir}/* && reboot"
fi
done
reboot
四 总结
通过脚本实现自动化安装是一个良好的习惯，可以达到事半功倍的效果，以后工作中要注意培养这种习惯！