编程知识 cdmana.com

Ice river teaches you to successfully install k8s cluster (based on one master two slave mode)!!

In the high concurrency scenario, the author has developed it , Simple to offer 、 Stable 、 Extensible delayed message queuing framework , It has precise timing task and delay queue processing function . Since the open source for more than half a year , It has successfully provided precise timing scheduling scheme for more than ten small and medium-sized enterprises , It has withstood the test of production environment . In order to benefit more children's shoes , Now give the open source framework address :

https://github.com/sunshinelyz/mykit-delay

PS: Welcome to Star Source code , It's fine too pr Your blazing code .

Write it at the front

Research K8S It's been a while , The beginning of learning K8S when , Install according to the online tutorial K8S The environment is always wrong . therefore , I changed my learning strategy , Let's not set up the environment . First through the official website to learn K8S The overall structure of , Underlying principle , Chew it hard again K8S Source code . Don't ask me how to learn for ha , It's just that I feel personally , This learning can help me better understand the whole cloud native system . see , This time, , I summed up how to install successfully at one time K8S Cluster approach . Let's talk about how to build based on one master two slave mode K8S colony . Back , Let's go on how to build completely without holes K8S High availability cluster solution . The article and what it takes to build the environment yml The document has been included in :https://github.com/sunshinelyz/technology-binghe and https://gitee.com/binghe001/technology-binghe . If the document is of some help to you , Don't forget to give it Star Oh !

Cluster planning

IP

Host name

node

Operating system version

192.168.175.101

binghe101

Master

CentOS 8.0.1905

192.168.175.102

binghe102

Worker

CentOS 8.0.1905

192.168.175.103

binghe103

Worker

CentOS 8.0.1905

Basic configuration

On three servers /etc/hosts Add the following configuration items to the file .

192.168.175.101  binghe101
192.168.175.102  binghe102
192.168.175.103  binghe103

Check the system environment

Check the environment of the system on three servers respectively .

1. Check the server operating system version

cat /etc/redhat-release

install Docker and K8S The server operating system version of the cluster needs to be in CentOS 7 above .

2. Check the host name of the server

hostname

Be careful : The host name of the server in the cluster cannot be localhost.

3. Check the server's CPU Check the number

lscpu

Be careful : Server in the cluster CPU Audit number cannot be less than 2 individual .

4. Look at the server network

With binghe101(Master) Server, for example . Execute on server ip route show Command to view the default network card of the server , As shown below .

[root@binghe101 ~]# ip route show
default via 192.168.175.2 dev ens33 proto static metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.175.0/24 dev ens33 proto kernel scope link src 192.168.175.101 metric 100 

In the output above, there is a line marked with binghe101 The default network card used by the server .

default via 192.168.175.2 dev ens33 proto static metric 100 

You can see ,binghe101 The default network card used by the server is ens33.

Next , Use ip address Command to view the IP Address , As shown below .

[root@binghe101 ~]# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:68:06:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.175.101/24 brd 192.168.175.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::890f:5a92:4171:2a11/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

You can see ,binghe101 The default network card on the server IP The address is 192.168.175.101,K8S Will use this IP The address communicates with other nodes in the cluster . All in the cluster K8S What is used IP Addresses must be interoperable .

Docker install

Install on three servers Docker And configure alicloud image Accelerator .

1. install Docker

newly build auto_install_docker.sh Script files

vim auto_install_docker.sh

The contents of the document are as follows .

export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com
dnf install yum*
yum install -y yum-utils  device-mapper-persistent-data  lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
dnf install https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.13-3.1.el7.x86_64.rpm
yum install docker-ce docker-ce-cli -y
systemctl enable docker.service
systemctl start docker.service
docker version

Or designate Docker To install the version of , here auto_install_docker.sh The contents of the script file are as follows .

export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com
dnf install yum*
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce-19.03.8 docker-ce-cli-19.03.8 containerd.io
systemctl enable docker.service
systemctl start docker.service
docker version

Use the following command to assign auto_install_docker.sh File executable rights .

chmod a+x ./auto_install_docker.sh

Next , Direct operation auto_install_docker.sh Script file installation Docker that will do .

./auto_install_docker.sh

2. Configure alicloud image Accelerator

New script file aliyun_docker_images.sh.

vim aliyun_docker_images.sh

The contents of the document are as follows .

mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://zz3sblpi.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload
systemctl restart docker

by aliyun_docker_images.sh Script files give executable rights , As shown below .

chmod a+x ./aliyun_docker_images.sh

perform aliyun_docker_images.sh Script file configuration alicloud image Accelerator .

./aliyun_docker_images.sh

System settings

Set up the system on three servers respectively .

1. install nfs-utils

yum install -y nfs-utils
yum install -y wget

2. Turn off firewall

systemctl stop firewalld
systemctl disable firewalld

3. close SeLinux

setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

4. close swap

swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab

5. modify /etc/sysctl.conf

newly build sys_config.sh Script files .

vim sys_config.sh

sys_config.sh The contents of the script file are as follows ,

#  If you have configuration , The modified 
sed -i "s#^net.ipv4.ip_forward.*#net.ipv4.ip_forward=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-ip6tables.*#net.bridge.bridge-nf-call-ip6tables=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-iptables.*#net.bridge.bridge-nf-call-iptables=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.disable_ipv6.*#net.ipv6.conf.all.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.default.disable_ipv6.*#net.ipv6.conf.default.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.lo.disable_ipv6.*#net.ipv6.conf.lo.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.forwarding.*#net.ipv6.conf.all.forwarding=1#g"  /etc/sysctl.conf
#  There may be no , Additional 
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.lo.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.forwarding = 1"  >> /etc/sysctl.conf
#  Execute a command to apply 
sysctl -p

Execute the following order to endow sys_config.sh File executable rights .

chmod a+x ./sys_config.sh

perform sys_config.sh Script files .

./sys_config.sh

install K8S

Install on three servers K8S.

1. To configure K8S yum Source

newly build k8s_yum.sh Script files .

vim k8s_yum.sh

The contents of the document are as follows .

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

give k8s_yum.sh Executable permissions for script files .

chmod a+x ./k8s_yum.sh

perform k8s_yum.sh file .

./k8s_yum.sh

2. Uninstall the old version of K8S

yum remove -y kubelet kubeadm kubectl

3. install kubelet、kubeadm、kubectl

yum install -y kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2

4. modify docker Cgroup Driver by systemd

sed -i "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service

5. restart docker, And start the kubelet

systemctl daemon-reload
systemctl restart docker
systemctl enable kubelet && systemctl start kubelet

Integrated installation script

Sum up , Above installation Docker、 Set up the system , install K8S Can be unified into auto_install_docker_k8s.sh Script . The contents of the script are as follows .

# install Docker 19.03.8
export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com
dnf install yum*
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce-19.03.8 docker-ce-cli-19.03.8 containerd.io
systemctl enable docker.service
systemctl start docker.service
docker version

# Configure alicloud image Accelerator 
mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://zz3sblpi.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload
systemctl restart docker

# install nfs-utils
yum install -y nfs-utils
yum install -y wget

# Turn off firewall 
systemctl stop firewalld
systemctl disable firewalld

# close SeLinux
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

#  close  swap
swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab

# modify  /etc/sysctl.conf
#  If you have configuration , The modified 
sed -i "s#^net.ipv4.ip_forward.*#net.ipv4.ip_forward=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-ip6tables.*#net.bridge.bridge-nf-call-ip6tables=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-iptables.*#net.bridge.bridge-nf-call-iptables=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.disable_ipv6.*#net.ipv6.conf.all.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.default.disable_ipv6.*#net.ipv6.conf.default.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.lo.disable_ipv6.*#net.ipv6.conf.lo.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.forwarding.*#net.ipv6.conf.all.forwarding=1#g"  /etc/sysctl.conf
#  There may be no , Additional 
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.lo.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.forwarding = 1"  >> /etc/sysctl.conf
#  Execute a command to apply 
sysctl -p

#  To configure K8S Of yum Source 
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

#  Uninstall old version K8S
yum remove -y kubelet kubeadm kubectl

#  install kubelet、kubeadm、kubectl, What I'm installing here is 1.18.2 edition , You can also install 1.17.2 edition 
yum install -y kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2

#  modify docker Cgroup Driver by systemd
# #  take /usr/lib/systemd/system/docker.service This line in the file  ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
# #  It is amended as follows  ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd
#  If not modified , Adding  worker  The following errors may be encountered when a node 
# [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". 
# Please follow the guide at https://kubernetes.io/docs/setup/cri/
sed -i "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service

#  Set up  docker  Mirror image , Improve  docker  Image download speed and stability 
#  If you visit  https://hub.docker.io  The speed is very stable , You can also skip this step 
# curl -sSL https://kuboard.cn/install-script/set_mirror.sh | sh -s ${REGISTRY_MIRROR}

#  restart  docker, And start the  kubelet
systemctl daemon-reload
systemctl restart docker
systemctl enable kubelet && systemctl start kubelet

docker version

Be careful : I installed K8S The version is 1.18.2, Everyone is installing K8S when , You can also choose another version to install

give auto_install_docker_k8s.sh Script file executable permissions .

chmod a+x ./auto_install_docker_k8s.sh

perform auto_install_docker_k8s.sh Script files .

./auto_install_docker_k8s.sh

Be careful : It needs to be executed on each server auto_install_docker_k8s.sh Script files .

initialization Master node

Only in binghe101 Operations performed on the server .

1. initialization Master Node network environment

#  Only in  master  Node execution 
# export  The order is only in the current  shell  Effective in conversation , Open a new  shell  After window , If you want to continue the installation process , Please re execute  export  command 
export MASTER_IP=192.168.175.101
#  Replace  k8s.master  by   What you want  dnsName
export APISERVER_NAME=k8s.master
# Kubernetes  The network segment of the container group , After the installation of the network segment , from  kubernetes  establish , It doesn't exist in your physical network in advance 
export POD_SUBNET=172.18.0.1/16
echo "${MASTER_IP}    ${APISERVER_NAME}" >> /etc/hosts

2. initialization Master node

stay binghe101 Create... On the server init_master.sh Script files , The contents of the document are as follows .

#!/bin/bash
#  Terminate execution on script error 
set -e

if [ ${#POD_SUBNET} -eq 0 ] || [ ${#APISERVER_NAME} -eq 0 ]; then
  echo -e "\033[31;1m Make sure you have set the environment variable  POD_SUBNET  and  APISERVER_NAME \033[0m"
  echo  At present POD_SUBNET=$POD_SUBNET
  echo  At present APISERVER_NAME=$APISERVER_NAME
  exit 1
fi


#  See full configuration options  https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2
rm -f ./kubeadm-config.yaml
cat <<EOF > ./kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.18.2
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
controlPlaneEndpoint: "${APISERVER_NAME}:6443"
networking:
  serviceSubnet: "10.96.0.0/16"
  podSubnet: "${POD_SUBNET}"
  dnsDomain: "cluster.local"
EOF

# kubeadm init
#  According to the network speed of your server , You need to wait  3 - 10  minute 
kubeadm init --config=kubeadm-config.yaml --upload-certs

#  To configure  kubectl
rm -rf /root/.kube/
mkdir /root/.kube/
cp -i /etc/kubernetes/admin.conf /root/.kube/config

#  install  calico  The network plugin 
#  Reference documents  https://docs.projectcalico.org/v3.13/getting-started/kubernetes/self-managed-onprem/onpremises
echo " install calico-3.13.1"
rm -f calico-3.13.1.yaml
wget https://kuboard.cn/install-script/calico/calico-3.13.1.yaml
kubectl apply -f calico-3.13.1.yaml

give init_master.sh Script file executable permissions .

chmod a+x ./init_master.sh

perform init_master.sh Script files .

./init_master.sh

3. see Master The initialization result of the node

(1) Make sure all container groups are in Running state

#  Execute the following command , wait for  3-10  minute , Until all the container groups are in  Running  state 
watch kubectl get pod -n kube-system -o wide

As shown below .

[root@binghe101 ~]# watch kubectl get pod -n kube-system -o wide
Every 2.0s: kubectl get pod -n kube-system -o wide                                                                                                                          binghe101: Sat May  2 23:40:33 2020

NAME                                       READY   STATUS    RESTARTS   AGE     IP                NODE        NOMINATED NODE   READINESS GATES
calico-kube-controllers-5b8b769fcd-l2tmm   1/1     Running   0          3m59s   172.18.203.67     binghe101   <none>           <none>
calico-node-8krsl                          1/1     Running   0          3m59s   192.168.175.101   binghe101   <none>           <none>
coredns-546565776c-rd2zr                   1/1     Running   0          3m59s   172.18.203.66     binghe101   <none>           <none>
coredns-546565776c-x8r7l                   1/1     Running   0          3m59s   172.18.203.65     binghe101   <none>           <none>
etcd-binghe101                             1/1     Running   0          4m14s   192.168.175.101   binghe101   <none>           <none>
kube-apiserver-binghe101                   1/1     Running   0          4m14s   192.168.175.101   binghe101   <none>           <none>
kube-controller-manager-binghe101          1/1     Running   0          4m14s   192.168.175.101   binghe101   <none>           <none>
kube-proxy-qnffb                           1/1     Running   0          3m59s   192.168.175.101   binghe101   <none>           <none>
kube-scheduler-binghe101                   1/1     Running   0          4m14s   192.168.175.101   binghe101   <none>           <none>

(2) see Master Node initialization result

kubectl get nodes -o wide

As shown below .

[root@binghe101 ~]# kubectl get nodes -o wide
NAME        STATUS   ROLES    AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION         CONTAINER-RUNTIME
binghe101   Ready    master   5m43s   v1.18.2   192.168.175.101   <none>        CentOS Linux 8 (Core)   4.18.0-80.el8.x86_64   docker://19.3.8

initialization Worker node

1. obtain join Command parameter

stay Master Execute the following command on the node to get join Command parameter .

kubeadm token create --print-join-command

The specific implementation is as follows .

[root@binghe101 ~]# kubeadm token create --print-join-command
W0502 23:44:55.218947   59318 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join k8s.master:6443 --token s0hoh1.2cwyf1fyyjl2h04a     --discovery-token-ca-cert-hash sha256:6d78e360dc64d84762611ac6beec8ac0f0fe9f72a5c2cca008df949e07827c19

among , There is a line of output .

kubeadm join k8s.master:6443 --token s0hoh1.2cwyf1fyyjl2h04a     --discovery-token-ca-cert-hash sha256:6d78e360dc64d84762611ac6beec8ac0f0fe9f72a5c2cca008df949e07827c19

This line of code is what you get join command .

Be careful :join In the command token The effective time of is 2 Hours ,2 Within hours , You can use this token Initialize any number of worker node .

2. initialization Worker node

For all worker Node execution , ad locum , Is in the binghe102 The server and binghe103 Execute on the server .

establish init_worker.sh Script files , The contents of the document are as follows .

#  Only in  worker  Node execution 
# 192.168.175.101  by  master  Node's intranet  IP
export MASTER_IP=192.168.175.101
#  Replace  k8s.master  For initialization  master  The... Used in node  APISERVER_NAME
export APISERVER_NAME=k8s.master
echo "${MASTER_IP}    ${APISERVER_NAME}" >> /etc/hosts

#  Replace with  master  Node  kubeadm token create  Command output join
kubeadm join k8s.master:6443 --token s0hoh1.2cwyf1fyyjl2h04a     --discovery-token-ca-cert-hash sha256:6d78e360dc64d84762611ac6beec8ac0f0fe9f72a5c2cca008df949e07827c19

among ,kubeadm join... Namely master Node kubeadm token create Command output join.

give init_worker.sh Script file executable permissions , And implement init_worker.sh Script files .

chmod a+x ./init_worker.sh
./init_worker.sh

3. View the initialization results

stay Master The node executes the following command to view the initialization result .

kubectl get nodes -o wide

As shown below .

[root@binghe101 ~]# kubectl get nodes
NAME        STATUS   ROLES    AGE     VERSION
binghe101   Ready    master   20m     v1.18.2
binghe102   Ready    <none>   2m46s   v1.18.2
binghe103   Ready    <none>   2m46s   v1.18.2

Be careful :kubectl get nodes Add... After the order -o wide Parameters can output more information .

restart K8S Problems caused by clustering

1.Worker The node fails to start

Master Node IP The address has changed , Lead to worker The node cannot be started . Need to be reinstalled K8S colony , And make sure that all nodes have a fixed intranet IP Address .

2.Pod Crash or no normal access

Restart the server and use the following command to check Pod Operating state .

kubectl get pods --all-namespaces

Find many Pod be not in Running state , here , You need to use the following command to delete the abnormal Pod.

kubectl delete pod <pod-name> -n <pod-namespece>

Be careful : If Pod It's using Deployment、StatefulSet Wait for the controller to create ,K8S A new Pod As a substitute , rebooted Pod It usually works .

This article is from WeChat official account. - Glacier Technology (hacker-binghe)

The source and reprint of the original text are detailed in the text , If there is any infringement , Please contact the yunjia_community@tencent.com Delete .

Original publication time : 2020-12-01

Participation of this paper Tencent cloud media sharing plan , You are welcome to join us , share .

版权声明
本文为[glacier]所创,转载请带上原文链接,感谢
https://cdmana.com/2020/12/20201224160737121T.html

Scroll to Top