Deploying a K8s cluster with kubeasz

wzdushu@gmail.com (wanzi) — Thu, 12 Dec 2019 10:22:42 +0800

Environment Preparation

Master nodes

1
2
3

172.16.244.14
172.16.244.16
172.16.244.18

Node nodes

1
2

172.16.244.25
172.16.244.27

Master node VIP: 172.16.243.13
Deployment tool: Ansible/kubeasz

Initialize Environment

Install Ansible

apt update
apt-get install ansible expect
git clone https://github.com/easzlab/kubeasz
cd kubeasz
cp * /etc/ansible/

1
2

ssh-keygen -t rsa -b 2048 # Generate key pair
./tools/yc-ssh-key-copy.sh hosts root 'rootpassword'

Prepare Binary Files

1
2

cd tools
./easzup -D # Downloads binaries to /etc/ansible/bin/ by default

Configure hosts file as follows:

[kube-master]
172.16.244.14
172.16.244.16
172.16.244.18

[etcd]
172.16.244.14 NODE_NAME=etcd1
172.16.244.16 NODE_NAME=etcd2
172.16.244.18 NODE_NAME=etcd3

# haproxy-keepalived
[haproxy]
172.16.244.14
172.16.244.16
172.16.244.18

[kube-node]
172.16.244.25
172.16.244.27

# [optional] load balance for accessing k8s from outside
[ex-lb]
172.16.244.14 LB_ROLE=backup EX_APISERVER_VIP=172.16.243.13 EX_APISERVER_PORT=8443
172.16.244.16 LB_ROLE=backup EX_APISERVER_VIP=172.16.243.13 EX_APISERVER_PORT=8443
172.16.244.18 LB_ROLE=master EX_APISERVER_VIP=172.16.243.13 EX_APISERVER_PORT=8443

# [optional] ntp server for the cluster
[chrony]
172.16.244.18

[all:vars]
# --------- Main Variables ---------------
# Cluster container-runtime supported: docker, containerd
CONTAINER_RUNTIME="docker"

# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
#CLUSTER_NETWORK="flannel"
CLUSTER_NETWORK="calico"

# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"

# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.68.0.0/16"

# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="10.101.0.0/16"

# NodePort Range
NODE_PORT_RANGE="20000-40000"

# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="cluster.local."

# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/opt/kube/bin"

# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"

# Deploy Directory (kubeasz workspace)
base_dir="/etc/ansible"

Deploy K8S Cluster

Initialize Configuration

1
2

cd /etc/ansible
ansible-playbook 01.prepare.yml

This step performs three main tasks:

1
2
3

chrony role: Synchronize time across cluster nodes [optional]
deploy role: Generate CA certificates, kubeconfig, kube-proxy.kubeconfig
prepare role: Distribute CA certificates, install kubectl client, configure environment

Install etcd Cluster

`1`	`ansible-playbook 02.etcd.yml`

Install Docker

`1`	`ansible-playbook 03.docker.yml`

Deploy Kubernetes Master

`1`	`ansible-playbook 04.kube-master.yml`

Deploy Kubernetes Node

`1`	`ansible-playbook 05.kube-node.yml`

Deploy Kubernetes Network (using Calico here)

`1`	`ansible-playbook 06.network.yml`

Deploy Ingress/K8s Dashboard/CoreDNS

Configure SSL certificate for Ingress; the default used in the repository is Traefik 1.7.12, which we plan to upgrade to version 2.0 later.

1
2

# kubectl create secret tls traefik-cert --key=test.cn.key --cert=test.cn.pem -n kube-system
secret/traefik-cert created

Deploy cluster add-ons

`1`	`ansible-playbook 07.cluster-addon.yml`

Remove Taint from Master Nodes

By default, after deployment, master nodes are tainted and excluded from scheduling:

# kubectl get node
NAME            STATUS                     ROLES    AGE   VERSION
172.16.244.14   Ready,SchedulingDisabled   master   91m   v1.16.2
172.16.244.16   Ready,SchedulingDisabled   master   91m   v1.16.2
172.16.244.18   Ready,SchedulingDisabled   master   91m   v1.16.2
172.16.244.25   Ready                      node     90m   v1.16.2
172.16.244.27   Ready                      node     90m   v1.16.2
# kubectl describe node 172.16.244.14 | grep Taint
Taints:             node.kubernetes.io/unschedulable:NoSchedule

Due to limited machine resources, we enable scheduling on master nodes:

# kubectl patch node 172.16.244.14 -p '{"spec":{"unschedulable":false}}'
# kubectl get node
NAME            STATUS   ROLES    AGE   VERSION
172.16.244.14   Ready    master   95m   v1.16.2
172.16.244.16   Ready    master   95m   v1.16.2
172.16.244.18   Ready    master   95m   v1.16.2
172.16.244.25   Ready    node     94m   v1.16.2
172.16.244.27   Ready    node     94m   v1.16.2

Deploy External Load Balancer (Keepalived + Haproxy)

`1`	`ansible-playbook roles/ex-lb/ex-lb.yml`

Deploy Rancher

Install Helm3

Refer to https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-rancher/

cd /opt/soft
wget https://get.helm.sh/helm-v3.0.1-linux-amd64.tar.gz
tar xf helm-v3.0.1-linux-amd64.tar.gz
cd linux-amd64/
cp helm /usr/local/bin/

Create Certificate

Since we use our own domain certificate, we create a certificate using Kubernetes secret. Alternatively, you can use cert-manager to issue a certificate for Rancher or use Let’s Encrypt.

`1`	`kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=test.cn.pem --key=test.cn.key`

Install Rancher

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update
helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=rancher-cicd.test.cn --set ingress.tls.source=secret

Check Ingress, Resources, and Deployment Status

# kubectl get ingress --all-namespaces
NAMESPACE       NAME      HOSTS                   ADDRESS   PORTS     AGE
cattle-system   rancher   rancher-cicd.test.cn             80, 443   20h
# kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available...
deployment "rancher" successfully rolled out
# kubectl -n cattle-system get deploy rancher
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
rancher   3/3     3            3           5m5s

At this point, the entire K8S cluster has been successfully deployed. If everything proceeds smoothly, the entire process should take around 10 minutes. The key is proper planning in advance.

Kubeasz on Wanzi's Blog – Notes on life, tech, and DevOps.