Kubernetes on Wanzi's Blog – Notes on life, tech, and DevOps.

One Week to CKA: Experience Sharing

wzdushu@gmail.com (wanzi) — Mon, 19 Aug 2024 10:10:42 +0800

I’ve long wanted to obtain a cloud-native & k8s related certification, and the sooner the better. Due to work, I kept postponing it until this year, and I watched its price increase twice (which is a bit painful to mention).

Recently, I finally had a week free to fully prepare for the exam. Since I have been using Kubernetes in my production environment at work, my plan was to absorb more exam points and do a lot of practice questions before the exam.

Below, I summarize my CKA exam journey over the past week, hoping it will be helpful to you.

Comprehensive Understanding of CKA

1. What is CKA?

CKA stands for Certified Kubernetes Administrator, which is the first Kubernetes-based certification offered by CNCF. The entire exam consists of practical questions, and the invigilation is extremely strict, making CKA one of the most recognized and valuable Kubernetes certifications globally.

2. How long is the CKA certificate valid?

The CKA certificate is valid for 3 years, but starting from April 1, 2024, the validity period for newly certified CKA certificates will change to 2 years. Candidates who pass the exam and obtain the certificate before April 1, 2024, will still have a 3-year validity period.

After the certificate expires, you need to retake the exam and meet the exam requirements. The exam is conducted online, with a duration of 2 hours, consisting of 17 questions, a total score of 100, and a passing score of 66.

3. What is the use of CKA?

For companies, Kubernetes Certified Service Providers (KCSP) generally require 3 CKA holders. For individuals, obtaining a CKA certification is an important indicator of their learning outcomes. It not only allows for systematic learning of knowledge but also serves as proof of technical capability to potential employers.

4. Comparison Analysis of CKA, CKAD, and CKS

In fact, the CNCF foundation mainly offers these three Kubernetes-related certifications, each with different focuses, allowing candidates to choose based on their actual situation.

CKA: Certified Kubernetes Administrator, focuses on Kubernetes system administration, such as basic installation, configuration, and management of production-grade clusters, involving key concepts like Kubernetes networking, storage, security, maintenance, logging, monitoring, application lifecycle, troubleshooting, and API object primitives.
CKAD: Certified Kubernetes Application Developer, focuses on designing, building, and deploying cloud-native applications on Kubernetes, such as using Kubernetes core primitives to create/migrate, configure, expose, and observe scalable applications.
CKS: Certified Kubernetes Security, focuses on Kubernetes security, such as building, deploying, and runtime protection of containerized applications and Kubernetes platform security.

By preparing for the three certifications, CKA covers a broader range of knowledge and is relatively more difficult. If you pass the CKA and then take the CKAD or CKS, you will find it easier to pass, as 60-80% of the knowledge points are interrelated.

5. What are the main knowledge points covered in the CKA exam?

Part 1: Cluster Architecture, Installation, and Configuration (25%):

Managing Role-Based Access Control (RBAC)
Installing a basic cluster using Kubeadm
Managing a highly available Kubernetes cluster
Setting up infrastructure to deploy a Kubernetes cluster
Performing version upgrades on a Kubernetes cluster using Kubeadm
Implementing etcd backup and restore

Part 2: Workloads and Scheduling (15%):

Understanding Deployments and how to perform rolling updates and rollbacks
Configuring applications using ConfigMaps and Secrets
Understanding how to scale applications
Understanding the primitives used to create robust, self-healing application deployments
Understanding how resource limits affect Pod scheduling
Understanding manifest management and general template tools

Part 3: Services and Networking (20%):

Understanding host network configuration on cluster nodes
Understanding connectivity between Pods
Understanding ClusterIP, NodePort, LoadBalancer service types, and endpoints
Understanding how to use Ingress controllers and Ingress resources
Understanding how to configure and use CoreDNS
Selecting the appropriate Container Network Interface (CNI) plugin

Part 4: Storage (10%):

Understanding StorageClasses and Persistent Volumes
Understanding volume modes, access modes, and volume reclaim policies
Understanding Persistent Volume Claim (PVC) primitives
Understanding how to configure applications with persistent storage

Part 5: Troubleshooting (30%):

Evaluating cluster and node logs
Understanding how to monitor applications
Managing container standard output and standard error logs
Troubleshooting application failures
Troubleshooting cluster component failures
Troubleshooting network issues

CKA Pre-Exam Preparation

Registration and Payment

Registration link: https://training.linuxfoundation.cn/certificates/1

The price has increased compared to before, so it’s recommended to wait for Black Friday when there’s usually a 50% discount.

You can choose between Chinese or English invigilation. It’s suggested to choose Chinese, although the difference is not significant, and even if the questions are in English, they are generally understandable.

After successful registration, log in to the backend and note down the exam voucher code, which will be needed later.

Activating the Exam Code

Refer to the following link for the entire exam process: https://training.linuxfoundation.cn/news/308

Step 1: Register a Linux Foundation ID (LFID). Make sure your name matches your ID card, passport, and real-name authentication, otherwise, you may not pass the exam reservation.

Registration and exam reservation link: https://trainingportal.linuxfoundation.org/

Step 2: Use the previously obtained exam voucher to register for the exam. Normally, without registering for the exam, the voucher is only valid for 1 month. After registering, the voucher is valid for 1 year.

Step 3: Schedule the exam

Exam time: The entire exam lasts 2 hours. It is recommended to choose a time slot around 1-2 AM Shanghai time, as the exam servers are overseas, and this time slot is less likely to experience lag.
Exam language and invigilation language: Choose Chinese directly.
Computer environment check: Before the exam, you need to check the exam system environment. It is recommended to use Chrome. Do not install PSI in advance; install it half an hour before the exam. The day before the exam, it is recommended to disable or uninstall software like Sunflower, Todesk, VMware, and Lemon, as these can cause delays.

Once the reservation is successful, you can start preparing for the exam.

Precautions for the Exam Environment

Before the exam, it is recommended to have a VPN ready, but it is suggested not to use it during the exam unless your network is extremely slow, causing you to be disconnected from the exam system. In that case, you can use the VPN.

From pre-exam preparation to entering the actual exam, the following five steps are required:

Step 1: PSI environment check. You must pass this check to proceed with the exam. It is essential to start this process half an hour before the exam, as it can take about 20 minutes.
Step 2: Facial recognition, which requires taking a photo with your computer’s camera.
Step 3: Identity verification, where you need to take photos of your ID card and passport with the camera. The invigilator will check these.
Step 4: The invigilator will ask you to show your surroundings with the laptop camera. There should be nothing on or under the table, and no items within 1 meter. No one should be talking near you, and it is best to place your phone more than 1 meter away where the invigilator can see it.
Step 5: The invigilator will ask you to raise your hands to check if you are holding anything and if you have earplugs in your ears.

Once all the above is confirmed, the invigilator will ask if you are ready to start the exam. Confirm with “OK,” and the exam time will begin from this point.

Exam Time, Questions, and Results

The reason for highlighting the exam time is that the total exam duration is 2 hours, with 17 questions. This can feel both long and short.

Since the exam server is in the US, you may frequently encounter lag, which can make answering each question take longer.

For those with very limited preparation time, it is recommended to place a clock or wear a watch opposite your computer, as the exam system time is in US time, making it easy to lose track of time.

Regarding the exam questions, it is suggested to start with the simpler ones, as 60-70% can be solved via command line;

Remember to execute kubectl config context xxxxk8s to select the specified cluster for each question;

Questions involving PV/PVC, Netpol, and YAML writing should be done later, and cluster upgrades can be left for last.

Generally, the results will be available 24 hours after the exam, with a passing score of 66. If you cannot enter the exam due to environmental issues, contact the invigilator as soon as possible. If the result is unsatisfactory, you have one more chance to retake the exam, and you can reschedule directly.

Question Analysis

Below are eight sample questions collected from the internet, which I personally found slightly challenging. These questions are similar in type to the actual exam questions, and it is recommended to try them out. Therefore, if you can write Kubernetes resource object YAML files by hand, do so, as it will save more time.

1. Create a new ClusterRole named deployment-clusterrole that only allows creating the following resource types: Deployment, StatefulSet, DaemonSet;
   In the existing namespace app-team1, create a new ServiceAccount named cicd-token, limited to the namespace app-team1, and bind the new ClusterRole deployment-clusterrole to the new ServiceAccount.

2. Find the pod running with high CPU usage through the pod label name=cpu-loader, and write the name of the pod with the highest CPU usage into the file /opt/KUTR000401/KUTR00401.txt (already exists).

3. In the existing namespace my-app, create a new NetworkPolicy named allow-port-from-namespace;
   Ensure the new NetworkPolicy allows Pods in the namespace echo to connect to Pods in the namespace my-app on port 9000;
   Further ensure the new NetworkPolicy: does not allow access to Pods not listening on port 9000, and does not allow access from Pods not in the namespace echo.

4. Reconfigure the existing deployment front-end and add a port specification named http to expose the existing container nginx's port 80/tcp;
   Create a new service named front-end-svc to expose the container port http;
   Configure this service to expose them via NodePort on the nodes where the Pods are located.

5. Create a new nginx Ingress resource with the following specifications: Name: ping, Namespace: ing-internal, exposing the service hello on path /hello using service port 5678. You can check the availability of the service hello using the following command, which should return hello: curl -kL <INTERNAL_IP>/hello

6. Create a new PersistentVolumeClaim named pv-volume, Class: csi-hostpath-sc, Size: 10Mi;
   Create a new Pod to mount the PersistentVolumeClaim as a volume: Name: web-server, Image: nginx:1.16, Mount Path: /usr/share/nginx/html, and configure the new Pod to have ReadWriteOnce permissions on the volume. Finally, use kubectl edit or kubectl patch to expand the PersistentVolumeClaim size to 70Mi and record this change.

7. Add a sidecar container named sidecar to the existing Pod 11-factor-app using the busybox image;
   The new sidecar container must run the following command: /bin/sh -c tail -n+1 -f /var/log/11-factor-app.log;
   Use a Volume mounted at /var/log to make the log file 11-factor-app.log available to the sidecar container;
   Apart from adding the necessary volume mount, do not change the specifications of the existing containers.

8. The existing Kubernetes cluster is running version 1.30. Upgrade all Kubernetes control plane and node components on the master node to version 1.31. Ensure to drain the master node before the upgrade and uncordon it after the upgrade.
   You can connect to the master node using the following command: ssh master01
   You can gain higher privileges on the master node using the following command: sudo -i
   Additionally, upgrade kubelet and kubectl on the master node. Do not upgrade worker nodes, etcd, container manager, CNI plugins, DNS services, or any other plugins.

CKA Exam Summary

Finally, here is a CKA certificate to wrap things up:

Overall, the CKA certification exam is moderately difficult. I believe that practical Kubernetes skills are crucial, especially for DevOps engineers.

To achieve a high score, there are no shortcuts—just absorb the knowledge points listed above, practice a lot, and get hands-on experience.

Lastly, I want to say that the exam is just a formality. The goal is to prove your proficiency in k8s and ultimately apply these skills in your actual work.

Achieving Multi-User Access on a Single GPU Card Using Alibaba Cloud's Open Source Solution

wzdushu@gmail.com (wanzi) — Fri, 30 Jun 2023 18:10:42 +0800

A new AI project has recently been launched, primarily providing online AI experiments for universities. The project has also purchased a GPU server, but it only has one Nvidia Tesla T4 card, which needs to support multiple students doing experiments online simultaneously.

The current online experiment system runs on Kubernetes, so we need to consider GPU sharing in the k8s environment. We have previously tested the Alibaba Cloud GPU card sharing solution; here, I will just record the steps for using it:

Kubernetes cluster version: 1.23.1

Adjusting the K8S Scheduler

Starting from 1.23.+, due to changes in the kube-scheduler scheduling policy, the previous deployment methods are no longer suitable.

Refer to: https://kubernetes.io/docs/reference/scheduling/policies/

1
2

git clone https://github.com/AliyunContainerService/gpushare-scheduler-extender.git
cp config/scheduler-policy-config.yaml /etc/kubernetes/

Since my k8s was installed via kubeasz, the Kubeconfig is located at /etc/kubernetes/kubelet.kubeconfig. You need to update /etc/kubernetes/scheduler-policy-config.yaml as follows:

---
apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /etc/kubernetes/kube-scheduler.kubeconfig
extenders:
- urlPrefix: "http://192.168.233.101:32766/gpushare-scheduler"
  filterVerb: filter
  bindVerb: bind
  enableHTTPS: false
  nodeCacheCapable: true
  managedResources:
  - name: aliyun.com/gpu-mem
    ignoredByScheduler: false
  ignorable: false

Modify the Kube-scheduler startup parameters, /etc/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/opt/kube/bin/kube-scheduler \
  --authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --bind-address=0.0.0.0 \
  --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --config=/etc/kubernetes/scheduler-policy-config.yaml \
  --leader-elect=true \
  --v=2
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

1
2

systemctl  daemon-reload
systemctl  restart  kube-scheduler.service

Installing the Scheduler Extender

Label the GPU machine:

kubectl label node mynode gpushare=true

1
2

cd gpushare-scheduler-extender/config
kubectl apply -f gpushare-schd-extender.yaml

Installing the Plugin

1
2

kubectl create -f https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-rbac.yaml
kubectl create -f https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-ds.yaml

After installation, it should look like this:

1
2
3

# kubectl get pod   -n kube-system |grep gpushare
gpushare-device-plugin-ds-j8blj              1/1     Running   0                30h
gpushare-schd-extender-74796c5f64-7g4bl      1/1     Running   0                29h

Testing the GPU

Modify samples/1.yaml as follows:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-1
  labels:
    app: binpack-1

spec:
  replicas: 1

  selector: # define how the deployment finds the pods it manages
    matchLabels:
      app: binpack-1

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-1

    spec:
      containers:
      - name: binpack-1
        image: cheyang/gpu-player:v2
        resources:
          limits:
            # GiB
            aliyun.com/gpu-count: 1
            aliyun.com/gpu-mem: 5

# kubectl apply -f  samples/1.yaml
deployment.apps/binpack-1 created
# kubectl get pod  |grep binpack
binpack-1-9995bdf69-pk2d4                 1/1     Running   0             12s
# kubectl logs -f binpack-1-9995bdf69-pk2d4
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=5
2023-06-30 09:40:50.890296: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2023-06-30 09:40:50.976283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:03:00.0
totalMemory: 14.75GiB freeMemory: 14.66GiB
2023-06-30 09:40:50.976313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:03:00.0, compute capability: 7.5)

From the logs, GPU sharing has been successful, and 5G of video memory has been allocated.

You can also check using the kubectl extension tool kubectl-inspect-gpushare:

# cd /usr/bin/
# wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare
# chmod u+x /usr/bin/kubectl-inspect-gpushare
# kubectl-inspect-gpushare
NAME             IPADDRESS        GPU0(Allocated/Total)  GPU Memory(GiB)
192.168.233.101  192.168.233.101  9/14                   9/14
--------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
9/14 (64%)

At this point, GPU card sharing is complete, and the next step is to schedule my GPU card using TensorFlow.

Reference documentation:

https://github.com/AliyunContainerService/gpushare-scheduler-extender/blob/master/docs/install.md

kind: Build a lightweight Kubernetes cluster

wzdushu@gmail.com (wanzi) — Thu, 13 Apr 2023 18:22:42 +0800

Recently revisiting Golang, I built a web application and successfully ran it locally. Now, I wanted to test it in a Kubernetes cluster, but due to hardware limitations, setting up a full-fledged K8s cluster was challenging. I remembered a friend mentioning that Kubernetes can also run inside Docker—so I decided to give it a try.

Today’s focus is kind. What is kind? And what can it do?

1. Introduction to kind

kind stands for Kubernetes In Docker — a tool that uses Docker container nodes to run local Kubernetes clusters. It’s primarily designed for testing Kubernetes itself and is ideal for local development or CI pipelines.

Currently, kind supports almost all official Kubernetes versions. At present, it only supports Docker as the runtime, but future versions will gradually add support for common CRI runtimes like containerd. In the near future, support for containerd is expected to be included.

Under the hood, kind leverages open-source tools such as kubeadm and kustomize to manage cluster creation and configuration.

Enough talk — let’s get started with installing and using kind.

2. Installing kind

Since I’m on macOS, I’ll use brew:

# brew install kind
==> Downloading https://mirrors.ustc.edu.cn/homebrew-bottles/kind-0.17.0.arm64_monterey.bottle.tar.gz
Already downloaded: /Users/wanzi/Library/Caches/Homebrew/downloads/bcd419997297730492f5cebc36be86fb51f21061a6ddb2e066e1e4d8ad33ddf3--kind-0.17.0.arm64_monterey.bottle.tar.gz
==> Pouring kind-0.17.0.arm64_monterey.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions
==> Summary
🍺  /opt/homebrew/Cellar/kind/0.17.0: 8 files, 8.7MB
==> Running `brew cleanup kind`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
# kind version
kind v0.17.0 go1.19.2 darwin/arm64

For Linux:

1
2
3

curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.17.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

If you’re on Windows, you can use Chocolatey or install via binary — both are officially supported.

3. Creating Your First Cluster

Before proceeding, ensure Docker and kubectl are installed (installation steps omitted here).

# kind help
kind creates and manages local Kubernetes clusters using Docker container 'nodes'

Usage:
  kind [command]

Available Commands:
  build       Build one of [node-image]
  completion  Output shell completion code for the specified shell (bash, zsh or fish)
  create      Creates one of [cluster]
  delete      Deletes one of [cluster]
  export      Exports one of [kubeconfig, logs]
  get         Gets one of [clusters, nodes, kubeconfig]
  help        Help about any command
  load        Loads images into nodes
  version     Prints the kind CLI version

Flags:
  -h, --help              help for kind
      --loglevel string   DEPRECATED: see -v instead
  -q, --quiet             silence all stderr output
  -v, --verbosity int32   info log verbosity, higher value produces more output

Use "kind [command] --help" for more information about a command.

From kind help, we see that kind supports commands like build, create, delete, get, load, etc. Use -h for detailed help on subcommands.

Now, create a cluster:

Ensure ~/.kube directory exists — kind writes cluster API address, certificates, and other config into kubeconfig during setup.

# kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

# kubectl get node
NAME                 STATUS   ROLES           AGE     VERSION
kind-control-plane   Ready    control-plane   9m21s   v1.25.3

# kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-565d847f94-kp6l7                     1/1     Running   0          9m15s
kube-system          coredns-565d847f94-ml28b                     1/1     Running   0          9m15s
kube-system          etcd-kind-control-plane                      1/1     Running   0          9m32s
kube-system          kindnet-hfsc8                                1/1     Running   0          9m15s
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          9m31s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          9m33s
kube-system          kube-proxy-m9zp5                             1/1     Running   0          9m15s
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          9m33s
local-path-storage   local-path-provisioner-684f458cdd-d7f29      1/1     Running   0          9m15s

Success! Your first kind cluster is up and running.

4. Cluster Operations

1. Custom Cluster Name

# kind create cluster --name ci-cluster
Creating cluster "ci-cluster" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-ci-cluster"
You can now use your cluster with:

2. Switch Context

# kubectl cluster-info --context kind-ci-cluster
Kubernetes control plane is running at https://127.0.0.1:56527
CoreDNS is running at https://127.0.0.1:56527/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Have a nice day! 👋

3. List All Clusters

1
2
3

# kind get clusters
ci-cluster
kind

4. Load Images into Nodes

By default, kind nodes cannot access host machine images — they must be manually loaded.

# kind load docker-image --name ci-cluster --nodes ci-cluster-control-plane traefik:v2.9.5
# docker exec -it ci-cluster-control-plane bash
root@ci-cluster-control-plane:/# crictl images | grep traefik
docker.io/library/traefik                  v2.9.5               a1252ce6bfaaa       132MB

5. Advanced Cluster Configuration

1. Multi-node Cluster

By default, kind creates a single-node cluster. Here’s how to define multiple nodes:

Create config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

Recreate the cluster:

# kind create cluster --config config.yaml
# kubectl get node
NAME                 STATUS   ROLES           AGE    VERSION
kind-control-plane   Ready    control-plane   110s   v1.25.3
kind-worker          Ready    <none>          74s    v1.25.3
kind-worker2         Ready    <none>          88s    v1.25.3

Note: Kind also supports multi-master HA setups — though not covered here for brevity.

2. Custom Kubernetes Version

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315

3. Mount Host Directory to Node Containers (Persistent Storage)

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/files
    containerPath: /files
  - hostPath: /Users/wanzi/tools/kind/other-files/
    containerPath: /other-files
    readOnly: true
    selinuxRelabel: false
    propagation: None

4. Port Mapping

Map host port 80 to nodePort 30950. The container port must match the Service’s nodePort.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30950
    hostPort: 80

And apply this service/pod example:

kind: Pod
apiVersion: v1
metadata:
  name: foo
  labels:
    app: foo
spec:
  containers:
  - name: foo
    image: hashicorp/http-echo:0.2.3
    args:
    - "-text=foo"
    ports:
    - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: foo
spec:
  type: NodePort
  ports:
  - name: http
    nodePort: 30950
    port: 5678
  selector:
    app: foo

5. Add Custom Labels

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  extraPortMappings:
  - containerPort: 30950
    hostPort: 80
  labels:
    tier: frontend
- role: worker
  labels:
    tier: backend

6. Customize kubeadm Configurations

Since kind uses kubeadm under the hood, you can customize the init process on the first control plane node:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label=true"

For deeper customization, four configuration types are available:

InitConfiguration
ClusterConfiguration
KubeProxyConfiguration
KubeletConfiguration

Example: Override API server flags using ClusterConfiguration:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      extraArgs:
        enable-admission-plugins: NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

For additional nodes joining the cluster, use JoinConfiguration:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label2=true"
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label3=true"

7. Configure Ingress

Allow local requests to ingress controller via ports 80/443 and set specific labels:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP

Apply ingress-nginx:

`1`	`kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml`

6. Complete Cluster Configuration Example

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  labels:
    app: front
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/wwwroot
    containerPath: /wwwroot
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  labels:
    app: backend
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/wwwroot
    containerPath: /wwwroot

networking:
  apiServerAddress: "127.0.0.1"
  apiServerPort: 6443
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
  # disableDefaultCNI: true # Default CNI is kindnetd; can disable to use others like Calico
  kubeProxyMode: "ipvs" # Set kube-proxy mode to ipvs; use "none" to disable

That concludes our journey with kind. You now have a fast, lightweight way to spin up Kubernetes clusters for testing and feature validation.

References:

Resolve Nginx file upload limits and 504 gateway timeout in Kubernetes

wzdushu@gmail.com (wanzi) — Mon, 08 Nov 2021 17:22:42 +0800

Recently, two recurring issues have arisen in our business operations using the Kubernetes cluster. Here’s a record of the solutions:

Frontend page file upload limited to 1M
POST requests from frontend to backend timing out with a 504 error

Solution for the first issue:
By default, Nginx limits upload size to 1M. To resolve this, add the following configuration in the http, server, or location blocks of the Nginx config:

`1`	`client_max_body_size 800M;`

Solution for the second issue:
By default, Nginx has a 1-minute timeout for reading responses from the backend. To extend this, add the following in the http, server, or location blocks:

1
2
3

proxy_connect_timeout 120;
proxy_read_timeout  900;
proxy_send_timeout 900;

Final Nginx configuration:

server {
    listen 80;
    server_name web_server;
    location / {
        root /usr/share/nginx/html/;
        index index.htm index.html;
        try_files $uri $uri/ /index.html;
    }
    location /business-operation {
        proxy_pass http://business-operat.common:5000;
        
        proxy_ignore_client_abort on; 
        keepalive_timeout 900s;
        client_body_timeout 900s;
        proxy_connect_timeout 120s;
        proxy_read_timeout  900s;
        proxy_send_timeout 900s;
        client_max_body_size 800m;
        proxy_pass_header Authorization;
        proxy_pass_header WWW-Authenticate;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
    }
}

After applying the above Nginx configuration, ensure these annotations are included in the Ingress definition:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "800M"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '120'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '900'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '900'
  name: enterprise-mark
  namespace: frontend
spec:
  rules:
    - host: enterprise-marketing.xxxx.com
      http:
        paths:
          - backend:
              serviceName: enterprise-mark
              servicePort: 80
            path: /
status:
  loadBalancer:
    ingress:
      - ip: 39.107.xxx.xxx

Alibaba Cloud ACK supports both public and internal SLB.

wzdushu@gmail.com (wanzi) — Thu, 21 Oct 2021 17:22:42 +0800

I. Background

You have an ACK cluster.
Nginx Ingress Controller has been successfully deployed and bound to a public-facing SLB.

Note: Kubernetes clusters created via the Alibaba Cloud Container Service console automatically deploy an Nginx Ingress Controller during initialization, which is default-mounted to a public SLB instance.

II. Configuration

1. Create an Internal SLB

In the Alibaba Cloud console, create an internal SLB and bind it to your VPC.

2. Configure Nginx Ingress Controller

# my-nginx-ingress-slb-intranet.yaml
# intranet nginx ingress slb service
apiVersion: v1
kind: Service
metadata:
  # Name the service as nginx-ingress-lb-intranet.
  name: nginx-ingress-lb-intranet
  namespace: kube-system
  labels:
    app: nginx-ingress-lb-intranet
  annotations:
    # Specify the SLB instance type as internal.
    service.beta.kubernetes.io/alicloud-loadbalancer-address-type: intranet
    # Replace with your internal SLB instance ID.
    service.beta.kubernetes.io/alicloud-loadbalancer-id: <YOUR_INTRANET_SLB_ID>
    # Whether to automatically create SLB port listeners (overrides existing ones); can also be configured manually.
    #service.beta.kubernetes.io/alicloud-loadbalancer-force-override-listeners: 'false'
spec:
  type: LoadBalancer
  # Route traffic to other nodes
  externalTrafficPolicy: "Cluster"
  ports:
  - port: 80
    name: http
    targetPort: 80
  - port: 443
    name: https
    targetPort: 443
  selector:
    # Select pods with app=ingress-nginx
    app: ingress-nginx

Apply the service resource:

`1`	`kubectl apply -f my-nginx-ingress-slb-intranet.yaml`

Retrieve the service resource:

1
2
3

# kubectl -n kube-system get svc | grep nginx-ingress-lb
nginx-ingress-lb            LoadBalancer   172.21.15.148   39.107.xxx.xxx   80:32076/TCP,443:30803/TCP     433d
nginx-ingress-lb-intranet   LoadBalancer   172.21.5.0      172.17.193.181   80:32282/TCP,443:30507/TCP     1d

After configuring Ingress to expose services, you can access the service via the public SLB. Other services within the same VPC can directly access the service through the internal SLB.

3. Create Ingress Service

Once the Ingress controller is configured, creating an Ingress requires no special setup—just like before. Example:

# Source: prometheusalert/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: RELEASE-NAME-prometheusalert
  labels:
    app.kubernetes.io/name: prometheusalert
    helm.sh/chart: prometheusalert-1.0.0
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/version: "1.2.0"
    app.kubernetes.io/managed-by: Helm
spec:
  rules:
    - host: "palert.con.sdi"
      http:
        paths:
          - path: /
            backend:
              serviceName: RELEASE-NAME-prometheusalert
              servicePort: 8080

For internal domain names, simply resolve them directly to the internal SLB address (e.g., 172.17.193.181).

Reference: Alibaba Cloud Documentation

Aggregating Prometheus Alert Messages Using Prometheus Alertmanager

wzdushu@gmail.com (wanzi) — Sat, 18 Sep 2021 17:22:42 +0800

Deploy PrometheusAlert

git clone https://github.com/feiyu563/PrometheusAlert.git
cd PrometheusAlert/example/helm/prometheusalert
# Update config/app.conf to set login user info and database configuration
helm install -n monitoring .

Create a WeChat Work Group Robot

After creating a WeChat Work group, right-click the group → “Add Group Robot”. This will generate a webhook URL for the robot. Record this URL for later use.

Prometheus Alert Configuration

Configure Alertmanager

Since I’m using https://github.com/prometheus-operator/kube-prometheus.git for Prometheus, directly modify the alertmanager-secret.yaml file. Add the webhook configuration under receivers:

apiVersion: v1
data: {}
kind: Secret
type: Opaque
metadata:
  name: alertmanager-main
  namespace: monitoring
stringData:
  alertmanager.yaml: |-
    global:
      resolve_timeout: "5m"
      # Email settings
      smtp_smarthost: 'smtp.exmail.qq.com:465'                   # SMTP server proxy
      smtp_from: 'devops_alert@xxx.com'                       # Sender email address
      smtp_auth_username: 'devops_alert@xxx.com'              # Email username
      smtp_auth_password: '1DevSop#'                             # Authorization password
      smtp_require_tls: false                            # TLS disabled (default is enabled)
    inhibit_rules:
    - equal:
      - namespace
      - alertname
      - instance
      source_match:
        severity: critical
      target_match_re:
        severity: warning|info
    - equal:
      - namespace
      - alertname
      - instance
      source_match:
        severity: warning
      target_match_re:
        severity: info
    receivers:
    - name: default
      email_configs:            # Email configuration
      - to: "xxxx@163.com"   # Recipient email for alerts
    - name: jf-k8s-node
      webhook_configs:
      - url: "http://palert.con.sdi/prometheusalert?type=wx&tpl=jf-k8s-node&wxurl=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=0da1f944-9a4e-4221-8036-f322351f7bd8"

Configure Prometheus Rules

Here’s an example of a disk usage alert rule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: external-node.rules
  namespace: monitoring
spec:
  groups:
  - name: k8s-node.rules
    rules:
    - alert: 磁盘容量
      expr: 100-(node_filesystem_avail_bytes{instance=~'39.100.*',fstype=~"ext4|xfs|fuse.glusterfs"}/node_filesystem_size_bytes {fstype=~"ext4|xfs|fuse.glusterfs"}*100) > 96
      for: 1m
      labels:
        severity: critical
        app: node-alert
      annotations:
        summary: "{{$labels.mountpoint}} 磁盘分区使用率过高！"
        description: "{{$labels.mountpoint }} 磁盘分区使用大于90%(目前使用:{{$value}}%)"

Customize Alert Templates

By default, Alertmanager uses its own template, which is not very effective. Here we unify the use of PrometheusAlert’s templates.

According to the official documentation, create a new template and specify the type as WeChat Work, purpose as Prometheus, with the following content:

{{ $var := .externalURL}}{{ range $k,$v:=.alerts }}
{{if eq $v.status "resolved"}}
#### [Prometheus Recovery Notification]({{$v.generatorURL}})
##### <font color="#FF0000">Alert Name</font>：[{{$v.labels.alertname}}]({{$var}})
##### <font color="#FF0000">Severity</font>：{{$v.labels.severity}}
##### <font color="#FF0000">Trigger Time</font>：{{GetCSTtime $v.startsAt}}
##### <font color="#02b340">Recovery Time</font>：{{GetCSTtime $v.endsAt}}
##### <font color="#FF0000">Instance</font>：{{$v.labels.instance}}
##### <font color="#FF0000">Details</font>：{{$v.annotations.description}}
{{else}}
#### [Prometheus Alert Notification]({{$v.generatorURL}})
##### <font color="#FF0000">Alert Name</font>：[{{$v.labels.alertname}}]({{$var}})
##### <font color="#FF0000">Severity</font>：{{$v.labels.severity}}
##### <font color="#FF0000">Trigger Time</font>：{{GetCSTtime $v.startsAt}}
##### <font color="#FF0000">Instance</font>：{{$v.labels.instance}}
##### <font color="#FF0000">Details</font>：{{$v.annotations.description}}
{{end}}
{{ end }}

After triggering a disk space alert, the actual result appears as follows:

With this setup, PrometheusAlert has successfully received Prometheus alert messages and forwarded them to the WeChat Work group robot.

Additionally, for multi-cluster and multiple alert sources, the power of PrometheusAlert truly shines.

Develop a Kubernetes cluster backup strategy

wzdushu@gmail.com (wanzi) — Sat, 11 Sep 2021 17:22:42 +0800

For backups, every internet company’s technical team must handle this task, and we are no exception. Today, I’ll share my own strategies for backing up production Kubernetes clusters.

My primary goals for Kubernetes backups are to prevent:

Accidental deletion of a namespace within the cluster
Accidental misconfiguration causing resource anomalies (e.g., deployments, configmaps)
Accidental deletion of partial resources in the cluster
Loss of etcd data

Backing Up etcd

Backing up etcd prevents catastrophic failures at the cluster level or loss of etcd data, which could render the entire cluster unusable. In such cases, only full cluster recovery can restore services.

Backup script for etcd:

#!/bin/bash
#ENDPOINTS="https://192.168.1.207:2379,https://192.168.1.208:2379,https://192.168.1.209:2379"
ENDPOINTS="127.0.0.1:2379"
CACERT="/etc/kubernetes/pki/etcd/ca.crt"
CERT="/etc/kubernetes/pki/etcd/server.crt"
KEY="/etc/kubernetes/pki/etcd/server.key"
DATE=`date +%Y%m%d-%H%M%S`
BACKUP_DIR="/home/centos/hostpath/backups/k8s/etcd"
ETCDCTL_API=3 /usr/local/bin/etcdctl --cacert=${CACERT} --cert=${CERT} --key=${KEY} --endpoints="${ENDPOINTS}" snapshot save ${BACKUP_DIR}/k8s-snapshot-${DATE}.db
find $BACKUP_DIR/ -type f -mtime +20 -exec rm -f {} \;

Cron job schedule:

`1`	`50 21 * * * /bin/bash /home/centos/hostpath/backups/k8s/etcdv3-bak.sh`

Setting Up MinIO Object Storage

Since our storage cluster uses GlusterFS, we use MinIO to set up object storage with GlusterFS as the underlying filesystem. If you’re using Alibaba Cloud OSS to back up your cluster resources, skip this step and refer to: https://github.com/AliyunContainerService/velero-plugin

To deploy MinIO in Kubernetes, persistent volumes (PV) and persistent volume claims (PVC) are required. For simplicity, we run it via Docker:

version: '2.0'
services:
  minio:
    image: minio/minio:latest
    container_name: minio
    ports:
      - "39000:9000"
      - "39001:9001"
    restart: always
    command: server --console-address ':9001' /data
    environment:
      MINIO_ACCESS_KEY: admin
      MINIO_SECRET_KEY: adminSD#123
    logging:
      options:
        max-size: "1000M" # Maximum file upload limit
        max-file: "100"
      driver: json-file
    volumes:
      - /home/centos/hostpath/backups/k8s/velero:/data # Mount path
    networks:
      - minio

networks:
  minio:
    ipam:
      config:
      - subnet: 10.210.1.0/24
        gateway: 10.210.1.1

Open your browser and access the following URL with the provided credentials to manage MinIO via the web console:

1
2
3

MinIO Web: http://192.168.1.214:39001
MinIO Admin: admin
MinIO Admin Password: adminSD#123

Installing Velero Client

`1`	`brew install velero`

Create a file credentials-velero for later use when setting up the Velero server to connect to object storage:

1
2
3

[default]
aws_access_key_id = admin
aws_secret_access_key = adminSD#123

Deploying Velero in Kubernetes

# velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.2.0 \
    --bucket k8s-jf \
    --secret-file ./credentials-velero \
    --use-volume-snapshots=false \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.1.214:39000

Velero Commands

Backup, View, Delete Operations

# Backup resources in the ingress-nginx namespace:
velero backup create ingress-nginx-backup --include-namespaces ingress-nginx

# View backup details:
velero backup describe ingress-nginx-backup
velero backup logs ingress-nginx-backup

# Delete a backup:
velero delete backup ingress-nginx-backup

# Backup all resources except those in ingress-nginx and test namespaces:
velero backup create k8s-full-test-backup --exclude-namespaces ingress-nginx,test

# Backup specific resource types:
velero backup create kube-system-backup --include-resources pod,secret

# Delete backup without confirmation:
velero backup delete kube-system-backup --confirm

# Backup pods with PVCs:
velero backup create pvc-backup --snapshot-volumes --include-namespaces test-velero

Note:

Use --include-resources to specify resource types to include.
Use --exclude-resources to exclude certain resource types.

Scheduled Backups

# Backup every 6 hours:
velero create schedule ${SCHEDULE_NAME} --schedule="0 */6 * * *"

# Backup every 6 hours using @every syntax:
velero create schedule ${SCHEDULE_NAME} --schedule="@every 6h"

# Daily backup for the web namespace:
velero create schedule ${SCHEDULE_NAME} --schedule="@every 24h" --include-namespaces web

# Weekly backup with TTL of 90 days:
velero create schedule ${SCHEDULE_NAME} --schedule="@every 168h" --ttl 2160h0m0s

Note: --ttl sets the backup retention period. After expiration, backups are automatically cleaned up. Default is 30 days.

Restoring Backups

# Create a restore from a backup:
velero restore create ${RESTORE_NAME} --from-backup ${BACKUP_NAME}

# Create a restore from a backup (default name: ${BACKUP_NAME}-<timestamp>):
velero restore create --from-backup ${BACKUP_NAME}

# Restore from the latest backup of a schedule:
velero restore create --from-schedule ${SCHEDULE_NAME}

# Restore specific resources from a backup:
velero restore create --from-backup backup-2 --include-resources pod,secret

# Restore all backups (existing services won't be overwritten):
velero restore create --from-backup all-ns-backup

# Restore only specific namespaces (e.g., default and nginx-example):
velero restore create --from-backup all-ns-backup --include-namespaces default,nginx-example

# Restore test-velero namespace to test-velero-1:
velero restore create restore-for-test --from-backup everyday-1-20210203131802 --namespace-mappings test-velero:test-velero-1

View backups:

velero get backup   # List backups
velero get schedule # List scheduled backups
velero get restore  # List restores
velero get plugins  # List installed plugins

Note: Velero allows restoring resources to different namespaces than their original source. Use the --namespace-mappings flag:

`1`	`velero restore create RESTORE_NAME --from-backup BACKUP_NAME --namespace-mappings old-ns-1:new-ns-1,old-ns-2:new-ns-2`

Velero Backup Practical Examples

Perform a full cluster backup:

`1`	`velero backup create k8s-jf-test-all`

Set up a 4-hourly backup with retention of 2 months:

1
2

# velero create schedule k8s-jf-cron-4h --exclude-namespaces test,tt --schedule="@every 4h" --ttl 1440h
Schedule "k8s-jf-cron-4h" created successfully.

Restore a single namespace from a full backup within the same cluster:

# velero restore create k8s-jf-test-all-restore --from-backup k8s-jf-test-all --include-namespaces test --namespace-mappings test:test10000
# velero restore describe k8s-jf-test-all-restore
Name:         k8s-jf-test-all-restore
Namespace:    velero
Labels:       <none>
Annotations:  <none>
Phase:                                 InProgress
Estimated total items to be restored:  141
Items restored so far:                 123
Started:    2021-09-07 10:47:44 +0800 CST
Completed:  <n/a>
Backup:  k8s-jf-test-all
Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>
Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto
Namespace mappings:  test=test10000
Label selector:  <none>
Restore PVs:  auto
Preserve Service NodePorts:  auto
# velero restore get
NAME                      BACKUP            STATUS       STARTED                         COMPLETED   ERRORS   WARNINGS   CREATED                         SELECTOR
k8s-jf-test-all-restore   k8s-jf-test-all   InProgress   2021-09-07 10:47:44 +0800 CST   <nil>       0        0          2021-09-07 10:47:44 +0800 CST   <none>

Cross-cluster restore using periodic backups

For cross-cluster restores, both clusters must use the same cloud provider’s persistent volume solution. Here, we use MinIO with bucket k8s-jf on both sides:

velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.2.0 \
    --bucket k8s-jf \
    --secret-file ./credentials-velero \
    --use-volume-snapshots=false \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.1.214:39000

Check existing backups:

# velero backup get
NAME                                STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
k8s-jf-all-cron-4h-20210907061716   Completed         0        0          2021-09-07 14:17:16 +0800 CST   59d       default            <none>
k8s-jf-all-cron-4h-20210907021627   Completed         0        0          2021-09-07 10:16:27 +0800 CST   59d       default            <none>
k8s-jf-test-all                     Completed         0        0          2021-09-07 10:19:45 +0800 CST   29d       default            <none>

Restore specific namespace data

Restore the argocd namespace from backup k8s-jf-all-cron-4h-20210907061716 into the argocd-dev namespace in the current cluster:

# velero restore create --from-backup k8s-jf-all-cron-4h-20210907061716 --include-namespaces argocd --namespace-mappings argocd:argocd-dev
Restore request "k8s-jf-all-cron-4h-20210907061716-20210907155450" submitted successfully.
Run `velero restore describe k8s-jf-all-cron-4h-20210907061716-20210907155450` or `velero restore logs k8s-jf-all-cron-4h-20210907061716-20210907155450` for more details.
# velero restore get
NAME                                               BACKUP                              STATUS       STARTED                         COMPLETED   ERRORS   WARNINGS   CREATED                         SELECTOR
k8s-jf-all-cron-4h-20210907061716-20210907155450   k8s-jf-all-cron-4h-20210907061716   InProgress   2021-09-07 15:54:51 +0800 CST   <nil>       0        0          2021-09-07 15:54:51 +0800 CST   <none>
# velero restore logs k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Attempting to restore Secret: argocd-application-controller-token-wv62v" logSource="pkg/restore/restore.go:1238" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
...
time="2021-09-07T08:11:30Z" level=info msg="Restored 61 items out of an estimated total of 61 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=argocd-server namespace=argocd-dev progress= resource=services restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:30Z" level=info msg="Waiting for all restic restores to complete" logSource="pkg/restore/restore.go:546" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
...
time="2021-09-07T08:11:30Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:480" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119

The logs confirm that data from the original argocd namespace has been successfully restored to the current cluster’s argocd-dev namespace.

Uninstalling Velero

Uninstall Velero — note that uninstall does not delete the namespace:

# velero uninstall
You are about to uninstall Velero.
Are you sure you want to continue (Y/N)? y
Velero uninstalled ⛵
# kubectl delete namespace/velero clusterrolebinding/velero
# kubectl delete crds -l component=velero

Troubleshooting Backup Issues

time="2021-09-07T07:22:35Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114"
time="2021-09-07T07:22:36Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117"
time="2021-09-07T07:22:36Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = RequestError: send request failed\ncaused by: Get http://minio.velero.svc:9000/velero?delimiter=%2F&list-type=2&prefix=backups%2F: dial tcp: lookup minio.velero.svc on 10.96.0.10:53: no such host" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:361" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182"
...

This indicates inconsistency between CRD BackupStorageLocation and the configured object storage credentials. Likely due to leftover resources after reinstallation.

Fix by reinstalling cleanly:

# velero uninstall
You are about to uninstall Velero.
Are you sure you want to continue (Y/N)? y
Velero uninstalled ⛵
# kubectl delete namespace/velero clusterrolebinding/velero
# kubectl delete crds -l component=velero
# velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.2.0 \
    --bucket k8s-jf \
    --secret-file ./credentials-velero \
    --use-volume-snapshots=false \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.1.214:39000
# kubectl -n velero get backupstoragelocation default -o yaml # Verify correct configuration
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  creationTimestamp: "2021-09-07T07:47:44Z"
  generation: 1
  labels:
    component: velero
  name: default
  namespace: velero
  resourceVersion: "1184696"
  selfLink: /apis/velero.io/v1/namespaces/velero/backupstoragelocations/default
  uid: 39502e43-272e-461f-a114-a9ec955f0510
spec:
  config:
    region: minio
    s3ForcePathStyle: "true"
    s3Url: http://192.168.1.214:39000
  default: true
  objectStorage:
    bucket: k8s-jf
  provider: aws
status:
  lastSyncedTime: "2021-09-07T07:50:00Z"
  lastValidationTime: "2021-09-07T07:50:00Z"
  phase: Available

Alibaba Cloud Shared GPU Solution Testing

wzdushu@gmail.com (wanzi) — Tue, 31 Aug 2021 14:22:42 +0800

Before deployment, ensure that nvidia-driver and nvidia-docker are installed on your Kubernetes nodes, and Docker’s default runtime has been set to nvidia.

# cat /etc/docker/daemon.json
{
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
      "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia"
}

1. Install gpushare-device-plugin via Helm

1
2
3

$ git clone https://github.com/AliyunContainerService/gpushare-scheduler-extender.git
$ cd gpushare-scheduler-extender/deployer/chart
$ helm install --name gpushare --namespace kube-system --set masterCount=3 gpushare-installer

2. Label GPU Nodes

1
2

$ kubectl label node sd-cluster-04 gpushare=true
$ kubectl label node sd-cluster-05 gpushare=true

3. Install kubectl-inspect-gpushare

Ensure kubectl is already installed (omitted here).

1
2
3

$ cd /usr/bin/
$ wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare
$ chmod u+x /usr/bin/kubectl-inspect-gpushare

Check current GPU resource usage in the cluster:

$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  0/14                   0/14
sd-cluster-05  192.168.1.215  8/14                   8/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
8/28 (28%)

To disable GPU sharing on a node, simply set gpushare=false:

1
2

$ kubectl label node sd-cluster-04 gpushare=false
$ kubectl label node sd-cluster-05 gpushare=false

II. Validation Test

1. Deploy First Application

Request 2GiB GPU memory; the application should be scheduled onto one GPU card.

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-1
  labels:
    app: binpack-1

spec:
  replicas: 1

  selector:
    matchLabels:
      app: binpack-1

  template:
    metadata:
      labels:
        app: binpack-1

    spec:
      containers:
      - name: binpack-1
        image: cheyang/gpu-player:v2
        resources:
          limits:
            # GiB
            aliyun.com/gpu-mem: 2

$ kubectl apply -f 1.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  0/14                   0/14
sd-cluster-05  192.168.1.215  2/14                   2/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
2/28 (7%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
$ kubectl logs -f binpack-1-6d6955c487-j4c4b -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=2
2021-08-13 02:47:22.395557: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 02:47:22.552831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 14.65GiB
2021-08-13 02:47:22.552873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

2. Deploy Second Application

Request 8GiB GPU memory per pod, with 2 replicas → total 16GiB.

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-2
  labels:
    app: binpack-2

spec:
  replicas: 2

  selector:
    matchLabels:
      app: binpack-2

  template:
    metadata:
      labels:
        app: binpack-2

    spec:
      containers:
      - name: binpack-2
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 8

$ kubectl apply -f 2.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  8/14                   8/14
sd-cluster-05  192.168.1.215  10/14                  10/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
18/28 (64%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          27m
$ kubectl logs -f binpack-2-58579b95f7-4wpbl -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=8
2021-08-13 02:48:41.246585: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 02:48:41.338992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 13.07GiB
2021-08-13 02:48:41.339031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

As shown by resource usage, the second application is now using 8GiB from two separate GPUs.

3. Deploy Third Application

Request 2GiB GPU memory.

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-3
  labels:
    app: binpack-3

spec:
  replicas: 1

  selector:
    matchLabels:
      app: binpack-3

  template:
    metadata:
      labels:
        app: binpack-3

    spec:
      containers:
      - name: binpack-3
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 2

$ kubectl apply -f 3.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  8/14                   8/14
sd-cluster-05  192.168.1.215  12/14                  12/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
20/28 (71%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          27m
binpack-2-58579b95f7-sjhwt         1/1     Running   0          27m
binpack-3-556bbd84f9-9xqg7         1/1     Running   0          14m
$ kubectl logs -f binpack-3-556bbd84f9-9xqg7 -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=2
2021-08-13 03:01:53.897423: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 03:01:54.008665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 7.08GiB
2021-08-13 03:01:54.008716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

After deploying the third application, the maximum available GPU memory is 8GiB. However, actual usage is limited to 6GiB per task because a single task cannot span across multiple GPU cards.

4. Deploy Fourth Application

Request 5GiB GPU memory — should be scheduled on sd-cluster-04.

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-4
  labels:
    app: binpack-4

spec:
  replicas: 1

  selector:
    matchLabels:
      app: binpack-4

  template:
    metadata:
      labels:
        app: binpack-4

    spec:
      containers:
      - name: binpack-4
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 5

$ kubectl apply -f 4.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  13/14                  13/14
sd-cluster-05  192.168.1.215  12/14                  12/14
------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
25/28 (89%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          26m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          24m
binpack-2-58579b95f7-sjhwt         1/1     Running   0          24m
binpack-3-556bbd84f9-9xqg7         1/1     Running   0          11m
binpack-4-6956458f85-cv62j         1/1     Running   0          6s
$ kubectl logs -f binpack-4-6956458f85-cv62j -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=5
2021-08-13 03:13:20.208122: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 03:13:20.361391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 6.46GiB
2021-08-13 03:13:20.361481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

III. Summary

The gpushare-device-plugin has the following limitations:

A single task cannot utilize GPUs across multiple machines (shared).
GPU resource allocation cannot be based on utilization percentage within a single GPU.

However, it is sufficient for algorithm team model testing scenarios. There are two alternative GPU sharing solutions, which are not covered here—refer directly to their official repositories if needed:

https://github.com/tkestack/gpu-manager
https://github.com/vmware/bitfusion-with-kubernetes-integration

References:
https://github.com/AliyunContainerService/gpushare-scheduler-extender/tree/master/deployer

Deploying a High-Availability Kubernetes Cluster with kubeadm

wzdushu@gmail.com (wanzi) — Sun, 15 Aug 2021 17:22:42 +0800

To facilitate later verification of private deployment, a quick Kubernetes cluster setup is required in the internal network environment. Previously, for larger clusters, I typically used Kubeasz or Kubespray. For this small-scale cluster, using kubeadm will be more efficient.

Below is the recorded process for deploying with kubeadm:

Cluster Nodes:

192.168.1.206 sd-cluster-206 node
192.168.1.207 sd-cluster-207 master,etcd
192.168.1.208 sd-cluster-208 master,etcd,haproxy,keepalived
192.168.1.209 sd-cluster-209 master,etcd,haproxy,keepalived

Image Versions:

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.5
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
docker pull registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.14.0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v0.48.1

I. Basic Environment Setup

1. Install Docker and Configure Hosts

yum install -y yum-utils device-mapper-persistent-data lvm2 git
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install docker-ce -y
systemctl start docker
systemctl enable docker
systemctl status docker

2. Configure `/etc/hosts`

cat >> /etc/hosts << hhhh
192.168.1.207 sd-cluster-207
192.168.1.208 sd-cluster-208
192.168.1.209 sd-cluster-209
hhhh

3. Disable Firewall and Set SELinux

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux

4. Disable Swap

Kubernetes 1.8+ requires disabling swap. If not disabled, kubelet will fail to start by default.
Option 1: Use --fail-swap-on=false in kubelet startup args.
Option 2: Disable system swap.

`1`	`# swapoff -a`

Edit /etc/fstab to comment out the swap mount line. Confirm swap is off using free -m.

5. Install kubeadm, kubelet, and ipvsadm on All Nodes

Reference: https://developer.aliyun.com/mirror/kubernetes

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y --nogpgcheck kubelet-1.18.3 kubeadm-1.18.3 ipvsadm
systemctl enable kubelet && systemctl start kubelet

Note: Due to lack of official sync, GPG checks may fail. In such cases, use yum install -y --nogpgcheck kubelet kubeadm kubectl.

Install ipvsadm. After reboot, reload modules (add to /etc/rc.local for auto-load):

cat <<EOF >> /etc/rc.local
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4
EOF
chmod +x /etc/rc.local

II. High Availability Load Balancer via keepalived + haproxy

`1`	`yum -y install keepalived haproxy`

1. keepalived Configuration

On 192.168.1.208 (MASTER):

vrrp_script chk_ha {
    script "killall -0 haproxy"
    interval 2
    weight -20
}

vrrp_instance VI_1 {
    state MASTER
    interface bond0
    virtual_router_id 33
    mcast_src_ip 192.168.1.208
    priority 100
    nopreempt
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass hak8s
    }
    track_script {
        chk_ha
    }
    virtual_ipaddress {
        192.168.1.205/24   # VIP
    }
}

On 192.168.1.209 (BACKUP):

vrrp_script chk_ha {
    script "killall -0 haproxy"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state BACKUP
    interface bond0
    virtual_router_id 33
    mcast_src_ip 192.168.1.209
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass hak8s
    }
    track_script {
        chk_ha
    }
    virtual_ipaddress {
        192.168.1.205/24
    }
}

2. haproxy Configuration

Both 192.168.1.208 and 192.168.1.209 use identical config:

global
    log /dev/log local0
    maxconn 65535
    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    stats socket /var/lib/haproxy/stats
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    retries 3
    option redispatch
    option dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000

frontend k8s-apiserver
    bind *:8443
    mode tcp
    balance roundrobin
    server sd-cluster-04 192.168.1.207:6443 weight 5 check inter 2000 rise 2 fall 3
    server sd-cluster-05 192.168.1.208:6443 weight 3 check inter 2000 rise 2 fall 3
    server sd-cluster-06 192.168.1.209:6443 weight 3 check inter 2000 rise 2 fall 3

Now the HA load balancer is complete. To verify: stop keepalived on 192.168.1.208 and confirm the VIP floats over to 192.168.1.209.

III. Initialize Kubernetes Cluster with kubeadm

1. Initialize First Master Node

kubeadm-config.yaml:

# kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.1.208
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: sd-cluster-208
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "192.168.1.208:8443"
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.18.3
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/16
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

Run initialization:

# kubeadm init --config kubeadm-config.yaml --upload-certs
W0828 03:02:37.249435   17451 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.3
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [sd-cluster-208 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.208 192.168.1.205]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [sd-cluster-208 localhost] and IPs [192.168.1.208 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [sd-cluster-208 localhost] and IPs [192.168.1.208 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0828 03:02:43.787797   17451 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0828 03:02:43.789895   17451 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 23.015352 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
0adac55426d376e72f21ec3aee2465f754e78810700843cb75c270be26eaeaf1
[mark-control-plane] Marking the node sd-cluster-208 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node sd-cluster-208 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.1.205:8443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:7400f57a033f7527c80a4b015b54b4b6a88ccb2184ab9a6e39b709fb56e10486 \
    --control-plane --certificate-key 0adac55426d376e72f21ec3aee2465f754e78810700843cb75c270be26eaeaf1

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.205:8443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:7400f57a033f7527c80a4b015b54b4b6a88ccb2184ab9a6e39b709fb56e10486

Configure kubeconfig for master node:

1
2
3

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

2. Add New Master Node to Cluster

# kubeadm join 192.168.1.205:8443 --token abcdef.0123456789abcdef \
>     --discovery-token-ca-cert-hash sha256:7400f57a033f7527c80a4b015b54b4b6a88ccb2184ab9a6e39b709fb56e10486 \
>     --control-plane --certificate-key 0adac55426d376e72f21ec3aee2465f754e78810700843cb75c270be26eaeaf1
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [sd-cluster-209 localhost] and IPs [192.168.1.209 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [sd-cluster-209 localhost] and IPs [192.168.1.209 127.0.0.1 ::1]
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [sd-cluster-209 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.209 192.168.1.205]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
W0828 03:12:57.996811   48328 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0828 03:12:58.009390   48328 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0828 03:12:58.011318   48328 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
{"level":"warn","ts":"2021-08-28T03:13:15.215-0400","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"passthrough:///https://192.168.1.209:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node sd-cluster-209 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node sd-cluster-209 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

3. Add New Worker Node to Cluster

1
2

kubeadm join 192.168.1.205:8443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:7400f57a033f7527c80a4b015b54b4b6a88ccb2184ab9a6e39b709fb56e10486

4. Install Flannel Network Plugin

# wget http://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# sed -i 's#quay.io/coreos/flannel:v0.14.0#registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.14.0#g' kube-flannel.yml
# kubectl apply -f kube-flannel.yml -n kube-system
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
# kubectl get pod -n kube-system | grep flannel
kube-flannel-ds-gtdl8                    1/1     Running   0          24s
kube-flannel-ds-jtndz                    1/1     Running   0          24s
kube-flannel-ds-nd79b                    1/1     Running   0          24s

5. Install Ingress Controller

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v0.48.1

wget https://github.com/kubernetes/ingress-nginx/releases/download/helm-chart-3.35.0/ingress-nginx-3.35.0.tgz
tar zxvf helm-chart-3.35.0.tar.gz
cd ingress-nginx
vim values.yaml  # Update image to the domestic Alibaba mirror version above

# helm install ingress-nginx --namespace ingress-nginx . -n ingress-nginx
NAME: ingress-nginx
LAST DEPLOYED: Tue Aug 31 17:02:28 2021
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
Get the application URL by running these commands:
  export HTTP_NODE_PORT=$(kubectl --namespace ingress-nginx get services -o jsonpath="{.spec.ports[0].nodePort}" ingress-nginx-controller)
  export HTTPS_NODE_PORT=$(kubectl --namespace ingress-nginx get services -o jsonpath="{.spec.ports[1].nodePort}" ingress-nginx-controller)
  export NODE_IP=$(kubectl --namespace ingress-nginx get nodes -o jsonpath="{.items[0].status.addresses[1].address}")

  echo "Visit http://$NODE_IP:$HTTP_NODE_PORT to access your application via HTTP."
  echo "Visit https://$NODE_IP:$HTTPS_NODE_PORT to access your application via HTTPS."

An example Ingress that makes use of the controller:

  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

For new Ingress resources, refer to the following template:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-nginx
spec:
  rules:
    - host: "my-nginx.test.com"
      http:
        paths:
          - path: /
            backend:
              serviceName: my-nginx
              servicePort: 8080

✅ Kubernetes high-availability cluster deployed via kubeadm completed.

Add a user to a Kubernetes cluster

wzdushu@gmail.com (wanzi) — Tue, 31 Dec 2019 10:22:42 +0800

Previously, an Kubernetes cluster environment was set up using Ansible. The current requirement is to add a user for daily management, restricted to a specific namespace. Below are the steps:

Kubernetes Users

In Kubernetes, there are two types of users: ServiceAccounts and regular users (User). ServiceAccounts are managed by Kubernetes, while regular users are typically managed externally. Kubernetes does not store user lists—meaning user creation, modification, or deletion must be handled externally, without interacting with the Kubernetes API. Although Kubernetes does not manage users directly, it can recognize the identity of users making API requests. In fact, every API request to Kubernetes must be associated with an identity (either a User or a ServiceAccount), allowing us to assign permissions within the cluster to specific users.

When Kubernetes receives API requests from users, it commonly uses one of three authentication methods: client certificates, static token files, or static password files. Here, we focus only on certificate-based authentication.

Generate User Certificate

Prepare a Certificate Signing Request (CSR) file, which will be signed by the Kubernetes CA. The default path for the Kubernetes API server’s CA is /etc/kubernetes/pki/. This process generates cicd-admin-key.pem (private key) and cicd-admin.pem (certificate). For details on generating CSR files, refer to: https://wnote.com/post/linux-openssl-issue-private-certificate/

The username passed in the CSR file is cicd-admin.

# vim cicd-admin-csr.json
{
  "CN": "cicd-admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

# cd /etc/kubernetes/pki/
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes cicd-admin-csr.json | cfssljson -bare cicd-admin
# ls -l cicd-admin*
-rw-r--r-- 1 root root 1001 Dec 19 16:51 cicd-admin.csr
-rw-r--r-- 1 root root  224 Dec 19 16:50 cicd-admin-csr.json
-rw------- 1 root root 1675 Dec 19 16:51 cicd-admin-key.pem
-rw-r--r-- 1 root root 1387 Dec 19 16:51 cicd-admin.pem

User Permission Control (RBAC)

Create Role

In Kubernetes, RBAC roles come in two types: Role and ClusterRole. A ClusterRole is a special kind of Role:

A Role belongs to a specific namespace; a ClusterRole applies across the entire cluster, including all namespaces.
A ClusterRole can grant cluster-wide permissions (e.g., managing Node resources) and can access resources across all namespaces (using --all-namespaces). By default, Kubernetes includes a built-in ClusterRole named admin, which is usually sufficient and doesn’t require manual creation.

Create a role named cicd-admin in the cicd namespace:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  namespace: cicd
  name: cicd-admin
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]

Bind User to Role

subjects specifies the account type, which can be either User or ServiceAccount. Here, we bind the user cicd-admin. roleRef defines the role referenced by the RoleBinding.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cicd-admin-binding
  namespace: cicd
subjects:
- kind: User
  name: cicd-admin
  apiGroup: ""
roleRef:
  kind: Role
  name: admin
  apiGroup: ""

Alternatively, use kubectl to create the binding:

`1`	`kubectl create rolebinding cicd-admin-binding --role=admin --user=cicd-admin --namespace=cicd`

Or bind the user to the default ClusterRole admin, which will limit the user’s permissions to the cicd namespace:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cicd-admin-binding
  namespace: cicd
subjects:
- kind: User
  name: cicd-admin
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: admin
  apiGroup: ""

Client Configuration

Set Cluster Parameters

export KUBE_APISERVER="https://cicd-k8s.test.cn:8443"
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=cicd-admin.kubeconfig

Set Client Authentication Parameters

kubectl config set-credentials cicd-admin \
  --client-certificate=/etc/kubernetes/ssl/cicd-admin.pem \
  --client-key=/etc/kubernetes/ssl/cicd-admin-key.pem \
  --embed-certs=true \
  --kubeconfig=cicd-admin.kubeconfig

Set Context Parameters

kubectl config set-context kubernetes \
  --cluster=kubernetes \
  --user=cicd-admin \
  --namespace=cicd \
  --kubeconfig=cicd-admin.kubeconfig

Set Default Context

`1`	`kubectl config use-context kubernetes --kubeconfig=cicd-admin.kubeconfig`

Client Usage

Copy the kubeconfig file to override ~/.kube/config:

`1`	`cp cicd-admin.kubeconfig ~/.kube/config`

With this, the setup for adding a user and granting them authorized access to the Kubernetes cluster is complete.

Deploy traefik2.1 in kubernetes cluster

wzdushu@gmail.com (wanzi) — Tue, 17 Dec 2019 10:22:42 +0800

Architecture & Concepts

Traefik 2.x has a big change compared to 1.7.x architecture. As shown in the architecture diagram above, the main function is to support TCP protocol and add the concept of Router.

Here we use Traefik 2.1 deployed in the kubernetes cluster. Business access is requested to traefik Ingress through haproxy. The following are some concepts involved in the construction process:

EntryPoints: Traefik’s network entry, defining the port where the request is accepted (regardless of http or tcp)
CRD: Extension of Kubernetes API
IngressRouter: forwards incoming requests to services that can handle requests. In addition, Middlewares can dynamically update requests before forwarding requests
Middlewares: dynamically process request parameters before the request reaches the service, such as header or forwarding rules, etc.
TraefikService: If this type is defined in CRD, IngressRouter can directly reference it. It is located between IngressRouter and the service, similar to the Maesh architecture. It is more suitable for more complex scenarios and can be omitted in general.

kubernetes configuration

Configure SSL certificate

Because the business service uses https, configure the SSL certificate here first:

`1`	`kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=test.cn.pem --key=test.cn.key`

Configure cluster access control (RBAC)

apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik-ingress-controller
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller

rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions resources: - ingresses verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses/status verbs: - update - apiGroups: - middlewares verbs: - get - list - watch - apiGroups: - traefik.containo.us resources: - ingressroutes verbs: - get - list - watch - apiGroups: - traefik. containo.us resources: - ingressroutetcps verbs: - get - list - watch - apiGroups: - traefik.containo.us resources: - tlsoptions verbs: - get - list - watch - apiGroups: - traefik.containo.us resources: - traefikservices
verbs:
- get
- list
- watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller

roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: kube-system

TLS parameter configuration

TLS1.2 is configured by default, of course TLS1.3 can also be used

apiVersion: traefik.containo.us/v1alpha1
kind: TLSOption
metadata:
name: mytlsoption
namespace: kube-system

spec:
minversion: VersionTLS12 snistrict: true ciphersuites: - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - TLS_RSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 - T LS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ``` ## CRD configuration
IngressRoute, Middleware, TLSOption, IngressRouteTCP, and TraefikService are defined here, of which TraefikService is a new CRD added in version 2.1

```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutes.traefik.containo.us

spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRoute
plural: ingressroutes
singular: ingressroute
scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: middlewares.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1 names: kind: Middleware plural: middlewares singular: middleware scope: Namespaced --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: tlsoptions.traefik.containo.us spec: group: traefik.containo.us version: v1alpha1 names: kind: TLSOption plural: tlsoptions singular: tlsoption scope: Namespaced --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: ingressroutetcps.traefik.containo.us spec: group: traefik.containo.us version: v1alpha1 names: kind: IngressRouteTCP
plural: ingressroutetcps
singular: ingressroutetcp
scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: traefikservices.traefik.containo.us

spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: TraefikService
plural: traefikservices
singular: traefikservice
scope: Namespaced

TraefikService configuration

TraefikService is somewhat similar to Maesh in solving the calling logic between services, but Maesh relies on coredns; in addition, traefik service can also set the backend service weight and configure the traffic mirroring of the service.

Here we configure the traefik dashboard and rancher’s traefikservice type service. For other service configurations, you can refer to rancher’s traefik service here. The traefik service will forward requests to the kubernetes service type (in the previous section, we have created the rancher service through helm3). Here is just an example:

apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:
name: traefik-webui-traefikservice
namespace: kube-system

spec:
weighted:
services:
- name: traefik-ingress-service
weight: 1
port: 8080

---
apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:t
name: rancher-traefikservice
namespace: cattle-system

spec:
weighted:
services:
- name: rancher
weight: 1
port: 80

Deployment configuration

Configure k8s standard service
Create traefik configmap, and configure entrypoints and default SSL certificate
Deploy traefik ingress controller in Deployment mode

kind: Service
apiVersion: v1
metadata:
name: traefik-ingress-service
namespace: kube-system
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- protocol: TCP
port: 80
nodePort: 23456
name: http
- protocol: TCP
port: 443
nodePort: 23457
name: https
type: NodePort

---
apiVersion: v1
kind: ConfigMap
metadata:
name: traefik-conf
namespace: kube-system data: traefik.toml: | [global] checkNewVersion = false sendAnonymousUsage = false [log] level = "DEBUG" [api] dashboard = true [metrics.prometheus] buckets = [0.1,0.3,1.2,5.0] entryPoint = "metrics" [entryPoints] [entryPoints.http] address = ":80" [entryPoints.https] address = ":4 43" [tls.stores] [tls.stores.default] [tls.stores.default.defaultCertificate] certFile = "/config/tls/test.cn.crt" keyFile = "/config/tls/test.cn.key" --- kind: Deployment apiVersion: apps/v1 metadata: name: traefik namespace: kube-system labels: k8s-app: traefik-ingress-lb spec: replicas: 1 selector: matchLabels: k8s-app: traefik-ingress-lb template: metadata: labels: k8s-app: traefik-ingress-lb name: traefik-ingress-lb spec: serviceAccountName: traefik-ingress-controller terminationGracePeriodSeconds: 60 nodeSelector: node- role.kubernetes.io/traefik: "true" volumes: - name: ssl secret: secretName: traefik-cert - name: config configMap: name: traefik-conf containers: - image: traefik:v2.1.1 name: traefik-ingress-lb volumeMounts: - mountPath: "/config" name: "config" - mountPath: "/config/tls" name: "ssl" resources: limits: cpu: 1000m memory: 800Mi requests: cpu: 500m memory: 600Mi ports: - name: http containerPort: 80 hostPort: 80 - name: https containerPort: 443 hostPort: 443 args: - --entrypoints.http.Address=:8 0 - --entrypoints.https.Address=:443 - --api - --accesslog - --providers.file.directory=/config/ - --providers.file.watch=true - --ping=true - --providers.kubernetescrd ```` ## IngressRouter configuration * Define middleware to configure X-Forwarded-Proto header information for rancher service
* Configure rancher's ingressroute and use the default certificate

```yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: redirect-https
namespace: kube-system
spec:
redirectScheme:
scheme: https

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: http-default-router
namespace: kube-system
spec:
entryPoints:
- http
routes:
- match: HostRegexp(`{host:.+}`)
kind: Rule
services:
- name: traefik-ingress-service
kind: Service
namespaces: kube-system
port: 80
middlewares:
- name: redirect-https
tls:
options:
name: mytlsoption
namespaces: kube-system
certResolver: default

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-webui
namespace: kube-system
spec:
entryPoints:
- https
routes:
- match: Host(`traefik.test.cn`)
kind: Rule
services:
- name: api@internal
kind: TraefikService
namespaces: kube-system

Configure Rancher’s IngressRoute

Since I use rancher to manage the cluster here, the actual environment can be configured according to your own needs

Define middleware to configure the X-Forwarded-Proto header information for the rancher service
Configure rancher’s ingressroute and use the default certificate

apiVersion: traefik.containo.us/v1alpha1 kind: Middleware metadata: name: rancher-https-headers namespace: cattle-system spec: headers: customRequestHeaders: .test.cn`) kind: Rule services: - name: rancher kind: Service namespaces: cattle-system port: 80 middlewares: - name: rancher-https-headers namespaces: cattle-system tls: certResolver: default ```` ## Unified deployment of traefik2.1

```shell
kubectl apply -f 01-crd.yaml
kubectl apply -f 02-rbac.yaml
kubectl apply -f 03-tlsoption.yaml
kubectl apply -f 04-traefikservices.yaml #Not required
kubectl apply -f 05-traefik.yaml
kubectl apply -f 06-ingressrouter.yaml
kubectl apply -f 06-ingressrouter-rancher.yaml

For more configuration information, please visit my github repository: https://github.com/iwz2099/kubecase/tree/master/traefik/v2

Deploying a K8s cluster with kubeasz

wzdushu@gmail.com (wanzi) — Thu, 12 Dec 2019 10:22:42 +0800

Environment Preparation

Master nodes

1
2
3

172.16.244.14
172.16.244.16
172.16.244.18

Node nodes

1
2

172.16.244.25
172.16.244.27

Master node VIP: 172.16.243.13
Deployment tool: Ansible/kubeasz

Initialize Environment

Install Ansible

apt update
apt-get install ansible expect
git clone https://github.com/easzlab/kubeasz
cd kubeasz
cp * /etc/ansible/

1
2

ssh-keygen -t rsa -b 2048 # Generate key pair
./tools/yc-ssh-key-copy.sh hosts root 'rootpassword'

Prepare Binary Files

1
2

cd tools
./easzup -D # Downloads binaries to /etc/ansible/bin/ by default

Configure hosts file as follows:

[kube-master]
172.16.244.14
172.16.244.16
172.16.244.18

[etcd]
172.16.244.14 NODE_NAME=etcd1
172.16.244.16 NODE_NAME=etcd2
172.16.244.18 NODE_NAME=etcd3

# haproxy-keepalived
[haproxy]
172.16.244.14
172.16.244.16
172.16.244.18

[kube-node]
172.16.244.25
172.16.244.27

# [optional] load balance for accessing k8s from outside
[ex-lb]
172.16.244.14 LB_ROLE=backup EX_APISERVER_VIP=172.16.243.13 EX_APISERVER_PORT=8443
172.16.244.16 LB_ROLE=backup EX_APISERVER_VIP=172.16.243.13 EX_APISERVER_PORT=8443
172.16.244.18 LB_ROLE=master EX_APISERVER_VIP=172.16.243.13 EX_APISERVER_PORT=8443

# [optional] ntp server for the cluster
[chrony]
172.16.244.18

[all:vars]
# --------- Main Variables ---------------
# Cluster container-runtime supported: docker, containerd
CONTAINER_RUNTIME="docker"

# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
#CLUSTER_NETWORK="flannel"
CLUSTER_NETWORK="calico"

# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"

# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.68.0.0/16"

# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="10.101.0.0/16"

# NodePort Range
NODE_PORT_RANGE="20000-40000"

# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="cluster.local."

# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/opt/kube/bin"

# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"

# Deploy Directory (kubeasz workspace)
base_dir="/etc/ansible"

Deploy K8S Cluster

Initialize Configuration

1
2

cd /etc/ansible
ansible-playbook 01.prepare.yml

This step performs three main tasks:

1
2
3

chrony role: Synchronize time across cluster nodes [optional]
deploy role: Generate CA certificates, kubeconfig, kube-proxy.kubeconfig
prepare role: Distribute CA certificates, install kubectl client, configure environment

Install etcd Cluster

`1`	`ansible-playbook 02.etcd.yml`

Install Docker

`1`	`ansible-playbook 03.docker.yml`

Deploy Kubernetes Master

`1`	`ansible-playbook 04.kube-master.yml`

Deploy Kubernetes Node

`1`	`ansible-playbook 05.kube-node.yml`

Deploy Kubernetes Network (using Calico here)

`1`	`ansible-playbook 06.network.yml`

Deploy Ingress/K8s Dashboard/CoreDNS

Configure SSL certificate for Ingress; the default used in the repository is Traefik 1.7.12, which we plan to upgrade to version 2.0 later.

1
2

# kubectl create secret tls traefik-cert --key=test.cn.key --cert=test.cn.pem -n kube-system
secret/traefik-cert created

Deploy cluster add-ons

`1`	`ansible-playbook 07.cluster-addon.yml`

Remove Taint from Master Nodes

By default, after deployment, master nodes are tainted and excluded from scheduling:

# kubectl get node
NAME            STATUS                     ROLES    AGE   VERSION
172.16.244.14   Ready,SchedulingDisabled   master   91m   v1.16.2
172.16.244.16   Ready,SchedulingDisabled   master   91m   v1.16.2
172.16.244.18   Ready,SchedulingDisabled   master   91m   v1.16.2
172.16.244.25   Ready                      node     90m   v1.16.2
172.16.244.27   Ready                      node     90m   v1.16.2
# kubectl describe node 172.16.244.14 | grep Taint
Taints:             node.kubernetes.io/unschedulable:NoSchedule

Due to limited machine resources, we enable scheduling on master nodes:

# kubectl patch node 172.16.244.14 -p '{"spec":{"unschedulable":false}}'
# kubectl get node
NAME            STATUS   ROLES    AGE   VERSION
172.16.244.14   Ready    master   95m   v1.16.2
172.16.244.16   Ready    master   95m   v1.16.2
172.16.244.18   Ready    master   95m   v1.16.2
172.16.244.25   Ready    node     94m   v1.16.2
172.16.244.27   Ready    node     94m   v1.16.2

Deploy External Load Balancer (Keepalived + Haproxy)

`1`	`ansible-playbook roles/ex-lb/ex-lb.yml`

Deploy Rancher

Install Helm3

Refer to https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-rancher/

cd /opt/soft
wget https://get.helm.sh/helm-v3.0.1-linux-amd64.tar.gz
tar xf helm-v3.0.1-linux-amd64.tar.gz
cd linux-amd64/
cp helm /usr/local/bin/

Create Certificate

Since we use our own domain certificate, we create a certificate using Kubernetes secret. Alternatively, you can use cert-manager to issue a certificate for Rancher or use Let’s Encrypt.

`1`	`kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=test.cn.pem --key=test.cn.key`

Install Rancher

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update
helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=rancher-cicd.test.cn --set ingress.tls.source=secret

Check Ingress, Resources, and Deployment Status

# kubectl get ingress --all-namespaces
NAMESPACE       NAME      HOSTS                   ADDRESS   PORTS     AGE
cattle-system   rancher   rancher-cicd.test.cn             80, 443   20h
# kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available...
deployment "rancher" successfully rolled out
# kubectl -n cattle-system get deploy rancher
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
rancher   3/3     3            3           5m5s

At this point, the entire K8S cluster has been successfully deployed. If everything proceeds smoothly, the entire process should take around 10 minutes. The key is proper planning in advance.