Kubernetes on 丸子有记, 记录生活、分享技术，专注于运维架构、运维开发(devops)。

一周拿下CKA经验分享

wzdushu@gmail.com (wanzi) — Mon, 19 Aug 2024 10:10:42 +0800

对于云原生&k8s相关证书很早就想考一个，越早考试越好，工作的原因一直推脱到今年，眼看着它价格涨了两次（说到这里有点小肉疼了）。

最近终于有空腾出一周时间全身心备考了，由于自己工作生产环境一直使用的kubernetes，所以，这次我自己的计划是考前多吸收考点，多做大量考题；

下面总结整理一下最近一周我CKA考试的历程，希望对你有帮助。

CKA全面认识

1、CKA是什么？

CKA全称：Certified Kubernetes Administrator 是由CNCF推出的第一个基于kubernetes的认证，整个考试全部是实操题，且监考极其严格，所以CKA是全球公认的Kubernetes含金量最高的证书之一。

2、CKA证书有效期多长时间？

CKA证书的有效期为3年，但自2024年4月1日起，新认证的CKA证书有效期变更为2年。在2024年4月1日前通过考试并获得证书的学员，其证书仍然保持3年的有效期。

证书过期后需重新参加考试，并且需要满足考试的要求。考试的形式是在线考试，考试时间2小时，考试题目17道，满分100分，66分及格。

3、CKA有什么用？

对企业而言，Kubernetes认证服务供应商(KCSP)一般需要有3名CKA。对个人而言，获得CKA认证，是检测自己学习成果的一个重要指标。一来可以系统的学习知识，二来可以通过证书向企业证明自己的技术实力。

4、CKA、CKAD、CKS对比分析

其实CNCF基金会下有关k8s的认证主要这三个，三个证书侧重点不一样，可以根据自己实际情况选择。

CKA:Certified Kubernetes Administrator, 侧重于kubernetes系统管理，比如基本安装以及配置和管理生产级集群的能力,涉及到Kubernetes网络、存储、安全、维护、日志记录和监控、应用生命周期、故障排除、API对象原语等关键概念。
CKAD:Certified Kubernetes Application Developer, 侧重于Kubernetes设计、构建和部署云原生应用程序，比如使用Kubernetes核心原语来创建/迁移、配置、公开和观察可伸缩的应用程序。
CKS:Certified Kubernetes Security，侧重于Kubernetes安全，比如构建、部署和运行时保护基于容器的应用程序和Kubernetes平台安全。

通过备考三个证书考点和考题，CKA的知识点要涉及更全面相对更难一些，如果你考过了CKA再去考CKAD或者CKS，你会很容易考试通过，而里面的知识点有60-80%都是有关联的。

5、CKA主要考试那些知识点？

第一部分：集群架构，安装和配置(占比25%)：

管理基于角色的访问控制（RBAC）
使用Kubeadm安装基本集群
管理高可用性的Kubernetes集群
设置基础架构以部署Kubernetes集群
使用Kubeadm在Kubernetes集群上执行版本升级
实施etcd备份和还原

第二部分：工作负载和调度(占比15%)：

了解部署以及如何执行滚动更新和回滚
使用ConfigMaps和Secrets配置应用程序
了解如何扩展应用程序
了解用于创建健壮的、自修复的应用程序部署的原语
了解资源限制如何影响Pod调度
了解清单管理和通用模板工具

第三部分：服务和网络(占比20%):

了解集群节点上的主机网络配置
理解Pods之间的连通性
了解ClusterIP、NodePort、LoadBalancer服务类型和端点
了解如何使用入口控制器和入口资源
了解如何配置和使用CoreDNS
选择适当的容器网络接口插件

第四部分：存储(占比10%):

了解存储类、持久卷
了解卷模式、访问模式和卷回收策略
理解持久容量声明原语
了解如何配置具有持久性存储的应用程序

第五部分：故障排除(占比30%):

评估集群和节点日志
了解如何监视应用程序
管理容器标准输出和标准错误日志
解决应用程序故障
对群集组件故障进行故障排除
排除网络故障

CKA考前准备

报名缴费

报名地址：https://training.linuxfoundation.cn/certificates/1

价格比以前都贵了，建议等黑五的下单，大概优惠五折的样子。

另外，这里可以选择中文监考或者英文监考，建议选择中文，当然二者差别不大，考题即便是英文也能大概看明白。

下单成功后，登录后台，查看订单的地方会有一个考试券，把这个记录下来，后面会用到。

激活考试码

整个考试流程可以参考这里：https://training.linuxfoundation.cn/news/308

第一步，需要注册一个 Linux Foundation ID（LFID），注意自己名称，一定要和身份证、护照保持一致，也建议和实名认证保持一致，不然后面预约考试过不了。

注册和预约考试地址：https://trainingportal.linuxfoundation.org/

第二步，拿到之前的考试券，注册考试，正常情况下不注册考试，考试券只有1个月的有效期，注册考试券后只需要1年内考试就可以。

第三步，考试预约

预约考试时间：整个考试时间2个小时，建议选择上海时间，凌晨1点-2点考试，因为考试服务器都在海外，这个时间点不会很卡。
预约考试语言：考试语言和监考老师语言，这里直接选择中文即可。
电脑环境检测：考试前需要对考试系统环境进行检测，浏览器建议用chrome，不建议提前安装PSI，考试前半个小时安装PSI即可，考试前头一天，建议将电脑上的向日葵、Todesk、Vmware、lemon等这些禁用或者卸载掉，我这次就是因为这个耽误了十几分钟。

预约成功以后，就可以安心备考了。

考试环境注意事项

考前，建议备一个VPN，但是考试的时候建议不要用，除非你的网络特别卡，卡的都退出考试系统了，这个时候可以选择用VPN。

从考前到进入真正考试时间，需要如下五步准备：

第一步：PSI环境检测，check成功才能考试，一定一定要提前半个小时，这个会折腾20分钟；
第二步：人脸识别，需要用电脑摄像头进行拍照；
第三步：身份认证信息，需要拿身份证和护照对着摄像头拍照，这个考官会检查
第四步：考官会让你拿着笔记本摄像头展示四周环境，考试桌子上桌子下面都不能有任何东西，且四周1米距离内不能有任何东西，身边不能有任何人谈话，手机建议放到1米外以上距离，让考官看到最好。
第五步：考官会让你举起手检查手里是否夹东西，耳朵两侧是否有耳塞等等。

上面都OK了，考官会给你确认是否可以考试，你回答OK即可，从这个时间才算考试时间。

考试时间&考题&考试结果

为什么考试时间需要单独拎出来说呢，就是因为考试时间总共就2个小时，17道题，说长也长，说短也短。

考试系统服务器再美国，经常会遇到卡顿现象，所以做一道题有的时候可能会很长。

对于备考时间非常短的同学，建议再电脑桌子对面放置一个钟表或者自己带一个手表，因为考试系统里时间是美国时间，很容易过点。

另外，关于考试题，建议刚开始先选择简单的，因为有60-70%都可以通过命令行解决掉；

每做一道题，切记一定要执行kubectl config context xxxxk8s选择考题指定集群；

对于比如PV/PVC、Netpol需要手写Yaml这些，建议后做，集群升级可以放到最后再做。

一般考试结束以后，成绩会在24小时后出来，考试通过分数是66分；如果因为考试环境问题无法进入考试，建议尽快联系监考老师；考试结果不理想，还有一次考试机会，后面直接重新约考即可。

考题解析

下面是一些从网上搜集的八道考题，从我个人角度觉得稍微有一些难度的考题，这些考题和实际考题类型基本相似，大家不妨自己做实验体验一下，所以，能手写k8s资源对象yaml文件，就手写，这样才能节省更多时间。

1、创建一个名为deployment-clusterrole且仅允许创建以下资源类型的新 ClusterRole：Deployment，StatefulSet，DaemonSet；
在现有的namespace app-team1中创建一个名为cicd-token的新ServiceAccount限于namespace app-team1中，将新的 ClusterRole deployment-clusterrole绑定到新的ServiceAccount

2、通过pod label name=cpu-loader，找到运行时占用大量 CPU 的 pod，并将占用 CPU 最高的 pod 名称写入文件 /opt/KUTR000401/KUTR00401.txt（已
存在）。

3、在现有的namespace my-app中创建一个名为allow-port-from-namespace的新NetworkPolicy；确保新的 NetworkPolicy 允许 namespace echo 中的 Pods 连接到 namespace
my-app 中的 Pods 的 9000 端口；进一步确保新的 NetworkPolicy：不允许对没有在监听 端口 9000 的 Pods 的访问，不允许非来自 namespace echo 中的 Pods 的访问。

4、请重新配置现有的deployment front-end以及添加名为http的端口规范来公开现有容器nginx的端口80/tcp；创建一个名为front-end-svc的新service，以公开容器端口http；配置此service，以通过各个Pod所在的节点上的NodePort来公开他们。

5、如下创建一个新的 nginx Ingress 资源：名称: ping，Namespace: ing-internal，使用服务端口5678在路径/hello上公开服务hello，可以使用以下命令检查服务hello的可用性，该命令应返回hello：curl -kL <INTERNAL_IP>/hello

6、创建一个新的 PersistentVolumeClaim，名称: pv-volume，Class: csi-hostpath-sc，容量: 10Mi；创建一个新的 Pod，来将PersistentVolumeClaim作为volume 进行挂载：名称：web-server，Image：nginx:1.16，挂载路径：/usr/share/nginx/html，配置新的Pod，以对volume具有 ReadWriteOnce权限。最后，使用 kubectl edit 或 kubectl patch 将 PersistentVolumeClaim 的容量扩展为70Mi，并记录此更改。

7、使用busybox Image来将名为sidecar的sidecar容器添加到现有的Pod 11-factor-app中；新的 sidecar 容器必须运行以下命令：/bin/sh -c tail -n+1 -f /var/log/11-factor-app.log；使用挂载在/var/log 的 Volume，使日志文件11-factor-app.log可用于sidecar 容器；除了添加所需要的 volume mount 以外，请勿更改现有容器的规格

8、现有的Kubernetes集群正在运行版本1.30,仅将master节点上的所有Kubernetes 控制平面和节点组件升级到版本 1.31,确保在升级之前drain master节点，并在升级后uncordon master节点.
可以使用以下命令，通过 ssh 连接到 master 节点：ssh master01
可以使用以下命令，在该 master 节点上获取更高权限：
sudo -i
另外，在主节点上升级 kubelet 和 kubectl。 请不要升级工作节点，etcd，container 管理器，CNI 插件， DNS 服务或任何其他插件。

CKA考试总结

最后，放一张CKA证书收尾：

通过这次CKA认证考试，整体来看难度适中，我觉得实操Kubernetes还是很重要的，尤其对于DevOps工程师；

其实，要想高分通过，没有任何捷径可言，就是多吸收上面列举的知识点，多练习，多实操。

最后，我想说的是考试只是一种形式，目的是为了证明自己熟练掌握k8s的技能，最终将这些技能应用到实际工作当中。

利用阿里云开源方案实现单GPU卡多人使用

wzdushu@gmail.com (wanzi) — Fri, 30 Jun 2023 18:10:42 +0800

最近新上了一个AI项目，主要是给大学提供AI在线实验，项目也采购了GPU服务器，但是只有一张Nvidia Tesla T4卡，需要支持多个学生同时在线做实验呢。

在线实验系统目前运行在Kubernetes上，因此，需要考虑k8s环境下，GPU共享，之前也测试过阿里云GPU卡共享方案，这里就直接记录使用步骤即可：

kubernetes集群版本：1.23.1

调整K8S调度器

从1.23.+起，由于kube-scheduler调度策略有调整，所以之前的部署方式就不好时了。

参考这里：https://kubernetes.io/docs/reference/scheduling/policies/

1
2

git clone https://github.com/AliyunContainerService/gpushare-scheduler-extender.git
cp config/scheduler-policy-config.yaml /etc/kubernetes/

由于我这里是通过kubeasz安装的k8s，所以Kubeconfig放在这里/etc/kubernetes/kubelet.kubeconfig，需要更新 /etc/kubernetes/scheduler-policy-config.yaml如下：

---
apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /etc/kubernetes/kube-scheduler.kubeconfig
extenders:
- urlPrefix: "http://192.168.233.101:32766/gpushare-scheduler"
  filterVerb: filter
  bindVerb: bind
  enableHTTPS: false
  nodeCacheCapable: true
  managedResources:
  - name: aliyun.com/gpu-mem
    ignoredByScheduler: false
  ignorable: false

修改Kube-scheduler启动启动参数，/etc/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/opt/kube/bin/kube-scheduler \
  --authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --bind-address=0.0.0.0 \
  --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --config=/etc/kubernetes/scheduler-policy-config.yaml \
  --leader-elect=true \
  --v=2
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

1
2

systemctl  daemon-reload
systemctl  restart  kube-scheduler.service

安装调度器扩展器

给GPU机器打上label:

kubectl label node mynode gpushare=true

1
2

cd gpushare-scheduler-extender/config
kubectl apply -f gpushare-schd-extender.yaml

安装插件

1
2

kubectl create -f https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-rbac.yaml
kubectl create -f https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-ds.yaml

安装完成以后如下

1
2
3

# kubectl get pod   -n kube-system |grep gpushare
gpushare-device-plugin-ds-j8blj              1/1     Running   0                30h
gpushare-schd-extender-74796c5f64-7g4bl      1/1     Running   0                29h

测试GPU

修改samples/1.yaml如下：

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-1
  labels:
    app: binpack-1

spec:
  replicas: 1

  selector: # define how the deployment finds the pods it mangages
    matchLabels:
      app: binpack-1

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-1

    spec:
      containers:
      - name: binpack-1
        image: cheyang/gpu-player:v2
        resources:
          limits:
            # GiB
            aliyun.com/gpu-count: 1
            aliyun.com/gpu-mem: 5

# kubectl apply -f  samples/1.yaml
deployment.apps/binpack-1 created
# kubectl get pod  |grep binpack
binpack-1-9995bdf69-pk2d4                 1/1     Running   0             12s
# kubectl logs -f binpack-1-9995bdf69-pk2d4
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=5
2023-06-30 09:40:50.890296: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2023-06-30 09:40:50.976283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:03:00.0
totalMemory: 14.75GiB freeMemory: 14.66GiB
2023-06-30 09:40:50.976313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:03:00.0, compute capability: 7.5)

通过日志看，GPU共享已经成功，且申请了5G显存。

当然你也可以通过kubectl扩展工具kubectl-inspect-gpushare来查看:

# cd /usr/bin/
# wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare
# chmod u+x /usr/bin/kubectl-inspect-gpushare
# kubectl-inspect-gpushare
NAME             IPADDRESS        GPU0(Allocated/Total)  GPU Memory(GiB)
192.168.233.101  192.168.233.101  9/14                   9/14
--------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
9/14 (64%)

至此，GPU卡共享已经完成，剩下就是通过tensorflow来调度我的GPU卡了。

参考文档：

https://github.com/AliyunContainerService/gpushare-scheduler-extender/blob/master/docs/install.md

kind构建轻量级kubernetes集群

wzdushu@gmail.com (wanzi) — Thu, 13 Apr 2023 18:22:42 +0800

最近在复习Golang，写了一个web应用，本地跑通以后想在k8s集群测试一番，苦于机器配置原因，搭建一套完整的k8s集群还是有一些力不从心，记得之前有朋友说过docker里也可以跑k8s，于是这就折腾了一番。

今天的主角是kind,那么kind是什么呢？kind可以用来干什么呢？

一、kind介绍

kind是Kubernetes In Docker缩写，一个使用docker容器节点来运行本地的kubernetes集群工具，kind主要用于测试kubernetes本身，适用于本地开发或者CI。

对于kind目前几乎支持kubernetes官方所有版本，对于运行时目前仅支持docker，未来会逐步支持CRI里常用运行时，相信未来一些日子，会慢慢支持conterned。

其实，kind管理集群过程，是在使用kubeadm、kustomize等开源组件来进行工作。

废话不多说，下面我们就快速安装使用kind。

二、kind安装

由于我系统是mac osx所以，我这里直接使用brew安装：

# brew install kind
==> Downloading https://mirrors.ustc.edu.cn/homebrew-bottles/kind-0.17.0.arm64_monterey.bottle.tar.gz
Already downloaded: /Users/wanzi/Library/Caches/Homebrew/downloads/bcd419997297730492f5cebc36be86fb51f21061a6ddb2e066e1e4d8ad33ddf3--kind-0.17.0.arm64_monterey.bottle.tar.gz
==> Pouring kind-0.17.0.arm64_monterey.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions
==> Summary
🍺  /opt/homebrew/Cellar/kind/0.17.0: 8 files, 8.7MB
==> Running `brew cleanup kind`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
# kind version
kind v0.17.0 go1.19.2 darwin/arm64

对于linux安装：

1
2
3

curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.17.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

当然，如果你是windows，也可以使用Chocolatey安装；或者使用二进制方式，这些官方都支持。

三、创建第一个集群

在操作之前，docker和kubectl是我们需要提前准备，具体安装我这里就先忽略。

# kind help
kind creates and manages local Kubernetes clusters using Docker container 'nodes'

Usage:
  kind [command]

Available Commands:
  build       Build one of [node-image]
  completion  Output shell completion code for the specified shell (bash, zsh or fish)
  create      Creates one of [cluster]
  delete      Deletes one of [cluster]
  export      Exports one of [kubeconfig, logs]
  get         Gets one of [clusters, nodes, kubeconfig]
  help        Help about any command
  load        Loads images into nodes
  version     Prints the kind CLI version

Flags:
  -h, --help              help for kind
      --loglevel string   DEPRECATED: see -v instead
  -q, --quiet             silence all stderr output
  -v, --verbosity int32   info log verbosity, higher value produces more output

Use "kind [command] --help" for more information about a command.

对于kind help命令，我们知道kind支持build,create,delete,get,load等操作，对于子命令可以通过-h获取更多讯息。

kind创建一个集群，在创建集群之前我们需要确保~/.kube目录存在，因为创建过程中，kind会将集群API地址、证书等信息写入kubeconfig。

# kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

# kubectl get node
NAME                 STATUS   ROLES           AGE     VERSION
kind-control-plane   Ready    control-plane   9m21s   v1.25.3
# kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-565d847f94-kp6l7                     1/1     Running   0          9m15s
kube-system          coredns-565d847f94-ml28b                     1/1     Running   0          9m15s
kube-system          etcd-kind-control-plane                      1/1     Running   0          9m32s
kube-system          kindnet-hfsc8                                1/1     Running   0          9m15s
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          9m31s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          9m33s
kube-system          kube-proxy-m9zp5                             1/1     Running   0          9m15s
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          9m33s
local-path-storage   local-path-provisioner-684f458cdd-d7f29      1/1     Running   0          9m15s

这样我们第一个集群就用kind创建完成。

四、集群操作：

1、kind自定义集群名称：

# kind create cluster --name ci-cluster
Creating cluster "ci-cluster" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-ci-cluster"
You can now use your cluster with:

2、切换集群：

# kubectl cluster-info --context kind-ci-cluster
Kubernetes control plane is running at https://127.0.0.1:56527
CoreDNS is running at https://127.0.0.1:56527/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Have a nice day! 👋

3、查看kind所有集群

1
2
3

# kind get clusters
ci-cluster
kind

4、导入镜像

默认kind node节点是无法读取宿主机镜像，需要手动导入到kind

node节点：

# kind load docker-image --name ci-cluster --nodes  ci-cluster-control-plane  traefik:v2.9.5
# docker exec -it ci-cluster-control-plane bash
root@ci-cluster-control-plane:/# crictl images  |grep traefik
docker.io/library/traefik                  v2.9.5               a1252ce6bfaaa       132MB

五、集群配置：

1、多节点集群

对于前面，我们介绍了kind创建集群，默认集群只有一个节点，下面介绍如何创建多个节点。

创建config.yaml内容如下：

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

重新创建集群

# kind create cluster --config config.yaml
# kubectl get node
NAME                 STATUS   ROLES           AGE    VERSION
kind-control-plane   Ready    control-plane   110s   v1.25.3
kind-worker          Ready    <none>          74s    v1.25.3
kind-worker2         Ready    <none>          88s    v1.25.3

kind本身也可以支持多master高可用部署，这里我主要测试就不简单介绍。

### 2、自定义kubernetes集群版本

```Yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315

3、挂载宿主机目录到节点容器持久化存储

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/files
    containerPath: /files
  - hostPath: /Users/wanzi/tools/kind/other-files/
    containerPath: /other-files
    readOnly: true
    selinuxRelabel: false
    propagation: None

4、端口映射

宿主机端口80映射到nodePort 30950，kind节点containerPort需要和Service的nodePort端口相同

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30950
    hostPort: 80

kind: Pod
apiVersion: v1
metadata:
  name: foo
  labels:
    app: foo
spec:
  containers:
  - name: foo
    image: hashicorp/http-echo:0.2.3
    args:
    - "-text=foo"
    ports:
    - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: foo
spec:
  type: NodePort
  ports:
  - name: http
    nodePort: 30950
    port: 5678
  selector:
    app: foo

5、自定义label

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  extraPortMappings:
  - containerPort: 30950
    hostPort: 80
  labels:
    tier: frontend
- role: worker
  labels:
    tier: backend

6、kubeadm自定义配置：

kind用kubeadm配置集群，他会在第一个控制平面节点上运行kubeadm init,我们可以对它进行自定义：

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label=true"

如果你想做更多的定制，有四种配置类型可供选择kubeadm init

InitConfiguration
ClusterConfiguration
KubeProxyConfiguration
KubeletConfiguration

例如，我们可以使用 kubeadm ClusterConfiguration ( spec ) 覆盖 apiserver 标志:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        extraArgs:
          enable-admission-plugins: NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

在 KIND管理HA的集群中配置的每个附加节点上，KIND运行kubeadm join可以使用JoinConfiguration自定义配置：

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label2=true"
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label3=true"

7、配置ingress

允许本地主机通过端口80/443向Ingress控制器发出请求，并配置控制器特定label

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP

`1`	`kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml`

六、完整集群配置

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
#增加多结点集群,并自定义node的集群版本
- role: control-plane
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  labels:
    app: front
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/wwwroot
    containerPath: /wwwroot
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  labels:
    app: backend
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/wwwroot
    containerPath: /wwwroot
networking:
  #自定义配置APIServer和网络
  apiServerAddress: "127.0.0.1"
  apiServerPort: 6443
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
  #disableDefaultCNI: true #默认CNI插件为kindnetd，也可以禁用使用其他插件比如calico
  kubeProxyMode: "ipvs" #设置kubeproxy模式为lvs,如果要禁用可以设置none

至此，使用kind来快速构建集群算是告一段落，平常就可以用它跑测试，验证功能了。

参考文档： https://kind.sigs.k8s.io https://github.com/kubernetes-sigs/kind

解决k8s下nginx文件上传限制和504网关超时

wzdushu@gmail.com (wanzi) — Mon, 08 Nov 2021 17:22:42 +0800

最近业务使用k8s集群经常有两个问题需要解决，这里记录一下：

前端页面上传文件1M限制
前端页面发送POST请求到后端，出现504超时

第一个问题主要解决方法：nginx默认上传大小为1M，nginx配置文件http,server, location区域增加如下内容：

`1`	`client_max_body_size 800M;`

第二个问题主要解决方法：nginx默认读取后端超时时间为1分钟，nginx配置文件http,server, location区域增加如下内容：

1
2
3

proxy_connect_timeout 120;
proxy_read_timeout  900;
proxy_send_timeout 900;

最终，nginx配置如下：

server {
    listen 80;
    server_name web_server;
    location / {
        root /usr/share/nginx/html/;
        index index.htm index.html;
        try_files $uri $uri/ /index.html;
    }
    location /business-operation {
        proxy_pass http://business-operat.common:5000;
        
        proxy_ignore_client_abort on; 
        keepalive_timeout 900s;
        client_body_timeout 900s;
        proxy_connect_timeout 120s;
        proxy_read_timeout  900s;
        proxy_send_timeout 900s;
        client_max_body_size 800m;
        proxy_pass_header Authorization;
        proxy_pass_header WWW-Authenticate;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
    }
}

最后，以上nginx配置调整以后，还需要在创建ingress的时候，注解里带上这些信息：

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "800M"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '120'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '900'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '900'
  name: enterprise-mark
  namespace: frontend
spec:
  rules:
    - host: enterprise-marketing.xxxx.com
      http:
        paths:
          - backend:
              serviceName: enterprise-mark
              servicePort: 80
            path: /
status:
  loadBalancer:
    ingress:
      - ip: 39.107.xxx.xxx

阿里云ACK同时支持公网和内网SLB

wzdushu@gmail.com (wanzi) — Thu, 21 Oct 2021 17:22:42 +0800

一、背景

有一套ACK集群
成功部署Nginx ingress controller并绑定了公网SLB

注意：通过阿里云容器服务管理控制台创建的Kubernetes集群在初始化时会自动部署一套Nginx Ingress Controller，默认其挂载在公网SLB实例上.

二、配置

1、创建内网LB

阿里云控制台，创建内网SLB并绑定VPC

2、配置Nginx ingress controller

# my-nginx-ingress-slb-intranet.yaml
# intranet nginx ingress slb service
apiVersion: v1
kind: Service
metadata:
  # 这里服务取名为nginx-ingress-lb-intranet。
  name: nginx-ingress-lb-intranet
  namespace: kube-system
  labels:
    app: nginx-ingress-lb-intranet
  annotations:
    # 指明SLB实例地址类型为私网类型。
    service.beta.kubernetes.io/alicloud-loadbalancer-address-type: intranet
    # 修改为您的私网SLB实例ID。
    service.beta.kubernetes.io/alicloud-loadbalancer-id: <YOUR_INTRANET_SLB_ID>
    # 是否自动创建SLB端口监听（会覆写已有端口监听），也可手动创建端口监听。
    #service.beta.kubernetes.io/alicloud-loadbalancer-force-override-listeners: 'false'
spec:
  type: LoadBalancer
  # route traffic to other nodes
  externalTrafficPolicy: "Cluster"
  ports:
  - port: 80
    name: http
    targetPort: 80
  - port: 443
    name: https
    targetPort: 443
  selector:
    # select app=ingress-nginx pods
    app: ingress-nginx

创建service资源：

`1`	`kubectl apply -f my-nginx-ingress-slb-intranet.yaml`

获取service资源：

1
2
3

# kubectl -n kube-system get svc | grep nginx-ingress-lb
nginx-ingress-lb            LoadBalancer   172.21.15.148   39.107.xxx.xxx   80:32076/TCP,443:30803/TCP     433d
nginx-ingress-lb-intranet   LoadBalancer   172.21.5.0      172.17.193.181   80:32282/TCP,443:30507/TCP     1d

通过Ingress对外暴露服务时，即可以通过公网SLB来访问该服务，同一个VPC下的其他服务也可以直接通过私网SLB来访问该服务。

3、创建ingress服务配置完ingress controller，ingress创建无须特殊配置，和以前创建ingress一摸一样，如下例子：

# Source: prometheusalert/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: RELEASE-NAME-prometheusalert
  labels:
    app.kubernetes.io/name: prometheusalert
    helm.sh/chart: prometheusalert-1.0.0
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/version: "1.2.0"
    app.kubernetes.io/managed-by: Helm
spec:
  rules:
    - host: "palert.con.sdi"
      http:
        paths:
          - path: /
            backend:
              serviceName: RELEASE-NAME-prometheusalert
              servicePort: 8080

对于内网域名直接解析到内网SLB地址（这里172.17.193.181）即可。

参考：阿里云文档

利用Prometheusalert打造Prometheus告警消息聚合

wzdushu@gmail.com (wanzi) — Sat, 18 Sep 2021 17:22:42 +0800

部署prometheusalert

git clone https://github.com/feiyu563/PrometheusAlert.git
cd PrometheusAlert/example/helm/prometheusalert
#更新config/app.conf设置登陆用户信息，并配置数据库信息
helm install -n monitoring .

创建企业微信群机器人

企业微信群以后，点击企业微信群，右键“添加群机器人”，创建群机器人会得到一个机器人的webhook地址，记录该地址备用。

Prometheus接入配置

配置alertmanager

由于我这里prometheus采用https://github.com/prometheus-operator/kube-prometheus.git 这里直接修改alertmanager-secret.yaml配置文件即可，将上面的webhook配置在receivers里即可：

apiVersion: v1
data: {}
kind: Secret
type: Opaque
metadata:
  name: alertmanager-main
  namespace: monitoring
stringData:
  alertmanager.yaml: |-
    global:
      resolve_timeout: "5m"
      #email
      smtp_smarthost: 'smtp.exmail.qq.com:465'                   # 邮箱smtp服务器代理
      smtp_from: 'devops_alert@xxx.com'                       # 发送邮箱名称
      smtp_auth_username: 'devops_alert@xxx.com'              # 邮箱名称
      smtp_auth_password: '1DevSop#'                             # 授权密码
      smtp_require_tls: false                            # 不开启tls 默认开启
    inhibit_rules:
    - equal:
      - namespace
      - alertname
      - instance
      source_match:
        severity: critical
      target_match_re:
        severity: warning|info
    - equal:
      - namespace
      - alertname
      - instance
      source_match:
        severity: warning
      target_match_re:
        severity: info
        receivers:
    - name: default
      email_configs:            # 邮箱配置
      - to: "xxxx@163.com"   # 接收警报的email配置
    - name: jf-k8s-node
      webhook_configs:
      - url: "http://palert.con.sdi/prometheusalert?type=wx&tpl=jf-k8s-node&wxurl=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=0da1f944-9a4e-4221-8036-f322351f7bd8"

配置Prometheus rules

这里举例磁盘告警规则

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: external-node.rules
  namespace: monitoring
spec:
  groups:
  - name: k8s-node.rules
    rules:
    - alert: 磁盘容量
      expr: 100-(node_filesystem_avail_bytes{instance=~'39.100.*',fstype=~"ext4|xfs|fuse.glusterfs"}/node_filesystem_size_bytes {fstype=~"ext4|xfs|fuse.glusterfs"}*100) > 96
      for: 1m
      labels:
        severity: critical
        app: node-alert
      annotations:
        summary: "{{$labels.mountpoint}} 磁盘分区使用率过高！"
        description: "{{$labels.mountpoint }} 磁盘分区使用大于90%(目前使用:{{$value}}%)"

定制消息模版

对于以上报警规则，默认情况下alertmanger有自己的告警消息模版不是很好用，这里我们统一使用prometheusalert的告警模版进行配置。

根据官方文档，我们新建模版，并指定模版类型为企业微信，模版用途Prometheus，模版内容如下：

{{ $var := .externalURL}}{{ range $k,$v:=.alerts }}
{{if eq $v.status "resolved"}}
#### [Prometheus恢复信息]({{$v.generatorURL}})
##### <font color="#FF0000">告警名称</font>：[{{$v.labels.alertname}}]({{$var}})
##### <font color="#FF0000">告警级别</font>：{{$v.labels.severity}}
##### <font color="#FF0000">触发时间</font>：{{GetCSTtime $v.startsAt}}
##### <font color="#02b340">恢复时间</font>：{{GetCSTtime $v.endsAt}}
##### <font color="#FF0000">故障实例</font>：{{$v.labels.instance}}
##### <font color="#FF0000">告警详情</font>：{{$v.annotations.description}}
{{else}}
#### [Prometheus告警信息]({{$v.generatorURL}})
##### <font color="#FF0000">告警名称</font>：[{{$v.labels.alertname}}]({{$var}})
##### <font color="#FF0000">告警级别</font>：{{$v.labels.severity}}
##### <font color="#FF0000">触发时间</font>：{{GetCSTtime $v.startsAt}}
##### <font color="#FF0000">故障实例</font>：{{$v.labels.instance}}
##### <font color="#FF0000">告警详情</font>：{{$v.annotations.description}}
{{end}}
{{ end }}

当磁盘空间告警以后，实际效果如下：

至此，用prometheusalert接受Prometheus告警消息，转发给企业微信群机器人配置完毕。

另外，对于多集群多消息源，prometheusalert的强大就体现出来了。

制定kubernetes集群备份策略

wzdushu@gmail.com (wanzi) — Sat, 11 Sep 2021 17:22:42 +0800

对于备份，每家互联网公司技术人员都要去做的一件事儿，当然我们也不例外，今天我主要针对生产环境kubernetes集群制定一些自己的策略，这里分享给大家。

我这里kubernetes备份目的主要为了防止如下情况出现：

防止误操作删除集群中某一个namespace
防止误操作导致集群中某一个资源出现异常，比如deployment、configmap等
防止误操作删除集群部分资源对象
防止etcd数据丢失

备份ETCD

备份etcd，防止k8s集群出现了集群级别故障或etcd数据丢失情况，导致整个集群不可用的情况，这种情况只能通过恢复集群恢复业务。

备份etcd脚本如下：

#!/bin/bash
#ENDPOINTS="https://192.168.1.207:2379,https://192.168.1.208:2379,https://192.168.1.209:2379"
ENDPOINTS="127.0.0.1:2379"
CACERT="/etc/kubernetes/pki/etcd/ca.crt"
CERT="/etc/kubernetes/pki/etcd/server.crt"
KEY="/etc/kubernetes/pki/etcd/server.key"
DATE=`date +%Y%m%d-%H%M%S`
BACKUP_DIR="/home/centos/hostpath/backups/k8s/etcd"
ETCDCTL_API=3 /usr/local/bin/etcdctl --cacert=${CACERT} --cert=${CERT} --key=${KEY}  --endpoints="${ENDPOINTS}" snapshot save ${BACKUP_DIR}/k8s-snapshot-${DATE}.db
find  $BACKUP_DIR/ -type f -mtime +20 -exec rm -f {} \;

cron任务计划：

`1`	`50 21 * * * /bin/bash /home/centos/hostpath/backups/k8s/etcdv3-bak.sh`

minio对象存储服务搭建

由于我们的存储集群采用的GlusterFS搭建，因此，这里我只能使用minio搭建对象存储，底层采用GlusterFS文件系统；如果你使用阿里云OSS备份你集群资源，请忽略这一步，可以移驾：https://github.com/AliyunContainerService/velero-plugin

由于minio在k8s环境下搭建需要提供对应存储pv/pvc，这里为了简单，直接启动docker方式：

version: '2.0'
services:
  minio:
    image: minio/minio:latest
    container_name: minio
    ports:
      - "39000:9000"
      - "39001:9001"
    restart: always
    command: server --console-address ':9001' /data
    environment:
      MINIO_ACCESS_KEY: admin
      MINIO_SECRET_KEY: adminSD#123
    logging:
      options:
        max-size: "1000M" # 最大文件上传限制
        max-file: "100"
      driver: json-file
    volumes:
      - /home/centos/hostpath/backups/k8s/velero:/data # 映射文件路径
    networks:
      - minio

networks:
  minio:
    ipam:
      config:
      - subnet: 10.210.1.0/24
        gateway: 10.210.1.1

打开浏览器输入如下地址和账户信息，就可以通过web控制台管理minio对象存储了

1
2
3

minio web：http://192.168.1.214:39001
minio admin：admin
minio admin passwd：adminSD#123

velero备份之安装客户端

`1`	`brew install velero`

创建文件credentials-velero内容如下,方便后边创建velero server端连接对象存储使用：

1
2
3

[default]
aws_access_key_id = admin
aws_secret_access_key = adminSD#123

velero备份之k8s部署velero

# velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.2.0 \
    --bucket k8s-jf \
    --secret-file ./credentials-velero \
    --use-volume-snapshots=false \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.1.214:39000

velero备份之velero命令

备份,查看,删除操作

#备份集群ingress-nginx namespace下资源:
velero backup create ingress-nginx-backup --include-namespaces ingress-nginx

#查看备份结果
velero backup describe ingress-nginx-backup
velero backup logs ingress-nginx-backup

#删除备份
velero delete backup ingress-nginx-backup

#备份非ingress-nginx和test命名空间下的资源：
velero backup create k8s-full-test-backup --exclude-namespaces ingress-nginx,test

#备份特定资源类型
velero backup create kube-system-backup --include-resources pod,secret

#--confirm 直接删除备份，无需确认：
velero backup delete kube-system-backup --confirm

#备份带pv pod
velero backup create pvc-backup  --snapshot-volumes --include-namespaces test-velero

注意：

–include-resources备份指定资源类型, –exclude-resources指定排除某些资源类型

定时备份：

# 每六个小时备份一次
velero create schedule ${SCHEDULE_NAME} --schedule="0 */6 * * *"

# 每六个小时使用every备份一次
velero create schedule ${SCHEDULE_NAME} --schedule="@every 6h"

# 创建一个web命名空间的天备份
velero create schedule ${SCHEDULE_NAME} --schedule="@every 24h" --include-namespaces web

# 创建一个周备份，持续时间为90天。
velero create schedule ${SCHEDULE_NAME} --schedule="@every 168h" --ttl 2160h0m0s

说明：–ttl可以指定backup的生存周期，在ttl超时后，backup会被定期清理,ttl默认30天

备份恢复：

#从backup创建restore
velero restore create ${RESTORE_NAME} --from-backup ${BACKUP_NAME}

# 从backup创建restore，restore默认名为 ${BACKUP_NAME}-<timestamp>
velero restore create --from-backup ${BACKUP_NAME}

# 从schedule最新一次的backup创建restore
velero restore create --from-schedule ${SCHEDULE_NAME}

# 指定backup中的某些资源创建restore
velero restore create --from-backup backup-2 --include-resources pod,secret

# 恢复集群所有备份，（对已经存在的服务不会覆盖）
velero restore create --from-backup all-ns-backup

# 仅恢复default nginx-example命名空间
velero restore create --from-backup all-ns-backup --include-namespaces default,nginx-example 

# 将test-velero 命名空间资源恢复到test-velero-1下面
velero restore create restore-for-test --from-backup everyday-1-20210203131802 --namespace-mappings test-velero:test-velero-1

查看备份

velero  get  backup   #备份查看
velero  get  schedule #查看定时备份
velero  get  restore  #查看已有的恢复
velero  get  plugins  #查看插件

说明：

velero restore create RESTORE_NAME –from-backup BACKUP_NAME –namespace-mappings old-ns-1:new-ns-1,old-ns-2:new-ns-2

Velero可以将资源还原到与其备份来源不同的命名空间中。为此，请使用–namespace-mappings标志

velero备份实战

velero对集群进行一次完整备份：

`1`	`velero backup create k8s-jf-test-all`

设置4小时定时备份一次，备份保留2个月：

1
2

# velero create schedule k8s-jf-cron-4h --exclude-namespaces test,tt --schedule="@every 4h" --ttl 1440h
Schedule "k8s-jf-cron-4h" created successfully.

同集群下，手动从完整备份中恢复其中一个namespace到指定namespace

# velero restore create  k8s-jf-test-all-restore --from-backup  k8s-jf-test-all --include-namespaces test --namespace-mappings  test:test10000
# velero restore describe k8s-jf-test-all-restore #查看恢复状态
Name:         k8s-jf-test-all-restore
Namespace:    velero
Labels:       <none>
Annotations:  <none>
Phase:                                 InProgress
Estimated total items to be restored:  141
Items restored so far:                 123
Started:    2021-09-07 10:47:44 +0800 CST
Completed:  <n/a>
Backup:  k8s-jf-test-all
Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>
Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto
Namespace mappings:  test=test10000
Label selector:  <none>
Restore PVs:  auto
Preserve Service NodePorts:  auto
# velero restore get
NAME                      BACKUP            STATUS       STARTED                         COMPLETED   ERRORS   WARNINGS   CREATED                         SELECTOR
k8s-jf-test-all-restore   k8s-jf-test-all   InProgress   2021-09-07 10:47:44 +0800 CST   <nil>       0        0          2021-09-07 10:47:44 +0800 CST   <none>

跨集群，将定期备份数据恢复

对于velero，跨集群需要保持2个集群使用的是同一个云厂商持久卷方案，这里我们统一使用minio，bucket都使用k8s-jf

velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.2.0 \
    --bucket k8s-jf \
    --secret-file ./credentials-velero \
    --use-volume-snapshots=false \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.1.214:39000

查看已经备份数据：

#velero backup get
NAME                                STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
k8s-jf-all-cron-4h-20210907061716   Completed         0        0          2021-09-07 14:17:16 +0800 CST   59d       default            <none>
k8s-jf-all-cron-4h-20210907021627   Completed         0        0          2021-09-07 10:16:27 +0800 CST   59d       default            <none>
k8s-jf-test-all                     Completed         0        0          2021-09-07 10:19:45 +0800 CST   29d       default            <none>

恢复指定命名空间数据

这里从k8s-jf-all-cron-4h-20210907061716这个备份恢复argocd命名空间数据到本集群argocd-dev下

# velero restore create  --from-backup  k8s-jf-all-cron-4h-20210907061716 --include-namespaces argocd --namespace-mappings  argocd:argocd-dev
Restore request "k8s-jf-all-cron-4h-20210907061716-20210907155450" submitted successfully.
Run `velero restore describe k8s-jf-all-cron-4h-20210907061716-20210907155450` or `velero restore logs k8s-jf-all-cron-4h-20210907061716-20210907155450` for more details.
# velero restore get
NAME                                               BACKUP                              STATUS       STARTED                         COMPLETED   ERRORS   WARNINGS   CREATED                         SELECTOR
k8s-jf-all-cron-4h-20210907061716-20210907155450   k8s-jf-all-cron-4h-20210907061716   InProgress   2021-09-07 15:54:51 +0800 CST   <nil>       0        0          2021-09-07 15:54:51 +0800 CST   <none>
# velero restore logs k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Attempting to restore Secret: argocd-application-controller-token-wv62v" logSource="pkg/restore/restore.go:1238" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Restored 2 items out of an estimated total of 61 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=argocd-application-controller-token-wv62v namespace=argocd-dev progress= resource=secrets restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Attempting to restore Secret: argocd-dex-server-token-9n4rs" logSource="pkg/restore/restore.go:1238" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Restored 3 items out of an estimated total of 61 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=argocd-dex-server-token-9n4rs namespace=argocd-dev progress= resource=secrets restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Attempting to restore Secret: argocd-secret" logSource="pkg/restore/restore.go:1238" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Restored 4 items out of an estimated total of 61 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=argocd-secret namespace=argocd-dev progress= resource=secrets restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Attempting to restore Secret: argocd-server-token-48vjd" logSource="pkg/restore/restore.go:1238" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Restored 5 items out of an estimated total of 61 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=argocd-server-token-48vjd namespace=argocd-dev progress= resource=secrets restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Attempting to restore Secret: cluster-192.168.1.210-3497337724" logSource="pkg/restore/restore.go:1238" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Restored 6 items out of an estimated total of 61 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=cluster-192.168.1.210-3497337724 namespace=argocd-dev progress= resource=secrets restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:23Z" level=info msg="Attempting to restore Secret: cluster-192.168.1.214-1096681010" logSource="pkg/restore/restore.go:1238" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
...
time="2021-09-07T08:11:30Z" level=info msg="Restored 61 items out of an estimated total of 61 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=argocd-server namespace=argocd-dev progress= resource=services restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:30Z" level=info msg="Waiting for all restic restores to complete" logSource="pkg/restore/restore.go:546" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:30Z" level=info msg="Done waiting for all restic restores to complete" logSource="pkg/restore/restore.go:562" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:30Z" level=info msg="Waiting for all post-restore-exec hooks to complete" logSource="pkg/restore/restore.go:566" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:30Z" level=info msg="Done waiting for all post-restore exec hooks to complete" logSource="pkg/restore/restore.go:574" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119
time="2021-09-07T08:11:30Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:480" restore=velero/k8s-jf-all-cron-4h-20210907061716-20210907161119

通过日志可以看到原集群argo下数据已经恢复到现在集群argo-dev下

卸载velero

卸载velero，注意这里的uninstall不会删除namespace：

# velero uninstall
You are about to uninstall Velero.
Are you sure you want to continue (Y/N)? y
Velero uninstalled ⛵
# kubectl delete namespace/velero clusterrolebinding/velero
# kubectl delete crds -l component=velero

备份中遇到异常

time="2021-09-07T07:22:35Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114"
time="2021-09-07T07:22:36Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117"
time="2021-09-07T07:22:36Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = RequestError: send request failed\ncaused by: Get http://minio.velero.svc:9000/velero?delimiter=%2F&list-type=2&prefix=backups%2F: dial tcp: lookup minio.velero.svc on 10.96.0.10:53: no such host" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:361" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182"
time="2021-09-07T07:22:36Z" level=error msg="Current backup storage locations available/unavailable/unknown: 0/1/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = RequestError: send request failed\ncaused by: Get http://minio.velero.svc:9000/velero?delimiter=%2F&list-type=2&prefix=: dial tcp: lookup minio.velero.svc on 10.96.0.10:53: no such host)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:164"
time="2021-09-07T07:22:36Z" level=error msg="Current backup storage locations available/unavailable/unknown: 0/1/0)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:166"

说明CRD资源BackupStorageLocation信息以及对象存储账户和自己设置的不一致，由于重新安装没有清理干净旧的资源信息

这里重新安装即可：

# velero uninstall
You are about to uninstall Velero.
Are you sure you want to continue (Y/N)? y
Velero uninstalled ⛵
# kubectl delete namespace/velero clusterrolebinding/velero
# kubectl delete crds -l component=velero
# velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.2.0 \
    --bucket k8s-jf \
    --secret-file ./credentials-velero \
    --use-volume-snapshots=false \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.1.214:39000
# kubectl -n velero get backupstoragelocation default -o yaml #查看资源信息已经是自己想要的
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  creationTimestamp: "2021-09-07T07:47:44Z"
  generation: 1
  labels:
    component: velero
  name: default
  namespace: velero
  resourceVersion: "1184696"
  selfLink: /apis/velero.io/v1/namespaces/velero/backupstoragelocations/default
  uid: 39502e43-272e-461f-a114-a9ec955f0510
spec:
  config:
    region: minio
    s3ForcePathStyle: "true"
    s3Url: http://192.168.1.214:39000
  default: true
  objectStorage:
    bucket: k8s-jf
  provider: aws
status:
  lastSyncedTime: "2021-09-07T07:50:00Z"
  lastValidationTime: "2021-09-07T07:50:00Z"
  phase: Available

阿里云共享GPU方案测试

wzdushu@gmail.com (wanzi) — Tue, 31 Aug 2021 14:22:42 +0800

一、k8s部署GPU共享插件

部署之前需要确保k8s节点上已安装nvidia-driver和nvidia-docker，同时已将docker默认运行时设置为nvidia

# cat /etc/docker/daemon.json
{
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
        "runtimeArgs": []
      }
  },
  "default-runtime": "nvidia",
}

1、helm安装gpushare-device-plugin

1
2
3

$ git clone https://github.com/AliyunContainerService/gpushare-scheduler-extender.git
$ cd gpushare-scheduler-extender/deployer/chart
$ helm install --name gpushare --namespace kube-system  --set masterCount=3 gpushare-installer

2、打Label，标记GPU节点

1
2

$ kubectl label node sd-cluster-04 gpushare=true
$ kubectl label node sd-cluster-05 gpushare=true

3、安装kubectl-inspect-gpushare

需要提前安装kubectl，这里就省略

1
2
3

$ cd /usr/bin/
$ wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare
$ chmod u+x /usr/bin/kubectl-inspect-gpushare

查看当前k8s集群GPU资源使用情况

$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  0/14                   0/14
sd-cluster-05  192.168.1.215  8/14                   8/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
8/28 (28%)

当然要禁用GPU节点共享GPU资源，直接设置gpushare=false即可

1
2

$ kubectl label node sd-cluster-04 gpushare=false
$ kubectl label node sd-cluster-05 gpushare=false

二、验证测试

1、部署第一个应用

申请2G GPU显存，测试应用，应该分配到一张卡上

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-1
  labels:
    app: binpack-1

spec:
  replicas: 1

  selector: # define how the deployment finds the pods it mangages
    matchLabels:
      app: binpack-1

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-1

    spec:
      containers:
      - name: binpack-1
        image: cheyang/gpu-player:v2
        resources:
          limits:
            # GiB
            aliyun.com/gpu-mem: 2

$ kubectl apply -f 1.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  0/14                   0/14
sd-cluster-05  192.168.1.215  2/14                   2/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
2/28 (7%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
$ kubectl logs -f binpack-1-6d6955c487-j4c4b -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=2
2021-08-13 02:47:22.395557: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 02:47:22.552831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 14.65GiB
2021-08-13 02:47:22.552873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

2、部署第二个应用

现在申请8G内存，2个实例，总共16G内存

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-2
  labels:
    app: binpack-2

spec:
  replicas: 2

  selector: # define how the deployment finds the pods it mangages
    matchLabels:
      app: binpack-2

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-2

    spec:
      containers:
      - name: binpack-2
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 8

$ kubectl apply -f 2.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  8/14                   8/14
sd-cluster-05  192.168.1.215  10/14                  10/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
18/28 (64%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          27m
$ kubectl logs -f binpack-2-58579b95f7-4wpbl -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=8
2021-08-13 02:48:41.246585: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 02:48:41.338992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 13.07GiB
2021-08-13 02:48:41.339031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

通过资源占用可以看到，第二个应用，已经分别使用了2张卡的8G内存

3、部署第三个应用

申请2G显存

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-3
  labels:
    app: binpack-3

spec:
  replicas: 1

  selector: # define how the deployment finds the pods it mangages
    matchLabels:
      app: binpack-3

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-3

    spec:
      containers:
      - name: binpack-3
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 2

$ kubectl apply -f 3.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  8/14                   8/14
sd-cluster-05  192.168.1.215  12/14                  12/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
20/28 (71%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          27m
binpack-2-58579b95f7-sjhwt         1/1     Running   0          27m
binpack-3-556bbd84f9-9xqg7         1/1     Running   0          14m
$ kubectl logs -f binpack-3-556bbd84f9-9xqg7 -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=2
2021-08-13 03:01:53.897423: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 03:01:54.008665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 7.08GiB
2021-08-13 03:01:54.008716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

部署完第三个应用以后，目前可用最大显存为8G，但是实际我们部署应用的时候最多只能使用6G，因为同一个任务不能分布式的跑到不同的GPU卡上

4、部署第四个应用

申请5G显存，应该是调度到sd-cluster-04上

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-4
  labels:
    app: binpack-4

spec:
  replicas: 1

  selector: # define how the deployment finds the pods it mangages
    matchLabels:
      app: binpack-4

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-4

    spec:
      containers:
      - name: binpack-4
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 5

$ kubectl apply -f 4.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  13/14                  13/14
sd-cluster-05  192.168.1.215  12/14                  12/14
------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
25/28 (89%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          26m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          24m
binpack-2-58579b95f7-sjhwt         1/1     Running   0          24m
binpack-3-556bbd84f9-9xqg7         1/1     Running   0          11m
binpack-4-6956458f85-cv62j         1/1     Running   0          6s
$ kubectl logs -f binpack-4-6956458f85-cv62j -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=5
2021-08-13 03:13:20.208122: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 03:13:20.361391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 6.46GiB
2021-08-13 03:13:20.361481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

三、总结

gpushare-device-plugin使用中有如下需求不能满足

同一个任务没法同时使用多台机器的GPU卡（共享）
同一张卡下没法通过GPU负载百分比来分配资源

不过对于算法团队模型测试已经完全够用了，对于GPU共享方案，其实还有另外两种，这里我就不介绍了，有需要的直接参考官方仓库配置即可。

https://github.com/tkestack/gpu-manager https://github.com/vmware/bitfusion-with-kubernetes-integration

参考资料： https://github.com/AliyunContainerService/gpushare-scheduler-extender/tree/master/deployer

kubeadm部署高可用kubernetes集群

wzdushu@gmail.com (wanzi) — Sun, 15 Aug 2021 17:22:42 +0800

为了后便后期验证私有化部署，最近内网环境需要快速搭建一套k8s集群，由于之前对于规模比较大的集群，我一般采用kubeasz和kubespray来搞定，这次对于小环境集群，直接用kubeadm部署会更加高效。

下面记录kubeadm部署过程：

集群节点：

192.168.1.206 sd-cluster-206 node
192.168.1.207 sd-cluster-207 master,etcd
192.168.1.208 sd-cluster-208 master,etcd,haproxy,keepalived
192.168.1.209 sd-cluster-209 master,etcd,haproxy,keepalived

镜像版本：

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.5
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
docker pull registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.14.0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v0.48.1

一、基本环境设置

1、安装docker并增加hosts

yum install -y yum-utils device-mapper-persistent-data lvm2 git
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install docker-ce -y
systemctl  start docker
systemctl  enable docker
systemctl  status docker

2、配置hosts

cat  >>  /etc/hosts << hhhh
192.168.1.207 sd-cluster-207
192.168.1.208 sd-cluster-208
192.168.1.209 sd-cluster-209
hhhh

3、禁用防火墙并设置selinux

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux

4、关闭swap

Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动。方法一,通过kubelet的启动参数–fail-swap-on=false更改这个限制。方法二,关闭系统的Swap。

`1`	`# swapoff -a`

修改/etc/fstab文件，注释掉SWAP的自动挂载，使用free -m确认swap已经关闭。

5、安装所有节点的kubeadm和kubelet、ipvsadm

参考：https://developer.aliyun.com/mirror/kubernetes

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y --nogpgcheck  kubelet-1.18.3 kubeadm-1.18.3 ipvsadm 
systemctl enable kubelet && systemctl start kubelet

ps: 由于官网未开放同步方式, 可能会有索引gpg检查失败的情况, 这时请用 yum install -y –nogpgcheck kubelet kubeadm kubectl 安装

增加ipvsadm，如果重新开机，需要重新加载（可以写在 /etc/rc.local 中开机自动加载）

cat <<EOF >> /etc/rc.local
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4
EOF
chmod +x /etc/rc.local

二、keepalived配合haproxy实现负载均衡器高可用

`1`	`yum -y install keepalived haproxy`

1、keepalived配置

192.168.1.208配置

vrrp_script chk_ha {
	script "killall -0 haproxy"  
	interval 2 		
	weight -20 					
}

vrrp_instance VI_1 {
	state MASTER
    interface bond0
	virtual_router_id 33
	mcast_src_ip 192.168.1.208
	priority 100 		
	nopreempt 			
	advert_int 1 				
	authentication {
		auth_type PASS
		auth_pass hak8s
	}

	track_script {
		chk_ha			## 执行haproxy监测
	}

	virtual_ipaddress {
		192.168.1.205/24		##VIP 
	}
}

192.168.1.209配置

vrrp_script chk_ha {
	script "killall -0 haproxy"
	interval 2
	weight -20
}
vrrp_instance VI_1 {
	state BACKUP
	interface bond0
	virtual_router_id 33
	mcast_src_ip 192.168.1.209
	priority 90
	advert_int 1
	authentication {
		auth_type PASS
		auth_pass hak8s
	}
	track_script {
		chk_ha
	}
	virtual_ipaddress {
		192.168.1.205/24
	}
}

2、haproxy配置

208和209两个master的haproxy配置一样：

global
    log /dev/log local0
    maxconn 65535
    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    stats socket /var/lib/haproxy/stats
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    retries 3
    option redispatch
    option dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000

frontend k8s-apiserver
    bind *:8443
    mode tcp
    balance roundrobin
    server sd-cluster-04 192.168.1.207:6443 weight 5 check inter 2000 rise 2 fall 3
    server sd-cluster-05 192.168.1.208:6443 weight 3 check inter 2000 rise 2 fall 3
    server sd-cluster-06 192.168.1.209:6443 weight 3 check inter 2000 rise 2 fall 3

至此，haproxy高可用已经搭建完成，验证方法，直接关闭208的keepalived服务，看vip是否飘逸到209机器上。

三、kubeadm初始化k8s集群

1、初始化第一台master

kubeadm-config.yaml配置

#kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.1.208
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: sd-cluster-208
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "192.168.1.208:8443"
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.18.3
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/16
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

# kubeadm init --config kubeadm-config.yaml --upload-certs
W0828 03:02:37.249435   17451 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.3
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [sd-cluster-208 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.208 192.168.1.205]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [sd-cluster-208 localhost] and IPs [192.168.1.208 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [sd-cluster-208 localhost] and IPs [192.168.1.208 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0828 03:02:43.787797   17451 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0828 03:02:43.789895   17451 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 23.015352 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
0adac55426d376e72f21ec3aee2465f754e78810700843cb75c270be26eaeaf1
[mark-control-plane] Marking the node sd-cluster-208 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node sd-cluster-208 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.1.205:8443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:7400f57a033f7527c80a4b015b54b4b6a88ccb2184ab9a6e39b709fb56e10486 \
    --control-plane --certificate-key 0adac55426d376e72f21ec3aee2465f754e78810700843cb75c270be26eaeaf1

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.205:8443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:7400f57a033f7527c80a4b015b54b4b6a88ccb2184ab9a6e39b709fb56e10486

配置master节点kubeconfig

1
2
3

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

2、新master节点加入集群

# kubeadm join 192.168.1.205:8443 --token abcdef.0123456789abcdef \
>     --discovery-token-ca-cert-hash sha256:7400f57a033f7527c80a4b015b54b4b6a88ccb2184ab9a6e39b709fb56e10486 \
>     --control-plane --certificate-key 0adac55426d376e72f21ec3aee2465f754e78810700843cb75c270be26eaeaf1
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [sd-cluster-209 localhost] and IPs [192.168.1.209 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [sd-cluster-209 localhost] and IPs [192.168.1.209 127.0.0.1 ::1]
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [sd-cluster-209 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.209 192.168.1.205]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
W0828 03:12:57.996811   48328 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0828 03:12:58.009390   48328 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0828 03:12:58.011318   48328 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
{"level":"warn","ts":"2021-08-28T03:13:15.215-0400","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"passthrough:///https://192.168.1.209:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node sd-cluster-209 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node sd-cluster-209 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

3、新node节点加入集群

1
2

kubeadm join 192.168.1.205:8443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:7400f57a033f7527c80a4b015b54b4b6a88ccb2184ab9a6e39b709fb56e10486

4、安装flannel网络组件

# wget http://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# sed -i 's#quay.io/coreos/flannel:v0.14.0#registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.14.0#g' kube-flannel.yml
# kubectl apply -f kube-flannel.yml -n kube-system
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
# kubectl get pod -n kube-system |grep flannel
kube-flannel-ds-gtdl8                    1/1     Running   0          24s
kube-flannel-ds-jtndz                    1/1     Running   0          24s
kube-flannel-ds-nd79b                    1/1     Running   0          24s

5、安装ingress

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v0.48.1

wget https://github.com/kubernetes/ingress-nginx/releases/download/helm-chart-3.35.0/ingress-nginx-3.35.0.tgz
tar zxvf helm-chart-3.35.0.tar.gz
cd ingress-nginx
vim values.yaml  #更新镜像为上面的国内阿里镜像版本

#helm install ingress-nginx --namespace ingress-nginx  . -n ingress-nginx
NAME: ingress-nginx
LAST DEPLOYED: Tue Aug 31 17:02:28 2021
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
Get the application URL by running these commands:
  export HTTP_NODE_PORT=$(kubectl --namespace ingress-nginx get services -o jsonpath="{.spec.ports[0].nodePort}" ingress-nginx-controller)
  export HTTPS_NODE_PORT=$(kubectl --namespace ingress-nginx get services -o jsonpath="{.spec.ports[1].nodePort}" ingress-nginx-controller)
  export NODE_IP=$(kubectl --namespace ingress-nginx get nodes -o jsonpath="{.items[0].status.addresses[1].address}")

  echo "Visit http://$NODE_IP:$HTTP_NODE_PORT to access your application via HTTP."
  echo "Visit https://$NODE_IP:$HTTPS_NODE_PORT to access your application via HTTPS."

An example Ingress that makes use of the controller:

  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

对于新建Ingress参考如下设置：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-nginx
spec:
  rules:
    - host: "my-nginx.test.com"
      http:
        paths:
          - path: /
            backend:
              serviceName: my-nginx
              servicePort: 8080

至此，kubeadm部署kubernetes高可用集群完成

kubernetes集群添加用户

wzdushu@gmail.com (wanzi) — Tue, 31 Dec 2019 10:22:42 +0800

之前通过ansible搭建了kubernetes集群环境,这里需求主要是添加一个用户进行日常管理，并限制到指定的namespace，接下来进行操作：

kubernetes中用户

K8S中有两种用户(User)——服务账号(ServiceAccount)和普通意义上的用户(User), ServiceAccount是由K8S管理的，而User通常是在外部管理，K8S不存储用户列表——也就是说，添加/编辑/删除用户都是在外部进行，无需与K8S API交互，虽然K8S并不管理用户，但是在K8S接收API请求时，是可以认知到发出请求的用户的，实际上，所有对K8S的API请求都需要绑定身份信息(User或者ServiceAccount)，这意味着，可以为User配置K8S集群中的请求权限

对于kubernetes接受用户请求的时候通常采取客户端证书、静态token文件、静态密码文件三种方式，这里我们只介绍证书验证。

生成用户证书

准备证书生成文件csr,通过kubernetes的ca签发证书,通常k8s api server的ca文件路径为/etc/kubernetes/pki/,这里会生成cicd-admin-key.pem(私钥)和cicd-admin.pem(证书), 具体csr如何生成可以参考：https://wnote.com/post/linux-openssl-issue-private-certificate/

这里的csr文件传递的用户名为cicd-admin

# vim cicd-admin-csr.json
{
  "CN": "cicd-admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
# cd /etc/kubernetes/pki/ 
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  cicd-admin-csr.json | cfssljson -bare cicd-admin
# ls -l cicd-admin*
-rw-r--r-- 1 root root 1001 Dec 19 16:51 cicd-admin.csr
-rw-r--r-- 1 root root  224 Dec 19 16:50 cicd-admin-csr.json
-rw------- 1 root root 1675 Dec 19 16:51 cicd-admin-key.pem
-rw-r--r-- 1 root root 1387 Dec 19 16:51 cicd-admin.pem

用户权限控制(RBAC)

创建角色

k8s中rbac的角色主要有两种,普通角色(Role)和集群角色(ClusterRole)，ClusterRole是特殊的Role

Role属于某个命名空间; 而ClusterRole属于整个集群，包括所有的命名空间
ClusterRole能够授予集群范围的权限，比如node资源的管理，可以请求全命名空间的资源(通过指定–all-namespaces), 默认情况下, K8S内置了一个名为admin的ClusterRole，实际使用中无需创建admin Role

在cicd的namespace下创建一个role为cicd-admin:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  namespace: cicd
  name: cicd-admin
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]

绑定用户到指定角色

subjects指定账户类型可以是User也可以是service account，这里指定的是用户cicd-admin, roleRef指定RoleBinding引用角色

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cicd-admin-binding
  namespace: cicd
subjects:
- kind: User
  name: cicd-admin
  apiGroup: ""
roleRef:
  kind: Role
  name: admin
  apiGroup: ""

或者通过kubectl进行绑定也可以:

`1`	`kubectl create rolebinding cicd-admin-binding --role=admin --user=cicd-admin --namespace=cicd`

或者给用户绑定到默认ClusterRole的admin角色,而这里cicd-admin如果绑定了admin，其权限也只会被限制在cicd的namespace

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cicd-admin-binding
  namespace: cicd
subjects:
- kind: User
  name: cicd-admin
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: admin
  apiGroup: ""

客户端配置

设置集群参数

export KUBE_APISERVER="https://cicd-k8s.test.cn:8443"
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=cicd-admin.kubeconfig

设置客户端认证参数

kubectl config set-credentials cicd-admin \
--client-certificate=/etc/kubernetes/ssl/cicd-admin.pem \
--client-key=/etc/kubernetes/ssl/cicd-admin-key.pem \
--embed-certs=true \
--kubeconfig=cicd-admin.kubeconfig

设置上下文参数

kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=cicd-admin \
--namespace=cicd \
--kubeconfig=cicd-admin.kubeconfig

设置默认上下文

`1`	`kubectl config use-context kubernetes --kubeconfig=cicd-admin.kubeconfig`

客户端使用

将kubeconfig文件覆盖~/.kube/config

`1`	`cp cicd-admin.kubeconfig ~/.kube/config`

至此，kubernetes添加用户并进行授权访问k8s集群的介绍就告一段落。

kubernetes集群部署traefik2.1

wzdushu@gmail.com (wanzi) — Tue, 17 Dec 2019 10:22:42 +0800

架构&概念

Traefik2.x版本相比1.7.x架构有很大变化，正如上边这张架构图，最主要的功能是支持了TCP协议、增加了Router概念。

这里我们采用在kubernetes集群部署Traefik2.1，业务访问通过haproxy请求到traefik Ingress，下边是搭建过程涉及到一些概念：

EntryPoints：Traefik的网络入口，定义请求接受的端口(不分http或tcp)
CRD：Kubernetes API的扩展
IngressRouter：将传入请求转发到可以处理请求的服务，另外转发请求之前可以通过Middlewares动态更新请求
Middlewares：请求到达服务之前进行动态处理请求参数，比如header或转发规则等等。
TraefikService：如果果CRD定义了了这种类型，IngressRouter可以直接引用，处在IngressRouter和服务之间,类似于Maesh架构，更适合较为复杂场景,一般情况可以不使用。

kubernetes配置

配置SSL证书

因为业务服务使用https,这里先配置下SSL证书:

`1`	`kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=test.cn.pem --key=test.cn.key`

配置集群访问控制(RBAC)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutetcps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - tlsoptions
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - traefikservices
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: kube-system

TLS参数配置

默认配置TLS1.2,当然也可以使用TLS1.3

apiVersion: traefik.containo.us/v1alpha1
kind: TLSOption
metadata:
  name: mytlsoption
  namespace: kube-system

spec:
  minversion: VersionTLS12
  snistrict: true
  ciphersuites:
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
    - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

CRD配置

这里定义了IngressRoute、Middleware、TLSOption、IngressRouteTCP、TraefikService,其中TraefikService是在2.1版本新增加的CRD

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced

TraefikService配置

TraefikService有点类似Maesh解决服务之间的调用逻辑，只是Maesh依赖coredns；另外traefik service还可以设置后端服务权重，配置服务的流量镜像。

这里我们配置traefik dashboard和rancher的traefikservice类型服务，其他服务配置可以参考这里rancher的traefik service, traefik service会转发请求到kubernetes 的服务类型(上一章节我们已经通过helm3创建了rancher服务)，这里只是举例说明：

apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:
  name: traefik-webui-traefikservice
  namespace: kube-system

spec:
  weighted:
    services:
      - name: traefik-ingress-service
        weight: 1
        port: 8080

---
apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:t
  name: rancher-traefikservice
  namespace: cattle-system

spec:
  weighted:
    services:
      - name: rancher
        weight: 1
        port: 80

Deployment配置

配置k8s标准service服务
创建traefik configmap,并配置entrypoints和默认SSL证书
Deployment方式部署traefik ingress controller

kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      nodePort: 23456
      name: http
    - protocol: TCP
      port: 443
      nodePort: 23457
      name: https
  type: NodePort

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-conf
  namespace: kube-system
data:
  traefik.toml: |
    [global]
      checkNewVersion = false
      sendAnonymousUsage = false
    [log]
      level = "DEBUG"
    [api]
      dashboard = true
    [metrics.prometheus]
      buckets = [0.1,0.3,1.2,5.0]
      entryPoint = "metrics"
    [entryPoints]
      [entryPoints.http]
        address = ":80"
      [entryPoints.https]
        address = ":443"
    [tls.stores]
      [tls.stores.default]
        [tls.stores.default.defaultCertificate]
          certFile = "/config/tls/test.cn.crt"
          keyFile  = "/config/tls/test.cn.key"
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      nodeSelector:
        node-role.kubernetes.io/traefik: "true"
      volumes:
      - name: ssl
        secret:
          secretName: traefik-cert
      - name: config
        configMap:
          name: traefik-conf
      containers:
      - image: traefik:v2.1.1
        name: traefik-ingress-lb
        volumeMounts:
        - mountPath: "/config"
          name: "config"
        - mountPath: "/config/tls"
          name: "ssl"
        resources:
          limits:
            cpu: 1000m
            memory: 800Mi
          requests:
            cpu: 500m
            memory: 600Mi
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443
        args:
        - --entrypoints.http.Address=:80
        - --entrypoints.https.Address=:443
        - --api
        - --accesslog
        - --providers.file.directory=/config/
        - --providers.file.watch=true
        - --ping=true
        - --providers.kubernetescrd

IngressRouter配置

定义middleware给rancher服务配置X-Forwarded-Proto的头信息
配置rancher的ingressroute，并使用默认证书

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: redirect-https
  namespace: kube-system
spec:
  redirectScheme:
    scheme: https

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: http-default-router
  namespace: kube-system
spec:
  entryPoints:
    - http
  routes:
  - match: HostRegexp(`{host:.+}`)
    kind: Rule
    services:
    - name: traefik-ingress-service
      kind: Service
      namespaces: kube-system
      port: 80
    middlewares:
    - name: redirect-https
  tls:
    options:
      name: mytlsoption
      namespaces: kube-system
    certResolver: default

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-webui
  namespace: kube-system
spec:
  entryPoints:
    - https
  routes:
  - match: Host(`traefik.test.cn`)
    kind: Rule
    services:
    - name: api@internal
      kind: TraefikService
      namespaces: kube-system

配置Rancher的IngressRoute

由于我这里使用rancher管理集群，实际环境可以根据你自己需求配置

定义middleware给rancher服务配置X-Forwarded-Proto的头信息
配置rancher的ingressroute，并使用默认证书

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: rancher-https-headers
  namespace: cattle-system
spec:
  headers:
    customRequestHeaders:
      X-Forwarded-Proto: "https"
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: rancher-tls
  namespace: cattle-system
spec:
  entryPoints:
  - https
  routes:
  - match: Host(`rancher.test.cn`)
    kind: Rule
    services:
    - name: rancher
      kind: Service
      namespaces: cattle-system
      port: 80
    middlewares:
    - name: rancher-https-headers
      namespaces: cattle-system

  tls:
    certResolver: default

统一部署traefik2.1

kubectl apply -f 01-crd.yaml
kubectl apply -f 02-rbac.yaml
kubectl apply -f 03-tlsoption.yaml
kubectl apply -f 04-traefikservices.yaml #非必须
kubectl apply -f 05-traefik.yaml
kubectl apply -f 06-ingressrouter.yaml
kubectl apply -f 06-ingressrouter-rancher.yaml

更多配置信息，请移步我的github仓库：https://github.com/iwz2099/kubecase/tree/master/traefik/v2

kubeasz部署k8s集群

wzdushu@gmail.com (wanzi) — Thu, 12 Dec 2019 10:22:42 +0800

环境准备

Master节点

1
2
3

172.16.244.14
172.16.244.16
172.16.244.18

Node节点

1
2

172.16.244.25
172.16.244.27

Master节点VIP地址: 172.16.243.13
部署工具:Ansible/kubeasz

初始化环境

安装Ansible

apt update
apt-get install ansible expect
git clone https://github.com/easzlab/kubeasz
cd kubeasz
cp * /etc/ansible/

配置ansible免密登录

1
2

ssh-keygen -t rsa -b 2048 #生成密钥
./tools/yc-ssh-key-copy.sh  hosts root 'rootpassword'

准备二进制文件

1
2

cd tools
./easzup -D #默认情况都会下载到/etc/ansible/bin/目录下

配置hosts文件如下:

[kube-master]
172.16.244.14
172.16.244.16
172.16.244.18

[etcd]
172.16.244.14 NODE_NAME=etcd1
172.16.244.16 NODE_NAME=etcd2
172.16.244.18 NODE_NAME=etcd3

#haproxy-keepalived
[haproxy]
172.16.244.14
172.16.244.16
172.16.244.18

[kube-node]
172.16.244.25
172.16.244.27


# [optional] loadbalance for accessing k8s from outside
[ex-lb]
172.16.244.14 LB_ROLE=backup EX_APISERVER_VIP=172.16.243.13 EX_APISERVER_PORT=8443
172.16.244.16 LB_ROLE=backup EX_APISERVER_VIP=172.16.243.13 EX_APISERVER_PORT=8443
172.16.244.18 LB_ROLE=master EX_APISERVER_VIP=172.16.243.13 EX_APISERVER_PORT=8443

# [optional] ntp server for the cluster
[chrony]
172.16.244.18

[all:vars]
# --------- Main Variables ---------------
# Cluster container-runtime supported: docker, containerd
CONTAINER_RUNTIME="docker"

# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
#CLUSTER_NETWORK="flannel"
CLUSTER_NETWORK="calico"

# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"

# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.68.0.0/16"

# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="10.101.0.0/16"

# NodePort Range
NODE_PORT_RANGE="20000-40000"

# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="cluster.local."

# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/opt/kube/bin"

# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"

# Deploy Directory (kubeasz workspace)
base_dir="/etc/ansible"

部署K8S集群

初始化配置

1
2

cd /etc/ansible
ansible-playbook 01.prepare.yml

这个过程主要做三件事：

1
2
3

chrony role: 集群节点时间同步[可选]
deploy role: 创建CA证书、kubeconfig、kube-proxy.kubeconfig
prepare role: 分发CA证书、kubectl客户端安装、环境配置

安装etcd集群

`1`	`ansible-playbook 02.etcd.yml`

安装docker

`1`	`ansible-playbook 03.docker.yml`

部署kubernetes master

`1`	`ansible-playbook 04.kube-master.yml`

部署kubernetes node

`1`	`ansible-playbook 05.kube-node.yml`

部署kubernetes network(这里选择calico)

`1`	`ansible-playbook 06.network.yml`

部署ingress/k8s dashbaord/coredns

配置ingress所用ssl证书, 这里仓库默认使用的traefik1.7.12,后边我们打算升级为2.0

1
2

# kubectl create secret tls traefik-cert --key=test.cn.key --cert=test.cn.pem  -n kube-system
secret/traefik-cert created

部署集群扩展

`1`	`ansible-playbook 07.cluster-addon.yml`

master节点取消污点

默认部署完后，master节点状态是打了污点，不在调度策略如下：

# kubectl  get node
NAME            STATUS                     ROLES    AGE   VERSION
172.16.244.14   Ready,SchedulingDisabled   master   91m   v1.16.2
172.16.244.16   Ready,SchedulingDisabled   master   91m   v1.16.2
172.16.244.18   Ready,SchedulingDisabled   master   91m   v1.16.2
172.16.244.25   Ready                      node     90m   v1.16.2
172.16.244.27   Ready                      node     90m   v1.16.2
# kubectl  describe node  172.16.244.14 |grep Taint
Taints:             node.kubernetes.io/unschedulable:NoSchedule

由于机器资源有限，所以把master也加入调度可用状态

# kubectl patch node 172.16.244.14 -p '{"spec":{"unschedulable":false}}'
# kubectl  get node
NAME            STATUS   ROLES    AGE   VERSION
172.16.244.14   Ready    master   95m   v1.16.2
172.16.244.16   Ready    master   95m   v1.16.2
172.16.244.18   Ready    master   95m   v1.16.2
172.16.244.25   Ready    node     94m   v1.16.2
172.16.244.27   Ready    node     94m   v1.16.2

部署外部负载均衡器(Keepalived+Haproxy)

`1`	`ansible-playbook roles/ex-lb/ex-lb.yml`

部署Rancher

安装Helm3

参考 https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-rancher/

cd /opt/soft
wget https://get.helm.sh/helm-v3.0.1-linux-amd64.tar.gz
tar xf helm-v3.0.1-linux-amd64.tar.gz
cd linux-amd64/
cp helm  /usr/local/bin/

创建证书

因为由自己域名证书,所以这里使用k8s的secret创建的证书,当然也可以使用cert-manager工具签发rancher自己的证书或者使用letsEncrypt

`1`	`kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=test.cn.pem --key=test.cn.key`

安装rancher

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update
helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=rancher-cicd.test.cn --set ingress.tls.source=secret

检查ingress、资源和部署状态

# kubectl  get ingress  --all-namespaces
NAMESPACE       NAME      HOSTS                   ADDRESS   PORTS     AGE
cattle-system   rancher   rancher-cicd.test.cn             80, 443   20h
# kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available...
deployment "rancher" successfully rolled out
# kubectl -n cattle-system get deploy rancher
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
rancher   3/3     3            3           5m5s

至此，整个K8S集群已经搭建完毕，如果顺利的话，整个过程应该在10分钟左右，重要的是提前规划好集群。