Aliyun on 丸子有记, 记录生活、分享技术，专注于运维架构、运维开发(devops)。

利用阿里云开源方案实现单GPU卡多人使用

wzdushu@gmail.com (wanzi) — Fri, 30 Jun 2023 18:10:42 +0800

最近新上了一个AI项目，主要是给大学提供AI在线实验，项目也采购了GPU服务器，但是只有一张Nvidia Tesla T4卡，需要支持多个学生同时在线做实验呢。

在线实验系统目前运行在Kubernetes上，因此，需要考虑k8s环境下，GPU共享，之前也测试过阿里云GPU卡共享方案，这里就直接记录使用步骤即可：

kubernetes集群版本：1.23.1

调整K8S调度器

从1.23.+起，由于kube-scheduler调度策略有调整，所以之前的部署方式就不好时了。

参考这里：https://kubernetes.io/docs/reference/scheduling/policies/

1
2

git clone https://github.com/AliyunContainerService/gpushare-scheduler-extender.git
cp config/scheduler-policy-config.yaml /etc/kubernetes/

由于我这里是通过kubeasz安装的k8s，所以Kubeconfig放在这里/etc/kubernetes/kubelet.kubeconfig，需要更新 /etc/kubernetes/scheduler-policy-config.yaml如下：

---
apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /etc/kubernetes/kube-scheduler.kubeconfig
extenders:
- urlPrefix: "http://192.168.233.101:32766/gpushare-scheduler"
  filterVerb: filter
  bindVerb: bind
  enableHTTPS: false
  nodeCacheCapable: true
  managedResources:
  - name: aliyun.com/gpu-mem
    ignoredByScheduler: false
  ignorable: false

修改Kube-scheduler启动启动参数，/etc/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/opt/kube/bin/kube-scheduler \
  --authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --bind-address=0.0.0.0 \
  --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --config=/etc/kubernetes/scheduler-policy-config.yaml \
  --leader-elect=true \
  --v=2
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

1
2

systemctl  daemon-reload
systemctl  restart  kube-scheduler.service

安装调度器扩展器

给GPU机器打上label:

kubectl label node mynode gpushare=true

1
2

cd gpushare-scheduler-extender/config
kubectl apply -f gpushare-schd-extender.yaml

安装插件

1
2

kubectl create -f https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-rbac.yaml
kubectl create -f https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-ds.yaml

安装完成以后如下

1
2
3

# kubectl get pod   -n kube-system |grep gpushare
gpushare-device-plugin-ds-j8blj              1/1     Running   0                30h
gpushare-schd-extender-74796c5f64-7g4bl      1/1     Running   0                29h

测试GPU

修改samples/1.yaml如下：

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-1
  labels:
    app: binpack-1

spec:
  replicas: 1

  selector: # define how the deployment finds the pods it mangages
    matchLabels:
      app: binpack-1

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-1

    spec:
      containers:
      - name: binpack-1
        image: cheyang/gpu-player:v2
        resources:
          limits:
            # GiB
            aliyun.com/gpu-count: 1
            aliyun.com/gpu-mem: 5

# kubectl apply -f  samples/1.yaml
deployment.apps/binpack-1 created
# kubectl get pod  |grep binpack
binpack-1-9995bdf69-pk2d4                 1/1     Running   0             12s
# kubectl logs -f binpack-1-9995bdf69-pk2d4
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=5
2023-06-30 09:40:50.890296: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2023-06-30 09:40:50.976283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:03:00.0
totalMemory: 14.75GiB freeMemory: 14.66GiB
2023-06-30 09:40:50.976313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:03:00.0, compute capability: 7.5)

通过日志看，GPU共享已经成功，且申请了5G显存。

当然你也可以通过kubectl扩展工具kubectl-inspect-gpushare来查看:

# cd /usr/bin/
# wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare
# chmod u+x /usr/bin/kubectl-inspect-gpushare
# kubectl-inspect-gpushare
NAME             IPADDRESS        GPU0(Allocated/Total)  GPU Memory(GiB)
192.168.233.101  192.168.233.101  9/14                   9/14
--------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
9/14 (64%)

至此，GPU卡共享已经完成，剩下就是通过tensorflow来调度我的GPU卡了。

参考文档：

https://github.com/AliyunContainerService/gpushare-scheduler-extender/blob/master/docs/install.md

解决k8s下nginx文件上传限制和504网关超时

wzdushu@gmail.com (wanzi) — Mon, 08 Nov 2021 17:22:42 +0800

最近业务使用k8s集群经常有两个问题需要解决，这里记录一下：

前端页面上传文件1M限制
前端页面发送POST请求到后端，出现504超时

第一个问题主要解决方法：nginx默认上传大小为1M，nginx配置文件http,server, location区域增加如下内容：

`1`	`client_max_body_size 800M;`

第二个问题主要解决方法：nginx默认读取后端超时时间为1分钟，nginx配置文件http,server, location区域增加如下内容：

1
2
3

proxy_connect_timeout 120;
proxy_read_timeout  900;
proxy_send_timeout 900;

最终，nginx配置如下：

server {
    listen 80;
    server_name web_server;
    location / {
        root /usr/share/nginx/html/;
        index index.htm index.html;
        try_files $uri $uri/ /index.html;
    }
    location /business-operation {
        proxy_pass http://business-operat.common:5000;
        
        proxy_ignore_client_abort on; 
        keepalive_timeout 900s;
        client_body_timeout 900s;
        proxy_connect_timeout 120s;
        proxy_read_timeout  900s;
        proxy_send_timeout 900s;
        client_max_body_size 800m;
        proxy_pass_header Authorization;
        proxy_pass_header WWW-Authenticate;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
    }
}

最后，以上nginx配置调整以后，还需要在创建ingress的时候，注解里带上这些信息：

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "800M"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '120'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '900'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '900'
  name: enterprise-mark
  namespace: frontend
spec:
  rules:
    - host: enterprise-marketing.xxxx.com
      http:
        paths:
          - backend:
              serviceName: enterprise-mark
              servicePort: 80
            path: /
status:
  loadBalancer:
    ingress:
      - ip: 39.107.xxx.xxx

阿里云ACK同时支持公网和内网SLB

wzdushu@gmail.com (wanzi) — Thu, 21 Oct 2021 17:22:42 +0800

一、背景

有一套ACK集群
成功部署Nginx ingress controller并绑定了公网SLB

注意：通过阿里云容器服务管理控制台创建的Kubernetes集群在初始化时会自动部署一套Nginx Ingress Controller，默认其挂载在公网SLB实例上.

二、配置

1、创建内网LB

阿里云控制台，创建内网SLB并绑定VPC

2、配置Nginx ingress controller

# my-nginx-ingress-slb-intranet.yaml
# intranet nginx ingress slb service
apiVersion: v1
kind: Service
metadata:
  # 这里服务取名为nginx-ingress-lb-intranet。
  name: nginx-ingress-lb-intranet
  namespace: kube-system
  labels:
    app: nginx-ingress-lb-intranet
  annotations:
    # 指明SLB实例地址类型为私网类型。
    service.beta.kubernetes.io/alicloud-loadbalancer-address-type: intranet
    # 修改为您的私网SLB实例ID。
    service.beta.kubernetes.io/alicloud-loadbalancer-id: <YOUR_INTRANET_SLB_ID>
    # 是否自动创建SLB端口监听（会覆写已有端口监听），也可手动创建端口监听。
    #service.beta.kubernetes.io/alicloud-loadbalancer-force-override-listeners: 'false'
spec:
  type: LoadBalancer
  # route traffic to other nodes
  externalTrafficPolicy: "Cluster"
  ports:
  - port: 80
    name: http
    targetPort: 80
  - port: 443
    name: https
    targetPort: 443
  selector:
    # select app=ingress-nginx pods
    app: ingress-nginx

创建service资源：

`1`	`kubectl apply -f my-nginx-ingress-slb-intranet.yaml`

获取service资源：

1
2
3

# kubectl -n kube-system get svc | grep nginx-ingress-lb
nginx-ingress-lb            LoadBalancer   172.21.15.148   39.107.xxx.xxx   80:32076/TCP,443:30803/TCP     433d
nginx-ingress-lb-intranet   LoadBalancer   172.21.5.0      172.17.193.181   80:32282/TCP,443:30507/TCP     1d

通过Ingress对外暴露服务时，即可以通过公网SLB来访问该服务，同一个VPC下的其他服务也可以直接通过私网SLB来访问该服务。

3、创建ingress服务配置完ingress controller，ingress创建无须特殊配置，和以前创建ingress一摸一样，如下例子：

# Source: prometheusalert/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: RELEASE-NAME-prometheusalert
  labels:
    app.kubernetes.io/name: prometheusalert
    helm.sh/chart: prometheusalert-1.0.0
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/version: "1.2.0"
    app.kubernetes.io/managed-by: Helm
spec:
  rules:
    - host: "palert.con.sdi"
      http:
        paths:
          - path: /
            backend:
              serviceName: RELEASE-NAME-prometheusalert
              servicePort: 8080

对于内网域名直接解析到内网SLB地址（这里172.17.193.181）即可。

参考：阿里云文档

阿里云共享GPU方案测试

wzdushu@gmail.com (wanzi) — Tue, 31 Aug 2021 14:22:42 +0800

一、k8s部署GPU共享插件

部署之前需要确保k8s节点上已安装nvidia-driver和nvidia-docker，同时已将docker默认运行时设置为nvidia

# cat /etc/docker/daemon.json
{
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
        "runtimeArgs": []
      }
  },
  "default-runtime": "nvidia",
}

1、helm安装gpushare-device-plugin

1
2
3

$ git clone https://github.com/AliyunContainerService/gpushare-scheduler-extender.git
$ cd gpushare-scheduler-extender/deployer/chart
$ helm install --name gpushare --namespace kube-system  --set masterCount=3 gpushare-installer

2、打Label，标记GPU节点

1
2

$ kubectl label node sd-cluster-04 gpushare=true
$ kubectl label node sd-cluster-05 gpushare=true

3、安装kubectl-inspect-gpushare

需要提前安装kubectl，这里就省略

1
2
3

$ cd /usr/bin/
$ wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare
$ chmod u+x /usr/bin/kubectl-inspect-gpushare

查看当前k8s集群GPU资源使用情况

$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  0/14                   0/14
sd-cluster-05  192.168.1.215  8/14                   8/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
8/28 (28%)

当然要禁用GPU节点共享GPU资源，直接设置gpushare=false即可

1
2

$ kubectl label node sd-cluster-04 gpushare=false
$ kubectl label node sd-cluster-05 gpushare=false

二、验证测试

1、部署第一个应用

申请2G GPU显存，测试应用，应该分配到一张卡上

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-1
  labels:
    app: binpack-1

spec:
  replicas: 1

  selector: # define how the deployment finds the pods it mangages
    matchLabels:
      app: binpack-1

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-1

    spec:
      containers:
      - name: binpack-1
        image: cheyang/gpu-player:v2
        resources:
          limits:
            # GiB
            aliyun.com/gpu-mem: 2

$ kubectl apply -f 1.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  0/14                   0/14
sd-cluster-05  192.168.1.215  2/14                   2/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
2/28 (7%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
$ kubectl logs -f binpack-1-6d6955c487-j4c4b -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=2
2021-08-13 02:47:22.395557: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 02:47:22.552831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 14.65GiB
2021-08-13 02:47:22.552873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

2、部署第二个应用

现在申请8G内存，2个实例，总共16G内存

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-2
  labels:
    app: binpack-2

spec:
  replicas: 2

  selector: # define how the deployment finds the pods it mangages
    matchLabels:
      app: binpack-2

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-2

    spec:
      containers:
      - name: binpack-2
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 8

$ kubectl apply -f 2.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  8/14                   8/14
sd-cluster-05  192.168.1.215  10/14                  10/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
18/28 (64%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          27m
$ kubectl logs -f binpack-2-58579b95f7-4wpbl -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=8
2021-08-13 02:48:41.246585: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 02:48:41.338992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 13.07GiB
2021-08-13 02:48:41.339031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

通过资源占用可以看到，第二个应用，已经分别使用了2张卡的8G内存

3、部署第三个应用

申请2G显存

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-3
  labels:
    app: binpack-3

spec:
  replicas: 1

  selector: # define how the deployment finds the pods it mangages
    matchLabels:
      app: binpack-3

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-3

    spec:
      containers:
      - name: binpack-3
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 2

$ kubectl apply -f 3.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  8/14                   8/14
sd-cluster-05  192.168.1.215  12/14                  12/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
20/28 (71%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          27m
binpack-2-58579b95f7-sjhwt         1/1     Running   0          27m
binpack-3-556bbd84f9-9xqg7         1/1     Running   0          14m
$ kubectl logs -f binpack-3-556bbd84f9-9xqg7 -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=2
2021-08-13 03:01:53.897423: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 03:01:54.008665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 7.08GiB
2021-08-13 03:01:54.008716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

部署完第三个应用以后，目前可用最大显存为8G，但是实际我们部署应用的时候最多只能使用6G，因为同一个任务不能分布式的跑到不同的GPU卡上

4、部署第四个应用

申请5G显存，应该是调度到sd-cluster-04上

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-4
  labels:
    app: binpack-4

spec:
  replicas: 1

  selector: # define how the deployment finds the pods it mangages
    matchLabels:
      app: binpack-4

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-4

    spec:
      containers:
      - name: binpack-4
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 5

$ kubectl apply -f 4.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  13/14                  13/14
sd-cluster-05  192.168.1.215  12/14                  12/14
------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
25/28 (89%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          26m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          24m
binpack-2-58579b95f7-sjhwt         1/1     Running   0          24m
binpack-3-556bbd84f9-9xqg7         1/1     Running   0          11m
binpack-4-6956458f85-cv62j         1/1     Running   0          6s
$ kubectl logs -f binpack-4-6956458f85-cv62j -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=5
2021-08-13 03:13:20.208122: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 03:13:20.361391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 6.46GiB
2021-08-13 03:13:20.361481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

三、总结

gpushare-device-plugin使用中有如下需求不能满足

同一个任务没法同时使用多台机器的GPU卡（共享）
同一张卡下没法通过GPU负载百分比来分配资源

不过对于算法团队模型测试已经完全够用了，对于GPU共享方案，其实还有另外两种，这里我就不介绍了，有需要的直接参考官方仓库配置即可。

https://github.com/tkestack/gpu-manager https://github.com/vmware/bitfusion-with-kubernetes-integration

参考资料： https://github.com/AliyunContainerService/gpushare-scheduler-extender/tree/master/deployer

阿里云PrivateZone+Bind9+Dnsmasq实现内部DNS

wzdushu@gmail.com (wanzi) — Sat, 10 Jul 2021 17:22:42 +0800

需求：

阿里云集群能够解析内部域名

办公网解析内部域名+办公网上网解析

解决方法：

对于第一个问题，直接使用阿里云PrivateZone解析即可
对于第二个问题，采用在PrivateZone配置内部域名zone，然后通过阿里云同步工具同步到办公网bind9服务器；对于办公网DNS解析入口，使用Dnsmasq处理，对于公网解析直接Forward到公网DNS，内部域名直接转发到bind9处理。

这里可能有人疑问，为什么不用bind直接实现所有内部解析呢？这里主要原因在实际使用中发现bind9的forward多个dns的时候并发有性能问题，偶尔会有超时现象，这一点dnsmasq做的相对出色很多。

一、阿里云PrivateZone配置

参考：https://help.aliyun.com/document_detail/64627.html

二、同步阿里云zone到bind9

1、docker-compose搭建bind9

version: '2'

services:
  bind:
    restart: always
    image: sameersbn/bind:9.16.1-20200524
    environment:
    - ROOT_PASSWORD=DNS2021#
    - WEBMIN_ENABLED=true
    - WEBMIN_INIT_SSL_ENABLED=false
    ports:
    - "15353:53/tcp"
    - "15353:53/udp"
    - "11953:953/tcp"
    - "10000:10000/tcp" #webmin管理
    volumes:
    - ./data:/data
    networks:
      - bind9

networks:
  bind9:
    ipam:
      config:
      - subnet: 10.220.0.0/16
        gateway: 10.220.0.1

2、修改bind配置文件

data/bind/etc/named.conf

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";
key "rndc-key" {
    algorithm hmac-sha256;
    secret "tREasaE2Jal1GfwfL5iii3a88eRGKWui41l5h3v89OM=";
};
controls {
	inet 127.0.0.1 port 953 allow { 127.0.0.1; } keys { rndc-key; };
	};
logging {
    channel query_log {
        file "query.log" versions 10 size 50M;
        severity info;
        print-category yes;
        print-severity yes;
        print-time yes;
    };
    category queries {
        query_log;
    };
};

data/bind/etc/named.conf.options

options {
	directory "/var/cache/bind";
    dnssec-validation no;
    dnssec-enable no;
    recursion yes;
    allow-recursion { any;};
    allow-transfer { any; };
    allow-query-cache { any; };
    listen-on-v6 { any; };
    listen-on port 53 { any; };
    forward first;
    forwarders {
        192.168.1.211 port 53;
        192.168.1.212 port 53;
        };
    transfer-format many-answers;
    transfers-per-ns 500;
    recursive-clients 100000;
    max-transfer-time-in 5;
    transfers-in 300;
    transfers-out 300;
    querylog yes;
};

data/bind/etc/named.conf.local

zone "sd.com" {
	type master;
	file "/etc/bind/zones/sd.com.zone";
    allow-update { 127.0.0.1; };
	};
zone "bgt.sdi" {
	type master;
	file "/etc/bind/zones/bgt.sdi.zone";
    allow-update { 127.0.0.1; };
	};
zone "con.sdi" {
	type master;
	file "/etc/bind/zones/con.sdi.zone";
    allow-update { 127.0.0.1; };
	};

3、同步域名zone配置到bind9

参考：https://help.aliyun.com/document_detail/102718.html

这里写入到shell，编写任务计划，批量执行同步即可。

`1`	`/5 * * * /bin/bash /opt/bind9/update.sh`

update.sh

#!/bin/bash
cd /opt/bind9/tools
./Zone_file_sync config.json
chown 101.101 /opt/bind9/data/bind/etc/zones/*
docker exec bind9_bind_1 bash -c "rndc -c /etc/bind/rndc.conf freeze;rndc -c /etc/bind/rndc.conf reload;rndc -c /etc/bind/rndc.conf thaw"

配置文件tools/config.json

{
  "accessKeyId": "LIDD5ssssmGExzGsdfsY6sJrSqo",
  "accessKeySecret": "C5N1TTESt74KhSTsswSSSWiz2",
  "zone": [
    {
      "zoneName": "sd.com",
      "zoneId": "2a4dc4e0sdsfa5d36a3b88ab6482saf",
      "filePath": "/opt/bind9/data/bind/etc/zones/sd.com.zone"
    },
    {
      "zoneName": "bgt.sdi",
      "zoneId": "f842ca07ccsd6f35d9e294d55a0c900",
      "filePath": "/opt/bind9/data/bind/etc/zones/bgt.sdi.zone"
    },
    {
      "zoneName": "con.sdi",
      "zoneId": "beb4d911addsf2bd86425ds280e7bbf2",
      "filePath": "/opt/bind9/data/bind/etc/zones/con.sdi.zone"
    }
  ]
}

三、dnsmasq部署

1、安装配置dnsmasq

# yum -y install dnsmasq
# cat >> /etc/dnsmasq.conf << Tag
port=53
proxy-dnssec
no-hosts  #不加载本地/etc/hosts
no-negcache
dns-forward-max=2000
server=114.114.114.114 #指定上游dns服务器
server=223.5.5.5 #指定上游dns服务器
server=/sd.com/127.0.0.1#15353  #转发到指定dns指定端口
server=/bgt.sdi/127.0.0.1#15353
server=/con.sdi/127.0.0.1#15353
log-queries  #记录dns查询日志
log-facility=/var/log/dnsmasq/dnsmasq.log #指定日志路径
log-async=50
cache-size=100000
Tag
# systemctl  start dnsmasq
# systemctl  enable dnsmasq

2、配置dnsmasq日志轮训策略：

#cat >> /etc/logrotate.d/dnsmasq Tag
/var/log/dnsmasq/dnsmasq.log {
notifempty
daily
dateext
rotate 15
sharedscripts
postrotate
[ ! -f /var/run/dnsmasq.pid ] || kill -USR2 `cat /var/run/dnsmasq.pid`
endscript
}
Tag

测试执行

`1`	`logrotate -vf /etc/logrotate.conf`

四、新增一个新zone，需要做哪些操作？

bind和dnsmasq需要如下变更:

例如：新增test.com

1、/etc/dnsmasq.conf增加server=/test.com/127.0.0.1#15353

2、/opt/bind9/data/bind/etc/named.conf.local新增如下：

zone "test.com" {
    type master;
    file "/etc/bind/zones/test.com.zone";
    allow-update { 127.0.0.1; };
    forwarders {};
    };

五、k8s集群内部如何接入内部DNS

1、修改集群节点/etc/resolv.conf为内部DNS

1
2
3

options timeout:1 attempts:1 rotate
nameserver 192.168.1.211
nameserver 192.168.1.212

2、coredns对于无法解析的直接forward走

Corefile: |
  .:53 {
      errors
      health
      kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods insecure
         upstream
         fallthrough in-addr.arpa ip6.arpa
      }
      prometheus :9153
      forward . /etc/resolv.conf
      cache 30
      loop
      reload
      loadbalance
  }

至此，整个内部DNS解析实现完成。

terraform自动化创建ECS

wzdushu@gmail.com (wanzi) — Fri, 26 Feb 2021 17:22:42 +0800

快速创建一台阿里云ECS主机

指定terraform版本

这里我们指定了阿里云provider版本信息，并设置了terraform的版本要求

# mkdir aliyun-ecs-one && cd aliyun-ecs-one
# touch versions.tf
# vim versions.tf
 terraform {
  required_providers {
    alicloud = {
      source  = "aliyun/alicloud"
      version = "1.115.1"
    }
  }

  required_version = ">= 0.12"
}

配置变量

这里主要指定密钥对、云region、ECS账户和镜像信息

# vim variables.tf
# 阿里云子账户access_key
variable "alicloud_access_key" {
  default     = "LTAI4GBXXXXXXXXXXXXXXXXXXXXXX"
  description = "The Alicloud Access Key ID to launch resources. Support to environment 'ALICLOUD_ACCESS_KEY'."
}

# 阿里云子账户secret_key
variable "alicloud_secret_key" {
  default     = "4Z4gbl3dXXXXXXXXXXXXXXXXXXXXX"
  description = "The Alicloud Access Secret Key to launch resources.  Support to environment 'ALICLOUD_SECRET_KEY'."
}

# 阿里云区域，这里为杭州地区
variable "region" {
  default     = "cn-hangzhou"
  description = "The Alicloud region resources.  Support to environment 'REGION'."
}

# 设置阿里云杭州区域可用机房，这里设置为cn-hangzhou-i
variable "availability_zone" {
  description = "The available zone to launch ecs instance and other resources."
  default     = "cn-hangzhou-i"
}

# 设置镜像版本
variable "image_id" {
  default = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
}

# 设置ECS实例类型，对于
variable "ecs_type" {
  default = "ecs.s6-c1m2.small"
}

# 指定ECS实例密码
variable "ecs_password" {
  default = "Test12345"
}

# 指定ECS实例磁盘类型，这里为普通云盘
variable "disk_category" {
  default = "cloud_efficiency"
}

# 设置磁盘大小
variable "disk_size" {
  default = "40"
}

# 设置上网扣费泪行，默认为PayByTraffic（按流量计费）
variable "internet_charge_type" {
  default = "PayByTraffic"
}

# 公共网络最大传出带宽，从1.7版本，默认设置大于0会自动申请独享公网IP地址
variable "internet_max_bandwidth_out" {
  default = 5
}

配置实例相关资源

这里由于是测试，创建实例需要提前创建vpc，vswitch，安全组，安全组规则

# vim main.tf
provider "alicloud" {
  region     = var.region
  access_key = var.alicloud_access_key
  secret_key = var.alicloud_secret_key
}

resource "alicloud_vpc" "vpc" {
  name       = "tf_test_foo"
  cidr_block = "10.100.0.0/16"
}

resource "alicloud_vswitch" "vsw" {
  vpc_id            = alicloud_vpc.vpc.id
  cidr_block        = "10.100.0.0/24"
  availability_zone = var.availability_zone
}

resource "alicloud_security_group" "default" {
  name   = "default"
  vpc_id = alicloud_vpc.vpc.id
}

resource "alicloud_security_group_rule" "allow_all_tcp" {
  type              = "ingress"
  ip_protocol       = "tcp"
  nic_type          = "intranet"
  policy            = "accept"
  port_range        = "1/65535"
  priority          = 1
  security_group_id = alicloud_security_group.default.id
  cidr_ip           = "0.0.0.0/0"
}

resource "alicloud_instance" "wanzi_test" {
  # cn-hangzhou
  availability_zone = var.availability_zone
  security_groups   = alicloud_security_group.default.*.id

  instance_type        = var.ecs_type
  system_disk_category = var.disk_category
  image_id             = var.image_id
  instance_name        = "wanzi_tf001"
  vswitch_id           = alicloud_vswitch.vsw.id
  password             = var.ecs_password
}

执行plan，模拟执行效果

# terraform  plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

创建云主机, 这个过程会请求阿里云API并在本地生成terraform state文件

# terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

  # alicloud_security_group.default will be created
  + resource "alicloud_security_group" "default" {
      + id                  = (known after apply)
      + inner_access        = (known after apply)
      + inner_access_policy = (known after apply)
      + name                = "default"
      + security_group_type = "normal"
      + vpc_id              = (known after apply)
    }
......
......
Plan: 5 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

alicloud_vpc.vpc: Creating...
alicloud_vpc.vpc: Creation complete after 9s [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_security_group.default: Creating...
alicloud_vswitch.vsw: Creating...
alicloud_security_group.default: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_security_group_rule.allow_all_tcp: Creating...
alicloud_security_group_rule.allow_all_tcp: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_vswitch.vsw: Creation complete after 4s [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_instance.wanzi_test: Creating...
alicloud_instance.wanzi_test: Still creating... [10s elapsed]
alicloud_instance.wanzi_test: Still creating... [20s elapsed]
alicloud_instance.wanzi_test: Creation complete after 22s [id=i-bp1gt9mb9asadff9r2zr]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

通过以上操作，我们可以看出已经完成资源的创建，这个过程当前目录也会生成对应tfstate文件，这个数据非常重要，千万不要删除。另外，后续可以通过terraform show查看我们创建资源数据信息。

批量创建多台ECS云主机

配置Module

由于https://registry.terraform.io/上已经有很多优秀的模块，我们这里直接拿来alibaba/ecs-instance/alicloud这个module进行操作即可。

更多关于官方ECS module参考这里：https://github.com/terraform-alicloud-modules/terraform-alicloud-ecs-instance

这里variables.tf和versions.tf配置还是基于第一步配置；这里我们在main.tf增加module资源批量创建ECS配置，如下：

module "tf-instances" {
 source                      = "alibaba/ecs-instance/alicloud"
 region                      = "cn-hangzhou"
 number_of_instances         = "3"
 vswitch_id                  = alicloud_vswitch.vsw.id
 group_ids                   = [alicloud_security_group.default.id]
 private_ips                 = ["10.100.0.10", "10.100.0.11", "10.100.0.12"]
 image_ids                   = ["ubuntu_18_04_64_20G_alibase_20190624.vhd"]
 instance_type               = var.ecs_type
 internet_max_bandwidth_out  = 10
 associate_public_ip_address = true
 instance_name               = "my_module_instances_"
 host_name                   = "wanzi-cluster"
 internet_charge_type        = "PayByTraffic"
 password                    = var.ecs_password
 system_disk_category        = "cloud_ssd"
 data_disks = [
  {
    disk_category = "cloud_ssd"
    disk_name     = "my_module_disk"
    disk_size     = "50"
  }
 ]
}

这里需要注意默认情况下，internet_max_bandwidth_out配置以后，会自动申请一个独享公网IP地址，对于没有这个需求的，可以不用配置。

批量创建资源

➜ terraform apply
alicloud_vpc.vpc: Refreshing state... [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_vswitch.vsw: Refreshing state... [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_security_group.default: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_security_group_rule.allow_all_tcp: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_instance.wanzi_test: Refreshing state... [id=i-bp1gt9mb9asadff9r2zr]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.tf-instances.alicloud_instance.this[0] will be created
  + resource "alicloud_instance" "this" {
      + availability_zone             = (known after apply)
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + description                   = "An ECS instance came from terraform-alicloud-modules/ecs-instance"
      + dry_run                       = false
      + host_name                     = "wanzi-cluster001"
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "my_module_instances_001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_charge_type          = "PayByTraffic"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 10
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = "10.100.0.10"
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_enhancement_strategy = "Active"
      + security_groups               = [
          + "sg-bp11s5pka9pxtj6pn4xq",
        ]
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_ssd"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + tags                          = {
          + "Name" = "my_module_instances_001"
        }
      + volume_tags                   = {
          + "Name" = "my_module_instances_001"
        }
      + vswitch_id                    = "vsw-bp1wgpgz9z8y2lfsl2beo"

      + data_disks {
          + category             = "cloud_efficiency"
          + delete_with_instance = true
          + encrypted            = false
          + name                 = "TF_ECS_Disk"
          + performance_level    = (known after apply)
          + size                 = 40
        }
    }

  # module.tf-instances.alicloud_instance.this[1] will be created
  + resource "alicloud_instance" "this" {
      + availability_zone             = (known after apply)
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + description                   = "An ECS instance came from terraform-alicloud-modules/ecs-instance"
      + dry_run                       = false
      + host_name                     = "wanzi-cluster002"
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "my_module_instances_002"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_charge_type          = "PayByTraffic"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 10
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = "10.100.0.11"
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_enhancement_strategy = "Active"
      + security_groups               = [
          + "sg-bp11s5pka9pxtj6pn4xq",
        ]
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_ssd"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + tags                          = {
          + "Name" = "my_module_instances_002"
        }
      + volume_tags                   = {
          + "Name" = "my_module_instances_002"
        }
      + vswitch_id                    = "vsw-bp1wgpgz9z8y2lfsl2beo"

      + data_disks {
          + category             = "cloud_efficiency"
          + delete_with_instance = true
          + encrypted            = false
          + name                 = "TF_ECS_Disk"
          + performance_level    = (known after apply)
          + size                 = 40
        }
    }

  # module.tf-instances.alicloud_instance.this[2] will be created
  + resource "alicloud_instance" "this" {
      + availability_zone             = (known after apply)
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + description                   = "An ECS instance came from terraform-alicloud-modules/ecs-instance"
      + dry_run                       = false
      + host_name                     = "wanzi-cluster003"
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "my_module_instances_003"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_charge_type          = "PayByTraffic"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 10
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = "10.100.0.12"
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_enhancement_strategy = "Active"
      + security_groups               = [
          + "sg-bp11s5pka9pxtj6pn4xq",
        ]
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_ssd"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + tags                          = {
          + "Name" = "my_module_instances_003"
        }
      + volume_tags                   = {
          + "Name" = "my_module_instances_003"
        }
      + vswitch_id                    = "vsw-bp1wgpgz9z8y2lfsl2beo"

      + data_disks {
          + category             = "cloud_efficiency"
          + delete_with_instance = true
          + encrypted            = false
          + name                 = "TF_ECS_Disk"
          + performance_level    = (known after apply)
          + size                 = 40
        }
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.tf-instances.alicloud_instance.this[2]: Creating...
module.tf-instances.alicloud_instance.this[1]: Creating...
module.tf-instances.alicloud_instance.this[0]: Creating...
module.tf-instances.alicloud_instance.this[1]: Still creating... [10s elapsed]
module.tf-instances.alicloud_instance.this[2]: Still creating... [10s elapsed]
module.tf-instances.alicloud_instance.this[0]: Still creating... [10s elapsed]
module.tf-instances.alicloud_instance.this[1]: Still creating... [20s elapsed]
module.tf-instances.alicloud_instance.this[0]: Still creating... [20s elapsed]
module.tf-instances.alicloud_instance.this[2]: Still creating... [20s elapsed]
module.tf-instances.alicloud_instance.this[0]: Creation complete after 21s [id=i-bp1hwbo4htk8sbwxtk6o]
module.tf-instances.alicloud_instance.this[1]: Creation complete after 21s [id=i-bp17lh41gywyih0xg6we]
module.tf-instances.alicloud_instance.this[2]: Creation complete after 22s [id=i-bp11zlrl6vxeaerz4ad0]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

至此，整个创建多ECS实例的操作已经完成，后续如果对当前已经部署ecs资源有调整，进行基本write/plan/apply操作即可，这个过程会重启阿里云实例。

terraform安装与命令详解

wzdushu@gmail.com (wanzi) — Thu, 25 Feb 2021 17:22:42 +0800

安装Terraform

Mac系统安装

1
2

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Linux系统安装

ubuntu安装

1
2
3

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install terraform

centos系统

1
2
3

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install terraform

验证安装

# terraform -v
Terraform v0.14.3

Your version of Terraform is out of date! The latest version
is 0.14.7. You can update by downloading from https://www.terraform.io/downloads.html
# terraform
Usage: terraform [global options] <subcommand> [args]

The available commands for execution are listed below.
The primary workflow commands are given first, followed by
less common or more advanced commands.

Main commands:
  init          Prepare your working directory for other commands
  validate      Check whether the configuration is valid
  plan          Show changes required by the current configuration
  apply         Create or update infrastructure
  destroy       Destroy previously-created infrastructure

All other commands:
  console       Try Terraform expressions at an interactive command prompt
  fmt           Reformat your configuration in the standard style
  force-unlock  Release a stuck lock on the current workspace
  get           Install or upgrade remote Terraform modules
  graph         Generate a Graphviz graph of the steps in an operation
  import        Associate existing infrastructure with a Terraform resource
  login         Obtain and save credentials for a remote host
  logout        Remove locally-stored credentials for a remote host
  output        Show output values from your root module
  providers     Show the providers required for this configuration
  refresh       Update the state to match remote systems
  show          Show the current state or a saved plan
  state         Advanced state management
  taint         Mark a resource instance as not fully functional
  untaint       Remove the 'tainted' state from a resource instance
  version       Show the current Terraform version
  workspace     Workspace management

Global options (use these before the subcommand, if any):
  -chdir=DIR    Switch to a different working directory before executing the
                given subcommand.
  -help         Show this help output, or the help for a specified subcommand.
  -version      An alias for the "version" subcommand.

terraform命令之资源管理

资源初始化

对于一个terraform资源项目，我这里创建了3个基本文件，分别为：main.tf（入口文件），variables.tf（变量信息），versions.tf（版本信息）

# ls 
main.tf     variables.tf      versions.tf
# terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of aliyun/alicloud from the dependency lock file
- Using aliyun/alicloud v1.115.1 from the shared cache directory

Terraform has been successfully initialized!

格式化terraform文件

fmt默认会回格式化处理当前目录下.tf文件，并格式为标准的tf格式。

# terraform fmt 
main.tf
variables.tf
versions.tf
# terraform fmt -diff  #格式化处理
main.tf
--- old/main.tf
+++ new/main.tf
@@ -1,7 +1,7 @@
 provider "alicloud" {
   region     = var.region
   access_key = var.alicloud_access_key
-  secret_key =  var.alicloud_secret_key
+  secret_key = var.alicloud_secret_key
 }

 resource "alicloud_vpc" "vpc" {
@@ -12,7 +12,7 @@
 resource "alicloud_vswitch" "vsw" {
   vpc_id            = alicloud_vpc.vpc.id
   cidr_block        = "10.100.0.0/24"
-  availability_zone =  var.availability_zone
+  availability_zone = var.availability_zone
 }

 resource "alicloud_security_group" "default" {
variables.tf
--- old/variables.tf
+++ new/variables.tf
@@ -4,7 +4,7 @@
 }

 variable "alicloud_secret_key" {
-  default                     = "4Z4gbl3d9TGz9jWobv9MPwInvyH2Kf"
+  default     = "4Z4gbl3d9TGz9jWobv9MPwInvyH2Kf"
   description = "The Alicloud Access Secret Key to launch resources.  Support to environment 'ALICLOUD_SECRET_KEY'."
 }

创建资源计划

terraform plan 会检查一组更改的执行计划是否符合您的期望，而不会更改实际资源或状态。

# terraform  plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

创建云资源

terraform apply 会自动生成一个资源创建计划，并批准执行该计划，同时在当前目录下会生成tfstate文件

# terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

  # alicloud_security_group.default will be created
  + resource "alicloud_security_group" "default" {
      + id                  = (known after apply)
      + inner_access        = (known after apply)
      + inner_access_policy = (known after apply)
      + name                = "default"
      + security_group_type = "normal"
      + vpc_id              = (known after apply)
    }
......
......
Plan: 5 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

alicloud_vpc.vpc: Creating...
alicloud_vpc.vpc: Creation complete after 9s [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_security_group.default: Creating...
alicloud_vswitch.vsw: Creating...
alicloud_security_group.default: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_security_group_rule.allow_all_tcp: Creating...
alicloud_security_group_rule.allow_all_tcp: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_vswitch.vsw: Creation complete after 4s [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_instance.wanzi_test: Creating...
alicloud_instance.wanzi_test: Still creating... [10s elapsed]
alicloud_instance.wanzi_test: Still creating... [20s elapsed]
alicloud_instance.wanzi_test: Creation complete after 22s [id=i-bp1gt9mb9asadff9r2zr]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

查看创建的资源信息

terraform show 会查看当前项目创建了哪些资源数据，

terraform show -json 以json形式查看数据

# terraform show
# alicloud_instance.wanzi_test:
resource "alicloud_instance" "wanzi_test" {
    availability_zone          = "cn-hangzhou-i"
    deletion_protection        = false
    dry_run                    = false
    host_name                  = "iZbp1gt9mb9asadff9r2zrZ"
    id                         = "i-bp1gt9mb9asadff9r2zr"
    image_id                   = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
    instance_charge_type       = "PostPaid"
    instance_name              = "wanzi_tf001"
    instance_type              = "ecs.s6-c1m2.small"
    internet_charge_type       = "PayByTraffic"
    internet_max_bandwidth_in  = -1
    internet_max_bandwidth_out = 0
    password                   = (sensitive value)
    private_ip                 = "10.100.0.234"
    security_groups            = [
        "sg-bp11s5pka9pxtj6pn4xq",
    ]
    spot_price_limit           = 0
    spot_strategy              = "NoSpot"
    status                     = "Running"
    subnet_id                  = "vsw-bp1wgpgz9z8y2lfsl2beo"
    system_disk_category       = "cloud_efficiency"
    system_disk_size           = 40
    volume_tags                = {}
    vswitch_id                 = "vsw-bp1wgpgz9z8y2lfsl2beo"
}

# alicloud_security_group.default:
resource "alicloud_security_group" "default" {
    id                  = "sg-bp11s5pka9pxtj6pn4xq"
    inner_access        = true
    inner_access_policy = "Accept"
    name                = "default"
    security_group_type = "normal"
    vpc_id              = "vpc-bp1kulcyygsi727aay4hd"
}

# alicloud_security_group_rule.allow_all_tcp:
resource "alicloud_security_group_rule" "allow_all_tcp" {
    cidr_ip           = "0.0.0.0/0"
    id                = "sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1"
    ip_protocol       = "tcp"
    nic_type          = "intranet"
    policy            = "accept"
    port_range        = "1/65535"
    priority          = 1
    security_group_id = "sg-bp11s5pka9pxtj6pn4xq"
    type              = "ingress"
}

# alicloud_vpc.vpc:
resource "alicloud_vpc" "vpc" {
    cidr_block        = "10.100.0.0/16"
    id                = "vpc-bp1kulcyygsi727aay4hd"
    name              = "tf_test_foo"
    resource_group_id = "rg-acfm2ogp24u3rcy"
    route_table_id    = "vtb-bp1wy8srerq12rta02r03"
    router_id         = "vrt-bp1apvobefvhshksnnwvm"
    router_table_id   = "vtb-bp1wy8srerq12rta02r03"
}

# alicloud_vswitch.vsw:
resource "alicloud_vswitch" "vsw" {
    availability_zone = "cn-hangzhou-i"
    cidr_block        = "10.100.0.0/24"
    id                = "vsw-bp1wgpgz9z8y2lfsl2beo"
    vpc_id            = "vpc-bp1kulcyygsi727aay4hd"
}

标记污点

terrraform taint 命令用于把某个资源标记为“被污染”状态，当再次执行 apply 命令时，这个被污染的资源将会被先释放，然后再创建一个新的，相当于对这个特定资源做了先删除后新建的操作。

1
2

# terraform taint alicloud_instance.wanzi_test
Resource instance alicloud_instance.wanzi_test has been marked as tainted.

而terrraform untaint正好相反，用于取消“被污染”标记，使其恢复到正常的状态。

1
2

# terraform untaint alicloud_instance.wanzi_test
Resource instance alicloud_instance.wanzi_test has been successfully untainted.

销毁云资源数据

terraform destory 将根据当前资源配置，销毁云端资源数据

#terraform destroy

Plan: 0 to add, 0 to change, 5 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

alicloud_security_group_rule.allow_all_tcp: Destroying... [id=sg-bp10tup89oothxz8tny1:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_instance.wanzi_test: Destroying... [id=i-bp10ukz4nlr894mhebgl]
alicloud_security_group_rule.allow_all_tcp: Destruction complete after 0s
alicloud_instance.wanzi_test: Still destroying... [id=i-bp10ukz4nlr894mhebgl, 10s elapsed]
alicloud_instance.wanzi_test: Still destroying... [id=i-bp10ukz4nlr894mhebgl, 20s elapsed]
alicloud_instance.wanzi_test: Destruction complete after 28s
alicloud_security_group.default: Destroying... [id=sg-bp10tup89oothxz8tny1]
alicloud_vswitch.vsw: Destroying... [id=vsw-bp1ap7ccst3fjxnw4pnza]
alicloud_security_group.default: Destruction complete after 9s
alicloud_vswitch.vsw: Still destroying... [id=vsw-bp1ap7ccst3fjxnw4pnza, 10s elapsed]
alicloud_vswitch.vsw: Destruction complete after 20s
alicloud_vpc.vpc: Destroying... [id=vpc-bp1obwt5ded2i0zlbu052]
alicloud_vpc.vpc: Destruction complete after 3s

Destroy complete! Resources: 5 destroyed.

将云端数据导入到本地项目

terraform import 通过云端实例ID来生成本地资源数据，本地目录会生成terraform.tfstate文件，对于本地项目已存在数据的导入前请先备份tfstate文件和.terraform目录；对于已经导入到本地的数据，可以通过terraform show展示出terrafrom文件格式，copy出来并进一步处理，即可得到tf资源文件内容。

# cat yunduan.tf
resource "alicloud_instance" "test999" {
  # (resource arguments)
}
#
# terraform import alicloud_instance.test999 i-bp1etiv4002h9q27lb97
alicloud_instance.test999: Importing from ID "i-bp1etiv4002h9q27lb97"...
alicloud_instance.test999: Import prepared!
  Prepared alicloud_instance for import
alicloud_instance.test999: Refreshing state... [id=i-bp1etiv4002h9q27lb97]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
# cat  terraform.tfstate
{
  "version": 4,
  "terraform_version": "0.14.3",
  "serial": 1,
  "lineage": "779fad5e-b076-8cfd-6041-f6eef8c88b8a",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "alicloud_instance",
      "name": "test999",
      "provider": "provider[\"registry.terraform.io/aliyun/alicloud\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "allocate_public_ip": null,
            "auto_release_time": "",
            "auto_renew_period": null,
            "availability_zone": "cn-hangzhou-i",
            "credit_specification": "",
            "data_disks": [],
            "deletion_protection": false,
            "description": "",
            "dry_run": null,
            "force_delete": null,
            "host_name": "iZbp1etiv4002h9q27lb97Z",
            "id": "i-bp1etiv4002h9q27lb97",
            "image_id": "ubuntu_18_04_64_20G_alibase_20190624.vhd",
            "include_data_disks": null,
            "instance_charge_type": "PostPaid",
            "instance_name": "wanzi_tf001",
            "instance_type": "ecs.s6-c1m2.small",
            "internet_charge_type": "PayByTraffic",
            "internet_max_bandwidth_in": -1,
            "internet_max_bandwidth_out": 0,
            "io_optimized": null,
            "is_outdated": null,
            "key_name": "",
            "kms_encrypted_password": null,
            "kms_encryption_context": null,
            "password": "",
            "period": null,
            "period_unit": null,
            "private_ip": "10.100.0.169",
            "public_ip": "",
            "renewal_status": null,
            "resource_group_id": "",
            "role_name": "",
            "security_enhancement_strategy": null,
            "security_groups": [
              "sg-bp14pij6g7sjmn9bz92a"
            ],
            "spot_price_limit": 0,
            "spot_strategy": "NoSpot",
            "status": "Running",
            "subnet_id": "vsw-bp1c966jdtiw1qwh2tng8",
            "system_disk_auto_snapshot_policy_id": "",
            "system_disk_category": "cloud_efficiency",
            "system_disk_description": null,
            "system_disk_name": null,
            "system_disk_performance_level": "",
            "system_disk_size": 40,
            "tags": {},
            "timeouts": {
              "create": null,
              "delete": null,
              "update": null
            },
            "user_data": "",
            "volume_tags": {},
            "vswitch_id": "vsw-bp1c966jdtiw1qwh2tng8"
          },
          "sensitive_attributes": [],
          "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6MTIwMDAwMDAwMDAwMCwidXBkYXRlIjo2MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjAifQ=="
        }
      ]
    }
  ]
}
# terraform show
# alicloud_instance.test999:
resource "alicloud_instance" "test999" {
    availability_zone          = "cn-hangzhou-i"
    deletion_protection        = false
    host_name                  = "iZbp1etiv4002h9q27lb97Z"
    id                         = "i-bp1etiv4002h9q27lb97"
    image_id                   = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
    instance_charge_type       = "PostPaid"
    instance_name              = "wanzi_tf001"
    instance_type              = "ecs.s6-c1m2.small"
    internet_charge_type       = "PayByTraffic"
    internet_max_bandwidth_in  = -1
    internet_max_bandwidth_out = 0
    private_ip                 = "10.100.0.169"
    security_groups            = [
        "sg-bp14pij6g7sjmn9bz92a",
    ]
    spot_price_limit           = 0
    spot_strategy              = "NoSpot"
    status                     = "Running"
    subnet_id                  = "vsw-bp1c966jdtiw1qwh2tng8"
    system_disk_category       = "cloud_efficiency"
    system_disk_size           = 40
    tags                       = {}
    volume_tags                = {}
    vswitch_id                 = "vsw-bp1c966jdtiw1qwh2tng8"

    timeouts {}
}

terraform资源关系绘图

每个模板定义的资源之间都存在不同程度的关系，terraform graph可以绘制资源关系大图，如下：

# terraform graph
digraph {
        compound = "true"
        newrank = "true"
        subgraph "root" {
                "[root] alicloud_instance.wanzi_test (expand)" [label = "alicloud_instance.wanzi_test", shape = "box"]
                "[root] alicloud_security_group.default (expand)" [label = "alicloud_security_group.default", shape = "box"]
                "[root] alicloud_security_group_rule.allow_all_tcp (expand)" [label = "alicloud_security_group_rule.allow_all_tcp", shape = "box"]
                "[root] alicloud_vpc.vpc (expand)" [label = "alicloud_vpc.vpc", shape = "box"]
                "[root] alicloud_vswitch.vsw (expand)" [label = "alicloud_vswitch.vsw", shape = "box"]
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" [label = "provider[\"registry.terraform.io/aliyun/alicloud\"]", shape = "diamond"]
                "[root] var.alicloud_access_key" [label = "var.alicloud_access_key", shape = "note"]
                "[root] var.alicloud_secret_key" [label = "var.alicloud_secret_key", shape = "note"]
                "[root] var.availability_zone" [label = "var.availability_zone", shape = "note"]
                "[root] var.disk_category" [label = "var.disk_category", shape = "note"]
                "[root] var.disk_size" [label = "var.disk_size", shape = "note"]
                "[root] var.ecs_password" [label = "var.ecs_password", shape = "note"]
                "[root] var.ecs_type" [label = "var.ecs_type", shape = "note"]
                "[root] var.image_id" [label = "var.image_id", shape = "note"]
                "[root] var.internet_charge_type" [label = "var.internet_charge_type", shape = "note"]
                "[root] var.internet_max_bandwidth_out" [label = "var.internet_max_bandwidth_out", shape = "note"]
                "[root] var.region" [label = "var.region", shape = "note"]
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] alicloud_security_group.default (expand)"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] alicloud_vswitch.vsw (expand)"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.disk_category"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.ecs_password"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.ecs_type"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.image_id"
                "[root] alicloud_security_group.default (expand)" -> "[root] alicloud_vpc.vpc (expand)"
                "[root] alicloud_security_group_rule.allow_all_tcp (expand)" -> "[root] alicloud_security_group.default (expand)"
                "[root] alicloud_vpc.vpc (expand)" -> "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]"
                "[root] alicloud_vswitch.vsw (expand)" -> "[root] alicloud_vpc.vpc (expand)"
                "[root] alicloud_vswitch.vsw (expand)" -> "[root] var.availability_zone"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] alicloud_instance.wanzi_test (expand)"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] alicloud_security_group_rule.allow_all_tcp (expand)"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] var.disk_size"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] var.internet_charge_type"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] var.internet_max_bandwidth_out"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"] (close)" -> "[root] alicloud_instance.wanzi_test (expand)"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"] (close)" -> "[root] alicloud_security_group_rule.allow_all_tcp (expand)"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" -> "[root] var.alicloud_access_key"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" -> "[root] var.alicloud_secret_key"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" -> "[root] var.region"
                "[root] root" -> "[root] meta.count-boundary (EachMode fixup)"
                "[root] root" -> "[root] provider[\"registry.terraform.io/aliyun/alicloud\"] (close)"
        }
}

该命令的结果还可以通过命令 terraform graph | dot -Tsvg > graph.svg 直接导出为一张图片（需要提前安装graphviz： brew install graphviz ）

`1`	`terraform graph \| dot -Tsvg > ~/Downloads/graph.svg`

查看graph.svg可以看到各个资源之间关系图谱：

terraform命令之State管理

查看当前state里存放所有资源

# terraform state list
alicloud_instance.wanzi_test
alicloud_security_group.default
alicloud_security_group_rule.allow_all_tcp
alicloud_vpc.vpc
alicloud_vswitch.vsw

查看某一个resource具体数据

# terraform state show alicloud_vswitch.vsw
# alicloud_vswitch.vsw:
resource "alicloud_vswitch" "vsw" {
    availability_zone = "cn-hangzhou-i"
    cidr_block        = "10.100.0.0/24"
    id                = "vsw-bp1wgpgz9z8y2lfsl2beo"
    vpc_id            = "vpc-bp1kulcyygsi727aay4hd"
}

移除特定资源

terraform state rm <资源类型>.<资源名称> state rm 命令用于将state中的某个资源移除，但是实际上并不会真正删除这个资源，另外也可以通过import操作从云端恢复到本地。

# terraform state rm  alicloud_security_group.default
Removed alicloud_security_group.default
Successfully removed 1 resource instance(s).
# terraform state list
alicloud_instance.wanzi_test
alicloud_vpc.vpc
alicloud_vswitch.vsw
# terraform import alicloud_security_group.default sg-bp11s5pka9pxtj6pn4xq
alicloud_security_group.default: Importing from ID "sg-bp11s5pka9pxtj6pn4xq"...
alicloud_security_group.default: Import prepared!
  Prepared alicloud_security_group for import
alicloud_security_group.default: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

刷新资源

terraform refresh刷新当前state内容，调用云API拉取最新数据写入state文件

# terraform refresh
alicloud_vpc.vpc: Refreshing state... [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_vswitch.vsw: Refreshing state... [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_security_group.default: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_instance.wanzi_test: Refreshing state... [id=i-bp1gt9mb9asadff9r2zr]

自动化编排工具Terraform介绍

wzdushu@gmail.com (wanzi) — Wed, 24 Feb 2021 17:22:42 +0800

Terraform是什么？：

Terraform是由HashiCorp公司在2014年左右推出的开源资源编排工具, 目前几乎所有的主流云服务商都支持Terraform，包括阿里云、腾讯云、华为云、AWS、Azure、百度云等等。目前很多公司都基于terraform构建自己的基础架构。

诞生背景：传统运维模式下，业务上线需经过设备采购，机器上架，网络环境搭建和系统安装等准备阶段。随着云计算的兴起，各大公有云厂商均提供了非常友好的交互界面，用户借助一个浏览器就可以按需采购各种云资源，快速实现业务架构的搭建。然而，随着业务架构的不断扩展，云资源采购的规模和种类也在持续增加。当用户需要快速采购大量不同类型的云资源时，云管理页面间大量的交互操作反而降低了云资源的采购效率。在阿里云控制台上初始化一个经典的VPC网络架构，从创建VPC、交换机VSwitch到创建Nat网关、弹性IP再到配置路由等工作，大概要花费20分钟甚至更久。同时，工作成果的不可复制性，带来的是跨Region和跨云平台场景下的重复劳动。

事实上，对业务运维人员而言，只关心对资源的配置，无需关心这些资源的创建步骤。如同喝咖啡，只需要告诉服务员喝什么，加不加冰等就够了。如果有一份完整的云资源采购清单，这张清单清楚的记录了所需要购买的云资源的种类，规格，数量以及各云资源之间的关系，然后一键完成购买，并且当业务需求发生变化时，只需要变更清单就可以实现对云资源的快速变更，那么效率就会提高很多。在云计算中这被称作资源编排，目前很多云平台也提供了资源编排的能力，如阿里云的ROS，AWS的CloudFormation等。

将云资源、服务或者操作步骤以代码的形式定义在模板中，借助编排引擎，实现资源的自动化管理，这就是基础设施即代码（Infrastructure as Code，简称IaC），也是资源编排最高效的实现模式。然而，多种云编排服务带来的是高昂的学习成本、低效的代码复用率和复杂的多云协同工作流程。每一种服务仅限于管理自家的单一云平台上，无法满足对多个云平台，多种层级（如IaaS，PaaS）资源的统一管理。如何解决如上问题，是否可以使用统一的编排工具，共用一套语法实现对包括阿里云在内的多云的统一管理呢？所以这个时候就诞生Terraform，来解决这些问题。

Terrafrom功能和作用：

功能点

IaC：infrastructure as code，用代码管理基础设施
执行计划：显示terraform apply时执行的操作
资源图：构建所有资源的图形
变更自动化：基于执行计划和资源图，可以清晰知道要变更的内容和顺序总结：terraform用于各类基础设施资源初始化，支持多种云平台，支持第三方服务对接

作用

使用不同provider的API，包装抽象成Terraform的标准代码结构
用户不需要了解每个云计算厂商的API细节，降低了部署难度

Terraform架构

Terraform本身是基于插件的架构，可扩展性很强，可以方便程序员对Terraform进行扩展。Terraform从逻辑上可以分为两层，核心层（Terraform Core）和插件层（Terraform Provider）。

核心层

核心层其实就是terraform的命令行工具，它是用go语言开发的，它负责：

读取.tf配置，进行变量替换
资源状态文件管理
分析资源关系，绘制图谱
依赖关系图谱，创建资源根据依赖关系，创建资源；对于没有依赖关系的资源，会并行进行创建(缺省10个并行进程），这也是Terraform能够高效快速管理云资源的原因。
用RPC调用插件层

插件层

插件层也是由go语言开发的，Terraform有超过250个不同的插件，它们负责：

接受核心层的RPC调用
具体提供某一项服务的执行

插件层又有两种：

Provider

Provider，负责与外界API的集成，比如阿里云Provider就提供了在阿里云创建、修改、删除云资源的功能。这个插件负责和阿里云云API的接口交互，并提供一层抽象，这样程序员可以在不了解API细节的情况下，通过terraform来编排资源。它负责：

始化以及外界API通信
外界API的认证
定义云资源与外界服务的关系

比如常见provider:

阿里云： https://github.com/aliyun/terraform-provider-alicloud
百度云：https://github.com/baidubce/terraform-provider-baiducloud
腾讯云：https://github.com/tencentcloudstack/terraform-provider-tencentcloud
华为云：https://github.com/huaweicloud/terraform-provider-huaweicloud
ucloud：https://github.com/ucloud/terraform-provider-ucloud
qingcloud：https://github.com/yunify/terraform-provider-qingcloud
AWS：https://github.com/hashicorp/terraform-provider-aws
Azure：https://github.com/terraform-providers/terraform-provider-azurerm
GoogleCloud：https://github.com/hashicorp/terraform-provider-google

Provisioner

Provisioner，负责在资源创建或者删除完成后，执行一些脚本。比如Puppet Provisioner就可以在云虚拟机资源创建完成后，在该资源上下载、安装、配置Puppet agent。

为了方便理解,网络上找了一个组件架构图，简单说明各个组件位置：

对于terraform日常操作，我画了一个基本workflow流程图如下：

terraform关键字解释：

声明式语言（HCL）：

Terraform是通过HashiCorp Configuration Language来编写代码的，HCL是声明式的，也就是说，程序员用HCL来描述整个基础架构应该是什么样的，然后把具体的实施工作交给Terraform就可以了，程序员不需要了解实施的具体步骤和细节，不需要了解terraform如何与云服务商的API进行对接。Terraform会根据代码，自动下载相应的Provider和Provisioner来负责具体步骤和细节。于声明式对应的是命令式。命令式语言是按照步骤执行的，先后顺序很重要，对固定输入执行命令式语言会得到固定的输出。声明式和命令式并无高下之分，只是在云资源编排这一领域，声明式会比较方便实现。我们日常见到的云资源编排工具都是声明式的，包括AWS CloudFormation、Azure Resource Template、Google Cloud Deoplyment Manager。大家如果通过调用腾讯云API来在腾讯云上实施资源编排，那通常就是命令式的。

资源状态文件(state)

Terraform初始化以后，会生成一个状态文件，该状态文件记录了最近一次操作的时间、各资源的相关属性、各变量的当前值、状态文件的版本、等等。

下一次再操作的时候，terraform首先会把当前状态文件与云服务商上的状态进行一次更新，找出是否后有被删除或者更改了的资源，然后再根据.tf文件，决定那些资源需要删除、更新、创建。操作完成后，会重新生成一个状态文件。

Terraform后台(backend)

资源状态文件的完整性比较重要，对于这些文件我们至少需要做到在操作开始时自动加锁，直到操作结束，这样别人无法更改；另外还需要对资源版本变更进行跟踪；对资源文件里敏感信息进行访问控制。

因此backend跟资源状态文件如何读取、存储、锁定，以及terraform apply如何执行严密相关。

terraform缺省使用本地后台，也就是说，状态文件会存放在当前目录下，terraform代码的执行也在本地虚拟机运行。这对一个人管理的云资源是没有问题的，但当团队人员数目加多以后，大家可能都有自己的工作台，但是需要一个共有的地方来存储资源状态文件。这是后就可以用到远程存储。目前terraform支持多种远程存储后台，包括AWS s3,Hashicorp Consul,etcd，Terraform云，以及terraform企业版等等，这些远程后台都提供在远程存储、锁定状态文件。其中terraform企业版提供远程运行terraform，以及其他一些企业级特性。

Terraform模块(module)

Terraform模块就是把一些高度可重用的代码写成模块，方便其他人使用。模块由输入参数、输出参数以及主逻辑组成。这就跟传统编程语言里的函数很像。Terraform提供了公开的模块注册器，模块编写完成以后，只要符合规范，就可以发布到模块注册器中让大家使用。https://registry.terraform.io/