Aliyun on Wanzi's Blog – Notes on life, tech, and DevOps.

Achieving Multi-User Access on a Single GPU Card Using Alibaba Cloud's Open Source Solution

wzdushu@gmail.com (wanzi) — Fri, 30 Jun 2023 18:10:42 +0800

A new AI project has recently been launched, primarily providing online AI experiments for universities. The project has also purchased a GPU server, but it only has one Nvidia Tesla T4 card, which needs to support multiple students doing experiments online simultaneously.

The current online experiment system runs on Kubernetes, so we need to consider GPU sharing in the k8s environment. We have previously tested the Alibaba Cloud GPU card sharing solution; here, I will just record the steps for using it:

Kubernetes cluster version: 1.23.1

Adjusting the K8S Scheduler

Starting from 1.23.+, due to changes in the kube-scheduler scheduling policy, the previous deployment methods are no longer suitable.

Refer to: https://kubernetes.io/docs/reference/scheduling/policies/

1
2

git clone https://github.com/AliyunContainerService/gpushare-scheduler-extender.git
cp config/scheduler-policy-config.yaml /etc/kubernetes/

Since my k8s was installed via kubeasz, the Kubeconfig is located at /etc/kubernetes/kubelet.kubeconfig. You need to update /etc/kubernetes/scheduler-policy-config.yaml as follows:

---
apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /etc/kubernetes/kube-scheduler.kubeconfig
extenders:
- urlPrefix: "http://192.168.233.101:32766/gpushare-scheduler"
  filterVerb: filter
  bindVerb: bind
  enableHTTPS: false
  nodeCacheCapable: true
  managedResources:
  - name: aliyun.com/gpu-mem
    ignoredByScheduler: false
  ignorable: false

Modify the Kube-scheduler startup parameters, /etc/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/opt/kube/bin/kube-scheduler \
  --authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --bind-address=0.0.0.0 \
  --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --config=/etc/kubernetes/scheduler-policy-config.yaml \
  --leader-elect=true \
  --v=2
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

1
2

systemctl  daemon-reload
systemctl  restart  kube-scheduler.service

Installing the Scheduler Extender

Label the GPU machine:

kubectl label node mynode gpushare=true

1
2

cd gpushare-scheduler-extender/config
kubectl apply -f gpushare-schd-extender.yaml

Installing the Plugin

1
2

kubectl create -f https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-rbac.yaml
kubectl create -f https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-ds.yaml

After installation, it should look like this:

1
2
3

# kubectl get pod   -n kube-system |grep gpushare
gpushare-device-plugin-ds-j8blj              1/1     Running   0                30h
gpushare-schd-extender-74796c5f64-7g4bl      1/1     Running   0                29h

Testing the GPU

Modify samples/1.yaml as follows:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-1
  labels:
    app: binpack-1

spec:
  replicas: 1

  selector: # define how the deployment finds the pods it manages
    matchLabels:
      app: binpack-1

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-1

    spec:
      containers:
      - name: binpack-1
        image: cheyang/gpu-player:v2
        resources:
          limits:
            # GiB
            aliyun.com/gpu-count: 1
            aliyun.com/gpu-mem: 5

# kubectl apply -f  samples/1.yaml
deployment.apps/binpack-1 created
# kubectl get pod  |grep binpack
binpack-1-9995bdf69-pk2d4                 1/1     Running   0             12s
# kubectl logs -f binpack-1-9995bdf69-pk2d4
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=5
2023-06-30 09:40:50.890296: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2023-06-30 09:40:50.976283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:03:00.0
totalMemory: 14.75GiB freeMemory: 14.66GiB
2023-06-30 09:40:50.976313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:03:00.0, compute capability: 7.5)

From the logs, GPU sharing has been successful, and 5G of video memory has been allocated.

You can also check using the kubectl extension tool kubectl-inspect-gpushare:

# cd /usr/bin/
# wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare
# chmod u+x /usr/bin/kubectl-inspect-gpushare
# kubectl-inspect-gpushare
NAME             IPADDRESS        GPU0(Allocated/Total)  GPU Memory(GiB)
192.168.233.101  192.168.233.101  9/14                   9/14
--------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
9/14 (64%)

At this point, GPU card sharing is complete, and the next step is to schedule my GPU card using TensorFlow.

Reference documentation:

https://github.com/AliyunContainerService/gpushare-scheduler-extender/blob/master/docs/install.md

Resolve Nginx file upload limits and 504 gateway timeout in Kubernetes

wzdushu@gmail.com (wanzi) — Mon, 08 Nov 2021 17:22:42 +0800

Recently, two recurring issues have arisen in our business operations using the Kubernetes cluster. Here’s a record of the solutions:

Frontend page file upload limited to 1M
POST requests from frontend to backend timing out with a 504 error

Solution for the first issue:
By default, Nginx limits upload size to 1M. To resolve this, add the following configuration in the http, server, or location blocks of the Nginx config:

`1`	`client_max_body_size 800M;`

Solution for the second issue:
By default, Nginx has a 1-minute timeout for reading responses from the backend. To extend this, add the following in the http, server, or location blocks:

1
2
3

proxy_connect_timeout 120;
proxy_read_timeout  900;
proxy_send_timeout 900;

Final Nginx configuration:

server {
    listen 80;
    server_name web_server;
    location / {
        root /usr/share/nginx/html/;
        index index.htm index.html;
        try_files $uri $uri/ /index.html;
    }
    location /business-operation {
        proxy_pass http://business-operat.common:5000;
        
        proxy_ignore_client_abort on; 
        keepalive_timeout 900s;
        client_body_timeout 900s;
        proxy_connect_timeout 120s;
        proxy_read_timeout  900s;
        proxy_send_timeout 900s;
        client_max_body_size 800m;
        proxy_pass_header Authorization;
        proxy_pass_header WWW-Authenticate;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
    }
}

After applying the above Nginx configuration, ensure these annotations are included in the Ingress definition:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "800M"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '120'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '900'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '900'
  name: enterprise-mark
  namespace: frontend
spec:
  rules:
    - host: enterprise-marketing.xxxx.com
      http:
        paths:
          - backend:
              serviceName: enterprise-mark
              servicePort: 80
            path: /
status:
  loadBalancer:
    ingress:
      - ip: 39.107.xxx.xxx

Alibaba Cloud ACK supports both public and internal SLB.

wzdushu@gmail.com (wanzi) — Thu, 21 Oct 2021 17:22:42 +0800

I. Background

You have an ACK cluster.
Nginx Ingress Controller has been successfully deployed and bound to a public-facing SLB.

Note: Kubernetes clusters created via the Alibaba Cloud Container Service console automatically deploy an Nginx Ingress Controller during initialization, which is default-mounted to a public SLB instance.

II. Configuration

1. Create an Internal SLB

In the Alibaba Cloud console, create an internal SLB and bind it to your VPC.

2. Configure Nginx Ingress Controller

# my-nginx-ingress-slb-intranet.yaml
# intranet nginx ingress slb service
apiVersion: v1
kind: Service
metadata:
  # Name the service as nginx-ingress-lb-intranet.
  name: nginx-ingress-lb-intranet
  namespace: kube-system
  labels:
    app: nginx-ingress-lb-intranet
  annotations:
    # Specify the SLB instance type as internal.
    service.beta.kubernetes.io/alicloud-loadbalancer-address-type: intranet
    # Replace with your internal SLB instance ID.
    service.beta.kubernetes.io/alicloud-loadbalancer-id: <YOUR_INTRANET_SLB_ID>
    # Whether to automatically create SLB port listeners (overrides existing ones); can also be configured manually.
    #service.beta.kubernetes.io/alicloud-loadbalancer-force-override-listeners: 'false'
spec:
  type: LoadBalancer
  # Route traffic to other nodes
  externalTrafficPolicy: "Cluster"
  ports:
  - port: 80
    name: http
    targetPort: 80
  - port: 443
    name: https
    targetPort: 443
  selector:
    # Select pods with app=ingress-nginx
    app: ingress-nginx

Apply the service resource:

`1`	`kubectl apply -f my-nginx-ingress-slb-intranet.yaml`

Retrieve the service resource:

1
2
3

# kubectl -n kube-system get svc | grep nginx-ingress-lb
nginx-ingress-lb            LoadBalancer   172.21.15.148   39.107.xxx.xxx   80:32076/TCP,443:30803/TCP     433d
nginx-ingress-lb-intranet   LoadBalancer   172.21.5.0      172.17.193.181   80:32282/TCP,443:30507/TCP     1d

After configuring Ingress to expose services, you can access the service via the public SLB. Other services within the same VPC can directly access the service through the internal SLB.

3. Create Ingress Service

Once the Ingress controller is configured, creating an Ingress requires no special setup—just like before. Example:

# Source: prometheusalert/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: RELEASE-NAME-prometheusalert
  labels:
    app.kubernetes.io/name: prometheusalert
    helm.sh/chart: prometheusalert-1.0.0
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/version: "1.2.0"
    app.kubernetes.io/managed-by: Helm
spec:
  rules:
    - host: "palert.con.sdi"
      http:
        paths:
          - path: /
            backend:
              serviceName: RELEASE-NAME-prometheusalert
              servicePort: 8080

For internal domain names, simply resolve them directly to the internal SLB address (e.g., 172.17.193.181).

Reference: Alibaba Cloud Documentation

Alibaba Cloud Shared GPU Solution Testing

wzdushu@gmail.com (wanzi) — Tue, 31 Aug 2021 14:22:42 +0800

Before deployment, ensure that nvidia-driver and nvidia-docker are installed on your Kubernetes nodes, and Docker’s default runtime has been set to nvidia.

# cat /etc/docker/daemon.json
{
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
      "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia"
}

1. Install gpushare-device-plugin via Helm

1
2
3

$ git clone https://github.com/AliyunContainerService/gpushare-scheduler-extender.git
$ cd gpushare-scheduler-extender/deployer/chart
$ helm install --name gpushare --namespace kube-system --set masterCount=3 gpushare-installer

2. Label GPU Nodes

1
2

$ kubectl label node sd-cluster-04 gpushare=true
$ kubectl label node sd-cluster-05 gpushare=true

3. Install kubectl-inspect-gpushare

Ensure kubectl is already installed (omitted here).

1
2
3

$ cd /usr/bin/
$ wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare
$ chmod u+x /usr/bin/kubectl-inspect-gpushare

Check current GPU resource usage in the cluster:

$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  0/14                   0/14
sd-cluster-05  192.168.1.215  8/14                   8/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
8/28 (28%)

To disable GPU sharing on a node, simply set gpushare=false:

1
2

$ kubectl label node sd-cluster-04 gpushare=false
$ kubectl label node sd-cluster-05 gpushare=false

II. Validation Test

1. Deploy First Application

Request 2GiB GPU memory; the application should be scheduled onto one GPU card.

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-1
  labels:
    app: binpack-1

spec:
  replicas: 1

  selector:
    matchLabels:
      app: binpack-1

  template:
    metadata:
      labels:
        app: binpack-1

    spec:
      containers:
      - name: binpack-1
        image: cheyang/gpu-player:v2
        resources:
          limits:
            # GiB
            aliyun.com/gpu-mem: 2

$ kubectl apply -f 1.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  0/14                   0/14
sd-cluster-05  192.168.1.215  2/14                   2/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
2/28 (7%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
$ kubectl logs -f binpack-1-6d6955c487-j4c4b -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=2
2021-08-13 02:47:22.395557: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 02:47:22.552831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 14.65GiB
2021-08-13 02:47:22.552873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

2. Deploy Second Application

Request 8GiB GPU memory per pod, with 2 replicas → total 16GiB.

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-2
  labels:
    app: binpack-2

spec:
  replicas: 2

  selector:
    matchLabels:
      app: binpack-2

  template:
    metadata:
      labels:
        app: binpack-2

    spec:
      containers:
      - name: binpack-2
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 8

$ kubectl apply -f 2.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  8/14                   8/14
sd-cluster-05  192.168.1.215  10/14                  10/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
18/28 (64%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          27m
$ kubectl logs -f binpack-2-58579b95f7-4wpbl -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=8
2021-08-13 02:48:41.246585: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 02:48:41.338992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 13.07GiB
2021-08-13 02:48:41.339031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

As shown by resource usage, the second application is now using 8GiB from two separate GPUs.

3. Deploy Third Application

Request 2GiB GPU memory.

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-3
  labels:
    app: binpack-3

spec:
  replicas: 1

  selector:
    matchLabels:
      app: binpack-3

  template:
    metadata:
      labels:
        app: binpack-3

    spec:
      containers:
      - name: binpack-3
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 2

$ kubectl apply -f 3.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  8/14                   8/14
sd-cluster-05  192.168.1.215  12/14                  12/14
----------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
20/28 (71%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          28m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          27m
binpack-2-58579b95f7-sjhwt         1/1     Running   0          27m
binpack-3-556bbd84f9-9xqg7         1/1     Running   0          14m
$ kubectl logs -f binpack-3-556bbd84f9-9xqg7 -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=2
2021-08-13 03:01:53.897423: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 03:01:54.008665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 7.08GiB
2021-08-13 03:01:54.008716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

After deploying the third application, the maximum available GPU memory is 8GiB. However, actual usage is limited to 6GiB per task because a single task cannot span across multiple GPU cards.

4. Deploy Fourth Application

Request 5GiB GPU memory — should be scheduled on sd-cluster-04.

apiVersion: apps/v1
kind: Deployment

metadata:
  name: binpack-4
  labels:
    app: binpack-4

spec:
  replicas: 1

  selector:
    matchLabels:
      app: binpack-4

  template:
    metadata:
      labels:
        app: binpack-4

    spec:
      containers:
      - name: binpack-4
        image: cheyang/gpu-player:v2
        resources:
          limits:
            aliyun.com/gpu-mem: 5

$ kubectl apply -f 4.yaml -n test
$ kubectl inspect gpushare
NAME           IPADDRESS      GPU0(Allocated/Total)  GPU Memory(GiB)
sd-cluster-04  192.168.1.214  13/14                  13/14
sd-cluster-05  192.168.1.215  12/14                  12/14
------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
25/28 (89%)
$ kubectl get pod -n test
NAME                               READY   STATUS    RESTARTS   AGE
binpack-1-6d6955c487-j4c4b         1/1     Running   0          26m
binpack-2-58579b95f7-4wpbl         1/1     Running   0          24m
binpack-2-58579b95f7-sjhwt         1/1     Running   0          24m
binpack-3-556bbd84f9-9xqg7         1/1     Running   0          11m
binpack-4-6956458f85-cv62j         1/1     Running   0          6s
$ kubectl logs -f binpack-4-6956458f85-cv62j -n test
ALIYUN_COM_GPU_MEM_DEV=14
ALIYUN_COM_GPU_MEM_CONTAINER=5
2021-08-13 03:13:20.208122: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-08-13 03:13:20.361391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:af:00.0
totalMemory: 14.75GiB freeMemory: 6.46GiB
2021-08-13 03:13:20.361481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)

III. Summary

The gpushare-device-plugin has the following limitations:

A single task cannot utilize GPUs across multiple machines (shared).
GPU resource allocation cannot be based on utilization percentage within a single GPU.

However, it is sufficient for algorithm team model testing scenarios. There are two alternative GPU sharing solutions, which are not covered here—refer directly to their official repositories if needed:

https://github.com/tkestack/gpu-manager
https://github.com/vmware/bitfusion-with-kubernetes-integration

References:
https://github.com/AliyunContainerService/gpushare-scheduler-extender/tree/master/deployer

Implementing Internal DNS with Alibaba Cloud PrivateZone + Bind9 + Dnsmasq

wzdushu@gmail.com (wanzi) — Sat, 10 Jul 2021 17:22:42 +0800

Requirements:

Alibaba Cloud cluster can resolve internal domain names

Office network resolves internal domain names + internet access resolution

Solution:

For the first requirement, directly use Alibaba Cloud PrivateZone for resolution.
For the second requirement, configure internal domain zones in PrivateZone, then synchronize them to the office network’s bind9 server using Alibaba Cloud’s synchronization tool. Use Dnsmasq as the DNS entry point for the office network: forward public queries to public DNS servers, and forward internal domain queries to the bind9 server.

Some may wonder: Why not use bind9 alone to handle all internal resolutions? The main reason is that in practice, bind9 exhibits performance issues when forwarding to multiple DNS servers simultaneously—occasional timeouts occur. In contrast, Dnsmasq handles this scenario significantly better.

1. Configure Alibaba Cloud PrivateZone

Reference: https://help.aliyun.com/document_detail/64627.html

2. Synchronize Alibaba Cloud Zones to bind9

1. Deploy bind9 via docker-compose

version: '2'

services:
  bind:
    restart: always
    image: sameersbn/bind:9.16.1-20200524
    environment:
    - ROOT_PASSWORD=DNS2021#
    - WEBMIN_ENABLED=true
    - WEBMIN_INIT_SSL_ENABLED=false
    ports:
    - "15353:53/tcp"
    - "15353:53/udp"
    - "11953:953/tcp"
    - "10000:10000/tcp" # webmin management
    volumes:
    - ./data:/data
    networks:
      - bind9

networks:
  bind9:
    ipam:
      config:
      - subnet: 10.220.0.0/16
        gateway: 10.220.0.1

2. Modify bind configuration files

data/bind/etc/named.conf

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";
key "rndc-key" {
    algorithm hmac-sha256;
    secret "tREasaE2Jal1GfwfL5iii3a88eRGKWui41l5h3v89OM=";
};
controls {
	inet 127.0.0.1 port 953 allow { 127.0.0.1; } keys { rndc-key; };
	};
logging {
    channel query_log {
        file "query.log" versions 10 size 50M;
        severity info;
        print-category yes;
        print-severity yes;
        print-time yes;
    };
    category queries {
        query_log;
    };
};

data/bind/etc/named.conf.options

options {
	directory "/var/cache/bind";
    dnssec-validation no;
    dnssec-enable no;
    recursion yes;
    allow-recursion { any;};
    allow-transfer { any; };
    allow-query-cache { any; };
    listen-on-v6 { any; };
    listen-on port 53 { any; };
    forward first;
    forwarders {
        192.168.1.211 port 53;
        192.168.1.212 port 53;
        };
    transfer-format many-answers;
    transfers-per-ns 500;
    recursive-clients 100000;
    max-transfer-time-in 5;
    transfers-in 300;
    transfers-out 300;
    querylog yes;
};

data/bind/etc/named.conf.local

zone "sd.com" {
	type master;
	file "/etc/bind/zones/sd.com.zone";
    allow-update { 127.0.0.1; };
	};
zone "bgt.sdi" {
	type master;
	file "/etc/bind/zones/bgt.sdi.zone";
    allow-update { 127.0.0.1; };
	};
zone "con.sdi" {
	type master;
	file "/etc/bind/zones/con.sdi.zone";
    allow-update { 127.0.0.1; };
	};

3. Synchronize zone configurations from Alibaba Cloud to bind9

Reference: https://help.aliyun.com/document_detail/102718.html

Create a shell script and schedule it via cron for batch execution.

`1`	`/5 * * * /bin/bash /opt/bind9/update.sh`

update.sh

#!/bin/bash
cd /opt/bind9/tools
./Zone_file_sync config.json
chown 101.101 /opt/bind9/data/bind/etc/zones/*
docker exec bind9_bind_1 bash -c "rndc -c /etc/bind/rndc.conf freeze;rndc -c /etc/bind/rndc.conf reload;rndc -c /etc/bind/rndc.conf thaw"

tools/config.json

{
  "accessKeyId": "LIDD5ssssmGExzGsdfsY6sJrSqo",
  "accessKeySecret": "C5N1TTESt74KhSTsswSSSWiz2",
  "zone": [
    {
      "zoneName": "sd.com",
      "zoneId": "2a4dc4e0sdsfa5d36a3b88ab6482saf",
      "filePath": "/opt/bind9/data/bind/etc/zones/sd.com.zone"
    },
    {
      "zoneName": "bgt.sdi",
      "zoneId": "f842ca07ccsd6f35d9e294d55a0c900",
      "filePath": "/opt/bind9/data/bind/etc/zones/bgt.sdi.zone"
    },
    {
      "zoneName": "con.sdi",
      "zoneId": "beb4d911addsf2bd86425ds280e7bbf2",
      "filePath": "/opt/bind9/data/bind/etc/zones/con.sdi.zone"
    }
  ]
}

3. Deploy dnsmasq

1. Install and configure dnsmasq

# yum -y install dnsmasq
# cat >> /etc/dnsmasq.conf << Tag
port=53
proxy-dnssec
no-hosts  # do not load local /etc/hosts
no-negcache
dns-forward-max=2000
server=114.114.114.114 # upstream DNS server
server=223.5.5.5 # upstream DNS server
server=/sd.com/127.0.0.1#15353  # forward to specific DNS on port
server=/bgt.sdi/127.0.0.1#15353
server=/con.sdi/127.0.0.1#15353
log-queries  # log DNS queries
log-facility=/var/log/dnsmasq/dnsmasq.log # specify log path
log-async=50
cache-size=100000
Tag
# systemctl start dnsmasq
# systemctl enable dnsmasq

2. Configure log rotation policy for dnsmasq:

# cat >> /etc/logrotate.d/dnsmasq Tag
/var/log/dnsmasq/dnsmasq.log {
    notifempty
    daily
    dateext
    rotate 15
    sharedscripts
    postrotate
        [ ! -f /var/run/dnsmasq.pid ] || kill -USR2 `cat /var/run/dnsmasq.pid`
    endscript
}
Tag

Test execution:

`1`	`logrotate -vf /etc/logrotate.conf`

4. Adding a New Zone: What Operations Are Needed?

Both bind9 and dnsmasq require updates.

Example: Add test.com

Add to /etc/dnsmasq.conf:
```
server=/test.com/127.0.0.1#15353
```

Add to /opt/bind9/data/bind/etc/named.conf.local:

zone "test.com" {
    type master;
    file "/etc/bind/zones/test.com.zone";
    allow-update { 127.0.0.1; };
    forwarders {};
};

5. How to Integrate Internal DNS into K8s Cluster

Update /etc/resolv.conf on cluster nodes to use internal DNS:

1
2
3

options timeout:1 attempts:1 rotate
nameserver 192.168.1.211
nameserver 192.168.1.212

Configure CoreDNS to forward unresolved queries:

Corefile: |
  .:53 {
      errors
      health
      kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods insecure
         upstream
         fallthrough in-addr.arpa ip6.arpa
      }
      prometheus :9153
      forward . /etc/resolv.conf
      cache 30
      loop
      reload
      loadbalance
  }

With these steps, the complete internal DNS resolution system is now implemented.

Automated ECS Creation with Terraform

wzdushu@gmail.com (wanzi) — Fri, 26 Feb 2021 17:22:42 +0800

Quickly Create an Alibaba Cloud ECS Instance

Specify Terraform Version

Here, we specify the Alibaba Cloud provider version and set the required Terraform version.

# mkdir aliyun-ecs-one && cd aliyun-ecs-one
# touch versions.tf
# vim versions.tf
terraform {
  required_providers {
    alicloud = {
      source  = "aliyun/alicloud"
      version = "1.115.1"
    }
  }

  required_version = ">= 0.12"
}

Configure Variables

Here we define key pairs, cloud region, ECS account, and image information.

# vim variables.tf
# Alibaba Cloud sub-account access_key
variable "alicloud_access_key" {
  default     = "LTAI4GBXXXXXXXXXXXXXXXXXXXXXX"
  description = "The Alicloud Access Key ID to launch resources. Support environment variable 'ALICLOUD_ACCESS_KEY'."
}

# Alibaba Cloud sub-account secret_key
variable "alicloud_secret_key" {
  default     = "4Z4gbl3dXXXXXXXXXXXXXXXXXXXXX"
  description = "The Alicloud Access Secret Key to launch resources. Support environment variable 'ALICLOUD_SECRET_KEY'."
}

# Alibaba Cloud region (Hangzhou in this case)
variable "region" {
  default     = "cn-hangzhou"
  description = "The Alicloud region for resources. Support environment variable 'REGION'."
}

# Available zone in Hangzhou region
variable "availability_zone" {
  description = "The available zone to launch ECS instance and other resources."
  default     = "cn-hangzhou-i"
}

# Image ID
variable "image_id" {
  default = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
}

# ECS instance type
variable "ecs_type" {
  default = "ecs.s6-c1m2.small"
}

# ECS instance password
variable "ecs_password" {
  default = "Test12345"
}

# Disk category (cloud efficiency disk here)
variable "disk_category" {
  default = "cloud_efficiency"
}

# Disk size
variable "disk_size" {
  default = "40"
}

# Internet charge type (default: PayByTraffic)
variable "internet_charge_type" {
  default = "PayByTraffic"
}

# Maximum outbound bandwidth on public network (default > 0 will automatically assign a dedicated public IP from v1.7 onwards)
variable "internet_max_bandwidth_out" {
  default = 5
}

Configure Instance Resources

Since this is for testing, we need to create VPC, VSwitch, Security Group, and Security Group Rules beforehand.

# vim main.tf
provider "alicloud" {
  region     = var.region
  access_key = var.alicloud_access_key
  secret_key = var.alicloud_secret_key
}

resource "alicloud_vpc" "vpc" {
  name       = "tf_test_foo"
  cidr_block = "10.100.0.0/16"
}

resource "alicloud_vswitch" "vsw" {
  vpc_id            = alicloud_vpc.vpc.id
  cidr_block        = "10.100.0.0/24"
  availability_zone = var.availability_zone
}

resource "alicloud_security_group" "default" {
  name   = "default"
  vpc_id = alicloud_vpc.vpc.id
}

resource "alicloud_security_group_rule" "allow_all_tcp" {
  type              = "ingress"
  ip_protocol       = "tcp"
  nic_type          = "intranet"
  policy            = "accept"
  port_range        = "1/65535"
  priority          = 1
  security_group_id = alicloud_security_group.default.id
  cidr_ip           = "0.0.0.0/0"
}

resource "alicloud_instance" "wanzi_test" {
  availability_zone = var.availability_zone
  security_groups   = alicloud_security_group.default.*.id

  instance_type        = var.ecs_type
  system_disk_category = var.disk_category
  image_id             = var.image_id
  instance_name        = "wanzi_tf001"
  vswitch_id           = alicloud_vswitch.vsw.id
  password             = var.ecs_password
}

Run plan to simulate the execution:

# terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

Create the cloud instance — this process calls Alibaba Cloud APIs and generates a local Terraform state file:

# terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

  # alicloud_security_group.default will be created
  + resource "alicloud_security_group" "default" {
      + id                  = (known after apply)
      + inner_access        = (known after apply)
      + inner_access_policy = (known after apply)
      + name                = "default"
      + security_group_type = "normal"
      + vpc_id              = (known after apply)
    }
......
......
Plan: 5 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

alicloud_vpc.vpc: Creating...
alicloud_vpc.vpc: Creation complete after 9s [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_security_group.default: Creating...
alicloud_vswitch.vsw: Creating...
alicloud_security_group.default: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_security_group_rule.allow_all_tcp: Creating...
alicloud_security_group_rule.allow_all_tcp: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_vswitch.vsw: Creation complete after 4s [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_instance.wanzi_test: Creating...
alicloud_instance.wanzi_test: Still creating... [10s elapsed]
alicloud_instance.wanzi_test: Still creating... [20s elapsed]
alicloud_instance.wanzi_test: Creation complete after 22s [id=i-bp1gt9mb9asadff9r2zr]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

After these steps, the resources have been successfully created. A .tfstate file is also generated in the current directory — this file is critical and must not be deleted. You can later use terraform show to view the created resource details.

Batch Create Multiple ECS Instances

Configure Module

Since many excellent modules are available on https://registry.terraform.io, we directly use the alibaba/ecs-instance/alicloud module.

For more information about the official ECS module, see: https://github.com/terraform-alicloud-modules/terraform-alicloud-ecs-instance

The variables.tf and versions.tf files remain as configured in Step 1. In main.tf, add the module configuration for batch creation of ECS instances:

module "tf-instances" {
  source                      = "alibaba/ecs-instance/alicloud"
  region                      = "cn-hangzhou"
  number_of_instances         = "3"
  vswitch_id                  = alicloud_vswitch.vsw.id
  group_ids                   = [alicloud_security_group.default.id]
  private_ips                 = ["10.100.0.10", "10.100.0.11", "10.100.0.12"]
  image_ids                   = ["ubuntu_18_04_64_20G_alibase_20190624.vhd"]
  instance_type               = var.ecs_type
  internet_max_bandwidth_out  = 10
  associate_public_ip_address = true
  instance_name               = "my_module_instances_"
  host_name                   = "wanzi-cluster"
  internet_charge_type        = "PayByTraffic"
  password                    = var.ecs_password
  system_disk_category        = "cloud_ssd"
  data_disks = [
    {
      disk_category = "cloud_ssd"
      disk_name     = "my_module_disk"
      disk_size     = "50"
    }
  ]
}

Note: By default, setting internet_max_bandwidth_out triggers automatic assignment of a dedicated public IP. If you don’t need this, you can omit the setting.

Batch Create Resources

➜ terraform apply
alicloud_vpc.vpc: Refreshing state... [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_vswitch.vsw: Refreshing state... [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_security_group.default: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_security_group_rule.allow_all_tcp: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_instance.wanzi_test: Refreshing state... [id=i-bp1gt9mb9asadff9r2zr]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.tf-instances.alicloud_instance.this[0] will be created
  + resource "alicloud_instance" "this" {
      + availability_zone             = (known after apply)
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + description                   = "An ECS instance came from terraform-alicloud-modules/ecs-instance"
      + dry_run                       = false
      + host_name                     = "wanzi-cluster001"
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "my_module_instances_001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_charge_type          = "PayByTraffic"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 10
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = "10.100.0.10"
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_enhancement_strategy = "Active"
      + security_groups               = [
          + "sg-bp11s5pka9pxtj6pn4xq",
        ]
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_ssd"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + tags                          = {
          + "Name" = "my_module_instances_001"
        }
      + volume_tags                   = {
          + "Name" = "my_module_instances_001"
        }
      + vswitch_id                    = "vsw-bp1wgpgz9z8y2lfsl2beo"

      + data_disks {
          + category             = "cloud_efficiency"
          + delete_with_instance = true
          + encrypted            = false
          + name                 = "TF_ECS_Disk"
          + performance_level    = (known after apply)
          + size                 = 40
        }
    }

  # module.tf-instances.alicloud_instance.this[1] will be created
  + resource "alicloud_instance" "this" {
      + availability_zone             = (known after apply)
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + description                   = "An ECS instance came from terraform-alicloud-modules/ecs-instance"
      + dry_run                       = false
      + host_name                     = "wanzi-cluster002"
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "my_module_instances_002"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_charge_type          = "PayByTraffic"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 10
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = "10.100.0.11"
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_enhancement_strategy = "Active"
      + security_groups               = [
          + "sg-bp11s5pka9pxtj6pn4xq",
        ]
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_ssd"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + tags                          = {
          + "Name" = "my_module_instances_002"
        }
      + volume_tags                   = {
          + "Name" = "my_module_instances_002"
        }
      + vswitch_id                    = "vsw-bp1wgpgz9z8y2lfsl2beo"

      + data_disks {
          + category             = "cloud_efficiency"
          + delete_with_instance = true
          + encrypted            = false
          + name                 = "TF_ECS_Disk"
          + performance_level    = (known after apply)
          + size                 = 40
        }
    }

  # module.tf-instances.alicloud_instance.this[2] will be created
  + resource "alicloud_instance" "this" {
      + availability_zone             = (known after apply)
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + description                   = "An ECS instance came from terraform-alicloud-modules/ecs-instance"
      + dry_run                       = false
      + host_name                     = "wanzi-cluster003"
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "my_module_instances_003"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_charge_type          = "PayByTraffic"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 10
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = "10.100.0.12"
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_enhancement_strategy = "Active"
      + security_groups               = [
          + "sg-bp11s5pka9pxtj6pn4xq",
        ]
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_ssd"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + tags                          = {
          + "Name" = "my_module_instances_003"
        }
      + volume_tags                   = {
          + "Name" = "my_module_instances_003"
        }
      + vswitch_id                    = "vsw-bp1wgpgz9z8y2lfsl2beo"

      + data_disks {
          + category             = "cloud_efficiency"
          + delete_with_instance = true
          + encrypted            = false
          + name                 = "TF_ECS_Disk"
          + performance_level    = (known after apply)
          + size                 = 40
        }
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.tf-instances.alicloud_instance.this[2]: Creating...
module.tf-instances.alicloud_instance.this[1]: Creating...
module.tf-instances.alicloud_instance.this[0]: Creating...
module.tf-instances.alicloud_instance.this[1]: Still creating... [10s elapsed]
module.tf-instances.alicloud_instance.this[2]: Still creating... [10s elapsed]
module.tf-instances.alicloud_instance.this[0]: Still creating... [10s elapsed]
module.tf-instances.alicloud_instance.this[1]: Still creating... [20s elapsed]
module.tf-instances.alicloud_instance.this[0]: Still creating... [20s elapsed]
module.tf-instances.alicloud_instance.this[2]: Still creating... [20s elapsed]
module.tf-instances.alicloud_instance.this[0]: Creation complete after 21s [id=i-bp1hwbo4htk8sbwxtk6o]
module.tf-instances.alicloud_instance.this[1]: Creation complete after 21s [id=i-bp17lh41gywyih0xg6we]
module.tf-instances.alicloud_instance.this[2]: Creation complete after 22s [id=i-bp11zlrl6vxeaerz4ad0]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

The batch creation of multiple ECS instances is now complete. For any future adjustments to deployed ECS resources, simply run write/plan/apply — this process may restart the Alibaba Cloud instances.

Terraform Installation and Command Reference

wzdushu@gmail.com (wanzi) — Thu, 25 Feb 2021 17:22:42 +0800

Installing Terraform

Installing on Mac

1
2

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Installing on Linux

Ubuntu installation

1
2
3

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install terraform

CentOS installation

1
2
3

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install terraform

Verifying Installation

# terraform -v
Terraform v0.14.3

Your version of Terraform is out of date! The latest version
is 0.14.7. You can update by downloading from https://www.terraform.io/downloads.html
# terraform
Usage: terraform [global options] <subcommand> [args]

The available commands for execution are listed below.
The primary workflow commands are given first, followed by
less common or more advanced commands.

Main commands:
  init          Prepare your working directory for other commands
  validate      Check whether the configuration is valid
  plan          Show changes required by the current configuration
  apply         Create or update infrastructure
  destroy       Destroy previously-created infrastructure

All other commands:
  console       Try Terraform expressions at an interactive command prompt
  fmt           Reformat your configuration in the standard style
  force-unlock  Release a stuck lock on the current workspace
  get           Install or upgrade remote Terraform modules
  graph         Generate a Graphviz graph of the steps in an operation
  import        Associate existing infrastructure with a Terraform resource
  login         Obtain and save credentials for a remote host
  logout        Remove locally-stored credentials for a remote host
  output        Show output values from your root module
  providers     Show the providers required for this configuration
  refresh       Update the state to match remote systems
  show          Show the current state or a saved plan
  state         Advanced state management
  taint         Mark a resource instance as not fully functional
  untaint       Remove the 'tainted' state from a resource instance
  version       Show the current Terraform version
  workspace     Workspace management

Global options (use these before the subcommand, if any):
  -chdir=DIR    Switch to a different working directory before executing the
                given subcommand.
  -help         Show this help output, or the help for a specified subcommand.
  -version      An alias for the "version" subcommand.

Terraform Commands for Resource Management

Initializing Resources

For a Terraform project, I created three basic files: main.tf (entry file), variables.tf (variable definitions), and versions.tf (version information).

# ls 
main.tf     variables.tf      versions.tf
# terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of aliyun/alicloud from the dependency lock file
- Using aliyun/alicloud v1.115.1 from the shared cache directory

Terraform has been successfully initialized!

Formatting Terraform Files

fmt automatically reformats .tf files in the current directory to conform to standard Terraform formatting.

# terraform fmt 
main.tf
variables.tf
versions.tf
# terraform fmt -diff  # Show differences after formatting
main.tf
--- old/main.tf
+++ new/main.tf
@@ -1,7 +1,7 @@
 provider "alicloud" {
   region     = var.region
   access_key = var.alicloud_access_key
-  secret_key =  var.alicloud_secret_key
+  secret_key = var.alicloud_secret_key
 }

 resource "alicloud_vpc" "vpc" {
@@ -12,7 +12,7 @@
 resource "alicloud_vswitch" "vsw" {
   vpc_id            = alicloud_vpc.vpc.id
   cidr_block        = "10.100.0.0/24"
-  availability_zone =  var.availability_zone
+  availability_zone = var.availability_zone
 }

 resource "alicloud_security_group" "default" {
variables.tf
--- old/variables.tf
+++ new/variables.tf
@@ -4,7 +4,7 @@
 }

 variable "alicloud_secret_key" {
-  default                     = "4Z4gbl3d9TGz9jWobv9MPwInvyH2Kf"
+  default     = "4Z4gbl3d9TGz9jWobv9MPwInvyH2Kf"
   description = "The Alicloud Access Secret Key to launch resources.  Support to environment 'ALICLOUD_SECRET_KEY'."
 }

Creating a Resource Plan

terraform plan checks whether the planned changes align with your expectations without modifying actual resources or state.

# terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

Creating Cloud Resources

terraform apply generates a creation plan and executes it, creating resources and generating a terraform.tfstate file in the current directory.

# terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

  # alicloud_security_group.default will be created
  + resource "alicloud_security_group" "default" {
      + id                  = (known after apply)
      + inner_access        = (known after apply)
      + inner_access_policy = (known after apply)
      + name                = "default"
      + security_group_type = "normal"
      + vpc_id              = (known after apply)
    }
......
......
Plan: 5 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

alicloud_vpc.vpc: Creating...
alicloud_vpc.vpc: Creation complete after 9s [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_security_group.default: Creating...
alicloud_vswitch.vsw: Creating...
alicloud_security_group.default: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_security_group_rule.allow_all_tcp: Creating...
alicloud_security_group_rule.allow_all_tcp: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_vswitch.vsw: Creation complete after 4s [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_instance.wanzi_test: Creating...
alicloud_instance.wanzi_test: Still creating... [10s elapsed]
alicloud_instance.wanzi_test: Still creating... [20s elapsed]
alicloud_instance.wanzi_test: Creation complete after 22s [id=i-bp1gt9mb9asadff9r2zr]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Viewing Created Resource Information

terraform show displays the resources created in the current project.

terraform show -json displays data in JSON format.

# terraform show
# alicloud_instance.wanzi_test:
resource "alicloud_instance" "wanzi_test" {
    availability_zone          = "cn-hangzhou-i"
    deletion_protection        = false
    dry_run                    = false
    host_name                  = "iZbp1gt9mb9asadff9r2zrZ"
    id                         = "i-bp1gt9mb9asadff9r2zr"
    image_id                   = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
    instance_charge_type       = "PostPaid"
    instance_name              = "wanzi_tf001"
    instance_type              = "ecs.s6-c1m2.small"
    internet_charge_type       = "PayByTraffic"
    internet_max_bandwidth_in  = -1
    internet_max_bandwidth_out = 0
    password                   = (sensitive value)
    private_ip                 = "10.100.0.234"
    security_groups            = [
        "sg-bp11s5pka9pxtj6pn4xq",
    ]
    spot_price_limit           = 0
    spot_strategy              = "NoSpot"
    status                     = "Running"
    subnet_id                  = "vsw-bp1wgpgz9z8y2lfsl2beo"
    system_disk_category       = "cloud_efficiency"
    system_disk_size           = 40
    volume_tags                = {}
    vswitch_id                 = "vsw-bp1wgpgz9z8y2lfsl2beo"
}

# alicloud_security_group.default:
resource "alicloud_security_group" "default" {
    id                  = "sg-bp11s5pka9pxtj6pn4xq"
    inner_access        = true
    inner_access_policy = "Accept"
    name                = "default"
    security_group_type = "normal"
    vpc_id              = "vpc-bp1kulcyygsi727aay4hd"
}

# alicloud_security_group_rule.allow_all_tcp:
resource "alicloud_security_group_rule" "allow_all_tcp" {
    cidr_ip           = "0.0.0.0/0"
    id                = "sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1"
    ip_protocol       = "tcp"
    nic_type          = "intranet"
    policy            = "accept"
    port_range        = "1/65535"
    priority          = 1
    security_group_id = "sg-bp11s5pka9pxtj6pn4xq"
    type              = "ingress"
}

# alicloud_vpc.vpc:
resource "alicloud_vpc" "vpc" {
    cidr_block        = "10.100.0.0/16"
    id                = "vpc-bp1kulcyygsi727aay4hd"
    name              = "tf_test_foo"
    resource_group_id = "rg-acfm2ogp24u3rcy"
    route_table_id    = "vtb-bp1wy8srerq12rta02r03"
    router_id         = "vrt-bp1apvobefvhshksnnwvm"
    router_table_id   = "vtb-bp1wy8srerq12rta02r03"
}

# alicloud_vswitch.vsw:
resource "alicloud_vswitch" "vsw" {
    availability_zone = "cn-hangzhou-i"
    cidr_block        = "10.100.0.0/24"
    id                = "vsw-bp1wgpgz9z8y2lfsl2beo"
    vpc_id            = "vpc-bp1kulcyygsi727aay4hd"
}

Marking a Resource as Tainted

terraform taint marks a resource as “tainted.” When apply is run again, the tainted resource will be destroyed first and then recreated—effectively performing a delete-and-recreate operation on that specific resource.

1
2

# terraform taint alicloud_instance.wanzi_test
Resource instance alicloud_instance.wanzi_test has been marked as tainted.

Conversely, terraform untaint removes the tainted status, restoring the resource to normal state.

1
2

# terraform untaint alicloud_instance.wanzi_test
Resource instance alicloud_instance.wanzi_test has been successfully untainted.

Destroying Cloud Resources

terraform destroy destroys cloud resources based on the current configuration.

# terraform destroy

Plan: 0 to add, 0 to change, 5 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

alicloud_security_group_rule.allow_all_tcp: Destroying... [id=sg-bp10tup89oothxz8tny1:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_instance.wanzi_test: Destroying... [id=i-bp10ukz4nlr894mhebgl]
alicloud_security_group_rule.allow_all_tcp: Destruction complete after 0s
alicloud_instance.wanzi_test: Still destroying... [id=i-bp10ukz4nlr894mhebgl, 10s elapsed]
alicloud_instance.wanzi_test: Still destroying... [id=i-bp10ukz4nlr894mhebgl, 20s elapsed]
alicloud_instance.wanzi_test: Destruction complete after 28s
alicloud_security_group.default: Destroying... [id=sg-bp10tup89oothxz8tny1]
alicloud_vswitch.vsw: Destroying... [id=vsw-bp1ap7ccst3fjxnw4pnza]
alicloud_security_group.default: Destruction complete after 9s
alicloud_vswitch.vsw: Still destroying... [id=vsw-bp1ap7ccst3fjxnw4pnza, 10s elapsed]
alicloud_vswitch.vsw: Destruction complete after 20s
alicloud_vpc.vpc: Destroying... [id=vpc-bp1obwt5ded2i0zlbu052]
alicloud_vpc.vpc: Destruction complete after 3s

Destroy complete! Resources: 5 destroyed.

Importing Cloud Data into Local Project

terraform import imports cloud instances into local state using their instance ID. It generates a terraform.tfstate file in the local directory. Before importing existing data, back up both terraform.tfstate and the .terraform directory. After importing, use terraform show to view the resource in Terraform format, copy it, and further process it to obtain the .tf resource file content.

# cat yunduan.tf
resource "alicloud_instance" "test999" {
  # (resource arguments)
}
#
# terraform import alicloud_instance.test999 i-bp1etiv4002h9q27lb97
alicloud_instance.test999: Importing from ID "i-bp1etiv4002h9q27lb97"...
alicloud_instance.test999: Import prepared!
  Prepared alicloud_instance for import
alicloud_instance.test999: Refreshing state... [id=i-bp1etiv4002h9q27lb97]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
# cat terraform.tfstate
{
  "version": 4,
  "terraform_version": "0.14.3",
  "serial": 1,
  "lineage": "779fad5e-b076-8cfd-6041-f6eef8c88b8a",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "alicloud_instance",
      "name": "test999",
      "provider": "provider[\"registry.terraform.io/aliyun/alicloud\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "allocate_public_ip": null,
            "auto_release_time": "",
            "auto_renew_period": null,
            "availability_zone": "cn-hangzhou-i",
            "credit_specification": "",
            "data_disks": [],
            "deletion_protection": false,
            "description": "",
            "dry_run": null,
            "force_delete": null,
            "host_name": "iZbp1etiv4002h9q27lb97Z",
            "id": "i-bp1etiv4002h9q27lb97",
            "image_id": "ubuntu_18_04_64_20G_alibase_20190624.vhd",
            "include_data_disks": null,
            "instance_charge_type": "PostPaid",
            "instance_name": "wanzi_tf001",
            "instance_type": "ecs.s6-c1m2.small",
            "internet_charge_type": "PayByTraffic",
            "internet_max_bandwidth_in": -1,
            "internet_max_bandwidth_out": 0,
            "io_optimized": null,
            "is_outdated": null,
            "key_name": "",
            "kms_encrypted_password": null,
            "kms_encryption_context": null,
            "password": "",
            "period": null,
            "period_unit": null,
            "private_ip": "10.100.0.169",
            "public_ip": "",
            "renewal_status": null,
            "resource_group_id": "",
            "role_name": "",
            "security_enhancement_strategy": null,
            "security_groups": [
              "sg-bp14pij6g7sjmn9bz92a"
            ],
            "spot_price_limit": 0,
            "spot_strategy": "NoSpot",
            "status": "Running",
            "subnet_id": "vsw-bp1c966jdtiw1qwh2tng8",
            "system_disk_auto_snapshot_policy_id": "",
            "system_disk_category": "cloud_efficiency",
            "system_disk_description": null,
            "system_disk_name": null,
            "system_disk_performance_level": "",
            "system_disk_size": 40,
            "tags": {},
            "timeouts": {
              "create": null,
              "delete": null,
              "update": null
            },
            "user_data": "",
            "volume_tags": {},
            "vswitch_id": "vsw-bp1c966jdtiw1qwh2tng8"
          },
          "sensitive_attributes": [],
          "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6MTIwMDAwMDAwMDAwMCwidXBkYXRlIjo2MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjAifQ=="
        }
      ]
    }
  ]
}
# terraform show
# alicloud_instance.test999:
resource "alicloud_instance" "test999" {
    availability_zone          = "cn-hangzhou-i"
    deletion_protection        = false
    host_name                  = "iZbp1etiv4002h9q27lb97Z"
    id                         = "i-bp1etiv4002h9q27lb97"
    image_id                   = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
    instance_charge_type       = "PostPaid"
    instance_name              = "wanzi_tf001"
    instance_type              = "ecs.s6-c1m2.small"
    internet_charge_type       = "PayByTraffic"
    internet_max_bandwidth_in  = -1
    internet_max_bandwidth_out = 0
    private_ip                 = "10.100.0.169"
    security_groups            = [
        "sg-bp14pij6g7sjmn9bz92a",
    ]
    spot_price_limit           = 0
    spot_strategy              = "NoSpot"
    status                     = "Running"
    subnet_id                  = "vsw-bp1c966jdtiw1qwh2tng8"
    system_disk_category       = "cloud_efficiency"
    system_disk_size           = 40
    tags                       = {}
    volume_tags                = {}
    vswitch_id                 = "vsw-bp1c966jdtiw1qwh2tng8"

    timeouts {}
}

Generating Resource Relationship Diagrams

Each template defines resources with varying degrees of interdependencies. terraform graph generates a visual diagram of these relationships:

# terraform graph
digraph {
        compound = "true"
        newrank = "true"
        subgraph "root" {
                "[root] alicloud_instance.wanzi_test (expand)" [label = "alicloud_instance.wanzi_test", shape = "box"]
                "[root] alicloud_security_group.default (expand)" [label = "alicloud_security_group.default", shape = "box"]
                "[root] alicloud_security_group_rule.allow_all_tcp (expand)" [label = "alicloud_security_group_rule.allow_all_tcp", shape = "box"]
                "[root] alicloud_vpc.vpc (expand)" [label = "alicloud_vpc.vpc", shape = "box"]
                "[root] alicloud_vswitch.vsw (expand)" [label = "alicloud_vswitch.vsw", shape = "box"]
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" [label = "provider[\"registry.terraform.io/aliyun/alicloud\"]", shape = "diamond"]
                "[root] var.alicloud_access_key" [label = "var.alicloud_access_key", shape = "note"]
                "[root] var.alicloud_secret_key" [label = "var.alicloud_secret_key", shape = "note"]
                "[root] var.availability_zone" [label = "var.availability_zone", shape = "note"]
                "[root] var.disk_category" [label = "var.disk_category", shape = "note"]
                "[root] var.disk_size" [label = "var.disk_size", shape = "note"]
                "[root] var.ecs_password" [label = "var.ecs_password", shape = "note"]
                "[root] var.ecs_type" [label = "var.ecs_type", shape = "note"]
                "[root] var.image_id" [label = "var.image_id", shape = "note"]
                "[root] var.internet_charge_type" [label = "var.internet_charge_type", shape = "note"]
                "[root] var.internet_max_bandwidth_out" [label = "var.internet_max_bandwidth_out", shape = "note"]
                "[root] var.region" [label = "var.region", shape = "note"]
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] alicloud_security_group.default (expand)"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] alicloud_vswitch.vsw (expand)"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.disk_category"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.ecs_password"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.ecs_type"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.image_id"
                "[root] alicloud_security_group.default (expand)" -> "[root] alicloud_vpc.vpc (expand)"
                "[root] alicloud_security_group_rule.allow_all_tcp (expand)" -> "[root] alicloud_security_group.default (expand)"
                "[root] alicloud_vpc.vpc (expand)" -> "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]"
                "[root] alicloud_vswitch.vsw (expand)" -> "[root] alicloud_vpc.vpc (expand)"
                "[root] alicloud_vswitch.vsw (expand)" -> "[root] var.availability_zone"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] alicloud_instance.wanzi_test (expand)"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] alicloud_security_group_rule.allow_all_tcp (expand)"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] var.disk_size"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] var.internet_charge_type"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] var.internet_max_bandwidth_out"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"] (close)" -> "[root] alicloud_instance.wanzi_test (expand)"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"] (close)" -> "[root] alicloud_security_group_rule.allow_all_tcp (expand)"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" -> "[root] var.alicloud_access_key"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" -> "[root] var.alicloud_secret_key"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" -> "[root] var.region"
                "[root] root" -> "[root] meta.count-boundary (EachMode fixup)"
                "[root] root" -> "[root] provider[\"registry.terraform.io/aliyun/alicloud\"] (close)"
        }
}

This output can also be exported directly as an SVG image using terraform graph | dot -Tsvg > graph.svg (requires prior installation of Graphviz: brew install graphviz).

`1`	`terraform graph \| dot -Tsvg > ~/Downloads/graph.svg`

Viewing graph.svg reveals the full resource relationship diagram:

Terraform Commands for State Management

Listing All Resources in Current State

# terraform state list
alicloud_instance.wanzi_test
alicloud_security_group.default
alicloud_security_group_rule.allow_all_tcp
alicloud_vpc.vpc
alicloud_vswitch.vsw

Viewing Specific Resource Data

# terraform state show alicloud_vswitch.vsw
# alicloud_vswitch.vsw:
resource "alicloud_vswitch" "vsw" {
    availability_zone = "cn-hangzhou-i"
    cidr_block        = "10.100.0.0/24"
    id                = "vsw-bp1wgpgz9z8y2lfsl2beo"
    vpc_id            = "vpc-bp1kulcyygsi727aay4hd"
}

Removing a Specific Resource from State

terraform state rm <resource_type>.<resource_name> removes a resource from state but does not delete it from the cloud. The resource can later be re-imported from the cloud.

# terraform state rm alicloud_security_group.default
Removed alicloud_security_group.default
Successfully removed 1 resource instance(s).
# terraform state list
alicloud_instance.wanzi_test
alicloud_vpc.vpc
alicloud_vswitch.vsw
# terraform import alicloud_security_group.default sg-bp11s5pka9pxtj6pn4xq
alicloud_security_group.default: Importing from ID "sg-bp11s5pka9pxtj6pn4xq"...
alicloud_security_group.default: Import prepared!
  Prepared alicloud_security_group for import
alicloud_security_group.default: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Refreshing Resource State

terraform refresh updates the local state by fetching the latest data from the cloud via API calls.

# terraform refresh
alicloud_vpc.vpc: Refreshing state... [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_vswitch.vsw: Refreshing state... [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_security_group.default: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_instance.wanzi_test: Refreshing state... [id=i-bp1gt9mb9asadff9r2zr]

Introduction to the Automation Orchestration Tool Terraform

wzdushu@gmail.com (wanzi) — Wed, 24 Feb 2021 17:22:42 +0800

What is Terraform?

Terraform is an open-source infrastructure orchestration tool introduced by HashiCorp around 2014. It is now supported by nearly all major cloud service providers, including Alibaba Cloud, Tencent Cloud, Huawei Cloud, AWS, Azure, Baidu Cloud, and more. Many companies today build their infrastructure using Terraform.

Background: In traditional operations, launching a business required multiple preparatory steps such as hardware procurement, server rack mounting, network setup, and system installation. With the rise of cloud computing, major public cloud providers offer user-friendly graphical interfaces—users can purchase various cloud resources via a browser and quickly set up their architecture. However, as business architectures expand, the scale and variety of cloud resource procurement continue to grow. When users need to rapidly acquire large numbers of diverse cloud resources, the numerous interactive operations across cloud management consoles actually reduce procurement efficiency. For example, initializing a classic VPC network on the Alibaba Cloud console—from creating the VPC and VSwitches to setting up NAT gateways, elastic IPs, and routing configurations—can take 20 minutes or even longer. Moreover, the non-reproducible nature of manual work leads to redundant efforts when operating across regions or multi-cloud environments.

In reality, operations personnel care only about resource configuration—not the step-by-step creation process. Think of it like ordering coffee: you just tell the waiter what kind of coffee and whether to add ice. If there were a complete list of required cloud resources—clearly specifying types, specifications, quantities, and relationships between resources—you could deploy everything with one click. And when business needs change, simply updating the list would allow rapid reconfiguration of cloud resources. This concept in cloud computing is known as infrastructure orchestration. Today, many cloud platforms provide similar capabilities, such as Alibaba Cloud’s ROS, AWS CloudFormation, etc.

Infrastructure as Code (IaC):
Defining cloud resources, services, or operational steps as code within templates, then using an orchestration engine to automate resource management—this is Infrastructure as Code (IaC), the most efficient approach to resource orchestration. However, different cloud orchestration services bring high learning costs, low code reusability, and complex workflows for multi-cloud coordination. Each service typically manages only its own cloud platform, making unified management of multiple clouds and various layers (e.g., IaaS, PaaS) difficult. How can we solve this? Can we use a single orchestration tool with a unified syntax to manage multiple clouds—including Alibaba Cloud—uniformly? That’s exactly why Terraform was created—to address these challenges.

Features and Functions of Terraform

Key Features

IaC: Manage infrastructure using code.
Execution Plan: Shows actions that will be performed during terraform apply.
Resource Graph: Visualizes all resources and their relationships.
Change Automation: Based on execution plans and resource graphs, clearly identifies what changes are needed and in what order.

Summary: Terraform is used for initializing various infrastructure resources, supports multiple cloud platforms, and enables integration with third-party services.

Benefits

Uses APIs from different providers and abstracts them into standardized Terraform code.
Users don’t need to understand the intricate details of each cloud provider’s API, significantly reducing deployment complexity.

Terraform Architecture

Terraform follows a plugin-based architecture, offering strong extensibility and allowing developers to easily extend its functionality. Logically, Terraform consists of two layers: the Core Layer (Terraform Core) and the Plugin Layer (Terraform Provider).

Core Layer

The core layer is essentially the Terraform command-line tool, written in Go. It handles:

Reading .tf configuration files and performing variable substitution.
Managing resource state files.
Analyzing resource dependencies and generating dependency graphs.
Creating resources based on dependency relationships. Resources without dependencies are created in parallel (default: 10 concurrent processes)—this is why Terraform efficiently manages cloud resources.
Calling the plugin layer via RPC.

Plugin Layer

Also written in Go, Terraform has over 250 different plugins. These plugins:

Receive RPC calls from the core layer.
Execute specific services.

There are two main types of plugins:

Provider

Providers integrate with external APIs—for example, the Alibaba Cloud Provider enables creation, modification, and deletion of Alibaba Cloud resources. This plugin interacts with Alibaba Cloud’s API endpoints and provides an abstraction layer so developers can manage resources via Terraform without needing to know API details. Responsibilities include:

Initializing communication with external APIs.
Handling authentication with external services.
Defining relationships between cloud resources and external services.

Common providers:

Alibaba Cloud: https://github.com/aliyun/terraform-provider-alicloud
Baidu Cloud: https://github.com/baidubce/terraform-provider-baiducloud
Tencent Cloud: https://github.com/tencentcloudstack/terraform-provider-tencentcloud
Huawei Cloud: https://github.com/huaweicloud/terraform-provider-huaweicloud
UCloud: https://github.com/ucloud/terraform-provider-ucloud
QingCloud: https://github.com/yunify/terraform-provider-qingcloud
AWS: https://github.com/hashicorp/terraform-provider-aws
Azure: https://github.com/terraform-providers/terraform-provider-azurerm
Google Cloud: https://github.com/hashicorp/terraform-provider-google

Provisioner

Provisioners execute scripts after resource creation or deletion. For instance, the Puppet Provisioner can download, install, and configure a Puppet agent on a newly created virtual machine.

For better understanding, here’s a simplified component architecture diagram from online sources:

Below is a basic workflow diagram for daily Terraform operations:

Terraform Keywords Explained

Declarative Language (HCL)

Terraform uses HashiCorp Configuration Language (HCL), which is declarative. This means developers describe what the infrastructure should look like, and Terraform takes care of how to achieve it—without requiring knowledge of implementation steps or how Terraform interacts with cloud provider APIs. Terraform automatically downloads the necessary Providers and Provisioners to handle the details.

In contrast, imperative languages execute commands step-by-step, where order matters. Fixed inputs produce fixed outputs. While neither paradigm is inherently superior, declarative approaches are more convenient for cloud resource orchestration. All mainstream cloud orchestration tools—like AWS CloudFormation, Azure Resource Manager Templates, Google Cloud Deployment Manager—are declarative. Using Tencent Cloud APIs directly for orchestration typically results in an imperative workflow.

State File

After initialization, Terraform generates a state file that records:

Timestamp of the last operation.
Attributes of each resource.
Current values of variables.
Version of the state file.

When running Terraform again, it first synchronizes the local state file with the actual state on the cloud provider to detect any deleted or modified resources. Then, based on the .tf configuration, it determines which resources need to be created, updated, or destroyed. After the operation completes, a new state file is generated.

Backend

The integrity of the state file is critical. We must ensure:

Automatic locking at the start of an operation and release only upon completion to prevent concurrent modifications.
Tracking version changes of resources.
Access control for sensitive information in the state file.

The backend defines how Terraform reads, stores, locks, and manages the state file—and is tightly coupled with how terraform apply executes.

By default, Terraform uses a local backend, storing the state file in the current directory and executing locally. This works fine for individual users managing small-scale infrastructure. But as team size grows, multiple developers may have separate workstations, requiring a shared location for state files. In such cases, remote backends become essential.

Terraform supports several remote backends:

AWS S3
HashiCorp Consul
etcd
Terraform Cloud
Terraform Enterprise

These remote backends provide secure remote storage and state locking. Terraform Enterprise also offers remote execution of Terraform and additional enterprise-grade features.

Module

A module is a reusable piece of Terraform code designed for easy sharing. A module consists of:

Input parameters (variables).
Output parameters.
Main logic.

This structure closely resembles functions in traditional programming languages. Terraform maintains a public module registry where modules can be published if they follow the standard format. Once published, others can use them directly.

https://registry.terraform.io/