Devops on Wanzi's Blog – Notes on life, tech, and DevOps.

Building an Enterprise Logging Platform with OpenObserve and Fluent Bit

wzdushu@gmail.com (wanzi) — Wed, 12 Nov 2025 09:30:00 +0800

Platform Architecture

Technology Selection

OpenObserve (formerly O2) is an open-source observability platform with significant advantages over traditional ELK solutions:

Low resource consumption: Memory usage is only 1/10 of Elasticsearch, with 90% storage space saved
High performance: Written in Rust, a single node can process 5TB+ logs/day
Full-featured: Integrates logs, metrics, and distributed tracing, replacing Grafana+Loki+Tempo
Strong compatibility: Supports Elasticsearch API, Prometheus API, and OpenTelemetry
Easy deployment: Single binary file with no external dependencies

Fluent Bit is a cloud-native log collector:

Lightweight: Memory usage <1MB, suitable for embedded and container environments
High performance: Multi-threaded asynchronous processing, throughput up to 100MB/s
Rich plugins: Supports 200+ input/output/filter/parser plugins
Kubernetes native: Automatic Pod discovery, label injection, metadata association

Architecture Design

┌─────────────────────────────────────────────────────────────────┐
│                        Business Application Layer                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐       │
│  │  Java App│  │  Go Service │ │ Nginx Logs│ │ System Logs│      │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘       │
│       │             │             │             │               │
│  stdout/stderr   stdout/stderr  /var/log/*   /var/log/*         │
└───────┼─────────────┼─────────────┼─────────────┼───────────────┘
        │             │             │             │
┌───────┴─────────────┴─────────────┴─────────────┴───────────────┐
│                      Kubernetes Cluster                          │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │              DaemonSet: Fluent Bit                      │   │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │   │
│  │  │  Tail Input  │→ │  Filter/Parse│→ │  Buffer      │  │   │
│  │  └──────────────┘  └──────────────┘  └──────────────┘  │   │
│  └─────────────────────────────────────────────────────────┘   │
└──────────────────────────────┬──────────────────────────────────┘
                               │ HTTP/HTTPS
┌──────────────────────────────┴──────────────────────────────────┐
│                    OpenObserve Cluster                           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │  Ingest Node │→ │  Data Storage│→ │  Query Engine│          │
│  │  (Write Opt) │  │  (Parquet)   │  │  (Full Text) │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
└─────────────────────────────────────────────────────────────────┘

Component Versions

Component	Version	Description
OpenObserve	0.15.0+	Stable version, supports distributed
Fluent Bit	4.0.0+	Latest stable version
Kubernetes	1.24+	Production environment
Helm	3.0+	Deployment tool

OpenObserve Deployment

Prerequisites

# 1. Create Namespace
kubectl create namespace logging

# 2. Add OpenObserve Helm repository
helm repo add openobserve https://charts.openobserve.ai
helm repo update

# 3. Create PV/PVC (use distributed storage for production)
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: openobserve-data
  namespace: logging
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: zstack-rbd  # Adjust according to actual SC
  resources:
    requests:
      storage: 500Gi
EOF

Helm Deployment of OpenObserve

Create values file openobserve-values.yaml:

# openobserve-values.yaml

image:
  repository: public.ecr.aws/zinclabs/openobserve
  tag: "0.15.1"
  pullPolicy: IfNotPresent

replicaCount: 3  # 3 nodes recommended for HA in production

# Resource configuration (adjust based on log volume)
resources:
  limits:
    cpu: "8"
    memory: 16Gi
  requests:
    cpu: "4"
    memory: 8Gi

# Data persistence
persistence:
  enabled: true
  existingClaim: openobserve-data
  mountPath: /data

# Environment variables
env:
  - name: ZO_ROOT_USER_EMAIL
    value: "admin@example.com"
  - name: ZO_ROOT_USER_PASSWORD
    valueFrom:
      secretKeyRef:
        name: openobserve-secret
        key: password
  - name: ZO_DATA_DIR
    value: "/data"
  - name: ZO_HTTP_PORT
    value: "5080"
  - name: ZO_MEMORY_CACHE_ENABLED
    value: "true"
  - name: ZO_MEMORY_CACHE_MAX_SIZE
    value: "4096"  # MB
  - name: ZO_COMPRESSION_ENABLED
    value: "true"
  - name: ZO_COMPRESSION_FORMAT
    value: "zstd"
  - name: ZO_PARQUET_COMPRESSION
    value: "zstd"
  - name: ZO_META_STORE
    value: "sqlite"  # Use PostgreSQL for production
  - name: ZO_METRICS_ENABLED
    value: "true"

# Configure PostgreSQL metadata store (recommended for production)
envFrom:
  - secretRef:
      name: postgres-connection

# Service configuration
service:
  type: ClusterIP
  port: 5080
  targetPort: 5080
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

# Ingress configuration
ingress:
  enabled: true
  className: "nginx"
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - host: logs.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: logs-tls
      hosts:
        - logs.example.com

# Pod scheduling
nodeSelector: {}
tolerations: []
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - openobserve
          topologyKey: kubernetes.io/hostname

# Monitoring configuration
monitoring:
  enabled: true
  serviceMonitor:
    enabled: true
    interval: 30s
    namespace: logging

Deploy OpenObserve

# 1. Create password Secret
kubectl create secret generic openobserve-secret \
  --from-literal=password='your-secure-password' \
  -n logging

# 2. Deploy
helm upgrade --install openobserve openobserve/openobserve \
  -n logging \
  -f openobserve-values.yaml \
  --version 0.10.2 \
  --timeout 15m

# 3. Wait for Pods to be ready
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=openobserve \
  -n logging \
  --timeout=300s

# 4. Check status
kubectl get pods -n logging
kubectl logs -f deployment/openobserve -n logging

# 5. Access Web UI
echo "Access URL: https://logs.example.com"
echo "Default user: admin@example.com"

Fluent Bit Deployment

Install Helm Chart

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
helm pull fluent/fluent-bit --version 0.50.0
helm install fluent-bit ./fluent-bit-0.50.0.tgz --namespace uganda-prod

Adjust DaemonSet Configuration

spec:
  volumes:
    - name: config
      configMap:
        name: fluent-bit
        defaultMode: 420
  containers:
    resources:
      requests:
        cpu: "1"
        memory: 2Gi
      limits:
        cpu: "2"
        memory: 4Gi
    volumeMounts:
      - name: config
        mountPath: /fluent-bit/etc/conf/custom_parsers.conf
        subPath: custom_parsers.conf
      - name: config
        mountPath: /fluent-bit/etc/conf/multiline_parsers.conf
        subPath: multiline_parsers.conf

Fluent Bit Collection Rule Adjustments

Complete ConfigMap Configuration

kind: ConfigMap
apiVersion: v1
metadata:
  name: fluent-bit-config
  namespace: uganda-prod
  labels:
    app: fluent-bit
    tier: logging
data:
  # ---------------------------------------------------------------------------
  # 1. Custom Parsers
  # Used to extract specific log fields, e.g., extract log level (INFO, ERROR, etc.) from Java logs
  # ---------------------------------------------------------------------------
  custom_parsers.conf: |
    [PARSER]
        Name        java_log_level
        Format      regex
        # Match format: 2023-10-27 10:00:00.123 INFO ...
        # The 'level' capture group will contain the log level
        Regex       ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}\s+(?<level>[A-Z]+)

  # ---------------------------------------------------------------------------
  # 2. Multiline Parsers
  # Specifically for handling multi-line logs like Java stack traces to prevent stacks from being split into multiple records
  # ---------------------------------------------------------------------------
  multiline_parsers.conf: |
    [MULTILINE_PARSER]
        Name          java_md_multiline
        Type          regex
        # Rule 1: If line starts with date (e.g., 2023-...), consider it start of new log
        rule      "start_state"   "/^\d{4}-\d{2}-\d{2}/"           "cont"
        # Rule 2: If line doesn't start with date, consider it continuation of previous line (stack part)
        rule      "cont"        "/^(?!\d{4}-\d{2}-\d{2}).+/"     "cont"
        # Flush timeout (seconds), force output current buffered multi-line log after timeout
        flush_timeout 5

  # ---------------------------------------------------------------------------
  # 3. Main Configuration File
  # ---------------------------------------------------------------------------
  fluent-bit.conf: |
    [SERVICE]
        Daemon              Off
        Flush               1
        Log_Level           info
        # Load standard and custom parsers
        Parsers_File        /fluent-bit/etc/conf/parsers.conf
        Parsers_File        /fluent-bit/etc/conf/custom_parsers.conf
        Parsers_File        /fluent-bit/etc/conf/multiline_parsers.conf
        
        # Enable built-in HTTP server for health checks and metrics exposure (/api/v1/metrics)
        HTTP_Server         On
        HTTP_Listen         0.0.0.0
        HTTP_Port           2020
        Health_Check        On

        # File system buffer configuration (prevent data loss when backend is unavailable)
        storage.path              /var/log/flb_storage
        storage.sync              normal
        storage.checksum          off
        storage.backlog.mem_limit 200M

    # -----------------------------------------------------------------------
    # [INPUT] Tail file collection
    # Collect standard container logs generated by Docker/Containerd
    # -----------------------------------------------------------------------
    [INPUT]
        Name                tail
        Path                /var/log/containers/*.log
        # Enable built-in multiline parsing (docker, cri format)
        multiline.parser    docker, cri
        Tag                 kube.*
        Mem_Buf_Limit       500MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Persist offset database, resume reading from checkpoint after restart
        DB                  /var/log/flb_kube.db
        DB.Sync             Normal
        Rotate_Wait         30
        Read_from_Head      Off

    # -----------------------------------------------------------------------
    # [FILTER] Stage 1: Multiline merging
    # Merge multiline logs before getting K8s metadata to ensure stack traces are handled as single records
    # -----------------------------------------------------------------------
    [FILTER]
        Name                multiline
        Match               kube.*
        multiline.key_content log
        multiline.parser    java_md_multiline

    # -----------------------------------------------------------------------
    # [FILTER] Stage 2: K8s metadata enhancement
        Call K8s API to get Pod details (Namespace, Pod Name, Labels, etc.)
    # -----------------------------------------------------------------------
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           Off       # Don't try to parse log field as JSON (avoid performance overhead or errors)
        Keep_Log            On        # Keep original log field
        K8S-Logging.Parser  On        # Allow Pod Annotation to specify parser
        K8S-Logging.Exclude Off       # Don't exclude logs with specific annotation

    # -----------------------------------------------------------------------
    # [FILTER] Stage 3: Log content parsing
    # Use custom parser to extract 'level' field
    # -----------------------------------------------------------------------
    [FILTER]
        Name                parser
        Match               kube.*
        Key_Name            log
        Parser              java_log_level
        Reserve_Data        On        # Keep original data
        Preserve_Key        On        # Keep original log key

    # -----------------------------------------------------------------------
    # [FILTER] Stage 4: Field cleaning
    # Copy extracted level field, remove unnecessary temporary fields
    # -----------------------------------------------------------------------
    [FILTER]
        Name                modify
        Match               kube.*
        Copy                level level
        Remove              _p
        Remove              stream
        Remove              time

    # -----------------------------------------------------------------------
    # [FILTER] Stage 5: Structure restructuring (Nest Lift)
    # Lift fields under kubernetes object to top level with 'k8s_' prefix
    # Purpose: Flatten data structure for easier rule matching
    # -----------------------------------------------------------------------
    [FILTER]
        Name                nest
        Match               kube.*
        Operation           lift
        Nested_under        kubernetes
        Add_prefix          k8s_

    # -----------------------------------------------------------------------
    # [FILTER] Stage 6: Remove redundant metadata
    # Delete large fields not needed to be sent to backend (e.g., annotations, docker_id)
    # -----------------------------------------------------------------------
    [FILTER]
        Name                modify
        Match               kube.*
        Remove              k8s_annotations
        Remove              k8s_docker_id
        Remove              k8s_container_hash

    # -----------------------------------------------------------------------
    # [FILTER] Stage 7: Structure restructuring (Nest Nest)
    # Repackage all 'k8s_' prefixed fields back under 'kubernetes' object, removing prefix
    # Purpose: Restore clean nested structure while cleaning up useless fields
    # -----------------------------------------------------------------------
    [FILTER]
        Name                nest
        Match               kube.*
        Operation           nest
        Wildcard            k8s_*
        Nested_under        kubernetes
        Remove_prefix       k8s_

    # =======================================================================
    # [FILTER] Stage 8: Dynamic routing (Rewrite Tag) - First hop: Split by environment
    # Modify Tag based on Namespace name to route logs to different processing streams
    # Syntax: Rule $field_name regex newTag keep_original(boolean)
    # =======================================================================
    
    # Route: uganda-uat environment
    [FILTER]
        Name                rewrite_tag
        Match               kube.*
        Rule                $kubernetes['namespace_name'] ^uganda-uat$ uganda-uat.temp false
        Emitter_Name        re_emitted_uganda-uat-temp

    # Route: uganda-test environment
    [FILTER]
        Name                rewrite_tag
        Match               kube.*
        Rule                $kubernetes['namespace_name'] ^uganda-test$ uganda-test.temp false
        Emitter_Name        re_emitted_uganda-test-temp

    # Route: uganda-prod environment (current deployment namespace)
    [FILTER]
        Name                rewrite_tag
        Match               kube.*
        Rule                $kubernetes['namespace_name'] ^uganda-prod$ uganda-prod.temp false
        Emitter_Name        re_emitted_uganda-prod-temp

    # Route: uganda-offline environment
    [FILTER]
        Name                rewrite_tag
        Match               kube.*
        Rule                $kubernetes['namespace_name'] ^uganda-offline$ uganda-offline.temp false
        Emitter_Name        re_emitted_uganda-offline-temp

    # =======================================================================
    # [FILTER] Stage 9: Dynamic routing (Rewrite Tag) - Second hop: Split by service
    # Secondary routing for specific containers in specific environments for fine-grained index isolation
    # =======================================================================

    # Example: UAT environment -> lms-backend service
    [FILTER]
        Name                rewrite_tag
        Match               uganda-uat.temp
        Rule                $kubernetes['container_name'] ^lms-backend$ uganda-uat-lms-backend false
        Emitter_Name        re_emitted_uganda-uat-lms-backend

    # Example: UAT environment -> other-service service (needs complete configuration)
    [FILTER]
        Name                rewrite_tag
        Match               uganda-uat.temp
        Rule                $kubernetes['container_name'] ^other-service$ uganda-uat-other-service false
        Emitter_Name        re_emitted_uganda-uat-other-service

    # Note: Production and other environments need similar container name filtering rules added
    # For example:
    # [FILTER]
    #     Name rewrite_tag
    #     Match uganda-prod.temp
    #     Rule $kubernetes['container_name'] ^payment-service$ uganda-prod-payment-service false
    #     Emitter_Name re_emitted_uganda-prod-payment-service

    # =======================================================================
    # [OUTPUT] Output plugin configuration
    # Send filtered logs to OpenObserve
    # =======================================================================

    # Output: UAT LMS Backend
    [OUTPUT]
        Name                http
        Match               uganda-uat-lms-backend
        URI                 /api/39NVPcXSEBOwGM5UnceQ35hQFNB/lms_backend/_json
        Host                openobserve.uganda-uat.svc.cluster.local
        Port                5080
        tls                 Off
        Format              json
        Json_date_key       _timestamp
        Json_date_format    iso8601
        
        HTTP_User           ops@test.com
        HTTP_Passwd         ROPe50N4BJjovJiT 
        
        compress            gzip
        Retry_Limit         False       # Infinite retry until successful
        net.connect_timeout 10
        net.io_timeout      30

    # Output: UAT Other Service (example)
    [OUTPUT]
        Name                http
        Match               uganda-uat-other-service
        URI                 /api/39NVPcXSEBOwGM5UnceQ35hQFNB/other_service/_json
        Host                openobserve.uganda-uat.svc.cluster.local
        Port                5080
        tls                 Off
        Format              json
        Json_date_key       _timestamp
        Json_date_format    iso8601
        HTTP_User           devops@test.com
        HTTP_Passwd         ROPe50N4BJjovJiT
        compress            gzip
        Retry_Limit         False

    # Default output (optional): Capture logs that don't match any specific rules to prevent data loss
    # [OUTPUT]
    #     Name http
    #     Match uganda-prod.temp
    #     URI /api/.../default/_json
    #     ...

Configuration Explanation

Filter Stage: Use rewrite_tag filter to filter logs by Namespace and container name
Output Stage: Send filtered logs to OpenObserve in different environments via HTTP
Multiline Logs: Use multiline filter to handle Java exception stacks
Log Optimization: gzip compression, retry on failure, TCP timeout configuration

Multiline Merge Rule Explanation

State	Matching Rule	Description
start_state	`/^\d{4}-\d{2}-\d{2}/`	Starts with date → New log starts, enters cont state
cont	`/^(?!\d{4}-\d{2}-\d{2}).+/`	Doesn’t start with date → Continue appending to previous log

Note: After modifying the ConfigMap, the fluent-bit service needs to be restarted.

Data Management

Create Data Stream

# Method 1: Create via API
curl -X POST "https://logs.example.com/api/demo/streams" \
  -u "admin@example.com:password" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "app-logs",
    "storage_type": "memory",
    "stream_type": "logs"
  }'

# Method 2: Auto-create (first write auto-creates, we use this method)
# Fluent Bit will auto-create new stream on first write

Set Data Retention Policy

OpenObserve’s dashboard allows setting retention policies based on index, with main strategies as follows:

Time-based retention
Size-based retention
Hybrid strategy (3650 days or 30TB, whichever comes first, we use this approach. Since it’s a financial service, we set retention for 10 years)

Data Archive Configuration

# Enable S3 archiving
env:
  - name: ZO_S3_STORE_ENABLED
    value: "true"
  - name: ZO_S3_STORE_BUCKET
    value: "logs-archive"
  - name: ZO_S3_STORE_REGION
    value: "us-east-1"
  - name: ZO_S3_STORE_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: s3-credentials
        key: access-key
  - name: ZO_S3_STORE_SECRET_KEY
    valueFrom:
      secretKeyRef:
        name: s3-credentials
        key: secret-key

Data Compression

# OpenObserve auto-compression
env:
  - name: ZO_COMPRESSION_ENABLED
    value: "true"
  - name: ZO_COMPRESSION_FORMAT
    value: "zstd"  # zstd/gzip/snappy
  - name: ZO_COMPRESSION_LEVEL
    value: "3"  # 1-19, higher compression ratio but higher CPU consumption

Query and Analysis

OpenObserve provides a powerful query engine supporting SQL mode and native query language (VQL). The following introduces basic queries and aggregation queries in different scenarios to help you quickly locate key logs.

💡 Core Tips

Mode Switch: Aggregation queries are recommended to be executed in SQL mode.

Table Name Convention: Stream names (table names) must be enclosed in double quotes, for example "lms".

Time Range: All queries are limited by the time selector in the upper right corner of the UI, please ensure the time range covers the target data.

1. Basic Search

Quickly locate a single log or specific event.

1.1 Full-text Index Search

Automatically scans all text fields, suitable for fuzzy searching.

`1`	`match_all('timeout')`

Scenario: Not sure which field the error appears in, quickly search for keywords globally.

1.2 Field-specific Search

Perform exact or inclusion matching on specific fields for better performance.

`1`	`str_match(log, 'tenant-backend')`

Scenario: Filter logs for specific services (like tenant-backend) or specific levels (str_match(level, 'ERROR')).

1.3 Multi-condition Combination Query

Use logical operators (AND, OR, NOT) to build complex filtering rules.

1
2

(level = 'INFO' OR level = 'ERROR') 
AND str_match(log, 'AuthInterceptor')

Scenario: Only view logs from the authentication module, including both normal and error processes.

2. Aggregation Queries 🚀

Through SQL’s GROUP BY and aggregate functions (COUNT, SUM, AVG, MAX, MIN), massive logs can be converted into statistical chart data.

2.1 Count by Log Level

Count the number of different log levels to quickly determine system health.

SELECT level, COUNT(*) as log_count 
FROM "lms" 
GROUP BY level 
ORDER BY log_count DESC

Output Example:

level log_count

INFO 15420

ERROR 32

WARN 105

level	log_count
INFO	15420
ERROR	32
WARN	105

2.2 Time Window Trend Analysis

Combine with the date_trunc function to count logs within a time unit, used for drawing trend charts.

SELECT date_trunc('minute', _timestamp) as time_window, COUNT(*) as hits
FROM "lms"
WHERE str_match(log, 'payment')
GROUP BY time_window
ORDER BY time_window ASC

Scenario: Observe the fluctuation of “payment” related logs per minute to identify sudden traffic or failure time points.
Note: 'minute' can be replaced with 'hour', 'day', etc.

2.3 Group by Service/Container

Analyze which microservice generates the most logs or errors.

SELECT kubernetes.container_name, COUNT(*) as total_logs
FROM "lms"
WHERE level = 'ERROR'
GROUP BY kubernetes.container_name
ORDER BY total_logs DESC
LIMIT 5

Scenario: Find the top 5 containers generating the most errors, prioritize troubleshooting.

2.4 Multi-dimensional Cross Analysis

Group by two or more dimensions simultaneously for deep dive analysis.

SELECT 
  date_trunc('hour', _timestamp) as hour,
  level,
  COUNT(*) as count
FROM "lms"
WHERE str_match(log, 'database')
GROUP BY hour, level
ORDER BY hour ASC, level

Scenario: View the distribution of INFO and ERROR database-related logs per hour.

2.5 Numeric Field Statistics

If logs contain extracted numeric fields (like response time duration), mathematical calculations can be performed.

SELECT 
  AVG(duration) as avg_latency,
  MAX(duration) as max_latency,
  MIN(duration) as min_latency
FROM "lms"
WHERE str_match(log, 'API_REQUEST')

Scenario: Calculate average and maximum latency for specific API requests to evaluate performance bottlenecks.

3. Advanced Techniques and Best Practices

Technique	Description	Example
Limit Result Set	Avoid returning too much data causing browser lag	Add `LIMIT 100`
Deduplication Count	Count unique error message types	`COUNT(DISTINCT log)`
Alias Optimization	Make output column names more readable	`COUNT(*) as "Total Errors"`
Null Value Handling	Exclude records with empty fields	`WHERE level IS NOT NULL`
Regex Matching	More flexible matching than `str_match`	`REGEXP_MATCH(log, 'Error.*\d+')`

⚠️ Performance Recommendations

Filter before aggregating: Be sure to narrow down the data range as much as possible in the WHERE clause (specifying time, keywords) before performing GROUP BY, which can significantly improve query speed.
Time granularity: When querying over large time ranges (like 7 days), use date_trunc('hour', ...) or date_trunc('day', ...), avoid using 'second' which causes too many data points.
Field indexing: For fields commonly used in GROUP BY (like level, container_name), it’s recommended to enable indexing in Stream Settings for best performance.

OpenObserve Optimization

# Memory cache
env:
  - name: ZO_MEMORY_CACHE_ENABLED
    value: "true"
  - name: ZO_MEMORY_CACHE_MAX_SIZE
    value: "8192"  # 8GB

# Query cache
  - name: ZO_QUERY_CACHE_ENABLED
    value: "true"
  - name: ZO_QUERY_CACHE_MAX_SIZE
    value: "4096"  # 4GB

# Parallel queries
  - name: ZO_QUERY_WORKER_THREADS
    value: "8"

Storage Optimization

# Parquet columnar storage
env:
  - name: ZO_PARQUET_COMPRESSION
    value: "zstd"
  - name: ZO_PARQUET_ROW_GROUP_SIZE
    value: "1048576"  # 1MB

# Data partitioning
  - name: ZO_PARTITION_ENABLED
    value: "true"
  - name: ZO_PARTITION_FIELDS
    value: "_timestamp,kubernetes.namespace_name"

Monitoring and Alerting

There are two parts that need monitoring: one is the OpenObserve service monitoring itself, and the other is business log monitoring. For monitoring of OpenObserve and Fluent Bit services themselves, Prometheus monitoring can be used directly. For core business order log monitoring, OpenObserve’s built-in alerting mechanism can meet the requirements directly.

Prometheus Metrics Collection

# Fluent Bit ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit
  endpoints:
    - port: metrics
      interval: 30s

# OpenObserve ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: openobserve
  namespace: logging
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: openobserve
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

Alert Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: logging-alerts
  namespace: logging
spec:
  groups:
    - name: logging
      rules:
        # Fluent Bit log loss alert
        - alert: FluentBitLogsDropping
          expr: rate(fluentbit_output_proc_records_failed_total[5m]) > 100
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Fluent Bit Log Loss"
            description: "Lost {{ $value }} logs in the last 5 minutes"

        # OpenObserve storage alert
        - alert: OpenObserveDiskSpaceLow
          expr: (node_filesystem_avail_bytes{mountpoint="/data"} / node_filesystem_size_bytes{mountpoint="/data"}) < 0.1
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "OpenObserve Disk Space Low"
            description: "Less than 10% disk space remaining"

        # Log ingestion delay
        - alert: LoggingIngestLag
          expr: openobserve_ingest_delay_seconds > 300
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Log Ingestion Lag"
            description: "Log lag is {{ $value }} seconds"

Business Log Alerts

This involves several key parts:

Alert Message Template

Settings –> Templates –> Add template, define Webhook alert message format, note that the message needs to be configured differently for different chat tools

Create: cdi-prod-template

    {
      "msgtype": "text",
      "text": {
        "content": "🚨 [Alert Notification]\nAlert Name: {alert_name}\nStream Name: {stream_name}\nSeverity: Error\nLog Info: {rows:1}\nTrigger Time: {alert_trigger_time_str}"
      }
    }

Note: The message here needs to be configured differently for different chat tools. My configuration here is for WeChat Work. For more information, please refer to https://openobserve.ai/docs/user-guide/management/templates/ to configure a suitable alert template for your needs.

Alert Address

Add a message assistant to the WeChat Work group to get the push message Webhook address, then add it in OpenObserve UI Settings –> Address –> Add Address.

Alert Rule Configuration

This needs to be determined based on internal R&D team communication, business log matching rules. Daily alerts should only match ERROR level logs. For core orders and risk control processes, phone alerts are recommended:

OpenObserve Open Source Edition Risks and Considerations

RBAC Permission Control

Open Source Edition Limitations:

No granular RBAC control (Enterprise edition supports)
All users have the same permissions
Cannot isolate data by organization/project

Solutions:

# 1. Use multi-tenancy (Stream-level isolation)
# Create separate Streams for each team
curl -X POST "https://logs.example.com/api/demo/streams" \
  -d '{"name":"team-a-logs"}'
curl -X POST "https://logs.example.com/api/demo/streams" \
  -d '{"name":"team-b-logs"}'

# 2. Use reverse proxy for permission control
# Nginx routes users to different Streams based on user
location /api/team-a/ {
  internal;
  proxy_pass http://openobserve.logging.svc/api/team-a-logs/;
}

# 3. Use API Key for simple authentication
# Generate separate API Key for each team

Here, since there are many projects, permission control rules are usually added at the reverse proxy, such as restricting IP and Host for POST requests.

Data Security

Risk Points:

Open source edition has no field-level encryption
Sensitive information may be stored in plain text
Missing audit logs

Best Practices:

# 1. Fluent Bit data masking
[FILTER]
    Name                modify
    Match               kube.*
    Remove              password,token,secret,key,ssn
    Rename              message  log_message

# 2. Use Lua filter for masking
[FILTER]
    Name                lua
    Match               kube.*
    Script              mask.lua
    Call                mask_sensitive

# mask.lua:
function mask_sensitive(tag, timestamp, record)
    if record["message"] ~= nil then
        record["message"] = string.gsub(record["message"], "password=%S+", "password=***")
    end
    return 1, timestamp, record
end

High Availability Risks

Single Point of Failure:

Open source edition has no automatic failover
Node downtime may cause data loss

Solutions:

# 1. Deploy multiple replicas
replicaCount: 3

# 2. Use shared storage
persistence:
  enabled: true
  storageClass: "nfs-client"  # Use distributed storage, here using Ceph RBD

# 3. Configure pod anti-affinity
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app.kubernetes.io/name
              operator: In
              values:
                - openobserve
        topologyKey: kubernetes.io/hostname

Capacity Planning

Log Volume	Fluent Bit Resources	OpenObserve Resources	Storage Space/Month
10GB/day	100m CPU / 100Mi	2 CPU / 4Gi	100GB
100GB/day	200m CPU / 200Mi	4 CPU / 8Gi	1TB
1TB/day	500m CPU / 500Mi	8 CPU / 16Gi	10TB

Summary

The log platform solution based on OpenObserve and Fluent Bit has the following advantages:

Cost Advantage: Compared to ELK solutions, storage costs are reduced by 90% and computing resources by 70%
High Performance: A single node supports 5TB+/day log ingestion with query responses <100ms
Simple and Easy to Use: Deployment time <30 minutes with low learning curve
Cloud Native: Kubernetes native integration with automatic scaling

It has been running stably in production for half a year with 5TB+ data volume, and query responses are stable within 200ms, basically meeting most business needs. If there are strong requirements for RBAC, it is recommended to directly purchase the commercial version or abandon this solution.

Kustomize Private Deployment of Multi-Scenario, Multi-Environment Container Services

wzdushu@gmail.com (wanzi) — Sat, 25 Oct 2025 10:30:00 +0800

I have been using Helm or native kubectl YAML combined with CI and ArgoCD to deploy applications for a long time. Recently, while deploying private services for a major financial client, I found the client’s environment to be quite complex, involving multiple countries and multiple sets of environments, such as Test, UAT, and Prod—about thirty environments in total. Modifying values and YAML files every time was exhausting. This time, I plan to use Kustomize to refactor our existing application deployment.

Kustomize Introduction

What is Kustomize

Kustomize is a Kubernetes-native configuration management tool that manages YAML resources through a templating approach:

Core Features:

No template syntax, pure YAML
Declarative configuration management
Supports multi-environment management
GitOps friendly
Integrated into kubectl 1.14+

Comparison with Helm:

Feature	Kustomize	Helm
Template Syntax	None (Pure YAML)	Go Template
Learning Curve	Low	Medium-High
Multi-Environment Management	Base + Overlay	Values files
Package Management	None	Chart Repository
Version Management	Git	Chart Version
Suitable Scenarios	Application Developer	Application Distributor

Basic Concepts

kustomization.yaml
├── bases/          # Base configuration (reuse)
├── overlays/       # Environment overrides
│   ├── dev/
│   ├── staging/
│   └── prod/
├── patches/        # Configuration patches
└── resources/      # Static resources

Differences between Kustomize and Helm

Design Philosophy

Kustomize Design Philosophy:

# Declarative configuration management
# Does not use templates, but modifies configuration via Overlay

# Base: Base configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: app
          image: my-app:latest

# Overlay: Environment differences
# overlays/production/kustomization.yaml
bases:
  - ../../base
patchesStrategicMerge:
  - |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-app
    spec:
      replicas: 10

Helm Design Philosophy:

# Templated configuration management
# Uses Go Template

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
        - name: app
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

# values.yaml (Different values for different environments)
replicaCount: 1
image:
  repository: my-app
  tag: latest

# values-production.yaml
replicaCount: 10
image:
  repository: my-app
  tag: v1.2.3

Feature Comparison

Feature	Kustomize	Helm
Templates	No templates, pure YAML	Go Template
Learning Curve	Low, only need to know YAML	Medium-High, need to learn template syntax
Reusability	Base + Overlay	Chart + Values
Debugging	kubectl kustomize	helm template –dry-run
Version Management	Git version control	Chart version number
Environment Management	Overlay directories	Multiple Values files
Package Distribution	None (use Git)	Chart repository
CI/CD Integration	Native kubectl	Requires helm installation
Rollback	kubectl rollout	helm rollback
Dependency Management	No direct support	Chart dependencies

Applicable Scenarios

Scenarios for choosing Kustomize:

Application developers managing their own K8s resources
Need GitOps workflows
Large configuration differences across environments
Do not want to learn template languages
Team is familiar with YAML and kubectl

Scenarios for choosing Helm:

Application distributors (software vendors)
Need package management and version control
Relatively fixed configuration
Complex application dependencies
Quick deployment of third-party applications

Deploying K8s Applications with Kustomize

Project Structure

my-app/
├── base/                           # Base configuration
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap.yaml
│   ├── serviceaccount.yaml
│   └── kustomization.yaml
├── overlays/                       # Environment configuration
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   ├── deployment-replica.yaml
│   │   ├── deployment-resources.yaml
│   │   └── configmap-env.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── deployment-replica.yaml
│   └── production/
│       ├── kustomization.yaml
│       ├── deployment-replica.yaml
│       ├── deployment-resources.yaml
│       └── configmap-prod.yaml
└── scripts/
    └── deploy.sh

Creating Base Configuration

base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
    version: v1.0.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: my-app
        version: v1.0.0
    spec:
      serviceAccountName: my-app
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: my-app
          image: my-app:latest
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          env:
            - name: JAVA_OPTS
              value: "-Xms512m -Xmx512m"
            - name: SPRING_PROFILES_ACTIVE
              value: "prod"
          resources:
            requests:
              cpu: 100m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - name: config
              mountPath: /config
            - name: logs
              mountPath: /logs
      volumes:
        - name: config
          configMap:
            name: my-app-config
        - name: logs
          emptyDir: {}

base/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: my-app

base/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-config
data:
  application.yml: |
    server:
      port: 8080
    spring:
      application:
        name: my-app
    logging:
      level:
        root: INFO

base/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app

base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Referenced resource files
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml
  - serviceaccount.yaml

# Common labels (added to all resources)
commonLabels:
  app.kubernetes.io/name: my-app
  app.kubernetes.io/instance: my-app
  app.kubernetes.io/managed-by: kustomize
  app.kubernetes.io/version: v1.0.0

# Common annotations
commonAnnotations:
  createdBy: kustomize
  env: base

# Name prefix
# namePrefix: dev-

# Name suffix
# nameSuffix: -v1

Overlay Environment Configuration

overlays/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Reference base
namespace: production
bases:
  - ../../base

# Replica count override
replicas:
  - name: my-app
    count: 10

# Image override
images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

# ConfigMap generator
configMapGenerator:
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml

# Secret generator
secretGenerator:
  - name: my-app-secret
    behavior: create
    envs:
      - .env.production

# Resource patches
patchesStrategicMerge:
  - deployment-resources.yaml
  - deployment-probes.yaml
  - pvc.yaml

# Namespace
namespace: production

# Common labels override
commonLabels:
  environment: production
  tier: backend

# Common annotations override
commonAnnotations:
  env: production
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

overlays/production/deployment-resources.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: my-app
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: 2000m
              memory: 4Gi

overlays/production/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 50Gi

overlays/production/application-production.yml

server:
  port: 8080

spring:
  application:
    name: my-app
  datasource:
    url: jdbc:mysql://mysql-prod.svc:3306/mydb
    username: prod_user
    password: ${DB_PASSWORD}

logging:
  level:
    root: INFO
    com.example: DEBUG
  file:
    path: /logs

overlays/production/.env.production

1
2
3

DB_PASSWORD=prod_secure_password_123
API_KEY=prod_api_key_xyz
REDIS_PASSWORD=prod_redis_pass

Deployment Commands

# Build configuration (do not apply, view final YAML)
kubectl kustomize overlays/production

# Save rendered result
kubectl kustomize overlays/production > rendered-prod.yaml

# View differences
kubectl diff -k overlays/production

# Apply directly
kubectl apply -k overlays/production

# Delete resources
kubectl delete -k overlays/production

# View resources
kubectl get all -l app.kubernetes.io/name=my-app -n production

Unified Deployment Management Configuration

1. Namespace Management

Method 1: Specify in Overlay

1
2

# overlays/production/kustomization.yaml
namespace: production

Method 2: Manage as Resource

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - namespace.yaml

# overlays/production/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    environment: production
    name: production

2. Secret Management

Method 1: Environment Variable File

# overlays/production/kustomization.yaml
secretGenerator:
  - name: my-app-secret
    type: Opaque
    envs:
      - .env.production

Method 2: Literals

secretGenerator:
  - name: my-app-secret
    type: Opaque
    literals:
      - DB_PASSWORD=prod123
      - API_KEY=xyz789

Method 3: Command Generation

secretGenerator:
  - name: tls-cert
    type: kubernetes.io/tls
    files:
      - tls.crt=tls.crt
      - tls.key=tls.key

Method 4: Reference from Existing Secret

# overlays/production/kustomization.yaml
generatorOptions:
  disableNameSuffixHash: true

resources:
  - secret.yaml

# overlays/production/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
type: Opaque
stringData:
  DB_PASSWORD: prod123

Method 5: Sealed Secrets (Recommended for Production)

# Generate sealed secret locally
kubeseal -f secret.yaml -w sealed-secret.yaml

# Reference in kustomization
resources:
  - sealed-secret.yaml

3. PVC Management

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - pvc-logs.yaml
  - pvc-data.yaml

# overlays/production/pvc-logs.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd  # Use SSD storage class
  resources:
    requests:
      storage: 50Gi

# overlays/production/pvc-data.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
spec:
  accessModes:
    - ReadWriteMany  # Multi-node shared
  storageClassName: nfs-storage
  resources:
    requests:
      storage: 100Gi

4. Image Version Management

Method 1: Direct Specification

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

Method 2: Use SHA256

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    digest: sha256:abc123def456...  # Safest method

Method 3: Multiple Images

images:
  - name: nginx
    newName: harbor.example.com/library/nginx
    newTag: 1.21-alpine
  - name: redis
    newName: harbor.example.com/library/redis
    newTag: 6.2-alpine
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

Method 4: CI/CD Dynamic Update

# In CI Pipeline
export IMAGE_TAG=$(git rev-parse --short HEAD)
cd overlays/production
kustomize edit set image my-app=harbor.example.com/prod/my-app:$IMAGE_TAG
kubectl apply -k .

5. Node Affinity Configuration

# overlays/production/deployment-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      affinity:
        # Node affinity
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - c5.2xlarge
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values:
                      - us-west-1a
        # Pod affinity
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - mysql
                topologyKey: kubernetes.io/hostname

Reference in kustomization:

1
2

patchesStrategicMerge:
  - deployment-affinity.yaml

6. Taint and Toleration Configuration

# overlays/production/deployment-taints.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      # Tolerate dedicated node taints
      tolerations:
        - key: workload
          operator: Equal
          value: dedicated
          effect: NoSchedule
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 300
        # Tolerate GPU nodes
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

7. Replica Count Management

# Method 1: replicas field (Recommended)
replicas:
  - name: my-app
    count: 10
  - name: my-app-sidecar
    count: 2

# Method 2: Patch method
# overlays/production/deployment-replica.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 10

8. ConfigMap Management

# Method 1: configMapGenerator (Recommended)
configMapGenerator:
  # Generate from file
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml
      - logback.xml=logback-production.xml

  # Generate from literals
  - name: my-app-env
    behavior: create
    literals:
      - SPRING_PROFILES_ACTIVE=production
      - LOG_LEVEL=INFO
      - JAVA_OPTS=-Xms2g -Xmx4g

  # Generate from env file
  - name: my-app-file-env
    behavior: create
    envs:
      - .env.production

# Method 2: Direct resource reference
resources:
  - configmap.yaml

Complete Production Environment Example

overlays/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Namespace
namespace: production

# Reference base
bases:
  - ../../base

# Additional resources
resources:
  - namespace.yaml
  - pvc-logs.yaml
  - pvc-data.yaml
  - hpa.yaml
  - service-monitor.yaml
  - network-policy.yaml

# Replica count
replicas:
  - name: my-app
    count: 10

# Image version
images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

# ConfigMap
configMapGenerator:
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=config/application-production.yml
  - name: my-app-env
    behavior: merge
    literals:
      - ENV=production
      - LOG_LEVEL=INFO

# Secret
secretGenerator:
  - name: my-app-secret
    behavior: create
    envs:
      - .env.production

# Patches
patchesStrategicMerge:
  - deployment-resources.yaml
  - deployment-affinity.yaml
  - deployment-taints.yaml
  - deployment-probes.yaml

# Common labels
commonLabels:
  environment: production
  tier: backend
  team: platform

# Common annotations
commonAnnotations:
  env: production
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

CI/CD Integration

GitLab CI Example

# .gitlab-ci.yml
stages:
  - build
  - deploy

variables:
  REGISTRY: harbor.example.com
  APP_NAME: my-app

build:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker build -t $REGISTRY/$APP_NAME:$CI_COMMIT_SHA .
    - docker push $REGISTRY/$APP_NAME:$CI_COMMIT_SHA
  only:
    - main
    - develop

deploy:dev:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: dev
  script:
    - cd overlays/dev
    - kustomize edit set image $REGISTRY/$APP_NAME=$REGISTRY/$APP_NAME:$CI_COMMIT_SHA
    - kubectl apply -k .
    - kubectl rollout status deployment/my-app -n dev
  only:
    - develop

deploy:prod:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: production
  script:
    - cd overlays/production
    - kustomize edit set image $REGISTRY/$APP_NAME=$REGISTRY/$APP_NAME:$CI_COMMIT_SHA
    - kubectl apply -k .
    - kubectl rollout status deployment/my-app -n production
  when: manual
  only:
    - main

ArgoCD Integration

# argocd/application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-production
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/example/k8s-manifests.git
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Summary

Kustomize Best Practices

Directory Structure Standards
- Store common configurations in base
- Manage environment differences in overlays
- Use clear naming conventions
Version Management
- Use Git to manage all configurations
- Use SHA256 or explicit tags for images
- Do not use latest
Secret Management
- Use Sealed Secrets in production environments
- Do not commit sensitive information to Git
- Use External Secrets Operator to integrate with Vault
Component Reuse
- Extract common configurations as components
- Establish a Kustomize component library
- Avoid duplicate configurations
CI/CD Integration
- Automate image tag updates
- Use kubectl diff to preview changes
- Apply configuration only after validation passes

Kustomize vs Helm Selection

Scenario	Recommended Solution	Reason
Microservice Configuration Management	Kustomize	High flexibility, Git-friendly
Third-party Application Deployment	Helm	Rich ecosystem, one-click deployment
Complex Template Requirements	Helm	Go Template is powerful
Simple Configuration	Kustomize	No need to learn templates
GitOps	Kustomize	Native support

Learning Path

Basic Syntax: base + overlay structure
Patch Strategies: StrategicMerge vs JSON6902
Component Reuse: Extract common configurations
CI/CD Integration: Automated deployment workflows
GitOps Practice: Combine with ArgoCD/Flux

Kustomize is an ideal tool for configuration management in the cloud-native era. Mastering it can significantly improve Kubernetes application management efficiency. In actual deployment scenarios, if the differentiation between environments for new projects is small, you can combine VibeCoding tools to generate one-click deployment scripts for rapid delivery of new environments.

Several Ways for Azure AKS Clusters to Access Blob Storage

wzdushu@gmail.com (wanzi) — Fri, 30 May 2025 21:22:42 +0800

When working with Azure, Blob Storage is frequently used. This article documents several common approaches for accessing Blob Storage from applications running in production environments:

一、Accessing Blob Storage via a Microsoft Entra ID Application Account

Advantages:

Works outside Azure environments (e.g., on-premises servers, AWS EC2)
Supports cross-subscription and cross-tenant access

Disadvantages:

Secrets must be securely stored (e.g., in Azure Key Vault)
Manual secret rotation is required upon expiration; otherwise, the application will break
Risk of credential leakage, especially if secrets are hardcoded in CI/CD pipelines

The specific configuration is as follows:

1、Configure an Application-Specific Account

Register an application in Entra ID, create a service principal, generate a client secret or certificate, and assign the application the Storage Blob Data Contributor role (for read/write access). Record the following information:

Tenant ID
Client ID
Client Secret (expires after a set period—manual renewal required)

2、Grant Blob Storage Permissions to the Application

(1)In the Azure portal, navigate to your storage account (e.g., mdblob001).

(2)Go to Access control (IAM) → + Add → Add role assignment.

(3)Select the role: Storage Blob Data Contributor (grants read & write access to blobs). Under Assign access to, choose User, group, or service principal, search for your app name (e.g., MyBlobApp), select it, and click Review + assign.

3、Access Blob Storage from Application Code

Use ClientSecretCredential (or ClientCertificateCredential) in your code to obtain tokens. Below is a Python SDK example:

Install required packages:

`1`	`pip install azure-identity azure-storage-blob`

from azure.identity import ClientSecretCredential
from azure.storage.blob import BlobServiceClient

# Entra ID App Configuration
TENANT_ID = "3c4e99f3-4831-4352-ae86-4d8f59512865"
CLIENT_ID = "3357a2b1-9895-484d-8754-42c162f2c9eb"
CLIENT_SECRET = "DoB8Q~4Gwl6sdzV_ocfUvXA3BtHYJLgrimwwMcj1"

# Storage Account & File Info
ACCOUNT_NAME = "mdblob001"
LOCAL_FILE_PATH = "/mnt/d/file.txt"
BLOB_NAME = "uploaded-file.txt"
CONTAINER_NAME = "mycontainer"  # Ensure this is defined

# Obtain Token Credential
credential = ClientSecretCredential(
    tenant_id=TENANT_ID,
    client_id=CLIENT_ID,
    client_secret=CLIENT_SECRET
)

# Create Blob Service Client
blob_service_client = BlobServiceClient(
    account_url=f"https://{ACCOUNT_NAME}.blob.core.windows.net",
    credential=credential
)

# List containers (test permissions)
try:
    containers = blob_service_client.list_containers()
    for container in containers:
        print("Container:", container.name)
except Exception as e:
    print("Error:", e)

# Upload file
try:
    blob_client = blob_service_client.get_blob_client(
        container=CONTAINER_NAME,
        blob=BLOB_NAME
    )

    with open(LOCAL_FILE_PATH, "rb") as data:
        blob_client.upload_blob(data, overwrite=True)

    print(f"File '{LOCAL_FILE_PATH}' successfully uploaded as '{BLOB_NAME}'")

except FileNotFoundError:
    print("Local file not found")
except Exception as e:
    print("Upload failed:", e)

二、Accessing Blob Storage via AKS System-Assigned Managed Identity

Azure provides system-assigned managed identities—automatically managed service principals—for supported resources like VMs, AKS clusters, and App Services. AKS enables this by default.

Advantages:

Lifecycle automatically managed (created/destroyed with the AKS cluster)
No secrets to manage—fully keyless access
Higher security: credentials never appear in code, config files, or environment variables
Deep integration with Azure RBAC
SDKs only need DefaultAzureCredential to auto-acquire tokens

Disadvantages:

Cannot be reused across workloads with different permission requirements
RBAC assignments must be configured manually
Not portable across clouds
Naming is not customizable

1、Enable system-assigned managed identity for AKS cluster

az aks create --resource-group test --name md-test-southafrica --generate-ssh-keys  # Enable system-assigned managed identity only for new clusters
az aks update --resource-group test --name md-test-southafrica --enable-managed-identity  # For existing clusters
az aks show --resource-group test -n md-test-southafrica --query "identity"  # View the system-assigned managed identity
{
  "delegatedResources": null,
  "principalId": "37e6bf32-b18d-4e3a-9f97-b86d0c9dc2cf",
  "tenantId": "3c4e99f3-4831-4352-ae86-4d8f59512865",
  "type": "SystemAssigned",
  "userAssignedIdentities": null
}

2、Assign Blob Storage Role to the Managed Identity

Roles:

Storage Blob Data Reader: read-only
Storage Blob Data Contributor: read/write
Storage Blob Data Owner: read/write + ACL management (for ADLS Gen2)

Use Azure CLI to grant the identity access to Blob Storage (for example, the Storage Blob Data Contributor role):

# Get the storage account ID
STORAGE_ACCOUNT_ID=$(az storage account show -n mdblob001 -g test --query id -o tsv)
echo $STORAGE_ACCOUNT_ID
/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001

# Assign the role
AKS_PRINCIPAL_ID=$(az aks show --resource-group test --name md-test-southafrica --query "identity.principalId" --output tsv)
echo "AKS Principal ID: $AKS_PRINCIPAL_ID"
AKS Principal ID: 37e6bf32-b18d-4e3a-9f97-b86d0c9dc2cf

az role assignment create \
  --assignee "37e6bf32-b18d-4e3a-9f97-b86d0c9dc2cf" \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001"
# Grant permissions to the AKS system-assigned managed identity

{
  "condition": null,
  "conditionVersion": null,
  "createdBy": "dc8278a7-78d6-4f6e-971c-8974d0ac4e2f",
  "createdOn": "2025-02-19T04:31:42.582306+00:00",
  "delegatedManagedIdentityResourceId": null,
  "description": null,
  "id": "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001/providers/Microsoft.Authorization/roleAssignments/81f9e7d1-b83c-4c4b-aab0-047fc4fb525f",
  "name": "81f9e7d1-b83c-4c4b-aab0-047fc4fb525f",
  "principalId": "37e6bf32-b18d-4e3a-9f97-b86d0c9dc2cf",
  "principalName": "ad674b19-a3b1-47ed-9463-128289ae5e34",
  "principalType": "ServicePrincipal",
  "resourceGroup": "Test",
  "roleDefinitionId": "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/providers/Microsoft.Authorization/roleDefinitions/ba92f5b4-2d11-453d-a403-e96b0029c9fe",
  "roleDefinitionName": "Storage Blob Data Contributor",
  "scope": "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001",
  "type": "Microsoft.Authorization/roleAssignments",
  "updatedBy": "dc8278a7-78d6-4f6e-971c-8974d0ac4e2f",
  "updatedOn": "2025-02-19T04:31:42.582306+00:00"
}

az role assignment list \
  --scope "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001" \
  --output table
# List role assignments under the storage account

Principal                             Role                           Scope
--------------------------------------------------------------------------------
ad674b19-a3b1-47ed-9463-128289ae5e34  Storage Blob Data Contributor  /subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001

3、Verification

Use the Python SDK to build a container and verify access:

from azure.storage.blob import BlobServiceClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
blob_service_client = BlobServiceClient(
    account_url="https://mdblob001.blob.core.windows.net",
    credential=credential
)

# List containers
containers = blob_service_client.list_containers()
for c in containers:
    print(c.name)

DefaultAzureCredential automatically attempts multiple authentication methods and, in AKS, retrieves a token via IMDS (Instance Metadata Service), provided that the pod is running on a node with managed identity enabled.

三、Accessing Blob Storage via AKS User-Assigned Managed Identity with Workload Identity

This is Microsoft’s recommended, standards-based (OIDC), secure approach for fine-grained workload identity.

1、Create a User-Assigned Managed Identity

az group create -n rg-aks-wi -l test  # Create a resource group; skip if it already exists
az identity create --name MD-Blob-Identity --resource-group test  # Create a user-assigned managed identity
{
  "clientId": "7d9b65f6-959d-4a74-ab3f-67d16adc8c37",
  "id": "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourcegroups/test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MD-Blob-Identity",
  "location": "southafricanorth",
  "name": "MD-Blob-Identity",
  "principalId": "bdd58864-063f-451d-b352-2dded3582b36",
  "resourceGroup": "test",
  "systemData": null,
  "tags": {},
  "tenantId": "3c4e99f3-4831-4352-ae86-4d8f59512865",
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities"
}

# Retrieve key information
export USER_ASSIGNED_CLIENT_ID="$(az identity show -g test -n MD-Blob-Identity --query clientId -o tsv)"
export USER_ASSIGNED_PRINCIPAL_ID="$(az identity show -g test -n MD-Blob-Identity --query principalId -o tsv)"

2、Assign Blob Storage Permissions

STORAGE_ACCOUNT_ID=$(az storage account show -n mdblob001 -g test --query id -o tsv)
echo $STORAGE_ACCOUNT_ID
/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001
az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee $USER_ASSIGNED_PRINCIPAL_ID \
  --scope $STORAGE_ACCOUNT_ID

3、Enable Workload Identity on AKS (if not already enabled)

1
2

az aks update -g test -n md-test-southafrica --enable-oidc-issuer
export AKS_OIDC_ISSUER=$(az aks show -g test -n md-test-southafrica --query "oidcIssuerProfile.issuerUrl" -o tsv)

4、Federate the Identity with a Kubernetes ServiceAccount

# Get the client ID of the user-assigned identity
export USER_ASSIGNED_CLIENT_ID=$(az identity show -g test -n MD-Blob-Identity --query clientId -o tsv)

# Create a federated credential (allowing a specific ServiceAccount in AKS to use this identity)
az identity federated-credential list --resource-group test --name md-test-southafrica

az identity federated-credential create --name my-fed-cred \
  --identity-name MD-Blob-Identity \
  --resource-group test \
  --issuer $AKS_OIDC_ISSUER \
  --subject system:serviceaccount:demo:md-blob-sa \
  --audience api://AzureADTokenExchange

{
  "audiences": [
    "api://AzureADTokenExchange"
  ],
  "id": "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourcegroups/test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MD-Blob-Identity/federatedIdentityCredentials/my-fed-cred",
  "issuer": "https://oidc.prod-aks.azure.com/3c4e99f3-4831-4352-ae86-4d8f59512865/",
  "name": "my-fed-cred",
  "resourceGroup": "test",
  "subject": "system:serviceaccount:demo:md-blob-sa",
  "systemData": null,
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials"
}

Subject format:：system:serviceaccount::

5、Create a ServiceAccount in Kubernetes and deploy the application.

Create Kubernetes ServiceAccount

# blob-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: blob-access-sa
  namespace: demo
  annotations:
    azure.workload.identity/client-id: "${USER_ASSIGNED_CLIENT_ID}"

apply

kubectl apply -f blob-sa.yaml

6、Deploy Application Pod

Here we use Python as an example.

# python-blob.yaml
apiVersion: v1
kind: Pod
metadata:
  name: blob-client
  namespace: demo
  labels:
    azure.workload.identity/use: "true"  # This label is required!
spec:
  serviceAccountName: blob-access-sa
  containers:
    - name: app
      image: python:3.11-slim
      command: ["sh", "-c"]
      args:
        - |
          pip install azure-identity azure-storage-blob && \
          python -c "
from azure.storage.blob import BlobServiceClient
from azure.identity import DefaultAzureCredential
import os

account = 'mdblob001'
url = f'https://mdblob001.blob.core.windows.net'
client = BlobServiceClient(account_url=url, credential=DefaultAzureCredential())
print('Containers:', [c.name for c in client.list_containers()])
"
      env:
        - name: AZURE_CLIENT_ID
          value: "${USER_ASSIGNED_CLIENT_ID}"  # Optional but recommended to explicitly specify
  restartPolicy: Never

Deploy and verify:

kubectl apply -f python-blob.yaml 
kubectl logs blob-client # The application can successfully list the Blob Storage containers, and it accesses Blob Storage using a token, achieving the entire process without any secrets.

With these three methods—Entra ID app credentials, system-assigned managed identity, and user-assigned identity with Workload Identity—you now have flexible, secure options for accessing Azure Blob Storage from AKS workloads.

Generate a Sitemap with a Python Script

wzdushu@gmail.com (wanzi) — Sun, 10 Mar 2024 10:22:42 +0800

Recently, I was helping a friend with a CMS cluster and found that the CMS did not have a sitemap feature. Since Python 3 was already installed on the server, I decided to temporarily generate a sitemap using a script. Here is a record of it.

Script Content:

#!/usr/bin/env python

import datetime
import mysql.connector
import xml.etree.cElementTree as ET
from lxml import etree

# Database connection parameters
config = {
    'host': '10.80.0.3', # For example: '192.168.1.100'
    'user': 'wnote_r',
    'password': 'Wnote#Pss2024',
    'database': 'wnote',
    'raise_on_warnings': True
}

# Sitemap path
sdir = '/opt/wwwroot/'

def mselect(site, n1, n2):
    # Connect to the database
    cnx = mysql.connector.connect(**config)
    cursor = cnx.cursor()

    # Get article IDs
    article_ids = []
    tag_ids = []

    # Query all records in the article table
    query1 = "select id from article order by newstime desc  limit {};".format(n1)
    cursor.execute(query1)
    for row in cursor:
        article_ids.append(row[0])

    # Query all tags
    query2 = "select id from phome_ecms_book order by newstime desc limit {};".format(n2)
    cursor.execute(query2)
    for row in cursor:
        tag_ids.append(row[0])

    # Generate URL lists
    article_urls = [f"https://{site}/article/{id}.html" for id in article_ids]
    tag_urls = [f"https://{site}/tags/{id}.html" for id in tag_ids]

    # Merge URL lists
    urls = article_urls + tag_urls

    # Create the XML tree structure for the sitemap
    root = ET.Element('urlset',
                  {'xmlns': 'http://www.sitemaps.org/schemas/sitemap/0.9',
                   'xmlns:xsi': 'http://www.w3.org/2001/XMLSchema-instance',
                   'xmlns:mobile': 'http://www.baidu.com/schemas/sitemap-mobile/1/',
                   'xsi:schemaLocation': 'http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd'})

    # Generate the sitemap file and build URL elements
    for url in urls:
        s_url = ET.SubElement(root, 'url')
        s_loc = ET.SubElement(s_url, 'loc')
        s_loc.text = url
        s_lastmod = ET.SubElement(s_url, 'lastmod')
        s_lastmod.text = datetime.datetime.now().strftime('%Y-%m-%dT%H:%M:%S+08:00')
        s_changefreq = ET.SubElement(s_url, 'changefreq')
        s_changefreq.text = 'always'
        s_priority = ET.SubElement(s_url, 'priority')
        s_priority.text = '0.95'

    # Generate the XML string
    sitemap_str = ET.tostring(root, encoding='unicode')

    # Save to file
    with open(spfile, "w", encoding="utf-8") as f:
        f.write(sitemap_str)

    # Close the database connection
    cursor.close()
    cnx.close()

with open('sites.list', 'r') as files:
    for tmp in files:
        site = tmp.strip().split()[0]
        spfile = sdir + site + '/' + tmp.strip().split()[1]
        n1 = tmp.strip().split()[2]
        n2 = tmp.strip().split()[3]
        print(site, spfile)
        mselect(site, n1, n2)

The content of sites.list is:

1
2

www.xxxxx.com 100 120
www.xxxxx.com 200 300

Summary

Writing a Python script is quite convenient. However, if performance is a concern, Golang might be a better choice.

Go Study Notes - Detailed Explanation of Pointers

wzdushu@gmail.com (wanzi) — Mon, 22 May 2023 18:22:42 +0800

What is a Pointer?

In the Go language, a pointer is a variable that stores the memory address of another variable. Therefore, a pointer variable points to the memory address of another variable, not the variable itself.

When declaring a pointer variable, you need to add * before the variable name, indicating that this is a pointer variable, for example:

`1`	`var p *int`

This indicates that a pointer named p has been declared, which points to an integer variable. The & operator can be used to get the address of a variable, for example:

1
2

x := 10
p := &x

This means getting the address of the variable x and assigning it to the pointer variable p.

When using a pointer to access a variable, you need to use the * operator, which is the dereference operator, for example:

1
2
3

x := 10
p := &x
fmt.Println(*p)

This means printing the value of the variable pointed to by the pointer variable p, which is the value of the variable x.

In addition to declaring pointer variables and obtaining the addresses of variables, you can also use the new function to create pointer variables. For example, creating a pointer to an integer variable:

`1`	`p := new(int)`

This will create a new integer variable, return its address, and then assign the address to the pointer variable p.

Pointers can also be used for function parameters and return values to share data between function calls.

When using pointers, be careful because if a pointer points to an invalid memory address, the program may crash or exhibit unpredictable behavior. Therefore, when using pointers, ensure that the memory address pointed to by the pointer is valid.

Use Cases for Pointers

Passing Parameters via Pointer Functions

Pointers can be used to pass parameters to functions. When a function needs to modify the value of an actual parameter, the address of the actual parameter can be passed as a formal parameter to the function, and the value of the actual parameter can be modified through the pointer. For example:

func modify(s *string) {
    *s = "hello, world"
}

func main() {
    s := "hello"
    modify(&s)
    fmt.Println(s) // Output: hello, world
}

Accessing Struct Fields via Pointers

Pointers can more conveniently access fields in a struct, especially when the struct is large, passing a pointer is more efficient than passing the entire struct. For example:

type Person struct {
    Name string
    Age  int
}

func main() {
    p := &Person{
        Name: "Alice",
        Age:  18,
    }
    p.Name = "Bob"
    fmt.Println(p) // Output: &{Bob 18}
}

Dynamically Allocating Memory via Pointers

Pointers can dynamically allocate memory at runtime, such as using the new() function to create a new variable and return its address. For example:

func main() {
    p := new(int)
    *p = 42
    fmt.Println(*p) // Output: 42
}

Passing Variadic Arguments

In Golang, you can use pointers to pass variadic arguments. This is because when using variadic arguments, what is passed is a slice (slice), and a slice is essentially a pointer to an array. This operation can avoid copying large amounts of data.

For example, consider the following function, which accepts variadic arguments and prints them out:

func printArgs(args ...int) {
    for _, arg := range args {
        fmt.Println(arg)
    }
}

Now, if you want to pass a slice as an argument to another function, you can use a pointer:

func printArgsPtr(args *[]int) {
    for _, arg := range *args {
        fmt.Println(arg)
    }
}

func main() {
    args := []int{1, 2, 3}
    printArgsPtr(&args)
}

In this example, we define a new function printArgsPtr that uses a pointer as a parameter to receive a slice. Inside the function, we use *args to dereference the pointer and get the actual slice, and then iterate over it to print each element. In the main function, we create a slice and pass its address to the printArgsPtr function.

It’s important to note that when using pointers to pass variadic arguments, if the slice is empty, you should not pass a nil pointer, but rather an empty slice. This is because in Golang, an empty slice and a nil pointer have different meanings, which I will explain later.

Besides the above aspects, pointers are also very useful in some specific scenarios, such as handling data structures, where pointers can more efficiently manipulate linked lists, trees, and other data structures. However, it’s important to note that overusing pointers can make code difficult to maintain, so they should be used with caution.

Common Nil Data Types

In the Go language, there are several nil data types, including:

Nil pointer (nil pointer): A pointer that points to a non-existent address, generally represented by nil.
Nil slice (nil slice): A slice with a length of 0, generally represented by nil.
Nil map (nil map): A map with a length of 0, generally represented by nil.
Nil interface (nil interface): An interface with both type and value as nil, generally represented by nil.
Nil channel (nil channel): A channel that has not been allocated storage space, generally represented by nil.

nil represents a null pointer, i.e., a pointer variable that does not point to any address, and can represent the zero value of pointer, interface, map, channel, and function types, but cannot be used to represent the zero value of other types such as int, float, bool, etc.

These nil data types are defined as the zero values of their respective types, where the zero values of pointers, slices, maps, interfaces, and channels are all nil, indicating that the default values of these types are nil, i.e., they do not point to any actual data. Therefore, when using these data types, they can be initialized to their zero values, or explicitly assigned to nil.

For example:

var s []int = nil // Define a nil slice
var p *int = nil  // Define a nil pointer
var m map[string]int = nil // Define a nil map
var i interface{} = nil // Define a nil interface
var c chan int = nil // Define a nil channel

Note the following points:

nil is not a keyword, but a predefined identifier. In Go, nil can be assigned to any type of pointer, slice, map, function, and channel variables.
Nil slices and nil maps need to be initialized before use, otherwise, a panic exception will occur.
Nil channels need to be initialized with the make function before they can be used.
Nil pointers can be used directly because their value is nil.
Nil interfaces can be used directly without initialization. Because a nil interface is essentially a variable of type interface{}, which can accept any type of value, including nil. So, when using a nil interface, if no explicit value is assigned, it is a nil interface and can be used directly.

Avoiding Nil Pointer Exceptions

When using pointer-type variables, perform a nil check to avoid nil pointer exceptions.

For example:

Integer Pointer Type

1
2

var p *int
fmt.Println(p == nil) // Outputs true

In the code, p is a variable of integer pointer type, and since it has not been initialized, its value is nil. You can check if it is a nil pointer by comparing p to nil.

Integer Slice Type

1
2

var s []int
fmt.Println(s == nil) // Outputs true

In the code, s is a variable of integer slice type, and since it has not been initialized, its value is nil. You can check if it is a nil slice by comparing s to nil.

Map Type

1
2

var m map[string]int
fmt.Println(m == nil) // Outputs true

In the code, m is a variable of map type, and since it has not been initialized, its value is nil. You can check if it is a nil map by comparing m to nil.

Function Type

1
2

var f func(int) int
fmt.Println(f == nil) // Outputs true

In the code, f is a variable of function type, and since it has not been initialized, its value is nil. You can check if it is a nil function by comparing f to nil.

Integer Channel Type

1
2

var ch chan int
fmt.Println(ch == nil) // Outputs true

In the code, ch is a variable of integer channel type, and since it has not been initialized, its value is nil. You can check if it is a nil channel by comparing ch to nil.

Finally, it is particularly important to note that empty arrays, empty slices, and empty maps can use the len() function to get their length, and the result will be 0. However, nil pointers, nil interfaces, and nil channels cannot use the len() function to get their length, otherwise, a runtime error will occur.

This is because, in Go, the len() function returns the length of a value, which is pre-defined and usually calculated by the compiler. The length of empty slices, empty arrays, and empty maps is 0 because they already have a block of memory, so the len() function can be used to get their length.

For nil interfaces and nil channels, which do not store data, their length is meaningless, so using the len() function will cause a runtime error. If you need to determine whether they are nil, you should use the nil value for the check.

Go Study Notes - Review of Go Fundamentals

wzdushu@gmail.com (wanzi) — Fri, 17 Mar 2023 18:22:42 +0800

I. Basic Knowledge

1. Basic Data Types

Value Types:

bool (1 byte)
byte (1 byte, alias of uint8)
rune (4 bytes, alias of int32, range -2^31 to 2^31)
int (32 or 64 bits, 4 or 8 bytes)
int8 (1 byte), int16 (2 bytes), int32 (4 bytes), int64 (8 bytes)
uint (32 or 64 bits, 4 or 8 bytes)
uint8 (1 byte), uint16 (2 bytes), uint32 (4 bytes), uint64 (8 bytes)
float32 (4 bytes), float64 (8 bytes)
string     // Sequence of fixed-length characters, underlying is a byte array
complex64, complex128 (default) // Complex number type
array   // Array (fixed length)
struct  // Structure

Reference Types (Pointer Types):

1
2
3

slice  // Slice, default value is nil
map    // Map, default value is nil
chan   // Channel, default value is nil

2. Encoding & Strings

Go’s default encoding is UTF-8.

Unicode: A global character encoding standard with variable-length encoding. Multiple implementations exist, such as UTF-8 and UTF-16. Character lengths range from 1 to 4 bytes. English characters typically occupy 1 byte; Chinese characters occupy 2 bytes. Each character corresponds to a hexadecimal number. Use utf8.RuneCountInString() to get the length of a Unicode string.
UTF-8: A variable-length character encoding, one implementation of Unicode. English characters take 1 byte, Chinese characters take 3 bytes, emojis take 4 bytes.

Go has two types of characters: byte and rune.

byte: Alias of uint8, commonly used for handling ASCII characters. Represents an ASCII string. Use []byte(s) to convert string s into a byte array.
rune: Alias of int32, equivalent to int32. Typically used to represent a Unicode code point, capable of handling any character. Use []rune(s) to convert string s into a Unicode rune array.

string: A sequence of fixed-length characters, internally implemented as a byte array. Strings are immutable. When strings contain only letters and digits, they can be converted to []byte or []rune. For multi-byte encoded characters like Chinese, conversion to []rune is required. The most universal method is []rune(s).

Single quotes: Used to denote byte or rune types. Default is rune. Content inside single quotes is an ASCII character. Use len() to get the length of an ASCII string.
Double quotes: Denote a string literal that supports escape sequences but cannot span multiple lines.
Backticks: Used to create raw string literals, supporting no escape sequences and allowing multiline content. Commonly used for writing multi-line messages, HTML, regular expressions, etc.

3. Built-in Functions

append  // Append elements to arrays or slices, returns modified array/slice
close   // Primarily used to close channels
delete  // Remove key-value pair from map
panic   // Stop normal goroutine execution (used with recover for error handling)
recover // Allows program-defined recovery from panic in goroutines
real    // Return real part of complex number (complex, real, imag for creating/manipulating complex numbers)
imag    // Return imaginary part of complex number
make    // Allocate memory, return the Type itself (only valid for slice, map, channel)
new     // Allocate memory, primarily for value types like int, struct, return pointer to type
cap     // Return maximum capacity of a type (only valid for slice and map)
copy    // Copy elements from one slice to another, return number of copied elements
len     // Get length (e.g., string, array, slice, map, channel)
print, println // Low-level print functions

II. Common Placeholders:

Placeholder  Description                            Example                     Output
%v         Default format of the value           Printf("%v", people)        {zhangsan}
%+v        Include field names when printing structs  Printf("%+v", people)    {Name:zhangsan}
%#v        Go syntax representation of the value   Printf("#v", people)      main.Human{Name:"zhangsan"}
%T         Go syntax representation of the type    Printf("%T", people)      main.Human
%%         Literal percent sign, not a value placeholder  Printf("%%")          %
%t         Boolean placeholder (true or false)     Printf("%t", true)        true
%c         Character placeholder, corresponding Unicode code point  Printf("%c", 0x4E2D)  中
%d         Integer placeholder, decimal representation  Printf("%d", 0x12)       18
%s         String placeholder (string type or []byte)  Printf("%s", []byte("Go语言"))  Go语言
%p         Pointer placeholder, hexadecimal with prefix 0x  Printf("%p", &people)     0x4f57f0

III. Arrays, Slices, and Maps

1. Arrays

var array_variable_name [number_of_elements]type

Array length must be a constant; once defined, it cannot change. Elements within the array can be modified.

`1`	`var a [5]int`

Declares an array of length 5 with type int.

a := [8]int{0, 1, 2, 3, 4, 5, 6, 7}
b := a[3:6]
// 8 8
fmt.Println(len(a), cap(a))
// Value: [3 4 5], Length: 3, Capacity: 5
fmt.Printf("Value: %d, Length: %d, Capacity: %d", b, len(b), cap(b))

2. Slices

A flexible, variable-length sequence of elements of the same type. Slices are built on top of arrays and support automatic resizing. Slices are reference types, internally containing address, length (len), and capacity (cap).

Syntax: var slice_variable_name []type

var a []string // Declares a slice of length 0, type string
var a []string{} // Declares a slice of length 0, type string, initialized

var myslice = []string{"Beijing", "Shanghai", "Shenzhen"}

for i := 0; i < len(myslice); i++ {
   fmt.Println("for loop:", myslice[i])
}

3. Maps

An unordered data structure based on key-value pairs. In Go, maps are reference types and must be initialized using make before use.

Syntax: map[key_type]value_type

Maps have a default initial value of nil. Memory must be allocated using the make() function. Note: cap does not apply to maps—cap returns the allocated space size of a slice/array. To get the map’s capacity, use len().

userInfo := map[string]string{
   "username": "xiaoli",
   "password": "1234556",
}
for k, v := range userInfo {
   fmt.Println(k, v, len(userInfo))
}

IV. Pointers

In Go, reference types require declaration and memory allocation; otherwise, values cannot be stored. For value types, memory is automatically allocated during declaration.

Pointer address: &a
Pointer dereference: *&a
Pointer type: &a → *int
& gets address, * dereferences to value

new: Allocates memory for a type and returns a pointer. Useful for instantiating structs, returning the pointer address of the struct.
make: Only used for creating memory for chan, map, and slice. Returns the actual type, not a pointer.

V. Structs

Go does not have classes or class inheritance. Instead, it uses structs combined with interfaces to achieve higher extensibility and flexibility compared to traditional object-oriented programming.

1. Ways to Instantiate Structs

Direct assignment (var), new, and key-value initialization.

package main

import "fmt"

// Define a struct
type member struct {
	name string
	city string
	age  int
}

func main() {
	var p1 member // Direct assignment instantiation
	p1.name = "Xiaoming"
	p1.city = "Beijing"
	p1.age = 32

	var p2 = new(member) // Instantiation using new
	p2.name = "Xiaoli"
	p2.city = "Shanghai"
	p2.age = 20

	p3 := &member{
		name: "Xiao Zhang",
		city: "Shenzhen",
		age:  18,
	} // Key-value initialization
	fmt.Printf("p1=%v \n", p1) // Output value
	fmt.Printf("p1=%+v \n", p1) // Output struct with field names
	fmt.Printf("p2=%#v \n", p2) // Output struct fields with Go syntax
	fmt.Printf("p3=%#v \n", p3)
}

Output:

p1={Xiaoming Beijing 32}
p1={name:Xiaoming city:Beijing age:32}
p2=&main.member{name:"Xiaoli", city:"Shanghai", age:20}
p3=&main.member{name:"Xiao Zhang", city:"Shenzhen", age:18}

2. Struct Methods and Receivers

In Go, methods are functions associated with a specific type, called the receiver. The receiver concept is similar to this or self in other languages.

Syntax:

1
2

func (receiver_variable receiver_type) method_name(parameter_list) (return_values) {
}

package main

import "fmt"

// Define a struct
type Person struct {
   name string
   age  int
   city string
}

// Value receiver
func (p Person) printInfo() {
   fmt.Printf("Name: %s, Age: %d, City: %s\n", p.name, p.age, p.city)
}

// Pointer receiver
func (p *Person) setInfo(name string, age int, city string) {
   p.name = name
   p.age = age
   p.city = city
}

func main() {
   // Instantiate struct
   p1 := Person{
      name: "Xiaoming",
      age:  15,
      city: "Tianjin",
   }
   p1.printInfo() // Name: Xiaoming, Age: 15, City: Tianjin
   p1.setInfo("Tiantian", 18, "Beijing")
   p1.printInfo() // Name: Tiantian, Age: 18, City: Beijing
}

Automated ECS Creation with Terraform

wzdushu@gmail.com (wanzi) — Fri, 26 Feb 2021 17:22:42 +0800

Quickly Create an Alibaba Cloud ECS Instance

Specify Terraform Version

Here, we specify the Alibaba Cloud provider version and set the required Terraform version.

# mkdir aliyun-ecs-one && cd aliyun-ecs-one
# touch versions.tf
# vim versions.tf
terraform {
  required_providers {
    alicloud = {
      source  = "aliyun/alicloud"
      version = "1.115.1"
    }
  }

  required_version = ">= 0.12"
}

Configure Variables

Here we define key pairs, cloud region, ECS account, and image information.

# vim variables.tf
# Alibaba Cloud sub-account access_key
variable "alicloud_access_key" {
  default     = "LTAI4GBXXXXXXXXXXXXXXXXXXXXXX"
  description = "The Alicloud Access Key ID to launch resources. Support environment variable 'ALICLOUD_ACCESS_KEY'."
}

# Alibaba Cloud sub-account secret_key
variable "alicloud_secret_key" {
  default     = "4Z4gbl3dXXXXXXXXXXXXXXXXXXXXX"
  description = "The Alicloud Access Secret Key to launch resources. Support environment variable 'ALICLOUD_SECRET_KEY'."
}

# Alibaba Cloud region (Hangzhou in this case)
variable "region" {
  default     = "cn-hangzhou"
  description = "The Alicloud region for resources. Support environment variable 'REGION'."
}

# Available zone in Hangzhou region
variable "availability_zone" {
  description = "The available zone to launch ECS instance and other resources."
  default     = "cn-hangzhou-i"
}

# Image ID
variable "image_id" {
  default = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
}

# ECS instance type
variable "ecs_type" {
  default = "ecs.s6-c1m2.small"
}

# ECS instance password
variable "ecs_password" {
  default = "Test12345"
}

# Disk category (cloud efficiency disk here)
variable "disk_category" {
  default = "cloud_efficiency"
}

# Disk size
variable "disk_size" {
  default = "40"
}

# Internet charge type (default: PayByTraffic)
variable "internet_charge_type" {
  default = "PayByTraffic"
}

# Maximum outbound bandwidth on public network (default > 0 will automatically assign a dedicated public IP from v1.7 onwards)
variable "internet_max_bandwidth_out" {
  default = 5
}

Configure Instance Resources

Since this is for testing, we need to create VPC, VSwitch, Security Group, and Security Group Rules beforehand.

# vim main.tf
provider "alicloud" {
  region     = var.region
  access_key = var.alicloud_access_key
  secret_key = var.alicloud_secret_key
}

resource "alicloud_vpc" "vpc" {
  name       = "tf_test_foo"
  cidr_block = "10.100.0.0/16"
}

resource "alicloud_vswitch" "vsw" {
  vpc_id            = alicloud_vpc.vpc.id
  cidr_block        = "10.100.0.0/24"
  availability_zone = var.availability_zone
}

resource "alicloud_security_group" "default" {
  name   = "default"
  vpc_id = alicloud_vpc.vpc.id
}

resource "alicloud_security_group_rule" "allow_all_tcp" {
  type              = "ingress"
  ip_protocol       = "tcp"
  nic_type          = "intranet"
  policy            = "accept"
  port_range        = "1/65535"
  priority          = 1
  security_group_id = alicloud_security_group.default.id
  cidr_ip           = "0.0.0.0/0"
}

resource "alicloud_instance" "wanzi_test" {
  availability_zone = var.availability_zone
  security_groups   = alicloud_security_group.default.*.id

  instance_type        = var.ecs_type
  system_disk_category = var.disk_category
  image_id             = var.image_id
  instance_name        = "wanzi_tf001"
  vswitch_id           = alicloud_vswitch.vsw.id
  password             = var.ecs_password
}

Run plan to simulate the execution:

# terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

Create the cloud instance — this process calls Alibaba Cloud APIs and generates a local Terraform state file:

# terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

  # alicloud_security_group.default will be created
  + resource "alicloud_security_group" "default" {
      + id                  = (known after apply)
      + inner_access        = (known after apply)
      + inner_access_policy = (known after apply)
      + name                = "default"
      + security_group_type = "normal"
      + vpc_id              = (known after apply)
    }
......
......
Plan: 5 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

alicloud_vpc.vpc: Creating...
alicloud_vpc.vpc: Creation complete after 9s [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_security_group.default: Creating...
alicloud_vswitch.vsw: Creating...
alicloud_security_group.default: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_security_group_rule.allow_all_tcp: Creating...
alicloud_security_group_rule.allow_all_tcp: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_vswitch.vsw: Creation complete after 4s [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_instance.wanzi_test: Creating...
alicloud_instance.wanzi_test: Still creating... [10s elapsed]
alicloud_instance.wanzi_test: Still creating... [20s elapsed]
alicloud_instance.wanzi_test: Creation complete after 22s [id=i-bp1gt9mb9asadff9r2zr]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

After these steps, the resources have been successfully created. A .tfstate file is also generated in the current directory — this file is critical and must not be deleted. You can later use terraform show to view the created resource details.

Batch Create Multiple ECS Instances

Configure Module

Since many excellent modules are available on https://registry.terraform.io, we directly use the alibaba/ecs-instance/alicloud module.

For more information about the official ECS module, see: https://github.com/terraform-alicloud-modules/terraform-alicloud-ecs-instance

The variables.tf and versions.tf files remain as configured in Step 1. In main.tf, add the module configuration for batch creation of ECS instances:

module "tf-instances" {
  source                      = "alibaba/ecs-instance/alicloud"
  region                      = "cn-hangzhou"
  number_of_instances         = "3"
  vswitch_id                  = alicloud_vswitch.vsw.id
  group_ids                   = [alicloud_security_group.default.id]
  private_ips                 = ["10.100.0.10", "10.100.0.11", "10.100.0.12"]
  image_ids                   = ["ubuntu_18_04_64_20G_alibase_20190624.vhd"]
  instance_type               = var.ecs_type
  internet_max_bandwidth_out  = 10
  associate_public_ip_address = true
  instance_name               = "my_module_instances_"
  host_name                   = "wanzi-cluster"
  internet_charge_type        = "PayByTraffic"
  password                    = var.ecs_password
  system_disk_category        = "cloud_ssd"
  data_disks = [
    {
      disk_category = "cloud_ssd"
      disk_name     = "my_module_disk"
      disk_size     = "50"
    }
  ]
}

Note: By default, setting internet_max_bandwidth_out triggers automatic assignment of a dedicated public IP. If you don’t need this, you can omit the setting.

Batch Create Resources

➜ terraform apply
alicloud_vpc.vpc: Refreshing state... [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_vswitch.vsw: Refreshing state... [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_security_group.default: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_security_group_rule.allow_all_tcp: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_instance.wanzi_test: Refreshing state... [id=i-bp1gt9mb9asadff9r2zr]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.tf-instances.alicloud_instance.this[0] will be created
  + resource "alicloud_instance" "this" {
      + availability_zone             = (known after apply)
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + description                   = "An ECS instance came from terraform-alicloud-modules/ecs-instance"
      + dry_run                       = false
      + host_name                     = "wanzi-cluster001"
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "my_module_instances_001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_charge_type          = "PayByTraffic"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 10
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = "10.100.0.10"
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_enhancement_strategy = "Active"
      + security_groups               = [
          + "sg-bp11s5pka9pxtj6pn4xq",
        ]
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_ssd"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + tags                          = {
          + "Name" = "my_module_instances_001"
        }
      + volume_tags                   = {
          + "Name" = "my_module_instances_001"
        }
      + vswitch_id                    = "vsw-bp1wgpgz9z8y2lfsl2beo"

      + data_disks {
          + category             = "cloud_efficiency"
          + delete_with_instance = true
          + encrypted            = false
          + name                 = "TF_ECS_Disk"
          + performance_level    = (known after apply)
          + size                 = 40
        }
    }

  # module.tf-instances.alicloud_instance.this[1] will be created
  + resource "alicloud_instance" "this" {
      + availability_zone             = (known after apply)
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + description                   = "An ECS instance came from terraform-alicloud-modules/ecs-instance"
      + dry_run                       = false
      + host_name                     = "wanzi-cluster002"
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "my_module_instances_002"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_charge_type          = "PayByTraffic"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 10
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = "10.100.0.11"
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_enhancement_strategy = "Active"
      + security_groups               = [
          + "sg-bp11s5pka9pxtj6pn4xq",
        ]
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_ssd"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + tags                          = {
          + "Name" = "my_module_instances_002"
        }
      + volume_tags                   = {
          + "Name" = "my_module_instances_002"
        }
      + vswitch_id                    = "vsw-bp1wgpgz9z8y2lfsl2beo"

      + data_disks {
          + category             = "cloud_efficiency"
          + delete_with_instance = true
          + encrypted            = false
          + name                 = "TF_ECS_Disk"
          + performance_level    = (known after apply)
          + size                 = 40
        }
    }

  # module.tf-instances.alicloud_instance.this[2] will be created
  + resource "alicloud_instance" "this" {
      + availability_zone             = (known after apply)
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + description                   = "An ECS instance came from terraform-alicloud-modules/ecs-instance"
      + dry_run                       = false
      + host_name                     = "wanzi-cluster003"
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "my_module_instances_003"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_charge_type          = "PayByTraffic"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 10
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = "10.100.0.12"
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_enhancement_strategy = "Active"
      + security_groups               = [
          + "sg-bp11s5pka9pxtj6pn4xq",
        ]
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_ssd"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + tags                          = {
          + "Name" = "my_module_instances_003"
        }
      + volume_tags                   = {
          + "Name" = "my_module_instances_003"
        }
      + vswitch_id                    = "vsw-bp1wgpgz9z8y2lfsl2beo"

      + data_disks {
          + category             = "cloud_efficiency"
          + delete_with_instance = true
          + encrypted            = false
          + name                 = "TF_ECS_Disk"
          + performance_level    = (known after apply)
          + size                 = 40
        }
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.tf-instances.alicloud_instance.this[2]: Creating...
module.tf-instances.alicloud_instance.this[1]: Creating...
module.tf-instances.alicloud_instance.this[0]: Creating...
module.tf-instances.alicloud_instance.this[1]: Still creating... [10s elapsed]
module.tf-instances.alicloud_instance.this[2]: Still creating... [10s elapsed]
module.tf-instances.alicloud_instance.this[0]: Still creating... [10s elapsed]
module.tf-instances.alicloud_instance.this[1]: Still creating... [20s elapsed]
module.tf-instances.alicloud_instance.this[0]: Still creating... [20s elapsed]
module.tf-instances.alicloud_instance.this[2]: Still creating... [20s elapsed]
module.tf-instances.alicloud_instance.this[0]: Creation complete after 21s [id=i-bp1hwbo4htk8sbwxtk6o]
module.tf-instances.alicloud_instance.this[1]: Creation complete after 21s [id=i-bp17lh41gywyih0xg6we]
module.tf-instances.alicloud_instance.this[2]: Creation complete after 22s [id=i-bp11zlrl6vxeaerz4ad0]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

The batch creation of multiple ECS instances is now complete. For any future adjustments to deployed ECS resources, simply run write/plan/apply — this process may restart the Alibaba Cloud instances.

Terraform Installation and Command Reference

wzdushu@gmail.com (wanzi) — Thu, 25 Feb 2021 17:22:42 +0800

Installing Terraform

Installing on Mac

1
2

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Installing on Linux

Ubuntu installation

1
2
3

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install terraform

CentOS installation

1
2
3

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install terraform

Verifying Installation

# terraform -v
Terraform v0.14.3

Your version of Terraform is out of date! The latest version
is 0.14.7. You can update by downloading from https://www.terraform.io/downloads.html
# terraform
Usage: terraform [global options] <subcommand> [args]

The available commands for execution are listed below.
The primary workflow commands are given first, followed by
less common or more advanced commands.

Main commands:
  init          Prepare your working directory for other commands
  validate      Check whether the configuration is valid
  plan          Show changes required by the current configuration
  apply         Create or update infrastructure
  destroy       Destroy previously-created infrastructure

All other commands:
  console       Try Terraform expressions at an interactive command prompt
  fmt           Reformat your configuration in the standard style
  force-unlock  Release a stuck lock on the current workspace
  get           Install or upgrade remote Terraform modules
  graph         Generate a Graphviz graph of the steps in an operation
  import        Associate existing infrastructure with a Terraform resource
  login         Obtain and save credentials for a remote host
  logout        Remove locally-stored credentials for a remote host
  output        Show output values from your root module
  providers     Show the providers required for this configuration
  refresh       Update the state to match remote systems
  show          Show the current state or a saved plan
  state         Advanced state management
  taint         Mark a resource instance as not fully functional
  untaint       Remove the 'tainted' state from a resource instance
  version       Show the current Terraform version
  workspace     Workspace management

Global options (use these before the subcommand, if any):
  -chdir=DIR    Switch to a different working directory before executing the
                given subcommand.
  -help         Show this help output, or the help for a specified subcommand.
  -version      An alias for the "version" subcommand.

Terraform Commands for Resource Management

Initializing Resources

For a Terraform project, I created three basic files: main.tf (entry file), variables.tf (variable definitions), and versions.tf (version information).

# ls 
main.tf     variables.tf      versions.tf
# terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of aliyun/alicloud from the dependency lock file
- Using aliyun/alicloud v1.115.1 from the shared cache directory

Terraform has been successfully initialized!

Formatting Terraform Files

fmt automatically reformats .tf files in the current directory to conform to standard Terraform formatting.

# terraform fmt 
main.tf
variables.tf
versions.tf
# terraform fmt -diff  # Show differences after formatting
main.tf
--- old/main.tf
+++ new/main.tf
@@ -1,7 +1,7 @@
 provider "alicloud" {
   region     = var.region
   access_key = var.alicloud_access_key
-  secret_key =  var.alicloud_secret_key
+  secret_key = var.alicloud_secret_key
 }

 resource "alicloud_vpc" "vpc" {
@@ -12,7 +12,7 @@
 resource "alicloud_vswitch" "vsw" {
   vpc_id            = alicloud_vpc.vpc.id
   cidr_block        = "10.100.0.0/24"
-  availability_zone =  var.availability_zone
+  availability_zone = var.availability_zone
 }

 resource "alicloud_security_group" "default" {
variables.tf
--- old/variables.tf
+++ new/variables.tf
@@ -4,7 +4,7 @@
 }

 variable "alicloud_secret_key" {
-  default                     = "4Z4gbl3d9TGz9jWobv9MPwInvyH2Kf"
+  default     = "4Z4gbl3d9TGz9jWobv9MPwInvyH2Kf"
   description = "The Alicloud Access Secret Key to launch resources.  Support to environment 'ALICLOUD_SECRET_KEY'."
 }

Creating a Resource Plan

terraform plan checks whether the planned changes align with your expectations without modifying actual resources or state.

# terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

Creating Cloud Resources

terraform apply generates a creation plan and executes it, creating resources and generating a terraform.tfstate file in the current directory.

# terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.wanzi_test will be created
  + resource "alicloud_instance" "wanzi_test" {
      + availability_zone             = "cn-hangzhou-i"
      + credit_specification          = (known after apply)
      + deletion_protection           = false
      + dry_run                       = false
      + host_name                     = (known after apply)
      + id                            = (known after apply)
      + image_id                      = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
      + instance_charge_type          = "PostPaid"
      + instance_name                 = "wanzi_tf001"
      + instance_type                 = "ecs.s6-c1m2.small"
      + internet_max_bandwidth_in     = (known after apply)
      + internet_max_bandwidth_out    = 0
      + key_name                      = (known after apply)
      + password                      = (sensitive value)
      + private_ip                    = (known after apply)
      + public_ip                     = (known after apply)
      + role_name                     = (known after apply)
      + security_groups               = (known after apply)
      + spot_strategy                 = "NoSpot"
      + status                        = "Running"
      + subnet_id                     = (known after apply)
      + system_disk_category          = "cloud_efficiency"
      + system_disk_performance_level = (known after apply)
      + system_disk_size              = 40
      + volume_tags                   = (known after apply)
      + vswitch_id                    = (known after apply)
    }

  # alicloud_security_group.default will be created
  + resource "alicloud_security_group" "default" {
      + id                  = (known after apply)
      + inner_access        = (known after apply)
      + inner_access_policy = (known after apply)
      + name                = "default"
      + security_group_type = "normal"
      + vpc_id              = (known after apply)
    }
......
......
Plan: 5 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

alicloud_vpc.vpc: Creating...
alicloud_vpc.vpc: Creation complete after 9s [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_security_group.default: Creating...
alicloud_vswitch.vsw: Creating...
alicloud_security_group.default: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_security_group_rule.allow_all_tcp: Creating...
alicloud_security_group_rule.allow_all_tcp: Creation complete after 1s [id=sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_vswitch.vsw: Creation complete after 4s [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_instance.wanzi_test: Creating...
alicloud_instance.wanzi_test: Still creating... [10s elapsed]
alicloud_instance.wanzi_test: Still creating... [20s elapsed]
alicloud_instance.wanzi_test: Creation complete after 22s [id=i-bp1gt9mb9asadff9r2zr]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Viewing Created Resource Information

terraform show displays the resources created in the current project.

terraform show -json displays data in JSON format.

# terraform show
# alicloud_instance.wanzi_test:
resource "alicloud_instance" "wanzi_test" {
    availability_zone          = "cn-hangzhou-i"
    deletion_protection        = false
    dry_run                    = false
    host_name                  = "iZbp1gt9mb9asadff9r2zrZ"
    id                         = "i-bp1gt9mb9asadff9r2zr"
    image_id                   = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
    instance_charge_type       = "PostPaid"
    instance_name              = "wanzi_tf001"
    instance_type              = "ecs.s6-c1m2.small"
    internet_charge_type       = "PayByTraffic"
    internet_max_bandwidth_in  = -1
    internet_max_bandwidth_out = 0
    password                   = (sensitive value)
    private_ip                 = "10.100.0.234"
    security_groups            = [
        "sg-bp11s5pka9pxtj6pn4xq",
    ]
    spot_price_limit           = 0
    spot_strategy              = "NoSpot"
    status                     = "Running"
    subnet_id                  = "vsw-bp1wgpgz9z8y2lfsl2beo"
    system_disk_category       = "cloud_efficiency"
    system_disk_size           = 40
    volume_tags                = {}
    vswitch_id                 = "vsw-bp1wgpgz9z8y2lfsl2beo"
}

# alicloud_security_group.default:
resource "alicloud_security_group" "default" {
    id                  = "sg-bp11s5pka9pxtj6pn4xq"
    inner_access        = true
    inner_access_policy = "Accept"
    name                = "default"
    security_group_type = "normal"
    vpc_id              = "vpc-bp1kulcyygsi727aay4hd"
}

# alicloud_security_group_rule.allow_all_tcp:
resource "alicloud_security_group_rule" "allow_all_tcp" {
    cidr_ip           = "0.0.0.0/0"
    id                = "sg-bp11s5pka9pxtj6pn4xq:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1"
    ip_protocol       = "tcp"
    nic_type          = "intranet"
    policy            = "accept"
    port_range        = "1/65535"
    priority          = 1
    security_group_id = "sg-bp11s5pka9pxtj6pn4xq"
    type              = "ingress"
}

# alicloud_vpc.vpc:
resource "alicloud_vpc" "vpc" {
    cidr_block        = "10.100.0.0/16"
    id                = "vpc-bp1kulcyygsi727aay4hd"
    name              = "tf_test_foo"
    resource_group_id = "rg-acfm2ogp24u3rcy"
    route_table_id    = "vtb-bp1wy8srerq12rta02r03"
    router_id         = "vrt-bp1apvobefvhshksnnwvm"
    router_table_id   = "vtb-bp1wy8srerq12rta02r03"
}

# alicloud_vswitch.vsw:
resource "alicloud_vswitch" "vsw" {
    availability_zone = "cn-hangzhou-i"
    cidr_block        = "10.100.0.0/24"
    id                = "vsw-bp1wgpgz9z8y2lfsl2beo"
    vpc_id            = "vpc-bp1kulcyygsi727aay4hd"
}

Marking a Resource as Tainted

terraform taint marks a resource as “tainted.” When apply is run again, the tainted resource will be destroyed first and then recreated—effectively performing a delete-and-recreate operation on that specific resource.

1
2

# terraform taint alicloud_instance.wanzi_test
Resource instance alicloud_instance.wanzi_test has been marked as tainted.

Conversely, terraform untaint removes the tainted status, restoring the resource to normal state.

1
2

# terraform untaint alicloud_instance.wanzi_test
Resource instance alicloud_instance.wanzi_test has been successfully untainted.

Destroying Cloud Resources

terraform destroy destroys cloud resources based on the current configuration.

# terraform destroy

Plan: 0 to add, 0 to change, 5 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

alicloud_security_group_rule.allow_all_tcp: Destroying... [id=sg-bp10tup89oothxz8tny1:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_instance.wanzi_test: Destroying... [id=i-bp10ukz4nlr894mhebgl]
alicloud_security_group_rule.allow_all_tcp: Destruction complete after 0s
alicloud_instance.wanzi_test: Still destroying... [id=i-bp10ukz4nlr894mhebgl, 10s elapsed]
alicloud_instance.wanzi_test: Still destroying... [id=i-bp10ukz4nlr894mhebgl, 20s elapsed]
alicloud_instance.wanzi_test: Destruction complete after 28s
alicloud_security_group.default: Destroying... [id=sg-bp10tup89oothxz8tny1]
alicloud_vswitch.vsw: Destroying... [id=vsw-bp1ap7ccst3fjxnw4pnza]
alicloud_security_group.default: Destruction complete after 9s
alicloud_vswitch.vsw: Still destroying... [id=vsw-bp1ap7ccst3fjxnw4pnza, 10s elapsed]
alicloud_vswitch.vsw: Destruction complete after 20s
alicloud_vpc.vpc: Destroying... [id=vpc-bp1obwt5ded2i0zlbu052]
alicloud_vpc.vpc: Destruction complete after 3s

Destroy complete! Resources: 5 destroyed.

Importing Cloud Data into Local Project

terraform import imports cloud instances into local state using their instance ID. It generates a terraform.tfstate file in the local directory. Before importing existing data, back up both terraform.tfstate and the .terraform directory. After importing, use terraform show to view the resource in Terraform format, copy it, and further process it to obtain the .tf resource file content.

# cat yunduan.tf
resource "alicloud_instance" "test999" {
  # (resource arguments)
}
#
# terraform import alicloud_instance.test999 i-bp1etiv4002h9q27lb97
alicloud_instance.test999: Importing from ID "i-bp1etiv4002h9q27lb97"...
alicloud_instance.test999: Import prepared!
  Prepared alicloud_instance for import
alicloud_instance.test999: Refreshing state... [id=i-bp1etiv4002h9q27lb97]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
# cat terraform.tfstate
{
  "version": 4,
  "terraform_version": "0.14.3",
  "serial": 1,
  "lineage": "779fad5e-b076-8cfd-6041-f6eef8c88b8a",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "alicloud_instance",
      "name": "test999",
      "provider": "provider[\"registry.terraform.io/aliyun/alicloud\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "allocate_public_ip": null,
            "auto_release_time": "",
            "auto_renew_period": null,
            "availability_zone": "cn-hangzhou-i",
            "credit_specification": "",
            "data_disks": [],
            "deletion_protection": false,
            "description": "",
            "dry_run": null,
            "force_delete": null,
            "host_name": "iZbp1etiv4002h9q27lb97Z",
            "id": "i-bp1etiv4002h9q27lb97",
            "image_id": "ubuntu_18_04_64_20G_alibase_20190624.vhd",
            "include_data_disks": null,
            "instance_charge_type": "PostPaid",
            "instance_name": "wanzi_tf001",
            "instance_type": "ecs.s6-c1m2.small",
            "internet_charge_type": "PayByTraffic",
            "internet_max_bandwidth_in": -1,
            "internet_max_bandwidth_out": 0,
            "io_optimized": null,
            "is_outdated": null,
            "key_name": "",
            "kms_encrypted_password": null,
            "kms_encryption_context": null,
            "password": "",
            "period": null,
            "period_unit": null,
            "private_ip": "10.100.0.169",
            "public_ip": "",
            "renewal_status": null,
            "resource_group_id": "",
            "role_name": "",
            "security_enhancement_strategy": null,
            "security_groups": [
              "sg-bp14pij6g7sjmn9bz92a"
            ],
            "spot_price_limit": 0,
            "spot_strategy": "NoSpot",
            "status": "Running",
            "subnet_id": "vsw-bp1c966jdtiw1qwh2tng8",
            "system_disk_auto_snapshot_policy_id": "",
            "system_disk_category": "cloud_efficiency",
            "system_disk_description": null,
            "system_disk_name": null,
            "system_disk_performance_level": "",
            "system_disk_size": 40,
            "tags": {},
            "timeouts": {
              "create": null,
              "delete": null,
              "update": null
            },
            "user_data": "",
            "volume_tags": {},
            "vswitch_id": "vsw-bp1c966jdtiw1qwh2tng8"
          },
          "sensitive_attributes": [],
          "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDAsImRlbGV0ZSI6MTIwMDAwMDAwMDAwMCwidXBkYXRlIjo2MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjAifQ=="
        }
      ]
    }
  ]
}
# terraform show
# alicloud_instance.test999:
resource "alicloud_instance" "test999" {
    availability_zone          = "cn-hangzhou-i"
    deletion_protection        = false
    host_name                  = "iZbp1etiv4002h9q27lb97Z"
    id                         = "i-bp1etiv4002h9q27lb97"
    image_id                   = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
    instance_charge_type       = "PostPaid"
    instance_name              = "wanzi_tf001"
    instance_type              = "ecs.s6-c1m2.small"
    internet_charge_type       = "PayByTraffic"
    internet_max_bandwidth_in  = -1
    internet_max_bandwidth_out = 0
    private_ip                 = "10.100.0.169"
    security_groups            = [
        "sg-bp14pij6g7sjmn9bz92a",
    ]
    spot_price_limit           = 0
    spot_strategy              = "NoSpot"
    status                     = "Running"
    subnet_id                  = "vsw-bp1c966jdtiw1qwh2tng8"
    system_disk_category       = "cloud_efficiency"
    system_disk_size           = 40
    tags                       = {}
    volume_tags                = {}
    vswitch_id                 = "vsw-bp1c966jdtiw1qwh2tng8"

    timeouts {}
}

Generating Resource Relationship Diagrams

Each template defines resources with varying degrees of interdependencies. terraform graph generates a visual diagram of these relationships:

# terraform graph
digraph {
        compound = "true"
        newrank = "true"
        subgraph "root" {
                "[root] alicloud_instance.wanzi_test (expand)" [label = "alicloud_instance.wanzi_test", shape = "box"]
                "[root] alicloud_security_group.default (expand)" [label = "alicloud_security_group.default", shape = "box"]
                "[root] alicloud_security_group_rule.allow_all_tcp (expand)" [label = "alicloud_security_group_rule.allow_all_tcp", shape = "box"]
                "[root] alicloud_vpc.vpc (expand)" [label = "alicloud_vpc.vpc", shape = "box"]
                "[root] alicloud_vswitch.vsw (expand)" [label = "alicloud_vswitch.vsw", shape = "box"]
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" [label = "provider[\"registry.terraform.io/aliyun/alicloud\"]", shape = "diamond"]
                "[root] var.alicloud_access_key" [label = "var.alicloud_access_key", shape = "note"]
                "[root] var.alicloud_secret_key" [label = "var.alicloud_secret_key", shape = "note"]
                "[root] var.availability_zone" [label = "var.availability_zone", shape = "note"]
                "[root] var.disk_category" [label = "var.disk_category", shape = "note"]
                "[root] var.disk_size" [label = "var.disk_size", shape = "note"]
                "[root] var.ecs_password" [label = "var.ecs_password", shape = "note"]
                "[root] var.ecs_type" [label = "var.ecs_type", shape = "note"]
                "[root] var.image_id" [label = "var.image_id", shape = "note"]
                "[root] var.internet_charge_type" [label = "var.internet_charge_type", shape = "note"]
                "[root] var.internet_max_bandwidth_out" [label = "var.internet_max_bandwidth_out", shape = "note"]
                "[root] var.region" [label = "var.region", shape = "note"]
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] alicloud_security_group.default (expand)"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] alicloud_vswitch.vsw (expand)"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.disk_category"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.ecs_password"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.ecs_type"
                "[root] alicloud_instance.wanzi_test (expand)" -> "[root] var.image_id"
                "[root] alicloud_security_group.default (expand)" -> "[root] alicloud_vpc.vpc (expand)"
                "[root] alicloud_security_group_rule.allow_all_tcp (expand)" -> "[root] alicloud_security_group.default (expand)"
                "[root] alicloud_vpc.vpc (expand)" -> "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]"
                "[root] alicloud_vswitch.vsw (expand)" -> "[root] alicloud_vpc.vpc (expand)"
                "[root] alicloud_vswitch.vsw (expand)" -> "[root] var.availability_zone"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] alicloud_instance.wanzi_test (expand)"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] alicloud_security_group_rule.allow_all_tcp (expand)"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] var.disk_size"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] var.internet_charge_type"
                "[root] meta.count-boundary (EachMode fixup)" -> "[root] var.internet_max_bandwidth_out"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"] (close)" -> "[root] alicloud_instance.wanzi_test (expand)"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"] (close)" -> "[root] alicloud_security_group_rule.allow_all_tcp (expand)"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" -> "[root] var.alicloud_access_key"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" -> "[root] var.alicloud_secret_key"
                "[root] provider[\"registry.terraform.io/aliyun/alicloud\"]" -> "[root] var.region"
                "[root] root" -> "[root] meta.count-boundary (EachMode fixup)"
                "[root] root" -> "[root] provider[\"registry.terraform.io/aliyun/alicloud\"] (close)"
        }
}

This output can also be exported directly as an SVG image using terraform graph | dot -Tsvg > graph.svg (requires prior installation of Graphviz: brew install graphviz).

`1`	`terraform graph \| dot -Tsvg > ~/Downloads/graph.svg`

Viewing graph.svg reveals the full resource relationship diagram:

Terraform Commands for State Management

Listing All Resources in Current State

# terraform state list
alicloud_instance.wanzi_test
alicloud_security_group.default
alicloud_security_group_rule.allow_all_tcp
alicloud_vpc.vpc
alicloud_vswitch.vsw

Viewing Specific Resource Data

# terraform state show alicloud_vswitch.vsw
# alicloud_vswitch.vsw:
resource "alicloud_vswitch" "vsw" {
    availability_zone = "cn-hangzhou-i"
    cidr_block        = "10.100.0.0/24"
    id                = "vsw-bp1wgpgz9z8y2lfsl2beo"
    vpc_id            = "vpc-bp1kulcyygsi727aay4hd"
}

Removing a Specific Resource from State

terraform state rm <resource_type>.<resource_name> removes a resource from state but does not delete it from the cloud. The resource can later be re-imported from the cloud.

# terraform state rm alicloud_security_group.default
Removed alicloud_security_group.default
Successfully removed 1 resource instance(s).
# terraform state list
alicloud_instance.wanzi_test
alicloud_vpc.vpc
alicloud_vswitch.vsw
# terraform import alicloud_security_group.default sg-bp11s5pka9pxtj6pn4xq
alicloud_security_group.default: Importing from ID "sg-bp11s5pka9pxtj6pn4xq"...
alicloud_security_group.default: Import prepared!
  Prepared alicloud_security_group for import
alicloud_security_group.default: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Refreshing Resource State

terraform refresh updates the local state by fetching the latest data from the cloud via API calls.

# terraform refresh
alicloud_vpc.vpc: Refreshing state... [id=vpc-bp1kulcyygsi727aay4hd]
alicloud_vswitch.vsw: Refreshing state... [id=vsw-bp1wgpgz9z8y2lfsl2beo]
alicloud_security_group.default: Refreshing state... [id=sg-bp11s5pka9pxtj6pn4xq]
alicloud_instance.wanzi_test: Refreshing state... [id=i-bp1gt9mb9asadff9r2zr]

Introduction to the Automation Orchestration Tool Terraform

wzdushu@gmail.com (wanzi) — Wed, 24 Feb 2021 17:22:42 +0800

What is Terraform?

Terraform is an open-source infrastructure orchestration tool introduced by HashiCorp around 2014. It is now supported by nearly all major cloud service providers, including Alibaba Cloud, Tencent Cloud, Huawei Cloud, AWS, Azure, Baidu Cloud, and more. Many companies today build their infrastructure using Terraform.

Background: In traditional operations, launching a business required multiple preparatory steps such as hardware procurement, server rack mounting, network setup, and system installation. With the rise of cloud computing, major public cloud providers offer user-friendly graphical interfaces—users can purchase various cloud resources via a browser and quickly set up their architecture. However, as business architectures expand, the scale and variety of cloud resource procurement continue to grow. When users need to rapidly acquire large numbers of diverse cloud resources, the numerous interactive operations across cloud management consoles actually reduce procurement efficiency. For example, initializing a classic VPC network on the Alibaba Cloud console—from creating the VPC and VSwitches to setting up NAT gateways, elastic IPs, and routing configurations—can take 20 minutes or even longer. Moreover, the non-reproducible nature of manual work leads to redundant efforts when operating across regions or multi-cloud environments.

In reality, operations personnel care only about resource configuration—not the step-by-step creation process. Think of it like ordering coffee: you just tell the waiter what kind of coffee and whether to add ice. If there were a complete list of required cloud resources—clearly specifying types, specifications, quantities, and relationships between resources—you could deploy everything with one click. And when business needs change, simply updating the list would allow rapid reconfiguration of cloud resources. This concept in cloud computing is known as infrastructure orchestration. Today, many cloud platforms provide similar capabilities, such as Alibaba Cloud’s ROS, AWS CloudFormation, etc.

Infrastructure as Code (IaC):
Defining cloud resources, services, or operational steps as code within templates, then using an orchestration engine to automate resource management—this is Infrastructure as Code (IaC), the most efficient approach to resource orchestration. However, different cloud orchestration services bring high learning costs, low code reusability, and complex workflows for multi-cloud coordination. Each service typically manages only its own cloud platform, making unified management of multiple clouds and various layers (e.g., IaaS, PaaS) difficult. How can we solve this? Can we use a single orchestration tool with a unified syntax to manage multiple clouds—including Alibaba Cloud—uniformly? That’s exactly why Terraform was created—to address these challenges.

Features and Functions of Terraform

Key Features

IaC: Manage infrastructure using code.
Execution Plan: Shows actions that will be performed during terraform apply.
Resource Graph: Visualizes all resources and their relationships.
Change Automation: Based on execution plans and resource graphs, clearly identifies what changes are needed and in what order.

Summary: Terraform is used for initializing various infrastructure resources, supports multiple cloud platforms, and enables integration with third-party services.

Benefits

Uses APIs from different providers and abstracts them into standardized Terraform code.
Users don’t need to understand the intricate details of each cloud provider’s API, significantly reducing deployment complexity.

Terraform Architecture

Terraform follows a plugin-based architecture, offering strong extensibility and allowing developers to easily extend its functionality. Logically, Terraform consists of two layers: the Core Layer (Terraform Core) and the Plugin Layer (Terraform Provider).

Core Layer

The core layer is essentially the Terraform command-line tool, written in Go. It handles:

Reading .tf configuration files and performing variable substitution.
Managing resource state files.
Analyzing resource dependencies and generating dependency graphs.
Creating resources based on dependency relationships. Resources without dependencies are created in parallel (default: 10 concurrent processes)—this is why Terraform efficiently manages cloud resources.
Calling the plugin layer via RPC.

Plugin Layer

Also written in Go, Terraform has over 250 different plugins. These plugins:

Receive RPC calls from the core layer.
Execute specific services.

There are two main types of plugins:

Provider

Providers integrate with external APIs—for example, the Alibaba Cloud Provider enables creation, modification, and deletion of Alibaba Cloud resources. This plugin interacts with Alibaba Cloud’s API endpoints and provides an abstraction layer so developers can manage resources via Terraform without needing to know API details. Responsibilities include:

Initializing communication with external APIs.
Handling authentication with external services.
Defining relationships between cloud resources and external services.

Common providers:

Alibaba Cloud: https://github.com/aliyun/terraform-provider-alicloud
Baidu Cloud: https://github.com/baidubce/terraform-provider-baiducloud
Tencent Cloud: https://github.com/tencentcloudstack/terraform-provider-tencentcloud
Huawei Cloud: https://github.com/huaweicloud/terraform-provider-huaweicloud
UCloud: https://github.com/ucloud/terraform-provider-ucloud
QingCloud: https://github.com/yunify/terraform-provider-qingcloud
AWS: https://github.com/hashicorp/terraform-provider-aws
Azure: https://github.com/terraform-providers/terraform-provider-azurerm
Google Cloud: https://github.com/hashicorp/terraform-provider-google

Provisioner

Provisioners execute scripts after resource creation or deletion. For instance, the Puppet Provisioner can download, install, and configure a Puppet agent on a newly created virtual machine.

For better understanding, here’s a simplified component architecture diagram from online sources:

Below is a basic workflow diagram for daily Terraform operations:

Terraform Keywords Explained

Declarative Language (HCL)

Terraform uses HashiCorp Configuration Language (HCL), which is declarative. This means developers describe what the infrastructure should look like, and Terraform takes care of how to achieve it—without requiring knowledge of implementation steps or how Terraform interacts with cloud provider APIs. Terraform automatically downloads the necessary Providers and Provisioners to handle the details.

In contrast, imperative languages execute commands step-by-step, where order matters. Fixed inputs produce fixed outputs. While neither paradigm is inherently superior, declarative approaches are more convenient for cloud resource orchestration. All mainstream cloud orchestration tools—like AWS CloudFormation, Azure Resource Manager Templates, Google Cloud Deployment Manager—are declarative. Using Tencent Cloud APIs directly for orchestration typically results in an imperative workflow.

State File

After initialization, Terraform generates a state file that records:

Timestamp of the last operation.
Attributes of each resource.
Current values of variables.
Version of the state file.

When running Terraform again, it first synchronizes the local state file with the actual state on the cloud provider to detect any deleted or modified resources. Then, based on the .tf configuration, it determines which resources need to be created, updated, or destroyed. After the operation completes, a new state file is generated.

Backend

The integrity of the state file is critical. We must ensure:

Automatic locking at the start of an operation and release only upon completion to prevent concurrent modifications.
Tracking version changes of resources.
Access control for sensitive information in the state file.

The backend defines how Terraform reads, stores, locks, and manages the state file—and is tightly coupled with how terraform apply executes.

By default, Terraform uses a local backend, storing the state file in the current directory and executing locally. This works fine for individual users managing small-scale infrastructure. But as team size grows, multiple developers may have separate workstations, requiring a shared location for state files. In such cases, remote backends become essential.

Terraform supports several remote backends:

AWS S3
HashiCorp Consul
etcd
Terraform Cloud
Terraform Enterprise

These remote backends provide secure remote storage and state locking. Terraform Enterprise also offers remote execution of Terraform and additional enterprise-grade features.

Module

A module is a reusable piece of Terraform code designed for easy sharing. A module consists of:

Input parameters (variables).
Output parameters.
Main logic.

This structure closely resembles functions in traditional programming languages. Terraform maintains a public module registry where modules can be published if they follow the standard format. Once published, others can use them directly.

https://registry.terraform.io/

SaltStack's SaltAPI

wzdushu@gmail.com (wanzi) — Thu, 15 Feb 2018 16:22:42 +0800

Installation

`1`	`yum -y install salt-api`

Configuration

cat /etc/salt/master.d/api.conf # Configure certificate and port

rest_cherrypy:
  port: 8888
  debug: True
  ssl_crt: /etc/pki/tls/certs/localhost.crt
  ssl_key: /etc/pki/tls/private/localhost_nopass.key

cat /etc/salt/master.d/eauth.conf # Set permissions

external_auth:
  pam:
    saltapi:
      - .*
      - '@wheel'
      - '@runner'

Add User

1
2

useradd -M -s /sbin/nologin saltapi
echo "saltapi_xxxxxx" | passwd saltapi --stdin

Start Service

1
2

systemctl enable salt-api
systemctl start salt-api

Test

Test using curl to obtain token information:

curl -k https://manage-op.test.cn:8888/login -H "Accept: application/x-yaml" \
-d username='saltapi' \
-d password='saltapi_xxxxxxxx' \
-d eauth='pam'

If using Python or Go, you can wrap your own client.

A Practical Guide to Salt Commands in SaltStack

wzdushu@gmail.com (wanzi) — Sat, 10 Feb 2018 16:22:42 +0800

Common Commands

salt -N 'ceph' test.ping                     # Test connectivity
salt -E '^server10*' test.ping                # Match by regex for connectivity
salt -S 192.168.150.101 test.ping             # Match by agent IP address
salt -S 192.168.150.0/24 test.ping            # Match by IP subnet
salt -N 'ceph' cmd.run 'df -Th'              # Check disk usage by group
salt -N 'ceph' cmd.exec_code python 'import os; print os.system("df -Th")'  # Execute Python code
salt -N 'ceph' cmd.exec_code perl 'print scalar localtime'  # Execute Perl code
salt -G 'osrelease:6.3' cmd.run 'python -V'   # Filter by grains and execute
salt '*' smbios.get system-serial-number      # Get server hardware serial number

Remote Script Execution:

`1`	`salt -N 'ceph' salt://scripts/runme.sh # Execute script remotely`

Remote File Copy:

salt -N 'ceph' cp.get_file salt://files/opencdn/a.txt /tmp/a.txt     # Pull file from master to agent
salt -N 'ceph' cp.get_file salt://files/opencdn/dir1/ /tmp           # Pull directory from master to agent
salt -N 'ceph' cp.get_url http://www.baidu.com /tmp/index.html       # Download URL
salt -N 'ceph' cp.push /etc/fstab                                   # Push file from minion to master (default: /var/cache/salt/master/minions/minion-id/files)
salt -N 'ceph' cp.push_dir /etc/modprobe.d/ glob='*.conf'          # Push specific config files from directory
salt-cp '*' fstab /etc/fstab                                         # Copy local fstab file to target node
salt-cp -E 'gpu0[1-2][1-5]' fstab /etc/fstab                        # Copy via regex match
salt-cp -G 'os:CentOS*' fstab /etc/fstab                             # Copy via grains match

Disk Information

1
2
3

salt -N 'ceph' disk.percent /data01        # Get disk usage percentage
salt -N 'ceph' disk.nodeusage /data01      # Get inode usage
salt -N 'ceph' xfs.info /data01            # Get XFS partition info

File Operations

1
2
3

salt -N 'ceph' file.chown /etc/passwd root root    # Change file ownership
salt -N 'ceph' file.copy /etc/passwd /tmp/passwd   # Copy file
salt -N 'ceph' file.directory_exists /etc/passwd   # Check if file exists

User Management

1
2
3

salt -N 'ceph' user.list_users         # List all system users
salt -N 'ceph' user.info lianghaiqiang  # Show user details
salt -N 'ceph' user.delete lianghaiqiang remove=True force=True  # Force delete user

Timezone, Reboot, and Cron

salt -N 'ceph' timezone.get_zone        # Get timezone
salt -N 'ceph' timezone.set_zone        # Set timezone
salt -N 'ceph' system.reboot            # Reboot system
salt -N 'ceph' cron.list_tab root       # List root's cron jobs
salt -N 'ceph' cron.raw_cron root       # List root's cron jobs in raw text format

File Extraction & Compression

1
2
3

salt -N 'ceph' archive.cmd_unzip template=jinja /tmp/zipfile.zip /tmp/{{grains.id}}/ excludes=file_1,file_2
salt -N 'ceph' archive.cmd_unzip /tmp/zipfile.zip /home/strongbad/ excludes=file_1,file_2
salt -N 'ceph' archive.cmd_zip template=jinja /tmp/zipfile.zip /tmp/sourcefile1,/tmp/{{grains.id}}.txt

Networking

salt -N 'ceph' network.dig www.baidu.com     # DNS lookup
salt -N 'ceph' network.get_hostname          # Get hostname
salt -N 'ceph' network.hw_addr eth0          # Get MAC address
salt -N 'ceph' network.ip_addrs              # Get IP addresses
salt -N 'ceph' network.ping www.baidu.com timeout=3  # Ping with timeout
salt -N 'ceph' pkg.install zlib               # Install package

Get Help on Objects:

e.g., grains/disk/pillar/pip/pkg

salt -N 'ceph' sys.doc sys                  # Get system help
salt 'gpu054' sys.list_functions grains      # List available grains functions
salt 'gpu054' sys.doc grains                 # View grains documentation
salt 'gpu054' sys.doc disk                   # View disk module documentation

salt 'gpu054' sys.list_functions pillar       # List pillar functions
salt 'gpu054' sys.doc pillar                  # View pillar documentation
salt 'gpu054' pillar.item                     # List supported pillar data sources

salt 'gpu054' sys.list_modules                # List available minion modules
salt 'gpu054' sys.list_functions pip          # List pip module functions
salt 'gpu054' sys.doc pip                     # View pip module help

salt 'gpu054' sys.list_state_modules          # List all state modules
salt 'gpu054' sys.list_state_functions pkg    # List pkg state functions
salt 'gpu054' sys.state_doc pkg               # View pkg state help

Job Management:

salt-run jobs.active                          # View active jobs on all minions (running saltutil.running)
salt-run jobs.lookup_jid <jid>                # Query results for a specific JID from master job cache
salt-run jobs.list_jobs                       # List all current jobs in master job cache
salt 'xxxx' saltutil.kill_job <jid>           # Kill a job on the master

Python Coding Standards

wzdushu@gmail.com (wanzi) — Sat, 20 Jan 2018 10:22:42 +0800

Encoding Declaration

Default encoding is not ASCII. You can specify it using:

`1`	`# coding=<encoding name>`

1
2

#!/usr/bin/python
# -*- coding: <encoding name> -*-

1
2

#!/usr/bin/python
# coding: utf-8

Code Layout

Use four spaces for indentation; avoid using tabs, and never mix tabs with spaces.
Limit line length to 79 characters. Use backslashes for line continuation, but prefer parentheses instead.
Leave two blank lines between classes and top-level functions; leave one blank line between methods within a class.
Module structure order: module description and docstring, import statements, global variables, constants, others. Within import, arrange in order: standard library, third-party, local modules. Avoid writing multiple imports on one line.
Add one space on both sides of operators. Omit spaces around assignment operators in default function arguments. Do not add spaces before closing brackets (], }, )).
Comments must be in English, preferably complete sentences. Start with a capital letter, end with a period (or other punctuation), followed by two spaces before the next sentence. For phrases, punctuation may be omitted.
Always write docstrings for public modules, functions, classes, and methods. Non-public ones don’t require docstrings, but comments are acceptable (placed on the line immediately after def).

Naming Conventions

Avoid using single lowercase letter l or uppercase O as they can easily be confused.
Module names should be short and use all lowercase letters, optionally separated by underscores.
Package names should be short and use all lowercase letters without underscores.
Class names use CapWords. Internal classes use _CapWords.
Exception names use CapWords with an Error suffix.
Global variables should ideally be limited to module scope (like static in C). Two ways to achieve this: use __all__ mechanism or prefix with an underscore.
Function names use all lowercase letters, optionally separated by underscores.
Constant names use all uppercase letters, optionally separated by underscores.
Class attributes (methods and variables) use all lowercase letters, optionally separated by underscores.
Class attributes have three scopes: public, non-public, and subclass API—similar to public, private, protected in C++. Prefix non-public attributes with a single underscore.
If a class attribute conflicts with a keyword, append an underscore. Avoid abbreviations or other workarounds.
To prevent naming conflicts with subclasses, prefix certain class attributes with double underscores. For example, in class Foo, define __a; access via Foo._Foo__a to avoid ambiguity. If a subclass also uses Foo, this protection fails.
The first parameter of instance methods must be self. The first parameter of static methods must be cls.

Code Checking Tools

`1`	`pip install flake8`

After installing flake8 successfully, open VSCode, go to File → Preferences → Settings, and add "python.linting.flake8Enabled": true in settings.json.

Auto Formatting Tool

`1`	`pip install yapf`

After installing yapf successfully, open VSCode, go to File → Preferences → Settings, and add "python.formatting.provider": "yapf" in settings.json.

Project Development Notes:

When starting a new project, keep the following in mind:

README.md: Write your project overview, quick start guide, etc. Although distutils requires no extension, GitHub renders .md files directly as HTML.
ChangeLog.txt: Store version change logs with a consistent format. Refer to web.py’s ChangeLog.txt for examples.
LICENSE.txt: Include the license used by your project. Do not create custom licenses.
requirements.txt: List dependencies on separate lines. This allows pip install to automatically install them.
setup.py: Installation script — detailed later.
docs: Store project documentation such as high-level design, detailed design, maintenance documents, and auto-generated pydoc content. Strongly recommend using Markdown format.
src: Place main module code here. Avoid placing module directories directly in the root directory. Module paths can be specified in setup.py.
tests: Store all unit tests and performance test scripts. Test files must start with test_ so that distutils will automatically include them in packaging. Run tests using:
python -m unittest discover -s ./ -p 'test_*.py' -v

References:

https://www.python.org/dev/peps/pep-0008/ https://wiki.woodpecker.org.cn/moin/PythonCodingRule