Linux on Wanzi's Blog – Notes on life, tech, and DevOps.

Self-hosted Enterprise-grade VPN Network with Headscale

wzdushu@gmail.com (wanzi) — Sat, 11 Jan 2025 10:00:00 +0800

Project Background

Business Pain Points

With business growth, there are currently over a dozen independent private network environments (ZStack/TStack) overseas, for example:

Overseas Private Network Region A
Overseas Private Network Region B
Overseas Private Network Region C

Each network is an independent entity requiring different VPN clients to dial in for resource access, posing significant challenges for daily O&M management:

O&M personnel need to frequently switch between multiple VPN clients.
Lack of unified permission management and access control.
Complex network configuration and difficult troubleshooting.
Inability to achieve cross-network automated O&M.

Solution

Adopting the Headscale (Server) + Tailscale (Client) architecture to build an enterprise-grade Zero Trust VPN network:

Deploy the Headscale control server on the Alibaba Cloud Singapore node.
Install Tailscale clients on private network nodes and register them with Headscale.
Achieve one-time connection to access all private network resources.
Support ACL access control and SSO.
Tailscale has excellent multi-platform support: iOS, macOS, Android, Windows, Linux, and supports IPv6.

Headscale Architecture Overview

Core Components

Headscale is an open-source implementation of the Tailscale control server, fully compatible with the Tailscale protocol:

Control Server (Headscale): Manages node registration, key distribution, address allocation, and ACL policies.
DERP Server: Relay server for traffic forwarding when NAT traversal fails.
Client (Tailscale): Deployed on nodes, establishes encrypted tunnels based on WireGuard.

Technical Advantages

Compared to traditional VPN solutions:

Zero Trust Architecture: Deny all access by default, explicitly authorize via ACL.
NAT Traversal: Supports STUN protocol for automatic hole punching, no public IP required.
Mesh Network: Direct communication between nodes without passing through a central server.
Auto Reconnection: Automatically re-establishes connections upon network changes.
Cross-platform Support: Linux/Windows/macOS/BSD/iOS/Android/FreeBSD.

Headscale Server Deployment

Environment Preparation

# System Requirements
# Ubuntu 20.04+ / CentOS 8+ / Debian 11+
# Public IP Server (Alibaba Cloud Singapore recommended)
# Open UDP ports: 3478 (DERP), 41641 (Headscale)

# Install Headscale
wget https://github.com/juanfont/headscale/releases/download/v0.23.0/headscale_0.23.0_linux_amd64.deb
sudo dpkg -i headscale_0.23.0_linux_amd64.deb

# Or install using yum (RHEL/CentOS)
sudo yum install -y https://github.com/juanfont/headscale/releases/download/v0.23.0/headscale_0.23.0_linux_amd64.rpm

Configuration File

Reference: https://github.com/juanfont/headscale/blob/main/config-example.yaml

Create configuration file /etc/headscale/config.yaml:

# Server URL
server_url: https://headscale.wnote.com:443

# Listen address
listen_addr: 0.0.0.0:8080

# Private key (auto-generated or specified)
private_key_path: /var/lib/headscale/private.key

# DERP relay server configuration 
derp:
  # Enable embedded DERP server
  server:
    enabled: false
    region_id: 999                     # Use an unused ID to avoid conflicts
    region_code: "self-derp"           # Short identifier
    region_name: "Self-hosted Co-located DERP"        # Descriptive name
    stun_listen_addr: "0.0.0.0:3478"   # STUN service listening, must open UDP 3478
    # [Important] Enter your server's actual public IPv4 address
    ipv4: "YOUR_SERVER_PUBLIC_IP"
    # If you have public IPv6, you can also fill it in
    # ipv6: "YOUR_SERVER_PUBLIC_IPV6"
    # Automatically add this region to DERP map
    automatically_add_embedded_derp_region: true
    # [Security] Strongly recommended to enable, only allow your Tailnet clients to use this relay
    verify_clients: true

  # You can choose to keep the official DERP as a backup, or disable it completely (clear urls)
  urls:
    - https://controlplane.tailscale.com/derpmap/default   # Keep backup (recommended)
  # If you have additional self-hosted DERP nodes (e.g., Singapore node), keep paths
  paths:
    - /etc/headscale/derp.yaml

  auto_update_enabled: true
  update_frequency: 24h

# Database (Default SQLite, PostgreSQL recommended for production)
database:
  type: sqlite
  sqlite:
    path: /var/lib/headscale/db.sqlite

# ACL policy file
acl_policy_path: /etc/headscale/acl.yaml

# DNS configuration
dns:
  nameservers:
    - 1.1.1.1
    - 8.8.8.8
  magic_dns: true  # Enable MagicDNS (node.namespace.vpn)

# OAuth configuration (optional)
# oidc:
#   issuer: "https://your-oidc-provider.com"
#   client_id: "your-client-id"
#   client_secret: "your-client-secret"

# Log configuration
log:
  format: text
  level: info

Configuring the DERP Relay Server

First, let’s understand the DERP concept:

The DERP server is primarily a reliable, low-latency backup relay solution to ensure devices can always connect when network conditions are poor (e.g., symmetric NAT, strict firewalls) preventing direct connections.

Direct Connection (P2P): Headscale’s ultimate goal is to let clients establish peer-to-peer direct connections via the STUN protocol “hole punching”, offering the fastest speed.
Relay (DERP): When hole punching fails (e.g., one party is behind strict symmetric NAT), traffic is forwarded via the DERP server. Although latency is higher than direct connection, it ensures uninterrupted connection.

If you want to self-host DERP and finely control multiple DERP nodes, you need to create a configuration file to define your private DERP network regions and node information.

The benefit of self-hosting DERP is that you can avoid the latency issues of official DERP servers overseas and keep relay traffic on your own servers, improving security and stability.

I don’t need this for now, just an example. Create /etc/headscale/derp.yaml:

# /etc/headscale/derp.yaml
regions:
  # First self-hosted region, e.g., East China
  901:
    regionid: 901
    regioncode: "cn-east"   # Region code, short
    regionname: "East China Self-hosted Node" # Region name
    nodes:
      - name: 901a
        regionid: 901
        # Enter your independent DERP server domain name here
        hostname: derp-shanghai.yourdomain.com
        # Your DERP server port, usually 443 or custom
        derpport: 12345
        # STUN port, usually 3478
        stunport: 3478
        stunonly: false

  # Second self-hosted region, e.g., North China
  902:
    regionid: 902
    regioncode: "cn-north"
    regionname: "North China Self-hosted Node"
    nodes:
      - name: 902a
        regionid: 902
        hostname: derp-beijing.yourdomain.com
        derpport: 443  # Assume this node uses standard port 443
        stunport: 3478
        stunonly: false

After creating the configuration, you can view the DERP map and connectivity on any Tailscale client:

tailscale debug derp-map
tailscale debug derp headscale # Test connectivity with a specific DERP server

To verify if the client is actually using the self-hosted relay, you can check the connection status via tailscale status. If you see derp=self-derp (or your custom region code), it means traffic is going through your self-hosted relay.

ACL Access Control Policy

Create /etc/headscale/acl.yaml:

# User/Group definitions
groups:
  group:admin:
    - admin@example.com
  group:devops:
    - devops1@example.com
    - devops2@example.com
  group:developer:
    - dev1@example.com
    - dev2@example.com

# Host tag definitions
hosts:
  sg-cloud: "10.100.0.1/32"
  network-a: "10.100.1.0/24"
  network-b: "10.100.2.0/24"
  network-c: "10.100.3.0/24"

# Access control rules
acls:
  # Admins can access all resources
  - action: accept
    src:
      - group:admin
    dst:
      - "*:*"

  # Ops team can access all servers via SSH
  - action: accept
    src:
      - group:devops
    dst:
      - "sg-cloud:22"
      - "network-a:22"
      - "network-b:22"
      - "network-c:22"

  # Dev team can access application ports
  - action: accept
    src:
      - group:developer
    dst:
      - "sg-cloud:80,443,8080"
      - "network-a:3306,6379"
      - "network-b:9200,9300"

  # Deny all other access by default
  - action: reject
    src:
      - "*"
    dst:
      - "*:*"

# Tag routing rules (optional)
tagOwners:
  tag:prod:
    - group:admin
    - group:devops
  tag:database:
    - group:admin

Starting the Service

# Create necessary directories
sudo mkdir -p /var/lib/headscale

# Start Headscale
sudo systemctl enable headscale
sudo systemctl start headscale
sudo systemctl status headscale

# View logs
sudo journalctl -u headscale -f

Configuring Nginx Reverse Proxy

Since we previously applied for a wildcard domain certificate via acme, we can directly specify the SSL certificate path. Configure according to actual scenarios in the enterprise:

server {
    listen 443 ssl http2;
    server_name headscale.wnote.com;

    ssl_certificate /etc/ssl/certs/headscale.crt;
    ssl_certificate_key /etc/ssl/private/headscale.key;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
        # Long connection timeout settings
        proxy_read_timeout 86400s;
        proxy_send_timeout 86400s;
        proxy_connect_timeout 300s;
    }
    # DERP relay path also needs proxying (embedded DERP serves via this path)
    location /derp {
        proxy_pass http://127.0.0.1:8080/derp;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Opening Firewall Ports

Ensure the server firewall allows the following traffic:

Port	Protocol	Purpose	Description
443	TCP	HTTPS (Control Plane + DERP Relay)	Opened via reverse proxy
3478	UDP	STUN Service	Must open UDP 3478 for NAT traversal detection

If using cloud service providers (like AWS, Alibaba Cloud), you also need to add inbound rules in the security group.

Tailscale Client Deployment

Installing Tailscale Client

Install on each private network node:

# Debian/Ubuntu
curl -fsSL https://tailscale.com/install.sh | sh

# RHEL/CentOS
curl -fsSL https://tailscale.com/install.sh | sh

# Windows
# Download installer: https://tailscale.com/download/windows

# macOS
brew install --cask tailscale

# Android/iOS
# Search "Tailscale" in App Store to install

Registering Nodes to Headscale

# Method 1: Use Auth Key (Recommended)

# Create Auth Key on server
sudo headscale apikeys create -e 8760h -o api-key.txt

# Client registration (replace YOUR_AUTH_KEY)
sudo tailscale up --login-server=https://headscale.wnote.com \
                  --auth-key=YOUR_AUTH_KEY \
                  --advertise-routes=10.0.0.0/24,192.168.1.0/24 \
                  --advertise-exit-node

# Method 2: Register via command line
# 1. Create user on server
sudo headscale users create user@example.com

# 2. Get registration command
sudo headscale namespaces create default

# 3. Execute registration on client and get URL
sudo tailscale up --login-server=https://headscale.wnote.com

# 4. Approve registration on server
sudo headscale nodes register -u user@example.com -n <NodeName>

Subnet Routing Configuration

Configure subnet routing on the gateway nodes of each private network:

# Enable IP forwarding
echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv6.conf.all.forwarding=1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Advertise subnet routes (execute on gateway node)
sudo tailscale up --advertise-routes=10.100.1.0/24,10.100.2.0/24

# Approve routes on server
sudo headscale routes list
sudo headscale routes enable -r <RouteID>

# Verify routing
sudo tailscale status
ping 10.100.1.10  # Test cross-network access

Exit Node Configuration (Optional)

Configure an exit node to access the internet via a specific node (designated internet gateway):

# Enable on exit node
sudo tailscale up --advertise-exit-node

# Approve on server
sudo headscale nodes list
sudo headscale nodes edit --exit-node <NodeID> --enable

# Client uses exit node
sudo tailscale up --exit-node=<ExitNodeIP> --exit-node-allow-lan-access

Specify to forward traffic via and allow simultaneous access to the local LAN. If --exit-node-allow-lan-access is not added, local LAN access (like printers, router admin pages) will also go through the exit node, potentially causing inaccessibility.

Management and Maintenance

Common Management Commands

# List all nodes
sudo headscale nodes list

# View node details
sudo headscale nodes get -i <NodeID>

# Delete node
sudo headscale nodes delete -i <NodeID>

# Rename node
sudo headscale nodes rename -i <NodeID> --new-name <NewName>

# Set node tags
sudo headscale nodes tag set -i <NodeID> -t "tag:prod,database"

# View routes
sudo headscale routes list

# Enable/Disable routes
sudo headscale routes enable -r <RouteID>
sudo headscale routes disable -r <RouteID>

# View namespaces
sudo headscale namespaces list

# Create namespace
sudo headscale namespaces create <NamespaceName>

# Add user
sudo headscale users create user@example.com

# List API keys
sudo headscale apikeys list

Monitoring and Alerting

# Enable Prometheus metrics
# Add in config.yaml:
metrics:
  listen_addr: 127.0.0.1:9090

# Access metrics
curl http://localhost:9090/metrics

# Grafana Dashboard
# Import Headscale official Dashboard: https://grafana.com/grafana/dashboards/18820

Backup and Recovery

# Backup database
sudo cp /var/lib/headscale/db.sqlite /backup/headscale-$(date +%Y%m%d).sqlite

# Backup configuration files
sudo tar -czf /backup/headscale-config-$(date +%Y%m%d).tar.gz /etc/headscale/

# Recovery
sudo systemctl stop headscale
sudo cp /backup/headscale-xxx.sqlite /var/lib/headscale/db.sqlite
sudo systemctl start headscale

Production Environment Best Practices

High Availability Deployment

# Use PostgreSQL as database backend
database:
  type: postgres
  postgres:
    host: pg-cluster.example.com
    port: 5432
    user: headscale
    password: "your-password"
    database: headscale
    sslmode: require

Security Hardening

Use mTLS Authentication

tls:
  letsencrypt:
    email: admin@example.com
    agree_tos: true
    listen_host: ":443"

Restrict IP Access

ip_prefixes:
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16

Regularly Rotate Keys

1
2
3

# Regenerate private key
sudo rm /var/lib/headscale/private.key
sudo systemctl restart headscale

Performance Optimization

Deploy Multiple DERP Servers
- Deploy a DERP node in each major region.
- Use CDN to accelerate DERP access.
Optimize Keep-alive

1
2

# Client configuration
sudo tailscale up --ping=20s  # Heartbeat interval

Single Point of Failure Risk If official DERP is completely disabled (urls: []) and there is only one self-hosted DERP server, all clients requiring relay will be unable to communicate if that server goes down or has network issues. For production, it is recommended to keep at least the official DERP as a backup or deploy multiple self-hosted DERP nodes.
Certificate Requirements Embedded DERP relies on Headscale’s HTTPS service, so your domain headscale.wnote.com must have a valid SSL certificate.

Troubleshooting

Common Issues

1. Node Cannot Register

# Check server logs
sudo journalctl -u headscale -n 50

# Check if client registration URL is correct
# Ensure firewall allows ports 8080/443

2. Cannot Access Subnet Routes

# Check if IP forwarding is enabled
sysctl net.ipv4.ip_forward

# Check if routes are approved
sudo headscale routes list

# Packet capture analysis
sudo tcpdump -i tailscale0 -n host 10.100.x.x

3. DERP Connection Failed

# Test DERP connectivity
tailscale ping --c 100 <DERPServerIP>

# Check DERP server status
curl https://your-derp-server.com/derp/latency-check

4. ACL Policy Not Taking Effect

# Validate ACL syntax
sudo headscale acl validate

# View effective ACL
sudo headscale acl get

Summary

The Headscale + Tailscale solution provides an out-of-the-box Zero Trust VPN solution for enterprises:

Simple Deployment: One-click server installation, automatic client configuration.
Powerful Features: Supports ACL, SSO, routing, and exit nodes.
Excellent Performance: Based on WireGuard, low latency, high throughput.
Secure and Reliable: End-to-end encryption, fine-grained permission control.

By deploying the Headscale control server on Alibaba Cloud Singapore and unifying access to various private networks, we achieved:

O&M personnel access all networks with a single login.
ACL-based fine-grained access control.
Automated network topology management.
Auditable operation logs.

This solution has been running stably in production with 10+ nodes, 100+ daily active connections, and a failure rate of <0.1%.

Some summaries and reflections on CDN construction

wzdushu@gmail.com (wanzi) — Wed, 07 Sep 2022 12:22:42 +0800

The ongoing COVID-19 pandemic has repeatedly disrupted daily life and work. Over the past half year, I’ve experienced a lot—family members fell seriously ill, my grandmother passed away, and there were numerous personal matters to handle, which led me to pause blogging for six months.

Amid economic downturns, many industries have begun layoffs and business scaling back. Taking advantage of recent free time, I’m summarizing my past experience with CDN services.

I. Architectural Analysis of Large-Scale Websites

In the mobile internet era, designing architectures for high-concurrency large-scale websites is far from simple. Here’s a concise overview across several key aspects:

1. Client Side:

Devices include desktop PCs, mobile phones, smart devices, etc.

Browser layers mainly consist of Chrome, Microsoft IE/Edge, Safari, Firefox, Opera, etc.—classification is primarily based on browser engines.

Protocol layer:

(1) Standard web pages: HTTP/1.1, HTTP/2, QUIC
(2) Streaming media: RTMP, HTTP-FLV, HLS (m3u8)

2. CDN Layer:

For small to medium-sized websites, commercial CDN services are usually sufficient.

For large or ultra-large websites, a hybrid approach—combining self-built and commercial CDN—is recommended. This will be discussed in detail later.

In the CDN domain, two core components dominate: caching layer and routing layer.

Depending on business scenarios, typical use cases include:

Static content acceleration
Dynamic content acceleration
Download acceleration
Audio acceleration
Video acceleration
Live streaming acceleration
P2P acceleration
Security acceleration

3. Static Services:

Frontend delivery models typically involve H5, mini-programs, Android, iOS. Core assets include HTML/CSS/JS.

Popular frontend frameworks: Vue, React, Angular.

Typically served via Nginx or OpenResty on the backend, with SSL certificates, expiration policies, CORS configurations, etc., properly set up.

4. Dynamic Services:

Based on the company’s primary programming language—commonly PHP, Java, Go, Python, etc.

Most companies now run dynamic services within Docker or Kubernetes clusters.

If true static/dynamic separation isn’t feasible and traffic must go through CDN, it’s advisable to configure origin-pull strategies to minimize routing overhead.

5. Data Caching Layer:

Common solutions: Redis, Memcached.

In cloud-native ecosystems, etcd and TiKV are also used for caching purposes.

6. Database Servers:

Traditional relational databases: MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, DB2.

NoSQL alternatives: MongoDB, HBase, Cassandra (Facebook), etc.

With increasing demands for performance and massive data handling, modern distributed databases are gaining traction:

TiDB (open-source by PingCAP)
Greenplum
DorisDB (open-source by Baidu)
ClickHouse

7. Distributed Storage:

Common choices: Ceph, HDFS, GlusterFS, GFS—selected based on specific use cases.

Currently, Ceph and HDFS dominate the market:

Ceph: Supports block storage, object storage, and file systems; widely adopted by cloud providers. It integrates well with OpenStack and Kubernetes.
HDFS: Ideal for massive data storage, often paired with Hadoop and Spark. However, its support for cloud-native environments is limited.

II. Self-Built CDN vs. Commercial CDN?

1. Choosing Between CDN Options

Deciding between self-building a CDN or using a commercial one depends on your company’s scale, cost constraints, and operational needs.

For general static file acceleration, if traffic is under 10M PV/month, avoid self-building—commercial CDNs suffice. Prioritize vendors offering low cost and strong performance.
For large file downloads, if your bandwidth is constrained, don’t build your own CDN—stick with commercial options.
For audio and on-demand video acceleration, if your R&D team lacks expertise, opt for commercial CDNs with wide node coverage, low pricing, and proven results.
For live streaming, where user experience is critical, prioritize CDN providers with excellent network quality across their edge nodes—preferably major cloud-based CDN vendors.
For dynamic acceleration and security acceleration, choose vendors with robust origin-pull paths and mature security mechanisms.

2. How to Build Your Own CDN?

Hardware: Select servers with high cost-performance ratios—focus on disk, memory, and CPU specs.
Data Center Bandwidth: Partner with telecom agents. Prefer data centers with:
- High outbound capacity
- Excellent network links
- Strong DDoS protection
Cache Nodes: Choose from Nginx, Apache Traffic Server, Varnish, Squid.
Routing System: Use third-party intelligent CDN solutions. If you have strong R&D capabilities, consider Bind View or Bind DLZ.
High Availability & Load Balancing: Options include LVS, Nginx, Keepalived, HAProxy.

3. How to Choose a Commercial CDN?

Domestic CDN Providers: Alibaba Cloud, Tencent Cloud, NetEase Cloud, Huawei Cloud, Baidu Cloud, Kingsoft Cloud, White Mountain Cloud, Qiniu Cloud, Blue Network, UpYun, Dalian, UCloud, QingCloud, etc.

International CDN Providers: AWS CloudFront, Akamai, Google Cloud CDN, Fastly, Limelight, Cloudflare, StackPath, CDNetworks, etc.

When selecting a CDN provider—domestic or international—consider these metrics:

Metric	Evaluation Criteria
(1) Node Distribution	More nodes closer to users → better acceleration
(2) First Screen Time	Shorter loading time → better UX
(3) Node Latency	Lower latency → better network path
(4) Download Speed	Faster download → strong server + network performance
(5) Packet Loss Rate	Low loss → stable end-user connectivity
(6) Origin Pull Rate	Lower ratio of origin requests/traffic → better offloading
(7) Hit Rate	Higher hit rate → more cached content → better performance
(8) Platform Features	Rich functionality, complete features, good UI/UX
(9) Support Responsiveness	How quickly the operations team responds and resolves issues
(10) Routing Strategy	Evaluate whether the vendor’s routing logic is rational and effective

III. Summary

In recent years, with the rise of cloud computing, China’s CDN industry has undergone significant consolidation. Former giants like NetEase Cloud and Blue Network have been largely overtaken by Alibaba Cloud and Tencent Cloud. Once-prominent Dalian’s business has declined, Kuaiwang was acquired by CenturyLink, White Mountain Cloud has pivoted toward edge computing, while mid-tier players like Kingsoft Cloud, UCloud, Qiniu Cloud, UpYun are gradually capturing market share.

In conclusion, the future direction of the CDN industry remains uncertain. But current trends show all major vendors aggressively investing in edge computing. With advancements in 5G, AI, and containerization technologies, CDN is no longer just a standalone technology—it must continuously integrate with other innovations and ultimately serve real-world applications in our daily lives.

Implementing Internal DNS with Alibaba Cloud PrivateZone + Bind9 + Dnsmasq

wzdushu@gmail.com (wanzi) — Sat, 10 Jul 2021 17:22:42 +0800

Requirements:

Alibaba Cloud cluster can resolve internal domain names

Office network resolves internal domain names + internet access resolution

Solution:

For the first requirement, directly use Alibaba Cloud PrivateZone for resolution.
For the second requirement, configure internal domain zones in PrivateZone, then synchronize them to the office network’s bind9 server using Alibaba Cloud’s synchronization tool. Use Dnsmasq as the DNS entry point for the office network: forward public queries to public DNS servers, and forward internal domain queries to the bind9 server.

Some may wonder: Why not use bind9 alone to handle all internal resolutions? The main reason is that in practice, bind9 exhibits performance issues when forwarding to multiple DNS servers simultaneously—occasional timeouts occur. In contrast, Dnsmasq handles this scenario significantly better.

1. Configure Alibaba Cloud PrivateZone

Reference: https://help.aliyun.com/document_detail/64627.html

2. Synchronize Alibaba Cloud Zones to bind9

1. Deploy bind9 via docker-compose

version: '2'

services:
  bind:
    restart: always
    image: sameersbn/bind:9.16.1-20200524
    environment:
    - ROOT_PASSWORD=DNS2021#
    - WEBMIN_ENABLED=true
    - WEBMIN_INIT_SSL_ENABLED=false
    ports:
    - "15353:53/tcp"
    - "15353:53/udp"
    - "11953:953/tcp"
    - "10000:10000/tcp" # webmin management
    volumes:
    - ./data:/data
    networks:
      - bind9

networks:
  bind9:
    ipam:
      config:
      - subnet: 10.220.0.0/16
        gateway: 10.220.0.1

2. Modify bind configuration files

data/bind/etc/named.conf

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";
key "rndc-key" {
    algorithm hmac-sha256;
    secret "tREasaE2Jal1GfwfL5iii3a88eRGKWui41l5h3v89OM=";
};
controls {
	inet 127.0.0.1 port 953 allow { 127.0.0.1; } keys { rndc-key; };
	};
logging {
    channel query_log {
        file "query.log" versions 10 size 50M;
        severity info;
        print-category yes;
        print-severity yes;
        print-time yes;
    };
    category queries {
        query_log;
    };
};

data/bind/etc/named.conf.options

options {
	directory "/var/cache/bind";
    dnssec-validation no;
    dnssec-enable no;
    recursion yes;
    allow-recursion { any;};
    allow-transfer { any; };
    allow-query-cache { any; };
    listen-on-v6 { any; };
    listen-on port 53 { any; };
    forward first;
    forwarders {
        192.168.1.211 port 53;
        192.168.1.212 port 53;
        };
    transfer-format many-answers;
    transfers-per-ns 500;
    recursive-clients 100000;
    max-transfer-time-in 5;
    transfers-in 300;
    transfers-out 300;
    querylog yes;
};

data/bind/etc/named.conf.local

zone "sd.com" {
	type master;
	file "/etc/bind/zones/sd.com.zone";
    allow-update { 127.0.0.1; };
	};
zone "bgt.sdi" {
	type master;
	file "/etc/bind/zones/bgt.sdi.zone";
    allow-update { 127.0.0.1; };
	};
zone "con.sdi" {
	type master;
	file "/etc/bind/zones/con.sdi.zone";
    allow-update { 127.0.0.1; };
	};

3. Synchronize zone configurations from Alibaba Cloud to bind9

Reference: https://help.aliyun.com/document_detail/102718.html

Create a shell script and schedule it via cron for batch execution.

`1`	`/5 * * * /bin/bash /opt/bind9/update.sh`

update.sh

#!/bin/bash
cd /opt/bind9/tools
./Zone_file_sync config.json
chown 101.101 /opt/bind9/data/bind/etc/zones/*
docker exec bind9_bind_1 bash -c "rndc -c /etc/bind/rndc.conf freeze;rndc -c /etc/bind/rndc.conf reload;rndc -c /etc/bind/rndc.conf thaw"

tools/config.json

{
  "accessKeyId": "LIDD5ssssmGExzGsdfsY6sJrSqo",
  "accessKeySecret": "C5N1TTESt74KhSTsswSSSWiz2",
  "zone": [
    {
      "zoneName": "sd.com",
      "zoneId": "2a4dc4e0sdsfa5d36a3b88ab6482saf",
      "filePath": "/opt/bind9/data/bind/etc/zones/sd.com.zone"
    },
    {
      "zoneName": "bgt.sdi",
      "zoneId": "f842ca07ccsd6f35d9e294d55a0c900",
      "filePath": "/opt/bind9/data/bind/etc/zones/bgt.sdi.zone"
    },
    {
      "zoneName": "con.sdi",
      "zoneId": "beb4d911addsf2bd86425ds280e7bbf2",
      "filePath": "/opt/bind9/data/bind/etc/zones/con.sdi.zone"
    }
  ]
}

3. Deploy dnsmasq

1. Install and configure dnsmasq

# yum -y install dnsmasq
# cat >> /etc/dnsmasq.conf << Tag
port=53
proxy-dnssec
no-hosts  # do not load local /etc/hosts
no-negcache
dns-forward-max=2000
server=114.114.114.114 # upstream DNS server
server=223.5.5.5 # upstream DNS server
server=/sd.com/127.0.0.1#15353  # forward to specific DNS on port
server=/bgt.sdi/127.0.0.1#15353
server=/con.sdi/127.0.0.1#15353
log-queries  # log DNS queries
log-facility=/var/log/dnsmasq/dnsmasq.log # specify log path
log-async=50
cache-size=100000
Tag
# systemctl start dnsmasq
# systemctl enable dnsmasq

2. Configure log rotation policy for dnsmasq:

# cat >> /etc/logrotate.d/dnsmasq Tag
/var/log/dnsmasq/dnsmasq.log {
    notifempty
    daily
    dateext
    rotate 15
    sharedscripts
    postrotate
        [ ! -f /var/run/dnsmasq.pid ] || kill -USR2 `cat /var/run/dnsmasq.pid`
    endscript
}
Tag

Test execution:

`1`	`logrotate -vf /etc/logrotate.conf`

4. Adding a New Zone: What Operations Are Needed?

Both bind9 and dnsmasq require updates.

Example: Add test.com

Add to /etc/dnsmasq.conf:
```
server=/test.com/127.0.0.1#15353
```

Add to /opt/bind9/data/bind/etc/named.conf.local:

zone "test.com" {
    type master;
    file "/etc/bind/zones/test.com.zone";
    allow-update { 127.0.0.1; };
    forwarders {};
};

5. How to Integrate Internal DNS into K8s Cluster

Update /etc/resolv.conf on cluster nodes to use internal DNS:

1
2
3

options timeout:1 attempts:1 rotate
nameserver 192.168.1.211
nameserver 192.168.1.212

Configure CoreDNS to forward unresolved queries:

Corefile: |
  .:53 {
      errors
      health
      kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods insecure
         upstream
         fallthrough in-addr.arpa ip6.arpa
      }
      prometheus :9153
      forward . /etc/resolv.conf
      cache 30
      loop
      reload
      loadbalance
  }

With these steps, the complete internal DNS resolution system is now implemented.

Summary of Daily Git Commands

wzdushu@gmail.com (wanzi) — Sat, 28 Jul 2018 17:22:42 +0800

Git Global Settings:

1
2

git config --global user.name "wanzi"
git config --global user.email "iwz2099@163.com"

Git Commit Code

git clone git@github.com:iwz2099/test.git
cd test
touch README.md
git add README.md
git commit -m "add README"
git push -u origin master
# The above command pushes the local master branch to the remote origin, and -u sets the default remote branch as origin. After this, you can use `git push` without any arguments.

# Push all local branches to origin
git push -u origin --all  

# If development is based on the local `case_dev_wanzi` branch, push it to the remote `case_dev` branch
git push origin case_dev_wanzi:case_dev

Git Query and Cleanup

git status   # Check current branch status
git log      # View commit history of current branch
git log -n3  # Show last three commits
git log -p -2  # Show patch differences for each of the last two commits
git log --stat  # Show summary statistics for each commit
git log --pretty=oneline # Display each commit on a single line; options include short, full, fuller
git log --pretty=format:"%h - %an, %ar : %s"
git log --pretty=format:"%h %s" --graph  # Use ASCII art to visualize branching and merge history
git reflog # Show all operations across all branches (including commits, resets, and deleted commits)
git reflog # Show last 10 entries
git grep -n wanzi  # Search for 'wanzi' in commit history and working directory
git grep --count wanzi  # Count occurrences of 'wanzi'
git clean  # Remove untracked files not ignored by .gitignore
git clean -d  # Clean entire working directory
git clean -d -n  #-n tests cleanup without actually removing files

Git Branch Operations:

git branch # List current branches
git branch -v # Show last commit for each branch
# Create local dev branch from remote master
git checkout -b dev origin/master 

git branch --merged # List branches merged into current branch
git branch --no-merged # List branches not yet merged into current branch

# Start new feature development: create and switch to new branch
git checkout -b dev-20180720-111111-wanzi
Equivalent to:
git branch r-20180720-111111-wanzi
git checkout r-20180720-111111-wanzi

# Merge completed feature into master
git checkout master
git merge dev-20180720-111111-wanzi

Git Merge Operations

Create a new local branch issue54 from remote master, then develop on it.

1
2

git remote add origin git@github.com:iwz2099/test.git
git checkout -b issue54 origin/master

After feature completion, merge changes back to remote branch.

git fetch origin    # Fetch updates from remote
git merge origin/master  # Merge remote master into current branch
git push origin master  # Push code to remote master

git log --no-merges issue54..origin/master  # View commits in master not yet merged into issue54

Git Tag Operations

git tag   # List current tags
git tag -l 'wanzi-2018*'   # List only tags matching pattern
git tag v1.8-20180720   # Create a lightweight tag (temporary use)
git tag -a v1.8-20180720 -m 'wanzi version 1.8'   # Create annotated tag with message; stores metadata
git show v1.8-20180720  # View tag details; compare with `git tag` output
git push origin tag_name    # Push specific tag to remote
git push origin --tags      # Push all tags to remote at once
git push origin :tag_name   # Delete remote tag
git checkout -b dev-20180720-111111-wanzi v1.2.0  # Create new branch from a tag
git branch -D dev-20180720-123333-wanzi   # Force delete an unmerged branch

Git Multi-Developer Collaboration:

# Developer A creates feature1 branch from remote master
git checkout -b feature1 origin/master

# Developer B creates feature11 branch from remote master
git checkout -b feature11 origin/master

# Developer A finishes first, pushes to remote feature1
# Developer B later finishes, wants to integrate A's changes into feature1
git fetch origin
git merge origin/feature1
git push -u origin feature11:feature1

Git Unstage Files and Undo Changes

# Code added to staging area but not committed yet; revert staging
git reset HEAD .  or
git reset HEAD a.txt
# This only affects staging area; working directory remains unchanged unless further actions are taken

git checkout -- .    # Discard all changes in working directory, revert to last commit
git checkout -- a.txt # Discard changes in a.txt, revert to last commit

Git Rollback to Previous Commit

git log
git reset --hard <commit_id>  # Roll back to specific commit
git reset --hard HEAD^  # Roll back to previous commit
git reset --hard HEAD^^ # Roll back to second-to-last commit
git reset --hard HEAD~4 # Roll back to fourth-to-last commit

git push origin HEAD --force # Force push to overwrite remote history (use with caution if you have project admin rights)

Git Rollback Master to a Specific Tag Version

Create a temporary rollback branch from the release tag, then force-push it to remote master:

`1`	`git checkout -b rollback_20180720 r-20180720-111225-wanzi`

In GitLab project settings, disable protection on the master branch
Force push the rollback branch to remote master (this will reset master to the specified tag version):

`1`	`git push origin rollback_20180720:master -f`

Git Other Operations

Fix commit message before pushing

`1`	`git commit --amend --only -m 'fix: fix a bug' # Amend commit message`

Correct author name/email before pushing

`1`	`git commit --amend --author="wanzi <iwz2099@163.com>"`

Dockerfile Multi-stage Builds

wzdushu@gmail.com (wanzi) — Mon, 02 Jul 2018 16:22:42 +0800

Understanding Docker Multi-stage Builds:

Building an image requires a base image; all subsequent operations are based on this base image.
Docker image files have a layered structure. Each RUN instruction adds a new layer, so reducing the number of layers helps minimize image size.
When multiple FROM instructions are used, only the last FROM image becomes the root image of the final build.

Example of multi-stage build in my own project deployment: Here, we compile a binary using the Golang base image and then directly copy it into a minimal Alpine-based image:

FROM golang:1.12.7 as build
MAINTAINER wanzi <iwz2099@163.com>

# Compilation configuration
ARG NAME=gaia
ARG FLAGS=-tags=jsoniter
ARG GOOS=linux
ARG GOARCH=amd64
ARG PORT_TO_EXPOSE=10020

ENV GOPROXY https://mirrors.aliyun.com/goproxy/
ENV GO111MODULE on

WORKDIR /opt/gaia
COPY . .
RUN GOOS=$GOOS GOARCH=$GOARCH go build -mod vendor -ldflags="-s -w" -o $NAME $FLAGS

FROM alpine
WORKDIR /opt/gaia
COPY --from=build /opt/gaia/gaia .
RUN mkdir -p /opt/gaia/conf
VOLUME ["/opt/gaia/conf"]

CMD ["./gaia"]
EXPOSE $PORT_TO_EXPOSE

Detailed Dockerfile syntax

wzdushu@gmail.com (wanzi) — Thu, 21 Jun 2018 16:22:42 +0800

FROM

Specifies the base image used to build the Docker image. FROM must be the first non-comment instruction in a Dockerfile. If the specified image is not available locally, Docker will automatically pull it from the public registry.

Example:

`1`	`FROM ubuntu:14.04 # Inherits from ubuntu:14.04`

MAINTAINER

Specifies the maintainer information.

`1`	`MAINTAINER wanzi "iwz2099@163.com"`

ENV

Sets environment variables that are available for subsequent RUN instructions and persist at runtime within the container.

ENV <key> <value> — sets a single variable
ENV <key>=<value> ... — allows setting multiple variables at once

Example:

1
2

ENV LANG en_US.UTF-8
ENV LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

RUN

Executes commands during image build time.

RUN <command>
RUN ["executable", "param1", "param2"]

Example:

1
2

RUN yum -y install bind-utils
RUN ["/bin/bash", "-c", "yum -y install bind-utils"]

The first form runs the command in a shell (/bin/sh -c), while the second uses exec execution. Use the second form to specify a different shell or interpreter.

Each RUN instruction executes a command on top of the current image layer, commits the result as a new layer, and subsequent RUN commands operate on the resulting image. Docker images are layered—any historical commit point can be used to create a new image, similar to version control in source code. Long commands can be split across lines using \.

COPY

Copies files or directories from the host filesystem into the container.

COPY <src> <dest>

Example:

`1`	`COPY script/ /build/script/`

Copies <src> (a relative path from the Dockerfile’s location) to <dest> inside the container. When copying local directories, prefer COPY over ADD.

ADD

Copies files or directories from the host filesystem into the container.

ADD <src> <dest>

Example:

1
2

ADD https://www.baidu.com/index.html   /var/www/html/
ADD ubuntu-xenial-core-cloudimg-amd64-root.tar.gz /

The <src> can be:

A relative path from the Dockerfile’s directory,
A URL (automatically downloaded and copied),
A local tar file (automatically extracted into a directory).

VOLUME

Declares a mount point for data volumes.

1
2

VOLUME [ "/data" ]
VOLUME [ "/var/lib/redis", "/var/log/redis" ]

Creates a mountable directory that can be mounted from the host or other containers. Typically used for databases or persistent data.

USER

Sets the user ID (UID) under which the container runs. This affects the user context for RUN commands.

USER <uid>

CMD

Supports three formats:

CMD ["executable","param1","param2"] — executes via exec, recommended.
CMD command param1 param2 — runs in /bin/sh, suitable for interactive applications.
CMD ["param1","param2"] — provides default parameters for ENTRYPOINT.

Specifies the default command to run when starting a container. Only one CMD instruction is allowed per Dockerfile; if multiple are defined, only the last one takes effect.

WORKDIR

Sets the working directory for RUN, CMD, and ENTRYPOINT instructions.

Example:

`1`	`WORKDIR /opt/nodeapp`

ONBUILD

Configures instructions to be executed when the built image is used as a base for another image.

ONBUILD [INSTRUCTION]

Example: A Dockerfile creating image-A contains:

1
2

ONBUILD ADD . /app/src
ONBUILD RUN /usr/local/bin/python-build --dir /app/src

When a new image is built using FROM image-A, these ONBUILD instructions are automatically added:

FROM image-A
# Automatically runs:
ADD . /app/src
RUN /usr/local/bin/python-build --dir /app/src

Images with ONBUILD are recommended to include this in their tag (e.g., ruby:1.9-onbuild).

ENTRYPOINT

Defines the default executable to run when the container starts. Unlike CMD, it cannot be overridden by command-line arguments unless explicitly changed via docker run --entrypoint.

Only one ENTRYPOINT instruction is allowed per Dockerfile; later ones override earlier ones.

Supported formats:

ENTRYPOINT [ "nodejs", "server.js" ]
ENTRYPOINT command param1 param2 — runs in shell

EXPOSE

Informs Docker that the container listens on a specific port at runtime.

EXPOSE <port>

When starting the container, use -P to let Docker assign a random port mapping automatically.

Example:

`1`	`EXPOSE 3000`

Notes & Best Practices

Keep the image minimal—avoid installing unnecessary packages.
Create a .dockerignore file to exclude unwanted files/directories (syntax similar to .gitignore).
Prefer official Docker Hub images as base layers to reduce image size.
Fix the initial instructions in the Dockerfile—avoid frequent changes to leverage caching effectively.
Use \ to join multiple RUN commands for better readability and maintainability.
Prefer COPY over ADD whenever possible.
Use the -t flag when building images for easier management.
Do not map public ports directly in the Dockerfile.
Always test the image locally before pushing it to a registry.

Docker Basic Commands

wzdushu@gmail.com (wanzi) — Wed, 20 Jun 2018 16:22:42 +0800

Common Commands

docker info     # View local Docker information
docker search openresty   # Search remote image repository
docker images      # View images in the local image repository
docker ps          # View currently running containers
docker pull centos  # Pull an image from the remote repository (default tag is 'latest' if not specified)
docker container run -p 8000:80 --rm -t -i centos:latest /bin/bash  # Map container port 80 to host port 8000; --rm removes the container after termination (useful for temporary debugging); -t allocates a pseudo-terminal and binds it to the container's stdin; -i keeps the container's stdin open
docker container exec -i -t b5d7bad57561 /bin/bash # Enter a running container
docker rmi $(docker images -f "dangling=true" -q) # Batch remove dangling (unnamed) images

Importing and Exporting Images

1
2
3

docker save -o centos7.tar centos  # Export image to tar file
docker load < centos7.tar   # Import image (via standard input, including original metadata such as tags)
docker load --input centos7.tar   # Load image from tar file (non-standard input method)

Importing and Exporting Containers

docker ps   # Default: show currently running containers
docker ps -a   # Show all containers, including stopped ones
docker ps -l   # Show the most recently created container (including stopped ones)

docker export 5a80afa126ba > centos7.5a80afa126ba.tar  # Export container snapshot
docker import 5a80afa126ba > centos7.5a80afa126ba.tar  # Import container snapshot into image repository

Note: Users can use docker load to import image files into the local image library, or docker import to import container snapshots. The key difference is that container snapshot files discard all historical records and metadata (preserving only the state at the time of the snapshot), while image storage files preserve full history and are larger in size. Additionally, when importing from a container snapshot, you can reassign metadata like tags.

Removing Images or Containers

docker rmi jdeathe/centos-ssh # Remove local image
or
docker image rm jdeathe/centos-ssh # Remove local image
docker rm dbd4d83097f5    # Remove local container
or
docker container rm dbd4d83097f5 # Remove local container

Stopping and Restarting Containers

1
2

docker container kill dbd4d83097f5  # Stop a running container
docker container restart dbd4d83097f5  # Restart a container

Building Images

1
2

docker image build .      # Build image from Dockerfile in current directory
docker image build -t cm-navigation:v0.0.1 .  # Build image from Dockerfile in current directory and assign a tag