阿里云 on Wanzi's Blog – Notes on life, tech, and DevOps.

Some summaries and reflections on CDN construction

wzdushu@gmail.com (wanzi) — Wed, 07 Sep 2022 12:22:42 +0800

The ongoing COVID-19 pandemic has repeatedly disrupted daily life and work. Over the past half year, I’ve experienced a lot—family members fell seriously ill, my grandmother passed away, and there were numerous personal matters to handle, which led me to pause blogging for six months.

Amid economic downturns, many industries have begun layoffs and business scaling back. Taking advantage of recent free time, I’m summarizing my past experience with CDN services.

I. Architectural Analysis of Large-Scale Websites

In the mobile internet era, designing architectures for high-concurrency large-scale websites is far from simple. Here’s a concise overview across several key aspects:

1. Client Side:

Devices include desktop PCs, mobile phones, smart devices, etc.

Browser layers mainly consist of Chrome, Microsoft IE/Edge, Safari, Firefox, Opera, etc.—classification is primarily based on browser engines.

Protocol layer:

(1) Standard web pages: HTTP/1.1, HTTP/2, QUIC
(2) Streaming media: RTMP, HTTP-FLV, HLS (m3u8)

2. CDN Layer:

For small to medium-sized websites, commercial CDN services are usually sufficient.

For large or ultra-large websites, a hybrid approach—combining self-built and commercial CDN—is recommended. This will be discussed in detail later.

In the CDN domain, two core components dominate: caching layer and routing layer.

Depending on business scenarios, typical use cases include:

Static content acceleration
Dynamic content acceleration
Download acceleration
Audio acceleration
Video acceleration
Live streaming acceleration
P2P acceleration
Security acceleration

3. Static Services:

Frontend delivery models typically involve H5, mini-programs, Android, iOS. Core assets include HTML/CSS/JS.

Popular frontend frameworks: Vue, React, Angular.

Typically served via Nginx or OpenResty on the backend, with SSL certificates, expiration policies, CORS configurations, etc., properly set up.

4. Dynamic Services:

Based on the company’s primary programming language—commonly PHP, Java, Go, Python, etc.

Most companies now run dynamic services within Docker or Kubernetes clusters.

If true static/dynamic separation isn’t feasible and traffic must go through CDN, it’s advisable to configure origin-pull strategies to minimize routing overhead.

5. Data Caching Layer:

Common solutions: Redis, Memcached.

In cloud-native ecosystems, etcd and TiKV are also used for caching purposes.

6. Database Servers:

Traditional relational databases: MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, DB2.

NoSQL alternatives: MongoDB, HBase, Cassandra (Facebook), etc.

With increasing demands for performance and massive data handling, modern distributed databases are gaining traction:

TiDB (open-source by PingCAP)
Greenplum
DorisDB (open-source by Baidu)
ClickHouse

7. Distributed Storage:

Common choices: Ceph, HDFS, GlusterFS, GFS—selected based on specific use cases.

Currently, Ceph and HDFS dominate the market:

Ceph: Supports block storage, object storage, and file systems; widely adopted by cloud providers. It integrates well with OpenStack and Kubernetes.
HDFS: Ideal for massive data storage, often paired with Hadoop and Spark. However, its support for cloud-native environments is limited.

II. Self-Built CDN vs. Commercial CDN?

1. Choosing Between CDN Options

Deciding between self-building a CDN or using a commercial one depends on your company’s scale, cost constraints, and operational needs.

For general static file acceleration, if traffic is under 10M PV/month, avoid self-building—commercial CDNs suffice. Prioritize vendors offering low cost and strong performance.
For large file downloads, if your bandwidth is constrained, don’t build your own CDN—stick with commercial options.
For audio and on-demand video acceleration, if your R&D team lacks expertise, opt for commercial CDNs with wide node coverage, low pricing, and proven results.
For live streaming, where user experience is critical, prioritize CDN providers with excellent network quality across their edge nodes—preferably major cloud-based CDN vendors.
For dynamic acceleration and security acceleration, choose vendors with robust origin-pull paths and mature security mechanisms.

2. How to Build Your Own CDN?

Hardware: Select servers with high cost-performance ratios—focus on disk, memory, and CPU specs.
Data Center Bandwidth: Partner with telecom agents. Prefer data centers with:
- High outbound capacity
- Excellent network links
- Strong DDoS protection
Cache Nodes: Choose from Nginx, Apache Traffic Server, Varnish, Squid.
Routing System: Use third-party intelligent CDN solutions. If you have strong R&D capabilities, consider Bind View or Bind DLZ.
High Availability & Load Balancing: Options include LVS, Nginx, Keepalived, HAProxy.

3. How to Choose a Commercial CDN?

Domestic CDN Providers: Alibaba Cloud, Tencent Cloud, NetEase Cloud, Huawei Cloud, Baidu Cloud, Kingsoft Cloud, White Mountain Cloud, Qiniu Cloud, Blue Network, UpYun, Dalian, UCloud, QingCloud, etc.

International CDN Providers: AWS CloudFront, Akamai, Google Cloud CDN, Fastly, Limelight, Cloudflare, StackPath, CDNetworks, etc.

When selecting a CDN provider—domestic or international—consider these metrics:

Metric	Evaluation Criteria
(1) Node Distribution	More nodes closer to users → better acceleration
(2) First Screen Time	Shorter loading time → better UX
(3) Node Latency	Lower latency → better network path
(4) Download Speed	Faster download → strong server + network performance
(5) Packet Loss Rate	Low loss → stable end-user connectivity
(6) Origin Pull Rate	Lower ratio of origin requests/traffic → better offloading
(7) Hit Rate	Higher hit rate → more cached content → better performance
(8) Platform Features	Rich functionality, complete features, good UI/UX
(9) Support Responsiveness	How quickly the operations team responds and resolves issues
(10) Routing Strategy	Evaluate whether the vendor’s routing logic is rational and effective

III. Summary

In recent years, with the rise of cloud computing, China’s CDN industry has undergone significant consolidation. Former giants like NetEase Cloud and Blue Network have been largely overtaken by Alibaba Cloud and Tencent Cloud. Once-prominent Dalian’s business has declined, Kuaiwang was acquired by CenturyLink, White Mountain Cloud has pivoted toward edge computing, while mid-tier players like Kingsoft Cloud, UCloud, Qiniu Cloud, UpYun are gradually capturing market share.

In conclusion, the future direction of the CDN industry remains uncertain. But current trends show all major vendors aggressively investing in edge computing. With advancements in 5G, AI, and containerization technologies, CDN is no longer just a standalone technology—it must continuously integrate with other innovations and ultimately serve real-world applications in our daily lives.

Implementing Internal DNS with Alibaba Cloud PrivateZone + Bind9 + Dnsmasq

wzdushu@gmail.com (wanzi) — Sat, 10 Jul 2021 17:22:42 +0800

Requirements:

Alibaba Cloud cluster can resolve internal domain names

Office network resolves internal domain names + internet access resolution

Solution:

For the first requirement, directly use Alibaba Cloud PrivateZone for resolution.
For the second requirement, configure internal domain zones in PrivateZone, then synchronize them to the office network’s bind9 server using Alibaba Cloud’s synchronization tool. Use Dnsmasq as the DNS entry point for the office network: forward public queries to public DNS servers, and forward internal domain queries to the bind9 server.

Some may wonder: Why not use bind9 alone to handle all internal resolutions? The main reason is that in practice, bind9 exhibits performance issues when forwarding to multiple DNS servers simultaneously—occasional timeouts occur. In contrast, Dnsmasq handles this scenario significantly better.

1. Configure Alibaba Cloud PrivateZone

Reference: https://help.aliyun.com/document_detail/64627.html

2. Synchronize Alibaba Cloud Zones to bind9

1. Deploy bind9 via docker-compose

version: '2'

services:
  bind:
    restart: always
    image: sameersbn/bind:9.16.1-20200524
    environment:
    - ROOT_PASSWORD=DNS2021#
    - WEBMIN_ENABLED=true
    - WEBMIN_INIT_SSL_ENABLED=false
    ports:
    - "15353:53/tcp"
    - "15353:53/udp"
    - "11953:953/tcp"
    - "10000:10000/tcp" # webmin management
    volumes:
    - ./data:/data
    networks:
      - bind9

networks:
  bind9:
    ipam:
      config:
      - subnet: 10.220.0.0/16
        gateway: 10.220.0.1

2. Modify bind configuration files

data/bind/etc/named.conf

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";
key "rndc-key" {
    algorithm hmac-sha256;
    secret "tREasaE2Jal1GfwfL5iii3a88eRGKWui41l5h3v89OM=";
};
controls {
	inet 127.0.0.1 port 953 allow { 127.0.0.1; } keys { rndc-key; };
	};
logging {
    channel query_log {
        file "query.log" versions 10 size 50M;
        severity info;
        print-category yes;
        print-severity yes;
        print-time yes;
    };
    category queries {
        query_log;
    };
};

data/bind/etc/named.conf.options

options {
	directory "/var/cache/bind";
    dnssec-validation no;
    dnssec-enable no;
    recursion yes;
    allow-recursion { any;};
    allow-transfer { any; };
    allow-query-cache { any; };
    listen-on-v6 { any; };
    listen-on port 53 { any; };
    forward first;
    forwarders {
        192.168.1.211 port 53;
        192.168.1.212 port 53;
        };
    transfer-format many-answers;
    transfers-per-ns 500;
    recursive-clients 100000;
    max-transfer-time-in 5;
    transfers-in 300;
    transfers-out 300;
    querylog yes;
};

data/bind/etc/named.conf.local

zone "sd.com" {
	type master;
	file "/etc/bind/zones/sd.com.zone";
    allow-update { 127.0.0.1; };
	};
zone "bgt.sdi" {
	type master;
	file "/etc/bind/zones/bgt.sdi.zone";
    allow-update { 127.0.0.1; };
	};
zone "con.sdi" {
	type master;
	file "/etc/bind/zones/con.sdi.zone";
    allow-update { 127.0.0.1; };
	};

3. Synchronize zone configurations from Alibaba Cloud to bind9

Reference: https://help.aliyun.com/document_detail/102718.html

Create a shell script and schedule it via cron for batch execution.

`1`	`/5 * * * /bin/bash /opt/bind9/update.sh`

update.sh

#!/bin/bash
cd /opt/bind9/tools
./Zone_file_sync config.json
chown 101.101 /opt/bind9/data/bind/etc/zones/*
docker exec bind9_bind_1 bash -c "rndc -c /etc/bind/rndc.conf freeze;rndc -c /etc/bind/rndc.conf reload;rndc -c /etc/bind/rndc.conf thaw"

tools/config.json

{
  "accessKeyId": "LIDD5ssssmGExzGsdfsY6sJrSqo",
  "accessKeySecret": "C5N1TTESt74KhSTsswSSSWiz2",
  "zone": [
    {
      "zoneName": "sd.com",
      "zoneId": "2a4dc4e0sdsfa5d36a3b88ab6482saf",
      "filePath": "/opt/bind9/data/bind/etc/zones/sd.com.zone"
    },
    {
      "zoneName": "bgt.sdi",
      "zoneId": "f842ca07ccsd6f35d9e294d55a0c900",
      "filePath": "/opt/bind9/data/bind/etc/zones/bgt.sdi.zone"
    },
    {
      "zoneName": "con.sdi",
      "zoneId": "beb4d911addsf2bd86425ds280e7bbf2",
      "filePath": "/opt/bind9/data/bind/etc/zones/con.sdi.zone"
    }
  ]
}

3. Deploy dnsmasq

1. Install and configure dnsmasq

# yum -y install dnsmasq
# cat >> /etc/dnsmasq.conf << Tag
port=53
proxy-dnssec
no-hosts  # do not load local /etc/hosts
no-negcache
dns-forward-max=2000
server=114.114.114.114 # upstream DNS server
server=223.5.5.5 # upstream DNS server
server=/sd.com/127.0.0.1#15353  # forward to specific DNS on port
server=/bgt.sdi/127.0.0.1#15353
server=/con.sdi/127.0.0.1#15353
log-queries  # log DNS queries
log-facility=/var/log/dnsmasq/dnsmasq.log # specify log path
log-async=50
cache-size=100000
Tag
# systemctl start dnsmasq
# systemctl enable dnsmasq

2. Configure log rotation policy for dnsmasq:

# cat >> /etc/logrotate.d/dnsmasq Tag
/var/log/dnsmasq/dnsmasq.log {
    notifempty
    daily
    dateext
    rotate 15
    sharedscripts
    postrotate
        [ ! -f /var/run/dnsmasq.pid ] || kill -USR2 `cat /var/run/dnsmasq.pid`
    endscript
}
Tag

Test execution:

`1`	`logrotate -vf /etc/logrotate.conf`

4. Adding a New Zone: What Operations Are Needed?

Both bind9 and dnsmasq require updates.

Example: Add test.com

Add to /etc/dnsmasq.conf:
```
server=/test.com/127.0.0.1#15353
```

Add to /opt/bind9/data/bind/etc/named.conf.local:

zone "test.com" {
    type master;
    file "/etc/bind/zones/test.com.zone";
    allow-update { 127.0.0.1; };
    forwarders {};
};

5. How to Integrate Internal DNS into K8s Cluster

Update /etc/resolv.conf on cluster nodes to use internal DNS:

1
2
3

options timeout:1 attempts:1 rotate
nameserver 192.168.1.211
nameserver 192.168.1.212

Configure CoreDNS to forward unresolved queries:

Corefile: |
  .:53 {
      errors
      health
      kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods insecure
         upstream
         fallthrough in-addr.arpa ip6.arpa
      }
      prometheus :9153
      forward . /etc/resolv.conf
      cache 30
      loop
      reload
      loadbalance
  }

With these steps, the complete internal DNS resolution system is now implemented.