Wanzi's Blog – Notes on life, tech, and DevOps.

笔记

wzdushu@gmail.com (wanzi) — Sun, 26 Nov 2023 15:00:52 +0800

Go语言学习手册 Golang新手学习手册, 涵盖Go常用数据类型、基本语法、函数、封装、继承、多态、并发编程、错误处理等等。
DevOps 面向运维开发、运维，涵盖Git、Jenkins、Gitlab、Github、Gerrit、Argo、ArgoCD、Kubernetes Operator、ChatBot等
Kubernetes 面向容器运维，主要讲解k8s部署、k8s基础、k8s常用控制器和资源清单、配置管理、RBAC、监控告警、Istio、Containerd等

About

wzdushu@gmail.com (wanzi) — Mon, 20 Aug 2018 21:38:52 +0800

About Me

I am Wanzi Shuo, a DevOps veteran navigating the internet industry. My tech career began with automation for CDN, live streaming, and VOD operations. I also experienced the first wave of AI, building AI compute clusters and DevOps platforms from 0 to 1, ensuring the stability of model training and inference.

Currently, I work in DevOps at a global internet finance company. On one hand, I implement financial-grade high availability, multi-active disaster recovery, and compliance standards into actual project infrastructure. On the other hand, I am advancing AI-empowered DevOps to make operations processes smarter, while using FinOps for effective cost control.

In my spare time, I enjoy reading and blogging, and I love meeting like-minded friends through writing—“Connecting through blogs” is a way of connection I believe in.

If you have any questions about this site, feel free to email me: wzdushu#gmail.com (replace # with @).

About This Site

Domain: wnote.com
“w” is taken from the first letter of the Pinyin for “Wanzi”, and “wnote” stands for “Wanzi You Ji” (Wanzi has notes)—recording thoughts and preserving memories.
Comment System: Powered by Giscus, implemented via GitHub Discussions. It is lightweight, open-source, and requires no additional database.
SSL Certificate: Automatically applied for via the ACME protocol from ZeroSSL’s free certificates to ensure secure access.

Wanzi’s Daily Life

Primary OS: Mac & Linux
I prefer Mac for its portability and Unix environment, allowing me to run commands and write code anytime, anywhere, maximizing efficiency.
Editor Stack:
- Vim: A sharp tool for text processing on Linux, fast as lightning;
- VS Code: Top choice for frontend development and Python script debugging;
- GoLand: Focused on Golang backend project development.
Programming Language Proficiency (high to low):
Shell → Python → Golang → HTML/CSS/JavaScript
Keyboard: Cherry G80-3000STKL (Bilibili Co-branded Edition)
Smooth touch, silent and non-intrusive, making typing feel like flowing water.

Purpose of the Site

Solidify Technical Knowledge
Technology changes rapidly, but memory fades. Writing helps organize thoughts and solidify experience; it serves as both a review and a roadmap for future reference.
Share Value
During the process of learning and practice, I have experienced confusion and gained growth. I hope my experience and insights can provide some inspiration or help to others—this is the simplest way to give back to the community.
Connect Through Blogs
In this fast-paced era, text remains a sincere form of dialogue. If you resonate with an article, feel free to leave a comment or send an email; I would love to exchange ideas with you.

wnote.com — Wanzi has notes, memories to record.

Developing Existing Projects with OpenSpec

wzdushu@gmail.com (wanzi) — Mon, 30 Mar 2026 22:10:42 +0800

Recently, I have been using OpenCode in conjunction with OpenSpec for secondary development on existing projects. I encountered some pitfalls and recorded them here:

1. Initialize Project:

First, run the CLI command in the project root directory to create the basic directory structure required by OpenSpec.

# openspec init 
Note: OpenSpec collects anonymous usage stats. Opt out: OPENSPEC_TELEMETRY=0
Detected tool directories: OpenCode (pre-selected for first-time setup)
✔ Select tools to set up (24 available) OpenCode
▌ OpenSpec structure created
✔ Setup complete for OpenCode

OpenSpec Setup Complete

Created: OpenCode
4 skills and 4 commands in .opencode/
Config: openspec/config.yaml (schema: spec-driven)

Getting started:
  Start your first change: /opsx:propose "your idea"

Learn more: https://github.com/Fission-AI/OpenSpec
Feedback:   https://github.com/Fission-AI/OpenSpec/issues

Restart your IDE for slash commands to take effect.

Additionally, to understand OpenSpec more deeply, it is recommended to enable all OpenSpec extension features for direct use in OpenCode or Claude Code later. You need to restart OpenCode or Claude Code after modification:

# openspec config profile

Current profile settings
  Delivery: both
  Workflows: 11 selected (custom)
  Delivery = where workflows are installed (skills, commands, or both)
  Workflows = which actions are available (propose, explore, apply, etc.)

✔ What do you want to configure? Delivery and workflows
✔ Delivery mode (how workflows are installed): Both (skills + commands) [current]
? Select workflows to make available:
❯[x] Propose change
 [x] Explore ideas
 [x] New change
 [x] Continue change
 [x] Apply tasks
 [x] Fast-forward
 [x] Sync specs
 [x] Archive change
 [x] Bulk archive
 [x] Verify change
 [x] Onboard

Create proposal, design, and tasks from a request
Space to toggle, Enter to confirm

2. Use Explore to Analyze Existing Code

This is the most critical step. You need to use the /opsx:explore command to guide the AI in analyzing your codebase and generating an initial specification.

How to use: Enter in the AI chat box:

/opsx:explore Analyze the architecture and core business logic of the current project, and generate corresponding specification documents

Function: The AI will scan your code files, understand the existing tech stack, directory structure, and functional modules, and then attempt to generate an initial proposal.md or specs document for you, serving as a baseline for subsequent development.

3. Complete Configuration

We all know that the core philosophy of OpenSpec is “define rules first, act later,” solving the “context loss” and “code chaos” issues AI often encounters in large projects through Spec-Driven Development (SDD).

Therefore, to help the AI understand your project more accurately, you can manually edit the OpenSpec “constitution” file openspec/config.yaml.

Content: Add your tech stack (e.g., React, Python, Go), coding conventions, domain knowledge, etc. This helps the AI generate code that better fits your project style.

Here, I directly asked the agent in OpenCode to output it:

Add the project's tech stack, coding conventions, domain knowledge, etc., to the openspec/config.yaml file to facilitate more accurate project understanding by OpenSpec in the future.

4. Start Incremental Development

After completing the above steps, you can proceed with development following the standard OpenSpec workflow:

Create change: Use /opsx:ff <feature_name> (fast mode) or /opsx:new <feature_name> (full mode) to start developing a new feature.
Define incremental spec: Describe this change (add/modify/delete) in the generated specs/ file.
Execute and archive: Use /opsx:apply to generate code. After verification, use /opsx:archive to archive. During archiving, the incremental spec will be automatically merged into the main spec. I apologize again for the inaccurate information provided previously; I hope this process helps you integrate OpenSpec smoothly.

Example: Adding a text translation tool to my technical toolbox:

/opsx:new Add a text translation tool that calls a large model in the background. Use the minimax API interface by default, referencing https://platform.minimaxi.com/docs/guides/text-generation. It should also support calling glm models, referencing the API interface at https://docs.bigmodel.cn/cn/guide/develop/claude/introduction. The translation function should be similar to https://fanyi.baidu.com/mtpe-individual/transText#/

The final project directory structure is as follows:

openspec
├── changes
│   └── archive
│       └── 2026-03-30-add-text-translator
├── config.yaml
└── specs
    └── text-translator
        └── spec.md

5. What to do if the feature doesn’t work?

If you are verifying after the AI has generated code and find that the code doesn’t run or doesn’t meet expectations, verification fails (/opsx:verify)

Do not modify the code directly: In the OpenSpec workflow, code is the product of the specification. If the code is wrong, it is usually because the “specification” was not written clearly enough, or the AI misunderstood it.
Correct repair steps:

1. Modify the specification (spec.md):

Open openspec/changes/new feature 1/specs/.../spec.md and write your corrective opinions into it.

For example: Add “Must handle network timeout situations” to the scenario description.

2. Re-apply (`/opsx:apply`):

Tell the AI: “I have updated the specification, please re-run /opsx:apply”. The AI will regenerate the code based on the new specification, overwriting the previous incorrect code.

3. Verify again:

Run /opsx:verify again to confirm the issue is resolved.

6. How to ensure synchronization with spec.md after modifying code?

In the OpenSpec workflow, code is the product of the specification. If you manually modify the code but spec.md is not synchronized, it will lead to “documentation and code disconnection.” The next time the AI generates code based on spec.md, your manual modifications will be overwritten or cause conflicts.

To ensure synchronization between the two, please choose one of the following three strategies based on when and why you modified the code:

1. Best Strategy: Modify Documentation First, Then Generate Code (Spec-First)

This is the method most recommended by OpenSpec. If you find a problem with the code (e.g., wrong logic, incorrect parameters), do not go directly to the code files to change them.

Steps: 1.1 Open openspec/changes/.../spec.md. 1.2 Modify the corresponding requirement description or scenario. 1.3 Run /opsx:apply. 1.4 Result: The AI will regenerate the code based on the new document, automatically overwriting the old implementation. This is the most thorough way to maintain synchronization.

2. Remedial Strategy: Code Modified, Reverse Sync Documentation (Reverse Sync)

If you are used to modifying code quickly first (e.g., fixing a bug or fine-tuning styles), you must immediately “backfill” the changes into the document after modification.

Steps:
1. Save code: Ensure your code modifications are saved.
2. Run verification: Enter /opsx:verify in the AI chat box.
  - The AI will compare your new code with the old spec.md.
  - It will discover inconsistencies (e.g., “Redis expiration time in code is 600 seconds, but document says 300 seconds”).
3. Accept suggestion: Let the AI update spec.md based on the code.
4. Result: The document is updated to match your code, and the two are consistent again.

3. Forced Synchronization During Archiving (`/opsx:archive`)

When you finish feature development and are ready to archive, OpenSpec will force a check on the synchronization status.

Mechanism:
- When you run /opsx:archive, OpenSpec will merge the spec.md in the current change directory into the project’s global specification library (openspec/specs/).
- Note: If you manually changed the code but didn’t update spec.md, your code logic will never be reflected in the document after archiving. The next time someone (or AI) views the archived document, they will see outdated logic.

7. Discovering spec.md lags behind code after archiving

To avoid this situation, you need to establish a set of “defensive” work habits.

1. Build “Verify -> Archive” Muscle Memory

This is the most direct and effective daily habit. Never treat /opsx:archive as the only action after development ends.

Bad habit: Change code -> Test pass -> /opsx:archive
Good habit: Change code -> /opsx:verify -> (Sync if there are differences) -> Test pass -> /opsx:archive

Specific operation: After you manually modify the code and before archiving, you must run /opsx:verify (or ask the AI to check in the AI chat box).

Tell the AI: “I manually modified xxx.ts, please compare it with spec.md, and if there are inconsistencies, please help me update spec.md.”
Only execute the archive after the AI confirms “Code is consistent with specification”.

2. Use Git Hooks (Pre-commit Hook) for Mandatory Checks

If you are worried you might occasionally forget, you can use Git’s mechanism for mandatory interception. This is a recommended practice for team collaboration.

You can add check logic to the project’s .git/hooks/pre-commit file. Although OpenSpec does not directly provide this hook, you can write a simple script:

Script logic idea:

Detect if any code files (e.g., .ts, .java, .py) have been modified.
Detect if the corresponding documents under openspec/changes/ have also been modified.
If code changed but document didn’t -> Intercept commit and prompt: “Code changes detected but OpenSpec document not updated, please run verify or update document first!”

#!/bin/sh
# .git/hooks/pre-commit

# Check if there are code changes in the staging area
CODE_CHANGES=$(git diff --cached --name-only | grep -E "\.(ts|js|java|py)$" | wc -l)
# Check if there are Spec changes in the staging area
SPEC_CHANGES=$(git diff --cached --name-only | grep "openspec/" | wc -l)

if [ "$CODE_CHANGES" -gt 0 ] && [ "$SPEC_CHANGES" -eq 0 ]; then
  echo "⚠️  Warning: Code changes detected, but no OpenSpec document changes found!"
  echo "⚠️  Please ensure your code logic is synchronized to spec.md, or run /opsx:verify."
  echo "To force commit, use --no-verify (not recommended)"
  exit 1
fi

exit 0

Additionally, for modern projects, it is recommended to use dedicated tools to manage hooks, so the configuration can be written into the codebase and shared by everyone.

Python projects / General: Use the pre-commit framework.
- Configure .pre-commit-config.yaml to define check rules (e.g., flake8, black, detect-secrets).
- Run pre-commit install to automatically configure hooks.
Frontend/Node projects: Use Husky.
- It allows you to easily define hook logic in package.json without manually handling shell script permissions.
Golang and Rust projects:
- Use lefthook: https://github.com/evilmartians/lefthook, a universal tool with parallel execution for Go development, extremely fast.

lefthook.yml generic configuration template:

# Lefthook configuration file
# Supports multi-language mixed projects (Polyglot/Monorepo)

# Global settings
min_version: 1.4.0

# pre-commit hook: Triggered on git commit
pre-commit:
  parallel: true # Core optimization: Execute all checks in parallel, greatly improving speed
  commands:
    
    # ==========================================
    # 1. Frontend (JavaScript / TypeScript)
    # ==========================================
    frontend-lint:
      glob: "*.{js,ts,jsx,tsx,vue}"
      run: npx eslint {staged_files} --fix
      # Note: If the project has no package.json, Lefthook will automatically skip or error, depending on configuration

    frontend-format:
      glob: "*.{js,ts,jsx,tsx,vue,css,scss,json}"
      run: npx prettier --write {staged_files}

    # ==========================================
    # 2. Go (Golang)
    # ==========================================
    go-fmt:
      glob: "*.go"
      # gofmt -w will modify files directly, ensuring the staging area is updated
      run: gofmt -w {staged_files}
    
    go-lint:
      glob: "*.go"
      # Assuming you have installed golangci-lint, this is the Go community standard lint tool
      run: golangci-lint run {staged_files}

    # ==========================================
    # 3. Rust
    # ==========================================
    rust-fmt:
      glob: "*.rs"
      run: cargo fmt -- {staged_files}

    rust-check:
      glob: "*.rs"
      # Use clippy for deep checks
      run: cargo clippy -- -D warnings

    # ==========================================
    # 4. Python
    # ==========================================
    python-lint:
      glob: "*.py"
      # Using flake8 as an example here, can also be replaced with black --check or pylint
      run: flake8 {staged_files}

    python-format:
      glob: "*.py"
      # If using black for formatting
      run: black {staged_files}

    # ==========================================
    # 5. Java / Kotlin
    # ==========================================
    # Java usually relies on build tools (Maven/Gradle) or specific linters (like checkstyle)
    # Here demonstrates calling Gradle's check task, which automatically handles all Java files
    java-check:
      glob: "*.java"
      # Note: Gradle tasks are usually slow and do not support checking only some files,
      # It is recommended to run full checks only in pre-push, or use plugins like spotless that support incremental checks
      run: ./gradlew checkstyleMain checkstyleTest --quiet

    # ==========================================

3. Remedial Action

Create a new change: Name it sync-logic-with-code or fix-docs-consistency.
Let AI reverse engineer: In this new change, tell the AI: “The current xxx feature code logic has changed, but spec.md is still old. Please rewrite/update spec.md based on the current code.”
Archive this “document fix”: Run /opsx:apply (at this point the AI will generate the updated document) and /opsx:archive.

Result: Your main specification library (openspec/specs/) will be updated by this new archive, becoming consistent with the code again. Although the history archive leaves a trace of “outdated documentation,” the latest main specification is accurate.

In conclusion, while using OpenSpec for project development, always keep the core principle of OpenSpec in mind:

Documentation (Spec) is the source, code is the result. When encountering problems, fix the documentation first, then let the AI rewrite the code.

Kaku Best Practices

wzdushu@gmail.com (wanzi) — Sat, 21 Mar 2026 10:30:00 +0800

Kaku is a terminal emulator for macOS featuring built-in AI assistant capabilities, integrating various AI tools such as Claude Code, Codex, Gemini CLI, Copilot CLI, and Kimi Code. This article details the installation, configuration, and best practices for Kaku, helping developers fully leverage the potential of AI-assisted programming.

Kaku Tool Introduction

What is Kaku

Kaku is a terminal emulator for macOS. It not only provides powerful terminal features but also includes built-in AI assistant capabilities, helping developers quickly complete tasks such as command diagnosis, code explanation, and AI conversations.

Core Features:

Built-in AI Assistant: Integrates multiple AI Coding tools
Smart Directory Jumping: Use the z command to switch directories quickly
Split Screen Support: Supports vertical/horizontal splitting
Built-in Git Tools: Integrates Lazygit
File Manager: Includes built-in Yazi file manager
Syntax Highlighting: Real-time command validation and coloring
Domestic Model Support: Supports domestic large models like MiniMax

Supported AI Tools

Kaku supports a variety of mainstream AI Coding tools:

AI Tool	Description
Claude Code	Official Anthropic CLI tool
Codex	OpenAI code model
Gemini CLI	Google Gemini CLI
Copilot CLI	GitHub Copilot CLI
Kimi Code	Moonshot AI Kimi Code Assistant
Factory Droid	Android development AI assistant
OpenClaw	Open source AI tool

Supported AI Providers

Provider	Base URL	Supported Models
VivGrid (Default)	`https://api.vivgrid.com/v1`	DeepSeek-V3.2
MiniMax	`https://api.minimax.io/v1`	MiniMax-M2.7, M2.7-highspeed, M2.5, M2.5-highspeed
OpenAI	`https://api.openai.com/v1`	GPT series (auto-detected)
Custom	Manual setup	Manual setup

Kaku Installation & Configuration

Installation Methods

Method 1: Download DMG (Recommended)

Visit https://github.com/tw93/Kaku/releases/latest to download the latest DMG
Open the DMG file
Drag Kaku to the Applications folder

Method 2: Homebrew Installation

`1`	`brew install tw93/tap/kakuku`

First-time Configuration

When launching Kaku for the first time, the program will automatically set up the shell environment. Configuration commands:

Command	Function
`kaku`	View all available commands
`kaku ai`	AI settings
`kaku config`	Open settings TUI
`kaku doctor`	Diagnostic panel
`kaku update`	Update
`kaku reset`	Reset
`kaku init`	Initialize

Configuration File

The Kaku configuration file is located at: ~/.config/kaku/kaku.lua

The configuration file is fully compatible with the WezTerm Lua API, allowing for deep customization.

AI Settings Details

Configuring AI Providers

Run the following command to enter AI settings:

kaku ai

Steps:

Select a Provider
Enter API Key
Select Model

Manual Configuration

Edit the ~/.config/kaku/assistant.toml file:

1
2
3

# Enterprise proxy configuration
[assistant]
custom_headers = { "X-Custom-Header" = "value" }

MiniMax Configuration

MiniMax is an AI provider available in China. The M2.7 model is fast and suitable for daily use:

1
2

# Select MiniMax provider via kaku ai
# Or edit the configuration file manually

Supported MiniMax models:

MiniMax-M2.7 (Standard)
MiniMax-M2.7-highspeed (High-speed)
MiniMax-M2.5
MiniMax-M2.5-highspeed

Kaku Assistant Features

What is Kaku Assistant

Kaku Assistant is the built-in AI assistant feature of Kaku that can:

Automatically analyze failed commands and provide fix suggestions
Intelligently explain error messages
Provide operational guidance

Enabling/Disabling Assistant

# Via TUI interface
kaku config

# Or set in configuration file
# ~/.config/kaku/kaku.lua
local kaku = require("kaku")
kaku.assistant.enabled = true

Using Assistant

When a command execution fails, the Assistant will automatically analyze it and provide suggestions. Follow the prompts to apply the fix.

Shortcut Keys

Global Operations

Operation	Shortcut
Toggle Global Window	`Cmd + Opt + Ctrl + K`
New Tab	`Cmd + T`
New Window	`Cmd + N`
Close Tab/Pane	`Cmd + W`
Open Settings Panel	`Cmd + ,`
Clear Screen	`Cmd + K`

Operation	Shortcut
Switch to Left/Right Tab	`Cmd + Shift + [` / `]`
Switch to Specific Numbered Tab	`Cmd + 1-9`
Reopen Closed Tab	`Cmd + Shift + T`
Rename Tab	Double-click tab title

Pane Operations

Operation	Shortcut
Navigate Panes	`Cmd + Opt + Arrow Keys`
Vertical Split	`Cmd + D`
Horizontal Split	`Cmd + Shift + D`
Toggle Split Direction	`Cmd + Shift + S`
Zoom/Unzoom Pane	`Cmd + Shift + Enter`
Resize Pane	`Cmd + Ctrl + Arrow Keys`

AI Function Shortcuts

Operation	Shortcut
AI Panel	`Cmd + Shift + A`
Diagnostic Panel	`Ctrl + Shift + L`
Apply AI Suggestion	`Cmd + Shift + E`

Git and File Management

Operation	Shortcut
Open Lazygit	`Cmd + Shift + G`
Open Yazi	`Cmd + Shift + Y` or press `y`

Display Adjustment

Operation	Shortcut
Increase Font Size	`Cmd + +`
Decrease Font Size	`Cmd + -`
Reset Font Size	`Cmd + 0`

Smart Directory Jumping (z command)

Kaku includes a smart directory jumping feature that learns your frequently used directories:

# Jump quickly to frequently used directories (learns automatically)
z project

# List matching directories
z -l proj

# Jump to recently visited directory
z -t

Other Shell Enhancements

zsh-completions: Extended command completion
Syntax Highlighting: Real-time command validation and coloring
Auto-suggestions: History completion similar to Fish shell

CLI Tool Integration

Tools Installed by kaku init

Running kaku init will automatically install the following CLI tools:

Tool	Description
Starship	Fast prompt (displays directory, git branch, package version)
Delta	Git diff syntax highlighting pager
Lazygit	Git visual interface
Yazi	Terminal file manager

kaku cli Commands

# View help
kaku cli --help

# Split pane
kaku cli split-pane

# Split and run command
kaku cli split-pane -- bash -c "echo Hello"

Advanced Configuration

Configuration File Example

Edit ~/.config/kaku/kaku.lua:

local kaku = require("kaku")

-- Window opacity
config.window_background_opacity = 0.8
config.macos_window_background_blur = 20

-- Working directory inheritance
config.window_inherit_working_directory = true
config.tab_inherit_working_directory = true
config.split_pane_inherit_working_directory = true

-- Disable copy on select
config.copy_on_select = false

-- Color overrides
config.color_overrides = {
  ['#6E6E6E'] = '#3A3942',
}

Built-in Shell Configuration

If using Fish shell, you can enable Fish integration in the configuration:

`1`	`config.fish_integration = true`

Troubleshooting

kaku command not found

If the kaku command cannot be found in the terminal, run:

1
2
3

/Applications/Kaku.app/Contents/MacOS/kaku init --update-only
exec zsh -l
kaku doctor

Diagnosing Issues

# Run diagnostic panel
kaku doctor

# Check configuration
kaku config

# Update Kaku
kaku update

# Reset configuration
kaku reset

Enterprise Proxy Configuration

If you need to access AI services via an enterprise proxy, edit ~/.config/kaku/assistant.toml:

[assistant]
custom_headers = {
    "Proxy-Authorization" = "Bearer your-token"
}

Differences from Claude Code

Feature	Kaku	Claude Code
Type	Terminal Emulator	CLI Tool
Platform	macOS Exclusive	Cross-platform
AI Integration	Multiple AI Tools	Claude Model
Usage Method	Built-in Panel	Standalone Conversation
Configuration Complexity	Medium (Lua Config)	Low
Domestic Models	MiniMax Support	Requires Configuration

Best Practices Summary

1. Daily Development Workflow

1. Use Cmd + Opt + Ctrl + K to open Kaku
2. Use the z command to quickly switch project directories
3. Use built-in Git tools (Cmd+Shift+G) to manage code
4. Use the AI panel (Cmd+Shift+A) to diagnose issues when commands fail
5. Use split screen features to view multiple terminals simultaneously

2. AI Feature Usage

Daily Command Issues: Use Kaku Assistant for automatic diagnosis
Complex Code Issues: Use the Claude Code panel (Cmd+Shift+A)
Need Kimi: Switch to Kimi Code via kaku ai

3. Shortcut Habits

It is recommended to build muscle memory for common shortcuts:

Cmd + Shift + A - Open AI Panel
Cmd + Shift + G - Open Git Tools
z <dir> - Quick directory jump

4. Personalized Configuration

Customize kaku.lua according to personal habits:

Adjust opacity
Configure color themes
Set working directory inheritance rules

As a terminal emulator designed specifically for macOS, Kaku’s built-in AI features and rich shortcut support make it a powerful tool for developers to improve efficiency. Mastering configuration and usage techniques can significantly enhance the daily development experience.

Implementing Resource Isolation in Doris Offline Analysis Data Warehouse Cluster

wzdushu@gmail.com (wanzi) — Sat, 28 Feb 2026 09:30:00 +0800

I. Background

With rapid business development, the current offline analysis data warehouse Doris cluster is facing resource contention issues due to diverse computing needs such as daily data science jobs, offline batch processing, and customer-side report queries. To address this, resource limits need to be implemented for different account resource requests. After research, the official Doris Workload Group solution has been adopted.

Doris version: doris-2.1.8-1-834d802457

II. Workload Group Introduction

Workload Group is an in-process resource isolation mechanism provided by Apache Doris that achieves resource isolation between different business loads through fine-grained division of CPU, memory, and IO resources within BE processes.

The principle is as shown in the diagram below:

Currently supported isolation capabilities include:

Resource Type	Isolation Method	Description
CPU	Soft limit / Hard limit	Soft limit allocates by weight, hard limit is absolute upper bound
Memory	Soft limit / Hard limit	Hard limit automatically kills queries to free up memory when exceeded
IO	Speed limit	Limits IO throughput for reading local/remote files
Concurrency	Queuing mechanism	Exceeding concurrency queries enter queue to wait

For more information: https://doris.apache.org/zh-CN/docs/2.1/admin-manual/workload-management/workload-group

III. Doris Cluster Adjustment to Support Cgroup (Manual Deployment)

3.1 Cgroup Environment Configuration

Check the system’s supported CGroup version:

root@doristest:~# cat /proc/filesystems | grep cgroup
nodev   cgroup
nodev   cgroup2

Confirm the effective CGroup version:

root@doristest:~# ls /sys/fs/cgroup/cpu/
ls: cannot access '/sys/fs/cgroup/cpu/': No such file or directory 
root@doristest:~# ls /sys/fs/cgroup/cgroup.controllers
/sys/fs/cgroup/cgroup.controllers # Indicates cgroup v2 is in effect

3.2 Create Doris CGroup Directory

# ========== CGroup v1 Operations ==========
mkdir /sys/fs/cgroup/cpu/doris

# Set permissions (root is the user running BE)
chmod 770 /sys/fs/cgroup/cpu/doris
chown -R root:root /sys/fs/cgroup/cpu/doris


# ========== CGroup v2 Operations ==========
mkdir /sys/fs/cgroup/doris

# Set permissions
chmod 770 /sys/fs/cgroup/doris
chown -R root:root /sys/fs/cgroup/doris  # Adjust as needed, currently root

3.3 CGroup v2 Additional Configuration

CGroup v2 has stricter permission control and requires additional configuration:

# Modify root directory cgroup.procs file permissions
chmod a+w /sys/fs/cgroup/cgroup.procs

# Enable CPU controller
# Enter doris directory
cd /sys/fs/cgroup/doris

# Enable CPU controller (by modifying parent subtree_control)
# Already set by default, can be ignored
echo +cpu > ../cgroup.subtree_control

# Verify: Check if cpu.max file appears in doris directory
/sys/fs/cgroup/doris/cpu.max
# And cgroup.controllers should contain cpu
# cat /sys/fs/cgroup/cgroup.controllers 
cpuset cpu io memory hugetlb pids rdma misc

3.4 Configure BE and Restart

Edit the be.conf file:

# ========== CGroup v1 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris

# ========== CGroup v2 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/doris

Restart BE service:

# Restart BE
./bin/stop_be.sh
./bin/start_be.sh

Verification: Check the be.INFO log

grep "add thread" log/be.INFO
# Appearance of "add thread xxx to group" indicates successful configuration

3.5 Persistent Configuration (Optional)

CGroup configuration will be cleared after machine reboot. It is recommended to use systemd to create an auto-configuration service on boot:

# /etc/systemd/system/doris-cgroup.service
[Unit]
Description=Doris CGroup Setup
Before=doris-be.service

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'mkdir -p /sys/fs/cgroup/cpu/doris && chmod 770 /sys/fs/cgroup/cpu/doris && chown -R doris:doris /sys/fs/cgroup/cpu/doris'

[Install]
WantedBy=multi-user.target

Set to auto-start on boot:

systemctl enable doris-cgroup.service

IV. Batch Enable Cgroup on Doris Nodes via Ansible

4.1 Preparation

Before executing batch deployment, confirm the following information:

Confirm CGroup version: CGroup version (v1 or v2) for all Doris BE nodes
Prepare host inventory: Edit Ansible’s hosts file to add all BE nodes
Confirm BE user: User running Doris BE service (currently root)

4.2 Configure Host Inventory

Edit the hosts file:

1
2
3

[doris_dabe_nodes]
# If nodes have different CGroup versions, they can be grouped. Here only specifying offline analysis warehouse BE nodes
10.18.7.160

4.3 Create Ansible Playbook

Create file setup_doris_cgroup.yaml:

---
- name: Setup Doris CGroup Configuration
  hosts: doris_dabe_nodes
  become: true
  vars:
    # CGroup configuration path (automatically selected based on version)
    cgroup_v1_path: "/sys/fs/cgroup/cpu/doris"
    cgroup_v2_path: "/sys/fs/cgroup/doris"
    doris_user: "root"
    doris_group: "root"
    
  tasks:
    - name: Detect CGroup version
      stat:
        path: /sys/fs/cgroup/cpu/
      register: cgroup_v1_check
      
    - name: Set CGroup path based on version
      set_fact:
        cgroup_path: "{{ cgroup_v1_path if cgroup_v1_check.stat.exists else cgroup_v2_path }}"
        cgroup_version: "{{ 'v1' if cgroup_v1_check.stat.exists else 'v2' }}"
        
    - name: Display detected CGroup version
      debug:
        msg: "Node {{ inventory_hostname }} uses CGroup {{ cgroup_version }}, path: {{ cgroup_path }}"

    - name: Create CGroup directory
      file:
        path: "{{ cgroup_path }}"
        state: directory
        mode: '0770'
        owner: "{{ doris_user }}"
        group: "{{ doris_group }}"
      register: cgroup_dir_result
      
    - name: Configure CGroup v2 specific settings
      block:
        - name: Enable write permission on cgroup.procs
          file:
            path: /sys/fs/cgroup/cgroup.procs
            mode: 'a+w'
            
        - name: Enable CPU controller for CGroup v2
          shell: |
            if ! grep -q '+cpu' /sys/fs/cgroup/cgroup.subtree_control 2>/dev/null; then
              echo '+cpu' > /sys/fs/cgroup/cgroup.subtree_control
            fi            
          args:
            executable: /bin/bash
          ignore_errors: yes
          
        - name: Verify CPU controller is enabled
          stat:
            path: "{{ cgroup_path }}/cpu.max"
          register: cpu_controller_check
          
        - name: Display CPU controller status
          debug:
            msg: "CPU controller is {{ 'enabled' if cpu_controller_check.stat.exists else 'NOT enabled' }}"
      when: cgroup_version == 'v2'

    - name: Verify CGroup directory structure
      shell: |
        echo "=== CGroup Version: {{ cgroup_version }} ==="
        echo "CGroup Path: {{ cgroup_path }}"
        ls -la {{ cgroup_path }} 2>/dev/null || echo "Directory verified"
        echo ""
        if [ "{{ cgroup_version }}" = "v2" ]; then
          echo "=== CPU Controller Check ==="
          cat /sys/fs/cgroup/cgroup.controllers 2>/dev/null
          test -f {{ cgroup_path }}/cpu.max && echo "cpu.max: EXISTS" || echo "cpu.max: NOT FOUND"
        fi        
      args:
        executable: /bin/bash
      register: verify_result
      
    - name: Display verification results
      debug:
        var: verify_result.stdout_lines

    - name: Create systemd service for CGroup persistence (optional)
      copy:
        dest: /etc/systemd/system/doris-cgroup.service
        content: |
          [Unit]
          Description=Doris CGroup Setup
          Before=doris-be.service

          [Service]
          Type=oneshot
          ExecStart=/bin/bash -c 'mkdir -p {{ cgroup_path }} && chmod 770 {{ cgroup_path }} && chown -R {{ doris_user }}:{{ doris_group }} {{ cgroup_path }}'
          {% if cgroup_version == 'v2' %}
          ExecStartPost=/bin/bash -c 'chmod a+w /sys/fs/cgroup/cgroup.procs'
          ExecStartPost=/bin/bash -c 'echo "+cpu" > /sys/fs/cgroup/cgroup.subtree_control 2>/dev/null || true'
          {% endif %}

          [Install]
          WantedBy=multi-user.target          
        mode: '0644'
      when: cgroup_dir_result is changed
      notify: Enable systemd service

  handlers:   
    - name: Enable systemd service
      systemd:
        name: doris-cgroup
        enabled: yes
        daemon_reload: yes

4.4 Execute Batch Deployment

Syntax check and dry run

# 1. Syntax check
ansible-playbook -i hosts setup_doris_cgroup.yaml --syntax-check

# 2. List all tasks to be executed
ansible-playbook -i hosts setup_doris_cgroup.yaml --list-tasks

# 3. Dry run (not actually executing, only showing changes to be made)
ansible-playbook -i hosts setup_doris_cgroup.yaml --check --diff

4.5 Official Execution and Verification of Deployment Results

Official execution

# Method 1: Execute on all BE nodes
ansible-playbook -i hosts setup_doris_cgroup.yaml

# Method 2: Execute only on specific nodes (e.g., test on single node first)
ansible-playbook -i hosts setup_doris_cgroup.yaml --limit 10.18.7.101

# Method 3: Execute step by step, each task requires confirmation
ansible-playbook -i hosts setup_doris_cgroup.yaml --step

Adjust Doris BE configuration via Doris Manager (rolling restart):

# ========== CGroup v1 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris

# ========== CGroup v2 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/doris

Verification:

# 1. Check if CGroup directory is created
ansible doris_dabe_nodes -i hosts -m shell -a "ls -la /sys/fs/cgroup/doris 2>/dev/null || ls -la /sys/fs/cgroup/cpu/doris 2>/dev/null"

# 2. Check CGroup initialization in BE logs (adjust according to actual BE log path)
ansible doris_dabe_nodes -i hosts -m shell -a "tail -100 /opt/doris/doris/be/log/be.INFO | grep -i 'cgroup\|add thread' | tail -5"

At this point, Doris BE nodes have enabled Cgroup resource limits.

V. Workload Group Management

5.1 Create Workload Group

Basic example (CPU soft limit)

CREATE WORKLOAD GROUP IF NOT EXISTS g1
PROPERTIES (
    "cpu_share" = "1024"
);

Complete configuration example

CREATE WORKLOAD GROUP IF NOT EXISTS etl_group
PROPERTIES (
    -- CPU configuration (soft limit mode)
    "cpu_share" = "2048",
    "cpu_hard_limit" = "40%",
    
    -- Memory configuration (hard limit mode)
    "memory_limit" = "40%",
    "enable_memory_overcommit" = "false",
    
    -- Concurrency control
    "max_concurrency" = "1",
    "max_queue_size" = "100",
    "queue_timeout" = "10000",
    
    -- IO speed limit
    "read_bytes_per_second" = "104857600",        -- 100MB/s
    "remote_read_bytes_per_second" = "52428800"   -- 50MB/s
);

5.2 Attribute Details

Attribute Name	Type	Default Value	Value Range	Description
cpu_share	INT	-1	[1, 10000]	CPU soft limit weight, higher value means higher priority
cpu_hard_limit	INT	-1	[1%, 100%]	CPU hard limit percentage (new in version 2.1)
memory_limit	FLOAT	-1	(0%, 100%]	Memory limit percentage
enable_memory_overcommit	BOOL	true	true/false	true=soft limit (can exceed), false=hard limit
max_concurrency	INT	2147483647	[0, 2147483647]	Maximum concurrent queries
max_queue_size	INT	0	[0, 2147483647]	Queue length (0=no queuing)
queue_timeout	INT	0	[0, 2147483647]	Queue timeout (milliseconds)
scan_thread_num	INT	-1	[1, 2147483647]	Scan thread count (-1=use BE configuration)
max_remote_scan_thread_num	INT	-1	[1, 2147483647]	Maximum scan thread count for external tables
min_remote_scan_thread_num	INT	-1	[1, 2147483647]	Minimum scan thread count for external tables
read_bytes_per_second	INT	-1	[1, 9223372036854775807]	Internal table read IO limit (bytes/second)
remote_read_bytes_per_second	INT	-1	[1, 9223372036854775807]	External table read IO limit (bytes/second)

Notes:

Must specify at least one attribute when creating
CGroup v1 cpu_share default is 1024, range 2-262144
CGroup v2 cpu_share default is 100, range 1-10000
Sum of all Workload Group memory_limit cannot exceed 100%
Sum of all Workload Group cpu_hard_limit cannot exceed 100%

5.3 Modify Workload Group

ALTER WORKLOAD GROUP g1 PROPERTIES('cpu_share' = '4096');

-- Change memory limit to hard limit
ALTER WORKLOAD GROUP g1 PROPERTIES(
    'memory_limit' = '30%',
    'enable_memory_overcommit' = 'false'
);

-- Modify concurrency and queuing parameters
ALTER WORKLOAD GROUP g1 PROPERTIES(
    'max_concurrency' = '100',
    'max_queue_size' = '200',
    'queue_timeout' = '30000'
);

-- Add CPU hard limit (hard limit mode must be enabled first)
ALTER WORKLOAD GROUP g1 PROPERTIES('cpu_hard_limit' = '20%');

5.4 Delete Workload Group

- Delete specified Workload Group
DROP WORKLOAD GROUP g1;

Note: The default normal group cannot be deleted.

VI. User Binding and Authorization

6.1 View Available Workload Groups

-- View Workload Groups that the current user has permission to use
SELECT name FROM information_schema.workload_groups;

6.2 Authorization

-- Grant user permission to use specified Workload Group
GRANT USAGE_PRIV ON WORKLOAD GROUP 'g1' TO 'user_1'@'%';

-- Grant permission to all Workload Groups
GRANT USAGE_PRIV ON WORKLOAD GROUP '*' TO 'user_1'@'%';

-- Revoke permission
REVOKE USAGE_PRIV ON WORKLOAD GROUP 'g1' FROM 'user_1'@'%';

6.3 Binding Method 1: User Properties (Recommended)

Set default Workload Group for user (persistent):

-- Set user's default Workload Group
SET PROPERTY FOR 'user_1' = 'default_workload_group' = 'g1';

-- View user properties
SHOW PROPERTY FOR 'user_1';

VII. Monitoring and Viewing

7.1 View Workload Group List

-- Show statement
SHOW WORKLOAD GROUPS;

-- System table query
SELECT * FROM information_schema.workload_groups;

7.2 View Resource Usage

-- View memory usage of each Workload Group (unit: MB)
SELECT 
    workload_group_id,
    name,
    MEMORY_USAGE_BYTES / 1024 / 1024 AS mem_used_mb,
    CPU_USAGE_PERCENT AS cpu_percent
FROM workload_group_resource_usage;

-- View details of specific Workload Group
SELECT * FROM information_schema.workload_groups WHERE name = 'g1';

7.3 System Table Field Description

workload_groups table

Field	Description
ID	Workload Group ID
NAME	Name
CPU_SHARE	CPU soft limit weight
MEMORY_LIMIT	Memory limit percentage
ENABLE_MEMORY_OVERCOMMIT	Whether memory overcommit is allowed
MAX_CONCURRENCY	Maximum concurrency
MAX_QUEUE_SIZE	Queue length
QUEUE_TIMEOUT	Queue timeout
CPU_HARD_LIMIT	CPU hard limit percentage
SCAN_THREAD_NUM	Scan thread count
READ_BYTES_PER_SECOND	Internal table IO limit
REMOTE_READ_BYTES_PER_SECOND	External table IO limit

VIII. CPU Soft/Hard Limit Mode Switching

8.1 Mode Description

Mode	Characteristics	Applicable Scenarios
CPU soft limit	Allocates CPU time by weight, can use all CPU when idle	Large load fluctuations, want to fully utilize resources
CPU hard limit	Absolute upper limit, cannot exceed even if CPU is idle	Need strict resource isolation, guarantee SLA

Note: Only one mode can be used in a cluster at the same time, cannot be mixed.

8.2 Switch from Soft Limit to Hard Limit

Step 1: Set hard limit values for all Workload Groups

-- Must set cpu_hard_limit for all Groups
ALTER WORKLOAD GROUP g1 PROPERTIES('cpu_hard_limit' = '30%');
ALTER WORKLOAD GROUP g2 PROPERTIES('cpu_hard_limit' = '20%');
ALTER WORKLOAD GROUP normal PROPERTIES('cpu_hard_limit' = '50%');

-- Verify: Sum of all Group's cpu_hard_limit <= 100%
SELECT SUM(cpu_hard_limit) FROM information_schema.workload_groups 
WHERE cpu_hard_limit > 0;

Step 2: Enable cluster hard limit switch

-- Takes effect immediately in memory (invalid after restart)
ADMIN SET FRONTEND CONFIG ("enable_cpu_hard_limit" = "true");

-- Persistent configuration: Modify fe.conf for all FEs
echo "experimental_enable_cpu_hard_limit = true" >> fe/conf/fe.conf

8.3 Switch from Hard Limit Back to Soft Limit

-- Disable hard limit switch (automatically switches back to soft limit mode)
ADMIN SET FRONTEND CONFIG ("enable_cpu_hard_limit" = "false");

For persistence, modify fe.conf to false or delete the configuration.

The above is the overall solution for adding resource limits to the Doris offline analysis data warehouse.

Reference document: https://doris.apache.org/zh-CN/docs/2.1/admin-manual/workload-management/workload-group

Choosing SDD Tools: OpenSpec or Spec-Kit

wzdushu@gmail.com (wanzi) — Fri, 27 Feb 2026 10:30:00 +0800

In the previous article “What is SDD,” we explored the importance of Specification-Driven Development. This article provides an in-depth comparison of two mainstream SDD tools—OpenSpec and Spec-Kit—to help developers choose the right tool for their needs.

Overview of OpenSpec and Spec-Kit

Introduction to OpenSpec

OpenSpec is a Specification-Driven Development (SDD) framework developed by Fission AI, designed specifically for AI coding assistants.

Project URL: https://github.com/Fission-AI/OpenSpec

Core Features:

Lightweight specification layer to align humans and AI before coding
Independent folder for each change, containing proposal, specification, design, and tasks
Supports 20+ AI assistants via slash command integration
Flexible iteration without rigid stage gates
Complete change lifecycle management

Introduction to Spec-Kit

Spec-Kit is an open-source Specification-Driven Development toolkit developed by GitHub.

Project URL: https://github.com/github/spec-kit

Core Features:

Transforms specification documents into executable specifications to directly generate working code
Supports multiple AI agents: Claude Code, Copilot, Gemini CLI, Codex, Cursor, Windsurf, etc.
Complete workflow: Constitution → Specify → Plan → Tasks → Implement
Rich extension ecosystem: Jira integration, Azure DevOps integration, code review, etc.
Supports both Greenfield and Brownfield projects

Installation and Configuration

OpenSpec Installation

Prerequisites: Node.js 20.19.0 or higher

# npm install
npm install -g @fission-ai/openspec@latest

# Also supports pnpm, yarn, bun, and nix
pnpm add -g @fission-ai/openspec

Initialize Project

# Initialize in project directory
openspec init

# Select workflow configuration
openspec config profile

# Refresh AI guidance
openspec update

Disable Telemetry (Optional)

1
2
3

export OPENSPEC_TELEMETRY=0
# or
export DO_NOT_TRACK=1

Spec-Kit Installation

Prerequisites: Python environment (uv recommended)

# Persistent installation (recommended)
uv tool install specify-cli --from git+https://github.com/github/spec-kit.git

# One-time use
uvx --from git+https://github.com/github/spec-kit.git specify init <PROJECT_NAME>

# Upgrade
uv tool install specify-cli --force --from git+https://github.com/github/spec-kit.git

Initialize Project

# Create new project
specify init <PROJECT_NAME>

# Initialize in existing project
specify init . --ai claude

# Initialize specifying AI agent
specify init --here --ai claude

# Check environment
specify check

Core Feature Comparison

Basic Features

Feature	OpenSpec	Spec-Kit
Implementation Language	Node.js	Python
CLI Tool	openspec	specify
AI Agent Support	20+	10+
Change Structure	Proposal+Spec+Design+Tasks	Constitution+Specify+Plan+Tasks
State Management	Full Lifecycle	Phased Workflow
Extension Ecosystem	Basic	Rich (Jira, Azure DevOps, etc.)
Parallel Execution	Basic Support	Work Package Mode

OpenSpec Slash Commands

Command	Function
`/opsx:propose "idea"`	Create new proposal
`/opsx:new`	Create new change
`/opsx:continue`	Continue change
`/opsx:apply`	Execute tasks
`/opsx:ff`	Fast forward to completion
`/opsx:verify`	Verify change
`/opsx:sync`	Sync status
`/opsx:archive`	Archive completed changes
`/opsx:bulk-archive`	Bulk archive
`/opsx:onboard`	Onboard new members

Spec-Kit Slash Commands

Command	Function
`/speckit.constitution`	Establish project governance principles
`/speckit.specify`	Define functional requirements
`/speckit.plan`	Technical planning
`/speckit.tasks`	Break down tasks
`/speckit.implement`	Execute implementation
`/speckit.clarify`	Clarify ambiguous requirements
`/speckit.analyze`	Consistency analysis
`/speckit.checklist`	Quality check

Workflow Comparison

OpenSpec Workflow

OpenSpec is change-centric, with each change having an independent folder structure:

Change Folder Structure:
├── proposal.md      # Proposal description
├── spec.md          # Specification
├── design.md        # Technical design
├── tasks.md         # Task list
└── artifacts/       # Artifacts

Development Loop

1. Propose Idea
   /opsx:propose "Add user authentication"

2. Create Change
   /opsx:new
   # Creates change folder

3. Write Specification
   Edit spec.md

4. Technical Design
   Edit design.md

5. Execute Tasks
   /opsx:apply

6. Verify Completion
   /opsx:verify

7. Archive
   /opsx:archive

Spec-Kit Workflow

Spec-Kit adopts a phased workflow, emphasizing the executability of specification documents:

Workflow:
Constitution → Specify → Plan → Tasks → Implement

Development Loop

1. Initialize Project
   specify init . --ai claude

2. Establish Principles
   /speckit.constitution
   # Define project governance standards

3. Define Specification
   /speckit.specify
   # Describe the functionality to build

4. Create Plan
   /speckit.plan
   # Provide tech stack and architecture

5. Break Down Tasks
   /speckit.tasks
   # Generate executable task list

6. Execute Implementation
   /speckit.implement
   # Implement features according to plan

AI Agent Support Comparison

OpenSpec Support

Claude Code ✅
GitHub Copilot ✅
Gemini CLI ✅
And 20+ other AI assistants

Spec-Kit Support

AI Agent	Support	Notes
Claude Code	✅
GitHub Copilot	✅
Gemini CLI	✅
Codex CLI	✅	Requires `--ai-skills` parameter
Cursor	✅
Windsurf	✅
Qoder CLI	✅
Kilo Code	✅
Generic	✅	Requires specifying `--ai-commands-dir`

Spec-Kit Extension Ecosystem

Spec-Kit boasts a rich set of extensions and presets:

Extensions

Extension	Function
AI-Driven Engineering (AIDE)	7-step structured workflow
Jira Integration	Sync tasks with Jira
Azure DevOps Integration	Sync with Azure DevOps
Review Extension	Code review
Verify Extension	Specification verification
Spec Sync	Detect deviations between spec and implementation

Presets

Preset	Function
AIDE In-Place Migration	Tech migration preset
Pirate Speak	Pirate style output

Use Case Comparison

Scenarios for Choosing OpenSpec

Projects requiring complete change tracking
Teams that value the Proposal → Spec → Design → Task workflow
Environments using a mix of multiple AI tools
Projects preferring flexible iteration over strict phases
Teams requiring frequent change reviews

Scenarios for Choosing Spec-Kit

Enterprise environments requiring Jira/Azure DevOps integration
Need for code review and quality gates
Teams that value the executability of specification documents
Scenarios requiring technical migration support
Users of mainstream AI agents like Claude Code and Copilot

Summary Comparison

┌─────────────────────────────────────────────────────────┐
│                   SDD Tool Comparison                    │
├─────────────────────────────────────────────────────────┤
│                                                         │
│   OpenSpec                │         Spec-Kit             │
│   ─────────               │         ─────────             │
│   • Node.js Implementation│         • Python Implementation│
│   • Change-driven         │         • Executable Specs    │
│   • 20+ AI Assistants     │         • 10+ AI Assistants   │
│   • Flexible Iteration    │         • Phased Workflow     │
│   • Full Change Lifecycle │         • Rich Ecosystem      │
│                          │         • Enterprise Integration│
│                                                         │
│   Best For:               │         Best For:             │
│   • Change Tracking       │         • Enterprise Env      │
│   • Flexible Iteration    │         • Quality Gates       │
│   • Multi-AI Tools        │         • Spec Verification   │
│                                                         │
└─────────────────────────────────────────────────────────┘

Final Recommendations

Choose OpenSpec if:

You need a complete Proposal → Spec → Design → Task workflow
You value historical tracking and review of changes
You prefer a flexible, iterative working style
You use various AI programming tools

Choose Spec-Kit if:

You need integration with enterprise tools (Jira, Azure DevOps)
You prioritize code review and quality verification
You need consistency checks between specifications and implementation
You use mainstream AI agents like Claude Code or Copilot

Can be used together: The two tools can complement each other—use OpenSpec for rapid change exploration and Spec-Kit for enterprise-level specification management.

What is SDD? Why is it needed in AI programming tools?

wzdushu@gmail.com (wanzi) — Wed, 25 Feb 2026 10:30:00 +0800

With the rising popularity of AI programming tools, a crucial concept is reshaping how we collaborate with AI to write code—SDD (Specification-Driven Development). This article will delve into the core philosophy of SDD and why mainstream AI programming tools are choosing to support the SDD model.

What is SDD

Definition of SDD

Specification-Driven Development (SDD) is a software development methodology that emphasizes writing clear, precise specifications before writing code, and then strictly implementing code according to those specifications.

Core Principles:

Spec First: A Spec must be written before code implementation.
Precise Description: Use formal or semi-formal languages to describe requirements.
Bidirectional Traceability: Specifications and code remain consistent and can be verified against each other.
Iterative Refinement: Start with a simple specification and gradually refine and improve it.

SDD vs Traditional Development Models

Dimension	Traditional Development	SDD
Starting Point	Requirement documents or verbal descriptions	Precisely written Spec files
Flexibility	Requirements change while writing	Implementation after specification confirmation
Verification	Mainly manual testing	Specifications serve as test cases
AI Collaboration	AI has broad creative freedom	AI executes according to specifications
Change Handling	High cost of requirement changes	Specification changes drive code modifications

Why AI Programming Needs SDD

Challenges in AI Programming

1. Context Understanding Bias

Although AI models are powerful, they cannot fully understand ambiguous human descriptions. Requirements in traditional development are often like:

"Implement a user login function with basic validation"

Such ambiguous descriptions lead to AI-generated code that does not match expectations.

2. Unstable Generated Code Quality

Without specification constraints, AI might:

Miss handling of edge cases
Ignore security considerations
Have inconsistent API design styles
Implement code inconsistent with the existing project style

3. Difficult Iteration and Maintenance

When AI generates code without specifications:

It is hard to judge if requirements are met
The scope of impact is unknown during modifications
It is difficult for new team members to understand the design intent

How SDD Solves These Problems

Specifications Provide Precise Intent

# Traditional requirement description
"Implement a user login function"

# SDD specification description
## Functional Requirements
1. Support username/email + password login
2. Generate JWT token upon successful login, valid for 24 hours
3. Lock account for 15 minutes after 3 incorrect password attempts
4. Login interface requires brute-force protection (Rate limiting: max 10 attempts per IP within 5 minutes)

## Interface Specification
- POST /api/v1/auth/login
- Request: { "username": string, "password": string }
- Response: { "token": string, "expiresIn": 86400 }

## Error Handling
- 400: Parameter validation failed
- 401: Username or password incorrect
- 423: Account locked

## Security Requirements
- Use bcrypt for password storage
- Returned token must not contain sensitive information

Specifications as Test Cases

SDD specifications can be directly or tool-converted into:

API Mock tests
Unit test cases
Integration test scenarios
Acceptance criteria

Value of SDD in AI Programming

1. Reduce Hallucinations

When constrained by clear specifications, AI is more likely to generate code that meets expectations rather than “creating” non-existent features.

2. Improve Code Consistency

Specifications define unified:

API design styles
Code organization structure
Naming conventions
Error handling patterns

3. Facilitate Review and Collaboration

The clarity of specifications allows:

Code reviews to be based on evidence
Team members to work in parallel
New members to quickly understand the project

4. Support Progressive Development

SDD supports starting with a simple specification and gradually expanding it:

# v1.0 Minimum Viable Version
## AuthService
- login(username, password): Promise<User>

# v1.1 Add token support
## AuthService
- login(username, password): Promise<LoginResult>
- logout(token): Promise<void>
- refreshToken(token): Promise<LoginResult>

# v1.2 Add third-party login
## AuthService
- login(username, password): Promise<LoginResult>
- loginWithOAuth(provider, code): Promise<LoginResult>
- logout(token): Promise<void>
- refreshToken(token): Promise<LoginResult>

Core Components of SDD

1. Spec File Structure

project/
├── SPEC.md              # Main specification file
├── api/
│   └── openapi.yaml     # API specification
├── models/
│   └── schema.md        # Data model specification
└── specs/
    ├── auth.md          # Authentication module specification
    ├── user.md          # User module specification
    └── ...

2. Specification Writing Principles

FIRRN Principles:

Factual: Describe the actual behavior of the system, not a wish list.
Independent: Each specification module describes one function independently.
Readable: Specifications should be easy to understand and maintain.
Realistic: Specifications must be achievable and testable.
Necessary: Every requirement is necessary.

3. Mapping Specifications to Code

## UserService.getUserById

### Description
Get user information by user ID

### Input
- userId: string (Required, User ID)

### Output
- User object or null

### Behavior
1. Validate userId format
2. Query user from database
3. Return user information (password field excluded)

### Errors
- InvalidUserIdError: userId format is invalid

Corresponding code:

async getUserById(userId: string): Promise<User | null> {
  // 1. Validate userId format
  if (!isValidUserId(userId)) {
    throw new InvalidUserIdError(userId);
  }

  // 2. Query user from database
  const user = await this.db.users.findOne({
    where: { id: userId },
    attributes: { exclude: ['password'] }
  });

  // 3. Return user information
  return user;
}

SDD Workflow

Complete Development Process

┌─────────────────────────────────────────────────────────┐
│  1. Requirement Analysis                                 │
│     - Collect business requirements                       │
│     - Identify key use cases                              │
│     - Determine constraints                               │
└───────────────────────┬─────────────────────────────────┘
                        ▼
┌─────────────────────────────────────────────────────────┐
│  2. Write Spec (Specification)                           │
│     - Write SPEC.md                                       │
│     - Define API interfaces                               │
│     - Describe data models                                │
│     - Clarify boundary conditions and error handling     │
└───────────────────────┬─────────────────────────────────┘
                        ▼
┌─────────────────────────────────────────────────────────┐
│  3. Review and Confirmation                               │
│     - Technical review                                   │
│     - Business confirmation                               │
│     - Lock specification after consensus                 │
└───────────────────────┬─────────────────────────────────┘
                        ▼
┌─────────────────────────────────────────────────────────┐
│  4. AI Generates Code                                    │
│     - Generate code based on Spec                         │
│     - Generate unit tests                                 │
│     - Generate API documentation                          │
└───────────────────────┬─────────────────────────────────┘
                        ▼
┌─────────────────────────────────────────────────────────┐
│  5. Verification and Iteration                            │
│     - Does code conform to Spec?                         │
│     - Is test coverage complete?                         │
│     - Update Spec or fix code when issues are found      │
└─────────────────────────────────────────────────────────┘

Spec Review Checklist

Functional description is clear and unambiguous
Inputs and outputs are clearly defined
Boundary conditions are covered
Error handling is complete
API design is consistent
Data model is reasonable
Security considerations are comprehensive
Performance requirements are clear

SDD Tool Ecosystem

Mainstream SDD Tools

Tool	Features	Use Cases
Spec-Kit	Designed for AI programming, CLI tool	Claude Code integration
OpenSpec	Open standard, community-driven	Cross-tool universal
Stopify	Simple Markdown format	Lightweight projects
Docusaurus	Documentation-driven	Projects with mature documentation

Claude Code and SDD

Claude Code natively supports the SDD workflow:

# 1. Create project and write specification
mkdir my-project && cd my-project
claude "Initialize a Node.js REST API project, create SPEC.md specification file"

# 2. Implement based on specification
claude "Implement the user authentication module according to SPEC.md"

# 3. Verify if implementation conforms to specification
claude "Check if the current implementation fully conforms to the definitions in SPEC.md"

# 4. Sync modifications when specification updates
# Edit SPEC.md to add new features
claude "Refactor related code according to the updated SPEC.md"

SDD Best Practices

Specification Writing Tips

1. Use Structured Format

## Function Name

### Description
Brief description of what the function does

### Input
- param1: type (description)
- param2: type (description)

### Output
- type: description

### Behavior
1. Step 1
2. Step 2

### Errors
- ErrorType: description

2. Use Concrete Examples

### Login Success Example
Input: { "username": "user@example.com", "password": "correct_password" }
Output: { "token": "eyJhbGciOiJIUzI1NiIs...", "expiresIn": 86400 }

### Password Error Example
Input: { "username": "user@example.com", "password": "wrong_password" }
Output: 401 { "error": "INVALID_CREDENTIALS", "message": "用户名或密码错误" }

3. Define Clear Acceptance Criteria

## Acceptance Criteria
- [ ] Username format validation is correct
- [ ] Password strength validation is correct
- [ ] Valid token returned on successful login
- [ ] Error count statistics are correct
- [ ] Account locking function works normally
- [ ] Token expiration handling is correct

Common Pitfalls

1. Specifications Are Too Detailed

# Bad practice: Dictating code implementation details
### Implementation
Use a for loop to iterate through the array, call the helper function on line 5

# Good practice: Only describing behavior
### Implementation
Process data items in the specified order

2. Specifications and Implementation Are Out of Sync

Update the Spec every time code is updated.
Spec changes must go through review.
Forbid “code implemented but Spec not updated”.

3. Over-abstraction

Specifications must be concrete and executable, not vague architectural designs.

Summary

Core Value of SDD

Clarity: Eliminates ambiguous requirements and reduces communication costs.
Verifiability: Specifications can be directly converted into test cases.
Maintainability: Code and documentation always remain consistent.
AI-Friendly: Provides clear execution goals for AI.

SDD Applicability Scenarios

Scenario	SDD Suitability	Reason
AI-assisted programming	Very Suitable	AI needs clear instructions
Complex business systems	Suitable	Reduces requirement understanding deviations
Rapid prototyping	Average	Specifications might become a burden
Personal small projects	Optional	Over-design might be wasteful
Multi-person collaborative projects	Very Suitable	Unified understanding and standards

Learning Suggestions

Start with Small Projects: Choose a simple project to practice the SDD workflow.
Use Good Tools: Recommend starting with Spec-Kit or simple Markdown.
Keep Specifications Updated: Specifications are not one-time; they require continuous maintenance.
Collaborate with AI: Let AI help you generate the specification draft, and you review and refine it.

SDD is not a silver bullet, but it provides a reliable framework for AI programming. As AI programming tools continue to evolve, SDD will become an important foundation for efficient human-computer collaboration.

Doris Manager Unified Deployment and Multi-Cluster Management

wzdushu@gmail.com (wanzi) — Thu, 12 Feb 2026 09:30:00 +0800

With business expansion and the diversification of product forms, there is a need to deploy multiple sets of offline and real-time data warehouse clusters. While the previous Ansible Playbook method for deploying Doris FE/BE was basically sufficient, it fell short for subsequent Doris version upgrades, scaling, visual management, and onboarding junior operations personnel. Therefore, this document was written to facilitate quick future reference.

1. Overview

1.1 Introduction to Doris Manager

Doris Manager is a graphical operation and maintenance tool launched by the Apache Doris community for unified deployment, management, and monitoring of Doris clusters. It simplifies the entire process from operating system initialization and component deployment to daily operation and maintenance management, making it particularly suitable for rapidly setting up and maintaining large-scale Doris clusters in production environments.

From the current perspective, the official open-source repository is no longer maintained. As SelectDB is a commercial company, for its own development, it has evolved Doris Manager into a free, closed-source operation and maintenance tool, though it can still be downloaded and used.

1.2 Main Features

Cluster Deployment: Automated deployment of FE, BE, Broker, and other components.
Node Management: Unified management of all nodes, supporting scaling out and scaling in.
Configuration Management: Centralized configuration management and parameter adjustment.
Monitoring and Alerting: Cluster status monitoring and alert notifications.
Log Viewing: Centralized log viewing and analysis.

2. Prerequisites

2.1 Environment Planning

Before deployment, please confirm the following information:

Planning Item	Description	Example
Node Planning	Determine FE and BE node IPs and quantities	FE: 3 nodes, BE: 10 nodes
Storage Planning	Plan data storage directories and disk types	/data/doris (ext4, noatime)
Network Planning	Confirm network connectivity between nodes and open ports	9030, 9031, 8060, 9000+
Timezone Planning	Configure timezone based on the business country	Africa/Abidjan (Côte d’Ivoire)
Domain Planning	Manager access domain	doris-manager.example.com

2.2 Software Preparation

Official download address: https://selectdb.com/download/enterprise#manager

Download to the company software repository in advance (needs to be downloaded from the selectdb official website):

Repository Address: https://repo.test.com/bigdata/

Software Package	Version	Description
doris-manager	24.1.5-x64-bin	Doris Manager Management Tool
apache-doris	2.1.8.1-bin-x64	Apache Doris Community Edition

Download Commands:

# Create download directory
mkdir -p /opt/downloads/doris_manager /opt/downloads/doris
cd /opt/downloads/doris_manager

# Download installation packages (from company repository)
wget https://repo.test.com/bigdata/doris-manager-24.1.5-x64-bin.tar.gz
wget https://repo.test.com/bigdata/apache-doris-2.1.8.1-bin-x64.tar.gz

2.3 System Requirements

Component	Minimum Configuration	Recommended Configuration
Operating System	CentOS 7+ / Ubuntu 18.04+	CentOS 7.9 / Ubuntu 20.04
CPU	4 Cores	8 Cores+
Memory	8GB	16GB+
Disk	100GB	SSD 500GB+
Network	Gigabit NIC	10 Gigabit NIC

3. System Initialization and Optimization

Before deploying Doris Manager and the Doris cluster, system initialization configuration is required for all nodes.

3.1 Data Disk Initialization

Important: Production environments must use an independent data disk; do not share it with the system disk.

Manual initialization (single node)

# 1. View data disk devices
lsblk
# Assume the data disk is /dev/vdb

# 2. Create ext4 file system (optimized parameters)
# -m 1: Reduce reserved space from 5% to 1% to increase available space
# -T largefile: Optimize for large file storage
# -L: Set volume label for easy identification
mkfs.ext4 -m 1 -T largefile -L doris-data /dev/vdb

# 3. Create mount point directory
mkdir -p /data

# 4. Mount data disk
mount /dev/vdb /data

# 5. Get disk UUID
blkid /dev/vdb
# Output example: /dev/vdb: UUID="e86ca90c-ca31-460d-b450-1fe914ebae70" TYPE="ext4"

# 6. Configure automatic mount on boot (replace with actual UUID)
echo "UUID=e86ca90c-ca31-460d-b450-1fe914ebae70 /data ext4 defaults,noatime 0 2" >> /etc/fstab
# noatime: Do not update access time to improve performance (especially for big data read/write)
# 0: Do not backup
# 2: Check file system on startup

# 7. Create Doris data directory
mkdir -p /data/doris

# 8. Verify mount
df -h | grep /data

3.2 Disable Swap Partition

Reason: When Doris is under high memory pressure, swap swaps memory pages to disk, causing GC pauses, query lag, or even node pseudo-death, severely affecting performance.

# 1. Temporarily turn off swap
sudo swapoff -a

# 2. Permanently disable swap (comment out the swap line in /etc/fstab)
sudo sed -i '/swap/s/^/#/' /etc/fstab

# 3. Verify swap is turned off
free -h
# Swap should show 0B

swapon --show
# Should have no output, indicating swap is fully closed

3.3 Optimize Kernel Parameters

3.3.1 Increase Virtual Memory Areas (VMA)

Reason: Doris maps a large number of files to memory, requiring sufficient virtual memory areas.

# Add kernel parameters
cat >> /etc/sysctl.conf << EOF
vm.max_map_count = 2000000
EOF

# Apply configuration
sysctl -p

# Verify
sysctl vm.max_map_count
# Should output: vm.max_map_count = 2000000

3.3.2 Optimize Network Parameters (Optional)

# Optimize TCP connection parameters for high concurrency scenarios
cat >> /etc/sysctl.conf << EOF
# TCP optimization parameters
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 2048
EOF

sysctl -p

3.4 Increase File Handle Limits

Reason: Doris BE opens a large number of files (data shards, indexes, logs, etc.), so the file descriptor limit must be increased.

# 1. Modify limits configuration
sudo tee -a /etc/security/limits.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

# 2. Create independent configuration in limits.d directory
sudo tee -a /etc/security/limits.d/20-nproc.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

# 3. Verify configuration
ulimit -n
# Should output: 65536

3.5 Apply All Configurations and Reboot

# Apply all kernel parameters
sysctl -p

# Reboot nodes for all configurations to take effect
reboot

# Verification after reboot
# 1. Check swap: free -h (Swap should be 0)
# 2. Check file handles: ulimit -n (should be 65536)
# 3. Check VMA: sysctl vm.max_map_count (should be 2000000)
# 4. Check data disk: df -h | grep /data

In actual deployment, you can customize virtual machine hosts or cloud host images in advance to include basic initialization operations.

4. Deploy Doris Manager

4.1 Extraction and Installation

# 1. Extract installation package
cd /opt/downloads/doris_manager
tar zxvf doris-manager-24.1.5-x64-bin.tar.gz

# 2. Create deployment directory
mkdir -p /opt/doris/manager

# 3. Move files to deployment directory
mv ./doris-manager-24.1.5-x64-bin/* /opt/doris/manager/

# 4. Prepare Doris installation package resources
# Manager needs local source files for subsequent cluster deployment
mkdir -p /opt/downloads/doris
cp /opt/downloads/doris_manager/apache-doris-2.1.8.1-bin-x64.tar.gz /opt/downloads/doris/
cp /opt/downloads/doris_manager/selectdb-doris-2.0.3-b4-bin-x64.tar.gz /opt/downloads/doris/

4.2 Start Doris Manager

# Start Doris Manager WebServer
cd /opt/doris/manager/
./webserver/bin/start.sh

# View logs (optional)
tail -f webserver/log/webserver.log

Startup Verification: Service listens on port 8004

# Check if port is listening normally
ss -an | grep 8004
# Expected output:
# tcp    LISTEN    0      100    *:8004    *:*

# Or use netstat
netstat -tlnp | grep 8004

4.3 Configure Auto-start on Boot (Recommended)

Use systemd to manage the Doris Manager service, ensuring automatic startup after node restarts.

# 1. Create systemd service file
cat > /etc/systemd/system/doris-manager.service <<'EOF'
[Unit]
Description=Doris Manager WebServer Service
Documentation=https://doris.apache.org/
After=network.target

[Service]
Type=forking
WorkingDirectory=/opt/doris/manager
ExecStart=/opt/doris/manager/webserver/bin/start.sh
ExecStop=/opt/doris/manager/webserver/bin/stop.sh
Restart=on-failure
RestartSec=10
User=root
Group=root
TimeoutStartSec=120
TimeoutStopSec=120
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

# 2. Reload systemd configuration
sudo systemctl daemon-reload

# 3. Enable auto-start on boot
sudo systemctl enable doris-manager

# 4. Check service status
sudo systemctl status doris-manager

4.4 Access Manager UI

Access the Doris Manager management interface via a browser:

http://{manager-ip}:8004

First Access Steps:

Enter the initialization user page to create the first Manager admin user.
The Manager admin account is independent of cluster accounts and is only used for Manager permission control.
Log in with that account after successful creation.

5. Configure Nginx Reverse Proxy

In production environments, it is recommended to configure domain names and HTTPS access via Nginx.

5.1 Nginx Configuration File

server {
  listen 80;
  listen 443 ssl;
  charset utf-8;
  server_name ug-test-doris-manager.test.com;
  index  index.html index.htm index.jsp index.php;

  # SSL configuration (optional)
  # if ($scheme = http) {
  #   rewrite ^(.*)$ https://$host$1 redirect;
  # }

  include rule.d/ssl.conf;
  include rule.d/ssl_test.conf;

  client_max_body_size 5000m;

  location  / {
    include rule.d/proxy_header.conf;
    include rule.d/proxy_connection_small.conf;
    proxy_pass http://10.156.179.58:8004;
  }

  access_log  /var/log/nginx/ug-test-doris-manager.test.com.access.log main;
  error_log   /var/log/nginx/ug-test-doris-manager.test.com.error.log;
}

5.2 Restart Nginx

# Check configuration file syntax
nginx -t

# Restart Nginx
systemctl restart nginx

# Verify
curl -I http://ug-test-doris-manager.test.com

Now you can access Doris Manager via the domain name:

https://ug-test-doris-manager.test.com

6. Initialize Manager Configuration

6.1 Create Admin Account

Access UI Interface: Access Doris Manager via the configured domain or IP.
Create Admin Account:
- Set username and password.
- Service Configuration: Since external network policies may not be open, it is recommended to turn off email alerts first.
- Click to start the doris-manager service.
Enter Management Console: After successful installation, enter the main interface.

6.2 Upload Doris Installation Packages

Upload the Doris installation package on the Manager interface:

apache-doris-2.1.8.1-bin-x64.tar.gz

Or use the command line to place in the specified directory:

1
2
3

# Ensure the path is accessible by Manager
mkdir -p /opt/downloads/doris
cp apache-doris-2.1.8.1-bin-x64.tar.gz /opt/downloads/doris/

7. Deploy Doris Agent

Agent is the proxy program of Doris Manager on each node, responsible for executing deployment and management tasks.

7.1 Get Installation Command

On the Manager interface:

Go to “Cluster Management” -> “Agent Management”.
Click the “Add Agent” button.
Get the installation command (usually two lines).

Example Command:

`1`	`wget http://10.156.179.58:8004/api/download/deploy.sh -O deploy_agent.sh && chmod +x deploy_agent.sh && ./deploy_agent.sh`

Note: Replace the IP address with the actual Manager service IP.

7.2 Execute Installation on Target Nodes

# Execute on every Doris node
wget http://{manager-ip}:8004/api/download/deploy.sh -O deploy_agent.sh
chmod +x deploy_agent.sh
./deploy_agent.sh

7.3 Configure Agent Auto-start on Boot

Create a systemd service file to manage Agent:

# 1. Create systemd service file
cat > /etc/systemd/system/doris-manager-agent.service <<'EOF'
[Unit]
Description=Doris Manager Agent Service
Documentation=https://doris.apache.org/
After=network.target

[Service]
Type=forking
# Type=forking is because the start.sh script starts the process in the background and exits immediately

WorkingDirectory=/root/manager-agent

ExecStart=/root/manager-agent/bin/start.sh
ExecStop=/root/manager-agent/bin/stop.sh

Restart=on-failure
RestartSec=10

User=root
Group=root

TimeoutStartSec=120
TimeoutStopSec=120

StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

# 2. Reload systemd configuration
sudo systemctl daemon-reload

# 3. Enable auto-start on boot
sudo systemctl enable doris-manager-agent

# 4. Start service
sudo systemctl start doris-manager-agent

# 5. Check service status
sudo systemctl status doris-manager-agent

# 6. View real-time logs (if startup fails)
sudo journalctl -u doris-manager-agent -f

7.4 Manually Create Data Directory

Manually create the data storage directory on all nodes:

`1`	`mkdir -p /data/doris`

7.5 Verify Agent in Manager Interface

On the “Agent Management” page of the Manager interface, you should see the installed Agent nodes with a status of “Online”.

8. Create Doris Cluster

8.1 Start Creating Cluster

Enter Cluster Management: Click “Cluster Management” on the Manager main interface.
Click “New Cluster”: Start creating a new Doris cluster.
Fill in Cluster Configuration:
- Cluster Name: e.g., prod-cluster
- Cluster Version: Select apache-doris-2.1.8.1
- Select Installation Package: Choose from the uploaded packages.

8.2 Add FE Nodes

Add FE Nodes:
- Select FE nodes from the list of nodes with Agent installed.
- It is recommended to have at least 3 FE nodes to form a high-availability cluster.
- Select FE role (Follower or Observer).
Configure FE Parameters:
- Ports: http_port=8030, query_port=9030, edit_log_port=9010
- Memory Allocation: Configure according to the node memory size.

8.3 Add BE Nodes

Add BE Nodes:
- Select BE nodes from the list of nodes with Agent installed.
- Select an appropriate number of BE nodes according to business requirements.
Configure BE Parameters:
- Ports: be_port=9060, webserver_port=8040
- Data Directory: /data/doris
- Memory Allocation: It is recommended to reserve system memory and allocate 70-80% of the remainder to BE.

8.4 Environment Detection

Before starting deployment, Manager will automatically perform environment detection:

System parameter checks (file handles, VMA, swap, etc.)
Disk space checks
Network connectivity checks
Port occupation checks

Wait for all detection items to pass, then click “Next” to continue deployment.

If there are failed detection items, please fix them according to the prompts and re-detect.

9. Cluster Deployment and Configuration

9.1 System Initialization Configuration

Before deploying the cluster, system parameter initialization configuration is required:

Execute the following system parameter adjustments on all nodes:

Modify System Limits (Limits):

# Modify file handle limits
sudo tee -a /etc/security/limits.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

sudo tee -a /etc/security/limits.d/20-nproc.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

Apply Configuration and Reboot:

# Apply kernel parameters
sysctl -p

# Reboot node
reboot

Start Agent After Reboot: After the node restart is complete, you must manually start the Agent service.

`1`	`systemctl start doris-manager-agent`

9.2 Deploy Cluster

Cluster Settings:
- Table Name Case Sensitivity: It is recommended to select “Insensitive” (for easier application compatibility).
- Confirm all configurations are correct.
Start Deployment: Click the “Deploy Cluster” button.
Monitor Deployment Progress:
- Manager will automatically deploy FE and BE components on each node.
- You can view real-time deployment logs on the interface.
- The deployment process usually takes 5-10 minutes.
Deployment Complete:
- All component statuses show as “Running”.
- Enter the cluster details page to view the cluster overview.

9.2 Configure Timezone (Important)

Business Scenario: To meet the business needs of different countries, the cluster timezone needs to be adjusted according to the deployment region.

Timezone Mapping Table:

Country/Region	Timezone Configuration	Example
China East 8 District	Asia/Shanghai	Mainland China
West Africa Côte d’Ivoire	Africa/Abidjan	CB/CDI Project
West Africa Uganda	Africa/Kampala	Uganda Project
UTC Standard Time	UTC	International General

9.2.1 Modify FE Timezone

On the Manager interface:

Go to “Cluster Management” -> Select Cluster -> “Configuration Management”.
Find the FE configuration file fe.conf.
Add or modify the following parameter:

1
2

# fe/conf/fe.conf
time_zone = Africa/Abidjan

Save the configuration and restart the FE node.

9.2.2 Modify BE Timezone

On the Manager interface:

Find the BE configuration file be.conf in “Configuration Management”.
Add or modify the following parameter:

1
2

# be/conf/be.conf
time_zone = Africa/Abidjan

Save the configuration and restart the BE node.

9.2.3 Verify Timezone Configuration

Method 1: Verify via MySQL Client

# Connect to FE
mysql -h {fe_host} -P9030 -u root -p

# View timezone settings
SHOW VARIABLES LIKE '%time_zone%';

# View current time
SELECT NOW();

Expected Output:

+---------------+--------+
| Variable_name | Value  |
+---------------+--------+
| time_zone     | AAI    |  # Abbreviation for Africa/Abidjan
+---------------+--------+

Method 2: Verify via Logs

# View FE logs
grep -i timezone /opt/doris/fe/log/fe.log

# View BE logs
grep -i timezone /opt/doris/be/log/be.INFO

10. Cluster Verification and Testing

10.1 Check Cluster Status

View in the Manager interface:

FE Node Status: Should be “ALIVE”
BE Node Status: Should be “ALIVE”
Cluster Health Status: Should be “Healthy”

10.2 Verify via Command Line

# 1. Connect to FE
mysql -h {fe_host} -P9030 -u root -p

# 2. View FE status
SHOW FRONTENDS;

# 3. View BE status
SHOW BACKENDS;

# 4. Create test database
CREATE DATABASE test_db;

# 5. Create test table
USE test_db;
CREATE TABLE test_table (
  id INT,
  name VARCHAR(100)
) DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 10;

# 6. Insert test data
INSERT INTO test_table VALUES (1, 'test'), (2, 'doris');

# 7. Query test data
SELECT * FROM test_table;

10.3 Verify Timezone

-- Query current time to confirm timezone is correct
SELECT NOW(), CURRENT_TIMESTAMP();

-- View timezone variables
SHOW VARIABLES LIKE '%time_zone%';

11. Troubleshooting

11.1 Agent Cannot Connect to Manager

Symptom: Manager interface shows Agent as offline.

Troubleshooting Steps:

# 1. Check if Agent process is running
ps aux | grep agent

# 2. Check Agent logs
tail -f /root/manager-agent/log/agent.log

# 3. Check network connectivity
curl -I http://{manager-ip}:8004

# 4. Check firewall and ports
telnet {manager-ip} 8004

# 5. Restart Agent
systemctl restart doris-manager-agent

11.2 FE Node Fails to Start

Common Causes:

Port occupied
Metadata directory permission issues
JAVA_HOME not configured

Troubleshooting Steps:

# 1. View FE logs
tail -f /opt/doris/fe/log/fe.log

# 2. Check port occupation
netstat -tlnp | grep -E '9030|9010|8030'

# 3. Check directory permissions
ls -la /data/doris/

# 4. Check Java environment
java -version
echo $JAVA_HOME

11.3 BE Node Fails to Start

Common Causes:

Data disk not mounted
Memory configuration too large
System parameters not effective

Troubleshooting Steps:

# 1. View BE logs
tail -f /opt/doris/be/log/be.INFO

# 2. Check data disk mount
df -h | grep /data

# 3. Check memory configuration
free -h
grep -i mem /opt/doris/be/conf/be.conf

# 4. Check system parameters
ulimit -n
sysctl vm.max_map_count

11.4 Timezone Configuration Not Taking Effect

Troubleshooting Steps:

# 1. Confirm configuration file has been modified
grep time_zone /opt/doris/fe/conf/fe.conf
grep time_zone /opt/doris/be/conf/be.conf

# 2. Restart FE and BE
# Restart via Manager interface or manually

# 3. View startup logs
grep -i timezone /opt/doris/fe/log/fe.log | tail -20

# 4. Verify via MySQL client
mysql -h {fe_host} -P9030 -u root -e "SHOW VARIABLES LIKE '%time_zone%';"

12. Operations Management

12.1 Cluster Scaling Out

Add BE Nodes:

Deploy Agent on the new node (refer to Chapter 7).
In Manager interface, select Cluster -> “Node Management” -> “Add Node”.
Select the BE nodes to add.
Confirm to automatically deploy and start BE.

Add FE Nodes:

Deploy Agent on the new node.
Add FE Follower node in Manager interface.
Wait for FE to complete data synchronization and join the cluster.

12.2 Cluster Scaling In

Delete BE Nodes:

Select the BE node to delete in Manager interface.
Click the “Delete” button.
The system will automatically perform data migration (takes a certain time).
The node will automatically go offline after data migration is complete.

Delete FE Nodes:

Deleting FE Follower nodes is not recommended in production environments.
If necessary, please ensure the cluster has at least 3 FE nodes first.

12.3 Configuration Modification

You can easily modify cluster configurations in the Manager interface:

Go to “Cluster Management” -> Select Cluster -> “Configuration Management”.
Select the configuration file to modify (fe.conf or be.conf).
Save after modifying parameters.
Click “Apply Configuration” to restart the corresponding components.

Common Configuration Adjustments:

Configuration Item	Location	Description
mem_limit	be.conf	BE memory limit
max_compaction_threads	be.conf	Maximum compaction threads
max_permissive_result_mem	fe.conf	Maximum memory for single query
enable_fuzzy_mode	fe.conf	Whether to enable fuzzy mode

12.4 Monitoring and Alerting

Cluster Monitoring Metrics:

CPU usage
Memory usage
Disk space usage
Query QPS
Query response time

Alert Configuration: Configure alert rules in the Manager interface:

Go to “Cluster Management” -> “Alert Management”.
Create alert rules (e.g., disk usage > 80%).
Configure alert receiving methods (Email, DingTalk, WeCom).

13. Appendix

13.1 Port Mapping Table

Component	Port	Description
Manager WebServer	8004	Manager Management Interface
FE http_port	8030	FE HTTP Service
FE query_port	9030	FE MySQL Protocol Port
FE edit_log_port	9010	FE Metadata Synchronization Port
BE be_port	9060	BE Communication Port
BE webserver_port	8040	BE Web Service

13.2 Directory Structure

/opt/doris/
├── manager/                    # Doris Manager
│   ├── webserver/
│   │   ├── bin/
│   │   └── log/
│   └── ...
├── fe/                         # Frontend
│   ├── bin/
│   ├── conf/                   # Configuration files
│   │   ├── fe.conf
│   │   └── fe_custom.conf
│   ├── log/                    # Log directory
│   │   ├── fe.log
│   │   └── fe.warn.log
│   └── ...
├── be/                         # Backend
│   ├── bin/
│   ├── conf/                   # Configuration files
│   │   ├── be.conf
│   │   └── be_custom.conf
│   ├── log/                    # Log directory
│   │   ├── be.INFO
│   │   └── be.out
│   └── ...
└── ...

/root/manager-agent/            # Agent working directory
├── bin/
├── conf/
└── log/

/data/doris/                    # Data storage directory
├── fe/                         # FE data
│   ├── meta/
│   └── ...
└── be/                         # BE data
    ├── storage/
    └── ...

13.3 Reference Documents

At this point, the deployment of Doris Manager and the setup of the Doris cluster are complete!

Best Practices for Using Claude Code

wzdushu@gmail.com (wanzi) — Thu, 05 Feb 2026 10:30:00 +0800

Claude Code is the official AI coding tool launched by Anthropic. It can read codebases, edit files, run commands, and integrate with development tools. It excels in complex reasoning and code understanding, making it one of the most powerful AI coding tools available today.

Detailed Introduction to Claude Code

What is Claude Code

Claude Code is an AI-driven agent coding assistant that helps developers build features, fix bugs, and automate development tasks. It understands the entire codebase and can work across multiple files and tools to complete tasks.

Core Advantages:

Powerful code understanding and reasoning capabilities
Supports multiple interfaces including Terminal, VS Code, and JetBrains
MCP (Model Context Protocol) extension support
Multi-agent collaboration features
Full Git workflow integration
Supports the full family of Claude models

Claude Code Installation

macOS / Linux (Recommended)

`1`	`curl -fsSL https://claude.ai/install.sh \| bash`

Windows PowerShell

`1`	`irm https://claude.ai/install.ps1 \| iex`

Windows CMD

`1`	`curl -fsSL https://claude.ai/install.cmd -o install.cmd && install.cmd && del install.cmd`

Homebrew

`1`	`brew install --cask claude-code`

WinGet (Windows)

`1`	`winget install Anthropic.ClaudeCode`

VS Code Extension

Search for “Claude Code” in the VS Code Extension Marketplace and install.

Quick Start

# Enter project directory
cd your-project

# Start Claude Code
claude

# Specify a task directly
claude "Help me create a user login feature"

Basic Usage of Claude Code

Basic Usage Tips

1. Task Decomposition Strategy

Do not ask the AI to complete complex tasks all at once:

# Inefficient ❌
claude "Help me write a complete e-commerce system"

# Efficient ✅
claude "Help me design the user database schema"
claude "Write a user registration API based on this structure"
claude "Add login verification logic"

2. Provide Complete Context

# Missing information ❌
claude "This code has an error"

# Complete information ✅
claude "I am writing a web service with Flask. The login endpoint returns a 500 error.
Error message: TypeError: Cannot read property 'name' of undefined
The stack trace shows it occurred at line 45 in app.py.
Help me analyze the cause and fix it."

3. Use Tool Calls

Claude Code can perform various operations:

# Read file
Read file: src/main.py

# Search code
Grep pattern: "def login" path: src/

# Execute command
Bash command: python -m pytest

# Write file
Write file: src/utils.py

# Batch operations
Glob pattern: "**/*.py"

Claude Code Model Selection

Claude Code supports multiple Claude models, which can be selected based on task complexity:

Model	Capability	Speed	Use Cases
Claude Opus 4.6	Highest	Slower	Complex architecture design, deep refactoring
Claude Sonnet 4.6	High	Medium	Daily development, code generation
Claude Haiku 4.5	Medium	Fast	Quick completion, simple tasks

1
2

# Switch model
/model claude-sonnet-4-6

Advanced Features of Claude Code

MCP Server Integration

MCP (Model Context Protocol) allows connecting Claude Code to external tools and data sources:

# Set up MCP servers in the configuration file
# ~/.claude/config.json
{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@anthropic/mcp-server-filesystem"],
      "allowedDirectories": ["~/projects"]
    }
  }
}

CLAUDE.md File

Create a CLAUDE.md file in the project root directory. Claude Code will read it at the start of every session:

# Project Description

## Code Standards
- Use TypeScript strict mode
- Follow Airbnb ESLint configuration

## Architecture Decisions
- Use dependency injection pattern
- All API calls must include error handling

## Preferred Libraries
- State Management: Zustand
- HTTP Client: Ky

Custom Commands (Skills)

Create reusable command templates:

# Create /review-pr command
# ~/.claude/commands/review-pr.md
# Review Pull Request

Please review this PR:
1. Check code quality and style consistency
2. Look for potential security issues
3. Evaluate if test coverage is sufficient
4. Verify documentation is updated

@pr-url

Hooks Configuration

Automatically execute shell commands before or after operations:

# ~/.claude/config.json
{
  "hooks": {
    "PostEdit": [
      {
        "match": "*.ts",
        "command": "npx eslint --fix {file}"
      }
    ]
  }
}

Scheduled Tasks

# Run Claude on a schedule
claude /schedule

# Automate morning PR reviews
claude /loop 8h /pr review

# Automate nightly CI failure analysis
claude /schedule "0 22 * * *" "Analyze today's CI failures"

Multi-Agent Collaboration

# Start multiple agents working in parallel
claude "Start a team to handle this feature:
- Agent 1: Frontend component development
- Agent 2: Backend API development
- Agent 3: Test writing
Coordinate their work and merge the results"

Git and Workflow Integration

Git Workflow

# Commit changes
claude "commit my changes with a descriptive message"

# Create a branch
claude "Create a feature branch to implement user registration"

# Analyze change impact
claude "Analyze what changes exist in the current branch compared to main"

Code Review Process

# 1. Let Claude read the code
claude "Please read all Python files in the src directory to understand the project structure"

# 2. Propose specific review requirements
claude "For the file src/api/v1/users.py:
- Check if input validation is complete
- Look for potential security vulnerabilities
- Evaluate if error handling is reasonable"

# 3. Request fix suggestions
claude "For the issues found, please provide fix solutions one by one"

Debugging Tips

# Provide error information
claude "The following error occurs when running 'python app.py':
TypeError: Cannot read property 'name' of undefined
The stack trace shows the error occurred at line 45.
Help me locate the problem and fix it."

# Provide reproduction steps
claude "1. Run 'npm run dev'
2. Visit http://localhost:3000
3. Click the login button
4. Enter correct credentials and click submit
Expected: Login successful and redirect
Actual: Page freezes with no response
Console shows: Pending POST /api/login"

Domestic LLM Configuration

Claude Code natively uses Claude models but also supports configuring other domestic models as alternatives.

GLM4 Series (Zhipu AI)

Zhipu’s GLM series excels in Chinese understanding and code generation.

GLM4.7 Features:

Ultra-long context support
Powerful Chinese understanding capabilities
Strong code generation capabilities
Tool Calling support

Configuration Method:

# Configure GLM via MCP
# ~/.claude/config.json
{
  "mcpServers": {
    "glm": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-openai"],
      "env": {
        "OPENAI_API_KEY": "your-zhipu-api-key",
        "OPENAI_BASE_URL": "https://open.bigmodel.cn/api/paas/v4"
      }
    }
  }
}

Kimi K2.5 (Moonshot AI)

Kimi K2.5 has unique advantages in long context and multimodal capabilities.

K2.5 Features:

256K ultra-long context
Multimodal understanding
Professional code generation, programming capability comparable to Claude

Pricing:

Input: ¥4.00/MTok
Output: ¥21.00/MTok
Cache Hit: ¥0.70/MTok

Configuration Method:

# Configure Kimi via MCP
# ~/.claude/config.json
{
  "mcpServers": {
    "kimi": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-openai"],
      "env": {
        "OPENAI_API_KEY": "your-kimi-api-key",
        "OPENAI_BASE_URL": "https://api.moonshot.cn/v1"
      }
    }
  }
}

MiniMax M2.5

MiniMax M2.5 is designed specifically for programming, focusing on fast response.

M2.5 Features:

High throughput, low latency
Optimized for production environments
SOTA programming and Agent capabilities
High cost-performance ratio

Configuration Method:

# Configure MiniMax via MCP
# ~/.claude/config.json
{
  "mcpServers": {
    "minimax": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-openai"],
      "env": {
        "OPENAI_API_KEY": "your-minimax-api-key",
        "OPENAI_BASE_URL": "https://api.minimax.chat/v1"
      }
    }
  }
}

Comparison with Other AI Coding Tools

Cursor

Cursor is one of the most popular AI-native IDEs, used by over half of the Fortune 500 companies.

Comparison with Claude Code:

Cursor is a complete AI editor, Claude Code is a CLI tool.
Cursor supports multiple models like GPT-5, Claude, and Gemini.
Claude Code has stronger reasoning capabilities, suitable for complex tasks.
Cursor has deeper AI integration.

Cursor Core Features:

Composer 2: Complete multi-file modifications with a single sentence.
Tab smart completion.
BugBot automatic code review.
Global codebase understanding.

Trae

Trae is an adaptive AI IDE focusing on local-first and privacy protection.

Comparison with Claude Code:

Trae has a Chinese-friendly interface and ample free quota.
Claude Code is more powerful and suitable for complex projects.
Trae is suitable for domestic users to get started quickly.

Trae Core Features:

Builder mode: Create files via conversation.
Chinese interface.
Privacy protection (data is not used for secondary training).
MCP protocol support.

OpenCode

OpenCode is a lightweight terminal AI coding tool developed in Go.

Comparison with Claude Code:

OpenCode is lighter and suitable for simple tasks.
Claude Code is more comprehensive in functionality.
OpenCode supports more model providers.

OpenCode Supported Models:

OpenAI: GPT-4.1, GPT-4.5, GPT-4o, O1, O3
Anthropic: Claude 4 Sonnet/Opus, Claude 3.5/3.7 Sonnet
Google: Gemini 2.5, 2.5 Flash, 2.0 Flash

Comparison Summary

Feature	Claude Code	Cursor	Trae	OpenCode
Type	CLI + Plugin	AI IDE	AI IDE	CLI Tool
Models	Claude	Multi-model	Multi-model	Multi-model
Reasoning	Strongest	Strong	Medium	Strong
Chinese Support	Average	Average	Excellent	Average
Learning Curve	Medium	Low	Low	Medium
Free Quota	Limited	Limited	Ample	Depends on API

Selection Advice:

Complex refactoring and deep tasks → Claude Code
Daily development and IDE experience → Cursor
Domestic users getting started quickly → Trae
Lightweight terminal tasks → OpenCode

Frequently Asked Questions

Q: Does Claude Code require payment?

A: Claude Code itself is free, but it requires a Claude API key. New users have a free quota, and subscribers have a more ample quota.

Q: Can domestic models replace Claude?

A: For daily programming tasks, domestic models are close to Claude’s level. Claude still has an advantage in complex reasoning tasks. Domestic models are cheaper.

Q: Can code written by AI be used directly?

A: No! AI code must be understood by yourself, tested locally, code reviewed, and confirmed safe before use.

Q: How do I choose a Claude model?

A: Use Haiku for simple tasks, Sonnet for daily development, and Opus for complex tasks.

Conclusion

Claude Code is one of the most powerful AI coding tools currently available, particularly suitable for:

Complex code understanding and refactoring
Batch operations across multiple files
Deep debugging and problem analysis
Tasks requiring strong reasoning capabilities

By configuring MCP, Claude Code can also use domestic models (GLM4, Kimi K2.5, MiniMax M2.5) as alternatives. Choose the appropriate tool and model based on task complexity and budget.

OpenCode Best Practices: Chinese Large Model Programming Assistant

wzdushu@gmail.com (wanzi) — Thu, 15 Jan 2026 21:22:42 +0800

OpenCode Overview

What is OpenCode

OpenCode is an open-source AI coding assistant built on VSCode with the following features:

Open Source & Free: Fully open source, self-hostable
Flexible Models: Supports multiple LLM backends
Local First: Protects code privacy
Highly Customizable: Extensible and customizable

Core Features

Feature	Description
Code Completion	Real-time code suggestions (single-line/multi-line)
Code Chat	Q&A on code
Code Generation	Generate code based on descriptions
Code Explanation	Explain the functionality of selected code
Refactoring Suggestions	Provide refactoring and optimization suggestions
Bug Fixing	Intelligently fix code errors

I. OpenCode Installation and Configuration

System Requirements

Operating System:
  - Windows 10/11
  - macOS 11+
  - Linux (Ubuntu 20.04+, CentOS 8+)

Hardware Requirements:
  - CPU: 4 cores+
  - RAM: 8GB+
  - Storage: 2GB available space

Software Dependencies:
  - Node.js 16+ (if building from source)
  - Git

Installation Steps

Method 1: Using Pre-built Versions (Recommended)

# 1. Download OpenCode
# Visit https://github.com/opencode-org/opencode/releases
# Download the installer for your platform

# Linux/macOS
wget https://github.com/opencode-org/opencode/releases/latest/download/opencv-linux-x64.tar.gz
tar -xzf opencv-linux-x64.tar.gz
cd opencv
./opencode

# macOS (using Homebrew Cask)
brew install --cask opencode

# Windows
# Download opencv-setup.exe and run the installer

# 2. Launch OpenCode
# The welcome page will appear on first launch

Method 2: Building from Source

# 1. Clone repository
git clone https://github.com/opencode-org/opencode.git
cd opencv

# 2. Install dependencies
npm install

# 3. Build project
npm run build

# 4. Run
npm run electron

Basic Configuration

settings.json Configuration

{
  // OpenCode Configuration
  "opencode.modelProvider": "openai",
  "opencode.apiKey": "your-api-key",
  "opencode.model": "gpt-4",
  "opencode.temperature": 0.3,
  "opencode.maxTokens": 2048,

  // Code Completion Configuration
  "opencode.autocomplete.enabled": true,
  "opencode.autocomplete.mode": "inline",
  "opencode.autocomplete.debounce": 150,

  // Code Chat Configuration
  "opencode.chat.enabled": true,
  "opencode.chat.contextLength": 10,

  // Privacy Configuration
  "opencode.telemetry.enabled": false,
  "opencode.anonymousUsageId": false,

  // Shortcut Configuration
  "opencode.suggestInline": "Ctrl+Enter",
  "opencode.openChat": "Ctrl+Shift+C",
  "opencode.explainCode": "Ctrl+Shift+E"
}

II. What OpenCode Can Do

1. Intelligent Code Completion

# Single-line completion
def calculate_discount(price, discount_rate):
    return price * (1 - discount_

# OpenCode auto-completes:
# rate)

# Multi-line completion
# Input comment:
# Create a function to calculate the nth term of the Fibonacci sequence

# OpenCode generates complete function:
def fibonacci(n):
    """
    Calculate the nth term of the Fibonacci sequence

    Args:
        n (int): The term number to calculate

    Returns:
        int: The value of the nth term
    """
    if n <= 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)

2. Code Explanation

# Select complex code, OpenCode provides explanation

def merge_sort(arr):
    if len(arr) <= 1:
        return arr

    mid = len(arr) // 2
    left = merge_sort(arr[:mid])
    right = merge_sort(arr[mid:])

    return merge(left, right)

# OpenCode explanation:
# This is an implementation of the merge sort algorithm.
# 1. Base case: If array length <= 1, return directly
# 2. Divide and conquer: Split the array in half, sort recursively
# 3. Merge: Merge two sorted arrays
# Time Complexity: O(n log n)
# Space Complexity: O(n)

3. Code Refactoring

# Original code
def process_user(user):
    if user is not None:
        if user.get("name") is not None:
            if user.get("age") is not None and user["age"] >= 18:
                return True
    return False

# OpenCode refactoring suggestion:
def process_user(user):
    """
    Check if user is a valid adult

    Args:
        user (dict): User info dictionary

    Returns:
        bool: Whether the user is valid
    """
    if not user:
        return False

    name = user.get("name")
    age = user.get("age")

    return bool(name and age >= 18)

4. Bug Detection and Fixing

# Buggy code
def divide_numbers(a, b):
    result = a / b
    return result

# Call
print(divide_numbers(10, 0))  # ZeroDivisionError

# OpenCode detects and fixes:
def divide_numbers(a, b):
    """
    Safely divide two numbers

    Args:
        a (float): Dividend
        b (float): Divisor

    Returns:
        float: Division result

    Raises:
        ValueError: When divisor is 0
    """
    if b == 0:
        raise ValueError("Divisor cannot be zero")

    return a / b

5. Unit Test Generation

# Original function
def add(a, b):
    return a + b

# OpenCode generates unit tests:
import unittest

class TestAdd(unittest.TestCase):
    def test_add_positive_numbers(self):
        """Test adding positive numbers"""
        self.assertEqual(add(2, 3), 5)

    def test_add_negative_numbers(self):
        """Test adding negative numbers"""
        self.assertEqual(add(-2, -3), -5)

    def test_add_mixed_numbers(self):
        """Test adding positive and negative numbers"""
        self.assertEqual(add(2, -3), -1)

    def test_add_zero(self):
        """Test adding zero"""
        self.assertEqual(add(5, 0), 5)

    def test_add_floats(self):
        """Test adding floats"""
        self.assertAlmostEqual(add(2.5, 3.1), 5.6, places=1)

if __name__ == '__main__':
    unittest.main()

III. Introduction to oh-my-opencode

What is oh-my-opencode

oh-my-opencode is an enhanced configuration management tool for OpenCode, providing:

Preset Configurations: Optimized for different scenarios
Model Switching: Quickly switch between different LLMs
Shortcut Bindings: Predefined shortcuts
Theme Customization: Customize appearance

Installing oh-my-opencode

# 1. Clone repository
git clone https://github.com/opencode-org/oh-my-opencode.git ~/.oh-my-opencode

# 2. Install
cd ~/.oh-my-opencode
./install.sh

# 3. Restart OpenCode

Preset Configurations

# List available presets
oh-my-opencode list

# Apply preset
oh-my-opencode apply python-developer
oh-my-opencode apply go-developer
oh-my-opencode apply fullstack-developer

# Create custom preset
oh-my-opencode create my-preset --template python-developer

IV. Domestic LLM Integration

Supported Domestic LLMs

Model	Provider	Features	Recommended Scenarios
GLM-4	Zhipu AI	Strong Chinese understanding	Chinese development
Minimax M2.5	MiniMax	Strong coding ability	Code generation
Kimi K2.5	Moonshot	Long context	Document understanding
Qwen3.5	Alibaba Cloud	Multilingual ability	Full-stack development

1. GLM-4 Configuration

{
  "opencode.modelProvider": "custom",
  "opencode.apiEndpoint": "https://open.bigmodel.cn/api/paas/v4/chat/completions",
  "opencode.apiKey": "your-glm-api-key",
  "opencode.model": "glm-4",
  "opencode.temperature": 0.3,
  "opencode.maxTokens": 4096,
  "opencode.contextLength": 8192,

  // Request Header Configuration
  "opencode.customHeaders": {
    "Authorization": "Bearer your-glm-api-key"
  },

  // Response Format
  "opencode.responseFormat": {
    "type": "text"
  }
}

2. Minimax M2.5 Configuration

{
  "opencode.modelProvider": "custom",
  "opencode.apiEndpoint": "https://api.minimax.chat/v1/text/chatcompletion_v2",
  "opencode.apiKey": "your-minimax-api-key",
  "opencode.model": "abab5.5-chat",
  "opencode.temperature": 0.7,
  "opencode.maxTokens": 2048,

  // Minimax Specific Parameters
  "opencode.customParams": {
    "group": "chat_group_name",
    "role_setting": "You are a professional programming assistant",
    "tokens_to_generate": 2048,
    "temperature": 0.7
  }
}

3. Kimi K2.5 Configuration

{
  "opencode.modelProvider": "custom",
  "opencode.apiEndpoint": "https://api.moonshot.cn/v1/chat/completions",
  "opencode.apiKey": "your-kimi-api-key",
  "opencode.model": "moonshot-v1-8k",
  "opencode.temperature": 0.3,
  "opencode.maxTokens": 8192,

  // Kimi Long Context Configuration
  "opencode.contextWindow": 32000,
  "opencode.includeContext": "full"
}

4. Qwen3.5 Configuration

{
  "opencode.modelProvider": "custom",
  "opencode.apiEndpoint": "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions",
  "opencode.apiKey": "your-qwen-api-key",
  "opencode.model": "qwen-plus",
  "opencode.temperature": 0.5,
  "opencode.maxTokens": 6000,

  // Qwen Specific Parameters
  "opencode.customParams": {
    "top_p": 0.8,
    "enable_search": false
  }
}

Multi-Model Configuration Switching

# Create model configuration files
~/.opencode/models/
├── glm-4.json
├── minimax-m2.5.json
├── kimi-k2.5.json
└── qwen3.5.json

# Quickly switch models
opencode model switch glm-4
opencode model switch minimax-m2.5
opencode model switch kimi-k2.5
opencode model switch qwen3.5

# Set default model
opencode model default qwen3.5

V. Best Practices

1. Prompt Engineering

Characteristics of Good Prompts

A good prompt should:
1. Clear Goal - Clearly state what to do
2. Provide Context - Give relevant background information
3. Specify Format - State the expected output format
4. Give Examples - Provide examples of desired results
5. Set Constraints - Define clear limitations

Example Prompts

# Bad Prompt
Write a sorting function

# Good Prompt
Please write a quick sort function in Python with the following requirements:
1. Input: List of integers
2. Output: Sorted list
3. Handle edge cases: empty list, single-element list
4. Include complete docstrings
5. Add type annotations
6. Include unit test examples

Please follow the Google Python Style Guide

2. Context Management

# Provide sufficient context information

# Bad example - Lacks context
# Help me optimize this function
def process(data):
    result = []
    for item in data:
        result.append(item * 2)
    return result

# Good example - Provides context
# Optimize the following function, requirements:
# Scenario: Processing large amounts of sensor data (millions)
# Current performance: Processing 100k items takes 5 seconds
# Goal: Process 100k items in <1 second
# Constraint: Data order must be preserved
# Language: Python
def process(data):
    result = []
    for item in data:
        result.append(item * 2)
    return result

3. Security Best Practices

Security Practices:
  API Key Management:
    - Store API keys in environment variables
    - Rotate keys regularly
    - Do not commit to code repositories

  Code Review:
    - AI-generated code requires human review
    - Pay special attention to security vulnerabilities
    - Test edge cases

  Sensitive Information:
    - Do not let AI access sensitive code
    - Use anonymized data
    - Configure ignore rules

4. Performance Optimization

{
  // Performance Optimization Configuration
  "opencode.cache.enabled": true,
  "opencode.cache.size": "1GB",
  "opencode.cache.ttl": 86400,

  // Request Optimization
  "opencode.batchRequests": true,
  "opencode.requestTimeout": 30000,
  "opencode.retryAttempts": 3,

  // Response Cache
  "opencode.responseCache.enabled": true,
  "opencode.responseCache.maxSize": 500
}

5. Team Collaboration

Team Collaboration:
  Configuration Sharing:
    - Create team configuration files
    - Unify model selection
    - Unify prompt templates

  Version Control:
    # .opencoderc
    {
      "model": "qwen3.5",
      "temperature": 0.3,
      "presets": {
        "python": "python-dev-preset",
        "go": "go-dev-preset"
      }
    }

  Knowledge Base:
    - Maintain team prompt library
    - Share best practices
    - Regular training

VI. Frequently Asked Questions

Q1: How to Configure API Keys

# Method 1: Environment Variables (Recommended)
export OPENCODE_API_KEY="your-api-key"
export OPENCODE_API_ENDPOINT="https://api.example.com"

# Method 2: Configuration File
~/.opencode/config.json
{
  "apiKey": "your-api-key",
  "apiEndpoint": "https://api.example.com"
}

# Method 3: OpenCode Settings UI
# Settings > OpenCode > API Key

Q2: How to Improve Generation Quality

{
  // Configuration to Improve Generation Quality
  "opencode.temperature": 0.3,        // Reduce randomness
  "opencode.topP": 0.9,               // Control diversity
  "opencode.frequencyPenalty": 0.5,   // Reduce repetition
  "opencode.presencePenalty": 0.3,    // Encourage new topics

  // Increase Context
  "opencode.contextLines": 50,        // Increase number of surrounding lines
  "opencode.includeFileHeader": true,  // Include file header
  "opencode.includeImports": true     // Include imports
}

Q3: Offline Usage Solutions

Offline Solutions:
  Local Models:
    - Use Ollama to run local models
    - Supports Llama, CodeLlama, etc.
    - Configure local API endpoint

  Configuration:
    opencode.apiEndpoint: "http://localhost:11434"
    opencode.model: "codellama:13b"

Summary

As an open-source AI coding assistant, OpenCode has the following advantages:

Open Source & Free: No subscription fees
Flexible Models: Supports multiple LLM backends
Domestic Models: Integrated with GLM, Minimax, Kimi, Qwen
Customizable: Highly configurable
Privacy Protection: Local deployment options

Combined with oh-my-opencode and domestic LLMs, an efficient AI-assisted development environment can be built.

Building an Enterprise Logging Platform with OpenObserve and Fluent Bit

wzdushu@gmail.com (wanzi) — Wed, 12 Nov 2025 09:30:00 +0800

Platform Architecture

Technology Selection

OpenObserve (formerly O2) is an open-source observability platform with significant advantages over traditional ELK solutions:

Low resource consumption: Memory usage is only 1/10 of Elasticsearch, with 90% storage space saved
High performance: Written in Rust, a single node can process 5TB+ logs/day
Full-featured: Integrates logs, metrics, and distributed tracing, replacing Grafana+Loki+Tempo
Strong compatibility: Supports Elasticsearch API, Prometheus API, and OpenTelemetry
Easy deployment: Single binary file with no external dependencies

Fluent Bit is a cloud-native log collector:

Lightweight: Memory usage <1MB, suitable for embedded and container environments
High performance: Multi-threaded asynchronous processing, throughput up to 100MB/s
Rich plugins: Supports 200+ input/output/filter/parser plugins
Kubernetes native: Automatic Pod discovery, label injection, metadata association

Architecture Design

┌─────────────────────────────────────────────────────────────────┐
│                        Business Application Layer                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐       │
│  │  Java App│  │  Go Service │ │ Nginx Logs│ │ System Logs│      │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘       │
│       │             │             │             │               │
│  stdout/stderr   stdout/stderr  /var/log/*   /var/log/*         │
└───────┼─────────────┼─────────────┼─────────────┼───────────────┘
        │             │             │             │
┌───────┴─────────────┴─────────────┴─────────────┴───────────────┐
│                      Kubernetes Cluster                          │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │              DaemonSet: Fluent Bit                      │   │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │   │
│  │  │  Tail Input  │→ │  Filter/Parse│→ │  Buffer      │  │   │
│  │  └──────────────┘  └──────────────┘  └──────────────┘  │   │
│  └─────────────────────────────────────────────────────────┘   │
└──────────────────────────────┬──────────────────────────────────┘
                               │ HTTP/HTTPS
┌──────────────────────────────┴──────────────────────────────────┐
│                    OpenObserve Cluster                           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │  Ingest Node │→ │  Data Storage│→ │  Query Engine│          │
│  │  (Write Opt) │  │  (Parquet)   │  │  (Full Text) │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
└─────────────────────────────────────────────────────────────────┘

Component Versions

Component	Version	Description
OpenObserve	0.15.0+	Stable version, supports distributed
Fluent Bit	4.0.0+	Latest stable version
Kubernetes	1.24+	Production environment
Helm	3.0+	Deployment tool

OpenObserve Deployment

Prerequisites

# 1. Create Namespace
kubectl create namespace logging

# 2. Add OpenObserve Helm repository
helm repo add openobserve https://charts.openobserve.ai
helm repo update

# 3. Create PV/PVC (use distributed storage for production)
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: openobserve-data
  namespace: logging
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: zstack-rbd  # Adjust according to actual SC
  resources:
    requests:
      storage: 500Gi
EOF

Helm Deployment of OpenObserve

Create values file openobserve-values.yaml:

# openobserve-values.yaml

image:
  repository: public.ecr.aws/zinclabs/openobserve
  tag: "0.15.1"
  pullPolicy: IfNotPresent

replicaCount: 3  # 3 nodes recommended for HA in production

# Resource configuration (adjust based on log volume)
resources:
  limits:
    cpu: "8"
    memory: 16Gi
  requests:
    cpu: "4"
    memory: 8Gi

# Data persistence
persistence:
  enabled: true
  existingClaim: openobserve-data
  mountPath: /data

# Environment variables
env:
  - name: ZO_ROOT_USER_EMAIL
    value: "admin@example.com"
  - name: ZO_ROOT_USER_PASSWORD
    valueFrom:
      secretKeyRef:
        name: openobserve-secret
        key: password
  - name: ZO_DATA_DIR
    value: "/data"
  - name: ZO_HTTP_PORT
    value: "5080"
  - name: ZO_MEMORY_CACHE_ENABLED
    value: "true"
  - name: ZO_MEMORY_CACHE_MAX_SIZE
    value: "4096"  # MB
  - name: ZO_COMPRESSION_ENABLED
    value: "true"
  - name: ZO_COMPRESSION_FORMAT
    value: "zstd"
  - name: ZO_PARQUET_COMPRESSION
    value: "zstd"
  - name: ZO_META_STORE
    value: "sqlite"  # Use PostgreSQL for production
  - name: ZO_METRICS_ENABLED
    value: "true"

# Configure PostgreSQL metadata store (recommended for production)
envFrom:
  - secretRef:
      name: postgres-connection

# Service configuration
service:
  type: ClusterIP
  port: 5080
  targetPort: 5080
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

# Ingress configuration
ingress:
  enabled: true
  className: "nginx"
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - host: logs.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: logs-tls
      hosts:
        - logs.example.com

# Pod scheduling
nodeSelector: {}
tolerations: []
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - openobserve
          topologyKey: kubernetes.io/hostname

# Monitoring configuration
monitoring:
  enabled: true
  serviceMonitor:
    enabled: true
    interval: 30s
    namespace: logging

Deploy OpenObserve

# 1. Create password Secret
kubectl create secret generic openobserve-secret \
  --from-literal=password='your-secure-password' \
  -n logging

# 2. Deploy
helm upgrade --install openobserve openobserve/openobserve \
  -n logging \
  -f openobserve-values.yaml \
  --version 0.10.2 \
  --timeout 15m

# 3. Wait for Pods to be ready
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=openobserve \
  -n logging \
  --timeout=300s

# 4. Check status
kubectl get pods -n logging
kubectl logs -f deployment/openobserve -n logging

# 5. Access Web UI
echo "Access URL: https://logs.example.com"
echo "Default user: admin@example.com"

Fluent Bit Deployment

Install Helm Chart

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
helm pull fluent/fluent-bit --version 0.50.0
helm install fluent-bit ./fluent-bit-0.50.0.tgz --namespace uganda-prod

Adjust DaemonSet Configuration

spec:
  volumes:
    - name: config
      configMap:
        name: fluent-bit
        defaultMode: 420
  containers:
    resources:
      requests:
        cpu: "1"
        memory: 2Gi
      limits:
        cpu: "2"
        memory: 4Gi
    volumeMounts:
      - name: config
        mountPath: /fluent-bit/etc/conf/custom_parsers.conf
        subPath: custom_parsers.conf
      - name: config
        mountPath: /fluent-bit/etc/conf/multiline_parsers.conf
        subPath: multiline_parsers.conf

Fluent Bit Collection Rule Adjustments

Complete ConfigMap Configuration

kind: ConfigMap
apiVersion: v1
metadata:
  name: fluent-bit-config
  namespace: uganda-prod
  labels:
    app: fluent-bit
    tier: logging
data:
  # ---------------------------------------------------------------------------
  # 1. Custom Parsers
  # Used to extract specific log fields, e.g., extract log level (INFO, ERROR, etc.) from Java logs
  # ---------------------------------------------------------------------------
  custom_parsers.conf: |
    [PARSER]
        Name        java_log_level
        Format      regex
        # Match format: 2023-10-27 10:00:00.123 INFO ...
        # The 'level' capture group will contain the log level
        Regex       ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}\s+(?<level>[A-Z]+)    

  # ---------------------------------------------------------------------------
  # 2. Multiline Parsers
  # Specifically for handling multi-line logs like Java stack traces to prevent stacks from being split into multiple records
  # ---------------------------------------------------------------------------
  multiline_parsers.conf: |
    [MULTILINE_PARSER]
        Name          java_md_multiline
        Type          regex
        # Rule 1: If line starts with date (e.g., 2023-...), consider it start of new log
        rule      "start_state"   "/^\d{4}-\d{2}-\d{2}/"           "cont"
        # Rule 2: If line doesn't start with date, consider it continuation of previous line (stack part)
        rule      "cont"        "/^(?!\d{4}-\d{2}-\d{2}).+/"     "cont"
        # Flush timeout (seconds), force output current buffered multi-line log after timeout
        flush_timeout 5    

  # ---------------------------------------------------------------------------
  # 3. Main Configuration File
  # ---------------------------------------------------------------------------
  fluent-bit.conf: |
    [SERVICE]
        Daemon              Off
        Flush               1
        Log_Level           info
        # Load standard and custom parsers
        Parsers_File        /fluent-bit/etc/conf/parsers.conf
        Parsers_File        /fluent-bit/etc/conf/custom_parsers.conf
        Parsers_File        /fluent-bit/etc/conf/multiline_parsers.conf
        
        # Enable built-in HTTP server for health checks and metrics exposure (/api/v1/metrics)
        HTTP_Server         On
        HTTP_Listen         0.0.0.0
        HTTP_Port           2020
        Health_Check        On

        # File system buffer configuration (prevent data loss when backend is unavailable)
        storage.path              /var/log/flb_storage
        storage.sync              normal
        storage.checksum          off
        storage.backlog.mem_limit 200M

    # -----------------------------------------------------------------------
    # [INPUT] Tail file collection
    # Collect standard container logs generated by Docker/Containerd
    # -----------------------------------------------------------------------
    [INPUT]
        Name                tail
        Path                /var/log/containers/*.log
        # Enable built-in multiline parsing (docker, cri format)
        multiline.parser    docker, cri
        Tag                 kube.*
        Mem_Buf_Limit       500MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Persist offset database, resume reading from checkpoint after restart
        DB                  /var/log/flb_kube.db
        DB.Sync             Normal
        Rotate_Wait         30
        Read_from_Head      Off

    # -----------------------------------------------------------------------
    # [FILTER] Stage 1: Multiline merging
    # Merge multiline logs before getting K8s metadata to ensure stack traces are handled as single records
    # -----------------------------------------------------------------------
    [FILTER]
        Name                multiline
        Match               kube.*
        multiline.key_content log
        multiline.parser    java_md_multiline

    # -----------------------------------------------------------------------
    # [FILTER] Stage 2: K8s metadata enhancement
        Call K8s API to get Pod details (Namespace, Pod Name, Labels, etc.)
    # -----------------------------------------------------------------------
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           Off       # Don't try to parse log field as JSON (avoid performance overhead or errors)
        Keep_Log            On        # Keep original log field
        K8S-Logging.Parser  On        # Allow Pod Annotation to specify parser
        K8S-Logging.Exclude Off       # Don't exclude logs with specific annotation

    # -----------------------------------------------------------------------
    # [FILTER] Stage 3: Log content parsing
    # Use custom parser to extract 'level' field
    # -----------------------------------------------------------------------
    [FILTER]
        Name                parser
        Match               kube.*
        Key_Name            log
        Parser              java_log_level
        Reserve_Data        On        # Keep original data
        Preserve_Key        On        # Keep original log key

    # -----------------------------------------------------------------------
    # [FILTER] Stage 4: Field cleaning
    # Copy extracted level field, remove unnecessary temporary fields
    # -----------------------------------------------------------------------
    [FILTER]
        Name                modify
        Match               kube.*
        Copy                level level
        Remove              _p
        Remove              stream
        Remove              time

    # -----------------------------------------------------------------------
    # [FILTER] Stage 5: Structure restructuring (Nest Lift)
    # Lift fields under kubernetes object to top level with 'k8s_' prefix
    # Purpose: Flatten data structure for easier rule matching
    # -----------------------------------------------------------------------
    [FILTER]
        Name                nest
        Match               kube.*
        Operation           lift
        Nested_under        kubernetes
        Add_prefix          k8s_

    # -----------------------------------------------------------------------
    # [FILTER] Stage 6: Remove redundant metadata
    # Delete large fields not needed to be sent to backend (e.g., annotations, docker_id)
    # -----------------------------------------------------------------------
    [FILTER]
        Name                modify
        Match               kube.*
        Remove              k8s_annotations
        Remove              k8s_docker_id
        Remove              k8s_container_hash

    # -----------------------------------------------------------------------
    # [FILTER] Stage 7: Structure restructuring (Nest Nest)
    # Repackage all 'k8s_' prefixed fields back under 'kubernetes' object, removing prefix
    # Purpose: Restore clean nested structure while cleaning up useless fields
    # -----------------------------------------------------------------------
    [FILTER]
        Name                nest
        Match               kube.*
        Operation           nest
        Wildcard            k8s_*
        Nested_under        kubernetes
        Remove_prefix       k8s_

    # =======================================================================
    # [FILTER] Stage 8: Dynamic routing (Rewrite Tag) - First hop: Split by environment
    # Modify Tag based on Namespace name to route logs to different processing streams
    # Syntax: Rule $field_name regex newTag keep_original(boolean)
    # =======================================================================
    
    # Route: uganda-uat environment
    [FILTER]
        Name                rewrite_tag
        Match               kube.*
        Rule                $kubernetes['namespace_name'] ^uganda-uat$ uganda-uat.temp false
        Emitter_Name        re_emitted_uganda-uat-temp

    # Route: uganda-test environment
    [FILTER]
        Name                rewrite_tag
        Match               kube.*
        Rule                $kubernetes['namespace_name'] ^uganda-test$ uganda-test.temp false
        Emitter_Name        re_emitted_uganda-test-temp

    # Route: uganda-prod environment (current deployment namespace)
    [FILTER]
        Name                rewrite_tag
        Match               kube.*
        Rule                $kubernetes['namespace_name'] ^uganda-prod$ uganda-prod.temp false
        Emitter_Name        re_emitted_uganda-prod-temp

    # Route: uganda-offline environment
    [FILTER]
        Name                rewrite_tag
        Match               kube.*
        Rule                $kubernetes['namespace_name'] ^uganda-offline$ uganda-offline.temp false
        Emitter_Name        re_emitted_uganda-offline-temp

    # =======================================================================
    # [FILTER] Stage 9: Dynamic routing (Rewrite Tag) - Second hop: Split by service
    # Secondary routing for specific containers in specific environments for fine-grained index isolation
    # =======================================================================

    # Example: UAT environment -> lms-backend service
    [FILTER]
        Name                rewrite_tag
        Match               uganda-uat.temp
        Rule                $kubernetes['container_name'] ^lms-backend$ uganda-uat-lms-backend false
        Emitter_Name        re_emitted_uganda-uat-lms-backend

    # Example: UAT environment -> other-service service (needs complete configuration)
    [FILTER]
        Name                rewrite_tag
        Match               uganda-uat.temp
        Rule                $kubernetes['container_name'] ^other-service$ uganda-uat-other-service false
        Emitter_Name        re_emitted_uganda-uat-other-service

    # Note: Production and other environments need similar container name filtering rules added
    # For example:
    # [FILTER]
    #     Name rewrite_tag
    #     Match uganda-prod.temp
    #     Rule $kubernetes['container_name'] ^payment-service$ uganda-prod-payment-service false
    #     Emitter_Name re_emitted_uganda-prod-payment-service

    # =======================================================================
    # [OUTPUT] Output plugin configuration
    # Send filtered logs to OpenObserve
    # =======================================================================

    # Output: UAT LMS Backend
    [OUTPUT]
        Name                http
        Match               uganda-uat-lms-backend
        URI                 /api/39NVPcXSEBOwGM5UnceQ35hQFNB/lms_backend/_json
        Host                openobserve.uganda-uat.svc.cluster.local
        Port                5080
        tls                 Off
        Format              json
        Json_date_key       _timestamp
        Json_date_format    iso8601
        
        HTTP_User           ops@test.com
        HTTP_Passwd         ROPe50N4BJjovJiT 
        
        compress            gzip
        Retry_Limit         False       # Infinite retry until successful
        net.connect_timeout 10
        net.io_timeout      30

    # Output: UAT Other Service (example)
    [OUTPUT]
        Name                http
        Match               uganda-uat-other-service
        URI                 /api/39NVPcXSEBOwGM5UnceQ35hQFNB/other_service/_json
        Host                openobserve.uganda-uat.svc.cluster.local
        Port                5080
        tls                 Off
        Format              json
        Json_date_key       _timestamp
        Json_date_format    iso8601
        HTTP_User           devops@test.com
        HTTP_Passwd         ROPe50N4BJjovJiT
        compress            gzip
        Retry_Limit         False

    # Default output (optional): Capture logs that don't match any specific rules to prevent data loss
    # [OUTPUT]
    #     Name http
    #     Match uganda-prod.temp
    #     URI /api/.../default/_json
    #     ...

Configuration Explanation

Filter Stage: Use rewrite_tag filter to filter logs by Namespace and container name
Output Stage: Send filtered logs to OpenObserve in different environments via HTTP
Multiline Logs: Use multiline filter to handle Java exception stacks
Log Optimization: gzip compression, retry on failure, TCP timeout configuration

Multiline Merge Rule Explanation

State	Matching Rule	Description
start_state	`/^\d{4}-\d{2}-\d{2}/`	Starts with date → New log starts, enters cont state
cont	`/^(?!\d{4}-\d{2}-\d{2}).+/`	Doesn’t start with date → Continue appending to previous log

Note: After modifying the ConfigMap, the fluent-bit service needs to be restarted.

Data Management

Create Data Stream

# Method 1: Create via API
curl -X POST "https://logs.example.com/api/demo/streams" \
  -u "admin@example.com:password" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "app-logs",
    "storage_type": "memory",
    "stream_type": "logs"
  }'

# Method 2: Auto-create (first write auto-creates, we use this method)
# Fluent Bit will auto-create new stream on first write

Set Data Retention Policy

OpenObserve’s dashboard allows setting retention policies based on index, with main strategies as follows:

Time-based retention
Size-based retention
Hybrid strategy (3650 days or 30TB, whichever comes first, we use this approach. Since it’s a financial service, we set retention for 10 years)

Data Archive Configuration

# Enable S3 archiving
env:
  - name: ZO_S3_STORE_ENABLED
    value: "true"
  - name: ZO_S3_STORE_BUCKET
    value: "logs-archive"
  - name: ZO_S3_STORE_REGION
    value: "us-east-1"
  - name: ZO_S3_STORE_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: s3-credentials
        key: access-key
  - name: ZO_S3_STORE_SECRET_KEY
    valueFrom:
      secretKeyRef:
        name: s3-credentials
        key: secret-key

Data Compression

# OpenObserve auto-compression
env:
  - name: ZO_COMPRESSION_ENABLED
    value: "true"
  - name: ZO_COMPRESSION_FORMAT
    value: "zstd"  # zstd/gzip/snappy
  - name: ZO_COMPRESSION_LEVEL
    value: "3"  # 1-19, higher compression ratio but higher CPU consumption

Query and Analysis

OpenObserve provides a powerful query engine supporting SQL mode and native query language (VQL). The following introduces basic queries and aggregation queries in different scenarios to help you quickly locate key logs.

💡 Core Tips

Mode Switch: Aggregation queries are recommended to be executed in SQL mode.

Table Name Convention: Stream names (table names) must be enclosed in double quotes, for example "lms".

Time Range: All queries are limited by the time selector in the upper right corner of the UI, please ensure the time range covers the target data.

1. Basic Search

Quickly locate a single log or specific event.

1.1 Full-text Index Search

Automatically scans all text fields, suitable for fuzzy searching.

`1`	`match_all('timeout')`

Scenario: Not sure which field the error appears in, quickly search for keywords globally.

1.2 Field-specific Search

Perform exact or inclusion matching on specific fields for better performance.

`1`	`str_match(log, 'tenant-backend')`

Scenario: Filter logs for specific services (like tenant-backend) or specific levels (str_match(level, 'ERROR')).

1.3 Multi-condition Combination Query

Use logical operators (AND, OR, NOT) to build complex filtering rules.

1
2

(level = 'INFO' OR level = 'ERROR') 
AND str_match(log, 'AuthInterceptor')

Scenario: Only view logs from the authentication module, including both normal and error processes.

2. Aggregation Queries 🚀

Through SQL’s GROUP BY and aggregate functions (COUNT, SUM, AVG, MAX, MIN), massive logs can be converted into statistical chart data.

2.1 Count by Log Level

Count the number of different log levels to quickly determine system health.

SELECT level, COUNT(*) as log_count 
FROM "lms" 
GROUP BY level 
ORDER BY log_count DESC

Output Example:

level log_count

INFO 15420

ERROR 32

WARN 105

2.2 Time Window Trend Analysis

Combine with the date_trunc function to count logs within a time unit, used for drawing trend charts.

SELECT date_trunc('minute', _timestamp) as time_window, COUNT(*) as hits
FROM "lms"
WHERE str_match(log, 'payment')
GROUP BY time_window
ORDER BY time_window ASC

Scenario: Observe the fluctuation of “payment” related logs per minute to identify sudden traffic or failure time points.
Note: 'minute' can be replaced with 'hour', 'day', etc.

2.3 Group by Service/Container

Analyze which microservice generates the most logs or errors.

SELECT kubernetes.container_name, COUNT(*) as total_logs
FROM "lms"
WHERE level = 'ERROR'
GROUP BY kubernetes.container_name
ORDER BY total_logs DESC
LIMIT 5

Scenario: Find the top 5 containers generating the most errors, prioritize troubleshooting.

2.4 Multi-dimensional Cross Analysis

Group by two or more dimensions simultaneously for deep dive analysis.

SELECT 
  date_trunc('hour', _timestamp) as hour,
  level,
  COUNT(*) as count
FROM "lms"
WHERE str_match(log, 'database')
GROUP BY hour, level
ORDER BY hour ASC, level

Scenario: View the distribution of INFO and ERROR database-related logs per hour.

2.5 Numeric Field Statistics

If logs contain extracted numeric fields (like response time duration), mathematical calculations can be performed.

SELECT 
  AVG(duration) as avg_latency,
  MAX(duration) as max_latency,
  MIN(duration) as min_latency
FROM "lms"
WHERE str_match(log, 'API_REQUEST')

Scenario: Calculate average and maximum latency for specific API requests to evaluate performance bottlenecks.

3. Advanced Techniques and Best Practices

Technique	Description	Example
Limit Result Set	Avoid returning too much data causing browser lag	Add `LIMIT 100`
Deduplication Count	Count unique error message types	`COUNT(DISTINCT log)`
Alias Optimization	Make output column names more readable	`COUNT(*) as "Total Errors"`
Null Value Handling	Exclude records with empty fields	`WHERE level IS NOT NULL`
Regex Matching	More flexible matching than `str_match`	`REGEXP_MATCH(log, 'Error.*\d+')`

⚠️ Performance Recommendations

Filter before aggregating: Be sure to narrow down the data range as much as possible in the WHERE clause (specifying time, keywords) before performing GROUP BY, which can significantly improve query speed.
Time granularity: When querying over large time ranges (like 7 days), use date_trunc('hour', ...) or date_trunc('day', ...), avoid using 'second' which causes too many data points.
Field indexing: For fields commonly used in GROUP BY (like level, container_name), it’s recommended to enable indexing in Stream Settings for best performance.

OpenObserve Optimization

# Memory cache
env:
  - name: ZO_MEMORY_CACHE_ENABLED
    value: "true"
  - name: ZO_MEMORY_CACHE_MAX_SIZE
    value: "8192"  # 8GB

# Query cache
  - name: ZO_QUERY_CACHE_ENABLED
    value: "true"
  - name: ZO_QUERY_CACHE_MAX_SIZE
    value: "4096"  # 4GB

# Parallel queries
  - name: ZO_QUERY_WORKER_THREADS
    value: "8"

Storage Optimization

# Parquet columnar storage
env:
  - name: ZO_PARQUET_COMPRESSION
    value: "zstd"
  - name: ZO_PARQUET_ROW_GROUP_SIZE
    value: "1048576"  # 1MB

# Data partitioning
  - name: ZO_PARTITION_ENABLED
    value: "true"
  - name: ZO_PARTITION_FIELDS
    value: "_timestamp,kubernetes.namespace_name"

Monitoring and Alerting

There are two parts that need monitoring: one is the OpenObserve service monitoring itself, and the other is business log monitoring. For monitoring of OpenObserve and Fluent Bit services themselves, Prometheus monitoring can be used directly. For core business order log monitoring, OpenObserve’s built-in alerting mechanism can meet the requirements directly.

Prometheus Metrics Collection

# Fluent Bit ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit
  endpoints:
    - port: metrics
      interval: 30s

# OpenObserve ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: openobserve
  namespace: logging
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: openobserve
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

Alert Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: logging-alerts
  namespace: logging
spec:
  groups:
    - name: logging
      rules:
        # Fluent Bit log loss alert
        - alert: FluentBitLogsDropping
          expr: rate(fluentbit_output_proc_records_failed_total[5m]) > 100
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Fluent Bit Log Loss"
            description: "Lost {{ $value }} logs in the last 5 minutes"

        # OpenObserve storage alert
        - alert: OpenObserveDiskSpaceLow
          expr: (node_filesystem_avail_bytes{mountpoint="/data"} / node_filesystem_size_bytes{mountpoint="/data"}) < 0.1
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "OpenObserve Disk Space Low"
            description: "Less than 10% disk space remaining"

        # Log ingestion delay
        - alert: LoggingIngestLag
          expr: openobserve_ingest_delay_seconds > 300
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Log Ingestion Lag"
            description: "Log lag is {{ $value }} seconds"

Business Log Alerts

This involves several key parts:

Alert Message Template

Settings –> Templates –> Add template, define Webhook alert message format, note that the message needs to be configured differently for different chat tools

Create: cdi-prod-template

    {
      "msgtype": "text",
      "text": {
        "content": "🚨 [Alert Notification]\nAlert Name: {alert_name}\nStream Name: {stream_name}\nSeverity: Error\nLog Info: {rows:1}\nTrigger Time: {alert_trigger_time_str}"
      }
    }

Note: The message here needs to be configured differently for different chat tools. My configuration here is for WeChat Work. For more information, please refer to https://openobserve.ai/docs/user-guide/management/templates/ to configure a suitable alert template for your needs.

Alert Address

Add a message assistant to the WeChat Work group to get the push message Webhook address, then add it in OpenObserve UI Settings –> Address –> Add Address.

Alert Rule Configuration

This needs to be determined based on internal R&D team communication, business log matching rules. Daily alerts should only match ERROR level logs. For core orders and risk control processes, phone alerts are recommended:

OpenObserve Open Source Edition Risks and Considerations

RBAC Permission Control

Open Source Edition Limitations:

No granular RBAC control (Enterprise edition supports)
All users have the same permissions
Cannot isolate data by organization/project

Solutions:

# 1. Use multi-tenancy (Stream-level isolation)
# Create separate Streams for each team
curl -X POST "https://logs.example.com/api/demo/streams" \
  -d '{"name":"team-a-logs"}'
curl -X POST "https://logs.example.com/api/demo/streams" \
  -d '{"name":"team-b-logs"}'

# 2. Use reverse proxy for permission control
# Nginx routes users to different Streams based on user
location /api/team-a/ {
  internal;
  proxy_pass http://openobserve.logging.svc/api/team-a-logs/;
}

# 3. Use API Key for simple authentication
# Generate separate API Key for each team

Here, since there are many projects, permission control rules are usually added at the reverse proxy, such as restricting IP and Host for POST requests.

Data Security

Risk Points:

Open source edition has no field-level encryption
Sensitive information may be stored in plain text
Missing audit logs

Best Practices:

# 1. Fluent Bit data masking
[FILTER]
    Name                modify
    Match               kube.*
    Remove              password,token,secret,key,ssn
    Rename              message  log_message

# 2. Use Lua filter for masking
[FILTER]
    Name                lua
    Match               kube.*
    Script              mask.lua
    Call                mask_sensitive

# mask.lua:
function mask_sensitive(tag, timestamp, record)
    if record["message"] ~= nil then
        record["message"] = string.gsub(record["message"], "password=%S+", "password=***")
    end
    return 1, timestamp, record
end

High Availability Risks

Single Point of Failure:

Open source edition has no automatic failover
Node downtime may cause data loss

Solutions:

# 1. Deploy multiple replicas
replicaCount: 3

# 2. Use shared storage
persistence:
  enabled: true
  storageClass: "nfs-client"  # Use distributed storage, here using Ceph RBD

# 3. Configure pod anti-affinity
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app.kubernetes.io/name
              operator: In
              values:
                - openobserve
        topologyKey: kubernetes.io/hostname

Capacity Planning

Log Volume	Fluent Bit Resources	OpenObserve Resources	Storage Space/Month
10GB/day	100m CPU / 100Mi	2 CPU / 4Gi	100GB
100GB/day	200m CPU / 200Mi	4 CPU / 8Gi	1TB
1TB/day	500m CPU / 500Mi	8 CPU / 16Gi	10TB

Summary

The log platform solution based on OpenObserve and Fluent Bit has the following advantages:

Cost Advantage: Compared to ELK solutions, storage costs are reduced by 90% and computing resources by 70%
High Performance: A single node supports 5TB+/day log ingestion with query responses <100ms
Simple and Easy to Use: Deployment time <30 minutes with low learning curve
Cloud Native: Kubernetes native integration with automatic scaling

It has been running stably in production for half a year with 5TB+ data volume, and query responses are stable within 200ms, basically meeting most business needs. If there are strong requirements for RBAC, it is recommended to directly purchase the commercial version or abandon this solution.

Kustomize Private Deployment of Multi-Scenario, Multi-Environment Container Services

wzdushu@gmail.com (wanzi) — Sat, 25 Oct 2025 10:30:00 +0800

I have been using Helm or native kubectl YAML combined with CI and ArgoCD to deploy applications for a long time. Recently, while deploying private services for a major financial client, I found the client’s environment to be quite complex, involving multiple countries and multiple sets of environments, such as Test, UAT, and Prod—about thirty environments in total. Modifying values and YAML files every time was exhausting. This time, I plan to use Kustomize to refactor our existing application deployment.

Kustomize Introduction

What is Kustomize

Kustomize is a Kubernetes-native configuration management tool that manages YAML resources through a templating approach:

Core Features:

No template syntax, pure YAML
Declarative configuration management
Supports multi-environment management
GitOps friendly
Integrated into kubectl 1.14+

Comparison with Helm:

Feature	Kustomize	Helm
Template Syntax	None (Pure YAML)	Go Template
Learning Curve	Low	Medium-High
Multi-Environment Management	Base + Overlay	Values files
Package Management	None	Chart Repository
Version Management	Git	Chart Version
Suitable Scenarios	Application Developer	Application Distributor

Basic Concepts

kustomization.yaml
├── bases/          # Base configuration (reuse)
├── overlays/       # Environment overrides
│   ├── dev/
│   ├── staging/
│   └── prod/
├── patches/        # Configuration patches
└── resources/      # Static resources

Differences between Kustomize and Helm

Design Philosophy

Kustomize Design Philosophy:

# Declarative configuration management
# Does not use templates, but modifies configuration via Overlay

# Base: Base configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: app
          image: my-app:latest

# Overlay: Environment differences
# overlays/production/kustomization.yaml
bases:
  - ../../base
patchesStrategicMerge:
  - |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-app
    spec:
      replicas: 10

Helm Design Philosophy:

# Templated configuration management
# Uses Go Template

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
        - name: app
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

# values.yaml (Different values for different environments)
replicaCount: 1
image:
  repository: my-app
  tag: latest

# values-production.yaml
replicaCount: 10
image:
  repository: my-app
  tag: v1.2.3

Feature Comparison

Feature	Kustomize	Helm
Templates	No templates, pure YAML	Go Template
Learning Curve	Low, only need to know YAML	Medium-High, need to learn template syntax
Reusability	Base + Overlay	Chart + Values
Debugging	kubectl kustomize	helm template –dry-run
Version Management	Git version control	Chart version number
Environment Management	Overlay directories	Multiple Values files
Package Distribution	None (use Git)	Chart repository
CI/CD Integration	Native kubectl	Requires helm installation
Rollback	kubectl rollout	helm rollback
Dependency Management	No direct support	Chart dependencies

Applicable Scenarios

Scenarios for choosing Kustomize:

Application developers managing their own K8s resources
Need GitOps workflows
Large configuration differences across environments
Do not want to learn template languages
Team is familiar with YAML and kubectl

Scenarios for choosing Helm:

Application distributors (software vendors)
Need package management and version control
Relatively fixed configuration
Complex application dependencies
Quick deployment of third-party applications

Deploying K8s Applications with Kustomize

Project Structure

my-app/
├── base/                           # Base configuration
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap.yaml
│   ├── serviceaccount.yaml
│   └── kustomization.yaml
├── overlays/                       # Environment configuration
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   ├── deployment-replica.yaml
│   │   ├── deployment-resources.yaml
│   │   └── configmap-env.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── deployment-replica.yaml
│   └── production/
│       ├── kustomization.yaml
│       ├── deployment-replica.yaml
│       ├── deployment-resources.yaml
│       └── configmap-prod.yaml
└── scripts/
    └── deploy.sh

Creating Base Configuration

base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
    version: v1.0.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: my-app
        version: v1.0.0
    spec:
      serviceAccountName: my-app
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: my-app
          image: my-app:latest
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          env:
            - name: JAVA_OPTS
              value: "-Xms512m -Xmx512m"
            - name: SPRING_PROFILES_ACTIVE
              value: "prod"
          resources:
            requests:
              cpu: 100m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - name: config
              mountPath: /config
            - name: logs
              mountPath: /logs
      volumes:
        - name: config
          configMap:
            name: my-app-config
        - name: logs
          emptyDir: {}

base/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: my-app

base/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-config
data:
  application.yml: |
    server:
      port: 8080
    spring:
      application:
        name: my-app
    logging:
      level:
        root: INFO

base/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app

base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Referenced resource files
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml
  - serviceaccount.yaml

# Common labels (added to all resources)
commonLabels:
  app.kubernetes.io/name: my-app
  app.kubernetes.io/instance: my-app
  app.kubernetes.io/managed-by: kustomize
  app.kubernetes.io/version: v1.0.0

# Common annotations
commonAnnotations:
  createdBy: kustomize
  env: base

# Name prefix
# namePrefix: dev-

# Name suffix
# nameSuffix: -v1

Overlay Environment Configuration

overlays/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Reference base
namespace: production
bases:
  - ../../base

# Replica count override
replicas:
  - name: my-app
    count: 10

# Image override
images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

# ConfigMap generator
configMapGenerator:
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml

# Secret generator
secretGenerator:
  - name: my-app-secret
    behavior: create
    envs:
      - .env.production

# Resource patches
patchesStrategicMerge:
  - deployment-resources.yaml
  - deployment-probes.yaml
  - pvc.yaml

# Namespace
namespace: production

# Common labels override
commonLabels:
  environment: production
  tier: backend

# Common annotations override
commonAnnotations:
  env: production
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

overlays/production/deployment-resources.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: my-app
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: 2000m
              memory: 4Gi

overlays/production/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 50Gi

overlays/production/application-production.yml

server:
  port: 8080

spring:
  application:
    name: my-app
  datasource:
    url: jdbc:mysql://mysql-prod.svc:3306/mydb
    username: prod_user
    password: ${DB_PASSWORD}

logging:
  level:
    root: INFO
    com.example: DEBUG
  file:
    path: /logs

overlays/production/.env.production

1
2
3

DB_PASSWORD=prod_secure_password_123
API_KEY=prod_api_key_xyz
REDIS_PASSWORD=prod_redis_pass

Deployment Commands

# Build configuration (do not apply, view final YAML)
kubectl kustomize overlays/production

# Save rendered result
kubectl kustomize overlays/production > rendered-prod.yaml

# View differences
kubectl diff -k overlays/production

# Apply directly
kubectl apply -k overlays/production

# Delete resources
kubectl delete -k overlays/production

# View resources
kubectl get all -l app.kubernetes.io/name=my-app -n production

Unified Deployment Management Configuration

1. Namespace Management

Method 1: Specify in Overlay

1
2

# overlays/production/kustomization.yaml
namespace: production

Method 2: Manage as Resource

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - namespace.yaml

# overlays/production/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    environment: production
    name: production

2. Secret Management

Method 1: Environment Variable File

# overlays/production/kustomization.yaml
secretGenerator:
  - name: my-app-secret
    type: Opaque
    envs:
      - .env.production

Method 2: Literals

secretGenerator:
  - name: my-app-secret
    type: Opaque
    literals:
      - DB_PASSWORD=prod123
      - API_KEY=xyz789

Method 3: Command Generation

secretGenerator:
  - name: tls-cert
    type: kubernetes.io/tls
    files:
      - tls.crt=tls.crt
      - tls.key=tls.key

Method 4: Reference from Existing Secret

# overlays/production/kustomization.yaml
generatorOptions:
  disableNameSuffixHash: true

resources:
  - secret.yaml

# overlays/production/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
type: Opaque
stringData:
  DB_PASSWORD: prod123

Method 5: Sealed Secrets (Recommended for Production)

# Generate sealed secret locally
kubeseal -f secret.yaml -w sealed-secret.yaml

# Reference in kustomization
resources:
  - sealed-secret.yaml

3. PVC Management

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - pvc-logs.yaml
  - pvc-data.yaml

# overlays/production/pvc-logs.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd  # Use SSD storage class
  resources:
    requests:
      storage: 50Gi

# overlays/production/pvc-data.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
spec:
  accessModes:
    - ReadWriteMany  # Multi-node shared
  storageClassName: nfs-storage
  resources:
    requests:
      storage: 100Gi

4. Image Version Management

Method 1: Direct Specification

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

Method 2: Use SHA256

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    digest: sha256:abc123def456...  # Safest method

Method 3: Multiple Images

images:
  - name: nginx
    newName: harbor.example.com/library/nginx
    newTag: 1.21-alpine
  - name: redis
    newName: harbor.example.com/library/redis
    newTag: 6.2-alpine
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

Method 4: CI/CD Dynamic Update

# In CI Pipeline
export IMAGE_TAG=$(git rev-parse --short HEAD)
cd overlays/production
kustomize edit set image my-app=harbor.example.com/prod/my-app:$IMAGE_TAG
kubectl apply -k .

5. Node Affinity Configuration

# overlays/production/deployment-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      affinity:
        # Node affinity
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - c5.2xlarge
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values:
                      - us-west-1a
        # Pod affinity
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - mysql
                topologyKey: kubernetes.io/hostname

Reference in kustomization:

1
2

patchesStrategicMerge:
  - deployment-affinity.yaml

6. Taint and Toleration Configuration

# overlays/production/deployment-taints.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      # Tolerate dedicated node taints
      tolerations:
        - key: workload
          operator: Equal
          value: dedicated
          effect: NoSchedule
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 300
        # Tolerate GPU nodes
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

7. Replica Count Management

# Method 1: replicas field (Recommended)
replicas:
  - name: my-app
    count: 10
  - name: my-app-sidecar
    count: 2

# Method 2: Patch method
# overlays/production/deployment-replica.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 10

8. ConfigMap Management

# Method 1: configMapGenerator (Recommended)
configMapGenerator:
  # Generate from file
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml
      - logback.xml=logback-production.xml

  # Generate from literals
  - name: my-app-env
    behavior: create
    literals:
      - SPRING_PROFILES_ACTIVE=production
      - LOG_LEVEL=INFO
      - JAVA_OPTS=-Xms2g -Xmx4g

  # Generate from env file
  - name: my-app-file-env
    behavior: create
    envs:
      - .env.production

# Method 2: Direct resource reference
resources:
  - configmap.yaml

Complete Production Environment Example

overlays/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Namespace
namespace: production

# Reference base
bases:
  - ../../base

# Additional resources
resources:
  - namespace.yaml
  - pvc-logs.yaml
  - pvc-data.yaml
  - hpa.yaml
  - service-monitor.yaml
  - network-policy.yaml

# Replica count
replicas:
  - name: my-app
    count: 10

# Image version
images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

# ConfigMap
configMapGenerator:
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=config/application-production.yml
  - name: my-app-env
    behavior: merge
    literals:
      - ENV=production
      - LOG_LEVEL=INFO

# Secret
secretGenerator:
  - name: my-app-secret
    behavior: create
    envs:
      - .env.production

# Patches
patchesStrategicMerge:
  - deployment-resources.yaml
  - deployment-affinity.yaml
  - deployment-taints.yaml
  - deployment-probes.yaml

# Common labels
commonLabels:
  environment: production
  tier: backend
  team: platform

# Common annotations
commonAnnotations:
  env: production
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

CI/CD Integration

GitLab CI Example

# .gitlab-ci.yml
stages:
  - build
  - deploy

variables:
  REGISTRY: harbor.example.com
  APP_NAME: my-app

build:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker build -t $REGISTRY/$APP_NAME:$CI_COMMIT_SHA .
    - docker push $REGISTRY/$APP_NAME:$CI_COMMIT_SHA
  only:
    - main
    - develop

deploy:dev:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: dev
  script:
    - cd overlays/dev
    - kustomize edit set image $REGISTRY/$APP_NAME=$REGISTRY/$APP_NAME:$CI_COMMIT_SHA
    - kubectl apply -k .
    - kubectl rollout status deployment/my-app -n dev
  only:
    - develop

deploy:prod:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: production
  script:
    - cd overlays/production
    - kustomize edit set image $REGISTRY/$APP_NAME=$REGISTRY/$APP_NAME:$CI_COMMIT_SHA
    - kubectl apply -k .
    - kubectl rollout status deployment/my-app -n production
  when: manual
  only:
    - main

ArgoCD Integration

# argocd/application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-production
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/example/k8s-manifests.git
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Summary

Kustomize Best Practices

Directory Structure Standards
- Store common configurations in base
- Manage environment differences in overlays
- Use clear naming conventions
Version Management
- Use Git to manage all configurations
- Use SHA256 or explicit tags for images
- Do not use latest
Secret Management
- Use Sealed Secrets in production environments
- Do not commit sensitive information to Git
- Use External Secrets Operator to integrate with Vault
Component Reuse
- Extract common configurations as components
- Establish a Kustomize component library
- Avoid duplicate configurations
CI/CD Integration
- Automate image tag updates
- Use kubectl diff to preview changes
- Apply configuration only after validation passes

Kustomize vs Helm Selection

Scenario	Recommended Solution	Reason
Microservice Configuration Management	Kustomize	High flexibility, Git-friendly
Third-party Application Deployment	Helm	Rich ecosystem, one-click deployment
Complex Template Requirements	Helm	Go Template is powerful
Simple Configuration	Kustomize	No need to learn templates
GitOps	Kustomize	Native support

Learning Path

Basic Syntax: base + overlay structure
Patch Strategies: StrategicMerge vs JSON6902
Component Reuse: Extract common configurations
CI/CD Integration: Automated deployment workflows
GitOps Practice: Combine with ArgoCD/Flux

Kustomize is an ideal tool for configuration management in the cloud-native era. Mastering it can significantly improve Kubernetes application management efficiency. In actual deployment scenarios, if the differentiation between environments for new projects is small, you can combine VibeCoding tools to generate one-click deployment scripts for rapid delivery of new environments.

In-depth Comparison of Enterprise Private Cloud Platforms: ZStack vs TStack

wzdushu@gmail.com (wanzi) — Sat, 14 Jun 2025 11:00:00 +0800

Comparison between ZStack and TStack (Tencent Cloud Private Cloud)

Due to the specific requirements of financial clients, a selection process for third-party private cloud platforms is necessary during private delivery. This comparison focuses primarily on ZStack and TStack:

In the private/hybrid cloud market, ZStack (Yunzhou Tech, backed by Alibaba Cloud) and TStack (Tencent Cloud’s proprietary full-stack cloud solution, now mostly integrated into the TCE/TCS ecosystem) represent two typical categories. The former is known for “productization, lightweight, independent vendor,” while the latter focuses on “full-stack capabilities, same source as public cloud, and big tech ecosystem.”

1. Core Positioning and Background

Dimension	ZStack (Yunzhou Tech)	TStack / TCE (Tencent Cloud)
Vendor Nature	Independent Software Vendor (ISV) focused on cloud platform software R&D with strong neutrality.	Cloud Service Provider (CSP) representing Tencent’s internal technology output, heavily dependent on the Tencent ecosystem.
Core Positioning	Productized Private Cloud / HCI Emphasizes “Simple, Scalable, Strong, Smart” (4S), positioning as a VMware replacement and for small-to-medium scale private clouds.	Full-Stack Private Cloud / Industry Cloud Emphasizes “public cloud experience in a private deployment,” providing full-stack IaaS+PaaS+SaaS capabilities, targeting digital transformation for large government and enterprises.
Technical DNA	Self-developed architecture (not a modified OpenStack), high code autonomy, lightweight.	Derived from Tencent Cloud (Tencent Cloud Public), deeply customized based on OpenStack/K8s, platform-heavy.
Typical Clients	SMEs, hospitals, schools, some SOEs, edge computing scenarios.	Large central SOEs, government bureaus, financial institutions, groups requiring complex PaaS capabilities.

2. Comparison of Architecture and Technical Features

2.1 Architecture Design

ZStack:
- Microservices Architecture: All components are process-based with no external dependencies (no need for separate MySQL or RabbitMQ deployment), featuring one-click installation and deployment completion within 30 minutes.
- Stateless Design: Management nodes support High Availability (HA) with automatic failover.
- Seamless Upgrades: Supports seamless online upgrades without business interruption.
TStack (TCE):
- Complex Distributed Architecture: Relies on a large number of middleware components; deployment usually requires professional team intervention with a longer cycle (days/weeks).
- Same Source as Public Cloud: Underlying technology is highly consistent with Tencent Cloud Public, suitable for scenarios requiring “hybrid cloud consistency.”
- Modular Delivery: Allows selection of Enterprise, Big Data, or AI editions, but high integration leads to relatively high coupling.

2.2 Virtualization and Containers

Feature	ZStack	TStack (TCE)
Virtualization Kernel	Deeply optimized based on KVM with a self-developed management engine, offering extremely low performance overhead.	Based on KVM, inheriting some features of Tencent Cloud Public’s Shenlong architecture, with strong performance.
Container Support	ZStack Zaku: Provides a container engine, supporting unified management of VMs and containers, lightweight K8s.	TCS (Tencent Cloud-native Suite): Enterprise-grade K8s platform with extremely rich features, supporting complex microservice governance and DevOps.
Compute-Storage Separation	Supports ZStack Cube HCI architecture with built-in distributed storage.	Supports integration with Tencent self-developed storage like CBS/CFS, as well as third-party storage.

3. Xinchuang (Localization) and Ecosystem Compatibility

With the deepening of domestic requirements, Xinchuang capabilities have become a critical factor in selection.

ZStack:
- Full-Stack Xinchuang: Claims “One Cloud, Eight Chips,” fully supporting domestic chips such as Phytium, Kunpeng, Hygon, Loongson, Zhaoxin, and Sunway.
- Ecosystem Adaptation: Adapted to 140+ domestic software and hardware products (OS, databases, middleware).
- Advantage: As an independent vendor, response speed for adaptation to niche domestic chips and operating systems is extremely fast with high flexibility.
TStack (TCE):
- Mainstream Xinchuang: Focuses on supporting mainstream domestic chips like Kunpeng and Phytium (especially vendors with strategic partnerships with Tencent).
- Ecosystem Binding: Deeply integrated with the Tencent Cloud ecosystem (e.g., TDSQL database, TDW big data), performing excellently in Tencent-based Xinchuang solutions.
- Advantage: In large Xinchuang projects for finance and government, can provide full-link backstop services from chip to application.

4. O&M and Delivery Experience

Dimension	ZStack	TStack (TCE)
Deployment Difficulty	⭐ (Extremely Simple) Single node startup, guided Web UI installation, extremely high automation.	⭐⭐⭐ (Complex) Usually requires on-site deployment by the vendor or certified partners, relying on complex pre-checks and environment preparation.
Daily O&M	⭐⭐ (Simple) Intuitive interface, focused features, fast learning curve for general ops staff.	⭐⭐⭐ (Professional) Extremely rich but complex features, requires a dedicated trained O&M team, steep learning curve.
Upgrade & Maintenance	Supports seamless online upgrades, fast version iteration.	Rigorous but cumbersome upgrade process, usually involving downtime windows or complex canary release strategies.
After-sales Service	Direct vendor service + partners, flexible response.	Relies on Tencent Cloud’s massive service system, comprehensive three-tier support mechanism, but relatively standardized processes (sometimes slightly rigid).

5. Cost Analysis (TCO)

ZStack:
- Licensing Model: Usually licensed by CPU socket or node count, transparent pricing.
- Hardware Requirements: Relatively low hardware configuration requirements, strong capability for legacy hardware reuse, can use generic x86 servers.
- Labor Cost: Simple O&M significantly reduces reliance on senior operations experts.
- Overall Assessment: Low initial investment, low long-term holding cost, suitable for budget-sensitive projects.
TStack (TCE):
- Licensing Model: Packaged total solution, including software licensing, implementation fees, and maintenance service fees, higher price.
- Hardware Requirements: Recommends dedicated hardware or high-performance servers, high requirements for network architecture (often requiring 10GbE起步).
- Labor Cost: Requires a dedicated cloud management team, high training costs.
- Overall Assessment: High initial investment, suitable for large projects with sufficient budget pursuing ultimate stability and full-stack capabilities.

6. Selection Decision Matrix

Please refer to the table below based on your actual scenario:

Scenario Characteristics	Recommended Solution	Rationale
VMware Replacement	ZStack	ZStack is designed specifically to replace VMware, with mature migration tools, low cost, lightweight architecture, and low risk.
Small/Medium Private Cloud (<50 nodes)	ZStack	Fast deployment, less O&M, high cost-performance ratio, features are fully sufficient.
Large Gov/Finance Core Cloud (>100 nodes)	TStack (TCE)	Requires extremely high stability, complex PaaS capabilities, and the brand endorsement/backstop service of a major vendor.
Need Deep PaaS/Big Data/AI Capabilities	TStack (TCE)	Tencent has natural advantages in big data, AI, and WeChat ecosystem integration; TCE can directly reuse public cloud capabilities.
Complex Xinchuang Environment (Mixed Domestic Chips)	ZStack	Independent vendors usually outperform major vendors’ standard products in compatibility and adaptation response speed for heterogeneous chips.
Hybrid Cloud Architecture (Linkage with Public Cloud)	TStack (TCE)	If there are plans to heavily use Tencent Cloud Public in the future, TCE can deliver a true “One Cloud” experience.
Limited Budget / Fast Launch	ZStack	Out-of-the-box, production-ready within days, significantly reducing TCO.

7. Summary and Recommendations

Choose ZStack if:
- You wish to rapidly build a private cloud and are tired of complex OpenStack deployments.
- Your primary goal is VMware replacement to reduce licensing costs.
- Your team size is small and lacks dedicated cloud operations experts.
- You require high autonomy/controllability and a neutral third-party software vendor.
Choose TStack (TCE) if:
- You are a large group, central SOE, or financial institution with near-stringent requirements for stability.
- You need full-stack capabilities (not just VMs, but also PaaS services like big data, AI, microservice governance).
- Your strategic direction is hybrid cloud, and you prefer using Tencent Cloud Public as the public side.
- You have sufficient budget to form a professional cloud management team and purchase advanced vendor services.

💡 Final Recommendation:

In the current market environment, “Small & Beautiful” and “Big & Complete” coexist. For 80% of generic enterprise scenarios, ZStack is often the more pragmatic and efficient choice;而对于那 20% 关乎国计民生、业务极度复杂的核心场景，TStack (TCE) 的底蕴则不可或缺。建议在选型前进行 POC 测试，重点验证部署效率、信创兼容性及实际业务负载下的性能表现。

Actually, in private deployment scenarios, ZStack and TStack already cover the IaaS layer (VMs and distributed storage) and the PaaS container layer. The only missing piece is the RDS platform. Therefore, when adding a third-party RDS platform to complement these two, attention must be paid to compatibility. Generally, ZStack is paired with Woqu, and TStack is paired with ActionTech.

After using both Woqu and ActionTech, I recommend using the Woqu platform, mainly because it performs better in terms of platform features, high availability, and UI usability. It supports MySQL, Oracle, SQL Server, PostgreSQL, Redis, MongoDB, OceanBase, TiDB, ES, RabbitMQ, Kafka, RocketMQ, etc.

Several Ways for Azure AKS Clusters to Access Blob Storage

wzdushu@gmail.com (wanzi) — Fri, 30 May 2025 21:22:42 +0800

When working with Azure, Blob Storage is frequently used. This article documents several common approaches for accessing Blob Storage from applications running in production environments:

一、Accessing Blob Storage via a Microsoft Entra ID Application Account

Advantages:

Works outside Azure environments (e.g., on-premises servers, AWS EC2)
Supports cross-subscription and cross-tenant access

Disadvantages:

Secrets must be securely stored (e.g., in Azure Key Vault)
Manual secret rotation is required upon expiration; otherwise, the application will break
Risk of credential leakage, especially if secrets are hardcoded in CI/CD pipelines

The specific configuration is as follows:

1、Configure an Application-Specific Account

Register an application in Entra ID, create a service principal, generate a client secret or certificate, and assign the application the Storage Blob Data Contributor role (for read/write access). Record the following information:

Tenant ID
Client ID
Client Secret (expires after a set period—manual renewal required)

2、Grant Blob Storage Permissions to the Application

(1)In the Azure portal, navigate to your storage account (e.g., mdblob001).

(2)Go to Access control (IAM) → + Add → Add role assignment.

(3)Select the role: Storage Blob Data Contributor (grants read & write access to blobs). Under Assign access to, choose User, group, or service principal, search for your app name (e.g., MyBlobApp), select it, and click Review + assign.

3、Access Blob Storage from Application Code

Use ClientSecretCredential (or ClientCertificateCredential) in your code to obtain tokens. Below is a Python SDK example:

Install required packages:

`1`	`pip install azure-identity azure-storage-blob`

from azure.identity import ClientSecretCredential
from azure.storage.blob import BlobServiceClient

# Entra ID App Configuration
TENANT_ID = "3c4e99f3-4831-4352-ae86-4d8f59512865"
CLIENT_ID = "3357a2b1-9895-484d-8754-42c162f2c9eb"
CLIENT_SECRET = "DoB8Q~4Gwl6sdzV_ocfUvXA3BtHYJLgrimwwMcj1"

# Storage Account & File Info
ACCOUNT_NAME = "mdblob001"
LOCAL_FILE_PATH = "/mnt/d/file.txt"
BLOB_NAME = "uploaded-file.txt"
CONTAINER_NAME = "mycontainer"  # Ensure this is defined

# Obtain Token Credential
credential = ClientSecretCredential(
    tenant_id=TENANT_ID,
    client_id=CLIENT_ID,
    client_secret=CLIENT_SECRET
)

# Create Blob Service Client
blob_service_client = BlobServiceClient(
    account_url=f"https://{ACCOUNT_NAME}.blob.core.windows.net",
    credential=credential
)

# List containers (test permissions)
try:
    containers = blob_service_client.list_containers()
    for container in containers:
        print("Container:", container.name)
except Exception as e:
    print("Error:", e)

# Upload file
try:
    blob_client = blob_service_client.get_blob_client(
        container=CONTAINER_NAME,
        blob=BLOB_NAME
    )

    with open(LOCAL_FILE_PATH, "rb") as data:
        blob_client.upload_blob(data, overwrite=True)

    print(f"File '{LOCAL_FILE_PATH}' successfully uploaded as '{BLOB_NAME}'")

except FileNotFoundError:
    print("Local file not found")
except Exception as e:
    print("Upload failed:", e)

二、Accessing Blob Storage via AKS System-Assigned Managed Identity

Azure provides system-assigned managed identities—automatically managed service principals—for supported resources like VMs, AKS clusters, and App Services. AKS enables this by default.

Advantages:

Lifecycle automatically managed (created/destroyed with the AKS cluster)
No secrets to manage—fully keyless access
Higher security: credentials never appear in code, config files, or environment variables
Deep integration with Azure RBAC
SDKs only need DefaultAzureCredential to auto-acquire tokens

Disadvantages:

Cannot be reused across workloads with different permission requirements
RBAC assignments must be configured manually
Not portable across clouds
Naming is not customizable

1、Enable system-assigned managed identity for AKS cluster

az aks create --resource-group test --name md-test-southafrica --generate-ssh-keys  # Enable system-assigned managed identity only for new clusters
az aks update --resource-group test --name md-test-southafrica --enable-managed-identity  # For existing clusters
az aks show --resource-group test -n md-test-southafrica --query "identity"  # View the system-assigned managed identity
{
  "delegatedResources": null,
  "principalId": "37e6bf32-b18d-4e3a-9f97-b86d0c9dc2cf",
  "tenantId": "3c4e99f3-4831-4352-ae86-4d8f59512865",
  "type": "SystemAssigned",
  "userAssignedIdentities": null
}

2、Assign Blob Storage Role to the Managed Identity

Roles:

Storage Blob Data Reader: read-only
Storage Blob Data Contributor: read/write
Storage Blob Data Owner: read/write + ACL management (for ADLS Gen2)

Use Azure CLI to grant the identity access to Blob Storage (for example, the Storage Blob Data Contributor role):

# Get the storage account ID
STORAGE_ACCOUNT_ID=$(az storage account show -n mdblob001 -g test --query id -o tsv)
echo $STORAGE_ACCOUNT_ID
/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001

# Assign the role
AKS_PRINCIPAL_ID=$(az aks show --resource-group test --name md-test-southafrica --query "identity.principalId" --output tsv)
echo "AKS Principal ID: $AKS_PRINCIPAL_ID"
AKS Principal ID: 37e6bf32-b18d-4e3a-9f97-b86d0c9dc2cf

az role assignment create \
  --assignee "37e6bf32-b18d-4e3a-9f97-b86d0c9dc2cf" \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001"
# Grant permissions to the AKS system-assigned managed identity

{
  "condition": null,
  "conditionVersion": null,
  "createdBy": "dc8278a7-78d6-4f6e-971c-8974d0ac4e2f",
  "createdOn": "2025-02-19T04:31:42.582306+00:00",
  "delegatedManagedIdentityResourceId": null,
  "description": null,
  "id": "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001/providers/Microsoft.Authorization/roleAssignments/81f9e7d1-b83c-4c4b-aab0-047fc4fb525f",
  "name": "81f9e7d1-b83c-4c4b-aab0-047fc4fb525f",
  "principalId": "37e6bf32-b18d-4e3a-9f97-b86d0c9dc2cf",
  "principalName": "ad674b19-a3b1-47ed-9463-128289ae5e34",
  "principalType": "ServicePrincipal",
  "resourceGroup": "Test",
  "roleDefinitionId": "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/providers/Microsoft.Authorization/roleDefinitions/ba92f5b4-2d11-453d-a403-e96b0029c9fe",
  "roleDefinitionName": "Storage Blob Data Contributor",
  "scope": "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001",
  "type": "Microsoft.Authorization/roleAssignments",
  "updatedBy": "dc8278a7-78d6-4f6e-971c-8974d0ac4e2f",
  "updatedOn": "2025-02-19T04:31:42.582306+00:00"
}

az role assignment list \
  --scope "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001" \
  --output table
# List role assignments under the storage account

Principal                             Role                           Scope
--------------------------------------------------------------------------------
ad674b19-a3b1-47ed-9463-128289ae5e34  Storage Blob Data Contributor  /subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001

3、Verification

Use the Python SDK to build a container and verify access:

from azure.storage.blob import BlobServiceClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
blob_service_client = BlobServiceClient(
    account_url="https://mdblob001.blob.core.windows.net",
    credential=credential
)

# List containers
containers = blob_service_client.list_containers()
for c in containers:
    print(c.name)

DefaultAzureCredential automatically attempts multiple authentication methods and, in AKS, retrieves a token via IMDS (Instance Metadata Service), provided that the pod is running on a node with managed identity enabled.

三、Accessing Blob Storage via AKS User-Assigned Managed Identity with Workload Identity

This is Microsoft’s recommended, standards-based (OIDC), secure approach for fine-grained workload identity.

1、Create a User-Assigned Managed Identity

az group create -n rg-aks-wi -l test  # Create a resource group; skip if it already exists
az identity create --name MD-Blob-Identity --resource-group test  # Create a user-assigned managed identity
{
  "clientId": "7d9b65f6-959d-4a74-ab3f-67d16adc8c37",
  "id": "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourcegroups/test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MD-Blob-Identity",
  "location": "southafricanorth",
  "name": "MD-Blob-Identity",
  "principalId": "bdd58864-063f-451d-b352-2dded3582b36",
  "resourceGroup": "test",
  "systemData": null,
  "tags": {},
  "tenantId": "3c4e99f3-4831-4352-ae86-4d8f59512865",
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities"
}

# Retrieve key information
export USER_ASSIGNED_CLIENT_ID="$(az identity show -g test -n MD-Blob-Identity --query clientId -o tsv)"
export USER_ASSIGNED_PRINCIPAL_ID="$(az identity show -g test -n MD-Blob-Identity --query principalId -o tsv)"

2、Assign Blob Storage Permissions

STORAGE_ACCOUNT_ID=$(az storage account show -n mdblob001 -g test --query id -o tsv)
echo $STORAGE_ACCOUNT_ID
/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourceGroups/Test/providers/Microsoft.Storage/storageAccounts/mdblob001
az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee $USER_ASSIGNED_PRINCIPAL_ID \
  --scope $STORAGE_ACCOUNT_ID

3、Enable Workload Identity on AKS (if not already enabled)

1
2

az aks update -g test -n md-test-southafrica --enable-oidc-issuer
export AKS_OIDC_ISSUER=$(az aks show -g test -n md-test-southafrica --query "oidcIssuerProfile.issuerUrl" -o tsv)

4、Federate the Identity with a Kubernetes ServiceAccount

# Get the client ID of the user-assigned identity
export USER_ASSIGNED_CLIENT_ID=$(az identity show -g test -n MD-Blob-Identity --query clientId -o tsv)

# Create a federated credential (allowing a specific ServiceAccount in AKS to use this identity)
az identity federated-credential list --resource-group test --name md-test-southafrica

az identity federated-credential create --name my-fed-cred \
  --identity-name MD-Blob-Identity \
  --resource-group test \
  --issuer $AKS_OIDC_ISSUER \
  --subject system:serviceaccount:demo:md-blob-sa \
  --audience api://AzureADTokenExchange

{
  "audiences": [
    "api://AzureADTokenExchange"
  ],
  "id": "/subscriptions/cdab1cd0-0556-41bc-8905-33ce907e818a/resourcegroups/test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MD-Blob-Identity/federatedIdentityCredentials/my-fed-cred",
  "issuer": "https://oidc.prod-aks.azure.com/3c4e99f3-4831-4352-ae86-4d8f59512865/",
  "name": "my-fed-cred",
  "resourceGroup": "test",
  "subject": "system:serviceaccount:demo:md-blob-sa",
  "systemData": null,
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials"
}

Subject format:：system:serviceaccount::

5、Create a ServiceAccount in Kubernetes and deploy the application.

Create Kubernetes ServiceAccount

# blob-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: blob-access-sa
  namespace: demo
  annotations:
    azure.workload.identity/client-id: "${USER_ASSIGNED_CLIENT_ID}"

apply

kubectl apply -f blob-sa.yaml

6、Deploy Application Pod

Here we use Python as an example.

# python-blob.yaml
apiVersion: v1
kind: Pod
metadata:
  name: blob-client
  namespace: demo
  labels:
    azure.workload.identity/use: "true"  # This label is required!
spec:
  serviceAccountName: blob-access-sa
  containers:
    - name: app
      image: python:3.11-slim
      command: ["sh", "-c"]
      args:
        - |
          pip install azure-identity azure-storage-blob && \
          python -c "
from azure.storage.blob import BlobServiceClient
from azure.identity import DefaultAzureCredential
import os

account = 'mdblob001'
url = f'https://mdblob001.blob.core.windows.net'
client = BlobServiceClient(account_url=url, credential=DefaultAzureCredential())
print('Containers:', [c.name for c in client.list_containers()])
"
      env:
        - name: AZURE_CLIENT_ID
          value: "${USER_ASSIGNED_CLIENT_ID}"  # Optional but recommended to explicitly specify
  restartPolicy: Never

Deploy and verify:

kubectl apply -f python-blob.yaml 
kubectl logs blob-client # The application can successfully list the Blob Storage containers, and it accesses Blob Storage using a token, achieving the entire process without any secrets.

With these three methods—Entra ID app credentials, system-assigned managed identity, and user-assigned identity with Workload Identity—you now have flexible, secure options for accessing Azure Blob Storage from AKS workloads.

Self-hosted Enterprise-grade VPN Network with Headscale

wzdushu@gmail.com (wanzi) — Sat, 11 Jan 2025 10:00:00 +0800

Project Background

Business Pain Points

With business growth, there are currently over a dozen independent private network environments (ZStack/TStack) overseas, for example:

Overseas Private Network Region A
Overseas Private Network Region B
Overseas Private Network Region C

Each network is an independent entity requiring different VPN clients to dial in for resource access, posing significant challenges for daily O&M management:

O&M personnel need to frequently switch between multiple VPN clients.
Lack of unified permission management and access control.
Complex network configuration and difficult troubleshooting.
Inability to achieve cross-network automated O&M.

Solution

Adopting the Headscale (Server) + Tailscale (Client) architecture to build an enterprise-grade Zero Trust VPN network:

Deploy the Headscale control server on the Alibaba Cloud Singapore node.
Install Tailscale clients on private network nodes and register them with Headscale.
Achieve one-time connection to access all private network resources.
Support ACL access control and SSO.
Tailscale has excellent multi-platform support: iOS, macOS, Android, Windows, Linux, and supports IPv6.

Headscale Architecture Overview

Core Components

Headscale is an open-source implementation of the Tailscale control server, fully compatible with the Tailscale protocol:

Control Server (Headscale): Manages node registration, key distribution, address allocation, and ACL policies.
DERP Server: Relay server for traffic forwarding when NAT traversal fails.
Client (Tailscale): Deployed on nodes, establishes encrypted tunnels based on WireGuard.

Technical Advantages

Compared to traditional VPN solutions:

Zero Trust Architecture: Deny all access by default, explicitly authorize via ACL.
NAT Traversal: Supports STUN protocol for automatic hole punching, no public IP required.
Mesh Network: Direct communication between nodes without passing through a central server.
Auto Reconnection: Automatically re-establishes connections upon network changes.
Cross-platform Support: Linux/Windows/macOS/BSD/iOS/Android/FreeBSD.

Headscale Server Deployment

Environment Preparation

# System Requirements
# Ubuntu 20.04+ / CentOS 8+ / Debian 11+
# Public IP Server (Alibaba Cloud Singapore recommended)
# Open UDP ports: 3478 (DERP), 41641 (Headscale)

# Install Headscale
wget https://github.com/juanfont/headscale/releases/download/v0.23.0/headscale_0.23.0_linux_amd64.deb
sudo dpkg -i headscale_0.23.0_linux_amd64.deb

# Or install using yum (RHEL/CentOS)
sudo yum install -y https://github.com/juanfont/headscale/releases/download/v0.23.0/headscale_0.23.0_linux_amd64.rpm

Configuration File

Reference: https://github.com/juanfont/headscale/blob/main/config-example.yaml

Create configuration file /etc/headscale/config.yaml:

# Server URL
server_url: https://headscale.wnote.com:443

# Listen address
listen_addr: 0.0.0.0:8080

# Private key (auto-generated or specified)
private_key_path: /var/lib/headscale/private.key

# DERP relay server configuration 
derp:
  # Enable embedded DERP server
  server:
    enabled: false
    region_id: 999                     # Use an unused ID to avoid conflicts
    region_code: "self-derp"           # Short identifier
    region_name: "Self-hosted Co-located DERP"        # Descriptive name
    stun_listen_addr: "0.0.0.0:3478"   # STUN service listening, must open UDP 3478
    # [Important] Enter your server's actual public IPv4 address
    ipv4: "YOUR_SERVER_PUBLIC_IP"
    # If you have public IPv6, you can also fill it in
    # ipv6: "YOUR_SERVER_PUBLIC_IPV6"
    # Automatically add this region to DERP map
    automatically_add_embedded_derp_region: true
    # [Security] Strongly recommended to enable, only allow your Tailnet clients to use this relay
    verify_clients: true

  # You can choose to keep the official DERP as a backup, or disable it completely (clear urls)
  urls:
    - https://controlplane.tailscale.com/derpmap/default   # Keep backup (recommended)
  # If you have additional self-hosted DERP nodes (e.g., Singapore node), keep paths
  paths:
    - /etc/headscale/derp.yaml

  auto_update_enabled: true
  update_frequency: 24h

# Database (Default SQLite, PostgreSQL recommended for production)
database:
  type: sqlite
  sqlite:
    path: /var/lib/headscale/db.sqlite

# ACL policy file
acl_policy_path: /etc/headscale/acl.yaml

# DNS configuration
dns:
  nameservers:
    - 1.1.1.1
    - 8.8.8.8
  magic_dns: true  # Enable MagicDNS (node.namespace.vpn)

# OAuth configuration (optional)
# oidc:
#   issuer: "https://your-oidc-provider.com"
#   client_id: "your-client-id"
#   client_secret: "your-client-secret"

# Log configuration
log:
  format: text
  level: info

Configuring the DERP Relay Server

First, let’s understand the DERP concept:

The DERP server is primarily a reliable, low-latency backup relay solution to ensure devices can always connect when network conditions are poor (e.g., symmetric NAT, strict firewalls) preventing direct connections.

Direct Connection (P2P): Headscale’s ultimate goal is to let clients establish peer-to-peer direct connections via the STUN protocol “hole punching”, offering the fastest speed.
Relay (DERP): When hole punching fails (e.g., one party is behind strict symmetric NAT), traffic is forwarded via the DERP server. Although latency is higher than direct connection, it ensures uninterrupted connection.

If you want to self-host DERP and finely control multiple DERP nodes, you need to create a configuration file to define your private DERP network regions and node information.

The benefit of self-hosting DERP is that you can avoid the latency issues of official DERP servers overseas and keep relay traffic on your own servers, improving security and stability.

I don’t need this for now, just an example. Create /etc/headscale/derp.yaml:

# /etc/headscale/derp.yaml
regions:
  # First self-hosted region, e.g., East China
  901:
    regionid: 901
    regioncode: "cn-east"   # Region code, short
    regionname: "East China Self-hosted Node" # Region name
    nodes:
      - name: 901a
        regionid: 901
        # Enter your independent DERP server domain name here
        hostname: derp-shanghai.yourdomain.com
        # Your DERP server port, usually 443 or custom
        derpport: 12345
        # STUN port, usually 3478
        stunport: 3478
        stunonly: false

  # Second self-hosted region, e.g., North China
  902:
    regionid: 902
    regioncode: "cn-north"
    regionname: "North China Self-hosted Node"
    nodes:
      - name: 902a
        regionid: 902
        hostname: derp-beijing.yourdomain.com
        derpport: 443  # Assume this node uses standard port 443
        stunport: 3478
        stunonly: false

After creating the configuration, you can view the DERP map and connectivity on any Tailscale client:

tailscale debug derp-map
tailscale debug derp headscale # Test connectivity with a specific DERP server

To verify if the client is actually using the self-hosted relay, you can check the connection status via tailscale status. If you see derp=self-derp (or your custom region code), it means traffic is going through your self-hosted relay.

ACL Access Control Policy

Create /etc/headscale/acl.yaml:

# User/Group definitions
groups:
  group:admin:
    - admin@example.com
  group:devops:
    - devops1@example.com
    - devops2@example.com
  group:developer:
    - dev1@example.com
    - dev2@example.com

# Host tag definitions
hosts:
  sg-cloud: "10.100.0.1/32"
  network-a: "10.100.1.0/24"
  network-b: "10.100.2.0/24"
  network-c: "10.100.3.0/24"

# Access control rules
acls:
  # Admins can access all resources
  - action: accept
    src:
      - group:admin
    dst:
      - "*:*"

  # Ops team can access all servers via SSH
  - action: accept
    src:
      - group:devops
    dst:
      - "sg-cloud:22"
      - "network-a:22"
      - "network-b:22"
      - "network-c:22"

  # Dev team can access application ports
  - action: accept
    src:
      - group:developer
    dst:
      - "sg-cloud:80,443,8080"
      - "network-a:3306,6379"
      - "network-b:9200,9300"

  # Deny all other access by default
  - action: reject
    src:
      - "*"
    dst:
      - "*:*"

# Tag routing rules (optional)
tagOwners:
  tag:prod:
    - group:admin
    - group:devops
  tag:database:
    - group:admin

Starting the Service

# Create necessary directories
sudo mkdir -p /var/lib/headscale

# Start Headscale
sudo systemctl enable headscale
sudo systemctl start headscale
sudo systemctl status headscale

# View logs
sudo journalctl -u headscale -f

Configuring Nginx Reverse Proxy

Since we previously applied for a wildcard domain certificate via acme, we can directly specify the SSL certificate path. Configure according to actual scenarios in the enterprise:

server {
    listen 443 ssl http2;
    server_name headscale.wnote.com;

    ssl_certificate /etc/ssl/certs/headscale.crt;
    ssl_certificate_key /etc/ssl/private/headscale.key;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
        # Long connection timeout settings
        proxy_read_timeout 86400s;
        proxy_send_timeout 86400s;
        proxy_connect_timeout 300s;
    }
    # DERP relay path also needs proxying (embedded DERP serves via this path)
    location /derp {
        proxy_pass http://127.0.0.1:8080/derp;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Opening Firewall Ports

Ensure the server firewall allows the following traffic:

Port	Protocol	Purpose	Description
443	TCP	HTTPS (Control Plane + DERP Relay)	Opened via reverse proxy
3478	UDP	STUN Service	Must open UDP 3478 for NAT traversal detection

If using cloud service providers (like AWS, Alibaba Cloud), you also need to add inbound rules in the security group.

Tailscale Client Deployment

Installing Tailscale Client

Install on each private network node:

# Debian/Ubuntu
curl -fsSL https://tailscale.com/install.sh | sh

# RHEL/CentOS
curl -fsSL https://tailscale.com/install.sh | sh

# Windows
# Download installer: https://tailscale.com/download/windows

# macOS
brew install --cask tailscale

# Android/iOS
# Search "Tailscale" in App Store to install

Registering Nodes to Headscale

# Method 1: Use Auth Key (Recommended)

# Create Auth Key on server
sudo headscale apikeys create -e 8760h -o api-key.txt

# Client registration (replace YOUR_AUTH_KEY)
sudo tailscale up --login-server=https://headscale.wnote.com \
                  --auth-key=YOUR_AUTH_KEY \
                  --advertise-routes=10.0.0.0/24,192.168.1.0/24 \
                  --advertise-exit-node

# Method 2: Register via command line
# 1. Create user on server
sudo headscale users create user@example.com

# 2. Get registration command
sudo headscale namespaces create default

# 3. Execute registration on client and get URL
sudo tailscale up --login-server=https://headscale.wnote.com

# 4. Approve registration on server
sudo headscale nodes register -u user@example.com -n <NodeName>

Subnet Routing Configuration

Configure subnet routing on the gateway nodes of each private network:

# Enable IP forwarding
echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv6.conf.all.forwarding=1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Advertise subnet routes (execute on gateway node)
sudo tailscale up --advertise-routes=10.100.1.0/24,10.100.2.0/24

# Approve routes on server
sudo headscale routes list
sudo headscale routes enable -r <RouteID>

# Verify routing
sudo tailscale status
ping 10.100.1.10  # Test cross-network access

Exit Node Configuration (Optional)

Configure an exit node to access the internet via a specific node (designated internet gateway):

# Enable on exit node
sudo tailscale up --advertise-exit-node

# Approve on server
sudo headscale nodes list
sudo headscale nodes edit --exit-node <NodeID> --enable

# Client uses exit node
sudo tailscale up --exit-node=<ExitNodeIP> --exit-node-allow-lan-access

Specify to forward traffic via and allow simultaneous access to the local LAN. If --exit-node-allow-lan-access is not added, local LAN access (like printers, router admin pages) will also go through the exit node, potentially causing inaccessibility.

Management and Maintenance

Common Management Commands

# List all nodes
sudo headscale nodes list

# View node details
sudo headscale nodes get -i <NodeID>

# Delete node
sudo headscale nodes delete -i <NodeID>

# Rename node
sudo headscale nodes rename -i <NodeID> --new-name <NewName>

# Set node tags
sudo headscale nodes tag set -i <NodeID> -t "tag:prod,database"

# View routes
sudo headscale routes list

# Enable/Disable routes
sudo headscale routes enable -r <RouteID>
sudo headscale routes disable -r <RouteID>

# View namespaces
sudo headscale namespaces list

# Create namespace
sudo headscale namespaces create <NamespaceName>

# Add user
sudo headscale users create user@example.com

# List API keys
sudo headscale apikeys list

Monitoring and Alerting

# Enable Prometheus metrics
# Add in config.yaml:
metrics:
  listen_addr: 127.0.0.1:9090

# Access metrics
curl http://localhost:9090/metrics

# Grafana Dashboard
# Import Headscale official Dashboard: https://grafana.com/grafana/dashboards/18820

Backup and Recovery

# Backup database
sudo cp /var/lib/headscale/db.sqlite /backup/headscale-$(date +%Y%m%d).sqlite

# Backup configuration files
sudo tar -czf /backup/headscale-config-$(date +%Y%m%d).tar.gz /etc/headscale/

# Recovery
sudo systemctl stop headscale
sudo cp /backup/headscale-xxx.sqlite /var/lib/headscale/db.sqlite
sudo systemctl start headscale

Production Environment Best Practices

High Availability Deployment

# Use PostgreSQL as database backend
database:
  type: postgres
  postgres:
    host: pg-cluster.example.com
    port: 5432
    user: headscale
    password: "your-password"
    database: headscale
    sslmode: require

Security Hardening

Use mTLS Authentication

tls:
  letsencrypt:
    email: admin@example.com
    agree_tos: true
    listen_host: ":443"

Restrict IP Access

ip_prefixes:
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16

Regularly Rotate Keys

1
2
3

# Regenerate private key
sudo rm /var/lib/headscale/private.key
sudo systemctl restart headscale

Performance Optimization

Deploy Multiple DERP Servers
- Deploy a DERP node in each major region.
- Use CDN to accelerate DERP access.
Optimize Keep-alive

1
2

# Client configuration
sudo tailscale up --ping=20s  # Heartbeat interval

Single Point of Failure Risk If official DERP is completely disabled (urls: []) and there is only one self-hosted DERP server, all clients requiring relay will be unable to communicate if that server goes down or has network issues. For production, it is recommended to keep at least the official DERP as a backup or deploy multiple self-hosted DERP nodes.
Certificate Requirements Embedded DERP relies on Headscale’s HTTPS service, so your domain headscale.wnote.com must have a valid SSL certificate.

Troubleshooting

Common Issues

1. Node Cannot Register

# Check server logs
sudo journalctl -u headscale -n 50

# Check if client registration URL is correct
# Ensure firewall allows ports 8080/443

2. Cannot Access Subnet Routes

# Check if IP forwarding is enabled
sysctl net.ipv4.ip_forward

# Check if routes are approved
sudo headscale routes list

# Packet capture analysis
sudo tcpdump -i tailscale0 -n host 10.100.x.x

3. DERP Connection Failed

# Test DERP connectivity
tailscale ping --c 100 <DERPServerIP>

# Check DERP server status
curl https://your-derp-server.com/derp/latency-check

4. ACL Policy Not Taking Effect

# Validate ACL syntax
sudo headscale acl validate

# View effective ACL
sudo headscale acl get

Summary

The Headscale + Tailscale solution provides an out-of-the-box Zero Trust VPN solution for enterprises:

Simple Deployment: One-click server installation, automatic client configuration.
Powerful Features: Supports ACL, SSO, routing, and exit nodes.
Excellent Performance: Based on WireGuard, low latency, high throughput.
Secure and Reliable: End-to-end encryption, fine-grained permission control.

By deploying the Headscale control server on Alibaba Cloud Singapore and unifying access to various private networks, we achieved:

O&M personnel access all networks with a single login.
ACL-based fine-grained access control.
Automated network topology management.
Auditable operation logs.

This solution has been running stably in production with 10+ nodes, 100+ daily active connections, and a failure rate of <0.1%.

Building an Enterprise-Grade VPN Network with WireGuard

wzdushu@gmail.com (wanzi) — Fri, 10 Jan 2025 14:22:42 +0800

Introduction to WireGuard

WireGuard is a modern, fast, simple, and secure Virtual Private Network (VPN) protocol designed to replace traditional VPN technologies like IPsec and OpenVPN. It is faster, simpler, leaner, and more practical than IPsec, while avoiding massive amounts of trouble. It offers higher performance compared to OpenVPN. Originally released for the Linux kernel, it is now cross-platform (Windows, macOS, BSD, iOS, Android) and widely deployable. Although currently under active development, it is likely already considered the most secure, easiest to use, and simplest VPN solution in the industry.

Main Use Cases:

Secure remote access to the enterprise intranet for employees working from home or traveling who need secure access to internal systems (e.g., file servers, ERP, databases).
Cross-branch interconnection (Site-to-Site), e.g., stable communication between a European office and the China headquarters.
Secure interconnection between cloud servers and local data centers, connecting business systems on Alibaba Cloud/AWS/Azure with local IDCs.
Secure interconnection between home networks and cloud servers, accessing home NAS files while traveling.
Secure communication for IoT devices, where smart cameras, industrial sensors, POS machines, etc., need to securely transmit data back to the data center.
Container and microservice network isolation, cross-host communication in container scenarios, e.g., using tools like wg-easy or Netmaker to quickly build overlay networks.

WireGuard Deployment

Installing WireGuard

WireGuard has been merged into the mainline kernel since Linux 5.6. Most modern distributions can install the userspace tools directly:

1
2

sudo apt update
sudo apt install wireguard wireguard-tools

Note: If using CentOS/RHEL, you need to enable EPEL or use the elrepo kernel module first; Windows/macOS clients can download the official GUI app from wireguard.com.

Generating Key Pairs

cd /etc/wireguard/
umask 077 # Ensure key file permissions are secure
wg genkey | tee server_private.key | wg pubkey > server_public.key # Generate server-side keys
wg genkey | tee server_private.key | wg pubkey > client1_public.key # Client A keys
wg genkey | tee server_private.key | wg pubkey > client2_public.key # Client B keys

WireGuard Configuration Management

WireGuard does not have a strict server/client distinction, but usually a node with a public IP is required to act as a relay server (Hub).

Edit the server-side (relay) configuration file /etc/wireguard/wg0.conf:

[Interface]
Address = 10.100.0.1/24
ListenPort = 51820
PrivateKey = wg genkey | tee server_private.key | wg pubkey > server_public.key # Generate server-side keys

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

# Client A
[Peer]
PublicKey = client1_public.key
AllowedIPs = 10.100.0.2/32

# Client B
[Peer]
PublicKey = client2_public.key
AllowedIPs = 10.100.0.3/32

Enable Auto-start on Boot

sudo systemctl enable wg-quick@wg0
sudo systemctl start wg-quick@wg0
# Check status
sudo wg show

Configure Kernel Forwarding and Firewall (CentOS/Ubuntu):

# Enable IPv4 forwarding
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p

# Open firewall port (optional, adjust based on actual situation)
sudo firewall-cmd --permanent --add-port=51820/udp
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=10.100.0.0/24 masquerade'
sudo firewall-cmd --reload
# If using iptables, refer to the PostUp/PostDown scripts above

WireGuard Dashboard

wg-easy (Lightweight Web UI)

Features: Single-page application, one-click Docker deployment, supports QR code scanning for connection, traffic monitoring, and online client toggling.
Use Cases: Small to medium teams (<50 users), quickly setting up a remote office VPN.
One-click deployment command:

docker run -d \
  --name=wg-easy \
  -e WG_HOST=your.domain.com \
  -e WG_PORT=51820 \
  -e WG_DEFAULT_ADDRESS=10.100.0.x \
  -e PASSWORD=your_admin_password \
  -v ～/.wg-easy:/etc/wireguard \
  -p 51821:51821 \
  -p 51820:51820/udp \
  weejewel/wg-easy

Note: wg-easy has no user permission isolation by default and is not suitable for multi-tenant or high-security scenarios.

Netmaker (Enterprise-grade SD-WAN Architecture)

Features:
- Supports Mesh networking (arbitrary node interconnection), Site-to-Site, and Kubernetes integration;
- Provides REST API, RBAC permission control, ACL policies, and metrics monitoring (Prometheus);
- Separation of control plane (netmaker) and data plane (netclient), supporting dynamic topology.
Architecture Components:
- Netmaker Server: Manages nodes, policies, and certificates;
- Netclient: Deployed on endpoints (servers/PCs/IoT), automatically registers and fetches configurations;
- MQTT Broker (Optional): Used for large-scale node status synchronization.
Deployment Methods: Supports Helm Chart (K8s), Docker Compose, and systemd.

Recommended for: Complex topologies such as cross-branch interconnection, hybrid cloud networking, and IoT device cluster management.

Tailscale / Headscale (Zero Trust Architecture)

Although not native WireGuard, it is based on the WireGuard protocol at the core;
Provides SSO (Google/Azure/OIDC), device authentication, and fine-grained ACLs;
Headscale is the open-source self-hosted version, suitable for enterprises with high data compliance requirements.

Tailscale is a commercial SaaS service (free version supports 3 users), while Headscale is its open-source, self-hosted alternative; the protocols are fully compatible.

Here is a strongly recommended solution: use the open-source Headscale for the server side and Tailscale for the client side.

Getting Started with Github Actions

wzdushu@gmail.com (wanzi) — Fri, 01 Nov 2024 17:22:42 +0800

What is Github Action?

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that lets you automate your build, test, and deployment processes. You can create workflows to build and test every pull request to your repository, or deploy merged pull requests to production.

GitHub Actions is more than just DevOps; it lets you run workflows when other events happen in your repository. For example, you can run a workflow to automatically add appropriate labels when someone creates a new issue in your repository.

GitHub offers Linux, Windows, and macOS virtual machines to run your workflows, or you can host your own self-hosted runners in your own data center or cloud infrastructure.

CICD tool selection

Currently, the CICD market is really flourishing. In addition to github actions, there are many other CICD tools to choose from, such as: Gitlab CI, Jenkins, Tekton focusing on cloud native, Bitbucket Pipelines, AWS CodePipeline, Bitbucket Pipelines, Azure DevOps, Alibaba Cloud Effect, etc.

Which method to choose to implement your business automation pipeline depends on your actual situation. The following recommendations start from the following aspects:

Cost, whether to choose an open source solution or a paid commercial version. If you put cost first, you need to consider the cost of building an open source software server and the licensing cost of paid software.
Ease of use and learning path, mainly consider whether the system interface is beautiful and whether the technical documentation is complete. The interface is difficult to judge, and the technical documentation can check whether the official documentation is detailed and whether the workflow uses yaml or a new language.
Scalability and flexibility, mainly consider whether the system plug-ins and custom requirements can be met. This also depends on personal needs. For example, some companies need CICD and jira to be connected, then it is better to choose Bitbucket Pipelines.
Integration capability, mainly consider whether it is integrated with the current code repository, test framework, deployment environment and the degree of API function perfection. For example, if your code repository is on Github, of course, it is better to choose github actions.
Security, mainly examine whether there are security features such as access control and data encryption. This is selected according to your actual needs.
Performance, mainly examine the performance bottleneck of large-scale deployment. Often this needs to be compared with cost or resource requirements. For example, if you have strict requirements for file transfer and bandwidth in daily construction, using github actions is not the best choice.
Enterprise-level solutions, mainly examine high concurrency, multi-tenancy, security compliance, etc. Often, for enterprises of a certain scale, there will be many customized requirements. See if the tool has enterprise solutions.
Cloud native support, mainly examine the degree of support for the business under kubernetes, especially the new version, in which Tekton has more advantages.
Community and ecology, mainly examine which users of the project or product are using it, and the community activity. In this regard, jenkins, gitlab, and github are relatively more advantageous.

Github Actions Components

1. workflow

From the literal meaning, we can see that workflow is a kind of work flow, and here it is mainly used to configure the CICD automation process of github actions. It is usually defined using yaml files. After the definition is completed, the workflow can be triggered automatically or manually.

The working directory of workflow is generally under the .github/workflow directory of the repository. A repository can have multiple workflows, and each workflow can perform a different set of tasks. The tasks here are also commonly referred to as job tasks. For example:

Build and test pull requests.
Deploy your application every time you release it.
Add tags every time you open a new issue.

Of course, workflows can also reference each other.

In actual use, we usually use workflow templates to quickly write workflows, and GitHub also provides workflow templates for various scenarios, such as

CI: continuous integration workflow
Deployment: deployment workflow
Automation: automation workflow
Code scanning: code scanning workflow
github pages: pages workflow

When writing workflows, you need to pay attention to the meaning of common keywords. Below I have briefly sorted them out for easy use later:

name: define the workflow name
run-name: define the workflow runtime name
on: define the trigger, which can be one event or multiple events or triggered based on matching
env: define workflow global variables
jobs: define workflow tasks
permissions: define workflow permissions for warehouses
default: define default configuration, which can be used globally or in a job, such as defining shells or working directories, etc.
on.schedule: define workflow time schedule
on.workflow_call: define information transmission during the input and output of different workflows
on.workflow_dispatch: define a specific workflow, enter some tags to trigger
jobs.<job_id>.name: define the task name of a single job
jobs.<job_id>.runs-on: select the jobs running environment, the default is github official, you can also choose a high-performance machine or host your own host
jobs.<job_id>.steps: define different running stages of a single job, for example, a job has checkout code and build
jobs.<job_id>.env: define a single job specific variable, belongs to the map type, and can also be set globally.
jobs.<job_id>.environment: define a reference in a single job, the content can be a variable or context expression
jobs.<job_id>.uses: defines which action to use. Sometimes, when deploying some complex scenarios, you need to use some open source actions.
jobs.<job_id>.with: specifies parameters. Generally, when building a task, you need to specify a version.
jobs.<job_id>.needs: generally, when there are multiple jobs, handle the dependencies between jobs.
jobs.<job_id>.if: specifies the job running conditions.
jobs.<job_id>.container: specifies the container service in a single job. Note that there are requirements for the runner’s operating system, such as Ubuntu.
jobs.<job_id>.outputs: defines the output of the job.

2. Event triggers

In the previous step, we know the workflow, so how to trigger the workflow? At this time, you need to define an event trigger, such as the common event trigger method:

Triggered by code repository changes, such as when pushed to the default branch of the code
Occurs outside GitHub and triggers the repository_dispatch event
Scheduled time
Manual trigger

Workflow triggers are defined using on, and there are several common configurations:

Set a trigger for an event, which is triggered when pushed to any branch in the workflow repository:

`1`	`on: push`

Set a trigger for multiple events, which is triggered when pushed to any branch in the workflow repository:

`1`	`on: [push, fork]`

When there are multiple branches, you need to filter specific branches, such as main or releases related branches, and trigger when there is a push:

on:
push:
branches:
- main
- 'releases/**'

GitHub itself can also match issues to trigger

on:
issues:
types:
- opened
- labeled

3. Jobs

Generally, a workflow consists of one or more jobs, which run in parallel by default. Note that jobs.<job_id> is the unique identifier of the job.

For example, we define a job named Docker build, and for this job, build is the unique identifier of the job.

jobs:
build:
name: Docker build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

In addition, dependencies can also be defined between jobs, defined by the jobs.<job_id>.needs keyword, such as:

jobs:
job1:
job2:
needs: job1
job3:
needs: [job1, job2]

In this example, job1 must be successfully completed before job2 starts, and job3 waits for job1 and job2 to complete.

4. Actions

Action is a custom application on Github that is used to perform complex but often repeated tasks. Using actions helps reduce the amount of repetitive code you write in your workflow files.

Actions can pull your Git repository from GitHub, set up the correct toolchain for your build environment, or set up authentication for your cloud provider.

Generally defined by the keyword uses, such as using the official: actions/checkout@v4

jobs:
deploy:
name: Deploy
runs-on: ubuntu-latest
environment: production

steps:
- name: Checkout
uses: actions/checkout@v4

You can write your own actions or find actions used in workflows in the GitHub Marketplace.

5. Runners

Runners are servers that run workflows when they are triggered. Define which type of runners to use by using the keyword runs-on. Each runner can run one job at a time, which is indeed very similar to the runner in gitlab CI.

GitHub provides Ubuntu Linux, Microsoft Windows, and macOS related runners to run workflows. Each workflow runs in a new, newly configured virtual machine.

Of course, if the default runners cannot meet your business needs, such as requiring an ARM architecture virtual machine, hosting your own server to GitHub for management; or if the resources are insufficient, you can also apply for larger runners.

The following sample workflow has two jobs, named Run-npm-on-Ubuntu and Run-PSScriptAnalyzer-on-Windows. When this workflow is triggered, GitHub provisions a new virtual machine for each job.

The job named Run-npm-on-Ubuntu is executed on a Linux VM because the job runs-on: specifies ubuntu-latest. The job named Run-PSScriptAnalyzer-on-Windows is executed on a Windows VM because the job runs-on: specifies windows-latest.

Install-Module PSScriptAnalyzer -ErrorAction Stop
- name: Get list of rules
shell: pwsh
run: |
Get-ScriptAnalyzerRule

This GitHub Actions workflow file defines two jobs:

Run-npm-on-Ubuntu: Runs the npm help command on an Ubuntu virtual machine.
Run-PSScriptAnalyzer-on-Windows: Installs the PSScriptAnalyzer module on a Windows virtual machine and lists all available rules.

These two jobs show how to run specific commands and scripts on different operating systems. In this way, you can ensure consistency and compatibility in different environments.

Common examples of Github Actions

1. Front-end build deployment

For example, a template for deploying a front-end Nodejs project to the Azure cloud, Deploy Node.js to Azure Web App:

on:
push:
branches: [ "master" ]
workflow_dispatch: # Allows manual triggering of workflows

env:
AZURE_WEBAPP_NAME: your-app-name # set this to your application's name
AZURE_WEBAPP_PACKAGE_PATH: '.' # set this to the path to your web app project, defaults to the repository root
NODE_VERSION: '20.x' # set this to the node version to use

permissions:
contents: read

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up Node.js
uses: actions/setup-node@v4
with: node-version: ${{ env.NODE_VERSION }} cache: 'npm' - name: npm install, build, and test run: | npm install npm run build --if-present npm run test --if-present - name: Upload artifact for deployment job uses: actions/upload-artifact@v3 with: name: node-app path: . deploy: permissions: contents: none runs-on: ubuntu-latest needs: build environment: name: 'Development' url: ${{ steps.deploy-to-webapp.outputs.webapp-url }} steps: - name: Download artifact from build job uses: actions/download-artifact@v3 with: name: node-app - name: 'Deploy to Azure WebApp' id: deploy-to-webapp uses: azure/webapps-deploy@v2
with:
app-name: ${{ env.AZURE_WEBAPP_NAME }}
publish-profile: ${{ secrets.AZURE_WEBAPP_PUBLISH_PROFILE }}
package: ${{ env.AZURE_WEBAPP_PACKAGE_PATH }}

This GitHub Actions workflow file defines two main jobs: build and deploy. The build job is responsible for checking out the code, setting up the Node.js environment, installing dependencies, building and testing the project, and uploading the build results as artifacts. The deploy job downloads the artifacts from the build job and deploys them to the Azure Web App. The entire process automates the entire process from code submission to application deployment.

2. Golang multi-version

Example: The following is a go multi-version build example

name: Go

on: [push]

jobs:
build:

runs-on: ubuntu-latest
strategy:
matrix:
go-version: [ '1.19', '1.20', '1.21.x' ]

steps:
- uses: actions/checkout@v4
- name: Setup Go ${{ matrix.go-version }}
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
# You can test your matrix by printing the current Go version
- name: Display Go version
run: go version

This GitHub Actions workflow file mainly defines the build support for multi-version golang environments.

3. Container example

Example: Run Docker container service in a node with runners as ubuntu

name: CI
on:
push:
branches: [ main ]
jobs:
container-test-job:
runs-on: ubuntu-latest
container:
image: node:18
env:
NODE_ENV: development
ports:
- 80 # Map the container port 80 to the host port 80.
volumes:
- my_docker_volume:/volume_mount # Mount a Docker volume named my_docker_volume to the container's /volume_mount path
options: --cpus 1 # Limit the number of CPUs used by the container to 1 core
steps:
- name: Check for dockerenv file
run: (ls /.dockerenv && echo Found dockerenv) || (echo No dockerenv)

This GitHub Actions workflow file defines a CI job that runs on a recent Ubuntu VM and uses the Node.js 18 Docker image. The job configures environment variables, port mappings, volume mounts, and CPU limits. The only step is to check if a .dockerenv file exists to confirm that the current environment is a Docker container.

4. Deploy the application to AWS ECS

# This workflow will build and push a new container image to Amazon ECR,
# and then will deploy a new task definition to Amazon ECS, when there is a push to the "master" branch.
#
# To use this workflow, you will need to complete the following set-up steps:
#
# 1. Create an ECR repository to store your images.
#    For example: `aws ecr create-repository --repository-name my-ecr-repo --region us-east-2`.
#    Replace the value of the `ECR_REPOSITORY` environment variable in the workflow below with your repository's name.
#    Replace the value of the `AWS_REGION` environment variable in the workflow below with your repository's region.
#
# 2. Create an ECS task definition, an ECS cluster, and an ECS service.
#    For example, follow the Getting Started guide on the ECS console:
#      https://us-east-2.console.aws.amazon.com/ecs/home?region=us-east-2#/firstRun
#    Replace the value of the `ECS_SERVICE` environment variable in the workflow below with the name you set for the Amazon ECS service.
#    Replace the value of the `ECS_CLUSTER` environment variable in the workflow below with the name you set for the cluster.
#
# 3. Store your ECS task definition as a JSON file in your repository.
#    The format should follow the output of `aws ecs register-task-definition --generate-cli-skeleton`.
#    Replace the value of the `ECS_TASK_DEFINITION` environment variable in the workflow below with the path to the JSON file.
#    Replace the value of the `CONTAINER_NAME` environment variable in the workflow below with the name of the container
#    in the `containerDefinitions` section of the task definition.
#
# 4. Store an IAM user access key in GitHub Actions secrets named `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
#    See the documentation for each action used below for the recommended IAM policies for this IAM user,
#    and best practices on handling the access key credentials.

name: Deploy to Amazon ECS

on:
  push:
    branches: [ "master" ]

env:
  AWS_REGION: MY_AWS_REGION                   # set this to your preferred AWS region, e.g. us-west-1
  ECR_REPOSITORY: MY_ECR_REPOSITORY           # set this to your Amazon ECR repository name
  ECS_SERVICE: MY_ECS_SERVICE                 # set this to your Amazon ECS service name
  ECS_CLUSTER: MY_ECS_CLUSTER                 # set this to your Amazon ECS cluster name
  ECS_TASK_DEFINITION: MY_ECS_TASK_DEFINITION # set this to the path to your Amazon ECS task definition
                                               # file, e.g. .aws/task-definition.json
  CONTAINER_NAME: MY_CONTAINER_NAME           # set this to the name of the container in the
                                               # containerDefinitions section of your task definition

permissions:
  contents: read

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    environment: production

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

    - name: Login to Amazon ECR
      id: login-ecr
      uses: aws-actions/amazon-ecr-login@v1

    - name: Build, tag, and push image to Amazon ECR
      id: build-image
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        IMAGE_TAG: ${{ github.sha }}
      run: |
        # Build a docker container and
        # push it to ECR so that it can
        # be deployed to ECS.
        docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
        docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
        echo "image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT        

    - name: Fill in the new image ID in the Amazon ECS task definition
      id: task-def
      uses: aws-actions/amazon-ecs-render-task-definition@v1
      with:
        task-definition: ${{ env.ECS_TASK_DEFINITION }}
        container-name: ${{ env.CONTAINER_NAME }}
        image: ${{ steps.build-image.outputs.image }}

    - name: Deploy Amazon ECS task definition
      uses: aws-actions/amazon-ecs-deploy-task-definition@v1
      with:
        task-definition: ${{ steps.task-def.outputs.task-definition }}
        service: ${{ env.ECS_SERVICE }}
        cluster: ${{ env.ECS_CLUSTER }}
        wait-for-service-stability: true

This GitHub Actions workflow The file defines a CI/CD process for building and deploying Docker images to Amazon ECS. The design content includes: checking out code, configuring AWS credentials, logging in to Amazon ECR, building tags and pushing images to ECR, and updating ECS task definitions. Image ID in, deploy the updated ECS task definition.

5. Combine with Helm to deploy the application to the AKS cluster

# This workflow will build and push an application to a Azure Kubernetes Service (AKS) cluster when you push your code
#
# This workflow assumes you have already created the target AKS cluster and have created an Azure Container Registry (ACR)
# The ACR should be attached to the AKS cluster
# For instructions see:
#   - https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal
#   - https://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-portal
#   - https://learn.microsoft.com/en-us/azure/aks/cluster-container-registry-integration?tabs=azure-cli#configure-acr-integration-for-existing-aks-clusters
#   - https://github.com/Azure/aks-create-action
#
# To configure this workflow:
#
# 1. Set the following secrets in your repository (instructions for getting these
#    https://docs.microsoft.com/en-us/azure/developer/github/connect-from-azure?tabs=azure-cli%2Clinux)):
#    - AZURE_CLIENT_ID
#    - AZURE_TENANT_ID
#    - AZURE_SUBSCRIPTION_ID
#
# 2. Set the following environment variables (or replace the values below):
#    - AZURE_CONTAINER_REGISTRY (name of your container registry / ACR)
#    - CONTAINER_NAME (name of the container image you would like to push up to your ACR)
#    - RESOURCE_GROUP (where your cluster is deployed)
#    - CLUSTER_NAME (name of your AKS cluster)
#    - IMAGE_PULL_SECRET_NAME (name of the ImagePullSecret that will be created to pull your ACR image)
#
# 3. Choose the appropriate render engine for the bake step https://github.com/Azure/k8s-bake. The config below assumes Helm.
#    Set your helmChart, overrideFiles, overrides, and helm-version to suit your configuration.
#    - CHART_PATH (path to your helm chart)
#    - CHART_OVERRIDE_PATH (path to your helm chart with override values)
#
# For more information on GitHub Actions for Azure, refer to https://github.com/Azure/Actions
# For more samples to get started with GitHub Action workflows to deploy to Azure, refer to https://github.com/Azure/actions-workflow-samples
# For more options with the actions used below please refer to https://github.com/Azure/login

name: Build and deploy an app to AKS with Helm

on:
  push:
    branches: ["master"]
  workflow_dispatch:

env:
  AZURE_CONTAINER_REGISTRY: "your-azure-container-registry"
  CONTAINER_NAME: "your-container-name"
  RESOURCE_GROUP: "your-resource-group"
  CLUSTER_NAME: "your-cluster-name"
  CHART_PATH: "your-chart-path"
  CHART_OVERRIDE_PATH: "your-chart-override-path"

jobs:
  buildImage:
    permissions:
      contents: read
      id-token: write
    runs-on: ubuntu-latest
    steps:
      # Checks out the repository this file is in
      - uses: actions/checkout@v4

      # Logs in with your Azure credentials
      - name: Azure login
        uses: azure/login@v1.4.6
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

      # Builds and pushes an image up to your Azure Container Registry
      - name: Build and push image to ACR
        run: |
          az acr build --image ${{ env.AZURE_CONTAINER_REGISTRY }}.azurecr.io/${{ env.CONTAINER_NAME }}:${{ github.sha }} --registry ${{ env.AZURE_CONTAINER_REGISTRY }} -g ${{ env.RESOURCE_GROUP }} .          

  deploy:
    permissions:
      actions: read
      contents: read
      id-token: write
    runs-on: ubuntu-latest
    needs: [buildImage]
    steps:
      # Checks out the repository this file is in
      - uses: actions/checkout@v4

      # Logs in with your Azure credentials
      - name: Azure login
        uses: azure/login@v1.4.6
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

      # Use kubelogin to configure your kubeconfig for Azure auth
      - name: Set up kubelogin for non-interactive login
        uses: azure/use-kubelogin@v1
        with:
          kubelogin-version: 'v0.0.25'

      # Retrieves your Azure Kubernetes Service cluster's kubeconfig file
      - name: Get K8s context
        uses: azure/aks-set-context@v3
        with:
          resource-group: ${{ env.RESOURCE_GROUP }}
          cluster-name: ${{ env.CLUSTER_NAME }}
          admin: 'false'
          use-kubelogin: 'true'

      # Runs Helm to create manifest files
      - name: Bake deployment
        uses: azure/k8s-bake@v2
        with:
          renderEngine: "helm"
          helmChart: ${{ env.CHART_PATH }}
          overrideFiles: ${{ env.CHART_OVERRIDE_PATH }}
          overrides: |
            replicas:2            
          helm-version: "latest"
        id: bake

      # Deploys application based on manifest files from previous step
      - name: Deploy application
        uses: Azure/k8s-deploy@v4
        with:
          action: deploy
          manifests: ${{ steps.bake.outputs.manifestsBundle }}
          images: |
            ${{ env.AZURE_CONTAINER_REGISTRY }}.azurecr.io/${{ env.CONTAINER_NAME }}:${{ github.sha }}

This GitHub Actions workflow file defines a CI/CD process for building and pushing a Docker image to Azure Container Registry (ACR), then Deploy the application to Azure Kubernetes Service (AKS) cluster. The main steps include: check out the code, log in with Azure credentials, build and push the Docker image to ACR, get the kubeconfig file of the AKS cluster, render the deployment manifest file with Helm, and deploy the application.

6. Combine with Helm to deploy the application to Alibaba Cloud ACK cluster

# This workflow will build and push a new container image to Alibaba Cloud Container Registry (ACR),
# and then will deploy it to Alibaba Cloud Container Service for Kubernetes (ACK), when there is a push to the "master" branch.
#
# To use this workflow, you will need to complete the following set-up steps:
#
# 1. Create an ACR repository to store your container images.
#    You can use ACR EE instance for more security and better performance.
#    For instructions see https://www.alibabacloud.com/help/doc-detail/142168.htm
#
# 2. Create an ACK cluster to run your containerized application.
#    You can use ACK Pro cluster for more security and better performance.
#    For instructions see https://www.alibabacloud.com/help/doc-detail/95108.htm
#
# 3. Store your AccessKey pair in GitHub Actions secrets named `ACCESS_KEY_ID` and `ACCESS_KEY_SECRET`.
#    For instructions on setting up secrets see: https://developer.github.com/actions/managing-workflows/storing-secrets/
#
# 4. Change the values for the REGION_ID, REGISTRY, NAMESPACE, IMAGE, ACK_CLUSTER_ID, and ACK_DEPLOYMENT_NAME.
#

name: Build and Deploy to ACK

on:
  push:
    branches: [ "master" ]

# Environment variables available to all jobs and steps in this workflow.
env:
  REGION_ID: cn-hangzhou
  REGISTRY: registry.cn-hangzhou.aliyuncs.com
  NAMESPACE: namespace
  IMAGE: repo
  TAG: ${{ github.sha }}
  ACK_CLUSTER_ID: clusterID
  ACK_DEPLOYMENT_NAME: nginx-deployment

  ACR_EE_REGISTRY: myregistry.cn-hangzhou.cr.aliyuncs.com
  ACR_EE_INSTANCE_ID: instanceID
  ACR_EE_NAMESPACE: namespace
  ACR_EE_IMAGE: repo
  ACR_EE_TAG: ${{ github.sha }}

permissions:
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    environment: production

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    # 1.1 Login to ACR
    - name: Login to ACR with the AccessKey pair
      uses: aliyun/acr-login@v1
      with:
        region-id: "${{ env.REGION_ID }}"
        access-key-id: "${{ secrets.ACCESS_KEY_ID }}"
        access-key-secret: "${{ secrets.ACCESS_KEY_SECRET }}"

    # 1.2 Build and push image to ACR
    - name: Build and push image to ACR
      run: |
        docker build --tag "$REGISTRY/$NAMESPACE/$IMAGE:$TAG" .
        docker push "$REGISTRY/$NAMESPACE/$IMAGE:$TAG"        

    # 1.3 Scan image in ACR
    - name: Scan image in ACR
      uses: aliyun/acr-scan@v1
      with:
        region-id: "${{ env.REGION_ID }}"
        access-key-id: "${{ secrets.ACCESS_KEY_ID }}"
        access-key-secret: "${{ secrets.ACCESS_KEY_SECRET }}"
        repository: "${{ env.NAMESPACE }}/${{ env.IMAGE }}"
        tag: "${{ env.TAG }}"

    # 2.1 (Optional) Login to ACR EE
    - uses: actions/checkout@v4
    - name: Login to ACR EE with the AccessKey pair
      uses: aliyun/acr-login@v1
      with:
        login-server: "https://${{ env.ACR_EE_REGISTRY }}"
        region-id: "${{ env.REGION_ID }}"
        access-key-id: "${{ secrets.ACCESS_KEY_ID }}"
        access-key-secret: "${{ secrets.ACCESS_KEY_SECRET }}"
        instance-id: "${{ env.ACR_EE_INSTANCE_ID }}"

    # 2.2 (Optional) Build and push image ACR EE
    - name: Build and push image to ACR EE
      run: |
        docker build -t "$ACR_EE_REGISTRY/$ACR_EE_NAMESPACE/$ACR_EE_IMAGE:$TAG" .
        docker push "$ACR_EE_REGISTRY/$ACR_EE_NAMESPACE/$ACR_EE_IMAGE:$TAG"        
    # 2.3 (Optional) Scan image in ACR EE
    - name: Scan image in ACR EE
      uses: aliyun/acr-scan@v1
      with:
        region-id: "${{ env.REGION_ID }}"
        access-key-id: "${{ secrets.ACCESS_KEY_ID }}"
        access-key-secret: "${{ secrets.ACCESS_KEY_SECRET }}"
        instance-id: "${{ env.ACR_EE_INSTANCE_ID }}"
        repository: "${{ env.ACR_EE_NAMESPACE}}/${{ env.ACR_EE_IMAGE }}"
        tag: "${{ env.ACR_EE_TAG }}"

    # 3.1 Set ACK context
    - name: Set K8s context
      uses: aliyun/ack-set-context@v1
      with:
        access-key-id: "${{ secrets.ACCESS_KEY_ID }}"
        access-key-secret: "${{ secrets.ACCESS_KEY_SECRET }}"
        cluster-id: "${{ env.ACK_CLUSTER_ID }}"

    # 3.2 Deploy the image to the ACK cluster
    - name: Set up Kustomize
      run: |-
        curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash /dev/stdin 3.8.6        
    - name: Deploy
      run: |-
        ./kustomize edit set image REGISTRY/NAMESPACE/IMAGE:TAG=$REGISTRY/$NAMESPACE/$IMAGE:$TAG
        ./kustomize build . | kubectl apply -f -
        kubectl rollout status deployment/$ACK_DEPLOYMENT_NAME
        kubectl get services -o wide

This GitHub Actions works The flow file defines a CI/CD process for building and pushing Docker images to Alibaba Cloud Container Registry (ACR), and then deploying the application to Alibaba Cloud Container Service for Kubernetes (ACK). The specific steps include: Checking out code, use AccessKey to log in to ACR, build and push Docker images to ACR, scan images in ACR, set the context of ACK cluster, set Kustomize, and deploy applications.

7. Combine with Helm to deploy applications to Google Cloud Cluster

# This workflow will build a docker container, publish it to Google Container
# Registry, and deploy it to GKE when there is a push to the "master"
# branch.
#
# To configure this workflow:
#
# 1. Enable the following Google Cloud APIs:
#
#    - Artifact Registry (artifactregistry.googleapis.com)
#    - Google Kubernetes Engine (container.googleapis.com)
#    - IAM Credentials API (iamcredentials.googleapis.com)
#
#    You can learn more about enabling APIs at
#    https://support.google.com/googleapi/answer/6158841.
#
# 2. Ensure that your repository contains the necessary configuration for your
#    Google Kubernetes Engine cluster, including deployment.yml,
#    kustomization.yml, service.yml, etc.
#
# 3. Create and configure a Workload Identity Provider for GitHub:
#    https://github.com/google-github-actions/auth#preferred-direct-workload-identity-federation.
#
#    Depending on how you authenticate, you will need to grant an IAM principal
#    permissions on Google Cloud:
#
#    - Artifact Registry Administrator (roles/artifactregistry.admin)
#    - Kubernetes Engine Developer (roles/container.developer)
#
#    You can learn more about setting IAM permissions at
#    https://cloud.google.com/iam/docs/manage-access-other-resources
#
# 5. Change the values in the "env" block to match your values.

name: 'Build and Deploy to GKE'

on:
  push:
    branches:
      - '"master"'

env:
  PROJECT_ID: 'my-project' # TODO: update to your Google Cloud project ID
  GAR_LOCATION: 'us-central1' # TODO: update to your region
  GKE_CLUSTER: 'cluster-1' # TODO: update to your cluster name
  GKE_ZONE: 'us-central1-c' # TODO: update to your cluster zone
  DEPLOYMENT_NAME: 'gke-test' # TODO: update to your deployment name
  REPOSITORY: 'samples' # TODO: update to your Artifact Registry docker repository name
  IMAGE: 'static-site'
  WORKLOAD_IDENTITY_PROVIDER: 'projects/123456789/locations/global/workloadIdentityPools/my-pool/providers/my-provider' # TODO: update to your workload identity provider

jobs:
  setup-build-publish-deploy:
    name: 'Setup, Build, Publish, and Deploy'
    runs-on: 'ubuntu-latest'
    environment: 'production'

    permissions:
      contents: 'read'
      id-token: 'write'

    steps:
      - name: 'Checkout'
        uses: 'actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332' # actions/checkout@v4

      # Configure Workload Identity Federation and generate an access token.
      #
      # See https://github.com/google-github-actions/auth for more options,
      # including authenticating via a JSON credentials file.
      - id: 'auth'
        name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/auth@f112390a2df9932162083945e46d439060d66ec2' # google-github-actions/auth@v2
        with:
          workload_identity_provider: '${{ env.WORKLOAD_IDENTITY_PROVIDER }}'

      # Authenticate Docker to Google Cloud Artifact Registry
      - name: 'Docker Auth'
        uses: 'docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567' # docker/login-action@v3
        with:
          username: 'oauth2accesstoken'
          password: '${{ steps.auth.outputs.auth_token }}'
          registry: '${{ env.GAR_LOCATION }}-docker.pkg.dev'

      # Get the GKE credentials so we can deploy to the cluster
      - name: 'Set up GKE credentials'
        uses: 'google-github-actions/get-gke-credentials@6051de21ad50fbb1767bc93c11357a49082ad116' # google-github-actions/get-gke-credentials@v2
        with:
          cluster_name: '${{ env.GKE_CLUSTER }}'
          location: '${{ env.GKE_ZONE }}'

      # Build the Docker image
      - name: 'Build and push Docker container'
        run: |-
          DOCKER_TAG="${GAR_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${REPOSITORY}/${IMAGE}:${GITHUB_SHA}"

          docker build \
            --tag "${DOCKER_TAG}" \
            --build-arg GITHUB_SHA="${GITHUB_SHA}" \
            --build-arg GITHUB_REF="${GITHUB_REF}" \
            .

          docker push "${DOCKER_TAG}"          

      # Set up kustomize
      - name: 'Set up Kustomize'
        run: |-
          curl -sfLo kustomize https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.4.3/kustomize_v5.4.3_linux_amd64.tar.gz
          chmod u+x ./kustomize          

      # Deploy the Docker image to the GKE cluster
      - name: 'Deploy to GKE'
        run: |-
          # replacing the image name in the k8s template
          ./kustomize edit set image LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/IMAGE:TAG=$GAR_LOCATION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY/$IMAGE:$GITHUB_SHA
          ./kustomize build . | kubectl apply -f -
          kubectl rollout status deployment/$DEPLOYMENT_NAME
          kubectl get services -o wide

This GitHub Actions workflow file defines a CI/CD process for building and pushing a Docker image to Google Cloud Artifact Registry (GAR) and then deploying the application to Google Kubernetes Engine (GKE ).

The specific steps include: check out the code, authenticate to Google Cloud using Workload Identity Provider, authenticate Docker to push the image to Artifact Registry, obtain the credentials of the GKE cluster, build and push the Docker image to Artifact Registry, set up Kustomize, and deploy the application program to GKE.

Github Actions Advantages and Disadvantages

Through using it for a few days, I feel that GitHub Actions is a powerful tool for automating software development workflows such as continuous integration (CI), continuous deployment (CD), testing, and release etc. It is directly integrated on the GitHub platform, providing rich functionality and flexibility.

However, like all tools, GitHub Actions also has its pros and cons. Here is a summary of its advantages and disadvantages:

Advantages

Seamless integration:

Seamless integration with GitHub: GitHub Actions is directly integrated into the GitHub platform, allowing easy access to code in the repository and metadata.
Automatic triggering: Workflows can be automatically triggered based on events such as code push, pull request, tag creation, etc.

Rich market:

Action market: GitHub provides A rich Action Marketplace, where users can find a large number of pre-built Actions for common tasks such as code quality checks, deployments, notifications, etc.
Community Support: A large developer community provides a wealth of resources and Best practices help users solve problems quickly.

Flexibility and scalability:

Custom workflow: Users can customize complex multi-step workflows to meet specific needs.
- Environment variables and key management*: Support environment variables and key management to ensure the security of sensitive information.
Parallel execution: Support parallel execution of multiple tasks to improve efficiency.

Built-in security and audit:

Security features: GitHub provides a variety of security features, such as dependency review, code scanning, etc.
Audit log: Detailed audit log records , easy to track and debug.

Cost-effectiveness:

Free plan: For open source projects and individual developers, GitHub provides free Actions usage quota.
Pay as you go: For Enterprise users can pay based on actual usage to avoid wasting resources.

Disadvantages

Learning curve:

Complexity: Although it provides rich functions, it requires configuration and management of complex workflows It may take some time to learn and experience.
Documentation and Examples: Although there is a lot of documentation and examples, it may take some time for beginners to find the right resources and best practices.

Performance and resource limits:

Resource limits: The free plan has limited resources, which may not be enough for large projects or high-frequency CI/CD pipelines.
Performance fluctuations: During peak hours, GitHub Actions Performance may be affected, resulting in slower builds and deployments.

Depends on GitHub:

Platform lock-in: Using GitHub Actions will increase your reliance on the GitHub platform, and the migration cost is high.
External integration: Although GitHub Actions supports integration with external tools and Service integration, but may require more configuration and maintenance than other CI/CD tools.

Difficult debugging:

Debugging tools: Although logs and outputs are provided, debugging is complex Multi-step workflows can be difficult, especially when problems arise.
Lack of visualization: GitHub Actions has limited visualization and monitoring capabilities compared to some commercial CI/CD tools.

*Security risks *:

Third-party Action: Using third-party Action may bring security risks and requires careful evaluation and selection.
Permission management: Improper permission management may lead to security vulnerabilities and requires careful configuration.

Summary

GitHub Actions is a powerful and flexible CI/CD tool that is particularly suitable for projects integrated with GitHub. Its advantages lie in seamless integration, rich market, flexibility and security.

However, it also has some disadvantages, such as a steep learning curve, resource limitations, platform lock-in, and debugging difficulties. The choice of whether to use GitHub Actions should be based on the specific needs of the project and the team’s technology stack.

For small projects and open source projects, GitHub Actions is a very good choice; for large enterprise projects, you may need to consider more professional CI/CD tools.

Wanzi's Blog – Notes on life, tech, and DevOps.

笔记

About

About Me

About This Site

Wanzi’s Daily Life

Purpose of the Site

Developing Existing Projects with OpenSpec

1. Initialize Project:

2. Use Explore to Analyze Existing Code

3. Complete Configuration

4. Start Incremental Development

5. What to do if the feature doesn’t work?

1. Modify the specification (spec.md):

2. Re-apply (/opsx:apply):

3. Verify again:

6. How to ensure synchronization with spec.md after modifying code?

1. Best Strategy: Modify Documentation First, Then Generate Code (Spec-First)

2. Remedial Strategy: Code Modified, Reverse Sync Documentation (Reverse Sync)

3. Forced Synchronization During Archiving (/opsx:archive)

7. Discovering spec.md lags behind code after archiving

1. Build “Verify -> Archive” Muscle Memory

2. Use Git Hooks (Pre-commit Hook) for Mandatory Checks

3. Remedial Action

Kaku Best Practices

Kaku Tool Introduction

What is Kaku

Supported AI Tools

Supported AI Providers

Kaku Installation & Configuration

Installation Methods

First-time Configuration

Configuration File

AI Settings Details

Configuring AI Providers

Manual Configuration

MiniMax Configuration

Kaku Assistant Features

What is Kaku Assistant

Enabling/Disabling Assistant

Using Assistant

Shortcut Keys

Global Operations

Tab Navigation

Pane Operations

AI Function Shortcuts

Git and File Management

Display Adjustment

Shell Navigation Features

Smart Directory Jumping (z command)

Other Shell Enhancements

CLI Tool Integration

Tools Installed by kaku init

kaku cli Commands

Advanced Configuration

Configuration File Example

Built-in Shell Configuration

Troubleshooting

kaku command not found

Diagnosing Issues

Enterprise Proxy Configuration

Differences from Claude Code

Best Practices Summary

1. Daily Development Workflow

2. AI Feature Usage

3. Shortcut Habits

4. Personalized Configuration

Implementing Resource Isolation in Doris Offline Analysis Data Warehouse Cluster

I. Background

II. Workload Group Introduction

III. Doris Cluster Adjustment to Support Cgroup (Manual Deployment)

3.1 Cgroup Environment Configuration

3.2 Create Doris CGroup Directory

3.3 CGroup v2 Additional Configuration

3.4 Configure BE and Restart

3.5 Persistent Configuration (Optional)

IV. Batch Enable Cgroup on Doris Nodes via Ansible

4.1 Preparation

4.2 Configure Host Inventory

4.3 Create Ansible Playbook

2. Re-apply (`/opsx:apply`):

3. Forced Synchronization During Archiving (`/opsx:archive`)