Doris on Wanzi's Blog – Notes on life, tech, and DevOps.

Implementing Resource Isolation in Doris Offline Analysis Data Warehouse Cluster

wzdushu@gmail.com (wanzi) — Sat, 28 Feb 2026 09:30:00 +0800

I. Background

With rapid business development, the current offline analysis data warehouse Doris cluster is facing resource contention issues due to diverse computing needs such as daily data science jobs, offline batch processing, and customer-side report queries. To address this, resource limits need to be implemented for different account resource requests. After research, the official Doris Workload Group solution has been adopted.

Doris version: doris-2.1.8-1-834d802457

II. Workload Group Introduction

Workload Group is an in-process resource isolation mechanism provided by Apache Doris that achieves resource isolation between different business loads through fine-grained division of CPU, memory, and IO resources within BE processes.

The principle is as shown in the diagram below:

Currently supported isolation capabilities include:

Resource Type	Isolation Method	Description
CPU	Soft limit / Hard limit	Soft limit allocates by weight, hard limit is absolute upper bound
Memory	Soft limit / Hard limit	Hard limit automatically kills queries to free up memory when exceeded
IO	Speed limit	Limits IO throughput for reading local/remote files
Concurrency	Queuing mechanism	Exceeding concurrency queries enter queue to wait

For more information: https://doris.apache.org/zh-CN/docs/2.1/admin-manual/workload-management/workload-group

III. Doris Cluster Adjustment to Support Cgroup (Manual Deployment)

3.1 Cgroup Environment Configuration

Check the system’s supported CGroup version:

root@doristest:~# cat /proc/filesystems | grep cgroup
nodev   cgroup
nodev   cgroup2

Confirm the effective CGroup version:

root@doristest:~# ls /sys/fs/cgroup/cpu/
ls: cannot access '/sys/fs/cgroup/cpu/': No such file or directory 
root@doristest:~# ls /sys/fs/cgroup/cgroup.controllers
/sys/fs/cgroup/cgroup.controllers # Indicates cgroup v2 is in effect

3.2 Create Doris CGroup Directory

# ========== CGroup v1 Operations ==========
mkdir /sys/fs/cgroup/cpu/doris

# Set permissions (root is the user running BE)
chmod 770 /sys/fs/cgroup/cpu/doris
chown -R root:root /sys/fs/cgroup/cpu/doris


# ========== CGroup v2 Operations ==========
mkdir /sys/fs/cgroup/doris

# Set permissions
chmod 770 /sys/fs/cgroup/doris
chown -R root:root /sys/fs/cgroup/doris  # Adjust as needed, currently root

3.3 CGroup v2 Additional Configuration

CGroup v2 has stricter permission control and requires additional configuration:

# Modify root directory cgroup.procs file permissions
chmod a+w /sys/fs/cgroup/cgroup.procs

# Enable CPU controller
# Enter doris directory
cd /sys/fs/cgroup/doris

# Enable CPU controller (by modifying parent subtree_control)
# Already set by default, can be ignored
echo +cpu > ../cgroup.subtree_control

# Verify: Check if cpu.max file appears in doris directory
/sys/fs/cgroup/doris/cpu.max
# And cgroup.controllers should contain cpu
# cat /sys/fs/cgroup/cgroup.controllers 
cpuset cpu io memory hugetlb pids rdma misc

3.4 Configure BE and Restart

Edit the be.conf file:

# ========== CGroup v1 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris

# ========== CGroup v2 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/doris

Restart BE service:

# Restart BE
./bin/stop_be.sh
./bin/start_be.sh

Verification: Check the be.INFO log

grep "add thread" log/be.INFO
# Appearance of "add thread xxx to group" indicates successful configuration

3.5 Persistent Configuration (Optional)

CGroup configuration will be cleared after machine reboot. It is recommended to use systemd to create an auto-configuration service on boot:

# /etc/systemd/system/doris-cgroup.service
[Unit]
Description=Doris CGroup Setup
Before=doris-be.service

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'mkdir -p /sys/fs/cgroup/cpu/doris && chmod 770 /sys/fs/cgroup/cpu/doris && chown -R doris:doris /sys/fs/cgroup/cpu/doris'

[Install]
WantedBy=multi-user.target

Set to auto-start on boot:

systemctl enable doris-cgroup.service

IV. Batch Enable Cgroup on Doris Nodes via Ansible

4.1 Preparation

Before executing batch deployment, confirm the following information:

Confirm CGroup version: CGroup version (v1 or v2) for all Doris BE nodes
Prepare host inventory: Edit Ansible’s hosts file to add all BE nodes
Confirm BE user: User running Doris BE service (currently root)

4.2 Configure Host Inventory

Edit the hosts file:

1
2
3

[doris_dabe_nodes]
# If nodes have different CGroup versions, they can be grouped. Here only specifying offline analysis warehouse BE nodes
10.18.7.160

4.3 Create Ansible Playbook

Create file setup_doris_cgroup.yaml:

---
- name: Setup Doris CGroup Configuration
  hosts: doris_dabe_nodes
  become: true
  vars:
    # CGroup configuration path (automatically selected based on version)
    cgroup_v1_path: "/sys/fs/cgroup/cpu/doris"
    cgroup_v2_path: "/sys/fs/cgroup/doris"
    doris_user: "root"
    doris_group: "root"
    
  tasks:
    - name: Detect CGroup version
      stat:
        path: /sys/fs/cgroup/cpu/
      register: cgroup_v1_check
      
    - name: Set CGroup path based on version
      set_fact:
        cgroup_path: "{{ cgroup_v1_path if cgroup_v1_check.stat.exists else cgroup_v2_path }}"
        cgroup_version: "{{ 'v1' if cgroup_v1_check.stat.exists else 'v2' }}"
        
    - name: Display detected CGroup version
      debug:
        msg: "Node {{ inventory_hostname }} uses CGroup {{ cgroup_version }}, path: {{ cgroup_path }}"

    - name: Create CGroup directory
      file:
        path: "{{ cgroup_path }}"
        state: directory
        mode: '0770'
        owner: "{{ doris_user }}"
        group: "{{ doris_group }}"
      register: cgroup_dir_result
      
    - name: Configure CGroup v2 specific settings
      block:
        - name: Enable write permission on cgroup.procs
          file:
            path: /sys/fs/cgroup/cgroup.procs
            mode: 'a+w'
            
        - name: Enable CPU controller for CGroup v2
          shell: |
            if ! grep -q '+cpu' /sys/fs/cgroup/cgroup.subtree_control 2>/dev/null; then
              echo '+cpu' > /sys/fs/cgroup/cgroup.subtree_control
            fi
          args:
            executable: /bin/bash
          ignore_errors: yes
          
        - name: Verify CPU controller is enabled
          stat:
            path: "{{ cgroup_path }}/cpu.max"
          register: cpu_controller_check
          
        - name: Display CPU controller status
          debug:
            msg: "CPU controller is {{ 'enabled' if cpu_controller_check.stat.exists else 'NOT enabled' }}"
      when: cgroup_version == 'v2'

    - name: Verify CGroup directory structure
      shell: |
        echo "=== CGroup Version: {{ cgroup_version }} ==="
        echo "CGroup Path: {{ cgroup_path }}"
        ls -la {{ cgroup_path }} 2>/dev/null || echo "Directory verified"
        echo ""
        if [ "{{ cgroup_version }}" = "v2" ]; then
          echo "=== CPU Controller Check ==="
          cat /sys/fs/cgroup/cgroup.controllers 2>/dev/null
          test -f {{ cgroup_path }}/cpu.max && echo "cpu.max: EXISTS" || echo "cpu.max: NOT FOUND"
        fi
      args:
        executable: /bin/bash
      register: verify_result
      
    - name: Display verification results
      debug:
        var: verify_result.stdout_lines

    - name: Create systemd service for CGroup persistence (optional)
      copy:
        dest: /etc/systemd/system/doris-cgroup.service
        content: |
          [Unit]
          Description=Doris CGroup Setup
          Before=doris-be.service

          [Service]
          Type=oneshot
          ExecStart=/bin/bash -c 'mkdir -p {{ cgroup_path }} && chmod 770 {{ cgroup_path }} && chown -R {{ doris_user }}:{{ doris_group }} {{ cgroup_path }}'
          {% if cgroup_version == 'v2' %}
          ExecStartPost=/bin/bash -c 'chmod a+w /sys/fs/cgroup/cgroup.procs'
          ExecStartPost=/bin/bash -c 'echo "+cpu" > /sys/fs/cgroup/cgroup.subtree_control 2>/dev/null || true'
          {% endif %}

          [Install]
          WantedBy=multi-user.target
        mode: '0644'
      when: cgroup_dir_result is changed
      notify: Enable systemd service

  handlers:   
    - name: Enable systemd service
      systemd:
        name: doris-cgroup
        enabled: yes
        daemon_reload: yes

4.4 Execute Batch Deployment

Syntax check and dry run

# 1. Syntax check
ansible-playbook -i hosts setup_doris_cgroup.yaml --syntax-check

# 2. List all tasks to be executed
ansible-playbook -i hosts setup_doris_cgroup.yaml --list-tasks

# 3. Dry run (not actually executing, only showing changes to be made)
ansible-playbook -i hosts setup_doris_cgroup.yaml --check --diff

4.5 Official Execution and Verification of Deployment Results

Official execution

# Method 1: Execute on all BE nodes
ansible-playbook -i hosts setup_doris_cgroup.yaml

# Method 2: Execute only on specific nodes (e.g., test on single node first)
ansible-playbook -i hosts setup_doris_cgroup.yaml --limit 10.18.7.101

# Method 3: Execute step by step, each task requires confirmation
ansible-playbook -i hosts setup_doris_cgroup.yaml --step

Adjust Doris BE configuration via Doris Manager (rolling restart):

# ========== CGroup v1 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris

# ========== CGroup v2 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/doris

Verification:

# 1. Check if CGroup directory is created
ansible doris_dabe_nodes -i hosts -m shell -a "ls -la /sys/fs/cgroup/doris 2>/dev/null || ls -la /sys/fs/cgroup/cpu/doris 2>/dev/null"

# 2. Check CGroup initialization in BE logs (adjust according to actual BE log path)
ansible doris_dabe_nodes -i hosts -m shell -a "tail -100 /opt/doris/doris/be/log/be.INFO | grep -i 'cgroup\|add thread' | tail -5"

At this point, Doris BE nodes have enabled Cgroup resource limits.

V. Workload Group Management

5.1 Create Workload Group

Basic example (CPU soft limit)

CREATE WORKLOAD GROUP IF NOT EXISTS g1
PROPERTIES (
    "cpu_share" = "1024"
);

Complete configuration example

CREATE WORKLOAD GROUP IF NOT EXISTS etl_group
PROPERTIES (
    -- CPU configuration (soft limit mode)
    "cpu_share" = "2048",
    "cpu_hard_limit" = "40%",
    
    -- Memory configuration (hard limit mode)
    "memory_limit" = "40%",
    "enable_memory_overcommit" = "false",
    
    -- Concurrency control
    "max_concurrency" = "1",
    "max_queue_size" = "100",
    "queue_timeout" = "10000",
    
    -- IO speed limit
    "read_bytes_per_second" = "104857600",        -- 100MB/s
    "remote_read_bytes_per_second" = "52428800"   -- 50MB/s
);

5.2 Attribute Details

Attribute Name	Type	Default Value	Value Range	Description
cpu_share	INT	-1	[1, 10000]	CPU soft limit weight, higher value means higher priority
cpu_hard_limit	INT	-1	[1%, 100%]	CPU hard limit percentage (new in version 2.1)
memory_limit	FLOAT	-1	(0%, 100%]	Memory limit percentage
enable_memory_overcommit	BOOL	true	true/false	true=soft limit (can exceed), false=hard limit
max_concurrency	INT	2147483647	[0, 2147483647]	Maximum concurrent queries
max_queue_size	INT	0	[0, 2147483647]	Queue length (0=no queuing)
queue_timeout	INT	0	[0, 2147483647]	Queue timeout (milliseconds)
scan_thread_num	INT	-1	[1, 2147483647]	Scan thread count (-1=use BE configuration)
max_remote_scan_thread_num	INT	-1	[1, 2147483647]	Maximum scan thread count for external tables
min_remote_scan_thread_num	INT	-1	[1, 2147483647]	Minimum scan thread count for external tables
read_bytes_per_second	INT	-1	[1, 9223372036854775807]	Internal table read IO limit (bytes/second)
remote_read_bytes_per_second	INT	-1	[1, 9223372036854775807]	External table read IO limit (bytes/second)

Notes:

Must specify at least one attribute when creating
CGroup v1 cpu_share default is 1024, range 2-262144
CGroup v2 cpu_share default is 100, range 1-10000
Sum of all Workload Group memory_limit cannot exceed 100%
Sum of all Workload Group cpu_hard_limit cannot exceed 100%

5.3 Modify Workload Group

ALTER WORKLOAD GROUP g1 PROPERTIES('cpu_share' = '4096');

-- Change memory limit to hard limit
ALTER WORKLOAD GROUP g1 PROPERTIES(
    'memory_limit' = '30%',
    'enable_memory_overcommit' = 'false'
);

-- Modify concurrency and queuing parameters
ALTER WORKLOAD GROUP g1 PROPERTIES(
    'max_concurrency' = '100',
    'max_queue_size' = '200',
    'queue_timeout' = '30000'
);

-- Add CPU hard limit (hard limit mode must be enabled first)
ALTER WORKLOAD GROUP g1 PROPERTIES('cpu_hard_limit' = '20%');

5.4 Delete Workload Group

- Delete specified Workload Group
DROP WORKLOAD GROUP g1;

Note: The default normal group cannot be deleted.

VI. User Binding and Authorization

6.1 View Available Workload Groups

-- View Workload Groups that the current user has permission to use
SELECT name FROM information_schema.workload_groups;

6.2 Authorization

-- Grant user permission to use specified Workload Group
GRANT USAGE_PRIV ON WORKLOAD GROUP 'g1' TO 'user_1'@'%';

-- Grant permission to all Workload Groups
GRANT USAGE_PRIV ON WORKLOAD GROUP '*' TO 'user_1'@'%';

-- Revoke permission
REVOKE USAGE_PRIV ON WORKLOAD GROUP 'g1' FROM 'user_1'@'%';

6.3 Binding Method 1: User Properties (Recommended)

Set default Workload Group for user (persistent):

-- Set user's default Workload Group
SET PROPERTY FOR 'user_1' = 'default_workload_group' = 'g1';

-- View user properties
SHOW PROPERTY FOR 'user_1';

VII. Monitoring and Viewing

7.1 View Workload Group List

-- Show statement
SHOW WORKLOAD GROUPS;

-- System table query
SELECT * FROM information_schema.workload_groups;

7.2 View Resource Usage

-- View memory usage of each Workload Group (unit: MB)
SELECT 
    workload_group_id,
    name,
    MEMORY_USAGE_BYTES / 1024 / 1024 AS mem_used_mb,
    CPU_USAGE_PERCENT AS cpu_percent
FROM workload_group_resource_usage;

-- View details of specific Workload Group
SELECT * FROM information_schema.workload_groups WHERE name = 'g1';

7.3 System Table Field Description

workload_groups table

Field	Description
ID	Workload Group ID
NAME	Name
CPU_SHARE	CPU soft limit weight
MEMORY_LIMIT	Memory limit percentage
ENABLE_MEMORY_OVERCOMMIT	Whether memory overcommit is allowed
MAX_CONCURRENCY	Maximum concurrency
MAX_QUEUE_SIZE	Queue length
QUEUE_TIMEOUT	Queue timeout
CPU_HARD_LIMIT	CPU hard limit percentage
SCAN_THREAD_NUM	Scan thread count
READ_BYTES_PER_SECOND	Internal table IO limit
REMOTE_READ_BYTES_PER_SECOND	External table IO limit

VIII. CPU Soft/Hard Limit Mode Switching

8.1 Mode Description

Mode	Characteristics	Applicable Scenarios
CPU soft limit	Allocates CPU time by weight, can use all CPU when idle	Large load fluctuations, want to fully utilize resources
CPU hard limit	Absolute upper limit, cannot exceed even if CPU is idle	Need strict resource isolation, guarantee SLA

Note: Only one mode can be used in a cluster at the same time, cannot be mixed.

8.2 Switch from Soft Limit to Hard Limit

Step 1: Set hard limit values for all Workload Groups

-- Must set cpu_hard_limit for all Groups
ALTER WORKLOAD GROUP g1 PROPERTIES('cpu_hard_limit' = '30%');
ALTER WORKLOAD GROUP g2 PROPERTIES('cpu_hard_limit' = '20%');
ALTER WORKLOAD GROUP normal PROPERTIES('cpu_hard_limit' = '50%');

-- Verify: Sum of all Group's cpu_hard_limit <= 100%
SELECT SUM(cpu_hard_limit) FROM information_schema.workload_groups 
WHERE cpu_hard_limit > 0;

Step 2: Enable cluster hard limit switch

-- Takes effect immediately in memory (invalid after restart)
ADMIN SET FRONTEND CONFIG ("enable_cpu_hard_limit" = "true");

-- Persistent configuration: Modify fe.conf for all FEs
echo "experimental_enable_cpu_hard_limit = true" >> fe/conf/fe.conf

8.3 Switch from Hard Limit Back to Soft Limit

-- Disable hard limit switch (automatically switches back to soft limit mode)
ADMIN SET FRONTEND CONFIG ("enable_cpu_hard_limit" = "false");

For persistence, modify fe.conf to false or delete the configuration.

The above is the overall solution for adding resource limits to the Doris offline analysis data warehouse.

Reference document: https://doris.apache.org/zh-CN/docs/2.1/admin-manual/workload-management/workload-group

Doris Manager Unified Deployment and Multi-Cluster Management

wzdushu@gmail.com (wanzi) — Thu, 12 Feb 2026 09:30:00 +0800

With business expansion and the diversification of product forms, there is a need to deploy multiple sets of offline and real-time data warehouse clusters. While the previous Ansible Playbook method for deploying Doris FE/BE was basically sufficient, it fell short for subsequent Doris version upgrades, scaling, visual management, and onboarding junior operations personnel. Therefore, this document was written to facilitate quick future reference.

1. Overview

1.1 Introduction to Doris Manager

Doris Manager is a graphical operation and maintenance tool launched by the Apache Doris community for unified deployment, management, and monitoring of Doris clusters. It simplifies the entire process from operating system initialization and component deployment to daily operation and maintenance management, making it particularly suitable for rapidly setting up and maintaining large-scale Doris clusters in production environments.

From the current perspective, the official open-source repository is no longer maintained. As SelectDB is a commercial company, for its own development, it has evolved Doris Manager into a free, closed-source operation and maintenance tool, though it can still be downloaded and used.

1.2 Main Features

Cluster Deployment: Automated deployment of FE, BE, Broker, and other components.
Node Management: Unified management of all nodes, supporting scaling out and scaling in.
Configuration Management: Centralized configuration management and parameter adjustment.
Monitoring and Alerting: Cluster status monitoring and alert notifications.
Log Viewing: Centralized log viewing and analysis.

2. Prerequisites

2.1 Environment Planning

Before deployment, please confirm the following information:

Planning Item	Description	Example
Node Planning	Determine FE and BE node IPs and quantities	FE: 3 nodes, BE: 10 nodes
Storage Planning	Plan data storage directories and disk types	/data/doris (ext4, noatime)
Network Planning	Confirm network connectivity between nodes and open ports	9030, 9031, 8060, 9000+
Timezone Planning	Configure timezone based on the business country	Africa/Abidjan (Côte d’Ivoire)
Domain Planning	Manager access domain	doris-manager.example.com

2.2 Software Preparation

Official download address: https://selectdb.com/download/enterprise#manager

Download to the company software repository in advance (needs to be downloaded from the selectdb official website):

Repository Address: https://repo.test.com/bigdata/

Software Package	Version	Description
doris-manager	24.1.5-x64-bin	Doris Manager Management Tool
apache-doris	2.1.8.1-bin-x64	Apache Doris Community Edition

Download Commands:

# Create download directory
mkdir -p /opt/downloads/doris_manager /opt/downloads/doris
cd /opt/downloads/doris_manager

# Download installation packages (from company repository)
wget https://repo.test.com/bigdata/doris-manager-24.1.5-x64-bin.tar.gz
wget https://repo.test.com/bigdata/apache-doris-2.1.8.1-bin-x64.tar.gz

2.3 System Requirements

Component	Minimum Configuration	Recommended Configuration
Operating System	CentOS 7+ / Ubuntu 18.04+	CentOS 7.9 / Ubuntu 20.04
CPU	4 Cores	8 Cores+
Memory	8GB	16GB+
Disk	100GB	SSD 500GB+
Network	Gigabit NIC	10 Gigabit NIC

3. System Initialization and Optimization

Before deploying Doris Manager and the Doris cluster, system initialization configuration is required for all nodes.

3.1 Data Disk Initialization

Important: Production environments must use an independent data disk; do not share it with the system disk.

Manual initialization (single node)

# 1. View data disk devices
lsblk
# Assume the data disk is /dev/vdb

# 2. Create ext4 file system (optimized parameters)
# -m 1: Reduce reserved space from 5% to 1% to increase available space
# -T largefile: Optimize for large file storage
# -L: Set volume label for easy identification
mkfs.ext4 -m 1 -T largefile -L doris-data /dev/vdb

# 3. Create mount point directory
mkdir -p /data

# 4. Mount data disk
mount /dev/vdb /data

# 5. Get disk UUID
blkid /dev/vdb
# Output example: /dev/vdb: UUID="e86ca90c-ca31-460d-b450-1fe914ebae70" TYPE="ext4"

# 6. Configure automatic mount on boot (replace with actual UUID)
echo "UUID=e86ca90c-ca31-460d-b450-1fe914ebae70 /data ext4 defaults,noatime 0 2" >> /etc/fstab
# noatime: Do not update access time to improve performance (especially for big data read/write)
# 0: Do not backup
# 2: Check file system on startup

# 7. Create Doris data directory
mkdir -p /data/doris

# 8. Verify mount
df -h | grep /data

3.2 Disable Swap Partition

Reason: When Doris is under high memory pressure, swap swaps memory pages to disk, causing GC pauses, query lag, or even node pseudo-death, severely affecting performance.

# 1. Temporarily turn off swap
sudo swapoff -a

# 2. Permanently disable swap (comment out the swap line in /etc/fstab)
sudo sed -i '/swap/s/^/#/' /etc/fstab

# 3. Verify swap is turned off
free -h
# Swap should show 0B

swapon --show
# Should have no output, indicating swap is fully closed

3.3 Optimize Kernel Parameters

3.3.1 Increase Virtual Memory Areas (VMA)

Reason: Doris maps a large number of files to memory, requiring sufficient virtual memory areas.

# Add kernel parameters
cat >> /etc/sysctl.conf << EOF
vm.max_map_count = 2000000
EOF

# Apply configuration
sysctl -p

# Verify
sysctl vm.max_map_count
# Should output: vm.max_map_count = 2000000

3.3.2 Optimize Network Parameters (Optional)

# Optimize TCP connection parameters for high concurrency scenarios
cat >> /etc/sysctl.conf << EOF
# TCP optimization parameters
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 2048
EOF

sysctl -p

3.4 Increase File Handle Limits

Reason: Doris BE opens a large number of files (data shards, indexes, logs, etc.), so the file descriptor limit must be increased.

# 1. Modify limits configuration
sudo tee -a /etc/security/limits.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

# 2. Create independent configuration in limits.d directory
sudo tee -a /etc/security/limits.d/20-nproc.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

# 3. Verify configuration
ulimit -n
# Should output: 65536

3.5 Apply All Configurations and Reboot

# Apply all kernel parameters
sysctl -p

# Reboot nodes for all configurations to take effect
reboot

# Verification after reboot
# 1. Check swap: free -h (Swap should be 0)
# 2. Check file handles: ulimit -n (should be 65536)
# 3. Check VMA: sysctl vm.max_map_count (should be 2000000)
# 4. Check data disk: df -h | grep /data

In actual deployment, you can customize virtual machine hosts or cloud host images in advance to include basic initialization operations.

4. Deploy Doris Manager

4.1 Extraction and Installation

# 1. Extract installation package
cd /opt/downloads/doris_manager
tar zxvf doris-manager-24.1.5-x64-bin.tar.gz

# 2. Create deployment directory
mkdir -p /opt/doris/manager

# 3. Move files to deployment directory
mv ./doris-manager-24.1.5-x64-bin/* /opt/doris/manager/

# 4. Prepare Doris installation package resources
# Manager needs local source files for subsequent cluster deployment
mkdir -p /opt/downloads/doris
cp /opt/downloads/doris_manager/apache-doris-2.1.8.1-bin-x64.tar.gz /opt/downloads/doris/
cp /opt/downloads/doris_manager/selectdb-doris-2.0.3-b4-bin-x64.tar.gz /opt/downloads/doris/

4.2 Start Doris Manager

# Start Doris Manager WebServer
cd /opt/doris/manager/
./webserver/bin/start.sh

# View logs (optional)
tail -f webserver/log/webserver.log

Startup Verification: Service listens on port 8004

# Check if port is listening normally
ss -an | grep 8004
# Expected output:
# tcp    LISTEN    0      100    *:8004    *:*

# Or use netstat
netstat -tlnp | grep 8004

4.3 Configure Auto-start on Boot (Recommended)

Use systemd to manage the Doris Manager service, ensuring automatic startup after node restarts.

# 1. Create systemd service file
cat > /etc/systemd/system/doris-manager.service <<'EOF'
[Unit]
Description=Doris Manager WebServer Service
Documentation=https://doris.apache.org/
After=network.target

[Service]
Type=forking
WorkingDirectory=/opt/doris/manager
ExecStart=/opt/doris/manager/webserver/bin/start.sh
ExecStop=/opt/doris/manager/webserver/bin/stop.sh
Restart=on-failure
RestartSec=10
User=root
Group=root
TimeoutStartSec=120
TimeoutStopSec=120
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

# 2. Reload systemd configuration
sudo systemctl daemon-reload

# 3. Enable auto-start on boot
sudo systemctl enable doris-manager

# 4. Check service status
sudo systemctl status doris-manager

4.4 Access Manager UI

Access the Doris Manager management interface via a browser:

http://{manager-ip}:8004

First Access Steps:

Enter the initialization user page to create the first Manager admin user.
The Manager admin account is independent of cluster accounts and is only used for Manager permission control.
Log in with that account after successful creation.

5. Configure Nginx Reverse Proxy

In production environments, it is recommended to configure domain names and HTTPS access via Nginx.

5.1 Nginx Configuration File

server {
  listen 80;
  listen 443 ssl;
  charset utf-8;
  server_name ug-test-doris-manager.test.com;
  index  index.html index.htm index.jsp index.php;

  # SSL configuration (optional)
  # if ($scheme = http) {
  #   rewrite ^(.*)$ https://$host$1 redirect;
  # }

  include rule.d/ssl.conf;
  include rule.d/ssl_test.conf;

  client_max_body_size 5000m;

  location  / {
    include rule.d/proxy_header.conf;
    include rule.d/proxy_connection_small.conf;
    proxy_pass http://10.156.179.58:8004;
  }

  access_log  /var/log/nginx/ug-test-doris-manager.test.com.access.log main;
  error_log   /var/log/nginx/ug-test-doris-manager.test.com.error.log;
}

5.2 Restart Nginx

# Check configuration file syntax
nginx -t

# Restart Nginx
systemctl restart nginx

# Verify
curl -I http://ug-test-doris-manager.test.com

Now you can access Doris Manager via the domain name:

https://ug-test-doris-manager.test.com

6. Initialize Manager Configuration

6.1 Create Admin Account

Access UI Interface: Access Doris Manager via the configured domain or IP.
Create Admin Account:
- Set username and password.
- Service Configuration: Since external network policies may not be open, it is recommended to turn off email alerts first.
- Click to start the doris-manager service.
Enter Management Console: After successful installation, enter the main interface.

6.2 Upload Doris Installation Packages

Upload the Doris installation package on the Manager interface:

apache-doris-2.1.8.1-bin-x64.tar.gz

Or use the command line to place in the specified directory:

1
2
3

# Ensure the path is accessible by Manager
mkdir -p /opt/downloads/doris
cp apache-doris-2.1.8.1-bin-x64.tar.gz /opt/downloads/doris/

7. Deploy Doris Agent

Agent is the proxy program of Doris Manager on each node, responsible for executing deployment and management tasks.

7.1 Get Installation Command

On the Manager interface:

Go to “Cluster Management” -> “Agent Management”.
Click the “Add Agent” button.
Get the installation command (usually two lines).

Example Command:

`1`	`wget http://10.156.179.58:8004/api/download/deploy.sh -O deploy_agent.sh && chmod +x deploy_agent.sh && ./deploy_agent.sh`

Note: Replace the IP address with the actual Manager service IP.

7.2 Execute Installation on Target Nodes

# Execute on every Doris node
wget http://{manager-ip}:8004/api/download/deploy.sh -O deploy_agent.sh
chmod +x deploy_agent.sh
./deploy_agent.sh

7.3 Configure Agent Auto-start on Boot

Create a systemd service file to manage Agent:

# 1. Create systemd service file
cat > /etc/systemd/system/doris-manager-agent.service <<'EOF'
[Unit]
Description=Doris Manager Agent Service
Documentation=https://doris.apache.org/
After=network.target

[Service]
Type=forking
# Type=forking is because the start.sh script starts the process in the background and exits immediately

WorkingDirectory=/root/manager-agent

ExecStart=/root/manager-agent/bin/start.sh
ExecStop=/root/manager-agent/bin/stop.sh

Restart=on-failure
RestartSec=10

User=root
Group=root

TimeoutStartSec=120
TimeoutStopSec=120

StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

# 2. Reload systemd configuration
sudo systemctl daemon-reload

# 3. Enable auto-start on boot
sudo systemctl enable doris-manager-agent

# 4. Start service
sudo systemctl start doris-manager-agent

# 5. Check service status
sudo systemctl status doris-manager-agent

# 6. View real-time logs (if startup fails)
sudo journalctl -u doris-manager-agent -f

7.4 Manually Create Data Directory

Manually create the data storage directory on all nodes:

`1`	`mkdir -p /data/doris`

7.5 Verify Agent in Manager Interface

On the “Agent Management” page of the Manager interface, you should see the installed Agent nodes with a status of “Online”.

8. Create Doris Cluster

8.1 Start Creating Cluster

Enter Cluster Management: Click “Cluster Management” on the Manager main interface.
Click “New Cluster”: Start creating a new Doris cluster.
Fill in Cluster Configuration:
- Cluster Name: e.g., prod-cluster
- Cluster Version: Select apache-doris-2.1.8.1
- Select Installation Package: Choose from the uploaded packages.

8.2 Add FE Nodes

Add FE Nodes:
- Select FE nodes from the list of nodes with Agent installed.
- It is recommended to have at least 3 FE nodes to form a high-availability cluster.
- Select FE role (Follower or Observer).
Configure FE Parameters:
- Ports: http_port=8030, query_port=9030, edit_log_port=9010
- Memory Allocation: Configure according to the node memory size.

8.3 Add BE Nodes

Add BE Nodes:
- Select BE nodes from the list of nodes with Agent installed.
- Select an appropriate number of BE nodes according to business requirements.
Configure BE Parameters:
- Ports: be_port=9060, webserver_port=8040
- Data Directory: /data/doris
- Memory Allocation: It is recommended to reserve system memory and allocate 70-80% of the remainder to BE.

8.4 Environment Detection

Before starting deployment, Manager will automatically perform environment detection:

System parameter checks (file handles, VMA, swap, etc.)
Disk space checks
Network connectivity checks
Port occupation checks

Wait for all detection items to pass, then click “Next” to continue deployment.

If there are failed detection items, please fix them according to the prompts and re-detect.

9. Cluster Deployment and Configuration

9.1 System Initialization Configuration

Before deploying the cluster, system parameter initialization configuration is required:

Execute the following system parameter adjustments on all nodes:

Modify System Limits (Limits):

# Modify file handle limits
sudo tee -a /etc/security/limits.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

sudo tee -a /etc/security/limits.d/20-nproc.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

Apply Configuration and Reboot:

# Apply kernel parameters
sysctl -p

# Reboot node
reboot

Start Agent After Reboot: After the node restart is complete, you must manually start the Agent service.

`1`	`systemctl start doris-manager-agent`

9.2 Deploy Cluster

Cluster Settings:
- Table Name Case Sensitivity: It is recommended to select “Insensitive” (for easier application compatibility).
- Confirm all configurations are correct.
Start Deployment: Click the “Deploy Cluster” button.
Monitor Deployment Progress:
- Manager will automatically deploy FE and BE components on each node.
- You can view real-time deployment logs on the interface.
- The deployment process usually takes 5-10 minutes.
Deployment Complete:
- All component statuses show as “Running”.
- Enter the cluster details page to view the cluster overview.

9.2 Configure Timezone (Important)

Business Scenario: To meet the business needs of different countries, the cluster timezone needs to be adjusted according to the deployment region.

Timezone Mapping Table:

Country/Region	Timezone Configuration	Example
China East 8 District	Asia/Shanghai	Mainland China
West Africa Côte d’Ivoire	Africa/Abidjan	CB/CDI Project
West Africa Uganda	Africa/Kampala	Uganda Project
UTC Standard Time	UTC	International General

9.2.1 Modify FE Timezone

On the Manager interface:

Go to “Cluster Management” -> Select Cluster -> “Configuration Management”.
Find the FE configuration file fe.conf.
Add or modify the following parameter:

1
2

# fe/conf/fe.conf
time_zone = Africa/Abidjan

Save the configuration and restart the FE node.

9.2.2 Modify BE Timezone

On the Manager interface:

Find the BE configuration file be.conf in “Configuration Management”.
Add or modify the following parameter:

1
2

# be/conf/be.conf
time_zone = Africa/Abidjan

Save the configuration and restart the BE node.

9.2.3 Verify Timezone Configuration

Method 1: Verify via MySQL Client

# Connect to FE
mysql -h {fe_host} -P9030 -u root -p

# View timezone settings
SHOW VARIABLES LIKE '%time_zone%';

# View current time
SELECT NOW();

Expected Output:

+---------------+--------+
| Variable_name | Value  |
+---------------+--------+
| time_zone     | AAI    |  # Abbreviation for Africa/Abidjan
+---------------+--------+

Method 2: Verify via Logs

# View FE logs
grep -i timezone /opt/doris/fe/log/fe.log

# View BE logs
grep -i timezone /opt/doris/be/log/be.INFO

10. Cluster Verification and Testing

10.1 Check Cluster Status

View in the Manager interface:

FE Node Status: Should be “ALIVE”
BE Node Status: Should be “ALIVE”
Cluster Health Status: Should be “Healthy”

10.2 Verify via Command Line

# 1. Connect to FE
mysql -h {fe_host} -P9030 -u root -p

# 2. View FE status
SHOW FRONTENDS;

# 3. View BE status
SHOW BACKENDS;

# 4. Create test database
CREATE DATABASE test_db;

# 5. Create test table
USE test_db;
CREATE TABLE test_table (
  id INT,
  name VARCHAR(100)
) DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 10;

# 6. Insert test data
INSERT INTO test_table VALUES (1, 'test'), (2, 'doris');

# 7. Query test data
SELECT * FROM test_table;

10.3 Verify Timezone

-- Query current time to confirm timezone is correct
SELECT NOW(), CURRENT_TIMESTAMP();

-- View timezone variables
SHOW VARIABLES LIKE '%time_zone%';

11. Troubleshooting

11.1 Agent Cannot Connect to Manager

Symptom: Manager interface shows Agent as offline.

Troubleshooting Steps:

# 1. Check if Agent process is running
ps aux | grep agent

# 2. Check Agent logs
tail -f /root/manager-agent/log/agent.log

# 3. Check network connectivity
curl -I http://{manager-ip}:8004

# 4. Check firewall and ports
telnet {manager-ip} 8004

# 5. Restart Agent
systemctl restart doris-manager-agent

11.2 FE Node Fails to Start

Common Causes:

Port occupied
Metadata directory permission issues
JAVA_HOME not configured

Troubleshooting Steps:

# 1. View FE logs
tail -f /opt/doris/fe/log/fe.log

# 2. Check port occupation
netstat -tlnp | grep -E '9030|9010|8030'

# 3. Check directory permissions
ls -la /data/doris/

# 4. Check Java environment
java -version
echo $JAVA_HOME

11.3 BE Node Fails to Start

Common Causes:

Data disk not mounted
Memory configuration too large
System parameters not effective

Troubleshooting Steps:

# 1. View BE logs
tail -f /opt/doris/be/log/be.INFO

# 2. Check data disk mount
df -h | grep /data

# 3. Check memory configuration
free -h
grep -i mem /opt/doris/be/conf/be.conf

# 4. Check system parameters
ulimit -n
sysctl vm.max_map_count

11.4 Timezone Configuration Not Taking Effect

Troubleshooting Steps:

# 1. Confirm configuration file has been modified
grep time_zone /opt/doris/fe/conf/fe.conf
grep time_zone /opt/doris/be/conf/be.conf

# 2. Restart FE and BE
# Restart via Manager interface or manually

# 3. View startup logs
grep -i timezone /opt/doris/fe/log/fe.log | tail -20

# 4. Verify via MySQL client
mysql -h {fe_host} -P9030 -u root -e "SHOW VARIABLES LIKE '%time_zone%';"

12. Operations Management

12.1 Cluster Scaling Out

Add BE Nodes:

Deploy Agent on the new node (refer to Chapter 7).
In Manager interface, select Cluster -> “Node Management” -> “Add Node”.
Select the BE nodes to add.
Confirm to automatically deploy and start BE.

Add FE Nodes:

Deploy Agent on the new node.
Add FE Follower node in Manager interface.
Wait for FE to complete data synchronization and join the cluster.

12.2 Cluster Scaling In

Delete BE Nodes:

Select the BE node to delete in Manager interface.
Click the “Delete” button.
The system will automatically perform data migration (takes a certain time).
The node will automatically go offline after data migration is complete.

Delete FE Nodes:

Deleting FE Follower nodes is not recommended in production environments.
If necessary, please ensure the cluster has at least 3 FE nodes first.

12.3 Configuration Modification

You can easily modify cluster configurations in the Manager interface:

Go to “Cluster Management” -> Select Cluster -> “Configuration Management”.
Select the configuration file to modify (fe.conf or be.conf).
Save after modifying parameters.
Click “Apply Configuration” to restart the corresponding components.

Common Configuration Adjustments:

Configuration Item	Location	Description
mem_limit	be.conf	BE memory limit
max_compaction_threads	be.conf	Maximum compaction threads
max_permissive_result_mem	fe.conf	Maximum memory for single query
enable_fuzzy_mode	fe.conf	Whether to enable fuzzy mode

12.4 Monitoring and Alerting

Cluster Monitoring Metrics:

CPU usage
Memory usage
Disk space usage
Query QPS
Query response time

Alert Configuration: Configure alert rules in the Manager interface:

Go to “Cluster Management” -> “Alert Management”.
Create alert rules (e.g., disk usage > 80%).
Configure alert receiving methods (Email, DingTalk, WeCom).

13. Appendix

13.1 Port Mapping Table

Component	Port	Description
Manager WebServer	8004	Manager Management Interface
FE http_port	8030	FE HTTP Service
FE query_port	9030	FE MySQL Protocol Port
FE edit_log_port	9010	FE Metadata Synchronization Port
BE be_port	9060	BE Communication Port
BE webserver_port	8040	BE Web Service

13.2 Directory Structure

/opt/doris/
├── manager/                    # Doris Manager
│   ├── webserver/
│   │   ├── bin/
│   │   └── log/
│   └── ...
├── fe/                         # Frontend
│   ├── bin/
│   ├── conf/                   # Configuration files
│   │   ├── fe.conf
│   │   └── fe_custom.conf
│   ├── log/                    # Log directory
│   │   ├── fe.log
│   │   └── fe.warn.log
│   └── ...
├── be/                         # Backend
│   ├── bin/
│   ├── conf/                   # Configuration files
│   │   ├── be.conf
│   │   └── be_custom.conf
│   ├── log/                    # Log directory
│   │   ├── be.INFO
│   │   └── be.out
│   └── ...
└── ...

/root/manager-agent/            # Agent working directory
├── bin/
├── conf/
└── log/

/data/doris/                    # Data storage directory
├── fe/                         # FE data
│   ├── meta/
│   └── ...
└── be/                         # BE data
    ├── storage/
    └── ...

13.3 Reference Documents

At this point, the deployment of Doris Manager and the setup of the Doris cluster are complete!