Bigdata on Wanzi's Blog – Notes on life, tech, and DevOps.

Implementing Resource Isolation in Doris Offline Analysis Data Warehouse Cluster

wzdushu@gmail.com (wanzi) — Sat, 28 Feb 2026 09:30:00 +0800

I. Background

With rapid business development, the current offline analysis data warehouse Doris cluster is facing resource contention issues due to diverse computing needs such as daily data science jobs, offline batch processing, and customer-side report queries. To address this, resource limits need to be implemented for different account resource requests. After research, the official Doris Workload Group solution has been adopted.

Doris version: doris-2.1.8-1-834d802457

II. Workload Group Introduction

Workload Group is an in-process resource isolation mechanism provided by Apache Doris that achieves resource isolation between different business loads through fine-grained division of CPU, memory, and IO resources within BE processes.

The principle is as shown in the diagram below:

Currently supported isolation capabilities include:

Resource Type	Isolation Method	Description
CPU	Soft limit / Hard limit	Soft limit allocates by weight, hard limit is absolute upper bound
Memory	Soft limit / Hard limit	Hard limit automatically kills queries to free up memory when exceeded
IO	Speed limit	Limits IO throughput for reading local/remote files
Concurrency	Queuing mechanism	Exceeding concurrency queries enter queue to wait

For more information: https://doris.apache.org/zh-CN/docs/2.1/admin-manual/workload-management/workload-group

III. Doris Cluster Adjustment to Support Cgroup (Manual Deployment)

3.1 Cgroup Environment Configuration

Check the system’s supported CGroup version:

root@doristest:~# cat /proc/filesystems | grep cgroup
nodev   cgroup
nodev   cgroup2

Confirm the effective CGroup version:

root@doristest:~# ls /sys/fs/cgroup/cpu/
ls: cannot access '/sys/fs/cgroup/cpu/': No such file or directory 
root@doristest:~# ls /sys/fs/cgroup/cgroup.controllers
/sys/fs/cgroup/cgroup.controllers # Indicates cgroup v2 is in effect

3.2 Create Doris CGroup Directory

# ========== CGroup v1 Operations ==========
mkdir /sys/fs/cgroup/cpu/doris

# Set permissions (root is the user running BE)
chmod 770 /sys/fs/cgroup/cpu/doris
chown -R root:root /sys/fs/cgroup/cpu/doris


# ========== CGroup v2 Operations ==========
mkdir /sys/fs/cgroup/doris

# Set permissions
chmod 770 /sys/fs/cgroup/doris
chown -R root:root /sys/fs/cgroup/doris  # Adjust as needed, currently root

3.3 CGroup v2 Additional Configuration

CGroup v2 has stricter permission control and requires additional configuration:

# Modify root directory cgroup.procs file permissions
chmod a+w /sys/fs/cgroup/cgroup.procs

# Enable CPU controller
# Enter doris directory
cd /sys/fs/cgroup/doris

# Enable CPU controller (by modifying parent subtree_control)
# Already set by default, can be ignored
echo +cpu > ../cgroup.subtree_control

# Verify: Check if cpu.max file appears in doris directory
/sys/fs/cgroup/doris/cpu.max
# And cgroup.controllers should contain cpu
# cat /sys/fs/cgroup/cgroup.controllers 
cpuset cpu io memory hugetlb pids rdma misc

3.4 Configure BE and Restart

Edit the be.conf file:

# ========== CGroup v1 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris

# ========== CGroup v2 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/doris

Restart BE service:

# Restart BE
./bin/stop_be.sh
./bin/start_be.sh

Verification: Check the be.INFO log

grep "add thread" log/be.INFO
# Appearance of "add thread xxx to group" indicates successful configuration

3.5 Persistent Configuration (Optional)

CGroup configuration will be cleared after machine reboot. It is recommended to use systemd to create an auto-configuration service on boot:

# /etc/systemd/system/doris-cgroup.service
[Unit]
Description=Doris CGroup Setup
Before=doris-be.service

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'mkdir -p /sys/fs/cgroup/cpu/doris && chmod 770 /sys/fs/cgroup/cpu/doris && chown -R doris:doris /sys/fs/cgroup/cpu/doris'

[Install]
WantedBy=multi-user.target

Set to auto-start on boot:

systemctl enable doris-cgroup.service

IV. Batch Enable Cgroup on Doris Nodes via Ansible

4.1 Preparation

Before executing batch deployment, confirm the following information:

Confirm CGroup version: CGroup version (v1 or v2) for all Doris BE nodes
Prepare host inventory: Edit Ansible’s hosts file to add all BE nodes
Confirm BE user: User running Doris BE service (currently root)

4.2 Configure Host Inventory

Edit the hosts file:

1
2
3

[doris_dabe_nodes]
# If nodes have different CGroup versions, they can be grouped. Here only specifying offline analysis warehouse BE nodes
10.18.7.160

4.3 Create Ansible Playbook

Create file setup_doris_cgroup.yaml:

---
- name: Setup Doris CGroup Configuration
  hosts: doris_dabe_nodes
  become: true
  vars:
    # CGroup configuration path (automatically selected based on version)
    cgroup_v1_path: "/sys/fs/cgroup/cpu/doris"
    cgroup_v2_path: "/sys/fs/cgroup/doris"
    doris_user: "root"
    doris_group: "root"
    
  tasks:
    - name: Detect CGroup version
      stat:
        path: /sys/fs/cgroup/cpu/
      register: cgroup_v1_check
      
    - name: Set CGroup path based on version
      set_fact:
        cgroup_path: "{{ cgroup_v1_path if cgroup_v1_check.stat.exists else cgroup_v2_path }}"
        cgroup_version: "{{ 'v1' if cgroup_v1_check.stat.exists else 'v2' }}"
        
    - name: Display detected CGroup version
      debug:
        msg: "Node {{ inventory_hostname }} uses CGroup {{ cgroup_version }}, path: {{ cgroup_path }}"

    - name: Create CGroup directory
      file:
        path: "{{ cgroup_path }}"
        state: directory
        mode: '0770'
        owner: "{{ doris_user }}"
        group: "{{ doris_group }}"
      register: cgroup_dir_result
      
    - name: Configure CGroup v2 specific settings
      block:
        - name: Enable write permission on cgroup.procs
          file:
            path: /sys/fs/cgroup/cgroup.procs
            mode: 'a+w'
            
        - name: Enable CPU controller for CGroup v2
          shell: |
            if ! grep -q '+cpu' /sys/fs/cgroup/cgroup.subtree_control 2>/dev/null; then
              echo '+cpu' > /sys/fs/cgroup/cgroup.subtree_control
            fi
          args:
            executable: /bin/bash
          ignore_errors: yes
          
        - name: Verify CPU controller is enabled
          stat:
            path: "{{ cgroup_path }}/cpu.max"
          register: cpu_controller_check
          
        - name: Display CPU controller status
          debug:
            msg: "CPU controller is {{ 'enabled' if cpu_controller_check.stat.exists else 'NOT enabled' }}"
      when: cgroup_version == 'v2'

    - name: Verify CGroup directory structure
      shell: |
        echo "=== CGroup Version: {{ cgroup_version }} ==="
        echo "CGroup Path: {{ cgroup_path }}"
        ls -la {{ cgroup_path }} 2>/dev/null || echo "Directory verified"
        echo ""
        if [ "{{ cgroup_version }}" = "v2" ]; then
          echo "=== CPU Controller Check ==="
          cat /sys/fs/cgroup/cgroup.controllers 2>/dev/null
          test -f {{ cgroup_path }}/cpu.max && echo "cpu.max: EXISTS" || echo "cpu.max: NOT FOUND"
        fi
      args:
        executable: /bin/bash
      register: verify_result
      
    - name: Display verification results
      debug:
        var: verify_result.stdout_lines

    - name: Create systemd service for CGroup persistence (optional)
      copy:
        dest: /etc/systemd/system/doris-cgroup.service
        content: |
          [Unit]
          Description=Doris CGroup Setup
          Before=doris-be.service

          [Service]
          Type=oneshot
          ExecStart=/bin/bash -c 'mkdir -p {{ cgroup_path }} && chmod 770 {{ cgroup_path }} && chown -R {{ doris_user }}:{{ doris_group }} {{ cgroup_path }}'
          {% if cgroup_version == 'v2' %}
          ExecStartPost=/bin/bash -c 'chmod a+w /sys/fs/cgroup/cgroup.procs'
          ExecStartPost=/bin/bash -c 'echo "+cpu" > /sys/fs/cgroup/cgroup.subtree_control 2>/dev/null || true'
          {% endif %}

          [Install]
          WantedBy=multi-user.target
        mode: '0644'
      when: cgroup_dir_result is changed
      notify: Enable systemd service

  handlers:   
    - name: Enable systemd service
      systemd:
        name: doris-cgroup
        enabled: yes
        daemon_reload: yes

4.4 Execute Batch Deployment

Syntax check and dry run

# 1. Syntax check
ansible-playbook -i hosts setup_doris_cgroup.yaml --syntax-check

# 2. List all tasks to be executed
ansible-playbook -i hosts setup_doris_cgroup.yaml --list-tasks

# 3. Dry run (not actually executing, only showing changes to be made)
ansible-playbook -i hosts setup_doris_cgroup.yaml --check --diff

4.5 Official Execution and Verification of Deployment Results

Official execution

# Method 1: Execute on all BE nodes
ansible-playbook -i hosts setup_doris_cgroup.yaml

# Method 2: Execute only on specific nodes (e.g., test on single node first)
ansible-playbook -i hosts setup_doris_cgroup.yaml --limit 10.18.7.101

# Method 3: Execute step by step, each task requires confirmation
ansible-playbook -i hosts setup_doris_cgroup.yaml --step

Adjust Doris BE configuration via Doris Manager (rolling restart):

# ========== CGroup v1 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris

# ========== CGroup v2 Configuration ==========
doris_cgroup_cpu_path = /sys/fs/cgroup/doris

Verification:

# 1. Check if CGroup directory is created
ansible doris_dabe_nodes -i hosts -m shell -a "ls -la /sys/fs/cgroup/doris 2>/dev/null || ls -la /sys/fs/cgroup/cpu/doris 2>/dev/null"

# 2. Check CGroup initialization in BE logs (adjust according to actual BE log path)
ansible doris_dabe_nodes -i hosts -m shell -a "tail -100 /opt/doris/doris/be/log/be.INFO | grep -i 'cgroup\|add thread' | tail -5"

At this point, Doris BE nodes have enabled Cgroup resource limits.

V. Workload Group Management

5.1 Create Workload Group

Basic example (CPU soft limit)

CREATE WORKLOAD GROUP IF NOT EXISTS g1
PROPERTIES (
    "cpu_share" = "1024"
);

Complete configuration example

CREATE WORKLOAD GROUP IF NOT EXISTS etl_group
PROPERTIES (
    -- CPU configuration (soft limit mode)
    "cpu_share" = "2048",
    "cpu_hard_limit" = "40%",
    
    -- Memory configuration (hard limit mode)
    "memory_limit" = "40%",
    "enable_memory_overcommit" = "false",
    
    -- Concurrency control
    "max_concurrency" = "1",
    "max_queue_size" = "100",
    "queue_timeout" = "10000",
    
    -- IO speed limit
    "read_bytes_per_second" = "104857600",        -- 100MB/s
    "remote_read_bytes_per_second" = "52428800"   -- 50MB/s
);

5.2 Attribute Details

Attribute Name	Type	Default Value	Value Range	Description
cpu_share	INT	-1	[1, 10000]	CPU soft limit weight, higher value means higher priority
cpu_hard_limit	INT	-1	[1%, 100%]	CPU hard limit percentage (new in version 2.1)
memory_limit	FLOAT	-1	(0%, 100%]	Memory limit percentage
enable_memory_overcommit	BOOL	true	true/false	true=soft limit (can exceed), false=hard limit
max_concurrency	INT	2147483647	[0, 2147483647]	Maximum concurrent queries
max_queue_size	INT	0	[0, 2147483647]	Queue length (0=no queuing)
queue_timeout	INT	0	[0, 2147483647]	Queue timeout (milliseconds)
scan_thread_num	INT	-1	[1, 2147483647]	Scan thread count (-1=use BE configuration)
max_remote_scan_thread_num	INT	-1	[1, 2147483647]	Maximum scan thread count for external tables
min_remote_scan_thread_num	INT	-1	[1, 2147483647]	Minimum scan thread count for external tables
read_bytes_per_second	INT	-1	[1, 9223372036854775807]	Internal table read IO limit (bytes/second)
remote_read_bytes_per_second	INT	-1	[1, 9223372036854775807]	External table read IO limit (bytes/second)

Notes:

Must specify at least one attribute when creating
CGroup v1 cpu_share default is 1024, range 2-262144
CGroup v2 cpu_share default is 100, range 1-10000
Sum of all Workload Group memory_limit cannot exceed 100%
Sum of all Workload Group cpu_hard_limit cannot exceed 100%

5.3 Modify Workload Group

ALTER WORKLOAD GROUP g1 PROPERTIES('cpu_share' = '4096');

-- Change memory limit to hard limit
ALTER WORKLOAD GROUP g1 PROPERTIES(
    'memory_limit' = '30%',
    'enable_memory_overcommit' = 'false'
);

-- Modify concurrency and queuing parameters
ALTER WORKLOAD GROUP g1 PROPERTIES(
    'max_concurrency' = '100',
    'max_queue_size' = '200',
    'queue_timeout' = '30000'
);

-- Add CPU hard limit (hard limit mode must be enabled first)
ALTER WORKLOAD GROUP g1 PROPERTIES('cpu_hard_limit' = '20%');

5.4 Delete Workload Group

- Delete specified Workload Group
DROP WORKLOAD GROUP g1;

Note: The default normal group cannot be deleted.

VI. User Binding and Authorization

6.1 View Available Workload Groups

-- View Workload Groups that the current user has permission to use
SELECT name FROM information_schema.workload_groups;

6.2 Authorization

-- Grant user permission to use specified Workload Group
GRANT USAGE_PRIV ON WORKLOAD GROUP 'g1' TO 'user_1'@'%';

-- Grant permission to all Workload Groups
GRANT USAGE_PRIV ON WORKLOAD GROUP '*' TO 'user_1'@'%';

-- Revoke permission
REVOKE USAGE_PRIV ON WORKLOAD GROUP 'g1' FROM 'user_1'@'%';

6.3 Binding Method 1: User Properties (Recommended)

Set default Workload Group for user (persistent):

-- Set user's default Workload Group
SET PROPERTY FOR 'user_1' = 'default_workload_group' = 'g1';

-- View user properties
SHOW PROPERTY FOR 'user_1';

VII. Monitoring and Viewing

7.1 View Workload Group List

-- Show statement
SHOW WORKLOAD GROUPS;

-- System table query
SELECT * FROM information_schema.workload_groups;

7.2 View Resource Usage

-- View memory usage of each Workload Group (unit: MB)
SELECT 
    workload_group_id,
    name,
    MEMORY_USAGE_BYTES / 1024 / 1024 AS mem_used_mb,
    CPU_USAGE_PERCENT AS cpu_percent
FROM workload_group_resource_usage;

-- View details of specific Workload Group
SELECT * FROM information_schema.workload_groups WHERE name = 'g1';

7.3 System Table Field Description

workload_groups table

Field	Description
ID	Workload Group ID
NAME	Name
CPU_SHARE	CPU soft limit weight
MEMORY_LIMIT	Memory limit percentage
ENABLE_MEMORY_OVERCOMMIT	Whether memory overcommit is allowed
MAX_CONCURRENCY	Maximum concurrency
MAX_QUEUE_SIZE	Queue length
QUEUE_TIMEOUT	Queue timeout
CPU_HARD_LIMIT	CPU hard limit percentage
SCAN_THREAD_NUM	Scan thread count
READ_BYTES_PER_SECOND	Internal table IO limit
REMOTE_READ_BYTES_PER_SECOND	External table IO limit

VIII. CPU Soft/Hard Limit Mode Switching

8.1 Mode Description

Mode	Characteristics	Applicable Scenarios
CPU soft limit	Allocates CPU time by weight, can use all CPU when idle	Large load fluctuations, want to fully utilize resources
CPU hard limit	Absolute upper limit, cannot exceed even if CPU is idle	Need strict resource isolation, guarantee SLA

Note: Only one mode can be used in a cluster at the same time, cannot be mixed.

8.2 Switch from Soft Limit to Hard Limit

Step 1: Set hard limit values for all Workload Groups

-- Must set cpu_hard_limit for all Groups
ALTER WORKLOAD GROUP g1 PROPERTIES('cpu_hard_limit' = '30%');
ALTER WORKLOAD GROUP g2 PROPERTIES('cpu_hard_limit' = '20%');
ALTER WORKLOAD GROUP normal PROPERTIES('cpu_hard_limit' = '50%');

-- Verify: Sum of all Group's cpu_hard_limit <= 100%
SELECT SUM(cpu_hard_limit) FROM information_schema.workload_groups 
WHERE cpu_hard_limit > 0;

Step 2: Enable cluster hard limit switch

-- Takes effect immediately in memory (invalid after restart)
ADMIN SET FRONTEND CONFIG ("enable_cpu_hard_limit" = "true");

-- Persistent configuration: Modify fe.conf for all FEs
echo "experimental_enable_cpu_hard_limit = true" >> fe/conf/fe.conf

8.3 Switch from Hard Limit Back to Soft Limit

-- Disable hard limit switch (automatically switches back to soft limit mode)
ADMIN SET FRONTEND CONFIG ("enable_cpu_hard_limit" = "false");

For persistence, modify fe.conf to false or delete the configuration.

The above is the overall solution for adding resource limits to the Doris offline analysis data warehouse.

Reference document: https://doris.apache.org/zh-CN/docs/2.1/admin-manual/workload-management/workload-group

Doris Manager Unified Deployment and Multi-Cluster Management

wzdushu@gmail.com (wanzi) — Thu, 12 Feb 2026 09:30:00 +0800

With business expansion and the diversification of product forms, there is a need to deploy multiple sets of offline and real-time data warehouse clusters. While the previous Ansible Playbook method for deploying Doris FE/BE was basically sufficient, it fell short for subsequent Doris version upgrades, scaling, visual management, and onboarding junior operations personnel. Therefore, this document was written to facilitate quick future reference.

1. Overview

1.1 Introduction to Doris Manager

Doris Manager is a graphical operation and maintenance tool launched by the Apache Doris community for unified deployment, management, and monitoring of Doris clusters. It simplifies the entire process from operating system initialization and component deployment to daily operation and maintenance management, making it particularly suitable for rapidly setting up and maintaining large-scale Doris clusters in production environments.

From the current perspective, the official open-source repository is no longer maintained. As SelectDB is a commercial company, for its own development, it has evolved Doris Manager into a free, closed-source operation and maintenance tool, though it can still be downloaded and used.

1.2 Main Features

Cluster Deployment: Automated deployment of FE, BE, Broker, and other components.
Node Management: Unified management of all nodes, supporting scaling out and scaling in.
Configuration Management: Centralized configuration management and parameter adjustment.
Monitoring and Alerting: Cluster status monitoring and alert notifications.
Log Viewing: Centralized log viewing and analysis.

2. Prerequisites

2.1 Environment Planning

Before deployment, please confirm the following information:

Planning Item	Description	Example
Node Planning	Determine FE and BE node IPs and quantities	FE: 3 nodes, BE: 10 nodes
Storage Planning	Plan data storage directories and disk types	/data/doris (ext4, noatime)
Network Planning	Confirm network connectivity between nodes and open ports	9030, 9031, 8060, 9000+
Timezone Planning	Configure timezone based on the business country	Africa/Abidjan (Côte d’Ivoire)
Domain Planning	Manager access domain	doris-manager.example.com

2.2 Software Preparation

Official download address: https://selectdb.com/download/enterprise#manager

Download to the company software repository in advance (needs to be downloaded from the selectdb official website):

Repository Address: https://repo.test.com/bigdata/

Software Package	Version	Description
doris-manager	24.1.5-x64-bin	Doris Manager Management Tool
apache-doris	2.1.8.1-bin-x64	Apache Doris Community Edition

Download Commands:

# Create download directory
mkdir -p /opt/downloads/doris_manager /opt/downloads/doris
cd /opt/downloads/doris_manager

# Download installation packages (from company repository)
wget https://repo.test.com/bigdata/doris-manager-24.1.5-x64-bin.tar.gz
wget https://repo.test.com/bigdata/apache-doris-2.1.8.1-bin-x64.tar.gz

2.3 System Requirements

Component	Minimum Configuration	Recommended Configuration
Operating System	CentOS 7+ / Ubuntu 18.04+	CentOS 7.9 / Ubuntu 20.04
CPU	4 Cores	8 Cores+
Memory	8GB	16GB+
Disk	100GB	SSD 500GB+
Network	Gigabit NIC	10 Gigabit NIC

3. System Initialization and Optimization

Before deploying Doris Manager and the Doris cluster, system initialization configuration is required for all nodes.

3.1 Data Disk Initialization

Important: Production environments must use an independent data disk; do not share it with the system disk.

Manual initialization (single node)

# 1. View data disk devices
lsblk
# Assume the data disk is /dev/vdb

# 2. Create ext4 file system (optimized parameters)
# -m 1: Reduce reserved space from 5% to 1% to increase available space
# -T largefile: Optimize for large file storage
# -L: Set volume label for easy identification
mkfs.ext4 -m 1 -T largefile -L doris-data /dev/vdb

# 3. Create mount point directory
mkdir -p /data

# 4. Mount data disk
mount /dev/vdb /data

# 5. Get disk UUID
blkid /dev/vdb
# Output example: /dev/vdb: UUID="e86ca90c-ca31-460d-b450-1fe914ebae70" TYPE="ext4"

# 6. Configure automatic mount on boot (replace with actual UUID)
echo "UUID=e86ca90c-ca31-460d-b450-1fe914ebae70 /data ext4 defaults,noatime 0 2" >> /etc/fstab
# noatime: Do not update access time to improve performance (especially for big data read/write)
# 0: Do not backup
# 2: Check file system on startup

# 7. Create Doris data directory
mkdir -p /data/doris

# 8. Verify mount
df -h | grep /data

3.2 Disable Swap Partition

Reason: When Doris is under high memory pressure, swap swaps memory pages to disk, causing GC pauses, query lag, or even node pseudo-death, severely affecting performance.

# 1. Temporarily turn off swap
sudo swapoff -a

# 2. Permanently disable swap (comment out the swap line in /etc/fstab)
sudo sed -i '/swap/s/^/#/' /etc/fstab

# 3. Verify swap is turned off
free -h
# Swap should show 0B

swapon --show
# Should have no output, indicating swap is fully closed

3.3 Optimize Kernel Parameters

3.3.1 Increase Virtual Memory Areas (VMA)

Reason: Doris maps a large number of files to memory, requiring sufficient virtual memory areas.

# Add kernel parameters
cat >> /etc/sysctl.conf << EOF
vm.max_map_count = 2000000
EOF

# Apply configuration
sysctl -p

# Verify
sysctl vm.max_map_count
# Should output: vm.max_map_count = 2000000

3.3.2 Optimize Network Parameters (Optional)

# Optimize TCP connection parameters for high concurrency scenarios
cat >> /etc/sysctl.conf << EOF
# TCP optimization parameters
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 2048
EOF

sysctl -p

3.4 Increase File Handle Limits

Reason: Doris BE opens a large number of files (data shards, indexes, logs, etc.), so the file descriptor limit must be increased.

# 1. Modify limits configuration
sudo tee -a /etc/security/limits.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

# 2. Create independent configuration in limits.d directory
sudo tee -a /etc/security/limits.d/20-nproc.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

# 3. Verify configuration
ulimit -n
# Should output: 65536

3.5 Apply All Configurations and Reboot

# Apply all kernel parameters
sysctl -p

# Reboot nodes for all configurations to take effect
reboot

# Verification after reboot
# 1. Check swap: free -h (Swap should be 0)
# 2. Check file handles: ulimit -n (should be 65536)
# 3. Check VMA: sysctl vm.max_map_count (should be 2000000)
# 4. Check data disk: df -h | grep /data

In actual deployment, you can customize virtual machine hosts or cloud host images in advance to include basic initialization operations.

4. Deploy Doris Manager

4.1 Extraction and Installation

# 1. Extract installation package
cd /opt/downloads/doris_manager
tar zxvf doris-manager-24.1.5-x64-bin.tar.gz

# 2. Create deployment directory
mkdir -p /opt/doris/manager

# 3. Move files to deployment directory
mv ./doris-manager-24.1.5-x64-bin/* /opt/doris/manager/

# 4. Prepare Doris installation package resources
# Manager needs local source files for subsequent cluster deployment
mkdir -p /opt/downloads/doris
cp /opt/downloads/doris_manager/apache-doris-2.1.8.1-bin-x64.tar.gz /opt/downloads/doris/
cp /opt/downloads/doris_manager/selectdb-doris-2.0.3-b4-bin-x64.tar.gz /opt/downloads/doris/

4.2 Start Doris Manager

# Start Doris Manager WebServer
cd /opt/doris/manager/
./webserver/bin/start.sh

# View logs (optional)
tail -f webserver/log/webserver.log

Startup Verification: Service listens on port 8004

# Check if port is listening normally
ss -an | grep 8004
# Expected output:
# tcp    LISTEN    0      100    *:8004    *:*

# Or use netstat
netstat -tlnp | grep 8004

4.3 Configure Auto-start on Boot (Recommended)

Use systemd to manage the Doris Manager service, ensuring automatic startup after node restarts.

# 1. Create systemd service file
cat > /etc/systemd/system/doris-manager.service <<'EOF'
[Unit]
Description=Doris Manager WebServer Service
Documentation=https://doris.apache.org/
After=network.target

[Service]
Type=forking
WorkingDirectory=/opt/doris/manager
ExecStart=/opt/doris/manager/webserver/bin/start.sh
ExecStop=/opt/doris/manager/webserver/bin/stop.sh
Restart=on-failure
RestartSec=10
User=root
Group=root
TimeoutStartSec=120
TimeoutStopSec=120
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

# 2. Reload systemd configuration
sudo systemctl daemon-reload

# 3. Enable auto-start on boot
sudo systemctl enable doris-manager

# 4. Check service status
sudo systemctl status doris-manager

4.4 Access Manager UI

Access the Doris Manager management interface via a browser:

http://{manager-ip}:8004

First Access Steps:

Enter the initialization user page to create the first Manager admin user.
The Manager admin account is independent of cluster accounts and is only used for Manager permission control.
Log in with that account after successful creation.

5. Configure Nginx Reverse Proxy

In production environments, it is recommended to configure domain names and HTTPS access via Nginx.

5.1 Nginx Configuration File

server {
  listen 80;
  listen 443 ssl;
  charset utf-8;
  server_name ug-test-doris-manager.test.com;
  index  index.html index.htm index.jsp index.php;

  # SSL configuration (optional)
  # if ($scheme = http) {
  #   rewrite ^(.*)$ https://$host$1 redirect;
  # }

  include rule.d/ssl.conf;
  include rule.d/ssl_test.conf;

  client_max_body_size 5000m;

  location  / {
    include rule.d/proxy_header.conf;
    include rule.d/proxy_connection_small.conf;
    proxy_pass http://10.156.179.58:8004;
  }

  access_log  /var/log/nginx/ug-test-doris-manager.test.com.access.log main;
  error_log   /var/log/nginx/ug-test-doris-manager.test.com.error.log;
}

5.2 Restart Nginx

# Check configuration file syntax
nginx -t

# Restart Nginx
systemctl restart nginx

# Verify
curl -I http://ug-test-doris-manager.test.com

Now you can access Doris Manager via the domain name:

https://ug-test-doris-manager.test.com

6. Initialize Manager Configuration

6.1 Create Admin Account

Access UI Interface: Access Doris Manager via the configured domain or IP.
Create Admin Account:
- Set username and password.
- Service Configuration: Since external network policies may not be open, it is recommended to turn off email alerts first.
- Click to start the doris-manager service.
Enter Management Console: After successful installation, enter the main interface.

6.2 Upload Doris Installation Packages

Upload the Doris installation package on the Manager interface:

apache-doris-2.1.8.1-bin-x64.tar.gz

Or use the command line to place in the specified directory:

1
2
3

# Ensure the path is accessible by Manager
mkdir -p /opt/downloads/doris
cp apache-doris-2.1.8.1-bin-x64.tar.gz /opt/downloads/doris/

7. Deploy Doris Agent

Agent is the proxy program of Doris Manager on each node, responsible for executing deployment and management tasks.

7.1 Get Installation Command

On the Manager interface:

Go to “Cluster Management” -> “Agent Management”.
Click the “Add Agent” button.
Get the installation command (usually two lines).

Example Command:

`1`	`wget http://10.156.179.58:8004/api/download/deploy.sh -O deploy_agent.sh && chmod +x deploy_agent.sh && ./deploy_agent.sh`

Note: Replace the IP address with the actual Manager service IP.

7.2 Execute Installation on Target Nodes

# Execute on every Doris node
wget http://{manager-ip}:8004/api/download/deploy.sh -O deploy_agent.sh
chmod +x deploy_agent.sh
./deploy_agent.sh

7.3 Configure Agent Auto-start on Boot

Create a systemd service file to manage Agent:

# 1. Create systemd service file
cat > /etc/systemd/system/doris-manager-agent.service <<'EOF'
[Unit]
Description=Doris Manager Agent Service
Documentation=https://doris.apache.org/
After=network.target

[Service]
Type=forking
# Type=forking is because the start.sh script starts the process in the background and exits immediately

WorkingDirectory=/root/manager-agent

ExecStart=/root/manager-agent/bin/start.sh
ExecStop=/root/manager-agent/bin/stop.sh

Restart=on-failure
RestartSec=10

User=root
Group=root

TimeoutStartSec=120
TimeoutStopSec=120

StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

# 2. Reload systemd configuration
sudo systemctl daemon-reload

# 3. Enable auto-start on boot
sudo systemctl enable doris-manager-agent

# 4. Start service
sudo systemctl start doris-manager-agent

# 5. Check service status
sudo systemctl status doris-manager-agent

# 6. View real-time logs (if startup fails)
sudo journalctl -u doris-manager-agent -f

7.4 Manually Create Data Directory

Manually create the data storage directory on all nodes:

`1`	`mkdir -p /data/doris`

7.5 Verify Agent in Manager Interface

On the “Agent Management” page of the Manager interface, you should see the installed Agent nodes with a status of “Online”.

8. Create Doris Cluster

8.1 Start Creating Cluster

Enter Cluster Management: Click “Cluster Management” on the Manager main interface.
Click “New Cluster”: Start creating a new Doris cluster.
Fill in Cluster Configuration:
- Cluster Name: e.g., prod-cluster
- Cluster Version: Select apache-doris-2.1.8.1
- Select Installation Package: Choose from the uploaded packages.

8.2 Add FE Nodes

Add FE Nodes:
- Select FE nodes from the list of nodes with Agent installed.
- It is recommended to have at least 3 FE nodes to form a high-availability cluster.
- Select FE role (Follower or Observer).
Configure FE Parameters:
- Ports: http_port=8030, query_port=9030, edit_log_port=9010
- Memory Allocation: Configure according to the node memory size.

8.3 Add BE Nodes

Add BE Nodes:
- Select BE nodes from the list of nodes with Agent installed.
- Select an appropriate number of BE nodes according to business requirements.
Configure BE Parameters:
- Ports: be_port=9060, webserver_port=8040
- Data Directory: /data/doris
- Memory Allocation: It is recommended to reserve system memory and allocate 70-80% of the remainder to BE.

8.4 Environment Detection

Before starting deployment, Manager will automatically perform environment detection:

System parameter checks (file handles, VMA, swap, etc.)
Disk space checks
Network connectivity checks
Port occupation checks

Wait for all detection items to pass, then click “Next” to continue deployment.

If there are failed detection items, please fix them according to the prompts and re-detect.

9. Cluster Deployment and Configuration

9.1 System Initialization Configuration

Before deploying the cluster, system parameter initialization configuration is required:

Execute the following system parameter adjustments on all nodes:

Modify System Limits (Limits):

# Modify file handle limits
sudo tee -a /etc/security/limits.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

sudo tee -a /etc/security/limits.d/20-nproc.conf <<EOF
* soft nofile 65536
* hard nofile 65536
EOF

Apply Configuration and Reboot:

# Apply kernel parameters
sysctl -p

# Reboot node
reboot

Start Agent After Reboot: After the node restart is complete, you must manually start the Agent service.

`1`	`systemctl start doris-manager-agent`

9.2 Deploy Cluster

Cluster Settings:
- Table Name Case Sensitivity: It is recommended to select “Insensitive” (for easier application compatibility).
- Confirm all configurations are correct.
Start Deployment: Click the “Deploy Cluster” button.
Monitor Deployment Progress:
- Manager will automatically deploy FE and BE components on each node.
- You can view real-time deployment logs on the interface.
- The deployment process usually takes 5-10 minutes.
Deployment Complete:
- All component statuses show as “Running”.
- Enter the cluster details page to view the cluster overview.

9.2 Configure Timezone (Important)

Business Scenario: To meet the business needs of different countries, the cluster timezone needs to be adjusted according to the deployment region.

Timezone Mapping Table:

Country/Region	Timezone Configuration	Example
China East 8 District	Asia/Shanghai	Mainland China
West Africa Côte d’Ivoire	Africa/Abidjan	CB/CDI Project
West Africa Uganda	Africa/Kampala	Uganda Project
UTC Standard Time	UTC	International General

9.2.1 Modify FE Timezone

On the Manager interface:

Go to “Cluster Management” -> Select Cluster -> “Configuration Management”.
Find the FE configuration file fe.conf.
Add or modify the following parameter:

1
2

# fe/conf/fe.conf
time_zone = Africa/Abidjan

Save the configuration and restart the FE node.

9.2.2 Modify BE Timezone

On the Manager interface:

Find the BE configuration file be.conf in “Configuration Management”.
Add or modify the following parameter:

1
2

# be/conf/be.conf
time_zone = Africa/Abidjan

Save the configuration and restart the BE node.

9.2.3 Verify Timezone Configuration

Method 1: Verify via MySQL Client

# Connect to FE
mysql -h {fe_host} -P9030 -u root -p

# View timezone settings
SHOW VARIABLES LIKE '%time_zone%';

# View current time
SELECT NOW();

Expected Output:

+---------------+--------+
| Variable_name | Value  |
+---------------+--------+
| time_zone     | AAI    |  # Abbreviation for Africa/Abidjan
+---------------+--------+

Method 2: Verify via Logs

# View FE logs
grep -i timezone /opt/doris/fe/log/fe.log

# View BE logs
grep -i timezone /opt/doris/be/log/be.INFO

10. Cluster Verification and Testing

10.1 Check Cluster Status

View in the Manager interface:

FE Node Status: Should be “ALIVE”
BE Node Status: Should be “ALIVE”
Cluster Health Status: Should be “Healthy”

10.2 Verify via Command Line

# 1. Connect to FE
mysql -h {fe_host} -P9030 -u root -p

# 2. View FE status
SHOW FRONTENDS;

# 3. View BE status
SHOW BACKENDS;

# 4. Create test database
CREATE DATABASE test_db;

# 5. Create test table
USE test_db;
CREATE TABLE test_table (
  id INT,
  name VARCHAR(100)
) DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 10;

# 6. Insert test data
INSERT INTO test_table VALUES (1, 'test'), (2, 'doris');

# 7. Query test data
SELECT * FROM test_table;

10.3 Verify Timezone

-- Query current time to confirm timezone is correct
SELECT NOW(), CURRENT_TIMESTAMP();

-- View timezone variables
SHOW VARIABLES LIKE '%time_zone%';

11. Troubleshooting

11.1 Agent Cannot Connect to Manager

Symptom: Manager interface shows Agent as offline.

Troubleshooting Steps:

# 1. Check if Agent process is running
ps aux | grep agent

# 2. Check Agent logs
tail -f /root/manager-agent/log/agent.log

# 3. Check network connectivity
curl -I http://{manager-ip}:8004

# 4. Check firewall and ports
telnet {manager-ip} 8004

# 5. Restart Agent
systemctl restart doris-manager-agent

11.2 FE Node Fails to Start

Common Causes:

Port occupied
Metadata directory permission issues
JAVA_HOME not configured

Troubleshooting Steps:

# 1. View FE logs
tail -f /opt/doris/fe/log/fe.log

# 2. Check port occupation
netstat -tlnp | grep -E '9030|9010|8030'

# 3. Check directory permissions
ls -la /data/doris/

# 4. Check Java environment
java -version
echo $JAVA_HOME

11.3 BE Node Fails to Start

Common Causes:

Data disk not mounted
Memory configuration too large
System parameters not effective

Troubleshooting Steps:

# 1. View BE logs
tail -f /opt/doris/be/log/be.INFO

# 2. Check data disk mount
df -h | grep /data

# 3. Check memory configuration
free -h
grep -i mem /opt/doris/be/conf/be.conf

# 4. Check system parameters
ulimit -n
sysctl vm.max_map_count

11.4 Timezone Configuration Not Taking Effect

Troubleshooting Steps:

# 1. Confirm configuration file has been modified
grep time_zone /opt/doris/fe/conf/fe.conf
grep time_zone /opt/doris/be/conf/be.conf

# 2. Restart FE and BE
# Restart via Manager interface or manually

# 3. View startup logs
grep -i timezone /opt/doris/fe/log/fe.log | tail -20

# 4. Verify via MySQL client
mysql -h {fe_host} -P9030 -u root -e "SHOW VARIABLES LIKE '%time_zone%';"

12. Operations Management

12.1 Cluster Scaling Out

Add BE Nodes:

Deploy Agent on the new node (refer to Chapter 7).
In Manager interface, select Cluster -> “Node Management” -> “Add Node”.
Select the BE nodes to add.
Confirm to automatically deploy and start BE.

Add FE Nodes:

Deploy Agent on the new node.
Add FE Follower node in Manager interface.
Wait for FE to complete data synchronization and join the cluster.

12.2 Cluster Scaling In

Delete BE Nodes:

Select the BE node to delete in Manager interface.
Click the “Delete” button.
The system will automatically perform data migration (takes a certain time).
The node will automatically go offline after data migration is complete.

Delete FE Nodes:

Deleting FE Follower nodes is not recommended in production environments.
If necessary, please ensure the cluster has at least 3 FE nodes first.

12.3 Configuration Modification

You can easily modify cluster configurations in the Manager interface:

Go to “Cluster Management” -> Select Cluster -> “Configuration Management”.
Select the configuration file to modify (fe.conf or be.conf).
Save after modifying parameters.
Click “Apply Configuration” to restart the corresponding components.

Common Configuration Adjustments:

Configuration Item	Location	Description
mem_limit	be.conf	BE memory limit
max_compaction_threads	be.conf	Maximum compaction threads
max_permissive_result_mem	fe.conf	Maximum memory for single query
enable_fuzzy_mode	fe.conf	Whether to enable fuzzy mode

12.4 Monitoring and Alerting

Cluster Monitoring Metrics:

CPU usage
Memory usage
Disk space usage
Query QPS
Query response time

Alert Configuration: Configure alert rules in the Manager interface:

Go to “Cluster Management” -> “Alert Management”.
Create alert rules (e.g., disk usage > 80%).
Configure alert receiving methods (Email, DingTalk, WeCom).

13. Appendix

13.1 Port Mapping Table

Component	Port	Description
Manager WebServer	8004	Manager Management Interface
FE http_port	8030	FE HTTP Service
FE query_port	9030	FE MySQL Protocol Port
FE edit_log_port	9010	FE Metadata Synchronization Port
BE be_port	9060	BE Communication Port
BE webserver_port	8040	BE Web Service

13.2 Directory Structure

/opt/doris/
├── manager/                    # Doris Manager
│   ├── webserver/
│   │   ├── bin/
│   │   └── log/
│   └── ...
├── fe/                         # Frontend
│   ├── bin/
│   ├── conf/                   # Configuration files
│   │   ├── fe.conf
│   │   └── fe_custom.conf
│   ├── log/                    # Log directory
│   │   ├── fe.log
│   │   └── fe.warn.log
│   └── ...
├── be/                         # Backend
│   ├── bin/
│   ├── conf/                   # Configuration files
│   │   ├── be.conf
│   │   └── be_custom.conf
│   ├── log/                    # Log directory
│   │   ├── be.INFO
│   │   └── be.out
│   └── ...
└── ...

/root/manager-agent/            # Agent working directory
├── bin/
├── conf/
└── log/

/data/doris/                    # Data storage directory
├── fe/                         # FE data
│   ├── meta/
│   └── ...
└── be/                         # BE data
    ├── storage/
    └── ...

13.3 Reference Documents

At this point, the deployment of Doris Manager and the setup of the Doris cluster are complete!

How to quickly set up a Greenplum cluster

wzdushu@gmail.com (wanzi) — Wed, 08 Sep 2021 17:22:42 +0800

Recently, our internal project has been supporting a big data initiative, requiring the simulation of customer scenarios using Greenplum (older version 4.2.2.4). Below is a record of the Greenplum cluster setup process—note that the procedure for higher versions of GP remains largely identical.

Building Base Image

CentOS 6 Dockerfile:

FROM centos:6

RUN mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
RUN curl -o /etc/yum.repos.d/CentOS-Base.repo https://www.xmpan.com/Centos-6-Vault-Aliyun.repo
RUN yum -y update; yum clean all
RUN yum install -y \
    net-tools \
    ntp \
    openssh-server \
    openssh-clients \
    less \
    iproute \
    lsof \
    wget \
    ed \
    which; yum clean all
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ''
RUN groupadd gpadmin
RUN useradd gpadmin -g gpadmin
RUN echo gpadmin | passwd gpadmin --stdin
ENTRYPOINT ["/usr/sbin/sshd", "-D"]

Build image:

`1`	`docker build -t harbor.test.com/sp/greenplum-base:centos6 .`

CentOS 7 Dockerfile:

FROM centos:7

RUN mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
RUN curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
RUN yum -y update; yum clean all
RUN yum install -y \
    net-tools \
    ntp \
    openssh-server \
    openssh-clients \
    less \
    iproute \
    lsof \
    wget \
    ed \
    which; yum clean all
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ''
RUN groupadd gpadmin
RUN useradd gpadmin -g gpadmin
RUN echo gpadmin | passwd gpadmin --stdin
ENTRYPOINT ["/usr/sbin/sshd", "-D"]

`1`	`docker build -t harbor.test.com/sp/greenplum-base:centos7 .`

Managing GP with Docker Compose

version: "2"

services:
  mdw:
    #image: "harbor.test.com/sp/greenplum-base:centos7"
    image: "harbor.test.com/sp/greenplum-base:centos6"
    container_name: gpdb-mdw
    volumes:
    - greenplum-mdw:/home/gpadmin
    ports:
      - "2222:22"
      - "15432:5432"
    hostname: mdw
    tty: true
    networks:
      - greenplum

  sdw1:
    #image: "harbor.test.com/sp/greenplum-base:centos7"
    image: "harbor.test.com/sp/greenplum-base:centos6"
    container_name: gpdb-sdw1
    volumes:
    - greenplum-mdw:/home/gpadmin
    hostname: sdw1
    tty: true
    networks:
      - greenplum

  sdw2:
    #image: "harbor.test.com/sp/greenplum-base:centos7"
    image: "harbor.test.com/sp/greenplum-base:centos6"
    container_name: gpdb-sdw2
    volumes:
    - greenplum-mdw:/home/gpadmin
    hostname: sdw2
    tty: true
    networks:
      - greenplum

networks:
  greenplum:
    ipam:
      config:
      - subnet: 10.188.0.0/24
        gateway: 10.188.0.1

volumes:
 greenplum-mdw:
 greenplum-sdw1:
 greenplum-sdw2:

Start services:

`1`	`docker-compose up -d`

Initializing Greenplum Installation

Copy Greenplum Binary to Container for Installation

docker cp greenplum-db-4.2.2.4-build-1-CE-RHEL5-x86_64.bin gpdb-mdw:/home/gpadmin
docker exec -it gpdb-mdw /bin/bash
su - gpadmin
bash greenplum-db-4.2.2.4-build-1-CE-RHEL5-x86_64.bin

Installation directory used here: /home/gpadmin/greenplum-db-new

Initialize Configuration Files:

# cd /home/gpadmin/gpconfig/
# touch hostlist seglist
# cat hostlist
mdw
sdw1
sdw2
# cat seglist 
sdw1
sdw2

Load Environment Variables and Configure SSH Keys

source ~/greenplum-db/greenplum_path.sh
gpssh-exkeys -f /home/gpadmin/gpconfig/hostlist
gpseginstall -f /home/gpadmin/gpconfig/seglist
gpssh -f ~/gpconfig/hostlist -e ls -l $GPHOME

Create Data Directories on mdw, sdw1, and sdw2 Hosts

1
2
3

mkdir -p ~/data/master
gpssh -f ~/gpconfig/seglist -e "mkdir -p ~/data/primary"
gpssh -f ~/gpconfig/seglist -e "mkdir -p ~/data/mirror"

Modify Initialization Configuration:

$ cp /home/gpadmin/greenplum-db/docs/cli_help/gpconfigs/gpinitsystem_config ~/gpconfig/gpinitsystem_config
$ egrep -v '^$|^#' gpinitsystem_config
ARRAY_NAME="EMC Greenplum DW"
SEG_PREFIX=gpseg
PORT_BASE=40000
MACHINE_LIST_FILE=/home/gpadmin/gpconfig/hostlist
declare -a DATA_DIRECTORY=(/home/gpadmin/data/primary)
MASTER_HOSTNAME=mdw
MASTER_DIRECTORY=/home/gpadmin/data/master
MASTER_PORT=5432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE

Initialize Greenplum Server

[gpadmin@mdw ~]$ gpinitsystem -c gpconfig/gpinitsystem_config -h gpconfig/seglist
20210723:08:04:39:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Checking configuration parameters, please wait...
/bin/mv: try to overwrite `gpconfig/gpinitsystem_config', overriding mode 0664 (rw-rw-r--)?
20210723:08:04:51:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Reading Greenplum configuration file gpconfig/gpinitsystem_config
20210723:08:04:51:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Locale has not been set in gpconfig/gpinitsystem_config, will set to default value
20210723:08:04:51:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Locale set to en_US.utf8
20210723:08:04:51:000282 gpinitsystem:mdw:gpadmin-[INFO]:-No DATABASE_NAME set, will exit following template1 updates
20210723:08:04:51:000282 gpinitsystem:mdw:gpadmin-[INFO]:-MASTER_MAX_CONNECT not set, will set to default value 250
20210723:08:04:52:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Checking configuration parameters, Completed
20210723:08:04:52:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Commencing multi-home checks, please wait...
..
20210723:08:04:52:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Configuring build for standard array
20210723:08:04:52:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Commencing multi-home checks, Completed
20210723:08:04:53:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Building primary segment instance array, please wait...
..
20210723:08:04:54:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Checking Master host
20210723:08:04:54:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Checking new segment hosts, please wait...
..
20210723:08:04:57:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Checking new segment hosts, Completed
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Greenplum Database Creation Parameters
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:---------------------------------------
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Master Configuration
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:---------------------------------------
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Master instance name       = EMC Greenplum DW
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Master hostname            = mdw
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Master port                = 5432
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Master instance dir        = /home/gpadmin/data/master/gpseg-1
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Master LOCALE              = en_US.utf8
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Greenplum segment prefix   = gpseg
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Master Database            =
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Master connections         = 250
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Master buffers             = 128000kB
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Segment connections        = 750
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Segment buffers            = 128000kB
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Checkpoint segments        = 8
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Encoding                   = UNICODE
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Postgres param file        = Off
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Initdb to be used          = /home/gpadmin/greenplum-db/./bin/initdb
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-GP_LIBRARY_PATH is         = /home/gpadmin/greenplum-db/./lib
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Ulimit check               = Passed
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Array host connect type    = Single hostname per node
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Master IP address [1]      = 10.188.0.2
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Standby Master             = Not Configured
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Primary segment #          = 1
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Total Database segments    = 2
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Trusted shell              = ssh
20210723:08:04:58:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Number segment hosts       = 2
20210723:08:04:59:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Mirroring config           = OFF
20210723:08:04:59:000282 gpinitsystem:mdw:gpadmin-[INFO]:----------------------------------------
20210723:08:04:59:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Greenplum Primary Segment Configuration
20210723:08:04:59:000282 gpinitsystem:mdw:gpadmin-[INFO]:----------------------------------------
20210723:08:04:59:000282 gpinitsystem:mdw:gpadmin-[INFO]:-sdw1 	/home/gpadmin/data/primary/gpseg0 	40000 	2 	0
20210723:08:04:59:000282 gpinitsystem:mdw:gpadmin-[INFO]:-sdw2 	/home/gpadmin/data/primary/gpseg1 	40000 	3 	1
Continue with Greenplum creation Yy/Nn>
Y
20210723:08:05:02:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Building the Master instance database, please wait...
20210723:08:05:13:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Starting the Master in admin mode
20210723:08:05:37:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Commencing parallel build of primary segment instances
20210723:08:05:37:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Spawning parallel processes    batch [1], please wait...
..
20210723:08:05:37:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Waiting for parallel processes batch [1], please wait...
.................
20210723:08:05:54:000282 gpinitsystem:mdw:gpadmin-[INFO]:------------------------------------------------
20210723:08:05:54:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Parallel process exit status
20210723:08:05:54:000282 gpinitsystem:mdw:gpadmin-[INFO]:------------------------------------------------
20210723:08:05:54:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Total processes marked as completed           = 2
20210723:08:05:54:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Total processes marked as killed              = 0
20210723:08:05:54:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Total processes marked as failed              = 0
20210723:08:05:54:000282 gpinitsystem:mdw:gpadmin-[INFO]:------------------------------------------------
20210723:08:05:55:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Deleting distributed backout files
20210723:08:05:55:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Removing back out file
20210723:08:05:55:000282 gpinitsystem:mdw:gpadmin-[INFO]:-No errors generated from parallel processes
20210723:08:05:55:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Restarting the Greenplum instance in production mode
20210723:08:05:55:010330 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -a -i -m -d /home/gpadmin/data/master/gpseg-1
20210723:08:05:55:010330 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20210723:08:05:56:010330 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20210723:08:05:56:010330 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20210723:08:05:57:010330 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 4.2.2.4 build 1 Community Edition'
20210723:08:05:57:010330 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database
20210723:08:05:57:010330 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='immediate'
20210723:08:05:57:010330 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw
20210723:08:05:57:010330 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=immediate
20210723:08:05:57:010330 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/home/gpadmin/data/master/gpseg-1
20210723:08:05:58:010413 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -a -d /home/gpadmin/data/master/gpseg-1
20210723:08:05:58:010413 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20210723:08:05:59:010413 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.2.2.4 build 1 Community Edition'
20210723:08:06:00:010413 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '201109210'
20210723:08:06:01:010413 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20210723:08:06:02:010413 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20210723:08:06:02:010413 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20210723:08:06:03:010413 gpstart:mdw:gpadmin-[INFO]:-Setting new master era
20210723:08:06:03:010413 gpstart:mdw:gpadmin-[INFO]:-Master Started...
20210723:08:06:03:010413 gpstart:mdw:gpadmin-[INFO]:-Checking for filespace consistency
20210723:08:06:03:010413 gpstart:mdw:gpadmin-[INFO]:-Obtaining current filespace entries used by TRANSACTION_FILES
20210723:08:06:03:010413 gpstart:mdw:gpadmin-[INFO]:-TRANSACTION_FILES OIDs are consistent for pg_system filespace
20210723:08:06:04:010413 gpstart:mdw:gpadmin-[INFO]:-TRANSACTION_FILES entries are consistent for pg_system filespace
20210723:08:06:04:010413 gpstart:mdw:gpadmin-[INFO]:-Checking for filespace consistency
20210723:08:06:04:010413 gpstart:mdw:gpadmin-[INFO]:-Obtaining current filespace entries used by TEMPORARY_FILES
20210723:08:06:05:010413 gpstart:mdw:gpadmin-[INFO]:-TEMPORARY_FILES OIDs are consistent for pg_system filespace
20210723:08:06:06:010413 gpstart:mdw:gpadmin-[INFO]:-TEMPORARY_FILES entries are consistent for pg_system filespace
20210723:08:06:06:010413 gpstart:mdw:gpadmin-[INFO]:-Shutting down master
20210723:08:06:10:010413 gpstart:mdw:gpadmin-[INFO]:-No standby master configured.  skipping...
20210723:08:06:10:010413 gpstart:mdw:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...
....
20210723:08:06:14:010413 gpstart:mdw:gpadmin-[INFO]:-Process results...
20210723:08:06:14:010413 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20210723:08:06:14:010413 gpstart:mdw:gpadmin-[INFO]:-   Successful segment starts                                            = 2
20210723:08:06:14:010413 gpstart:mdw:gpadmin-[INFO]:-   Failed segment starts                                                = 0
20210723:08:06:14:010413 gpstart:mdw:gpadmin-[INFO]:-   Skipped segment starts (segments are marked down in configuration)   = 0
20210723:08:06:14:010413 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20210723:08:06:14:010413 gpstart:mdw:gpadmin-[INFO]:-
20210723:08:06:14:010413 gpstart:mdw:gpadmin-[INFO]:-Successfully started 2 of 2 segment instances
20210723:08:06:14:010413 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20210723:08:06:14:010413 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /home/gpadmin/data/master/gpseg-1
20210723:08:06:16:010413 gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active
20210723:08:06:17:010413 gpstart:mdw:gpadmin-[INFO]:-Database successfully started
20210723:08:06:17:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Completed restart of Greenplum instance in production mode
20210723:08:06:17:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Loading gp_toolkit...
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Scanning utility log file for any warning messages
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[WARN]:-*******************************************************
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[WARN]:-Scan of log file indicates that some warnings or errors
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[WARN]:-were generated during the array creation
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Please review contents of log file
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-/home/gpadmin/gpAdminLogs/gpinitsystem_20210723.log
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-To determine level of criticality
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[WARN]:-*******************************************************
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Greenplum Database instance successfully created
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-------------------------------------------------------
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-To complete the environment configuration, please
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-update gpadmin .bashrc file with the following
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-1. Ensure that the greenplum_path.sh file is sourced
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-2. Add "export MASTER_DATA_DIRECTORY=/home/gpadmin/data/master/gpseg-1"
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-   to access the Greenplum scripts for this instance:
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-   or, use -d /home/gpadmin/data/master/gpseg-1 option for the Greenplum scripts
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-   Example gpstate -d /home/gpadmin/data/master/gpseg-1
20210723:08:06:18:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Script log file = /home/gpadmin/gpAdminLogs/gpinitsystem_20210723.log
20210723:08:06:19:000282 gpinitsystem:mdw:gpadmin-[INFO]:-To remove instance, run gpdeletesystem utility
20210723:08:06:19:000282 gpinitsystem:mdw:gpadmin-[INFO]:-To initialize a Standby Master Segment for this Greenplum instance
20210723:08:06:19:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Review options for gpinitstandby
20210723:08:06:19:000282 gpinitsystem:mdw:gpadmin-[INFO]:-------------------------------------------------------
20210723:08:06:19:000282 gpinitsystem:mdw:gpadmin-[INFO]:-The Master /home/gpadmin/data/master/gpseg-1/pg_hba.conf post gpinitsystem
20210723:08:06:19:000282 gpinitsystem:mdw:gpadmin-[INFO]:-has been configured to allow all hosts within this new
20210723:08:06:19:000282 gpinitsystem:mdw:gpadmin-[INFO]:-array to intercommunicate. Any hosts external to this
20210723:08:06:19:000282 gpinitsystem:mdw:gpadmin-[INFO]:-new array must be explicitly added to this file
20210723:08:06:19:000282 gpinitsystem:mdw:gpadmin-[INFO]:-Refer to the Greenplum Admin support guide which is
20210723:08:06:19:000282 gpinitsystem:mdw:gpadmin-[INFO]:-located in the /home/gpadmin/greenplum-db/./docs directory
20210723:08:06:19:000282 gpinitsystem:mdw:gpadmin-[INFO]:-------------------------------------------------------

Greenplum User Management

Create user admini with permissions to create databases and log in:

1
2

create role admini password 'SdAdm2021#' createdb login;
select rolname,oid from pg_roles;

If password is forgotten, it can be reset:

`1`	`alter user gpadmin with password 'SdAdm2021#';`

Update /home/gpadmin/data/master/gpseg-1/pg_hba.conf to allow admini remote login:

1
2
3

host     all         admini         127.0.0.1/32       md5
host     all         admini         10.188.0.2/32       md5
host     all         admini         0.0.0.0/0       md5

Reload configuration:

[gpadmin@mdw greenplum-db-new]$ pwd
/home/gpadmin/greenplum-db-new
[gpadmin@mdw greenplum-db-new]$ ./bin/gpstop  -u
20210728:01:50:22:112099 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -u
20210728:01:50:22:112099 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20210728:01:50:22:112099 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20210728:01:50:22:112099 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20210728:01:50:23:112099 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 4.2.2.4 build 1 Community Edition'
20210728:01:50:23:112099 gpstop:mdw:gpadmin-[INFO]:-Signalling all postmaster processes to reload

Starting GP Server

Since GP was manually installed inside containers, it must be manually started after machine or Docker restarts.

root@sd-cluster-01:/opt/greenplum# docker exec -it gpdb-mdw bash
[root@mdw /]# su - gpadmin
[gpadmin@mdw ~]$ gpstart  -a
Log in to other nodes via master:
[gpadmin@mdw ~]$ ssh sdw1
Last login: Tue Aug 10 10:19:29 2021 from gpdb-mdw.greenplum_greenplum
[gpadmin@sdw1 ~]$ exit
logout
Connection to sdw1 closed.
[gpadmin@mdw ~]$ ssh sdw2
Last login: Mon Aug  9 12:54:16 2021 from gpdb-mdw.greenplum_greenplum
[gpadmin@sdw2 ~]$ exit
logout
Connection to sdw2 closed.

GP Performance Optimization

As GP is built on PostgreSQL, optimization focuses primarily on PostgreSQL tuning. For default settings, I’ve adjusted several key parameters:

Log into the master container and modify: /home/gpadmin/data/master/gpseg-1/postgresql.conf

max_connections = 800 # Maximum client connections; avoid exceeding available RAM when multiplied by work_mem
shared_buffers = 12GB    # Increase data cached in shared_buffers; typically set to 10% of total RAM
work_mem = 128MB # Reduce external sorting; improves performance
maintenance_work_mem = 512M  # Speeds up index creation and VACUUM operations
effective_cache_size = 20GB  # Available system memory