Argocd on Wanzi's Blog – Notes on life, tech, and DevOps.

Kustomize Private Deployment of Multi-Scenario, Multi-Environment Container Services

wzdushu@gmail.com (wanzi) — Sat, 25 Oct 2025 10:30:00 +0800

I have been using Helm or native kubectl YAML combined with CI and ArgoCD to deploy applications for a long time. Recently, while deploying private services for a major financial client, I found the client’s environment to be quite complex, involving multiple countries and multiple sets of environments, such as Test, UAT, and Prod—about thirty environments in total. Modifying values and YAML files every time was exhausting. This time, I plan to use Kustomize to refactor our existing application deployment.

Kustomize Introduction

What is Kustomize

Kustomize is a Kubernetes-native configuration management tool that manages YAML resources through a templating approach:

Core Features:

No template syntax, pure YAML
Declarative configuration management
Supports multi-environment management
GitOps friendly
Integrated into kubectl 1.14+

Comparison with Helm:

Feature	Kustomize	Helm
Template Syntax	None (Pure YAML)	Go Template
Learning Curve	Low	Medium-High
Multi-Environment Management	Base + Overlay	Values files
Package Management	None	Chart Repository
Version Management	Git	Chart Version
Suitable Scenarios	Application Developer	Application Distributor

Basic Concepts

kustomization.yaml
├── bases/          # Base configuration (reuse)
├── overlays/       # Environment overrides
│   ├── dev/
│   ├── staging/
│   └── prod/
├── patches/        # Configuration patches
└── resources/      # Static resources

Differences between Kustomize and Helm

Design Philosophy

Kustomize Design Philosophy:

# Declarative configuration management
# Does not use templates, but modifies configuration via Overlay

# Base: Base configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: app
          image: my-app:latest

# Overlay: Environment differences
# overlays/production/kustomization.yaml
bases:
  - ../../base
patchesStrategicMerge:
  - |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-app
    spec:
      replicas: 10

Helm Design Philosophy:

# Templated configuration management
# Uses Go Template

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
        - name: app
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

# values.yaml (Different values for different environments)
replicaCount: 1
image:
  repository: my-app
  tag: latest

# values-production.yaml
replicaCount: 10
image:
  repository: my-app
  tag: v1.2.3

Feature Comparison

Feature	Kustomize	Helm
Templates	No templates, pure YAML	Go Template
Learning Curve	Low, only need to know YAML	Medium-High, need to learn template syntax
Reusability	Base + Overlay	Chart + Values
Debugging	kubectl kustomize	helm template –dry-run
Version Management	Git version control	Chart version number
Environment Management	Overlay directories	Multiple Values files
Package Distribution	None (use Git)	Chart repository
CI/CD Integration	Native kubectl	Requires helm installation
Rollback	kubectl rollout	helm rollback
Dependency Management	No direct support	Chart dependencies

Applicable Scenarios

Scenarios for choosing Kustomize:

Application developers managing their own K8s resources
Need GitOps workflows
Large configuration differences across environments
Do not want to learn template languages
Team is familiar with YAML and kubectl

Scenarios for choosing Helm:

Application distributors (software vendors)
Need package management and version control
Relatively fixed configuration
Complex application dependencies
Quick deployment of third-party applications

Deploying K8s Applications with Kustomize

Project Structure

my-app/
├── base/                           # Base configuration
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap.yaml
│   ├── serviceaccount.yaml
│   └── kustomization.yaml
├── overlays/                       # Environment configuration
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   ├── deployment-replica.yaml
│   │   ├── deployment-resources.yaml
│   │   └── configmap-env.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── deployment-replica.yaml
│   └── production/
│       ├── kustomization.yaml
│       ├── deployment-replica.yaml
│       ├── deployment-resources.yaml
│       └── configmap-prod.yaml
└── scripts/
    └── deploy.sh

Creating Base Configuration

base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
    version: v1.0.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: my-app
        version: v1.0.0
    spec:
      serviceAccountName: my-app
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: my-app
          image: my-app:latest
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          env:
            - name: JAVA_OPTS
              value: "-Xms512m -Xmx512m"
            - name: SPRING_PROFILES_ACTIVE
              value: "prod"
          resources:
            requests:
              cpu: 100m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - name: config
              mountPath: /config
            - name: logs
              mountPath: /logs
      volumes:
        - name: config
          configMap:
            name: my-app-config
        - name: logs
          emptyDir: {}

base/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: my-app

base/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-config
data:
  application.yml: |
    server:
      port: 8080
    spring:
      application:
        name: my-app
    logging:
      level:
        root: INFO

base/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app

base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Referenced resource files
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml
  - serviceaccount.yaml

# Common labels (added to all resources)
commonLabels:
  app.kubernetes.io/name: my-app
  app.kubernetes.io/instance: my-app
  app.kubernetes.io/managed-by: kustomize
  app.kubernetes.io/version: v1.0.0

# Common annotations
commonAnnotations:
  createdBy: kustomize
  env: base

# Name prefix
# namePrefix: dev-

# Name suffix
# nameSuffix: -v1

Overlay Environment Configuration

overlays/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Reference base
namespace: production
bases:
  - ../../base

# Replica count override
replicas:
  - name: my-app
    count: 10

# Image override
images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

# ConfigMap generator
configMapGenerator:
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml

# Secret generator
secretGenerator:
  - name: my-app-secret
    behavior: create
    envs:
      - .env.production

# Resource patches
patchesStrategicMerge:
  - deployment-resources.yaml
  - deployment-probes.yaml
  - pvc.yaml

# Namespace
namespace: production

# Common labels override
commonLabels:
  environment: production
  tier: backend

# Common annotations override
commonAnnotations:
  env: production
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

overlays/production/deployment-resources.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: my-app
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: 2000m
              memory: 4Gi

overlays/production/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 50Gi

overlays/production/application-production.yml

server:
  port: 8080

spring:
  application:
    name: my-app
  datasource:
    url: jdbc:mysql://mysql-prod.svc:3306/mydb
    username: prod_user
    password: ${DB_PASSWORD}

logging:
  level:
    root: INFO
    com.example: DEBUG
  file:
    path: /logs

overlays/production/.env.production

1
2
3

DB_PASSWORD=prod_secure_password_123
API_KEY=prod_api_key_xyz
REDIS_PASSWORD=prod_redis_pass

Deployment Commands

# Build configuration (do not apply, view final YAML)
kubectl kustomize overlays/production

# Save rendered result
kubectl kustomize overlays/production > rendered-prod.yaml

# View differences
kubectl diff -k overlays/production

# Apply directly
kubectl apply -k overlays/production

# Delete resources
kubectl delete -k overlays/production

# View resources
kubectl get all -l app.kubernetes.io/name=my-app -n production

Unified Deployment Management Configuration

1. Namespace Management

Method 1: Specify in Overlay

1
2

# overlays/production/kustomization.yaml
namespace: production

Method 2: Manage as Resource

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - namespace.yaml

# overlays/production/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    environment: production
    name: production

2. Secret Management

Method 1: Environment Variable File

# overlays/production/kustomization.yaml
secretGenerator:
  - name: my-app-secret
    type: Opaque
    envs:
      - .env.production

Method 2: Literals

secretGenerator:
  - name: my-app-secret
    type: Opaque
    literals:
      - DB_PASSWORD=prod123
      - API_KEY=xyz789

Method 3: Command Generation

secretGenerator:
  - name: tls-cert
    type: kubernetes.io/tls
    files:
      - tls.crt=tls.crt
      - tls.key=tls.key

Method 4: Reference from Existing Secret

# overlays/production/kustomization.yaml
generatorOptions:
  disableNameSuffixHash: true

resources:
  - secret.yaml

# overlays/production/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
type: Opaque
stringData:
  DB_PASSWORD: prod123

Method 5: Sealed Secrets (Recommended for Production)

# Generate sealed secret locally
kubeseal -f secret.yaml -w sealed-secret.yaml

# Reference in kustomization
resources:
  - sealed-secret.yaml

3. PVC Management

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - pvc-logs.yaml
  - pvc-data.yaml

# overlays/production/pvc-logs.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd  # Use SSD storage class
  resources:
    requests:
      storage: 50Gi

# overlays/production/pvc-data.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
spec:
  accessModes:
    - ReadWriteMany  # Multi-node shared
  storageClassName: nfs-storage
  resources:
    requests:
      storage: 100Gi

4. Image Version Management

Method 1: Direct Specification

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

Method 2: Use SHA256

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    digest: sha256:abc123def456...  # Safest method

Method 3: Multiple Images

images:
  - name: nginx
    newName: harbor.example.com/library/nginx
    newTag: 1.21-alpine
  - name: redis
    newName: harbor.example.com/library/redis
    newTag: 6.2-alpine
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

Method 4: CI/CD Dynamic Update

# In CI Pipeline
export IMAGE_TAG=$(git rev-parse --short HEAD)
cd overlays/production
kustomize edit set image my-app=harbor.example.com/prod/my-app:$IMAGE_TAG
kubectl apply -k .

5. Node Affinity Configuration

# overlays/production/deployment-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      affinity:
        # Node affinity
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - c5.2xlarge
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values:
                      - us-west-1a
        # Pod affinity
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - mysql
                topologyKey: kubernetes.io/hostname

Reference in kustomization:

1
2

patchesStrategicMerge:
  - deployment-affinity.yaml

6. Taint and Toleration Configuration

# overlays/production/deployment-taints.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      # Tolerate dedicated node taints
      tolerations:
        - key: workload
          operator: Equal
          value: dedicated
          effect: NoSchedule
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 300
        # Tolerate GPU nodes
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

7. Replica Count Management

# Method 1: replicas field (Recommended)
replicas:
  - name: my-app
    count: 10
  - name: my-app-sidecar
    count: 2

# Method 2: Patch method
# overlays/production/deployment-replica.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 10

8. ConfigMap Management

# Method 1: configMapGenerator (Recommended)
configMapGenerator:
  # Generate from file
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml
      - logback.xml=logback-production.xml

  # Generate from literals
  - name: my-app-env
    behavior: create
    literals:
      - SPRING_PROFILES_ACTIVE=production
      - LOG_LEVEL=INFO
      - JAVA_OPTS=-Xms2g -Xmx4g

  # Generate from env file
  - name: my-app-file-env
    behavior: create
    envs:
      - .env.production

# Method 2: Direct resource reference
resources:
  - configmap.yaml

Complete Production Environment Example

overlays/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Namespace
namespace: production

# Reference base
bases:
  - ../../base

# Additional resources
resources:
  - namespace.yaml
  - pvc-logs.yaml
  - pvc-data.yaml
  - hpa.yaml
  - service-monitor.yaml
  - network-policy.yaml

# Replica count
replicas:
  - name: my-app
    count: 10

# Image version
images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

# ConfigMap
configMapGenerator:
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=config/application-production.yml
  - name: my-app-env
    behavior: merge
    literals:
      - ENV=production
      - LOG_LEVEL=INFO

# Secret
secretGenerator:
  - name: my-app-secret
    behavior: create
    envs:
      - .env.production

# Patches
patchesStrategicMerge:
  - deployment-resources.yaml
  - deployment-affinity.yaml
  - deployment-taints.yaml
  - deployment-probes.yaml

# Common labels
commonLabels:
  environment: production
  tier: backend
  team: platform

# Common annotations
commonAnnotations:
  env: production
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

CI/CD Integration

GitLab CI Example

# .gitlab-ci.yml
stages:
  - build
  - deploy

variables:
  REGISTRY: harbor.example.com
  APP_NAME: my-app

build:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker build -t $REGISTRY/$APP_NAME:$CI_COMMIT_SHA .
    - docker push $REGISTRY/$APP_NAME:$CI_COMMIT_SHA
  only:
    - main
    - develop

deploy:dev:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: dev
  script:
    - cd overlays/dev
    - kustomize edit set image $REGISTRY/$APP_NAME=$REGISTRY/$APP_NAME:$CI_COMMIT_SHA
    - kubectl apply -k .
    - kubectl rollout status deployment/my-app -n dev
  only:
    - develop

deploy:prod:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: production
  script:
    - cd overlays/production
    - kustomize edit set image $REGISTRY/$APP_NAME=$REGISTRY/$APP_NAME:$CI_COMMIT_SHA
    - kubectl apply -k .
    - kubectl rollout status deployment/my-app -n production
  when: manual
  only:
    - main

ArgoCD Integration

# argocd/application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-production
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/example/k8s-manifests.git
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Summary

Kustomize Best Practices

Directory Structure Standards
- Store common configurations in base
- Manage environment differences in overlays
- Use clear naming conventions
Version Management
- Use Git to manage all configurations
- Use SHA256 or explicit tags for images
- Do not use latest
Secret Management
- Use Sealed Secrets in production environments
- Do not commit sensitive information to Git
- Use External Secrets Operator to integrate with Vault
Component Reuse
- Extract common configurations as components
- Establish a Kustomize component library
- Avoid duplicate configurations
CI/CD Integration
- Automate image tag updates
- Use kubectl diff to preview changes
- Apply configuration only after validation passes

Kustomize vs Helm Selection

Scenario	Recommended Solution	Reason
Microservice Configuration Management	Kustomize	High flexibility, Git-friendly
Third-party Application Deployment	Helm	Rich ecosystem, one-click deployment
Complex Template Requirements	Helm	Go Template is powerful
Simple Configuration	Kustomize	No need to learn templates
GitOps	Kustomize	Native support

Learning Path

Basic Syntax: base + overlay structure
Patch Strategies: StrategicMerge vs JSON6902
Component Reuse: Extract common configurations
CI/CD Integration: Automated deployment workflows
GitOps Practice: Combine with ArgoCD/Flux

Kustomize is an ideal tool for configuration management in the cloud-native era. Mastering it can significantly improve Kubernetes application management efficiency. In actual deployment scenarios, if the differentiation between environments for new projects is small, you can combine VibeCoding tools to generate one-click deployment scripts for rapid delivery of new environments.

Automated Application Deployment with ArgoCD and Jenkins Pipeline

wzdushu@gmail.com (wanzi) — Wed, 29 Jul 2020 15:22:42 +0800

Create Helm Repository

First, create a basic Helm template repository:

`1`	`helm create template .`

For actual deployments, you’ll need to customize the Helm template according to your business requirements. Here, we directly use an internal custom generic template for rapid deployment. Alternatively, you can refer to Bitnami’s maintained Helm charts: https://github.com/bitnami/charts/tree/master/bitnami

Jenkins Credential Configuration: ArgoCD Token

Configure Jenkins Pipeline

We’ll use the gotest project (https://code.test.cn/hqliang/gotest) as an example.

pipeline {
    environment {
        GOPROXY = "http://repo.test.cn/repository/goproxy/"
        HUB_URL = "registry.test.cn"
        ARGOCD_SERVER = "qacd.test.cn"
        ARGOCD_PROJ = "test-project"
        ARGOCD_APP = "gotest"
        ARGOCD_REPO = "https://code.test.cn/devops/cicd/qa.git"
        ARGOCD_PATH = "devops/gotest"
        ARGOCD_CLUSTER = "https://172.16.19.250:8443"
        ARGOCD_NS = "default"
    }

    agent {
        node {
            label 'aiops'
        }
    }

    stages {
        stage('Docker_Build') {
            steps {
                withCredentials([[$class: 'UsernamePasswordMultiBinding',
                    credentialsId: '12ff2942-972c-4df3-8d2d-2cfcb25e00de',
                    usernameVariable: 'DOCKER_HUB_USER',
                    passwordVariable: 'DOCKER_HUB_PASSWORD']]) {
                    sh '''
                    TAG=$(git describe --tags `git rev-list --tags --max-count=1`)
                    docker login registry.test.cn -u ${DOCKER_HUB_USER} -p ${DOCKER_HUB_PASSWORD}
                    make docker-all VERSION=$TAG
                    '''
                }
            }
        }

        stage('Deploy_K8S') {
            steps {
                withCredentials([string(credentialsId: "qa-argocd", variable: 'ARGOCD_AUTH_TOKEN')]) {
                    sh '''
                    TAG=$(git describe --tags `git rev-list --tags --max-count=1`)
                    # Create app
                    ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app create $ARGOCD_APP \
                        --project $ARGOCD_PROJ \
                        --repo $ARGOCD_REPO \
                        --path $ARGOCD_PATH \
                        --dest-namespace $ARGOCD_NS \
                        --dest-server $ARGOCD_CLUSTER \
                        --upsert --insecure
                    # Deploy app
                    ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app set $ARGOCD_APP \
                        -p containers.tag=$TAG --insecure
                    # Sync to ArgoCD
                    ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app sync $ARGOCD_APP \
                        --force --insecure
                    ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app wait $ARGOCD_APP \
                        --timeout 600 --insecure
                    '''
                }
            }
        }
    }
}

Configure GitLab Webhook to Trigger Jenkins

Jenkins Configuration: Trigger Build

GitLab Webhook Configuration

Push Tag to Remote Branch

Jenkins Pipeline Build

Check the Jenkins pipeline build result:

Docker automatically builds and pushes the image to Harbor:

Create Argo application and sync deployment to Kubernetes cluster:

Finally, check the sync status — indicating successful deployment to the Kubernetes cluster:

View the application workflow graph in ArgoCD’s dashboard — all components are running normally:

Use kubectl to verify the created resources:

Finally, validate if the service is accessible: visit http://gotest.test.cn/df to check container disk space.

Summary & Optimization

Whether using Jenkins pipelines or GitLab CI, we can better integrate ArgoCD to achieve one-click deployment of applications.

When deploying a Deployment in Argo CD, you encounter the error: `no space left on device`.

wzdushu@gmail.com (wanzi) — Mon, 18 May 2020 10:22:42 +0800

Failure Phenomenon

This morning, I deployed several business applications via ArgoCD. After successfully deploying two applications, subsequent deployments from a third-party source consistently failed—despite using identical configurations, only the target cluster differed. Why would this happen?

I checked the logs and found the following:

1
2
3

  Warning  Failed     1m                kubelet, 172.16.25.13  Error: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/ba37165607862efb350093e5e287207e2547759fd81dc4e5e356a86ac5e28324-init/merged: no space left on device
  Warning  Failed     1m                kubelet, 172.16.25.13  Error: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/f69b62f360fc2a94487aca041b08d0929810beab0602e0ec8b90c94b2e893337-init/merged: no space left on device
  Warning  Failed     48s               kubelet, 172.16.25.13  Error: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/a8d20a44183b39ae989eee8a442960124ff23844482f726ea7ab39a292aecbb3-init/merged: no space left on device

Solution

Check disk space—no issues found:

1
2
3

root@gpu613:~# df -Th /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda2      ext4  1.8T  359G  1.3T  22% /

After Googling, I discovered this might be due to exhausted inotify watches.

Check current limit:

1
2

#cat /proc/sys/fs/inotify/max_user_watches
8192

Increase the number of watchable directories:

1
2

echo "fs.inotify.max_user_watches=100000" >> /etc/sysctl.conf
sysctl -p

After applying the change, re-triggered the ArgoCD sync. This time, the deployment was successfully created—confirming that inotify watch exhaustion was indeed the root cause.

Add multiple clusters to ArgoCD

wzdushu@gmail.com (wanzi) — Tue, 05 May 2020 15:22:42 +0800

Generate Argo CD Management User Token

Log in to the dashboard, go to Settings → Accounts → admin → Generate New.
After generation, record the token information, which looks like this:

fyJhbGciOiJ3UzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI2OWI0M2M0Mi01MmZiLTRlZmItODIxOC0yOWU3NGM5MWI0NDIiLCJpYXQiOjE1OTUzMTEx3zQsImlzcyI6ImFyZ29jZCIsIm5iZiI6MTU5NTMxMTE3NCwic3ViIjoib3duZXIifQ.9u4XzArEeaz7G2Q2TWusnTkakEmq9BYDAUHr3dC6wG5

Configure Argo CD Config

For Argo CD with HTTPS enabled, adding clusters becomes cumbersome—it requires logging into the server pod for configuration. Follow these steps:

# cat ~/.argocd/config
contexts:
- name: argocd-server.argocd
  server: qacd.test.cn
  user: argocd-server.argocd
current-context: argocd-server.argocd
servers:
- grpc-web-root-path: ""
  insecure: true
  server: qacd.test.cn
users:
- auth-token: xxxxxx # This is the token generated in step 1
  name: argocd-server.argocd

Configure kubeconfig

Skip detailed configuration here—refer to previous documentation. Ensure you can access the cluster and have cluster administrator privileges. Set the CONTEXT to idc-bj-k8s.

Add Cluster

# argocd --grpc-web cluster add idc-bj-k8s --kubeconfig ~/.kube/config
INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" updated
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" updated
Cluster 'https://172.16.16.250:8443' added
# argocd --grpc-web cluster list
SERVER                          NAME        VERSION  STATUS      MESSAGE
https://172.16.16.250:8443      idc-bj-k8s  1.14     Successful
https://kubernetes.default.svc              1.14     Successful

The Beijing IDC cluster has now been successfully added to Argo CD. You can proceed to deploy applications to it.

Remove Cluster

# argocd --grpc-web cluster rm https://172.16.16.250:8443
# argocd --grpc-web cluster list
SERVER                          NAME        VERSION  STATUS      MESSAGE
https://kubernetes.default.svc              1.14     Successful

ArgoCD Installation and Deployment

wzdushu@gmail.com (wanzi) — Fri, 01 May 2020 15:22:42 +0800

Installation and Deployment

Deploying ArgoCD is straightforward. Use the official high-availability (HA) deployment method:

1
2

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v1.5.2/manifests/ha/install.yaml

You can customize the deployment file as needed. After the pods are successfully started:

# kubectl -n argocd get pod
NAME                                             READY   STATUS    RESTARTS   AGE
argocd-application-controller-66fbf66657-ghf2c   1/1     Running   0          6d17h
argocd-application-controller-66fbf66657-gpm7d   1/1     Running   0          6d17h
argocd-application-controller-66fbf66657-tr5kd   1/1     Running   0          6d17h
argocd-dex-server-5c5f986596-c8ftv               1/1     Running   0          9d
argocd-redis-ha-haproxy-69c6df79c6-2fxd6         1/1     Running   0          9d
argocd-redis-ha-haproxy-69c6df79c6-mksg2         1/1     Running   0          9d
argocd-redis-ha-haproxy-69c6df79c6-wq57f         1/1     Running   0          9d
argocd-redis-ha-server-0                         2/2     Running   0          9d
argocd-redis-ha-server-1                         2/2     Running   0          9d
argocd-redis-ha-server-2                         2/2     Running   0          9d
argocd-repo-server-76bbb56cc7-d8fp5              1/1     Running   0          7d
argocd-repo-server-76bbb56cc7-qvl5z              1/1     Running   0          7d
argocd-repo-server-76bbb56cc7-xqrfn              1/1     Running   0          7d
argocd-server-6464c7bcd-fgktr                    1/1     Running   0          6d19h
argocd-server-6464c7bcd-jkqdb                    1/1     Running   0          6d19h
argocd-server-6464c7bcd-nfdwn                    1/1     Running   0          6d19h

Configure Ingress for ArgoCD Access

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/redirect-entry-point: https
spec:
  rules:
    - host: cd.testcn
      http:
        paths:
        - backend:
            serviceName: argocd-server
            servicePort: https
          path: /

Access ArgoCD via https://cd.test.cn/. The default username is admin, and the password is the name of one of the pods. Retrieve the password using:

`1`	`# kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name \| cut -d'/' -f 2`

User Management

ArgoCD defaults to local users but supports LDAP integration. Manage local users by editing the argocd-cm ConfigMap. To add a user:

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd
data:
  accounts.chlai: apiKey,login  # Allows login and API key generation
  users.anonymous.enabled: "true"  # Enables anonymous login

Changes take effect immediately after saving.

By default, two roles exist: readonly and admin. Grant admin privileges by modifying the argocd-rbac-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.csv: |-
    g, chlai, role:admin
  policy.default: role:readonly  # Anonymous users default to readonly role

To generate a login password, use the admin user via CLI to log in to the server, then update the password with:

argocd login <argocd-server>
argocd account list
argocd account update-password \
  --account <name> \
  --current-password <current-admin> \
  --new-password <new-user-password>

Creating Applications via ArgoCD UI

Click the + NEW APP button to create an application.

Fill in the application details:

Name: guestbook
Project: default
Sync Policy: Manual

Configure the source. Here, use Git; set the repository URL to: https://github.com/argoproj/argocd-example-apps.git; select HEAD as Revision; choose the path: guestbook.

Select the target cluster for deployment: https://kubernetes.default.svc. Since ArgoCD is deployed within a Kubernetes cluster, it has already registered the current cluster—use it directly. Set Namespace to my-app. Create this namespace in Kubernetes using: kubectl create namespace my-app.

After filling in all fields, click CREATE.

Since the application hasn’t been deployed yet and no Kubernetes resources have been created, the status remains OutOfSync. Click SYNC to synchronize (deploy). You can also install the ArgoCD CLI and sync via command: argocd app sync guestbook.

The application is now created but still OutOfSync. Manually sync to start deployment.

Wait for the application to be fully deployed.

Check the application in the Kubernetes cluster.