Kustomize Private Deployment of Multi-Scenario, Multi-Environment Container Services

wzdushu@gmail.com (wanzi) — Sat, 25 Oct 2025 10:30:00 +0800

I have been using Helm or native kubectl YAML combined with CI and ArgoCD to deploy applications for a long time. Recently, while deploying private services for a major financial client, I found the client’s environment to be quite complex, involving multiple countries and multiple sets of environments, such as Test, UAT, and Prod—about thirty environments in total. Modifying values and YAML files every time was exhausting. This time, I plan to use Kustomize to refactor our existing application deployment.

Kustomize Introduction

What is Kustomize

Kustomize is a Kubernetes-native configuration management tool that manages YAML resources through a templating approach:

Core Features:

No template syntax, pure YAML
Declarative configuration management
Supports multi-environment management
GitOps friendly
Integrated into kubectl 1.14+

Comparison with Helm:

Feature	Kustomize	Helm
Template Syntax	None (Pure YAML)	Go Template
Learning Curve	Low	Medium-High
Multi-Environment Management	Base + Overlay	Values files
Package Management	None	Chart Repository
Version Management	Git	Chart Version
Suitable Scenarios	Application Developer	Application Distributor

Basic Concepts

kustomization.yaml
├── bases/          # Base configuration (reuse)
├── overlays/       # Environment overrides
│   ├── dev/
│   ├── staging/
│   └── prod/
├── patches/        # Configuration patches
└── resources/      # Static resources

Differences between Kustomize and Helm

Design Philosophy

Kustomize Design Philosophy:

# Declarative configuration management
# Does not use templates, but modifies configuration via Overlay

# Base: Base configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: app
          image: my-app:latest

# Overlay: Environment differences
# overlays/production/kustomization.yaml
bases:
  - ../../base
patchesStrategicMerge:
  - |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-app
    spec:
      replicas: 10

Helm Design Philosophy:

# Templated configuration management
# Uses Go Template

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
        - name: app
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

# values.yaml (Different values for different environments)
replicaCount: 1
image:
  repository: my-app
  tag: latest

# values-production.yaml
replicaCount: 10
image:
  repository: my-app
  tag: v1.2.3

Feature Comparison

Feature	Kustomize	Helm
Templates	No templates, pure YAML	Go Template
Learning Curve	Low, only need to know YAML	Medium-High, need to learn template syntax
Reusability	Base + Overlay	Chart + Values
Debugging	kubectl kustomize	helm template –dry-run
Version Management	Git version control	Chart version number
Environment Management	Overlay directories	Multiple Values files
Package Distribution	None (use Git)	Chart repository
CI/CD Integration	Native kubectl	Requires helm installation
Rollback	kubectl rollout	helm rollback
Dependency Management	No direct support	Chart dependencies

Applicable Scenarios

Scenarios for choosing Kustomize:

Application developers managing their own K8s resources
Need GitOps workflows
Large configuration differences across environments
Do not want to learn template languages
Team is familiar with YAML and kubectl

Scenarios for choosing Helm:

Application distributors (software vendors)
Need package management and version control
Relatively fixed configuration
Complex application dependencies
Quick deployment of third-party applications

Deploying K8s Applications with Kustomize

Project Structure

my-app/
├── base/                           # Base configuration
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap.yaml
│   ├── serviceaccount.yaml
│   └── kustomization.yaml
├── overlays/                       # Environment configuration
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   ├── deployment-replica.yaml
│   │   ├── deployment-resources.yaml
│   │   └── configmap-env.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── deployment-replica.yaml
│   └── production/
│       ├── kustomization.yaml
│       ├── deployment-replica.yaml
│       ├── deployment-resources.yaml
│       └── configmap-prod.yaml
└── scripts/
    └── deploy.sh

Creating Base Configuration

base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
    version: v1.0.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: my-app
        version: v1.0.0
    spec:
      serviceAccountName: my-app
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: my-app
          image: my-app:latest
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          env:
            - name: JAVA_OPTS
              value: "-Xms512m -Xmx512m"
            - name: SPRING_PROFILES_ACTIVE
              value: "prod"
          resources:
            requests:
              cpu: 100m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - name: config
              mountPath: /config
            - name: logs
              mountPath: /logs
      volumes:
        - name: config
          configMap:
            name: my-app-config
        - name: logs
          emptyDir: {}

base/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: my-app

base/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-config
data:
  application.yml: |
    server:
      port: 8080
    spring:
      application:
        name: my-app
    logging:
      level:
        root: INFO

base/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app

base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Referenced resource files
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml
  - serviceaccount.yaml

# Common labels (added to all resources)
commonLabels:
  app.kubernetes.io/name: my-app
  app.kubernetes.io/instance: my-app
  app.kubernetes.io/managed-by: kustomize
  app.kubernetes.io/version: v1.0.0

# Common annotations
commonAnnotations:
  createdBy: kustomize
  env: base

# Name prefix
# namePrefix: dev-

# Name suffix
# nameSuffix: -v1

Overlay Environment Configuration

overlays/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Reference base
namespace: production
bases:
  - ../../base

# Replica count override
replicas:
  - name: my-app
    count: 10

# Image override
images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

# ConfigMap generator
configMapGenerator:
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml

# Secret generator
secretGenerator:
  - name: my-app-secret
    behavior: create
    envs:
      - .env.production

# Resource patches
patchesStrategicMerge:
  - deployment-resources.yaml
  - deployment-probes.yaml
  - pvc.yaml

# Namespace
namespace: production

# Common labels override
commonLabels:
  environment: production
  tier: backend

# Common annotations override
commonAnnotations:
  env: production
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

overlays/production/deployment-resources.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: my-app
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: 2000m
              memory: 4Gi

overlays/production/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 50Gi

overlays/production/application-production.yml

server:
  port: 8080

spring:
  application:
    name: my-app
  datasource:
    url: jdbc:mysql://mysql-prod.svc:3306/mydb
    username: prod_user
    password: ${DB_PASSWORD}

logging:
  level:
    root: INFO
    com.example: DEBUG
  file:
    path: /logs

overlays/production/.env.production

1
2
3

DB_PASSWORD=prod_secure_password_123
API_KEY=prod_api_key_xyz
REDIS_PASSWORD=prod_redis_pass

Deployment Commands

# Build configuration (do not apply, view final YAML)
kubectl kustomize overlays/production

# Save rendered result
kubectl kustomize overlays/production > rendered-prod.yaml

# View differences
kubectl diff -k overlays/production

# Apply directly
kubectl apply -k overlays/production

# Delete resources
kubectl delete -k overlays/production

# View resources
kubectl get all -l app.kubernetes.io/name=my-app -n production

Unified Deployment Management Configuration

1. Namespace Management

Method 1: Specify in Overlay

1
2

# overlays/production/kustomization.yaml
namespace: production

Method 2: Manage as Resource

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - namespace.yaml

# overlays/production/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    environment: production
    name: production

2. Secret Management

Method 1: Environment Variable File

# overlays/production/kustomization.yaml
secretGenerator:
  - name: my-app-secret
    type: Opaque
    envs:
      - .env.production

Method 2: Literals

secretGenerator:
  - name: my-app-secret
    type: Opaque
    literals:
      - DB_PASSWORD=prod123
      - API_KEY=xyz789

Method 3: Command Generation

secretGenerator:
  - name: tls-cert
    type: kubernetes.io/tls
    files:
      - tls.crt=tls.crt
      - tls.key=tls.key

Method 4: Reference from Existing Secret

# overlays/production/kustomization.yaml
generatorOptions:
  disableNameSuffixHash: true

resources:
  - secret.yaml

# overlays/production/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
type: Opaque
stringData:
  DB_PASSWORD: prod123

Method 5: Sealed Secrets (Recommended for Production)

# Generate sealed secret locally
kubeseal -f secret.yaml -w sealed-secret.yaml

# Reference in kustomization
resources:
  - sealed-secret.yaml

3. PVC Management

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - pvc-logs.yaml
  - pvc-data.yaml

# overlays/production/pvc-logs.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd  # Use SSD storage class
  resources:
    requests:
      storage: 50Gi

# overlays/production/pvc-data.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
spec:
  accessModes:
    - ReadWriteMany  # Multi-node shared
  storageClassName: nfs-storage
  resources:
    requests:
      storage: 100Gi

4. Image Version Management

Method 1: Direct Specification

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

Method 2: Use SHA256

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    digest: sha256:abc123def456...  # Safest method

Method 3: Multiple Images

images:
  - name: nginx
    newName: harbor.example.com/library/nginx
    newTag: 1.21-alpine
  - name: redis
    newName: harbor.example.com/library/redis
    newTag: 6.2-alpine
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

Method 4: CI/CD Dynamic Update

# In CI Pipeline
export IMAGE_TAG=$(git rev-parse --short HEAD)
cd overlays/production
kustomize edit set image my-app=harbor.example.com/prod/my-app:$IMAGE_TAG
kubectl apply -k .

5. Node Affinity Configuration

# overlays/production/deployment-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      affinity:
        # Node affinity
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - c5.2xlarge
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values:
                      - us-west-1a
        # Pod affinity
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - mysql
                topologyKey: kubernetes.io/hostname

Reference in kustomization:

1
2

patchesStrategicMerge:
  - deployment-affinity.yaml

6. Taint and Toleration Configuration

# overlays/production/deployment-taints.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      # Tolerate dedicated node taints
      tolerations:
        - key: workload
          operator: Equal
          value: dedicated
          effect: NoSchedule
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 300
        # Tolerate GPU nodes
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

7. Replica Count Management

# Method 1: replicas field (Recommended)
replicas:
  - name: my-app
    count: 10
  - name: my-app-sidecar
    count: 2

# Method 2: Patch method
# overlays/production/deployment-replica.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 10

8. ConfigMap Management

# Method 1: configMapGenerator (Recommended)
configMapGenerator:
  # Generate from file
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml
      - logback.xml=logback-production.xml

  # Generate from literals
  - name: my-app-env
    behavior: create
    literals:
      - SPRING_PROFILES_ACTIVE=production
      - LOG_LEVEL=INFO
      - JAVA_OPTS=-Xms2g -Xmx4g

  # Generate from env file
  - name: my-app-file-env
    behavior: create
    envs:
      - .env.production

# Method 2: Direct resource reference
resources:
  - configmap.yaml

Complete Production Environment Example