Kustomize私有化部署多场景多环境容器服务

wzdushu@gmail.com (wanzi) — Sat, 25 Oct 2025 10:30:00 +0800

早期一直在使用helm或者kubectl原生yaml结合CI和ArgoCD部署应用，最近在给一个金融大客户部署私有化服务的时候，发现客户的环境比较复杂，涉及多个国家多套环境，比如Test、UAT、Prod，差不多有三十个环境，每次修改的values和yaml文件快吐了，这次打算用Kustomize来改造现有的部署应用试试。

Kustomize介绍

什么是Kustomize

Kustomize是Kubernetes原生的配置管理工具，通过模板化的方式管理YAML资源：

核心特性：

无模板语法，纯YAML
声明式配置管理
支持多环境管理
GitOps友好
已集成kubectl 1.14+

与Helm对比：

特性	Kustomize	Helm
模板语法	无（纯YAML）	Go Template
学习曲线	低	中高
多环境管理	Base + Overlay	Values文件
包管理	无	Chart Repository
版本管理	Git	Chart版本
适合场景	应用开发者	应用分发者

基本概念

kustomization.yaml
├── bases/          # 基础配置（复用）
├── overlays/       # 环境覆盖
│   ├── dev/
│   ├── staging/
│   └── prod/
├── patches/        # 配置补丁
└── resources/      # 静态资源

Kustomize与Helm区别

设计理念

Kustomize设计理念：

# 声明式配置管理
# 不使用模板，而是通过叠加（Overlay）方式修改配置

# Base: 基础配置
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: app
          image: my-app:latest

# Overlay: 环境差异
# overlays/production/kustomization.yaml
bases:
  - ../../base
patchesStrategicMerge:
  - |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-app
    spec:
      replicas: 10

Helm设计理念：

# 模板化配置管理
# 使用Go Template模板

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
        - name: app
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

# values.yaml (不同环境不同values)
replicaCount: 1
image:
  repository: my-app
  tag: latest

# values-production.yaml
replicaCount: 10
image:
  repository: my-app
  tag: v1.2.3

功能对比

功能	Kustomize	Helm
模板	无模板，纯YAML	Go Template
学习曲线	低，只需会YAML	中高，需学模板语法
复用性	Base + Overlay	Chart + Values
调试	kubectl kustomize	helm template –dry-run
版本管理	Git版本控制	Chart版本号
环境管理	Overlay目录	多个Values文件
包分发	无（用Git）	Chart仓库
CI/CD集成	原生kubectl	需安装helm
Rollback	kubectl rollout	helm rollback
依赖管理	无直接支持	Chart dependencies

适用场景

选择Kustomize的场景：

应用开发者管理自己的K8s资源
需要GitOps工作流
多环境配置差异大
不想学习模板语言
团队熟悉YAML和kubectl

选择Helm的场景：

应用分发者（软件供应商）
需要包管理和版本控制
配置相对固定
复杂的应用依赖关系
快速部署第三方应用

Kustomize部署K8s应用

项目结构

my-app/
├── base/                           # 基础配置
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap.yaml
│   ├── serviceaccount.yaml
│   └── kustomization.yaml
├── overlays/                       # 环境配置
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   ├── deployment-replica.yaml
│   │   ├── deployment-resources.yaml
│   │   └── configmap-env.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── deployment-replica.yaml
│   └── production/
│       ├── kustomization.yaml
│       ├── deployment-replica.yaml
│       ├── deployment-resources.yaml
│       └── configmap-prod.yaml
└── scripts/
    └── deploy.sh

Base配置创建

base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
    version: v1.0.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: my-app
        version: v1.0.0
    spec:
      serviceAccountName: my-app
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: my-app
          image: my-app:latest
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          env:
            - name: JAVA_OPTS
              value: "-Xms512m -Xmx512m"
            - name: SPRING_PROFILES_ACTIVE
              value: "prod"
          resources:
            requests:
              cpu: 100m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - name: config
              mountPath: /config
            - name: logs
              mountPath: /logs
      volumes:
        - name: config
          configMap:
            name: my-app-config
        - name: logs
          emptyDir: {}

base/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: my-app

base/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-config
data:
  application.yml: |
    server:
      port: 8080
    spring:
      application:
        name: my-app
    logging:
      level:
        root: INFO

base/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app

base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# 引用的资源文件
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml
  - serviceaccount.yaml

# 通用标签（所有资源添加）
commonLabels:
  app.kubernetes.io/name: my-app
  app.kubernetes.io/instance: my-app
  app.kubernetes.io/managed-by: kustomize
  app.kubernetes.io/version: v1.0.0

# 通用注解
commonAnnotations:
  createdBy: kustomize
  env: base

# 名称前缀
# namePrefix: dev-

# 名称后缀
# nameSuffix: -v1

Overlay环境配置

overlays/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# 引用base
namespace: production
bases:
  - ../../base

# 副本数覆盖
replicas:
  - name: my-app
    count: 10

# 镜像覆盖
images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

# ConfigMap生成器
configMapGenerator:
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml

# Secret生成器
secretGenerator:
  - name: my-app-secret
    behavior: create
    envs:
      - .env.production

# 资源补丁
patchesStrategicMerge:
  - deployment-resources.yaml
  - deployment-probes.yaml
  - pvc.yaml

# 命名空间
namespace: production

# 通用标签覆盖
commonLabels:
  environment: production
  tier: backend

# 通用注解覆盖
commonAnnotations:
  env: production
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

overlays/production/deployment-resources.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: my-app
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: 2000m
              memory: 4Gi

overlays/production/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 50Gi

overlays/production/application-production.yml

server:
  port: 8080

spring:
  application:
    name: my-app
  datasource:
    url: jdbc:mysql://mysql-prod.svc:3306/mydb
    username: prod_user
    password: ${DB_PASSWORD}

logging:
  level:
    root: INFO
    com.example: DEBUG
  file:
    path: /logs

overlays/production/.env.production

1
2
3

DB_PASSWORD=prod_secure_password_123
API_KEY=prod_api_key_xyz
REDIS_PASSWORD=prod_redis_pass

部署命令

# 构建配置（不应用，查看最终YAML）
kubectl kustomize overlays/production

# 保存渲染结果
kubectl kustomize overlays/production > rendered-prod.yaml

# 查看差异
kubectl diff -k overlays/production

# 直接应用
kubectl apply -k overlays/production

# 删除资源
kubectl delete -k overlays/production

# 查看资源
kubectl get all -l app.kubernetes.io/name=my-app -n production

统一部署管理配置

1. Namespace管理

方式一：在Overlay中指定

1
2

# overlays/production/kustomization.yaml
namespace: production

方式二：作为资源管理

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - namespace.yaml

# overlays/production/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    environment: production
    name: production

2. Secret管理

方案一：环境变量文件

# overlays/production/kustomization.yaml
secretGenerator:
  - name: my-app-secret
    type: Opaque
    envs:
      - .env.production

方案二：字面值

secretGenerator:
  - name: my-app-secret
    type: Opaque
    literals:
      - DB_PASSWORD=prod123
      - API_KEY=xyz789

方案三：命令生成

secretGenerator:
  - name: tls-cert
    type: kubernetes.io/tls
    files:
      - tls.crt=tls.crt
      - tls.key=tls.key

方案四：从现有Secret引用

# overlays/production/kustomization.yaml
generatorOptions:
  disableNameSuffixHash: true

resources:
  - secret.yaml

# overlays/production/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
type: Opaque
stringData:
  DB_PASSWORD: prod123

方案五：Sealed Secrets（推荐生产使用）

# 本地生成sealed secret
kubeseal -f secret.yaml -w sealed-secret.yaml

# 在kustomization中引用
resources:
  - sealed-secret.yaml

3. PVC管理

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - pvc-logs.yaml
  - pvc-data.yaml

# overlays/production/pvc-logs.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd  # 使用SSD存储类
  resources:
    requests:
      storage: 50Gi

# overlays/production/pvc-data.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
spec:
  accessModes:
    - ReadWriteMany  # 多节点共享
  storageClassName: nfs-storage
  resources:
    requests:
      storage: 100Gi

4. 镜像版本管理

方式一：直接指定

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

方式二：使用SHA256

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    digest: sha256:abc123def456...  # 最安全的方式

方式三：多个镜像

images:
  - name: nginx
    newName: harbor.example.com/library/nginx
    newTag: 1.21-alpine
  - name: redis
    newName: harbor.example.com/library/redis
    newTag: 6.2-alpine
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

方式四：CI/CD动态更新

# 在CI Pipeline中
export IMAGE_TAG=$(git rev-parse --short HEAD)
cd overlays/production
kustomize edit set image my-app=harbor.example.com/prod/my-app:$IMAGE_TAG
kubectl apply -k .

5. 节点亲和性配置

# overlays/production/deployment-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      affinity:
        # 节点亲和性
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - c5.2xlarge
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values:
                      - us-west-1a
        # Pod亲和性
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - mysql
                topologyKey: kubernetes.io/hostname

在kustomization中引用：

1
2

patchesStrategicMerge:
  - deployment-affinity.yaml

6. 污点和容忍配置

# overlays/production/deployment-taints.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      # 容忍专用节点污点
      tolerations:
        - key: workload
          operator: Equal
          value: dedicated
          effect: NoSchedule
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 300
        # 容许GPU节点
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

7. 副本数量管理

# 方式一：replicas字段（推荐）
replicas:
  - name: my-app
    count: 10
  - name: my-app-sidecar
    count: 2

# 方式二：Patch方式
# overlays/production/deployment-replica.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 10

8. ConfigMap管理

# 方式一：configMapGenerator（推荐）
configMapGenerator:
  # 从文件生成
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml
      - logback.xml=logback-production.xml

  # 从字面值生成
  - name: my-app-env
    behavior: create
    literals:
      - SPRING_PROFILES_ACTIVE=production
      - LOG_LEVEL=INFO
      - JAVA_OPTS=-Xms2g -Xmx4g

  # 从env文件生成
  - name: my-app-file-env
    behavior: create
    envs:
      - .env.production

# 方式二：直接引用资源
resources:
  - configmap.yaml

完整生产环境示例