早期一直在使用helm或者kubectl原生yaml结合CI和ArgoCD部署应用，最近在给一个金融大客户部署私有化服务的时候，发现客户的环境比较复杂，涉及多个国家多套环境，比如Test、UAT、Prod，差不多有三十个环境，每次修改的values和yaml文件快吐了，这次打算用Kustomize来改造现有的部署应用试试。

Kustomize介绍

什么是Kustomize

Kustomize是Kubernetes原生的配置管理工具，通过模板化的方式管理YAML资源：

核心特性：

  • 无模板语法，纯YAML
  • 声明式配置管理
  • 支持多环境管理
  • GitOps友好
  • 已集成kubectl 1.14+

与Helm对比：

特性 Kustomize Helm
模板语法 无（纯YAML） Go Template
学习曲线 低 中高
多环境管理 Base + Overlay Values文件
包管理 无 Chart Repository
版本管理 Git Chart版本
适合场景 应用开发者 应用分发者

基本概念

kustomization.yaml
├── bases/          # 基础配置（复用）
├── overlays/       # 环境覆盖
│   ├── dev/
│   ├── staging/
│   └── prod/
├── patches/        # 配置补丁
└── resources/      # 静态资源

Kustomize与Helm区别

设计理念

Kustomize设计理念：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# 声明式配置管理
# 不使用模板，而是通过叠加（Overlay）方式修改配置

# Base: 基础配置
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: app
          image: my-app:latest

# Overlay: 环境差异
# overlays/production/kustomization.yaml
bases:
  - ../../base
patchesStrategicMerge:
  - |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-app
    spec:
      replicas: 10

Helm设计理念：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# 模板化配置管理
# 使用Go Template模板

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
        - name: app
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

# values.yaml (不同环境不同values)
replicaCount: 1
image:
  repository: my-app
  tag: latest

# values-production.yaml
replicaCount: 10
image:
  repository: my-app
  tag: v1.2.3

功能对比

功能 Kustomize Helm
模板 无模板，纯YAML Go Template
学习曲线 低，只需会YAML 中高，需学模板语法
复用性 Base + Overlay Chart + Values
调试 kubectl kustomize helm template –dry-run
版本管理 Git版本控制 Chart版本号
环境管理 Overlay目录 多个Values文件
包分发 无（用Git） Chart仓库
CI/CD集成 原生kubectl 需安装helm
Rollback kubectl rollout helm rollback
依赖管理 无直接支持 Chart dependencies

适用场景

选择Kustomize的场景：

  • 应用开发者管理自己的K8s资源
  • 需要GitOps工作流
  • 多环境配置差异大
  • 不想学习模板语言
  • 团队熟悉YAML和kubectl

选择Helm的场景：

  • 应用分发者（软件供应商）
  • 需要包管理和版本控制
  • 配置相对固定
  • 复杂的应用依赖关系
  • 快速部署第三方应用

Kustomize部署K8s应用

项目结构

my-app/
├── base/                           # 基础配置
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap.yaml
│   ├── serviceaccount.yaml
│   └── kustomization.yaml
├── overlays/                       # 环境配置
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   ├── deployment-replica.yaml
│   │   ├── deployment-resources.yaml
│   │   └── configmap-env.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── deployment-replica.yaml
│   └── production/
│       ├── kustomization.yaml
│       ├── deployment-replica.yaml
│       ├── deployment-resources.yaml
│       └── configmap-prod.yaml
└── scripts/
    └── deploy.sh

Base配置创建

base/deployment.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
    version: v1.0.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: my-app
        version: v1.0.0
    spec:
      serviceAccountName: my-app
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: my-app
          image: my-app:latest
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          env:
            - name: JAVA_OPTS
              value: "-Xms512m -Xmx512m"
            - name: SPRING_PROFILES_ACTIVE
              value: "prod"
          resources:
            requests:
              cpu: 100m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - name: config
              mountPath: /config
            - name: logs
              mountPath: /logs
      volumes:
        - name: config
          configMap:
            name: my-app-config
        - name: logs
          emptyDir: {}

base/service.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: my-app

base/configmap.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-config
data:
  application.yml: |
    server:
      port: 8080
    spring:
      application:
        name: my-app
    logging:
      level:
        root: INFO

base/serviceaccount.yaml

1
2
3
4

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app

base/kustomization.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# 引用的资源文件
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml
  - serviceaccount.yaml

# 通用标签（所有资源添加）
commonLabels:
  app.kubernetes.io/name: my-app
  app.kubernetes.io/instance: my-app
  app.kubernetes.io/managed-by: kustomize
  app.kubernetes.io/version: v1.0.0

# 通用注解
commonAnnotations:
  createdBy: kustomize
  env: base

# 名称前缀
# namePrefix: dev-

# 名称后缀
# nameSuffix: -v1

Overlay环境配置

overlays/production/kustomization.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# 引用base
namespace: production
bases:
  - ../../base

# 副本数覆盖
replicas:
  - name: my-app
    count: 10

# 镜像覆盖
images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

# ConfigMap生成器
configMapGenerator:
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml

# Secret生成器
secretGenerator:
  - name: my-app-secret
    behavior: create
    envs:
      - .env.production

# 资源补丁
patchesStrategicMerge:
  - deployment-resources.yaml
  - deployment-probes.yaml
  - pvc.yaml

# 命名空间
namespace: production

# 通用标签覆盖
commonLabels:
  environment: production
  tier: backend

# 通用注解覆盖
commonAnnotations:
  env: production
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

overlays/production/deployment-resources.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: my-app
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: 2000m
              memory: 4Gi

overlays/production/pvc.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 50Gi

overlays/production/application-production.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

server:
  port: 8080

spring:
  application:
    name: my-app
  datasource:
    url: jdbc:mysql://mysql-prod.svc:3306/mydb
    username: prod_user
    password: ${DB_PASSWORD}

logging:
  level:
    root: INFO
    com.example: DEBUG
  file:
    path: /logs

overlays/production/.env.production

1
2
3

DB_PASSWORD=prod_secure_password_123
API_KEY=prod_api_key_xyz
REDIS_PASSWORD=prod_redis_pass

部署命令

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# 构建配置（不应用，查看最终YAML）
kubectl kustomize overlays/production

# 保存渲染结果
kubectl kustomize overlays/production > rendered-prod.yaml

# 查看差异
kubectl diff -k overlays/production

# 直接应用
kubectl apply -k overlays/production

# 删除资源
kubectl delete -k overlays/production

# 查看资源
kubectl get all -l app.kubernetes.io/name=my-app -n production

统一部署管理配置

1. Namespace管理

方式一：在Overlay中指定

1
2

# overlays/production/kustomization.yaml
namespace: production

方式二：作为资源管理

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - namespace.yaml

# overlays/production/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    environment: production
    name: production

2. Secret管理

方案一：环境变量文件

1
2
3
4
5
6

# overlays/production/kustomization.yaml
secretGenerator:
  - name: my-app-secret
    type: Opaque
    envs:
      - .env.production

方案二：字面值

1
2
3
4
5
6

secretGenerator:
  - name: my-app-secret
    type: Opaque
    literals:
      - DB_PASSWORD=prod123
      - API_KEY=xyz789

方案三：命令生成

1
2
3
4
5
6

secretGenerator:
  - name: tls-cert
    type: kubernetes.io/tls
    files:
      - tls.crt=tls.crt
      - tls.key=tls.key

方案四：从现有Secret引用

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# overlays/production/kustomization.yaml
generatorOptions:
  disableNameSuffixHash: true

resources:
  - secret.yaml

# overlays/production/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
type: Opaque
stringData:
  DB_PASSWORD: prod123

方案五：Sealed Secrets（推荐生产使用）

1
2
3
4
5
6

# 本地生成sealed secret
kubeseal -f secret.yaml -w sealed-secret.yaml

# 在kustomization中引用
resources:
  - sealed-secret.yaml

3. PVC管理

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# overlays/production/kustomization.yaml
resources:
  - ../../base
  - pvc-logs.yaml
  - pvc-data.yaml

# overlays/production/pvc-logs.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-logs
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd  # 使用SSD存储类
  resources:
    requests:
      storage: 50Gi

# overlays/production/pvc-data.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
spec:
  accessModes:
    - ReadWriteMany  # 多节点共享
  storageClassName: nfs-storage
  resources:
    requests:
      storage: 100Gi

4. 镜像版本管理

方式一：直接指定

1
2
3
4

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

方式二：使用SHA256

1
2
3
4

images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    digest: sha256:abc123def456...  # 最安全的方式

方式三：多个镜像

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

images:
  - name: nginx
    newName: harbor.example.com/library/nginx
    newTag: 1.21-alpine
  - name: redis
    newName: harbor.example.com/library/redis
    newTag: 6.2-alpine
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

方式四：CI/CD动态更新

1
2
3
4
5

# 在CI Pipeline中
export IMAGE_TAG=$(git rev-parse --short HEAD)
cd overlays/production
kustomize edit set image my-app=harbor.example.com/prod/my-app:$IMAGE_TAG
kubectl apply -k .

5. 节点亲和性配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

# overlays/production/deployment-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      affinity:
        # 节点亲和性
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - c5.2xlarge
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values:
                      - us-west-1a
        # Pod亲和性
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - mysql
                topologyKey: kubernetes.io/hostname

在kustomization中引用：

1
2

patchesStrategicMerge:
  - deployment-affinity.yaml

6. 污点和容忍配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# overlays/production/deployment-taints.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      # 容忍专用节点污点
      tolerations:
        - key: workload
          operator: Equal
          value: dedicated
          effect: NoSchedule
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 300
        # 容许GPU节点
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

7. 副本数量管理

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# 方式一：replicas字段（推荐）
replicas:
  - name: my-app
    count: 10
  - name: my-app-sidecar
    count: 2

# 方式二：Patch方式
# overlays/production/deployment-replica.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 10

8. ConfigMap管理

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# 方式一：configMapGenerator（推荐）
configMapGenerator:
  # 从文件生成
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=application-production.yml
      - logback.xml=logback-production.xml

  # 从字面值生成
  - name: my-app-env
    behavior: create
    literals:
      - SPRING_PROFILES_ACTIVE=production
      - LOG_LEVEL=INFO
      - JAVA_OPTS=-Xms2g -Xmx4g

  # 从env文件生成
  - name: my-app-file-env
    behavior: create
    envs:
      - .env.production

# 方式二：直接引用资源
resources:
  - configmap.yaml

完整生产环境示例

overlays/production/kustomization.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# 命名空间
namespace: production

# 引用base
bases:
  - ../../base

# 额外资源
resources:
  - namespace.yaml
  - pvc-logs.yaml
  - pvc-data.yaml
  - hpa.yaml
  - service-monitor.yaml
  - network-policy.yaml

# 副本数
replicas:
  - name: my-app
    count: 10

# 镜像版本
images:
  - name: my-app
    newName: harbor.example.com/prod/my-app
    newTag: v1.2.3

# ConfigMap
configMapGenerator:
  - name: my-app-config
    behavior: replace
    files:
      - application.yml=config/application-production.yml
  - name: my-app-env
    behavior: merge
    literals:
      - ENV=production
      - LOG_LEVEL=INFO

# Secret
secretGenerator:
  - name: my-app-secret
    behavior: create
    envs:
      - .env.production

# Patch
patchesStrategicMerge:
  - deployment-resources.yaml
  - deployment-affinity.yaml
  - deployment-taints.yaml
  - deployment-probes.yaml

# 通用标签
commonLabels:
  environment: production
  tier: backend
  team: platform

# 通用注解
commonAnnotations:
  env: production
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

CI/CD集成

GitLab CI示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

# .gitlab-ci.yml
stages:
  - build
  - deploy

variables:
  REGISTRY: harbor.example.com
  APP_NAME: my-app

build:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker build -t $REGISTRY/$APP_NAME:$CI_COMMIT_SHA .
    - docker push $REGISTRY/$APP_NAME:$CI_COMMIT_SHA
  only:
    - main
    - develop

deploy:dev:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: dev
  script:
    - cd overlays/dev
    - kustomize edit set image $REGISTRY/$APP_NAME=$REGISTRY/$APP_NAME:$CI_COMMIT_SHA
    - kubectl apply -k .
    - kubectl rollout status deployment/my-app -n dev
  only:
    - develop

deploy:prod:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: production
  script:
    - cd overlays/production
    - kustomize edit set image $REGISTRY/$APP_NAME=$REGISTRY/$APP_NAME:$CI_COMMIT_SHA
    - kubectl apply -k .
    - kubectl rollout status deployment/my-app -n production
  when: manual
  only:
    - main

ArgoCD集成

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# argocd/application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-production
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/example/k8s-manifests.git
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

总结

Kustomize最佳实践

  1. 目录结构规范

    • base存放通用配置
    • overlays管理环境差异
    • 清晰的命名约定

  2. 版本管理

    • 使用Git管理所有配置
    • 镜像使用SHA256或明确的tag
    • 不要使用latest

  3. Secret管理

    • 生产环境使用Sealed Secrets
    • 不要将敏感信息提交到Git
    • 使用External Secrets Operator集成Vault

  4. 组件复用

    • 抽取通用配置为组件
    • 建立Kustomize组件库
    • 避免重复配置

  5. CI/CD集成

    • 自动化镜像tag更新
    • 使用kubectl diff预览变更
    • 配置验证通过后再应用

Kustomize vs Helm选择

场景 推荐方案 原因
微服务配置管理 Kustomize 灵活性高，Git友好
第三方应用部署 Helm 生态丰富，一键部署
复杂模板需求 Helm Go Template功能强大
简单配置 Kustomize 无需学习模板
GitOps Kustomize 天然支持

学习路径

  1. 基础语法：base + overlay结构
  2. Patch策略：StrategicMerge vs JSON6902
  3. 组件复用：提取通用配置
  4. CI/CD集成：自动化部署流程
  5. GitOps实践：与ArgoCD/Flux结合

Kustomize是云原生时代配置管理的理想工具，掌握它能显著提升Kubernetes应用管理效率，在实际部署场景下，对于新项目不同环境差异化很小，可以结合VibeCoding工具生成一键部署脚本，快速交付新环境。