Docker on Wanzi's Blog – Notes on life, tech, and DevOps.

kind: Build a lightweight Kubernetes cluster

wzdushu@gmail.com (wanzi) — Thu, 13 Apr 2023 18:22:42 +0800

Recently revisiting Golang, I built a web application and successfully ran it locally. Now, I wanted to test it in a Kubernetes cluster, but due to hardware limitations, setting up a full-fledged K8s cluster was challenging. I remembered a friend mentioning that Kubernetes can also run inside Docker—so I decided to give it a try.

Today’s focus is kind. What is kind? And what can it do?

1. Introduction to kind

kind stands for Kubernetes In Docker — a tool that uses Docker container nodes to run local Kubernetes clusters. It’s primarily designed for testing Kubernetes itself and is ideal for local development or CI pipelines.

Currently, kind supports almost all official Kubernetes versions. At present, it only supports Docker as the runtime, but future versions will gradually add support for common CRI runtimes like containerd. In the near future, support for containerd is expected to be included.

Under the hood, kind leverages open-source tools such as kubeadm and kustomize to manage cluster creation and configuration.

Enough talk — let’s get started with installing and using kind.

2. Installing kind

Since I’m on macOS, I’ll use brew:

# brew install kind
==> Downloading https://mirrors.ustc.edu.cn/homebrew-bottles/kind-0.17.0.arm64_monterey.bottle.tar.gz
Already downloaded: /Users/wanzi/Library/Caches/Homebrew/downloads/bcd419997297730492f5cebc36be86fb51f21061a6ddb2e066e1e4d8ad33ddf3--kind-0.17.0.arm64_monterey.bottle.tar.gz
==> Pouring kind-0.17.0.arm64_monterey.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions
==> Summary
🍺  /opt/homebrew/Cellar/kind/0.17.0: 8 files, 8.7MB
==> Running `brew cleanup kind`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
# kind version
kind v0.17.0 go1.19.2 darwin/arm64

For Linux:

1
2
3

curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.17.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

If you’re on Windows, you can use Chocolatey or install via binary — both are officially supported.

3. Creating Your First Cluster

Before proceeding, ensure Docker and kubectl are installed (installation steps omitted here).

# kind help
kind creates and manages local Kubernetes clusters using Docker container 'nodes'

Usage:
  kind [command]

Available Commands:
  build       Build one of [node-image]
  completion  Output shell completion code for the specified shell (bash, zsh or fish)
  create      Creates one of [cluster]
  delete      Deletes one of [cluster]
  export      Exports one of [kubeconfig, logs]
  get         Gets one of [clusters, nodes, kubeconfig]
  help        Help about any command
  load        Loads images into nodes
  version     Prints the kind CLI version

Flags:
  -h, --help              help for kind
      --loglevel string   DEPRECATED: see -v instead
  -q, --quiet             silence all stderr output
  -v, --verbosity int32   info log verbosity, higher value produces more output

Use "kind [command] --help" for more information about a command.

From kind help, we see that kind supports commands like build, create, delete, get, load, etc. Use -h for detailed help on subcommands.

Now, create a cluster:

Ensure ~/.kube directory exists — kind writes cluster API address, certificates, and other config into kubeconfig during setup.

# kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

# kubectl get node
NAME                 STATUS   ROLES           AGE     VERSION
kind-control-plane   Ready    control-plane   9m21s   v1.25.3

# kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-565d847f94-kp6l7                     1/1     Running   0          9m15s
kube-system          coredns-565d847f94-ml28b                     1/1     Running   0          9m15s
kube-system          etcd-kind-control-plane                      1/1     Running   0          9m32s
kube-system          kindnet-hfsc8                                1/1     Running   0          9m15s
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          9m31s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          9m33s
kube-system          kube-proxy-m9zp5                             1/1     Running   0          9m15s
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          9m33s
local-path-storage   local-path-provisioner-684f458cdd-d7f29      1/1     Running   0          9m15s

Success! Your first kind cluster is up and running.

4. Cluster Operations

1. Custom Cluster Name

# kind create cluster --name ci-cluster
Creating cluster "ci-cluster" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-ci-cluster"
You can now use your cluster with:

2. Switch Context

# kubectl cluster-info --context kind-ci-cluster
Kubernetes control plane is running at https://127.0.0.1:56527
CoreDNS is running at https://127.0.0.1:56527/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Have a nice day! 👋

3. List All Clusters

1
2
3

# kind get clusters
ci-cluster
kind

4. Load Images into Nodes

By default, kind nodes cannot access host machine images — they must be manually loaded.

# kind load docker-image --name ci-cluster --nodes ci-cluster-control-plane traefik:v2.9.5
# docker exec -it ci-cluster-control-plane bash
root@ci-cluster-control-plane:/# crictl images | grep traefik
docker.io/library/traefik                  v2.9.5               a1252ce6bfaaa       132MB

5. Advanced Cluster Configuration

1. Multi-node Cluster

By default, kind creates a single-node cluster. Here’s how to define multiple nodes:

Create config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

Recreate the cluster:

# kind create cluster --config config.yaml
# kubectl get node
NAME                 STATUS   ROLES           AGE    VERSION
kind-control-plane   Ready    control-plane   110s   v1.25.3
kind-worker          Ready    <none>          74s    v1.25.3
kind-worker2         Ready    <none>          88s    v1.25.3

Note: Kind also supports multi-master HA setups — though not covered here for brevity.

2. Custom Kubernetes Version

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315

3. Mount Host Directory to Node Containers (Persistent Storage)

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/files
    containerPath: /files
  - hostPath: /Users/wanzi/tools/kind/other-files/
    containerPath: /other-files
    readOnly: true
    selinuxRelabel: false
    propagation: None

4. Port Mapping

Map host port 80 to nodePort 30950. The container port must match the Service’s nodePort.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30950
    hostPort: 80

And apply this service/pod example:

kind: Pod
apiVersion: v1
metadata:
  name: foo
  labels:
    app: foo
spec:
  containers:
  - name: foo
    image: hashicorp/http-echo:0.2.3
    args:
    - "-text=foo"
    ports:
    - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: foo
spec:
  type: NodePort
  ports:
  - name: http
    nodePort: 30950
    port: 5678
  selector:
    app: foo

5. Add Custom Labels

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  extraPortMappings:
  - containerPort: 30950
    hostPort: 80
  labels:
    tier: frontend
- role: worker
  labels:
    tier: backend

6. Customize kubeadm Configurations

Since kind uses kubeadm under the hood, you can customize the init process on the first control plane node:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label=true"

For deeper customization, four configuration types are available:

InitConfiguration
ClusterConfiguration
KubeProxyConfiguration
KubeletConfiguration

Example: Override API server flags using ClusterConfiguration:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      extraArgs:
        enable-admission-plugins: NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

For additional nodes joining the cluster, use JoinConfiguration:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label2=true"
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "my-label3=true"

7. Configure Ingress

Allow local requests to ingress controller via ports 80/443 and set specific labels:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP

Apply ingress-nginx:

`1`	`kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml`

6. Complete Cluster Configuration Example

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  labels:
    app: front
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/wwwroot
    containerPath: /wwwroot
- role: worker
  images: kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315
  labels:
    app: backend
  extraMounts:
  - hostPath: /Users/wanzi/tools/kind/wwwroot
    containerPath: /wwwroot

networking:
  apiServerAddress: "127.0.0.1"
  apiServerPort: 6443
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
  # disableDefaultCNI: true # Default CNI is kindnetd; can disable to use others like Calico
  kubeProxyMode: "ipvs" # Set kube-proxy mode to ipvs; use "none" to disable

That concludes our journey with kind. You now have a fast, lightweight way to spin up Kubernetes clusters for testing and feature validation.

References:

Customize Jenkins Agent Integration with Docker and kubectl Tools

wzdushu@gmail.com (wanzi) — Wed, 22 Mar 2023 18:22:42 +0800

Since the official Jenkins agent image does not include the tools we need—such as Helm, kubectl, curl, and ArgoCD—during Jenkins builds, we must integrate these tools ourselves.

Note: The official image names have changed:

jenkins/agent (formerly jenkins/slave) — renamed starting from version 4.3-2

jenkins/inbound-agent (formerly jenkins/jnlp-slave) — renamed starting from version 4.3-2

Dockerfile

FROM jenkins/inbound-agent:4.11-1-alpine-jdk11

USER root

ADD docker/docker /usr/bin/docker
ADD kubectl /usr/bin/kubectl
ADD helm /usr/bin/helm

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN chmod +x /usr/bin/docker /usr/bin/kubectl /usr/bin/helm
RUN apk add curl

ENTRYPOINT ["/usr/local/bin/jenkins-agent"]

Build

`1`	`docker build -t harbor.test.com/tools/jnlp-docker:4.11-1-alpine-jdk11 -f Dockerfile .`

Dockerfile Multi-stage Builds

wzdushu@gmail.com (wanzi) — Mon, 02 Jul 2018 16:22:42 +0800

Understanding Docker Multi-stage Builds:

Building an image requires a base image; all subsequent operations are based on this base image.
Docker image files have a layered structure. Each RUN instruction adds a new layer, so reducing the number of layers helps minimize image size.
When multiple FROM instructions are used, only the last FROM image becomes the root image of the final build.

Example of multi-stage build in my own project deployment: Here, we compile a binary using the Golang base image and then directly copy it into a minimal Alpine-based image:

FROM golang:1.12.7 as build
MAINTAINER wanzi <iwz2099@163.com>

# Compilation configuration
ARG NAME=gaia
ARG FLAGS=-tags=jsoniter
ARG GOOS=linux
ARG GOARCH=amd64
ARG PORT_TO_EXPOSE=10020

ENV GOPROXY https://mirrors.aliyun.com/goproxy/
ENV GO111MODULE on

WORKDIR /opt/gaia
COPY . .
RUN GOOS=$GOOS GOARCH=$GOARCH go build -mod vendor -ldflags="-s -w" -o $NAME $FLAGS

FROM alpine
WORKDIR /opt/gaia
COPY --from=build /opt/gaia/gaia .
RUN mkdir -p /opt/gaia/conf
VOLUME ["/opt/gaia/conf"]

CMD ["./gaia"]
EXPOSE $PORT_TO_EXPOSE

Detailed Dockerfile syntax

wzdushu@gmail.com (wanzi) — Thu, 21 Jun 2018 16:22:42 +0800

FROM

Specifies the base image used to build the Docker image. FROM must be the first non-comment instruction in a Dockerfile. If the specified image is not available locally, Docker will automatically pull it from the public registry.

Example:

`1`	`FROM ubuntu:14.04 # Inherits from ubuntu:14.04`

MAINTAINER

Specifies the maintainer information.

`1`	`MAINTAINER wanzi "iwz2099@163.com"`

ENV

Sets environment variables that are available for subsequent RUN instructions and persist at runtime within the container.

ENV <key> <value> — sets a single variable
ENV <key>=<value> ... — allows setting multiple variables at once

Example:

1
2

ENV LANG en_US.UTF-8
ENV LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

RUN

Executes commands during image build time.

RUN <command>
RUN ["executable", "param1", "param2"]

Example:

1
2

RUN yum -y install bind-utils
RUN ["/bin/bash", "-c", "yum -y install bind-utils"]

The first form runs the command in a shell (/bin/sh -c), while the second uses exec execution. Use the second form to specify a different shell or interpreter.

Each RUN instruction executes a command on top of the current image layer, commits the result as a new layer, and subsequent RUN commands operate on the resulting image. Docker images are layered—any historical commit point can be used to create a new image, similar to version control in source code. Long commands can be split across lines using \.

COPY

Copies files or directories from the host filesystem into the container.

COPY <src> <dest>

Example:

`1`	`COPY script/ /build/script/`

Copies <src> (a relative path from the Dockerfile’s location) to <dest> inside the container. When copying local directories, prefer COPY over ADD.

ADD

Copies files or directories from the host filesystem into the container.

ADD <src> <dest>

Example:

1
2

ADD https://www.baidu.com/index.html   /var/www/html/
ADD ubuntu-xenial-core-cloudimg-amd64-root.tar.gz /

The <src> can be:

A relative path from the Dockerfile’s directory,
A URL (automatically downloaded and copied),
A local tar file (automatically extracted into a directory).

VOLUME

Declares a mount point for data volumes.

1
2

VOLUME [ "/data" ]
VOLUME [ "/var/lib/redis", "/var/log/redis" ]

Creates a mountable directory that can be mounted from the host or other containers. Typically used for databases or persistent data.

USER

Sets the user ID (UID) under which the container runs. This affects the user context for RUN commands.

USER <uid>

CMD

Supports three formats:

CMD ["executable","param1","param2"] — executes via exec, recommended.
CMD command param1 param2 — runs in /bin/sh, suitable for interactive applications.
CMD ["param1","param2"] — provides default parameters for ENTRYPOINT.

Specifies the default command to run when starting a container. Only one CMD instruction is allowed per Dockerfile; if multiple are defined, only the last one takes effect.

WORKDIR

Sets the working directory for RUN, CMD, and ENTRYPOINT instructions.

Example:

`1`	`WORKDIR /opt/nodeapp`

ONBUILD

Configures instructions to be executed when the built image is used as a base for another image.

ONBUILD [INSTRUCTION]

Example: A Dockerfile creating image-A contains:

1
2

ONBUILD ADD . /app/src
ONBUILD RUN /usr/local/bin/python-build --dir /app/src

When a new image is built using FROM image-A, these ONBUILD instructions are automatically added:

FROM image-A
# Automatically runs:
ADD . /app/src
RUN /usr/local/bin/python-build --dir /app/src

Images with ONBUILD are recommended to include this in their tag (e.g., ruby:1.9-onbuild).

ENTRYPOINT

Defines the default executable to run when the container starts. Unlike CMD, it cannot be overridden by command-line arguments unless explicitly changed via docker run --entrypoint.

Only one ENTRYPOINT instruction is allowed per Dockerfile; later ones override earlier ones.

Supported formats:

ENTRYPOINT [ "nodejs", "server.js" ]
ENTRYPOINT command param1 param2 — runs in shell

EXPOSE

Informs Docker that the container listens on a specific port at runtime.

EXPOSE <port>

When starting the container, use -P to let Docker assign a random port mapping automatically.

Example:

`1`	`EXPOSE 3000`

Notes & Best Practices

Keep the image minimal—avoid installing unnecessary packages.
Create a .dockerignore file to exclude unwanted files/directories (syntax similar to .gitignore).
Prefer official Docker Hub images as base layers to reduce image size.
Fix the initial instructions in the Dockerfile—avoid frequent changes to leverage caching effectively.
Use \ to join multiple RUN commands for better readability and maintainability.
Prefer COPY over ADD whenever possible.
Use the -t flag when building images for easier management.
Do not map public ports directly in the Dockerfile.
Always test the image locally before pushing it to a registry.

Docker Basic Commands

wzdushu@gmail.com (wanzi) — Wed, 20 Jun 2018 16:22:42 +0800

Common Commands

docker info     # View local Docker information
docker search openresty   # Search remote image repository
docker images      # View images in the local image repository
docker ps          # View currently running containers
docker pull centos  # Pull an image from the remote repository (default tag is 'latest' if not specified)
docker container run -p 8000:80 --rm -t -i centos:latest /bin/bash  # Map container port 80 to host port 8000; --rm removes the container after termination (useful for temporary debugging); -t allocates a pseudo-terminal and binds it to the container's stdin; -i keeps the container's stdin open
docker container exec -i -t b5d7bad57561 /bin/bash # Enter a running container
docker rmi $(docker images -f "dangling=true" -q) # Batch remove dangling (unnamed) images

Importing and Exporting Images

1
2
3

docker save -o centos7.tar centos  # Export image to tar file
docker load < centos7.tar   # Import image (via standard input, including original metadata such as tags)
docker load --input centos7.tar   # Load image from tar file (non-standard input method)

Importing and Exporting Containers

docker ps   # Default: show currently running containers
docker ps -a   # Show all containers, including stopped ones
docker ps -l   # Show the most recently created container (including stopped ones)

docker export 5a80afa126ba > centos7.5a80afa126ba.tar  # Export container snapshot
docker import 5a80afa126ba > centos7.5a80afa126ba.tar  # Import container snapshot into image repository

Note: Users can use docker load to import image files into the local image library, or docker import to import container snapshots. The key difference is that container snapshot files discard all historical records and metadata (preserving only the state at the time of the snapshot), while image storage files preserve full history and are larger in size. Additionally, when importing from a container snapshot, you can reassign metadata like tags.

Removing Images or Containers

docker rmi jdeathe/centos-ssh # Remove local image
or
docker image rm jdeathe/centos-ssh # Remove local image
docker rm dbd4d83097f5    # Remove local container
or
docker container rm dbd4d83097f5 # Remove local container

Stopping and Restarting Containers

1
2

docker container kill dbd4d83097f5  # Stop a running container
docker container restart dbd4d83097f5  # Restart a container

Building Images

1
2

docker image build .      # Build image from Dockerfile in current directory
docker image build -t cm-navigation:v0.0.1 .  # Build image from Dockerfile in current directory and assign a tag