Jenkins on Wanzi's Blog – Notes on life, tech, and DevOps.

Customize Jenkins Agent Integration with Docker and kubectl Tools

wzdushu@gmail.com (wanzi) — Wed, 22 Mar 2023 18:22:42 +0800

Since the official Jenkins agent image does not include the tools we need—such as Helm, kubectl, curl, and ArgoCD—during Jenkins builds, we must integrate these tools ourselves.

Note: The official image names have changed:

jenkins/agent (formerly jenkins/slave) — renamed starting from version 4.3-2

jenkins/inbound-agent (formerly jenkins/jnlp-slave) — renamed starting from version 4.3-2

Dockerfile

FROM jenkins/inbound-agent:4.11-1-alpine-jdk11

USER root

ADD docker/docker /usr/bin/docker
ADD kubectl /usr/bin/kubectl
ADD helm /usr/bin/helm

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN chmod +x /usr/bin/docker /usr/bin/kubectl /usr/bin/helm
RUN apk add curl

ENTRYPOINT ["/usr/local/bin/jenkins-agent"]

Build

`1`	`docker build -t harbor.test.com/tools/jnlp-docker:4.11-1-alpine-jdk11 -f Dockerfile .`

Jenkins Workspace and Local Persistent Storage for Maven Repository

wzdushu@gmail.com (wanzi) — Wed, 22 Mar 2023 16:22:42 +0800

Create Local Storage

Since this is a test cluster, we can directly use local volume storage here.

As the Jenkins server runs under the jenkins user with UID 1000, we need to pre-assign permissions for /opt/jenkins_agent/ and /opt/jenkins_maven/ on node01:

1
2

chown 1000.1000 -R /opt/jenkins_agent/
chown 1000.1000 -R /opt/jenkins_maven/

Local Storage: agent-pv-pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-agent-pv
spec:
  storageClassName: local # Local PV
  capacity:
    storage: 30Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  local:
    path: /opt/jenkins_agent
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - node01
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-agent-pvc
  namespace: kube-ops
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

Local Storage: maven-pv-pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-maven-pv
spec:
  storageClassName: local # Local PV
  capacity:
    storage: 30Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  local:
    path: /opt/jenkins_maven
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - node01
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-maven-pvc
  namespace: kube-ops
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

Create PV and PVC Resources

# kubectl apply -f agent-pv-pvc.yaml
persistentvolume/jenkins-agent-pv created
persistentvolumeclaim/jenkins-agent-pvc created

# kubectl apply -f maven-pv-pvc.yaml
persistentvolume/jenkins-maven-pv created
persistentvolumeclaim/jenkins-maven-pvc created

Deploying Jenkins in a Kubernetes Environment

wzdushu@gmail.com (wanzi) — Thu, 16 Mar 2023 16:22:42 +0800

1. Install and Deploy Jenkins

1. Manual Installation

Manual installation is straightforward—just prepare the YAML configuration in advance. The complete CICD resource file jenkins-install.yaml is as follows:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  storageClassName: local # Local PV
  capacity:
    storage: 30Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  local:
    path: /opt/jenkins
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - node02
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: kube-ops
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments", "ingresses"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["pods/log", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: kube-ops
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: kube-ops
spec:
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccount: jenkins
      initContainers:
        - name: fix-permissions
          image: busybox
          command: ["sh", "-c", "chown -R 1000:1000 /var/jenkins_home"]
          securityContext:
            privileged: true
          volumeMounts:
            - name: jenkinshome
              mountPath: /var/jenkins_home
      containers:
        - name: jenkins
          #image: jenkins/jenkins:lts
          image: jenkins/jenkins:2.375.3-lts # Use latest stable version
          imagePullPolicy: IfNotPresent
          env:
          - name: JAVA_OPTS
            value: -Dhudson.model.DownloadService.noSignatureCheck=true
          ports:
            - containerPort: 8080
              name: web
              protocol: TCP
            - containerPort: 50000
              name: agent
              protocol: TCP
          resources:
            limits:
              cpu: 1500m
              memory: 4096Mi
            requests:
              cpu: 1500m
              memory: 2048Mi
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          volumeMounts:
            - name: jenkinshome
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkinshome
          persistentVolumeClaim:
            claimName: jenkins-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: kube-ops
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  ports:
    - name: web
      port: 8080
      targetPort: web
    - name: agent
      port: 50000
      targetPort: agent
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  namespace: kube-ops
spec:
  ingressClassName: traefik
  rules:
  - host: jenkins.test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: jenkins
            port:
              number: 8080

Apply the Jenkins installation:

`1`	`kubectl apply -f jenkins-install.yaml`

This installation involves the following resources:

Local storage: using hostPath
RBAC permissions: ServiceAccount, ClusterRole, ClusterRoleBinding
Deployment and services: Deployment, Service, Ingress

2. Configure Jenkins

Note: Since Jenkins performs signature verification on update-center.json, you must disable this check beforehand to avoid failures when changing plugin sources. Set the environment variable JAVA_OPTS=-Dhudson.model.DownloadService.noSignatureCheck=true.

After installing Jenkins, open https://jenkins.test.com and enter the admin password. Retrieve it via:

1
2

kubectl exec -it jenkins-d995d6d4-srfr8 -n kube-ops -- cat /var/jenkins_home/secrets/initialAdminPassword
a571eb7a8ac54256ae638bb4a61fd31b

Skip plugin installation initially. Choosing default plugins will be very slow (you may opt for recommended plugins instead). Click the close button in the top-right corner to skip plugin selection. After configuring the domestic mirror source, proceed with installing essential plugins.

Go to: https://jenkins.test.com/chinese/ and configure the domestic mirror source. I use Alibaba Cloud’s mirror: https://mirrors.aliyun.com/jenkins/updates/update-center.json

3. Install Plugins

Visit: https://jenkins.test.com/manage/pluginManager/available

I mainly install the following plugins: Kubernetes, Chinese language pack, Pipeline, etc. Specific steps are omitted here.

Jenkins Localization (Chinese): https://plugins.jenkins.io/localization-zh-cn/
Build With Parameters: https://plugins.jenkins.io/build-with-parameters
Kubernetes: https://plugins.jenkins.io/kubernetes/
Pipeline: https://plugins.jenkins.io/workflow-aggregator/
Git: https://plugins.jenkins.io/git/
zentimestamp: https://plugins.jenkins.io/zentimestamp/

2. Connect to Kubernetes Cluster

1. Configure Kubernetes Cluster Info

Go to: https://jenkins.test.com/configureClouds/

Create a new cloud, select Kubernetes, and configure the cluster address and namespace:

2. Configure Jenkins Access Address

Set the Jenkins URL used within the Kubernetes cluster, including the Jenkins agent channel address:

3. Pod Template Configuration

Configure namespace and labels:

4. Container Template Configuration

Define slave build container settings. Note that the container name must be jnlp.

The Docker image I use is self-built, pre-integrated with docker, kubectl, and other tools. Leave the Command and Arguments fields blank; otherwise, container creation will fail.

Configure volumes:

Mount Docker socket to enable Docker-in-Docker (sharing host Docker).
Mount kubeconfig to allow kubectl, helm, and other commands to manage Kubernetes resources.

Finally, set the Service Account to jenkins, which matches the account created earlier.

3. Simple Test

1. Create a Freestyle Job

2. Bind Slave Pod

3. Configure Shell Test Command

4. Execute a Build Task

After execution, check the build history. You’ll observe that a slave Pod is automatically created during the process and then automatically deleted after completion.

4. Summary

Continuous integration and delivery (CI/CD) are essential components of modern development workflows. Most companies currently deploy Jenkins clusters to build CI/CD environments. However, traditional Jenkins Slave architectures (one master, multiple slaves) face several challenges:

Single point of failure: If the Master fails, the entire pipeline becomes unavailable.
Inconsistent slave configurations: Different languages require different build environments, leading to complex and hard-to-manage configurations.
Uneven resource distribution: Some Slaves are overloaded while others remain idle, causing job queuing.
Resource waste: Slaves often run on physical or virtual machines, consuming resources even when idle.

These issues drive the need for a more efficient and reliable solution. Containerization technologies like Docker, especially when combined with Kubernetes, effectively address these pain points. Below is a simplified diagram of a Jenkins cluster built on Kubernetes:

As shown in the diagram, both Jenkins Master and Slave run as Pods on Kubernetes nodes. The Master runs on one node and stores its configuration data in a Volume. Slaves are dynamically created and destroyed based on demand, rather than running continuously.

The workflow is as follows: When the Jenkins Master receives a build request, it dynamically creates a Jenkins Slave Pod based on the configured label, registers it with the Master, executes the job, then deregisters and deletes the Pod upon completion—returning to the original state.

Benefits of this approach include:

High availability: If the Jenkins Master fails, Kubernetes automatically creates a new container and assigns the Volume, preserving data and ensuring service continuity.
Dynamic scaling: Resources are allocated only when needed. After each job completes, the slave is deregistered and the Pod is deleted, freeing up resources. Kubernetes intelligently schedules slaves onto idle nodes, reducing bottlenecks caused by high utilization on specific nodes.
Excellent scalability: When cluster resources become insufficient and jobs queue up, simply add a new Kubernetes Node to scale horizontally.

Are all the previous problems solved in a Kubernetes environment? It appears nearly perfect.

Ultimately, deploying CI/CD using Jenkins + Kubernetes offers clear advantages in both workflow efficiency and cost optimization.

Setting up Jenkins using Docker Compose

wzdushu@gmail.com (wanzi) — Mon, 11 Nov 2019 16:22:42 +0800

docker-compose Configuration

version: '2'
 
services:
  jenkins:
    image: jenkins/jenkins:latest
    restart: always
    environment:
      JAVA_OPTS: "-Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Shanghai -Djava.awt.headless=true -Dmail.smtp.starttls.enable=true"
    ports:
      - "80:8080"
      - "50000:50000"
    volumes:
      - '/ssd/jenkins:/var/jenkins_home'
      - '/var/run/docker.sock:/var/run/docker.sock'
      - '/etc/localtime:/etc/localtime:ro'
    dns: 223.5.5.5
    networks:
      - extnetwork
networks:
   extnetwork:
      ipam:
         config:
         - subnet: 172.255.0.0/16

Start Services

`1`	`docker-compose up -d`