Cicd on Wanzi's Blog – Notes on life, tech, and DevOps.

Getting Started with Github Actions

wzdushu@gmail.com (wanzi) — Fri, 01 Nov 2024 17:22:42 +0800

What is Github Action?

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that lets you automate your build, test, and deployment processes. You can create workflows to build and test every pull request to your repository, or deploy merged pull requests to production.

GitHub Actions is more than just DevOps; it lets you run workflows when other events happen in your repository. For example, you can run a workflow to automatically add appropriate labels when someone creates a new issue in your repository.

GitHub offers Linux, Windows, and macOS virtual machines to run your workflows, or you can host your own self-hosted runners in your own data center or cloud infrastructure.

CICD tool selection

Currently, the CICD market is really flourishing. In addition to github actions, there are many other CICD tools to choose from, such as: Gitlab CI, Jenkins, Tekton focusing on cloud native, Bitbucket Pipelines, AWS CodePipeline, Bitbucket Pipelines, Azure DevOps, Alibaba Cloud Effect, etc.

Which method to choose to implement your business automation pipeline depends on your actual situation. The following recommendations start from the following aspects:

Cost, whether to choose an open source solution or a paid commercial version. If you put cost first, you need to consider the cost of building an open source software server and the licensing cost of paid software.
Ease of use and learning path, mainly consider whether the system interface is beautiful and whether the technical documentation is complete. The interface is difficult to judge, and the technical documentation can check whether the official documentation is detailed and whether the workflow uses yaml or a new language.
Scalability and flexibility, mainly consider whether the system plug-ins and custom requirements can be met. This also depends on personal needs. For example, some companies need CICD and jira to be connected, then it is better to choose Bitbucket Pipelines.
Integration capability, mainly consider whether it is integrated with the current code repository, test framework, deployment environment and the degree of API function perfection. For example, if your code repository is on Github, of course, it is better to choose github actions.
Security, mainly examine whether there are security features such as access control and data encryption. This is selected according to your actual needs.
Performance, mainly examine the performance bottleneck of large-scale deployment. Often this needs to be compared with cost or resource requirements. For example, if you have strict requirements for file transfer and bandwidth in daily construction, using github actions is not the best choice.
Enterprise-level solutions, mainly examine high concurrency, multi-tenancy, security compliance, etc. Often, for enterprises of a certain scale, there will be many customized requirements. See if the tool has enterprise solutions.
Cloud native support, mainly examine the degree of support for the business under kubernetes, especially the new version, in which Tekton has more advantages.
Community and ecology, mainly examine which users of the project or product are using it, and the community activity. In this regard, jenkins, gitlab, and github are relatively more advantageous.

Github Actions Components

1. workflow

From the literal meaning, we can see that workflow is a kind of work flow, and here it is mainly used to configure the CICD automation process of github actions. It is usually defined using yaml files. After the definition is completed, the workflow can be triggered automatically or manually.

The working directory of workflow is generally under the .github/workflow directory of the repository. A repository can have multiple workflows, and each workflow can perform a different set of tasks. The tasks here are also commonly referred to as job tasks. For example:

Build and test pull requests.
Deploy your application every time you release it.
Add tags every time you open a new issue.

Of course, workflows can also reference each other.

In actual use, we usually use workflow templates to quickly write workflows, and GitHub also provides workflow templates for various scenarios, such as

CI: continuous integration workflow
Deployment: deployment workflow
Automation: automation workflow
Code scanning: code scanning workflow
github pages: pages workflow

When writing workflows, you need to pay attention to the meaning of common keywords. Below I have briefly sorted them out for easy use later:

name: define the workflow name
run-name: define the workflow runtime name
on: define the trigger, which can be one event or multiple events or triggered based on matching
env: define workflow global variables
jobs: define workflow tasks
permissions: define workflow permissions for warehouses
default: define default configuration, which can be used globally or in a job, such as defining shells or working directories, etc.
on.schedule: define workflow time schedule
on.workflow_call: define information transmission during the input and output of different workflows
on.workflow_dispatch: define a specific workflow, enter some tags to trigger
jobs.<job_id>.name: define the task name of a single job
jobs.<job_id>.runs-on: select the jobs running environment, the default is github official, you can also choose a high-performance machine or host your own host
jobs.<job_id>.steps: define different running stages of a single job, for example, a job has checkout code and build
jobs.<job_id>.env: define a single job specific variable, belongs to the map type, and can also be set globally.
jobs.<job_id>.environment: define a reference in a single job, the content can be a variable or context expression
jobs.<job_id>.uses: defines which action to use. Sometimes, when deploying some complex scenarios, you need to use some open source actions.
jobs.<job_id>.with: specifies parameters. Generally, when building a task, you need to specify a version.
jobs.<job_id>.needs: generally, when there are multiple jobs, handle the dependencies between jobs.
jobs.<job_id>.if: specifies the job running conditions.
jobs.<job_id>.container: specifies the container service in a single job. Note that there are requirements for the runner’s operating system, such as Ubuntu.
jobs.<job_id>.outputs: defines the output of the job.

2. Event triggers

In the previous step, we know the workflow, so how to trigger the workflow? At this time, you need to define an event trigger, such as the common event trigger method:

Triggered by code repository changes, such as when pushed to the default branch of the code
Occurs outside GitHub and triggers the repository_dispatch event
Scheduled time
Manual trigger

Workflow triggers are defined using on, and there are several common configurations:

Set a trigger for an event, which is triggered when pushed to any branch in the workflow repository:

`1`	`on: push`

Set a trigger for multiple events, which is triggered when pushed to any branch in the workflow repository:

`1`	`on: [push, fork]`

When there are multiple branches, you need to filter specific branches, such as main or releases related branches, and trigger when there is a push:

on:
push:
branches:
- main
- 'releases/**'

GitHub itself can also match issues to trigger

on:
issues:
types:
- opened
- labeled

3. Jobs

Generally, a workflow consists of one or more jobs, which run in parallel by default. Note that jobs.<job_id> is the unique identifier of the job.

For example, we define a job named Docker build, and for this job, build is the unique identifier of the job.

jobs:
build:
name: Docker build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

In addition, dependencies can also be defined between jobs, defined by the jobs.<job_id>.needs keyword, such as:

jobs:
job1:
job2:
needs: job1
job3:
needs: [job1, job2]

In this example, job1 must be successfully completed before job2 starts, and job3 waits for job1 and job2 to complete.

4. Actions

Action is a custom application on Github that is used to perform complex but often repeated tasks. Using actions helps reduce the amount of repetitive code you write in your workflow files.

Actions can pull your Git repository from GitHub, set up the correct toolchain for your build environment, or set up authentication for your cloud provider.

Generally defined by the keyword uses, such as using the official: actions/checkout@v4

jobs:
deploy:
name: Deploy
runs-on: ubuntu-latest
environment: production

steps:
- name: Checkout
uses: actions/checkout@v4

You can write your own actions or find actions used in workflows in the GitHub Marketplace.

5. Runners

Runners are servers that run workflows when they are triggered. Define which type of runners to use by using the keyword runs-on. Each runner can run one job at a time, which is indeed very similar to the runner in gitlab CI.

GitHub provides Ubuntu Linux, Microsoft Windows, and macOS related runners to run workflows. Each workflow runs in a new, newly configured virtual machine.

Of course, if the default runners cannot meet your business needs, such as requiring an ARM architecture virtual machine, hosting your own server to GitHub for management; or if the resources are insufficient, you can also apply for larger runners.

The following sample workflow has two jobs, named Run-npm-on-Ubuntu and Run-PSScriptAnalyzer-on-Windows. When this workflow is triggered, GitHub provisions a new virtual machine for each job.

The job named Run-npm-on-Ubuntu is executed on a Linux VM because the job runs-on: specifies ubuntu-latest. The job named Run-PSScriptAnalyzer-on-Windows is executed on a Windows VM because the job runs-on: specifies windows-latest.

Install-Module PSScriptAnalyzer -ErrorAction Stop
- name: Get list of rules
shell: pwsh
run: |
Get-ScriptAnalyzerRule

This GitHub Actions workflow file defines two jobs:

Run-npm-on-Ubuntu: Runs the npm help command on an Ubuntu virtual machine.
Run-PSScriptAnalyzer-on-Windows: Installs the PSScriptAnalyzer module on a Windows virtual machine and lists all available rules.

These two jobs show how to run specific commands and scripts on different operating systems. In this way, you can ensure consistency and compatibility in different environments.

Common examples of Github Actions

1. Front-end build deployment

For example, a template for deploying a front-end Nodejs project to the Azure cloud, Deploy Node.js to Azure Web App:

on:
push:
branches: [ "master" ]
workflow_dispatch: # Allows manual triggering of workflows

env:
AZURE_WEBAPP_NAME: your-app-name # set this to your application's name
AZURE_WEBAPP_PACKAGE_PATH: '.' # set this to the path to your web app project, defaults to the repository root
NODE_VERSION: '20.x' # set this to the node version to use

permissions:
contents: read

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up Node.js
uses: actions/setup-node@v4
with: node-version: ${{ env.NODE_VERSION }} cache: 'npm' - name: npm install, build, and test run: | npm install npm run build --if-present npm run test --if-present - name: Upload artifact for deployment job uses: actions/upload-artifact@v3 with: name: node-app path: . deploy: permissions: contents: none runs-on: ubuntu-latest needs: build environment: name: 'Development' url: ${{ steps.deploy-to-webapp.outputs.webapp-url }} steps: - name: Download artifact from build job uses: actions/download-artifact@v3 with: name: node-app - name: 'Deploy to Azure WebApp' id: deploy-to-webapp uses: azure/webapps-deploy@v2
with:
app-name: ${{ env.AZURE_WEBAPP_NAME }}
publish-profile: ${{ secrets.AZURE_WEBAPP_PUBLISH_PROFILE }}
package: ${{ env.AZURE_WEBAPP_PACKAGE_PATH }}

This GitHub Actions workflow file defines two main jobs: build and deploy. The build job is responsible for checking out the code, setting up the Node.js environment, installing dependencies, building and testing the project, and uploading the build results as artifacts. The deploy job downloads the artifacts from the build job and deploys them to the Azure Web App. The entire process automates the entire process from code submission to application deployment.

2. Golang multi-version

Example: The following is a go multi-version build example

name: Go

on: [push]

jobs:
build:

runs-on: ubuntu-latest
strategy:
matrix:
go-version: [ '1.19', '1.20', '1.21.x' ]

steps:
- uses: actions/checkout@v4
- name: Setup Go ${{ matrix.go-version }}
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
# You can test your matrix by printing the current Go version
- name: Display Go version
run: go version

This GitHub Actions workflow file mainly defines the build support for multi-version golang environments.

3. Container example

Example: Run Docker container service in a node with runners as ubuntu

name: CI
on:
push:
branches: [ main ]
jobs:
container-test-job:
runs-on: ubuntu-latest
container:
image: node:18
env:
NODE_ENV: development
ports:
- 80 # Map the container port 80 to the host port 80.
volumes:
- my_docker_volume:/volume_mount # Mount a Docker volume named my_docker_volume to the container's /volume_mount path
options: --cpus 1 # Limit the number of CPUs used by the container to 1 core
steps:
- name: Check for dockerenv file
run: (ls /.dockerenv && echo Found dockerenv) || (echo No dockerenv)

This GitHub Actions workflow file defines a CI job that runs on a recent Ubuntu VM and uses the Node.js 18 Docker image. The job configures environment variables, port mappings, volume mounts, and CPU limits. The only step is to check if a .dockerenv file exists to confirm that the current environment is a Docker container.

4. Deploy the application to AWS ECS

# This workflow will build and push a new container image to Amazon ECR,
# and then will deploy a new task definition to Amazon ECS, when there is a push to the "master" branch.
#
# To use this workflow, you will need to complete the following set-up steps:
#
# 1. Create an ECR repository to store your images.
#    For example: `aws ecr create-repository --repository-name my-ecr-repo --region us-east-2`.
#    Replace the value of the `ECR_REPOSITORY` environment variable in the workflow below with your repository's name.
#    Replace the value of the `AWS_REGION` environment variable in the workflow below with your repository's region.
#
# 2. Create an ECS task definition, an ECS cluster, and an ECS service.
#    For example, follow the Getting Started guide on the ECS console:
#      https://us-east-2.console.aws.amazon.com/ecs/home?region=us-east-2#/firstRun
#    Replace the value of the `ECS_SERVICE` environment variable in the workflow below with the name you set for the Amazon ECS service.
#    Replace the value of the `ECS_CLUSTER` environment variable in the workflow below with the name you set for the cluster.
#
# 3. Store your ECS task definition as a JSON file in your repository.
#    The format should follow the output of `aws ecs register-task-definition --generate-cli-skeleton`.
#    Replace the value of the `ECS_TASK_DEFINITION` environment variable in the workflow below with the path to the JSON file.
#    Replace the value of the `CONTAINER_NAME` environment variable in the workflow below with the name of the container
#    in the `containerDefinitions` section of the task definition.
#
# 4. Store an IAM user access key in GitHub Actions secrets named `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
#    See the documentation for each action used below for the recommended IAM policies for this IAM user,
#    and best practices on handling the access key credentials.

name: Deploy to Amazon ECS

on:
  push:
    branches: [ "master" ]

env:
  AWS_REGION: MY_AWS_REGION                   # set this to your preferred AWS region, e.g. us-west-1
  ECR_REPOSITORY: MY_ECR_REPOSITORY           # set this to your Amazon ECR repository name
  ECS_SERVICE: MY_ECS_SERVICE                 # set this to your Amazon ECS service name
  ECS_CLUSTER: MY_ECS_CLUSTER                 # set this to your Amazon ECS cluster name
  ECS_TASK_DEFINITION: MY_ECS_TASK_DEFINITION # set this to the path to your Amazon ECS task definition
                                               # file, e.g. .aws/task-definition.json
  CONTAINER_NAME: MY_CONTAINER_NAME           # set this to the name of the container in the
                                               # containerDefinitions section of your task definition

permissions:
  contents: read

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    environment: production

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

    - name: Login to Amazon ECR
      id: login-ecr
      uses: aws-actions/amazon-ecr-login@v1

    - name: Build, tag, and push image to Amazon ECR
      id: build-image
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        IMAGE_TAG: ${{ github.sha }}
      run: |
        # Build a docker container and
        # push it to ECR so that it can
        # be deployed to ECS.
        docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
        docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
        echo "image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT

    - name: Fill in the new image ID in the Amazon ECS task definition
      id: task-def
      uses: aws-actions/amazon-ecs-render-task-definition@v1
      with:
        task-definition: ${{ env.ECS_TASK_DEFINITION }}
        container-name: ${{ env.CONTAINER_NAME }}
        image: ${{ steps.build-image.outputs.image }}

    - name: Deploy Amazon ECS task definition
      uses: aws-actions/amazon-ecs-deploy-task-definition@v1
      with:
        task-definition: ${{ steps.task-def.outputs.task-definition }}
        service: ${{ env.ECS_SERVICE }}
        cluster: ${{ env.ECS_CLUSTER }}
        wait-for-service-stability: true

This GitHub Actions workflow The file defines a CI/CD process for building and deploying Docker images to Amazon ECS. The design content includes: checking out code, configuring AWS credentials, logging in to Amazon ECR, building tags and pushing images to ECR, and updating ECS task definitions. Image ID in, deploy the updated ECS task definition.

5. Combine with Helm to deploy the application to the AKS cluster

# This workflow will build and push an application to a Azure Kubernetes Service (AKS) cluster when you push your code
#
# This workflow assumes you have already created the target AKS cluster and have created an Azure Container Registry (ACR)
# The ACR should be attached to the AKS cluster
# For instructions see:
#   - https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal
#   - https://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-portal
#   - https://learn.microsoft.com/en-us/azure/aks/cluster-container-registry-integration?tabs=azure-cli#configure-acr-integration-for-existing-aks-clusters
#   - https://github.com/Azure/aks-create-action
#
# To configure this workflow:
#
# 1. Set the following secrets in your repository (instructions for getting these
#    https://docs.microsoft.com/en-us/azure/developer/github/connect-from-azure?tabs=azure-cli%2Clinux)):
#    - AZURE_CLIENT_ID
#    - AZURE_TENANT_ID
#    - AZURE_SUBSCRIPTION_ID
#
# 2. Set the following environment variables (or replace the values below):
#    - AZURE_CONTAINER_REGISTRY (name of your container registry / ACR)
#    - CONTAINER_NAME (name of the container image you would like to push up to your ACR)
#    - RESOURCE_GROUP (where your cluster is deployed)
#    - CLUSTER_NAME (name of your AKS cluster)
#    - IMAGE_PULL_SECRET_NAME (name of the ImagePullSecret that will be created to pull your ACR image)
#
# 3. Choose the appropriate render engine for the bake step https://github.com/Azure/k8s-bake. The config below assumes Helm.
#    Set your helmChart, overrideFiles, overrides, and helm-version to suit your configuration.
#    - CHART_PATH (path to your helm chart)
#    - CHART_OVERRIDE_PATH (path to your helm chart with override values)
#
# For more information on GitHub Actions for Azure, refer to https://github.com/Azure/Actions
# For more samples to get started with GitHub Action workflows to deploy to Azure, refer to https://github.com/Azure/actions-workflow-samples
# For more options with the actions used below please refer to https://github.com/Azure/login

name: Build and deploy an app to AKS with Helm

on:
  push:
    branches: ["master"]
  workflow_dispatch:

env:
  AZURE_CONTAINER_REGISTRY: "your-azure-container-registry"
  CONTAINER_NAME: "your-container-name"
  RESOURCE_GROUP: "your-resource-group"
  CLUSTER_NAME: "your-cluster-name"
  CHART_PATH: "your-chart-path"
  CHART_OVERRIDE_PATH: "your-chart-override-path"

jobs:
  buildImage:
    permissions:
      contents: read
      id-token: write
    runs-on: ubuntu-latest
    steps:
      # Checks out the repository this file is in
      - uses: actions/checkout@v4

      # Logs in with your Azure credentials
      - name: Azure login
        uses: azure/login@v1.4.6
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

      # Builds and pushes an image up to your Azure Container Registry
      - name: Build and push image to ACR
        run: |
          az acr build --image ${{ env.AZURE_CONTAINER_REGISTRY }}.azurecr.io/${{ env.CONTAINER_NAME }}:${{ github.sha }} --registry ${{ env.AZURE_CONTAINER_REGISTRY }} -g ${{ env.RESOURCE_GROUP }} .

  deploy:
    permissions:
      actions: read
      contents: read
      id-token: write
    runs-on: ubuntu-latest
    needs: [buildImage]
    steps:
      # Checks out the repository this file is in
      - uses: actions/checkout@v4

      # Logs in with your Azure credentials
      - name: Azure login
        uses: azure/login@v1.4.6
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

      # Use kubelogin to configure your kubeconfig for Azure auth
      - name: Set up kubelogin for non-interactive login
        uses: azure/use-kubelogin@v1
        with:
          kubelogin-version: 'v0.0.25'

      # Retrieves your Azure Kubernetes Service cluster's kubeconfig file
      - name: Get K8s context
        uses: azure/aks-set-context@v3
        with:
          resource-group: ${{ env.RESOURCE_GROUP }}
          cluster-name: ${{ env.CLUSTER_NAME }}
          admin: 'false'
          use-kubelogin: 'true'

      # Runs Helm to create manifest files
      - name: Bake deployment
        uses: azure/k8s-bake@v2
        with:
          renderEngine: "helm"
          helmChart: ${{ env.CHART_PATH }}
          overrideFiles: ${{ env.CHART_OVERRIDE_PATH }}
          overrides: |
            replicas:2
          helm-version: "latest"
        id: bake

      # Deploys application based on manifest files from previous step
      - name: Deploy application
        uses: Azure/k8s-deploy@v4
        with:
          action: deploy
          manifests: ${{ steps.bake.outputs.manifestsBundle }}
          images: |
            ${{ env.AZURE_CONTAINER_REGISTRY }}.azurecr.io/${{ env.CONTAINER_NAME }}:${{ github.sha }}

This GitHub Actions workflow file defines a CI/CD process for building and pushing a Docker image to Azure Container Registry (ACR), then Deploy the application to Azure Kubernetes Service (AKS) cluster. The main steps include: check out the code, log in with Azure credentials, build and push the Docker image to ACR, get the kubeconfig file of the AKS cluster, render the deployment manifest file with Helm, and deploy the application.

6. Combine with Helm to deploy the application to Alibaba Cloud ACK cluster

# This workflow will build and push a new container image to Alibaba Cloud Container Registry (ACR),
# and then will deploy it to Alibaba Cloud Container Service for Kubernetes (ACK), when there is a push to the "master" branch.
#
# To use this workflow, you will need to complete the following set-up steps:
#
# 1. Create an ACR repository to store your container images.
#    You can use ACR EE instance for more security and better performance.
#    For instructions see https://www.alibabacloud.com/help/doc-detail/142168.htm
#
# 2. Create an ACK cluster to run your containerized application.
#    You can use ACK Pro cluster for more security and better performance.
#    For instructions see https://www.alibabacloud.com/help/doc-detail/95108.htm
#
# 3. Store your AccessKey pair in GitHub Actions secrets named `ACCESS_KEY_ID` and `ACCESS_KEY_SECRET`.
#    For instructions on setting up secrets see: https://developer.github.com/actions/managing-workflows/storing-secrets/
#
# 4. Change the values for the REGION_ID, REGISTRY, NAMESPACE, IMAGE, ACK_CLUSTER_ID, and ACK_DEPLOYMENT_NAME.
#

name: Build and Deploy to ACK

on:
  push:
    branches: [ "master" ]

# Environment variables available to all jobs and steps in this workflow.
env:
  REGION_ID: cn-hangzhou
  REGISTRY: registry.cn-hangzhou.aliyuncs.com
  NAMESPACE: namespace
  IMAGE: repo
  TAG: ${{ github.sha }}
  ACK_CLUSTER_ID: clusterID
  ACK_DEPLOYMENT_NAME: nginx-deployment

  ACR_EE_REGISTRY: myregistry.cn-hangzhou.cr.aliyuncs.com
  ACR_EE_INSTANCE_ID: instanceID
  ACR_EE_NAMESPACE: namespace
  ACR_EE_IMAGE: repo
  ACR_EE_TAG: ${{ github.sha }}

permissions:
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    environment: production

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    # 1.1 Login to ACR
    - name: Login to ACR with the AccessKey pair
      uses: aliyun/acr-login@v1
      with:
        region-id: "${{ env.REGION_ID }}"
        access-key-id: "${{ secrets.ACCESS_KEY_ID }}"
        access-key-secret: "${{ secrets.ACCESS_KEY_SECRET }}"

    # 1.2 Build and push image to ACR
    - name: Build and push image to ACR
      run: |
        docker build --tag "$REGISTRY/$NAMESPACE/$IMAGE:$TAG" .
        docker push "$REGISTRY/$NAMESPACE/$IMAGE:$TAG"

    # 1.3 Scan image in ACR
    - name: Scan image in ACR
      uses: aliyun/acr-scan@v1
      with:
        region-id: "${{ env.REGION_ID }}"
        access-key-id: "${{ secrets.ACCESS_KEY_ID }}"
        access-key-secret: "${{ secrets.ACCESS_KEY_SECRET }}"
        repository: "${{ env.NAMESPACE }}/${{ env.IMAGE }}"
        tag: "${{ env.TAG }}"

    # 2.1 (Optional) Login to ACR EE
    - uses: actions/checkout@v4
    - name: Login to ACR EE with the AccessKey pair
      uses: aliyun/acr-login@v1
      with:
        login-server: "https://${{ env.ACR_EE_REGISTRY }}"
        region-id: "${{ env.REGION_ID }}"
        access-key-id: "${{ secrets.ACCESS_KEY_ID }}"
        access-key-secret: "${{ secrets.ACCESS_KEY_SECRET }}"
        instance-id: "${{ env.ACR_EE_INSTANCE_ID }}"

    # 2.2 (Optional) Build and push image ACR EE
    - name: Build and push image to ACR EE
      run: |
        docker build -t "$ACR_EE_REGISTRY/$ACR_EE_NAMESPACE/$ACR_EE_IMAGE:$TAG" .
        docker push "$ACR_EE_REGISTRY/$ACR_EE_NAMESPACE/$ACR_EE_IMAGE:$TAG"
    # 2.3 (Optional) Scan image in ACR EE
    - name: Scan image in ACR EE
      uses: aliyun/acr-scan@v1
      with:
        region-id: "${{ env.REGION_ID }}"
        access-key-id: "${{ secrets.ACCESS_KEY_ID }}"
        access-key-secret: "${{ secrets.ACCESS_KEY_SECRET }}"
        instance-id: "${{ env.ACR_EE_INSTANCE_ID }}"
        repository: "${{ env.ACR_EE_NAMESPACE}}/${{ env.ACR_EE_IMAGE }}"
        tag: "${{ env.ACR_EE_TAG }}"

    # 3.1 Set ACK context
    - name: Set K8s context
      uses: aliyun/ack-set-context@v1
      with:
        access-key-id: "${{ secrets.ACCESS_KEY_ID }}"
        access-key-secret: "${{ secrets.ACCESS_KEY_SECRET }}"
        cluster-id: "${{ env.ACK_CLUSTER_ID }}"

    # 3.2 Deploy the image to the ACK cluster
    - name: Set up Kustomize
      run: |-
        curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash /dev/stdin 3.8.6
    - name: Deploy
      run: |-
        ./kustomize edit set image REGISTRY/NAMESPACE/IMAGE:TAG=$REGISTRY/$NAMESPACE/$IMAGE:$TAG
        ./kustomize build . | kubectl apply -f -
        kubectl rollout status deployment/$ACK_DEPLOYMENT_NAME
        kubectl get services -o wide

This GitHub Actions works The flow file defines a CI/CD process for building and pushing Docker images to Alibaba Cloud Container Registry (ACR), and then deploying the application to Alibaba Cloud Container Service for Kubernetes (ACK). The specific steps include: Checking out code, use AccessKey to log in to ACR, build and push Docker images to ACR, scan images in ACR, set the context of ACK cluster, set Kustomize, and deploy applications.

7. Combine with Helm to deploy applications to Google Cloud Cluster

# This workflow will build a docker container, publish it to Google Container
# Registry, and deploy it to GKE when there is a push to the "master"
# branch.
#
# To configure this workflow:
#
# 1. Enable the following Google Cloud APIs:
#
#    - Artifact Registry (artifactregistry.googleapis.com)
#    - Google Kubernetes Engine (container.googleapis.com)
#    - IAM Credentials API (iamcredentials.googleapis.com)
#
#    You can learn more about enabling APIs at
#    https://support.google.com/googleapi/answer/6158841.
#
# 2. Ensure that your repository contains the necessary configuration for your
#    Google Kubernetes Engine cluster, including deployment.yml,
#    kustomization.yml, service.yml, etc.
#
# 3. Create and configure a Workload Identity Provider for GitHub:
#    https://github.com/google-github-actions/auth#preferred-direct-workload-identity-federation.
#
#    Depending on how you authenticate, you will need to grant an IAM principal
#    permissions on Google Cloud:
#
#    - Artifact Registry Administrator (roles/artifactregistry.admin)
#    - Kubernetes Engine Developer (roles/container.developer)
#
#    You can learn more about setting IAM permissions at
#    https://cloud.google.com/iam/docs/manage-access-other-resources
#
# 5. Change the values in the "env" block to match your values.

name: 'Build and Deploy to GKE'

on:
  push:
    branches:
      - '"master"'

env:
  PROJECT_ID: 'my-project' # TODO: update to your Google Cloud project ID
  GAR_LOCATION: 'us-central1' # TODO: update to your region
  GKE_CLUSTER: 'cluster-1' # TODO: update to your cluster name
  GKE_ZONE: 'us-central1-c' # TODO: update to your cluster zone
  DEPLOYMENT_NAME: 'gke-test' # TODO: update to your deployment name
  REPOSITORY: 'samples' # TODO: update to your Artifact Registry docker repository name
  IMAGE: 'static-site'
  WORKLOAD_IDENTITY_PROVIDER: 'projects/123456789/locations/global/workloadIdentityPools/my-pool/providers/my-provider' # TODO: update to your workload identity provider

jobs:
  setup-build-publish-deploy:
    name: 'Setup, Build, Publish, and Deploy'
    runs-on: 'ubuntu-latest'
    environment: 'production'

    permissions:
      contents: 'read'
      id-token: 'write'

    steps:
      - name: 'Checkout'
        uses: 'actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332' # actions/checkout@v4

      # Configure Workload Identity Federation and generate an access token.
      #
      # See https://github.com/google-github-actions/auth for more options,
      # including authenticating via a JSON credentials file.
      - id: 'auth'
        name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/auth@f112390a2df9932162083945e46d439060d66ec2' # google-github-actions/auth@v2
        with:
          workload_identity_provider: '${{ env.WORKLOAD_IDENTITY_PROVIDER }}'

      # Authenticate Docker to Google Cloud Artifact Registry
      - name: 'Docker Auth'
        uses: 'docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567' # docker/login-action@v3
        with:
          username: 'oauth2accesstoken'
          password: '${{ steps.auth.outputs.auth_token }}'
          registry: '${{ env.GAR_LOCATION }}-docker.pkg.dev'

      # Get the GKE credentials so we can deploy to the cluster
      - name: 'Set up GKE credentials'
        uses: 'google-github-actions/get-gke-credentials@6051de21ad50fbb1767bc93c11357a49082ad116' # google-github-actions/get-gke-credentials@v2
        with:
          cluster_name: '${{ env.GKE_CLUSTER }}'
          location: '${{ env.GKE_ZONE }}'

      # Build the Docker image
      - name: 'Build and push Docker container'
        run: |-
          DOCKER_TAG="${GAR_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${REPOSITORY}/${IMAGE}:${GITHUB_SHA}"

          docker build \
            --tag "${DOCKER_TAG}" \
            --build-arg GITHUB_SHA="${GITHUB_SHA}" \
            --build-arg GITHUB_REF="${GITHUB_REF}" \
            .

          docker push "${DOCKER_TAG}"

      # Set up kustomize
      - name: 'Set up Kustomize'
        run: |-
          curl -sfLo kustomize https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.4.3/kustomize_v5.4.3_linux_amd64.tar.gz
          chmod u+x ./kustomize

      # Deploy the Docker image to the GKE cluster
      - name: 'Deploy to GKE'
        run: |-
          # replacing the image name in the k8s template
          ./kustomize edit set image LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/IMAGE:TAG=$GAR_LOCATION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY/$IMAGE:$GITHUB_SHA
          ./kustomize build . | kubectl apply -f -
          kubectl rollout status deployment/$DEPLOYMENT_NAME
          kubectl get services -o wide

This GitHub Actions workflow file defines a CI/CD process for building and pushing a Docker image to Google Cloud Artifact Registry (GAR) and then deploying the application to Google Kubernetes Engine (GKE ).

The specific steps include: check out the code, authenticate to Google Cloud using Workload Identity Provider, authenticate Docker to push the image to Artifact Registry, obtain the credentials of the GKE cluster, build and push the Docker image to Artifact Registry, set up Kustomize, and deploy the application program to GKE.

Github Actions Advantages and Disadvantages

Through using it for a few days, I feel that GitHub Actions is a powerful tool for automating software development workflows such as continuous integration (CI), continuous deployment (CD), testing, and release etc. It is directly integrated on the GitHub platform, providing rich functionality and flexibility.

However, like all tools, GitHub Actions also has its pros and cons. Here is a summary of its advantages and disadvantages:

Advantages

Seamless integration:

Seamless integration with GitHub: GitHub Actions is directly integrated into the GitHub platform, allowing easy access to code in the repository and metadata.
Automatic triggering: Workflows can be automatically triggered based on events such as code push, pull request, tag creation, etc.

Rich market:

Action market: GitHub provides A rich Action Marketplace, where users can find a large number of pre-built Actions for common tasks such as code quality checks, deployments, notifications, etc.
Community Support: A large developer community provides a wealth of resources and Best practices help users solve problems quickly.

Flexibility and scalability:

Custom workflow: Users can customize complex multi-step workflows to meet specific needs.
- Environment variables and key management*: Support environment variables and key management to ensure the security of sensitive information.
Parallel execution: Support parallel execution of multiple tasks to improve efficiency.

Built-in security and audit:

Security features: GitHub provides a variety of security features, such as dependency review, code scanning, etc.
Audit log: Detailed audit log records , easy to track and debug.

Cost-effectiveness:

Free plan: For open source projects and individual developers, GitHub provides free Actions usage quota.
Pay as you go: For Enterprise users can pay based on actual usage to avoid wasting resources.

Disadvantages

Learning curve:

Complexity: Although it provides rich functions, it requires configuration and management of complex workflows It may take some time to learn and experience.
Documentation and Examples: Although there is a lot of documentation and examples, it may take some time for beginners to find the right resources and best practices.

Performance and resource limits:

Resource limits: The free plan has limited resources, which may not be enough for large projects or high-frequency CI/CD pipelines.
Performance fluctuations: During peak hours, GitHub Actions Performance may be affected, resulting in slower builds and deployments.

Depends on GitHub:

Platform lock-in: Using GitHub Actions will increase your reliance on the GitHub platform, and the migration cost is high.
External integration: Although GitHub Actions supports integration with external tools and Service integration, but may require more configuration and maintenance than other CI/CD tools.

Difficult debugging:

Debugging tools: Although logs and outputs are provided, debugging is complex Multi-step workflows can be difficult, especially when problems arise.
Lack of visualization: GitHub Actions has limited visualization and monitoring capabilities compared to some commercial CI/CD tools.

*Security risks *:

Third-party Action: Using third-party Action may bring security risks and requires careful evaluation and selection.
Permission management: Improper permission management may lead to security vulnerabilities and requires careful configuration.

Summary

GitHub Actions is a powerful and flexible CI/CD tool that is particularly suitable for projects integrated with GitHub. Its advantages lie in seamless integration, rich market, flexibility and security.

However, it also has some disadvantages, such as a steep learning curve, resource limitations, platform lock-in, and debugging difficulties. The choice of whether to use GitHub Actions should be based on the specific needs of the project and the team’s technology stack.

For small projects and open source projects, GitHub Actions is a very good choice; for large enterprise projects, you may need to consider more professional CI/CD tools.

Deploy static files to a cloud server via SSH using GitHub Actions

wzdushu@gmail.com (wanzi) — Wed, 30 Oct 2024 21:22:42 +0800

Previously, we used GitHub Actions to configure automated deployment for Hugo. Today, I’ll mainly document how to set up deployment to Tencent Cloud via SSH using GitHub Actions.

Since wnote.com uses DNS-based intelligent routing—serving GitHub Pages for overseas users and a containerized environment on a Tencent Cloud host for users in mainland China—it’s necessary to integrate GitHub Actions with Tencent Cloud.

Configuring Secrets Here, we primarily configure the SSH-related credentials for Tencent Cloud:

We configure the following secrets:

Adding Tencent Cloud Deployment to the Workflow Configuration File

name: Deploy Hugo site to Pages

on:
  # Runs on pushes targeting the default branch
  push:
    branches: ["master"]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
  contents: read
  pages: write
  id-token: write

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
  group: "pages"
  cancel-in-progress: false

# Default to bash
defaults:
  run:
    shell: bash

jobs:
  # Build job and Deploy hugo to pages
  build-deploy:
    runs-on: ubuntu-latest
    env:
      HUGO_VERSION: 0.121.0
    steps:
      - name: Install Hugo CLI
        run: |
          wget -O ${{ runner.temp }}/hugo.deb https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_linux-amd64.deb \
          && sudo dpkg -i ${{ runner.temp }}/hugo.deb

      - name: Install Dart Sass
        run: sudo snap install dart-sass

      - name: Checkout
        uses: actions/checkout@v4
        with:
          submodules: recursive

      - name: Build with Hugo
        env:
          HUGO_CACHEDIR: ${{ runner.temp }}/hugo_cache
          HUGO_ENVIRONMENT: production
        run: |
          hugo \
            --minify \
            --baseURL "https://wnote.com/"
            #--baseURL "${{ steps.pages.outputs.base_url }}/"

      - name: Upload artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: ./public
          
      - name: Deploy to GitHub Pages
        uses: peaceiris/actions-gh-pages@v4
        with:
          deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }}
          external_repository: wzdushu/wzdushu.github.io
          publish_dir: ./public
          publish_branch: master
          cname: wnote.com
          commit_message: ${{ github.event.head_commit.message }}

      - name: Deploy to Tencent Cloud
        uses: easingthemes/ssh-deploy@v5.1.0
        with:
          SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
          ARGS: "-avzr --delete"
          SSH_CMD_ARGS: "-o StrictHostKeyChecking=no"
          SOURCE: "public/"
          REMOTE_HOST: ${{ secrets.TENCENT_REMOTE_HOST }}
          REMOTE_PORT: ${{ secrets.TENCENT_REMOTE_PORT }}
          REMOTE_USER: ${{ secrets.TENCENT_REMOTE_USR }}
          TARGET: ${{ secrets.TENCENT_REMOTE_TARGET }} # /opt/wwwroot/xxx
          EXCLUDE: "/dist/, /node_modules/"
          SCRIPT_BEFORE: |
            whoami
            ls -al
          SCRIPT_AFTER: |
            whoami
            ls -al
            echo $RSYNC_STDOUT

At this point, the GitHub Actions workflow for deploying to Tencent Cloud via SSH is complete.

Github Actions automatically deploys hugo to Github Pages

wzdushu@gmail.com (wanzi) — Thu, 17 Oct 2024 17:22:42 +0800

Preface

I recently planned to upgrade the theme of my blog. Since I was playing around with Github Actions, I planned to use it to implement CICD. After writing and submitting the article, the static files generated by hugo will be automatically deployed to Github pages, Netlify, Vercel, Cloud flare Pages and other third-party platforms. Today, I will record the automatic deployment process of Github pages, which will save a lot of trouble.

Since the Hugo static website generator used by the website involves sensitive configuration, I will put the source code of Hugo in a private warehouse, and then put the front-end static pages generated by Hugo in the Github public warehouse. In this way, we need to prepare two Github warehouses, and then implement CICD through github actions in the private warehouse.

Github Actions official documentation: https://docs.github.com/zh/actions

Github Pages official standard: https://pages.github.com/

Github Token configuration

Since it involves deployment, authentication will be used in this process, and Github provides three token authentication methods:

github_token
deploy_key
personal_token

In fact, in each workflow, GitHub will automatically create a unique GITHUB_TOKEN key for workflow authentication. GITHUB_TOKEN will expire when the job is completed or up to 24 hours later. This is a temporary authentication strategy.

Note: Since we are here involving cross-warehouse deployment, github_token cannot be used.

For more github_token, please refer to: https://docs.github.com/actions/security-guides/automatic-token-authentication

personal_token is mainly used to generate tokens with personal accounts.

https://github.com/settings/tokens

For deploy_token, you mainly generate a key pair yourself, attach the public key to the github repository, and store the private key yourself.

For more deploy_token references: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys

Here we use the deploy_token method for authentication, attaching the public key to the deploy-token of the public repository and the private key to the secret of the private repository.

Create deploy_token

1
2

ssh-keygen -t rsa -b 4096 -C "$(git config user.email)" -f gh-pages-wzdushu -N ""
# The key pair gh-pages-wzdushu and gh-pages-wzdushu.pub will be generated in the current directory

Public repository wzdushu/wzdushu.github.io configures Deploy keys and sets the content of gh-pages-wzdushu.pub.

Create a secret in a private repository, specify the name: ACTIONS_DEPLOY_KEY The content is gh-pages-wzdushu

Github Actions configuration

Create a workflow in a private repository, my path here is: .github/workflows/cicd.yml, edit this file as follows:

name: Deploy Hugo site to Pages

on:
# Runs on pushes targeting the default branch
push:
branches: ["master"]

# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
contents: read
pages: write
id-token: write

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued. # However, do NOT cancel in-progress runs as we want to allow these production deployments to complete. concurrency: group: "pages" cancel-in-progress: false # Default to bash defaults: run: shell: bash jobs: # Build job and Deploy hugo to pages build-deploy: runs-on: ubuntu-latest env: HUGO_VERSION: 0.121.0 steps: - name: Install Hugo CLI run: | wget -O ${{ runner.temp }}/hugo.deb https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_linux-amd64.deb \ && sudo dpkg -i ${{ runner.temp }}/hugo.deb - name: Install Dart Sass run: sudo snap install dart-sass - name: Checkout uses: actions/checkout@v4 with: submodules: recursive - name: Build with Hugo env: HUGO_CACHEDIR: ${{ runner.temp }}/hugo_cache HUGO_ENVIRONMENT: production run: | hugo \ --minify \ --baseURL "${{ steps.pages.outputs .base_url }}/" - name: Upload artifact uses: actions/upload-pages-artifact@v2 with: path: ./public - name: Deploy to GitHub Pages uses: peaceiris/actions-gh-pages@v4 with: deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }} external_repository: wzdushu/wzdushu.github.io
publish_dir: ./public
publish_branch: master
cname: wnote.com
commit_message: ${{ github.event.head_commit.message }}

on There are three triggering methods here:

push: execute when git push is pushed.
workflow_dispatch: manually execute the actions of the project repository.
schedule: scheduled execution

Since the new version of the hugo theme here depends on Sass, it needs to be installed here, which is not required for everyone.

The deployment here does not use the official actions, but uses the open source peaceiris/actions-gh-pages, mainly because this repository is more friendly to Hugo, MkDocs, Gatsby, mdBook, Next, Nuxt, etc.

For other configurations, please refer to the official documentation of github actions, and will not be explained one by one.

Execute deployment

Later, we happily submitted the article to Github as usual, triggered GitHub Actions to generate static files and pushed them to GitHub Pages, and spent the rest of the time drinking a cup of coffee and waiting for it to finish its work.

Summary: The function of Github actions to automatically deploy is not limited to deploying static pages today. For actual projects and production environments, applications are often deployed to k8s clusters or some other PAAS platforms, such as Alibaba Cloud ACK, AWS’s EKS, Azure ’s AKS, etc. Of course, Github Actions also has other functions similar to Jenkins or Gitlab CI.

Customize Jenkins Agent Integration with Docker and kubectl Tools

wzdushu@gmail.com (wanzi) — Wed, 22 Mar 2023 18:22:42 +0800

Since the official Jenkins agent image does not include the tools we need—such as Helm, kubectl, curl, and ArgoCD—during Jenkins builds, we must integrate these tools ourselves.

Note: The official image names have changed:

jenkins/agent (formerly jenkins/slave) — renamed starting from version 4.3-2

jenkins/inbound-agent (formerly jenkins/jnlp-slave) — renamed starting from version 4.3-2

Dockerfile

FROM jenkins/inbound-agent:4.11-1-alpine-jdk11

USER root

ADD docker/docker /usr/bin/docker
ADD kubectl /usr/bin/kubectl
ADD helm /usr/bin/helm

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN chmod +x /usr/bin/docker /usr/bin/kubectl /usr/bin/helm
RUN apk add curl

ENTRYPOINT ["/usr/local/bin/jenkins-agent"]

Build

`1`	`docker build -t harbor.test.com/tools/jnlp-docker:4.11-1-alpine-jdk11 -f Dockerfile .`

Jenkins Workspace and Local Persistent Storage for Maven Repository

wzdushu@gmail.com (wanzi) — Wed, 22 Mar 2023 16:22:42 +0800

Create Local Storage

Since this is a test cluster, we can directly use local volume storage here.

As the Jenkins server runs under the jenkins user with UID 1000, we need to pre-assign permissions for /opt/jenkins_agent/ and /opt/jenkins_maven/ on node01:

1
2

chown 1000.1000 -R /opt/jenkins_agent/
chown 1000.1000 -R /opt/jenkins_maven/

Local Storage: agent-pv-pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-agent-pv
spec:
  storageClassName: local # Local PV
  capacity:
    storage: 30Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  local:
    path: /opt/jenkins_agent
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - node01
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-agent-pvc
  namespace: kube-ops
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

Local Storage: maven-pv-pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-maven-pv
spec:
  storageClassName: local # Local PV
  capacity:
    storage: 30Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  local:
    path: /opt/jenkins_maven
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - node01
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-maven-pvc
  namespace: kube-ops
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

Create PV and PVC Resources

# kubectl apply -f agent-pv-pvc.yaml
persistentvolume/jenkins-agent-pv created
persistentvolumeclaim/jenkins-agent-pvc created

# kubectl apply -f maven-pv-pvc.yaml
persistentvolume/jenkins-maven-pv created
persistentvolumeclaim/jenkins-maven-pvc created

Deploying Jenkins in a Kubernetes Environment

wzdushu@gmail.com (wanzi) — Thu, 16 Mar 2023 16:22:42 +0800

1. Install and Deploy Jenkins

1. Manual Installation

Manual installation is straightforward—just prepare the YAML configuration in advance. The complete CICD resource file jenkins-install.yaml is as follows:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  storageClassName: local # Local PV
  capacity:
    storage: 30Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  local:
    path: /opt/jenkins
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - node02
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: kube-ops
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments", "ingresses"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["pods/log", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: kube-ops
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: kube-ops
spec:
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccount: jenkins
      initContainers:
        - name: fix-permissions
          image: busybox
          command: ["sh", "-c", "chown -R 1000:1000 /var/jenkins_home"]
          securityContext:
            privileged: true
          volumeMounts:
            - name: jenkinshome
              mountPath: /var/jenkins_home
      containers:
        - name: jenkins
          #image: jenkins/jenkins:lts
          image: jenkins/jenkins:2.375.3-lts # Use latest stable version
          imagePullPolicy: IfNotPresent
          env:
          - name: JAVA_OPTS
            value: -Dhudson.model.DownloadService.noSignatureCheck=true
          ports:
            - containerPort: 8080
              name: web
              protocol: TCP
            - containerPort: 50000
              name: agent
              protocol: TCP
          resources:
            limits:
              cpu: 1500m
              memory: 4096Mi
            requests:
              cpu: 1500m
              memory: 2048Mi
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          volumeMounts:
            - name: jenkinshome
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkinshome
          persistentVolumeClaim:
            claimName: jenkins-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: kube-ops
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  ports:
    - name: web
      port: 8080
      targetPort: web
    - name: agent
      port: 50000
      targetPort: agent
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  namespace: kube-ops
spec:
  ingressClassName: traefik
  rules:
  - host: jenkins.test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: jenkins
            port:
              number: 8080

Apply the Jenkins installation:

`1`	`kubectl apply -f jenkins-install.yaml`

This installation involves the following resources:

Local storage: using hostPath
RBAC permissions: ServiceAccount, ClusterRole, ClusterRoleBinding
Deployment and services: Deployment, Service, Ingress

2. Configure Jenkins

Note: Since Jenkins performs signature verification on update-center.json, you must disable this check beforehand to avoid failures when changing plugin sources. Set the environment variable JAVA_OPTS=-Dhudson.model.DownloadService.noSignatureCheck=true.

After installing Jenkins, open https://jenkins.test.com and enter the admin password. Retrieve it via:

1
2

kubectl exec -it jenkins-d995d6d4-srfr8 -n kube-ops -- cat /var/jenkins_home/secrets/initialAdminPassword
a571eb7a8ac54256ae638bb4a61fd31b

Skip plugin installation initially. Choosing default plugins will be very slow (you may opt for recommended plugins instead). Click the close button in the top-right corner to skip plugin selection. After configuring the domestic mirror source, proceed with installing essential plugins.

Go to: https://jenkins.test.com/chinese/ and configure the domestic mirror source. I use Alibaba Cloud’s mirror: https://mirrors.aliyun.com/jenkins/updates/update-center.json

3. Install Plugins

Visit: https://jenkins.test.com/manage/pluginManager/available

I mainly install the following plugins: Kubernetes, Chinese language pack, Pipeline, etc. Specific steps are omitted here.

Jenkins Localization (Chinese): https://plugins.jenkins.io/localization-zh-cn/
Build With Parameters: https://plugins.jenkins.io/build-with-parameters
Kubernetes: https://plugins.jenkins.io/kubernetes/
Pipeline: https://plugins.jenkins.io/workflow-aggregator/
Git: https://plugins.jenkins.io/git/
zentimestamp: https://plugins.jenkins.io/zentimestamp/

2. Connect to Kubernetes Cluster

1. Configure Kubernetes Cluster Info

Go to: https://jenkins.test.com/configureClouds/

Create a new cloud, select Kubernetes, and configure the cluster address and namespace:

2. Configure Jenkins Access Address

Set the Jenkins URL used within the Kubernetes cluster, including the Jenkins agent channel address:

3. Pod Template Configuration

Configure namespace and labels:

4. Container Template Configuration

Define slave build container settings. Note that the container name must be jnlp.

The Docker image I use is self-built, pre-integrated with docker, kubectl, and other tools. Leave the Command and Arguments fields blank; otherwise, container creation will fail.

Configure volumes:

Mount Docker socket to enable Docker-in-Docker (sharing host Docker).
Mount kubeconfig to allow kubectl, helm, and other commands to manage Kubernetes resources.

Finally, set the Service Account to jenkins, which matches the account created earlier.

3. Simple Test

1. Create a Freestyle Job

2. Bind Slave Pod

3. Configure Shell Test Command

4. Execute a Build Task

After execution, check the build history. You’ll observe that a slave Pod is automatically created during the process and then automatically deleted after completion.

4. Summary

Continuous integration and delivery (CI/CD) are essential components of modern development workflows. Most companies currently deploy Jenkins clusters to build CI/CD environments. However, traditional Jenkins Slave architectures (one master, multiple slaves) face several challenges:

Single point of failure: If the Master fails, the entire pipeline becomes unavailable.
Inconsistent slave configurations: Different languages require different build environments, leading to complex and hard-to-manage configurations.
Uneven resource distribution: Some Slaves are overloaded while others remain idle, causing job queuing.
Resource waste: Slaves often run on physical or virtual machines, consuming resources even when idle.

These issues drive the need for a more efficient and reliable solution. Containerization technologies like Docker, especially when combined with Kubernetes, effectively address these pain points. Below is a simplified diagram of a Jenkins cluster built on Kubernetes:

As shown in the diagram, both Jenkins Master and Slave run as Pods on Kubernetes nodes. The Master runs on one node and stores its configuration data in a Volume. Slaves are dynamically created and destroyed based on demand, rather than running continuously.

The workflow is as follows: When the Jenkins Master receives a build request, it dynamically creates a Jenkins Slave Pod based on the configured label, registers it with the Master, executes the job, then deregisters and deletes the Pod upon completion—returning to the original state.

Benefits of this approach include:

High availability: If the Jenkins Master fails, Kubernetes automatically creates a new container and assigns the Volume, preserving data and ensuring service continuity.
Dynamic scaling: Resources are allocated only when needed. After each job completes, the slave is deregistered and the Pod is deleted, freeing up resources. Kubernetes intelligently schedules slaves onto idle nodes, reducing bottlenecks caused by high utilization on specific nodes.
Excellent scalability: When cluster resources become insufficient and jobs queue up, simply add a new Kubernetes Node to scale horizontally.

Are all the previous problems solved in a Kubernetes environment? It appears nearly perfect.

Ultimately, deploying CI/CD using Jenkins + Kubernetes offers clear advantages in both workflow efficiency and cost optimization.

Getting Started with Argo Events

wzdushu@gmail.com (wanzi) — Wed, 07 Jul 2021 16:22:42 +0800

Previously, we introduced how to install Argo Workflow and trigger tasks. In this article, we focus on a new tool:

What is ArgoEvents?

Argo Events is an event-driven Kubernetes workflow automation framework. It supports over 20 different event sources (e.g., webhooks, S3 drops, cronjobs, message queues such as Kafka, GCP PubSub, SNS, SQS, etc.).

Features:

Supports events from over 20 event sources and more than 10 trigger types.
Enables customization of business-level constraints for workflow automation.
Manages everything from simple, linear, real-time workflows to complex, multi-source event scenarios.
Complies with the CloudEvents specification.

Components:

EventSource (similar to a gateway; sends messages to the event bus)
EventBus (event message queue; implemented using high-performance distributed messaging middleware NATS — note that NATS has ceased maintenance after 2023, so architectural changes may be expected in the future)
EventSensor (subscribes to the message queue, parameterizes events, and filters them)

Deploying ArgoEvents

Deploy argo-events:

1
2

kubectl create ns argo-events
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/v1.2.3/manifests/install.yaml

Deploy argo-eventbus:

`1`	`kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/eventbus/native.yaml`

RBAC Account Authorization

Create `operate-workflow-sa` account

Grant operate-workflow-sa permission to create Argo Workflows within the argo-events namespace — required for EventSensor to automatically create workflows later.

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: argo-events
  name: operate-workflow-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: operate-workflow-role
  namespace: argo-events
rules:
  - apiGroups:
      - argoproj.io
    verbs:
      - "*"
    resources:
      - workflows
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: operate-workflow-role-binding
  namespace: argo-events
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: operate-workflow-role
subjects:
  - kind: ServiceAccount
    name: operate-workflow-sa
    namespace: argo-events

Create `workflow-pods-sa` account

Grant workflow-pods-sa permission to create pods via workflows within the argo-events namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: argo-events
  name: workflow-pods-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workflow-pods-role
rules:
  - apiGroups:
      - ""
    verbs:
      - "*"
    resources:
      - pods
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: workflow-pods-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: workflow-pods-role
subjects:
  - kind: ServiceAccount
    name: workflow-pods-sa
    namespace: argo-events

Automating Task Triggers with ArgoEvents

Start an event source to receive requests:

`1`	`kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/event-sources/webhook.yaml`

Note: The name field in the event source is example. You must specify this exact name when creating a sensor in production environments.

Create a webhook-based sensor to consume requests:

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: webhook
spec:
  template:
    serviceAccountName: operate-workflow-sa
  dependencies:
    - name: test-dep
      eventSourceName: webhook
      eventName: example
  triggers:
    - template:
        name: webhook-workflow-trigger
        k8s:
          group: argoproj.io
          version: v1alpha1
          resource: workflows
          operation: create
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: webhook-
              spec:
                serviceAccountName: workflow-pods-sa
                entrypoint: whalesay
                arguments:
                  parameters:
                  - name: message
                    # Value will be overridden by payload from test-dep
                    value: hello world
                templates:
                - name: whalesay
                  inputs:
                    parameters:
                    - name: message
                  container:
                    image: docker/whalesay:latest
                    command: [cowsay]
                    args: ["{{inputs.parameters.message}}"]
          parameters:
            - src:
                dependencyName: test-dep
              dest: spec.arguments.parameters.0.value

Reference: https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/sensors/webhook.yaml

Forward local requests to remote:

`1`	`kubectl -n argo-events port-forward <name-of-event-source-pod> <local port>:12000`

Send data to the event source:

`1`	`curl -d '{"message":"ok"}' -H "Content-Type: application/json" -X POST http://localhost:12000/example`

Now, you can automate workflow task creation using ArgoEvents.

Argo Workflow Practice I: Installation and Deployment

wzdushu@gmail.com (wanzi) — Fri, 07 May 2021 16:22:42 +0800

Introduction & Architecture

Argo Workflows is an open-source, container-level workflow engine designed to orchestrate parallel jobs on Kubernetes. It leverages Kubernetes Custom Resource Definitions (CRDs) to implement its full architecture, including Workflow, Workflow Template, and Cron Workflow.

What Can Argo Workflows Do?

Define workflows where each step is a container.
Model multi-step workflows as a sequence of tasks or capture task dependencies using Directed Acyclic Graphs (DAGs).
Easily run compute-intensive jobs—such as machine learning or data processing—on Kubernetes within a short time frame.
Run CI/CD pipelines natively on Kubernetes without configuring complex software development tooling.

Key Features of Argo Workflows:

Workflow: Orchestrates multiple workflow templates with customizable execution order.
Workflow Template: A reusable template definition for workflows; can be invoked by other workflows or templates within the same namespace or cluster.
Cluster Workflow Template: A cluster-scoped workflow template accessible across all namespaces via ClusterRole permissions.
Cron Workflow: Scheduled workflow type, equivalent to an advanced version of Kubernetes CronJob.

Installation & Configuration

Install Argo Workflows

We are installing the stable version 2.12.10. The installation process sets up ServiceAccount, Role, ClusterRole, Deployment, and other necessary components.

1
2

kubectl create ns argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/v2.12.10/manifests/install.yaml

Configure Ingress for External Access

Since our cluster uses Traefik as the ingress controller, and Argo Workflows defaults to internal HTTPS access, we need to add specific annotations to route external requests to argo-server.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/redirect-entry-point: https
  name: argo-server
  namespace: argo
spec:
  rules:
  - host: argo.test.cn
    http:
      paths:
      - backend:
          serviceName: argo-server
          servicePort: web
        path: /
status:
  loadBalancer: {}

External access URL: https://argo.test.cn

Simple Workflow Test

Let’s test with a “Hello World” example:

# argo submit -n argo --watch https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/hello-world.yaml
Name:                hello-world-4mffd
Namespace:           argo
ServiceAccount:      default
Status:              Pending
Created:             Fri May 07 16:11:25 +0800 (now)
Name:                hello-world-4mffd
Namespace:           argo
ServiceAccount:      default
Status:              Running
Created:             Fri May 07 16:11:25 +0800 (now)
Started:             Fri May 07 16:11:25 +0800 (now)
Duration:            0 seconds

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ◷ hello-world-4mffd  whalesay  hello-world-4mffd  0s
Name:                hello-world-4mffd
Namespace:           argo
ServiceAccount:      default
Status:              Running
Created:             Fri May 07 16:11:25 +0800 (10 seconds ago)
Started:             Fri May 07 16:11:25 +0800 (10 seconds ago)
Duration:            10 seconds

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ◷ hello-world-4mffd  whalesay  hello-world-4mffd  10s       ContainerCreating
Name:                hello-world-4mffd
Namespace:           argo
ServiceAccount:      default
Status:              Succeeded
Conditions:
 Completed           True
Created:             Fri May 07 16:11:25 +0800 (2 minutes ago)
Started:             Fri May 07 16:11:25 +0800 (2 minutes ago)
Finished:            Fri May 07 16:14:18 +0800 (now)
Duration:            2 minutes 53 seconds
ResourcesDuration:   1m18s*cpu,1m18s*memory

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ✔ hello-world-4mffd  whalesay  hello-world-4mffd  2m

# argo list -n argo
NAME                STATUS      AGE   DURATION   PRIORITY
hello-world-4mffd   Succeeded   3m    2m         0

# argo logs -f hello-world-4mffd -n argo
hello-world-4mffd:  _____________
hello-world-4mffd: < hello world >
hello-world-4mffd:  -------------
hello-world-4mffd:     \
hello-world-4mffd:      \
hello-world-4mffd:       \
hello-world-4mffd:                     ##        .
hello-world-4mffd:               ## ## ##       ==
hello-world-4mffd:            ## ## ## ##      ===
hello-world-4mffd:        /""""""""""""""""___/ ===
hello-world-4mffd:   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
hello-world-4mffd:        \______ o          __/
hello-world-4mffd:         \    \        __/
hello-world-4mffd:           \____\______/

Configure GitLab Runner with Ceph S3

wzdushu@gmail.com (wanzi) — Fri, 26 Mar 2021 17:22:42 +0800

When building frontend projects with npm, it’s common for dependency downloads to take a long time, and reusing artifacts or caches across different jobs is also challenging. Whether using artifacts or cache, we ultimately need persistent reuse of files. Here, we’ll use cache as an example.

Note: The GitLab Runner is deployed to the Kubernetes cluster via Helm chart (deployment details are omitted). You must prepare a Ceph S3 key pair in advance for configuring accesskey and secretkey.

Create Kubernetes Secret

Used by the GitLab Runner to connect to Ceph S3.

apiVersion: v1
data:
  accesskey: N1NMT0hIRzYxddfsgxVzVssddfsdY=
  secretkey: d25Uc0NDQVdsfsdUkssCQ1VsdEwxeUsdsNwb2R4TnRzZDliTG1DTUN6cQ==
kind: Secret
metadata:
  name: gitlab-runner-s3
  namespace: gitlab-managed-apps
type: Opaque

Deploy GitLab Runner via Helm

Refer to the official GitLab Helm chart: https://gitlab.com/gitlab-org/charts/gitlab-runner.git. Modify the values.yaml file in the Helm chart directory as follows:

cache:
  ## General settings
  cacheType: s3
  cachePath: "devops" # Specify the Ceph S3 cache path; here we use department names for separation
  cacheShared: true

  ## S3 settings
  s3ServerAddress: "ops-rgw.test.cn"
  s3BucketName: "runners-cache"
  s3BucketLocation:
  s3CacheInsecure: true
  secretName: "gitlab-runner-s3"

Update the Helm configuration:

1
2

cd gitlab-runner
helm upgrade runner-devops -f values.yaml -n gitlab-runner .

Test GitLab CI Job

Configure GitLab CI by modifying .gitlab-ci.yaml. Here’s an example for a frontend project build:

stages:
  - Build

build-and-deploy:
  image: registry.test.cn/devops/node:latest
  stage: Build
  cache:
    key: devops-vue
    paths:
      - node_modules/
      - .yarn
  tags:
    - devopstest
  script:
    - yarn config set registry https://r.cnpmjs.org
    - yarn config set @test:registry https://npm.test.cn/
    - yarn --pure-lockfile --cache-folder .yarn --network-timeout 600000
    - yarn build
  when: always

After triggering the GitLab CI job, observe the real-time job execution. At this point, cache files have been uploaded to Ceph S3. Subsequent builds will be significantly faster!

64 Creating cache devops-vue-4...
65 node_modules/: found 30710 matching files and directories
66 .yarn: found 34390 matching files and directories 
67 Uploading cache.zip to https://ops-rgw.test.cn/runners-cache/devops/project/4187/devops-vue-js-starter-4
68 Created cache
70 Cleaning up file based variables
00:00
72 Job succeeded

Automated Application Deployment with ArgoCD and Jenkins Pipeline

wzdushu@gmail.com (wanzi) — Wed, 29 Jul 2020 15:22:42 +0800

Create Helm Repository

First, create a basic Helm template repository:

`1`	`helm create template .`

For actual deployments, you’ll need to customize the Helm template according to your business requirements. Here, we directly use an internal custom generic template for rapid deployment. Alternatively, you can refer to Bitnami’s maintained Helm charts: https://github.com/bitnami/charts/tree/master/bitnami

Jenkins Credential Configuration: ArgoCD Token

Configure Jenkins Pipeline

We’ll use the gotest project (https://code.test.cn/hqliang/gotest) as an example.

pipeline {
    environment {
        GOPROXY = "http://repo.test.cn/repository/goproxy/"
        HUB_URL = "registry.test.cn"
        ARGOCD_SERVER = "qacd.test.cn"
        ARGOCD_PROJ = "test-project"
        ARGOCD_APP = "gotest"
        ARGOCD_REPO = "https://code.test.cn/devops/cicd/qa.git"
        ARGOCD_PATH = "devops/gotest"
        ARGOCD_CLUSTER = "https://172.16.19.250:8443"
        ARGOCD_NS = "default"
    }

    agent {
        node {
            label 'aiops'
        }
    }

    stages {
        stage('Docker_Build') {
            steps {
                withCredentials([[$class: 'UsernamePasswordMultiBinding',
                    credentialsId: '12ff2942-972c-4df3-8d2d-2cfcb25e00de',
                    usernameVariable: 'DOCKER_HUB_USER',
                    passwordVariable: 'DOCKER_HUB_PASSWORD']]) {
                    sh '''
                    TAG=$(git describe --tags `git rev-list --tags --max-count=1`)
                    docker login registry.test.cn -u ${DOCKER_HUB_USER} -p ${DOCKER_HUB_PASSWORD}
                    make docker-all VERSION=$TAG
                    '''
                }
            }
        }

        stage('Deploy_K8S') {
            steps {
                withCredentials([string(credentialsId: "qa-argocd", variable: 'ARGOCD_AUTH_TOKEN')]) {
                    sh '''
                    TAG=$(git describe --tags `git rev-list --tags --max-count=1`)
                    # Create app
                    ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app create $ARGOCD_APP \
                        --project $ARGOCD_PROJ \
                        --repo $ARGOCD_REPO \
                        --path $ARGOCD_PATH \
                        --dest-namespace $ARGOCD_NS \
                        --dest-server $ARGOCD_CLUSTER \
                        --upsert --insecure
                    # Deploy app
                    ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app set $ARGOCD_APP \
                        -p containers.tag=$TAG --insecure
                    # Sync to ArgoCD
                    ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app sync $ARGOCD_APP \
                        --force --insecure
                    ARGOCD_SERVER=$ARGOCD_SERVER argocd --grpc-web app wait $ARGOCD_APP \
                        --timeout 600 --insecure
                    '''
                }
            }
        }
    }
}

Configure GitLab Webhook to Trigger Jenkins

Jenkins Configuration: Trigger Build

GitLab Webhook Configuration

Push Tag to Remote Branch

Jenkins Pipeline Build

Check the Jenkins pipeline build result:

Docker automatically builds and pushes the image to Harbor:

Create Argo application and sync deployment to Kubernetes cluster:

Finally, check the sync status — indicating successful deployment to the Kubernetes cluster:

View the application workflow graph in ArgoCD’s dashboard — all components are running normally:

Use kubectl to verify the created resources:

Finally, validate if the service is accessible: visit http://gotest.test.cn/df to check container disk space.

Summary & Optimization

Whether using Jenkins pipelines or GitLab CI, we can better integrate ArgoCD to achieve one-click deployment of applications.

When deploying a Deployment in Argo CD, you encounter the error: `no space left on device`.

wzdushu@gmail.com (wanzi) — Mon, 18 May 2020 10:22:42 +0800

Failure Phenomenon

This morning, I deployed several business applications via ArgoCD. After successfully deploying two applications, subsequent deployments from a third-party source consistently failed—despite using identical configurations, only the target cluster differed. Why would this happen?

I checked the logs and found the following:

1
2
3

  Warning  Failed     1m                kubelet, 172.16.25.13  Error: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/ba37165607862efb350093e5e287207e2547759fd81dc4e5e356a86ac5e28324-init/merged: no space left on device
  Warning  Failed     1m                kubelet, 172.16.25.13  Error: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/f69b62f360fc2a94487aca041b08d0929810beab0602e0ec8b90c94b2e893337-init/merged: no space left on device
  Warning  Failed     48s               kubelet, 172.16.25.13  Error: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/a8d20a44183b39ae989eee8a442960124ff23844482f726ea7ab39a292aecbb3-init/merged: no space left on device

Solution

Check disk space—no issues found:

1
2
3

root@gpu613:~# df -Th /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda2      ext4  1.8T  359G  1.3T  22% /

After Googling, I discovered this might be due to exhausted inotify watches.

Check current limit:

1
2

#cat /proc/sys/fs/inotify/max_user_watches
8192

Increase the number of watchable directories:

1
2

echo "fs.inotify.max_user_watches=100000" >> /etc/sysctl.conf
sysctl -p

After applying the change, re-triggered the ArgoCD sync. This time, the deployment was successfully created—confirming that inotify watch exhaustion was indeed the root cause.

Add multiple clusters to ArgoCD

wzdushu@gmail.com (wanzi) — Tue, 05 May 2020 15:22:42 +0800

Generate Argo CD Management User Token

Log in to the dashboard, go to Settings → Accounts → admin → Generate New.
After generation, record the token information, which looks like this:

fyJhbGciOiJ3UzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI2OWI0M2M0Mi01MmZiLTRlZmItODIxOC0yOWU3NGM5MWI0NDIiLCJpYXQiOjE1OTUzMTEx3zQsImlzcyI6ImFyZ29jZCIsIm5iZiI6MTU5NTMxMTE3NCwic3ViIjoib3duZXIifQ.9u4XzArEeaz7G2Q2TWusnTkakEmq9BYDAUHr3dC6wG5

Configure Argo CD Config

For Argo CD with HTTPS enabled, adding clusters becomes cumbersome—it requires logging into the server pod for configuration. Follow these steps:

# cat ~/.argocd/config
contexts:
- name: argocd-server.argocd
  server: qacd.test.cn
  user: argocd-server.argocd
current-context: argocd-server.argocd
servers:
- grpc-web-root-path: ""
  insecure: true
  server: qacd.test.cn
users:
- auth-token: xxxxxx # This is the token generated in step 1
  name: argocd-server.argocd

Configure kubeconfig

Skip detailed configuration here—refer to previous documentation. Ensure you can access the cluster and have cluster administrator privileges. Set the CONTEXT to idc-bj-k8s.

Add Cluster

# argocd --grpc-web cluster add idc-bj-k8s --kubeconfig ~/.kube/config
INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" updated
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" updated
Cluster 'https://172.16.16.250:8443' added
# argocd --grpc-web cluster list
SERVER                          NAME        VERSION  STATUS      MESSAGE
https://172.16.16.250:8443      idc-bj-k8s  1.14     Successful
https://kubernetes.default.svc              1.14     Successful

The Beijing IDC cluster has now been successfully added to Argo CD. You can proceed to deploy applications to it.

Remove Cluster

# argocd --grpc-web cluster rm https://172.16.16.250:8443
# argocd --grpc-web cluster list
SERVER                          NAME        VERSION  STATUS      MESSAGE
https://kubernetes.default.svc              1.14     Successful

ArgoCD Installation and Deployment

wzdushu@gmail.com (wanzi) — Fri, 01 May 2020 15:22:42 +0800

Installation and Deployment

Deploying ArgoCD is straightforward. Use the official high-availability (HA) deployment method:

1
2

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v1.5.2/manifests/ha/install.yaml

You can customize the deployment file as needed. After the pods are successfully started:

# kubectl -n argocd get pod
NAME                                             READY   STATUS    RESTARTS   AGE
argocd-application-controller-66fbf66657-ghf2c   1/1     Running   0          6d17h
argocd-application-controller-66fbf66657-gpm7d   1/1     Running   0          6d17h
argocd-application-controller-66fbf66657-tr5kd   1/1     Running   0          6d17h
argocd-dex-server-5c5f986596-c8ftv               1/1     Running   0          9d
argocd-redis-ha-haproxy-69c6df79c6-2fxd6         1/1     Running   0          9d
argocd-redis-ha-haproxy-69c6df79c6-mksg2         1/1     Running   0          9d
argocd-redis-ha-haproxy-69c6df79c6-wq57f         1/1     Running   0          9d
argocd-redis-ha-server-0                         2/2     Running   0          9d
argocd-redis-ha-server-1                         2/2     Running   0          9d
argocd-redis-ha-server-2                         2/2     Running   0          9d
argocd-repo-server-76bbb56cc7-d8fp5              1/1     Running   0          7d
argocd-repo-server-76bbb56cc7-qvl5z              1/1     Running   0          7d
argocd-repo-server-76bbb56cc7-xqrfn              1/1     Running   0          7d
argocd-server-6464c7bcd-fgktr                    1/1     Running   0          6d19h
argocd-server-6464c7bcd-jkqdb                    1/1     Running   0          6d19h
argocd-server-6464c7bcd-nfdwn                    1/1     Running   0          6d19h

Configure Ingress for ArgoCD Access

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/redirect-entry-point: https
spec:
  rules:
    - host: cd.testcn
      http:
        paths:
        - backend:
            serviceName: argocd-server
            servicePort: https
          path: /

Access ArgoCD via https://cd.test.cn/. The default username is admin, and the password is the name of one of the pods. Retrieve the password using:

`1`	`# kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name \| cut -d'/' -f 2`

User Management

ArgoCD defaults to local users but supports LDAP integration. Manage local users by editing the argocd-cm ConfigMap. To add a user:

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd
data:
  accounts.chlai: apiKey,login  # Allows login and API key generation
  users.anonymous.enabled: "true"  # Enables anonymous login

Changes take effect immediately after saving.

By default, two roles exist: readonly and admin. Grant admin privileges by modifying the argocd-rbac-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.csv: |-
    g, chlai, role:admin
  policy.default: role:readonly  # Anonymous users default to readonly role

To generate a login password, use the admin user via CLI to log in to the server, then update the password with:

argocd login <argocd-server>
argocd account list
argocd account update-password \
  --account <name> \
  --current-password <current-admin> \
  --new-password <new-user-password>

Creating Applications via ArgoCD UI

Click the + NEW APP button to create an application.

Fill in the application details:

Name: guestbook
Project: default
Sync Policy: Manual

Configure the source. Here, use Git; set the repository URL to: https://github.com/argoproj/argocd-example-apps.git; select HEAD as Revision; choose the path: guestbook.

Select the target cluster for deployment: https://kubernetes.default.svc. Since ArgoCD is deployed within a Kubernetes cluster, it has already registered the current cluster—use it directly. Set Namespace to my-app. Create this namespace in Kubernetes using: kubectl create namespace my-app.

After filling in all fields, click CREATE.

Since the application hasn’t been deployed yet and no Kubernetes resources have been created, the status remains OutOfSync. Click SYNC to synchronize (deploy). You can also install the ArgoCD CLI and sync via command: argocd app sync guestbook.

The application is now created but still OutOfSync. Manually sync to start deployment.

Wait for the application to be fully deployed.

Check the application in the Kubernetes cluster.

Deploy GitLab Runner on K8S

wzdushu@gmail.com (wanzi) — Thu, 14 Nov 2019 17:22:42 +0800

Deploy gitlab-runner

Deploy using Helm, refer to: https://gitlab.com/gitlab-org/charts/gitlab-runner.git

`1`	`helm install --namespace gitlab-managed-apps --name k8s-gitlab-runner -f values.yaml`

Note: The values.yaml file must set privileged: true.

Build Base Image (Docker-in-Docker)

Content of the Dockerfile:

FROM docker:19.03.1-dind
WORKDIR /opt
RUN echo "nameserver 114.114.114.114" >> /etc/resolv.conf
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN apk update
RUN apk upgrade
RUN apk add g++ gcc make docker docker-compose git

Build the image and push it to Harbor:

1
2

docker build -t registry.test.cn/devops/docker-tool:19.03.1 .
docker push registry.test.cn/devops/docker-tool:19.03.1

GitLab CI Test

Content of .gitlab-ci.yml:

image: registry.test.cn/devops/docker-tool:19.03.1

variables:
  REPO_NAME: gitlab.test.cn/xxx/xxxx
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

before_script:
  - mkdir -p $GOPATH/src/$(dirname $REPO_NAME)
  - ln -svf $CI_PROJECT_DIR $GOPATH/src/$REPO_NAME
  - cd $GOPATH/src/$REPO_NAME

stages:
  - deploy

deploy:
  tags:
    - k8s-gitlab-runner # Specify runner
  only:
    - tags
  stage: deploy
  services:
    - registry.test.cn/devops/docker-tool:19.03.1
  script:
    - export DOCKER_HOST='tcp://localhost:2375'
    - docker login -u "$Harbor_bce_user" -p "$Harbor_ecs_passwd" $Harbor_ecs_address # Set global variables in GitLab UI
    - make deploy VERSION=$CI_COMMIT_REF_NAME # Build image based on Git tag

Setting up Jenkins using Docker Compose

wzdushu@gmail.com (wanzi) — Mon, 11 Nov 2019 16:22:42 +0800

docker-compose Configuration

version: '2'
 
services:
  jenkins:
    image: jenkins/jenkins:latest
    restart: always
    environment:
      JAVA_OPTS: "-Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Shanghai -Djava.awt.headless=true -Dmail.smtp.starttls.enable=true"
    ports:
      - "80:8080"
      - "50000:50000"
    volumes:
      - '/ssd/jenkins:/var/jenkins_home'
      - '/var/run/docker.sock:/var/run/docker.sock'
      - '/etc/localtime:/etc/localtime:ro'
    dns: 223.5.5.5
    networks:
      - extnetwork
networks:
   extnetwork:
      ipam:
         config:
         - subnet: 172.255.0.0/16

Start Services

`1`	`docker-compose up -d`