Configure GitLab Runner with Ceph S3

wzdushu@gmail.com (wanzi) — Fri, 26 Mar 2021 17:22:42 +0800

When building frontend projects with npm, it’s common for dependency downloads to take a long time, and reusing artifacts or caches across different jobs is also challenging. Whether using artifacts or cache, we ultimately need persistent reuse of files. Here, we’ll use cache as an example.

Note: The GitLab Runner is deployed to the Kubernetes cluster via Helm chart (deployment details are omitted). You must prepare a Ceph S3 key pair in advance for configuring accesskey and secretkey.

Create Kubernetes Secret

Used by the GitLab Runner to connect to Ceph S3.

apiVersion: v1
data:
  accesskey: N1NMT0hIRzYxddfsgxVzVssddfsdY=
  secretkey: d25Uc0NDQVdsfsdUkssCQ1VsdEwxeUsdsNwb2R4TnRzZDliTG1DTUN6cQ==
kind: Secret
metadata:
  name: gitlab-runner-s3
  namespace: gitlab-managed-apps
type: Opaque

Deploy GitLab Runner via Helm

Refer to the official GitLab Helm chart: https://gitlab.com/gitlab-org/charts/gitlab-runner.git. Modify the values.yaml file in the Helm chart directory as follows:

cache:
  ## General settings
  cacheType: s3
  cachePath: "devops" # Specify the Ceph S3 cache path; here we use department names for separation
  cacheShared: true

  ## S3 settings
  s3ServerAddress: "ops-rgw.test.cn"
  s3BucketName: "runners-cache"
  s3BucketLocation:
  s3CacheInsecure: true
  secretName: "gitlab-runner-s3"

Update the Helm configuration:

1
2

cd gitlab-runner
helm upgrade runner-devops -f values.yaml -n gitlab-runner .

Test GitLab CI Job

Configure GitLab CI by modifying .gitlab-ci.yaml. Here’s an example for a frontend project build:

stages:
  - Build

build-and-deploy:
  image: registry.test.cn/devops/node:latest
  stage: Build
  cache:
    key: devops-vue
    paths:
      - node_modules/
      - .yarn
  tags:
    - devopstest
  script:
    - yarn config set registry https://r.cnpmjs.org
    - yarn config set @test:registry https://npm.test.cn/
    - yarn --pure-lockfile --cache-folder .yarn --network-timeout 600000
    - yarn build
  when: always

After triggering the GitLab CI job, observe the real-time job execution. At this point, cache files have been uploaded to Ceph S3. Subsequent builds will be significantly faster!

64 Creating cache devops-vue-4...
65 node_modules/: found 30710 matching files and directories
66 .yarn: found 34390 matching files and directories 
67 Uploading cache.zip to https://ops-rgw.test.cn/runners-cache/devops/project/4187/devops-vue-js-starter-4
68 Created cache
70 Cleaning up file based variables
00:00
72 Job succeeded

Deploy GitLab Runner on K8S

wzdushu@gmail.com (wanzi) — Thu, 14 Nov 2019 17:22:42 +0800

Deploy gitlab-runner

Deploy using Helm, refer to: https://gitlab.com/gitlab-org/charts/gitlab-runner.git

`1`	`helm install --namespace gitlab-managed-apps --name k8s-gitlab-runner -f values.yaml`

Note: The values.yaml file must set privileged: true.

Build Base Image (Docker-in-Docker)

Content of the Dockerfile:

FROM docker:19.03.1-dind
WORKDIR /opt
RUN echo "nameserver 114.114.114.114" >> /etc/resolv.conf
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN apk update
RUN apk upgrade
RUN apk add g++ gcc make docker docker-compose git

Build the image and push it to Harbor:

1
2

docker build -t registry.test.cn/devops/docker-tool:19.03.1 .
docker push registry.test.cn/devops/docker-tool:19.03.1

GitLab CI Test

Content of .gitlab-ci.yml:

image: registry.test.cn/devops/docker-tool:19.03.1

variables:
  REPO_NAME: gitlab.test.cn/xxx/xxxx
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

before_script:
  - mkdir -p $GOPATH/src/$(dirname $REPO_NAME)
  - ln -svf $CI_PROJECT_DIR $GOPATH/src/$REPO_NAME
  - cd $GOPATH/src/$REPO_NAME

stages:
  - deploy

deploy:
  tags:
    - k8s-gitlab-runner # Specify runner
  only:
    - tags
  stage: deploy
  services:
    - registry.test.cn/devops/docker-tool:19.03.1
  script:
    - export DOCKER_HOST='tcp://localhost:2375'
    - docker login -u "$Harbor_bce_user" -p "$Harbor_ecs_passwd" $Harbor_ecs_address # Set global variables in GitLab UI
    - make deploy VERSION=$CI_COMMIT_REF_NAME # Build image based on Git tag

Gitlab on Wanzi's Blog – Notes on life, tech, and DevOps.