Gitlab runner配置ceph s3

wzdushu@gmail.com (wanzi) — Fri, 26 Mar 2021 17:22:42 +0800

对于前端项目Npm构建的时候，经常拉取前端库耗时比较长，另外不同的job之间复用也是一个问题，无论是artifacts或者cache最终我们需要持久化复用文件，这里我们以cache为例

注意：这里的Gitlab runner是采用helm chart方式部署到k8s集群，runner部署忽略；需要提前准备ceph s3的密钥对，用于配置accesskey和secretkey

创建k8s secret

给后边Gitlab Runner连接ceph s3使用。

apiVersion: v1
data:
  accesskey: N1NMT0hIRzYxddfsgxVzVssddfsdY=
  secretkey: d25Uc0NDQVdsfsdUkssCQ1VsdEwxeUsdsNwb2R4TnRzZDliTG1DTUN6cQ==
kind: Secret
metadata:
  name: gitlab-runner-s3
  namespace: gitlab-managed-apps
type: Opaque

helm部署gitlab runner

具体Gitlab helm chart参考：https://gitlab.com/gitlab-org/charts/gitlab-runner.git，修改helm chart目录下values.yaml内容如下：

cache:
  ## General settings
  cacheType: s3
  cachePath: "devops" #指定ceph s3缓存路径，这里我们以部门来区分
  cacheShared: true
   
  ## S3 settings
  s3ServerAddress: "ops-rgw.test.cn"
  s3BucketName: "runners-cache"
  s3BucketLocation:
  s3CacheInsecure: true
  secretName: "gitlab-runner-s3"

更新helm配置

1
2

cd gitlab-runner
helm upgrade   runner-devops -f values.yaml -n gitlab-runner .

测试gitlab CI任务

配置gitlab Ci，修改.gitlab-ci.yaml，这里以前端项目构建为例：

stages:
  - Build
   
build-and-deploy:
  image: registry.test.cn/devops/node:latest
  stage: Build
  cache:
    key: devops-vue
    paths:
      - node_modules/
      - .yarn
  tags:
    - devopstest
  script:
    - yarn config set registry https://r.cnpmjs.org
    - yarn config set @test:registry https://npm.test.cn/
    - yarn --pure-lockfile --cache-folder .yarn --network-timeout 600000
    - yarn build
  when: always

Gitlab CI触发构建任务以后，我们观察JOB构建任务实时情况，这时缓存文件已经上传到ceph s3，后期再构建编译就大大提高了效率！

64 Creating cache devops-vue-4...
65 node_modules/: found 30710 matching files and directories
66 .yarn: found 34390 matching files and directories 
67 Uploading cache.zip to https://ops-rgw.test.cn/runners-cache/devops/project/4187/devops-vue-js-starter-4
68 Created cache
70Cleaning up file based variables
00:00
72 Job succeeded

基于K8S部署gitlab-runner

wzdushu@gmail.com (wanzi) — Thu, 14 Nov 2019 17:22:42 +0800

部署gitlab-runner

这里基于helm部署，参考：https://gitlab.com/gitlab-org/charts/gitlab-runner.git

`1`	`helm install --namespace gitlab-managed-apps --name k8s-gitlab-runner -f values.yaml`

注意：values.yaml文件需要设置privileged: true

构建基础镜像(docker in docker)

Dockerfile文件内容：

FROM docker:19.03.1-dind
WORKDIR /opt
RUN echo "nameserver 114.114.114.114" >> /etc/resolv.conf
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN apk update
RUN apk upgrade
RUN apk add g++ gcc make docker docker-compose  git

构建镜像并推送到harbor

1
2

docker build -t registry.test.cn/devops/docker-tool:19.03.1  .
docker push  registry.test.cn/devops/docker-tool:19.03.1

Gitlab-CI测试

.gitlab-ci.yml文件内容如下:

image: registry.test.cn/devops/docker-tool:19.03.1

variables:
  REPO_NAME: gitlab.test.cn/xxx/xxxx
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""
  
before_script:
  - mkdir -p $GOPATH/src/$(dirname $REPO_NAME)
  - ln -svf $CI_PROJECT_DIR $GOPATH/src/$REPO_NAME
  - cd $GOPATH/src/$REPO_NAME

stages:
  - deploy

deploy:
  tags:
    - k8s-gitlab-runner #指定runner
  only:
    - tags
  stage: deploy
  services:
    - registry.test.cn/devops/docker-tool:19.03.1
  script:
    - export DOCKER_HOST='tcp://localhost:2375'
    - docker login -u "$Harbor_bce_user" -p "$Harbor_ecs_passwd" $Harbor_ecs_address #gitlab后台设置统一变量
    - make deploy VERSION=$CI_COMMIT_REF_NAME #根据git tag构建镜像

Gitlab-Runner on 丸子有记, 记录生活、分享技术，专注于运维架构、运维开发(devops)。