gitlab-ci-k8s-runner部署及缓存配置说明

因gitlab-ci的k8s-runner会将job里的每个stage都交由单独的pod来执行，导致默认的本地缓存无法使用。
本文记录了基于k8s pv的gitlab-ci的缓存配置方案，无需使用S3分布式存储。
并记录了gitlab-ci-k8s-runner相关的配置、部署步骤等。

gitlab-ci-k8s-runner部署及缓存配置说明

一 gitlab-ci-cache机制说明

参考：https://blog.csdn.net/xichenguan/article/details/101439395

Gitlab cache 机制可以大大加快 CI/CD Job 的执行速度。基础知识可以参看 Gitlab Cache。
下面直接总结在 Kubernetes 环境中的三种 Cache 的解决方案。

1. Distributed runners caching

gitlab runner job执行前从分布式存储中检查下载解压 cache 文件，job执行后，压缩上传 cache 文件到分布式存储。这是 gitlab 提供的通用的正宗的方法，在非 Kubernetes 环境中也可以使用；

2. 给 Job Executor Pod 挂载同一个 volume

gitlab runner 提供了本地存储 cache 的方式，如果远程存储没有配置，gitlab runner 照样会压缩 cache 文件，然后按照目录规则存储到指定的目录。这种方式的原理是将此目录配置为 Kubernetes Volume，每个 Job 执行时都挂载此 Volume ，这样就相当于所有的 Job 有了一个集中式的存储。可以参考使用GitLab CI在Kubernetes服务上运行GitLab Runner并执行Pipeline 尝试配置。

3. 不使用缓存，在 Kubernetes 集群中安装各类仓库的私服

Gitlab CI/CI Job 执行时，配置为使用这些私服，速度也很快。

注意：方式1一般指S3存储，建议使用方式2

二准备缓存pv、pvc

1.配置runner nfs缓存目录

export CACHE_DIR=/home/runner-cache
mkdir -p $CACHE_DIR
chmod -R 777 $CACHE_DIR
echo "$CACHE_DIR *(rw,sync,no_root_squash,all_squash,no_subtree_check,anonuid=0,anongid=0)" >> /etc/exports
exportfs -a
showmount -e localhost

2.创建pv、pvc

kubectl -n=命名空间 apply -f gitlab-runner-pv-pvc.yaml

3. 创建imagePullSecret

参考：https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account

kubectl -n=命名空间 create secret docker-registry myregistrykey --docker-server=DUMMY_SERVER \
        --docker-username=DUMMY_USERNAME --docker-password=DUMMY_DOCKER_PASSWORD 
kubectl -n=命名空间 get secret myregistrykey
kubectl -n=命名空间 get secret myregistrykey --output=json

4. 可选：将gitlab-runner、gitlab-runner-helper镜像上传到私有仓库

gitlab-runner镜像

docker tag registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v15.8.0 私有仓库/gitlab-runner:alpine-v15.8.0
docker push 私有仓库/gitlab-runner:alpine-v15.8.0

gitlab-runner-helper镜像

docker tag registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-12335144 私有仓库/gitlab-runner-helper:x86_64-12335144
docker push 私有仓库/gitlab-runner-helper:x86_64-12335144

三根据官方helm生成k8s-runner部署文件

参考：https://docs.gitlab.com/runner/install/kubernetes.html
默认values.yaml文件：https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/main/values.yaml

helm repo add gitlab https://charts.gitlab.io
helm repo update gitlab
helm install --dry-run --debug --namespace 命名空间 \
gitlab-runner -f values.yaml gitlab/gitlab-runner > gitlab-runner-all.yaml

kubectl -n=命名空间 apply -f gitlab-runner-all.yaml

四附：docker-runner部署

运行

docker run -d --name gitlab-runner --restart always \
  -v /home/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

注册

docker run --rm -it -v /home/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

重启

docker restart gitlab-runner
docker logs -f gitlab-runner

修改配置

在gitlab修改配置runner，支持运行未标记的作业

五 value.yaml和pv-pvc文件参考

gitlab-runner-pv-pvc.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: gitlab-runner-cache-pv
spec:
  storageClassName: gitlab-runner
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: $NFS_SERVER
    path: "$NFS_PATH"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gitlab-runner-cache-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: gitlab-runner
  resources:
    requests:
      storage: 100Gi

gitlab-k8s-runner-helm的value.yaml文件配置

参考：

image:
  registry: "自行填写"
  image: "自行填写"
  tag: "自行填写"
imagePullPolicy: IfNotPresent
imagePullSecrets:
  - name: "自行填写"
gitlabUrl: "自行填写"
runnerRegistrationToken: "自行填写"
# runnerToken: ""
# unregisterRunners: true
terminationGracePeriodSeconds: 3600
# certsSecretName:
concurrent: 10
checkInterval: 30
# logLevel:
# logFormat:
# sentryDsn:
preEntrypointScript: |
  echo "hello"
sessionServer:
  enabled: false
  # annotations: {}
  # timeout: 1800
  # internalPort: 8093
  # externalPort: 9000
  # publicIP: ""
  # loadBalancerSourceRanges:
  #   - 1.2.3.4/32

## For RBAC support:
rbac:
  create: true
  rules: []
  # - resources: ["configmaps", "pods", "pods/attach", "secrets", "services"]
  #   verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
  # - apiGroups: [""]
  #   resources: ["pods/exec"]
  #   verbs: ["create", "patch", "delete"]
  clusterWideAccess: false
  # serviceAccountName: default
  # serviceAccountAnnotations: {}
  podSecurityPolicy:
    enabled: false
    resourceNames:
      - gitlab-runner

  ## Specify one or more imagePullSecrets used for pulling the runner image
  ##
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account
  ##
  imagePullSecrets: []

metrics:
  enabled: false
  portName: metrics
  port: 9252
  serviceMonitor:
    enabled: false
    ## labels: {}
    # interval: ""
    # scheme: "http"
    # tlsConfig: {}
    # path: "/metrics"
    # metricRelabelings: []
    ## relabelings: []
service:
  enabled: false
  # labels: {}
  # annotations: {}
  # clusterIP: ""
  # externalIPs: []
  # loadBalancerIP: ""
  # loadBalancerSourceRanges: []
  type: ClusterIP
    # metrics:
  # nodePort: ""
  # additionalPorts: []

runners:
  # runner configuration, where the multi line strings is evaluated as
  # template so you can specify helm values inside of it.
  #
  # tpl: https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function
  # runner configuration: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
  # 见：1. https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerskubernetes-section
  #  2. https://docs.gitlab.com/runner/executors/kubernetes.html#overwriting-container-resources
  config: |
    [[runners]]
      cache_dir = "/cache"
      [runners.kubernetes]
        namespace = "自行填写"
        image = "自行填写"
        privileged = true
        image_pull_secrets = ["自行填写"]
        image_pull_policy = "if-not-present"
        helper_image = "自行填写"
        helper_cpu_limit = "1000m"
        helper_memory_limit = "2048Mi"
        helper_cpu_requests = "200m"
        helper_memory_requests = "512Mi"
        service_cpu_limit = "1000m"
        service_memory_limit = "2048Mi"
        service_cpu_requests = "200m"
        service_memory_requests = "512Mi"
        cpu_limit = "1000m"
        cpu_limit_overwrite_max_allowed = "5000m"
        memory_limit = "2048Mi"
        memory_limit_overwrite_max_allowed = "10240Mi"
        cpu_requests = "200m"
        cpu_requests_overwrite_max_allowed = "5000m"
        memory_requests = "512Mi"
        memory_requests_overwrite_max_allowed = "10240Mi"
        [[runners.kubernetes.volumes.host_path]]
          name = "docker"
          mount_path = "/var/run/docker.sock"
          read_only = true
          host_path = "/var/run/docker.sock"
        [[runners.kubernetes.volumes.pvc]]
           name = "gitlab-runner-cache-pvc"
           mount_path = "/cache"

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: false
  runAsNonRoot: true
  privileged: false
  capabilities:
    drop: ["ALL"]

podSecurityContext:
  runAsUser: 100
  # runAsGroup: 65533
  fsGroup: 65533
  # supplementalGroups: [65533]
  ## Note: values for the ubuntu image:
  # runAsUser: 999
  # fsGroup: 999

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
   limits:
     memory: 2048Mi
     cpu: 1000m
   requests:
     memory: 512Mi
     cpu: 200m

affinity: {}
nodeSelector: {}
tolerations: []
# envVars:
#   - name: RUNNER_EXECUTOR
#     value: kubernetes
hostAliases: []
  # Example:
  # - ip: "127.0.0.1"
  #   hostnames:
  #   - "foo.local"
  #   - "bar.local"
  # - ip: "10.1.2.3"
  #   hostnames:
  #   - "foo.remote"
#   - "bar.remote"

podAnnotations: {}
  # Example:
# iam.amazonaws.com/role: <my_role_arn>
podLabels: {}
  # Example:
  # owner.team: <my_cool_team>

priorityClassName: ""

secrets: []
  # Example:
  # - name: my-secret
  # - name: myOtherSecret
  #   items:
  #     - key: key_one
#       path: path_one

configMaps: {}

volumeMounts:
  - name: cache
    mountPath: /cache
volumes:
  - name: cache
    persistentVolumeClaim:
      claimName: gitlab-runner-cache-pvc

Q.E.D.

gitlab-ci-k8s-runner部署及缓存配置说明

gitlab-ci-k8s-runner部署及缓存配置说明

一 gitlab-ci-cache机制说明

1. Distributed runners caching

2. 给 Job Executor Pod 挂载同一个 volume

3. 不使用缓存，在 Kubernetes 集群中安装各类仓库的私服

二准备缓存pv、pvc

1.配置runner nfs缓存目录

2.创建pv、pvc

3. 创建imagePullSecret

4. 可选：将gitlab-runner、gitlab-runner-helper镜像上传到私有仓库

三根据官方helm生成k8s-runner部署文件

四附：docker-runner部署

运行

注册

重启

修改配置

五 value.yaml和pv-pvc文件参考

gitlab-runner-pv-pvc.yaml

gitlab-k8s-runner-helm的value.yaml文件配置

WSL开发系列-gui篇(wslg)

grafana第三方集成以及使用jwt(jwks)实现免登录访问

且炼时光

gitlab-ci-k8s-runner部署及缓存配置说明

一 gitlab-ci-cache机制说明

1. Distributed runners caching

2. 给 Job Executor Pod 挂载同一个 volume

3. 不使用缓存，在 Kubernetes 集群中安装各类仓库的私服

二 准备缓存pv、pvc

1.配置runner nfs缓存目录

2.创建pv、pvc

3. 创建imagePullSecret

4. 可选：将gitlab-runner、gitlab-runner-helper镜像上传到私有仓库

三 根据官方helm生成k8s-runner部署文件

四 附：docker-runner部署

运行

注册

重启

修改配置

五 value.yaml和pv-pvc文件参考

gitlab-runner-pv-pvc.yaml

gitlab-k8s-runner-helm的value.yaml文件配置

WSL开发系列-gui篇(wslg)

grafana第三方集成以及使用jwt(jwks)实现免登录访问

且炼时光

二准备缓存pv、pvc

三根据官方helm生成k8s-runner部署文件

四附：docker-runner部署