Job Queue in GWS

Table of Contents

When you commit something to GWS-backed git server, it’s possible to run jobq, which is an integrated post-receive hook, which enqueues jobs in a special FIFO (sidenote: first-in-first-out) queue. You can configure one queue per repository, or a queue shared between many repositories, or a mix of those, and you don’t have to worry about DDoS-ing your git server, even on low-end machines. This makes it suitable for a lightweight CI/CD system or a master node which notifies specialised workers.

Gitolite Configuration

To enable a job queue for a repository, we must enable a jobq post-receive hook and configure some options:

  • JOBQ_COMMAND: command which will run when post-receive hook triggers
  • JOBQ_REF_FILTER: list of git ref names that are allowed to trigger the command
  • JOBQ_QUEUE: name of the queue

On top of those, we can run jobs synchronously or asynchronously, i.e. they might block committer’s console or run in the background: an option controlled by JOBQ_ASYNC.

The most common scenario is to run a command in the background for master branch and all tags. Most repositories store their particular deployment instruction sets inside the repository itself and we’re going to suppose that they’re stored in a Makefile.ci file, which is the ordinarey makefile.

Gitolite configuration in conf/gitolite.conf will look like this:

repo mygitrepo
    option hook.post-receive = jobq
    option ENV.JOBQ_QUEUE = "deployment"
    option ENV.JOBQ_ASYNC = "1"
    option ENV.JOBQ_COMMAND = "run-deployment"
    option ENV.JOBQ_REF_FILTER = "refs/heads/master refs/tags/*"

Helper script: run-deployment

We will run run-deployment helper script, which will prepare a repository and do rudimentary discovery.

All commands specified in JOBQ_COMMAND must be visible by GWS, which basically means that we must put them in GWS’ $PATH. To do so, we’re going to create a derivative GWS image and copy the script to /usr/local/bin.

FROM docker.io/mgoral/gws:0.4.1
USER root:root
RUN mkdir -p /usr/local/bin
COPY run-deployment /usr/local/bin/run-deployment
USER git:git

Git post-receive hooks run inside bare git repositories. You won’t see ordinary files in bare repositories, so to see a makefile and run make, we must first:

  1. checkout the repository in the correct revision to a temporary location
  2. enter that location
  3. verify that Makefile.ci exists
  4. run make
  5. remove temporary files
#!/usr/bin/bash

set -o errexit
set -o nounset
set -o pipefail

tmpd=$(mktemp -d)

# trap will automatically remove temporary directory as soon as this script
# exits. Note the interesting "quoting hack".
trap "rm -rf '${tmpd}'" EXIT

# Log commands which Bash runs: this will make our life easier when
# debugging failed jobs
set -x

git --work-tree="${tmpd}" --git-dir="${GIT_DIR}" checkout --force "${GIT_REV_NEW}"
cd "${tmpd}"

if [[ -e Makefile.ci ]]; then
    make --file=Makefile.ci deploy
fi

Now whenever we push a new commit to master branch or release a new tag, GWS will automatically schedule above script to run.

To see how our job is doing, we could examine the job queue for the repo by running ssh <gitserver> jobq mygitrepo. Or we could see logs of specific job by passing its ID: ssh <gitserver> jobq mygitrepo 0. Accessing logs of still running jobs will attach to their output in real time. Neat!

Podman Inside Podman

Running bare make is so 90’s. Everyone use containers now.

It is possible to run a container from within GWS container, a technique known as “Podman inside Podman”, (sidenote: More details about this technique is in the Dan Walsh’s blog post.) but we must prepare our Podman image for this.

The original GWS image runs on top of Debian. It differs from official Podman images, which are built on top of Fedora. Fedora does a few extra things by itself, so our Containerfile will have some extra steps:

  1. we must install podman and all necessary dependencies, including libcap2-bin, which provides setcap command. We don’t uninstall it, because doing so could unintentionally uninstall some optional GWS’ dependencies (like nginx)
  2. we must drop setuid bit and grant CAP_SETUID/CAP_SETGID as file capabilities
  3. we must add a lot of IDs to subuid/subgid maps

Containerfile

FROM docker.io/mgoral/gws:0.4.1
USER root:root
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
    podman \
    slirp4netns \
    fuse-overlayfs \
    uidmap \
    containers-storage \
    libcap2-bin \
        && rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/*

RUN chmod 0755 /usr/bin/newuidmap /usr/bin/newgidmap && \
    setcap cap_setuid=ep /usr/bin/newuidmap && \
    setcap cap_setgid=ep /usr/bin/newgidmap

RUN echo "git:1:999" > /etc/subuid && \
    echo "git:1001:64535" >> /etc/subuid && \
    echo "git:1:999" > /etc/subgid && \
    echo "git:1001:64535" >> /etc/subgid

COPY containers.conf /etc/containers/containers.conf
COPY podman-containers.conf /config/.config/containers/containers.conf

RUN sed -e 's|^#mount_program|mount_program|g' \
        -e '/additionalimage.*/a "/var/lib/shared",' \
        -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' \
        /usr/share/containers/storage.conf > /etc/containers/storage.conf

ENV _CONTAINERS_USERNS_CONFIGURED=""

RUN mkdir -p /usr/local/bin
COPY run-deployment /usr/local/bin/run-deployment

USER git:git

This Containerfile uses few more configuration files, which are shamelessly copied from the official Podman image repository and pasted here for reference.

containers.conf

[containers]
netns="host"
userns="host"
ipcns="host"
utsns="host"
cgroupns="host"
cgroups="disabled"
log_driver = "k8s-file"
[engine]
cgroup_manager = "cgroupfs"
events_logger="file"
runtime="crun"

podman-containers.conf

[containers]
volumes = [
    "/proc:/proc",
    ]
    default_sysctls = []

With modified image we can change run-deployment script to run podman instead of make:

podman run \
    --env "GIT_REF" \
    --env "GL_REPO" \
    --volume "${tmpd}:/source" \
    --workdir=/source \
    --timeout 600 \
    --rm \
    ubuntu \
    make -f Makefile.ci deploy

To give us more flexibility, instead of hardcoding image name (“ubuntu”) and command, we could pass them to run-deployment from Gitolite configuration. This way we can run different deployment strategies for different repositories. All we have to do is to handle arguments in run-deployment script:

declare -r image=$1; shift
podman run ... "${image}" "$@"