Forgejo Actions Runner Management

Purpose

This document is a Standard Operating Procedure for adding, removing, and managing Forgejo Actions runners on Fedora Forge.

Scope

This SOP applies to Fedora Infrastructure team members who manage Forgejo Actions runners on forge.fedoraproject.org and forge.stg.fedoraproject.org.

Overview

Forgejo Actions runners execute lightweight workflows on Fedora Forge. Runners are defined in the ansible repo, registered in the Forgejo application via its CLI, and run as podman containers on a KubeVirt VM (forgejo-runnerhost-vm) in the forgejo OpenShift namespace.

This document covers Fedora-specific runner management. For detailed runner configuration options, refer to the official Forgejo Actions documentation.

Architecture

  Ansible repo                      Private repo (batcave01)
  ============                      ========================
  runners/<env>/<name>.yml              forgejo_runner_secrets:
  (name, scope, labels, image)            <name>: "<40-char hex>"
            \                                    /
             \                                  /
              v                                v
        load-forgejo-runners.yaml  ──>  forgejo_runners[]
                    |
          ┌─────────┼──────────────────────────┐
          v         v                          v
    K8s Secret    Control host              Forgejo pod
   forgejo-       forgejo-runners-          forgejo-cli
   runner-config   vars.yaml               actions register
          |                                    |
          v                                    v
    Runner host VM                     Runner appears in
   (watcher syncs secret               Forgejo admin UI
    to local file)                     with correct labels
          |
          v
    ansible-role-forgejo-runner
   (ansible-pull on boot, then
    starts runner containers)

Key components

Component Location Purpose

Runner definitions

roles/openshift-apps/forgejo/runners/{production,staging}/

Public config (name, scope, labels, image)

Runner secrets

/srv/private/ansible/vars.yml on batcave01

Registration secrets (40-char hex)

Forgejo playbook

playbooks/openshift-apps/forgejo.yml

Orchestrates everything

Runner host VM

KubeVirt VM forgejo-runnerhost-vm in forgejo namespace

Runs all runners as unprivileged systemd user services (rootless podman containers)

External runner role

codeberg.org/fedora/ansible-role-forgejo-runner

Configures runner daemons on the VM

Adding a new runner

Step 1: Create the runner definition file

Create a YAML file in roles/openshift-apps/forgejo/runners/<env>/ where <env> is production or staging. The filename must match the name field.

Simple label example (runners/production/roadrunner-1.yml):

---
name: roadrunner-1
scope: roadrunner
labels:
  - meep-meep
image: code.forgejo.org/forgejo/runner:12

Custom image mapping example (runners/production/roadrunner-2.yml):

---
name: roadrunner-2
scope: roadrunner
labels:
  - "docker:docker://node:22-bookworm"
  - "docker-slim:docker://node:22-alpine"
image: code.forgejo.org/forgejo/runner:12

With capacity (optional, limits concurrent jobs):

---
name: roadrunner-3
scope: roadrunner
labels:
  - meep-meep
image: code.forgejo.org/forgejo/runner:12
capacity: 4

Field reference

Field Required Description

name

Yes

Unique runner name. Must match the filename (without .yml).

scope

Yes

Forgejo organization name for org-scoped runners, or global for instance-wide. On the production instance the standard is org-scoped runners.

labels

Yes

List of workflow runs-on labels. Quote values containing colons.

image

Yes

Runner container image (e.g., code.forgejo.org/forgejo/runner:12).

capacity

No

Maximum concurrent jobs for this runner.

See also the Forgejo Runner Configuration example for details on label types and runner options.

Label format

Labels can be simple names or include an image mapping:

  • Simple: podman, docker, fedora — the runner uses its default container image.

  • With image: "docker:docker://node:22-bookworm" — the part before : is the label name (what workflows use in runs-on), the part after is the container image the runner uses for that label.

During registration in the Forgejo application, only the label name (before :) is used. The full label:image mapping is passed to the runner daemon configuration.

Step 2: Add the registration secret to private vars

On batcave01, edit /srv/private/ansible/vars.yml and add the runner’s secret to the appropriate dict.

Generate a new secret:

openssl rand -hex 20

Add it to the private vars:

# For production
forgejo_runner_secrets:
  roadrunner-1: "a1b2c3d4e5f6..."   # 40-char hex

# For staging
forgejo_stg_runner_secrets:
  roadrunner-1: "f6e5d4c3b2a1..."

Step 3: Commit and deploy

# Commit the runner definition (public repo)
cd /srv/web/infra/ansible
git add roles/openshift-apps/forgejo/runners/<env>/<name>.yml
git commit -m "forgejo: add <name> runner for <env>"
git push

# Run the playbook
sudo rbac-playbook openshift-apps/forgejo.yml -l <limit>

Where <limit> is:

  • staging for staging runners

  • production for production runners

What the playbook does automatically

  1. Loads runner definitions — scans runners/<env>/*.yml, merges each with its secret from private vars.

  2. Renders runner config — produces forgejo-runners-vars.yaml containing the Forgejo instance URL, full runner list (with secrets and label:image mappings), and Zabbix agent config.

  3. Updates K8s Secret — deploys the rendered config as forgejo-runner-config Secret in the forgejo namespace.

  4. Registers runners in Forgejo — execs forgejo-cli actions register inside the Forgejo application pod for each runner (idempotent with --secret). Label names are extracted (stripping :image suffixes).

  5. Watcher syncs to VM — the watcher service on forgejo-runnerhost-vm detects the Secret update and writes the config to /home/<user>/forgejo-runners-vars.yaml.

  6. Runner daemon picks up config — the ansible-role-forgejo-runner on the VM reads the updated config and starts/restarts runner containers.

Removing a runner

Step 1: Delete the runner definition file

git rm roles/openshift-apps/forgejo/runners/<env>/<name>.yml
git commit -m "forgejo: remove <name> runner from <env>"
git push

Step 2: Run the playbook

sudo rbac-playbook openshift-apps/forgejo.yml -l <limit>

This updates the K8s Secret (the runner will no longer be in the config), and the watcher syncs the change to the VM. The runner daemon stops the removed runner’s container.

Step 3: Remove from the Forgejo application (manual)

The playbook does not automatically deregister runners from Forgejo. Remove it via the Forgejo admin UI:

  1. Go to Site Administration > Actions > Runners.

  2. Find the runner by name.

  3. Delete it.

Step 4: Clean up the secret (optional)

Remove the runner’s secret from /srv/private/ansible/vars.yml on batcave01.

Modifying a runner

To change labels, image, scope, or capacity:

  1. Edit the runner definition file in runners/<env>/<name>.yml.

  2. Commit and push.

  3. Run the playbook: sudo rbac-playbook openshift-apps/forgejo.yml -l <limit>

  4. If the watcher doesn’t pick up the change, restart it on the VM (see Troubleshooting).

If you change a runner’s labels, the forgejo-cli actions register command with the same --secret will update the existing registration. Workflows using the old label will stop matching; workflows using the new label will start matching.

Runner host VM operations

Accessing the VM console

virtctl console forgejo-runnerhost-vm -n forgejo

Checking runner status on the VM

# Check if runner containers are running
podman ps

# Check the watcher service
systemctl --user list-units | grep -iE 'watch|secret|sync|forgejo'

# Check ansible-pull log (from initial boot)
cat ~/ansible-pull.log

# Check the synced config
cat ~/forgejo-runners-vars.yaml

Restarting the VM

# Restart (preserves data volume, cloud-init does NOT re-run)
virtctl restart forgejo-runnerhost-vm -n forgejo

# Full rebuild (deletes data volume, cloud-init runs fresh)
oc delete vm forgejo-runnerhost-vm -n forgejo
# Then re-run the playbook to recreate
sudo rbac-playbook openshift-apps/forgejo.yml -l <limit>

Re-running ansible-pull on the VM

If you need to re-apply the external runner role without rebuilding the VM:

ansible-pull -U https://codeberg.org/fedora/ansible-role-forgejo-runner.git -C main \
  playbooks/ansible-pull.yml >> ~/ansible-pull.log 2>&1

Troubleshooting

Runner not appearing in Forgejo UI

  1. Check if the playbook ran the registration task successfully:

    sudo rbac-playbook openshift-apps/forgejo.yml -l <limit> -v

  2. Verify the runner’s secret matches between the definition and private vars.

  3. Check if the Forgejo pod is running:

    oc get pods -n forgejo -l app.kubernetes.io/name=forgejo

Runner registered but not picking up jobs

  1. Check if the runner daemon is running on the VM:

    virtctl console forgejo-runnerhost-vm -n forgejo
podman ps

  2. Verify the config was synced to the VM:

    cat ~/forgejo-runners-vars.yaml | grep -A5 <runner-name>

  3. Verify labels match between the runner definition, Forgejo UI, and the workflow runs-on.

K8s Secret updated but VM config is stale

The watcher service may have lost its watch connection. Restart it:

# On the VM
systemctl --user restart forgejo-runner-config-watcher.service

# Verify the file was updated
cat ~/forgejo-runners-vars.yaml | grep -A5 <runner-name>

Cloud-init failures on VM boot

Cloud-init only runs on first boot. Common issues:

  • Home directory owned by root: Fixed by defer: true on write_files and chown -R in runcmd.

  • ansible-pull fails: Check ~/ansible-pull.log on the VM.

  • Packages fail to install: Check /var/log/cloud-init-output.log on the VM.

Non-breaking space characters in runner secrets

If runner definitions were copy-pasted from web pages, they may contain invisible \xa0 (non-breaking space) characters. Detect with:

grep -rP '\xc2\xa0' /srv/private/ansible/vars.yml

Fix by retyping the affected lines or:

sed -i 's/\xc2\xa0/ /g' /srv/private/ansible/vars.yml

Reference

Playbook command

# Staging
sudo rbac-playbook openshift-apps/forgejo.yml -l staging

# Production
sudo rbac-playbook openshift-apps/forgejo.yml -l production

File locations on batcave01

Path Purpose

/srv/web/infra/ansible/roles/openshift-apps/forgejo/runners/

Runner definitions

/srv/private/ansible/vars.yml

Runner secrets

/srv/web/infra/ansible/playbooks/openshift-apps/forgejo.yml

Playbook

File locations on runner host VM

Path Purpose

~/forgejo-runners-vars.yaml

Synced runner config (from K8s Secret)

~/ansible-pull.log

ansible-pull output from boot

/mnt/sa-token/

Mounted SA token for K8s API access

Kubernetes resources in forgejo namespace

Resource Name Purpose

Secret

forgejo-runner-config

Runner config (synced to VM)

Secret

forgejo-runner-vm-token

SA token for VM K8s API access

ServiceAccount

forgejo-runner-vm

VM identity

Role

forgejo-runner-config-reader

Read secrets permission

RoleBinding

forgejo-runner-config-reader

Binds SA to Role

VirtualMachine

forgejo-runnerhost-vm

Runner host VM

Service

forgejo-runnerhost

Headless service for VM DNS

Want to help? Learn how to contribute to Fedora Docs ›