Watch
0
0
Fork
You've already forked hyperhive
0
hyperhive/docs/4-implementation/scheduler/ci.md
iris a19745d30d docs: the repo's own CI checks move from the CI runner page to the project process docs
docs/4-implementation/scheduler/ci.md described both the swarm-ci-runner product (hosting, registration, security, store maintenance) and this repo's own pipeline (which jobs run on a PR, the local equivalents, the nightly VM tests). The second part is project process: it moves, whole and in order, to docs/5-project/process/ci-checks.md, with a pointer left behind, an entry in the docs index and the one doc comment in hive-forge that named the old section updated.
2026-11-20 02:15:46 +01:00

19 KiB

swarm-ci-runner: Forgejo Actions Runner

The swarm-ci-runner module runs the swarm's Forgejo Actions runner in a swarm-ci-runner nixos-container, executing CI jobs from .forgejo/workflows/ci.yml on every PR. It's a swarm service: it runs on whichever host sets it, and needs neither the forge nor a hive on that host.

For operators

Enabling it's one line: services.hyperhive.deploy.ci-runner.enable = true in the NixOS config of the host that should run it. No manual token provisioning: swarm-controller publishes the forge's runner registration token to the secret store, and the runner reads it with a store identity of its own. On the store's host glue-bao-tls.nix mints that identity; on any other host, copy ci-runner.pem and ci-runner-key.pem from the store host's /var/lib/swarm-bao-pki/ and name them in services.hyperhive.deploy.bao.ciRunnerClientCertFile and ciRunnerClientKeyFile.

Two things worth knowing before you rely on it:

  • Only run CI for contributors you trust. Builds run unsandboxed (an nspawn limitation, not a choice) — a malicious build script in a PR could make network requests or write to the container filesystem during the build. Fine for a small trusted-team hive where everyone already has forge access; if you take outside/fork contributions, gate CI behind Forgejo's fork-PR-approval setting or restrict the workflow to push-only triggers (forks can't push to your branches). See Security below for the full threat model.
  • Watch your disk. CI builds through the host's nix store with no automatic garbage collection of its own — a busy CI day can fill /nix/store until every job fails with ENOSPC. Add the daily + disk-pressure GC config from Host store maintenance below to your host's NixOS config (not optional if you plan to actually use this).

Everything below this point is implementation detail: registration internals, container design, and the full security writeup.

This repo's own checks

Which jobs run on this repo's pull requests, how to run them locally and how the nightly VM tests work are project process, in the repo's CI checks. This page covers the runner itself.

Configuration reference

services.hyperhive.deploy.ci-runner.enable = true on the host that runs it (see For operators above). The runner reaches the forge at https://<swarm.forge.domain>, so that name has to resolve from the host: the gateway's dnsmasq answers it locally where the forge runs, and forwards it to the operator's DNS everywhere else.

Optional tuning under services.hyperhive.deploy.ci-runner: name (runner name in the forge admin panel), concurrency (parallel job capacity), labels (workflow targeting; the default hive-ci:host is what this repo's workflows name), jobTimeout (per-job wall-clock cap, default "1h", Go duration string for example "3h" — a job that exceeds it's killed so a hung or runaway build can't hold the runner's single slot indefinitely).

The old services.hyperhive.deploy.forgejo.ci.{enable,name,concurrency,labels,package,jobTimeout} and services.hyperhive.swarm.forge.ci.{enable,name,concurrency,labels,package} are still accepted: each forwards to the same-named option under deploy.ci-runner, and evaluation warns naming both paths. Set the new name to silence the warning. The two name defaults differ (hive-ci before, swarm-ci-runner now), so a host that never set the old name shows up under the new default.

The forwarded enable carries the new module's requirements, not the old module's: evaluation refuses a host that sets only the old enable unless services.hyperhive.deploy.bao.ciRunnerClientCertFile and ciRunnerClientKeyFile are both set. They default to the identity glue-bao-tls.nix mints when services.hyperhive.deploy.bao.enable is true; elsewhere place them as described under For operators. The runner no longer needs deploy.forgejo.enable or a hive on its host.

Container design

  • Private netns, bridge-attached: the container runs in its own network namespace (privateNetwork = true, hostBridge), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at 127.0.0.1:7000 and the raw forge port are unreachable from CI.
  • Non-ephemeral: runner credentials (.runner) persist across restarts, written to the container's state dir on registration and reused after.
  • Sandbox fallback: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets nix.settings.sandbox-fallback = true in the container — nix builds run unsandboxed (safe because the container is already isolated). See docs/5-project/process/gotchas.md.
  • Credential isolation: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own .runner, and the host unit that fetches the token holds a store identity that reads that one path. Only swarm-ci-runner-register.service loads the registration token; the daemon unit and the jobs it runs have it neither in their environment nor among their credentials.

Registration

  1. swarm-controller (swarm-controller/src/forge/runner_token.rs) reads the forge's instance-wide token from GET /api/v1/admin/runners/registration-token at start and every five minutes, and writes it to the secret store at swarm/services/ci-runner/registration-token when it differs from the value it last wrote. Its store grant on that path is write-only.

  2. swarm-bao-ci-runner.service on the runner's host logs in to the store with the ci-runner leaf, whose swarm-ci-runner role reads that one path and nothing else, retrying every 30 seconds until the token is there. It writes the token to a root-only file in /run/swarm-ci-runner on the host, which the container sees read-only, and restarts the runner when the value changes. /run is tmpfs, so no copy reaches persistent disk; after a reboot the unit fetches the token again, and until the store answers only a runner that has no .runner yet waits. It runs at boot, on deploy, and every ten minutes on a timer.

  3. swarm-ci-runner-register.service in the container is the only unit that loads that file, with LoadCredential=. systemd hands a unit's credentials to all of its processes, and for a host label act_runner runs every job step as a process of the daemon with the daemon's whole environment, so the daemon unit carries the token in neither. The register unit is a oneshot under the daemon's dynamic user and state directory. It reads the token from $CREDENTIALS_DIRECTORY and runs act_runner register only when .runner is missing, so the .runner it writes is the daemon's own. The runner unit (gitea-runner-swarm) starts it in a pre-start step and runs the daemon once it succeeds. A root step of the register unit, ahead of registering, removes .runner in three cases:

    • The forge has no record of the runner. Forgejo answers every call from an unknown runner with unregistered runner, and the daemon exits 1 when its startup Declare gets that answer. On the next start the step finds the message in the daemon's journal since its last start, and registers again. A daemon that's already running when an admin deletes its runner keeps polling and logging the error without exiting; the next restart of the unit or container registers it again.
    • .runner names another forge address than the configured one.
    • .runner names other labels than the configured ones, compared in the form the daemon writes back: <name>:<scheme>, with host for a label that names no scheme.

    A changed token alone doesn't register again: the daemon authenticates with the per-runner token in .runner. Without .runner and with no token yet the register unit fails and the runner unit's pre-start step with it; Restart=on-failure on the runner unit starts both again every 15 seconds, with no start limit.

    Host jobs run as the runner's own user, so they can still read .runner and act as this runner; see the security section below.

A new registration is a new runner on the forge: the previous entry stays in the forge admin panel as offline until an admin deletes it. The runner the old hive-ci module registered is one such entry, since this runner registers under its own name and token. Delete it under Site administration, Actions, Runners.

Actions mirrors

With services.hyperhive.deploy.forgejo.actionsFromSelf enabled, the swarm controller seeds actions/checkout and actions/upload-artifact pull-mirrors (from github.com/actions/*) on the forge and Forgejo's DEFAULT_ACTIONS_URL points at the forge itself. CI uses: actions/<name>@vN steps then resolve on the forge, with no external DNS on the CI critical path. A workflow that names any other bare actions/* action fails at job setup with "repository not found" until that action joins the list in nix/host-modules/hive-forge/default.nix.

A forge host that runs the swarm controller therefore resolves those actions from itself, with or without a runner on that host. Set actionsFromSelf = false there to resolve actions from data.forgejo.org and seed no mirror.

The option defaults to whether the forge's host also runs the swarm controller (services.hyperhive.deploy.swarm-controller.enable; set by glue-forge-controller.nix): on when it does, off otherwise. Setting it true on a forge host without the controller leaves the actions unresolved, because nothing seeds the mirrors; evaluation warns about it.

The swarm controller seeds the mirror in its swarm-wide forge-objects pass (swarm-controller/src/forge/objects.rs), beside the orgs it ensures. glue-forge-controller.nix forwards the effective mirror list as SWARM_CONTROLLER_FORGE_MIRRORS in the swarm-controller service environment (JSON-encoded [{upstream, dest}] list). It reads the list of the host the controller runs on: mirrors declared on any other host aren't seeded, and evaluation warns about them.

The hyperhive mirror: the same list carries hyperhive/hyperhive, from services.hyperhive.deploy.forgejo.hyperhiveUpstream (default https://forge.darkest.space/hyperhive/hyperhive on a host that runs the controller, unset elsewhere). The controller creates the mirror only when hyperhive/hyperhive doesn't exist, and it leaves a repo that exists but is not a mirror unchanged. deploy.forgejo.mirrors takes precedence for the same dest.

General-purpose mirrors: you can pre-seed any external repo as a pull-mirror via services.hyperhive.deploy.forgejo.mirrors:

services.hyperhive.deploy.forgejo.mirrors = [
  { upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
  { upstream = "https://github.com/example/tool";    dest = "mirrors/tool"; }
];

The controller creates each entry as a real Forgejo pull-mirror — not a one-off clone. Forgejo re-syncs the mirror on every pull (git-upload-pack request), so a DNS blip during that sync will propagate back to the runner as a hard git clone failure. The controller autocreates the <owner> org in dest. Keep mirror dests out of the hive-c0re-managed namespaces (config/, shared/, agents/, core/) to avoid provisioning collisions.

Security: unsandboxed builds and trusted contributors

The runner should only run CI for trusted contributors. The security boundary is weaker than it looks:

What unsandboxed builds mean

nspawn containers can't create user-namespaces, so the module sets nix.settings.sandbox-fallback = true in the container. This means every nix build (and nix flake check) runs without a build sandbox — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.

A malicious default.nix or build script in a PR can therefore:

  • Make arbitrary network requests to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at https://<forge.domain> (no admin credentials) and can't reach host-loopback services: the unauthenticated core dashboard at 127.0.0.1:7000 and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
  • Write to the container filesystem, including corrupting the runner's state dir or .runner credentials.

No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only swarm-ci-runner-register.service loads, so a job step can't read it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read .runner and act as this runner; it can't issue admin API calls.

Note: nix flake check --no-build (eval-only) reduces the exposure but doesn't eliminate it — builtins.fetchGit, builtins.fetchurl, and import-from-derivation can reach the network and filesystem during evaluation. The CI workflow's nix flake check job builds every check derivation, which is the higher-risk path.

Mitigation

For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.

For repos with external contributors or fork PRs:

  • Use Forgejo's fork PR approval workflow (repository.settings → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
  • Or restrict the CI workflow trigger to push events on branches (not pull_request from forks) — forks can't push to upstream branches.

The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.

The CI runner builds derivations through the host nix-daemon — the swarm-ci-runner container shares the host store and has no daemon of its own. Build outputs accumulate in /nix/store with no automatic collection, and a busy CI day can fill the disk until every job fails fast with ENOSPC.

Store GC is a host-level concern, so it belongs in the host's own NixOS configuration, not in the hyperhive service modules — a single service should not reach out and change the host's global nix-daemon options. The module only warns at nixos-rebuild time when the host sets neither nix.gc.automatic nor nix.settings.min-free. Add the following to your host config:

{
  # Daily GC: delete store paths not referenced by a live root and older
  # than a day. Keeps the store bounded between builds.
  #
  # `randomizedDelaySec` because a fixed daily time lands on the same wall
  # clock as whatever else runs nightly. A collection that overlaps a CI
  # job doesn't fail it — the runner builds through the host nix-daemon,
  # so an in-flight build's temp roots are protected — but the first build
  # after a collection rebuilds what the collection deleted, so a run that
  # lands just behind one pays for it in wall time.
  nix.gc = {
    automatic = true;
    dates = "daily";
    randomizedDelaySec = "45min";
    options = "--delete-older-than 1d";
  };

  # Disk-pressure GC: when free space drops below min-free mid-build, the
  # daemon collects garbage up to max-free before continuing. This is the
  # real-time net the daily timer can't provide — a same-day build burst is
  # what fills the disk. Tune to your disk size.
  nix.settings.min-free = 20 * 1024 * 1024 * 1024;  # 20 GiB
  nix.settings.max-free = 50 * 1024 * 1024 * 1024;  # 50 GiB
}

Measure a collection on your own hosts before relying on that stagger. randomizedDelaySec spreads the start, not the run, and the sweep over unused .links entries dominates the run — which scales with the size of the store, not with what this collection deletes. On a large store it can take most of an hour, so two hosts both set to daily may overlap however much jitter you give them. journalctl -u nix-gc.service reports each run's wall clock.

Remote builders: if CI dispatches builds to a remote builder (for example via nix.buildMachines / ssh-ng://), the build outputs land in that host's store, so apply the same GC config wherever the builder runs — GC on the coordinator host won't reclaim space on the builder.

⚠️ Two stores also means the temp-root protection above doesn't span them. Each daemon holds roots only for the builds it runs, so a collection on the builder knows nothing about a coordinator build waiting on a path there. Stagger the two hosts' GC windows rather than giving them the same dates, and prefer min-free/max-free on the builder — disk-pressure collection at least runs while nothing is mid-copy.

References

  • nix/host-modules/swarm-ci-runner.nix: runner configuration, token fetch, re-registration step, container setup.
  • swarm-controller/src/forge/runner_token.rs: publishes the registration token.
  • .forgejo/workflows/ci.yml: workflow definition.
  • docs/5-project/process/gotchas.md: nix sandboxing limitations in containers.

Host setup

One Forgejo Actions runner, in a swarm-ci-runner container on the host that sets deploy.ci-runner.enable. It doesn't derive from allSwarmServices: a runner executes pull-request code, so running one is a decision of its own.

It needs neither the forge nor a hive on its host. It reaches the forge at https://<swarm.forge.domain>, and registers with the token swarm-controller publishes to the secret store, which it reads under its own ci-runner leaf. On a host without the store, that leaf is the one file to copy.

The runner's built-in Prometheus endpoint is on, bound to 127.0.0.1:9101 in the container's own network namespace. The container's journal forwarder scrapes it (services.hyperhive.swarmJournalForwarder.extraScrapeTargets) and pushes the series to the swarm collector under the job gitea-runner, so nothing listens on the bridge. Series, all prefixed gitea_runner_: job_running, capacity, job_capacity_utilization_ratio, uptime_seconds, info (labels version and name), job_total{status}, job_duration_seconds, poll_fetch_total{result}, poll_fetch_duration_seconds, poll_backoff_seconds, report_log_*, report_state_* and client_errors_total. A pull-request job runs in the same namespace and can read the endpoint, which carries counts, durations, the runner's name and its version, and no secrets.

The ci runner board in Grafana shows these series. Its Runner filter (the series' service.instance.id, the scrape address) narrows every panel, and uptime, job slots and slot use show one series per runner, so a runner registered by hand appears beside the swarm's once its metrics reach the store under its own instance id. The forge's CI runs tab is the other view: run times and queue wait of every run, whichever runner ran it.