# swarm-ci-runner: Forgejo Actions Runner The `swarm-ci-runner` module runs the swarm's Forgejo Actions runner in a `swarm-ci-runner` nixos-container, executing CI jobs from `.forgejo/workflows/ci.yml` on every PR. It's a swarm service: it runs on whichever host sets it, and needs neither the forge nor a hive on that host. ## For operators **Enabling it's one line**: `services.hyperhive.deploy.ci-runner.enable = true` in the NixOS config of the host that should run it. No manual token provisioning: swarm-controller publishes the forge's runner registration token to the secret store, and the runner reads it with a store identity of its own. On the store's host `glue-bao-tls.nix` mints that identity; on any other host, copy `ci-runner.pem` and `ci-runner-key.pem` from the store host's `/var/lib/swarm-bao-pki/` and name them in `services.hyperhive.deploy.bao.ciRunnerClientCertFile` and `ciRunnerClientKeyFile`. Two things worth knowing before you rely on it: - **Only run CI for contributors you trust.** Builds run **unsandboxed** (an nspawn limitation, not a choice) — a malicious build script in a PR could make network requests or write to the container filesystem during the build. Fine for a small trusted-team hive where everyone already has forge access; if you take outside/fork contributions, gate CI behind Forgejo's fork-PR-approval setting or restrict the workflow to push-only triggers (forks can't push to your branches). See _Security_ below for the full threat model. - **Watch your disk.** CI builds through the host's nix store with no automatic garbage collection of its own — a busy CI day can fill `/nix/store` until every job fails with `ENOSPC`. Add the daily + disk-pressure GC config from _Host store maintenance_ below to your host's NixOS config (not optional if you plan to actually use this). Everything below this point is implementation detail: registration internals, container design, and the full security writeup. ## CI checks Several jobs run on every PR (and on `workflow_dispatch` for manual re-triggers), defined in [`.forgejo/workflows/ci.yml`](../../.forgejo/workflows/ci.yml). The first four below are required checks (forge branch protection) — a hit on any of them blocks merge. The two vale jobs split what used to be a single check: the error-only one exists so it _can_ become a required check without also gating on the full job's standing warning/suggestion backlog, but flipping that switch on the forge is a separate operator action, not something this table can assert ahead of it happening. | Job | What it runs | Required? | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | | **nix flake check** | treefmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation | yes | | **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) | yes | | **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) | yes | | **doc-pointer lint** | flags `docs/…md` paths and relative markdown links that no longer resolve (`scripts/check-doc-refs.sh`) | yes | | **attribution-trailer lint** | flags `Co-Authored-By` trailers with Claude/Anthropic attribution in PR commits (`scripts/check-attribution-trailers.sh`) | not yet — intended to be | | **prose lint (vale, errors)** | vale, `docs/` tree, error-severity rule hits only | not yet — intended to be | | **prose lint (vale)** | vale, `docs/` tree, full severity range (warnings + suggestions too) | no — advisory, standing backlog | `hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger without an empty commit. **`ci-rerun --pr` verifies the code but doesn't reliably move the PR's own status checks.** Measured directly (raw `GET /repos/.../actions/tasks` JSON): a genuine PR-triggered run carries `event: pull_request` with the `#` pseudo-ref as its `head_branch`; a `ci-rerun --pr`-dispatched run carries `event: workflow_dispatch` with the real branch name instead. Forgejo's PR commit-status tracking appears scoped to `pull_request`-event runs specifically, so a dispatched run — even a genuinely successful one — never writes to that status slot. Reproduced with three dispatches on one PR over 20+ minutes, all genuinely green, the PR's own status unmoved throughout. **Always re-check `pr status` after a dispatch and believe what it says — don't push a commit just to unwedge it, that's the empty-commit anti-pattern this verb exists to avoid.** If `hive-forge ci-log --run --job 0` shows the code green (verdict is the last line) but the status is stuck, ask the operator to select "rerun" in the forge web UI — it's CSRF-gated, so only they can do it. ### Running checks locally Don't run `nix flake check` directly — it dispatches to the shared build farm and wastes a remote-builder slot. Use the devshell equivalents instead: ```sh nix develop -c cargo clippy --all-targets -- -D warnings nix develop -c cargo test nix fmt # authoritative; the formatter set is nix/treefmt.nix sh scripts/check-issue-refs.sh # tracker-tag lint sh scripts/check-comment-blocks.sh # comment-block lint ``` `scripts/pre-push` provides a git pre-push hook that automates the two lint checks. Install it once per clone: ```sh ln -sf ../../scripts/pre-push .git/hooks/pre-push ``` After that, any `git push` automatically runs both lints and aborts with a diagnostic if either fails — catching the issue locally before CI sees it. Note that the hook does **not** run `cargo clippy` or `cargo test` (those are slow); run those manually before pushing Rust changes. ## Configuration reference `services.hyperhive.deploy.ci-runner.enable = true` on the host that runs it (see _For operators_ above). The runner reaches the forge at `https://`, so that name has to resolve from the host: the gateway's dnsmasq answers it locally where the forge runs, and forwards it to the operator's DNS everywhere else. Optional tuning under `services.hyperhive.deploy.ci-runner`: `name` (runner name in the forge admin panel), `concurrency` (parallel job capacity), `labels` (workflow targeting; the default `hive-ci:host` is what this repo's workflows name), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration string for example `"3h"` — a job that exceeds it's killed so a hung or runaway build can't hold the runner's single slot indefinitely). ## Container design - **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. - **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after. - **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`. - **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. ## Registration 1. **swarm-controller** (`swarm-controller/src/forge/runner_token.rs`) reads the forge's instance-wide token from `GET /api/v1/admin/runners/registration-token` at start and every five minutes, and writes it to the secret store at `swarm/services/ci-runner/registration-token` when it differs from the value it last wrote. Its store grant on that path is write-only. 2. **`swarm-bao-ci-runner.service`** on the runner's host logs in to the store with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path and nothing else, retrying every 30 seconds until the token is there. It writes `TOKEN=` to the runner's `EnvironmentFile` inside the container, and restarts the runner when the value changes. It runs at boot and on deploy, not on a timer. 3. **The runner unit** (`gitea-runner-swarm` in the container) registers in the upstream pre-start step whenever `.runner` is missing or the token or labels changed. A pre-start step of this module's, ahead of it, removes `.runner` in two cases upstream doesn't check: - The forge has no record of the runner. Forgejo answers every call from an unknown runner with `unregistered runner`, and the daemon exits 1 when its startup `Declare` gets that answer. On the next start the step finds the message in the daemon's journal since its last start, and registers again. A daemon that's already running when an admin deletes its runner keeps polling and logging the error without exiting; the next restart of the unit or container registers it again. - `.runner` names another forge address than the configured one. Until a token arrives the step fails, and `Restart=on-failure` retries every 15 seconds with no start limit. A new registration is a new runner on the forge: the previous entry stays in the forge admin panel as offline until an admin deletes it. ## Actions checkout mirror With `services.hyperhive.deploy.forgejo.actionsFromSelf` enabled, the swarm controller seeds an `actions/checkout` pull-mirror on the forge and Forgejo's `DEFAULT_ACTIONS_URL` points at the forge itself. CI `uses: actions/checkout@vN` steps then resolve on the forge, with no external DNS on the CI critical path. The option defaults to whether the forge's host also runs the swarm controller (`services.hyperhive.deploy.swarm-controller.enable`): on when it does, off otherwise. Setting it `true` on a forge host without the controller leaves `actions/checkout` unresolved, because nothing seeds the mirror; evaluation warns about it. The **swarm controller** seeds the mirror in its swarm-wide forge-objects pass (`swarm-controller/src/forge/objects.rs`), beside the orgs it ensures. The nix module forwards the effective mirror list as `SWARM_CONTROLLER_FORGE_MIRRORS` in the `swarm-controller` service environment (JSON-encoded `[{upstream, dest}]` list). It reads the list of the host the controller runs on: mirrors declared on any other host aren't seeded, and evaluation warns about them. **General-purpose mirrors**: you can pre-seed any external repo as a pull-mirror via `services.hyperhive.deploy.forgejo.mirrors`: ```nix services.hyperhive.deploy.forgejo.mirrors = [ { upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; } { upstream = "https://github.com/example/tool"; dest = "mirrors/tool"; } ]; ``` The controller creates each entry as a real Forgejo pull-mirror — not a one-off clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack` request), so a DNS blip during that sync will propagate back to the runner as a hard `git clone` failure. The controller autocreates the `` org in `dest`. Keep mirror dests out of the hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`) to avoid provisioning collisions. ## Security: unsandboxed builds and trusted contributors **The runner should only run CI for trusted contributors.** The security boundary is weaker than it looks: ### What unsandboxed builds mean nspawn containers can't create user-namespaces, so the module sets `nix.settings.sandbox-fallback = true` in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase. A malicious `default.nix` or build script in a PR can therefore: - **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic). - **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials. No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which reaches the container's env-file. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls. Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path. ### Mitigation For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway. For repos with external contributors or fork PRs: - Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR. - Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches. The current design is appropriate for a trusted-team hive where all contributors have implicit forge access. ## Host store maintenance (recommended) The CI runner builds derivations through the **host** nix-daemon — the `swarm-ci-runner` container shares the host store and has no daemon of its own. Build outputs accumulate in `/nix/store` with no automatic collection, and a busy CI day can fill the disk until every job fails fast with `ENOSPC`. Store GC is a **host-level** concern, so it belongs in the host's own NixOS configuration, not in the hyperhive service modules — a single service should not reach out and change the host's global nix-daemon options. Add the following to your host config: ```nix { # Daily GC: delete store paths not referenced by a live root and older # than a day. Keeps the store bounded between builds. # # `randomizedDelaySec` because a fixed daily time lands on the same wall # clock as whatever else runs nightly. A collection that overlaps a CI # job doesn't fail it — the runner builds through the host nix-daemon, # so an in-flight build's temp roots are protected — but the first build # after a collection rebuilds what the collection deleted, so a run that # lands just behind one pays for it in wall time. nix.gc = { automatic = true; dates = "daily"; randomizedDelaySec = "45min"; options = "--delete-older-than 1d"; }; # Disk-pressure GC: when free space drops below min-free mid-build, the # daemon collects garbage up to max-free before continuing. This is the # real-time net the daily timer can't provide — a same-day build burst is # what fills the disk. Tune to your disk size. nix.settings.min-free = 20 * 1024 * 1024 * 1024; # 20 GiB nix.settings.max-free = 50 * 1024 * 1024 * 1024; # 50 GiB } ``` **Measure a collection on your own hosts before relying on that stagger.** `randomizedDelaySec` spreads the _start_, not the run, and the sweep over unused `.links` entries dominates the run — which scales with the size of the store, not with what this collection deletes. On a large store it can take most of an hour, so two hosts both set to `daily` may overlap however much jitter you give them. `journalctl -u nix-gc.service` reports each run's wall clock. **Remote builders:** if CI dispatches builds to a remote builder (for example via `nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_ store, so apply the same GC config wherever the builder runs — GC on the coordinator host won't reclaim space on the builder. ⚠️ Two stores also means the temp-root protection above **doesn't span them**. Each daemon holds roots only for the builds it runs, so a collection on the _builder_ knows nothing about a coordinator build waiting on a path there. Stagger the two hosts' GC windows rather than giving them the same `dates`, and prefer `min-free`/`max-free` on the builder — disk-pressure collection at least runs while nothing is mid-copy. ## References - `nix/host-modules/swarm-ci-runner.nix`: runner configuration, token fetch, re-registration step, container setup. - `swarm-controller/src/forge/runner_token.rs`: publishes the registration token. - `.forgejo/workflows/ci.yml`: workflow definition. - `docs/process/gotchas.md`: nix sandboxing limitations in containers.