hyperhive/docs/scheduler/ci.md

# swarm-ci-runner: Forgejo Actions Runner

The `swarm-ci-runner` module runs the swarm's Forgejo Actions runner in a
`swarm-ci-runner` nixos-container, executing CI jobs from
`.forgejo/workflows/ci.yml` on every PR. It's a swarm service: it runs on
whichever host sets it, and needs neither the forge nor a hive on that host.

## For operators

**Enabling it's one line**: `services.hyperhive.deploy.ci-runner.enable = true`
in the NixOS config of the host that should run it. No manual token
provisioning: swarm-controller publishes the forge's runner registration token
to the secret store, and the runner reads it with a store identity of its own.
On the store's host `glue-bao-tls.nix` mints that identity; on any other host, copy
`ci-runner.pem` and `ci-runner-key.pem` from the store host's
`/var/lib/swarm-bao-pki/` and name them in
`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
`ciRunnerClientKeyFile`.

Two things worth knowing before you rely on it:

- **Only run CI for contributors you trust.** Builds run **unsandboxed**
  (an nspawn limitation, not a choice) — a malicious build script in a
  PR could make network requests or write to the container filesystem
  during the build. Fine for a small trusted-team hive where everyone
  already has forge access; if you take outside/fork contributions,
  gate CI behind Forgejo's fork-PR-approval setting or restrict the
  workflow to push-only triggers (forks can't push to your branches).
  See _Security_ below for the full threat model.
- **Watch your disk.** CI builds through the host's nix store with no
  automatic garbage collection of its own — a busy CI day can fill
  `/nix/store` until every job fails with `ENOSPC`. Add the daily +
  disk-pressure GC config from _Host store maintenance_ below to your
  host's NixOS config (not optional if you plan to actually use this).

Everything below this point is implementation detail: registration
internals, container design, and the full security writeup.

## CI checks

<!-- vale write-good.Passive = NO -->

Several jobs run on every PR (and on `workflow_dispatch` for manual
re-triggers), defined in
[`.forgejo/workflows/ci.yml`](../../.forgejo/workflows/ci.yml). The first four
below are required checks (forge branch protection) — a hit on any of them
blocks merge. The two vale jobs split what used to be a single check: the
error-only one exists so it _can_ become a required check without also
gating on the full job's standing warning/suggestion backlog, but flipping
that switch on the forge is a separate operator action, not something this
table can assert ahead of it happening.

<!-- vale write-good.Passive = YES -->

| Job                           | What it runs                                                                                                              | Required?                       |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
| **nix flake check**           | treefmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation                                           | yes                             |
| **tracker-tag lint**          | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`)                                            | yes                             |
| **comment-block lint**        | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`)                                         | yes                             |
| **doc-pointer lint**          | flags `docs/…md` paths and relative markdown links that no longer resolve (`scripts/check-doc-refs.sh`)                   | yes                             |
| **attribution-trailer lint**  | flags `Co-Authored-By` trailers with Claude/Anthropic attribution in PR commits (`scripts/check-attribution-trailers.sh`) | not yet — intended to be        |
| **prose lint (vale, errors)** | vale, `docs/` tree, error-severity rule hits only                                                                         | not yet — intended to be        |
| **prose lint (vale)**         | vale, `docs/` tree, full severity range (warnings + suggestions too)                                                      | no — advisory, standing backlog |

`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
without an empty commit.

<!-- vale write-good.Passive = NO -->

**`ci-rerun --pr` verifies the code but doesn't reliably move the PR's own
status checks.** Measured directly (raw `GET /repos/.../actions/tasks`
JSON): a genuine PR-triggered run carries `event: pull_request` with the
`#<n>` pseudo-ref as its `head_branch`; a `ci-rerun --pr`-dispatched run
carries `event: workflow_dispatch` with the real branch name instead.
Forgejo's PR commit-status tracking appears scoped to `pull_request`-event
runs specifically, so a dispatched run — even a genuinely successful one —
never writes to that status slot. Reproduced with three dispatches on one
PR over 20+ minutes, all genuinely green, the PR's own status unmoved
throughout. **Always re-check `pr status` after a dispatch and believe
what it says — don't push a commit just to unwedge it, that's the
empty-commit anti-pattern this verb exists to avoid.** If `hive-forge
ci-log --run <n> --job 0` shows the code green (verdict is the
last line) but the status is stuck, ask the operator to select "rerun" in
the forge web UI — it's CSRF-gated, so only they can do it.

<!-- vale write-good.Passive = YES -->

### Running checks locally

Don't run `nix flake check` directly — it dispatches to the shared build farm and
wastes a remote-builder slot. Use the devshell equivalents instead:

```sh
nix develop -c cargo clippy --all-targets -- -D warnings
nix develop -c cargo test
nix fmt                         # authoritative; the formatter set is nix/treefmt.nix
sh scripts/check-issue-refs.sh  # tracker-tag lint
sh scripts/check-comment-blocks.sh  # comment-block lint
```

`scripts/pre-push` provides a git pre-push hook that automates the two
lint checks. Install it once per clone:

```sh
ln -sf ../../scripts/pre-push .git/hooks/pre-push
```

After that, any `git push` automatically runs both lints and aborts with a
diagnostic if either fails — catching the issue locally before CI sees it.
Note that the hook does **not** run `cargo clippy` or `cargo test` (those are
slow); run those manually before pushing Rust changes.

## Configuration reference

`services.hyperhive.deploy.ci-runner.enable = true` on the host that runs it
(see _For operators_ above). The runner reaches the forge at
`https://<swarm.forge.domain>`, so that name has to resolve from the host: the
gateway's dnsmasq answers it locally where the forge runs, and forwards it to
the operator's DNS everywhere else.

Optional tuning under `services.hyperhive.deploy.ci-runner`: `name` (runner
name in the forge admin panel), `concurrency` (parallel job capacity), `labels`
(workflow targeting; the default `hive-ci:host` is what this repo's workflows
name), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration string
for example `"3h"` — a job that exceeds it's killed so a hung or runaway build
can't hold the runner's single slot indefinitely).

## Container design

- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
- **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after.
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`.
- **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. Only `swarm-ci-runner-register.service` loads the registration token; the daemon unit and the jobs it runs have it neither in their environment nor among their credentials.

## Registration

1. **swarm-controller** (`swarm-controller/src/forge/runner_token.rs`) reads
   the forge's instance-wide token from
   `GET /api/v1/admin/runners/registration-token` at start and every five
   minutes, and writes it to the secret store at
   `swarm/services/ci-runner/registration-token` when it differs from the
   value it last wrote. Its store grant on that path is write-only.
2. **`swarm-bao-ci-runner.service`** on the runner's host logs in to the store
   with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path
   and nothing else, retrying every 30 seconds until the token is there. It
   writes the token to a root-only file inside the container, and restarts
   the runner when the value changes. It runs at boot and on deploy, not on a
   timer.
3. **`swarm-ci-runner-register.service`** in the container is the only unit
   that loads that file, with `LoadCredential=`. systemd hands a unit's
   credentials to all of its processes, and for a `host` label act_runner
   runs every job step as a process of the daemon with the daemon's whole
   environment, so the daemon unit carries the token in neither. The
   register unit is a oneshot under the daemon's dynamic user and state
   directory. It reads the token from `$CREDENTIALS_DIRECTORY` and runs
   `act_runner register` only when `.runner` is missing, so the `.runner` it
   writes is the daemon's own. **The runner unit** (`gitea-runner-swarm`)
   starts it in a pre-start step and runs the daemon once it succeeds.
   A root step of the register unit, ahead of registering, removes `.runner`
   in three cases:
   - The forge has no record of the runner. Forgejo answers every call from an
     unknown runner with `unregistered runner`, and the daemon exits 1 when its
     startup `Declare` gets that answer. On the next start the step finds the
     message in the daemon's journal since its last start, and registers again.
     A daemon that's already running when an admin deletes its runner keeps polling
     and logging the error without exiting; the next restart of the unit or
     container registers it again.
   - `.runner` names another forge address than the configured one.
   - `.runner` names other labels than the configured ones, compared in the
     form the daemon writes back: `<name>:<scheme>`, with `host` for a
     label that names no scheme.

   A changed token alone doesn't register again: the daemon authenticates with
   the per-runner token in `.runner`. Without `.runner` and with no token yet
   the register unit fails and the runner unit's pre-start step with it;
   `Restart=on-failure` on the runner unit starts both again every 15 seconds,
   with no start limit.

   Host jobs run as the runner's own user, so they can still read `.runner`
   and act as this runner; see the security section below.

A new registration is a new runner on the forge: the previous entry stays in
the forge admin panel as offline until an admin deletes it.

## Actions checkout mirror

<!-- vale write-good.Passive = NO -->

With `services.hyperhive.deploy.forgejo.actionsFromSelf` enabled, the swarm
controller seeds an `actions/checkout` pull-mirror on the forge and Forgejo's
`DEFAULT_ACTIONS_URL` points at the forge itself. CI
`uses: actions/checkout@vN` steps then resolve on the forge, with no external
DNS on the CI critical path.

The option defaults to whether the forge's host also runs the swarm controller
(`services.hyperhive.deploy.swarm-controller.enable`): on when it does, off
otherwise. Setting it `true` on a forge host without the controller leaves
`actions/checkout` unresolved, because nothing seeds the mirror; evaluation
warns about it.

<!-- vale write-good.Passive = YES -->

The **swarm controller** seeds the mirror in its swarm-wide forge-objects
pass (`swarm-controller/src/forge/objects.rs`), beside the orgs it ensures.
The nix module forwards the effective mirror list as
`SWARM_CONTROLLER_FORGE_MIRRORS` in the `swarm-controller` service
environment (JSON-encoded `[{upstream, dest}]` list). It reads the list of
the host the controller runs on: mirrors declared on any other host aren't
seeded, and evaluation warns about them.

**General-purpose mirrors**: you can pre-seed any external repo as a
pull-mirror via `services.hyperhive.deploy.forgejo.mirrors`:

```nix
services.hyperhive.deploy.forgejo.mirrors = [
  { upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
  { upstream = "https://github.com/example/tool";    dest = "mirrors/tool"; }
];
```

The controller creates each entry as a real Forgejo pull-mirror — not a one-off
clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
request), so a DNS blip during that sync will propagate back to the
runner as a hard `git clone` failure. The controller
autocreates the `<owner>` org in `dest`. Keep mirror dests out of the
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
to avoid provisioning collisions.

## Security: unsandboxed builds and trusted contributors

**The runner should only run CI for trusted contributors.** The security boundary is weaker than it looks:

### What unsandboxed builds mean

nspawn containers can't create user-namespaces, so the module sets `nix.settings.sandbox-fallback = true` in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.

A malicious `default.nix` or build script in a PR can therefore:

- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.

No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only `swarm-ci-runner-register.service` loads, so a job step can't read it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls.

Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.

### Mitigation

For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.

For repos with external contributors or fork PRs:

- Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
- Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches.

The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.

## Host store maintenance (recommended)

The CI runner builds derivations through the **host** nix-daemon — the
`swarm-ci-runner` container shares the host store and has no daemon of its own. Build
outputs accumulate in `/nix/store` with no automatic collection, and a busy
CI day can fill the disk until every job fails fast with `ENOSPC`.

Store GC is a **host-level** concern, so it belongs in the host's own NixOS
configuration, not in the hyperhive service modules — a single service should
not reach out and change the host's global nix-daemon options. Add the
following to your host config:

```nix
{
  # Daily GC: delete store paths not referenced by a live root and older
  # than a day. Keeps the store bounded between builds.
  #
  # `randomizedDelaySec` because a fixed daily time lands on the same wall
  # clock as whatever else runs nightly. A collection that overlaps a CI
  # job doesn't fail it — the runner builds through the host nix-daemon,
  # so an in-flight build's temp roots are protected — but the first build
  # after a collection rebuilds what the collection deleted, so a run that
  # lands just behind one pays for it in wall time.
  nix.gc = {
    automatic = true;
    dates = "daily";
    randomizedDelaySec = "45min";
    options = "--delete-older-than 1d";
  };

  # Disk-pressure GC: when free space drops below min-free mid-build, the
  # daemon collects garbage up to max-free before continuing. This is the
  # real-time net the daily timer can't provide — a same-day build burst is
  # what fills the disk. Tune to your disk size.
  nix.settings.min-free = 20 * 1024 * 1024 * 1024;  # 20 GiB
  nix.settings.max-free = 50 * 1024 * 1024 * 1024;  # 50 GiB
}
```

**Measure a collection on your own hosts before relying on that stagger.**
`randomizedDelaySec` spreads the _start_, not the run, and the sweep over
unused `.links` entries dominates the run — which scales with the size of the
store, not with what this collection deletes. On a large store it can take most
of an hour, so two hosts both set to `daily` may overlap however much jitter you
give them. `journalctl -u nix-gc.service` reports each run's wall clock.

**Remote builders:** if CI dispatches builds to a remote builder (for example via
`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
store, so apply the same GC config wherever the builder runs —
GC on the coordinator host won't reclaim space on the builder.

⚠️ Two stores also means the temp-root protection above **doesn't span
them**. Each daemon holds roots only for the builds it runs, so a
collection on the _builder_ knows nothing about a coordinator build
waiting on a path there. Stagger the two hosts' GC windows rather than
giving them the same `dates`, and prefer `min-free`/`max-free` on the
builder — disk-pressure collection at least runs while nothing is mid-copy.

## References

- `nix/host-modules/swarm-ci-runner.nix`: runner configuration, token fetch, re-registration step, container setup.
- `swarm-controller/src/forge/runner_token.rs`: publishes the registration token.
- `.forgejo/workflows/ci.yml`: workflow definition.
- `docs/process/gotchas.md`: nix sandboxing limitations in containers.