hyperhive/docs/ci.md

# hive-ci: Forgejo Actions Runner

The `hive-ci` module runs a Forgejo Actions runner in a `hive-ci` nixos-container,
executing CI jobs from `.forgejo/workflows/ci.yml` on every PR.

## For operators

**Enabling it is one line**: `services.hyperhive.deploy.forgejo.ci.enable = true`
in the host NixOS config. No manual token provisioning — hive-c0re
registers the runner with the forge automatically.

Two things worth knowing before you rely on it:

- **Only run CI for contributors you trust.** Builds run **unsandboxed**
  (an nspawn limitation, not a choice) — a malicious build script in a
  PR could make network requests or write to the container filesystem
  during the build. Fine for a small trusted-team hive where everyone
  already has forge access; if you take outside/fork contributions,
  gate CI behind Forgejo's fork-PR-approval setting or restrict the
  workflow to push-only triggers (forks can't push to your branches).
  See *Security* below for the full threat model.
- **Watch your disk.** CI builds through the host's nix store with no
  automatic garbage collection of its own — a busy CI day can fill
  `/nix/store` until every job fails with `ENOSPC`. Add the daily +
  disk-pressure GC config from *Host store maintenance* below to your
  host's NixOS config (not optional if you plan to actually use this).

Everything below this point is implementation detail: exact
auto-registration internals, container design, and the full security
writeup.

## CI checks

Three jobs run on every PR (and on `workflow_dispatch` for manual re-triggers),
defined in [`.forgejo/workflows/ci.yml`](../.forgejo/workflows/ci.yml). All
three are required checks (forge branch protection) — a hit on any of them
blocks merge.

| Job | What it runs |
| --- | --- |
| **nix flake check** | treefmt + rustfmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation |
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) |
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) |

`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
without an empty commit.

**`ci-rerun --pr` verifies the code but does not reliably move the PR's own
status checks.** Measured directly (raw `GET /repos/.../actions/tasks`
JSON): a genuine PR-triggered run carries `event: pull_request` with the
`#<n>` pseudo-ref as its `head_branch`; a `ci-rerun --pr`-dispatched run
carries `event: workflow_dispatch` with the real branch name instead.
Forgejo's PR commit-status tracking appears scoped to `pull_request`-event
runs specifically, so a dispatched run — even a genuinely successful one —
never writes to that status slot. Reproduced with three dispatches on one
PR over 20+ minutes, all genuinely green, the PR's own status unmoved
throughout. **Always re-check `pr-status` after a dispatch and believe
what it says — don't push a commit just to unwedge it, that's the
empty-commit anti-pattern this verb exists to avoid.** If the code is
verified green (`hive-forge ci-log --run <n> --job 0`, verdict is the
last line) but the status is stuck, ask the operator to click "rerun" in
the forge web UI — it's CSRF-gated, so only they can do it.

### Running checks locally

Don't run `nix flake check` directly — it dispatches to the shared build farm and
wastes a remote-builder slot. Use the devshell equivalents instead:

```sh
nix develop -c cargo clippy --all-targets -- -D warnings
nix develop -c cargo test
nix develop -c treefmt          # same as nix fmt; treefmt covers rustfmt + nixfmt + taplo
sh scripts/check-issue-refs.sh  # tracker-tag lint
sh scripts/check-comment-blocks.sh  # comment-block lint
```

A git pre-push hook that automates the two lint checks is provided at
`scripts/pre-push`. Install it once per clone:

```sh
ln -sf ../../scripts/pre-push .git/hooks/pre-push
```

After that, any `git push` automatically runs both lints and aborts with a
diagnostic if either fails — catching the issue locally before CI sees it.
Note that the hook does **not** run `cargo clippy` or `cargo test` (those are
slow); run those manually before pushing Rust changes.

## Configuration reference

The internal forge is always present (mandatory), so the runner always has a
hive-forge instance to register against — nothing extra to enable beyond
`services.hyperhive.deploy.forgejo.ci.enable = true` (see *For operators* above).

Optional tuning: `services.hyperhive.deploy.forgejo.ci.name` (runner name in forge
admin panel), `concurrency` (parallel job capacity), `labels` (workflow
targeting), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration
string e.g. `"3h"` — a job that exceeds it is killed so a hung or runaway
build can't hold the runner's single slot indefinitely).

## Container design

- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`) and reaches hive-forge through the gateway at `http://<forge.domain>` (resolved to the bridge IP via `networking.extraHosts`). It cannot reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. Requires `forge.behindGateway = true`.
- **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/gotchas.md`.
- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. hive-c0re holds it and performs all forge API calls (runner validation + registration-token mint, in `forge/ci_runner.rs`); via hive-priv it writes only the runner registration token to the host env-file `/run/hive-ci/runner-token`, which the container bind-mounts read-only.

## Auto-registration flow

Registration is **off the container's boot-critical path** — hive-c0re owns
it and runs it out of band, so a slow forge or core-token never delays the
container's start. Gotcha: don't gate the container's own start on a forge
round-trip (a host-side unit that did this could exceed the nspawn start
timeout and trip a restart loop) — registration must stay something
hive-c0re drives after the container is already up. The core admin token is
held only by hive-c0re on the host; only the runner registration token
reaches the container.

### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep)

Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.service` when `deploy.forgejo.ci.enable`). Best-effort — failures are logged and never abort the sweep; a healthy runner is never restarted.

1. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`, validate its id against `GET /api/v1/admin/runners/{id}` with the core admin token:
   - **200**: still registered — done, no restart.
   - **404 / other non-200 / malformed**: stale — re-register (below).
   - **transport error** (forge unreachable): keep the existing creds; a network blip must not wipe a valid runner.
2. If absent or stale: mint a fresh token from `GET /api/v1/admin/runners/registration-token`, then hand it to hive-priv's `RegisterCiRunner`, which (as root) writes `TOKEN=<real>` **in place** to the host env-file `/run/hive-ci/runner-token` (preserving the inode nspawn pinned into the container at start) and restarts `gitea-runner-hive.service` inside the container so it picks up the credential and registers.

### Container side

- The container boots immediately — nothing gates its start on registration.
- tmpfiles seeds `/run/hive-ci/runner-token` with `TOKEN=placeholder` so the runner's `EnvironmentFile` always exists.
- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it is already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
- **Convergence**: because the token write targets the *host* file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.

## Actions checkout mirror

When `deploy.forgejo.ci.enable` is set, hive-c0re auto-seeds an
`actions/checkout` pull-mirror on the local forge and sets Forgejo's
`DEFAULT_ACTIONS_URL` to point at the local instance. This means CI
`uses: actions/checkout@vN` steps resolve entirely on loopback — no
external DNS on the CI critical path.

The mirror is seeded by **hive-c0re** itself during its forge
provisioning sweep (`forge/repos.rs::ensure_mirrors`). The nix module
forwards the effective mirror list as `HYPERHIVE_FORGE_MIRRORS` in the
`hive-c0re` service environment (JSON-encoded `[{upstream, dest}]`
list). hive-c0re already holds the admin token for the rest of the
forge provisioning sweep (orgs, agent accounts, etc.), so mirror
seeding lives in the same place rather than a separate host-side unit.

**General-purpose mirrors**: you can pre-seed any external repo as a
pull-mirror via `services.hyperhive.swarm.forge.mirrors`:

```nix
services.hyperhive.swarm.forge.mirrors = [
  { upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
  { upstream = "https://github.com/example/tool";    dest = "mirrors/tool"; }
];
```

Each entry is created as a real Forgejo pull-mirror — not a one-off
clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
request), so a DNS blip during that sync will propagate back to the
runner as a hard `git clone` failure. The
`<owner>` org in `dest` is auto-created. Keep mirror dests out of the
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
to avoid provisioning collisions.

## Security: unsandboxed builds and trusted contributors

**hive-ci should only run CI for trusted contributors.** The security boundary is weaker than it looks:

### What unsandboxed builds mean

nspawn containers cannot create user-namespaces, so `nix.settings.sandbox-fallback = true` is set in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.

A malicious `default.nix` or build script in a PR can therefore:

- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge only through the gateway (`http://<forge.domain>`, public/read endpoints — no admin credentials) and **cannot** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (bridge→127.0.0.0/8 is dropped).
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.

The core admin token (`forge-core-token`) is **not** bind-mounted into the container. It is held and used only by hive-c0re on the host (`forge/ci_runner.rs`), which mints per-runner registration tokens; only that registration token reaches the container's env-file. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls.

Note: `nix flake check --no-build` (eval-only) reduces the attack surface but does not eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.

### Mitigation

For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.

For repos with external contributors or fork PRs:

- Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
- Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches.

The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.

## Host store maintenance (recommended)

The CI runner builds derivations through the **host** nix-daemon — the
hive-ci container shares the host store and has no daemon of its own. Build
outputs accumulate in `/nix/store` with no automatic collection, and a busy
CI day can fill the disk until every job fails fast with `ENOSPC`.

Store GC is a **host-level** concern, so it belongs in the host's own NixOS
configuration, not in the hyperhive service modules — a single service should
not reach out and change the host's global nix-daemon options. Add the
following to your host config:

```nix
{
  # Daily GC: delete store paths not referenced by a live root and older
  # than a day. Keeps the store bounded between builds.
  nix.gc = {
    automatic = true;
    dates = "daily";
    options = "--delete-older-than 1d";
  };

  # Disk-pressure GC: when free space drops below min-free mid-build, the
  # daemon collects garbage up to max-free before continuing. This is the
  # real-time net the daily timer can't provide — a same-day build burst is
  # what fills the disk. Tune to your disk size.
  nix.settings.min-free = 20 * 1024 * 1024 * 1024;  # 20 GiB
  nix.settings.max-free = 50 * 1024 * 1024 * 1024;  # 50 GiB
}
```

**Remote builders:** if CI dispatches builds to a remote builder (e.g. via
`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
store, so the same GC config should be applied wherever the builder runs —
GC on the coordinator host won't reclaim space on the builder.

## References

- `nix/host-modules/hive-ci.nix`: runner configuration, auto-registration script, container setup.
- `.forgejo/workflows/ci.yml`: workflow definition.
- `docs/gotchas.md`: nix sandboxing limitations in containers.