hyperhive/docs/ci.md

hive-ci: Forgejo Actions Runner

The hive-ci module runs a Forgejo Actions runner in a hive-ci nixos-container, executing CI jobs from .forgejo/workflows/ci.yml on every PR.

For operators

Enabling it is one line: services.hyperhive.forge.ci.enable = true in the host NixOS config. No manual token provisioning — hive-c0re registers the runner with the forge automatically.

Two things worth knowing before you rely on it:

• Only run CI for contributors you trust. Builds run unsandboxed (an nspawn limitation, not a choice) — a malicious build script in a PR could make network requests or write to the container filesystem during the build. Fine for a small trusted-team hive where everyone already has forge access; if you take outside/fork contributions, gate CI behind Forgejo's fork-PR-approval setting or restrict the workflow to push-only triggers (forks can't push to your branches). See Security below for the full threat model.
• Watch your disk. CI builds through the host's nix store with no automatic garbage collection of its own — a busy CI day can fill /nix/store until every job fails with ENOSPC. Add the daily + disk-pressure GC config from Host store maintenance below to your host's NixOS config (not optional if you plan to actually use this).

Everything below this point is implementation detail: exact auto-registration internals, container design, and the full security writeup.

CI checks

Three jobs run on every PR (and on workflow_dispatch for manual re-triggers), defined in .forgejo/workflows/ci.yml. All three are required checks (forge branch protection) — a hit on any of them blocks merge.

Job	What it runs
nix flake check	treefmt + rustfmt formatting, cargo clippy -D warnings, cargo test, module evaluation
tracker-tag lint	flags #NNN issue tags in source and comments (scripts/check-issue-refs.sh)
comment-block lint	flags contiguous comment blocks over 30 lines (scripts/check-comment-blocks.sh)

hive-forge ci-rerun --pr N dispatches a workflow_dispatch retrigger without an empty commit.

Running checks locally

Don't run nix flake check directly — it dispatches to the shared build farm and wastes a remote-builder slot. Use the devshell equivalents instead:

nix develop -c cargo clippy --all-targets -- -D warnings
nix develop -c cargo test
nix develop -c treefmt          # same as nix fmt; treefmt covers rustfmt + nixfmt + taplo
sh scripts/check-issue-refs.sh  # tracker-tag lint
sh scripts/check-comment-blocks.sh  # comment-block lint

A git pre-push hook that automates the two lint checks is provided at scripts/pre-push. Install it once per clone:

ln -sf ../../scripts/pre-push .git/hooks/pre-push

After that, any git push automatically runs both lints and aborts with a diagnostic if either fails — catching the issue locally before CI sees it. Note that the hook does not run cargo clippy or cargo test (those are slow); run those manually before pushing Rust changes.

Configuration reference

The internal forge is always present (mandatory), so the runner always has a hive-forge instance to register against — nothing extra to enable beyond services.hyperhive.forge.ci.enable = true (see For operators above).

Optional tuning: services.hyperhive.forge.ci.name (runner name in forge admin panel), concurrency (parallel job capacity), labels (workflow targeting), jobTimeout (per-job wall-clock cap, default "1h", Go duration string e.g. "3h" — a job that exceeds it is killed so a hung or runaway build can't hold the runner's single slot indefinitely).

Container design

• Private netns, bridge-attached: the container runs in its own network namespace (privateNetwork = true, hostBridge) and reaches hive-forge through the gateway at http://<forge.domain> (resolved to the bridge IP via networking.extraHosts). It cannot reach host-loopback services — the core dashboard at 127.0.0.1:7000 and the raw forge port are unreachable from CI. Requires forge.behindGateway = true.
• Non-ephemeral: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
• Sandbox fallback: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets nix.settings.sandbox-fallback = true in the container — nix builds run unsandboxed (safe because the container is already isolated). See docs/gotchas.md.
• Credential isolation: the forge admin token (forge-core-token) never enters the container. hive-c0re holds it and performs all forge API calls (runner validation + registration-token mint, in forge/ci_runner.rs); via hive-priv it writes only the runner registration token to the host env-file /run/hive-ci/runner-token, which the container bind-mounts read-only.

Auto-registration flow

Registration is off the container's boot-critical path — hive-c0re owns it and runs it out of band, so a slow forge or core-token never delays the container's start. Gotcha: don't gate the container's own start on a forge round-trip (a host-side unit that did this could exceed the nspawn start timeout and trip a restart loop) — registration must stay something hive-c0re drives after the container is already up. The core admin token is held only by hive-c0re on the host; only the runner registration token reaches the container.

hive-c0re side (forge/ci_runner.rs, run during the startup sweep)

Gated on HYPERHIVE_FORGE_CI_ENABLED (the nix module sets it on hive-c0re.service when forge.ci.enable). Best-effort — failures are logged and never abort the sweep; a healthy runner is never restarted.

1. If .runner exists at /var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner, validate its id against GET /api/v1/admin/runners/{id} with the core admin token:
   • 200: still registered — done, no restart.
   • 404 / other non-200 / malformed: stale — re-register (below).
   • transport error (forge unreachable): keep the existing creds; a network blip must not wipe a valid runner.
2. If absent or stale: mint a fresh token from GET /api/v1/admin/runners/registration-token, then hand it to hive-priv's RegisterCiRunner, which (as root) writes TOKEN=<real> in place to the host env-file /run/hive-ci/runner-token (preserving the inode nspawn pinned into the container at start) and restarts gitea-runner-hive.service inside the container so it picks up the credential and registers.

Container side

• The container boots immediately — nothing gates its start on registration.
• tmpfiles seeds /run/hive-ci/runner-token with TOKEN=placeholder so the runner's EnvironmentFile always exists.
• gitea-runner-hive.service has an ExecStartPre precond (ahead of the nix-daemon wait) that fails fast unless it is already registered (.runner present) or a real, non-placeholder token is in place. Restart=on-failure (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
• Convergence: because the token write targets the host file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.

Actions checkout mirror

When forge.ci.enable is set, hive-c0re auto-seeds an actions/checkout pull-mirror on the local forge and sets Forgejo's DEFAULT_ACTIONS_URL to point at the local instance. This means CI uses: actions/checkout@vN steps resolve entirely on loopback — no external DNS on the CI critical path.

The mirror is seeded by hive-c0re itself during its forge provisioning sweep (forge.rs::ensure_mirrors). The nix module forwards the effective mirror list as HYPERHIVE_FORGE_MIRRORS in the hive-c0re service environment (JSON-encoded [{upstream, dest}] list). hive-c0re already holds the admin token for the rest of the forge provisioning sweep (orgs, agent accounts, etc.), so mirror seeding lives in the same place rather than a separate host-side unit.

General-purpose mirrors: you can pre-seed any external repo as a pull-mirror via services.hyperhive.forge.mirrors:

services.hyperhive.forge.mirrors = [
  { upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
  { upstream = "https://github.com/example/tool";    dest = "mirrors/tool"; }
];

Each entry is created as a real Forgejo pull-mirror — not a one-off clone. Forgejo re-syncs the mirror on every pull (git-upload-pack request), so a DNS blip during that sync will propagate back to the runner as a hard git clone failure. The <owner> org in dest is auto-created. Keep mirror dests out of the hive-c0re-managed namespaces (config/, shared/, agents/, core/) to avoid provisioning collisions.

Security: unsandboxed builds and trusted contributors

hive-ci should only run CI for trusted contributors. The security boundary is weaker than it looks:

What unsandboxed builds mean

nspawn containers cannot create user-namespaces, so nix.settings.sandbox-fallback = true is set in the container. This means every nix build (and nix flake check) runs without a build sandbox — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.

A malicious default.nix or build script in a PR can therefore:

• Make arbitrary network requests to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge only through the gateway (http://<forge.domain>, public/read endpoints — no admin credentials) and cannot reach host-loopback services: the unauthenticated core dashboard at 127.0.0.1:7000 and the raw forge port are off-limits (bridge→127.0.0.0/8 is dropped).
• Write to the container filesystem, including corrupting the runner's state dir or .runner credentials.

The core admin token (forge-core-token) is not bind-mounted into the container. It is held and used only by hive-c0re on the host (forge/ci_runner.rs), which mints per-runner registration tokens; only that registration token reaches the container's env-file. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls.

Note: nix flake check --no-build (eval-only) reduces the attack surface but does not eliminate it — builtins.fetchGit, builtins.fetchurl, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full nix flake check (builds derivations), which is the higher-risk path.

Mitigation

For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.

For repos with external contributors or fork PRs:

• Use Forgejo's fork PR approval workflow (repository.settings → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
• Or restrict the CI workflow trigger to push events on branches (not pull_request from forks) — forks can't push to upstream branches.

The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.

Host store maintenance (recommended)

The CI runner builds derivations through the host nix-daemon — the hive-ci container shares the host store and has no daemon of its own. Build outputs accumulate in /nix/store with no automatic collection, and a busy CI day can fill the disk until every job fails fast with ENOSPC.

Store GC is a host-level concern, so it belongs in the host's own NixOS configuration, not in the hyperhive service modules — a single service should not reach out and change the host's global nix-daemon options. Add the following to your host config:

{
  # Daily GC: delete store paths not referenced by a live root and older
  # than a day. Keeps the store bounded between builds.
  nix.gc = {
    automatic = true;
    dates = "daily";
    options = "--delete-older-than 1d";
  };

  # Disk-pressure GC: when free space drops below min-free mid-build, the
  # daemon collects garbage up to max-free before continuing. This is the
  # real-time net the daily timer can't provide — a same-day build burst is
  # what fills the disk. Tune to your disk size.
  nix.settings.min-free = 20 * 1024 * 1024 * 1024;  # 20 GiB
  nix.settings.max-free = 50 * 1024 * 1024 * 1024;  # 50 GiB
}

Remote builders: if CI dispatches builds to a remote builder (e.g. via nix.buildMachines / ssh-ng://), the build outputs land in that host's store, so the same GC config should be applied wherever the builder runs — GC on the coordinator host won't reclaim space on the builder.

References

• nix/host-modules/hive-ci.nix: runner configuration, auto-registration script, container setup.
• .forgejo/workflows/ci.yml: workflow definition.
• docs/gotchas.md: nix sandboxing limitations in containers.