hyperhive/docs/ci.md

hive-ci: Forgejo Actions Runner

The hive-ci module runs a Forgejo Actions runner in a hive-ci nixos-container, executing CI jobs from .forgejo/workflows/ci.yml on every PR.

CI checks

Three jobs run on every PR (and on workflow_dispatch for manual re-triggers):

Job	What it runs	Currently required
nix flake check	treefmt + rustfmt formatting, cargo clippy -D warnings, cargo test, module evaluation	yes
tracker-tag lint	flags #NNN issue tags in source and comments (scripts/check-issue-refs.sh)	no (red, non-blocking)
comment-block lint	flags contiguous comment blocks over 30 lines (scripts/check-comment-blocks.sh)	no (red, non-blocking)

The tracker-tag and comment-block checks are non-blocking today (a hit fails the check but does not prevent merge) while the legacy backlog is cleaned up. They are expected to become required checks once the tree is clean.

Running checks locally

Don't run nix flake check directly — it dispatches to the shared build farm and wastes a remote-builder slot. Use the devshell equivalents instead:

nix develop -c cargo clippy --all-targets -- -D warnings
nix develop -c cargo test
nix develop -c treefmt          # same as nix fmt; treefmt covers rustfmt + nixfmt + taplo
sh scripts/check-issue-refs.sh  # tracker-tag lint
sh scripts/check-comment-blocks.sh  # comment-block lint

A git pre-push hook that automates the two lint checks is provided at scripts/pre-push. Install it once per clone:

ln -sf ../../scripts/pre-push .git/hooks/pre-push

After that, any git push automatically runs both lints and aborts with a diagnostic if either fails — catching the issue locally before CI sees it. Note that the hook does not run cargo clippy or cargo test (those are slow); run those manually before pushing Rust changes.

Operator bootstrap

Set services.hyperhive.forge.ci.enable = true in the host NixOS config. That's it — no manual token provisioning.

Requirements:

• The internal forge is always present (mandatory), so the runner always has a hive-forge instance to register against — nothing extra to enable.
• Optional: tune services.hyperhive.forge.ci.name (runner name in forge admin panel), concurrency (parallel job capacity), labels (workflow targeting), jobTimeout (per-job wall-clock cap, default "1h", Go duration string e.g. "3h" — a job that exceeds it is killed so a hung or runaway build can't hold the runner's single slot indefinitely).

Container design

• Shared host netns: container reaches hive-forge at http://127.0.0.1:<httpPort> (same as hive-gateway).
• Non-ephemeral: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
• Sandbox fallback: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets nix.settings.sandbox-fallback = true in the container — nix builds run unsandboxed (safe because the container is already isolated). See docs/gotchas.md.
• Credential isolation: the forge admin token (forge-core-token) never enters the container. A host-side oneshot service (hive-ci-prefetch.service) performs all forge API calls and writes only the runner registration token into the container via a read-only bind-mount at /run/hive-ci/runner-token.

Auto-registration flow

hive-ci-prefetch.service is a host-side oneshot that runs on every boot before nixos-container@hive-ci.service. It handles both first-run registration and stale-credential detection. The core admin token is accessed only on the host and never bind-mounted into the container.

Every boot

1. Check for the core admin token at /var/lib/hyperhive/forge-core-token. If absent (forge still initialising), write TOKEN=placeholder and exit — the runner service will fail gracefully until the next boot.
2. If .runner exists at /var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner: validate the runner ID against GET /api/v1/admin/runners/{id} using the core token:
   • 200: runner still registered — write TOKEN=placeholder to /run/hive-ci/runner-token and exit; runner reuses .runner credentials.
   • 404: runner was deleted from forge (e.g. after a wipe) — delete .runner, proceed to re-registration below.
   • 000 (forge unreachable): keep existing .runner; write placeholder token; the runner itself will surface the connectivity error.
   • other non-200: treat as stale, delete .runner and re-register.
   • malformed .runner (no id field): delete and re-register.
3. If .runner is absent (first boot or purged above): fetch a fresh registration token from GET /api/v1/admin/runners/registration-token. Retries for 30s in case forge is still starting. Writes TOKEN=<real> to /run/hive-ci/runner-token.
4. Container starts with /run/hive-ci/runner-token bind-mounted read-only. gitea-actions-runner reads the token, registers itself, and persists credentials to .runner. On subsequent boots step 2 validates these credentials and fast-paths past registration.

Actions checkout mirror

When forge.ci.enable is set, hive-c0re auto-seeds an actions/checkout pull-mirror on the local forge and sets Forgejo's DEFAULT_ACTIONS_URL to point at the local instance. This means CI uses: actions/checkout@vN steps resolve entirely on loopback — no external DNS on the CI critical path.

The mirror is seeded by hive-c0re itself during its forge provisioning sweep (forge.rs::ensure_mirrors). The nix module forwards the effective mirror list as HYPERHIVE_FORGE_MIRRORS in the hive-c0re service environment (JSON-encoded [{upstream, dest}] list). hive-c0re already holds the admin token for the rest of the forge provisioning sweep (orgs, agent accounts, etc.), so mirror seeding lives in the same place rather than a separate host-side unit.

General-purpose mirrors: you can pre-seed any external repo as a pull-mirror via services.hyperhive.forge.mirrors:

services.hyperhive.forge.mirrors = [
  { upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
  { upstream = "https://github.com/example/tool";    dest = "mirrors/tool"; }
];

Each entry is created as a real Forgejo pull-mirror — not a one-off clone. Forgejo re-syncs the mirror on every pull (git-upload-pack request), so a DNS blip during that sync will propagate back to the runner as a hard git clone failure. The <owner> org in dest is auto-created. Keep mirror dests out of the hive-c0re-managed namespaces (config/, shared/, agents/, core/) to avoid provisioning collisions.

CI workflow

Three jobs are defined in .forgejo/workflows/ci.yml:

name: CI
on:
  pull_request:
    branches: ["**"]
  # workflow_dispatch lets `hive-forge ci-rerun --pr N` retrigger without
  # an empty commit.
  workflow_dispatch:

jobs:
  check:
    name: nix flake check
    runs-on: [hive-ci]
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v3
      - run: nix flake check

  tracker-tags:
    name: tracker-tag lint
    runs-on: [hive-ci]
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v3
      - run: sh scripts/check-issue-refs.sh

  comment-blocks:
    name: comment-block lint
    runs-on: [hive-ci]
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v3
      - run: sh scripts/check-comment-blocks.sh

• nix flake check — all flake checks: formatting (treefmt+rustfmt), cargo test, cargo clippy, module evaluation, hivectl-docs freshness. No --no-build: the check derivations are the canonical source of truth. 30-minute wall-clock cap.
• tracker-tag lint — flags #NNN tracker references in non-markdown source (hive convention: use prose in code, not tags). Non-blocking for now (doesn't gate merge) while the legacy backlog is cleaned up.
• comment-block lint — flags contiguous comment blocks over 30 lines. Prose belongs in docs/, not inline. Also non-blocking while the tree settles.

All three jobs run in parallel on each PR push. hive-forge ci-rerun --pr N dispatches a fresh workflow_dispatch run without an empty commit — the correct way to retry a transient CI failure.

Security: unsandboxed builds and trusted contributors

hive-ci should only run CI for trusted contributors. The security boundary is weaker than it looks:

What unsandboxed builds mean

nspawn containers cannot create user-namespaces, so nix.settings.sandbox-fallback = true is set in the container. This means every nix build (and nix flake check) runs without a build sandbox — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.

A malicious default.nix or build script in a PR can therefore:

• Make arbitrary network requests to any address reachable from the container. The container shares host netns, so http://127.0.0.1:<forgePort> is reachable with the forge API — without admin credentials, but public/read endpoints are accessible.
• Write to the container filesystem, including corrupting the runner's state dir or .runner credentials.

The core admin token (forge-core-token) is not bind-mounted into the container. It is accessed only by the host-side hive-ci-prefetch.service before the container starts. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls.

Note: nix flake check --no-build (eval-only) reduces the attack surface but does not eliminate it — builtins.fetchGit, builtins.fetchurl, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full nix flake check (builds derivations), which is the higher-risk path.

Mitigation

For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.

For repos with external contributors or fork PRs:

• Use Forgejo's fork PR approval workflow (repository.settings → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
• Or restrict the CI workflow trigger to push events on branches (not pull_request from forks) — forks can't push to upstream branches.

The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.

Host store maintenance (recommended)

The CI runner builds derivations through the host nix-daemon — the hive-ci container shares the host store and has no daemon of its own. Build outputs accumulate in /nix/store with no automatic collection, and a busy CI day can fill the disk until every job fails fast with ENOSPC.

Store GC is a host-level concern, so it belongs in the host's own NixOS configuration, not in the hyperhive service modules — a single service should not reach out and change the host's global nix-daemon options. Add the following to your host config:

{
  # Daily GC: delete store paths not referenced by a live root and older
  # than a day. Keeps the store bounded between builds.
  nix.gc = {
    automatic = true;
    dates = "daily";
    options = "--delete-older-than 1d";
  };

  # Disk-pressure GC: when free space drops below min-free mid-build, the
  # daemon collects garbage up to max-free before continuing. This is the
  # real-time net the daily timer can't provide — a same-day build burst is
  # what fills the disk. Tune to your disk size.
  nix.settings.min-free = 20 * 1024 * 1024 * 1024;  # 20 GiB
  nix.settings.max-free = 50 * 1024 * 1024 * 1024;  # 50 GiB
}

Remote builders: if CI dispatches builds to a remote builder (e.g. via nix.buildMachines / ssh-ng://), the build outputs land in that host's store, so the same GC config should be applied wherever the builder runs — GC on the coordinator host won't reclaim space on the builder.

References

• nix/modules/hive-ci.nix: runner configuration, auto-registration script, container setup.
• .forgejo/workflows/ci.yml: workflow definition.
• docs/gotchas.md: nix sandboxing limitations in containers.