hyperhive/docs/ci.md

# hive-ci: Forgejo Actions Runner

The `hive-ci` module runs a Forgejo Actions runner in a `hive-ci` nixos-container,
executing CI jobs from `.forgejo/workflows/ci.yml` on every PR.

## CI checks

Three jobs run on every PR (and on `workflow_dispatch` for manual re-triggers):

| Job | What it runs | Currently required |
| --- | --- | --- |
| **nix flake check** | treefmt + rustfmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation | yes |
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) | no (red, non-blocking) |
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) | no (red, non-blocking) |

The tracker-tag and comment-block checks are non-blocking today (a hit fails the
check but does not prevent merge) while the legacy backlog is cleaned up. They are
expected to become required checks once the tree is clean.

### Running checks locally

Don't run `nix flake check` directly — it dispatches to the shared build farm and
wastes a remote-builder slot. Use the devshell equivalents instead:

```sh
nix develop -c cargo clippy --all-targets -- -D warnings
nix develop -c cargo test
nix develop -c treefmt          # same as nix fmt; treefmt covers rustfmt + nixfmt + taplo
sh scripts/check-issue-refs.sh  # tracker-tag lint
sh scripts/check-comment-blocks.sh  # comment-block lint
```

A git pre-push hook that automates the two lint checks is provided at
`scripts/pre-push`. Install it once per clone:

```sh
ln -sf ../../scripts/pre-push .git/hooks/pre-push
```

After that, any `git push` automatically runs both lints and aborts with a
diagnostic if either fails — catching the issue locally before CI sees it.
Note that the hook does **not** run `cargo clippy` or `cargo test` (those are
slow); run those manually before pushing Rust changes.

## Operator bootstrap

Set `services.hyperhive.forge.ci.enable = true` in the host NixOS config. That's it — no manual token provisioning.

**Requirements:**

- The internal forge is always present (mandatory), so the runner always has a hive-forge instance to register against — nothing extra to enable.
- Optional: tune `services.hyperhive.forge.ci.name` (runner name in forge admin panel), `concurrency` (parallel job capacity), `labels` (workflow targeting), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration string e.g. `"3h"` — a job that exceeds it is killed so a hung or runaway build can't hold the runner's single slot indefinitely).

## Container design

- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`) and reaches hive-forge through the gateway at `http://<forge.domain>` (resolved to the bridge IP via `networking.extraHosts`). It cannot reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. Requires `forge.behindGateway = true`.
- **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/gotchas.md`.
- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. A host-side oneshot service (`hive-ci-prefetch.service`) performs all forge API calls and writes only the runner registration token into the container via a read-only bind-mount at `/run/hive-ci/runner-token`.

## Auto-registration flow

`hive-ci-prefetch.service` is a host-side oneshot that runs on **every boot** before `nixos-container@hive-ci.service`. It handles both first-run registration and stale-credential detection. The core admin token is accessed only on the host and never bind-mounted into the container.

### Every boot

1. Check for the core admin token at `/var/lib/hyperhive/forge-core-token`. If absent (forge still initialising), write `TOKEN=placeholder` and exit — the runner service will fail gracefully until the next boot.
2. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`: validate the runner ID against `GET /api/v1/admin/runners/{id}` using the core token:
   - **200**: runner still registered — write `TOKEN=placeholder` to `/run/hive-ci/runner-token` and exit; runner reuses `.runner` credentials.
   - **404**: runner was deleted from forge (e.g. after a wipe) — delete `.runner`, proceed to re-registration below.
   - **000** (forge unreachable): keep existing `.runner`; write placeholder token; the runner itself will surface the connectivity error.
   - **other non-200**: treat as stale, delete `.runner` and re-register.
   - **malformed `.runner`** (no `id` field): delete and re-register.
3. If `.runner` is absent (first boot or purged above): fetch a fresh registration token from `GET /api/v1/admin/runners/registration-token`. Retries for 30s in case forge is still starting. Writes `TOKEN=<real>` to `/run/hive-ci/runner-token`.
4. Container starts with `/run/hive-ci/runner-token` bind-mounted read-only. `gitea-actions-runner` reads the token, registers itself, and persists credentials to `.runner`. On subsequent boots step 2 validates these credentials and fast-paths past registration.

## Actions checkout mirror

When `forge.ci.enable` is set, hive-c0re auto-seeds an
`actions/checkout` pull-mirror on the local forge and sets Forgejo's
`DEFAULT_ACTIONS_URL` to point at the local instance. This means CI
`uses: actions/checkout@vN` steps resolve entirely on loopback — no
external DNS on the CI critical path.

The mirror is seeded by **hive-c0re** itself during its forge
provisioning sweep (`forge.rs::ensure_mirrors`). The nix module
forwards the effective mirror list as `HYPERHIVE_FORGE_MIRRORS` in the
`hive-c0re` service environment (JSON-encoded `[{upstream, dest}]`
list). hive-c0re already holds the admin token for the rest of the
forge provisioning sweep (orgs, agent accounts, etc.), so mirror
seeding lives in the same place rather than a separate host-side unit.

**General-purpose mirrors**: you can pre-seed any external repo as a
pull-mirror via `services.hyperhive.forge.mirrors`:

```nix
services.hyperhive.forge.mirrors = [
  { upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
  { upstream = "https://github.com/example/tool";    dest = "mirrors/tool"; }
];
```

Each entry is created as a real Forgejo pull-mirror — not a one-off
clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
request), so a DNS blip during that sync will propagate back to the
runner as a hard `git clone` failure. The
`<owner>` org in `dest` is auto-created. Keep mirror dests out of the
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
to avoid provisioning collisions.

## CI workflow

Three jobs are defined in [`.forgejo/workflows/ci.yml`](../.forgejo/workflows/ci.yml):
`nix flake check`, `tracker-tag lint`, and `comment-block lint`. All three are
required — a lint failure blocks merge. `hive-forge ci-rerun --pr N` dispatches
a `workflow_dispatch` retrigger without an empty commit.

## Security: unsandboxed builds and trusted contributors

**hive-ci should only run CI for trusted contributors.** The security boundary is weaker than it looks:

### What unsandboxed builds mean

nspawn containers cannot create user-namespaces, so `nix.settings.sandbox-fallback = true` is set in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.

A malicious `default.nix` or build script in a PR can therefore:

- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge only through the gateway (`http://<forge.domain>`, public/read endpoints — no admin credentials) and **cannot** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (bridge→127.0.0.0/8 is dropped).
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.

The core admin token (`forge-core-token`) is **not** bind-mounted into the container. It is accessed only by the host-side `hive-ci-prefetch.service` before the container starts. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls.

Note: `nix flake check --no-build` (eval-only) reduces the attack surface but does not eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.

### Mitigation

For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.

For repos with external contributors or fork PRs:

- Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
- Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches.

The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.

## Host store maintenance (recommended)

The CI runner builds derivations through the **host** nix-daemon — the
hive-ci container shares the host store and has no daemon of its own. Build
outputs accumulate in `/nix/store` with no automatic collection, and a busy
CI day can fill the disk until every job fails fast with `ENOSPC`.

Store GC is a **host-level** concern, so it belongs in the host's own NixOS
configuration, not in the hyperhive service modules — a single service should
not reach out and change the host's global nix-daemon options. Add the
following to your host config:

```nix
{
  # Daily GC: delete store paths not referenced by a live root and older
  # than a day. Keeps the store bounded between builds.
  nix.gc = {
    automatic = true;
    dates = "daily";
    options = "--delete-older-than 1d";
  };

  # Disk-pressure GC: when free space drops below min-free mid-build, the
  # daemon collects garbage up to max-free before continuing. This is the
  # real-time net the daily timer can't provide — a same-day build burst is
  # what fills the disk. Tune to your disk size.
  nix.settings.min-free = 20 * 1024 * 1024 * 1024;  # 20 GiB
  nix.settings.max-free = 50 * 1024 * 1024 * 1024;  # 50 GiB
}
```

**Remote builders:** if CI dispatches builds to a remote builder (e.g. via
`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
store, so the same GC config should be applied wherever the builder runs —
GC on the coordinator host won't reclaim space on the builder.

## References

- `nix/host-modules/hive-ci.nix`: runner configuration, auto-registration script, container setup.
- `.forgejo/workflows/ci.yml`: workflow definition.
- `docs/gotchas.md`: nix sandboxing limitations in containers.