hive-ci: Forgejo Actions Runner

The hive-ci module runs a Forgejo Actions runner in a hive-ci nixos-container, executing CI jobs from .forgejo/workflows/ci.yml (e.g., nix flake check on every PR).

Operator bootstrap

Set services.hyperhive.forge.ci.enable = true in the host NixOS config. That's it — no manual token provisioning.

Requirements:

services.hyperhive.forge.enable = true must also be set (the runner registers against hive-forge).
Optional: tune services.hyperhive.forge.ci.name (runner name in forge admin panel), concurrency (parallel job capacity), labels (workflow targeting), jobTimeout (per-job wall-clock cap, default "1h", Go duration string e.g. "3h" — a job that exceeds it is killed so a hung or runaway build can't hold the runner's single slot indefinitely).

Container design

Shared host netns: container reaches hive-forge at http://127.0.0.1:<httpPort> (same as hive-gateway).
Non-ephemeral: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
Sandbox fallback: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets nix.settings.sandbox-fallback = true in the container — nix builds run unsandboxed (safe because the container is already isolated). See docs/gotchas.md.
Credential isolation: the forge admin token (forge-core-token) never enters the container. A host-side oneshot service (hive-ci-prefetch.service) performs all forge API calls and writes only the runner registration token into the container via a read-only bind-mount at /run/hive-ci/runner-token.

Auto-registration flow

hive-ci-prefetch.service is a host-side oneshot that runs on every boot before nixos-container@hive-ci.service. It handles both first-run registration and stale-credential detection. The core admin token is accessed only on the host and never bind-mounted into the container.

Every boot

Check for the core admin token at /var/lib/hyperhive/forge-core-token. If absent (forge still initialising), write TOKEN=placeholder and exit — the runner service will fail gracefully until the next boot.
If .runner exists at /var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner: validate the runner ID against GET /api/v1/admin/runners/{id} using the core token:
- 200: runner still registered — write TOKEN=placeholder to /run/hive-ci/runner-token and exit; runner reuses .runner credentials.
- 404: runner was deleted from forge (e.g. after a wipe) — delete .runner, proceed to re-registration below.
- 000 (forge unreachable): keep existing .runner; write placeholder token; the runner itself will surface the connectivity error.
- other non-200: treat as stale, delete .runner and re-register.
- malformed .runner (no id field): delete and re-register.
If .runner is absent (first boot or purged above): fetch a fresh registration token from GET /api/v1/admin/runners/registration-token. Retries for 30s in case forge is still starting. Writes TOKEN=<real> to /run/hive-ci/runner-token.
Container starts with /run/hive-ci/runner-token bind-mounted read-only. gitea-actions-runner reads the token, registers itself, and persists credentials to .runner. On subsequent boots step 2 validates these credentials and fast-paths past registration.

CI workflow

The single CI job is defined in .forgejo/workflows/ci.yml:

name: CI
on:
  pull_request:
    branches: ["**"]
jobs:
  check:
    name: nix flake check
    runs-on: [hive-ci]
    steps:
      - uses: actions/checkout@v3
      - name: check
        run: nix flake check

This runs on every PR, executing all flake checks (treefmt, rustfmt, cargo test, cargo clippy, module evaluation). No --no-build: the checks' derivations are the canonical source of truth.

Security: unsandboxed builds and trusted contributors

hive-ci should only run CI for trusted contributors. The security boundary is weaker than it looks:

What unsandboxed builds mean

nspawn containers cannot create user-namespaces, so nix.settings.sandbox-fallback = true is set in the container. This means every nix build (and nix flake check) runs without a build sandbox — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.

A malicious default.nix or build script in a PR can therefore:

Make arbitrary network requests to any address reachable from the container. The container shares host netns, so http://127.0.0.1:<forgePort> is reachable with the forge API — without admin credentials, but public/read endpoints are accessible.
Write to the container filesystem, including corrupting the runner's state dir or .runner credentials.

The core admin token (forge-core-token) is not bind-mounted into the container. It is accessed only by the host-side hive-ci-prefetch.service before the container starts. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls.

Note: nix flake check --no-build (eval-only) reduces the attack surface but does not eliminate it — builtins.fetchGit, builtins.fetchurl, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full nix flake check (builds derivations), which is the higher-risk path.

Mitigation

For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.

For repos with external contributors or fork PRs:

Use Forgejo's fork PR approval workflow (repository.settings → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
Or restrict the CI workflow trigger to push events on branches (not pull_request from forks) — forks can't push to upstream branches.

The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.

Host store maintenance (recommended)

The CI runner builds derivations through the host nix-daemon — the hive-ci container shares the host store and has no daemon of its own. Build outputs accumulate in /nix/store with no automatic collection, and a busy CI day can fill the disk until every job fails fast with ENOSPC.

Store GC is a host-level concern, so it belongs in the host's own NixOS configuration, not in the hyperhive service modules — a single service should not reach out and change the host's global nix-daemon options. Add the following to your host config:

{
  # Daily GC: delete store paths not referenced by a live root and older
  # than a day. Keeps the store bounded between builds.
  nix.gc = {
    automatic = true;
    dates = "daily";
    options = "--delete-older-than 1d";
  };

  # Disk-pressure GC: when free space drops below min-free mid-build, the
  # daemon collects garbage up to max-free before continuing. This is the
  # real-time net the daily timer can't provide — a same-day build burst is
  # what fills the disk. Tune to your disk size.
  nix.settings.min-free = 20 * 1024 * 1024 * 1024;  # 20 GiB
  nix.settings.max-free = 50 * 1024 * 1024 * 1024;  # 50 GiB
}

Remote builders: if CI dispatches builds to a remote builder (e.g. via nix.buildMachines / ssh-ng://), the build outputs land in that host's store, so the same GC config should be applied wherever the builder runs — GC on the coordinator host won't reclaim space on the builder.

References

nix/modules/hive-ci.nix: runner configuration, auto-registration script, container setup.
.forgejo/workflows/ci.yml: workflow definition.
docs/gotchas.md: nix sandboxing limitations in containers.

7.8 KiB Raw Blame History