From 14b602b35410cba4d9960d4b879f3e444b3a2ab7 Mon Sep 17 00:00:00 2001 From: atlas Date: Thu, 16 Jul 2026 14:29:25 +0200 Subject: [PATCH] docs(#2415): update docs/ci.md for c0re-owned runner registration #2415 moved hive-ci runner registration off the boot-critical path into hive-c0re (forge/ci_runner.rs) + hive-priv, retiring the host-side hive-ci-prefetch.service. Update the doc's Container-design, Auto-registration flow, and Security sections to describe the new flow: c0re validates .runner / mints the token out of band, hive-priv writes the host env-file + restarts the runner, the container boots immediately with a fail-fast precond + Restart=on-failure self-heal. --- docs/ci.md | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/docs/ci.md b/docs/ci.md index 5ea5a8d6..95160b90 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -56,23 +56,28 @@ Set `services.hyperhive.forge.ci.enable = true` in the host NixOS config. That's - **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`) and reaches hive-forge through the gateway at `http://` (resolved to the bridge IP via `networking.extraHosts`). It cannot reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. Requires `forge.behindGateway = true`. - **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter). - **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/gotchas.md`. -- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. A host-side oneshot service (`hive-ci-prefetch.service`) performs all forge API calls and writes only the runner registration token into the container via a read-only bind-mount at `/run/hive-ci/runner-token`. +- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. hive-c0re holds it and performs all forge API calls (runner validation + registration-token mint, in `forge/ci_runner.rs`); via hive-priv it writes only the runner registration token to the host env-file `/run/hive-ci/runner-token`, which the container bind-mounts read-only. ## Auto-registration flow -`hive-ci-prefetch.service` is a host-side oneshot that runs on **every boot** before `nixos-container@hive-ci.service`. It handles both first-run registration and stale-credential detection. The core admin token is accessed only on the host and never bind-mounted into the container. +Registration is **off the container's boot-critical path** — hive-c0re owns it and runs it out of band, so a slow forge or core-token never delays the container's start. (The earlier design ran a host-side `hive-ci-prefetch.service` that gated `container@hive-ci` start on a forge round-trip, which could exceed the nspawn start timeout and trip a restart loop; moving registration into hive-c0re removed that.) The core admin token is held only by hive-c0re on the host; only the runner registration token reaches the container. -### Every boot +### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep) -1. Check for the core admin token at `/var/lib/hyperhive/forge-core-token`. If absent (forge still initialising), write `TOKEN=placeholder` and exit — the runner service will fail gracefully until the next boot. -2. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`: validate the runner ID against `GET /api/v1/admin/runners/{id}` using the core token: - - **200**: runner still registered — write `TOKEN=placeholder` to `/run/hive-ci/runner-token` and exit; runner reuses `.runner` credentials. - - **404**: runner was deleted from forge (e.g. after a wipe) — delete `.runner`, proceed to re-registration below. - - **000** (forge unreachable): keep existing `.runner`; write placeholder token; the runner itself will surface the connectivity error. - - **other non-200**: treat as stale, delete `.runner` and re-register. - - **malformed `.runner`** (no `id` field): delete and re-register. -3. If `.runner` is absent (first boot or purged above): fetch a fresh registration token from `GET /api/v1/admin/runners/registration-token`. Retries for 30s in case forge is still starting. Writes `TOKEN=` to `/run/hive-ci/runner-token`. -4. Container starts with `/run/hive-ci/runner-token` bind-mounted read-only. `gitea-actions-runner` reads the token, registers itself, and persists credentials to `.runner`. On subsequent boots step 2 validates these credentials and fast-paths past registration. +Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.service` when `forge.ci.enable`). Best-effort — failures are logged and never abort the sweep; a healthy runner is never restarted. + +1. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`, validate its id against `GET /api/v1/admin/runners/{id}` with the core admin token: + - **200**: still registered — done, no restart. + - **404 / other non-200 / malformed**: stale — re-register (below). + - **transport error** (forge unreachable): keep the existing creds; a network blip must not wipe a valid runner. +2. If absent or stale: mint a fresh token from `GET /api/v1/admin/runners/registration-token`, then hand it to hive-priv's `RegisterCiRunner`, which (as root) writes `TOKEN=` **in place** to the host env-file `/run/hive-ci/runner-token` (preserving the inode nspawn pinned into the container at start) and restarts `gitea-runner-hive.service` inside the container so it picks up the credential and registers. + +### Container side + +- The container boots immediately — nothing gates its start on registration. +- tmpfiles seeds `/run/hive-ci/runner-token` with `TOKEN=placeholder` so the runner's `EnvironmentFile` always exists. +- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it is already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net). +- **Convergence**: because the token write targets the *host* file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own. ## Actions checkout mirror @@ -128,7 +133,7 @@ A malicious `default.nix` or build script in a PR can therefore: - **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge only through the gateway (`http://`, public/read endpoints — no admin credentials) and **cannot** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (bridge→127.0.0.0/8 is dropped). - **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials. -The core admin token (`forge-core-token`) is **not** bind-mounted into the container. It is accessed only by the host-side `hive-ci-prefetch.service` before the container starts. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls. +The core admin token (`forge-core-token`) is **not** bind-mounted into the container. It is held and used only by hive-c0re on the host (`forge/ci_runner.rs`), which mints per-runner registration tokens; only that registration token reaches the container's env-file. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls. Note: `nix flake check --no-build` (eval-only) reduces the attack surface but does not eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.