fix: move core-token out of hive-ci container (host-side prefetch service)

This commit is contained in:
atlas 2026-06-03 22:14:17 +02:00 committed by mara
commit 9209094397
2 changed files with 113 additions and 87 deletions

View file

@ -15,22 +15,23 @@ Set `services.hyperhive.ci.enable = true` in the host NixOS config. That's it
- **Shared host netns**: container reaches hive-forge at `http://127.0.0.1:<httpPort>` (same as hive-gateway).
- **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/gotchas.md`.
- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. A host-side oneshot service (`hive-ci-prefetch.service`) performs all forge API calls and writes only the runner registration token into the container via a read-only bind-mount at `/run/hive-ci/runner-token`.
## Auto-registration flow
`hive-ci-register.service` is a oneshot that runs on **every boot** before `gitea-runner-hive.service`. It handles both first-run registration and stale-credential detection.
`hive-ci-prefetch.service` is a host-side oneshot that runs on **every boot** before `nixos-container@hive-ci.service`. It handles both first-run registration and stale-credential detection. The core admin token is accessed only on the host and never bind-mounted into the container.
### Every boot
1. Read the core admin token from `/run/hive-ci/core-token` (bind-mounted from `/var/lib/hyperhive/forge-core-token`).
2. If `.runner` exists: validate the runner ID against `GET /api/v1/admin/runners/{id}` using the core token:
- **200**: runner still registered — write dummy `TOKEN=placeholder` and exit; runner reuses `.runner` credentials.
1. Check for the core admin token at `/var/lib/hyperhive/forge-core-token`. If absent (forge still initialising), write `TOKEN=placeholder` and exit — the runner service will fail gracefully until the next boot.
2. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`: validate the runner ID against `GET /api/v1/admin/runners/{id}` using the core token:
- **200**: runner still registered — write `TOKEN=placeholder` to `/run/hive-ci/runner-token` and exit; runner reuses `.runner` credentials.
- **404**: runner was deleted from forge (e.g. after a wipe) — delete `.runner`, proceed to re-registration below.
- **000** (forge unreachable): keep existing `.runner`; the runner itself will surface the connectivity error.
- **000** (forge unreachable): keep existing `.runner`; write placeholder token; the runner itself will surface the connectivity error.
- **other non-200**: treat as stale, delete `.runner` and re-register.
- **malformed `.runner`** (no `id` field): delete and re-register.
3. If `.runner` is absent (first boot or purged above): fetch a fresh registration token from `GET /api/v1/admin/runners/registration-token`. Retries for 30s in case forge is still starting. Writes `TOKEN=<real>` to `/run/hive-ci/runner-token`.
4. `gitea-actions-runner` reads the token, registers itself, and persists credentials to `.runner`. On subsequent boots step 2 validates these credentials and fast-paths past registration.
4. Container starts with `/run/hive-ci/runner-token` bind-mounted read-only. `gitea-actions-runner` reads the token, registers itself, and persists credentials to `.runner`. On subsequent boots step 2 validates these credentials and fast-paths past registration.
## CI workflow
@ -63,10 +64,11 @@ nspawn containers cannot create user-namespaces, so `nix.settings.sandbox-fallba
A malicious `default.nix` or build script in a PR can therefore:
- **Read the core admin token** at `/run/hive-ci/core-token` (bind-mounted into the container for runner auto-registration). This token has `read:admin + write:admin` scopes on hive-forge — enough to read any repo, enumerate users, and issue forge admin API calls.
- **Make arbitrary network requests** to any address reachable from the container. The container shares host netns, so `http://127.0.0.1:<forgePort>` is reachable.
- **Make arbitrary network requests** to any address reachable from the container. The container shares host netns, so `http://127.0.0.1:<forgePort>` is reachable with the forge API — without admin credentials, but public/read endpoints are accessible.
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
The core admin token (`forge-core-token`) is **not** bind-mounted into the container. It is accessed only by the host-side `hive-ci-prefetch.service` before the container starts. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls.
Note: `nix flake check --no-build` (eval-only) reduces the attack surface but does not eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.
### Mitigation
@ -77,7 +79,6 @@ For repos with external contributors or fork PRs:
- Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
- Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches.
- As a structural fix, move `core-token` out of the container bind-mount tree and use a scoped registration-only token. That work is tracked separately.
The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.