From 92090943970c5cbf3a47b9d2afb7f042ed99b2ea Mon Sep 17 00:00:00 2001 From: atlas Date: Wed, 3 Jun 2026 22:14:17 +0200 Subject: [PATCH] fix: move core-token out of hive-ci container (host-side prefetch service) --- docs/ci.md | 19 +++-- nix/modules/hive-ci.nix | 181 +++++++++++++++++++++++----------------- 2 files changed, 113 insertions(+), 87 deletions(-) diff --git a/docs/ci.md b/docs/ci.md index 9b43f675..66292879 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -15,22 +15,23 @@ Set `services.hyperhive.ci.enable = true` in the host NixOS config. That's it - **Shared host netns**: container reaches hive-forge at `http://127.0.0.1:` (same as hive-gateway). - **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter). - **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/gotchas.md`. +- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. A host-side oneshot service (`hive-ci-prefetch.service`) performs all forge API calls and writes only the runner registration token into the container via a read-only bind-mount at `/run/hive-ci/runner-token`. ## Auto-registration flow -`hive-ci-register.service` is a oneshot that runs on **every boot** before `gitea-runner-hive.service`. It handles both first-run registration and stale-credential detection. +`hive-ci-prefetch.service` is a host-side oneshot that runs on **every boot** before `nixos-container@hive-ci.service`. It handles both first-run registration and stale-credential detection. The core admin token is accessed only on the host and never bind-mounted into the container. ### Every boot -1. Read the core admin token from `/run/hive-ci/core-token` (bind-mounted from `/var/lib/hyperhive/forge-core-token`). -2. If `.runner` exists: validate the runner ID against `GET /api/v1/admin/runners/{id}` using the core token: - - **200**: runner still registered — write dummy `TOKEN=placeholder` and exit; runner reuses `.runner` credentials. +1. Check for the core admin token at `/var/lib/hyperhive/forge-core-token`. If absent (forge still initialising), write `TOKEN=placeholder` and exit — the runner service will fail gracefully until the next boot. +2. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`: validate the runner ID against `GET /api/v1/admin/runners/{id}` using the core token: + - **200**: runner still registered — write `TOKEN=placeholder` to `/run/hive-ci/runner-token` and exit; runner reuses `.runner` credentials. - **404**: runner was deleted from forge (e.g. after a wipe) — delete `.runner`, proceed to re-registration below. - - **000** (forge unreachable): keep existing `.runner`; the runner itself will surface the connectivity error. + - **000** (forge unreachable): keep existing `.runner`; write placeholder token; the runner itself will surface the connectivity error. - **other non-200**: treat as stale, delete `.runner` and re-register. - **malformed `.runner`** (no `id` field): delete and re-register. 3. If `.runner` is absent (first boot or purged above): fetch a fresh registration token from `GET /api/v1/admin/runners/registration-token`. Retries for 30s in case forge is still starting. Writes `TOKEN=` to `/run/hive-ci/runner-token`. -4. `gitea-actions-runner` reads the token, registers itself, and persists credentials to `.runner`. On subsequent boots step 2 validates these credentials and fast-paths past registration. +4. Container starts with `/run/hive-ci/runner-token` bind-mounted read-only. `gitea-actions-runner` reads the token, registers itself, and persists credentials to `.runner`. On subsequent boots step 2 validates these credentials and fast-paths past registration. ## CI workflow @@ -63,10 +64,11 @@ nspawn containers cannot create user-namespaces, so `nix.settings.sandbox-fallba A malicious `default.nix` or build script in a PR can therefore: -- **Read the core admin token** at `/run/hive-ci/core-token` (bind-mounted into the container for runner auto-registration). This token has `read:admin + write:admin` scopes on hive-forge — enough to read any repo, enumerate users, and issue forge admin API calls. -- **Make arbitrary network requests** to any address reachable from the container. The container shares host netns, so `http://127.0.0.1:` is reachable. +- **Make arbitrary network requests** to any address reachable from the container. The container shares host netns, so `http://127.0.0.1:` is reachable with the forge API — without admin credentials, but public/read endpoints are accessible. - **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials. +The core admin token (`forge-core-token`) is **not** bind-mounted into the container. It is accessed only by the host-side `hive-ci-prefetch.service` before the container starts. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls. + Note: `nix flake check --no-build` (eval-only) reduces the attack surface but does not eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path. ### Mitigation @@ -77,7 +79,6 @@ For repos with external contributors or fork PRs: - Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR. - Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches. -- As a structural fix, move `core-token` out of the container bind-mount tree and use a scoped registration-only token. That work is tracked separately. The current design is appropriate for a trusted-team hive where all contributors have implicit forge access. diff --git a/nix/modules/hive-ci.nix b/nix/modules/hive-ci.nix index 88d42838..0a3f18cc 100644 --- a/nix/modules/hive-ci.nix +++ b/nix/modules/hive-ci.nix @@ -11,60 +11,77 @@ let # hive-c0re writes its own admin token here on first forge startup. # The token has read:admin + write:admin scopes — sufficient to call # the runner registration-token API endpoint. + # This path is HOST-ONLY. It is never bind-mounted into hive-ci. coreTokenPath = "/var/lib/hyperhive/forge-core-token"; - # Oneshot run before gitea-runner-hive.service on every boot. + # Container state root on the host. Non-ephemeral containers keep + # their filesystem here across reboots. The prefetch service accesses + # the runner's .runner file via this path to validate credentials + # without entering the container. + containerRoot = "/var/lib/nixos-containers/hive-ci"; + + # Host-side oneshot. Runs before `nixos-container@hive-ci.service`. # - # The nixpkgs gitea-actions-runner module uses tokenFile as a systemd - # EnvironmentFile (sets TOKEN= in the environment). EnvironmentFile is - # loaded before any ExecStartPre, so the file MUST exist at service start — - # an ExecStartPre override is too late. We solve this with: - # 1. tmpfiles: pre-create /run/hive-ci/runner-token with TOKEN=placeholder - # 2. hive-ci-register.service (this script): runs on every boot: - # a. If .runner exists: validate the runner ID against the forge admin - # API. If the runner was deleted from forge (404), delete .runner so - # the next step re-registers. If forge is unreachable (000), keep - # the existing credentials (the runner itself will surface the error). - # If valid, write a dummy token and exit — runner reuses .runner creds. - # b. If .runner absent (first boot or stale creds were purged): fetch a - # fresh registration token and write TOKEN= to the EnvironmentFile. - # 3. gitea-runner-hive.service: After=hive-ci-register.service - registerScript = pkgs.writeShellScript "hive-ci-register" '' + # The core token stays on the host. The container only ever sees the + # TOKEN= env-file populated here, never the core token itself. + # + # Flow: + # 1. If forge-core-token doesn't exist yet (forge still initialising): + # write TOKEN=placeholder and exit — the container starts but the + # runner will fail to register. systemd will restart on next boot. + # 2. If .runner exists: validate the runner ID against forge. + # 404 → purge .runner (re-registration needed). + # 000 (forge unreachable) → keep credentials, write placeholder. + # other non-200 → purge .runner. + # 3. If .runner absent or just purged: fetch a fresh registration token + # (retry 30s for forge to start) and write TOKEN=. + prefetchScript = pkgs.writeShellScript "hive-ci-prefetch" '' set -euo pipefail TOKEN_FILE=/run/hive-ci/runner-token - CORE_TOKEN=$(cat /run/hive-ci/core-token) FORGE_URL="http://127.0.0.1:${toString forgeCfg.httpPort}" - RUNNER_FILE=/var/lib/gitea-runner/hive/.runner + RUNNER_FILE="${containerRoot}/var/lib/gitea-runner/hive/.runner" - # If .runner exists, validate runner credentials against forge. - # Delete .runner if the runner entry is gone (404) — this triggers - # fresh re-registration below. Skip validation if forge is unreachable - # (000) so a transient forge outage doesn't wipe valid credentials. + # Guard: if the core token hasn't been written yet (forge still + # initialising) write a placeholder and exit cleanly. The runner + # service will fail gracefully; the next boot will re-run this + # service and pick up the real token. + if [ ! -f "${coreTokenPath}" ]; then + echo "hive-ci-prefetch: core token absent, deferring registration" >&2 + echo "TOKEN=placeholder" > "$TOKEN_FILE" + chmod 600 "$TOKEN_FILE" + exit 0 + fi + CORE_TOKEN=$(cat ${coreTokenPath}) + + # Validate existing .runner credentials against forge. Purge if + # the runner was deleted (404) or the file is malformed. Keep if + # forge is unreachable (000) — transient outage shouldn't wipe creds. if [ -f "$RUNNER_FILE" ]; then RUNNER_ID=$(${pkgs.jq}/bin/jq -r '.id // empty' "$RUNNER_FILE" 2>/dev/null || true) if [ -z "''${RUNNER_ID:-}" ] || [ "$RUNNER_ID" = "0" ]; then - echo "hive-ci: .runner malformed (no id), purging for re-registration" >&2 + echo "hive-ci-prefetch: .runner malformed (no id), purging for re-registration" >&2 rm -f "$RUNNER_FILE" else HTTP=$(${pkgs.curl}/bin/curl -s -o /dev/null -w "%{http_code}" \ -H "Authorization: token $CORE_TOKEN" \ "$FORGE_URL/api/v1/admin/runners/$RUNNER_ID" || echo "000") if [ "$HTTP" = "404" ]; then - echo "hive-ci: runner $RUNNER_ID not found in forge, purging .runner" >&2 + echo "hive-ci-prefetch: runner $RUNNER_ID gone from forge, purging .runner" >&2 rm -f "$RUNNER_FILE" elif [ "$HTTP" = "000" ]; then - echo "hive-ci: forge unreachable, keeping existing .runner credentials" >&2 + echo "hive-ci-prefetch: forge unreachable, keeping existing credentials" >&2 elif [ "$HTTP" != "200" ]; then - echo "hive-ci: runner validation returned HTTP $HTTP, purging .runner" >&2 + echo "hive-ci-prefetch: runner validation HTTP $HTTP, purging .runner" >&2 rm -f "$RUNNER_FILE" fi fi fi - # If .runner still valid, write dummy token and exit — nixpkgs register - # step exits early when .runner exists; TOKEN value is irrelevant. + # .runner valid → write placeholder; gitea-actions-runner skips + # re-registration when .runner exists (TOKEN value is irrelevant). if [ -f "$RUNNER_FILE" ]; then echo "TOKEN=placeholder" > "$TOKEN_FILE" + chmod 600 "$TOKEN_FILE" exit 0 fi @@ -80,13 +97,12 @@ let done if [ -z "''${REG_TOKEN:-}" ] || [ "$REG_TOKEN" = "null" ]; then - echo "hive-ci: failed to fetch runner registration token from forge" >&2 + echo "hive-ci-prefetch: failed to fetch runner registration token from forge" >&2 exit 1 fi - # Write in EnvironmentFile format: TOKEN=. - # File is already 0600 (set by tmpfiles on boot). echo "TOKEN=$REG_TOKEN" > "$TOKEN_FILE" + chmod 600 "$TOKEN_FILE" ''; in { @@ -97,13 +113,12 @@ in # survive restarts (gitea-actions-runner writes them to its stateDir # on first registration and reuses them on every subsequent start). # - # Auto-registration: on every boot a oneshot service validates existing - # runner credentials against forge (purging stale/.runner if the runner - # was deleted from forge). On first boot (or after credential purge) it - # fetches a fresh registration token via the forge admin API, using the - # core token hive-c0re writes to /var/lib/hyperhive/forge-core-token. - # No manual token handling needed — `forge.ci.enable = true` is the - # full operator bootstrap. + # Credential isolation: the forge admin token (`forge-core-token`) + # never enters the hive-ci container. A host-side oneshot service + # (`hive-ci-prefetch.service`) performs all forge API calls before + # the container starts and writes only the runner registration token + # to `/run/hive-ci/runner-token`. The container bind-mounts this + # file read-only and never has access to the wider admin token. # # Nix builds inside the container use the shared /nix/store (standard # nixos-container behaviour) with sandbox-fallback = true, because @@ -185,6 +200,39 @@ in } ]; + # Create /run/hive-ci/ on the host and seed runner-token with a + # placeholder. hive-ci-prefetch.service overwrites it with the real + # token (or a fresh placeholder) before the container starts. The + # placeholder ensures the EnvironmentFile is always present even if + # the prefetch service hasn't run yet (e.g. tmpfiles-setup timing). + systemd.tmpfiles.rules = [ + "d /run/hive-ci 0700 root root -" + "f /run/hive-ci/runner-token 0600 root root - TOKEN=placeholder" + ]; + + # Host-side oneshot: validates/refreshes runner credentials before + # the container starts. The core admin token stays on the host and + # is never bind-mounted into the container. Runs on every boot so + # stale .runner credentials (runner deleted from forge) are detected + # and the container re-registers on the next start. + systemd.services.hive-ci-prefetch = { + description = "Pre-fetch hive-ci runner registration token (host-side)"; + # Run before the container starts but after tmpfiles so the token + # file directory exists. After hive-c0re so the forge token is + # likely written (best-effort — the script handles the absent case). + after = [ + "systemd-tmpfiles-setup.service" + "hive-c0re.service" + ]; + before = [ "nixos-container@hive-ci.service" ]; + wantedBy = [ "nixos-container@hive-ci.service" ]; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + ExecStart = prefetchScript; + }; + }; + containers.hive-ci = { autoStart = true; ephemeral = false; @@ -192,10 +240,11 @@ in privateNetwork = false; bindMounts = { - # Core token — used by hive-ci-register.service on first boot. - # Read-only: the container only reads it, never modifies it. - "/run/hive-ci/core-token" = { - hostPath = coreTokenPath; + # Pre-filled by hive-ci-prefetch.service (host-side) before the + # container starts. Read-only: the container reads TOKEN= from + # here; the core admin token never enters the container. + "/run/hive-ci/runner-token" = { + hostPath = "/run/hive-ci/runner-token"; isReadOnly = true; }; }; @@ -221,9 +270,10 @@ in enable = true; name = cfg.name; url = "http://127.0.0.1:${toString forgeCfg.httpPort}"; - # EnvironmentFile providing TOKEN= for the nixpkgs register step. - # Pre-created by tmpfiles (placeholder); overwritten with the real - # token by hive-ci-register.service on first boot only. + # EnvironmentFile providing TOKEN= — pre-filled by the + # host-side hive-ci-prefetch.service before the container + # starts; bind-mounted read-only from /run/hive-ci/runner-token + # on the host. tokenFile = "/run/hive-ci/runner-token"; labels = cfg.labels; settings = { @@ -233,39 +283,15 @@ in }; }; - # Pre-create the EnvironmentFile so gitea-runner-hive.service can - # always load it. The nixpkgs module sets EnvironmentFile=tokenFile - # which systemd loads before any ExecStartPre — the file must exist - # at service-start time. hive-ci-register.service overwrites this - # placeholder before the runner starts (real token on first/re-reg, - # dummy placeholder when .runner credentials are still valid). - # - # Note: `f` doesn't create parent directories, but /run/hive-ci/ - # is guaranteed to exist by the time container systemd starts: - # nspawn creates mount-point directories for all bindMounts before - # launching the container's init. So the dir is there when - # systemd-tmpfiles-setup.service runs. - systemd.tmpfiles.rules = [ - "f /run/hive-ci/runner-token 0600 root root - TOKEN=placeholder" - ]; + # No tmpfiles rule: /run/hive-ci/runner-token is bind-mounted + # read-only from the host (pre-filled before container start). + # nspawn creates the /run/hive-ci/ mount-point directory + # automatically before launching the container's init. - # Oneshot that validates/refreshes runner credentials on every boot: - # validates existing .runner against forge (purges if stale/404), - # then fetches a fresh registration token if .runner is absent. - # Runs before gitea-runner-hive.service via the After= dependency. - systemd.services.hive-ci-register = { - description = "Validate/register hive-ci runner against Forgejo"; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - ExecStart = registerScript; - }; - }; - - # Runner must start after the register oneshot so the EnvironmentFile - # contains the real token on first boot. - systemd.services."gitea-runner-hive".after = [ "hive-ci-register.service" ]; - systemd.services."gitea-runner-hive".wants = [ "hive-ci-register.service" ]; + # No hive-ci-register.service inside the container: all forge + # API calls (runner validation, token fetch) moved to the + # host-side hive-ci-prefetch.service. The core admin token + # never enters this container. # git is already in the gitea-actions-runner service PATH (the # nixpkgs module builds it from the package's runtime deps). @@ -275,7 +301,6 @@ in # to prepend nix's bin dir to PATH without touching the # environment.PATH the nixpkgs module sets — overriding that # would lose git, curl, nodejs, and other runner deps. - # curl/jq in the register script use absolute store paths. environment.systemPackages = [ pkgs.git pkgs.nix