vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: move core-token out of hive-ci container (host-side prefetch service)

Browse source
This commit is contained in:
atlas 2026-06-03 22:14:17 +02:00 • committed by mara
commit 9209094397
2 changed files with 113 additions and 87 deletions
Download patch file Download diff file Expand all files Collapse all files

19
docs/ci.md
View file

@ -15,22 +15,23 @@ Set `services.hyperhive.ci.enable = true` in the host NixOS config. That's it
- **Shared host netns**: container reaches hive-forge at `http://127.0.0.1:<httpPort>` (same as hive-gateway).
- **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/gotchas.md`.
- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. A host-side oneshot service (`hive-ci-prefetch.service`) performs all forge API calls and writes only the runner registration token into the container via a read-only bind-mount at `/run/hive-ci/runner-token`.
## Auto-registration flow
`hive-ci-register.service` is a oneshot that runs on **every boot** before `gitea-runner-hive.service`. It handles both first-run registration and stale-credential detection.
`hive-ci-prefetch.service` is a host-side oneshot that runs on **every boot** before `nixos-container@hive-ci.service`. It handles both first-run registration and stale-credential detection. The core admin token is accessed only on the host and never bind-mounted into the container.
### Every boot
1. Read the core admin token from `/run/hive-ci/core-token` (bind-mounted from `/var/lib/hyperhive/forge-core-token`).
2. If `.runner` exists: validate the runner ID against `GET /api/v1/admin/runners/{id}` using the core token:
- **200**: runner still registered — write dummy `TOKEN=placeholder` and exit; runner reuses `.runner` credentials.
1. Check for the core admin token at `/var/lib/hyperhive/forge-core-token`. If absent (forge still initialising), write `TOKEN=placeholder` and exit — the runner service will fail gracefully until the next boot.
2. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`: validate the runner ID against `GET /api/v1/admin/runners/{id}` using the core token:
- **200**: runner still registered — write `TOKEN=placeholder` to `/run/hive-ci/runner-token` and exit; runner reuses `.runner` credentials.
- **404**: runner was deleted from forge (e.g. after a wipe) — delete `.runner`, proceed to re-registration below.
- **000** (forge unreachable): keep existing `.runner`; the runner itself will surface the connectivity error.
- **000** (forge unreachable): keep existing `.runner`; write placeholder token; the runner itself will surface the connectivity error.
- **other non-200**: treat as stale, delete `.runner` and re-register.
- **malformed `.runner`** (no `id` field): delete and re-register.
3. If `.runner` is absent (first boot or purged above): fetch a fresh registration token from `GET /api/v1/admin/runners/registration-token`. Retries for 30s in case forge is still starting. Writes `TOKEN=<real>` to `/run/hive-ci/runner-token`.
4. `gitea-actions-runner` reads the token, registers itself, and persists credentials to `.runner`. On subsequent boots step 2 validates these credentials and fast-paths past registration.
4. Container starts with `/run/hive-ci/runner-token` bind-mounted read-only. `gitea-actions-runner` reads the token, registers itself, and persists credentials to `.runner`. On subsequent boots step 2 validates these credentials and fast-paths past registration.
## CI workflow
@ -63,10 +64,11 @@ nspawn containers cannot create user-namespaces, so `nix.settings.sandbox-fallba
A malicious `default.nix` or build script in a PR can therefore:
- **Read the core admin token** at `/run/hive-ci/core-token` (bind-mounted into the container for runner auto-registration). This token has `read:admin + write:admin` scopes on hive-forge — enough to read any repo, enumerate users, and issue forge admin API calls.
- **Make arbitrary network requests** to any address reachable from the container. The container shares host netns, so `http://127.0.0.1:<forgePort>` is reachable.
- **Make arbitrary network requests** to any address reachable from the container. The container shares host netns, so `http://127.0.0.1:<forgePort>` is reachable with the forge API — without admin credentials, but public/read endpoints are accessible.
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
The core admin token (`forge-core-token`) is **not** bind-mounted into the container. It is accessed only by the host-side `hive-ci-prefetch.service` before the container starts. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls.
Note: `nix flake check --no-build` (eval-only) reduces the attack surface but does not eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.
### Mitigation
@ -77,7 +79,6 @@ For repos with external contributors or fork PRs:
- Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
- Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches.
- As a structural fix, move `core-token` out of the container bind-mount tree and use a scoped registration-only token. That work is tracked separately.
The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.

181
nix/modules/hive-ci.nix
View file

@ -11,60 +11,77 @@ let
# hive-c0re writes its own admin token here on first forge startup.
# The token has read:admin + write:admin scopes — sufficient to call
# the runner registration-token API endpoint.
# This path is HOST-ONLY. It is never bind-mounted into hive-ci.
coreTokenPath = "/var/lib/hyperhive/forge-core-token";
# Oneshot run before gitea-runner-hive.service on every boot.
# Container state root on the host. Non-ephemeral containers keep
# their filesystem here across reboots. The prefetch service accesses
# the runner's .runner file via this path to validate credentials
# without entering the container.
containerRoot = "/var/lib/nixos-containers/hive-ci";
# Host-side oneshot. Runs before `nixos-container@hive-ci.service`.
#
# The nixpkgs gitea-actions-runner module uses tokenFile as a systemd
# EnvironmentFile (sets TOKEN= in the environment). EnvironmentFile is
# loaded before any ExecStartPre, so the file MUST exist at service start —
# an ExecStartPre override is too late. We solve this with:
# 1. tmpfiles: pre-create /run/hive-ci/runner-token with TOKEN=placeholder
# 2. hive-ci-register.service (this script): runs on every boot:
# a. If .runner exists: validate the runner ID against the forge admin
# API. If the runner was deleted from forge (404), delete .runner so
# the next step re-registers. If forge is unreachable (000), keep
# the existing credentials (the runner itself will surface the error).
# If valid, write a dummy token and exit — runner reuses .runner creds.
# b. If .runner absent (first boot or stale creds were purged): fetch a
# fresh registration token and write TOKEN=<real> to the EnvironmentFile.
# 3. gitea-runner-hive.service: After=hive-ci-register.service
registerScript = pkgs.writeShellScript "hive-ci-register" ''
# The core token stays on the host. The container only ever sees the
# TOKEN= env-file populated here, never the core token itself.
#
# Flow:
# 1. If forge-core-token doesn't exist yet (forge still initialising):
# write TOKEN=placeholder and exit — the container starts but the
# runner will fail to register. systemd will restart on next boot.
# 2. If .runner exists: validate the runner ID against forge.
# 404 → purge .runner (re-registration needed).
# 000 (forge unreachable) → keep credentials, write placeholder.
# other non-200 → purge .runner.
# 3. If .runner absent or just purged: fetch a fresh registration token
# (retry 30s for forge to start) and write TOKEN=<real>.
prefetchScript = pkgs.writeShellScript "hive-ci-prefetch" ''
set -euo pipefail
TOKEN_FILE=/run/hive-ci/runner-token
CORE_TOKEN=$(cat /run/hive-ci/core-token)
FORGE_URL="http://127.0.0.1:${toString forgeCfg.httpPort}"
RUNNER_FILE=/var/lib/gitea-runner/hive/.runner
RUNNER_FILE="${containerRoot}/var/lib/gitea-runner/hive/.runner"
# If .runner exists, validate runner credentials against forge.
# Delete .runner if the runner entry is gone (404) — this triggers
# fresh re-registration below. Skip validation if forge is unreachable
# (000) so a transient forge outage doesn't wipe valid credentials.
# Guard: if the core token hasn't been written yet (forge still
# initialising) write a placeholder and exit cleanly. The runner
# service will fail gracefully; the next boot will re-run this
# service and pick up the real token.
if [ ! -f "${coreTokenPath}" ]; then
echo "hive-ci-prefetch: core token absent, deferring registration" >&2
echo "TOKEN=placeholder" > "$TOKEN_FILE"
chmod 600 "$TOKEN_FILE"
exit 0
fi
CORE_TOKEN=$(cat ${coreTokenPath})
# Validate existing .runner credentials against forge. Purge if
# the runner was deleted (404) or the file is malformed. Keep if
# forge is unreachable (000) — transient outage shouldn't wipe creds.
if [ -f "$RUNNER_FILE" ]; then
RUNNER_ID=$(${pkgs.jq}/bin/jq -r '.id // empty' "$RUNNER_FILE" 2>/dev/null || true)
if [ -z "''${RUNNER_ID:-}" ] || [ "$RUNNER_ID" = "0" ]; then
echo "hive-ci: .runner malformed (no id), purging for re-registration" >&2
echo "hive-ci-prefetch: .runner malformed (no id), purging for re-registration" >&2
rm -f "$RUNNER_FILE"
else
HTTP=$(${pkgs.curl}/bin/curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: token $CORE_TOKEN" \
"$FORGE_URL/api/v1/admin/runners/$RUNNER_ID" || echo "000")
if [ "$HTTP" = "404" ]; then
echo "hive-ci: runner $RUNNER_ID not found in forge, purging .runner" >&2
echo "hive-ci-prefetch: runner $RUNNER_ID gone from forge, purging .runner" >&2
rm -f "$RUNNER_FILE"
elif [ "$HTTP" = "000" ]; then
echo "hive-ci: forge unreachable, keeping existing .runner credentials" >&2
echo "hive-ci-prefetch: forge unreachable, keeping existing credentials" >&2
elif [ "$HTTP" != "200" ]; then
echo "hive-ci: runner validation returned HTTP $HTTP, purging .runner" >&2
echo "hive-ci-prefetch: runner validation HTTP $HTTP, purging .runner" >&2
rm -f "$RUNNER_FILE"
fi
fi
fi
# If .runner still valid, write dummy token and exit — nixpkgs register
# step exits early when .runner exists; TOKEN value is irrelevant.
# .runner valid → write placeholder; gitea-actions-runner skips
# re-registration when .runner exists (TOKEN value is irrelevant).
if [ -f "$RUNNER_FILE" ]; then
echo "TOKEN=placeholder" > "$TOKEN_FILE"
chmod 600 "$TOKEN_FILE"
exit 0
fi
@ -80,13 +97,12 @@ let
done
if [ -z "''${REG_TOKEN:-}" ] || [ "$REG_TOKEN" = "null" ]; then
echo "hive-ci: failed to fetch runner registration token from forge" >&2
echo "hive-ci-prefetch: failed to fetch runner registration token from forge" >&2
exit 1
fi
# Write in EnvironmentFile format: TOKEN=<value>.
# File is already 0600 (set by tmpfiles on boot).
echo "TOKEN=$REG_TOKEN" > "$TOKEN_FILE"
chmod 600 "$TOKEN_FILE"
'';
in
{
@ -97,13 +113,12 @@ in
# survive restarts (gitea-actions-runner writes them to its stateDir
# on first registration and reuses them on every subsequent start).
#
# Auto-registration: on every boot a oneshot service validates existing
# runner credentials against forge (purging stale/.runner if the runner
# was deleted from forge). On first boot (or after credential purge) it
# fetches a fresh registration token via the forge admin API, using the
# core token hive-c0re writes to /var/lib/hyperhive/forge-core-token.
# No manual token handling needed — `forge.ci.enable = true` is the
# full operator bootstrap.
# Credential isolation: the forge admin token (`forge-core-token`)
# never enters the hive-ci container. A host-side oneshot service
# (`hive-ci-prefetch.service`) performs all forge API calls before
# the container starts and writes only the runner registration token
# to `/run/hive-ci/runner-token`. The container bind-mounts this
# file read-only and never has access to the wider admin token.
#
# Nix builds inside the container use the shared /nix/store (standard
# nixos-container behaviour) with sandbox-fallback = true, because
@ -185,6 +200,39 @@ in
}
];
# Create /run/hive-ci/ on the host and seed runner-token with a
# placeholder. hive-ci-prefetch.service overwrites it with the real
# token (or a fresh placeholder) before the container starts. The
# placeholder ensures the EnvironmentFile is always present even if
# the prefetch service hasn't run yet (e.g. tmpfiles-setup timing).
systemd.tmpfiles.rules = [
"d /run/hive-ci 0700 root root -"
"f /run/hive-ci/runner-token 0600 root root - TOKEN=placeholder"
];
# Host-side oneshot: validates/refreshes runner credentials before
# the container starts. The core admin token stays on the host and
# is never bind-mounted into the container. Runs on every boot so
# stale .runner credentials (runner deleted from forge) are detected
# and the container re-registers on the next start.
systemd.services.hive-ci-prefetch = {
description = "Pre-fetch hive-ci runner registration token (host-side)";
# Run before the container starts but after tmpfiles so the token
# file directory exists. After hive-c0re so the forge token is
# likely written (best-effort — the script handles the absent case).
after = [
"systemd-tmpfiles-setup.service"
"hive-c0re.service"
];
before = [ "nixos-container@hive-ci.service" ];
wantedBy = [ "nixos-container@hive-ci.service" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
ExecStart = prefetchScript;
};
};
containers.hive-ci = {
autoStart = true;
ephemeral = false;
@ -192,10 +240,11 @@ in
privateNetwork = false;
bindMounts = {
# Core token — used by hive-ci-register.service on first boot.
# Read-only: the container only reads it, never modifies it.
"/run/hive-ci/core-token" = {
hostPath = coreTokenPath;
# Pre-filled by hive-ci-prefetch.service (host-side) before the
# container starts. Read-only: the container reads TOKEN= from
# here; the core admin token never enters the container.
"/run/hive-ci/runner-token" = {
hostPath = "/run/hive-ci/runner-token";
isReadOnly = true;
};
};
@ -221,9 +270,10 @@ in
enable = true;
name = cfg.name;
url = "http://127.0.0.1:${toString forgeCfg.httpPort}";
# EnvironmentFile providing TOKEN= for the nixpkgs register step.
# Pre-created by tmpfiles (placeholder); overwritten with the real
# token by hive-ci-register.service on first boot only.
# EnvironmentFile providing TOKEN= — pre-filled by the
# host-side hive-ci-prefetch.service before the container
# starts; bind-mounted read-only from /run/hive-ci/runner-token
# on the host.
tokenFile = "/run/hive-ci/runner-token";
labels = cfg.labels;
settings = {
@ -233,39 +283,15 @@ in
};
};
# Pre-create the EnvironmentFile so gitea-runner-hive.service can
# always load it. The nixpkgs module sets EnvironmentFile=tokenFile
# which systemd loads before any ExecStartPre — the file must exist
# at service-start time. hive-ci-register.service overwrites this
# placeholder before the runner starts (real token on first/re-reg,
# dummy placeholder when .runner credentials are still valid).
#
# Note: `f` doesn't create parent directories, but /run/hive-ci/
# is guaranteed to exist by the time container systemd starts:
# nspawn creates mount-point directories for all bindMounts before
# launching the container's init. So the dir is there when
# systemd-tmpfiles-setup.service runs.
systemd.tmpfiles.rules = [
"f /run/hive-ci/runner-token 0600 root root - TOKEN=placeholder"
];
# No tmpfiles rule: /run/hive-ci/runner-token is bind-mounted
# read-only from the host (pre-filled before container start).
# nspawn creates the /run/hive-ci/ mount-point directory
# automatically before launching the container's init.
# Oneshot that validates/refreshes runner credentials on every boot:
# validates existing .runner against forge (purges if stale/404),
# then fetches a fresh registration token if .runner is absent.
# Runs before gitea-runner-hive.service via the After= dependency.
systemd.services.hive-ci-register = {
description = "Validate/register hive-ci runner against Forgejo";
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
ExecStart = registerScript;
};
};
# Runner must start after the register oneshot so the EnvironmentFile
# contains the real token on first boot.
systemd.services."gitea-runner-hive".after = [ "hive-ci-register.service" ];
systemd.services."gitea-runner-hive".wants = [ "hive-ci-register.service" ];
# No hive-ci-register.service inside the container: all forge
# API calls (runner validation, token fetch) moved to the
# host-side hive-ci-prefetch.service. The core admin token
# never enters this container.
# git is already in the gitea-actions-runner service PATH (the
# nixpkgs module builds it from the package's runtime deps).
@ -275,7 +301,6 @@ in
# to prepend nix's bin dir to PATH without touching the
# environment.PATH the nixpkgs module sets — overriding that
# would lose git, curl, nodejs, and other runner deps.
# curl/jq in the register script use absolute store paths.
environment.systemPackages = [
pkgs.git
pkgs.nix