swarm-ci-runner: run the forge CI runner as a swarm service

The CI runner was a hive deploy module: hive-ci.nix ran it beside the
forge, and hive-c0re registered it through the forge's admin API with
the `core` token, writing the registration token through hive-priv. It
could only run on the forge's host, and it kept an admin-scoped forge
token on a hive.

Now the runner is a swarm service:

- swarm-ci-runner.nix runs it in a `swarm-ci-runner` container on the
  host that sets deploy.ci-runner.enable, reaching the forge by name.
  It needs neither the forge nor a hive there.
- swarm-controller reads the forge's runner registration token with its
  own admin token and writes it to swarm/services/ci-runner/
  registration-token whenever it differs from what it last wrote. Its
  grant on that path is create/update only.
- The runner's own leaf (CN swarm-ci-runner, role and policy of the same
  name) reads that one path. swarm-bao-ci-runner.service fetches it into
  the runner's EnvironmentFile and restarts the runner on a change.
- A pre-start step removes .runner when the daemon's journal since its
  last start holds Forgejo's "unregistered runner" refusal (the daemon
  exits 1 when its startup Declare gets it), or when .runner names
  another forge address; upstream's pre-start step then registers again.

Removed from the hive side: hive-ci.nix and deploy.forgejo.ci.*,
forge/ci_runner.rs, PrivRequest::RegisterCiRunner and its handler and
ReadWritePaths entry, InfraContainer::Ci with hivectl's --ci and the
dashboard's hive-ci journal entry, HYPERHIVE_FORGE_CI_ENABLED, and
read:admin,write:admin from CORE_TOKEN_SCOPES: nothing else in hive-c0re
calls /api/v1/admin/*. The old ci options, under both swarm.forge.ci and
deploy.forgejo.ci, now fail evaluation naming deploy.ci-runner.

The forge's half of the old ci.enable, the actions/checkout mirror and
DEFAULT_ACTIONS_URL = self, is deploy.forgejo.actionsFromSelf.

Closes #4954

docs/scheduler/ci.md

@@ -1,13 +1,21 @@
-# hive-ci: Forgejo Actions Runner
+# swarm-ci-runner: Forgejo Actions Runner
 
-The `hive-ci` module runs a Forgejo Actions runner in a `hive-ci` nixos-container,
-executing CI jobs from `.forgejo/workflows/ci.yml` on every PR.
+The `swarm-ci-runner` module runs the swarm's Forgejo Actions runner in a
+`swarm-ci-runner` nixos-container, executing CI jobs from
+`.forgejo/workflows/ci.yml` on every PR. It's a swarm service: it runs on
+whichever host sets it, and needs neither the forge nor a hive on that host.
 
 ## For operators
 
-**Enabling it's one line**: `services.hyperhive.deploy.forgejo.ci.enable = true`
-in the host NixOS config. No manual token provisioning — hive-c0re
-registers the runner with the forge automatically.
+**Enabling it's one line**: `services.hyperhive.deploy.ci-runner.enable = true`
+in the NixOS config of the host that should run it. No manual token
+provisioning: swarm-controller publishes the forge's runner registration token
+to the secret store, and the runner reads it with a store identity of its own.
+On the store's host `glue-bao-tls.nix` mints that identity; on any other host, copy
+`ci-runner.pem` and `ci-runner-key.pem` from the store host's
+`/var/lib/swarm-bao-pki/` and name them in
+`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
+`ciRunnerClientKeyFile`.
 
 Two things worth knowing before you rely on it:
 
@@ -25,9 +33,8 @@ Two things worth knowing before you rely on it:
   disk-pressure GC config from _Host store maintenance_ below to your
   host's NixOS config (not optional if you plan to actually use this).
 
-Everything below this point is implementation detail: exact
-autoregistration internals, container design, and the full security
-writeup.
+Everything below this point is implementation detail: registration
+internals, container design, and the full security writeup.
 
 ## CI checks
 
@@ -105,62 +112,68 @@ slow); run those manually before pushing Rust changes.
 
 ## Configuration reference
 
-The runner registers against the forge on its own host, so enable it on the
-swarm's forge host (`services.hyperhive.deploy.forgejo.enable`, see
-[`../swarm/services.md`](../swarm/services.md)). Evaluation fails on any other
-host. There, nothing extra to enable beyond
-`services.hyperhive.deploy.forgejo.ci.enable = true` (see _For operators_ above).
+`services.hyperhive.deploy.ci-runner.enable = true` on the host that runs it
+(see _For operators_ above). The runner reaches the forge at
+`https://<swarm.forge.domain>`, so that name has to resolve from the host: the
+gateway's dnsmasq answers it locally where the forge runs, and forwards it to
+the operator's DNS everywhere else.
 
-Optional tuning: `services.hyperhive.deploy.forgejo.ci.name` (runner name in forge
-admin panel), `concurrency` (parallel job capacity), `labels` (workflow
-targeting), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration
-string for example `"3h"` — a job that exceeds it's killed so a hung or runaway
-build can't hold the runner's single slot indefinitely).
+Optional tuning under `services.hyperhive.deploy.ci-runner`: `name` (runner
+name in the forge admin panel), `concurrency` (parallel job capacity), `labels`
+(workflow targeting; the default `hive-ci:host` is what this repo's workflows
+name), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration string
+for example `"3h"` — a job that exceeds it's killed so a hung or runaway build
+can't hold the runner's single slot indefinitely).
 
 ## Container design
 
-- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`) and reaches hive-forge through the gateway at `http://<forge.domain>` (resolved to the bridge IP via `networking.extraHosts`). It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
-- **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
+- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
+- **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after.
 - **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`.
-- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. hive-c0re holds it and performs all forge API calls (runner validation + registration-token mint, in `forge/ci_runner.rs`); via hive-priv it writes only the runner registration token to the host env-file `/run/hive-ci/runner-token`, which the container bind-mounts read-only.
+- **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path.
 
-## Autoregistration flow
+## Registration
 
-Registration is **off the container's boot-critical path** — hive-c0re owns
-it and runs it out of band, so a slow forge or core-token never delays the
-container's start. Gotcha: don't gate the container's own start on a forge
-round-trip (a host-side unit that did this could exceed the nspawn start
-timeout and trip a restart loop) — registration must stay something
-hive-c0re drives after the container is already up. hive-c0re holds the
-core admin token only on the host; only the runner registration token
-reaches the container.
+1. **swarm-controller** (`swarm-controller/src/forge/runner_token.rs`) reads
+   the forge's instance-wide token from
+   `GET /api/v1/admin/runners/registration-token` at start and every five
+   minutes, and writes it to the secret store at
+   `swarm/services/ci-runner/registration-token` when it differs from the
+   value it last wrote. Its store grant on that path is write-only.
+2. **`swarm-bao-ci-runner.service`** on the runner's host logs in to the store
+   with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path
+   and nothing else, retrying every 30 seconds until the token is there. It
+   writes `TOKEN=<value>` to the runner's `EnvironmentFile` inside the
+   container, and restarts the runner when the value changes. It runs at boot
+   and on deploy, not on a timer.
+3. **The runner unit** (`gitea-runner-swarm` in the container) registers in
+   the upstream pre-start step whenever `.runner` is missing or the token or
+   labels changed. A pre-start step of this module's, ahead of it, removes
+   `.runner` in two cases upstream doesn't check:
+   - The forge has no record of the runner. Forgejo answers every call from an
+     unknown runner with `unregistered runner`, and the daemon exits 1 when its
+     startup `Declare` gets that answer. On the next start the step finds the
+     message in the daemon's journal since its last start, and registers again.
+     A daemon that's already running when an admin deletes its runner keeps polling
+     and logging the error without exiting; the next restart of the unit or
+     container registers it again.
+   - `.runner` names another forge address than the configured one.
 
-### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep)
+   Until a token arrives the step fails, and `Restart=on-failure` retries every
+   15 seconds with no start limit.
 
-Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.service` when `deploy.forgejo.ci.enable`). Best-effort — hive-c0re logs failures and never aborts the sweep; a healthy runner is never restarted.
-
-1. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`, validate its id against `GET /api/v1/admin/runners/{id}` with the core admin token:
-   - **200**: still registered — done, no restart.
-   - **404 / other non-200 / malformed**: stale — re-register (below).
-   - **transport error** (forge unreachable): keep the existing creds; a network blip must not wipe a valid runner.
-2. If absent or stale: mint a fresh token from `GET /api/v1/admin/runners/registration-token`, then hand it to hive-priv's `RegisterCiRunner`, which (as root) writes `TOKEN=<real>` **in place** to the host env-file `/run/hive-ci/runner-token` (preserving the inode nspawn pinned into the container at start) and restarts `gitea-runner-hive.service` inside the container so it picks up the credential and registers.
-
-### Container side
-
-- The container boots immediately — nothing gates its start on registration.
-- tmpfiles seeds `/run/hive-ci/runner-token` with `TOKEN=placeholder` so the runner's `EnvironmentFile` always exists.
-- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it's already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
-- **Convergence**: because the token write targets the _host_ file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
+A new registration is a new runner on the forge: the previous entry stays in
+the forge admin panel as offline until an admin deletes it.
 
 ## Actions checkout mirror
 
 <!-- vale write-good.Passive = NO -->
 
-When `deploy.forgejo.ci.enable` is set, the swarm controller autoseeds an
-`actions/checkout` pull-mirror on the local forge and sets Forgejo's
-`DEFAULT_ACTIONS_URL` to point at the local instance. This means CI
-`uses: actions/checkout@vN` steps resolve entirely on loopback — no
-external DNS on the CI critical path.
+With `services.hyperhive.deploy.forgejo.actionsFromSelf` set on the forge's
+host, the swarm controller seeds an `actions/checkout` pull-mirror on the forge
+and Forgejo's `DEFAULT_ACTIONS_URL` points at the forge itself. CI
+`uses: actions/checkout@vN` steps then resolve on the forge, with no external
+DNS on the CI critical path.
 
 <!-- vale write-good.Passive = YES -->
 
@@ -192,7 +205,7 @@ to avoid provisioning collisions.
 
 ## Security: unsandboxed builds and trusted contributors
 
-**hive-ci should only run CI for trusted contributors.** The security boundary is weaker than it looks:
+**The runner should only run CI for trusted contributors.** The security boundary is weaker than it looks:
 
 ### What unsandboxed builds mean
 
@@ -200,10 +213,10 @@ nspawn containers can't create user-namespaces, so the module sets `nix.settings
 
 A malicious `default.nix` or build script in a PR can therefore:
 
-- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge only through the gateway (`http://<forge.domain>`, public/read endpoints — no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
+- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
 - **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
 
-The core admin token (`forge-core-token`) **isn't** bind-mounted into the container. it's held and used only by hive-c0re on the host (`forge/ci_runner.rs`), which mints per-runner registration tokens; only that registration token reaches the container's env-file. A build process can still reach forge over the network, but can't use the admin token to issue privileged API calls.
+No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which reaches the container's env-file. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls.
 
 Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.
 
@@ -221,7 +234,7 @@ The current design is appropriate for a trusted-team hive where all contributors
 ## Host store maintenance (recommended)
 
 The CI runner builds derivations through the **host** nix-daemon — the
-hive-ci container shares the host store and has no daemon of its own. Build
+`swarm-ci-runner` container shares the host store and has no daemon of its own. Build
 outputs accumulate in `/nix/store` with no automatic collection, and a busy
 CI day can fill the disk until every job fails fast with `ENOSPC`.
 
@@ -278,6 +291,7 @@ builder — disk-pressure collection at least runs while nothing is mid-copy.
 
 ## References
 
-- `nix/host-modules/hive-ci.nix`: runner configuration, autoregistration script, container setup.
+- `nix/host-modules/swarm-ci-runner.nix`: runner configuration, token fetch, re-registration step, container setup.
+- `swarm-controller/src/forge/runner_token.rs`: publishes the registration token.
 - `.forgejo/workflows/ci.yml`: workflow definition.
 - `docs/process/gotchas.md`: nix sandboxing limitations in containers.

docs/scheduler/coordinator.md

@@ -474,8 +474,8 @@ Two things to know about the weights:
   same value and the weight does _not_ rank agents against each other.
   What `80` buys is that agents yield to everything **not** on this
   drop-in path: host services (nginx and dnsmasq among them) and the
-  infra containers (`hive-ci`, `hive-forge`, `hive-gateway`,
-  `hive-matrix`), which stay at the kernel default of `100`.
+  infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which
+  stay at the kernel default of `100`.
 - `IOWeight=` is only honoured when the backing device runs the BFQ
   scheduler or has blk-iocost QoS enabled. On a host using
   `none`/`mq-deadline`/`kyber` without iocost, systemd writes the value