mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-ci-runner: run the forge CI runner as a swarm service

Browse source 
The CI runner was a hive deploy module: hive-ci.nix ran it beside the
forge, and hive-c0re registered it through the forge's admin API with
the `core` token, writing the registration token through hive-priv. It
could only run on the forge's host, and it kept an admin-scoped forge
token on a hive.

Now the runner is a swarm service:

- swarm-ci-runner.nix runs it in a `swarm-ci-runner` container on the
  host that sets deploy.ci-runner.enable, reaching the forge by name.
  It needs neither the forge nor a hive there.
- swarm-controller reads the forge's runner registration token with its
  own admin token and writes it to swarm/services/ci-runner/
  registration-token whenever it differs from what it last wrote. Its
  grant on that path is create/update only.
- The runner's own leaf (CN swarm-ci-runner, role and policy of the same
  name) reads that one path. swarm-bao-ci-runner.service fetches it into
  the runner's EnvironmentFile and restarts the runner on a change.
- A pre-start step removes .runner when the daemon's journal since its
  last start holds Forgejo's "unregistered runner" refusal (the daemon
  exits 1 when its startup Declare gets it), or when .runner names
  another forge address; upstream's pre-start step then registers again.

Removed from the hive side: hive-ci.nix and deploy.forgejo.ci.*,
forge/ci_runner.rs, PrivRequest::RegisterCiRunner and its handler and
ReadWritePaths entry, InfraContainer::Ci with hivectl's --ci and the
dashboard's hive-ci journal entry, HYPERHIVE_FORGE_CI_ENABLED, and
read:admin,write:admin from CORE_TOKEN_SCOPES: nothing else in hive-c0re
calls /api/v1/admin/*. The old ci options, under both swarm.forge.ci and
deploy.forgejo.ci, now fail evaluation naming deploy.ci-runner.

The forge's half of the old ci.enable, the actions/checkout mirror and
DEFAULT_ACTIONS_URL = self, is deploy.forgejo.actionsFromSelf.

Closes #4954
This commit is contained in:
atlas 2026-10-05 02:25:05 +02:00 • committed by mara
commit e2da5509ae
60 changed files with 1088 additions and 1201 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.forgejo/workflows/coverage.yml
View file

@ -4,7 +4,7 @@ name: coverage
# than printed into the run log, so a scheduled run leaves something to fetch. # than printed into the run log, so a scheduled run leaves something to fetch.
# #
# ⚠️ The hour is deliberate: at the default job capacity of 1 # ⚠️ The hour is deliberate: at the default job capacity of 1
# (`services.hyperhive.deploy.forgejo.ci.concurrency`) this job holds the # (`services.hyperhive.deploy.ci-runner.concurrency`) this job holds the
# runner for its whole timeout and everything else queues behind it. # runner for its whole timeout and everything else queues behind it.
# #
# Separate from ci.yml because a `workflow_dispatch` trigger there fires every # Separate from ci.yml because a `workflow_dispatch` trigger there fires every

2
.forgejo/workflows/flake-update.yml
View file

@ -11,7 +11,7 @@ name: flake update
# apply. Stale PRs just stack; newest lock wins. # apply. Stale PRs just stack; newest lock wins.
# #
# `PR_BOT_TOKEN` (repo secret) is the `flake-bot` account's write token. # `PR_BOT_TOKEN` (repo secret) is the `flake-bot` account's write token.
# Self-signed forge cert: not worked around with `curl -k` - hive-ci.nix # Self-signed forge cert: not worked around with `curl -k` - swarm-ci-runner.nix
# already sets SSL_CERT_FILE to a bundle with the hive CA (wiring # already sets SSL_CERT_FILE to a bundle with the hive CA (wiring
# confirmed; anchor coverage for the swarm forge specifically isn't). # confirmed; anchor coverage for the swarm forge specifically isn't).
on: on:

2
README.md
View file

@ -55,7 +55,7 @@ the swarm — one of each, each on whichever host you choose
├── swarm-grafana dashboards over both stores ├── swarm-grafana dashboards over both stores
│ │
└── opt-in, not part of singleHostSwarm └── opt-in, not part of singleHostSwarm
├── hive-ci Forgejo Actions runner (deploy.forgejo.ci) ├── swarm-ci-runner Forgejo Actions runner (deploy.ci-runner)
├── swarm-snapshot-store btrfs-receive endpoint for agent snapshots ├── swarm-snapshot-store btrfs-receive endpoint for agent snapshots
└── wg-hive WireGuard mesh between hives (deploy.wireguard) └── wg-hive WireGuard mesh between hives (deploy.wireguard)

2
docs/README.md
View file

@ -91,7 +91,7 @@ declarations.
[`scheduler/jobq.md`](scheduler/jobq.md) — operator-facing, no implementation detail. [`scheduler/jobq.md`](scheduler/jobq.md) — operator-facing, no implementation detail.
- **How does the rebuild queue work? What are the concrete step kinds, - **How does the rebuild queue work? What are the concrete step kinds,
queue sources, scheduler internals?** → [`scheduler/coordinator.md`](scheduler/coordinator.md). queue sources, scheduler internals?** → [`scheduler/coordinator.md`](scheduler/coordinator.md).
- **How does the CI runner work? What's the autoregistration flow?** → - **How does the CI runner work? How does it register?** →
[`scheduler/ci.md`](scheduler/ci.md). [`scheduler/ci.md`](scheduler/ci.md).
- **How do I export Claude Code metrics (tokens, cost, tool calls) to - **How do I export Claude Code metrics (tokens, cost, tool calls) to
Prometheus/Grafana?** → [`scheduler/observability.md`](scheduler/observability.md). Prometheus/Grafana?** → [`scheduler/observability.md`](scheduler/observability.md).

14
docs/integrations/forge.md
View file

@ -31,12 +31,14 @@ by a test). swarm-controller mints every agent token with it:
| `read:notification` | Poll `GET /notifications` for unread events. | | `read:notification` | Poll `GET /notifications` for unread events. |
| `write:notification` | Mark notifications read via `PATCH /notifications/threads/{id}`. | | `write:notification` | Mark notifications read via `PATCH /notifications/threads/{id}`. |
**`CORE_TOKEN_SCOPES`** (`hive-c0re::forge`, for its own `core` user): **`CORE_TOKEN_SCOPES`** (`hive-c0re::forge`, for its own `core` user): the
everything in `AGENT_TOKEN_SCOPES` plus `read:admin` and `write:admin`. same set as `AGENT_TOKEN_SCOPES`, with no admin scope. Forgejo's token scope
Site-admin membership alone isn't sufficient — Forgejo's token scope gate runs gate runs before the user-permission check, so this token gets
before the user-permission check, so `/api/v1/admin/*` returns `403 Forbidden` on `/api/v1/admin/*` even though `core` is a site admin.
`403 Forbidden` for any token without the admin scope bits, even when hive-c0re only mints the token when the file at
the bearer is a site admin. `/var/lib/hyperhive/forge-core-token` is missing or the forge rejects it, so
a token minted with admin scopes keeps them until an operator deletes that
file and restarts hive-c0re.
--- ---

20
docs/networking/network.md
View file

@ -34,21 +34,21 @@ that touches it.
│ hive-br0 10.42.0.1/24 │ │ hive-br0 10.42.0.1/24 │
│ ┌──────────┼──────────────┐ │ │ ┌──────────┼──────────────┐ │
└──────────────┼──────────┼──────────────┼───────────────────────────┘ └──────────────┼──────────┼──────────────┼───────────────────────────┘
vb-h-<a> vb-h-<b> vb-hive-ci veth pairs vb-h-<a> vb-h-<b> vb-<runner> veth pairs
│ │ │ │ │ │
┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns ┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns
│ agent a │ │ agent b │ │ hive-ci │ each; eth0 leases │ agent a │ │ agent b │ │ runner │ each; eth0 leases
│ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool │ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool
└─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘
``` ```
| container | netns | IPv4 | listens / reached via | | container | netns | IPv4 | listens / reached via |
| -------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- | | ----------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
| gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge | | gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost | | `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost | | `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
| `hive-ci` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface | | `swarm-ci-runner` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 | | `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
The flows, end to end: The flows, end to end:
@ -172,7 +172,7 @@ boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
from the container's (empty) port list, and deletes the host's from the container's (empty) port list, and deletes the host's
`nixos-nat-*` chains without recreating them, silently wiping the `nixos-nat-*` chains without recreating them, silently wiping the
bridge holes above plus the agents' NAT. Private-netns containers bridge holes above plus the agents' NAT. Private-netns containers
(agents, hive-ci) may keep their own firewall — it's scoped to their (agents, swarm-ci-runner) may keep their own firewall — it's scoped to their
namespace. namespace.
### Reaching host services (`exposeHostPorts`) ### Reaching host services (`exposeHostPorts`)

126
docs/scheduler/ci.md
View file

@ -1,13 +1,21 @@
# hive-ci: Forgejo Actions Runner # swarm-ci-runner: Forgejo Actions Runner
The `hive-ci` module runs a Forgejo Actions runner in a `hive-ci` nixos-container, The `swarm-ci-runner` module runs the swarm's Forgejo Actions runner in a
executing CI jobs from `.forgejo/workflows/ci.yml` on every PR. `swarm-ci-runner` nixos-container, executing CI jobs from
`.forgejo/workflows/ci.yml` on every PR. It's a swarm service: it runs on
whichever host sets it, and needs neither the forge nor a hive on that host.
## For operators ## For operators
**Enabling it's one line**: `services.hyperhive.deploy.forgejo.ci.enable = true` **Enabling it's one line**: `services.hyperhive.deploy.ci-runner.enable = true`
in the host NixOS config. No manual token provisioning — hive-c0re in the NixOS config of the host that should run it. No manual token
registers the runner with the forge automatically. provisioning: swarm-controller publishes the forge's runner registration token
to the secret store, and the runner reads it with a store identity of its own.
On the store's host `glue-bao-tls.nix` mints that identity; on any other host, copy
`ci-runner.pem` and `ci-runner-key.pem` from the store host's
`/var/lib/swarm-bao-pki/` and name them in
`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
`ciRunnerClientKeyFile`.
Two things worth knowing before you rely on it: Two things worth knowing before you rely on it:
@ -25,9 +33,8 @@ Two things worth knowing before you rely on it:
disk-pressure GC config from _Host store maintenance_ below to your disk-pressure GC config from _Host store maintenance_ below to your
host's NixOS config (not optional if you plan to actually use this). host's NixOS config (not optional if you plan to actually use this).
Everything below this point is implementation detail: exact Everything below this point is implementation detail: registration
autoregistration internals, container design, and the full security internals, container design, and the full security writeup.
writeup.
## CI checks ## CI checks
@ -105,62 +112,68 @@ slow); run those manually before pushing Rust changes.
## Configuration reference ## Configuration reference
The runner registers against the forge on its own host, so enable it on the `services.hyperhive.deploy.ci-runner.enable = true` on the host that runs it
swarm's forge host (`services.hyperhive.deploy.forgejo.enable`, see (see _For operators_ above). The runner reaches the forge at
[`../swarm/services.md`](../swarm/services.md)). Evaluation fails on any other `https://<swarm.forge.domain>`, so that name has to resolve from the host: the
host. There, nothing extra to enable beyond gateway's dnsmasq answers it locally where the forge runs, and forwards it to
`services.hyperhive.deploy.forgejo.ci.enable = true` (see _For operators_ above). the operator's DNS everywhere else.
Optional tuning: `services.hyperhive.deploy.forgejo.ci.name` (runner name in forge Optional tuning under `services.hyperhive.deploy.ci-runner`: `name` (runner
admin panel), `concurrency` (parallel job capacity), `labels` (workflow name in the forge admin panel), `concurrency` (parallel job capacity), `labels`
targeting), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration (workflow targeting; the default `hive-ci:host` is what this repo's workflows
string for example `"3h"` — a job that exceeds it's killed so a hung or runaway name), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration string
build can't hold the runner's single slot indefinitely). for example `"3h"` — a job that exceeds it's killed so a hung or runaway build
can't hold the runner's single slot indefinitely).
## Container design ## Container design
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`) and reaches hive-forge through the gateway at `http://<forge.domain>` (resolved to the bridge IP via `networking.extraHosts`). It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. - **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
- **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter). - **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after.
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`. - **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`.
- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. hive-c0re holds it and performs all forge API calls (runner validation + registration-token mint, in `forge/ci_runner.rs`); via hive-priv it writes only the runner registration token to the host env-file `/run/hive-ci/runner-token`, which the container bind-mounts read-only. - **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path.
## Autoregistration flow ## Registration
Registration is **off the container's boot-critical path** — hive-c0re owns 1. **swarm-controller** (`swarm-controller/src/forge/runner_token.rs`) reads
it and runs it out of band, so a slow forge or core-token never delays the the forge's instance-wide token from
container's start. Gotcha: don't gate the container's own start on a forge `GET /api/v1/admin/runners/registration-token` at start and every five
round-trip (a host-side unit that did this could exceed the nspawn start minutes, and writes it to the secret store at
timeout and trip a restart loop) — registration must stay something `swarm/services/ci-runner/registration-token` when it differs from the
hive-c0re drives after the container is already up. hive-c0re holds the value it last wrote. Its store grant on that path is write-only.
core admin token only on the host; only the runner registration token 2. **`swarm-bao-ci-runner.service`** on the runner's host logs in to the store
reaches the container. with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path
and nothing else, retrying every 30 seconds until the token is there. It
writes `TOKEN=<value>` to the runner's `EnvironmentFile` inside the
container, and restarts the runner when the value changes. It runs at boot
and on deploy, not on a timer.
3. **The runner unit** (`gitea-runner-swarm` in the container) registers in
the upstream pre-start step whenever `.runner` is missing or the token or
labels changed. A pre-start step of this module's, ahead of it, removes
`.runner` in two cases upstream doesn't check:
- The forge has no record of the runner. Forgejo answers every call from an
unknown runner with `unregistered runner`, and the daemon exits 1 when its
startup `Declare` gets that answer. On the next start the step finds the
message in the daemon's journal since its last start, and registers again.
A daemon that's already running when an admin deletes its runner keeps polling
and logging the error without exiting; the next restart of the unit or
container registers it again.
- `.runner` names another forge address than the configured one.
### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep) Until a token arrives the step fails, and `Restart=on-failure` retries every
15 seconds with no start limit.
Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.service` when `deploy.forgejo.ci.enable`). Best-effort — hive-c0re logs failures and never aborts the sweep; a healthy runner is never restarted. A new registration is a new runner on the forge: the previous entry stays in
the forge admin panel as offline until an admin deletes it.
1. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`, validate its id against `GET /api/v1/admin/runners/{id}` with the core admin token:
- **200**: still registered — done, no restart.
- **404 / other non-200 / malformed**: stale — re-register (below).
- **transport error** (forge unreachable): keep the existing creds; a network blip must not wipe a valid runner.
2. If absent or stale: mint a fresh token from `GET /api/v1/admin/runners/registration-token`, then hand it to hive-priv's `RegisterCiRunner`, which (as root) writes `TOKEN=<real>` **in place** to the host env-file `/run/hive-ci/runner-token` (preserving the inode nspawn pinned into the container at start) and restarts `gitea-runner-hive.service` inside the container so it picks up the credential and registers.
### Container side
- The container boots immediately — nothing gates its start on registration.
- tmpfiles seeds `/run/hive-ci/runner-token` with `TOKEN=placeholder` so the runner's `EnvironmentFile` always exists.
- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it's already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
- **Convergence**: because the token write targets the _host_ file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
## Actions checkout mirror ## Actions checkout mirror
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
When `deploy.forgejo.ci.enable` is set, the swarm controller autoseeds an With `services.hyperhive.deploy.forgejo.actionsFromSelf` set on the forge's
`actions/checkout` pull-mirror on the local forge and sets Forgejo's host, the swarm controller seeds an `actions/checkout` pull-mirror on the forge
`DEFAULT_ACTIONS_URL` to point at the local instance. This means CI and Forgejo's `DEFAULT_ACTIONS_URL` points at the forge itself. CI
`uses: actions/checkout@vN` steps resolve entirely on loopback — no `uses: actions/checkout@vN` steps then resolve on the forge, with no external
external DNS on the CI critical path. DNS on the CI critical path.
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->
@ -192,7 +205,7 @@ to avoid provisioning collisions.
## Security: unsandboxed builds and trusted contributors ## Security: unsandboxed builds and trusted contributors
**hive-ci should only run CI for trusted contributors.** The security boundary is weaker than it looks: **The runner should only run CI for trusted contributors.** The security boundary is weaker than it looks:
### What unsandboxed builds mean ### What unsandboxed builds mean
@ -200,10 +213,10 @@ nspawn containers can't create user-namespaces, so the module sets `nix.settings
A malicious `default.nix` or build script in a PR can therefore: A malicious `default.nix` or build script in a PR can therefore:
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge only through the gateway (`http://<forge.domain>`, public/read endpoints — no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic). - **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials. - **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
The core admin token (`forge-core-token`) **isn't** bind-mounted into the container. it's held and used only by hive-c0re on the host (`forge/ci_runner.rs`), which mints per-runner registration tokens; only that registration token reaches the container's env-file. A build process can still reach forge over the network, but can't use the admin token to issue privileged API calls. No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which reaches the container's env-file. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls.
Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path. Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.
@ -221,7 +234,7 @@ The current design is appropriate for a trusted-team hive where all contributors
## Host store maintenance (recommended) ## Host store maintenance (recommended)
The CI runner builds derivations through the **host** nix-daemon — the The CI runner builds derivations through the **host** nix-daemon — the
hive-ci container shares the host store and has no daemon of its own. Build `swarm-ci-runner` container shares the host store and has no daemon of its own. Build
outputs accumulate in `/nix/store` with no automatic collection, and a busy outputs accumulate in `/nix/store` with no automatic collection, and a busy
CI day can fill the disk until every job fails fast with `ENOSPC`. CI day can fill the disk until every job fails fast with `ENOSPC`.
@ -278,6 +291,7 @@ builder — disk-pressure collection at least runs while nothing is mid-copy.
## References ## References
- `nix/host-modules/hive-ci.nix`: runner configuration, autoregistration script, container setup. - `nix/host-modules/swarm-ci-runner.nix`: runner configuration, token fetch, re-registration step, container setup.
- `swarm-controller/src/forge/runner_token.rs`: publishes the registration token.
- `.forgejo/workflows/ci.yml`: workflow definition. - `.forgejo/workflows/ci.yml`: workflow definition.
- `docs/process/gotchas.md`: nix sandboxing limitations in containers. - `docs/process/gotchas.md`: nix sandboxing limitations in containers.

4
docs/scheduler/coordinator.md
View file

@ -474,8 +474,8 @@ Two things to know about the weights:
same value and the weight does _not_ rank agents against each other. same value and the weight does _not_ rank agents against each other.
What `80` buys is that agents yield to everything **not** on this What `80` buys is that agents yield to everything **not** on this
drop-in path: host services (nginx and dnsmasq among them) and the drop-in path: host services (nginx and dnsmasq among them) and the
infra containers (`hive-ci`, `hive-forge`, `hive-gateway`, infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which
`hive-matrix`), which stay at the kernel default of `100`. stay at the kernel default of `100`.
- `IOWeight=` is only honoured when the backing device runs the BFQ - `IOWeight=` is only honoured when the backing device runs the BFQ
scheduler or has blk-iocost QoS enabled. On a host using scheduler or has blk-iocost QoS enabled. On a host using
`none`/`mq-deadline`/`kyber` without iocost, systemd writes the value `none`/`mq-deadline`/`kyber` without iocost, systemd writes the value

2
docs/swarm/README.md
View file

@ -265,7 +265,7 @@ one per hive. It ensures them at start and every five minutes after
whitelists = the `operators` team, and no user. The controller leaves a whitelists = the `operators` team, and no user. The controller leaves a
repo with no `main` rule alone; repo with no `main` rule alone;
- the pull-mirrors from `deploy.forgejo.mirrors` on the controller's - the pull-mirrors from `deploy.forgejo.mirrors` on the controller's
host (with the `actions/checkout` one `deploy.forgejo.ci.enable` adds); host (with the `actions/checkout` one `deploy.forgejo.actionsFromSelf` adds);
- `internal/docs` (private) and `internal/knowledge` (public, with a - `internal/docs` (private) and `internal/knowledge` (public, with a
seed `README.md` while empty); seed `README.md` while empty);
- the `agent-configs` org avatar - the `agent-configs` org avatar

2
docs/swarm/ca.md
View file

@ -213,7 +213,7 @@ Two consumers, and only one of them is fine:
next to every agent's flake. The bundle is the runtime-to-build-time next to every agent's flake. The bundle is the runtime-to-build-time
bridge. bridge.
- **The Matrix container crosses the same bridge**, via the shared - **The Matrix container crosses the same bridge**, via the shared
`lib/hive-ca-trust.nix` helper that `hive-ci` and `hive-forge` already `lib/hive-ca-trust.nix` helper that `hive-forge` and `swarm-ci-runner`
use: the bundle is bind-mounted read-only into the container, and a use: the bundle is bind-mounted read-only into the container, and a
oneshot concatenates it with the system CAs before tuwunel starts. oneshot concatenates it with the system CAs before tuwunel starts.

1
docs/swarm/credentials.md
View file

@ -69,6 +69,7 @@ of the cell says how.
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into the runner's `EnvironmentFile`; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ❌ `swarm-bao-ci-runner` reads it when it starts, at boot or deploy, and restarts the runner when the value changed |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | | _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

17
docs/swarm/secrets.md
View file

@ -302,7 +302,7 @@ next controller restart.
### Per-principal identities ### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. These four units read one path identity and sharing one merges the identities. These five units read one path
each and each holds a leaf, a role and a policy of its own: each and each holds a leaf, a role and a policy of its own:
| unit | option pair under `deploy.bao.` | reads | | unit | option pair under `deploy.bao.` | reads |
@ -311,20 +311,23 @@ each and each holds a leaf, a role and a policy of its own:
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` | | `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` | | `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` | | `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
| `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` |
The first exists **per hive**, because the path it reads carries a hive name The first exists **per hive**, because the path it reads carries a hive name
and every hive runs its own reader. Its subject is and every hive runs its own reader. Its subject is
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that `<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
composed spelling as a hive name, so nobody can name a hive into another hive's composed spelling as a hive name, so nobody can name a hive into another hive's
role. The other three read a path that names a swarm service rather than a hive, role. The other four read a path that names a swarm service rather than a hive,
so one role each is enough and their subjects are the flat so one role each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName` and `deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`. `deploy.bao.forwarderOidcCommonName` and `deploy.bao.ciRunnerCommonName`.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults On a host that mints its own PKI, `glue-bao-tls.nix` signs all five and defaults
all eight options, and there is nothing to do. Elsewhere you issue each leaf from all ten options, and there is nothing to do. Elsewhere you issue each leaf from
that CA out of band and name it here — one file per principal rather than one file that CA out of band and name it here — one file per principal rather than one file
shared by four, which is the whole of what this buys. shared by five, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as
`swarm-grafana.nix` does for Grafana.
On a host without the store, where the forwarder never runs, forgetting one of On a host without the store, where the forwarder never runs, forgetting one of
the other six isn't a quiet degrade. A hand-configured remote-store hive that the other six isn't a quiet degrade. A hand-configured remote-store hive that

13
docs/swarm/services.md
View file

@ -221,3 +221,16 @@ has one either way, reached by its swarm name through the gateway.
Agent-side configuration, and what a hive's own collector does, are in Agent-side configuration, and what a hive's own collector does, are in
[`../scheduler/observability.md`](../scheduler/observability.md). [`../scheduler/observability.md`](../scheduler/observability.md).
### CI runner
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
runner executes pull-request code, so running one is a decision of its own.
It needs neither the forge nor a hive on its host. It reaches the forge at
`https://<swarm.forge.domain>`, and registers with the token swarm-controller
publishes to the secret store, which it reads under its own `ci-runner` leaf.
On a host without the store, that leaf is the one file to copy. Registration,
the container and the threat model are in
[`../scheduler/ci.md`](../scheduler/ci.md).

3
docs/tools/hivectl-cli.md
View file

@ -600,7 +600,6 @@ Bare `hivectl stop` stops everything; scope flags narrow it to specific sub-agen
* `--agents` — All sub-agent containers * `--agents` — All sub-agent containers
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b` * `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
* `--ci` — The CI runner container (`hive-ci`)
* `--forge` — The forge container (`hive-forge`) * `--forge` — The forge container (`hive-forge`)
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container * `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
* `--matrix` — The matrix container (`hive-matrix`) * `--matrix` — The matrix container (`hive-matrix`)
@ -621,7 +620,6 @@ Bare `hivectl start` restores the agents stopped by the last broad-scope `stop`
* `--agents` — All sub-agent containers * `--agents` — All sub-agent containers
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b` * `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
* `--ci` — The CI runner container (`hive-ci`)
* `--forge` — The forge container (`hive-forge`) * `--forge` — The forge container (`hive-forge`)
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container * `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
* `--matrix` — The matrix container (`hive-matrix`) * `--matrix` — The matrix container (`hive-matrix`)
@ -641,7 +639,6 @@ Bare `hivectl restart` restarts everything; scope flags narrow it.
* `--agents` — All sub-agent containers * `--agents` — All sub-agent containers
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b` * `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
* `--ci` — The CI runner container (`hive-ci`)
* `--forge` — The forge container (`hive-forge`) * `--forge` — The forge container (`hive-forge`)
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container * `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
* `--matrix` — The matrix container (`hive-matrix`) * `--matrix` — The matrix container (`hive-matrix`)

8
docs/trust-boundary/boundary.md
View file

@ -56,13 +56,13 @@ Concretely, the core daemon's dashboard `/api` carries **no
application-layer authentication** — the core daemon serves operator-authority application-layer authentication** — the core daemon serves operator-authority
routes unauthenticated at the HTTP layer. Their protection is entirely (a) the routes unauthenticated at the HTTP layer. Their protection is entirely (a) the
gateway, which fronts all operator traffic and is where operator auth lives, gateway, which fronts all operator traffic and is where operator auth lives,
and (b) network isolation, which keeps agents — and `hive-ci`'s untrusted PR and (b) network isolation, which keeps agents — and the CI runner's untrusted
builds — off host-loopback so nothing can reach `127.0.0.1:<dashboard_port>` PR builds — off host-loopback so nothing can reach `127.0.0.1:<dashboard_port>`
directly. This is deliberate given the load-bearing role of network isolation directly. This is deliberate given the load-bearing role of network isolation
above, but it's a standing invariant: the `/api` must never bind to a above, but it's a standing invariant: the `/api` must never bind to a
non-loopback address or get exposed outside the gateway, and every new non-loopback address or get exposed outside the gateway, and every new
operator-authority route inherits that assumption. hyperhive treats `hive-ci` operator-authority route inherits that assumption. hyperhive treats
like an agent for this purpose — it runs untrusted PR code and is `swarm-ci-runner` like an agent for this purpose — it runs untrusted PR code and is
netns-isolated for the same reason. netns-isolated for the same reason.
The boundary rests on three layers: The boundary rests on three layers:

9
docs/trust-boundary/security.md
View file

@ -205,9 +205,9 @@ Unsandboxed nix builds run as `nixbld` users (non-root, typically UIDs 30001-300
Without sandbox isolation, a build derivation's builder script has read access to Without sandbox isolation, a build derivation's builder script has read access to
any file in the container that the nixbld user can read. any file in the container that the nixbld user can read.
The blast radius also has a **network** dimension. hive-ci runs its unsandboxed The blast radius also has a **network** dimension. `swarm-ci-runner` runs its
builds of untrusted PR code in its own private netns behind the hive bridge: a unsandboxed builds of untrusted PR code in its own private netns behind the
build reaches the forge only through the gateway and can't reach host-loopback bridge: a build reaches the forge by name and can't reach host-loopback
services — including the core dashboard at `127.0.0.1:<dashboard_port>`, which services — including the core dashboard at `127.0.0.1:<dashboard_port>`, which
has no application-layer auth of its own (see [`docs/scheduler/ci.md`](../scheduler/ci.md)). The `0600` has no application-layer auth of its own (see [`docs/scheduler/ci.md`](../scheduler/ci.md)). The `0600`
token policy bounds file reads; network isolation bounds network reach. token policy bounds file reads; network isolation bounds network reach.
@ -286,7 +286,6 @@ known operations; there is no arbitrary command pass-through:
| `ControlInfraContainer` | `systemctl <action> container@<container>.service` — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re` has no variant, so no request can name it) | | `ControlInfraContainer` | `systemctl <action> container@<container>.service` — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re` has no variant, so no request can name it) |
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` | | `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop | | `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
| `RegisterCiRunner` | write `/run/hive-ci/runner-token` (host path, root-owned) then `systemctl --machine=hive-ci restart gitea-runner-hive.service`. Only the registration token crosses; the forge admin token never enters the container |
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs | | `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first | | `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy | | `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
@ -301,7 +300,7 @@ known operations; there is no arbitrary command pass-through:
**Container allowlist** — `hive-priv` validates every request against **Container allowlist** — `hive-priv` validates every request against
an allowlist before any operation: the allowlist accepts only names an allowlist before any operation: the allowlist accepts only names
matching the agent-name convention (char-validated) or the known matching the agent-name convention (char-validated) or the known
sibling service containers (`hive-forge`, `hive-matrix`, `hive-ci`), sibling service containers (`hive-forge`, `hive-matrix`),
and rejects arbitrary container names. `hive-gateway` is a host unit, and rejects arbitrary container names. `hive-gateway` is a host unit,
not a container, so it's not in this list — see `ReloadGatewayNginx` not a container, so it's not in this list — see `ReloadGatewayNginx`
above for how the hard-coded unit name scopes its access instead. above for how the hard-coded unit name scopes its access instead.

6
docs/web-ui/dashboard.md
View file

@ -515,15 +515,15 @@ deep-link directly to a specific agent's journal. A "fetched N ago"
chip appears after the `↻ refresh` button following each successful chip appears after the `↻ refresh` button following each successful
fetch and ticks every 30 s. fetch and ticks every 30 s.
**INFRA sub-tab** — journald viewer for the four hive infrastructure **INFRA sub-tab** — journald viewer for the three hive infrastructure
containers (`hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`), a containers (`hive-forge`, `hive-gateway`, `hive-matrix`), a
fixed client-side list (`INFRA_NAMES` in `logs.js` — no dashboard API fixed client-side list (`INFRA_NAMES` in `logs.js` — no dashboard API
exposes just the name list). No unit filter (infra containers don't run exposes just the name list). No unit filter (infra containers don't run
the per-agent hive daemons) — always the full machine journal (or, for the per-agent hive daemons) — always the full machine journal (or, for
the gateway, the host journal filtered to its own unit). Fetches the gateway, the host journal filtered to its own unit). Fetches
`GET /api/journal/{name}?lines=500`, same "fetched N ago" ticker as `GET /api/journal/{name}?lines=500`, same "fetched N ago" ticker as
AGENT. A `?agent=<name>` deep-link routes here instead of AGENT when AGENT. A `?agent=<name>` deep-link routes here instead of AGENT when
the name is one of the four infra containers. the name is one of the three infra containers.
**SYSTEM sub-tab** — host-side service logs. Unit selector **SYSTEM sub-tab** — host-side service logs. Unit selector
(`hive-c0re.service` / `hive-priv.service`). Fetches (`hive-c0re.service` / `hive-priv.service`). Fetches

1
flake.nix
View file

@ -167,7 +167,6 @@
lib.mkDefault lib.mkDefault
self.packages.${pkgs.stdenv.hostPlatform.system}.swagger-ui-theme; self.packages.${pkgs.stdenv.hostPlatform.system}.swagger-ui-theme;
}; };
hive-ci = ./nix/host-modules/hive-ci.nix;
hive-forge = ./nix/host-modules/hive-forge; hive-forge = ./nix/host-modules/hive-forge;
}; };

2
frontend/packages/dashboard/src/core.html
View file

@ -20,7 +20,7 @@
/builds.html (the build lifecycle hub). Default tab: K3PT ST4T3. /builds.html (the build lifecycle hub). Default tab: K3PT ST4T3.
The section <div> ids (tombstones-section, container-load-section) The section <div> ids (tombstones-section, container-load-section)
match what core.js's renderers target. Hive infra containers match what core.js's renderers target. Hive infra containers
(`hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`) have no (`hive-forge`, `hive-gateway`, `hive-matrix`) have no
dashboard panel — `hivectl stop`/`start`/`restart` is the only dashboard panel — `hivectl stop`/`start`/`restart` is the only
control surface. --> control surface. -->
<header class="page-header"> <header class="page-header">

6
frontend/packages/dashboard/src/logs.html
View file

@ -60,11 +60,11 @@ select an agent above</pre
> >
</section> </section>
<!-- INFRA: journald viewer for infrastructure containers (hive-ci, <!-- INFRA: journald viewer for infrastructure containers (hive-forge,
hive-forge, hive-gateway, hive-matrix). Always fetches the full hive-gateway, hive-matrix). Always fetches the full
machine journal — no unit filter (infra containers don't run the machine journal — no unit filter (infra containers don't run the
per-agent hive daemons). Backed by GET /api/journal/{name}?lines=N. per-agent hive daemons). Backed by GET /api/journal/{name}?lines=N.
Deep-link: ?agent=hive-ci routes here instead of the AGENT tab. --> Deep-link: ?agent=hive-forge routes here instead of the AGENT tab. -->
<section <section
class="logs-pane" class="logs-pane"
id="logs-pane-infra" id="logs-pane-infra"

Some files were not shown because too many files have changed in this diff Show more