vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

treefmt: apply prettier

Browse source 
Pure `nix fmt` output from the commit before this one — no hand edits.
203 files: 52 md, 42 tsx, 32 js, 32 css, 21 ts, 13 html, 8 json, 3 mjs.

Reproduce with `nix develop -c nix fmt` on the parent commit; the result
should be byte-identical to this tree.

None of the 13 `.prettierignore` entries appears here — verified by
intersecting the changed-file list against the ignore file, with a
control proving the intersection finds a match when one exists.
This commit is contained in:
atlas 2026-09-02 14:29:33 +02:00
commit 39b95c2ede
203 changed files with 10090 additions and 6085 deletions
Download patch file Download diff file Expand all files Collapse all files

24
docs/agent-lifecycle/agent-hierarchy.md
View file

@ -28,7 +28,7 @@ requester-as-parent edge written at its `init_config` approval (so
agent, including the bootstrap container (`ruth`) — it's just another
root. The manager is reparentable like any other agent; there's no
"structurally root" carve-out. Its privileges live on its MCP socket,
not its tree position (see *Manager special-casing today* below).
not its tree position (see _Manager special-casing today_ below).
### Reparenting
@ -88,15 +88,15 @@ umount-old / mount-new / restart-cascade step.
## Planned topology semantics (once ancestor-based enforcement lands)
| operation | who can do it |
| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `kill` / `start` / `restart` / `update` (any descendant) | any ancestor |
| `request_init_config` (spawn a new child) | any agent, child added under self |
| config change via forge PR (any descendant's config) | any ancestor |
| `get_logs` (any descendant) | any ancestor |
| moderate reminders (cancel any open thread of a descendant) | any ancestor |
| `send` / `recv` routing | parent ↔ same-parent siblings ↔ self ↔ descendants; explicit allow-list for anyone else |
| `request_update_meta_inputs` (bump meta lock) | root agents only (today: just `manager`) |
| operation | who can do it |
| ----------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `kill` / `start` / `restart` / `update` (any descendant) | any ancestor |
| `request_init_config` (spawn a new child) | any agent, child added under self |
| config change via forge PR (any descendant's config) | any ancestor |
| `get_logs` (any descendant) | any ancestor |
| moderate reminders (cancel any open thread of a descendant) | any ancestor |
| `send` / `recv` routing | parent ↔ same-parent siblings ↔ self ↔ descendants; explicit allow-list for anyone else |
| `request_update_meta_inputs` (bump meta lock) | root agents only (today: just `manager`) |
"Ancestor" walks `ContainerView.parent` chains; cycles are guarded by a
visited-set at dispatch time (a malformed `topology.json` can't lock
@ -138,11 +138,11 @@ other agents don't:
capability.
- **Prompt/tools** — the system prompt uses `<!-- role:agent -->` /
`<!-- role:manager -->` marker blocks, and a `Flavor::{Agent,
Manager}` switch picks the MCP tool allow-list claude sees. Both are
Manager}` switch picks the MCP tool allow-list claude sees. Both are
already parametrised on a single flavour value, so the planned
per-capability-group version (`cap:<group>` prompt blocks + a
matching tool allow-list) is additive rather than a rewrite.
- **State dirs** — *not* special-cased: `HYPERHIVE_STATE_DIR` is
- **State dirs** — _not_ special-cased: `HYPERHIVE_STATE_DIR` is
injected uniformly via `systemd.globalEnvironment` for every
container including the manager, so all token/state paths resolve
through it the same way everywhere.

2
docs/getting-started/setup.md
View file

@ -114,7 +114,7 @@ Nothing to run — it is served on the swarm apex
decide whether you can actually open it:
- **You are in `admins`** (step 3). The gateway asks authelia whether
you have a session; the rule that makes it mean *operator* wants the
you have a session; the rule that makes it mean _operator_ wants the
group. Without it you log in and still get bounced.
- **The name resolves to this host.** It is published to the hive's own
resolver and to `/etc/hosts` when `gateway.localHostsEntry` is on; from

18
docs/integrations/forge.md
View file

@ -112,7 +112,7 @@ On a **successful** broker delivery, `forge_notify` marks the thread
read on forge straight away (`PATCH /notifications/threads/{id}`). The
broker inbox is the durable work queue now — each delivered wake is a
sqlite row with its own ack lifecycle — so the forge unread flag no
longer needs to track whether the agent has *processed* a
longer needs to track whether the agent has _processed_ a
notification. Clearing it on delivery keeps forge's unread set **tiny
by construction**: at rest it holds only threads that failed to
deliver plus whatever arrived since the last 30s poll.
@ -132,19 +132,19 @@ own read-state is the only durable record this design needs.
**Read-before-comment guard doesn't block a fresh wake.** hive-forge's
read-before-comment guard (which keys off forge unread-state) does not
force the agent to view a thread before commenting on it: the broker
wake already carries the notification body, so *delivery is the read*.
wake already carries the notification body, so _delivery is the read_.
An agent that wants the full thread still runs `hive-forge comments` /
`view`.
**In-process dedupe (tiny, ephemeral).** A single-process map (thread
id → last-delivered `updated_at`) guards the narrow window where a
mark-read call *transiently fails* and the thread reappears unread in
mark-read call _transiently fails_ and the thread reappears unread in
the next poll before its `updated_at` bumps — so a flaky PATCH doesn't
re-fire the wake. It is **not persisted** and resets on restart (forge
read-state covers the durable case). Each poll prunes it to the ids in
the single `limit=UNREAD_FETCH_LIMIT` (50) fetch page, so it can never
exceed that many entries (a debug assertion pins the invariant; the
fetch limit and the bound are the same constant). A failed *delivery*
fetch limit and the bound are the same constant). A failed _delivery_
is left unread and out of the map, so it resurfaces next tick.
Self-echo notifications (the agent's own writes, see below) are marked
@ -225,13 +225,13 @@ lookahead.
Five shapes, distinguished by the notification's classification:
| Trigger | Wrapper |
| ----------------------------------- | --------------------------------------------------------------------------------- |
| Trigger | Wrapper |
| ----------------------------------- | ------------------------------------------------------------------------------ |
| Comment on issue / PR | `[comment on PR #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
| Review submission | `[PR approved #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
| New issue / PR | `[new PR #N owner/repo] title\nurl: ...\nassignee: ...` |
| Later activity (open, not creation) | `[activity on PR #N owner/repo] title\nurl: ...\nassignee: ...` |
| State change | `[PR merged #N owner/repo] title\nurl: ...\nassignee: ...` |
| New issue / PR | `[new PR #N owner/repo] title\nurl: ...\nassignee: ...` |
| Later activity (open, not creation) | `[activity on PR #N owner/repo] title\nurl: ...\nassignee: ...` |
| State change | `[PR merged #N owner/repo] title\nurl: ...\nassignee: ...` |
The issue/PR's own description is deliberately never embedded here (only a
comment/review body is — that's the actual new content a notification

8
docs/integrations/github.md
View file

@ -109,7 +109,7 @@ Forgejo's notifications API is modelled on GitHub's, so one tolerant
parse serves both: `id`, `repository.full_name`,
`subject {title,url,latest_comment_url}` and `updated_at` line up field
for field. The two real differences are absorbed by lenient
deserializers — GitHub sends the thread id as a *string* where Forgejo
deserializers — GitHub sends the thread id as a _string_ where Forgejo
sends a number, and says `PullRequest` where Forgejo says `Pull`. Todo
keys are prefixed `gh:<id>` so a github thread id cannot collide with a
forge one.
@ -118,16 +118,16 @@ Two host differences worth knowing before touching this code:
- **Auth scheme, not just value.** Forgejo takes
`Authorization: token <t>`; GitHub wants `Bearer <t>` plus `Accept:
application/vnd.github+json`, `X-GitHub-Api-Version` and a
application/vnd.github+json`, `X-GitHub-Api-Version` and a
`User-Agent`. Sending Forgejo's form to GitHub does not error — it
authenticates as *nobody* and silently drops to the unauthenticated
authenticates as _nobody_ and silently drops to the unauthenticated
rate limit. The cheap way to tell the two apart is the rate-limit
header: `x-ratelimit-remaining` near 5000 is an authenticated user,
near 60 is anonymous.
- **GitHub sets the cadence.** It returns `X-Poll-Interval` (60s in
practice, slower than our own tick) and rate-limits callers who ignore
it, so the loop re-arms to the server's interval whenever that is
*slower* than ours. A hint faster than our own tick is not a reason to
_slower_ than ours. A hint faster than our own tick is not a reason to
poll harder.
⚠️ **This needs the `notifications` scope on the PAT.** A token minted

2
docs/integrations/knowledge.md
View file

@ -42,7 +42,7 @@ hive-c0re maintains the local clone at
`internal/knowledge` (see `docs/swarm/README.md` § Swarm-wide forge
webhooks). On any push to main, including merge commits, it sends an
event to every hive over the swarm queue and each hive runs `git
pull`, so agents see the new content on their next turn.
pull`, so agents see the new content on their next turn.
A hive that is offline when the event is sent does not get it on
reconnect — the periodic pull below is what closes that gap. So one

34
docs/networking/network.md
View file

@ -43,13 +43,13 @@ untouched by any of it.
└─────────┘ └─────────┘ └─────────┘
```
| container | netns | IPv4 | listens / reached via |
| -------------- | ----------------------- | -------------------- | -------------------------------------------------------------------------------------------- |
| `hive-gateway` | host (shared) | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
| `hive-ci` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
| container | netns | IPv4 | listens / reached via |
| -------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
| `hive-gateway` | host (shared) | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
| `hive-ci` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
The flows, end to end:
@ -57,7 +57,7 @@ The flows, end to end:
host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
- **DNS** — agents and the service containers query `bridgeIp:53`; hive
zones are answered authoritatively with the bridge IP, everything else
forwards to the host's resolvers (see *Resolver behaviour* below). Each
forwards to the host's resolvers (see _Resolver behaviour_ below). Each
container points its own `resolv.conf` there, and one that instead
inherits the host's resolves no swarm name at all — those records exist
only on the bridge.
@ -169,7 +169,7 @@ agent containers.
The **host** firewall is the only firewall. The shared-netns infra
containers (gateway, forge, matrix) set
`networking.firewall.enable = false`: a NixOS firewall inside a
shared-netns container runs against the *host* ruleset — at container
shared-netns container runs against the _host_ ruleset — at container
boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
from the container's (empty) port list, and deletes the host's
`nixos-nat-*` chains without recreating them, silently wiping the
@ -183,7 +183,7 @@ By default agents can only reach the host on 80/443 (+53 DNS), so a
host-side service on another port — e.g. a dev OTLP collector you want
agents to reach directly — is unreachable. (hyperhive's own telemetry
needs none of this: `otel.enable` opens its collector's port itself, and
`otel.endpoint` is the *upstream*, which no agent ever dials. See
`otel.endpoint` is the _upstream_, which no agent ever dials. See
`docs/scheduler/observability.md`.)
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
@ -206,12 +206,12 @@ Each agent container runs in a private network namespace with a dedicated
veth pair attached to the bridge. The following table summarises what
the nix side sets up unconditionally:
| effect | mechanism |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
| effect | mechanism |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them |
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
@ -262,7 +262,7 @@ wiring is runtime:
**Why isolation is safe**: hive-c0re's control-plane sockets are unix
domain sockets bind-mounted into containers, not network listeners — see
the *Control plane (no network)* bullet under [Network
the _Control plane (no network)_ bullet under [Network
map](#network-map) above. `PRIVATE_NETWORK=1` has no effect on a path
that never touches the network stack.

24
docs/networking/snapshot-store.md
View file

@ -43,8 +43,8 @@ Note that the mesh is gated on `swarm.wireguard.enable`, **not** on
## Pointing a hive at it
The block above configures the host that *receives*. Every hive that
*pushes* separately needs to be told where the store is:
The block above configures the host that _receives_. Every hive that
_pushes_ separately needs to be told where the store is:
```nix
services.hyperhive.swarm.snapshotStore = {
@ -65,9 +65,9 @@ inconsistency:
Note the option lives under `swarm.*` while the receiving host's lives
under `services.hyperhive.snapshotStore`. That is the distinction the
two namespaces carry throughout: `swarm.*` describes *the swarm* as seen
from this host, and a bare `services.hyperhive.<service>` describes *a
role this host performs*. A store host sets both --- one to run the
two namespaces carry throughout: `swarm.*` describes _the swarm_ as seen
from this host, and a bare `services.hyperhive.<service>` describes _a
role this host performs_. A store host sets both --- one to run the
receiver, one only if it also runs a hive that pushes.
With it set, `hivectl agent <name> subvol snapshot push <label>
@ -88,7 +88,7 @@ that address provably came from the holder of that private key. A
packet that reaches the receiver has therefore already been
authenticated by the kernel.
Layering TLS client certs on top would authenticate *the same fact* a
Layering TLS client certs on top would authenticate _the same fact_ a
second time, and add a credential with an expiry --- a migration that
fails because a renewal quietly didn't happen, discovered on the day
you need to move an agent.
@ -98,14 +98,14 @@ you need to move an agent.
The destination is keyed by **agent**.
This is not cosmetic. After a migration, an agent's next incremental
send arrives from a *different* hive than the previous one. Keying by
send arrives from a _different_ hive than the previous one. Keying by
hive would split that agent's snapshot chain across two directories,
and `btrfs send -p` would fail to find its parent --- breaking exactly
the case the store exists to serve.
## What the sender can and cannot choose
A `btrfs send` stream carries no notion of *which agent* it belongs to,
A `btrfs send` stream carries no notion of _which agent_ it belongs to,
and the subvolume name inside it is chosen by the sender. So the
protocol is one `agent <name>` header line, then the raw stream.
@ -129,7 +129,7 @@ address being set are assertions, not documentation --- bound to
state.
Binding is not sufficient on its own. NixOS's firewall is default-deny
and filters in netfilter, *before* a packet reaches a bound socket, so
and filters in netfilter, _before_ a packet reaches a bound socket, so
the port is opened explicitly --- and scoped to the mesh interface:
```nix
@ -176,8 +176,8 @@ and means it should get the treatment a backup host gets --- restricted
access, and a decision (rather than an omission) on encryption at rest.
The trap is the label: this box holds backup-grade data while not being
called a backup, so it can end up with backup-grade *exposure* and
non-backup-grade *controls*. Nobody puts a migration staging area on
called a backup, so it can end up with backup-grade _exposure_ and
non-backup-grade _controls_. Nobody puts a migration staging area on
the access-review list.
### What a snapshot contains
@ -202,7 +202,7 @@ gap is tracked separately.
### Retention
Retention lives on the *sending* side (last-N by count, swept
Retention lives on the _sending_ side (last-N by count, swept
periodically), not here. Count rather than age is deliberate: a count
is bounded by construction, whereas an age policy silently scales disk
usage with how hot a hive runs.

10
docs/process/gotchas.md
View file

@ -78,8 +78,8 @@ existing containers can't be started.
### `RestrictAddressFamilies` fails as "Address family not supported by protocol"
A unit whose `RestrictAddressFamilies` omits a family gets `EAFNOSUPPORT`
(errno 97) back from `socket()`. Clients surface that as *"tcp open error:
Address family not supported by protocol"* — the message names the
(errno 97) back from `socket()`. Clients surface that as _"tcp open error:
Address family not supported by protocol"_ — the message names the
**protocol** and never the **sandbox**, so it reads like a dead network, a
missing route, or an IPv6 problem.
@ -98,7 +98,7 @@ re-checks it when the program changes.** A unit that only served a unix
socket when it was written is correct at `[ "AF_UNIX" ]` and silently wrong
the day someone adds an HTTP client. Check the unit in the same commit as
the client — and when narrowing it, prefer a test that derives the required
families from the code (which fails on the *next* client too) over one that
families from the code (which fails on the _next_ client too) over one that
asserts today's list.
### `register_agent` is idempotent
@ -120,7 +120,7 @@ operator's host-level `allowUnfree` does **not** propagate in.
Operators don't need to set anything on their side.
That same isolation is why an agent can't pick a claude out of a
*different* nixpkgs by itself: a container only ever sees the one
_different_ nixpkgs by itself: a container only ever sees the one
nixpkgs the meta flake injects, so an `agent.nix` naming the host's
`nixpkgs-unstable` has nothing to name. A release channel can trail
unstable by weeks on this package, which is what
@ -140,7 +140,7 @@ an input only because a docs tree has no runtime dependencies.
The `storePath` trap is worth spelling out, because it is not confined
to options the operator writes: **any** option of type `package` fed a
store-path *string* coerces through `lib.toDerivation`, i.e.
store-path _string_ coerces through `lib.toDerivation`, i.e.
`builtins.storePath`. `environment.systemPackages` and
`systemd.services.<name>.path` both do it (the latter takes plain
strings like `/run/wrappers` happily, but anything under

6
docs/process/pr-review-gate.md
View file

@ -8,7 +8,7 @@ human review that already happened.
Whether a PR can merge, and what counts toward "can", is configured
per repo in its branch-protection settings — not a fact true of every
hive or every repo. The pieces a repo *can* require:
hive or every repo. The pieces a repo _can_ require:
- **CI is green** — the repo's required status checks pass on the
PR's current head commit, if the repo requires any.
@ -40,8 +40,8 @@ independently.
Auto-merge isn't "no human ever looked at this." Whoever arms it has
already judged the PR sound at a coarse level — the signal it sends is
roughly *"apart from maybe minor tweaks a reviewer can still catch,
I think this is fine."* That's the human-in-the-loop step, and it
roughly _"apart from maybe minor tweaks a reviewer can still catch,
I think this is fine."_ That's the human-in-the-loop step, and it
already happened. No large changes are expected to surface after
that point — a reviewer's job past that point is to flag it if one
does, not to assume none ever will.

18
docs/scheduler/ci.md
View file

@ -18,11 +18,11 @@ Two things worth knowing before you rely on it:
already has forge access; if you take outside/fork contributions,
gate CI behind Forgejo's fork-PR-approval setting or restrict the
workflow to push-only triggers (forks can't push to your branches).
See *Security* below for the full threat model.
See _Security_ below for the full threat model.
- **Watch your disk.** CI builds through the host's nix store with no
automatic garbage collection of its own — a busy CI day can fill
`/nix/store` until every job fails with `ENOSPC`. Add the daily +
disk-pressure GC config from *Host store maintenance* below to your
disk-pressure GC config from _Host store maintenance_ below to your
host's NixOS config (not optional if you plan to actually use this).
Everything below this point is implementation detail: exact
@ -36,11 +36,11 @@ defined in [`.forgejo/workflows/ci.yml`](../../.forgejo/workflows/ci.yml). All
three are required checks (forge branch protection) — a hit on any of them
blocks merge.
| Job | What it runs |
| --- | --- |
| **nix flake check** | treefmt + rustfmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation |
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) |
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) |
| Job | What it runs |
| ---------------------- | ----------------------------------------------------------------------------------------- |
| **nix flake check** | treefmt + rustfmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation |
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) |
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) |
`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
without an empty commit.
@ -90,7 +90,7 @@ slow); run those manually before pushing Rust changes.
The internal forge is always present (mandatory), so the runner always has a
hive-forge instance to register against — nothing extra to enable beyond
`services.hyperhive.deploy.forgejo.ci.enable = true` (see *For operators* above).
`services.hyperhive.deploy.forgejo.ci.enable = true` (see _For operators_ above).
Optional tuning: `services.hyperhive.deploy.forgejo.ci.name` (runner name in forge
admin panel), `concurrency` (parallel job capacity), `labels` (workflow
@ -131,7 +131,7 @@ Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.serv
- The container boots immediately — nothing gates its start on registration.
- tmpfiles seeds `/run/hive-ci/runner-token` with `TOKEN=placeholder` so the runner's `EnvironmentFile` always exists.
- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it is already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
- **Convergence**: because the token write targets the *host* file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
- **Convergence**: because the token write targets the _host_ file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
## Actions checkout mirror

124
docs/scheduler/coordinator.md
View file

@ -19,7 +19,7 @@ changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
concurrency comes from the resource classes below, not from multiple workers.
Special cases like graceful stop, deferred starts, and the meta-update
cascade need no bespoke code paths — each is expressed as a DAG *shape*
cascade need no bespoke code paths — each is expressed as a DAG _shape_
built from the same primitive nodes.
### Two levels: DAG and node
@ -42,54 +42,54 @@ there is no malformed spec to reject.
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| Node | Wraps |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta |
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
| `DeployWindow` | resource-holding root of the merge-config-PR deploy subtree — declares the build slot, the lease and the meta window, then completes immediately so its children run under them (see _Approvals_ below) |
| `DeployApply` | the deploy's irreversible half: ff-merge the reviewed PR head, two-phase meta deploy, container rebuild |
| Node | Wraps |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta |
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
| `DeployWindow` | resource-holding root of the merge-config-PR deploy subtree — declares the build slot, the lease and the meta window, then completes immediately so its children run under them (see _Approvals_ below) |
| `DeployApply` | the deploy's irreversible half: ff-merge the reviewed PR head, two-phase meta deploy, container rebuild |
Cheap — no build slot:
| Node | Behavior |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `MergeVerify` | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation |
| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail: (1) rolls `applied/main` back from the parked `refs/hyperhive/rollback/<id>` and aborts the staged meta lock when the deploy never confirmed good; (2) mirrors whichever deploy tag landed to the forge config repo, always, best-effort; (3) posts the failing build log back onto the config PR when the deploy failed. Named for (2)/(3), which run on the success path too — not `AbortDeploy`. Infallible by construction |
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here |
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
| `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/<id>`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree |
| `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent |
| `EmitRebuilt` | tail of a rebuild/perm-change — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent *and* per outcome |
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
| `Reparent` | `set-parent` / `set-parent-bulk`: apply every `(child, new_parent)` move under one `META_LOCK` commit (`meta::bulk_commit_topology`), send the per-agent move notifications, rescan + diff-emit. Agentless like `MetaLock` — a bulk move can span multiple agents, and a reparent touches the meta repo, not any one container. `moves` is typed `(Ident, Option<Ident>)` pairs, not raw strings. Rides the existing `Template::MetaUpdate` variant rather than a dedicated one — it's internal-only (never reaches the graph wire), so the stand-in only affects `terminal_hook` dispatch (resolves to no hook either way) and history-retention bucketing |
| `ForgeSweep` | one-shot boot-time forge user/token sweep for every container (`forge::ensure_all`) as a first-class node, so it shows as real work on the dashboard instead of running invisibly in a bare `tokio::spawn`. Agentless |
| `MatrixSweep` | same as `ForgeSweep`, for matrix (`matrix::ensure_all`). The periodic 30-min re-sweep stays a background loop in `main.rs`; only the boot-time instance is a node |
| `WebhookRegister` | one-shot boot-time Forgejo webhook registration (`internal/knowledge` push→pull, `agent-configs` PR→approval). No-op until the core token, hive domain, and HMAC secret are all available. Agentless |
| `KnowledgePull` | one-shot boot-time `/knowledge` pull (`knowledge::pull`), reconciling commits that landed while `hive-c0re` was down. Same rationale as `MatrixSweep`: the periodic hourly re-pull stays a background loop |
| `WantedPull` | one-shot boot-time pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. No background loop behind this one — boot is the whole cadence; the deploy event (`swarm_status`) is the fast path, this repairs a missed one. Agentless |
| Node | Behavior |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MergeVerify` | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation |
| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail: (1) rolls `applied/main` back from the parked `refs/hyperhive/rollback/<id>` and aborts the staged meta lock when the deploy never confirmed good; (2) mirrors whichever deploy tag landed to the forge config repo, always, best-effort; (3) posts the failing build log back onto the config PR when the deploy failed. Named for (2)/(3), which run on the success path too — not `AbortDeploy`. Infallible by construction |
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here |
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
| `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/<id>`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree |
| `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent |
| `EmitRebuilt` | tail of a rebuild/perm-change — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent _and_ per outcome |
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
| `Reparent` | `set-parent` / `set-parent-bulk`: apply every `(child, new_parent)` move under one `META_LOCK` commit (`meta::bulk_commit_topology`), send the per-agent move notifications, rescan + diff-emit. Agentless like `MetaLock` — a bulk move can span multiple agents, and a reparent touches the meta repo, not any one container. `moves` is typed `(Ident, Option<Ident>)` pairs, not raw strings. Rides the existing `Template::MetaUpdate` variant rather than a dedicated one — it's internal-only (never reaches the graph wire), so the stand-in only affects `terminal_hook` dispatch (resolves to no hook either way) and history-retention bucketing |
| `ForgeSweep` | one-shot boot-time forge user/token sweep for every container (`forge::ensure_all`) as a first-class node, so it shows as real work on the dashboard instead of running invisibly in a bare `tokio::spawn`. Agentless |
| `MatrixSweep` | same as `ForgeSweep`, for matrix (`matrix::ensure_all`). The periodic 30-min re-sweep stays a background loop in `main.rs`; only the boot-time instance is a node |
| `WebhookRegister` | one-shot boot-time Forgejo webhook registration (`internal/knowledge` push→pull, `agent-configs` PR→approval). No-op until the core token, hive domain, and HMAC secret are all available. Agentless |
| `KnowledgePull` | one-shot boot-time `/knowledge` pull (`knowledge::pull`), reconciling commits that landed while `hive-c0re` was down. Same rationale as `MatrixSweep`: the periodic hourly re-pull stays a background loop |
| `WantedPull` | one-shot boot-time pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. No background loop behind this one — boot is the whole cadence; the deploy event (`swarm_status`) is the fast path, this repairs a missed one. Agentless |
There is deliberately **no `GitCommit` node**: `meta.rs` fuses each mutation
with its commit under its internal `META_LOCK` mutex, so a standalone commit
node would open a dirty-working-tree window between nodes.
Two further layers protect the meta repo across *windows* that span multiple
Two further layers protect the meta repo across _windows_ that span multiple
`META_LOCK` acquisitions — above all the approval deploy's prepare→finalize
span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole
container build:
@ -104,7 +104,7 @@ container build:
because a resource is held by a subtree root across its whole subtree, which
a `MutexGuard` (bounded by one executor fn) cannot — that is what lets a
multi-node deploy own one window. For the same reason the window must stay
*off* long store-only work: the rebuild's meta preamble is its own
_off_ long store-only work: the rebuild's meta preamble is its own
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
toplevel build runs outside the window and `buildSlots > 1` still gives
concurrent rebuilds across agents.
@ -119,10 +119,10 @@ container build:
The `stop` / `start` power ops write the durable `wanted` intent via a head
`SetWanted` node (not a pre-submit side effect) — it holds the agent lease,
so intent-write + reconcile is atomic per-agent. `restart` is the exception:
it writes *no* intent (no `SetWanted` head) — it bounces the container and
it writes _no_ intent (no `SetWanted` head) — it bounces the container and
lets the tail `Reconcile` converge to the agent's existing `wanted`, so a
deliberately-stopped agent is not forced back up by a hive-wide restart. The
hive-wide power ops — `restart`, `stop`, and `start` — take an agent *list*:
hive-wide power ops — `restart`, `stop`, and `start` — take an agent _list_:
a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
subgraph each (independent roots, run concurrently on their own leases), not
N separate DAGs.
@ -132,12 +132,12 @@ async `lifecycle::is_running` read), so they live in `job_queue/power.rs`,
not the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
a head `SetWanted` (intent) — `restart` does not; the tail `Reconcile`
(convergence guarantee — cheap, noops when already converged) is ALWAYS
present; only the *mechanical* nodes (`Signal`/`Drain`/`StopForUpdate`) are
state-conditional — skipped for a *down* agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
present; only the _mechanical_ nodes (`Signal`/`Drain`/`StopForUpdate`) are
state-conditional — skipped for a _down_ agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
closes the TOCTOU window: if an agent flips state between the `is_running`
read and node exec, the tail `Reconcile` still converges it in-DAG (with
`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale*
sweep. `start` folds the per-agent stale-rev upgrade in (a _down + stale_
agent's subgraph is a rebuild-then-start).
```text
@ -163,7 +163,7 @@ Notable collapses:
leaves it stopped).
- **The swap-failure recovery-start is structural**: `Reconcile` deps on
`Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
terminal ok *or* fail, bringing a wanted-up agent back on its old config.
terminal ok _or_ fail, bringing a wanted-up agent back on its old config.
- **Deferred start is automatic**: `Reconcile` holds no build slot, so the
next DAG's `Prebuild` starts as soon as `Swap` frees the slot.
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
@ -171,7 +171,7 @@ Notable collapses:
drains overlap; each DAG's tail `Reconcile` does the actual stop.
- **The meta-update cascade grows in the same DAG on completion**:
`MetaLock`'s executor computes the affected agent set after the bump lands
and grows one `rebuild` subgraph per agent into its *own* DAG via
and grows one `rebuild` subgraph per agent into its _own_ DAG via
`append_subgraph` (rooted on the `MetaLock`, `relock = false` so the cascade
doesn't revert the bump). Not child DAGs — one DAG, no `parent_id`. A failed
bump appends nothing (no cancel-children dance). Same shape as the startup
@ -180,9 +180,9 @@ Notable collapses:
### Desired-state (spec vs status)
Per-agent power *intent* — `wanted: Up | Offline` — is durable as the
Per-agent power _intent_ — `wanted: Up | Offline` — is durable as the
`agent_power` table in the coordinator DB (`hive-c0re/src/stores/power.rs`).
`container_view` remains the observed *status*; `Reconcile` nodes converge the
`container_view` remains the observed _status_; `Reconcile` nodes converge the
two. Setting `wanted` is never a queued node: the power layer
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
@ -237,15 +237,15 @@ than to one DAG shape it happens to appear in. A **brace** is the one sanctioned
exception: a pure-resource-holder root that declares on behalf of a subtree
coordinated with itself, whose members then declare nothing.
It is forced rather than stylistic. Declaring a resource means *"I need this
exclusively"*, and the agent lease is single-unit — so **two siblings that both
It is forced rather than stylistic. Declaring a resource means _"I need this
exclusively"_, and the agent lease is single-unit — so **two siblings that both
declared it could never run concurrently.** For a subtree whose whole point is
concurrency (`Prebuild` beside the `Signal` → `Drain` quiesce window), declaring
the requirement truthfully on every node and running those nodes in parallel are
mutually exclusive. One holder above them speaks for the subtree.
This is the opposite of the failure the declare-your-own rule exists to prevent,
not a relapse into it: there the requirement was *implicit*, inferred from a
not a relapse into it: there the requirement was _implicit_, inferred from a
node's kind and true only by accident of placement. Here it is explicit, on one
node, with the omission below it documented on the brace itself.
@ -257,7 +257,7 @@ Two consequences worth knowing:
leave a gap another DAG could claim the agent in, mid-bounce. A brace supplies
that continuity directly, so the nesting is no longer load-bearing.
- **Observability is unaffected.** `running_transients` keys off a node's
*payload* agent, not off a declared lease edge, so every child still lights its
_payload_ agent, not off a declared lease edge, so every child still lights its
own dashboard pill and still reports its own `takes_container_down` to the
crash watcher. A brace itself reports `false`: it parents the stopping nodes
but does not stop anything, and claiming otherwise would widen crash
@ -323,12 +323,12 @@ cancelled-while-queued, which fails the approval instead of dangling it).
### Wire shape
`RebuildQueueChanged { seq }` (event name kept) — **a bare trigger, no
payload.** It says *the queue changed*; a client that wants to know how
payload.** It says _the queue changed_; a client that wants to know how
re-fetches `GET /api/jobq/graph`.
That endpoint serves the graph generically (`hive-jobq-wire`): every node
carries `id`, `parent`, `deps`, `state`, `label` (the node kind's own wire
string — the kind *is* the phase label, there is no separate sub-step
string — the kind _is_ the phase label, there is no separate sub-step
string) and free-form `data` for what only some kinds have (`agent`,
`approval_id`, `inputs`, `build_log_id`). Group roots ride as ordinary
nodes, so a group's state is just the root's own `state`.
@ -341,7 +341,7 @@ live-log panel off the running node.
The event carries no payload by design: shipping a typed projection of
the whole queue in the event itself would be a second rendering of the
same graph that has to be kept in agreement by hand with the endpoint
every consumer actually reads. Telling a client *when* to refetch is
every consumer actually reads. Telling a client _when_ to refetch is
the event's whole job.
---
@ -362,7 +362,7 @@ On startup, `auto_update::run` classifies every agent by rev freshness (the
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
vs the current flake path) and persisted `wanted` intent, then:
1. **Config path** — when *any* marker is stale, submit one `Boot`
1. **Config path** — when _any_ marker is stale, submit one `Boot`
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that grows an in-DAG
`Rebuild` subgraph for each stale agent whose `wanted = Up` (topology-sorted,
parents first). Stale but wanted-offline agents get no boot-time nix work — their
@ -453,7 +453,7 @@ graft's completion then plants the deploy tag — so "did the agent come back
up?" is answered by `Reconcile` succeeding, the same way it is for every
other rebuild, instead of by a fused inline start.
The grafted nodes land *inside* `DeployWindow`'s subtree, so they re-enter
The grafted nodes land _inside_ `DeployWindow`'s subtree, so they re-enter
the meta window and build slot it already holds rather than deadlocking
against it.
@ -514,7 +514,7 @@ The same drop-in carries `CPUWeight=` / `IOWeight=` from
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
setting: the quota and the memory max are **hard caps** that throttle
an agent even on a completely idle host, while the weights are cgroup
v2 **relative shares** that only decide who yields *under contention*.
v2 **relative shares** that only decide who yields _under contention_.
A low-weight container still gets the whole machine when nothing else
wants it.
@ -529,7 +529,7 @@ Two things to know about the weights:
- They are **hive-wide** — unlike the caps there is no per-agent
override in `meta/resource-limits.json`, so every agent carries the
same value and the weight does *not* rank agents against each other.
same value and the weight does _not_ rank agents against each other.
What `80` buys is that agents yield to everything **not** on this
drop-in path: host services (nginx and dnsmasq among them) and the
infra containers (`hive-ci`, `hive-forge`, `hive-gateway`,

20
docs/scheduler/jobq.md
View file

@ -2,7 +2,7 @@
Every container operation — rebuild, first-spawn, a config-PR deploy,
power changes — runs through one shared job queue. This page explains
what the job queue *is*, as a general idea, independent of what any one
what the job queue _is_, as a general idea, independent of what any one
subsystem uses it for. For the hive-c0re-specific step catalogue and the
engineering internals (scheduler, leases, resource windows) see
[`coordinator.md`](coordinator.md) instead.
@ -23,7 +23,7 @@ Two ideas are all there is to it:
that's the whole reason things queue instead of all firing at once.
The engine's whole job is: whenever a step's ordering and resource needs
are both satisfied, run it. It has no opinion on what the steps *do* —
are both satisfied, run it. It has no opinion on what the steps _do_ —
that's supplied by whoever builds the graph. hive-c0re is the one thing
building graphs on it today, but nothing about the engine is specific to
containers or rebuilds; there's nothing stopping another subsystem from
@ -38,14 +38,14 @@ nested under it are that job's steps, in order (occasionally a couple run
side by side). A step shows one of:
| Glyph | Meaning |
| ----- | -------------------------------------------------------- |
| `⏸` | queued, waiting its turn |
| `▶` | running |
| `◐` | its own work is done, waiting on a step nested under it |
| `✔` | finished successfully |
| `✖` | failed |
| `⊘` | cancelled |
| `·` | skipped (not needed for this run) |
| ----- | ------------------------------------------------------- |
| `⏸` | queued, waiting its turn |
| `▶` | running |
| `◐` | its own work is done, waiting on a step nested under it |
| `✔` | finished successfully |
| `✖` | failed |
| `⊘` | cancelled |
| `·` | skipped (not needed for this run) |
A step that isn't needed for a given run shows as `·` rather than being
left out of the tree entirely, so the same kind of operation keeps a

94
docs/scheduler/observability.md
View file

@ -46,7 +46,7 @@ no longer survives independently of anything host-side.
**It has no application-level auth.** The receiver takes any OTLP that reaches
it; what bounds who can reach it is the firewall — `exposeHostPorts` opens the
port on the bridge interface only. So "unauthenticated to a bridge address"
means *reachable from an agent container*, not *presents a credential*.
means _reachable from an agent container_, not _presents a credential_.
The consequence, stated because it is a choice rather than an oversight: **any
agent can push arbitrary OTLP, and it is forwarded on under the operator's
@ -56,10 +56,10 @@ smuggled out in resource attributes on an otherwise-legitimate export.
That is a **different risk from the one the collector fixes**, and strictly
smaller than what preceded it: before, every agent held the upstream credential
itself, so it could do all of the above *and* use the token anywhere else. The
itself, so it could do all of the above _and_ use the token anywhere else. The
collector removes the token and keeps the pipe. Agents are inside the trust
boundary (`docs/trust-boundary/security.md`: capability = accepted risk), so an agent being
able to *send* is an accepted extension of that boundary — but it is not
able to _send_ is an accepted extension of that boundary — but it is not
closed by this design, and nothing here should be read as closing it.
**The `agent` label is self-reported, and no planned authentication changes
@ -73,13 +73,13 @@ is getting a credential:
- **agent→collector** (this section's hop) stays open on the bridge. Nothing
downstream can tell one agent's export from another's.
- **hive→swarm** is where the planned ingest auth goes. The swarm tier stamps
`hive=` from the connection it authenticated, so *that* label becomes
`hive=` from the connection it authenticated, so _that_ label becomes
unforgeable.
So a verified `hive` is reachable and a verified `agent` is not — and that falls
out of the topology rather than being a gap someone forgot to close. The swarm
runs one collector, and the mechanism gives it no finer grain: a bearer-token
check never reveals *which* token matched, and a receiver reads request metadata
check never reveals _which_ token matched, and a receiver reads request metadata
rather than the claims it authenticated with.
If you need per-agent numbers you can act on, take them from the agent's own
@ -100,10 +100,10 @@ two-tier architecture, the security model, and how the options interact.
Telemetry crosses two collectors, and which one you configure depends on what
the host is:
| | runs where | receives from | does |
|---|---|---|---|
| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination |
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's store and exports upstream |
| | runs where | receives from | does |
| ------------------------------------ | ---------------------- | --------------------------------- | --------------------------------------------------------------------- |
| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination |
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's store and exports upstream |
An all-local host runs both, and needs nothing said about the hop between them.
@ -119,16 +119,16 @@ services.hyperhive.otel = {
every agent needs the credential to authenticate — and the harness delivers
that token into the agent's own `~/.claude/settings.json`, a file the agent can
read. `0600` protects it from other containers, not from the agent itself. As
long as the direct path stays *selectable*, that hole stays selectable; an
long as the direct path stays _selectable_, that hole stays selectable; an
option that can reintroduce it is a hole with extra steps.
**Why the tiers stay separate on one box.** They are not collapsed when
co-located: an all-local hive is a statement about *where* processes run, not
co-located: an all-local hive is a statement about _where_ processes run, not
about the shape of the deployment. A boundary that disappears locally is one
the local deployment stops testing.
**`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier
redefines it — the agent-facing value is *derived*
redefines it — the agent-facing value is _derived_
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
automatically; there is nothing to open by hand.
@ -160,9 +160,9 @@ swarm tier and no metrics appearing for that hive.
⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
exporter takes an *address*: `https://host/path` is a legal
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in
address"*. The collector's config is validated at build time, so a mismatch is
exporter takes an _address_: `https://host/path` is a legal
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in
address"_. The collector's config is validated at build time, so a mismatch is
a build error naming the reason rather than telemetry silently going nowhere.
## Network access
@ -176,7 +176,7 @@ services.hyperhive.network.exposeHostPorts = [ 5432 ];
```
and point whatever consumes it at `10.42.0.1:5432` rather than loopback: inside
a container, loopback is the *container*. The bridge IP is the host's address on
a container, loopback is the _container_. The bridge IP is the host's address on
the `hive-br0` bridge. The service must also bind an address the bridge can
reach — a `127.0.0.1`-only listener stays unreachable no matter what the
firewall allows. See `docs/networking/network.md::Reaching host services` for details.
@ -196,12 +196,12 @@ yourself from a tool call or `hivectl shell`.
Every agent's export therefore includes these resource attributes
automatically:
| Attribute | Value |
|-----------|-------|
| `service.name` | `hyperhive-agent` (constant) |
| `agent` | agent logical name (e.g. `iris`) |
| `hive` | hive display name (`services.hyperhive.hiveName`) |
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
| Attribute | Value |
| -------------- | ------------------------------------------------------------ |
| `service.name` | `hyperhive-agent` (constant) |
| `agent` | agent logical name (e.g. `iris`) |
| `hive` | hive display name (`services.hyperhive.hiveName`) |
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
Additional labels can be appended via `extraResourceAttributes` (see option
reference above); custom per-data-point labels can be passed with
@ -222,14 +222,14 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
`service.name = hyperhive-c0re`; each data point is tagged `container.name`
(= the `h-<agent>` machine) and the hive `agent` label:
| Metric | Unit | Kind | Source |
|--------|------|------|--------|
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | state dir + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
| Metric | Unit | Kind | Source |
| ----------------------------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------- |
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | state dir + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
@ -252,14 +252,14 @@ turn, its own outcome classification, the loose-ends backlog, and session
boundaries. Token usage, cost, and tool-call counts are **not** duplicated
here; that's already covered by Claude's own export.
| Metric | Unit | Kind | Attributes |
|--------|------|------|------------|
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `CLAUDE.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
| Metric | Unit | Kind | Attributes |
| -------------------------------------- | ---- | --------- | ----------------------------------------------------------------------------------------------------------- |
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `CLAUDE.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
Resource attributes (`service.name`, `agent`, `hive`, `swarm`) come from the
same container-wide `OTEL_RESOURCE_ATTRIBUTES` as everything else in this
@ -277,11 +277,11 @@ that hosts no agents still reports, and "this hive is quiet" is
distinguishable from "this hive is gone". Select them with
`{hive!="",agent=""}`.
| Metric | Unit | Kind | Meaning |
|--------|------|------|---------|
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
| Metric | Unit | Kind | Meaning |
| ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
Both levels are reported every cycle, `0` included, so a healthy hive is
visible as zeros rather than as missing series.
@ -303,10 +303,10 @@ Forgejo's own native `/metrics` endpoint has no equivalent: it exposes
counts of durable rows (issues, comments, repos), and neither a commit nor a
push is stored anywhere as a row to count.
| Metric | Unit | Kind | Attributes |
|--------|------|------|------------|
| `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) |
| `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) |
| Metric | Unit | Kind | Attributes |
| ---------------------------- | ---- | ------- | ------------------- |
| `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) |
| `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) |
A push with zero commits (a branch delete, or a force-push that doesn't add
new commits) still increments `push.count`; `commit.count` only advances

48
docs/swarm/README.md
View file

@ -7,7 +7,7 @@ the additional config needed when the swarm spans multiple hosts.
For the full option reference rather than prose: `services.hyperhive.swarm.*`
(swarm-wide facts, identical on every host) and `services.hyperhive.deploy.*`
(this host's own deployment decisions — does *this* machine run grafana,
(this host's own deployment decisions — does _this_ machine run grafana,
the swarm controller, authelia, …) are separate generated pages, `nix
build .#docs-swarm` / `.#docs-deploy` or the website's `/options/swarm.html`
/ `/options/deploy.html`.
@ -20,7 +20,7 @@ build .#docs-swarm` / `.#docs-deploy` or the website's `/options/swarm.html`
- **swarm** — one or more hives whose operators have declared them
as peers. Agents can be qualified as `agent@hive-domain`.
- **peer hive** — any hive in `services.hyperhive.swarm.hives` other
than this one. Peers are *derived*, not declared: the directory lists
than this one. Peers are _derived_, not declared: the directory lists
every hive including yourself, and `hiveName` says which one you are.
## Hive identity config
@ -47,7 +47,7 @@ and deploys — an eval failure asking the operator to write the address
down is the cheaper outcome. **Upgrading past this release means setting
both once.**
`domain` is required too, but you no longer *write* it: it is read from
`domain` is required too, but you no longer _write_ it: it is read from
this hive's own entry in the directory, whose `domain` defaults to
`<name>.<swarm.domain>`. So a conventional swarm states no addresses at
all, and a hive addressed by something else states it in the one place
@ -70,7 +70,7 @@ form qualified labels (`iris@pr1ma.example.com`).
`swarm.name` is purely display — it surfaces in the dashboard chrome
header and per-agent system prompts, and federated hives at different
domains can share one. `hiveName` surfaces in the same places but is
*not* only display: it is the leftmost label of the hive's domain. That
_not_ only display: it is the leftmost label of the hive's domain. That
`swarm.name` sits under `swarm` and `hiveName` does not is the whole
distinction — one names this hive, the other names the group it belongs
to.
@ -120,14 +120,14 @@ services.hyperhive.swarm.hives = {
```
One attrset describing **every** hive in the swarm, **including this
one**, keyed by that hive's `hiveName`. It is meant to be *identical on
every host* — write it once, share it, and each host reads it correctly
one**, keyed by that hive's `hiveName`. It is meant to be _identical on
every host_ — write it once, share it, and each host reads it correctly
because `services.hyperhive.hiveName` says which entry is itself.
Empty (the default) means this host isn't in a swarm. Once non-empty it
**must** contain an entry for `hiveName`; eval fails naming the missing
hive. That assertion is load-bearing rather than pedantic — "my peers"
is derived as *everything that isn't me*, so a directory that doesn't
is derived as _everything that isn't me_, so a directory that doesn't
contain you derives every hive as a peer and you peer with yourself.
`domain` is required per entry and deliberately undefaulted: it is
@ -138,7 +138,7 @@ evaluates cleanly points at a real machine that isn't the one you meant.
> inside a swarm comes from the swarm root ([`ca.md`](ca.md)): every
> hive chains to it, so one anchor replaces per-hive pinning entirely.
> What that genuinely drops is trusting a hive whose root this swarm
> does *not* own — another swarm's, or one keeping its own CA. That is
> does _not_ own — another swarm's, or one keeping its own CA. That is
> a cross-swarm problem and wants a mechanism designed for it. (An
> earlier `certFingerprint` field existed for exactly that gap, pinning
> a peer's TLS leaf for hive-c0re's own peer HTTPS checks — removed
@ -162,7 +162,7 @@ evaluates cleanly points at a real machine that isn't the one you meant.
⚠️ **That container currently trusts no swarm-internal CA**, so a
self-signed gateway certificate does not federate. The swarm root
can't simply be listed there: `security.pki.certificateFiles` is
read when the system is *built*, and the root is a runtime file (its
read when the system is _built_, and the root is a runtime file (its
key must never enter the store), so there is no build-time name for
it. Bridging that needs a runtime mechanism and is tracked as its own
issue. Until then, federation needs CA-issued certs (ACME). See
@ -184,7 +184,7 @@ swarm.hives = { … }; swarm.hives = { … }; # byte-identical
```
That is the point of the shape, and it removes a class of bug rather
than saving typing: a per-host peer list let two hosts hold *different*
than saving typing: a per-host peer list let two hosts hold _different_
facts about the same third hive — a stale endpoint, a rotated
fingerprint — with nothing to detect the disagreement. One entry per
hive makes it unrepresentable.
@ -277,7 +277,7 @@ port}` tells this hive where the swarm's `btrfs receive` endpoint is, so
`hivectl agent <name> subvol snapshot push` has somewhere to stream to.
It is genuinely swarm-scoped rather than per-peer — a swarm has exactly
one store, because the receiver keys destinations by *agent* so a
one store, because the receiver keys destinations by _agent_ so a
migrating agent keeps one unbroken incremental chain. See
[snapshot-store.md](../networking/snapshot-store.md).
@ -305,12 +305,12 @@ A hive publishes only once it has been given the three
`swarm.statusPublish` coordinates below. A hive that has not reads
`never_reported` — it is not broken, it just has nothing to say upward.
| freshness | what to do about it |
|---|---|
| `fresh` | nothing — reported within `staleAfterSeconds` |
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
| `unknown` | something is publishing under a name that is not in `swarm.hives` — a typo in the roster, or a hive that was removed and is still running |
| freshness | what to do about it |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `fresh` | nothing — reported within `staleAfterSeconds` |
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
| `unknown` | something is publishing under a name that is not in `swarm.hives` — a typo in the roster, or a hive that was removed and is still running |
Every row also carries `last_seen_unix` and `age_seconds` if you want to
apply your own threshold. The timestamp is the one the queue recorded on
@ -328,11 +328,11 @@ Three options, on the **hive**, set together or not at all — a
half-configured hive is an eval error rather than one that quietly never
reports:
| option | what to set it to |
|---|---|
| `natsUrl` | where the swarm queue listens, as this hive reaches it |
| `tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
| `clientSecretFile` | path to this hive's client secret, plaintext |
| option | what to set it to |
| ------------------ | ------------------------------------------------------ |
| `natsUrl` | where the swarm queue listens, as this hive reaches it |
| `tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
| `clientSecretFile` | path to this hive's client secret, plaintext |
On a host that runs the queue and the IdP itself, all three default to
the local ones and there is nothing to set. Any other hive needs them
@ -364,8 +364,8 @@ on the `agent-configs` org, both under
`https://<swarm.domain>/webhook/forge/`.
The controller **interprets** a delivery and sends hives a specific
message — *the knowledge repo changed*, *deploy agent `foo` at rev
`abc123`* — rather than forwarding forge payloads for each hive to
message — _the knowledge repo changed_, _deploy agent `foo` at rev
`abc123`_ — rather than forwarding forge payloads for each hive to
re-derive. Approval happens once, at the swarm level: a hive receives a
decision, not an event to adjudicate.

18
docs/swarm/ca.md
View file

@ -12,9 +12,9 @@ O(n²) hand-pinning into one anchor per swarm.
What differs is who puts the artifacts on disk:
| | swarm root | this hive's CA |
| --- | --- | --- |
| default | operator-provided | operator-provided, else self-signed as before |
| | swarm root | this hive's CA |
| ---------------------------------------- | --------------------------------------------- | ---------------------------------------------- |
| default | operator-provided | operator-provided, else self-signed as before |
| `autoConfigure = true` (all on one host) | generated by `swarm-ca.service` on first boot | issued by `hive-tls-ca.service` under the root |
`services.hyperhive.swarm.ca.autoConfigure` selects between them, and is
@ -49,13 +49,13 @@ swarm-services leaf can, and this host signs that one only under
and every client sees a name mismatch.
So `hive-tls` emits an eval-time **warning** naming the uncovered names.
It is deliberately not an assertion: this module can see what *it* is
It is deliberately not an assertion: this module can see what _it_ is
able to issue, but not an operator-installed sub-CA in
`swarm.ca.stateDir`, an external ACME setup, or a certificate delivered
by any other means. If you have arranged one, the warning is expected
and can be ignored. Otherwise either install the sub-CA, or pin the
names back under the hive domain — a supported migration, since the
sub-CA is constrained to the *configured* names and the swarm root
sub-CA is constrained to the _configured_ names and the swarm root
carries no name constraints at all.
## Constraints on the material
@ -96,7 +96,7 @@ CA has nothing to install — `peer-config` says so instead.
Handing a peer this hive's own `ca.pem` would not work even as a
one-off: once a hive CA is an intermediate under the swarm root, it is
no longer something a verifier can build a chain *to* — OpenSSL will not
no longer something a verifier can build a chain _to_ — OpenSSL will not
terminate a chain at a trusted non-self-signed certificate without
`-partial_chain`. That is why the tool distributes the root, not a
per-hive file.
@ -127,7 +127,7 @@ that has not restarted yet still validates. Removing `ca-previous.pem`
is a deliberate later step: how long is long enough is a property of the
deployment, not something the unit can know.
A marker file (`.swarm-ca-adopted`) records that this ran. Its *absence*
A marker file (`.swarm-ca-adopted`) records that this ran. Its _absence_
is the trigger, so adoption fires once per hive rather than being
re-decided on every activation.
@ -157,7 +157,7 @@ this: it self-signs exactly as it always has.
The root **key** is a runtime file for the reason in
[Constraints on the material](#constraints-on-the-material).
The root **certificate** is a runtime file as a *consequence* — it lives
The root **certificate** is a runtime file as a _consequence_ — it lives
beside the key under `swarm.ca.stateDir` — and that has a cost worth
naming, because it is not obvious and it bites at a distance:
@ -185,7 +185,7 @@ Two consumers, and only one of them is fine:
an OpenSSL-linked binary.
> ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.**
> `openssl-probe` uses it *instead of* the default store, so naming
> `openssl-probe` uses it _instead of_ the default store, so naming
> just the bundle would drop every public CA and break federation with
> the wider matrix network — a much bigger outage than the one being
> fixed. The same caveat applies to `hive-forge` (Go) for the same

66
docs/swarm/secrets.md
View file

@ -29,34 +29,34 @@ container that starts after it.
Every row below is read against one of these. This is a different cut
from the [deployment shapes](services.md#deployment-shapes) --- those
say *where services run*, these say *who is responsible for a secret
file being there* --- so the two lists do not line up one-to-one, and
say _where services run_, these say _who is responsible for a secret
file being there_ --- so the two lists do not line up one-to-one, and
neither is a renaming of the other.
| topology | what it means | who places secrets |
|---|---|---|
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — each secret is generated where it is read, or copied by a host unit |
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
| **hive elsewhere** | a hive that federates with a swarm it does not host | the operator provides the file and names it in config |
| topology | what it means | who places secrets |
| ------------------ | ---------------------------------------------------------- | ---------------------------------------------------------------------------- |
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — each secret is generated where it is read, or copied by a host unit |
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
| **hive elsewhere** | a hive that federates with a swarm it does not host | the operator provides the file and names it in config |
## Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere |
|---|---|---|---|
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it is public |
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `swarm.otel.clientSecretFile` — the collector need not share a host with authelia |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `swarm.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `nats.calloutUserSeedFile` / `nats.calloutIssuerSeedFile` |
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a *client* of it and holds none of this |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it is stated rather than inferred | n/a — only the host running the store seals anything |
| secret | generated by | lives at | hive elsewhere |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it is public |
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `swarm.otel.clientSecretFile` — the collector need not share a host with authelia |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `swarm.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `nats.calloutUserSeedFile` / `nats.calloutIssuerSeedFile` |
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it is stated rather than inferred | n/a — only the host running the store seals anything |
The three keys authelia mints for itself are generated in-container precisely
because nothing outside that container ever reads them. **That is the test worth
@ -81,7 +81,7 @@ The copy is only made when authelia is enabled **on this host** and something
published is being scraped; otherwise no secret is needed and none is placed.
⚠️ **Do not read that delivery unit as the only way this collector is fed.**
Whether it authenticates follows the *credential*, never another service's
Whether it authenticates follows the _credential_, never another service's
placement: a swarm collector may run on a host that holds neither store and no
authelia, and then the secret is an ordinary operator-provided file named in
`services.hyperhive.swarm.otel.clientSecretFile` — the same shape as the hive
@ -126,13 +126,13 @@ when the system builds. The server names the offending file and refuses to run.
## Hive-level — one of each per hive
| secret | generated by | lives at |
|---|---|---|
| hive CA cert + key | `hive-tls.nix` first-boot unit | `<deploy.hive-controller.tls.stateDir>/ca.pem`, `ca-key.pem` (`0600`) |
| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| matrix registration token | a host activation script, on first boot | `/var/lib/hyperhive/matrix-register-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| secret | generated by | lives at |
| ---------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| hive CA cert + key | `hive-tls.nix` first-boot unit | `<deploy.hive-controller.tls.stateDir>/ca.pem`, `ca-key.pem` (`0600`) |
| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| matrix registration token | a host activation script, on first boot | `/var/lib/hyperhive/matrix-register-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
Both delivery units wait for authelia's first boot to mint the secret — a
bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip
@ -141,7 +141,7 @@ layers from its cause.
The store's **first reader** is the matrix registration token, and it is worth
saying why that one: it is an opaque 32-byte value with no second file and no
format. Authelia's OIDC secret needs a `.secret` *and* a matching `.digest`, so
format. Authelia's OIDC secret needs a `.secret` _and_ a matching `.digest`, so
starting there would have meant debugging "can a reader authenticate and get
bytes back" and "did we write authelia's file format right" at once, with an
SSO outage as the failure mode.
@ -182,7 +182,7 @@ possible at all. Two things follow:
containers through a bind-mounted bundle assembled at boot instead.
- **On any topology other than all-local, placing that key is an operations
decision**, not something this module tree makes for you. A hive that hosts no
swarm services needs only the root's *cert*, to trust what others issue.
swarm services needs only the root's _cert_, to trust what others issue.
## Adding a secret

31
docs/swarm/services.md
View file

@ -10,7 +10,7 @@ services.hyperhive.deploy.allSwarmServices = true;
```
**`deploy.allSwarmServices` is what "the swarm's shared services run
here" means: every once-per-swarm service that is *optional* takes its
here" means: every once-per-swarm service that is _optional_ takes its
`enable` from it.** That is the whole rule, stated once — the per-service
sections below do not repeat it, so a service that stops deriving is a
visible difference rather than one more paragraph saying the same thing.
@ -23,7 +23,7 @@ which wins, so "all local except X" needs no further option.
**Both default to off**, and that is deliberate: a host cannot tell
whether it is meant to be the swarm's service host, so this is an
operator saying so rather than something inferred. With them off, a hive
is a *client* of those services — it configures how to reach them and
is a _client_ of those services — it configures how to reach them and
runs none of them.
The forge is the exception, and not because it is per-hive: it is
@ -112,11 +112,11 @@ services.hyperhive.deploy.grafana.enable = false;
⚠️ **This starts a database that grows for as long as the swarm runs.**
See `retentionPeriod` below before leaving it at its default.
| Option | When you'd touch it |
|---|---|
| Option | When you'd touch it |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you cannot get back. |
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who should not be able to reconfigure Grafana. |
| `swarm.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store on this host, which is the only thing it can reach. |
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who should not be able to reconfigure Grafana. |
| `swarm.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store on this host, which is the only thing it can reach. |
**Logging in.** Grafana is behind swarm SSO, so the accounts are the
authelia ones — there is no separate Grafana password, and the local
@ -138,16 +138,16 @@ the hive's daemons and infra containers, and the harness units inside every
agent container. The collector below is what writes to it.
**Reading them.** Open Grafana, pick **Explore**, and choose the
`VictoriaLogs` datasource — it is provisioned for you. Grafana's *Logs
Drilldown* app is deliberately not installed: it only supports Loki, and
`VictoriaLogs` datasource — it is provisioned for you. Grafana's _Logs
Drilldown_ app is deliberately not installed: it only supports Loki, and
no setting here changes that, so Explore is the log browser for this
swarm.
| Option | When you'd touch it |
|---|---|
| Option | When you'd touch it |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are usually read within days of being written. Raise it if you need to answer questions about last quarter. |
| `swarm.victorialogs.domain` | Only to rename it. |
| `swarm.victorialogs.port` | Only if something else on the services host already claims `9428`. |
| `swarm.victorialogs.domain` | Only to rename it. |
| `swarm.victorialogs.port` | Only if something else on the services host already claims `9428`. |
Like the metrics store, it binds loopback and takes no credential of its
own: the gateway vhost is the only way in, and the collector is the only
@ -173,10 +173,10 @@ loopback URL an operator has to redirect. There is nothing to set on a hive
that does not run the swarm's services; the name resolves through the
gateway either way.
| Option | When you'd touch it |
|---|---|
| Option | When you'd touch it |
| ------------------- | --------------------------------------------------------------------------------------- |
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
| `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
| `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
With neither `otel.endpoint` nor the store enabled, this collector is
refused at eval — a tier that receives samples and drops them looks
@ -184,4 +184,3 @@ healthy while losing data.
Agent-side configuration, and what a hive's own collector does, are in
[`../scheduler/observability.md`](../scheduler/observability.md).

16
docs/swarm/sso.md
View file

@ -157,12 +157,12 @@ evaluation error.
The registration half is identical; what each service does with the
result is not.
| | forge | matrix |
|---|---|---|
| how it learns the config | a oneshot calls `forgejo admin auth`, writing a login-source row into its database | tuwunel reads a `[[global.identity_provider]]` entry from its config file |
| how it reads the secret | a path inside its container | the same path, handed on by `LoadCredential` |
| callback URL | `<root>/user/oauth2/<source>/callback` | `<homeserver>/_matrix/client/unstable/login/sso/callback/<client_id>`, a shape tuwunel fixes rather than accepts |
| cost of a malformed entry | the login source is missing | the homeserver can refuse to start |
| | forge | matrix |
| ------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| how it learns the config | a oneshot calls `forgejo admin auth`, writing a login-source row into its database | tuwunel reads a `[[global.identity_provider]]` entry from its config file |
| how it reads the secret | a path inside its container | the same path, handed on by `LoadCredential` |
| callback URL | `<root>/user/oauth2/<source>/callback` | `<homeserver>/_matrix/client/unstable/login/sso/callback/<client_id>`, a shape tuwunel fixes rather than accepts |
| cost of a malformed entry | the login source is missing | the homeserver can refuse to start |
Two consequences worth stating plainly:
@ -213,13 +213,13 @@ friendly "SSO is unavailable" page; that page is a file, so it returns
HTML instead of the failure that actually happened:
- `/api/authz/auth-request` — nginx `auth_request` treats **any 2xx as
success**, so a down provider means *access granted*
success**, so a down provider means _access granted_
- `/api/oidc/introspection` — a token check that answers `200`
- `/api/oidc/token`, `/.well-known/openid-configuration` — a client
parsing an error page as its JSON document
So authelia's `/api/` and `/.well-known/` prefixes are routed **without**
error interception. The split is by *audience*, not by an enumerated path
error interception. The split is by _audience_, not by an enumerated path
list: a human gets the page, every machine caller gets the status.
Enumerating endpoints individually would leave the next one added
silently intercepted.

20
docs/swarm/ui.md
View file

@ -4,7 +4,7 @@ The swarm's own web surface, served by the gateway on the **swarm apex**
(`services.hyperhive.swarm.domain`) and readable only by operators.
Distinct from the per-hive dashboard, which lives on the hive domain and
answers for one host. This one is the view *across* hives.
answers for one host. This one is the view _across_ hives.
## Enabling
@ -15,7 +15,7 @@ services.hyperhive.deploy.swarm-ui.enable = true; # defaults to deploy.swarm-c
Derived from the controller rather than from `allSwarmServices`:
the UI is a view onto the controller's state and reaches it over that
daemon's socket, so the host that runs the controller is the host that
can serve the UI. A hive that merely *uses* a swarm has nothing to serve.
can serve the UI. A hive that merely _uses_ a swarm has nothing to serve.
The UI answers on `services.hyperhive.swarm.domain` and nothing else. It
shares that name with the swarm-controller it fronts — one service to a
@ -29,8 +29,8 @@ reader and to a certificate — so there is no separate option to pin.
## 🔑 You must be in the `admins` group
**This is the step that separates "protected" from "locked out".** The
vhost's `auth_request` asks authelia *"is there a session"*; the rule
that makes it mean *"is this an operator"* is an `access_control` entry
vhost's `auth_request` asks authelia _"is there a session"_; the rule
that makes it mean _"is this an operator"_ is an `access_control` entry
requiring `group:admins`. An account without that group authenticates
fine and still gets bounced.
@ -41,7 +41,7 @@ swarmctl user add <you> --group admins
`admins` deliberately, not a new word: [`../getting-started/setup.md`](../getting-started/setup.md) has
told every operator to create exactly that group since the bootstrap step
existed, so an account made by following the guide already passes. This
is the first rule that *consumes* a group name — inventing a second one
is the first rule that _consumes_ a group name — inventing a second one
would have meant those accounts silently failing a check they were
supposed to pass.
@ -50,7 +50,7 @@ An account created without any group needs re-adding with the flag —
what changes.
Why a group and not a list of usernames: agents are getting authelia
accounts of their own (matrix SSO), and *authenticated* would then
accounts of their own (matrix SSO), and _authenticated_ would then
include every agent in the hive. The group is the only thing standing
between "an operator's page" and "anyone with a session".
@ -68,10 +68,10 @@ Adding a swarm service name means touching two things. Missing the
second ships as a different flavour of "works from the host, broken from
a container":
| site | file |
| --- | --- |
| site | file |
| ---------------------------- | --------------------------------------------------------------- |
| vhost + `gateway.localNames` | the service's own module (e.g. `nix/host-modules/swarm-ui.nix`) |
| **certificate name** | `nix/host-modules/swarm.nix` (`serviceDomains`) |
| **certificate name** | `nix/host-modules/swarm.nix` (`serviceDomains`) |
The DNS record and the local-dev `/etc/hosts` entry need no separate
edit: both are derived from `services.hyperhive.gateway.localNames`,
@ -83,7 +83,7 @@ matrix discovery); a swarm service declares its own vhost next to its
own options, the way `swarm-ui.nix` and `swarm-authelia.nix` do.
⚠️ The certificate one is the least obvious and the most visible when
missed. `serviceDomains` is *both* the services sub-CA's
missed. `serviceDomains` is _both_ the services sub-CA's
`nameConstraints` set and the leaf's SAN list, and the apex is a
**sibling** of `forge.<swarm>` / `chat.<swarm>` / `auth.<swarm>`, not a
parent — no CA in the hierarchy issues for it implicitly. Left out, the

2
docs/tools/README.md
View file

@ -1,6 +1,6 @@
# Tools
`hivectl` is *your* tool — the operator's own host CLI. Everything
`hivectl` is _your_ tool — the operator's own host CLI. Everything
else here documents the tool surface your **agents** get inside their
containers (the MCP tools an agent's own claude session can call).
You never call these directly, but they're the reference for what an

6
docs/tools/bash.md
View file

@ -20,15 +20,15 @@ output. Handle it on a future turn — unless `wait_seconds` already
delivered the terminal result inline, in which case no todo is created
(see `status` below).
* `timeout_secs` — kill the task after N seconds and mark it
- `timeout_secs` — kill the task after N seconds and mark it
`timed_out`. Omit for no timeout (runs until natural exit).
* `wait_seconds` — inline poll before returning (capped at 30).
- `wait_seconds` — inline poll before returning (capped at 30).
When the task finishes within the window the full status is
returned immediately and no todo is created; when the window expires
the task keeps running and the normal `task started: id=<id>`
response is returned. **Defaults to 3** — pass `wait_seconds: 0`
to disable inline waiting and always get the immediate response.
* `name` — optional caller-chosen task id. When set it replaces the
- `name` — optional caller-chosen task id. When set it replaces the
auto-generated hex id, so it surfaces in `status(<name>)` lookups and
the loose-ends list — a memorable label instead of an opaque id. A name
is **reusable once its previous task has finished**; submitting a

6
docs/tools/hivectl.md
View file

@ -172,7 +172,7 @@ resume drains the backlog rather than dropping it. Points worth knowing:
- **Sticky.** The marker lives on the persistent harness mount, so a
paused agent stays paused across a container restart — and pausing a
*stopped* agent makes it come up parked.
_stopped_ agent makes it come up parked.
- **Not a DAG.** Unlike `restart`/`stop`, there's no container operation
to sequence, so it applies immediately with nothing to wait on.
- **Stopping a paused agent is still fast.** The graceful-stop
@ -201,7 +201,7 @@ they go into a systemd drop-in verbatim, and a typo there makes the
unit fail to start.
**Declarative, not incremental**: each invocation replaces the agent's
whole entry. `set-limits sock --memory-max 8G` leaves `sock` with *only*
whole entry. `set-limits sock --memory-max 8G` leaves `sock` with _only_
a memory override, reverting any previously-set CPU quota to the hive
default. To avoid a forgotten flag silently wiping an override, a bare
`set-limits <name>` with no flags is rejected — clearing requires the
@ -236,7 +236,7 @@ Bare `choom` starts a fresh blank session. `--resume <value>` passes
through as `claude --resume <value>` to rejoin a prior session by its
session id — the flag name deliberately matches the claude flag it maps
to. (choom never uses claude's `--continue`: that's a bare flag that
takes no argument and resumes the cwd's *latest* session, i.e. the
takes no argument and resumes the cwd's _latest_ session, i.e. the
harness's; a value after it would be consumed as the first prompt,
silently poking the live harness session.) A value is required when the
flag is given. Either way choom never collides with the harness's live

2
docs/tools/lifecycle.md
View file

@ -74,7 +74,7 @@ agents after the approval resolves.
| `kill` / `start` / `restart` / `update` | No | Direct children |
| `list_containers` | No | All descendants |
| `request_init_config` | Yes (InitConfig) | New direct child only |
| `request_update_meta_inputs` | Yes (MetaUpdate) | Meta flake (global) |
| `request_update_meta_inputs` | Yes (MetaUpdate) | Meta flake (global) |
## See also

65
docs/tools/swarmctl-cli.md
View file

@ -4,12 +4,12 @@ This document contains the help content for the `swarmctl` command-line program.
**Command Overview:**
* [`swarmctl`↴](#swarmctl)
* [`swarmctl user`↴](#swarmctl-user)
* [`swarmctl user add`↴](#swarmctl-user-add)
* [`swarmctl user update`↴](#swarmctl-user-update)
* [`swarmctl user list`↴](#swarmctl-user-list)
* [`swarmctl completions`↴](#swarmctl-completions)
- [`swarmctl`↴](#swarmctl)
- [`swarmctl user`↴](#swarmctl-user)
- [`swarmctl user add`↴](#swarmctl-user-add)
- [`swarmctl user update`↴](#swarmctl-user-update)
- [`swarmctl user list`↴](#swarmctl-user-list)
- [`swarmctl completions`↴](#swarmctl-completions)
## `swarmctl`
@ -19,17 +19,15 @@ swarm-level operator CLI
###### **Subcommands:**
* `user` — Manage subjects in the swarm's SSO provider
* `completions` — Generate a shell completion script for `swarmctl` and print it to stdout
- `user` — Manage subjects in the swarm's SSO provider
- `completions` — Generate a shell completion script for `swarmctl` and print it to stdout
###### **Options:**
* `--authelia-bin <PATH>` — authelia binary used to hash passwords. The argon2 parameters must match the verifier's, so this has to be the *configured* package rather than whatever is on `PATH`
* `--users-file <PATH>` — Host-side path of authelia's users database — i.e. the path inside the container, prefixed with the container's root.
This is the only user store: it is read before every change and written in place, and `swarm-authelia-bridge` writes the same file.
- `--authelia-bin <PATH>` — authelia binary used to hash passwords. The argon2 parameters must match the verifier's, so this has to be the _configured_ package rather than whatever is on `PATH`
- `--users-file <PATH>` — Host-side path of authelia's users database — i.e. the path inside the container, prefixed with the container's root.
This is the only user store: it is read before every change and written in place, and `swarm-authelia-bridge` writes the same file.
## `swarmctl user`
@ -39,11 +37,9 @@ Manage subjects in the swarm's SSO provider
###### **Subcommands:**
* `add` — Add a user, generating a password for them
* `update` — Change an existing user's attributes
* `list` — List every user in authelia's users database
- `add` — Add a user, generating a password for them
- `update` — Change an existing user's attributes
- `list` — List every user in authelia's users database
## `swarmctl user add`
@ -53,15 +49,13 @@ Add a user, generating a password for them
###### **Arguments:**
* `<USERNAME>` — Login name. Conservative ASCII only — it is a YAML map key and reaches access-control rules and logs
- `<USERNAME>` — Login name. Conservative ASCII only — it is a YAML map key and reaches access-control rules and logs
###### **Options:**
* `--display-name <TEXT>` — Name shown in the SSO UI. Defaults to the username
* `--email <ADDRESS>`
* `--group <GROUP>` — Repeatable
- `--display-name <TEXT>` — Name shown in the SSO UI. Defaults to the username
- `--email <ADDRESS>`
- `--group <GROUP>` — Repeatable
## `swarmctl user update`
@ -73,16 +67,14 @@ Every flag is optional and they compose, so one call can set several things at o
###### **Arguments:**
* `<USERNAME>` — Login name of an existing user
- `<USERNAME>` — Login name of an existing user
###### **Options:**
* `--display-name <TEXT>` — Name shown in the SSO UI
* `--email <ADDRESS>`
* `--add-group <GROUP>` — Repeatable. Adding a group the user is already in is not an error
* `--remove-group <GROUP>` — Repeatable. Fails if the user is not in the group — a revocation that reports success without revoking is the failure nobody re-checks
- `--display-name <TEXT>` — Name shown in the SSO UI
- `--email <ADDRESS>`
- `--add-group <GROUP>` — Repeatable. Adding a group the user is already in is not an error
- `--remove-group <GROUP>` — Repeatable. Fails if the user is not in the group — a revocation that reports success without revoking is the failure nobody re-checks
## `swarmctl user list`
@ -92,8 +84,6 @@ Read-only: it never writes the file. Shows every subject in it, including agent
**Usage:** `swarmctl user list`
## `swarmctl completions`
Generate a shell completion script for `swarmctl` and print it to stdout.
@ -106,16 +96,13 @@ Dispatched before `PathArgs::resolve()` for the same reason as `markdown-docs`:
###### **Arguments:**
* `<SHELL>` — Shell to emit completions for
- `<SHELL>` — Shell to emit completions for
Possible values: `bash`, `elvish`, `fish`, `powershell`, `zsh`
<hr/>
<small><i>
This document was generated automatically by
<a href="https://crates.io/crates/clap-markdown"><code>clap-markdown</code></a>.
This document was generated automatically by
<a href="https://crates.io/crates/clap-markdown"><code>clap-markdown</code></a>.
</i></small>

18
docs/trust-boundary/boundary.md
View file

@ -99,11 +99,11 @@ exact same path, so the group grant always holds.
`/run/hive-agent/<name>/` is shared by **three principals that share no
group**, which is why its mode is what it is:
| principal | reaches | needs |
|---|---|---|
| principal | reaches | needs |
| ------------------- | ---------------------------------------- | ------------ |
| the agent's harness | binds + unlinks `agent.sock`, `web.sock` | owner, `rwx` |
| `hive-c0re` | dials `agent.sock` (todo wakes) | traverse |
| the gateway's nginx | dials `web.sock` | traverse |
| `hive-c0re` | dials `agent.sock` (todo wakes) | traverse |
| the gateway's nginx | dials `web.sock` | traverse |
The last two land in "other", so the dir is **`0751`, owned by the
agent's container uid/gid** — `o=--x` is traverse without listing, and
@ -111,13 +111,13 @@ both sockets are `0666`, which is all a dialer needs.
**Ownership is declared, not repaired.** The tmpfiles.d entry written by
`SyncAgentTmpfiles` names the uid/gid directly. Do not add a chown
alongside it: `d` re-applies on every boot *and* every agent
alongside it: `d` re-applies on every boot _and_ every agent
spawn/destroy, so ownership set afterwards is reverted the next time any
agent changes — which is exactly how this dir spent a long time at
`0777 root root` while a privileged chown appeared to be fixing it.
The mode is load-bearing, not cosmetic. Write permission on a
*directory* is what confers the right to unlink its entries, whoever owns
_directory_ is what confers the right to unlink its entries, whoever owns
them, and the sticky bit is the only thing that would restrain that (it
is not set here). A world-writable socket dir therefore lets anything
able to reach the path delete an agent's socket and bind its own — and
@ -129,11 +129,11 @@ nginx reaches all of `/run/hive-agent` as a plain host path. Dropping
them and the rest of the host. That costs no network isolation: nginx
binds the host's `:80`/`:443` and reaches `localhost` upstreams, which a
netns would have to be opened up for anyway.
🔑 It does mean nothing *implicitly* scopes the privileged reload verb —
🔑 It does mean nothing _implicitly_ scopes the privileged reload verb —
see [`docs/trust-boundary/security.md`](security.md#hive-c0re-privilege-separation) for
how `PrivRequest::ReloadGatewayNginx`'s containment works.
⚠️ Contrast `/shared`, which *is* sticky world-writable (`1777`): it has
⚠️ Contrast `/shared`, which _is_ sticky world-writable (`1777`): it has
many legitimate writers, so sticky is the best available answer there.
This dir has exactly one writer, so it needs no world write at all.
@ -141,7 +141,7 @@ This dir has exactly one writer, so it needs no world write at all.
`hivectl` drives the whole hive — spawn / kill / destroy / rebuild /
deploy — over the **host admin socket** `/run/hyperhive/host.sock`,
socket-activated by the `hive-c0re.socket` unit. That socket *is* the
socket-activated by the `hive-c0re.socket` unit. That socket _is_ the
full-control surface, so who can connect to it is a real trust
boundary.

32
docs/trust-boundary/security.md
View file

@ -207,22 +207,22 @@ for every operation that genuinely requires root.
**Narrow interface** — `PrivRequest` variants map 1:1 to specific
known operations; there is no arbitrary command pass-through:
| Operation | What it runs |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
| `CreateContainer` / `UpdateContainer` | `nixos-container create/update <name> --flake <ref>` |
| `DestroyContainer` | `nixos-container destroy <name>` |
| `ListContainers` | `nixos-container list` |
| `ReadContainerJournal` | `journalctl -M <container> -n <n> [filters...]` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
| `DaemonReload` | `systemctl daemon-reload` |
| `RunForgeAdmin` | `nixos-container run hive-forge -- runuser -u forgejo -- forgejo admin <args>` |
| `WriteAgentForgeToken` / `WriteAgentMatrixToken` | write `0600` credential file into agent state dir |
| `RestartMatrixDaemon` | `systemctl --machine=h-<name> restart hive-matrix-daemon.service` |
| Operation | What it runs |
| ------------------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
| `CreateContainer` / `UpdateContainer` | `nixos-container create/update <name> --flake <ref>` |
| `DestroyContainer` | `nixos-container destroy <name>` |
| `ListContainers` | `nixos-container list` |
| `ReadContainerJournal` | `journalctl -M <container> -n <n> [filters...]` |
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
| `DaemonReload` | `systemctl daemon-reload` |
| `RunForgeAdmin` | `nixos-container run hive-forge -- runuser -u forgejo -- forgejo admin <args>` |
| `WriteAgentForgeToken` / `WriteAgentMatrixToken` | write `0600` credential file into agent state dir |
| `RestartMatrixDaemon` | `systemctl --machine=h-<name> restart hive-matrix-daemon.service` |
**Container allowlist** — every request is validated against an
allowlist before any operation: only names matching the agent-name

29
docs/turn-loop/claude-invocation.md
View file

@ -73,7 +73,7 @@ percentage-of-window ctx badge.
harness-owned session title (`turn::session_title()`, default
`hive-session`, override `HIVE_SESSION_TITLE`). The durable
`hive_claude::InfiniteSession` (built once by the serve loop via
`turn::make_session`, then reused) `--resume <title>`s it; the *first* use
`turn::make_session`, then reused) `--resume <title>`s it; the _first_ use
(bootstrap, post-archive, post-purge) misses and the session re-runs the
same prompt once with `--name <title>` to mint it. That single self-heal
rule is the whole identity system — there is **no** scraped session-id
@ -85,7 +85,7 @@ context (it won't carry our title). claude stores sessions in
`~/.claude/projects/<cwd-slug>/<uuid>.jsonl` (bind-mounted persistently);
`--name` writes the title into the file as a `custom-title` event, which
is what `--resume <title>` resolves against. We never pass bare
`--continue` (it resumes the *latest* session in the cwd — the hijack
`--continue` (it resumes the _latest_ session in the cwd — the hijack
vector). Auto-compact, auto-memory, and dynamic workflows (the `/workflows`
feature) are disabled via the managed settings at
`/etc/claude-code/managed-settings.json`: hyperhive owns compaction
@ -94,12 +94,12 @@ feature) are disabled via the managed settings at
harness's autonomous turns.
**Session reset** is available via `POST /api/new-session` (or
`/new-session` slash command). It does *not* touch the session inline —
`/new-session` slash command). It does _not_ touch the session inline —
that would race a mid-write claude process. Instead `Bus::request_session_reset()`
sets a one-shot flag consumed at the next turn boundary by `drive_turn`,
which **archives** the current session: the backing `<uuid>.jsonl` is
renamed to `<uuid>.jsonl.archived` (dropped out of claude's `*.jsonl`
resolution glob, history preserved on disk, only the file carrying *our*
resolution glob, history preserved on disk, only the file carrying _our_
title — any `choom` session sharing the cwd is left alone). The next
turn's `--resume <title>` then misses and self-heals into a fresh session.
@ -111,19 +111,19 @@ at `/etc/claude-code/managed-settings.json`); hyperhive owns it. The
window with two triggers baked into its `run`:
- **Reactive** — claude-code prints `Prompt is too long`. The session is
*already* past the window, so no turn can run on it — the session
_already_ past the window, so no turn can run on it — the session
`/compact`s straight away and retries the same wake-up prompt once. No
notes-checkpoint turn is possible here: the detail is gone. If the retry
*still* overflows, `run` surfaces `Error::PromptTooLong`; `drive_turn`
_still_ overflows, `run` surfaces `Error::PromptTooLong`; `drive_turn`
then archives the session (session lifecycle stays hive-side) and the
serve loop requeues the message so it redelivers into a fresh session
(see [Turn outcomes](README.md#turn-outcomes) — the wake prompt itself is tiny, so
the overflow was the accumulated context the archive clears).
- **Proactive** — a turn finishes cleanly but the last inference's context
size crossed the policy watermark. While the session is still healthy it
runs one synthetic *notes-checkpoint* turn (`CHECKPOINT_PROMPT` —
runs one synthetic _notes-checkpoint_ turn (`CHECKPOINT_PROMPT` —
"context is filling up, flush durable state into `/state` now") and
*then* `/compact`s, so the agent can persist in-flight state before the
_then_ `/compact`s, so the agent can persist in-flight state before the
detail collapses into a summary.
The **when** is a `hive_claude::CompactionPolicy` injected by the harness:
@ -166,7 +166,7 @@ still applies.
session (same mechanism as the operator reset — rename `<uuid>.jsonl` →
`.archived`) so the next turn's `--resume <title>` misses and starts
fresh. Unlike proactive compaction the session is dropped entirely, not
compacted — and *no* preceding checkpoint turn runs, because any turn
compacted — and _no_ preceding checkpoint turn runs, because any turn
before the reset would just re-warm the cache and defeat the purpose.
Set `HIVE_AUTO_RESET_WATERMARK_TOKENS=0` to disable. Auto-reset and the
operator reset are mutually exclusive per turn (both archive → fresh
@ -236,11 +236,11 @@ needs to `chown` a bind mount), once at startup:
**`hive_identity` / `swarm_identity` shape.** Each carries a
leading space + backticked name (` on hive \`pr1ma\``,
` in swarm \`constellat1on\``) when the corresponding env var
is set, otherwise empty string. The independence lets the
template drop one or both into the opener prose without
breaking single-hive deployments that never set the option;
the renderer also treats `Some("")` from a caller as `None` so
` in swarm \`constellat1on\``) when the corresponding env var
is set, otherwise empty string. The independence lets the
template drop one or both into the opener prose without
breaking single-hive deployments that never set the option;
the renderer also treats `Some("")`from a caller as`None` so
empty-string env vars and missing env vars round-trip the
same way.
@ -251,4 +251,3 @@ telemetry-to-bus bridge — lives in `hive-agent`'s `turn` module; see its
The actual claude spawn, stream classification, and the
reactive/proactive compaction loop are in the `hive-claude` crate.
Login-wait lives in `hive-agent`'s `login` module.

14
docs/turn-loop/config.md
View file

@ -75,11 +75,11 @@ Declares extra navigation links that appear in the per-agent page
header alongside the built-in forge / config / container links. Each
entry has:
| Field | Required | Description |
|-------|----------|-------------|
| `label` | yes | Display text shown in the icon strip tooltip and meta-nav. |
| `url` | yes | Absolute URL — may include a different port (the dashboard renders it as a plain anchor). |
| `icon` | no | Emoji or short glyph prefix. Defaults to empty string. |
| Field | Required | Description |
| ------- | -------- | ----------------------------------------------------------------------------------------- |
| `label` | yes | Display text shown in the icon strip tooltip and meta-nav. |
| `url` | yes | Absolute URL — may include a different port (the dashboard renders it as a plain anchor). |
| `icon` | no | Emoji or short glyph prefix. Defaults to empty string. |
The list is written to `<state>/hyperhive-dashboard-links.json` by a
one-shot systemd unit at container boot. The harness's own web UI
@ -213,6 +213,7 @@ hyperhive.claudePluginsAutoUpdate = false; # default
> to list `skill-creator@claude-plugins-official` and `base@hyperhive`
> explicitly alongside its own entries — likewise for the two default
> entries in `claudeMarketplaces`.
- **`claudePluginsAutoUpdate`** — when `true`, runs
`claude plugin marketplace update` before installing plugins to pull
the latest index. Disabled by default to keep boot times short and
@ -265,7 +266,7 @@ is a no-op without the other:
(outside the nix store, one `KEY=value` per line, systemd
`EnvironmentFile` syntax) supplying the credentials `claude` itself
reads from the environment — typically `ANTHROPIC_API_KEY` and
`ANTHROPIC_BASE_URL`. Loaded as an *optional* `EnvironmentFile`
`ANTHROPIC_BASE_URL`. Loaded as an _optional_ `EnvironmentFile`
(leading `-`), so setting the option before the file exists doesn't
strand the harness at boot.
@ -299,4 +300,3 @@ Switching an already-provisioned OAuth agent to `useApiKey` leaves
cleaned up automatically. Cost shape also changes: subscription pricing
→ per-request billing with no built-in monthly cap, worth knowing before
pointing a busy agent at a metered backend.

3
docs/turn-loop/mcp.md
View file

@ -106,7 +106,7 @@ already in flight.
`{state_dir}/hyperhive-status`. Pass `""` to clear.
- `get_agent_meta` — fetch identity + status metadata for an agent:
`{ name, hyperhive_rev, running, status_text, status_set_at,
hive_name?, swarm_name?, matrix_accounts? }`. `matrix_accounts` is a
hive_name?, swarm_name?, matrix_accounts? }`. `matrix_accounts` is a
list of matrix identities the agent can act as (`name`, `user_id?`,
`homeserver`); omitted for agents with no matrix provisioning. Omit
`name` to query self.
@ -229,4 +229,3 @@ built-in shell there stays within the existing trust boundary. The bash MCP serv
`WebFetch` / `WebSearch` are off by default; enable the `web_tools`
tool group in the P3RM1SS10NS tab and rebuild the agent to enable them.

2
docs/web-ui.md
View file

@ -9,7 +9,7 @@ Both are SPAs — `GET /` returns a static shell, `/api/state`
returns JSON, JS renders. No full-page reloads.
This doc has been split for readability. Pick the section you need —
or, if you're the operator looking for how to actually *use* the
or, if you're the operator looking for how to actually _use_ the
dashboard rather than how it's built, start at
**[web-ui/README.md](web-ui/README.md)** instead.

4
docs/web-ui/README.md
View file

@ -18,7 +18,7 @@ of your time. It's a single page with exactly four tabs:
- **SW4RM** — every agent, live. This is the default tab and the one
you'll check most.
- **Y3R C4LL** — anything waiting on *you*: pending approvals. If an
- **Y3R C4LL** — anything waiting on _you_: pending approvals. If an
agent needs a decision from you, it's here.
- **P3RM1SS10NS** — what tools and system-level access each agent has.
- **SCH3DUL3S** — scheduled prompts and agent self-reminders.
@ -47,7 +47,7 @@ and quick links (stats, screen, forge profile). Click the name to open
its terminal and watch it work in real time.
**Approve something an agent is waiting on.** Y3R C4LL is the one tab
worth checking regularly — it's everything that needs *you*: approvals
worth checking regularly — it's everything that needs _you_: approvals
for config changes. The tab's count pill tells you at a glance if
anything's pending.

10
docs/web-ui/design-guide.md
View file

@ -2,7 +2,7 @@
Scope: **swarm-ui only** (the swarm-level Preact app — not the per-hive
dashboard, which has its own older visual language). This doc is the
*why*: the principles behind how swarm-ui looks and behaves, and the
_why_: the principles behind how swarm-ui looks and behaves, and the
concrete rules that follow from them. It deliberately doesn't show what
things look like — that's `ComponentsPage` (`/components`), the living,
always-current demo of every primitive in `src/ui/`. Code can't go stale
@ -21,14 +21,14 @@ repeated here.
distinction, not decoration for its own sake.
- **Whimsy** — small, delightful touches are welcome (the per-hive
dashboard's home-page matrix-rain background, `packages/dashboard/src/
home.js`, is the reference example — currently in the dashboard
home.js`, is the reference example — currently in the dashboard
package, not swarm-ui itself, but the pattern it sets applies here
too). Whimsy still has to clear the accessibility bar below (motion,
in particular). swarm-ui's own reference example: `Panel`'s optional
`icon` prop (`src/ui/panel/Panel.tsx`), a small emoji glyph in a
panel's header, chosen per panel with no default — grew out of a
one-off emoji dropped into a single page's copy, which wasn't whimsy
in this sense (a *consistent*, reusable touch) until it became a real
in this sense (a _consistent_, reusable touch) until it became a real
prop every panel can opt into. `aria-hidden`, since it's decorative —
the title text is still the actual label.
- **Efficient navigation** — minimize clicks/hops for a common task.
@ -62,7 +62,7 @@ smooth them instead.
The mechanical contract (base16 slots, semantic vars, what a page's CSS
is and isn't allowed to reference) lives in `docs/web-ui/css-vars.md` —
read that for the how. This section is the *policy* layered on top:
read that for the how. This section is the _policy_ layered on top:
- **User-theming compatible by construction.** The whole point of the
base16/`colors.css` swap contract is that a user's own theme (stylix
@ -132,7 +132,7 @@ just bare column headers floating over nothing.
touch target (`2.75em` ≈ 44px, WCAG 2.5.5) by default, so this isn't
something each page has to remember.
- **Phone is a second-class citizen, not an unsupported one.** Things
must not *break* at phone width, but don't over-invest in phone
must not _break_ at phone width, but don't over-invest in phone
optimization beyond that. swarm-ui is installable as a PWA, so a
phone can check status or do basic interactions.
- **A narrow viewport isn't only a phone** — a tiling-window-manager