Three dashboards over metrics already in VictoriaMetrics: the swarm
controller's job graph and forge push/commit counters, the forge
process's runtime, and the OTEL collectors' self-telemetry.
The three delivery units made /run/swarm-grafana-secrets only after
`bao login` and `bao kv get`, so with the store sealed or unreachable the
bind source was never created by them and a switch could restart the
container without it. Each script now starts with the install, from one
shared `ensureSecretDir`, and the tmpfiles comment says what is true.
The module-eval case asserts the install precedes `bao login` in all three
scripts. docs/swarm/sso.md states that after a reboot Grafana stays down
until the secret store answers.
Part of #4998 (slice 2b: grafana).
The three delivery units still read from bao under Grafana's own leaf, but
now write to /run/swarm-grafana-secrets on the host (tmpfs), which the
container bind-mounts read-only at the same path. Before, they wrote to
/var/lib/nixos-containers/swarm-grafana/var/lib/{grafana-oidc,grafana-admin,
grafana-metrics}/ on persistent disk, where a superseded value stayed after
rotation and went into backups of the container root. Host tmpfiles now
removes the three old files.
Rotation is unchanged: refresh_consumer restarts Grafana (or starts it if it
failed) when a file changes, since Grafana reads each $__file{} once as it
starts. After a reboot the files are fetched again; until the store answers
Grafana stays down, where before it came up on the last copy from disk.
Part of #4998 (slice 2b).
The swarm UI has no switch of its own since it runs wherever the
controller runs; its package option follows it under
deploy.swarm-controller.ui. deploy.swarm-ui.package and
swarm.ui.package stay as renames and warn at evaluation. The flake's
default, the vhost root and the module-eval stub use the new path.
A spawn into a missing cwd fails with a bare ENOENT; a daemon newer than its
container's bind now inherits its own cwd again, as before the work-dir default.
The system prompt now names the work dir for worktrees, clones and build
trees, and keeps state/ for notes. A subagent started without a dir runs
in the agent's work dir instead of the daemon's own cwd.
swarm-bao-matrix-oidc still reads the secret from bao under matrix-ctl's
leaf, but now writes it to /run/hive-matrix-oidc on the host (tmpfs), which
the container bind-mounts read-only for tuwunel's LoadCredential. Before, it
wrote it to
/var/lib/nixos-containers/hive-matrix/var/lib/tuwunel-oidc/<id>.secret on
persistent disk, where a superseded secret stayed after rotation and went
into backups of the container root. Host tmpfiles now removes that old file.
clientSecretFile must now lie under /run/hive-matrix-oidc/, the one directory
the container shares; any other value fails at evaluation.
Rotation is unchanged: the ten-minute timer rewrites the file when the value
differs and refresh_consumer restarts tuwunel. After a reboot the secret is
fetched again; until the store answers, tuwunel does not start.
Part of #4998 (slice 4: tuwunel).
swarm-bao-forwarder-oidc still reads the secret from bao under the
forwarder's own leaf, but now writes it to /run/swarm-bao-forwarder-oidc on
the host (tmpfs), which the swarm-bao container bind-mounts read-only for the
collector's LoadCredential. Before, it wrote it to
/var/lib/nixos-containers/swarm-bao/var/lib/swarm-bao-otel-oidc/<id>.secret on
persistent disk, where a superseded secret stayed after rotation and went
into backups of the container root. Host tmpfiles now removes that old file.
Rotation is unchanged: the ten-minute timer rewrites the file when the value
differs and refresh_consumer restarts the forwarder. After a reboot the
secret is fetched again; the file is not pre-created, so until the store
answers the forwarder stays down (LoadCredential refuses a missing source) and
resumes from its journal cursor afterwards. The container now starts after
systemd-tmpfiles-resetup, since the new bind source is made there.
Part of #4998 (slice 3: forwarder).
swarm-bao-otel-oidc still reads the secret from bao under the collector's
own leaf, but now writes it to /run/swarm-otel-oidc on the host (tmpfs),
which the container bind-mounts read-only for the collector's
LoadCredential. Before, it wrote it to
/var/lib/nixos-containers/swarm-otel/var/lib/swarm-otel-oidc/<id>.secret on
persistent disk, where a superseded secret stayed after rotation and went
into backups of the container root. Host tmpfiles now removes that old file.
Rotation is unchanged: the ten-minute timer rewrites the file when the value
differs and refresh_consumer restarts the collector. After a reboot the
secret is fetched again; until the store answers the collector starts on an
empty file and the stores refuse its pushes and scrapes.
Part of #4998 (slice 2a).
swarm-bao-ci-runner still reads the token from bao under the runner's own
ci-runner leaf, but now writes it to /run/swarm-ci-runner on the host
(tmpfs), which the container bind-mounts read-only for the register
unit's LoadCredential. Before, it wrote the token to
/var/lib/nixos-containers/swarm-ci-runner/var/lib/swarm-ci-runner/registration-token
on persistent disk, where a superseded token stayed after rotation and
went into backups of the container root. The container's tmpfiles now
remove that old file.
Rotation is unchanged: the ten-minute timer rewrites the file when the
value differs and refresh_consumer restarts the runner. After a reboot
the token is fetched again; until the store answers, a runner that is
already registered keeps running and only a new registration waits.
Part of #4998 (slice 1).
The dashboard, its API, the swagger UI and the agent routes move to their
own vhost, `swarm.dashboardDomain` = `<hive>.hives.<swarm.domain>`, under
the swarm domain so authelia's session cookie reaches it. `/`, `/api/` and
`/api/docs/` ask authelia through `auth_request`, and authelia's
`operatorsOnly` rule for each hive's dashboard name lets only the admins
group in. The hive domain keeps matrix discovery and redirects every other
path to the new name.
The name rides as a SAN on the hive's own gateway leaf: the per-hive bao
role allows it, and hive-tls asks for it and checks coverage.
HTTP Basic auth is gone: the htpasswd writer and its test, the
`hivectl gateway` verbs and host-socket requests, the `gateway.auth`
options (now a removed-option error), the unauthorized page and the bcrypt
dependency. tmpfiles deletes a leftover gateway.htpasswd, which held
bcrypt hashes at 0644.
c0re hands out dashboard and agent URLs on the new name
(HYPERHIVE_DASHBOARD_DOMAIN), and the swarm controller's hive roster
carries each hive's dashboard name for the swarm UI's hive list.
Closes#3469
swarm-bao-forge-oidc fetched the forge's OIDC client secret from bao and wrote
it to /var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/<id>.secret,
on the container's persistent root, where it outlived a bao rotation and was
in every backup of that tree.
Now the fetch writes it to /run/hive-forge-oidc/<id>.secret on the host
(tmpfs, root 0400). The directory is bound read-only into the container at
the same path, and forgejo-sso-source loads it through LoadCredential. A
tmpfiles rule creates the directory at boot, so the bind source always
exists, and removes the old persistent copy.
After a reboot the file is absent until the fetch succeeds, and
forgejo-sso-source fails to start until then; the fetch starts it again once
it writes the file. Logins keep working meanwhile: forgejo holds the secret
in its login-source row.
deploy.forgejo.sso.clientSecretFile (and its old swarm.forge.sso name) is
removed: it only chose where in the container the fetched file landed. The
fetch no longer needs the container's forgejo uid, so it no longer fails on a
fresh host before the container has created its state dir.
Part of #4998.
The shell had two body widths, a 60em readable column and a 90em wide body
that a short list of table routes opted into. Every route now gets the
90em body, and the opt-in route list, its modifier class and its test go
away.
Until its first successful read of its key, the harness read its
permissions as the default set, `execution` included. So did the
subagent daemon. An agent the operator had narrowed ran wider than
granted after every restart while the queue was unreachable, and in the
gap before the first 10 s poll. During the meta-repo import, every
non-ruth key is still unset, and the agent read that as the default too,
for as long as the import stayed pending.
mara picked option (a) on #5293: "a, removal of the old path as a follow
up in 1w".
hive-c0re now writes each agent's entry from
swarm-agent-permissions.json into the agent's runtime dir as
swarm-permissions.json (/run/hive in the container), together with
whether the import is still pending. It writes the file at every start
(write_dropins) and after every 30 s pass of the permissions worker. The
harness and the subagent daemon hold that stored list until their own
first read succeeds. While the import is pending, an unset key keeps
the stored list rather than reverting it to the default. A queue that
is down never blocks a start: the agent runs on the stored grant.
The decision lives in swarm_queue_client::agent_permissions::HiveCopy,
shared by both readers.
Part of #5170.
Declaring an agent Destroyed now deletes its `agent-permissions` key,
next to the queue-credential revoke and with the same failure handling:
logged at error, the destroy goes ahead, and declaring it destroyed
again re-runs it. Agent names are reusable after a destroy, and a create
left on the default stores no key, so the old key handed a new agent of
the same name the destroyed agent's grant (argus on #5293).
Also: migrate.rs's create-time-setup note named ruth's tool-group seed
in `ensure_root_agent`, which this PR deletes; it now says she holds the
code-constant default when the swarm sets nothing.
Part of #5170.
mara on #5170: "dont split caps and perms. its all perms. / dont include
the now empty tool groups / nothing set -> default / also add perms to
agent creation screen".
- hive-sh4re: one `Permission` enum (messaging, meta, inbox, execution,
web_tools, manage_root_agent) replaces `ToolGroup` and `Capability`;
`lifecycle`/`forge` are gone. `Permission::default_for` is the default
for an agent the swarm sets nothing for.
- The record is `{ "permissions": [...] }`. Missing key, unset field or
no successful read yet = default; `[]` = none; a failed read after a
good one keeps the last value.
- hive-agent reads its own key every 10s and builds each turn's --tools,
--allowedTools and MCP config (rewritten before every turn) from it;
the subagent daemon reads it on each start/continue. Agents get a
read-only grant on their own key.
- hive-c0re restarts an agent only when its manage_root_agent changes.
The import now merges meta/capabilities.json and meta/tool-groups.json
into one list per agent and deletes both; the "import pending, don't
read" decision is `ImportOutcome::read_allowed`, table-tested.
- swarm-controller: `GET /api/permissions` (names + default) and an
optional `permissions` on `POST /api/agents`, stored before the job is
queued. swarm-ui's create-agent form gets a permissions field.
- Deleted: HIVE_TOOL_GROUPS and its forward to the subagent daemon,
tool_groups.rs, ruth's tool-group seed, the permissions rebuild job
(PermPayload/WritePermFile/perm_change), the dashboard permissions tab
and its stale-entries panel.
Part of #5170.
An agent's capabilities move from the hive's meta repo
(`meta/capabilities.json`, injected as `HIVE_CAPABILITIES`) into the
`capabilities` field of its key in a new `agent-permissions` KV bucket,
which swarm-controller alone writes.
- swarm-queue-client: `agent_permissions` (bucket, key, `AgentPermissions`,
direct-get read, the import subject and its wire types).
- swarm-controller: creates the bucket at start; `GET|PUT|DELETE
/api/agents/{name}/permissions` (deny_unknown_fields and unknown
capability ⇒ 422, duplicate ⇒ 400, agent must be declared by a hive);
answers each hive's one-shot import, writing a key only where there is
none and only for agents that hive's wanted state declares.
- swarmctl: `agent permissions get|set|clear`.
- swarm-nats-auth: a hive reads every key of the bucket and sends its
import on its own subject; the controller subscribes to the imports and
may answer them (`responses`).
- hive-c0re: reads each of its agents' keys every 30s into
/var/lib/hyperhive/swarm-agent-permissions.json and restarts an agent,
gracefully, when its capabilities change. Unset is the compiled-in
default (ruth: manage_root_agent; everyone else: none). A failed read
keeps the stored value. On first boot it seeds that file from
meta/capabilities.json, sends the grants to the controller, and deletes
the file once the controller answers.
Deleted: the capabilities.json reader/writer, `HIVE_CAPABILITIES` in the
rendered meta flake, ruth's capability seed, the capabilities half of the
permissions rebuild job (`PermPayload::Capabilities`/`Combined`), the
dashboard's capabilities table and `capabilities_changed` event, and the
now-unused `Capability::description`.
Part of #5170.
The crate README rewrite was unrolled at eval time: one sed block per
crate, each carrying one -e expression per other crate. The installPhase
string grew with the square of the crate count, and at 40 crates
(d5de1e494 added swarm-claude-login) it reached 131708 bytes. Linux
caps a single environment variable at 128 KiB, so exec of the builder
failed with "Argument list too long" and every consumer of
reference-docs failed with it, the website repo included.
The rewrite is now a shell loop over $members at build time, building
the same sed expressions per crate. The phase text no longer depends on
the crate count. Output is byte-identical to the old form at 6aab2ebb6
(39 crates, the last commit where it built).
Closes#5321
swarm-bao-grafana-controller-token logs in to the store with Grafana's
leaf, like its three Grafana siblings, and glue-bao-readers-store-order.nix
already orders it after the store's container and PKI unit. The
services-switch case that pins that ordering did not list it, so dropping
it from the glue would have passed. It is now in the roster, and the count
of readers rendered off the store's host goes from 7 to 8: the unit exists
wherever Grafana does.
credentials.md gets the row for its store path, beside the controller's
forge token.
swarm-bao-grafana-controller-token, on the host that runs Grafana, logs in
to Grafana's API over its socket as the built-in admin (the password
swarm-bao-grafana-admin minted, #5138) and keeps a `swarm-controller`
service account that is an enabled Viewer. It mints a token for that
account and writes it to bao at
swarm/controller/swarm-controller/grafana/token under Grafana's leaf. A
ten-minute timer runs it again. The unit keeps a stored token Grafana
accepts. If Grafana refuses the stored token with 401, or bao holds
none, the unit deletes every token of the account and mints a new one.
Grafana's grant gains create/read/update on that leaf, and the
controller's gains read.
swarm-controller serves GET /api/alerts from Grafana's
/api/prometheus/grafana/api/v1/alerts. It reads the token from bao,
keeps it in memory, and re-reads bao once on a 401. The HTTP client has
5s connect and 15s request timeouts. Any failure is a 503 that names its
cause. swarm-ui gets an /alerts page listing them, built the same way as
the hives page.
This is option (c) on #5129, the operator's choice.
A store answers the revision it wrote and what it replaced, and restoring
compares against that revision, so a write that landed after the creation's
own is kept instead of overwritten. A write that changed nothing rolls back
nothing, and a key that held no value is removed rather than set to an empty
one. A test fails the second store after the first and checks the earlier
caps come back.
create_agent puts back the caps and inference settings it replaced if a
later store or the job insert fails, so a refused creation leaves nothing
behind for the name. The shared field components take an id prefix so the
create dialog and an open editor never share a DOM id. The queue check now
follows the hive roster check, the 422 is documented, and the copy says
empty rather than closed and that destroy keeps stored settings.
POST /api/agents takes optional limits and inference objects, checked by
the same rules as the two PUT routes before anything is queued and stored
ahead of the job graph. The create form gets collapsed sections for both,
reusing the fields of the detail-panel editors.
The three issue-report handlers awaited the forge bare, so a forge that
accepts a connection and never answers hung the request. Wrap each in
within_forge_deadline, as pulls.rs does, and answer 503 on overrun.
The minter parses `Request.agent` with `Ident::parse`, as the controller
does, before it makes a scratch directory, runs `claude auth login` or
touches the store, and answers `Failed` otherwise. The name becomes a
directory name and a store path in the one service that writes every
agent's credentials, so it is checked at this service's own boundary.
The doc states that the `operatorsOnly` authelia rule, not the
controller, keeps the routes to operators, and that a stale version record
waits for a relogin. The env-strip comment says it is defence in depth.
Refs #5268
A new swarm service, `swarm-claude-login`, in its own container on the host
that sets `deploy.claude-login.enable` (defaulted on by `allSwarmServices`
where the host has the minter's store leaf, like the review classifier).
- swarm-ui: an agent card showing "login needed" gets a "log in" button.
Its dialog starts the login, shows the URL claude printed, and takes the
pasted code.
- swarm-controller: `POST/DELETE /api/agents/{name}/claude-login` and
`POST /api/agents/{name}/claude-login/code`, behind the swarm UI's
authelia-gated `/api/`, relay the steps to the minter as requests on
`$SWARM.claude-login.request`.
- swarm-claude-login: runs `claude auth login` over piped stdio with HOME
and CLAUDE_CONFIG_DIR in a per-login scratch dir, scrapes the first
https URL, writes the code to stdin, and stores the `.credentials.json`
at `swarm/agents/<agent>/claude-credentials` with a KV v2 write, a new
version over any earlier one. The scratch dir goes on every exit path; a
login waits 15 minutes for its code.
- hive-agent: while parked in `wait_for_login`, reads that leaf every 30 s
under the agent's own cert and installs it only when its KV version is
newer than the one recorded in `<harness>/claude-credentials-version`.
An equal or older version is never put back, because claude rotates the
refresh token and the stored copy is stale after the first refresh.
- swarm-secret-client: `claude_login` paths and `SecretStore::read_versioned`.
- Credentials: a `claude-login` store leaf minted by glue-bao-tls, a
`swarm-claude-login` policy (create/update on
`swarm/agents/+/claude-credentials`, read on its own queue client
secret), an authelia machine client, and a `--claude-login-client`
grant in the queue's auth-callout responder. The unit runs as a static
system user.
The hive's own web-UI login is unchanged.
Refs #5268
Swarm::http() built a reqwest client with no timeout. A token refresh runs
under the exclusive flock on the login cache (kept: authelia rotates the
refresh token, and a second refresher presenting the used one revokes the
login), so an authelia that accepted the refresh and never answered held
that lock forever and every other `swarmctl --swarm` invocation blocked
behind it in flock.
The client now has a 5 s connect timeout and a 30 s total timeout, the
pair swarm-controller's own clients of swarm services use. No caller
streams a response (every gateway call and every login POST reads one
whole body), so a total bound fits all of them. The lock is unchanged; it
is now held at most for one bounded request.
Closes#5308
The wide diff's context drops from 200 to 25 lines, so a block's
enclosing gate stays in view when it opens within 25 lines above the
change (the forge's own diff carries 3). The fallback note, the doc and
the gate fixture (now 20 lines out, not 47) follow.
A failure reading the changed files (a forge error on a tree or blob, a
truncated tree, a non-base64 blob) or joining the diff task made the PR
read fail, so the review posted no verdict. It now falls back to the
forge's 3-line .diff with "reading the files failed: …" as the note.
render wrote each file's whole diff before wide compared it with
MAX_BYTES. It now writes through a writer that refuses once past the cap,
which stops similar's formatter mid-file, so one huge file costs about
MAX_BYTES of output, not its full diff.
The classifier read the forge's PR `.diff`, whose context is fixed at 3
lines. A block moved into a glue module lost its enclosing gate (a
`config = lib.mkIf …` 13 to 191 lines up in the misfired moves #5199,
#5200, #5201), so the glue restating that gate read as a new condition.
The controller now builds the diff itself: it lists the PR's changed files,
reads each at the merge base and at the head through the git tree and blob
API with the token it already holds, and diffs them with 200 lines of
context. It sends the forge's `.diff` instead when the PR changes more than
100 files, the forge names no merge base, or the wide diff is over 256 KiB,
and the posted review then says so with the reason.
Fixes#5227
The granting units (granterPolicyUnit's 17, swarm-bao-granter-role,
swarm-bao-controller-policy, swarm-bao-agent-pki) and swarm-bao-nats-tls
copy store-retry's restart budget by hand, are Type=oneshot, and set no
TimeoutStartSec, so systemd gives each attempt no start timeout. A store
that accepts a connection and never answers leaves the attempt running
forever: Restart=on-failure never fires, units ordered after it wait, and
swarm-bao-nats-tls's daily timer starts a unit that is still activating.
Each now sets TimeoutStartSec: 60 for the granting units and the
granter's role, 2min for controller-policy (it may generate the services
root), 5min for agent-pki and nats-tls (each may wait on a blocking
restart of a server in a container). module-eval-bao-grants and
module-eval-nats-tls assert the bound is finite.
The restart budget itself is unchanged; that is #5305.
Closes#5306
swarm-services-cert, swarm-services-cert-renew and hive-tls-resign are
oneshots, which systemd gives no start timeout, and the root-only
`ca/pem` curl set no connect or total timeout. A store that accepted the
connection and never answered held the attempt forever: no failure, so
no store-retry restart, and a hung renew unit swallowed every later
timer tick.
The curl now gives up after 10s to connect and 30s overall. Each unit
gets a TimeoutStartSec: 20min for the services-leaf pair (the script
waits up to 10min on its lock), 10min for the gateway leaf. The
`--no-block` comment no longer cites a TimeoutStartSec that did not
exist.
module-eval-hive-tls gains a case pinning both.
Closes#5303
The metrics vhost forwarded the store's read API (query, series, labels,
export, vmui, ...) with no credential, so anything that could reach
metrics.<swarm> could read every hive's and agent's metrics. mara ruled on
checked by an auth_request responder, and operators use their authelia
session.
- swarm-metrics-auth (new crate): auth_request responder on the metrics
store's host. Compares the presented bearer, in constant time, with the
one stored at swarm/services/<grafana client id>/metrics/bearer, read
under its own bao leaf and cached for 60s. Static user, the post-#5264
swarm-logs-auth unit shape.
- swarm-bao-grafana-metrics-bearer: on Grafana's host, creates that bearer
in bao if absent (create without update, Grafana's existing leaf) and
writes "Bearer <value>" into the container; the datasource sends it as
its Authorization header via $__file{}.
- The vhost: read locations take auth_request through a map (a bearer goes
to swarm-metrics-auth, anything else to authelia, no login redirect);
vmui takes an authelia session with the login redirect; forceSSL like
the logs vhost. The authelia subrequest location is renamed from
/__metrics_push_authz to /__metrics_authelia since it now serves reads.
- Authelia: the collector keeps every path; everyone else is refused the
collector's push and /metrics paths; group admins is admitted to the
rest; then deny.
- bao: a swarm-metrics-auth cert role and policy (read on the one leaf),
its leaf minted by glue-bao-tls, the CN reserved as a hive name, and
Grafana's grant gains create on the bearer leaf.
Fixes#3596
The retries covered about a minute, and the swarm-bao restart that lost
#5286's review lasted 3m19s. Each try now gets its own forge deadline,
a try with no answer by then is retried like a refused connection, and
the waits are 5s, 15s, 45s, 90s and 150s: six tries over about five
minutes.
The docs also said to request the review again after a post-failed run.
That does nothing while the account's earlier request is still pending,
so they now say to remove the request and then request it again.
A review request's verdict was lost when the post failed: on 2026-10-14
the classifier approved #5286, but its post hit swarm-bao mid-restart
(cert login: connection refused), the run was recorded post-failed, and
nothing on the PR showed it.
post() now tries the post up to four times, waiting 5s, 15s and 45s,
when it fails because the connection to the secret store or the forge
failed or timed out. Any other error fails at once, as before. The
verdict is computed once and reposted as is. All tries share the one
forge deadline the single post had, and the warn log carries the number
of tries.
