Ten service modules (forge, matrix chat, authelia, grafana, otel, nats,
swarm-ui, victorialogs, victoriametrics, bao UI) wrote
services.nginx.virtualHosts themselves, each applying the gateway's
listen set, per-name TLS attrs and security headers by hand. They now
declare services.hyperhive.gateway.vhosts.<name> with only their own
locations and options; ./declared-vhosts.nix renders it into the nginx
vhost with listen, TLS and headers added.
An eval assertion refuses a declared vhost that sets listen, addSSL,
onlySSL, enableACME, useACMEHost, sslCertificate or sslCertificateKey,
and one declared while the gateway is off. nats' own leaf is the new
`certificate` field. The rendered nginx vhosts are unchanged.
open_stream takes its deadlines as parameters so the tests can use short
ones. The header-write test connects to a peer that holds the connection
without reading and sends a header larger than any loopback buffer; the
connect deadline is tested through the shared bounded helper with a
future that never completes, since a connect that hangs is not
reproducible without the network. The module-eval case asserts the
unit's RuntimeMaxSec and that the receiver script reads its header with
a timeout.
The receiver blocked in read until a peer sent a header, and a stalled
btrfs receive was never reaped, so a silent or stuck connection held a
root service instance indefinitely. The receiver now gives the header 30 s
and the unit has RuntimeMaxSec=6h.
The sender had no deadline on the TCP connect or the header write, so an
unreachable mesh address held the admin-socket handler until the kernel
gave up. Both now time out after 15 s and report it like any other
failure. The stream body stays unbounded on the sender: the receiver's
RuntimeMaxSec already caps it, and btrfs send writes it outside this
process.
vale's write-good.Passive and Microsoft.Contractions rules (error level)
flagged 'is killed', 'is capped' and 'has not'. Same bounds stated: 2
minutes per call, 4 minutes for the sequence.
The calls run serially before the web UI is served, so a per-call deadline
alone left N x 2 min of boot delay (8 min with the default two
marketplaces and two plugins, 10 with auto-update) when the fetch path is
blackholed. One shared 4 minute budget now bounds the sequence: each call
gets min(2 min, what is left), and calls after the budget is spent fail at
once with the same warning.
Chosen over spawning the install: boot keeps installing plugins before the
first turn, as today, and the diff is one Instant threaded through the three
call sites. docs/turn-loop/config.md states the 2 minute and 4 minute bounds.
The claude plugin marketplace add/update and install calls fetch over
the network and are awaited before the web UI is spawned. One hung
fetch stopped the agent from booting at all, against the module's own
promise that plugin failures never block boot.
Each call now runs under a 2 minute deadline; on expiry the child is
killed, a warning is logged and boot continues, as for any other
plugin failure.
Each container now runs the journal forwarder the queue's and the builder's
containers run, pushing to a receiver of its own on the swarm collector. Each
authenticates as a new machine OIDC client of its own
(swarm.<service>.otel.clientId), registered by a glue file wherever authelia
runs, minted by authelia and published to the store by the secret publisher.
Grafana's browser login client is unchanged.
The fetch unit inside each container reads the client secret with the store
identity the host already holds for that service (the runner's, Grafana's,
and the log store responder's), now also handed into the container as an
nspawn system credential. Each of those three policies gains one read stanza
on its forwarder's client path. The host units that use those identities are
unchanged.
The three containers drop --link-journal=host, and the host collector stops
reading them.
The parameter was discarded with a comment claiming kept_state_names
takes &self; it is an associated fn with no receiver. Removes the
parameter, the false comment and the argument at both callers.
The gateway nginx serves stub_status on a loopback server of its own,
127.0.0.1:8085 (services.hyperhive.gateway.statusPort). The hive-tier
collector reads it with the contrib nginx receiver, which the pinned
otelcol-contrib 0.151.0 ships, so no exporter daemon is added. The
receiver dials http:// only (a unix: or http+unix: endpoint fails with
"unsupported protocol scheme"), so the listener is loopback TCP with
allow 127.0.0.1 / deny all rather than a unix socket.
The new gateway dashboard plots nginx.requests, nginx.connections_current
by state, and nginx.connections_accepted against handled, per hive.
Part of #5342.
Three git spawns in workers/auto_update.rs and workers/knowledge.rs used
a bare tokio Command, so they ignored HYPERHIVE_GIT and, when the job
queue's deadline dropped their future (the knowledge pull has a 5 minute
one), the git child was left running instead of killed.
They now use lifecycle::git_command(), as the other git calls in
knowledge.rs already do.
Each container now runs the journal forwarder the queue's container runs,
pushing to a receiver of its own on the swarm collector as the service's
existing OIDC client, with the client's secret read inside the container
by the fetch unit. Their store identities are system credentials, so the
fetch unit loads them too. The three containers drop --link-journal=host,
which hid their journals from the forwarder, and the host collector stops
reading them.
The swarm-nats container now runs its own otel collector, which reads the
container's journal and pushes it to the swarm collector at
https://<swarm.otel.domain>/swarm-nats, authenticated as the queue's
existing OIDC client (swarm.nats.clientId). Records carry
service.name=nats and no hive label.
The collector lives in a reusable container module,
nix/container-modules/swarm-journal-forwarder.nix, which follows
swarm-bao's forwarder: a journald receiver over /var/log/journal with
merge and a file_storage cursor, an otlphttp exporter with an
oauth2client authenticator that requests its own client id as audience,
self-metrics on a loopback telemetry port, and the client secret handed
in through LoadCredential. Unlike swarm-bao, the secret is fetched inside
the container by swarm-journal-forwarder-oidc.service, under the
container's own store identity (the swarm-nats-auth leaf, which already
holds a read grant on secret/swarm/services/swarm-nats/oidc/client).
The fetch retries until the store answers, and the collector restarts
until the file exists. A 10-minute timer re-reads the secret and restarts
the collector when it changed.
The swarm collector's authenticated producer route was hard-wired to
swarm-bao. It is now rendered from swarm.otel.serviceProducers
(internal), one receiver, oidc authenticator, gateway location and
pipeline entry per service: swarm-bao (port 4391, unchanged names) and
swarm-nats (port 4392). The reserved-fragment and port-collision
assertions cover every entry.
The queue's OIDC client registration gains its own id as audience and
RS256-signed access tokens, which the receiver verifies offline.
swarm-nats drops --link-journal=host, so its journal stays in the guest
where the forwarder reads it, as swarm-bao did in 44009dc51. The new
swarm.nats.otel.telemetryPort (8891) keeps the forwarder off the
collectors' 8888-8890 in the shared host netns, with an assertion.
module-eval-nats-authelia gains four cases: the container's forwarder
and fetch unit on the store host and on a queue-only host, the
responder role's read grant, the swarm collector's swarm-nats receiver
on a host without the queue, and the client registration.
Part of #4499
Part of #5342
The otel enable description and the exposeHostPorts example in network.md
still described agents dialing a bridge address. Rewrap the two long lines
the earlier edit left.
Agents are handed `http://hive-collector.<swarm domain>:<port>` as their
otel endpoint instead of `http://<bridge IP>:<port>`. Every hive running
its collector contributes the name to `gateway.localNames`, so its own
dnsmasq answers it with its bridge IP, the way the queue's name already
works. The rendered meta flake then carries no hive-specific address,
which a swarm-level render needs (#4857, operator ruling 100744: "b").
hive-c0re's own exporter keeps the bridge address: on the host the name
may resolve to loopback through `gateway.localHostsEntry`, where the
receiver does not listen.
The name sits under the swarm domain and starts with `hive`, a fragment
no hive name may contain, so no hive's default domain can take it.
credentials.md, services.md and secrets.md each started with text that
belongs elsewhere, ahead of the heading: a cell fragment glued onto the
heading line, and two table rows. Put each back in its table or cell.
secrets.md was already broken on main (the swarm-builder row of the
per-unit table sat above the heading); the other two came in with this
branch.
Grafana's VictoriaLogs datasource now targets https://logs.<swarm> and sends
the same static bearer as the metrics datasource. The log store's vhost routes
a 64-hex Authorization bearer to a second swarm-metrics-auth instance
(swarm-logs-grafana-auth, cert role swarm-logs-auth, which gains a read on the
bearer's store path). swarm-agent. tokens and everything else keep their routes.
Part of #3742 (DoD 3, logs).
Three dashboards over metrics already in VictoriaMetrics: the swarm
controller's job graph and forge push/commit counters, the forge
process's runtime, and the OTEL collectors' self-telemetry.
The three delivery units made /run/swarm-grafana-secrets only after
`bao login` and `bao kv get`, so with the store sealed or unreachable the
bind source was never created by them and a switch could restart the
container without it. Each script now starts with the install, from one
shared `ensureSecretDir`, and the tmpfiles comment says what is true.
The module-eval case asserts the install precedes `bao login` in all three
scripts. docs/swarm/sso.md states that after a reboot Grafana stays down
until the secret store answers.
Part of #4998 (slice 2b: grafana).
The three delivery units still read from bao under Grafana's own leaf, but
now write to /run/swarm-grafana-secrets on the host (tmpfs), which the
container bind-mounts read-only at the same path. Before, they wrote to
/var/lib/nixos-containers/swarm-grafana/var/lib/{grafana-oidc,grafana-admin,
grafana-metrics}/ on persistent disk, where a superseded value stayed after
rotation and went into backups of the container root. Host tmpfiles now
removes the three old files.
Rotation is unchanged: refresh_consumer restarts Grafana (or starts it if it
failed) when a file changes, since Grafana reads each $__file{} once as it
starts. After a reboot the files are fetched again; until the store answers
Grafana stays down, where before it came up on the last copy from disk.
Part of #4998 (slice 2b).
The swarm UI has no switch of its own since it runs wherever the
controller runs; its package option follows it under
deploy.swarm-controller.ui. deploy.swarm-ui.package and
swarm.ui.package stay as renames and warn at evaluation. The flake's
default, the vhost root and the module-eval stub use the new path.
A spawn into a missing cwd fails with a bare ENOENT; a daemon newer than its
container's bind now inherits its own cwd again, as before the work-dir default.
The system prompt now names the work dir for worktrees, clones and build
trees, and keeps state/ for notes. A subagent started without a dir runs
in the agent's work dir instead of the daemon's own cwd.
swarm-bao-matrix-oidc still reads the secret from bao under matrix-ctl's
leaf, but now writes it to /run/hive-matrix-oidc on the host (tmpfs), which
the container bind-mounts read-only for tuwunel's LoadCredential. Before, it
wrote it to
/var/lib/nixos-containers/hive-matrix/var/lib/tuwunel-oidc/<id>.secret on
persistent disk, where a superseded secret stayed after rotation and went
into backups of the container root. Host tmpfiles now removes that old file.
clientSecretFile must now lie under /run/hive-matrix-oidc/, the one directory
the container shares; any other value fails at evaluation.
Rotation is unchanged: the ten-minute timer rewrites the file when the value
differs and refresh_consumer restarts tuwunel. After a reboot the secret is
fetched again; until the store answers, tuwunel does not start.
Part of #4998 (slice 4: tuwunel).
swarm-bao-forwarder-oidc still reads the secret from bao under the
forwarder's own leaf, but now writes it to /run/swarm-bao-forwarder-oidc on
the host (tmpfs), which the swarm-bao container bind-mounts read-only for the
collector's LoadCredential. Before, it wrote it to
/var/lib/nixos-containers/swarm-bao/var/lib/swarm-bao-otel-oidc/<id>.secret on
persistent disk, where a superseded secret stayed after rotation and went
into backups of the container root. Host tmpfiles now removes that old file.
Rotation is unchanged: the ten-minute timer rewrites the file when the value
differs and refresh_consumer restarts the forwarder. After a reboot the
secret is fetched again; the file is not pre-created, so until the store
answers the forwarder stays down (LoadCredential refuses a missing source) and
resumes from its journal cursor afterwards. The container now starts after
systemd-tmpfiles-resetup, since the new bind source is made there.
Part of #4998 (slice 3: forwarder).
swarm-bao-otel-oidc still reads the secret from bao under the collector's
own leaf, but now writes it to /run/swarm-otel-oidc on the host (tmpfs),
which the container bind-mounts read-only for the collector's
LoadCredential. Before, it wrote it to
/var/lib/nixos-containers/swarm-otel/var/lib/swarm-otel-oidc/<id>.secret on
persistent disk, where a superseded secret stayed after rotation and went
into backups of the container root. Host tmpfiles now removes that old file.
Rotation is unchanged: the ten-minute timer rewrites the file when the value
differs and refresh_consumer restarts the collector. After a reboot the
secret is fetched again; until the store answers the collector starts on an
empty file and the stores refuse its pushes and scrapes.
Part of #4998 (slice 2a).
swarm-bao-ci-runner still reads the token from bao under the runner's own
ci-runner leaf, but now writes it to /run/swarm-ci-runner on the host
(tmpfs), which the container bind-mounts read-only for the register
unit's LoadCredential. Before, it wrote the token to
/var/lib/nixos-containers/swarm-ci-runner/var/lib/swarm-ci-runner/registration-token
on persistent disk, where a superseded token stayed after rotation and
went into backups of the container root. The container's tmpfiles now
remove that old file.
Rotation is unchanged: the ten-minute timer rewrites the file when the
value differs and refresh_consumer restarts the runner. After a reboot
the token is fetched again; until the store answers, a runner that is
already registered keeps running and only a new registration waits.
Part of #4998 (slice 1).
The dashboard, its API, the swagger UI and the agent routes move to their
own vhost, `swarm.dashboardDomain` = `<hive>.hives.<swarm.domain>`, under
the swarm domain so authelia's session cookie reaches it. `/`, `/api/` and
`/api/docs/` ask authelia through `auth_request`, and authelia's
`operatorsOnly` rule for each hive's dashboard name lets only the admins
group in. The hive domain keeps matrix discovery and redirects every other
path to the new name.
The name rides as a SAN on the hive's own gateway leaf: the per-hive bao
role allows it, and hive-tls asks for it and checks coverage.
HTTP Basic auth is gone: the htpasswd writer and its test, the
`hivectl gateway` verbs and host-socket requests, the `gateway.auth`
options (now a removed-option error), the unauthorized page and the bcrypt
dependency. tmpfiles deletes a leftover gateway.htpasswd, which held
bcrypt hashes at 0644.
c0re hands out dashboard and agent URLs on the new name
(HYPERHIVE_DASHBOARD_DOMAIN), and the swarm controller's hive roster
carries each hive's dashboard name for the swarm UI's hive list.
Closes#3469
swarm-bao-forge-oidc fetched the forge's OIDC client secret from bao and wrote
it to /var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/<id>.secret,
on the container's persistent root, where it outlived a bao rotation and was
in every backup of that tree.
Now the fetch writes it to /run/hive-forge-oidc/<id>.secret on the host
(tmpfs, root 0400). The directory is bound read-only into the container at
the same path, and forgejo-sso-source loads it through LoadCredential. A
tmpfiles rule creates the directory at boot, so the bind source always
exists, and removes the old persistent copy.
After a reboot the file is absent until the fetch succeeds, and
forgejo-sso-source fails to start until then; the fetch starts it again once
it writes the file. Logins keep working meanwhile: forgejo holds the secret
in its login-source row.
deploy.forgejo.sso.clientSecretFile (and its old swarm.forge.sso name) is
removed: it only chose where in the container the fetched file landed. The
fetch no longer needs the container's forgejo uid, so it no longer fails on a
fresh host before the container has created its state dir.
Part of #4998.
The shell had two body widths, a 60em readable column and a 90em wide body
that a short list of table routes opted into. Every route now gets the
90em body, and the opt-in route list, its modifier class and its test go
away.
Until its first successful read of its key, the harness read its
permissions as the default set, `execution` included. So did the
subagent daemon. An agent the operator had narrowed ran wider than
granted after every restart while the queue was unreachable, and in the
gap before the first 10 s poll. During the meta-repo import, every
non-ruth key is still unset, and the agent read that as the default too,
for as long as the import stayed pending.
mara picked option (a) on #5293: "a, removal of the old path as a follow
up in 1w".
hive-c0re now writes each agent's entry from
swarm-agent-permissions.json into the agent's runtime dir as
swarm-permissions.json (/run/hive in the container), together with
whether the import is still pending. It writes the file at every start
(write_dropins) and after every 30 s pass of the permissions worker. The
harness and the subagent daemon hold that stored list until their own
first read succeeds. While the import is pending, an unset key keeps
the stored list rather than reverting it to the default. A queue that
is down never blocks a start: the agent runs on the stored grant.
The decision lives in swarm_queue_client::agent_permissions::HiveCopy,
shared by both readers.
Part of #5170.
Declaring an agent Destroyed now deletes its `agent-permissions` key,
next to the queue-credential revoke and with the same failure handling:
logged at error, the destroy goes ahead, and declaring it destroyed
again re-runs it. Agent names are reusable after a destroy, and a create
left on the default stores no key, so the old key handed a new agent of
the same name the destroyed agent's grant (argus on #5293).
Also: migrate.rs's create-time-setup note named ruth's tool-group seed
in `ensure_root_agent`, which this PR deletes; it now says she holds the
code-constant default when the swarm sets nothing.
Part of #5170.
mara on #5170: "dont split caps and perms. its all perms. / dont include
the now empty tool groups / nothing set -> default / also add perms to
agent creation screen".
- hive-sh4re: one `Permission` enum (messaging, meta, inbox, execution,
web_tools, manage_root_agent) replaces `ToolGroup` and `Capability`;
`lifecycle`/`forge` are gone. `Permission::default_for` is the default
for an agent the swarm sets nothing for.
- The record is `{ "permissions": [...] }`. Missing key, unset field or
no successful read yet = default; `[]` = none; a failed read after a
good one keeps the last value.
- hive-agent reads its own key every 10s and builds each turn's --tools,
--allowedTools and MCP config (rewritten before every turn) from it;
the subagent daemon reads it on each start/continue. Agents get a
read-only grant on their own key.
- hive-c0re restarts an agent only when its manage_root_agent changes.
The import now merges meta/capabilities.json and meta/tool-groups.json
into one list per agent and deletes both; the "import pending, don't
read" decision is `ImportOutcome::read_allowed`, table-tested.
- swarm-controller: `GET /api/permissions` (names + default) and an
optional `permissions` on `POST /api/agents`, stored before the job is
queued. swarm-ui's create-agent form gets a permissions field.
- Deleted: HIVE_TOOL_GROUPS and its forward to the subagent daemon,
tool_groups.rs, ruth's tool-group seed, the permissions rebuild job
(PermPayload/WritePermFile/perm_change), the dashboard permissions tab
and its stale-entries panel.
Part of #5170.
An agent's capabilities move from the hive's meta repo
(`meta/capabilities.json`, injected as `HIVE_CAPABILITIES`) into the
`capabilities` field of its key in a new `agent-permissions` KV bucket,
which swarm-controller alone writes.
- swarm-queue-client: `agent_permissions` (bucket, key, `AgentPermissions`,
direct-get read, the import subject and its wire types).
- swarm-controller: creates the bucket at start; `GET|PUT|DELETE
/api/agents/{name}/permissions` (deny_unknown_fields and unknown
capability ⇒ 422, duplicate ⇒ 400, agent must be declared by a hive);
answers each hive's one-shot import, writing a key only where there is
none and only for agents that hive's wanted state declares.
- swarmctl: `agent permissions get|set|clear`.
- swarm-nats-auth: a hive reads every key of the bucket and sends its
import on its own subject; the controller subscribes to the imports and
may answer them (`responses`).
- hive-c0re: reads each of its agents' keys every 30s into
/var/lib/hyperhive/swarm-agent-permissions.json and restarts an agent,
gracefully, when its capabilities change. Unset is the compiled-in
default (ruth: manage_root_agent; everyone else: none). A failed read
keeps the stored value. On first boot it seeds that file from
meta/capabilities.json, sends the grants to the controller, and deletes
the file once the controller answers.
Deleted: the capabilities.json reader/writer, `HIVE_CAPABILITIES` in the
rendered meta flake, ruth's capability seed, the capabilities half of the
permissions rebuild job (`PermPayload::Capabilities`/`Combined`), the
dashboard's capabilities table and `capabilities_changed` event, and the
now-unused `Capability::description`.
Part of #5170.
The crate README rewrite was unrolled at eval time: one sed block per
crate, each carrying one -e expression per other crate. The installPhase
string grew with the square of the crate count, and at 40 crates
(d5de1e494 added swarm-claude-login) it reached 131708 bytes. Linux
caps a single environment variable at 128 KiB, so exec of the builder
failed with "Argument list too long" and every consumer of
reference-docs failed with it, the website repo included.
The rewrite is now a shell loop over $members at build time, building
the same sed expressions per crate. The phase text no longer depends on
the crate count. Output is byte-identical to the old form at 6aab2ebb6
(39 crates, the last commit where it built).
Closes#5321
swarm-bao-grafana-controller-token logs in to the store with Grafana's
leaf, like its three Grafana siblings, and glue-bao-readers-store-order.nix
already orders it after the store's container and PKI unit. The
services-switch case that pins that ordering did not list it, so dropping
it from the glue would have passed. It is now in the roster, and the count
of readers rendered off the store's host goes from 7 to 8: the unit exists
wherever Grafana does.
credentials.md gets the row for its store path, beside the controller's
forge token.
swarm-bao-grafana-controller-token, on the host that runs Grafana, logs in
to Grafana's API over its socket as the built-in admin (the password
swarm-bao-grafana-admin minted, #5138) and keeps a `swarm-controller`
service account that is an enabled Viewer. It mints a token for that
account and writes it to bao at
swarm/controller/swarm-controller/grafana/token under Grafana's leaf. A
ten-minute timer runs it again. The unit keeps a stored token Grafana
accepts. If Grafana refuses the stored token with 401, or bao holds
none, the unit deletes every token of the account and mints a new one.
Grafana's grant gains create/read/update on that leaf, and the
controller's gains read.
swarm-controller serves GET /api/alerts from Grafana's
/api/prometheus/grafana/api/v1/alerts. It reads the token from bao,
keeps it in memory, and re-reads bao once on a 401. The HTTP client has
5s connect and 15s request timeouts. Any failure is a 503 that names its
cause. swarm-ui gets an /alerts page listing them, built the same way as
the hives page.
This is option (c) on #5129, the operator's choice.
A store answers the revision it wrote and what it replaced, and restoring
compares against that revision, so a write that landed after the creation's
own is kept instead of overwritten. A write that changed nothing rolls back
nothing, and a key that held no value is removed rather than set to an empty
one. A test fails the second store after the first and checks the earlier
caps come back.
create_agent puts back the caps and inference settings it replaced if a
later store or the job insert fails, so a refused creation leaves nothing
behind for the name. The shared field components take an id prefix so the
create dialog and an open editor never share a DOM id. The queue check now
follows the hive roster check, the 422 is documented, and the copy says
empty rather than closed and that destroy keeps stored settings.
POST /api/agents takes optional limits and inference objects, checked by
the same rules as the two PUT routes before anything is queued and stored
ahead of the job graph. The create form gets collapsed sections for both,
reusing the fields of the detail-panel editors.
