The four-way client-cert split gives each store reader its own leaf, and
three of the four readers render only where their own leaf exists. On a
host that mints its own PKI glue-bao-tls.nix defaults all eight, so there
is nothing to do; on a hand-configured remote-store hive, omitting one
pair used to mean that unit silently did not render — a privilege-
narrowing unit absent from a green build, with the missing unit as the
only evidence.
Each of the three now asserts its own pair, shaped after
swarm-grafana.nix's haveClientIdentity assertion and named to the pair it
needs. What differs from Grafana's is the gate: these fire only where the
host demonstrably reads the store (it holds deploy.bao.clientCertFile and
clientKeyFile) and the consumer is on. A host with no store identity is
the supported no-store deployment and still evaluates; the collector's
no-secret degrade is untouched, because that host holds no clientCertFile
either.
Also rewords three passive-voice sentences in docs/swarm/secrets.md that
vale flagged, and documents what the refusal costs and where it stays
silent.
Four units read one path each out of the store, and all four logged in
holding `deploy.bao.clientCertFile` — the hive's own leaf. Bao identifies
a principal by the subject of the certificate it presents, so four
readers behind one certificate were ONE principal, and the only grant
expressible was the union of what the four need: read on
`swarm/agents/*`, `swarm/hives/<hive>/*` and `swarm/services/*`. The unit
fetching Grafana's OIDC client secret could fetch every agent credential
in the swarm; the one fetching this hive's matrix token could fetch
Grafana's. Least privilege was not misconfigured here, it was
unrepresentable.
Each now holds a leaf, a cert-auth role and a policy of its own, and each
policy is the single `secret/data/…` path that unit's own script names —
spelled to the leaf, not to a prefix, the way matrix-ctl's already is.
Following the four exemplars in-tree rather than building a mechanism:
`signLeaf` mints the leaves, `swarm-bao.nix` writes the roles from the
bootstrap token, the consumers name their own pair.
Two of the four are written PER HIVE and two are not, which is the shape
of the paths rather than a preference. A matrix appservice token and a
queue credential live under `swarm/hives/<name>/` and every hive runs a
reader for its own, so one role for all of them would have to be granted
`hives/*` — letting one hive read another's, a reach no hive has today.
An OIDC client secret lives under `swarm/services/<client-id>/` and a
swarm registers each exactly once, so one role each is enough. The
per-hive subjects are `<prefix>-<hive>` and swarm.nix reserves every
composed spelling as a hive name, so a hive cannot be named into another
hive's role.
The shared leaf stays: hive-c0re still passes it into its container, the
`bao` CLI wrapper still defaults to it, and the three
`glue-*-bao-identity.nix` files derive the PKI directory from it.
module-eval-bao-grants gains a negative arm per principal — each pins the
three stanzas the hive's leaf carried and the two wildcards a later
widening would reach for, so a policy that grows fails here rather than
in a store. Plus the consuming side: repointing a unit back at the hive's
leaf would evaluate, deploy and log in, and silently restore the union.
A hive that reads a store on another machine now places one leaf per
principal instead of one shared by four. That cost is the point, and
docs/swarm/secrets.md lists the pairs.
Review response on #4620: not having SSO is not a supported
deployment, so the type should not permit it, and the docs paragraph
explaining why SSO is always present is redundant once the type says
so.
- swarm.authelia.url drops types.nullOr.
- Every consumer's null-arm is gone: two option defaults
(swarm-controller's and swarm's own statusPublish.tokenEndpoint)
that produced an empty/null placeholder when the URL was null now
unconditionally compute the real derived URL. Five now-dead
"assertion = ... != null" guards (swarm-authelia's bridge,
swarm-grafana, swarm-otel, swarm-nats, hive-forge, hive-matrix) are
removed as unreachable — in every case the same URL was already
interpolated unconditionally a few lines below the guard.
- grafanaNoSso, the module-eval fixture whose sole purpose was
exercising the now-unsupported no-IdP refusal, is removed along
with its dedicated test case; swarm.authelia.url = null is a type
error now, not a value that reaches that assertion.
- docs/swarm/services.md: cut the clause about setting the option to
null and the sentence explaining why the URL is co-location-
independent — both redundant now that the type enforces it.
`swarm.authelia.url` defaulted to `https://<domain>` only when this host
ran the container, and to `null` otherwise — so the address a client is
given was a statement about co-location rather than about the swarm. A
swarm has one SSO provider; every hive addresses the same name and
resolution decides which address that reaches, exactly as
`swarm.otel.domain` already works.
The option stays nullable: "this swarm has no IdP" is still expressible,
it is just now something an operator states rather than something not
running the container produces. The Grafana fixture that exercised the
no-IdP refusal says it explicitly.
Closes#4536
`services.opentelemetry-collector` already defines
`serviceConfig.Restart = "always"` at the normal priority. The forwarder
added in #4537 defined `"on-failure"` beside it, and two definitions at
one priority are a conflict the module system refuses to resolve — so
`containers.swarm-bao` stopped evaluating at all (#4615).
Drop our definition rather than force a value over it, which is what the
sibling collector in swarm-otel.nix already does. The resolved value is
`"always"`, which is the one we want here: a forwarder that exits for any
reason, clean or not, has stopped shipping the store's journal.
`RestartSec` stays — it conflicts with nothing and is what keeps the
restarts during first-boot secret delivery under the default start-rate
limit.
bao's metrics were scraped by the SWARM collector over loopback, via a
`swarm.otel.scrapeTargets.bao` entry gated on `deploy.swarm-otel.enable`
— "does the swarm's collector run on THIS host". It had to be: loopback
only reaches a reader that landed on the same host.
What that rendered everywhere else was nothing at all. Off that host the
metrics listener was not emitted, so the store's metrics reached the
store nowhere, and a host with no entry is indistinguishable from a host
nobody asked to scrape.
Moves the scrape into the collector this container already runs, per
mara on #4537: "move the existing scraper to the local collector". The
container shares the host netns (privateNetwork = false), so the scrape
still dials 127.0.0.1 — the listener keeps its address, its
`metrics_only` narrowing and its loopback-only bind, and the API
listener's `tls_require_and_verify_client_cert` is untouched.
The listener and its `prometheus_retention_time` lose their gate: the
reader ships with the store now, so there is no host where the endpoint
has none. The metrics pipeline reuses the logs pipeline's `resource`
processor and `otlphttp` exporter, so both signals carry the same
`service.name` and leave by the one hop.
Logs are unaffected: `journaldUnits` and --link-journal=host stay until
every sibling swarm container has a collector of its own.
The module-eval absence arm "a store with no collector beside it serves
no metrics" is inverted rather than dropped — the condition it asserted
is the bug. Three cases join it: the job is in swarm-bao AND gone from
swarm-otel (a move, not a copy), the scrape target and listener are both
pinned to loopback, and the metrics pipeline shares its exporter with
the logs one.
Both earlier versions asked the wrong host. `hyperhive.otel.enable` asked
whether this host runs a HIVE collector; `deploy.swarm-otel.enable` asked
whether this host runs the SWARM one. Neither answers the question the
forwarder actually has — "is there a collector to forward to" — and that
question cannot be false: a swarm always runs at least one instance of
every swarm-level service. So the forwarder renders under the condition
already enclosing it, that the store is deployed here, and nothing else.
`scrapeHere` deliberately keeps its `deploy.*` gate one line up. It is a
loopback metrics listener, which genuinely only works where the scraper
is — the two are different tiers, and the name says so.
The module-eval case that pins it is the split topology: the swarm
collector on another host, nothing local naming it, and the forwarder
still enabled and still addressed at `swarm.otel.domain`'s route. Both
removed gates render nothing in that fixture, which the co-located ones
they shipped with could not show.
The forwarder pointed at the hive bridge address and was gated on the
hive's `otel.enable`, so it existed only where a hive collector stood
beside it. It now exports to `swarm.otel.domain` — the gateway-served
name that resolves locally when co-located and over the network
otherwise — on the swarm tier's own producer route, and is gated on
`deploy.swarm-otel.enable` like its sibling `scrapeHere`.
Refs #4526
Every container is supposed to run a collector that passes its logs and
metrics to the next hop. swarm-bao did not: its journal reached the store
only because `--link-journal=host` puts it in the host tree, where the
swarm collector — a different container — reads it through a unit
allowlist. That is the topology being retired, and in this deployment it
delivers nothing: no `_SYSTEMD_UNIT` value in the seven-day store mentions
openbao at all.
So the store's container now runs its own journal forwarder, copied from
an agent container's (nix/agent-modules/otel.nix): the whole journal, no
unit allowlist, pushed to the same first hop every agent on the host
already exports to. A local collector reads the local journal, so there is
nothing for a list of unit names to disagree with.
The `swarm.otel.journaldUnits` entry and `--link-journal=host` both stay.
Every sibling swarm container still rides the shared collector, and they
come out once each of them has a forwarder of its own.
Closes#4526
mara: 'i asked for a table. ask when doing something different.' Right
call — the PR body flagged the bargauge substitution as an open question,
not a decision, and I should have waited for an answer instead of
treating the silent absence of an objection as one.
Single 'Busiest agents' table panel (replaces the 5 per-metric bargauges):
5 table-format instant queries (turns, input, output, cache-read, cost)
joined on the agent label (joinByField), renamed to the screenshot's own
column names via organize, sorted by turns descending via sortBy — same
column set and sort order as the attached /stats screenshot.
This is a genuinely novel schema shape for this repo: no table panel,
transformation, or field-override config exists anywhere else in
nix/host-modules/swarm-grafana/dashboards/*.json to verify the join/
organize/sortBy option shapes against. Structurally verified (valid
JSON, jq empty, unique panel ids, every panel referenced exactly once,
nix fmt clean, dashboard-description lint clean) but the join/rename
field-name mechanics (whether Grafana names the joined columns exactly
'Value #A'/'Value #B'/etc.) are built from general Grafana schema
knowledge, not a working local precedent — flagging that explicitly so
the actual render gets checked before merge.
Reworked per review feedback: no new tab, and 3 of the added stat
panels duplicated existing ones (Active agents ~ panel-1 Agents
reporting, Cost = panel-2 Cost verbatim, Tokens ~ panel-3 + the new
input/output/cache-read breakdown). Removed panel-50/52/56 and the
overview tab entirely.
What's left is genuinely new (no prior panel covered it): a Turns KPI
and turns-by-agent bargauge in the activity tab (hyperhive.agent.turn.count
was emitted but unused anywhere), and input/output/cache-read token KPIs
plus cost/input/output/cache-read-by-agent bargauges in the cost tab
(the existing cost/token panels have no per-agent breakdown, and none
sort busiest-first).
New 'overview' tab on the agents dashboard, extending it per the
hive stats page's own summary layout (busiest-agents table + top KPI
row):
- KPI row: active agents, turns, tokens (all types), input, output,
cache-read tokens, cost. All from metrics already flowing into
this dashboard (claude_code.token.usage / claude_code.cost.usage /
claude_code.session.count) plus hyperhive.agent.turn.count, which
the harness already emits but nothing visualized yet.
- Busiest-agents replicated as five sorted bargauge panels (turns,
cost, input, output, cache-read, each by agent) rather than one
literal multi-column table -- this schema (Grafana v2beta1) has no
existing table+merge-transform panel anywhere in this repo to
pattern-match against, and every other panel here follows the
proven sort_desc(...) + bargauge shape already used for
'Cost by subagent name' (panel-43). Trading the exact table layout
for schema I can verify against a working precedent.
- Left out the stats page's 'window' tile -- redundant with the
dashboard's own always-visible time-range picker.
No new instrumentation needed. Structurally verified (valid JSON, no
duplicate panel ids, every panel referenced exactly once in the
layout) but not visually verified -- no local Grafana to render
against.
A swarm runs one homeserver and every hive on it logged in as the same
`@hive:` localpart, holding the same access token out of one swarm-wide
store path. That is one matrix identity for N hives: the homeserver
cannot attribute an action to the hive that took it, and revoking one
hive's standing revokes every hive's.
Three changes, and the third is the one that makes the other two real:
- **The localpart carries the hive's name** (`hive-<hive>`), derived in
one place, `swarm_secret_client::matrix::hive_localpart`.
`hive-matrix.nix` renders the same string as the appservice
registration's `sender_localpart`, so the shared account stops being
created rather than merely stops being used.
- **The store path is templated by hive**, not a constant. The
"a swarm runs one homeserver, so this is a constant rather than a
parameter" rationale went with it; it stopped holding the moment two
hives shared the homeserver it describes.
- **The path moved out from under the grant every hive has.** It sat at
`swarm/services/matrix/sender-token`, inside the
`secret/data/swarm/services/*` read stanza `policy::render` gives every
hive. It now sits under that hive's own stanza,
`secret/data/swarm/hives/<hive>/*`, which interpolates the reader's
name — so a hive reads its own token and is refused another's. The
policy renderer itself is unchanged: narrowing the `services/*` grant
would break the OIDC-secret read it exists for, and moving the
credential is what this needed instead. A policy test walks the
rendered stanzas and asserts none of hive alpha's covers hive beta's
sender token, so a later stanza that widened it fails here.
`swarm-matrix-ctl` takes a new required `MATRIX_MINT_HIVE` and writes
that hive's path; its store grant in `swarm-bao.nix` follows, scoped to
one hive's leaf via the new `deploy.bao.matrixCtlHiveName` (defaulting to
this host's `hiveName`) rather than a `hives/*` wildcard, which would
hand the matrix container every hive's token back.
Migration: no outage at deploy. `ensure_hive_user` short-circuits on the
local token file, so a hive keeps running on what it has; with no such
file it reads the new per-hive path, finds nothing, and falls through to
the existing register-or-appservice-login ladder against its own
localpart — which needs only the per-hive `as_token` on local disk. The
old shared object is read by nothing afterwards. Rooms do not follow the
identity, and that is the one operator step; both ways out are written
into `docs/integrations/matrix.md`.
No admin standing is granted to the per-hive accounts: `admin_execute`
stays empty and the assertion pinning it is untouched.
Renames `swarm-matrix-minter` and reshapes it around subcommands. Minting
is now `swarm-matrix-ctl mint`.
Running rust inside `containers.hive-matrix` is not free: it needs its own
store identity, its own cert role and its own bind mounts, and every one of
those is per-*container*, not per-task. A second single-purpose crate would
have had to duplicate that plumbing to add one action, so the next thing
that has to run in there should be a verb here rather than a new crate.
The old name guaranteed the opposite.
`main.rs` is clap dispatch; the minting logic moves to `mint.rs` unchanged.
A bare invocation is refused: `mint` writes a credential, so "no verb"
defaulting to it would make a typo in the unit mint rather than fail.
The environment prefix moves with it, `MATRIX_MINTER_*` → `MATRIX_MINT_*`.
Scoped to the verb and not to the binary, because a binary-scoped prefix is
one the next verb has to share or widen, and a widened one never narrows
again. A test asserts every variable carries the verb's prefix.
The principal renames too. The cert role, bao policy, granting unit, leaf
filename and `certAuthCns` entry all have to spell one string the same way,
so leaving them as `swarm-matrix-minter` would have rebuilt the naming
split this branch exists to remove. Renaming the nix options alongside is
free here: every one of them is introduced by this PR and has never been
released, so no operator config names them yet.
`ExecStart` now names the verb, which is a contract between a nix string
and a clap enum that fails at deploy time with no local signal. Both ends
assert it: `mint_is_spelled_the_way_the_unit_invokes_it` in the crate, and
a new module-eval arm reading the rendered `ExecStart`.
docs/getting-started/setup.md drops the sender token from its "live on the
host" list: setup does not touch this credential, so a setup guide has no
reason to name it.
The store path and every identifier around it called this an admin
token. It is not one: of ~15 hive-c0re call sites only two need
homeserver admin, and the homeserver no longer promotes the account at
boot, so the name overstated both what the credential is and what it may
do.
Renaming it to the account was not enough either. "The `@hive:` token"
reads as the token of a hive user, and no such user is provisioned —
`@hive:<server_name>` is the appservice registration's own
`sender_localpart`, an account the homeserver creates for itself when it
loads the registration.
So it is the **sender token**: the matrix appservice sender account's
access token, at `swarm/services/matrix/sender-token`. The name says
what it authenticates as rather than what it may do, which is the part
that was wrong.
The path has one constructor, and the bao grant, the grant assertion and
three unit tests pin its literal independently — so a half-finished
rename fails a check rather than leaving the minter and its readers
disagreeing at runtime. `tracing` messages are renamed with the code, so
the journal reads the way the source does.
The host-side file keeps its name (`matrix/access-token`): it carried no
admin framing, and renaming it would orphan the file on every deployed
hive for nothing.
`docs/tools/hivectl-cli.md` is regenerated from the clap tree.
A swarm runs one homeserver and a homeserver has one appservice sender
account, so "mint it once" is a property of the thing being minted
rather than something a lock has to enforce. That is what makes this
account the one to move first: no trigger route, no controller change
and no agent list — a boot-time oneshot beside tuwunel is the whole
mechanism.
`swarm-matrix-minter` runs inside `containers.hive-matrix`, which
already holds the appservice token: the rendered registration is bound
in read-only because that is how tuwunel is handed it. What the
container lacked was an identity of its own, so this adds one — a leaf
from the store's CA with a grant of exactly one path, not the hive's
leaf, which reads every secret in the store.
Both ends of the credential ship here. The minter reads the path it
publishes to before it touches the homeserver, and returning on a
non-empty read IS the "only once"; `hive-c0re`'s `ensure_hive_user`
reads the same path, authenticating with the hive name already in
`HYPERHIVE_HIVE_NAME`. The existing mint-then-`M_USER_IN_USE`-login
ladder stays as the fallback for a store that is empty, unconfigured or
unreachable, which is every swarm deployed before this — so nothing
needs backfilling and nothing breaks if the rest of the sequence never
lands.
The credential is not an admin credential, and is not named like one.
It is the access token of the appservice registration's own
`sender_localpart` — `@hive:<server_name>`, an account the homeserver
creates for itself when it loads the registration. The store path is
`swarm/services/matrix/sender-token`, the host path is
`matrix/access-token`, and the homeserver no longer runs an
`admin_execute` promotion for that account at boot. Everything the hive
provisions with it — the Space, the chat room, their hierarchy and join
rules, the invites — rides on being the creator of those rooms at power
level 100, not on homeserver admin; there is no Synapse admin API here
to need, tuwunel has none.
Two operations do need an admin *sender* and therefore stop working:
`hivectl matrix promote-user` and `hivectl matrix reset-password`, both
`!admin …` messages into `#admins:<server>`, plus the password-reset
recovery path that an agent with a lost password file falls back to.
They are swarm-level operations and are left failing loudly rather than
served by an over-privileged token every other call site would also
carry. The sweep's own admin-rights check and self-repair go with them:
an account that is deliberately not an admin has nothing to check.
`ephemeral = false` stays, and hive root can still read the container's
filesystem. Accepted: what this buys is identity separation — no hive
*process* holds or reads the appservice token — not physical isolation.
Refs #4345
'Rows by unit' and 'Rows by host' use VictoriaLogs LogsQL, not PromQL,
so the sort_desc() fix from the PromQL panels doesn't port directly.
Verified live against the real store (swarm-logs query) that
'| sort by (rows) desc' after 'stats by (...)' orders correctly on our
VictoriaLogs version -- the VictoriaMetrics/VictoriaLogs#258 sort-order
bug doesn't reproduce here. Append the sort clause to both panel
queries.
The 41-line comment on logLevelRules tripped the comment-block lint
(30-line max). Move the detailed walkthrough (why each rule shape is
what it is, the three silent-failure modes, the query a level button
emits) into docs/scheduler/observability.md's existing 'Log severity'
section, which already covered the mapping at a higher level. The nix
comment now carries the short why/contract and points at the doc for
the full detail — no information dropped, just relocated.
Grafana's log-level buttons filter on a field called `level`; no row in
the store has one. The store's name for it is `severity_text`, chosen by
VictoriaLogs' OTLP ingester rather than by us — v1.52.0's
`app/vlinsert/opentelemetry/pb.go` writes it unconditionally and the
ingest parameters have no `_level_field` to rename it with. So the
mapping is made on the reader: the VictoriaLogs datasource, which was
provisioned with no `jsonData` at all, now carries a `logLevelRules`
entry per severity the journald parser can emit.
`logLevelRules` is the datasource plugin's only level-related jsonData
key — there is no field-name setting and no OpenTelemetry preset to
switch on. It is read off `instanceSettings.jsonData` in the plugin's
`datasource.ts` and typed in its `configuration/LogLevelRules/types.ts`,
both recovered from the sourcemap shipped in the pinned artifact
(`grafanaPlugins.victoriametrics-logs-datasource` 0.26.3). Each enabled
rule appends an `OR severity_text:="INFO"`-shaped term to the query a
level button emits, next to the `level:…` term that matches nothing.
A wrong rule here fails silently: Grafana provisions unknown jsonData
without complaint and the buttons go on returning zero rows. The three
ways to get one wrong — a non-literal `enabled`, a non-canonical `level`
spelling, a value that is not the stored text — are recorded at the
binding, and a module-eval arm pins them along with the real failure
mode, a severity added to `nix/journald-severity.nix` and not here.
`Unspecified` is left unmapped on purpose: it is the store's own
rendering of an absent severity and the thing the logs dashboard's "no
severity" panel counts.
Refs #4560
Slipped past the #4574 lint — 1473 chars vs. the 100-char cap, caught
by pre-push while working on an unrelated branch. Same treatment as
every other panel: state what the number is, not the reasoning.
Cut every panel/board description down to a short statement of what the
number is. Drops the design-reasoning prose, edge-case notes, and
operator-attributed quotes that had accumulated in an earlier pass —
useful context while building the boards, wrong shape for a tooltip an
operator glances at mid-incident.
The mapping above needs something that says whether it is still working
after whoever wrote it has gone. A timeseries rather than a stat, so a
regression is a line lifting off zero rather than a number nobody reads.
Two series, and the split is the point: rows that carried a PRIORITY and
arrived with no severity anyway (a broken mapping — this one must reach
zero and stay there), against rows that never had a priority to map. The
latter is Claude Code's own OTLP telemetry, which emits log records with no
severity set at the source; mapping cannot reach it, so it is named rather
than folded into one number that never goes to zero.
Needs no provisioning change — logstore.json is already in the shipped
dashboard list.
Records reached VictoriaLogs carrying the journal's raw PRIORITY and
severity_text "Unspecified" — every line in the store, at every tier, with
no level a query or a dashboard could read. VictoriaLogs has no ingest
parameter naming a level field; it auto-detects one by field name, so the
mapping has to happen in the collector.
A stanza severity_parser on each journald receiver, from one shared file
rather than a copy per tier: the two receivers are unrelated config (a
fixed stanza in the agent container, a parameterised block inside the
swarm-otel container) and a drifted copy fails silently — every line still
arrives, labelled as the wrong thing.
Two details that are easy to get wrong and quiet when wrong. PRIORITY
counts down in urgency where the OTEL severity counts up, so the table is
written as a table. And overwrite_text is required: without it the parser
sets the severity number and leaves the text as the raw digit, so
severity_text arrives as the literal "6" — populated, and not a level
anything renders.
Two swarm-wide facts were being read off this machine's deploy set, so the
answer differed between two hosts of one swarm:
- `swarm.authelia.oidc.hiveIdentities` defaulted to `deploy.nats.enable`,
so whether a hive gets an identity at all depended on whether the IdP
host happened to also run the queue. It is on by default now: a swarm's
hives have identities, and the clients are inert until used.
- `swarm.statusPublish.tokenEndpoint` defaulted through `queueLocal`
(`deploy.nats.enable && deploy.authelia.enable`), so a hive that was not
the swarm host had no token endpoint even when the swarm's IdP was
reachable and named. It follows `swarm.authelia.url` now — the same
derivation `swarm-controller.nix`'s own `queue.tokenEndpoint` already
uses, which is correct for a remote provider.
`deploy.nix:1-30` is what makes this a rule rather than a preference:
`swarm.*` is "identical on every host, byte for byte" and `deploy.*` is
"necessarily different on every host". A swarm value derived from a deploy
value cannot satisfy both.
The all-or-nothing status-publish assertion follows: the token endpoint is
no longer one of the coordinates that says this hive publishes — every hive
in a swarm with an IdP has one — so the two per-host coordinates are what
must agree, and they now require the endpoint rather than being counted
beside it.
`queueLocal` itself stays for the three remaining host-local addresses
(`natsUrl`, `clientSecretFile`, `agentNatsUrl`): each of those is a
`deploy.*` value that genuinely differs per host.
Closes#4048
Fix wording flagged on #4521 review: "Single nginx in front of every
hyperhive web surface" and "Runs on the host" claimed a deployment-wide
topology. There's one nginx per host that has something on it, and
potentially more inside service containers. Reword to describe what
this module builds on this host, with no count claim swarm-wide.
`services.hyperhive.gateway.enable`, `gateway.dns.enable` and
`network.enable` replace the `hyperhive.enable` gate on all three. Each
defaults to false; the modules that need one assert it with `mkDefault
true` from inside the guard their own deployment already carries, and
`swarm-required-services.nix` — the module that owns what the
swarm-services toggle implies — asserts all three explicitly.
hive-c0re asserts all three unconditionally, so an ordinary hive keeps
getting them with no opt-in: it is the host's only knowledge that agent
containers exist.
The resolver moves to its own `hive-gateway/dns.nix` so it can be gated
without reindenting the nginx half of the module.
Reinstates `network.enable`, dropping its `mkRemovedOptionModule` shim.
A config still carrying `network.enable = false` from before the removal
now switches the bridge off instead of failing eval.
Also deletes a duplicate `centralToggleOff` fixture in nix/module-eval.nix.
Two sibling slices added it independently (c5f60fd5, ce3b3d94); the merge
was textually clean and left `main` failing to evaluate at all, so this
file could not be gated without removing one.
The receiver was defined for `scrapeTargets != {} || publishedScrapeTargets
!= {}`, but the swarm-tier pipeline named it only for `scrapeTargets != {}`.
A collector with published targets and no loopback ones therefore rendered
scrape configs that belonged to no pipeline: requested, parsed, delivered
nowhere, and valid enough to deploy.
Latent, not live: the module seeds `scrapeTargets.collector` under its own
`enable`, so the loopback set is never empty in a real deploy and the
disagreeing arm is unreachable today. Nothing would have noticed if that
seeding became conditional, which is why the module-eval case pins the
pipeline rather than trusting the seed to keep masking it.
Refs #4511
`services.hyperhive.enable` is going away (Refs #4500). These four blocks
ANDed it with a second condition that was already the load-bearing one:
the bridge-firewall hole asks whether an operator named any
`network.exposeHostPorts`, and the three swarm-level services ask their
own `deploy.*.enable`, each of which defaults false (swarm-ui derives
from the controller's toggle, a sibling deployment decision) and none of
which is derived from the hive toggle. So dropping the conjunct turns
nothing on by itself — the remaining condition still decides.
The way that claim fails is by something becoming unconditional, so
module-eval gets absence arms for all four, each asserted on a host with
the hive ON and on one with it OFF, plus the two controls that make the
absences mean something.
The swarm-controller arm is probed by the credential oneshot and by the
daemon's `ExecStart` rather than by the unit name: hive-tls defines an
environment key on that name, which leaves an inert fragment behind on
any hive with a CA whether or not the controller runs there.
hive-forge's publicUrl and behindGateway, and swarm-controller's
forgeTokenFile, no longer gate their default on
config.services.hyperhive.enable — none of the three has a reader that
depends on hyperhive being enabled to make sense of the value, so the
extra condition only added a hidden coupling. module-eval.nix gains a
centralToggleOff fixture plus four cases asserting each affected
default now resolves identically whether the toggle is on or off.
Refs #4500
`swarm-controller` mints an agent's mTLS leaf at creation and publishes it
at `swarm/agents/<agent>/bao-mtls`. Nothing read it back. This adds the
hop that carries it the rest of the way, and the in-container consumer
that proves the hop works.
Host side, `lifecycle::agent_identity` reads the row under *this hive's*
own certificate — the hive is a principal the store already knows — and
stages the leaf and its key `0600` under a new `agent-identity/<name>`
state dir, deliberately outside every bind-mounted tree. Both files go in
as systemd credentials rather than binds, the same answer and the same
mode reason as the queue secret beside it: the staged key is unreadable
to the unprivileged agent user, and the container manager reads a
`--load-credential` source as root before re-exposing it under the
consuming unit's own `User=`. The agent is never asked to authenticate in
order to obtain the thing it authenticates with.
Container side, `hive-agent-bao-identity.service` logs in with that
certificate and reads the agent's own path back, failing the unit when
either step does not succeed. It fails loudly where the hive-side readers
degrade quietly, because a refused certificate means an agent that
believes it reaches the store and never does — a cause only the login
itself can name.
The address is the whole switch, no separate `enable`, matching how
`queue.nix` and `logs.nix` already gate themselves. A hive with a store
forwards `HIVE_AGENT_BAO_ADDR` and every agent on it gets the check; a
hive without one forwards nothing and no agent does. That is what keeps
the delivery from landing in a container with nothing to read it.
The hive can now reach an agent's identity, so hive privilege covers
agent privilege. Accepted, not mitigated: the alternative is an agent
fetching its own credential with a credential it does not yet have.
Refs #4137
`swarm/agents/<agent>/bao-mtls` did not exist, and neither did any
per-agent identity at the secret store: `policy::agent_object_name`,
`render_agent` and `render_agent_with_queue` had been written and never
called outside their own tests. An agent's only "per-agent" secret today
is read under the HIVE's certificate, through a wide grant on
`swarm/agents/*` — so "per-agent" was presentational.
The swarm now mints the certificate, so no hive ever needs the capability
to mint one. `swarm-controller` is the service that does it: it already
logs in to the store, and its existing grant already covers exactly the
three objects written here (`create/update` on
`secret/data/swarm/agents/*`, `sys/policies/acl/hive-*` and
`auth/cert/certs/hive-*`). No new bao grant, and nothing co-located — a
cert-auth role pins its authority by value, per role, so the controller
issues from its own CA on its own host and pins that CA in the role it
writes. No existing role changes.
The mint node does not report success on a write. After publishing it
connects again, with the leaf it just issued and under the role it just
wrote, and reads the path back — so the policy, the role, the common name
and the leaf are exercised in production on every agent creation. A
certificate this code mints that the role this code writes will not accept
turns the job node red at creation time instead of surfacing later as an
agent container that cannot start.
`TriggerDeploy` gains an `after_any` edge on the mint, not `after_ok`: a
hive cannot pass down a certificate the swarm has not published, but a
host with no authority configured must still create agents exactly as it
does today.
The private key is generated in memory and never written to disk on the
controller — `SecretStore::connect_with_identity` takes the PEM the minter
is already holding, so nothing is written out purely to be logged in with.
Refs #4137
mara: "the subagent should already stamp the agent on the metric. so i want
to see count of distinct subagent names per agent name over time" — it
does. New timeseries panel-46 on the subagents tab: one line per parent
agent, counting distinct subagent label values with a turn in the trailing
$__rate_interval at each point. Rolling, not cumulative — can fall as well
as rise, same shape as any other rate()-derived panel here.
Placed at y:15, below where #4491's panel-45 (stat tile, y:11-15) sits once
merged, and numbered panel-46 rather than reusing panel-45 — argus caught
that this PR and #4491 both independently added a panel-45 in the same grid
slot from a shared unmerged base, which would have collided on whichever
merged second.
Narrower than a true concurrency gauge (subagents open at this exact
instant) — that needs hive-subagent-mcp to publish its own metric, which
nothing does today; noted as a separate, harder ask on #4488.
Bargauge panels display series in the order the datasource returns them —
Grafana has no native sort for that (long-standing upstream gap, grafana#17245).
The reliable fix is sorting at the PromQL level with sort_desc() on an
instant vector, which forge.json's 'Issues by label' panel already does.
Applies the same pattern to the other 14 multi-series bargauge panels across
agents.json, claude-usage.json, and authelia.json (the latter two also
needed instant: true added, since sort_desc only works on instant vectors).
logstore.json's two VictoriaLogs-backed bargauge panels use LogsQL, not
PromQL, and are left out of this pass — LogsQL's sort-after-stats has a
documented ordering bug in some versions; needs its own verification.
mara: "show subagent count in time range" — the existing 'Subagent turns'
tile answers volume (how many turns), not this (how many differently-named
subagents ran at all). New panel-45 counts subagent label values with at
least one turn in the selected range.
readOnly makes a bad socketPath inexpressible rather than diagnosed
after the fact. Since readOnly rejects any definition including one
arriving through a rename shim, drop the deploy.nix rename entry for
it and the fixture line exercising it, and update the controllerOldPath
comment's shim count from seven to six.
Refs #4208
The socket is 0666 (nginx needs to connect(2) as a different user), so its
directory is the only thing bounding what the gateway's nginx can reach —
the crate's README already says so. A unit test pins DEFAULT_SOCKET, but
that only catches a developer editing the const; an operator setting the
documented socketPath option to a shared dir (notably /run/hyperhive,
which holds the host admin socket, or /run/hive) went unenforced.
Denylist rather than the general "not any other module's socket dir"
form — narrower, but it's what the README names and lands without pulling
in other modules' constants.
Refs #4208
First slice of the move off the hand-rolled swarm services sub-CA: the
store gains the three objects the eventual minter needs, and nothing
else.
- a `pki` secrets mount, enabled ask-first off the same `bao secrets
list` snapshot the kv-v2 mount beside it uses, so a rebuild that finds
it mounted does nothing;
- a `swarm-services` role on it, `allowed_domains` read straight out of
`swarm.serviceDomains` — the same swarm-tier list swarm-ca.nix
name-constrains its sub-CA to and hive-tls.nix carries as the leaf's
SANs — with subdomains, localhost, globs and IP SANs all off;
- a `swarm-services-issuer` policy granting `update` on
`pki/issue/swarm-services`.
Inert: the mount has no issuer generated into it, the role therefore
cannot issue, and no login role attaches the policy, so no token in the
swarm carries it. No consumer changes; swarm-ca.nix still mints the
services sub-CA exactly as before. Reversible with
`bao secrets disable pki`.
The cert-auth role that attaches the policy waits for the leaf carrying
its CN, which glue-bao-tls.nix mints — a later step.
Refs #4256
An agent can reach VictoriaLogs only through the gateway, and since the
machine query route landed the way to read it has been to hand-roll a
client_credentials token request and a curl, per query. This is the CLI
that closes that: `swarm-logs query '<LogsQL>'`, matched log lines on
stdout, so the answer pipes into grep like any other command's.
Built to the plan posted on the tracker thread: own crate, own
docs/tools reference generated off the clap tree, `query` as the one
verb, and the JSON error body surfaced on a non-200 rather than
swallowed. No `tail`: streaming is a different endpoint with a different
response shape, and folding it in here would be a fatter scope than the
ask.
Minting the token is NOT implemented here — swarm-queue-client already
owns the client_credentials request, its error type and its CA handling,
and a token-endpoint fix has to be findable in one place. What this crate
adds is the agent-shaped half: the client id arrives as a *file* beside
the secret, so nothing outside nix/agent-modules/queue.nix spells
`hive-<name>-agent` twice. That is the same problem hive-agent's
swarm_queue module solves, and swarm-logs/src/auth.rs is its `decide`
restated over this binary's inputs.
⚠️ The plan named one thing to verify empirically before calling the auth
settled: whether authelia's bearer policy for the logs vhost accepts the
agent client's audience. Measured from inside a container: it does not.
The client minted a token fine but with `aud: []` and `scp: []`, asking
for the logs URL as an audience answered `invalid_target`, and presenting
the audience-less token to the gateway answered a bare 401. So
swarm-authelia.nix's agentClients gains `authelia.bearer.authz` and the
query URL as a second audience — authelia authorises a bearer token by
the URL being requested, and that URL is now one binding read by three
places rather than three spellings of one address.
The URL reaches an agent the same way its queue coordinates do: computed
on the host (a container cannot derive a gateway address), forwarded by
hive_c0re::meta into the container's option set, and consumed by a new
agent module that installs the binary *wrapped* with its coordinates —
the shape swarm-controller.nix installs swarmctl in. Gated on the queue
credential as well as on the URL: a binary that can only answer 401 is
worse than no binary, because an agent reads a 401 as "no logs", which is
the exact confusion the store's machine route was added to end.
Adds services.hyperhive.deploy.swarm-controller.matrixHomeserverUrl,
threaded to the daemon as SWARM_CONTROLLER_MATRIX_HOMESERVER_URL, and a
Rust helper (homeserver_or_configured_default) that lets a caller-supplied
homeserver keep overriding it. Config plumbing only: put_matrix_account
does not call the helper yet, so this is a no-op for every current caller.
Refs #4345
Adds approle to the auth methods enabled during swarm-bao bootstrap,
using the same idempotent check pattern as cert. No role or policy
is configured yet — those are handled in later slices.
Refs #4386
`glue-matrix-bao-token.nix` has read
`secret/swarm/hives/<hive>/matrix/appservice-token` since it landed, but
nothing ever wrote that path. The store was empty in every deployment, so
every read degraded to "keep what activation minted" and each hive stayed
the origin of a value the swarm has to agree on — two hives never
converged.
`swarm-secret-publish` is now the producer. It already holds a store
identity, already writes under the hive prefix, and already runs per
hive in the roster, so the mint is a third loop beside the two OIDC
copies rather than a second shape of this unit.
Idempotence comes from a record of its own, not from the store: this
principal is granted `create`/`update` with no `read`, so it cannot ask
whether a hive already has a token. It keeps what it minted under
`StateDirectory=` (0700 dir, 0600 file) and mints only when that file is
missing or empty; the `put` runs every time, because re-putting the same
bytes changes nothing for a reader while a mint whose publish failed must
not be left as a token this host holds and no hive can reach.
The token never becomes a nix literal and never reaches argv: the mint
redirects into a file, and the publish hands bao `value=@<path>` so bao
opens it itself — the same handling the OIDC loops use.
`hive-matrix.nix`'s activation mint stays as the genuine first-boot
fallback. It already fires only when the token file is absent, so it
cannot clobber a value the store delivered; `hs_token` has no swarm half
and is still minted there for real.
Refs #4402
Nothing reads it any more: hive-c0re creates accounts as the hive's
appservice, so the mint, the host file, the bind mount, the
`LoadCredential` entry and tuwunel's `registration_token_file` all go.
⚠️ `allow_registration` has to go to `false` in the same change, and not
as hardening. tuwunel refuses to START when registration is allowed with
no token configured — it demands
`yes_i_am_very_very_sure_…_open_registration_…` instead — so dropping the
token and leaving the flag true is not a lax homeserver, it is one that
does not boot. The flag is checked only for requests arriving without an
appservice token, so hive-c0re provisions exactly as before and everyone
else is refused outright.
The swarm secret store keeps its role, repointed at the credential that
replaced the token (`swarm/hives/<hive>/matrix/appservice-token`). Its
unit now also re-runs hive-matrix's own registration renderer after
writing the file: the token is half an agreement, and a registration
still naming the previous value authenticates nobody. The renderer is
shared through an internal option rather than copied, so the
registration's shape has one home.
Both spellings of `registrationTokenFile` become
`mkRemovedOptionModule` with a message naming what replaced them. A hive
that never set the option — the default — is unaffected; one that pinned
it fails to evaluate with instructions instead of a silent no-op.
An upgraded hive needs no intervention: the activation script has both
halves in place before the homeserver restarts, existing agents keep the
tokens their devices already hold, and the old token file is left on
disk read by nothing. docs/integrations/matrix.md spells the path out.
Refs #4402
The hive creates matrix accounts with a shared registration token today,
which means the secret that authorises account creation has to reach both
hive-c0re and tuwunel and stay identical in both. An appservice
registration replaces that with an identity: one token that says "this is
the hive's provisioner", carried in an ordinary credential file.
`url = null`, so nothing is served and no daemon is introduced — with no
URL the homeserver never calls out, and the registration exists purely to
give the `as_token` meaning.
Delivered through `appservice_dir` rather than a `[global.appservice.<id>]`
stanza, because a stanza's `as_token` would be a nix literal and a nix
literal is a world-readable store path. The file is minted and rendered by
a host activation script, bind-mounted into the container, and handed to
the homeserver by `LoadCredential` — the same two steps the registration
token and the OIDC client secret already take, and for the same reason
(0600 root on the host, `DynamicUser=true` in the container).
`sender_localpart` is the hive admin account on purpose: loading a
registration creates its sender user on a zero-user database inside
`Services::start()`, and the `admin_execute` promotion runs after that and
still before the HTTP listener accepts anything. So a fresh homeserver has
a joined, power-level-100 admin on its first boot without anyone having
won the first-registered-user grant. `admin_execute_errors_ignore` is set
because a failing startup command otherwise aborts startup outright.
Nothing reads the registration yet — hive-c0re still provisions through
the registration token, which is untouched here.
Refs #4402
`swarmctl agent create <name> --hive <hive>` POSTs `/api/agents` to
swarm-controller over the daemon's unix socket and prints the queued
job's node id.
It deliberately does not wait. The endpoint queues a DAG whose last node
*publishes* a deploy message; the hive's `hive-c0re` then converges on
its own clock, out of the controller's sight. So even a fully settled
graph would not mean the agent is up, and there is nothing this CLI
could wait for that would let it claim otherwise. Printing the id is
exactly what the response says and all of what it says.
Transport is a bare hyper HTTP/1.1 client handshaked onto a tokio
`UnixStream` via `hyper_util::rt::TokioIo` — the same crate family
`hivectl/src/watch.rs` and `hive-agent/src/web_ui/proxy.rs` already use,
all of it already workspace-pinned. The request/response shapes are a
local mirror rather than a shared crate: the controller's own types are
private to its binary and this crate does not link it, the same
separation `hivectl` keeps from `hive-c0re`.
Errors are reduced to one actionable line — the controller answers
RFC 9457 problem+json, so an unknown `--hive` reaches the operator as
the roster of hives that would have worked rather than a body dump.
Response `warnings` are printed when non-empty.
The nix module wraps the binary with `SWARM_CONTROLLER_SOCKET`, read
from the same `socketPath` the daemon binds.
Refs #4399
The swarm can already tell whether an agent is alive — the `agent-status`
KV bucket republishes once a minute — but not what it is doing right now.
A header bar wants the second thing, and a minute-old answer to "is this
agent thinking" is the wrong answer most of the time it is read.
`hive-agent` now publishes a turn-state header to
`$SWARM.agent-state.<hive>.<agent>`, a core subject beside the terminal
rows it already sends. It goes out **on transition, not on a timer**: the
publisher watches the event bus, rebuilds the header, and sends only when
the serialised result differs from the last one it sent — so a second
periodic writer, which is the problem this exists to fix, is not what
replaces the bucket.
The payload is the published contract a swarm-level renderer is written
against, so the test asserts on the serialised JSON keys rather than on
Rust field names. Two fields deliberately depart from the per-agent web
UI's `StateSnapshot`: `turn_state_since` is an ISO 8601 UTC string rather
than unix seconds, matching the sibling `$SWARM.term` subject's stamp, and
`agent_state` carries the swarm's own `AgentState` vocabulary rather than
a `paused` boolean, so a reader can compare actual against wanted without
translating. `turn_state` and `agent_state` stay two separate fields:
neither vocabulary contains the other's values.
Swarm-side, `GET /api/agents/{name}/state/stream` relays the subject as
SSE, resolving the agent's hive at request time exactly as the terminal
stream does and passing the bytes through without parsing them.
The broker grant is a second `--agent-publish-subject` rather than a
widening of the existing one, so the terminal family and the header family
stay independently revocable, and a `module-eval` arm pins the rendered
flag and its argument together — the doubled dollar included, since a
single one expands to nothing in `ExecStart` and yields a grant that
matches nothing.
Refs #3802
The swarm collector's OIDC client secret only existed where authelia
did: `swarm-otel-oidc-secret.service` copied the minted plaintext out
of authelia's container tree, reachable only because the two share a
host's network namespace. A swarm that placed authelia elsewhere
delivered nothing, and the option's own description said so —
"a deployment that places authelia elsewhere points this at a file it
delivers itself." Same gap as #3853 and #4234, and this is the
swarm-otel twin of #4234's fix for Grafana.
Mirrors PR #4361 (Grafana) almost exactly:
- `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, in every
deployment, replacing the co-located copy unit outright — one
delivery route, not two, per the ruling that landed under #4234.
- Client registration moved out of `swarm-otel.nix`'s own `config`
block (gated on this host running the collector) into
`glue-swarm-otel-oidc-client.nix` (gated on this host running
authelia), the same split `glue-grafana-oidc-client.nix` made. It
was broken the same way: a split deployment registered the client
nowhere at all, so authelia never minted a secret for the publisher
to send on.
- The publisher's `services` prefix (write grant in `swarm-bao.nix`,
hive read grant in `policy::render`) already covers any service's
path — nothing to add there. `swarm-secret-publisher.nix` only grew
`serviceClientIds` by one entry.
One judgement call, stated rather than buried: the store-reading unit
renders only where this host holds a client identity
(`deploy.bao.clientCertFile`/`clientKeyFile`), rather than asserting
it the way `swarm-grafana.nix` does. Grafana's local login form is
disabled unconditionally, so a Grafana with no OIDC secret has no way
in at all — that earns a hard refusal. This collector without a
credential still receives every hive's telemetry; only its own pushes
to the stores go out unauthenticated and get refused there, an
already-supported degrade the module's own `haveCollectorSecret` flag
named before this change. So the reading unit follows the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use
for their own optional readers: no unit when the identity is absent,
not a build refusal.
Fixtures mirror #4361's: `otelBaoWithAuthelia`/`otelBaoRemoteAuthelia`
are the positive pair (co-located and split, both reading through the
store), `otelNoIdentity` is the negative — no reading unit, no
assertion firing, `clientSecretFile` left null.
Refs #4258
The v2-conversion script carried over the local-test dashboard's own
title ('agents-full-test') instead of the real one, since it read
spec.title straight from the generated ground-truth file without
overriding it back. Caught on a diff review, not by CI — nothing
schema-level flags a wrong-but-valid string.
Confirmed empirically (real Grafana 13.0.7, a file-based provisioner
identical in shape to swarm-grafana.nix's) that the deployed version
provisions v2-schema dashboards natively, tabs included, no migration
path needed. Supersedes the row-based split from the previous commit
on this branch: same four groups (activity/cost/health/subagents), same
17 panels with identical content (targets, fieldConfig, options,
descriptions all carried over unchanged — only the document shape
around them moved from a flat panels array + row wrapper panels to the
v2 schema's spec.elements map + spec.layout TabsLayout referencing them
by id), now rendered as real browser tabs instead of accordion rows.
Generated the panel-content half of this file mechanically rather than
hand-authoring the new schema: posted the pre-conversion (v1, flat,
17-panel) dashboard to a local Grafana 13.0.7 and read it back through
the v2beta1 API, which is Grafana's own conversion of the exact same
panel content into the v2 element shape — then replaced only the
generated layout (a flat GridLayout) with a hand-authored TabsLayout
grouping the same element references into the four tabs. Verified the
result end-to-end against the same local instance: file-provisioned
(confirmed via the loaded object's own grafana.app/managedBy annotation),
and a real screenshot of each of the four tabs rendering with its
correct panels.
Cost, health (cpu/mem/disk), and subagents, plus an activity row for the
panels that fit none of the three (agent/turn counts, lines-of-code and
CLAUDE.md-size trends). Rows rather than tabs: Grafana's tab layout is a
v2/Scenes-schema feature, and this deployment's file-based provisioner
doesn't reliably load v2-schema dashboards, so classic-schema collapsible
rows are the closest available equivalent — one section open, the rest
collapsed. activity stays expanded so the dashboard opens on something
useful instead of a wall of accordion headers.
