New 'overview' tab on the agents dashboard, extending it per the
hive stats page's own summary layout (busiest-agents table + top KPI
row):
- KPI row: active agents, turns, tokens (all types), input, output,
cache-read tokens, cost. All from metrics already flowing into
this dashboard (claude_code.token.usage / claude_code.cost.usage /
claude_code.session.count) plus hyperhive.agent.turn.count, which
the harness already emits but nothing visualized yet.
- Busiest-agents replicated as five sorted bargauge panels (turns,
cost, input, output, cache-read, each by agent) rather than one
literal multi-column table -- this schema (Grafana v2beta1) has no
existing table+merge-transform panel anywhere in this repo to
pattern-match against, and every other panel here follows the
proven sort_desc(...) + bargauge shape already used for
'Cost by subagent name' (panel-43). Trading the exact table layout
for schema I can verify against a working precedent.
- Left out the stats page's 'window' tile -- redundant with the
dashboard's own always-visible time-range picker.
No new instrumentation needed. Structurally verified (valid JSON, no
duplicate panel ids, every panel referenced exactly once in the
layout) but not visually verified -- no local Grafana to render
against.
A swarm runs one homeserver and every hive on it logged in as the same
`@hive:` localpart, holding the same access token out of one swarm-wide
store path. That is one matrix identity for N hives: the homeserver
cannot attribute an action to the hive that took it, and revoking one
hive's standing revokes every hive's.
Three changes, and the third is the one that makes the other two real:
- **The localpart carries the hive's name** (`hive-<hive>`), derived in
one place, `swarm_secret_client::matrix::hive_localpart`.
`hive-matrix.nix` renders the same string as the appservice
registration's `sender_localpart`, so the shared account stops being
created rather than merely stops being used.
- **The store path is templated by hive**, not a constant. The
"a swarm runs one homeserver, so this is a constant rather than a
parameter" rationale went with it; it stopped holding the moment two
hives shared the homeserver it describes.
- **The path moved out from under the grant every hive has.** It sat at
`swarm/services/matrix/sender-token`, inside the
`secret/data/swarm/services/*` read stanza `policy::render` gives every
hive. It now sits under that hive's own stanza,
`secret/data/swarm/hives/<hive>/*`, which interpolates the reader's
name — so a hive reads its own token and is refused another's. The
policy renderer itself is unchanged: narrowing the `services/*` grant
would break the OIDC-secret read it exists for, and moving the
credential is what this needed instead. A policy test walks the
rendered stanzas and asserts none of hive alpha's covers hive beta's
sender token, so a later stanza that widened it fails here.
`swarm-matrix-ctl` takes a new required `MATRIX_MINT_HIVE` and writes
that hive's path; its store grant in `swarm-bao.nix` follows, scoped to
one hive's leaf via the new `deploy.bao.matrixCtlHiveName` (defaulting to
this host's `hiveName`) rather than a `hives/*` wildcard, which would
hand the matrix container every hive's token back.
Migration: no outage at deploy. `ensure_hive_user` short-circuits on the
local token file, so a hive keeps running on what it has; with no such
file it reads the new per-hive path, finds nothing, and falls through to
the existing register-or-appservice-login ladder against its own
localpart — which needs only the per-hive `as_token` on local disk. The
old shared object is read by nothing afterwards. Rooms do not follow the
identity, and that is the one operator step; both ways out are written
into `docs/integrations/matrix.md`.
No admin standing is granted to the per-hive accounts: `admin_execute`
stays empty and the assertion pinning it is untouched.
Renames `swarm-matrix-minter` and reshapes it around subcommands. Minting
is now `swarm-matrix-ctl mint`.
Running rust inside `containers.hive-matrix` is not free: it needs its own
store identity, its own cert role and its own bind mounts, and every one of
those is per-*container*, not per-task. A second single-purpose crate would
have had to duplicate that plumbing to add one action, so the next thing
that has to run in there should be a verb here rather than a new crate.
The old name guaranteed the opposite.
`main.rs` is clap dispatch; the minting logic moves to `mint.rs` unchanged.
A bare invocation is refused: `mint` writes a credential, so "no verb"
defaulting to it would make a typo in the unit mint rather than fail.
The environment prefix moves with it, `MATRIX_MINTER_*` → `MATRIX_MINT_*`.
Scoped to the verb and not to the binary, because a binary-scoped prefix is
one the next verb has to share or widen, and a widened one never narrows
again. A test asserts every variable carries the verb's prefix.
The principal renames too. The cert role, bao policy, granting unit, leaf
filename and `certAuthCns` entry all have to spell one string the same way,
so leaving them as `swarm-matrix-minter` would have rebuilt the naming
split this branch exists to remove. Renaming the nix options alongside is
free here: every one of them is introduced by this PR and has never been
released, so no operator config names them yet.
`ExecStart` now names the verb, which is a contract between a nix string
and a clap enum that fails at deploy time with no local signal. Both ends
assert it: `mint_is_spelled_the_way_the_unit_invokes_it` in the crate, and
a new module-eval arm reading the rendered `ExecStart`.
docs/getting-started/setup.md drops the sender token from its "live on the
host" list: setup does not touch this credential, so a setup guide has no
reason to name it.
The store path and every identifier around it called this an admin
token. It is not one: of ~15 hive-c0re call sites only two need
homeserver admin, and the homeserver no longer promotes the account at
boot, so the name overstated both what the credential is and what it may
do.
Renaming it to the account was not enough either. "The `@hive:` token"
reads as the token of a hive user, and no such user is provisioned —
`@hive:<server_name>` is the appservice registration's own
`sender_localpart`, an account the homeserver creates for itself when it
loads the registration.
So it is the **sender token**: the matrix appservice sender account's
access token, at `swarm/services/matrix/sender-token`. The name says
what it authenticates as rather than what it may do, which is the part
that was wrong.
The path has one constructor, and the bao grant, the grant assertion and
three unit tests pin its literal independently — so a half-finished
rename fails a check rather than leaving the minter and its readers
disagreeing at runtime. `tracing` messages are renamed with the code, so
the journal reads the way the source does.
The host-side file keeps its name (`matrix/access-token`): it carried no
admin framing, and renaming it would orphan the file on every deployed
hive for nothing.
`docs/tools/hivectl-cli.md` is regenerated from the clap tree.
Three leftovers from the rename, plus the gating prose job.
`nix/packages/default.nix` still described the minter as the "matrix
admin credential's minter", and shipped that claim in the package's
`meta.description` — a PR-visible string.
`promote_user_to_admin`'s doc comment pointed at the `admin_execute`
entry in `hive-matrix.nix` as the thing that bootstraps the first
admin. That entry is gone, so the comment referenced nothing. It now
records that the account is ordinary, that the call therefore has no
working sender, and that rehoming at swarm level is the fix rather than
re-granting. `reset_user_password` gained the matching warning; it had
none.
The prose fixes clear all 8 `CI / prose lint (vale, errors)` failures,
all of which were in docs this branch touches. No vale config change,
no exception, no carve-out: contractions, one recast sentence, one
de-hyphenation and one dropped "simply".
Tense: four docs described system behaviour in the future ("will
refuse", "will fetch and trust"). Reference docs get read from the
other side of the change, so they say what the system does.
A swarm runs one homeserver and a homeserver has one appservice sender
account, so "mint it once" is a property of the thing being minted
rather than something a lock has to enforce. That is what makes this
account the one to move first: no trigger route, no controller change
and no agent list — a boot-time oneshot beside tuwunel is the whole
mechanism.
`swarm-matrix-minter` runs inside `containers.hive-matrix`, which
already holds the appservice token: the rendered registration is bound
in read-only because that is how tuwunel is handed it. What the
container lacked was an identity of its own, so this adds one — a leaf
from the store's CA with a grant of exactly one path, not the hive's
leaf, which reads every secret in the store.
Both ends of the credential ship here. The minter reads the path it
publishes to before it touches the homeserver, and returning on a
non-empty read IS the "only once"; `hive-c0re`'s `ensure_hive_user`
reads the same path, authenticating with the hive name already in
`HYPERHIVE_HIVE_NAME`. The existing mint-then-`M_USER_IN_USE`-login
ladder stays as the fallback for a store that is empty, unconfigured or
unreachable, which is every swarm deployed before this — so nothing
needs backfilling and nothing breaks if the rest of the sequence never
lands.
The credential is not an admin credential, and is not named like one.
It is the access token of the appservice registration's own
`sender_localpart` — `@hive:<server_name>`, an account the homeserver
creates for itself when it loads the registration. The store path is
`swarm/services/matrix/sender-token`, the host path is
`matrix/access-token`, and the homeserver no longer runs an
`admin_execute` promotion for that account at boot. Everything the hive
provisions with it — the Space, the chat room, their hierarchy and join
rules, the invites — rides on being the creator of those rooms at power
level 100, not on homeserver admin; there is no Synapse admin API here
to need, tuwunel has none.
Two operations do need an admin *sender* and therefore stop working:
`hivectl matrix promote-user` and `hivectl matrix reset-password`, both
`!admin …` messages into `#admins:<server>`, plus the password-reset
recovery path that an agent with a lost password file falls back to.
They are swarm-level operations and are left failing loudly rather than
served by an over-privileged token every other call site would also
carry. The sweep's own admin-rights check and self-repair go with them:
an account that is deliberately not an admin has nothing to check.
`ephemeral = false` stays, and hive root can still read the container's
filesystem. Accepted: what this buys is identity separation — no hive
*process* holds or reads the appservice token — not physical isolation.
Refs #4345
'Rows by unit' and 'Rows by host' use VictoriaLogs LogsQL, not PromQL,
so the sort_desc() fix from the PromQL panels doesn't port directly.
Verified live against the real store (swarm-logs query) that
'| sort by (rows) desc' after 'stats by (...)' orders correctly on our
VictoriaLogs version -- the VictoriaMetrics/VictoriaLogs#258 sort-order
bug doesn't reproduce here. Append the sort clause to both panel
queries.
The 41-line comment on logLevelRules tripped the comment-block lint
(30-line max). Move the detailed walkthrough (why each rule shape is
what it is, the three silent-failure modes, the query a level button
emits) into docs/scheduler/observability.md's existing 'Log severity'
section, which already covered the mapping at a higher level. The nix
comment now carries the short why/contract and points at the doc for
the full detail — no information dropped, just relocated.
Grafana's log-level buttons filter on a field called `level`; no row in
the store has one. The store's name for it is `severity_text`, chosen by
VictoriaLogs' OTLP ingester rather than by us — v1.52.0's
`app/vlinsert/opentelemetry/pb.go` writes it unconditionally and the
ingest parameters have no `_level_field` to rename it with. So the
mapping is made on the reader: the VictoriaLogs datasource, which was
provisioned with no `jsonData` at all, now carries a `logLevelRules`
entry per severity the journald parser can emit.
`logLevelRules` is the datasource plugin's only level-related jsonData
key — there is no field-name setting and no OpenTelemetry preset to
switch on. It is read off `instanceSettings.jsonData` in the plugin's
`datasource.ts` and typed in its `configuration/LogLevelRules/types.ts`,
both recovered from the sourcemap shipped in the pinned artifact
(`grafanaPlugins.victoriametrics-logs-datasource` 0.26.3). Each enabled
rule appends an `OR severity_text:="INFO"`-shaped term to the query a
level button emits, next to the `level:…` term that matches nothing.
A wrong rule here fails silently: Grafana provisions unknown jsonData
without complaint and the buttons go on returning zero rows. The three
ways to get one wrong — a non-literal `enabled`, a non-canonical `level`
spelling, a value that is not the stored text — are recorded at the
binding, and a module-eval arm pins them along with the real failure
mode, a severity added to `nix/journald-severity.nix` and not here.
`Unspecified` is left unmapped on purpose: it is the store's own
rendering of an absent severity and the thing the logs dashboard's "no
severity" panel counts.
Refs #4560
Slipped past the #4574 lint — 1473 chars vs. the 100-char cap, caught
by pre-push while working on an unrelated branch. Same treatment as
every other panel: state what the number is, not the reasoning.
Cut every panel/board description down to a short statement of what the
number is. Drops the design-reasoning prose, edge-case notes, and
operator-attributed quotes that had accumulated in an earlier pass —
useful context while building the boards, wrong shape for a tooltip an
operator glances at mid-incident.
Both journald receivers import one PRIORITY mapping, and each tier's
suite was asserting the whole table against its own receiver. That checks
one file twice: invert the table and two cases fail saying the same
thing, which tells you nothing about which of the two possible defects
you have.
Split by subject instead. The table's contents — the inverted direction,
overwrite_text, parse_from/on_error — belong to the file that holds them,
so they get a suite of their own reading that file directly, with no
fixture at all. Each tier keeps a case, reduced to the question only it
can answer: does MY receiver carry the shared mapping.
Both tier cases stay. They cover different receivers over different
journals — the agent container's own and the swarm collector's host
journal — and one tier quietly losing its parser while the other keeps
one is exactly the half-fixed state worth catching.
Membership rather than equality of the whole operator list, so a tier
that later grows an unrelated operator of its own still passes.
Checked against seven defect scenarios: each fails exactly one case, and
names the right one.
The direction is the part a reviewer cannot check by looking, so it is
asserted at both ends of the table and in both tiers' groups: an inverted
mapping still maps every value to something, and a case that only asks
whether a severity parser exists passes on the exact defect. The reader
that turns a rendered operator list back into a PRIORITY -> name function
lives in lib.nix, since both tiers need it.
The panel is asserted on its query rather than its title, because a panel
that keeps the title and loses the expression renders an empty graph that
looks exactly like zero prioless lines.
The mapping above needs something that says whether it is still working
after whoever wrote it has gone. A timeseries rather than a stat, so a
regression is a line lifting off zero rather than a number nobody reads.
Two series, and the split is the point: rows that carried a PRIORITY and
arrived with no severity anyway (a broken mapping — this one must reach
zero and stay there), against rows that never had a priority to map. The
latter is Claude Code's own OTLP telemetry, which emits log records with no
severity set at the source; mapping cannot reach it, so it is named rather
than folded into one number that never goes to zero.
Needs no provisioning change — logstore.json is already in the shipped
dashboard list.
Records reached VictoriaLogs carrying the journal's raw PRIORITY and
severity_text "Unspecified" — every line in the store, at every tier, with
no level a query or a dashboard could read. VictoriaLogs has no ingest
parameter naming a level field; it auto-detects one by field name, so the
mapping has to happen in the collector.
A stanza severity_parser on each journald receiver, from one shared file
rather than a copy per tier: the two receivers are unrelated config (a
fixed stanza in the agent container, a parameterised block inside the
swarm-otel container) and a drifted copy fails silently — every line still
arrives, labelled as the wrong thing.
Two details that are easy to get wrong and quiet when wrong. PRIORITY
counts down in urgency where the OTEL severity counts up, so the table is
written as a table. And overwrite_text is required: without it the parser
sets the severity number and leaves the text as the raw digit, so
severity_text arrives as the literal "6" — populated, and not a level
anything renders.
The single module-eval derivation forced ~62 full nixosSystem
fixtures live at once to compute its cases list: 10.6GB peak RSS /
5m25s to evaluate, by far the dominant cost in nix flake check.
Splits it into 21 independent checks.module-eval-* derivations
(1-7 fixtures each) sharing builders/helpers via module-eval/lib.nix,
so no single derivation needs more than a handful of fixtures live
at once. A few cases spanning two clusters carry a small duplicated
fixture rather than threading shared state through lib.nix.
mkRenamedOptionModule/mkRemovedOptionModule are nixpkgs' own tested
machinery; proving each call site resolves is redundant with what the
library already guarantees. Removes 10 of the ~79 full nixosSystem
fixtures (wireguard/forge/matrix/controller/ui/stores/statusPublish/
peers old-path + matrixMovedAppserviceToken/matrixRemovedToken) and
their 15 dedicated cases. Fixtures still exercised by genuine
behavioral cases (natsOldPath, autheliaOldPath, grafanaOldPath) are
kept, minus their now-redundant rename-only case.
Two swarm-wide facts were being read off this machine's deploy set, so the
answer differed between two hosts of one swarm:
- `swarm.authelia.oidc.hiveIdentities` defaulted to `deploy.nats.enable`,
so whether a hive gets an identity at all depended on whether the IdP
host happened to also run the queue. It is on by default now: a swarm's
hives have identities, and the clients are inert until used.
- `swarm.statusPublish.tokenEndpoint` defaulted through `queueLocal`
(`deploy.nats.enable && deploy.authelia.enable`), so a hive that was not
the swarm host had no token endpoint even when the swarm's IdP was
reachable and named. It follows `swarm.authelia.url` now — the same
derivation `swarm-controller.nix`'s own `queue.tokenEndpoint` already
uses, which is correct for a remote provider.
`deploy.nix:1-30` is what makes this a rule rather than a preference:
`swarm.*` is "identical on every host, byte for byte" and `deploy.*` is
"necessarily different on every host". A swarm value derived from a deploy
value cannot satisfy both.
The all-or-nothing status-publish assertion follows: the token endpoint is
no longer one of the coordinates that says this hive publishes — every hive
in a swarm with an IdP has one — so the two per-host coordinates are what
must agree, and they now require the endpoint rather than being counted
beside it.
`queueLocal` itself stays for the three remaining host-local addresses
(`natsUrl`, `clientSecretFile`, `agentNatsUrl`): each of those is a
`deploy.*` value that genuinely differs per host.
Closes#4048
Fix wording flagged on #4521 review: "Single nginx in front of every
hyperhive web surface" and "Runs on the host" claimed a deployment-wide
topology. There's one nginx per host that has something on it, and
potentially more inside service containers. Reword to describe what
this module builds on this host, with no count claim swarm-wide.
`services.hyperhive.gateway.enable`, `gateway.dns.enable` and
`network.enable` replace the `hyperhive.enable` gate on all three. Each
defaults to false; the modules that need one assert it with `mkDefault
true` from inside the guard their own deployment already carries, and
`swarm-required-services.nix` — the module that owns what the
swarm-services toggle implies — asserts all three explicitly.
hive-c0re asserts all three unconditionally, so an ordinary hive keeps
getting them with no opt-in: it is the host's only knowledge that agent
containers exist.
The resolver moves to its own `hive-gateway/dns.nix` so it can be gated
without reindenting the nginx half of the module.
Reinstates `network.enable`, dropping its `mkRemovedOptionModule` shim.
A config still carrying `network.enable = false` from before the removal
now switches the bridge off instead of failing eval.
Also deletes a duplicate `centralToggleOff` fixture in nix/module-eval.nix.
Two sibling slices added it independently (c5f60fd5, ce3b3d94); the merge
was textually clean and left `main` failing to evaluate at all, so this
file could not be gated without removing one.
The rebase onto main resolved an additive conflict in
module-eval.nix by keeping both fixture blocks, leaving
centralToggleOff defined twice and failing nix flake check with
'attribute centralToggleOff already defined'. Drop the later
duplicate; the earlier definition already covers every downstream
case that references it.
`systemd.paths.forge-avatar-sync` was gated on `agent.icon != null` alone,
while the `systemd.services.forge-avatar-sync` it triggers is gated on
`agent.icon != null && agent.forge.url != null`. An agent with an icon and no
forge URL therefore rendered a `.path` unit, pulled into multi-user.target,
watching for a forge-token whose arrival would activate a unit that does not
exist.
The module already documents the fixed behaviour: `forge.url`'s own option
description says the tea-login and avatar-sync units are "not generated at all"
when it is null --- an absent integration, never a misdirected one. That
sentence was true of the oneshot and false of its watcher.
Latent, not live: hive-c0re renders `forge.url` into every agent's config from
the host's `HIVE_FORGE_URL`, so on a real hive it is always set and the
asymmetric arm is unreachable. It is reachable wherever the agent modules are
evaluated outside a hive.
A module-eval case pins both halves absent for an agent with an icon and no
forge; it fails on the parent commit, where the path unit renders.
The receiver was defined for `scrapeTargets != {} || publishedScrapeTargets
!= {}`, but the swarm-tier pipeline named it only for `scrapeTargets != {}`.
A collector with published targets and no loopback ones therefore rendered
scrape configs that belonged to no pipeline: requested, parsed, delivered
nowhere, and valid enough to deploy.
Latent, not live: the module seeds `scrapeTargets.collector` under its own
`enable`, so the loopback set is never empty in a real deploy and the
disagreeing arm is unreachable today. Nothing would have noticed if that
seeding became conditional, which is why the module-eval case pins the
pipeline rather than trusting the seed to keep masking it.
Refs #4511
The base set (skill-creator + base@hyperhive) was declared via the
option's `default`, so a per-agent definition of claudePlugins
replaced it wholesale. Move the base set to a plain `config`
definition instead: a listOf option merges multiple plain definitions
by concatenation, so an agent's own list now adds to the base set
rather than replacing it, while lib.mkForce / lib.mkOverride on the
agent side still replace the whole merged list deliberately (mkDefault
was ruled out explicitly).
Also de-dup at the JSON-render site with lib.unique, so an agent that
names a base-set entry itself doesn't get it installed twice, and
reword the option doc, which still claimed the old REPLACES semantics.
Four module-eval cases cover the unset / agent-adds / mkForce-replaces
/ duplicate-entry shapes.
Refs #4467
`services.hyperhive.enable` is going away (Refs #4500). These four blocks
ANDed it with a second condition that was already the load-bearing one:
the bridge-firewall hole asks whether an operator named any
`network.exposeHostPorts`, and the three swarm-level services ask their
own `deploy.*.enable`, each of which defaults false (swarm-ui derives
from the controller's toggle, a sibling deployment decision) and none of
which is derived from the hive toggle. So dropping the conjunct turns
nothing on by itself — the remaining condition still decides.
The way that claim fails is by something becoming unconditional, so
module-eval gets absence arms for all four, each asserted on a host with
the hive ON and on one with it OFF, plus the two controls that make the
absences mean something.
The swarm-controller arm is probed by the credential oneshot and by the
daemon's `ExecStart` rather than by the unit name: hive-tls defines an
environment key on that name, which leaves an inert fragment behind on
any hive with a CA whether or not the controller runs there.
hive-forge's publicUrl and behindGateway, and swarm-controller's
forgeTokenFile, no longer gate their default on
config.services.hyperhive.enable — none of the three has a reader that
depends on hyperhive being enabled to make sense of the value, so the
extra condition only added a hidden coupling. module-eval.nix gains a
centralToggleOff fixture plus four cases asserting each affected
default now resolves identically whether the toggle is on or off.
Refs #4500
`swarm-controller` mints an agent's mTLS leaf at creation and publishes it
at `swarm/agents/<agent>/bao-mtls`. Nothing read it back. This adds the
hop that carries it the rest of the way, and the in-container consumer
that proves the hop works.
Host side, `lifecycle::agent_identity` reads the row under *this hive's*
own certificate — the hive is a principal the store already knows — and
stages the leaf and its key `0600` under a new `agent-identity/<name>`
state dir, deliberately outside every bind-mounted tree. Both files go in
as systemd credentials rather than binds, the same answer and the same
mode reason as the queue secret beside it: the staged key is unreadable
to the unprivileged agent user, and the container manager reads a
`--load-credential` source as root before re-exposing it under the
consuming unit's own `User=`. The agent is never asked to authenticate in
order to obtain the thing it authenticates with.
Container side, `hive-agent-bao-identity.service` logs in with that
certificate and reads the agent's own path back, failing the unit when
either step does not succeed. It fails loudly where the hive-side readers
degrade quietly, because a refused certificate means an agent that
believes it reaches the store and never does — a cause only the login
itself can name.
The address is the whole switch, no separate `enable`, matching how
`queue.nix` and `logs.nix` already gate themselves. A hive with a store
forwards `HIVE_AGENT_BAO_ADDR` and every agent on it gets the check; a
hive without one forwards nothing and no agent does. That is what keeps
the delivery from landing in a container with nothing to read it.
The hive can now reach an agent's identity, so hive privilege covers
agent privilege. Accepted, not mitigated: the alternative is an agent
fetching its own credential with a credential it does not yet have.
Refs #4137
`swarm/agents/<agent>/bao-mtls` did not exist, and neither did any
per-agent identity at the secret store: `policy::agent_object_name`,
`render_agent` and `render_agent_with_queue` had been written and never
called outside their own tests. An agent's only "per-agent" secret today
is read under the HIVE's certificate, through a wide grant on
`swarm/agents/*` — so "per-agent" was presentational.
The swarm now mints the certificate, so no hive ever needs the capability
to mint one. `swarm-controller` is the service that does it: it already
logs in to the store, and its existing grant already covers exactly the
three objects written here (`create/update` on
`secret/data/swarm/agents/*`, `sys/policies/acl/hive-*` and
`auth/cert/certs/hive-*`). No new bao grant, and nothing co-located — a
cert-auth role pins its authority by value, per role, so the controller
issues from its own CA on its own host and pins that CA in the role it
writes. No existing role changes.
The mint node does not report success on a write. After publishing it
connects again, with the leaf it just issued and under the role it just
wrote, and reads the path back — so the policy, the role, the common name
and the leaf are exercised in production on every agent creation. A
certificate this code mints that the role this code writes will not accept
turns the job node red at creation time instead of surfacing later as an
agent container that cannot start.
`TriggerDeploy` gains an `after_any` edge on the mint, not `after_ok`: a
hive cannot pass down a certificate the swarm has not published, but a
host with no authority configured must still create agents exactly as it
does today.
The private key is generated in memory and never written to disk on the
controller — `SecretStore::connect_with_identity` takes the PEM the minter
is already holding, so nothing is written out purely to be logged in with.
Refs #4137
`services.hyperhive.agent.matrix.enable` was a second source of truth for
a fact the account set already carried: after ①-③ the hive-internal
`main` account is an ordinary `matrixAccounts` entry, so "does this agent
have matrix" and "does this agent have an account" were the same question
asked twice, with the boolean able to disagree.
The option is gone and a non-empty `matrixAccounts` now gates the daemon
unit, its token path-watcher and the injected `extraMcpServers.matrix`
entry.
That is only a real condition because `matrixAccounts.main` is itself
gated: it is declared when `matrix.url != null`, never unconditionally. A
`main` with no homeserver is an account the daemon can never log in as,
so declaring one always would have made the signal trivially true and
turned matrix on for every agent in every hive. With the URL gate, the
empty set is reachable exactly for an agent the hive gave no homeserver
and whose operator declared no account of its own — the state the old
`enable = false` expressed.
Assertions: "extras require enable" is deleted, having become the
definition of the thing it checked (an external-only account with its own
homeserver is now rendered rather than rejected). `main.tokenFile` stays
pinned, re-guarded on `? main` instead of on the flag, since `main` is
absent whenever the URL is null and an unguarded index would throw there.
Both spellings of the option get `mkRemovedOptionModule`, following
../host-modules/deploy.nix's registrationTokenFile pair rather than a
silent delete: the definition whose meaning changes is `false`, and left
undeclared it would be ignored and hand the agent the tools its operator
turned off. Failing the eval with the replacement spelling is the only
outcome that cannot.
module-eval gains the three arms — URL, nothing, external-only — with the
middle one carrying why it exists: it is the only thing in the suite that
would notice `main` becoming unconditional again.
Refs #4475
mara: "the subagent should already stamp the agent on the metric. so i want
to see count of distinct subagent names per agent name over time" — it
does. New timeseries panel-46 on the subagents tab: one line per parent
agent, counting distinct subagent label values with a turn in the trailing
$__rate_interval at each point. Rolling, not cumulative — can fall as well
as rise, same shape as any other rate()-derived panel here.
Placed at y:15, below where #4491's panel-45 (stat tile, y:11-15) sits once
merged, and numbered panel-46 rather than reusing panel-45 — argus caught
that this PR and #4491 both independently added a panel-45 in the same grid
slot from a shared unmerged base, which would have collided on whichever
merged second.
Narrower than a true concurrency gauge (subagents open at this exact
instant) — that needs hive-subagent-mcp to publish its own metric, which
nothing does today; noted as a separate, harder ask on #4488.
`matrixAccounts` is meant to be the agent's full account list, but the
hive-internal `main` account was outside it: the nix module emitted only
the extras and `hive-matrix-daemon` prepended a `main` it synthesized
from the per-agent paths, with the option schema forbidding the name
outright.
nix/agent-modules/matrix.nix now declares `main` itself, as an ordinary
entry under `matrix.enable`, from the state-dir paths the module already
used for its token path-watcher (now a shared `stateDir` binding) plus
`matrix.url`. The whole set, `main` included, is serialized to
HIVE_MATRIX_ACCOUNTS.
accounts::configured therefore synthesizes `main` only when the parsed
list carries none, and otherwise takes the declared one verbatim —
hoisting it to index 0, since the daemon reads index 0 as the primary
and nix serializes an attrset, so `main` sorts wherever its key falls.
Declared xor synthesized: an agent whose harness predates this entry
keeps working, a current one gets its own, and there is no arrangement
where `main` is duplicated or missing.
The reserved-name assertion is replaced rather than dropped: the name
must now be legal (the module uses it), but `main`'s tokenFile stays
pinned to `<state>/matrix-token`, since hive-c0re provisions the
hive-internal token there and nowhere else — a retarget would evaluate
fine and then never restore. The other two fields are mkDefault and free
to override.
Refs #4475
Bargauge panels display series in the order the datasource returns them —
Grafana has no native sort for that (long-standing upstream gap, grafana#17245).
The reliable fix is sorting at the PromQL level with sort_desc() on an
instant vector, which forge.json's 'Issues by label' panel already does.
Applies the same pattern to the other 14 multi-series bargauge panels across
agents.json, claude-usage.json, and authelia.json (the latter two also
needed instant: true added, since sort_desc only works on instant vectors).
logstore.json's two VictoriaLogs-backed bargauge panels use LogsQL, not
PromQL, and are left out of this pass — LogsQL's sort-after-stats has a
documented ordering bug in some versions; needs its own verification.
mara: "show subagent count in time range" — the existing 'Subagent turns'
tile answers volume (how many turns), not this (how many differently-named
subagents ran at all). New panel-45 counts subagent label values with at
least one turn in the selected range.
Every per-agent harness option lived at the top-level `hyperhive.*` while
the host tier has always been `services.hyperhive.*`. Move all 52 agent-tier
option leaves (33 top-level names across 16 modules) to
`services.hyperhive.agent.*`, repoint every read, and keep existing agent
configs evaluating through one `mkRenamedOptionModule` per old leaf path in
the new nix/agent-modules/renamed-options.nix.
The shims are per leaf rather than per namespace: `user`, `mcp`, `otel`,
`queue`, `docs`, `forge`, `frontend`, `github`, `gui`, `logs`, `matrix` and
`cargo` are plain attrsets of declarations, not submodule-typed options, so
a parent-path rename would not reach their children. Three read-only
options (`frontend.mergedDist`, `queue.clientIdFile`,
`queue.clientSecretFile`) deliberately get no shim — a rename contributes a
definition, which a read-only option refuses; the exclusions are commented
in place.
Refs #4473
Same slice, same fact: socketPath lost its pre-rename path, so only three
old paths are defined in the fixture now, not four. Missed this sentence
while fixing the count three lines below it.
readOnly means socketPath can only ever hold the default, so the fixture's
old custom value can no longer reach the unit and the check.module-eval
assertion for it was failing. Assert the default instead of deleting the
check -- it still catches the env var vanishing or being misspelled, which
deleting it would not.
Also corrects the block's stale 'FOUR movers' count: socketPath lost its
pre-rename shim in the same slice, so only three of the asserted values
are still rename-shim movers.
readOnly makes a bad socketPath inexpressible rather than diagnosed
after the fact. Since readOnly rejects any definition including one
arriving through a rename shim, drop the deploy.nix rename entry for
it and the fixture line exercising it, and update the controllerOldPath
comment's shim count from seven to six.
Refs #4208
The socket is 0666 (nginx needs to connect(2) as a different user), so its
directory is the only thing bounding what the gateway's nginx can reach —
the crate's README already says so. A unit test pins DEFAULT_SOCKET, but
that only catches a developer editing the const; an operator setting the
documented socketPath option to a shared dir (notably /run/hyperhive,
which holds the host admin socket, or /run/hive) went unenforced.
Denylist rather than the general "not any other module's socket dir"
form — narrower, but it's what the README names and lands without pulling
in other modules' constants.
Refs #4208
First slice of the move off the hand-rolled swarm services sub-CA: the
store gains the three objects the eventual minter needs, and nothing
else.
- a `pki` secrets mount, enabled ask-first off the same `bao secrets
list` snapshot the kv-v2 mount beside it uses, so a rebuild that finds
it mounted does nothing;
- a `swarm-services` role on it, `allowed_domains` read straight out of
`swarm.serviceDomains` — the same swarm-tier list swarm-ca.nix
name-constrains its sub-CA to and hive-tls.nix carries as the leaf's
SANs — with subdomains, localhost, globs and IP SANs all off;
- a `swarm-services-issuer` policy granting `update` on
`pki/issue/swarm-services`.
Inert: the mount has no issuer generated into it, the role therefore
cannot issue, and no login role attaches the policy, so no token in the
swarm carries it. No consumer changes; swarm-ca.nix still mints the
services sub-CA exactly as before. Reversible with
`bao secrets disable pki`.
The cert-auth role that attaches the policy waits for the leaf carrying
its CN, which glue-bao-tls.nix mints — a later step.
Refs #4256
An agent can reach VictoriaLogs only through the gateway, and since the
machine query route landed the way to read it has been to hand-roll a
client_credentials token request and a curl, per query. This is the CLI
that closes that: `swarm-logs query '<LogsQL>'`, matched log lines on
stdout, so the answer pipes into grep like any other command's.
Built to the plan posted on the tracker thread: own crate, own
docs/tools reference generated off the clap tree, `query` as the one
verb, and the JSON error body surfaced on a non-200 rather than
swallowed. No `tail`: streaming is a different endpoint with a different
response shape, and folding it in here would be a fatter scope than the
ask.
Minting the token is NOT implemented here — swarm-queue-client already
owns the client_credentials request, its error type and its CA handling,
and a token-endpoint fix has to be findable in one place. What this crate
adds is the agent-shaped half: the client id arrives as a *file* beside
the secret, so nothing outside nix/agent-modules/queue.nix spells
`hive-<name>-agent` twice. That is the same problem hive-agent's
swarm_queue module solves, and swarm-logs/src/auth.rs is its `decide`
restated over this binary's inputs.
⚠️ The plan named one thing to verify empirically before calling the auth
settled: whether authelia's bearer policy for the logs vhost accepts the
agent client's audience. Measured from inside a container: it does not.
The client minted a token fine but with `aud: []` and `scp: []`, asking
for the logs URL as an audience answered `invalid_target`, and presenting
the audience-less token to the gateway answered a bare 401. So
swarm-authelia.nix's agentClients gains `authelia.bearer.authz` and the
query URL as a second audience — authelia authorises a bearer token by
the URL being requested, and that URL is now one binding read by three
places rather than three spellings of one address.
The URL reaches an agent the same way its queue coordinates do: computed
on the host (a container cannot derive a gateway address), forwarded by
hive_c0re::meta into the container's option set, and consumed by a new
agent module that installs the binary *wrapped* with its coordinates —
the shape swarm-controller.nix installs swarmctl in. Gated on the queue
credential as well as on the URL: a binary that can only answer 401 is
worse than no binary, because an agent reads a 401 as "no logs", which is
the exact confusion the store's machine route was added to end.
A unicast DHCP renewal reply currently reaches dhcpcd only by matching
the firewall's ESTABLISHED,RELATED conntrack rule against the outbound
request. When that conntrack entry has already expired the reply is
dropped silently, with no log line anywhere. The client's broadcast
paths (DISCOVER, rebind) bypass netfilter entirely via a raw BPF
socket and never depend on this state — only the unicast renewal path
does.
This removes that dependency by accepting DHCP client traffic
unconditionally, gated on the firewall being enabled at all. It does
not identify or claim to fix the cause of any particular observed
renewal failure.
Refs #3389
Adds services.hyperhive.deploy.swarm-controller.matrixHomeserverUrl,
threaded to the daemon as SWARM_CONTROLLER_MATRIX_HOMESERVER_URL, and a
Rust helper (homeserver_or_configured_default) that lets a caller-supplied
homeserver keep overriding it. Config plumbing only: put_matrix_account
does not call the helper yet, so this is a no-op for every current caller.
Refs #4345
The subagent daemon put `model` straight onto claude's argv with no
validation, so an agent could spawn nested sessions on any model the
operator had deliberately kept off its harness. Forward the existing
`hyperhive.availableModels` onto the daemon unit as
HIVE_AVAILABLE_MODELS (same rail `HIVE_TOOL_GROUPS` uses) and check
`start`/`continue` against it before building the config.
Default open: an absent var restricts nothing, so an agent deployed
before this keeps working. An omitted `model` is always allowed — it
lets claude pick its own default rather than naming one.
Refs #4436
Adds approle to the auth methods enabled during swarm-bao bootstrap,
using the same idempotent check pattern as cert. No role or policy
is configured yet — those are handled in later slices.
Refs #4386
`glue-matrix-bao-token.nix` has read
`secret/swarm/hives/<hive>/matrix/appservice-token` since it landed, but
nothing ever wrote that path. The store was empty in every deployment, so
every read degraded to "keep what activation minted" and each hive stayed
the origin of a value the swarm has to agree on — two hives never
converged.
`swarm-secret-publish` is now the producer. It already holds a store
identity, already writes under the hive prefix, and already runs per
hive in the roster, so the mint is a third loop beside the two OIDC
copies rather than a second shape of this unit.
Idempotence comes from a record of its own, not from the store: this
principal is granted `create`/`update` with no `read`, so it cannot ask
whether a hive already has a token. It keeps what it minted under
`StateDirectory=` (0700 dir, 0600 file) and mints only when that file is
missing or empty; the `put` runs every time, because re-putting the same
bytes changes nothing for a reader while a mint whose publish failed must
not be left as a token this host holds and no hive can reach.
The token never becomes a nix literal and never reaches argv: the mint
redirects into a file, and the publish hands bao `value=@<path>` so bao
opens it itself — the same handling the OIDC loops use.
`hive-matrix.nix`'s activation mint stays as the genuine first-boot
fallback. It already fires only when the token file is absent, so it
cannot clobber a value the store delivered; `hs_token` has no swarm half
and is still minted there for real.
Refs #4402
Nothing reads it any more: hive-c0re creates accounts as the hive's
appservice, so the mint, the host file, the bind mount, the
`LoadCredential` entry and tuwunel's `registration_token_file` all go.
⚠️ `allow_registration` has to go to `false` in the same change, and not
as hardening. tuwunel refuses to START when registration is allowed with
no token configured — it demands
`yes_i_am_very_very_sure_…_open_registration_…` instead — so dropping the
token and leaving the flag true is not a lax homeserver, it is one that
does not boot. The flag is checked only for requests arriving without an
appservice token, so hive-c0re provisions exactly as before and everyone
else is refused outright.
The swarm secret store keeps its role, repointed at the credential that
replaced the token (`swarm/hives/<hive>/matrix/appservice-token`). Its
unit now also re-runs hive-matrix's own registration renderer after
writing the file: the token is half an agreement, and a registration
still naming the previous value authenticates nobody. The renderer is
shared through an internal option rather than copied, so the
registration's shape has one home.
Both spellings of `registrationTokenFile` become
`mkRemovedOptionModule` with a message naming what replaced them. A hive
that never set the option — the default — is unaffected; one that pinned
it fails to evaluate with instructions instead of a silent no-op.
An upgraded hive needs no intervention: the activation script has both
halves in place before the homeserver restarts, existing agents keep the
tokens their devices already hold, and the old token file is left on
disk read by nothing. docs/integrations/matrix.md spells the path out.
Refs #4402
The hive creates matrix accounts with a shared registration token today,
which means the secret that authorises account creation has to reach both
hive-c0re and tuwunel and stay identical in both. An appservice
registration replaces that with an identity: one token that says "this is
the hive's provisioner", carried in an ordinary credential file.
`url = null`, so nothing is served and no daemon is introduced — with no
URL the homeserver never calls out, and the registration exists purely to
give the `as_token` meaning.
Delivered through `appservice_dir` rather than a `[global.appservice.<id>]`
stanza, because a stanza's `as_token` would be a nix literal and a nix
literal is a world-readable store path. The file is minted and rendered by
a host activation script, bind-mounted into the container, and handed to
the homeserver by `LoadCredential` — the same two steps the registration
token and the OIDC client secret already take, and for the same reason
(0600 root on the host, `DynamicUser=true` in the container).
`sender_localpart` is the hive admin account on purpose: loading a
registration creates its sender user on a zero-user database inside
`Services::start()`, and the `admin_execute` promotion runs after that and
still before the HTTP listener accepts anything. So a fresh homeserver has
a joined, power-level-100 admin on its first boot without anyone having
won the first-registered-user grant. `admin_execute_errors_ignore` is set
because a failing startup command otherwise aborts startup outright.
Nothing reads the registration yet — hive-c0re still provisions through
the registration token, which is untouched here.
Refs #4402
`build_config` now resolves a subagent's `--tools` from
`HIVE_TOOL_GROUPS`, the same var the harness resolves its own session
from — but the meta renderer writes that var onto the `hive-agent` unit
alone (`systemd.services.${service}.environment`), and the subagent
daemon is a separate unit. It would therefore have resolved the default
groups no matter what the agent was actually granted.
That direction is safe — the default groups add no built-ins, so the
resolution is a subset of the parent's either way, never a superset — but
it isn't what the code says it does: an agent granted `web_tools` would
spawn subagents silently without `WebFetch`/`WebSearch`, and the "same
set as the parent" property would be true only for agents whose groups
happen not to matter.
Forward the var onto the daemon's unit, read off the harness unit rather
than re-derived, so there is one place it is decided. Absent stays
absent: `or null`, which systemd drops from the unit, leaving the daemon
the same fallback the harness would take.
Refs #4416
