A record written to stdout carries no priority, so journald files the
whole stream at one level and the swarm log store shows `info` whatever
level `tracing` gave it. Under a systemd unit the process's stdout
already *is* the journal, so the fix is to speak the journal protocol
directly and let each record carry its own severity.
New `hive-log` crate holds the one sink chooser, called by `hive-c0re`,
`hive-agent` and `swarm-controller`. It builds the same `EnvFilter`
those binaries always built, then installs exactly one layer — never
both, since a journald layer stacked on the `fmt` layer under a unit
stores every record twice.
The choice is an fstat compare, not a presence test: a child inherits
`$JOURNAL_STREAM` even when its own stdout was redirected elsewhere, so
the variable existing proves nothing. The crate parses `dev:inode` out
of it and compares both numbers against an fstat of stdout, the
descriptor the `fmt` layer writes to by default. No match, unset, or
unparseable takes the `fmt` branch. A journald layer that fails to
construct despite a match falls back to `fmt` and warns through it —
a process must never fail to start because of its logger.
mara: 'i asked for a table. ask when doing something different.' Right
call — the PR body flagged the bargauge substitution as an open question,
not a decision, and I should have waited for an answer instead of
treating the silent absence of an objection as one.
Single 'Busiest agents' table panel (replaces the 5 per-metric bargauges):
5 table-format instant queries (turns, input, output, cache-read, cost)
joined on the agent label (joinByField), renamed to the screenshot's own
column names via organize, sorted by turns descending via sortBy — same
column set and sort order as the attached /stats screenshot.
This is a genuinely novel schema shape for this repo: no table panel,
transformation, or field-override config exists anywhere else in
nix/host-modules/swarm-grafana/dashboards/*.json to verify the join/
organize/sortBy option shapes against. Structurally verified (valid
JSON, jq empty, unique panel ids, every panel referenced exactly once,
nix fmt clean, dashboard-description lint clean) but the join/rename
field-name mechanics (whether Grafana names the joined columns exactly
'Value #A'/'Value #B'/etc.) are built from general Grafana schema
knowledge, not a working local precedent — flagging that explicitly so
the actual render gets checked before merge.
Reworked per review feedback: no new tab, and 3 of the added stat
panels duplicated existing ones (Active agents ~ panel-1 Agents
reporting, Cost = panel-2 Cost verbatim, Tokens ~ panel-3 + the new
input/output/cache-read breakdown). Removed panel-50/52/56 and the
overview tab entirely.
What's left is genuinely new (no prior panel covered it): a Turns KPI
and turns-by-agent bargauge in the activity tab (hyperhive.agent.turn.count
was emitted but unused anywhere), and input/output/cache-read token KPIs
plus cost/input/output/cache-read-by-agent bargauges in the cost tab
(the existing cost/token panels have no per-agent breakdown, and none
sort busiest-first).
New 'overview' tab on the agents dashboard, extending it per the
hive stats page's own summary layout (busiest-agents table + top KPI
row):
- KPI row: active agents, turns, tokens (all types), input, output,
cache-read tokens, cost. All from metrics already flowing into
this dashboard (claude_code.token.usage / claude_code.cost.usage /
claude_code.session.count) plus hyperhive.agent.turn.count, which
the harness already emits but nothing visualized yet.
- Busiest-agents replicated as five sorted bargauge panels (turns,
cost, input, output, cache-read, each by agent) rather than one
literal multi-column table -- this schema (Grafana v2beta1) has no
existing table+merge-transform panel anywhere in this repo to
pattern-match against, and every other panel here follows the
proven sort_desc(...) + bargauge shape already used for
'Cost by subagent name' (panel-43). Trading the exact table layout
for schema I can verify against a working precedent.
- Left out the stats page's 'window' tile -- redundant with the
dashboard's own always-visible time-range picker.
No new instrumentation needed. Structurally verified (valid JSON, no
duplicate panel ids, every panel referenced exactly once in the
layout) but not visually verified -- no local Grafana to render
against.
Both agent-facing tools go away end to end, with no replacement. This is
an intentional capability removal: agents can no longer enumerate their
own subtree, and can no longer queue a meta-flake input bump.
The system prompt and docs/tools/lifecycle.md land in this same commit
on purpose. A tool named in the prompt but absent from the server makes
agents confidently call something that doesn't exist, and the failure
then surfaces far from its cause.
Removed:
- MCP registrations and bodies (hive-agent-mcp), plus the now-unused
UpdateMetaInputsArgs.
- Wire variants Request::ListDescendants,
Request::RequestUpdateMetaInputs and Response::Containers, plus
ContainerInfo, whose only consumer was that response.
- hive-c0re's handle_list_descendants (its whole module) and
handle_request_update_meta_inputs, the two dispatch arms, and the
require_group(agent, "approvals", ...) gate on the meta-inputs verb.
- The stream_enrich emoji entry and argument formatter.
- docs/tools/lifecycle.md (both tools it documented are gone), its two
referrers, the tool-group tables and the agent-hierarchy prose.
Tool groups are kept, deliberately. ToolGroup::Lifecycle listed exactly
one tool and now lists none — it is vestigial, but the variant stays so
existing meta/capabilities.json grants still parse; retiring it is a
separate decision. ToolGroup::Approvals also listed exactly one tool,
but the group is NOT dead: check_can_cancel_approval still gates
cancel_loose_end's approval-cancel arm on it server-side.
ApprovalKind::UpdateMetaInputs stays too. Nothing in production code
produces it any more, but pre-existing approval rows may still carry it,
and the operator's own path to a meta update is unaffected — the
dashboard's POST /api/meta-update inserts the meta_update job directly,
bypassing approvals entirely.
The two format_ack tests in hive-agent-mcp that named
request_update_meta_inputs were only using it as a label string while
exercising the generic OkWarn/Ok renderer, so they are retargeted to a
surviving tool rather than deleted.
Note hive-c0re's priv_client::list_containers is a different thing (the
host-side privileged container listing behind hive-priv) and is
untouched.
Closes#4591
The 'Last event Ns ago' line used to repeat the full working-vs-wedged
explanation on every single status call. That's already covered by
the status tool's own MCP description and by docs/tools/subagent.md's
'Is it working, or is it wedged?' section, so restating it every call
was pure repetition on a line meant to be skimmed, possibly several
times a minute. Down to just the number and the state name.
A swarm runs one homeserver and every hive on it logged in as the same
`@hive:` localpart, holding the same access token out of one swarm-wide
store path. That is one matrix identity for N hives: the homeserver
cannot attribute an action to the hive that took it, and revoking one
hive's standing revokes every hive's.
Three changes, and the third is the one that makes the other two real:
- **The localpart carries the hive's name** (`hive-<hive>`), derived in
one place, `swarm_secret_client::matrix::hive_localpart`.
`hive-matrix.nix` renders the same string as the appservice
registration's `sender_localpart`, so the shared account stops being
created rather than merely stops being used.
- **The store path is templated by hive**, not a constant. The
"a swarm runs one homeserver, so this is a constant rather than a
parameter" rationale went with it; it stopped holding the moment two
hives shared the homeserver it describes.
- **The path moved out from under the grant every hive has.** It sat at
`swarm/services/matrix/sender-token`, inside the
`secret/data/swarm/services/*` read stanza `policy::render` gives every
hive. It now sits under that hive's own stanza,
`secret/data/swarm/hives/<hive>/*`, which interpolates the reader's
name — so a hive reads its own token and is refused another's. The
policy renderer itself is unchanged: narrowing the `services/*` grant
would break the OIDC-secret read it exists for, and moving the
credential is what this needed instead. A policy test walks the
rendered stanzas and asserts none of hive alpha's covers hive beta's
sender token, so a later stanza that widened it fails here.
`swarm-matrix-ctl` takes a new required `MATRIX_MINT_HIVE` and writes
that hive's path; its store grant in `swarm-bao.nix` follows, scoped to
one hive's leaf via the new `deploy.bao.matrixCtlHiveName` (defaulting to
this host's `hiveName`) rather than a `hives/*` wildcard, which would
hand the matrix container every hive's token back.
Migration: no outage at deploy. `ensure_hive_user` short-circuits on the
local token file, so a hive keeps running on what it has; with no such
file it reads the new per-hive path, finds nothing, and falls through to
the existing register-or-appservice-login ladder against its own
localpart — which needs only the per-hive `as_token` on local disk. The
old shared object is read by nothing afterwards. Rooms do not follow the
identity, and that is the one operator step; both ways out are written
into `docs/integrations/matrix.md`.
No admin standing is granted to the per-hive accounts: `admin_execute`
stays empty and the assertion pinning it is untouched.
Three hits the recent write-good.Passive=error flip (hyperhive#4546) now
catches on this branch's content:
- "stored password is gone" -> "has lost its stored password"
- "the ordinary appservice re-login above is unaffected" -> named the
actual actor (the admin-sender limitation) instead of leaving it
passive
- "Federation is enabled" -> "tuwunel enables federation" (tuwunel is
already named one clause earlier in the same paragraph)
No suppressions, no content dropped.
Both spots hedged the swarm-level framing with a temporal qualifier
("until then", "today") that read as future work awaiting a fix, on
top of the framing itself. Drops the qualifiers; the operations are
swarm-level and rehomed there now, not pending anything from the hive.
Renames `swarm-matrix-minter` and reshapes it around subcommands. Minting
is now `swarm-matrix-ctl mint`.
Running rust inside `containers.hive-matrix` is not free: it needs its own
store identity, its own cert role and its own bind mounts, and every one of
those is per-*container*, not per-task. A second single-purpose crate would
have had to duplicate that plumbing to add one action, so the next thing
that has to run in there should be a verb here rather than a new crate.
The old name guaranteed the opposite.
`main.rs` is clap dispatch; the minting logic moves to `mint.rs` unchanged.
A bare invocation is refused: `mint` writes a credential, so "no verb"
defaulting to it would make a typo in the unit mint rather than fail.
The environment prefix moves with it, `MATRIX_MINTER_*` → `MATRIX_MINT_*`.
Scoped to the verb and not to the binary, because a binary-scoped prefix is
one the next verb has to share or widen, and a widened one never narrows
again. A test asserts every variable carries the verb's prefix.
The principal renames too. The cert role, bao policy, granting unit, leaf
filename and `certAuthCns` entry all have to spell one string the same way,
so leaving them as `swarm-matrix-minter` would have rebuilt the naming
split this branch exists to remove. Renaming the nix options alongside is
free here: every one of them is introduced by this PR and has never been
released, so no operator config names them yet.
`ExecStart` now names the verb, which is a contract between a nix string
and a clap enum that fails at deploy time with no local signal. Both ends
assert it: `mint_is_spelled_the_way_the_unit_invokes_it` in the crate, and
a new module-eval arm reading the rendered `ExecStart`.
docs/getting-started/setup.md drops the sender token from its "live on the
host" list: setup does not touch this credential, so a setup guide has no
reason to name it.
The store path and every identifier around it called this an admin
token. It is not one: of ~15 hive-c0re call sites only two need
homeserver admin, and the homeserver no longer promotes the account at
boot, so the name overstated both what the credential is and what it may
do.
Renaming it to the account was not enough either. "The `@hive:` token"
reads as the token of a hive user, and no such user is provisioned —
`@hive:<server_name>` is the appservice registration's own
`sender_localpart`, an account the homeserver creates for itself when it
loads the registration.
So it is the **sender token**: the matrix appservice sender account's
access token, at `swarm/services/matrix/sender-token`. The name says
what it authenticates as rather than what it may do, which is the part
that was wrong.
The path has one constructor, and the bao grant, the grant assertion and
three unit tests pin its literal independently — so a half-finished
rename fails a check rather than leaving the minter and its readers
disagreeing at runtime. `tracing` messages are renamed with the code, so
the journal reads the way the source does.
The host-side file keeps its name (`matrix/access-token`): it carried no
admin framing, and renaming it would orphan the file on every deployed
hive for nothing.
`docs/tools/hivectl-cli.md` is regenerated from the clap tree.
Three leftovers from the rename, plus the gating prose job.
`nix/packages/default.nix` still described the minter as the "matrix
admin credential's minter", and shipped that claim in the package's
`meta.description` — a PR-visible string.
`promote_user_to_admin`'s doc comment pointed at the `admin_execute`
entry in `hive-matrix.nix` as the thing that bootstraps the first
admin. That entry is gone, so the comment referenced nothing. It now
records that the account is ordinary, that the call therefore has no
working sender, and that rehoming at swarm level is the fix rather than
re-granting. `reset_user_password` gained the matching warning; it had
none.
The prose fixes clear all 8 `CI / prose lint (vale, errors)` failures,
all of which were in docs this branch touches. No vale config change,
no exception, no carve-out: contractions, one recast sentence, one
de-hyphenation and one dropped "simply".
Tense: four docs described system behaviour in the future ("will
refuse", "will fetch and trust"). Reference docs get read from the
other side of the change, so they say what the system does.
`swarm/services/matrix/sender-token` has a real minter
(`swarm-matrix-minter`, in the `hive-matrix` container) and two
readers now: the minter's own pre-flight read-back before it mints,
and hive-c0re's `stored_sender_token()` under the hive's own
certificate. The credentials table catalogs minter/reader/renewal for
every credential path; this one was missing a row.
Refs #4345
A swarm runs one homeserver and a homeserver has one appservice sender
account, so "mint it once" is a property of the thing being minted
rather than something a lock has to enforce. That is what makes this
account the one to move first: no trigger route, no controller change
and no agent list — a boot-time oneshot beside tuwunel is the whole
mechanism.
`swarm-matrix-minter` runs inside `containers.hive-matrix`, which
already holds the appservice token: the rendered registration is bound
in read-only because that is how tuwunel is handed it. What the
container lacked was an identity of its own, so this adds one — a leaf
from the store's CA with a grant of exactly one path, not the hive's
leaf, which reads every secret in the store.
Both ends of the credential ship here. The minter reads the path it
publishes to before it touches the homeserver, and returning on a
non-empty read IS the "only once"; `hive-c0re`'s `ensure_hive_user`
reads the same path, authenticating with the hive name already in
`HYPERHIVE_HIVE_NAME`. The existing mint-then-`M_USER_IN_USE`-login
ladder stays as the fallback for a store that is empty, unconfigured or
unreachable, which is every swarm deployed before this — so nothing
needs backfilling and nothing breaks if the rest of the sequence never
lands.
The credential is not an admin credential, and is not named like one.
It is the access token of the appservice registration's own
`sender_localpart` — `@hive:<server_name>`, an account the homeserver
creates for itself when it loads the registration. The store path is
`swarm/services/matrix/sender-token`, the host path is
`matrix/access-token`, and the homeserver no longer runs an
`admin_execute` promotion for that account at boot. Everything the hive
provisions with it — the Space, the chat room, their hierarchy and join
rules, the invites — rides on being the creator of those rooms at power
level 100, not on homeserver admin; there is no Synapse admin API here
to need, tuwunel has none.
Two operations do need an admin *sender* and therefore stop working:
`hivectl matrix promote-user` and `hivectl matrix reset-password`, both
`!admin …` messages into `#admins:<server>`, plus the password-reset
recovery path that an agent with a lost password file falls back to.
They are swarm-level operations and are left failing loudly rather than
served by an over-privileged token every other call site would also
carry. The sweep's own admin-rights check and self-repair go with them:
an account that is deliberately not an admin has nothing to check.
`ephemeral = false` stays, and hive root can still read the container's
filesystem. Accepted: what this buys is identity separation — no hive
*process* holds or reads the appservice token — not physical isolation.
Refs #4345
'Rows by unit' and 'Rows by host' use VictoriaLogs LogsQL, not PromQL,
so the sort_desc() fix from the PromQL panels doesn't port directly.
Verified live against the real store (swarm-logs query) that
'| sort by (rows) desc' after 'stats by (...)' orders correctly on our
VictoriaLogs version -- the VictoriaMetrics/VictoriaLogs#258 sort-order
bug doesn't reproduce here. Append the sort clause to both panel
queries.
2 real active-voice rewrites (per mara: main picked these up via a
rebase+merge that landed after write-good.Passive went error-gate,
never checked against the current rule):
- "The datasource is told instead" -> nix/host-modules/swarm-grafana.nix
tells the datasource instead.
- "`value` is compared with `===`" -> the builder compares `value`
with `===`.
No suppressions, no content dropped.
.vale.ini's own comments named the issues that gated
write-good.Passive, but the tracker-tag lint's extension list didn't
cover .ini, so it never caught its own violation. Extend the scope
and reword the comment to prose that stands on its own.
qualify_ref (ci_runs.rs:84-93) reads '#N' and an all-digit value as a PR,
anything else as a branch — so '--branch #4540' and '--branch
ops-4502-swarm-agent-icon' for the same PR/head commit answer two disjoint
questions with no indication which one was asked. Print the resolved ref
to stderr whenever --branch is given, so the output states which question
it answered.
Stdout is untouched in both plain and --json mode, so no existing
invocation's parsed output changes. An all-digit branch name stays
reachable via --branch refs/heads/<name> (already documented, unchanged).
The comment moved from swarm-grafana.nix into observability.md by the
previous commit became visible to vale (it doesn't lint nix comments).
Reword six flagged spots (contractions, a sentence-initial 'So', a
period outside quotes) without dropping any fact.
The 41-line comment on logLevelRules tripped the comment-block lint
(30-line max). Move the detailed walkthrough (why each rule shape is
what it is, the three silent-failure modes, the query a level button
emits) into docs/scheduler/observability.md's existing 'Log severity'
section, which already covered the mapping at a higher level. The nix
comment now carries the short why/contract and points at the doc for
the full detail — no information dropped, just relocated.
Grafana's log-level buttons filter on a field called `level`; no row in
the store has one. The store's name for it is `severity_text`, chosen by
VictoriaLogs' OTLP ingester rather than by us — v1.52.0's
`app/vlinsert/opentelemetry/pb.go` writes it unconditionally and the
ingest parameters have no `_level_field` to rename it with. So the
mapping is made on the reader: the VictoriaLogs datasource, which was
provisioned with no `jsonData` at all, now carries a `logLevelRules`
entry per severity the journald parser can emit.
`logLevelRules` is the datasource plugin's only level-related jsonData
key — there is no field-name setting and no OpenTelemetry preset to
switch on. It is read off `instanceSettings.jsonData` in the plugin's
`datasource.ts` and typed in its `configuration/LogLevelRules/types.ts`,
both recovered from the sourcemap shipped in the pinned artifact
(`grafanaPlugins.victoriametrics-logs-datasource` 0.26.3). Each enabled
rule appends an `OR severity_text:="INFO"`-shaped term to the query a
level button emits, next to the `level:…` term that matches nothing.
A wrong rule here fails silently: Grafana provisions unknown jsonData
without complaint and the buttons go on returning zero rows. The three
ways to get one wrong — a non-literal `enabled`, a non-canonical `level`
spelling, a value that is not the stored text — are recorded at the
binding, and a module-eval arm pins them along with the real failure
mode, a severity added to `nix/journald-severity.nix` and not here.
`Unspecified` is left unmapped on purpose: it is the store's own
rendering of an absent severity and the thing the logs dashboard's "no
severity" panel counts.
Refs #4560
Both blockers are done — #4548 (353 hits in hand-written docs) and
#4549 (46 hits in generated CLI reference docs, hivectl/swarmctl/hive-forge's
own clap help text) — and the previous commit suppresses the reviewed
false-positive remainder. vale --minAlertLevel=error docs now returns
0 errors, matching CI's existing prose-lint-errors job.
Closes#4546.
133 hits across 38 files, all previously classified during #4548's sweep
and deliberately left un-rewritten (predicate-adjective state/necessity
description, design-intent idiom, structural/type-description idiom,
no-single-actor topology claim, parallel-triple exception, vale
substring-match artifact — see hyperhive#4548's per-PR bodies for the
per-hit reasoning).
Wraps each one in a scoped <!-- vale write-good.Passive = NO/YES -->
pair (the supported mechanism — TokenIgnores has a known offset-drift
bug) rather than a blanket per-file or per-rule silence, so a *new*
passive-voice hit anywhere in these files still fails once the rule
gates CI (next commit). Table/list false positives (docs/swarm/credentials.md's
renewal-table cells) wrap the whole block, not each cell.
Part of #4546.
Slipped past the #4574 lint — 1473 chars vs. the 100-char cap, caught
by pre-push while working on an unrelated branch. Same treatment as
every other panel: state what the number is, not the reasoning.
Checks every grafana dashboard panel/board description against a length
cap and a small keyword list (operator name, tracker-tag references) so
the descriptions just shortened don't regress. Wired into both CI and
pre-push, same as the other content lints.
Cut every panel/board description down to a short statement of what the
number is. Drops the design-reasoning prose, edge-case notes, and
operator-attributed quotes that had accumulated in an earlier pass —
useful context while building the boards, wrong shape for a tooltip an
operator glances at mid-incident.
Both journald receivers import one PRIORITY mapping, and each tier's
suite was asserting the whole table against its own receiver. That checks
one file twice: invert the table and two cases fail saying the same
thing, which tells you nothing about which of the two possible defects
you have.
Split by subject instead. The table's contents — the inverted direction,
overwrite_text, parse_from/on_error — belong to the file that holds them,
so they get a suite of their own reading that file directly, with no
fixture at all. Each tier keeps a case, reduced to the question only it
can answer: does MY receiver carry the shared mapping.
Both tier cases stay. They cover different receivers over different
journals — the agent container's own and the swarm collector's host
journal — and one tier quietly losing its parser while the other keeps
one is exactly the half-fixed state worth catching.
Membership rather than equality of the whole operator list, so a tier
that later grows an unrelated operator of its own still passes.
Checked against seven defect scenarios: each fails exactly one case, and
names the right one.
The direction is the part a reviewer cannot check by looking, so it is
asserted at both ends of the table and in both tiers' groups: an inverted
mapping still maps every value to something, and a case that only asks
whether a severity parser exists passes on the exact defect. The reader
that turns a rendered operator list back into a PRIORITY -> name function
lives in lib.nix, since both tiers need it.
The panel is asserted on its query rather than its title, because a panel
that keeps the title and loses the expression renders an empty graph that
looks exactly like zero prioless lines.
The mapping above needs something that says whether it is still working
after whoever wrote it has gone. A timeseries rather than a stat, so a
regression is a line lifting off zero rather than a number nobody reads.
Two series, and the split is the point: rows that carried a PRIORITY and
arrived with no severity anyway (a broken mapping — this one must reach
zero and stay there), against rows that never had a priority to map. The
latter is Claude Code's own OTLP telemetry, which emits log records with no
severity set at the source; mapping cannot reach it, so it is named rather
than folded into one number that never goes to zero.
Needs no provisioning change — logstore.json is already in the shipped
dashboard list.
Records reached VictoriaLogs carrying the journal's raw PRIORITY and
severity_text "Unspecified" — every line in the store, at every tier, with
no level a query or a dashboard could read. VictoriaLogs has no ingest
parameter naming a level field; it auto-detects one by field name, so the
mapping has to happen in the collector.
A stanza severity_parser on each journald receiver, from one shared file
rather than a copy per tier: the two receivers are unrelated config (a
fixed stanza in the agent container, a parameterised block inside the
swarm-otel container) and a drifted copy fails silently — every line still
arrives, labelled as the wrong thing.
Two details that are easy to get wrong and quiet when wrong. PRIORITY
counts down in urgency where the OTEL severity counts up, so the table is
written as a table. And overwrite_text is required: without it the parser
sets the severity number and leaves the text as the raw digit, so
severity_text arrives as the literal "6" — populated, and not a level
anything renders.
4 genuine passive-voice rewrites (actor named — you, the swarm, nix);
3 false positives left untouched — a design-constraint description
with no single correct actor across both provisioning modes ('is
issued with pathlen:1'), a governance/definitional statement ('who is
allowed to decide'), and the already-established predicate-adjective
classification ('is misconfigured').
5 genuine passive-voice rewrites (actor named — create-user, hivectl,
the operator, claude, you the CLI user); 2 false positives left
untouched — predicate-adjective necessity descriptions ('is needed',
'is required'), same category already established across this sweep.
