The `pki` mount had no issuer and no principal could log in to it, so the
swarm's service certificates were still minted by two openssl hops from a
root key on disk. Close both halves and retire the openssl path with them.
The mount now generates its own root, once. The granting unit asks bao
whether an issuer already exists (`bao list pki/issuers`) before calling
`pki/root/generate/internal`, so a rebuild or a reboot re-asserts the role
and the grant without touching the anchor — a root that changed per boot
would invalidate every certificate issued under it and every browser
taught to trust it. The guard asks the store rather than looking for a
marker file on this host's disk: a file is a claim about a mount that may
have been restored from a snapshot or disabled and re-enabled underneath
it.
`swarm-services-issuer` stops being an inert policy. A fourth cert-auth
role attaches it, following the shape the controller, the publisher and
matrix-ctl already use, and glue-bao-tls.nix signs the leaf carrying its
CN — that credential is what opens the mount, so it cannot come out of it.
`swarm-services-cert.service` logs in with that leaf, calls
`pki/issue/swarm-services`, and writes the result to the path
hive-tls.nix already wrote and the gateway already copies from. The
sub-CA layer does not move; it stops existing. The role's
`allowed_domains`, read from the same `swarm.serviceDomains` the SANs
come from, enforces at issue time what the sub-CA encoded in x509
`nameConstraints`, and with the root inside the mount there is nothing
left for an intermediate to be an intermediate of.
Not a flag day: the issuing root is published beside the leaf as
`swarm-services-root.pem` (0644) and joins `trust-bundle.pem`, where the
swarm root still sits. A leaf chaining to the old sub-CA and one issued
by the store both verify against the same bundle, so hives can be
rebuilt in any order. The same file is what an operator hands a browser
— readable without a store login, which matters because every listener
demands a client certificate.
The eval-time warning about uncovered service names is gone rather than
reworded. It fired on "this host does not hold the swarm root key", which
was the reason a hive could end up serving its own leaf on a
swarm-service name. Every hive now asks the store with its own identity,
so that stopped being the thing that decides.
Closes#4586
mint_and_verify reads the queue credential back before writing it, so a
re-run keeps the value a live agent already authenticates with. that read
is read_optional, which maps only a 404 to absence — so with create/update
alone every mint aborted on a 403 at its first store read.
read on the same paths the stanza already grants create and update, and
nothing else: no list, no delete, no patch.
Cut ~27 lines of blackout-measurement and cross-reference narrative
(already in the PR body / issue) down to the three things a reader
actually needs at this call site: the [Unit]-vs-[Service] trap, why
Restart is absent, and the window-vs-burst constraint.
Every deploy on a hive host, the replacement opentelemetry-collector
reaches bind() while the outgoing process still holds
127.0.0.1:8888 (its self-scrape endpoint). nixpkgs sets
Restart = "always" with no RestartSec, so the unit spends its five
default attempts in under two seconds, hits start-limit-hit and stops
retrying — ~27s of telemetry blackout per deploy.
RestartSec = 5 with a 12-attempt burst over a 120s window rides the
race out instead: the blackout ends within one interval of the port
coming free, and 55s of it being held is survivable where 2s was not.
StartLimitBurst/StartLimitIntervalSec go at the systemd.services attr
level, which NixOS renders into [Unit]; under serviceConfig systemd
ignores them silently. module-eval-hive-otel asserts the placement.
`is_forbidden` and the webhook-secret load/regenerate path were the last two
entries on the shortlist in hyperhive/hyperhive#3950; the other three landed in
hyperhive/hyperhive#4650. Both are classification logic whose failure mode is
silence, which is why they are worth a test rather than a coverage line.
`is_forbidden` gates the arms that tell an operator a Forgejo admin PATCH was
refused for want of a scope, and which credential to delete and re-mint to fix
it. One of those PATCHes is `ensure_repo_creation_disabled` — the lockdown that
stops an agent creating a repo it owns and self-merging in it. Two tests: a 403
is recognised in both shapes the typed client produces (the spec-listed
`Forbidden` kind and the bare `UnexpectedStatusCode`), and nothing else is —
not a 401, whose remedy is the automatic re-mint one function down, and not a
transport error that never reached the forge at all.
`load_or_generate` grows the path-taking half `load_or_generate_at`, the same
seam `swarm-controller`'s `webhook::load_or_generate_at` already has and for
the same stated reason. Three tests over it: a valid stored secret is returned
verbatim and never rotated (the newline this module writes itself makes the
trim load-bearing, not defensive); a malformed one is replaced by a secret that
reaches *disk*, not just the caller, and is then stable; and each near miss —
empty, whitespace, 63 chars, 65 chars, right length with a non-hex char — is
refused. That last one is the security case: `Hmac::new_from_slice` accepts a
key of any length, empty included, so a relaxed check fails nowhere and just
keys every signature off a guessable value.
Every test was confirmed able to fail: six mutations of the code under test,
each watched red, then reverted. The two halves of the validity check and the
two arms of `is_forbidden` were broken separately, so neither test passes on
one arm alone.
- hive-c0re::webhook_secret::verify_signature — the HMAC comparison
verify_hmac (the only gate on the public webhook endpoint) delegates
to. Correct-signature and mismatched-signature (wrong secret, tampered
body) cases.
- hive-forge credential-helper get (host= check) — subprocess
integration tests since the check is inlined in run(), which reads
real stdin/env and prints real stdout. Host mismatch (error, token
withheld), host match (credentials printed), and no host= line
(backwards compat) cases.
- hive-priv::{validate_credential_name, validate_snapshot_name,
ensure_plain_filename} — the only gate on the root-privileged socket.
Empty/charset/dot/slash rejection, hive- prefix requirement, and
./../slash rejection respectively.
The swarm-tier collector's journald receiver had no storage extension, so
it started each run with no cursor: journalctl --follow --lines=0 ships
only what arrives after the receiver starts. Every collector restart
therefore dropped whatever was written to the journal while it was down,
silently — no error and no replay.
Wire the receiver to a file_storage extension, matching the agent-tier
collector in nix/agent-modules/otel.nix, so a restart resumes from the
persisted cursor instead.
Refs #4527.
The four-way client-cert split gives each store reader its own leaf, and
three of the four readers render only where their own leaf exists. On a
host that mints its own PKI glue-bao-tls.nix defaults all eight, so there
is nothing to do; on a hand-configured remote-store hive, omitting one
pair used to mean that unit silently did not render — a privilege-
narrowing unit absent from a green build, with the missing unit as the
only evidence.
Each of the three now asserts its own pair, shaped after
swarm-grafana.nix's haveClientIdentity assertion and named to the pair it
needs. What differs from Grafana's is the gate: these fire only where the
host demonstrably reads the store (it holds deploy.bao.clientCertFile and
clientKeyFile) and the consumer is on. A host with no store identity is
the supported no-store deployment and still evaluates; the collector's
no-secret degrade is untouched, because that host holds no clientCertFile
either.
Also rewords three passive-voice sentences in docs/swarm/secrets.md that
vale flagged, and documents what the refusal costs and where it stays
silent.
Four units read one path each out of the store, and all four logged in
holding `deploy.bao.clientCertFile` — the hive's own leaf. Bao identifies
a principal by the subject of the certificate it presents, so four
readers behind one certificate were ONE principal, and the only grant
expressible was the union of what the four need: read on
`swarm/agents/*`, `swarm/hives/<hive>/*` and `swarm/services/*`. The unit
fetching Grafana's OIDC client secret could fetch every agent credential
in the swarm; the one fetching this hive's matrix token could fetch
Grafana's. Least privilege was not misconfigured here, it was
unrepresentable.
Each now holds a leaf, a cert-auth role and a policy of its own, and each
policy is the single `secret/data/…` path that unit's own script names —
spelled to the leaf, not to a prefix, the way matrix-ctl's already is.
Following the four exemplars in-tree rather than building a mechanism:
`signLeaf` mints the leaves, `swarm-bao.nix` writes the roles from the
bootstrap token, the consumers name their own pair.
Two of the four are written PER HIVE and two are not, which is the shape
of the paths rather than a preference. A matrix appservice token and a
queue credential live under `swarm/hives/<name>/` and every hive runs a
reader for its own, so one role for all of them would have to be granted
`hives/*` — letting one hive read another's, a reach no hive has today.
An OIDC client secret lives under `swarm/services/<client-id>/` and a
swarm registers each exactly once, so one role each is enough. The
per-hive subjects are `<prefix>-<hive>` and swarm.nix reserves every
composed spelling as a hive name, so a hive cannot be named into another
hive's role.
The shared leaf stays: hive-c0re still passes it into its container, the
`bao` CLI wrapper still defaults to it, and the three
`glue-*-bao-identity.nix` files derive the PKI directory from it.
module-eval-bao-grants gains a negative arm per principal — each pins the
three stanzas the hive's leaf carried and the two wildcards a later
widening would reach for, so a policy that grows fails here rather than
in a store. Plus the consuming side: repointing a unit back at the hive's
leaf would evaluate, deploy and log in, and silently restore the union.
A hive that reads a store on another machine now places one leaf per
principal instead of one shared by four. That cost is the point, and
docs/swarm/secrets.md lists the pairs.
mara's ruling on the review: documentation describes functionality as
is. The dashboard doc carried a "Topology tree" section marked dormant,
still spelling out the indent lanes, joints and continuation bars the
renderer paints — for a renderer that, with no parent field to walk,
puts every container at depth 0 and emits no prefix column at all. A
section labelled dormant is still a section describing a feature the
code does not have.
Each one now states what the page renders today: SW4RM's C0NTAINERS is
a flat alphabetical list, one row per container, no indent and no glyph;
the tree section says nothing nests and names the code that decides so;
the selection bar lists the bulk actions it has, without a note about
the M0V3 picker it doesn't (agent-hierarchy.md's removal table is where
that record belongs).
Two more the -U15 context sweep turned up outside that section, neither
naming a removed identifier so neither reachable by grep: approvals.md
told an agent to clone "the child's" config repo, and hivectl.md sold
`agent restart` as a way round "the agent hierarchy".
The prose added by this branch named the tracker item in seventeen
places, which check-issue-refs.sh rejects: a `#N` tag is dead weight for
anyone reading the public mirror, where no issue data exists. Each one
now states the fact it was pointing at — the parent field is gone — so
the sentence stands on its own.
Two of those lines also carried a rustdoc break: `[`write`]` in
topology.rs is ambiguous between the module's own `write` fn and the
`write!` macro, which `-D rustdoc::broken-intra-doc-links` fails. Spelled
`[`write()`]`, per rustdoc's own suggestion.
The host_config.rs rewrite is two lines rather than three so the doc
block stays under check-comment-blocks.sh's 30-line ceiling.
The topology doc keeps its filename and its second half (manager
special-casing, harness unit shape) — both are cross-referenced from
other pages and neither is about the parent field. Its first half is
rewritten: what topology.json is now, and a table of what the removal
took with it, so a reader who finds `<parent>` or `set-parent` in an old
issue thread learns it went away rather than moved.
The dashboard's tree-rendering section is marked dormant rather than
deleted: the walk is still in swarm.js and retiring it is the frontend
owner's call.
`topology.json` was a map of `name -> parent | null`, and that value fed
the whole agent hierarchy: `<parent>` / `<children>` recipient sentinels,
the reparenting API (CLI verb, wire verb, dashboard endpoints, DAG node),
the dashboard tree, the rebuild depth sort, and an unconditional
bind-mount grant giving every agent RW on its direct children's state.
Per the operator's ruling the field goes, and with it all of the above.
The file survives as what remains once the value is gone: the roster of
agent names, which is the set `ManageRootAgent` grants mounts over. It is
now a JSON array; `read` still accepts the old map shape and keeps its
keys, so a hive that upgrades across this does not blank its roster (and
so no capability holder loses its mounts for the length of that window).
Two sites kept their behaviour under a different recipient rather than
losing it. Both addressed `<parent>`, which the broker already resolved to
`operator` for a root agent, and every agent is now what that fallback
called a root:
- the harness's turn-failure / plugin-failure notification
(`Surface::send_to_parent` -> `send_to_operator`), and
- the send allow-list's always-permitted escape hatch, so an agent with a
restrictive allow-list still has a way to say it is stuck.
What is NOT preserved, deliberately: an agent with no capability no longer
sees any other agent's dirs. `ManageRootAgent`'s own grant is unchanged --
still every agent in the roster, still state RW + config RO, still no
`harness`.
The dashboard's reparenting control (the M0V3 picker) is deleted with its
CSS. The tree rendering that reads `ContainerView.parent` is left for the
frontend owner -- it degrades to a flat list with the field gone.
--run <id> against a run that was itself PR-triggered has the identical
defect #4632 fixed for --pr: a workflow_dispatch run writes no commit
status, so it can't clear the red (pull_request) check on that PR's sha
no matter how the dispatched run turns out.
resolve_run already distinguishes this case -- it recognizes a run's
prettyref as a PR pseudo-ref (#<n>) via pr_number_from_run_ref, then
used to call branch_for_pr to keep going. It now bails with the same
pr_refusal_message --pr uses instead, before ever building a dispatch
request. branch_for_pr has no other caller (--pr refuses before
touching it too, since #4632), so it's removed rather than left dead.
Push (non-PR-triggered) and --branch are untouched.
Regenerated docs/tools/forge-cli.md from clap help; corrected
docs/tools/forge.md's claim that --run always works.
The doc comment still described the old (never-shipped) behavior --
exactly the lie this whole PR exists to remove, just in --help instead
of the module doc or forge.md. Say what the flag actually does: refuse,
because a workflow_dispatch run writes no commit status and can't clear
a red (pull_request) check.
Regenerated docs/tools/forge-cli.md (derived from clap help via
'hive-forge markdown-docs').
A workflow_dispatch run writes no commit status, so ci-rerun --pr could
never clear the red (pull_request) check it claimed to be re-running for
- it dispatched a fresh run and printed a success message regardless,
even though the check stays red no matter how that run turns out.
--pr now refuses up front, before dispatching, naming the mechanism and
the working alternative (re-run from the web UI). --run and --branch are
unchanged: --run's own PR-pseudo-ref resolution and --branch's direct
dispatch are both untouched.
Fixes the exit-code/honesty defect from #4613; the workflow_dispatch vs.
pull_request event-type question (whether to close+reopen the PR to fire
a real pull_request event) is a separate, parked decision.
Every agent on a hive authenticates to the swarm queue with the same
hive-scoped OIDC client, so at the auth callout one agent is
indistinguishable from its co-hived neighbours. The commit before this
one mints a secret per agent at swarm level into
secret/swarm/agents/<agent>/queue; nothing read it.
Read it here, and read it from the container itself. A hive courier in
the path would be the hive vouching for which agent this is, which is
the property a per-agent credential exists to remove -- so the agent
logs in to the store with the certificate hive-agent-bao-identity
already proves it can log in with, and reads its own path. The store
certificate is for reaching the store and nothing else: what the new
unit writes to /run is the secret it read back, and nothing hands a
BAO_CLIENT_* path to anything queue-shaped.
The read needs no policy change. render_agent grants read on
secret/data/swarm/agents/<agent>/*, which covers this path and the
bao-mtls one beside it alike -- which is also why this unit degrades
where the identity check fails. A refusal this unit sees and that check
did not cannot be a policy that drifted; it is an object not yet minted,
the ordinary state of every agent created before its swarm knew to mint
one.
The harness resolves the path and reports which credential this agent
can present. It does not yet present it: the auth-callout responder
still verifies only the hive-scoped token, and an agent offering a
credential nothing on the other end reads back would simply be refused.
Teaching swarm-nats-auth to read the same path is the next slice.
`vale --minAlertLevel=error docs` is a CI gate and origin/main passes it
with zero findings, so the twelve this branch added were a red build, not
a backlog: eight Microsoft.Contractions, one write-good.So, two
write-good.Passive, across the new credential-matrix row, the backfill
runbook and the mint-identity help text.
Contractions and the sentence that started with "So" are mechanical.
The two passive hits are rewrites: "`--hive` is required" becomes
"`--hive` has no default", which is the actual claim -- the flag has no
value to fall back on -- and the help text's "the agent's queue secret is
left exactly as it is" becomes "it leaves an existing queue secret
exactly as it stands", which also names who does the leaving.
docs/tools/swarmctl-cli.md is regenerated, not hand-edited; the wording
lives in swarmctl's clap doc comments.
The mint route shipped without tests while the create route beside it
has three, so the two properties that make it a backfill rather than a
second creation route were unasserted: that it refuses before queuing,
and that it queues the mint node and nothing else.
Queuing a config-repo scaffold or a deploy against an agent that already
exists is the failure the second of those catches, and it is invisible
from the status code -- a version that queued the whole creation graph
would answer 200 with a node id just the same.
The agent name gets its own arm. create_agent's hive is checked against
the swarm roster, which incidentally rejects a name that is not an
identifier; an agent name has no roster to check against, so the parse
is the only thing between a traversal and a store path built out of it.
MintAgentIdentityResponse derives Clone + Debug to match
CreateAgentResponse -- expect_err on the refusal arms needs Debug on the
success type.
Agent creation at swarm level is event-driven and nothing sweeps for
agents missing a credential, so an agent created before a credential
joined the mint never receives one -- nothing comes back around to it.
Without a way to re-run the mint by hand, the only route to giving an
existing agent its queue credential would be to delete and recreate the
agent.
POST /api/agents/{name}/identity enqueues the same MintAgentIdentity
node POST /api/agents declares, rather than writing inline: a second
code path that mints an identity is a second place for the four strings
that have to agree to disagree. swarmctl agent mint-identity is the
operator end, the same POST-and-print-the-node-id shape agent create
already has.
--hive is required on both ends. Neither the CLI nor the controller
keeps a roster of which agent runs where, and the credentials this mints
name a hive, so a default would be a guess that hands an agent subjects
on a hive it does not run on.
Documents the backfill as a runbook step, and fills in the renewal cell
the credential matrix requires for the new row.
Every agent on a hive authenticates to the swarm queue with the same
hive-scoped secret today, so at the auth callout one agent is
indistinguishable from its co-hived neighbours and no subject can be
scoped to one of them.
Mint a secret per agent instead, at swarm level, into
secret/swarm/agents/<agent>/queue -- inside the stanza every agent's ACL
document already grants, so no policy changes and no existing agent's
document is rewritten. It is written by the same node that already mints
the agent's certificate, and read back under the agent's own token
before that node reports success.
The secret is not derived from the agent's mTLS identity: the two
credentials answer different questions and coupling their lifetimes
would mean renewing either implied renewing the other. Nothing here
rotates a queue secret -- a re-run keeps the existing value and only
corrects the principal it names, because this function is re-run
deliberately against agents that are already connected. Revoking one
means deleting the path.
Nothing reads the new credential yet; this is the minting half.
swarm-logs covers every host-tier unit this tool could reach, so the
second, capability-gated path into host journald earns nothing and is
removed outright rather than disabled behind a flag.
Removed end to end: the MCP tool definition + handler, the
GetHostJournal/HostJournal wire variants, hive-c0re's
dispatch_host_journal handler, the ReadHostJournal capability, and the
harness-side capability->--allowedTools gate. get_host_journal was the
only capability that mapped to an MCP tool, so allowed_capability_tools
could only ever return an empty vec; it goes too rather than linger as a
function that provably does nothing.
capabilities::has_cap/caps_for stay: #4624 gave ManageRootAgent's
bind-mount enforcement (hive-c0re/src/lifecycle/host_config.rs) a
second caller of has_cap, so they're no longer callerless once this
lands on top of it.
hive-sh4re's journal module (JournalPriority) had no consumer outside
this tool and is deleted.
An existing capabilities.json still naming read_host_journal does not
error: capabilities::prune_unknown drops unrecognised names with a
warn!, and an agent left with no capabilities has its entry removed. No
migration step is needed.
Untouched: hive-c0re/src/dashboard/journal.rs's
read_host_journal_response, which matches the name but is the private
helper behind the operator-only GET /api/journal-host dashboard route
and carries no capability check.
Rewrite two bare issue references (#4472, #4477) as self-standing
prose per check-issue-refs.sh's requirement — no markdown exemption,
hash-number tags are dead weight to a public forge-mirror reader.
Apply iris's vale fix to docs/agent-lifecycle/approvals.md (passive
voice, two contractions, one auto- hyphenation).
`roles.json` had a seeding path: `topology::reconcile_roles` granted
`can_manage_top_level_agents` to MANAGER_NAME on every meta sync, and that
role is what put every agent's state/config dir, `/applied` and `/meta` into
the root agent's nspawn binds. Collapsing the role into the capability left
the capability store with no equivalent, so the root agent would have lost
its recovery mounts at its next container rebuild — silently, and only then,
since nspawn bakes bind flags at container start.
Seed it in `ensure_root_agent`, before the create/start branch so a hive
whose root container already exists still gets the grant. Seeded once rather
than re-ensured every boot: the role kept an empty-list tombstone so a revoke
stuck, and the capability store deletes an emptied entry instead, so "no
entry for the manager" cannot tell a fresh hive from a deliberate revoke.
File existence can — every grant and revoke writes capabilities.json, even
when the result is `{}` — so the seed fires only while the store has never
been written, and is inert forever after.
The target is `lifecycle::MANAGER_NAME`, the same const the deleted role seed
used and the one definition of "the root agent" in the tree; no name literal
at the seeding site. The written string comes from
`Capability::ManageRootAgent` via `IntoStaticStr` rather than being spelled
out, so it cannot drift into a name `prune_unknown` would drop.
The hive had two spellings of "this agent may act on agents that aren't
its children": the `ManageRootAgent` capability, which nothing checked,
and a `can_manage_top_level_agents` role in a third meta store,
`roles.json`, which owned the real grant — the bind mounts that put
another agent's state (rw) and config (ro) inside the holder's
container. The two drifted independently, and with the parent/child
hierarchy removed the role's set (`parent.is_none()`) silently became
every agent while nothing said so.
Collapse them. The mount grant now hangs off
`Capability::ManageRootAgent`, looked up through the one capability
path that already exists (`capabilities::has_cap` over
`capabilities.json`) rather than a second mechanism. `roles.json` and
everything that read, wrote or reconciled it is gone, along with its
`meta.rs` staging and commit-label wiring; nothing in the tree reads
that file any more.
The enum variant keeps its name deliberately. Renaming it would turn
every `manage_root_agent` already stored in `capabilities.json` into an
unrecognised name that `prune_unknown` drops without asking. Its
meaning, not its spelling, is what changed: "may manage any agent". The
doc comment and the description string now say that.
`top_level_agents()`/`top_level_agents_in()` are replaced by
`all_agents()`/`all_agents_in()`. Under "manage any agent" the mounted
set is every agent by definition, so the code states it instead of
deriving it from a predicate that no longer discriminates — and the
call-site comment explains that, because it otherwise reads as a
widening. The holder is no longer bound as its own virtual child: that
reproduced the own-state and own-config mounts exactly, so dropping it
loses nothing.
Review response on #4620: not having SSO is not a supported
deployment, so the type should not permit it, and the docs paragraph
explaining why SSO is always present is redundant once the type says
so.
- swarm.authelia.url drops types.nullOr.
- Every consumer's null-arm is gone: two option defaults
(swarm-controller's and swarm's own statusPublish.tokenEndpoint)
that produced an empty/null placeholder when the URL was null now
unconditionally compute the real derived URL. Five now-dead
"assertion = ... != null" guards (swarm-authelia's bridge,
swarm-grafana, swarm-otel, swarm-nats, hive-forge, hive-matrix) are
removed as unreachable — in every case the same URL was already
interpolated unconditionally a few lines below the guard.
- grafanaNoSso, the module-eval fixture whose sole purpose was
exercising the now-unsupported no-IdP refusal, is removed along
with its dedicated test case; swarm.authelia.url = null is a type
error now, not a value that reaches that assertion.
- docs/swarm/services.md: cut the clause about setting the option to
null and the sentence explaining why the URL is co-location-
independent — both redundant now that the type enforces it.
`swarm.authelia.url` defaulted to `https://<domain>` only when this host
ran the container, and to `null` otherwise — so the address a client is
given was a statement about co-location rather than about the swarm. A
swarm has one SSO provider; every hive addresses the same name and
resolution decides which address that reaches, exactly as
`swarm.otel.domain` already works.
The option stays nullable: "this swarm has no IdP" is still expressible,
it is just now something an operator states rather than something not
running the container produces. The Grafana fixture that exercised the
no-IdP refusal says it explicitly.
Closes#4536
turn.rs has no tests. The knobs read env through two helpers and the
difference between them is load-bearing: env_u64 keeps a parsed 0, so
HIVE_TURN_IDLE_SECS=0 disables the watchdog and
HIVE_AUTO_RESET_WATERMARK_TOKENS=0 disables auto-reset; env_u64_positive
discards it, so a 0 sleep or cache TTL falls back to the default. Calling
the wrong one is a one-word edit that changes whether 0 turns a feature
off or does nothing.
Splitting the parse and zero-rejection halves out of the env lookup makes
both testable — std::env is process-global, so the lookup itself is not
safely settable from a threaded test runner, and the env-to-knob mapping
stays uncovered for that reason.
No behaviour change: each of the four knobs already agreed with its own
doc comment.
`services.opentelemetry-collector` already defines
`serviceConfig.Restart = "always"` at the normal priority. The forwarder
added in #4537 defined `"on-failure"` beside it, and two definitions at
one priority are a conflict the module system refuses to resolve — so
`containers.swarm-bao` stopped evaluating at all (#4615).
Drop our definition rather than force a value over it, which is what the
sibling collector in swarm-otel.nix already does. The resolved value is
`"always"`, which is the one we want here: a forwarder that exits for any
reason, clean or not, has stopped shipping the store's journal.
`RestartSec` stays — it conflicts with nothing and is what keeps the
restarts during first-boot secret delivery under the default start-rate
limit.
mara's ruling on this PR: the principle that landed for the role path is
general, so the no-role path moves the task out of the system prompt as
well. `system_prompt_and_trigger`'s no-role branch still passed
`prompt_file` straight through as `--append-system-prompt-file` —
byte-identical to main's pre-PR behaviour, and the exact bug this PR
exists to fix. Since no agent ships roles yet, every real dispatch takes
this path, so the standing-system-prompt bug (a task re-asserting itself
as an instruction on turn 2/3/N under continue/goal) was still live for
100% of usage; only the unused role path had actually been fixed.
`compose_prompt` now always folds the task into `trigger`, role or not.
Without a role there is no role text to hold a system-prompt file open
for, so none exists at all — `--append-system-prompt-file` is omitted
from the spawn entirely, not pointed at anything task-shaped.
Replaces `without_a_role_the_task_file_is_what_claude_is_pointed_at`
(which pinned the bug as "unchanged") with a test asserting the task is
absent from what reaches `--append-system-prompt-file` and present in
the trigger. Role-path tests are untouched in behavior; only mechanical
fallout from `compose_prompt`'s `Option<PathBuf>` return and `&str`
trigger params (clippy needless_pass_by_value once both branches only
borrowed it).
Blast radius: this changes behaviour for every existing dispatch, since
prompt_file's content has always gone into the system-prompt file before
this fix.
A `start` may now name a role: `role: "reviewer"` loads the spawning
agent's own `subagent_roles/reviewer.md` and renders it, alone, into one
per-session file that `--append-system-prompt-file` points at. The role is
the system prompt; the task is the turn, never the other way round — a
task baked into the system prompt would re-assert itself as an
instruction on every later turn of a continued session, not just the one
it was written for. The task instructions (`prompt_file`) are read and
folded ahead of the turn's own prompt instead, the same channel that
carries them to the subagent without a role.
The argument is optional, so every existing call is unchanged — pinned by
a test that a pre-role payload still deserializes with `role` absent from
the schema's required set, and another that the no-role path reaches
claude with the caller's own file, unrendered, and the trigger untouched.
With a role, one test pins the system-prompt file to the role's text and
nothing of the task, and another pins the task still reaching the
subagent as the turn's prompt.
A role name with no file fails the call, before the session name is even
reserved, and the error lists the roles the directory does hold. No agent
ships roles yet, so named-but-missing is the ordinary first-run state; a
fallback there would spawn a subagent under a prompt missing every clause
the role existed to carry. An empty file and a name that is not a plain
identifier refuse the same way.
A subagent's backgrounded bash child is reaped along with the rest of
its process tree at turn end, which silently orphans anything still
running past the CLI's default 2-minute timeout — the run reports
normally but the log file is empty or truncated. Raise the default on
hive-subagent-daemon's own unit rather than in managed-settings, since
managed-settings is also read by the main agent's session and the
operator's ruling is explicit that the main agent's environment stays
as is.
The previous commit deleted `schedule_authorized` and with it three
denial tests, leaving the permit behaviour it introduced unasserted:
nothing in the suite would notice a subtree check creeping back into
the scheduling handlers.
These four pin the inverse of the decision the deleted predicate used
to make. They sit at the handler level because that is the lowest layer
where the decision still exists to be observed — the predicate, its pure
`_in` form and the wrapper are gone, so there is no function left whose
return value *is* the decision, and the layer above (`dispatch`) reaches
these verbs through `require_group("scheduling")`, a different gate that
is still present and not the one under test.
`cancel` and `edit` assert the row afterwards rather than stopping at
`Response::Ok`: a check that silently declines to act while still
answering `Ok` is the regression a response-code assertion misses.
`list_schedules` is the structurally different case — it never rejected,
it filtered per row, so its assertion is that another owner's row is
present at all. `fire` asserts the response only; the fan-out legitimately
finds no live container under test and `fire_now` reports that inside
`Ok(report)`, so the response code is the only honest signal there.
Verified by mutation, not by inspection: reintroducing an ancestry walk
into the four handlers flips all four tests to failing (the three
mutating verbs on the refusal, `list_schedules` on an empty list) and
leaves the two pre-existing ghost-filter tests untouched.
These four cover the complete set of gate points this branch removes.
The roster listing that used to be the fifth is no longer one: `main`
removed `list_containers`, `Request::ListDescendants` and the handler
behind them outright, so there is nothing left there to assert.
Refs #4472
The topology predicate `is_descendant_of` gated the four schedule-
managing verbs: a caller could only name a schedule owned by an agent at
or below itself in `topology.json`. Those gates now permit any requester,
so the predicate, its pure `_in` form and the `schedule_authorized`
wrapper built on it are gone rather than left returning a constant. The
other two wrappers went earlier with the verbs they served —
`require_descendant` with the lifecycle MCP verbs in 87970a8c, and
`resolve_agent_state_target` with `get_loose_ends`'s agent parameter.
`require_group(agent, "scheduling", ...)` is untouched and still fires at
dispatch for every one of the five scheduling verbs, so holding the tool
group remains the gate; what goes is the ownership restriction layered on
top of it.
The three schedule-mutating verbs keep their row lookup as a plain
existence check, so a caller naming a schedule that does not exist still
gets `not found` rather than a message from deeper in the cancel path.
`list_schedules` stops filtering per row: it would only have hidden rows
the requester may act on anyway.
Error messages, tool descriptions and docs that described the subtree
relation are reworded — a refusal message naming a topology that no
longer decides anything is worse than none.
The six `is_descendant_of_in` unit tests go with the function they test;
the permit behaviour they leave unasserted is picked up by the next
commit.
Refs #4472
bao's metrics were scraped by the SWARM collector over loopback, via a
`swarm.otel.scrapeTargets.bao` entry gated on `deploy.swarm-otel.enable`
— "does the swarm's collector run on THIS host". It had to be: loopback
only reaches a reader that landed on the same host.
What that rendered everywhere else was nothing at all. Off that host the
metrics listener was not emitted, so the store's metrics reached the
store nowhere, and a host with no entry is indistinguishable from a host
nobody asked to scrape.
Moves the scrape into the collector this container already runs, per
mara on #4537: "move the existing scraper to the local collector". The
container shares the host netns (privateNetwork = false), so the scrape
still dials 127.0.0.1 — the listener keeps its address, its
`metrics_only` narrowing and its loopback-only bind, and the API
listener's `tls_require_and_verify_client_cert` is untouched.
The listener and its `prometheus_retention_time` lose their gate: the
reader ships with the store now, so there is no host where the endpoint
has none. The metrics pipeline reuses the logs pipeline's `resource`
processor and `otlphttp` exporter, so both signals carry the same
`service.name` and leave by the one hop.
Logs are unaffected: `journaldUnits` and --link-journal=host stay until
every sibling swarm container has a collector of its own.
The module-eval absence arm "a store with no collector beside it serves
no metrics" is inverted rather than dropped — the condition it asserted
is the bug. Three cases join it: the job is in swarm-bao AND gone from
swarm-otel (a move, not a copy), the scrape target and listener are both
pinned to loopback, and the metrics pipeline shares its exporter with
the logs one.
Both earlier versions asked the wrong host. `hyperhive.otel.enable` asked
whether this host runs a HIVE collector; `deploy.swarm-otel.enable` asked
whether this host runs the SWARM one. Neither answers the question the
forwarder actually has — "is there a collector to forward to" — and that
question cannot be false: a swarm always runs at least one instance of
every swarm-level service. So the forwarder renders under the condition
already enclosing it, that the store is deployed here, and nothing else.
`scrapeHere` deliberately keeps its `deploy.*` gate one line up. It is a
loopback metrics listener, which genuinely only works where the scraper
is — the two are different tiers, and the name says so.
The module-eval case that pins it is the split topology: the swarm
collector on another host, nothing local naming it, and the forwarder
still enabled and still addressed at `swarm.otel.domain`'s route. Both
removed gates render nothing in that fixture, which the co-located ones
they shipped with could not show.
The forwarder pointed at the hive bridge address and was gated on the
hive's `otel.enable`, so it existed only where a hive collector stood
beside it. It now exports to `swarm.otel.domain` — the gateway-served
name that resolves locally when co-located and over the network
otherwise — on the swarm tier's own producer route, and is gated on
`deploy.swarm-otel.enable` like its sibling `scrapeHere`.
Refs #4526
Every container is supposed to run a collector that passes its logs and
metrics to the next hop. swarm-bao did not: its journal reached the store
only because `--link-journal=host` puts it in the host tree, where the
swarm collector — a different container — reads it through a unit
allowlist. That is the topology being retired, and in this deployment it
delivers nothing: no `_SYSTEMD_UNIT` value in the seven-day store mentions
openbao at all.
So the store's container now runs its own journal forwarder, copied from
an agent container's (nix/agent-modules/otel.nix): the whole journal, no
unit allowlist, pushed to the same first hop every agent on the host
already exports to. A local collector reads the local journal, so there is
nothing for a list of unit names to disagree with.
The `swarm.otel.journaldUnits` entry and `--link-journal=host` both stay.
Every sibling swarm container still rides the shared collector, and they
come out once each of them has a forwarder of its own.
Closes#4526
mara: 'hive ui agent term: logout button gone? must have dropped with
one of the redesigns, pls bring it back or tell me where to find it'
It did drop, silently. The old app.js overflow menu had an
unconditional 🔓 logout item (dashboard-link + rebuild + new-session +
logout); the Preact rewrite's Root.tsx comment explicitly documents
dashboard-link moving to MetaNav and rebuild being removed outright,
but says nothing about logout — it just didn't come along. `/logout`
still works (TermInput's slash-command list, type-twice-to-confirm),
but nothing in the UI points at it or offers a click path anymore.
Fix: add a 'logout' entry to the status badge's dropdown (StatusChips'
StatusMenu) — the same menu that already hosts pause/resume and
cancel-turn, reusing its exact click-then-confirm-click pattern
(danger: true, armed/confirm state reset on close). Unlike cancel-turn
it's not gated on `thinking` — always offered, matching the old
overflow menu's unconditional entry. Wired through onLogout ->
termActions.ts's existing postLogout (already used by the slash
command, so the only new code is the menu entry + prop threading).
Scoped to just what was asked: /new-session has the identical
gone-from-any-menu gap (same old overflow item, no restored click
path), left alone here and flagged separately rather than folded in.
tsc --noEmit clean, nix fmt clean.
mara: 'filling just one of the fields should not touch the other.'
The cpu/mem override form always posts both fields, and the server
correctly treats an empty field as 'clear this override' (documented,
tested behavior in agent_config/resource_limits.rs). The bug was on the
client: the inputs only ever showed the current effective value as a
placeholder, never as the actual value — so a field left untouched
looked filled to the eye but posted empty, silently clearing whatever
override (or lack of one) was already in effect for that field.
Fix: set .value to the current effective value (already fetched and
displayed correctly in the read-only cap columns next to this form) so
leaving a field alone round-trips it unchanged. One resulting nuance
worth flagging: an agent with no per-agent override on a field, that's
purely tracking the hive-wide default, now pins that field to today's
default the first time *any* field on the form is edited, rather than
continuing to silently follow future hive-default changes. Fixing that
fully would need the server to expose whether each field is a real
per-agent override or just the resolved default, which is more than
this bug report asked for.
A record written to stdout carries no priority, so journald files the
whole stream at one level and the swarm log store shows `info` whatever
level `tracing` gave it. Under a systemd unit the process's stdout
already *is* the journal, so the fix is to speak the journal protocol
directly and let each record carry its own severity.
New `hive-log` crate holds the one sink chooser, called by `hive-c0re`,
`hive-agent` and `swarm-controller`. It builds the same `EnvFilter`
those binaries always built, then installs exactly one layer — never
both, since a journald layer stacked on the `fmt` layer under a unit
stores every record twice.
The choice is an fstat compare, not a presence test: a child inherits
`$JOURNAL_STREAM` even when its own stdout was redirected elsewhere, so
the variable existing proves nothing. The crate parses `dev:inode` out
of it and compares both numbers against an fstat of stdout, the
descriptor the `fmt` layer writes to by default. No match, unset, or
unparseable takes the `fmt` branch. A journald layer that fails to
construct despite a match falls back to `fmt` and warns through it —
a process must never fail to start because of its logger.
mara: 'i asked for a table. ask when doing something different.' Right
call — the PR body flagged the bargauge substitution as an open question,
not a decision, and I should have waited for an answer instead of
treating the silent absence of an objection as one.
Single 'Busiest agents' table panel (replaces the 5 per-metric bargauges):
5 table-format instant queries (turns, input, output, cache-read, cost)
joined on the agent label (joinByField), renamed to the screenshot's own
column names via organize, sorted by turns descending via sortBy — same
column set and sort order as the attached /stats screenshot.
This is a genuinely novel schema shape for this repo: no table panel,
transformation, or field-override config exists anywhere else in
nix/host-modules/swarm-grafana/dashboards/*.json to verify the join/
organize/sortBy option shapes against. Structurally verified (valid
JSON, jq empty, unique panel ids, every panel referenced exactly once,
nix fmt clean, dashboard-description lint clean) but the join/rename
field-name mechanics (whether Grafana names the joined columns exactly
'Value #A'/'Value #B'/etc.) are built from general Grafana schema
knowledge, not a working local precedent — flagging that explicitly so
the actual render gets checked before merge.
Reworked per review feedback: no new tab, and 3 of the added stat
panels duplicated existing ones (Active agents ~ panel-1 Agents
reporting, Cost = panel-2 Cost verbatim, Tokens ~ panel-3 + the new
input/output/cache-read breakdown). Removed panel-50/52/56 and the
overview tab entirely.
What's left is genuinely new (no prior panel covered it): a Turns KPI
and turns-by-agent bargauge in the activity tab (hyperhive.agent.turn.count
was emitted but unused anywhere), and input/output/cache-read token KPIs
plus cost/input/output/cache-read-by-agent bargauges in the cost tab
(the existing cost/token panels have no per-agent breakdown, and none
sort busiest-first).
New 'overview' tab on the agents dashboard, extending it per the
hive stats page's own summary layout (busiest-agents table + top KPI
row):
- KPI row: active agents, turns, tokens (all types), input, output,
cache-read tokens, cost. All from metrics already flowing into
this dashboard (claude_code.token.usage / claude_code.cost.usage /
claude_code.session.count) plus hyperhive.agent.turn.count, which
the harness already emits but nothing visualized yet.
- Busiest-agents replicated as five sorted bargauge panels (turns,
cost, input, output, cache-read, each by agent) rather than one
literal multi-column table -- this schema (Grafana v2beta1) has no
existing table+merge-transform panel anywhere in this repo to
pattern-match against, and every other panel here follows the
proven sort_desc(...) + bargauge shape already used for
'Cost by subagent name' (panel-43). Trading the exact table layout
for schema I can verify against a working precedent.
- Left out the stats page's 'window' tile -- redundant with the
dashboard's own always-visible time-range picker.
No new instrumentation needed. Structurally verified (valid JSON, no
duplicate panel ids, every panel referenced exactly once in the
layout) but not visually verified -- no local Grafana to render
against.
