The nix-store gauge comment claimed a semconv name it deliberately does
not use; it now says why the name is hyperhive-scoped. Two doc pages
still described the removed banner for the limits-read and
knowledge-pull failures. The warning alert's comment now says
pending_logins pages on purpose.
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
The external export is a setting of the swarm collector, so it moves to
services.hyperhive.deploy.swarm-otel.upstream. Renamed:
- otel.endpoint -> upstream.endpoint
- otel.protocol -> upstream.protocol
- otel.headersCredential -> upstream.headersCredential
- otel.collector.upstreamHeaderName -> upstream.headerName
The unreleased otel.upstreamSignals is declared as upstream.signals.
Hosts keep working through the rename warnings. Eval test covers the old
paths reaching the new options. Docs and stale comments point at the
new paths.
Adds services.hyperhive.otel.upstreamSignals, a list of "metrics" and/or
"logs" naming the signals the swarm collector sends to otel.endpoint. The
swarm's own stores always receive everything. The upstream exporter is
only defined when at least one signal uses it, since otelcol refuses an
exporter no pipeline references.
On deploy: nothing changes until the option is set; the default is
[ "metrics" "logs" ], today's behaviour. To stop the 404 flood from an
upstream that has no /v1/logs, set services.hyperhive.otel.upstreamSignals
= [ "metrics" ].
swarm-bao-granter-role exits 4 when the store is unsealed but refuses the
bootstrap token, and Restart=on-failure retried that every 30s for as long
as the token file existed. RestartPreventExitStatus=4 makes that permanent
failure leave the unit failed once. Exit 1 (store sealed or unreachable)
keeps retrying.
On deploy: the unit goes red once instead of looping every 30s. The stale
token on muede-lpt2 still needs the operator step (remove it or mint a
fresh one); this does not clear it.
pull waits (bounded) for the swarm queue connection instead of failing at boot while the client is still connecting. The watch runs one pull after every attach, including re-attaches, and a failed attach or ended stream is retried with a bounded backoff instead of waiting for the next boot. A first deploy is not queued again while one for the agent is queued or running.
An agent whose first deploy was lost to a c0re restart is now deployed on the next boot, or the next watch re-attach, without anyone re-declaring it.
nixos-rebuild switch no longer hangs when nginx is (re)started in the
same transaction: the cert unit is Before=nginx, so its synchronous
reload and nginx's queued start job waited on each other. The reload is
now queued with --no-block try-reload-or-restart, as the nixpkgs ACME
module does.
sync_agents returned early whenever the rendered flake.nix matched disk.
A sync stopped after writing flake.nix but before its commit therefore
never re-locked or committed. The early return now also requires that
flake.nix and flake.lock have nothing uncommitted; otherwise the normal
lock+commit path runs and logs that it is resuming.
A hung forge fetch inside `nix flake update hyperhive` (or any other meta
git/nix child) now fails after its deadline with an error naming the command,
the child is killed, and META_LOCK, BuildSlot and MetaWindow are released. The
boot sweep already treats that failure as a warning and fans out the rebuilds.
A wait on META_LOCK past 60s logs a warning naming the waiter and the holder.
The logs/self pipeline shipped the collector's own journal to the upstream
exporter too. When the upstream failed, the collector logged the failure,
journald/self read it, and logs/self sent it upstream again: ~8 lines/s.
logs/self now exports only to otlphttp/victorialogs, and a filter processor
drops the collector's failure lines that name that exporter, so the store
exporter cannot rebuild the loop one hop later. Other collector lines,
including upstream failures, still reach the store.
On deploy, swarm-otel's store rows drop from ~8/s of error spam to its real
output.
The collector's container ran with --link-journal=host and the host's
journal bound over /var/log/journal, so its own journal was never
written where anything read it. It now keeps its journal in the guest,
reads it with a second journald receiver, and gets the host's journal
at /run/host-journal.
On deploy swarm-otel rows start arriving in the store, stamped
service.name swarm-otel.
/messages reads GET /api/messages/stream. A history select picks the window
(live only up to 24 hours) and a filter keeps the messages whose sender or
recipient contains the text. Each message is a row of the terminal list an
agent's terminal uses: one summary line with the time, the two parties and
the start of the body, and the whole body under it. The list follows the
newest message and holds the last 2000.
The feed has no event id, so a reconnect sends the history again. The page
empties its list each time the connection opens and fills it from the
replay, where the terminal stream hook, which keeps its rows across a
reconnect, would list every message twice.
The stream URL, filter, cap and row logic is in its own module with tests,
and docs/swarm/ui.md describes the page.
The refusal sat inside the collapsed section, and the general error panel
hides while a refusal is set, so a refused submit with the section closed
showed nothing. The refusal now renders under the section, outside the
details element.
The form gets a collapsed "instructions" section with a plain text box. Its
text goes in the optional agents_md field of the create request, and is left
out while the box is empty or only whitespace. The controller's size refusal
shows under the box instead of in the form's general error panel. The text
box empties after a successful create.
The send and refusal logic sits in its own module with tests, and
docs/swarm/ui.md describes the field.
no behaviour change: a pure move of items and tests into new modules, plus
visibility edits (pub(crate)) and path updates in sibling modules that named
crate-root items. same routes, same OpenAPI, 561 tests before and after.
why: #5470, new features stop landing in one 6.4k-line file by default.
files outside swarm-controller/src: docs/agent-lifecycle/agent-roster.md,
swarm-ui JobsPage.tsx and AgentsPage.tsx, nix/host-modules/swarm-controller.nix
and swarm-grafana.nix hold comments that named main.rs for items that moved.
The page opens on the firing ones. The button lists every instance again,
with the state filter and the tones back, so the table's filtering is
visible. The saved-filters key goes back to its original name, since the
state column is a filter again.
The page listed every non-normal instance. It now keeps the ones in the
Alerting state and links to Grafana's alert-groups view for the pending,
no-data and error ones.
The instructions row of the agent detail panel gets an edit button that
opens the agent's AGENTS.md, as the harness last published it, in a text
box. Save sends a unified diff against the revision that was loaded; when
the agent changed the file meanwhile the editor keeps the operator's text,
shows the agent's version and offers to save over it or take it. An unsaved
edit survives closing the dialog.
The diff is built in the browser (LCS over lines, three lines of context,
git's no-newline marker) and was checked against git apply on 600 random
before/after pairs.
docs/swarm/README.md gets the routes, the revision, the apply semantics
and the stopped-agent behaviour; the persistence doc points there from
the AGENTS.md entry. The swarm-controller, swarm-controller-api and
swarmctl READMEs and the repo map in AGENTS.md list the new bodies and
routes. docs/tools/swarmctl-cli.md is regenerated for
`swarmctl agent agents-md`, whose help text loses two passive phrasings
vale flags.
A subagent's terminal opened directly showed nothing until the subagent
published again: the relay was a core subscription, though its rows sit
in term-sub-<agent> for 24 hours. It now goes through the agent route's
replay-then-tail, so it opens with the subagent's newest 100 rows and
resumes after Last-Event-ID.
- term_stream.rs: the replay takes a stream name and a filter subject
instead of an agent, and `replay_or_live` (replay, else the live core
subscription) is the one fn both routes call. When the filter is one of
several subjects in the stream, a fresh reader's consumer starts at the
oldest row and drops all but the newest REPLAY_ROWS it counts as
pending, since the stream's newest sequences may belong to sibling
subagents. A single-subject stream (term-main-<agent>) picks its start
with replay_start exactly as before. Tests for both subagent cases.
- subagent_term.rs: the relay calls replay_or_live on the subagent's
subject in term-sub-<agent> and reads Last-Event-ID; module doc and
OpenAPI text no longer say "live only".
- main.rs: the handler test passes the new headers argument.
- swarm-queue-client agent_term.rs: replay_consumer_config takes the
filter subject instead of the agent.
- swarm-queue-client subagent_term.rs: consumer_create_grant, the
extended CONSUMER.CREATE form with a subagent subject as filter.
- swarm-nats-auth policy.rs: the controller holds that grant; the agent
grant's `$SWARM.term.*` filter is two tokens short of a subagent
subject, and the test that refused it now admits it.
- docs/swarm/ui.md, README.md: a subagent terminal opens with its newest
rows.
Fixes#5527
Pick agents from a grid of their icons (running or all), type the text, send.
The page posts the names and the text to the controller and shows what
happened to each agent, with a retry that selects just the ones that failed.
The selection rules and the controller's answer are read by plain functions
with tests.
Claude Code (>= 2.1.277) and opencode both read AGENTS.md, but Claude
Code reads it only when no CLAUDE.md exists in its cwd or above. So
this is a plain rename with no CLAUDE.md stub left behind.
- hive-agent/src/agents_md_migrate.rs: new. At harness start, before
the first turn, renames state/CLAUDE.md to state/AGENTS.md when only
the former exists; logs an error naming both paths when both exist
and touches neither; does nothing otherwise. Idempotent.
- hive-agent/src/main.rs: calls it in serve_main; checkpoint prompt
and a comment name AGENTS.md.
- hive-agent/src/claude_md_watch.rs: the size watch reads AGENTS.md;
todo text, doc comments and tests follow. Module name, todo
subsystem key ("claude_md") and metric name
(hyperhive.agent.claude_md.lines) are unchanged so existing todo
rows and dashboard history carry over.
- hive-agent/src/turn.rs, hive-agent/prompts/system.md: prompts tell
the agent its file is AGENTS.md.
- hive-agent/src/{otel_turn_metrics,stats}.rs,
frontend/packages/agent/src/stats.js, hive-subagent-mcp/src/*.rs:
comments and a tool parameter description naming the agent's file.
- nix/host-modules/swarm-grafana/dashboards/agents.json: panel title
and description.
- claude-plugins skills claude-subagents, state-hygiene: they mean the
agent's own state file.
- CLAUDE.md -> AGENTS.md at the repo root; docs/crates/README.md link
(checked by check-doc-refs.sh), hive-c0re/README.md,
.prettierignore and the nix/sources.nix comment follow.
- docs/: every prose mention of the agent's file; persistence.md
documents the boot-time rename, claude-invocation.md the shadowing.
Fixes#5509
The swarm UI's broadcast page (#5524) needs the controller to do the
fan-out, so POST /api/agents/broadcast takes {agents, body} and answers
200 with {results: [{agent, sent, reason?}]} in request order.
- agent_input.rs: the single-agent send is now send_one, which both
routes call; the single route maps its outcome to the same statuses
and texts as before. The broadcast validates the whole request first
(sender, empty body, empty or duplicated agents, more than 64, a name
outside the roster), then sends up to 16 at once with the single
route's no-responders window.
- main.rs: wires the route.
- docs/swarm/ui.md: describes the route next to the single-agent one.
An N running subagents badge opens a list of the agent's subagents, each a
link to its terminal, so you can switch without going back to the agents page.
A subagent counts as running when it published a row in the last 15 minutes;
the quiet ones are listed below it, dimmed. The list under the preview and the
menu now share one fetch hook and one definition of running.
/builds lists the builder's recent builds newest first, one card each with how
it ended, and opens a failed build's nix tail. /builds?run=<id> opens that
build's card and scrolls to it. The swarm-builder commit status now links
there instead of to the raw JSON record.
The forge and matrix containers belong to the services host, not to a
hive. hive-priv's container-name validator still accepted hive-forge and
hive-matrix for WriteNspawnFlags, WriteResourceLimits and
RemoveServiceDropin; hive-c0re only ever sends h-<agent> names there.
- hive-priv-sock/src/lib.rs: drop SIBLING_CONTAINERS, its only user was
the validator, and its test.
- hive-priv/src/main.rs: the validator accepts only h-<name> and returns
the logical name, so write_resource_limits loses its non-h- fallback;
test that all three requests refuse both names.
- hive-priv/README.md, docs/trust-boundary/security.md: the allowlist
no longer lists the forge and matrix.
Fixes#5508
