`/api/cancel` stopped a turn only by SIGINTing a child process whose argv0 is
`claude`, so on an ACP agent it reported "no claude process to interrupt"
and the turn ran on. The serve loop now stores the session's canceller on
the `Bus` when the runtime has one, and `/api/cancel` uses it: the agent is
sent `session/cancel`, and the next wake prompt carries the interrupted hint,
as for a signalled claude turn. Without a canceller (claude), the SIGINT path
is untouched.
An ACP turn stopped by the idle watchdog (`HIVE_TURN_IDLE_SECS`) becomes
`TurnError::AgentStall(note)`, handled like `ApiStall`: park for
`HIVE_STALL_SLEEP_SECS`, requeue, and record `api_stall`. The TurnEnd note
is the runtime's message (how long the agent was silent, whether it had to
be killed, and that a silently retried provider error such as HTTP 429
looks like this), not the claude-specific `ApiStall` text. A cancel the
agent ignores is a `Failed` turn.
Refs #4391
AgentSession becomes hive_runtime::AgentRuntime, picked at startup from
the environment. Unset HIVE_RUNTIME keeps the claude backend, built
from the same title, store and PercentPolicy as before; drive_turn,
the 401 retry and the error mapping see the claude errors unchanged.
On acp the harness keeps the session id in the harness dir, answers
permission requests like the claude built-in allow-list (no built-in
shell, web fetch only with web_tools), maps runtime errors to Failed,
and reports an operator /compact as skipped instead of done.
Refs #4391
The auth callout grants an agent that presents its own queue credential
one more subject, `$KV.agent-icons.<agent>`: its own key in the
agent-icons bucket and no other. The hive's shared agent client is
granted none of the bucket, since every agent on a hive presents it.
hive-agent writes `/etc/hyperhive/icon.svg`, the file its `GET /icon`
serves, to that key once per start, as a JetStream publish straight to
the subject (what `kv::Store::put` sends, minus the bucket lookup), so
the one subject is the whole grant. No icon deletes the key. A failed
write, including one that arrives before the bucket exists, is retried
with backoff until acked. An agent connected with the hive's shared
client publishes nothing.
swarm-controller creates the bucket as soon as its queue connection is
up, instead of on the first icon read, so an agent's write does not
wait for someone to look.
Measured against a local nats-server with a user allowed publish on
`$KV.agent-icons.atlas` only: the write to its own key is stored and
readable, a write to `$KV.agent-icons.argus` is refused (the ack times
out), the DEL marker makes the key read as absent, and a write before
the bucket exists fails with "no responders".
rustls is built with both `ring` (async-nats's `ring` feature) and
`aws-lc-rs` (reqwest's `rustls` feature), so it cannot pick a
process-level default by itself. Since the queue started requiring TLS
(1d261b3f), async-nats builds its config with `ClientConfig::builder()`,
which panics without an installed default. The panic kills the async-nats
connector task, and every queue client (swarm-controller, hive-c0re, all
hive-agents) has sat in `Pending` since the 2026-09-25 23:04Z deploy.
Add `swarm_queue_client::install_crypto_provider()`, which installs
aws-lc-rs and ignores the "already installed" error. It is called first in
`main` of every binary that links async-nats: hive-agent, hive-c0re,
swarm-controller, swarm-nats-auth. `connect()` also calls it, so a new
binary that dials through this crate is covered without remembering to.
aws-lc-rs because reqwest already falls back to it when no default is
installed, so HTTPS in these processes keeps its current provider. The
other rustls users in the tree reach it only through reqwest, which never
panics here.
Closes#4738
hive-sock-client: each attempt now bounds connect (5s), write (10s) and
the wait for the response (60s by default). The response bound is per
call through the new `request_within`, which hive-agent's serve-loop
`Recv` uses with its 180s long-poll plus 30s headroom. A response
timeout is terminal rather than retried: the server holds the request,
so a retry re-sends something it may still act on and multiplies the
wait by the backoff schedule.
Outbound HTTP: the matrix login/whoami clients in swarm-controller and
hive-c0re's dashboard (5s connect, 30s request), the authelia-bridge
client (5s/30s; ensuring an identity runs an argon2 hash first) and the
ci-runner forge calls (5s/15s, config_pr_poll's forge budget) get a
connect_timeout and a request timeout. Timeout errors name the bound
that fired.
hive-agent's unix-socket extra web proxy bounds the connect (5s) and
the wait for the response head (30s, the http sibling's budget); the
body read stays unbounded.
Refs #4723
The prose added by this branch named the tracker item in seventeen
places, which check-issue-refs.sh rejects: a `#N` tag is dead weight for
anyone reading the public mirror, where no issue data exists. Each one
now states the fact it was pointing at — the parent field is gone — so
the sentence stands on its own.
Two of those lines also carried a rustdoc break: `[`write`]` in
topology.rs is ambiguous between the module's own `write` fn and the
`write!` macro, which `-D rustdoc::broken-intra-doc-links` fails. Spelled
`[`write()`]`, per rustdoc's own suggestion.
The host_config.rs rewrite is two lines rather than three so the doc
block stays under check-comment-blocks.sh's 30-line ceiling.
`topology.json` was a map of `name -> parent | null`, and that value fed
the whole agent hierarchy: `<parent>` / `<children>` recipient sentinels,
the reparenting API (CLI verb, wire verb, dashboard endpoints, DAG node),
the dashboard tree, the rebuild depth sort, and an unconditional
bind-mount grant giving every agent RW on its direct children's state.
Per the operator's ruling the field goes, and with it all of the above.
The file survives as what remains once the value is gone: the roster of
agent names, which is the set `ManageRootAgent` grants mounts over. It is
now a JSON array; `read` still accepts the old map shape and keeps its
keys, so a hive that upgrades across this does not blank its roster (and
so no capability holder loses its mounts for the length of that window).
Two sites kept their behaviour under a different recipient rather than
losing it. Both addressed `<parent>`, which the broker already resolved to
`operator` for a root agent, and every agent is now what that fallback
called a root:
- the harness's turn-failure / plugin-failure notification
(`Surface::send_to_parent` -> `send_to_operator`), and
- the send allow-list's always-permitted escape hatch, so an agent with a
restrictive allow-list still has a way to say it is stuck.
What is NOT preserved, deliberately: an agent with no capability no longer
sees any other agent's dirs. `ManageRootAgent`'s own grant is unchanged --
still every agent in the roster, still state RW + config RO, still no
`harness`.
The dashboard's reparenting control (the M0V3 picker) is deleted with its
CSS. The tree rendering that reads `ContainerView.parent` is left for the
frontend owner -- it degrades to a flat list with the field gone.
A record written to stdout carries no priority, so journald files the
whole stream at one level and the swarm log store shows `info` whatever
level `tracing` gave it. Under a systemd unit the process's stdout
already *is* the journal, so the fix is to speak the journal protocol
directly and let each record carry its own severity.
New `hive-log` crate holds the one sink chooser, called by `hive-c0re`,
`hive-agent` and `swarm-controller`. It builds the same `EnvFilter`
those binaries always built, then installs exactly one layer — never
both, since a journald layer stacked on the `fmt` layer under a unit
stores every record twice.
The choice is an fstat compare, not a presence test: a child inherits
`$JOURNAL_STREAM` even when its own stdout was redirected elsewhere, so
the variable existing proves nothing. The crate parses `dev:inode` out
of it and compares both numbers against an fstat of stdout, the
descriptor the `fmt` layer writes to by default. No match, unset, or
unparseable takes the `fmt` branch. A journald layer that fails to
construct despite a match falls back to `fmt` and warns through it —
a process must never fail to start because of its logger.
get_loose_ends now always returns the caller's own loose ends, for every
caller including the manager (ruth) — there is no separate manager
surface, ruth is a normal agent with different default capabilities.
- AgentGetLooseEndsArgs removed; get_loose_ends takes no args.
- Wire Request::GetLooseEnds collapses from an Option<String> target to
a unit variant.
- hive-c0re's handle_get_loose_ends drops the "*" hive-wide branch and
the subtree/capability resolver (resolve_agent_state_target); both
are gone since there is no longer a target to resolve.
- loose_ends::hive_wide and Capability::QueryAgentState removed as
dead code — their only callers were the two functions above.
- is_descendant_of is untouched (still used by lifecycle_handlers.rs
and schedules.rs independently of this change).
- Docs updated: docs/turn-loop/mcp.md, docs/web-ui/dashboard.md,
docs/process/conventions.md (Loose-ends wire shape + capabilities
table), plus the doc comments in hive-core-agent-sock, mcp_config.rs
and capabilities.rs that described the old shape.
Refs #4480
The swarm can already tell whether an agent is alive — the `agent-status`
KV bucket republishes once a minute — but not what it is doing right now.
A header bar wants the second thing, and a minute-old answer to "is this
agent thinking" is the wrong answer most of the time it is read.
`hive-agent` now publishes a turn-state header to
`$SWARM.agent-state.<hive>.<agent>`, a core subject beside the terminal
rows it already sends. It goes out **on transition, not on a timer**: the
publisher watches the event bus, rebuilds the header, and sends only when
the serialised result differs from the last one it sent — so a second
periodic writer, which is the problem this exists to fix, is not what
replaces the bucket.
The payload is the published contract a swarm-level renderer is written
against, so the test asserts on the serialised JSON keys rather than on
Rust field names. Two fields deliberately depart from the per-agent web
UI's `StateSnapshot`: `turn_state_since` is an ISO 8601 UTC string rather
than unix seconds, matching the sibling `$SWARM.term` subject's stamp, and
`agent_state` carries the swarm's own `AgentState` vocabulary rather than
a `paused` boolean, so a reader can compare actual against wanted without
translating. `turn_state` and `agent_state` stay two separate fields:
neither vocabulary contains the other's values.
Swarm-side, `GET /api/agents/{name}/state/stream` relays the subject as
SSE, resolving the agent's hive at request time exactly as the terminal
stream does and passing the bytes through without parsing them.
The broker grant is a second `--agent-publish-subject` rather than a
widening of the existing one, so the terminal family and the header family
stay independently revocable, and a `module-eval` arm pins the rendered
flag and its argument together — the doubled dollar included, since a
single one expands to nothing in `ExecStart` and yields a grant that
matches nothing.
Refs #3802
The harness has had its queue coordinates since the credential reached
the container, but nothing used them. This offers each terminal row
upward on `$SWARM.term.<hive>.<agent>`, so a swarm-level terminal can
render an agent without reaching into the hive that hosts it.
It publishes the same `TermMsg` the web UI is handed rather than a
second model of the same events, so a new tool or a reclassified event
changes both surfaces together. It subscribes to the event bus rather
than to the SSE handler: the handler classifies per connected browser,
so hanging this off it would mean an agent nobody is watching publishes
nothing. That also means its own long-lived `ClassifyCtx`, since a
publisher restarting its correlation state would lose the `tool_use` →
name mapping a `tool_result` needs to render.
The hive in the subject is derived from the queue client id, not from
the harness's hive display name. Those come from different sources with
no rule tying them together, and the responder builds its grant from the
client id — so deriving it from the display name yields a publish the
broker refuses, reaching an operator as a terminal that is merely empty.
The prefix and suffix that bracket the hive are the responder's flags,
which the agent is not told; it restates their defaults, and the symptom
of a deployment retuning one without changing this is every publish
refused rather than a wrong subject accepted.
Oversize rows degrade in the publisher. Exceeding `max_payload` is not a
truncation: the server refuses the message and closes the connection, so
an oversize publish costs the row, the connection, and the rows racing
behind it through the reconnect. The body is the only unbounded field —
summaries are already trimmed at classification — so it is the field
spent, and the row keeps its icon, level, summary and coalesce key. A
row that does not fit even then is logged and dropped rather than sent.
The limit is read off the connection, so `8388608` stays spelled once in
the queue's own module; size is measured by serializing, because JSON
escaping separates character count from wire length by an unbounded
factor on exactly the rows already near the limit.
Best-effort throughout: no queue, an unparseable client id and a failed
connect each disable the publisher with one log line, and a failed
publish loses its row and nothing else. The turn loop and the web UI
never block on the queue.
Refs #3805
hive-c0re stats the two files `swarm-bao-queue-agent` lands and forwards
them into every agent container as systemd credentials, and the harness
resolves a `QueueConfig` out of them at boot. Nothing connects yet.
A credential and not a bind mount, and the mode is what forces it: the
secret is root:0600 and the harness runs as the unprivileged agent user,
so a bind would deliver a file that user cannot open. nspawn's
`--load-credential` is read by the container manager as root and
re-exposed under the consuming unit's own `User=`. hive-c0re never reads
the bytes either way, which is just as well — it runs as `hive-core`.
Absent files stay legal and become visible rather than silent: the
publisher lives on the authelia host and mints on its first boot, so
"nothing at that path" is the ordinary early state of a swarm. c0re
forwards nothing and logs why; the harness logs that it has no queue.
The client id comes out of the delivered file rather than being rebuilt
from `hiveName` in nix, which is the agreement `swarm-secret-client`
states. `QueueConfig::from_env` wants it as a value, so the harness reads
the file itself — assigning the variable instead would need
`std::env::set_var` in a process that has already spawned threads.
Refs #3805
Eleven doc comments pointed at `docs/` files as markdown links. Ten of
them render as broken hyperlinks in the docs rustdoc CI builds, and
nothing in the tree can tell.
Rustdoc renders a page at `target/doc/<crate>/<module…>/`, so a relative
link resolves against that directory and not against the source file it
was typed in. Every one of these except the single crate-root `//!` was
written for a reader resolving from the source tree, which is one `../`
short at module level and two short one directory deeper.
Two measurements on a throwaway crate, same build and same
`RUSTDOCFLAGS="-D rustdoc::all"`:
* a bogus intra-doc link `[`no_such_item`]` is a hard error, so the
`docs-rustdoc` check in nix/checks.nix works for its class;
* a relative link to a nonexistent file in the same comment produces
no diagnostic at all and lands in the html verbatim as
href="../../../docs/does-not-exist.md".
So the class is invisible to the one gate whose stated purpose is to
stop a doc pointer dangling — and it is worse than the plain-text
failure that gate's comment describes, because a broken href still
looks clickable.
Fixing the depths was the other option and is rejected: the correct
depth is a function of how deeply the module is nested, so any module
move silently breaks it again, and no check we have would notice.
The link text was already the canonical pointer — `docs/x.md::Section`,
the same repo-root-relative form used everywhere else in the tree and
the form scripts/check-doc-refs.sh gates. Dropping the `[…](…)` wrapper
keeps every byte of information a reader uses and removes the only part
that was ever wrong.
Refs #3926.
Move terminal-row classification server-side into a new
hive-agent/src/term_msg.rs, replacing the old JSON-mutation
enrich()/stamped-field approach in stream_enrich.rs with one
uniform wire shape: {icon?, level: debug|info|warn|error, summary,
body?, body_format?: markdown|diff, coalesce_key?}. No more per-row
`kind` tag or raw claude-JSON passthrough — every row is the same
shape, with structural identity carried by icon + summary text
instead of a CSS class per row kind.
hive-agent/src/web_ui/stream.rs's history + SSE endpoints now both
call term_msg::classify() and serve TermEnvelope{ts, seq?, msgs}
frames; events that classify to zero rows (agent-state changes,
drop-noise) never reach the wire.
Frontend: classifyEvent.ts collapses from a large per-tool dispatch
tree to a thin TermMsg -> StreamRow adapter. streamRow.ts/Row.tsx
drop the now-dead meta/childText fields. terminal.css switches from
a dozen-odd per-row-kind classes to four level-based color rules.
Expand/collapse of a bodied row is now a uniform client-side
decision (the operator's preference), no server-side per-tool
override.
docs/terminal-rendering.md rewritten to match.
New state_entry_watch.rs, mirroring disk_watch.rs's shape exactly:
periodic in-process probe, Todos::upsert with a stable count-bucketed
summary (anti-nag - drifting inside one bucket stays silent, crossing
a bucket speaks up again), clears once back under threshold.
Top-level entry count only, deliberately - a large subdirectory (git
clone, build tree) counts as one entry regardless of what's inside it,
which is disk_watch's problem to catch on its own axis (bytes), not
this one's.
Wired into main.rs's spawn_todo_socket alongside disk_watch::run.
closes#3464
Closes the #3110 split — lib.rs is now just the crate doc comment and
the pub mod list.
journal.rs's new doc comment fixes a pre-existing bug: the old
JournalPriority doc text in lib.rs was actually half Capability's doc
(a leftover from an earlier reorder that moved the code but not the
comment above it).
Part of hyperhive#1898 (b): every docs subdir should have a top-level
README.md link, achieved by moving/renaming where an existing file
already fits the role.
docs/turn-loop.md already served as the hub + index for the three
sub-pages under turn-loop/ (claude-invocation.md, config.md, mcp.md),
so it moves wholesale rather than leaving a redundant top-level
pointer stub. Fixes every inbound/relative link across the repo
(top-level README.md, CLAUDE.md, docs/persistence.md,
docs/tools/scheduling.md, the sub-pages own back-link, hive-agent
README + doc comments, hive-agent/Cargo.toml, .prettierignore per-file
exemption entry) - grepped the whole tree for both turn-loop.md and
turn-loop/ to find every reference rather than trusting a partial
list.
nix fmt clean, cargo check -p hive-agent clean.
Six places in the tree hand-rolled the same connect / write one JSON
line / read one JSON line back. Two of them — the harness serve loop's
client and the MCP server's — were byte-identical apart from a six-line
wrapper, ~145 lines of literal copy-paste. The other four each
reimplemented a subset, and the subsets had drifted: some named the
socket path in their errors and some did not, one classified transient
against fatal failures and the rest retried nothing at all, two drained
the response and two decoded it.
That duplication was defended when the daemons were split out, on the
grounds that a daemon's socket etiquette should stay visible in the
crate that depends on it. The etiquette genuinely does differ. The code
does not, and five copies is where "each daemon documents its own
etiquette" stops paying for itself.
`hive-sock-client` now owns the transport once, generic over the
request and response types so it is protocol-agnostic: the host-served
control socket and the harness's in-agent socket both use it with their
own wire-type crates. The two real differences become values instead of
forks. Retry is `Retry::RideOutRestart` (2/4/8/16/30s, sized to ride out
a service restart) for callers with no natural retry of their own, or
`Retry::None` for callers already inside a poll loop where the poll
interval is the retry — and the reason each caller picked one is a
comment at the call site rather than a reimplementation. The response is
either decoded (`request`) or half-closed and drained (`notify`, where
the drain exists so the server's write-back doesn't land on a closed
socket). Whether a failure propagates or is logged and swallowed stays
at the call site, because that is the caller's choice and not a property
of the transport.
Errors always name the socket path now, everywhere. That detail is
load-bearing: a permission problem on a socket that reads as "is the
daemon running?" sends the operator to fix the wrong thing.
The transient-against-fatal enum is gone rather than moved. Serialising
happens before the retry loop and deserialising after it, so only
connect, I/O and short-read failures can reach the loop at all — a
deterministic failure is now unretryable by construction instead of by
classification.
It is deliberately a new crate and not part of `hive-agent-sock`. The
`*-sock` crates are pure wire types by convention — `hive-agent-sock`
depends on serde and nothing else — and the two largest copies talk to
the host socket, whose types live in a different crate entirely. A
transport in either wire-type crate would drag tokio into it and point
the wrong way besides.
No wire-format change: same JSON line in, same line out.
The poller was a `tokio::spawn` inside the `hive-agent` serve loop. It
never needed anything from that loop except a socket path, so being
in-process bought nothing and cost two things: a harness restart took
forge notifications down with it, and the whole forge/HTTP dependency
tree was linked into the serve-loop binary.
It is now `hive-forge-notify`, a per-agent daemon with its own systemd
unit, a sibling of `hive-bash-daemon` and `hive-matrix-daemon`. Same
contract as those two: it reaches the harness only by upserting todos on
the in-agent socket, and nowhere else.
The module moves verbatim (`notify.rs`) — the formatters, the activation
gates, the dedupe map and all 33 tests are unchanged. Only the socket
call sites are rewritten, onto a small local `todo_client` rather than
the harness's. That mirrors what both sibling daemons already do, and
the etiquette differs on purpose: the harness's client carries a 60s
backoff schedule sized to ride out a hive-c0re restart, which its
callers need because they have no retry of their own. This poller's two
call sites both sit inside the 30s poll loop and both treat a failure as
"leave the thread unread, try next tick", so the poll interval already
is the retry; a second backoff would only stack sleeps and delay the
rest of the batch.
The unit is `Restart=on-failure`, not `always`. An agent with no forge
account is a supported configuration and the poller reports it by
logging why and exiting 0 — under `always` that clean exit would be a
restart loop on every forge-less agent.
`forgejo-api`, `url` and `time` drop out of `hive-agent`'s dependencies
with the module.
Also corrects docs that outlived the code they described: the persisted
`forge_cursor` field is long gone (forge's own read-state is the durable
record of what has been delivered), but `docs/persistence.md` and the
`harness_state` module docs still documented it as live.
Closes 2718.
The operator has been going through agent dirs by hand with ncdu,
deleting 20+GB target dirs. Agents had no way to know they were the
ones sitting on the space.
New `disk_watch` module in the harness: every 15 minutes it statvfs's
the filesystem backing the agent's state dir and, past 80%, raises a
keyed `disk` todo telling the agent to free space — with the operator's
rules inline: only delete things that are actually big, build output
first, and never delete something still needed, ask for more space
instead.
Over threshold it also walks the agent's own tree (`/agents/<label>`
plus `$HOME`) and names the directories worth looking at, so the todo
says where the bytes actually went rather than just that the disk is
full. The walk is bounded on every axis — entry budget, recursion cap,
report depth — pinned to the state dir's device so it can't wander into
`/nix` or the shared bind mounts, and it does not traverse symlinks. It
reports the deepest oversized directory on each branch, so the agent
gets pointed at `<workspace>/target` rather than at `/agents/<label>`.
Anti-nag is the whole design constraint. The todo is keyed, and the
summary is deliberately stable: the percentage is bucketed to 5 points
and no raw byte counts appear anywhere in it. An unchanged situation
re-upserts as `changed == false` and never fires the wake, so a disk
that has been steady at 89% for a week sits quietly in the loose-ends
list; crossing into a new bucket speaks up once. Dropping back under
the threshold clears the row.
Harness-local by construction, per the operator's call that this gets
no core wiring: hive-c0re cannot push a todo at all (the store and its
wake live inside the container), and running in-process means this
skips even the in-agent socket and calls `Todos::upsert` directly.
Worth recording, since it shaped the scope: btrfs does NOT fold qgroup
limits into statfs. Measured with quota counting enabled and a 20G
limit set on a real subvolume, statvfs returns byte-identical whole-FS
numbers for that subvolume, an ordinary agent dir, and the root. So
this watches host-FS pressure, which is valid before and after the
planned subvolume migration; per-agent quota awareness would need the
limit handed to the agent explicitly.
A paused agent keeps its container, its claude session and its
dashboard/todo servers up, but stops driving turns. Messages queue
unacked and are drained on resume.
The whole protocol is a single marker file, `<harness>/paused`. That
directory is already a bind-mount shared between host and container, so
both sides just stat the same path: the harness reads it to decide
whether to drive a turn, hive-c0re reads it to render the badge and
writes/removes it for `hivectl pause|resume`. No new wire protocol, no
container round-trip, and it is sticky across restarts by construction.
Not calling `recv_next` while paused *is* the queueing semantic, so
there is no fencing to get wrong: reminders buffer in their unbounded
channel, the todo `Notify` permit coalesces, and a `request_next_turn`
that raced the pause survives because the gate sits above
`self_continue.take()`.
Graceful stop is handled host-side rather than in the harness: a paused
agent provably has no turn in flight, so `run_signal` skips the fence
entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for
a checkpoint turn that will never run.
`paused` is reported on `ContainerView` / `AgentStatusRow` for the
dashboard, orthogonal to `running` and reported for stopped containers
too.
Closes: hyperhive/hyperhive issue 2271
diagnostic instrumentation for #2678 (phantom 'you have todos' wakes
after clearing bash-task todos). logs subsystem/key/id/changed on
UpsertTodo, subsystem/key/all/count on ClearTodo, id/count on
MarkTodoDone, and a marker when the serve loop actually consumes a
todo_wake notification. no behavior change - RUST_LOG=debug only.
