Two sites spelled it as a brace group, services.hyperhive.{hiveName,
swarmName}, which no anchored rewrite can handle correctly now that only
one of the two moves; both are written out separately. One of them is an
MCP tool description, so it is rendered into every agent's system prompt.
hive-c0re pushes todos into each agent over the in-agent socket, and
every one of those dials has been failing with EACCES. The socket is
created by todo_server::bind with no mode set at all, so it lands at
0777 & ~umask -- typically 0755. connect(2) on a unix socket requires
*write* permission, and hive-core is neither the socket's owner nor in
its group, so it is locked out.
The tell is the sibling socket. web.sock is bound in the same
directory, by the same process, as the same user, and does set its mode
(0666) immediately after bind. Only the socket missing that call fails,
which is also why no ownership or chown theory explained it: both
sockets share every directory they live in, so anything at the
directory level would have broken them together.
Fix is the two lines web.sock already had. Access control for these
sockets is the containing directory's job, not the socket's -- the mode
here only has to not exclude the host daemon that is supposed to reach
it.
Observable effect: scheduled prompts and message wakes reach agents
again. An agent whose wake is dropped still sees its messages whenever
something else wakes it, so the failure presents as agents that look
healthy but answer late, or not at all if nothing else is waking them.
Part of hyperhive#1898 (b): every docs subdir should have a top-level
README.md link, achieved by moving/renaming where an existing file
already fits the role.
docs/turn-loop.md already served as the hub + index for the three
sub-pages under turn-loop/ (claude-invocation.md, config.md, mcp.md),
so it moves wholesale rather than leaving a redundant top-level
pointer stub. Fixes every inbound/relative link across the repo
(top-level README.md, CLAUDE.md, docs/persistence.md,
docs/tools/scheduling.md, the sub-pages own back-link, hive-agent
README + doc comments, hive-agent/Cargo.toml, .prettierignore per-file
exemption entry) - grepped the whole tree for both turn-loop.md and
turn-loop/ to find every reference rather than trusting a partial
list.
nix fmt clean, cargo check -p hive-agent clean.
Adds services.hyperhive.forge.publicUrl (defaults to the gateway vhost
URL when behindGateway=true, null otherwise). HIVE_FORGE_PUBLIC_URL is
now sourced from it instead of hardcoding https://${forge.domain}
whenever behindGateway is on.
The 4 frontend call sites that built a forge link from
state.forge_public_url now hide the link when that's absent, rather
than guessing http://<browser-hostname>:3000 — a guess that's only
correct by accident once the operator isn't on plain localhost. Fixes
the dashboard H0M3 tile, per-agent-row forge links + agent menu, the
approval-queue PR link, and the per-agent page's own meta-nav forge
link (found during this pass, same defect, not in the original
3-site inventory).
Docs + doc-comments updated to match.
Closes#2831. post_logout's wipe_summary only checked whether cleared.wiped
was empty, so a real .credentials.json blocked by EPERM (nothing made it
into wiped) got reported as 'no credential files present (already logged
out)' in the same breath as a warning saying the delete failed -- self
contradictory and misleading, since the file is still there un-deleted.
Extracted the branch into a pure wipe_summary() helper (wiped file(s) named >
warnings-only failure > genuinely nothing found) with unit tests covering
the exact permission-error shape from the issue.
The per-agent web UI todos flyout (loose-ends v2) had no mark-done
affordance at all — dismissing a todo was only possible via the
cancel_loose_end MCP tool, one id at a time. Add a checkbox per row,
a select-all/select-none/mark-done bulk row, and a new
POST /api/todos/mark-done handler that loops the existing single-id
MarkTodoDone request over the in-agent socket (no new wire request
type needed — the todos list is small, so N same-host round-trips is
cheap).
Fixes#2917
The todo fired on filesystem usage alone, so every agent on a busy host
got woken by a disk it had no power over. Measured case: a 927G volume
at 81%, 742G used, of which one agent's entire tree was 5.3G. The todo
cost that agent a full turn to arrive at "not actionable".
`summary_for` now also requires a non-empty `big_dirs()` result. An
agent that owns no oversized directory cannot free meaningful space, so
there is nothing to say to it; the shared store filling up is the host's
signal. Agents that ARE sitting on a stale `target/` still get told, with
the same bucketed anti-nag summary as before.
Drops the conditional around the "Biggest directories" section, which is
now unreachable when empty. Existing tests passed `&[]` as a
don't-care — they now pass a directory, since that argument became
load-bearing.
Closes#2759
Six places in the tree hand-rolled the same connect / write one JSON
line / read one JSON line back. Two of them — the harness serve loop's
client and the MCP server's — were byte-identical apart from a six-line
wrapper, ~145 lines of literal copy-paste. The other four each
reimplemented a subset, and the subsets had drifted: some named the
socket path in their errors and some did not, one classified transient
against fatal failures and the rest retried nothing at all, two drained
the response and two decoded it.
That duplication was defended when the daemons were split out, on the
grounds that a daemon's socket etiquette should stay visible in the
crate that depends on it. The etiquette genuinely does differ. The code
does not, and five copies is where "each daemon documents its own
etiquette" stops paying for itself.
`hive-sock-client` now owns the transport once, generic over the
request and response types so it is protocol-agnostic: the host-served
control socket and the harness's in-agent socket both use it with their
own wire-type crates. The two real differences become values instead of
forks. Retry is `Retry::RideOutRestart` (2/4/8/16/30s, sized to ride out
a service restart) for callers with no natural retry of their own, or
`Retry::None` for callers already inside a poll loop where the poll
interval is the retry — and the reason each caller picked one is a
comment at the call site rather than a reimplementation. The response is
either decoded (`request`) or half-closed and drained (`notify`, where
the drain exists so the server's write-back doesn't land on a closed
socket). Whether a failure propagates or is logged and swallowed stays
at the call site, because that is the caller's choice and not a property
of the transport.
Errors always name the socket path now, everywhere. That detail is
load-bearing: a permission problem on a socket that reads as "is the
daemon running?" sends the operator to fix the wrong thing.
The transient-against-fatal enum is gone rather than moved. Serialising
happens before the retry loop and deserialising after it, so only
connect, I/O and short-read failures can reach the loop at all — a
deterministic failure is now unretryable by construction instead of by
classification.
It is deliberately a new crate and not part of `hive-agent-sock`. The
`*-sock` crates are pure wire types by convention — `hive-agent-sock`
depends on serde and nothing else — and the two largest copies talk to
the host socket, whose types live in a different crate entirely. A
transport in either wire-type crate would drag tokio into it and point
the wrong way besides.
No wire-format change: same JSON line in, same line out.
The poller was a `tokio::spawn` inside the `hive-agent` serve loop. It
never needed anything from that loop except a socket path, so being
in-process bought nothing and cost two things: a harness restart took
forge notifications down with it, and the whole forge/HTTP dependency
tree was linked into the serve-loop binary.
It is now `hive-forge-notify`, a per-agent daemon with its own systemd
unit, a sibling of `hive-bash-daemon` and `hive-matrix-daemon`. Same
contract as those two: it reaches the harness only by upserting todos on
the in-agent socket, and nowhere else.
The module moves verbatim (`notify.rs`) — the formatters, the activation
gates, the dedupe map and all 33 tests are unchanged. Only the socket
call sites are rewritten, onto a small local `todo_client` rather than
the harness's. That mirrors what both sibling daemons already do, and
the etiquette differs on purpose: the harness's client carries a 60s
backoff schedule sized to ride out a hive-c0re restart, which its
callers need because they have no retry of their own. This poller's two
call sites both sit inside the 30s poll loop and both treat a failure as
"leave the thread unread, try next tick", so the poll interval already
is the retry; a second backoff would only stack sleeps and delay the
rest of the batch.
The unit is `Restart=on-failure`, not `always`. An agent with no forge
account is a supported configuration and the poller reports it by
logging why and exiting 0 — under `always` that clean exit would be a
restart loop on every forge-less agent.
`forgejo-api`, `url` and `time` drop out of `hive-agent`'s dependencies
with the module.
Also corrects docs that outlived the code they described: the persisted
`forge_cursor` field is long gone (forge's own read-state is the durable
record of what has been delivered), but `docs/persistence.md` and the
`harness_state` module docs still documented it as live.
After each notification poll, query GET /api/v1/issues/search with
assigned=true for both issues and pulls, read X-Total-Count, and
upsert/clear a keyed 'rollup' todo (subsystem=forge, key=rollup).
When the total is > 0 the summary reads e.g. '3 open assigned: 2 issues,
1 PR'. When it drops to 0 the rollup todo is cleared. The 'rollup' key is
distinct from per-thread numeric keys so clearing it never touches
notification todos.
Closes#2725.
Closes 2718.
The operator has been going through agent dirs by hand with ncdu,
deleting 20+GB target dirs. Agents had no way to know they were the
ones sitting on the space.
New `disk_watch` module in the harness: every 15 minutes it statvfs's
the filesystem backing the agent's state dir and, past 80%, raises a
keyed `disk` todo telling the agent to free space — with the operator's
rules inline: only delete things that are actually big, build output
first, and never delete something still needed, ask for more space
instead.
Over threshold it also walks the agent's own tree (`/agents/<label>`
plus `$HOME`) and names the directories worth looking at, so the todo
says where the bytes actually went rather than just that the disk is
full. The walk is bounded on every axis — entry budget, recursion cap,
report depth — pinned to the state dir's device so it can't wander into
`/nix` or the shared bind mounts, and it does not traverse symlinks. It
reports the deepest oversized directory on each branch, so the agent
gets pointed at `<workspace>/target` rather than at `/agents/<label>`.
Anti-nag is the whole design constraint. The todo is keyed, and the
summary is deliberately stable: the percentage is bucketed to 5 points
and no raw byte counts appear anywhere in it. An unchanged situation
re-upserts as `changed == false` and never fires the wake, so a disk
that has been steady at 89% for a week sits quietly in the loose-ends
list; crossing into a new bucket speaks up once. Dropping back under
the threshold clears the row.
Harness-local by construction, per the operator's call that this gets
no core wiring: hive-c0re cannot push a todo at all (the store and its
wake live inside the container), and running in-process means this
skips even the in-agent socket and calls `Todos::upsert` directly.
Worth recording, since it shaped the scope: btrfs does NOT fold qgroup
limits into statfs. Measured with quota counting enabled and a 20G
limit set on a real subvolume, statvfs returns byte-identical whole-FS
numbers for that subvolume, an ordinary agent dir, and the root. So
this watches host-FS pressure, which is valid before and after the
planned subvolume migration; per-agent quota awareness would need the
limit handed to the agent explicitly.
A paused agent keeps its container, its claude session and its
dashboard/todo servers up, but stops driving turns. Messages queue
unacked and are drained on resume.
The whole protocol is a single marker file, `<harness>/paused`. That
directory is already a bind-mount shared between host and container, so
both sides just stat the same path: the harness reads it to decide
whether to drive a turn, hive-c0re reads it to render the badge and
writes/removes it for `hivectl pause|resume`. No new wire protocol, no
container round-trip, and it is sticky across restarts by construction.
Not calling `recv_next` while paused *is* the queueing semantic, so
there is no fencing to get wrong: reminders buffer in their unbounded
channel, the todo `Notify` permit coalesces, and a `request_next_turn`
that raced the pause survives because the gate sits above
`self_continue.take()`.
Graceful stop is handled host-side rather than in the harness: a paused
agent provably has no turn in flight, so `run_signal` skips the fence
entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for
a checkpoint turn that will never run.
`paused` is reported on `ContainerView` / `AgentStatusRow` for the
dashboard, orthogonal to `running` and reported for stopped containers
too.
Closes: hyperhive/hyperhive issue 2271
per mara's review on #2679: replace the separate dispatch_todo sub-match
(with its trailing unreachable! arm) with four small handler functions
called directly from dispatch's existing match. same behavior, no
unreachable! left in the todo path.
diagnostic instrumentation for #2678 (phantom 'you have todos' wakes
after clearing bash-task todos). logs subsystem/key/id/changed on
UpsertTodo, subsystem/key/all/count on ClearTodo, id/count on
MarkTodoDone, and a marker when the serve loop actually consumes a
todo_wake notification. no behavior change - RUST_LOG=debug only.
