Its three error paths returned a bare string with a status code, which forces
a caller to treat the whole body as prose. hive-c0re converted some time ago,
so swarm-controller was the last backend on the old shape -- and it is the one
behind the hive status page's 503, where the body is frequently the entire
diagnosis rather than a summary.
Adds the commitment to docs/conventions.md, since it was implied by the code
in one daemon and written down nowhere: an endpoint of ours answering with a
bare string is a bug to file, not something callers work around.
The test asserts the rendered response -- media type plus an addressable
detail -- rather than the problem_details value, because a handler that built
the value correctly and then returned it as a string would satisfy a test
written against the type alone.
Every admitted client got the same unrestricted grant, so any hive could
write any other hive's status key. The responder now derives a
permission set from the caller's identity and mints it into the user
JWT.
A hive may publish to its own KV key and the two JetStream subjects
needed to reach it; the controller may list and fetch every key and
write none; anything else is denied outright. Deny is the default
because every other shape fails open, and silently: a client that
matched no rule and kept the old grant would make the policy advisory.
The subject sets are measured rather than reasoned about, and two of
them are counter-intuitive. `$KV.<bucket>.<key>` alone does not let a
client write that key, because the client resolves the bucket first. And
`$JS.API.>` is not "the JetStream permission": it also covers
`$JS.API.STREAM.DELETE`, with which a hive correctly refused on a
neighbour's key can delete the whole bucket and every hive's data with
it. Granting it would have made per-key scoping decorative, so the
subjects are named individually and a test asserts the wildcard does not
come back as a convenience.
Minimality is by removal: each subject was dropped in turn to confirm
the client breaks without it. That is not pedantry — an additive search
had called a set minimal while two of its five subjects were never
needed, which ships an unnecessary grant with a measurement attached
making it look earned.
Both grants include `STREAM.CREATE` on the one named stream, because
`status::open_or_create` is called by both ends: either may arrive first
on a fresh swarm, and without it a new swarm never gets a bucket at all.
`CREATE` is not `UPDATE`, so a second arrival cannot reshape the bucket
the first one made.
`status::BUCKET` moves out from behind the `kv` feature so this
responder can share it. The name is a `&str` with no dependencies and
only `open_or_create` needs JetStream; gating the name forced a third
consumer to choose between a stack it does not use and a copied literal,
and the copied literal is exactly the disagreement that module exists to
prevent.
Only publish is scoped. Subscription permissions are unrestricted and
unmeasured, and the module docs say so rather than implying a property
nothing established.
The controller reads per-hive status out of a JetStream KV bucket and
nothing was writing one, so every hive rendered `never_reported`. This is
the half that makes the read path mean anything.
A hive offers; the controller never reaches down to collect. The gateway
has gone down in a way where every recovery channel ran through the one
broken thing, so a status path that depended on the controller would go
dark exactly when it is needed to diagnose the controller's own network.
What it publishes is what the hive already says about itself —
`warnings::readiness()`, the same value `/health/ready` serves. Nothing
here stamps a time: freshness is derived by the reader from when the value
landed, so a hive cannot make itself look fresher than it is, and a hive
with a wrong clock skews only its own payload.
The key is this hive's `hiveName`, which `swarm.nix` already asserts is a
key of `swarm.hives` — so a hive that evaluates at all publishes under a
name the roster knows, rather than by convention.
Publish first, then wait: a hive that has just come up is the one whose
status someone is looking at, and sleeping first would make every restart
read stale for a full interval. The interval is one decision with the
controller's staleness threshold, not two — a ratio of 2 means one lost
publish still reads fresh and two consecutive misses read stale.
Failures go to the dashboard banner through SweepHealth, debounced, at
`warn` and deliberately not `crit`: `crit` is what makes this hive report
itself degraded, and a hive that cannot reach the queue is not unhealthy —
the swarm's view of it is. Publishing `degraded` because the publish
failed would be both false and self-erasing on the next tick.
Operator ruling: libs should not use anyhow. The queue connect was moved
here verbatim from swarm-controller, which is a binary, so it arrived
still wearing a binary's error handling — the move changed what the code
is without changing how it reports.
Callers get variants they can match on, split by what an operator does
about them: a half-configured environment is a deployment bug, a refused
token is an identity-provider config problem, an unreachable queue is a
network one. The binaries that consume this keep anyhow and `?` converts,
so nothing downstream is more verbose for it. Same split hive-claude uses.
One thing anyhow was doing unpaid: the auth callback hands async-nats a
plain string, and a Display that stops at the top message drops the cause
— the half that says why the mint failed. `chain()` walks the source
chain, which is what `{:#}` was doing before.
Its only user was queue.rs, which is now swarm-queue-client. An unused
Cargo.toml dependency is not a build error, which is exactly why it
survives: the next reader takes it as still needed and copies it
forward.
A hive publishing its own status needs the same connect the controller
already has - mint an authelia token, present it at CONNECT for the
callout responder, let async-nats re-run the callback per attempt. Only
the use differs: the controller reads, a hive writes.
Copying it would put credential handling in two places, and a
token-refresh fix would then have to be found twice. That is the same
reasoning that already put hive-sock-client in its own crate rather than
in each daemon that speaks to a unix socket.
`from_env` takes a prefix rather than hardcoding SWARM_CONTROLLER_*: the
variables belong to the consuming unit, since a NixOS module sets them
alongside its other options. What is shared is the RULE - all four
together or none at all - not the spelling. The half-set case gains a
test, because it is the case the rule exists for and it previously had
none.
No jetstream/kv feature on the crate: it ends at a connected client, and
what a consumer does with it should be visible in that consumer's own
Cargo.toml.
Behaviour-preserving, and proven that way rather than by inspection: the
full behavioural gate (real nats-server, credential rotation, mutation)
is 20/0 unchanged, and the controller's own tests still pass.
The controller connects to the swarm queue as its own client and serves
what each hive last said about itself at GET /api/hives/status.
THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream
KV bucket (history 1) and the controller reads it per request, keeping no
copy. A cache here would be a second answer to the same question, free to
disagree with the first, and the disagreement surfaces as a hive reading
healthy on a dashboard while the bucket says otherwise. Whichever side
arrives first creates the bucket; both want the same shape.
Rows come from the roster rather than from the bucket, so an empty bucket
renders as a swarm nobody has heard from instead of a healthy one, and
`never_reported` stays distinct from `stale` - went quiet is a fault,
never spoke is usually a deployment that has not happened. Freshness is
derived at read time and never stored as a flag, because a stored
`healthy` boolean goes stale silently the moment nothing arrives, which
is the failure this endpoint is designed against. The timestamp is the
NATS server's, applied when the value landed, so a publisher cannot make
itself look fresher than it is.
Authentication is per connection attempt, not per process. Authelia
issues `client_credentials` tokens that expire in 3599s, and auth happens
at CONNECT, so a long-lived connection is fine but a reconnect an hour
later needs a token minted an hour later. `with_auth_callback` is re-run
by async-nats for each attempt, which handles expiry by construction
rather than by a timer - the alternative fails in the way this subsystem
exists to prevent, with the controller still serving while its data
quietly stops updating.
Three failure shapes are deliberate:
- A half-set environment is fatal; an absent one is not. Silently
behaving like an unconfigured host is how every hive ends up reading
`never_reported` with nothing to point at.
- The endpoint answers 503 rather than an empty list when the store
cannot be read. "I cannot reach the store" and "every hive is silent"
are different answers, and rendering the second turns a local fault
into an apparent swarm-wide outage.
- `retry_on_initial_connect` makes the daemon and the queue bootable in
either order, and the status handler refuses when the client is not
Connected rather than issuing a request into it - a request made in
that window does not fail, it waits, so every poll would hang and
learn nothing. `Pending` is the state a never-connected client is in,
which is why the test is `!= Connected` and not `== Disconnected`.
The rendering rules are a pure function over a map, so the semantics are
tested against a table rather than against a running server. The KV read,
the credential rotation and the 503 paths are covered behaviourally
instead: a real NATS server with a rotating token endpoint, asserting
that the controller recovers only when the credential rotates, and
mutation-tested by holding the credential wrong for the same window.
Per mara on the PR: all deps go into workspace level so versions stay
consistent. async-nats, data-encoding, nkeys and nats-jwt are new
[workspace.dependencies] entries; sha2 and futures-util now come from
there.
The crate had asked for sha2 0.10 while the workspace standard is 0.11,
and for the futures facade where the workspace carries futures-util. Both
resolved without adding a Cargo.lock entry - each was already in the tree
via something else - so the drift was invisible in the lock and would only
have surfaced as two versions of a hashing crate in one binary.
Slice 1 shipped the NATS container with an auth_callout block and no
responder, which is the fail-closed state: the server answers
auth_required and admits nobody. This crate is what lets it say yes.
Connects as the callout-exempt user by nkey (never by name - the server
refuses to start if that entry carries a username), subscribes to
$SYS.REQ.USER.AUTH, validates the presented bearer token against
authelia's introspection endpoint, and replies with a signed NATS user
JWT. A denial is a signed response carrying an error, never silence: a
server that hears nothing cannot tell a refusing responder from a dead
one, so staying quiet would turn every rejection into a timeout and hide
an outage inside what looks like ordinary denials.
Everything that is not an explicit active:true denies - network error,
timeout, non-2xx, unparseable body, no token at all. Those are exactly
the conditions under which an attacker would most like this to fall
open. The introspection budget is held under the server's own 2s
auth_callout timeout by a test, since the two numbers live in different
languages in different files.
nats-jwt mints the user JWT. It cannot mint the authorization_response
wrapper - its claim enum is closed and its claims carry no aud, which
the response needs so a reply cannot be replayed at another server in
the cluster - so that half is hand-written, and a test builds a user
token both ways and requires the bytes to match. That is the only
honest basis for trusting the hand-written path on the shape the crate
does not model.
async-nats is taken with default-features off: the default set carries
jetstream, kv, object-store, websockets and service, none of which a
callout responder speaks.
Fixes hyperhive#3223.
swarm-controller: GET /api/hives (utoipa-annotated same as /health),
serving the swarm's hive directory (name + domain) loaded once at
startup from a new SWARM_CONTROLLER_HIVES env var. The controller's
NixOS module sets it from services.hyperhive.swarm.hives, JSON-encoded
the same way hive-c0re already builds HYPERHIVE_PEERS for its own peer
list (environment.nix) — the full directory here rather than
peers-minus-self, since a swarm-level daemon has no 'self' hive to
exclude. Unset/malformed both fall back to an empty list with a
warning rather than failing startup, so /health stays answerable even
if this one env var is wrong.
swarm-ui: App.tsx's Home route fetches /api/hives and renders it
through the already-merged <Table>/<StatusChip>/<Panel> primitives —
name, domain (linking out to that hive's own gateway-routed
dashboard), and a static "configured" status chip until a real
online/stale/offline rollup exists server-side. Also gave swarm-ui a
base <a> color (theme's --blue) — base.css covers body/typography but
not links, and this is genuinely page-level rather than any one
component's concern.
Verified end to end, not just source-reading: ran the real
swarm-controller binary with SWARM_CONTROLLER_HIVES set, curled
/api/hives + /health over its actual unix socket; separately served
the real swarm-ui dist against a mock /api/hives and screenshotted the
rendered table. Also re-verified the nginx wiring evaluates (same
throwaway nixosSystem eval technique as #3212) — SWARM_CONTROLLER_HIVES
resolves to the expected JSON shape.
cargo test/clippy -p swarm-controller clean (2 tests, including a new
load_hives one covering missing/malformed/valid env var states). npm
run build + typecheck clean.
Rust half mirrors hive-c0re/src/dashboard/mod.rs's utoipa pattern
exactly: an ApiDoc root, #[utoipa::path(...)] on /health (the one
existing route), and a raw JSON route at /api/openapi.json served
via OpenApiRouter::split_for_parts(). Only annotated routes appear
in the spec.
Gateway wiring extends the swarm-UI vhost (the only vhost swarm-
controller is reachable from) with:
- /api/ — proxied to the controller's unix socket untouched (no URI
segment after the socket path), so a route swarm-controller
registers is the path nginx forwards, no prefix-stripping to keep
in sync by hand.
- /api/docs/ (+ the bare /api/docs redirect) — the same
swagger-ui-theme dist the per-hive dashboard already serves at its
own /api/docs/, reused as-is since it's generic.
Both new locations reuse the same auth_request block the vhost's own
'/' already applies, factored into a shared swarmAuthRequest string —
auth_request does not inherit across sibling nginx locations, so
without this the page itself would be gated while its own API and
API docs sat open.
cargo test -p swarm-controller + cargo clippy --all-targets both
clean. Verified the new nginx wiring evaluates correctly with a
throwaway nixosSystem eval (services.hyperhive.swarm.{controller,ui}
enabled): /api/ proxies to the socket, /api/docs redirects, and both
require auth_request the same as the vhost root.
Fixes hyperhive#3212
Mirrors hivectl exactly: a `completions <shell>` verb that walks the
live clap tree, and a package that pipes it into installShellCompletion
for bash/zsh/fish. Generating from the command tree rather than writing a
script by hand is what keeps completions from drifting away from the
verbs they complete — the same reason `markdown-docs` renders the docs
from that tree.
Dispatched before PathArgs::resolve() for the same reason markdown-docs
is: emitting a completion script needs none of the SWARMCTL_AUTHELIA_*
deployment env vars, and requiring them would make the package's own
build-time invocation fail — exactly where it runs.
swarmctl leaves mkBinPackage for its own derivation, since the extractor
installs a binary and nothing else.
Adds swarmctl markdown-docs (a hidden Verb, same clap-markdown +
hide=true shape as hivectl markdown-docs) and generates
docs/tools/swarmctl-cli.md from it. Wires a swarmctl-docs freshness
check into nix/checks.nix, same shape as hivectl-docs, diffing against
packages.swarmctl.
One real gotcha: PathArgs::resolve() reads required
SWARMCTL_AUTHELIA_* deployment env vars and errors if unset -
swarmctl markdown-docs must not go through that path (it needs none of
those vars, and the docs build runs it outside any real deployment).
Restructured main() so resolve() only runs for the User arm, not
unconditionally before the match.
Also links the new doc from docs/tools/README.md (new 'for the swarm
operator' section), CLAUDE.md's swarmctl bullet, and
docs/conventions.md's flake-check list.
Verified: cargo check/clippy -D warnings/test/fmt -p swarmctl all
clean; swarmctl markdown-docs diffs clean against the committed doc
(checked against both a plain cargo build and the actual nix build.
#swarmctl output); scripts/check-issue-refs.sh clean.
The swarm-authelia module states that its users database is written by
swarm-controller, but nothing ever granted the means. This adds the tool
that does it.
swarmctl runs as root on the controller's host and acts directly. The
rootless alternative was examined and does not work: relocating the users
file into a directory the controller owns only turns a write problem into
a read problem, because authelia must then reach across the same boundary
in the other direction. Making that read work needs either a hand-pinned
gid or world-readable password hashes.
The user store is two files, one authoritative: users.json is canonical,
users.yml is a rendered artifact. That split is what lets the crate work
without a YAML parser -- the workspace has none, and adding one costs a
crates.io fetch, a lock update and a vendor hash for a schema we fully
control and only ever emit.
Passwords are generated by authelia rather than passed to it: argv is
world-readable, so a password on a command line is readable by any local
process for the lifetime of the call.
The three derived facts swarmctl needs about the authelia container --
machine, unit and the host-side users path -- become readOnly options on
the authelia module rather than literals repeated at the call site.
First half of the swarm-controller slice: the crate, its workspace entry
and its daemonBins entry, so the systemd unit that follows has a binary
to point at.
It serves one health endpoint and owns no state. That is the whole
intent -- this makes the unit real (service user, runtime and state
directories, socket, nginx reachability) so the swarm-level surfaces
that follow have somewhere to land. Inventing those surfaces now would
bake in a shape nobody has agreed to.
The socket gets its own runtime directory rather than sharing
hive-c0re's. nginx reaches a unix upstream by having the socket's
directory bind-mounted into the gateway container, so co-locating this
socket with the host admin socket would hand the gateway that socket
too.
Three things, all from review:
Accepted outcomes were built from a hand-listed [Done, Failed, Cancelled,
Skipped] array. Exhaustive today, silently short the day someone adds a
variant — the new outcome would vanish from every edge that accepts it.
BitFlags::ALL asks the type instead.
TerminalState carried rename_all = "snake_case" while its sibling State did
not, so one enum shipped "done" and the other "Done". A rename is a second
spelling of a name that then has to be kept in agreement by hand; both now
serialise their variant names verbatim. Nothing else reads TerminalState off
a wire, so no consumer moves. GraphDep's tag values likewise.
The endpoint documented its body as serde_json::Value, which tells a spec
reader nothing. hive-jobq-wire now derives ToSchema. State and TerminalState
are foreign types here and utoipa stays out of the scheduler crate, so the
schema points at local mirror enums. A mirror that drifts is worse than none:
the conversions are exhaustive (a new upstream variant fails the build) and a
test asserts each documented name equals the serialised one, since an
exhaustive match still compiles when only the spellings diverge.
The wire types were in hive-host-sock, which is the host *socket* crate — so
anything living there is core-shaped by construction, and the projection had
quietly grown two core dependencies to match: it selected roots by matching
NodeKind::Dag, and rendered payloads through free functions in hive-c0re that
nothing obliged a second host to write.
hive-jobq is the wrong home too. That crate is the scheduler — logic — and
folding presentation in means every consumer of it carries a JSON vocabulary
it may never serve.
So: a new hive-jobq-wire. A host implements WireNode for its payload N and
WireResource for its resource name R; GraphWire::wire_snapshot is
blanket-implemented for Graph<N, R> when both hold, and for nothing else. A
payload that has never said how it displays has no way onto the wire.
wire_snapshot takes the roots to serve rather than reading Graph::roots
itself. Nothing is ever removed from a Graph, so retention is a policy only
the host can hold; hive-c0re passes visible_roots(), which is the existing
MAX_HISTORY_DAGS bound selected structurally (a root is a node with no
parent) instead of by node kind.
`jobs::NodeView` can only ever display hive-c0re's queue. Five of its
fields are domain knowledge: `approval_id` is only ever on a
`DeployWindow`, `inputs` only on a `MetaLock`, `build_log_id` only on the
nix-heavy kinds, `agent` is derived from the payload, and `kind` is a
payload tag consumers branch on. A component built against that shape
cannot render a second jobq.
`graph::GraphNode` is `hive_jobq::Node` with both generics erased: the
crate's own field set, with everything domain-specific in one opaque
`payload.data` slot the consumer renders without branching on. That is
the crate boundary made visible — hive-jobq owns structure, its host owns
meaning — and it is the same split #2957 drew inside the code.
Two details that are easy to get wrong and are pinned by tests:
`GraphDep::Node` carries `accepts` as the **set** of terminal outcomes,
not a strong/weak flag. A template emits its tails as a pair edged on the
same upstream node, and the only thing telling them apart is which
outcomes each accepts; collapsing that renders two structurally different
nodes identically.
There is **no roll-up field**. A group root ships as an ordinary node
with `parent: None`, and its own `state` is its subtree's answer —
`Finishing` means "own logic done, children still running", the terminal
states are the rolled-up outcome. A separate field would be a lossier
copy: `DagView::rollup_state` flattens `Running` and `Finishing` into
one, which is exactly the distinction a viewer wants.
`State` and `TerminalState` are re-exported from `hive-jobq` rather than
redeclared, so they cannot drift from the scheduler that produces them.
Follow-up from #2955 (mara: 'make core able to give agent a todo').
First migration slice: Spawned was pure FYI-check-when-convenient
material, not something needing an immediate turn.
Coordinator::push_todo/push_todo_submitter do a best-effort live dial
of the target agent's own hive-agent-sock (hive_host_sock::agent_todo_
socket), sending the exact UpsertTodo request in-container producers
(matrix/bash/forge-notify) already send. Push, not queue: agent
offline (socket absent) or dial failure is a silent no-op, no retry,
no fallback delivery -- matches mara's 'not available if offline'
call exactly.
HelperEvent::Spawned removed entirely (enum variant + all 3 call
sites migrated: handle_spawn's two arms, finish_approval's Spawn
approval-kind arm) rather than kept alongside a translation layer --
per mara's correction on the first design attempt, migrating the
producer means deleting the old path, not bridging it.
Verified: cargo build/clippy/test -p hive-c0re -p hive-host-sock
-p hive-sh4re clean (318 tests), nix fmt clean.
Swagger UI itself is nginx-hosted now (iris's 86a39c4c), so c0re
carrying its own vendored copy via utoipa-swagger-ui was a straight
duplicate — dropped the dependency (root Cargo.toml + hive-c0re's),
swapped the SwaggerUi::new(...).url(...) mount for a plain
/api/openapi.json GET route serving the same OpenApi doc as JSON.
Verified: cargo build/clippy/test -p hive-c0re clean, Cargo.lock
dropped utoipa-swagger-ui + utoipa-swagger-ui-vendored with no other
changes, nix fmt clean.
Third time the operator asked for a guid and got a substitute: first an
i64 index, then a {random job id, per-builder counter} pair. The pair was
defensible in isolation -- a foreign handle misses rather than colliding,
with no new dependency -- but "an equivalent that avoids a dep" is a
counter-proposal, not an implementation.
It is also simpler as a guid, which was the question asked: NodeGuid(Uuid)
drops the `job` field, the `next_seq` counter, `fresh_job_id()` and the
`Cell` import, and halves the type's doc. One random draw per node rather
than one per builder -- noise next to what a node does when it runs.
uuid 1.24 was already in Cargo.lock as a transitive dependency, so this
adds an edge rather than a package.
fixeshyperhive/hyperhive#2841. hive-claude was extracted to its own
repo a while back (hyperhive/hive-claude) but this workspace kept
vendoring a path dependency on an in-tree copy - two copies with
nothing keeping them in sync.
drops "hive-claude" from workspace members, switches the dependency
to a git source pinned at hive-claude's current main (2900cc2), and
deletes the in-tree hive-claude/ directory.
this is the workspace's first git-sourced cargo dependency. verified:
cargo build/test/clippy -p hive-agent (the only consumer) all clean,
nix fmt 0 files changed, nix flake check all green (crane vendors the
git dep from Cargo.lock via its own fixed-output derivation, no extra
plumbing needed). network reachability + credentials for the
canonical host checked separately on hive-claude#1 - public repo, no
auth needed, hive-ci confirmed reachable; hive-priv (host netns) is
the one path nobody can test from a container, atlas is running a
one-agent canary rebuild once this merges as the real confirmation.
`hive_host_sock::jobs::State` was a hand-maintained copy of
`hive_jobq::State` — five variants spelled the same in both, kept in sync
by whoever remembered. Adding `Skipped` last week meant adding it twice.
The wire crate now re-exports the scheduler's enum and `to_wire_state` is
gone.
Two states that were hidden now reach clients. `to_wire_state` renamed
`Pending` to `Queued` and folded `Finishing` into `Running`, so the
dashboard could not distinguish a node waiting on its dependencies from
one whose own work is done while its sub-nodes still run. Both are now
visible, and consumers say which they mean.
Every consumer had to move with it, and only the Rust ones said so: the
exhaustive matches in `hivectl` and `DagView::rollup_state` failed to
compile, while the dashboard's fourteen string comparisons would have
gone quietly wrong — a `finishing` node no longer counting as running,
a `pending` node no longer as queued.
The frontend also builds CSS class names out of the state string
(`rqe-` + state, `rqe-node-` + state) and keys its glyph map on it, all
lowercase. Those go through a `stateSlug` helper now; comparisons use the
wire spelling, presentation lowercases. Without that split every queue
entry and node chip would have silently lost its styling.
Dropping the `State as JobState` alias in hive-c0re falls out of this:
the alias only existed to tell two `State` types apart, and there is one
now.
Review follow-ups on #2785.
`DepWhen` wraps `BitFlags<TerminalState>` instead of a hand-rolled `u8`,
so the bit manipulation belongs to the library and `TerminalState` gains
its flag value from `#[bitflags]` rather than a `bit()` match anyone
could get wrong. `of`/`accepts`/`is_empty` become one-liners over it.
Serialization is written out by hand rather than derived: clippy's
`unsafe_derive_deserialize` fires on deriving over a type with unsafe
internals, and the honest fix is to say what the wire form is. It is now
the list of accepted outcomes — `["done","failed"]` — which reads better
than a bitmask and survives the bits being renumbered.
Also drops the `pub use` re-export of `DepWhen` / `TerminalState` from
hive-c0re's `model`. It existed so that `use super::model::…` kept
compiling, which is a shim for one consumer's convenience; the sites
import from `hive_jobq` directly now.
And removes comments narrating what the code used to be. Git holds that.
The DagView / NodeView / Source / State / PermPayload types only ever
travel on the host admin socket and the dashboard channels hive-c0re
serves off the same snapshot; their whole consumer set is hive-c0re,
hivectl and the socket protocol crate itself. Living in hive-sh4re made
the five other crates that depend on it carry job-queue types they never
name.
Pure move: git mv of the module plus the import sweep, no type changes.
hive-sh4re keeps its own chrono (wire_time still needs it).
`Coordinator::set_paused` wrote the marker directly with `std::fs::write`
from hive-c0re, which runs as the unprivileged `hive-core` user. The
agent's harness dir is chowned to the agent user on every container boot
(`user.nix`'s activation chown), mode 0755 — so hive-core can stat the
marker but gets EACCES creating or unlinking it. Pause therefore only
ever worked on an agent that had never booted; the read side works
because a stat needs traverse, not write, which is why the paused pill
and `is_paused` looked healthy.
Route both directions through hive-priv, the root helper that already
owns the other writes into agent-owned directories:
- `PrivRequest::SetAgentPaused { agent_name, paused }`, with the marker
filename constant moved to hive-priv-sock. That is the narrowest crate
all three sides share (hive-priv deliberately does not depend on
hive-sh4re, which re-exports it for the in-container resolver). A
private copy on any one side would break pause silently, since every
reader just sees "no marker".
- `write_agent_state_file` generalised to `write_agent_dir_file`, taking
the target directory: `state/` and `harness/` are both agent-owned,
which is the same reason both need root.
- resume unlinks via `remove_file`, which acts on the leaf and never
follows a symlink — an agent could otherwise plant a link at the
marker path and have root delete an arbitrary file.
`Coordinator::set_paused` becomes an async round-trip; its three call
sites were already async. Both directions stay idempotent because the
dashboard toggle and `hivectl pause|resume` fire without reading the
current state first.
The agents root is 0700 and owned by the daemon's user, so hivectl's
client-side existence guard hit EACCES on traversal for anyone not root.
It reported that as "this command needs root; re-run with sudo", which
turned three verbs' pre-flight check into a permission error about the
wrong thing: `choom`, `subvol upgrade` and `subvol snapshot create` all
failed at the guard rather than at whatever they actually needed.
The daemon runs as the owning user and already answers this question for
its own provisioning paths, so expose it on the host socket as
`AgentExists` and have hivectl ask. Operators reach that socket through
the `hive-admin` group, so the guard now works without sudo.
`choom` still needs root for `machinectl shell` — we ship no polkit rule
granting those actions — so it now checks the effective uid and says so
directly instead of failing later inside systemd's authorisation.
Six places in the tree hand-rolled the same connect / write one JSON
line / read one JSON line back. Two of them — the harness serve loop's
client and the MCP server's — were byte-identical apart from a six-line
wrapper, ~145 lines of literal copy-paste. The other four each
reimplemented a subset, and the subsets had drifted: some named the
socket path in their errors and some did not, one classified transient
against fatal failures and the rest retried nothing at all, two drained
the response and two decoded it.
That duplication was defended when the daemons were split out, on the
grounds that a daemon's socket etiquette should stay visible in the
crate that depends on it. The etiquette genuinely does differ. The code
does not, and five copies is where "each daemon documents its own
etiquette" stops paying for itself.
`hive-sock-client` now owns the transport once, generic over the
request and response types so it is protocol-agnostic: the host-served
control socket and the harness's in-agent socket both use it with their
own wire-type crates. The two real differences become values instead of
forks. Retry is `Retry::RideOutRestart` (2/4/8/16/30s, sized to ride out
a service restart) for callers with no natural retry of their own, or
`Retry::None` for callers already inside a poll loop where the poll
interval is the retry — and the reason each caller picked one is a
comment at the call site rather than a reimplementation. The response is
either decoded (`request`) or half-closed and drained (`notify`, where
the drain exists so the server's write-back doesn't land on a closed
socket). Whether a failure propagates or is logged and swallowed stays
at the call site, because that is the caller's choice and not a property
of the transport.
Errors always name the socket path now, everywhere. That detail is
load-bearing: a permission problem on a socket that reads as "is the
daemon running?" sends the operator to fix the wrong thing.
The transient-against-fatal enum is gone rather than moved. Serialising
happens before the retry loop and deserialising after it, so only
connect, I/O and short-read failures can reach the loop at all — a
deterministic failure is now unretryable by construction instead of by
classification.
It is deliberately a new crate and not part of `hive-agent-sock`. The
`*-sock` crates are pure wire types by convention — `hive-agent-sock`
depends on serde and nothing else — and the two largest copies talk to
the host socket, whose types live in a different crate entirely. A
transport in either wire-type crate would drag tokio into it and point
the wrong way besides.
No wire-format change: same JSON line in, same line out.
The poller was a `tokio::spawn` inside the `hive-agent` serve loop. It
never needed anything from that loop except a socket path, so being
in-process bought nothing and cost two things: a harness restart took
forge notifications down with it, and the whole forge/HTTP dependency
tree was linked into the serve-loop binary.
It is now `hive-forge-notify`, a per-agent daemon with its own systemd
unit, a sibling of `hive-bash-daemon` and `hive-matrix-daemon`. Same
contract as those two: it reaches the harness only by upserting todos on
the in-agent socket, and nowhere else.
The module moves verbatim (`notify.rs`) — the formatters, the activation
gates, the dedupe map and all 33 tests are unchanged. Only the socket
call sites are rewritten, onto a small local `todo_client` rather than
the harness's. That mirrors what both sibling daemons already do, and
the etiquette differs on purpose: the harness's client carries a 60s
backoff schedule sized to ride out a hive-c0re restart, which its
callers need because they have no retry of their own. This poller's two
call sites both sit inside the 30s poll loop and both treat a failure as
"leave the thread unread, try next tick", so the poll interval already
is the retry; a second backoff would only stack sleeps and delay the
rest of the batch.
The unit is `Restart=on-failure`, not `always`. An agent with no forge
account is a supported configuration and the poller reports it by
logging why and exiting 0 — under `always` that clean exit would be a
restart loop on every forge-less agent.
`forgejo-api`, `url` and `time` drop out of `hive-agent`'s dependencies
with the module.
Also corrects docs that outlived the code they described: the persisted
`forge_cursor` field is long gone (forge's own read-state is the durable
record of what has been delivered), but `docs/persistence.md` and the
`harness_state` module docs still documented it as live.
Closes 2718.
The operator has been going through agent dirs by hand with ncdu,
deleting 20+GB target dirs. Agents had no way to know they were the
ones sitting on the space.
New `disk_watch` module in the harness: every 15 minutes it statvfs's
the filesystem backing the agent's state dir and, past 80%, raises a
keyed `disk` todo telling the agent to free space — with the operator's
rules inline: only delete things that are actually big, build output
first, and never delete something still needed, ask for more space
instead.
Over threshold it also walks the agent's own tree (`/agents/<label>`
plus `$HOME`) and names the directories worth looking at, so the todo
says where the bytes actually went rather than just that the disk is
full. The walk is bounded on every axis — entry budget, recursion cap,
report depth — pinned to the state dir's device so it can't wander into
`/nix` or the shared bind mounts, and it does not traverse symlinks. It
reports the deepest oversized directory on each branch, so the agent
gets pointed at `<workspace>/target` rather than at `/agents/<label>`.
Anti-nag is the whole design constraint. The todo is keyed, and the
summary is deliberately stable: the percentage is bucketed to 5 points
and no raw byte counts appear anywhere in it. An unchanged situation
re-upserts as `changed == false` and never fires the wake, so a disk
that has been steady at 89% for a week sits quietly in the loose-ends
list; crossing into a new bucket speaks up once. Dropping back under
the threshold clears the row.
Harness-local by construction, per the operator's call that this gets
no core wiring: hive-c0re cannot push a todo at all (the store and its
wake live inside the container), and running in-process means this
skips even the in-agent socket and calls `Todos::upsert` directly.
Worth recording, since it shaped the scope: btrfs does NOT fold qgroup
limits into statfs. Measured with quota counting enabled and a 20G
limit set on a real subvolume, statvfs returns byte-identical whole-FS
numbers for that subvolume, an ordinary agent dir, and the root. So
this watches host-FS pressure, which is valid before and after the
planned subvolume migration; per-agent quota awareness would need the
limit handed to the agent explicitly.
Node gains started_at/finished_at (chrono DateTime<Utc>, serialized
RFC 3339 on the wire per hive_sh4re::wire_time) plus error (String).
Graph::set_state self-stamps started_at on the first Running transition
and finished_at on the first terminal one, via an internal now_utc()
clock (keeps settle/complete signatures stable). Outcome::Failed(String)
carries the failure reason, set on the terminal transition.
hive-c0re complete_node builds Outcome::Failed(msg); its node_rt
side-table stays i64 for now (double-write) until #2637 reads the Node.
Toward #2637: the jobq graph becomes the source of truth for per-node
lifecycle so the queue can be sent to the client as-is.
