approvals.md (655 lines) is genuinely dense implementation reference
(webhook mechanics, DAG phase names, db column names, git-ref rollback
internals) with nothing written for an operator deciding whether to
click approve. Unlike web-ui/, this is a single top-level file with no
subdirectory to give a separate landing page to, so the fix here is
in-file: a "For operators" section right after the intro, covering
what actually shows up on the dashboard (or hivectl CLI) and what each
approval kind does when you click approve or deny, before the existing
implementation detail.
Every claim in the new section was checked against the rest of this
file plus dashboard.md and hivectl-cli.md rather than assumed - caught
one real error before pushing (hivectl approvals list doesn't exist,
the verb is `pending`) and one incomplete claim (denial reasons are
dashboard-only with an optional prompt; the CLI deny has no reason
argument).
github.md mixed operator content (enabling, provisioning, security)
with deep implementation detail (the gh wrapper/credential-helper
mechanics, the notification poller's internals) in file order, so an
operator reading top-to-bottom hits internals before finishing the
part they actually need.
Pure reorder, no rewrite: Enabling -> Provisioning -> Security (all
operator-facing) now come first: How the agent uses it and
Notifications (both pure impl detail) move to the end, with a one-line
marker between them. Every word of existing content is unchanged, only
section order moved - lowest-risk shape for a file like this with no
subdirectory to split into (see hyperhive#1898).
argus review on hyperhive#2986: the README listed Stats/Peers/Settings
alongside Permissions/Schedules as dashboard tabs. Checked the actual
frontend rather than trusting dashboard.md prose (which is internally
inconsistent on this - some section headers say "tab" for things that
turned out not to be):
- stats.html and settings.html are real separate bundles
(frontend/packages/dashboard/src/{stats,settings}.{html,js}) - same
shape as builds.html/core.html/logs.html, moved to the "own page"
list.
- Peers is neither a tab nor a page - swarm.js::renderPeerHives
confirms it is a card list rendered inside the SW4RM tab
(#peers-block/#peers-section), gated on state.peer_hives being
non-empty. Folded into the SW4RM bullet instead of listing it as its
own item anywhere.
The dashboard tab strip is genuinely just four: SW4RM, Y3R C4LL,
P3RM1SS10NS, SCH3DUL3S.
docs/web-ui/ had four pages, all written as deep implementation
reference (dashboard.md alone is 1466 lines of wire shapes and DOM
mechanics) - there was nothing here written for an operator actually
using the dashboard day to day.
Add docs/web-ui/README.md: genuinely task-oriented content (what the
dashboard is, where the tabs are, the things you'd actually do -
checking an agent, answering a pending question, approving a config
change, granting a permission, reading logs) with pointers into the
existing pages for implementation depth. Point docs/web-ui.md at it as
the operator starting point.
Deliberately not touching dashboard.md/agent.md/shape.md/css-vars.md -
reflowing 1466 lines of dense, working reference content in place
turned out to be a much bigger and more error-prone task than "lead
with the user fact" suggested at a glance (see hyperhive#1898). The
subdir's landing page carries the user-facing content; the existing
pages stay exactly as they are, as the depth layer underneath it.
Verified the new page actually renders as the subdir's index via the
website repo's new subdir-landing-page support (hyperhive/website
PR #46): a real build of the prose-docs derivation confirms
web-ui/index.html now exists, the root index links straight to it, and
every internal link resolves correctly.
Follow-up from #2955 (mara: 'make core able to give agent a todo').
First migration slice: Spawned was pure FYI-check-when-convenient
material, not something needing an immediate turn.
Coordinator::push_todo/push_todo_submitter do a best-effort live dial
of the target agent's own hive-agent-sock (hive_host_sock::agent_todo_
socket), sending the exact UpsertTodo request in-container producers
(matrix/bash/forge-notify) already send. Push, not queue: agent
offline (socket absent) or dial failure is a silent no-op, no retry,
no fallback delivery -- matches mara's 'not available if offline'
call exactly.
HelperEvent::Spawned removed entirely (enum variant + all 3 call
sites migrated: handle_spawn's two arms, finish_approval's Spawn
approval-kind arm) rather than kept alongside a translation layer --
per mara's correction on the first design attempt, migrating the
producer means deleting the old path, not bridging it.
Verified: cargo build/clippy/test -p hive-c0re -p hive-host-sock
-p hive-sh4re clean (318 tests), nix fmt clean.
mara, PR review: 'less impl details in user facing docs'. Dropped the
utoipa crate link, the ApiDoc source-file pointer, and the
#[utoipa::path(...)] annotation-syntax mention from the Dashboard
endpoints intro paragraph -- kept only what a reader needs (where to
find it, that it's a growing subset not yet the full list). Trimmed
the H0M3-page mention the same way.
Closes#2965.
hive-c0re auto-generates an OpenAPI 3 spec via utoipa
(hive-c0re/src/dashboard/mod.rs's ApiDoc), served raw at
/api/openapi.json and browsable as a Swagger UI at /api/docs, but
docs/ never mentioned either — genuinely zero hits grepping the whole
docs/ tree. Documented both in docs/web-ui/dashboard.md's Dashboard
endpoints + H0M3 page sections.
Also added the H0M3 hub's API tile mara suggested ("maybe also add
home page app that opens swagger ui") -- a plain static link to
/api/docs, no gating needed since the endpoint always exists
(unlike Forge/Matrix, which are conditionally enabled).
Per mara: 'i wanted you to put this in .../settings.html' — the toggle
belongs with the other operator-local browser preferences, not buried
in each agent's own overflow menu.
Extracted the get/set + localStorage key into @hive/shared/prefs.js so
settings.html (writer) and every per-agent app.js (reader, via
HiveTerminal.create's expandDetails option) agree on the exact same key
without two independently-typed copies that could drift. Removed the
now-unused overflow-menu toggle + its agent.css rules from the agent
page. Docs moved from docs/web-ui/agent.md's overflow-button section to
docs/web-ui/dashboard.md's S3TT1NGS section, next to the existing
browser-notifications preference.
Adds a browser-local (localStorage only, no backend field) toggle in
the per-agent overflow menu's new settings section: whether otherwise-
collapsed <details> rows in the live terminal (long tool-results,
Write/Edit diffs, ...) default open. Message-bearing rows that already
default open (send/ask/answer/recv) are unaffected either way.
The shared terminal factory (frontend/packages/shared/src/terminal/terminal.js)
gains an optional expandDetails option (boolean or zero-arg function),
read live on every details()/detailsDiff() call rather than captured
once, so flipping the toggle mid-session applies to the next rendered
row without a reload. Unused by the dashboard's own terminal pane, so
its default-closed behaviour is unchanged.
Closes#2961.
Resources were derived from the node's kind: `templates::node` called
`NodeKind::resource_deps()`, which fanned out to `needs_build_slot` /
`needs_lease` / `needs_meta_window`. That made the requirement a property
of the *kind*, so a kind that happened to run under an ancestor already
holding the resource could get away with declaring nothing.
Three did. `Start`, `Stop` and `PostSwap` appear in none of the three
predicates, and that was only safe because one construction site fans
them out from inside a lease-holding `Reconcile` — a fact about today's
DAG shape, not about the nodes.
Each of the 41 construction sites now says what it holds. `Start` /
`Stop` / `PostSwap` declare the agent lease; per the contract that is a
re-entrant borrow, which a new test pins rather than argues.
`running_transients` reads the node's declared deps instead of
re-deriving from the kind. That closes the blank-pill gap: the pill went
blank during container start, stop and the post-swap tail because the
declaration was missing, not because the filter was wrong.
The deleted predicates carried the only written record of three design
decisions; each moved to the `Resource` variant it constrains rather than
dying with its function.
The submit-time petgraph `toposort` this described is gone — a cycle
needs an edge pointing at a node declared later, and a handle only
exists for a node already declared. Say why the validation pass is
absent rather than leaving a description of one that isn't there.
`templates.rs`'s module doc was 35 lines and over the comment-block
lint's max; it now points here for the reasoning instead of restating
it, and drops the power-op paragraph that `submit.rs` already owns.
docs/web-ui/agent.md still described a "loose-ends" and "tasks" flyout
that were superseded by the todos flyout when loose-ends-v2 landed —
GET /api/loose-ends and GET /api/bash-tasks are both gone server-side.
Replaced with an accurate description of the todos flyout (including
the mark-done bulk action from #2919), and noted that the ask->operator
inline-answer binding this doc also describes is currently
non-functional (its data source was the same removed endpoint) —
tracked separately as #2922, not fixed here.
buildLooseEndsList in app.js rendered the old loose-ends flyout and had
zero call sites left; removed it. buildAnswerForm stays — reconcileAskBinds
still calls it, even though that path is currently dead per #2922.
Fixes#2920
Phase 4 (repoint every container onto `meta#<n>`) and phase 5 (rename
the `root` container to `h-root`) were marker-guarded one-shots for
layouts no live hive still has: containers are rendered onto `meta#<n>`
at creation, and the `h-` prefix has been the naming for far longer than
any deployment predates. A one-shot nobody can still trigger is dead
weight, so both are gone along with `repoint_container`,
`rename_manager_container`, `CONTAINER_TIMEOUT` and the two marker paths.
Phase 6 was not obsolete, only misplaced. Ruth's tool groups are now
seeded by `ensure_root_agent` on the one path that creates her, rather
than re-asserted on every hive-c0re boot. The skip-if-already-set guard
survives the move: a destroy+recreate under the same name must not reset
an operator's chosen group set back to MANAGER_DEFAULT.
That also settles a latent bug. Phase 4's marker check was a `return`,
not a skip, so on any hive carrying the marker phases 5 and 6 never ran
at all — the tool-group backfill, whose whole job was preventing a silent
privilege downgrade, has not executed here in a long time. Moving it to
create-time removes the question rather than answering it.
What stays is convergence: three unguarded, idempotent phases that re-run
each boot and no-op once their state is right. The module doc now names
the three categories so the next person can tell which kind they're
adding.
argus flagged on !2910 that the dashboard doc still described the pill's
old shape. It did, in two ways that now teach the wrong thing:
- it listed a fixed vocabulary, where the label is now the running
node's own wire tag — the `NodeKind::as_str` strings `NodeView.kind`
already carries. A client that switches on specific values is now
wrong, and `restarting` in particular no longer exists at all.
- it described the transient as operator-initiated ("set the moment the
operator clicks"), which was the distinction between it and the
rebuild-queue fallback. That is no longer true: the transient is
derived from the running node, so worker-driven work the operator
never clicked lights the same pill.
The queue-`kind` half of that section is untouched — that path did not
change and its vocabulary is still fixed.
Also notes on the wire-event list that `transient_kind` is a display
string to render, not an enum to branch on, since that is the property a
client would otherwise have to infer from a now-open set.
Docs only, no code change.
The connectivity-overrides section still documented
`hyperhive.forge.url = "http://localhost:3000"` as the default and said
the empty string was a valid value. Both stopped being true when the
option became `nullOr str` with `default = null`.
Also says what `null` means and why, since "the default is null" alone
reads like an oversight rather than the point: a loopback default is
only correct when the forge shares the agent's netns, and the units that
would consume the URL are simply not generated without one.
Refs #2860
docs/tools/hivectl-cli.md is generated by `hivectl markdown-docs` and
diffed against a fresh run by the hivectl-docs flake check, so adding
the push verb without regenerating it would have failed CI. The
regeneration also retires two copies of a sentence describing the
cross-hive leg as an ssh pipe that "isn't wired up yet" -- ssh was
dropped when the WireGuard mesh became the authentication, and the leg
is wired up now.
snapshot-store.md documented only the receiving host, so a reader had
no way to learn how a pushing hive is told where the store is. It now
covers services.hyperhive.swarm.snapshotStore, including why address
has no default and port does: an address is a deployment fact that
cannot be guessed, a port is a convention both ends read from the same
docs. It also states the namespace rule the two options illustrate --
swarm.* describes the swarm as seen from here, a bare
services.hyperhive.<service> describes a role this host performs.
swarm.md never mentioned the store even though the option lives in its
namespace, so a reader configuring swarm.peers had no signal it exists.
Reverting the namespace move from the previous commit — mara's reason
is better than mine was.
I grouped it with swarm.peers and swarm.wireguard because the module
serves the swarm tier. But those two describe THE SWARM: who is in it,
how it is meshed. snapshotStore describes THIS HOST'S ROLE. On a
standalone store box the operator enables one service, and nesting it
under `swarm` implies they are configuring a swarm when they are not.
The swarm- prefix on the file and units stands: the name says which
tier the component serves, the option path says what you are turning
on. Those are different questions and they are allowed different
answers.
mara, in preparation for the swarm tier: the store is a swarm-level
role, not a hive one, so hive- was misleading about which tier it
belongs to. Module, units, syslog identifier, log lines and docs all
move to swarm-snapshot-store.
Also moved the option under services.hyperhive.swarm.snapshotStore, to
sit with swarm.peers and swarm.wireguard rather than dangling off the
top level. That is a judgement call beyond the literal rename — flagged
on the PR, and cheap precisely now: the option has never shipped, so
there is no deployment to migrate, whereas doing it after a release
would be a breaking change for no new benefit.
mara: the option was the wrong shape for the concern. "this host runs
nothing else" is a deployment expectation, not something a module
should assert about its own host — and asserting it made co-location
look like a config toggle rather than what it is.
Replaced with docs/snapshot-store.md, which the module had no docs
page at all before: enabling it, why the mesh is the authentication
(cryptokey routing already binds source address to pubkey, so certs
would authenticate the same fact twice and add an expiry), why the
destination is keyed per agent (a per-hive prefix splits an agent's
chain the first time it migrates), what the sender may and may not
choose, why the firewall rule is interface-scoped, what a snapshot
does and does not contain, and what the pull side still needs.
The dedicated-host expectation is stated there as an operational
assumption with its own failure mode — true on day one, quietly false
the day someone notices the box has spare disk — rather than as an
assertion someone flips to false to make the build proceed.
Linked from CLAUDE.md's reading paths.
hive-forge-notify grows a second binary, hive-github-notify. The two
share the notification half of the job — tolerant parse, classification,
formatting, dedupe, todo delivery — and nothing else: each binary owns
its host's protocol outright.
Two binaries rather than one multi-source daemon, and rather than a
cargo feature. A feature would unify across the workspace and cost every
crate its build cache. Two binaries keep the decision in nix: forge.nix
installs the forge unit, github.nix installs the github one under
hyperhive.github.enable, so a hive built without that module has no
github poller in its closure at all — GitHub access is separable (a
tier, a policy boundary), not merely switched off. Both binaries ship
from the existing derivation, so packages.nix is untouched.
The split is real at the code level too, not just at the unit level.
source.rs is a trait; the impls live in the binaries that use them, so
neither binary links the other's protocol code and the library names no
host at all. The forge-only assigned-issue rollup moves into the forge
binary for the same reason: it asks the forge what is assigned to this
agent, which is not a notification-protocol concern.
At runtime the github unit needs a PAT at <state>/github-token, the same
dashboard-provisioned token the gh wrapper and the git credential helper
already use. No PAT: it logs why and exits 0, which is why the unit is
Restart=on-failure and not always.
Forgejo's notifications API is modelled on GitHub's, so one tolerant
parse serves both — the differences (string thread ids, PullRequest vs
Pull) are absorbed by lenient deserializers rather than a second parse
path. Thread ids normalise to String at the parse boundary; they are
only ever opaque keys. Todo keys gain a per-source prefix so the two
hosts cannot collide, and the forge's is deliberately empty to keep
existing forge todo keys stable across the deploy that lands this.
The github loop honours the server's X-Poll-Interval, re-arming only
when the server asks for a slower cadence than ours; the hint is read
before the status check, because it arrives on error and empty pages too
and that is exactly when it matters. Reading the notification stream
needs the notifications scope on the PAT, which a token minted for push
access typically lacks; the failure mode is silence, so docs/github.md
says so explicitly.
The queue carried a per-DAG `HookKind` that fired an inline side effect
from outside the graph when a container rolled up terminal. mara asked
three times why this could not be an ordinary node; the answer in the
code was a doc-comment claiming a node could not work, and it was wrong.
`DepWhen::AfterAny` already existed with two live users, and a weak edge
is satisfied by a `Cancelled` dep, so a tail node runs on success,
failure and cancel alike. What was genuinely missing was smaller than a
hook: a node had no way to learn how the work it followed ended.
So: `Claim` now carries `deps: Vec<DepOutcome>`, snapshotted at claim
time from the graph the scheduler already holds (no `hive-jobq` change).
`Claim::deps_state()` / `deps_error()` roll that up, and two new kinds
consume it — `ResolveApproval { approval_id }` and `EmitRebuilt { agent }`.
Templates append one as a group-root with `AfterAny` edges onto the DAG's
other group roots; a root's state is its subtree's roll-up, so that
covers every node without fanning out to each of them.
Deleted: `HookKind`, `DagSpec.hook`, `NodeKind::Dag.hook`, `DagMeta.hook`,
`TerminalDag`, `terminal_dag()`, `terminal_summary()`, `dag_agents()`,
`dag_rollup()`, `fire_terminal_hook()`, `run_terminal_hook()`,
`emit_rebuilt()`. `complete_node` returns `()`.
Load-bearing details:
- `JobQueue::cancel` spares tail nodes instead of cancelling the whole
subtree, and returns `bool`. Without this a cancelled approval DAG
would dangle its approval forever — the hazard `tests.rs` already
named. The spared tail's deps are `Cancelled`, which satisfies its weak
edge, so the scheduler claims it and it resolves the row as cancelled.
`hive-jobq` anticipated exactly this: `cancel_node`'s doc already says
to settle afterwards so "a weak-edge terminal node observing the
cancellation" can advance.
- The existing `complete(container)` call after cancelling is kept and is
deliberately a no-op when a tail was spared (a non-terminal child parks
the container back in `Finishing`), so power ops still settle
synchronously with no branch.
- `DeployTail` is NOT `is_tail()`: it does real compensating work, and a
cancelled DAG has nothing to compensate.
- `exec::failure_reason` falls back to `first_error(dag_id)` because a
group root that rolled up `Failed` from a child carries no error of its
own — without it every tail-reported failure would lose its reason.
- `EmitRebuilt` is per agent, so a multi-agent DAG reports each agent's
own outcome rather than painting all of them with the DAG roll-up.
- `ResolveApproval` is agentless: the approval row already names its
agent, and that is also what lets one tail close a multi-agent DAG.
Transients-derived-from-running-nodes and the frontend's node-kind
strings stay out of this change; they touch iris's slice and review
better next to their own diff.
Agents run whatever `claude-code` the meta flake's `nixpkgs` resolves
to, and that is normally a release channel. This one package moves fast
enough that stable trails unstable by weeks — 26.05 is on 2.1.187 while
unstable carries 2.1.220 — and an agent cannot fix it for itself: it
only ever sees the single nixpkgs hive-c0re injects, so an `agent.nix`
has no other tree to reach for.
New host option `services.hyperhive.c0re.claudeCodePackage` takes the
package directly and rides the existing `hyperhiveDocs` threading path —
serveConfigJson -> HiveEnv -> render_flake — to reach each agent as
`hyperhive.claudeCodePath`. Null (the default) is today's behaviour.
What travels is the store *path*, as a plain string literal, not a flake
input: containers share the host's `/nix/store`, so the build is already
reachable inside them with its whole closure and has nothing to travel.
An input would be worse than useless — a `path:/nix/store/<pkg>` input
is re-copied as a reference-less `-source`, which strips exactly the
closure the binary needs.
The catch is that a path written into a generated flake is text, so
nothing in the container's closure keeps the binary alive. The host does
that instead, and gets it for free: the package is interpolated into
`/etc/hyperhive/serve.json`, `builtins.toJSON` preserves string context,
so the /etc entry references it and the system closure gc-roots it for
as long as that generation is the one the agents were rendered from. An
assertion pins that property, because losing the context is invisible at
eval and at deploy — it would surface only as every agent failing to
spawn `claude` whenever the next gc ran.
Container side wraps the path in a symlink farm rather than putting it
on PATH directly: `systemd.services.<name>.path` and
`environment.systemPackages` both coerce a store-path *string* through
`lib.toDerivation`, i.e. `builtins.storePath`, which pure evaluation
rejects. Interpolating the path into a builder is just text and
evaluates anywhere. `claude-code` drops out of systemPackages when a
pin is set, so there is exactly one claude in the container.
Refs #2693
The PR added CPUWeight=/IOWeight= to the drop-in but left the prose
docs describing a two-setting file. Covers the cap-vs-share
distinction, the hive-wide-only scope (no resource-limits.json
override), and the iocost/BFQ caveat that makes IOWeight= inert on
most hosts.
The agents root is 0700 and owned by the daemon's user, so hivectl's
client-side existence guard hit EACCES on traversal for anyone not root.
It reported that as "this command needs root; re-run with sudo", which
turned three verbs' pre-flight check into a permission error about the
wrong thing: `choom`, `subvol upgrade` and `subvol snapshot create` all
failed at the guard rather than at whatever they actually needed.
The daemon runs as the owning user and already answers this question for
its own provisioning paths, so expose it on the host socket as
`AgentExists` and have hivectl ask. Operators reach that socket through
the `hive-admin` group, so the guard now works without sudo.
`choom` still needs root for `machinectl shell` — we ship no polkit rule
granting those actions — so it now checks the effective uid and says so
directly instead of failing later inside systemd's authorisation.
The poller was a `tokio::spawn` inside the `hive-agent` serve loop. It
never needed anything from that loop except a socket path, so being
in-process bought nothing and cost two things: a harness restart took
forge notifications down with it, and the whole forge/HTTP dependency
tree was linked into the serve-loop binary.
It is now `hive-forge-notify`, a per-agent daemon with its own systemd
unit, a sibling of `hive-bash-daemon` and `hive-matrix-daemon`. Same
contract as those two: it reaches the harness only by upserting todos on
the in-agent socket, and nowhere else.
The module moves verbatim (`notify.rs`) — the formatters, the activation
gates, the dedupe map and all 33 tests are unchanged. Only the socket
call sites are rewritten, onto a small local `todo_client` rather than
the harness's. That mirrors what both sibling daemons already do, and
the etiquette differs on purpose: the harness's client carries a 60s
backoff schedule sized to ride out a hive-c0re restart, which its
callers need because they have no retry of their own. This poller's two
call sites both sit inside the 30s poll loop and both treat a failure as
"leave the thread unread, try next tick", so the poll interval already
is the retry; a second backoff would only stack sleeps and delay the
rest of the batch.
The unit is `Restart=on-failure`, not `always`. An agent with no forge
account is a supported configuration and the poller reports it by
logging why and exiting 0 — under `always` that clean exit would be a
restart loop on every forge-less agent.
`forgejo-api`, `url` and `time` drop out of `hive-agent`'s dependencies
with the module.
Also corrects docs that outlived the code they described: the persisted
`forge_cursor` field is long gone (forge's own read-state is the durable
record of what has been delivered), but `docs/persistence.md` and the
`harness_state` module docs still documented it as live.
Agents had the official marketplace configured out of the box but an
empty plugin list, so nothing was installed from it unless an agent's
own config asked. skill-creator is the one plugin that pays for itself
generically: it teaches an agent to write, refine, and evaluate its own
skills, which is exactly the capability an agent can't bootstrap by
being told about it once in a prompt.
Defaulting the option keeps this consistent with claudeMarketplaces,
which already ships the official marketplace the same way. Documented
the list-option semantics next to both: a per-agent definition replaces
the default rather than extending it, so an agent that sets its own
plugin list has to name skill-creator alongside its entries.
Both groups only ever act on a single managed agent's state dir, so
they belong in the `agents` namespace rather than as top-level verbs
next to `forge` / `matrix` / `wg`.
Renames `quota limit` -> `quota set`: the enclosing group already
carries the noun, so the bare verb matches the flat `set-parent` /
`set-limits` spelling without stuttering, and it removes the
`set-limits` (cpu/mem) vs `quota limit` (disk) ambiguity. Adds a
cross-pointer from `set-limits` to `agents quota`.
Handlers stay in their own modules; `run_agents` gains the reparenting
glue. Regenerates docs/tools/hivectl-cli.md.
Refs #2724
The gateway container's /etc/resolv.conf is a one-shot copy: nixos-container
cps it in from the host in its start script, and nspawn's --resolv-conf=auto
copies (not binds) for a writable host-netns container. systemd-nspawn(1)
states the consequence outright — "no further propagation of configuration is
generally done after the one-time early initialization (this is because the
file is usually updated through copying and renaming)".
dnsmasq has no explicit upstream and follows that file, so a host network
change strands it on a resolver that no longer answers and every non-hive
lookup from every agent hangs. Agents' own resolvers point at the static
bridge IP and never go stale, which is why the symptom presents as "the
gateway needs a kick".
Add a host-side hive-gateway-resolv path unit watching /etc/resolv.conf.
On change it machinectl copy-to's the file into the container and reloads
dnsmasq — ExecReload is kill -HUP, so upstreams are re-read and the cache
flushed without dropping anything; nginx never notices.
- watched from the HOST: a rename on the host doesn't cross the nspawn mount
namespace, so an in-container path unit can't see it (same reason c0re
reloads nginx from the host side)
- copy, not a file bind-mount: openresolv renames over the file, so a bind
would pin the first inode forever — strictly worse than today
- machinectl copy-to writes through the container's own mount namespace, so
this holds regardless of how the container assembles /etc
- armed Before=network-pre.target so the boot's first DHCP write is caught,
and re-run on gateway start for changes made while it was down
- a host file with no nameserver line is skipped, not pushed, so a
mid-rewrite snapshot can't blank hive DNS
- deliberately no fallback server=: dnsmasq queries all known upstreams in
parallel, so a hardcoded public resolver would take a share of normal
traffic rather than only covering the gap
Closes 2718.
The operator has been going through agent dirs by hand with ncdu,
deleting 20+GB target dirs. Agents had no way to know they were the
ones sitting on the space.
New `disk_watch` module in the harness: every 15 minutes it statvfs's
the filesystem backing the agent's state dir and, past 80%, raises a
keyed `disk` todo telling the agent to free space — with the operator's
rules inline: only delete things that are actually big, build output
first, and never delete something still needed, ask for more space
instead.
Over threshold it also walks the agent's own tree (`/agents/<label>`
plus `$HOME`) and names the directories worth looking at, so the todo
says where the bytes actually went rather than just that the disk is
full. The walk is bounded on every axis — entry budget, recursion cap,
report depth — pinned to the state dir's device so it can't wander into
`/nix` or the shared bind mounts, and it does not traverse symlinks. It
reports the deepest oversized directory on each branch, so the agent
gets pointed at `<workspace>/target` rather than at `/agents/<label>`.
Anti-nag is the whole design constraint. The todo is keyed, and the
summary is deliberately stable: the percentage is bucketed to 5 points
and no raw byte counts appear anywhere in it. An unchanged situation
re-upserts as `changed == false` and never fires the wake, so a disk
that has been steady at 89% for a week sits quietly in the loose-ends
list; crossing into a new bucket speaks up once. Dropping back under
the threshold clears the row.
Harness-local by construction, per the operator's call that this gets
no core wiring: hive-c0re cannot push a todo at all (the store and its
wake live inside the container), and running in-process means this
skips even the in-agent socket and calls `Todos::upsert` directly.
Worth recording, since it shaped the scope: btrfs does NOT fold qgroup
limits into statfs. Measured with quota counting enabled and a 20G
limit set on a real subvolume, statvfs returns byte-identical whole-FS
numbers for that subvolume, an ordinary agent dir, and the root. So
this watches host-FS pressure, which is valid before and after the
planned subvolume migration; per-agent quota awareness would need the
limit handed to the agent explicitly.
The `step` label was taken off the wire in #2661, when each deploy phase
became a first-class DAG node. Since then it has been written but never
read: `NodeRuntime` derives only `Debug, Default, Clone` — no serde — so
the field could not reach any client, and the only reads of it were the
dedup checks inside its own setters. This deletes the machinery.
Removed:
- `NodeRuntime.step`, `set_step`, `set_step_running`, and the
`rt.step = None` clear in `complete_node`. `NodeRuntime` keeps its
remaining `build_log_id` field (deliberately still a struct — collapsing
it to a bare `Option<i64>` would churn every call site for no gain).
- `Ctx::step` and its ~15 call sites in `job_queue/exec.rs`. `Ctx` itself
stays: it is the build-log sink, which `run_prebuild` and `run_swap`
still use.
- `Coordinator::set_queue_step` and its 11 callers in `actions.rs`.
- `JobQueue::running_node_of`, reachable only from `set_queue_step`.
- `swap_update`'s `on_step` parameter and its one body call.
- The `set_step_only_on_running_and_signals_change` test.
Dropping the calls orphaned parameters, which are removed with their call
sites: `ctx` on ten executors that used it only as a step sink, and
`queue_entry_id` on `run_deploy_merge_verify` / `run_deploy_apply` /
`run_finalize_deploy` plus both `coord` and `queue_entry_id` on
`prepare_applied_target`. `run_deploy_tail` KEEPS its `queue_entry_id` —
that one has a genuine surviving use (the build-log link in the failure
comment posted to the PR).
One behavioural change, called out so it is not mistaken for a dropped
dashboard refresh: `Ctx::step` and `set_queue_step` each emitted a
`rebuild_queue_changed` snapshot when the label changed, and those
emissions go away with them. This is safe — the snapshot payload has no
step field, so those pushes carried nothing a client could observe. Real
state transitions still emit from the scheduler's claim and completion
paths, from `submit`, and from the three `actions.rs` sites. Net effect is
strictly fewer redundant SSE pushes.
Docs: `docs/coordinator.md` still listed `step` as a `NodeView` wire field
and `docs/web-ui/dashboard.md` documented a cyan `↳ <step>` sub-line under
each queue row. Neither has existed since #2661 — both corrected here, plus
the `job_queue/model.rs` module doc.
Not touched: `frontend/packages/dashboard/src/system-sections.css` has a
dead `.rqe-step` rule with no JS referencing it. Left for the frontend
owner rather than deleted here.
Closes: #2664
The hive applies one `agentCpuQuota` / `agentMemoryMax` to every
container. That's the right default and the wrong ceiling: a build-heavy
agent needs headroom the other twelve don't, and raising the hive-wide
value to suit it hands that headroom to everyone.
Adds a per-agent override, persisted host-side and resolved per-field
against the hive defaults.
Follows the existing `meta/*.json` pattern (`capabilities.json`,
`tool-groups.json`): a host-side map read by `hive-c0re`, staged and
committed in the meta repo so every change lands in the audit trail.
```json
{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }
```
Fallback is **per field**, not per agent: an entry with only
`memory_max` leaves that agent on the hive-wide CPU quota. Absent file,
absent agent and absent field all resolve to the hive default, so the
feature is inert until someone opts an agent in.
Unlike the other meta files this one is **not** injected into the
container — a limit is something done *to* an agent, not something it
reads about itself.
```
hivectl agents set-limits sock --cpu-quota 400% --memory-max 8G
hivectl agents set-limits sock --reset
```
Values are validated before they're persisted: they go into a systemd
drop-in verbatim, and a typo there makes the unit fail to *start* —
turning a fat-fingered quota into a container that won't come back.
The command is declarative: each call replaces the agent's whole entry.
That makes a forgotten flag a silent revert, so a bare `set-limits
<name>` is rejected at the clap layer and clearing needs an explicit
`--reset`.
`ContainerView` gains `cpu_quota` / `memory_max`, both always populated:
there's no "unset" state to render, only "same as everyone else". They
reflect what the drop-in *says* — what the next start will enforce — not
a live cgroup reading.
The write goes through `meta::commit_resource_limits` rather than the
bare setter, so it's staged and committed under `META_LOCK`. Writing
without committing would leave the meta working tree dirty for the next
`prepare_deploy` to trip over.
Docs: `persistence.md` (the new meta file, and why it isn't injected),
`tools/hivectl.md` (the prose guide), `tools/hivectl-cli.md`
(regenerated clap dump).
Closes: internal/requests issue 25
A paused agent keeps its container, its claude session and its
dashboard/todo servers up, but stops driving turns. Messages queue
unacked and are drained on resume.
The whole protocol is a single marker file, `<harness>/paused`. That
directory is already a bind-mount shared between host and container, so
both sides just stat the same path: the harness reads it to decide
whether to drive a turn, hive-c0re reads it to render the badge and
writes/removes it for `hivectl pause|resume`. No new wire protocol, no
container round-trip, and it is sticky across restarts by construction.
Not calling `recv_next` while paused *is* the queueing semantic, so
there is no fencing to get wrong: reminders buffer in their unbounded
channel, the todo `Notify` permit coalesces, and a `request_next_turn`
that raced the pause survives because the gate sits above
`self_continue.take()`.
Graceful stop is handled host-side rather than in the harness: a paused
agent provably has no turn in flight, so `run_signal` skips the fence
entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for
a checkpoint turn that will never run.
`paused` is reported on `ContainerView` / `AgentStatusRow` for the
dashboard, orthogonal to `running` and reported for stopped containers
too.
Closes: hyperhive/hyperhive issue 2271
The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.
Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.
The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.
`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.
The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.
Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
