Destroy was a straight-line async fn with no queue node behind it, so
nothing in the graph could answer "is this container going down on
purpose?". That gap is why an imperative crash-watch suppression guard
existed: an RAII handle held for the operation's duration, a second way
to say what every other lifecycle op already says through its node.
Reuse the existing Stop node rather than teaching a new node to stop
things:
Stop -> DestroyContainer -> (PurgeState) -> DestroyBookkeeping
Stop already declares takes_container_down honestly, so the suppression
is now derived from the graph like every other op's. It also turns the
precondition into an edge: DestroyContainer runs only under a completed
Stop, so it operates on an already-stopped container and carries
takes_container_down = false permanently. A container still alive at
that point is a real bug and stays loud instead of being absorbed by a
flag -- which matters because a wrong true silently swallows a crash
while a wrong false only costs a spurious event.
Removes suppress_crash_watch, CrashWatchSuppression, crash_suppressed,
crash_watch_suppressed and NO_NODE_LABEL. The migration call sites went
with the obsolete startup migrations, so destroy was the last caller and
intent now has exactly one home.
destroy() becomes a submit-and-return, matching every sibling endpoint
(rebuild, kill, restart, start, pause, resume) -- it was the only
lifecycle op that awaited its work. The container rescan moves into the
bookkeeping tail, so ContainerRemoved now arrives after the 200 rather
than before it.
Also drops an orphaned doc-comment in coordinator.rs: two stacked blocks
where only the second described crash_suppressed, the first documenting
a field that no longer exists. Removing the field would have re-pointed
it at recent_transient.
TREE IS RED ON PURPOSE — there is no compiling intermediate between
deleting submit and converting every caller. Checkpoint commit so the
work is durable; do not "fix" it by restoring submit.
Done:
- JobQueue::submit -> JobQueue::insert (no source/reason/container;
returns the ids insert_job names).
- submit.rs deleted. Its 6 pure chain builders + 3 async *_many
gatherers were NOT wrapper code and are rescued into
job_queue/power.rs (templates.rs documents power ops as living
outside it, because their shape needs a live is_running read).
- Converted: meta_inputs 1, topology 2, permissions 3, auto_update 2,
actions 3, lifecycle_handlers 3.
- Dropped source/reason at every converted site: nothing ever read
NodeKind::Dag's fields (only `{ .. }` matches exist), so they are
write-only. Dead reason-only locals deleted; the boot sweep's summary
became a tracing::info! rather than being lost.
Remaining: dashboard/lifecycle_ops 7, server.rs 7, and the test suite —
tests.rs has its own submit() helper whose u64 return is used as the
handle to navigate the inserted DAG, so those need a different way to
find nodes, not a mechanical port.
It reads as a swap thing but is post-*rebuild* bookkeeping: rev marker, forge and
matrix sync, manager kick, container rescan, meta-inputs snapshot. Wire label
follows (`post_swap` -> `rebuild_bookkeeping`); the graph view renders whatever
label it is sent, so nothing keys on the old string.
Also corrects a doc the previous commit falsified: this node's comment still said
it declares the agent lease as a re-entrant borrow, after that declaration moved
to the brace. Clippy and the whole suite pass over a stale doc comment, so it
took reading the file to find.
Verified by grepping the new name for places it has no business being, which
caught the sed rewriting a *historical* test name in a `// Replaces ...` comment
- reverted, since prose about the past must keep its old spelling.
`nix_logged` wrote its `build_logs` row with `node_id = None`, so a deploy's
eval/relock log was reachable from the agent+kind+time listing but not from
the queue node that ran it.
The comment justifying the `None` said both callers are "reached from outside
the queue as well as from inside it". That is no longer true: `verify_commit`
and `prepare_deploy` have exactly one caller each, and both sit under the
`MergeVerify` / `DeployApply` arms of `exec.rs`'s node dispatch, where the
`NodeId` is already in scope.
Threads `Option<u64>` from the dispatch down, mirroring `prebuild_toplevel`'s
existing `Some(id.get())` at exec.rs:241. Kept as `Option` rather than a bare
`u64` because `meta::prepare_deploy` / `meta::verify_commit` are public API and
a future non-queue caller has no node to name; the comment now says that
instead of the stale claim.
No behaviour change beyond the log row gaining its node link.
DagSpec described the graph the templates were about to build, one layer
below the templates themselves. Per #2972 the templates should be that
unit, so the spec type is gone and every declarer writes onto the job
builder directly.
- delete DagSpec<F> and its hand-written Debug impl
- submit(source, reason, declare: impl FnOnce(&Job)) replaces the
pre-built-spec signature; submit_and_emit follows
- all six templates take &Job; the Source is now the caller's to pass,
which spawn and approval_deploy previously hardcoded while the other
four did not
- power_dag dissolves into stop_nodes/start_nodes/restart_nodes, which
borrow their targets instead of owning them
315 tests pass unchanged.
Follow-up from #2955 (mara: 'make core able to give agent a todo').
First migration slice: Spawned was pure FYI-check-when-convenient
material, not something needing an immediate turn.
Coordinator::push_todo/push_todo_submitter do a best-effort live dial
of the target agent's own hive-agent-sock (hive_host_sock::agent_todo_
socket), sending the exact UpsertTodo request in-container producers
(matrix/bash/forge-notify) already send. Push, not queue: agent
offline (socket absent) or dial failure is a silent no-op, no retry,
no fallback delivery -- matches mara's 'not available if offline'
call exactly.
HelperEvent::Spawned removed entirely (enum variant + all 3 call
sites migrated: handle_spawn's two arms, finish_approval's Spawn
approval-kind arm) rather than kept alongside a translation layer --
per mara's correction on the first design attempt, migrating the
producer means deleting the old path, not bridging it.
Verified: cargo build/clippy/test -p hive-c0re -p hive-host-sock
-p hive-sh4re clean (318 tests), nix fmt clean.
mara on !2910: "also remove imperative path for the things that are not
nodes yet, file follow up issue to fix that".
`TransientGuard`, the stored map and both manual set/clear are gone.
`transient_snapshot()` is derived and nothing else — destroy and
migration show no pill, because there is no node to derive one from. The
pill returns for free when they become nodes.
What those three guards were actually doing, though, was suppressing the
crash watcher, not drawing a pill. `migrate.rs` said so in its own
comment: without it, `crash_watch` fires `ContainerCrash` for every
migrated agent and the manager tries to recover containers that were
stopped on purpose. Destroy is the same — the container disappears
deliberately and nothing in the graph says so.
Deleting them outright would therefore have traded a dashboard pill for
false crash alerts on every destroy and every migration. So the
suppression survives as its own thing, `suppress_crash_watch`, with a
name that says what it is. It is still RAII, and still held for the
operation rather than stamped once, because the crash watcher's grace
window is finite and a destroy is not — a single tombstone would expire
mid-operation. The drop stamps the tombstone, covering the poll that
lands just after.
That leaves RAII in the codebase for exactly one purpose instead of two.
Untangling the pill from the suppression is what made the transient layer
deletable at all.
Follow-up issue for making destroy + migration real queue nodes to
follow; at that point this guard goes too.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (322 + 41 passed) and `nix fmt`.
The dashboard pill was declared once per DAG at submit time, so a rebuild
reported `rebuilding` for its entire life — through the prebuild, the
stop, the swap, the tail and the reconcile. It named the intent of the
request, not what was happening.
It is now read off the nodes actually running. A node lights a pill when
it is `Running` and declares the agent's own resource. Declaring is the
test, not targeting: `Prebuild` and `MetaSync` name an agent but are
lease-exempt on purpose (the container keeps serving), so they must not
light one. It is also not the lease *owner* — `resource_state()` answers
"who holds the slot", which is a different question from "what is
running", and a descendant that borrows an ancestor's grant never
appears in that map.
`TransientKind` is gone entirely rather than being re-derived. The label
is the node's own wire tag (`NodeKind::as_str`) — the same vocabulary
`NodeView.kind` already ships, so a pill and a DAG node name an operation
identically and there is no second taxonomy to keep in step. Work with no
node behind it (destroy, migration) supplies its own literal.
`DagSpec::transient`, `Claim::transient`, `DagMeta::transient` and
`NodeKind::Dag`'s `transient` field all go with it.
## the safety half, which is deliberately not the display half
`crash_watch::is_deliberate_stop` used to match a `TransientKind` to
decide whether a vanished container was intentional or a crash. That made
a pill's display vocabulary decide an alerting question, so renaming or
adding a label would silently move the alerting boundary.
`TransientState` now carries two independent fields: `label` (rendered,
nothing branches on it) and `deliberate_stop` (read only by the crash
watcher). The producer sets the second, because the producer is the only
thing that knows — it is not recoverable from the first.
For queue work that value is `NodeKind::takes_container_down()`, and it
is emphatically not "holds a lease": `Create` and `Start` hold the
agent's lease exactly like `Stop` does, and a container dying *while
starting* is a real crash that must keep reporting as one. The default is
`false` on purpose — a wrong `false` costs a spurious crash event, a
wrong `true` swallows a real crash silently.
## known cost, accepted on the issue
A restart no longer reads `restarting`. No `NodeKind` is unique to a
restart — `restart_chain` reuses `Signal` / `StopForUpdate` / `Drain` /
`Reconcile` — because "restart" is a property of the DAG's shape, not of
any node. A restart now reads `signal` / `stop_for_update`, then the
agent returns.
`Start` / `Stop` / `PostSwap` run inside a lease-holding ancestor and
re-declare nothing, so they light no pill and the agent reads idle for
those windows. Closing that is the resources-where-constructed work
(#2818), not this change.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (321 + 40 passed) and `nix fmt`.
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:
- `Skipped` — the node's own edges ruled it out. Expected; the failure
branch of a run that succeeded is `Skipped`. A parent's roll-up
**ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
not-success for the roll-up, as before.
Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.
`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.
With that, the templates stop computing outcomes and let the graph pick:
- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
of the two the graph let run.
Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.
Deletes, all of them #2770's host-side debt:
- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
and the dep-snapshotting loop in `claim_ready`. Executors read their
own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
is derived from the edges: `cancel` keeps a node iff one of its edges
accepts `Cancelled`. An approval tail names it and survives to resolve
the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
claim that this couldn't dissolve was only true while `AFTER_ANY`
accepted cancellation.
`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.
Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
The queue carried a per-DAG `HookKind` that fired an inline side effect
from outside the graph when a container rolled up terminal. mara asked
three times why this could not be an ordinary node; the answer in the
code was a doc-comment claiming a node could not work, and it was wrong.
`DepWhen::AfterAny` already existed with two live users, and a weak edge
is satisfied by a `Cancelled` dep, so a tail node runs on success,
failure and cancel alike. What was genuinely missing was smaller than a
hook: a node had no way to learn how the work it followed ended.
So: `Claim` now carries `deps: Vec<DepOutcome>`, snapshotted at claim
time from the graph the scheduler already holds (no `hive-jobq` change).
`Claim::deps_state()` / `deps_error()` roll that up, and two new kinds
consume it — `ResolveApproval { approval_id }` and `EmitRebuilt { agent }`.
Templates append one as a group-root with `AfterAny` edges onto the DAG's
other group roots; a root's state is its subtree's roll-up, so that
covers every node without fanning out to each of them.
Deleted: `HookKind`, `DagSpec.hook`, `NodeKind::Dag.hook`, `DagMeta.hook`,
`TerminalDag`, `terminal_dag()`, `terminal_summary()`, `dag_agents()`,
`dag_rollup()`, `fire_terminal_hook()`, `run_terminal_hook()`,
`emit_rebuilt()`. `complete_node` returns `()`.
Load-bearing details:
- `JobQueue::cancel` spares tail nodes instead of cancelling the whole
subtree, and returns `bool`. Without this a cancelled approval DAG
would dangle its approval forever — the hazard `tests.rs` already
named. The spared tail's deps are `Cancelled`, which satisfies its weak
edge, so the scheduler claims it and it resolves the row as cancelled.
`hive-jobq` anticipated exactly this: `cancel_node`'s doc already says
to settle afterwards so "a weak-edge terminal node observing the
cancellation" can advance.
- The existing `complete(container)` call after cancelling is kept and is
deliberately a no-op when a tail was spared (a non-terminal child parks
the container back in `Finishing`), so power ops still settle
synchronously with no branch.
- `DeployTail` is NOT `is_tail()`: it does real compensating work, and a
cancelled DAG has nothing to compensate.
- `exec::failure_reason` falls back to `first_error(dag_id)` because a
group root that rolled up `Failed` from a child carries no error of its
own — without it every tail-reported failure would lose its reason.
- `EmitRebuilt` is per agent, so a multi-agent DAG reports each agent's
own outcome rather than painting all of them with the DAG roll-up.
- `ResolveApproval` is agentless: the approval row already names its
agent, and that is also what lets one tail close a multi-agent DAG.
Transients-derived-from-running-nodes and the frontend's node-kind
strings stay out of this change; they touch iris's slice and review
better next to their own diff.
A config deploy could silently discard committed agent config. sock's
icon commit carried a full proposal/approved/building/deployed tag set
yet was not an ancestor of `main` — genuinely deployed, then dropped.
Two gaps compounded.
`prepare_applied_target` is documented as "fast-forward applied/main to
target", but `git_update_ref` is `git update-ref <ref> <target>` with no
old-value guard: an unconditional force move. Anything reachable from the
old `main` but not from `target` leaves the branch without a word.
And nothing checked that it *was* a fast-forward. `run_deploy_merge_verify`
asserts exactly one thing about history — that the live PR head still
equals the reviewed sha. That is a drift gate on the *head*; it says
nothing about the *base*. A PR opened from a stale base passes it
unchanged and then rewinds `main` when it lands.
Adds, in the order they run:
- an ancestry gate as step 3 of MergeVerify — the reviewed head must
descend from `applied/main`, else bail before the irreversible merge.
It sits after the fetch (the commit has to be local to check
reachability) and before the eval, so it stays inside the region where
nothing is mutated and the node is still safely retryable.
- `git_update_ref_cas`, used for the `applied/main` move.
`git update-ref <ref> <new> <old>` refuses, and leaves the ref alone,
when the current value is not `old`. The ancestry gate only proves the
target is safe against the `main` observed *then*; the CAS is what
keeps that proof true *now*. `run_deploy_apply` already reads
`prev_main` to park the rollback ref, so that value is threaded in —
re-reading it inside the callee would reintroduce the race.
`git_is_ancestor` returns `Ok(false)` for exit 1 rather than treating
"not an ancestor" as a failure. Its doc comment notes this is not the
"did this branch land upstream" question: a squash-merge rewrites the
commit, so `--is-ancestor` correctly answers false for a branch whose
contents were merged. Different question, same command.
Tests cover both directions of the ancestry check, and that a stale CAS
both errors *and* leaves the ref where it was — a guard that fails while
still moving the ref would be worse than none.
Not covered here, deliberately: the non-PR apply path also writes `main`
and wants the same treatment. Kept separate to stay reviewable.
The `step` label was taken off the wire in #2661, when each deploy phase
became a first-class DAG node. Since then it has been written but never
read: `NodeRuntime` derives only `Debug, Default, Clone` — no serde — so
the field could not reach any client, and the only reads of it were the
dedup checks inside its own setters. This deletes the machinery.
Removed:
- `NodeRuntime.step`, `set_step`, `set_step_running`, and the
`rt.step = None` clear in `complete_node`. `NodeRuntime` keeps its
remaining `build_log_id` field (deliberately still a struct — collapsing
it to a bare `Option<i64>` would churn every call site for no gain).
- `Ctx::step` and its ~15 call sites in `job_queue/exec.rs`. `Ctx` itself
stays: it is the build-log sink, which `run_prebuild` and `run_swap`
still use.
- `Coordinator::set_queue_step` and its 11 callers in `actions.rs`.
- `JobQueue::running_node_of`, reachable only from `set_queue_step`.
- `swap_update`'s `on_step` parameter and its one body call.
- The `set_step_only_on_running_and_signals_change` test.
Dropping the calls orphaned parameters, which are removed with their call
sites: `ctx` on ten executors that used it only as a step sink, and
`queue_entry_id` on `run_deploy_merge_verify` / `run_deploy_apply` /
`run_finalize_deploy` plus both `coord` and `queue_entry_id` on
`prepare_applied_target`. `run_deploy_tail` KEEPS its `queue_entry_id` —
that one has a genuine surviving use (the build-log link in the failure
comment posted to the PR).
One behavioural change, called out so it is not mistaken for a dropped
dashboard refresh: `Ctx::step` and `set_queue_step` each emitted a
`rebuild_queue_changed` snapshot when the label changed, and those
emissions go away with them. This is safe — the snapshot payload has no
step field, so those pushes carried nothing a client could observe. Real
state transitions still emit from the scheduler's claim and completion
paths, from `submit`, and from the three `actions.rs` sites. Net effect is
strictly fewer redundant SSE pushes.
Docs: `docs/coordinator.md` still listed `step` as a `NodeView` wire field
and `docs/web-ui/dashboard.md` documented a cyan `↳ <step>` sub-line under
each queue row. Neither has existed since #2661 — both corrected here, plus
the `job_queue/model.rs` module doc.
Not touched: `frontend/packages/dashboard/src/system-sections.css` has a
dead `.rqe-step` rule with no JS referencing it. Left for the frontend
owner rather than deleted here.
Closes: #2664
The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.
Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.
The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.
`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.
The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.
Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
`run_approval_merge_config_pr` and `run_merge_config_pr` are gone; the
three phases are `run_deploy_merge_verify` (drift gate, fetch, verify —
mutates nothing), `run_deploy_apply` (merge + build) and
`run_deploy_tail` (compensation + push).
The rollback state is a git ref in the applied repo
(`refs/hyperhive/rollback/<approval-id>`) rather than a value handed
between nodes, because hive-c0re can restart between the apply and the
tail and the tail still has to know what to undo.
Rolling `main` back on a *successful* deploy is the worst thing the tail
can do, so it is guarded twice: the apply drops the rollback ref before
it plants `deployed/<id>`, and the tail refuses to compensate at all if
`deployed/<id>` resolves. It takes two independent git failures to get
there.
`run_deploy_tail` returns nothing and warns on every error — a failing
compensation must not mask the deploy's own verdict, which the terminal
hook takes from the DAG's roll-up.
Collapse the scattered ensure_agent_runtime_dir calls into the lifecycle
functions themselves so callers have a single responsibility:
- lifecycle::spawn: calls ensure_agent_runtime_dir before write_dropins.
Callers (handle_spawn, ensure_root_agent) no longer need a separate
preamble step.
- lifecycle::rebuild_no_meta spawn path: calls ensure_agent_runtime_dir
before write_dropins. apply_commit / merge_config_pr flows no longer
need a manual ensure_agent_runtime_dir.
- run_create (job-queue): drops ensure_agent_runtime_dir + register_agent.
The tail Reconcile's converge_start_preamble handles the runtime dir
and mcp_sockets::spawn_poll handles the listener. Create stays purely
'provision + create', not 'create + start'.
- handle_spawn (server.rs): drops manual preamble; lifecycle::spawn owns it.
Drops unneeded unregister_agent on failure (supervisor handles listener).
- ensure_root_agent (auto_update.rs): drops manual ensure_agent_runtime_dir.
- actions.rs apply_commit / merge_config_pr: drop manual
ensure_agent_runtime_dir; rebuild_no_meta's spawn path handles it.
Result: ensure_agent_runtime_dir lives in exactly two places —
lifecycle::spawn (direct spawn) and converge_start_preamble (start/reconcile
path). All other callers are clean call sites.
- lifecycle::ensure_agent_runtime_dir(name): pure filesystem op, no
Coordinator dep. Creates /run/hyperhive/agents/<name> without touching
the MCP listener map.
- workers/mcp_sockets::spawn_poll(coord): 10 s reconcile loop (same shape
as agent_sockets::spawn_poll). Converges 'agent running => MCP listener
bound'. First tick is immediate so hive-c0re restarts re-register all
running agents without waiting a full interval. Fixes the dead-listener-
after-daemon-restart gap.
- All ensure_runtime() call sites updated:
- Prebuild/Swap/WriteDropin: Coordinator::agent_dir() (pure, no IO)
- Reconcile-Start: ensure_agent_runtime_dir + agent_dir (dir may be
missing after reboot; listener deferred to supervisor)
- run_create / handle_spawn: ensure_agent_runtime_dir + register_agent
(eager on first spawn so socket ready before harness first turn)
- apply_commit / merge_config_pr: ensure_agent_runtime_dir + agent_dir
- Manager (auto_update): ensure_agent_runtime_dir + agent_dir
(manager has no MCP listener; socket_server::start_manager owns it)
- ensure_runtime() retained in Coordinator with updated doc pointing at
the preferred split form. No callers remain outside tests.
Root cause of the boot outage: container@h-* units try to start before
hive-c0re reaches ensure_runtime, so bind-mount source dirs are missing.
Fix: hive-c0re (via hive-priv, which runs as root) writes
/etc/tmpfiles.d/hyperhive-agents.conf whenever the agent set changes.
systemd-tmpfiles-setup.service (sysinit.target) reads it at every boot
BEFORE any container units start, pre-creating:
/run/hyperhive/agents/<name> — MCP socket dir (bind -> /run/hive)
/run/hive-agent/<name> — web socket dir (bind -> /run/hive-agent)
This alone removes the outage class: even if hive-c0re is slow to start,
the bind-mount sources exist and container units can activate.
Added:
- PrivRequest::SyncAgentTmpfiles { agents } in hive-sh4re
- sync_agent_tmpfiles() in hive-priv: generates content, writes atomically,
calls systemd-tmpfiles --create to apply immediately
- priv_client::sync_agent_tmpfiles() wrapper
- lifecycle::sync_tmpfiles() best-effort helper (list + priv call)
- Call sites: hive-c0re startup, handle_spawn success, destroy success
replaces three hand-rolled six-arm matches (actions.rs ×2,
state_snapshot.rs); approvals::kind_to_str delegates. a new kind can
no longer silently miss one of them
jobs are now DAGs of primitive nodes (prebuild, stop-for-update, swap,
reconcile, signal, drain, ...) driven by one scheduler with N build
slots + per-agent lifecycle leases. per-agent power intent (wanted
up/offline) is durable in agent_power.sqlite; Reconcile nodes converge
observed state to it. kills the graceful-stop watcher thread, the
deferred-start follow-up, and the cascade pre-enqueue (fan-out on
MetaLock completion instead). tracker: #2166
- Replace the #-number tracker references in code comments with prose
(tracker-tag lint; hive convention is prose in source).
- ensure_agent_subvolume now treats a chown failure on the freshly
created subvolume as fatal: it rolls the subvolume back (deletes it)
and returns an error, instead of warning and leaving a root-owned
subvol that hive-c0re can't write into (which would also make the
c0re-side exists-check skip the retry, wedging the agent).
Progressive enhancement: a brand-new agent's state root under
/var/lib/hyperhive/agents is created as a btrfs subvolume when the host
filesystem is btrfs, otherwise it falls back to a plain directory. No
existing agent is auto-migrated — the new path only fires when the root
does not yet exist, so plain-dir agents are left untouched until an
explicit opt-in upgrade.
Two new privileged ops (subvolume create/delete are root-only):
EnsureAgentSubvolume statfs-gates on btrfs, creates the subvolume, and
chowns it to the hive-core user so the normal state/claude/harness
mkdirs succeed inside it; DeleteAgentSubvolume btrfs-subvolume-deletes
the root iff it is actually a subvolume. hive-c0re calls Ensure before
the per-agent dirs are created (spawn/rebuild/InitConfig) and Delete on
the purge path only — destroy keeps the subvolume for revival, matching
plain-dir semantics. btrfs-progs added to the hive-priv unit PATH.
Per-subvolume usage accounting + optional quota is a separate
follow-up.
