The revert hook is dead by construction, so it can only ever be wrong.
DAG state `Cancelled` has exactly one producer: `JobQueue::cancel`, which
refuses unless every work node is still `Pending`. A cancel *cascade*
(some node failed, downstream cancelled) rolls up `Failed` instead —
`dag_rollup` short-circuits on any failed subtree node. So on a DAG that
reaches `Cancelled`, no node ever executed: the `SetWanted` head provably
never ran and `wanted` still reads whatever the operator last set it to.
There is therefore nothing to revert, and `revert_intent` did not revert
anything — it wrote `Wanted::from_running(observed)`, i.e. the agent's
*observed* state, over an intent the DAG never touched. Harmless when
observed already matched, silent corruption otherwise: cancel a queued
start for an agent that is down but `wanted = Up` (crashed, or caught
mid-bounce) and the intent flips to `Offline`, leaving it
deliberately-stopped as far as reconcile and crash-watch are concerned.
The hook made sense when `set_wanted` was a pre-submit side effect
written before the DAG ran; moving it into the DAG as a node left the
hook vestigial.
Drop `HookKind::RevertIntent`, `revert_intent`, and the power-op arm of
`terminal_hook` — start / stop / graceful-stop now settle with no
terminal hook, same as restart always did. The test asserts the general
statement across restart/stop/start x graceful x running: stop and start
carry a `SetWanted` head, and cancelling them still fires no hook.
`terminal_hook` mapped `Restart` / `GracefulRestart` to `RevertIntent`, but
a restart never writes `wanted` — `restart_chain` deliberately has no
`SetWanted` head, so the tail `Reconcile` converges to the agent's existing
intent and a deliberately-stopped agent isn't forced up by a hive-wide
restart.
`revert_intent` writes `Wanted::from_running(observed)` unconditionally on a
cancelled DAG. So for an agent that is `wanted = Up` but currently down
(crashed, or caught behind another queued op), submitting a restart and then
cancelling it writes `wanted = Offline` — reverting an intent the DAG never
touched, to a value nobody asked for. Reconcile and crash-watch both then
read the agent as deliberately-stopped and leave it down.
It's invisible for a running agent, since `from_running(true)` equals the
intent already on file, which is why it went unnoticed. `cancel` only
succeeds while every node is still `Pending`, so the reachable window is
exactly "queued restart + observed != intent" — precisely when someone
restarts and then thinks better of it.
Drop both restart templates from the `RevertIntent` arm; they fall through
to no terminal hook, which is correct for a DAG that writes no intent.
Document the invariant on `HookKind::RevertIntent` and on `revert_intent`
itself: the hook writes *observed* state, so dispatching it for a template
with no `SetWanted` head doesn't restore an intent, it invents one.
Test covers all four restart shapes (graceful x running), asserting both
that the spec carries no `SetWanted` and that a cancelled restart dispatches
no hook, with a contrast arm pinning stop's revert in place.
Fixeshyperhive/hyperhive#2710
Adds CPU/memory cap columns and an inline edit form to the container-load
table in /core.html, backed by a new POST /api/resource-limits/{name}
dashboard endpoint.
## Backend (hive-c0re)
lifecycle_ops.rs — new post_resource_limits handler:
- Parses ResourceLimitsForm { cpu_quota, memory_max } (both optional; empty
string = clear override, fall back to hive-wide default).
- Validates each non-empty value via resource_limits::validate_cpu_quota /
validate_memory_max — returns 422 UNPROCESSABLE_ENTITY with a human-
readable message on invalid input so the dashboard can surface it inline.
- Calls meta::commit_resource_limits (staged git write under META_LOCK, same
as hivectl set-limits).
- Re-applies the drop-in immediately via lifecycle::write_dropins so the new
ceilings take effect on the next container start without waiting for a
rebuild.
- Triggers rescan_containers_and_emit so ContainerView.cpu_quota/memory_max
update via SSE without waiting for the next periodic sweep.
dashboard/mod.rs — registers the route:
POST /api/resource-limits/{name}
## Frontend (core.js + system-sections.css)
core.js:
- containersState derived from /api/state snapshot alongside tombstonesState
— supplies configured cpu_quota/memory_max to the LOAD table.
- lastLoadRows stash lets SSE-triggered re-renders call renderContainerLoad
without waiting for the next 5s poll.
- renderContainerLoad: adds cpu cap / mem cap columns (muted; tooltip
'configured ceiling — takes effect on next start') sourced from
ContainerView, plus a per-row S3T toggle button that expands an inline
edit form with cpu_quota / memory_max text inputs and a S4V3 button.
The edit form shows a restart hint, surfaces validation errors inline, and
collapses on success.
- container_state_changed SSE handler: updates containersState in place and
re-renders the LOAD table so the cap columns flip immediately after a save.
system-sections.css:
- CSS for the new cap columns (.cload-cap-th, .cload-cap) and inline edit
form (.cload-edit-row, .cload-edit-form, .cload-edit-label, etc.).
- Remove dead .rqe-step rule (step sub-step label retired from the wire in
'job_queue: retire the now-off-wire step sub-step label').
A config deploy could silently discard committed agent config. sock's
icon commit carried a full proposal/approved/building/deployed tag set
yet was not an ancestor of `main` — genuinely deployed, then dropped.
Two gaps compounded.
`prepare_applied_target` is documented as "fast-forward applied/main to
target", but `git_update_ref` is `git update-ref <ref> <target>` with no
old-value guard: an unconditional force move. Anything reachable from the
old `main` but not from `target` leaves the branch without a word.
And nothing checked that it *was* a fast-forward. `run_deploy_merge_verify`
asserts exactly one thing about history — that the live PR head still
equals the reviewed sha. That is a drift gate on the *head*; it says
nothing about the *base*. A PR opened from a stale base passes it
unchanged and then rewinds `main` when it lands.
Adds, in the order they run:
- an ancestry gate as step 3 of MergeVerify — the reviewed head must
descend from `applied/main`, else bail before the irreversible merge.
It sits after the fetch (the commit has to be local to check
reachability) and before the eval, so it stays inside the region where
nothing is mutated and the node is still safely retryable.
- `git_update_ref_cas`, used for the `applied/main` move.
`git update-ref <ref> <new> <old>` refuses, and leaves the ref alone,
when the current value is not `old`. The ancestry gate only proves the
target is safe against the `main` observed *then*; the CAS is what
keeps that proof true *now*. `run_deploy_apply` already reads
`prev_main` to park the rollback ref, so that value is threaded in —
re-reading it inside the callee would reintroduce the race.
`git_is_ancestor` returns `Ok(false)` for exit 1 rather than treating
"not an ancestor" as a failure. Its doc comment notes this is not the
"did this branch land upstream" question: a squash-merge rewrites the
commit, so `--is-ancestor` correctly answers false for a branch whose
contents were merged. Different question, same command.
Tests cover both directions of the ancestry check, and that a stale CAS
both errors *and* leaves the ref where it was — a guard that fails while
still moving the ref would be worse than none.
Not covered here, deliberately: the non-PR apply path also writes `main`
and wants the same treatment. Kept separate to stay reviewable.
The `step` label was taken off the wire in #2661, when each deploy phase
became a first-class DAG node. Since then it has been written but never
read: `NodeRuntime` derives only `Debug, Default, Clone` — no serde — so
the field could not reach any client, and the only reads of it were the
dedup checks inside its own setters. This deletes the machinery.
Removed:
- `NodeRuntime.step`, `set_step`, `set_step_running`, and the
`rt.step = None` clear in `complete_node`. `NodeRuntime` keeps its
remaining `build_log_id` field (deliberately still a struct — collapsing
it to a bare `Option<i64>` would churn every call site for no gain).
- `Ctx::step` and its ~15 call sites in `job_queue/exec.rs`. `Ctx` itself
stays: it is the build-log sink, which `run_prebuild` and `run_swap`
still use.
- `Coordinator::set_queue_step` and its 11 callers in `actions.rs`.
- `JobQueue::running_node_of`, reachable only from `set_queue_step`.
- `swap_update`'s `on_step` parameter and its one body call.
- The `set_step_only_on_running_and_signals_change` test.
Dropping the calls orphaned parameters, which are removed with their call
sites: `ctx` on ten executors that used it only as a step sink, and
`queue_entry_id` on `run_deploy_merge_verify` / `run_deploy_apply` /
`run_finalize_deploy` plus both `coord` and `queue_entry_id` on
`prepare_applied_target`. `run_deploy_tail` KEEPS its `queue_entry_id` —
that one has a genuine surviving use (the build-log link in the failure
comment posted to the PR).
One behavioural change, called out so it is not mistaken for a dropped
dashboard refresh: `Ctx::step` and `set_queue_step` each emitted a
`rebuild_queue_changed` snapshot when the label changed, and those
emissions go away with them. This is safe — the snapshot payload has no
step field, so those pushes carried nothing a client could observe. Real
state transitions still emit from the scheduler's claim and completion
paths, from `submit`, and from the three `actions.rs` sites. Net effect is
strictly fewer redundant SSE pushes.
Docs: `docs/coordinator.md` still listed `step` as a `NodeView` wire field
and `docs/web-ui/dashboard.md` documented a cyan `↳ <step>` sub-line under
each queue row. Neither has existed since #2661 — both corrected here, plus
the `job_queue/model.rs` module doc.
Not touched: `frontend/packages/dashboard/src/system-sections.css` has a
dead `.rqe-step` rule with no JS referencing it. Left for the frontend
owner rather than deleted here.
Closes: #2664
`container_view::build_all` renders every container on every scan, and
each agent's row resolved its limits through `effective()`, which reads
and parses `resource-limits.json` from disk. That is one file read per
agent per scan of a file that is identical for all of them.
Split the resolution in two: `effective_from` takes an already-loaded
map, and `effective` keeps the read-then-resolve shape for the
single-agent callers (`write_dropins`, which runs once per spawn and
rebuild and has no map to hand). `build_all` now loads the map once at
the top — the same treatment `topology::read()` already gets there —
and calls `effective_from` per agent.
No behaviour change: the fallback matrix lives in `resolve`, which both
paths still go through, and its tests are untouched.
The now-redundant `limits_for` is gone; `effective_from` covers its one
caller.
The hive applies one `agentCpuQuota` / `agentMemoryMax` to every
container. That's the right default and the wrong ceiling: a build-heavy
agent needs headroom the other twelve don't, and raising the hive-wide
value to suit it hands that headroom to everyone.
Adds a per-agent override, persisted host-side and resolved per-field
against the hive defaults.
Follows the existing `meta/*.json` pattern (`capabilities.json`,
`tool-groups.json`): a host-side map read by `hive-c0re`, staged and
committed in the meta repo so every change lands in the audit trail.
```json
{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }
```
Fallback is **per field**, not per agent: an entry with only
`memory_max` leaves that agent on the hive-wide CPU quota. Absent file,
absent agent and absent field all resolve to the hive default, so the
feature is inert until someone opts an agent in.
Unlike the other meta files this one is **not** injected into the
container — a limit is something done *to* an agent, not something it
reads about itself.
```
hivectl agents set-limits sock --cpu-quota 400% --memory-max 8G
hivectl agents set-limits sock --reset
```
Values are validated before they're persisted: they go into a systemd
drop-in verbatim, and a typo there makes the unit fail to *start* —
turning a fat-fingered quota into a container that won't come back.
The command is declarative: each call replaces the agent's whole entry.
That makes a forgotten flag a silent revert, so a bare `set-limits
<name>` is rejected at the clap layer and clearing needs an explicit
`--reset`.
`ContainerView` gains `cpu_quota` / `memory_max`, both always populated:
there's no "unset" state to render, only "same as everyone else". They
reflect what the drop-in *says* — what the next start will enforce — not
a live cgroup reading.
The write goes through `meta::commit_resource_limits` rather than the
bare setter, so it's staged and committed under `META_LOCK`. Writing
without committing would leave the meta working tree dirty for the next
`prepare_deploy` to trip over.
Docs: `persistence.md` (the new meta file, and why it isn't injected),
`tools/hivectl.md` (the prose guide), `tools/hivectl-cli.md`
(regenerated clap dump).
Closes: internal/requests issue 25
When ContainerView.paused is true, show a clickable yellow `⏸ paused`
badge on the agent card that POSTs to the new /api/resume/{name} endpoint
to un-park the turn loop. The badge doubles as the resume button so the
state is self-documenting and one click to fix.
The agent action menu gains ⏸ P4US3 (when not paused) and ▶ R3SUM3
(when paused), orthogonal to the running/stopped start/stop actions.
On the backend, /api/pause/{name} and /api/resume/{name} POST routes
wire to Coordinator::set_paused and trigger an immediate rescan so the
badge flips via the existing SSE ContainerUpdate without polling.
Depends on the ContainerView.paused field and Coordinator::set_paused
added in the parent PR.
Add NodeView::parent to the wire (hive-sh4re + hive-c0re dag_view), then
render the recursive parent/child tree in the dashboard build queue instead
of the previous flat chain/fan-out layout.
Wire change (hive-sh4re, hive-c0re):
- NodeView gains parent: Option<NodeId> (skip_serializing_if = None)
- dag_view() projects node.parent, filtering out the Dag container id
(top-level work nodes become parent: None on the wire)
Frontend (builds.js):
- Replace nodeComponents + splitFanOut with buildNodeTree (uses parent
edges directly) + topoSort helper (orders siblings by deps)
- renderTreeNode walks the tree depth-first, rendering indented rows
with └─/├─ connectors and agent label per chip
- Flat chains and fan-out heuristics are gone; structure comes straight
from the scheduler's parent axis
Closes: none (parent issue tracked in forge)
A paused agent keeps its container, its claude session and its
dashboard/todo servers up, but stops driving turns. Messages queue
unacked and are drained on resume.
The whole protocol is a single marker file, `<harness>/paused`. That
directory is already a bind-mount shared between host and container, so
both sides just stat the same path: the harness reads it to decide
whether to drive a turn, hive-c0re reads it to render the badge and
writes/removes it for `hivectl pause|resume`. No new wire protocol, no
container round-trip, and it is sticky across restarts by construction.
Not calling `recv_next` while paused *is* the queueing semantic, so
there is no fencing to get wrong: reminders buffer in their unbounded
channel, the todo `Notify` permit coalesces, and a `request_next_turn`
that raced the pause survives because the gate sits above
`self_continue.take()`.
Graceful stop is handled host-side rather than in the harness: a paused
agent provably has no turn in flight, so `run_signal` skips the fence
entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for
a checkpoint turn that will never run.
`paused` is reported on `ContainerView` / `AgentStatusRow` for the
dashboard, orthogonal to `running` and reported for stopped containers
too.
Closes: hyperhive/hyperhive issue 2271
The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.
Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.
The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.
`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.
The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.
Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
Rewrite the approval flow's step 4 as the three phases, with the reason
the rollback state is a git ref, and refresh the coordinator's node
inventory + dispatch table. Fix four doc comments that still pointed at
the deleted `run_merge_config_pr`.
Two cases, both pinning the load-bearing property that the tail is
reached on every path: a failed apply (AfterAny dep is terminal) and a
failed verify (apply is cancel-cascaded, tail still claimable). Both
assert the DAG rolls up to Failed — an Ok tail must not launder a failed
deploy into a success.
Non-derivable per-node payload rides the node that owns it. Tagging all
four deploy nodes with the approval id would render the same card four
times in `dag_view`.
Also fix `set_queue_step`'s doc comment, which claimed the DAG-id lookup
was exact because approval DAGs are single-node. They are not anymore;
what actually holds is that the chain is strictly sequential with the
root parked in Finishing, so at most one node is ever Running.
`run_approval_merge_config_pr` and `run_merge_config_pr` are gone; the
three phases are `run_deploy_merge_verify` (drift gate, fetch, verify —
mutates nothing), `run_deploy_apply` (merge + build) and
`run_deploy_tail` (compensation + push).
The rollback state is a git ref in the applied repo
(`refs/hyperhive/rollback/<approval-id>`) rather than a value handed
between nodes, because hive-c0re can restart between the apply and the
tail and the tail still has to know what to undo.
Rolling `main` back on a *successful* deploy is the worst thing the tail
can do, so it is guarded twice: the apply drops the rollback ref before
it plants `deployed/<id>`, and the tail refuses to compensate at all if
`deployed/<id>` resolves. It takes two independent git failures to get
there.
`run_deploy_tail` returns nothing and warns on every error — a failing
compensation must not mask the deploy's own verdict, which the terminal
hook takes from the DAG's roll-up.
A config-PR deploy was one opaque node that fetched, verified, merged,
built and compensated. That shape made three things impossible: the
nix-heavy phases could not take the meta window without the cheap ones
holding it too, a crash mid-build left no node to run the rollback, and
the dashboard could only ever show "deploying" for the whole thing.
Replace it with a `DeployWindow` group root over `MergeVerify ->
DeployApply` (AfterOk) plus a `DeployTail` hanging off the apply with
AfterAny, so the tail runs whether the apply succeeded, failed, or was
cancel-cascaded by a failing verify.
The two-phase approval deploy keeps a bumped `flake.lock` staged
uncommitted for the whole container build, so no other meta mutation may
land inside that span — until now enforced by a process-global
`meta::exclusive()` mutex held inside each executor fn.
A `MutexGuard` cannot outlive the fn that takes it, which is what blocks
decomposing the opaque `ApprovalDeploy` node into scheduler-visible
sub-nodes: the window has to span them. Replace the mutex with
`Resource::MetaWindow`, a global capacity-1 queue resource declared by
every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources
are held by a subtree root across its whole subtree, so a later increment
can hang the deploy's phases under one window-holding parent.
Same global serialisation as before, and the scheduler now blocks a node
from being claimed rather than parking a worker on a mutex.
Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync`
node. `Prebuild` must NOT hold the window: the old mutex was deliberately
scoped to drop before the multi-minute toplevel build, which only reads
the store, and a cap-1 global held across it would serialise every
agent's rebuild behind every other's. `MetaSync` is a sibling root that
`Prebuild` deps `AfterOk` on — not its parent, since a parent's resource
covers its whole subtree and would reintroduce exactly that problem.
Queue tests: shape assertions gain the extra node, which is the point of
the change (phases become nodes). The concurrency invariants are intact
but observed one step later — the `MetaSync` heads take turns on the
window, exactly as the runtime mutex made them, so those tests now
complete the heads before asserting that the prebuilds overlap.
Two tests asserted the old behaviour where completed nodes/DAGs stayed
in the snapshot. Under the redesign, Done nodes are filtered off the wire
(a fully-Done DAG disappears; a Failed one lingers + is history-capped):
- failed_node test: the completed reconcile is Done → assert it's absent,
not Done-present (its run is already verified by the claim).
- history-eviction test: fail the nodes so the DAGs linger (Done ones
would vanish), then assert the grace window + per-template cap.
The raw-graph wire drops the inline build_log_id; the client fetches a
node's captured output on demand. Two handlers resolve node id -> log-row
id (JobQueue::build_log_id_of, now keyed by the wire u64) then delegate to
the existing get_full / raw handlers: /api/build-log/<node_id> serves the
BuildLogFull JSON ({stdout, stderr} + header), /raw serves text/plain.
404 when the node has no linked log.
Reconcile with mara + argus's review on the frontend PR (#2660):
- DagView regains started_at/finished_at (DateTime<Utc>), computed
host-side as min/max over ALL subtree nodes (including the Done ones
filtered off the wire). The client can't derive these — the
earliest/only-started node is often Done and absent — so the backend
sets them, per mara's call.
- NodeView gains has_log: bool = build_log_id.is_some(), the precise
old 'node has a captured build log' guard so the dashboard only shows
a log link for nodes that actually produce one.
Now that Template is internal to hive-c0re (not a pub wire enum), the
dead-code lint flags the two variants nothing ever constructs. Removed
them + their as_str arms; terminal_hook's catch-all arm is unaffected.
- server::await_dags: a DAG is settled when gone from the snapshot (fully
Done) or present with all nodes terminal; pending only with a non-terminal
node (DagView no longer carries a rolled-up state).
- DagView::rollup_state() added to hive-sh4re — the shared node-set roll-up
derivation every Rust consumer uses.
- JobQueue::build_log_id_of(node_id) — the node_id -> build_logs lookup the
query endpoint will use; tests assert log-id via it now.
- tests: derive roll-up state; drop the off-wire step/build_log_id wire asserts.
hive-c0re side of the raw-graph wire: dag_view now projects the slim
DagView, reading started_at/finished_at/error straight off the
hive_jobq Node (removes the node_rt double-write from #2645), excludes
Done nodes, and rides approval_id/inputs on the owning node. Template
moved into hive-c0re (model.rs) — no longer on the wire. Still WIP:
build-log endpoint + hivectl derive + compile fixes to follow.
Node gains started_at/finished_at (chrono DateTime<Utc>, serialized
RFC 3339 on the wire per hive_sh4re::wire_time) plus error (String).
Graph::set_state self-stamps started_at on the first Running transition
and finished_at on the first terminal one, via an internal now_utc()
clock (keeps settle/complete signatures stable). Outcome::Failed(String)
carries the failure reason, set on the terminal transition.
hive-c0re complete_node builds Outcome::Failed(msg); its node_rt
side-table stays i64 for now (double-write) until #2637 reads the Node.
Toward #2637: the jobq graph becomes the source of truth for per-node
lifecycle so the queue can be sent to the client as-is.
Replace the in-tree scheduler with the domain-agnostic hive-jobq crate
(merged in #2615): parent-axis grouping + borrow/subtree-reservation
resource model + roll-up completion (State::Finishing).
Host adaptation:
- NodeSpec gains an explicit `parent` axis; templates declare grouping +
sibling ordering directly (deps order execution, parent groups a subtree
whose resource the descendants borrow).
- Rebuild is a nested two-root subtree: Prebuild (root, owns the build slot
for the whole subtree, lease-exempt) -> StopForUpdate (child, owns the
agent lease) -> Swap/PostSwap (children, borrow both); Reconcile is a
separate top-level root (AfterAny Prebuild) so it survives the cancel-
cascade of any failed step (recovery-start invariant) and converges to
the persisted `wanted` on a fresh lease. This is the multi-root
correction to the single-root-chain sketch: node0=root broke lease-
exemption (hoisting the lease onto Prebuild) and recovery-reconcile
(root failure cancels all children).
- Spawn / perm-change / power-ops (stop/start/restart) group-rooted the
same way; per-agent power-op subgraphs stay independent roots so a
multi-agent DAG runs them concurrently, each on its own lease.
- insert_group honours the explicit parent axis (no lease hoisting); the
DAG terminal node deps AfterAny on every group root and runs once the
whole op rolls up. Drop the old Graph::add_dep terminal wiring.
36/36 job_queue tests, full hive-c0re suite green, clippy --all-targets.
