A paused agent keeps its container, its claude session and its
dashboard/todo servers up, but stops driving turns. Messages queue
unacked and are drained on resume.
The whole protocol is a single marker file, `<harness>/paused`. That
directory is already a bind-mount shared between host and container, so
both sides just stat the same path: the harness reads it to decide
whether to drive a turn, hive-c0re reads it to render the badge and
writes/removes it for `hivectl pause|resume`. No new wire protocol, no
container round-trip, and it is sticky across restarts by construction.
Not calling `recv_next` while paused *is* the queueing semantic, so
there is no fencing to get wrong: reminders buffer in their unbounded
channel, the todo `Notify` permit coalesces, and a `request_next_turn`
that raced the pause survives because the gate sits above
`self_continue.take()`.
Graceful stop is handled host-side rather than in the harness: a paused
agent provably has no turn in flight, so `run_signal` skips the fence
entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for
a checkpoint turn that will never run.
`paused` is reported on `ContainerView` / `AgentStatusRow` for the
dashboard, orthogonal to `running` and reported for stopped containers
too.
Closes: hyperhive/hyperhive issue 2271
`fn main() -> Result<()>` let anyhow's Debug impl render failures with a
bare `Error:` header. hive-forge is almost always invoked from an agent's
bash task, where the completion wake points at the task's .out file - so a
failure that only writes to .err is easy to miss entirely (mara's "had no
clue it failed" on #2624).
Wrap the dispatch in a run() and own the failure path in main():
- prefix with the binary name (`hive-forge: FAILED: ...`) so the line is
unmistakably ours in a mixed transcript,
- render with {:#} (alternate Display), which keeps the full context chain
inline - plain Display would have dropped every `.context()` below the
top one,
- return ExitCode::FAILURE explicitly rather than relying on the Termination
impl.
Half of #2624: the other half (surfacing .err in the bash-mcp completion
when a task exits non-zero) is damocles's, per the issue thread.
The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.
Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.
The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.
`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.
The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.
Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
Rewrite the approval flow's step 4 as the three phases, with the reason
the rollback state is a git ref, and refresh the coordinator's node
inventory + dispatch table. Fix four doc comments that still pointed at
the deleted `run_merge_config_pr`.
Two cases, both pinning the load-bearing property that the tail is
reached on every path: a failed apply (AfterAny dep is terminal) and a
failed verify (apply is cancel-cascaded, tail still claimable). Both
assert the DAG rolls up to Failed — an Ok tail must not launder a failed
deploy into a success.
Non-derivable per-node payload rides the node that owns it. Tagging all
four deploy nodes with the approval id would render the same card four
times in `dag_view`.
Also fix `set_queue_step`'s doc comment, which claimed the DAG-id lookup
was exact because approval DAGs are single-node. They are not anymore;
what actually holds is that the chain is strictly sequential with the
root parked in Finishing, so at most one node is ever Running.
`run_approval_merge_config_pr` and `run_merge_config_pr` are gone; the
three phases are `run_deploy_merge_verify` (drift gate, fetch, verify —
mutates nothing), `run_deploy_apply` (merge + build) and
`run_deploy_tail` (compensation + push).
The rollback state is a git ref in the applied repo
(`refs/hyperhive/rollback/<approval-id>`) rather than a value handed
between nodes, because hive-c0re can restart between the apply and the
tail and the tail still has to know what to undo.
Rolling `main` back on a *successful* deploy is the worst thing the tail
can do, so it is guarded twice: the apply drops the rollback ref before
it plants `deployed/<id>`, and the tail refuses to compensate at all if
`deployed/<id>` resolves. It takes two independent git failures to get
there.
`run_deploy_tail` returns nothing and warns on every error — a failing
compensation must not mask the deploy's own verdict, which the terminal
hook takes from the DAG's roll-up.
A config-PR deploy was one opaque node that fetched, verified, merged,
built and compensated. That shape made three things impossible: the
nix-heavy phases could not take the meta window without the cheap ones
holding it too, a crash mid-build left no node to run the rollback, and
the dashboard could only ever show "deploying" for the whole thing.
Replace it with a `DeployWindow` group root over `MergeVerify ->
DeployApply` (AfterOk) plus a `DeployTail` hanging off the apply with
AfterAny, so the tail runs whether the apply succeeded, failed, or was
cancel-cascaded by a failing verify.
The two-phase approval deploy keeps a bumped `flake.lock` staged
uncommitted for the whole container build, so no other meta mutation may
land inside that span — until now enforced by a process-global
`meta::exclusive()` mutex held inside each executor fn.
A `MutexGuard` cannot outlive the fn that takes it, which is what blocks
decomposing the opaque `ApprovalDeploy` node into scheduler-visible
sub-nodes: the window has to span them. Replace the mutex with
`Resource::MetaWindow`, a global capacity-1 queue resource declared by
every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources
are held by a subtree root across its whole subtree, so a later increment
can hang the deploy's phases under one window-holding parent.
Same global serialisation as before, and the scheduler now blocks a node
from being claimed rather than parking a worker on a mutex.
Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync`
node. `Prebuild` must NOT hold the window: the old mutex was deliberately
scoped to drop before the multi-minute toplevel build, which only reads
the store, and a cap-1 global held across it would serialise every
agent's rebuild behind every other's. `MetaSync` is a sibling root that
`Prebuild` deps `AfterOk` on — not its parent, since a parent's resource
covers its whole subtree and would reintroduce exactly that problem.
Queue tests: shape assertions gain the extra node, which is the point of
the change (phases become nodes). The concurrency invariants are intact
but observed one step later — the `MetaSync` heads take turns on the
window, exactly as the runtime mutex made them, so those tests now
complete the heads before asserting that the prebuilds overlap.
Container nix invocations hard-failed whenever the remote builder
muede-pc2 was unreachable, while the identical build on the host
succeeded. Both go through the same host nix-daemon, so the difference
looked impossible.
The cause is that `fallback` is a client-side option: the nix client
transmits it to the daemon per connection (`tryFallback` in setOptions),
so the caller decides whether a failed remote dispatch may degrade to a
local build — even when the build itself runs on the host daemon under
NIX_REMOTE=daemon. Only genuinely daemon-side settings such as
`builders` are inherited from the host. The containers never set
`fallback`, so they took nix's default of false.
Set it in the agent-container base module and in the CI container, and
correct the hive-ci comment that claimed fallback was inherited from
the host daemon along with buildMachines and max-jobs.
Verified in an agent container: `nix fmt` fails outright on the remote
store's connection reset, while the same command with fallback enabled
reports the same connection error and then builds locally and succeeds.
per mara's review on #2679: replace the separate dispatch_todo sub-match
(with its trailing unreachable! arm) with four small handler functions
called directly from dispatch's existing match. same behavior, no
unreachable! left in the todo path.
diagnostic instrumentation for #2678 (phantom 'you have todos' wakes
after clearing bash-task todos). logs subsystem/key/id/changed on
UpsertTodo, subsystem/key/all/count on ClearTodo, id/count on
MarkTodoDone, and a marker when the serve loop actually consumes a
todo_wake notification. no behavior change - RUST_LOG=debug only.
`container_run` logs every stdout line at INFO (target `nixos-container`)
as operation progress. For the read-only `list` op — called on the hot
path (dashboard rescan, forge + boot sweeps) — that stdout is the return
value, not progress, so every call logs the full ~28-line container roster
at INFO. hive-c0re calls it several times a second, flooding the host
journal.
Gate the stdout per-line INFO logging on the op not being `list`. Mutating
ops still log their progress; stderr is still logged for every op (errors
matter regardless). No behaviour change beyond log volume.
Removes entryAgents() and the two places it rendered agent names:
- rqe-agent code element in the entry header
- rqe-node-agent-label prefix per component chain when multi-component
The DAG structure split (WCC + fan-out) already communicates subgraph
boundaries visually via the separate .rqe-nodes rows; the agent-name
labels on top of that caused layout breakage (#2666) and duplicate
information. Closes#2666.
The live build log header still labels liveNode.agent (a single specific
node, not the whole DAG) — that .rqe-agent rule is kept.
Also removes the now-unused .rqe-node-agent-label CSS rule and its
comment.
Drop the API path from --org and the "Forgejo applies it to the initial
commit" mechanics from --default-branch (kept the user-facing caveat:
only takes effect with --auto-init). Swept the remaining verbs
(attachment-get, pr-reviews, attach, repo-add-collaborator, comment,
clone, pr-cmd router, …) — already user-relevant, no changes needed.
Continue trimming clap arg help to user-relevant info: drop the
token-bounded-paging rationale (list --page), the why-it's-required
note (lint no-reviewer), the `Forgejo Do:`/`force_merge` API internals
(pr-merge), and tighten diff --full. pr-status was already clean.
Drop implementation detail from the clap arg help (the `<verb> --help`
surface) — which API/endpoint, page-count math, persisted-vs-streamer
log-source internals, refspec shapes — keeping only what/when-to-use for
each flag. Module `//!` docs (dev-facing, not shown by `--help`) left
intact.
Drop implementation mechanics from the `--help` surface, keep only
what a user needs to run the command:
- global `-r`/`-f`/`--json`: remove token-file paths, the bash-helper
history, and the "already-JSON verbs ignore --json" aside.
- verb `about` strings (repo-create/repo-labels/repo-search/artifact-get/
ci-log/ci-rerun/pr-commits): drop which-API / "no REST endpoint" /
web-route / workflow-dispatch internals and cross-refs.
Per-verb arg help (verbs/*.rs) trimmed in follow-up commits.
Per mara (#2660): the client shouldn't re-derive a friendly name from
node kinds — that re-bakes the domain knowledge the raw-graph redesign
removed. The queue card / hivectl header now shows what the backend sends
(the DAG's source + the raw node kinds); the node chain conveys the
operation. Removes the DagView::label() helper (hivectl was its only
consumer after iris dropped the frontend map in #2660) and points
hivectl's header at d.source instead.
Two tests asserted the old behaviour where completed nodes/DAGs stayed
in the snapshot. Under the redesign, Done nodes are filtered off the wire
(a fully-Done DAG disappears; a Failed one lingers + is history-capped):
- failed_node test: the completed reconcile is Done → assert it's absent,
not Done-present (its run is already verified by the claim).
- history-eviction test: fail the nodes so the DAGs linger (Done ones
would vanish), then assert the grace window + per-template cap.
hivectl/dag_progress.rs was reading the now-removed DagView.kind/state.
Derive both from the node set via the shared DagView::rollup_state() +
DagView::label() helpers (added to hive-sh4re). Timestamps are DateTime<Utc>
now — elapsed calcs compare in unix seconds. Dropped the live step display
(step left the wire). Test fixtures updated to the slim shape.
The raw-graph wire drops the inline build_log_id; the client fetches a
node's captured output on demand. Two handlers resolve node id -> log-row
id (JobQueue::build_log_id_of, now keyed by the wire u64) then delegate to
the existing get_full / raw handlers: /api/build-log/<node_id> serves the
BuildLogFull JSON ({stdout, stderr} + header), /raw serves text/plain.
404 when the node has no linked log.
Reconcile with mara + argus's review on the frontend PR (#2660):
- DagView regains started_at/finished_at (DateTime<Utc>), computed
host-side as min/max over ALL subtree nodes (including the Done ones
filtered off the wire). The client can't derive these — the
earliest/only-started node is often Done and absent — so the backend
sets them, per mara's call.
- NodeView gains has_log: bool = build_log_id.is_some(), the precise
old 'node has a captured build log' guard so the dashboard only shows
a log link for nodes that actually produce one.
Now that Template is internal to hive-c0re (not a pub wire enum), the
dead-code lint flags the two variants nothing ever constructs. Removed
them + their as_str arms; terminal_hook's catch-all arm is unaffected.
- server::await_dags: a DAG is settled when gone from the snapshot (fully
Done) or present with all nodes terminal; pending only with a non-terminal
node (DagView no longer carries a rolled-up state).
- DagView::rollup_state() added to hive-sh4re — the shared node-set roll-up
derivation every Rust consumer uses.
- JobQueue::build_log_id_of(node_id) — the node_id -> build_logs lookup the
query endpoint will use; tests assert log-id via it now.
- tests: derive roll-up state; drop the off-wire step/build_log_id wire asserts.
hive-c0re side of the raw-graph wire: dag_view now projects the slim
DagView, reading started_at/finished_at/error straight off the
hive_jobq Node (removes the node_rt double-write from #2645), excludes
Done nodes, and rides approval_id/inputs on the owning node. Template
moved into hive-c0re (model.rs) — no longer on the wire. Still WIP:
build-log endpoint + hivectl derive + compile fixes to follow.
WIP (hive-sh4re only; hive-c0re construction follows). Reshapes the
dashboard/hivectl queue wire per mara's redesign (#2637):
- DagView is now a thin projection: { id, source, reason, created_at, nodes }.
Dropped the rolled-up kind (Template)/state/started_at/finished_at/inputs/
approval_id — the client derives label + roll-up state + DAG timestamps
from the node set.
- NodeView drops step + inline build_log_id; started_at/finished_at are
chrono DateTime<Utc> (off the hive_jobq Node); non-derivable per-node
payload rides the owning node (approval_id on the approval node, inputs
on the meta_lock node).
- Deleted the Template enum entirely.
Done nodes are excluded from the wire (a fully-done DAG disappears; a
failed DAG lingers until the history cap). Build logs move to an on-demand
GET /api/build-log/<node_id> query (next commits).
Land argus's review nit from #2654 (it was pushed to that PR's branch as
fc4230f2 but got orphaned when the branch was merged + auto-deleted before
the fix landed). The hive-sh4re README opening said 'Agent / Manager
request + response shapes', which reads as if the wire envelopes still live
here; they moved to the per-socket crates. Now says 'shared payload
vocabulary' and notes where the envelopes live.
Second increment of the per-crate README effort, covering the rest of the
infra/wire/priv column: hive-priv, hive-metric, hive-types, hive-sh4re,
hive-core-agent-sock, hive-agent-sock. Same shape as the first batch —
purpose + when-to-use, and point at the crate-root //! docs plus the
relevant docs/ pages rather than duplicating them. Wires
readme = "README.md" into each Cargo.toml [package].
Disjoint from the batch-1 crates, so the two increments compose cleanly.
Adds crate READMEs (matching the hive-claude precedent) and wires
readme = "README.md" into each Cargo.toml [package] for hive-jobq,
hive-host-sock, and hive-priv-sock — the crates squarely in the infra
lane. Each README leads with purpose + when-to-use and points at the
crate-root //! docs for depth rather than duplicating them.
First increment of the per-crate-README effort; the shape here is the
proposed template for the remaining crates (see issue discussion).
hive-c0re's broker reminder store and /api/reminders endpoint were
removed in PR #2644 (reminders migrated to in-container sqlite store).
The dashboard had a QU3U3D R3M1ND3RS section on the schedules tab backed
by that endpoint, and a per-agent badge driven by pending_reminders
(always 0 after the migration).
Remove both. Per-agent reminders now surface through get_loose_ends /
the todos pill on the agent page, consistent with the todos migration.
- frontend/packages/dashboard/src/schedules.js: drop refreshReminders,
renderReminders, applyRemindersChanged; drop appendLinkified import
(no longer used); update module comment.
- frontend/packages/dashboard/src/tabs.js: drop applyRemindersChanged
and refreshReminders imports; remove reminders-section from managed
list; remove refreshReminders call site; drop reminders_changed from
SSE dispatch; simplify countdown ticker (reminder-due gone).
- frontend/packages/dashboard/src/dashboard.html: remove QU3U3D
R3M1ND3RS section.
- frontend/packages/dashboard/src/dashboard.css: remove reminder list +
row CSS.
- frontend/packages/dashboard/src/common.css: remove .badge-reminder.
- frontend/packages/dashboard/src/swarm.js: remove pending_reminders
from fingerprint key and badge render.
pending is the claude CLI's normal init-event race for a freshly
spawned stdio server (status reported before the handshake completes,
then flips to connected moments later in the same turn — see
anthropics/claude-agent-sdk-typescript#368). flagging it fired on
nearly every turn for all three configured servers with zero actual
impact, which is why #2038's persistent-vs-one-turn repro question
looked answered when it wasn't — that was this false positive, not
the real bug.
