A config deploy could silently discard committed agent config. sock's
icon commit carried a full proposal/approved/building/deployed tag set
yet was not an ancestor of `main` — genuinely deployed, then dropped.
Two gaps compounded.
`prepare_applied_target` is documented as "fast-forward applied/main to
target", but `git_update_ref` is `git update-ref <ref> <target>` with no
old-value guard: an unconditional force move. Anything reachable from the
old `main` but not from `target` leaves the branch without a word.
And nothing checked that it *was* a fast-forward. `run_deploy_merge_verify`
asserts exactly one thing about history — that the live PR head still
equals the reviewed sha. That is a drift gate on the *head*; it says
nothing about the *base*. A PR opened from a stale base passes it
unchanged and then rewinds `main` when it lands.
Adds, in the order they run:
- an ancestry gate as step 3 of MergeVerify — the reviewed head must
descend from `applied/main`, else bail before the irreversible merge.
It sits after the fetch (the commit has to be local to check
reachability) and before the eval, so it stays inside the region where
nothing is mutated and the node is still safely retryable.
- `git_update_ref_cas`, used for the `applied/main` move.
`git update-ref <ref> <new> <old>` refuses, and leaves the ref alone,
when the current value is not `old`. The ancestry gate only proves the
target is safe against the `main` observed *then*; the CAS is what
keeps that proof true *now*. `run_deploy_apply` already reads
`prev_main` to park the rollback ref, so that value is threaded in —
re-reading it inside the callee would reintroduce the race.
`git_is_ancestor` returns `Ok(false)` for exit 1 rather than treating
"not an ancestor" as a failure. Its doc comment notes this is not the
"did this branch land upstream" question: a squash-merge rewrites the
commit, so `--is-ancestor` correctly answers false for a branch whose
contents were merged. Different question, same command.
Tests cover both directions of the ancestry check, and that a stale CAS
both errors *and* leaves the ref where it was — a guard that fails while
still moving the ref would be worse than none.
Not covered here, deliberately: the non-PR apply path also writes `main`
and wants the same treatment. Kept separate to stay reviewable.
The `step` label was taken off the wire in #2661, when each deploy phase
became a first-class DAG node. Since then it has been written but never
read: `NodeRuntime` derives only `Debug, Default, Clone` — no serde — so
the field could not reach any client, and the only reads of it were the
dedup checks inside its own setters. This deletes the machinery.
Removed:
- `NodeRuntime.step`, `set_step`, `set_step_running`, and the
`rt.step = None` clear in `complete_node`. `NodeRuntime` keeps its
remaining `build_log_id` field (deliberately still a struct — collapsing
it to a bare `Option<i64>` would churn every call site for no gain).
- `Ctx::step` and its ~15 call sites in `job_queue/exec.rs`. `Ctx` itself
stays: it is the build-log sink, which `run_prebuild` and `run_swap`
still use.
- `Coordinator::set_queue_step` and its 11 callers in `actions.rs`.
- `JobQueue::running_node_of`, reachable only from `set_queue_step`.
- `swap_update`'s `on_step` parameter and its one body call.
- The `set_step_only_on_running_and_signals_change` test.
Dropping the calls orphaned parameters, which are removed with their call
sites: `ctx` on ten executors that used it only as a step sink, and
`queue_entry_id` on `run_deploy_merge_verify` / `run_deploy_apply` /
`run_finalize_deploy` plus both `coord` and `queue_entry_id` on
`prepare_applied_target`. `run_deploy_tail` KEEPS its `queue_entry_id` —
that one has a genuine surviving use (the build-log link in the failure
comment posted to the PR).
One behavioural change, called out so it is not mistaken for a dropped
dashboard refresh: `Ctx::step` and `set_queue_step` each emitted a
`rebuild_queue_changed` snapshot when the label changed, and those
emissions go away with them. This is safe — the snapshot payload has no
step field, so those pushes carried nothing a client could observe. Real
state transitions still emit from the scheduler's claim and completion
paths, from `submit`, and from the three `actions.rs` sites. Net effect is
strictly fewer redundant SSE pushes.
Docs: `docs/coordinator.md` still listed `step` as a `NodeView` wire field
and `docs/web-ui/dashboard.md` documented a cyan `↳ <step>` sub-line under
each queue row. Neither has existed since #2661 — both corrected here, plus
the `job_queue/model.rs` module doc.
Not touched: `frontend/packages/dashboard/src/system-sections.css` has a
dead `.rqe-step` rule with no JS referencing it. Left for the frontend
owner rather than deleted here.
Closes: #2664
logs.html/logs.js previously showed infra containers (hive-ci, hive-forge,
hive-gateway, hive-matrix) in an optgroup within the AGENT tab selector.
This was confusing because infra containers don't run the per-agent hive
daemons, making the unit filter meaningless for them.
Changes:
- Add INFRA tab (between AGENT and SYSTEM) with its own container selector
and full-machine-journal fetch (no unit filter).
- Remove the infra optgroup from the AGENT tab — it now shows agents only.
- loadContainerLists() replaces loadAgentList(): fetches /api/state once and
populates both selectors, avoiding a duplicate network request.
- Deep-link (?agent=hive-ci) now routes to the INFRA tab when the named
container is an infra container, falling back to AGENT otherwise.
- Remove syncUnitSelectForSelection() — no longer needed since the AGENT
tab no longer contains infra containers.
- Extend the 30s timestamp ticker to cover the INFRA tab fetch time.
No backend changes: /api/journal/{name} already supports infra container names.
`container_view::build_all` renders every container on every scan, and
each agent's row resolved its limits through `effective()`, which reads
and parses `resource-limits.json` from disk. That is one file read per
agent per scan of a file that is identical for all of them.
Split the resolution in two: `effective_from` takes an already-loaded
map, and `effective` keeps the read-then-resolve shape for the
single-agent callers (`write_dropins`, which runs once per spawn and
rebuild and has no map to hand). `build_all` now loads the map once at
the top — the same treatment `topology::read()` already gets there —
and calls `effective_from` per agent.
No behaviour change: the fallback matrix lives in `resolve`, which both
paths still go through, and its tests are untouched.
The now-redundant `limits_for` is gone; `effective_from` covers its one
caller.
The hive applies one `agentCpuQuota` / `agentMemoryMax` to every
container. That's the right default and the wrong ceiling: a build-heavy
agent needs headroom the other twelve don't, and raising the hive-wide
value to suit it hands that headroom to everyone.
Adds a per-agent override, persisted host-side and resolved per-field
against the hive defaults.
Follows the existing `meta/*.json` pattern (`capabilities.json`,
`tool-groups.json`): a host-side map read by `hive-c0re`, staged and
committed in the meta repo so every change lands in the audit trail.
```json
{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }
```
Fallback is **per field**, not per agent: an entry with only
`memory_max` leaves that agent on the hive-wide CPU quota. Absent file,
absent agent and absent field all resolve to the hive default, so the
feature is inert until someone opts an agent in.
Unlike the other meta files this one is **not** injected into the
container — a limit is something done *to* an agent, not something it
reads about itself.
```
hivectl agents set-limits sock --cpu-quota 400% --memory-max 8G
hivectl agents set-limits sock --reset
```
Values are validated before they're persisted: they go into a systemd
drop-in verbatim, and a typo there makes the unit fail to *start* —
turning a fat-fingered quota into a container that won't come back.
The command is declarative: each call replaces the agent's whole entry.
That makes a forgotten flag a silent revert, so a bare `set-limits
<name>` is rejected at the clap layer and clearing needs an explicit
`--reset`.
`ContainerView` gains `cpu_quota` / `memory_max`, both always populated:
there's no "unset" state to render, only "same as everyone else". They
reflect what the drop-in *says* — what the next start will enforce — not
a live cgroup reading.
The write goes through `meta::commit_resource_limits` rather than the
bare setter, so it's staged and committed under `META_LOCK`. Writing
without committing would leave the meta working tree dirty for the next
`prepare_deploy` to trip over.
Docs: `persistence.md` (the new meta file, and why it isn't injected),
`tools/hivectl.md` (the prose guide), `tools/hivectl-cli.md`
(regenerated clap dump).
Closes: internal/requests issue 25
When ContainerView.paused is true, show a clickable yellow `⏸ paused`
badge on the agent card that POSTs to the new /api/resume/{name} endpoint
to un-park the turn loop. The badge doubles as the resume button so the
state is self-documenting and one click to fix.
The agent action menu gains ⏸ P4US3 (when not paused) and ▶ R3SUM3
(when paused), orthogonal to the running/stopped start/stop actions.
On the backend, /api/pause/{name} and /api/resume/{name} POST routes
wire to Coordinator::set_paused and trigger an immediate rescan so the
badge flips via the existing SSE ContainerUpdate without polling.
Depends on the ContainerView.paused field and Coordinator::set_paused
added in the parent PR.
dag_progress.rs's test-only node() constructor was missing the new
parent field added to NodeView in hive-sh4re. Add parent: None to
silence the missing-field compile error.
Replace the Unicode prefix string approach (└─ / ├─ / │ built up as text
in a single <span class="rqe-tree-indent">) with positioned DOM elements
that draw real lines:
- rqe-tree-guide: fixed-width ancestor column, optionally draws a full
vertical border-left when the ancestor has siblings below it
(.rqe-tree-guide-line).
- rqe-tree-connector: draws the L/T shape via ::before (vertical stem,
top→center for last child, full height for mid child) and ::after
(horizontal spur, center→right). .rqe-tree-connector-last vs
.rqe-tree-connector-mid controls stem length.
renderTreeNode() now takes ancestorLines: boolean[] instead of a prefix
string. Each entry is true when the ancestor at that depth was not the
last child (so a vertical guide is still needed through that column).
childAncestorLines propagates depth === 0 correctly (root nodes have no
guide columns, so their children start with an empty array).
Lines are drawn with var(--border) so they follow the theme and work at
any font size without alignment drift. Addresses the review note on PR 2686.
Add NodeView::parent to the wire (hive-sh4re + hive-c0re dag_view), then
render the recursive parent/child tree in the dashboard build queue instead
of the previous flat chain/fan-out layout.
Wire change (hive-sh4re, hive-c0re):
- NodeView gains parent: Option<NodeId> (skip_serializing_if = None)
- dag_view() projects node.parent, filtering out the Dag container id
(top-level work nodes become parent: None on the wire)
Frontend (builds.js):
- Replace nodeComponents + splitFanOut with buildNodeTree (uses parent
edges directly) + topoSort helper (orders siblings by deps)
- renderTreeNode walks the tree depth-first, rendering indented rows
with └─/├─ connectors and agent label per chip
- Flat chains and fan-out heuristics are gone; structure comes straight
from the scheduler's parent axis
Closes: none (parent issue tracked in forge)
A paused agent keeps its container, its claude session and its
dashboard/todo servers up, but stops driving turns. Messages queue
unacked and are drained on resume.
The whole protocol is a single marker file, `<harness>/paused`. That
directory is already a bind-mount shared between host and container, so
both sides just stat the same path: the harness reads it to decide
whether to drive a turn, hive-c0re reads it to render the badge and
writes/removes it for `hivectl pause|resume`. No new wire protocol, no
container round-trip, and it is sticky across restarts by construction.
Not calling `recv_next` while paused *is* the queueing semantic, so
there is no fencing to get wrong: reminders buffer in their unbounded
channel, the todo `Notify` permit coalesces, and a `request_next_turn`
that raced the pause survives because the gate sits above
`self_continue.take()`.
Graceful stop is handled host-side rather than in the harness: a paused
agent provably has no turn in flight, so `run_signal` skips the fence
entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for
a checkpoint turn that will never run.
`paused` is reported on `ContainerView` / `AgentStatusRow` for the
dashboard, orthogonal to `running` and reported for stopped containers
too.
Closes: hyperhive/hyperhive issue 2271
`fn main() -> Result<()>` let anyhow's Debug impl render failures with a
bare `Error:` header. hive-forge is almost always invoked from an agent's
bash task, where the completion wake points at the task's .out file - so a
failure that only writes to .err is easy to miss entirely (mara's "had no
clue it failed" on #2624).
Wrap the dispatch in a run() and own the failure path in main():
- prefix with the binary name (`hive-forge: FAILED: ...`) so the line is
unmistakably ours in a mixed transcript,
- render with {:#} (alternate Display), which keeps the full context chain
inline - plain Display would have dropped every `.context()` below the
top one,
- return ExitCode::FAILURE explicitly rather than relying on the Termination
impl.
Half of #2624: the other half (surfacing .err in the bash-mcp completion
when a task exits non-zero) is damocles's, per the issue thread.
The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.
Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.
The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.
`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.
The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.
Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
Rewrite the approval flow's step 4 as the three phases, with the reason
the rollback state is a git ref, and refresh the coordinator's node
inventory + dispatch table. Fix four doc comments that still pointed at
the deleted `run_merge_config_pr`.
Two cases, both pinning the load-bearing property that the tail is
reached on every path: a failed apply (AfterAny dep is terminal) and a
failed verify (apply is cancel-cascaded, tail still claimable). Both
assert the DAG rolls up to Failed — an Ok tail must not launder a failed
deploy into a success.
Non-derivable per-node payload rides the node that owns it. Tagging all
four deploy nodes with the approval id would render the same card four
times in `dag_view`.
Also fix `set_queue_step`'s doc comment, which claimed the DAG-id lookup
was exact because approval DAGs are single-node. They are not anymore;
what actually holds is that the chain is strictly sequential with the
root parked in Finishing, so at most one node is ever Running.
`run_approval_merge_config_pr` and `run_merge_config_pr` are gone; the
three phases are `run_deploy_merge_verify` (drift gate, fetch, verify —
mutates nothing), `run_deploy_apply` (merge + build) and
`run_deploy_tail` (compensation + push).
The rollback state is a git ref in the applied repo
(`refs/hyperhive/rollback/<approval-id>`) rather than a value handed
between nodes, because hive-c0re can restart between the apply and the
tail and the tail still has to know what to undo.
Rolling `main` back on a *successful* deploy is the worst thing the tail
can do, so it is guarded twice: the apply drops the rollback ref before
it plants `deployed/<id>`, and the tail refuses to compensate at all if
`deployed/<id>` resolves. It takes two independent git failures to get
there.
`run_deploy_tail` returns nothing and warns on every error — a failing
compensation must not mask the deploy's own verdict, which the terminal
hook takes from the DAG's roll-up.
A config-PR deploy was one opaque node that fetched, verified, merged,
built and compensated. That shape made three things impossible: the
nix-heavy phases could not take the meta window without the cheap ones
holding it too, a crash mid-build left no node to run the rollback, and
the dashboard could only ever show "deploying" for the whole thing.
Replace it with a `DeployWindow` group root over `MergeVerify ->
DeployApply` (AfterOk) plus a `DeployTail` hanging off the apply with
AfterAny, so the tail runs whether the apply succeeded, failed, or was
cancel-cascaded by a failing verify.
The two-phase approval deploy keeps a bumped `flake.lock` staged
uncommitted for the whole container build, so no other meta mutation may
land inside that span — until now enforced by a process-global
`meta::exclusive()` mutex held inside each executor fn.
A `MutexGuard` cannot outlive the fn that takes it, which is what blocks
decomposing the opaque `ApprovalDeploy` node into scheduler-visible
sub-nodes: the window has to span them. Replace the mutex with
`Resource::MetaWindow`, a global capacity-1 queue resource declared by
every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources
are held by a subtree root across its whole subtree, so a later increment
can hang the deploy's phases under one window-holding parent.
Same global serialisation as before, and the scheduler now blocks a node
from being claimed rather than parking a worker on a mutex.
Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync`
node. `Prebuild` must NOT hold the window: the old mutex was deliberately
scoped to drop before the multi-minute toplevel build, which only reads
the store, and a cap-1 global held across it would serialise every
agent's rebuild behind every other's. `MetaSync` is a sibling root that
`Prebuild` deps `AfterOk` on — not its parent, since a parent's resource
covers its whole subtree and would reintroduce exactly that problem.
Queue tests: shape assertions gain the extra node, which is the point of
the change (phases become nodes). The concurrency invariants are intact
but observed one step later — the `MetaSync` heads take turns on the
window, exactly as the runtime mutex made them, so those tests now
complete the heads before asserting that the prebuilds overlap.
Container nix invocations hard-failed whenever the remote builder
muede-pc2 was unreachable, while the identical build on the host
succeeded. Both go through the same host nix-daemon, so the difference
looked impossible.
The cause is that `fallback` is a client-side option: the nix client
transmits it to the daemon per connection (`tryFallback` in setOptions),
so the caller decides whether a failed remote dispatch may degrade to a
local build — even when the build itself runs on the host daemon under
NIX_REMOTE=daemon. Only genuinely daemon-side settings such as
`builders` are inherited from the host. The containers never set
`fallback`, so they took nix's default of false.
Set it in the agent-container base module and in the CI container, and
correct the hive-ci comment that claimed fallback was inherited from
the host daemon along with buildMachines and max-jobs.
Verified in an agent container: `nix fmt` fails outright on the remote
store's connection reset, while the same command with fallback enabled
reports the same connection error and then builds locally and succeeds.
per mara's review on #2679: replace the separate dispatch_todo sub-match
(with its trailing unreachable! arm) with four small handler functions
called directly from dispatch's existing match. same behavior, no
unreachable! left in the todo path.
diagnostic instrumentation for #2678 (phantom 'you have todos' wakes
after clearing bash-task todos). logs subsystem/key/id/changed on
UpsertTodo, subsystem/key/all/count on ClearTodo, id/count on
MarkTodoDone, and a marker when the serve loop actually consumes a
todo_wake notification. no behavior change - RUST_LOG=debug only.
`container_run` logs every stdout line at INFO (target `nixos-container`)
as operation progress. For the read-only `list` op — called on the hot
path (dashboard rescan, forge + boot sweeps) — that stdout is the return
value, not progress, so every call logs the full ~28-line container roster
at INFO. hive-c0re calls it several times a second, flooding the host
journal.
Gate the stdout per-line INFO logging on the op not being `list`. Mutating
ops still log their progress; stderr is still logged for every op (errors
matter regardless). No behaviour change beyond log volume.
Removes entryAgents() and the two places it rendered agent names:
- rqe-agent code element in the entry header
- rqe-node-agent-label prefix per component chain when multi-component
The DAG structure split (WCC + fan-out) already communicates subgraph
boundaries visually via the separate .rqe-nodes rows; the agent-name
labels on top of that caused layout breakage (#2666) and duplicate
information. Closes#2666.
The live build log header still labels liveNode.agent (a single specific
node, not the whole DAG) — that .rqe-agent rule is kept.
Also removes the now-unused .rqe-node-agent-label CSS rule and its
comment.
Drop the API path from --org and the "Forgejo applies it to the initial
commit" mechanics from --default-branch (kept the user-facing caveat:
only takes effect with --auto-init). Swept the remaining verbs
(attachment-get, pr-reviews, attach, repo-add-collaborator, comment,
clone, pr-cmd router, …) — already user-relevant, no changes needed.
Continue trimming clap arg help to user-relevant info: drop the
token-bounded-paging rationale (list --page), the why-it's-required
note (lint no-reviewer), the `Forgejo Do:`/`force_merge` API internals
(pr-merge), and tighten diff --full. pr-status was already clean.
Drop implementation detail from the clap arg help (the `<verb> --help`
surface) — which API/endpoint, page-count math, persisted-vs-streamer
log-source internals, refspec shapes — keeping only what/when-to-use for
each flag. Module `//!` docs (dev-facing, not shown by `--help`) left
intact.
Drop implementation mechanics from the `--help` surface, keep only
what a user needs to run the command:
- global `-r`/`-f`/`--json`: remove token-file paths, the bash-helper
history, and the "already-JSON verbs ignore --json" aside.
- verb `about` strings (repo-create/repo-labels/repo-search/artifact-get/
ci-log/ci-rerun/pr-commits): drop which-API / "no REST endpoint" /
web-route / workflow-dispatch internals and cross-refs.
Per-verb arg help (verbs/*.rs) trimmed in follow-up commits.
Per mara (#2660): the client shouldn't re-derive a friendly name from
node kinds — that re-bakes the domain knowledge the raw-graph redesign
removed. The queue card / hivectl header now shows what the backend sends
(the DAG's source + the raw node kinds); the node chain conveys the
operation. Removes the DagView::label() helper (hivectl was its only
consumer after iris dropped the frontend map in #2660) and points
hivectl's header at d.source instead.
Two tests asserted the old behaviour where completed nodes/DAGs stayed
in the snapshot. Under the redesign, Done nodes are filtered off the wire
(a fully-Done DAG disappears; a Failed one lingers + is history-capped):
- failed_node test: the completed reconcile is Done → assert it's absent,
not Done-present (its run is already verified by the claim).
- history-eviction test: fail the nodes so the DAGs linger (Done ones
would vanish), then assert the grace window + per-template cap.
hivectl/dag_progress.rs was reading the now-removed DagView.kind/state.
Derive both from the node set via the shared DagView::rollup_state() +
DagView::label() helpers (added to hive-sh4re). Timestamps are DateTime<Utc>
now — elapsed calcs compare in unix seconds. Dropped the live step display
(step left the wire). Test fixtures updated to the slim shape.
The raw-graph wire drops the inline build_log_id; the client fetches a
node's captured output on demand. Two handlers resolve node id -> log-row
id (JobQueue::build_log_id_of, now keyed by the wire u64) then delegate to
the existing get_full / raw handlers: /api/build-log/<node_id> serves the
BuildLogFull JSON ({stdout, stderr} + header), /raw serves text/plain.
404 when the node has no linked log.
Reconcile with mara + argus's review on the frontend PR (#2660):
- DagView regains started_at/finished_at (DateTime<Utc>), computed
host-side as min/max over ALL subtree nodes (including the Done ones
filtered off the wire). The client can't derive these — the
earliest/only-started node is often Done and absent — so the backend
sets them, per mara's call.
- NodeView gains has_log: bool = build_log_id.is_some(), the precise
old 'node has a captured build log' guard so the dashboard only shows
a log link for nodes that actually produce one.
Now that Template is internal to hive-c0re (not a pub wire enum), the
dead-code lint flags the two variants nothing ever constructs. Removed
them + their as_str arms; terminal_hook's catch-all arm is unaffected.
