Two tests asserted the old behaviour where completed nodes/DAGs stayed
in the snapshot. Under the redesign, Done nodes are filtered off the wire
(a fully-Done DAG disappears; a Failed one lingers + is history-capped):
- failed_node test: the completed reconcile is Done → assert it's absent,
not Done-present (its run is already verified by the claim).
- history-eviction test: fail the nodes so the DAGs linger (Done ones
would vanish), then assert the grace window + per-template cap.
The raw-graph wire drops the inline build_log_id; the client fetches a
node's captured output on demand. Two handlers resolve node id -> log-row
id (JobQueue::build_log_id_of, now keyed by the wire u64) then delegate to
the existing get_full / raw handlers: /api/build-log/<node_id> serves the
BuildLogFull JSON ({stdout, stderr} + header), /raw serves text/plain.
404 when the node has no linked log.
Reconcile with mara + argus's review on the frontend PR (#2660):
- DagView regains started_at/finished_at (DateTime<Utc>), computed
host-side as min/max over ALL subtree nodes (including the Done ones
filtered off the wire). The client can't derive these — the
earliest/only-started node is often Done and absent — so the backend
sets them, per mara's call.
- NodeView gains has_log: bool = build_log_id.is_some(), the precise
old 'node has a captured build log' guard so the dashboard only shows
a log link for nodes that actually produce one.
Now that Template is internal to hive-c0re (not a pub wire enum), the
dead-code lint flags the two variants nothing ever constructs. Removed
them + their as_str arms; terminal_hook's catch-all arm is unaffected.
- server::await_dags: a DAG is settled when gone from the snapshot (fully
Done) or present with all nodes terminal; pending only with a non-terminal
node (DagView no longer carries a rolled-up state).
- DagView::rollup_state() added to hive-sh4re — the shared node-set roll-up
derivation every Rust consumer uses.
- JobQueue::build_log_id_of(node_id) — the node_id -> build_logs lookup the
query endpoint will use; tests assert log-id via it now.
- tests: derive roll-up state; drop the off-wire step/build_log_id wire asserts.
hive-c0re side of the raw-graph wire: dag_view now projects the slim
DagView, reading started_at/finished_at/error straight off the
hive_jobq Node (removes the node_rt double-write from #2645), excludes
Done nodes, and rides approval_id/inputs on the owning node. Template
moved into hive-c0re (model.rs) — no longer on the wire. Still WIP:
build-log endpoint + hivectl derive + compile fixes to follow.
Node gains started_at/finished_at (chrono DateTime<Utc>, serialized
RFC 3339 on the wire per hive_sh4re::wire_time) plus error (String).
Graph::set_state self-stamps started_at on the first Running transition
and finished_at on the first terminal one, via an internal now_utc()
clock (keeps settle/complete signatures stable). Outcome::Failed(String)
carries the failure reason, set on the terminal transition.
hive-c0re complete_node builds Outcome::Failed(msg); its node_rt
side-table stays i64 for now (double-write) until #2637 reads the Node.
Toward #2637: the jobq graph becomes the source of truth for per-node
lifecycle so the queue can be sent to the client as-is.
Replace the in-tree scheduler with the domain-agnostic hive-jobq crate
(merged in #2615): parent-axis grouping + borrow/subtree-reservation
resource model + roll-up completion (State::Finishing).
Host adaptation:
- NodeSpec gains an explicit `parent` axis; templates declare grouping +
sibling ordering directly (deps order execution, parent groups a subtree
whose resource the descendants borrow).
- Rebuild is a nested two-root subtree: Prebuild (root, owns the build slot
for the whole subtree, lease-exempt) -> StopForUpdate (child, owns the
agent lease) -> Swap/PostSwap (children, borrow both); Reconcile is a
separate top-level root (AfterAny Prebuild) so it survives the cancel-
cascade of any failed step (recovery-start invariant) and converges to
the persisted `wanted` on a fresh lease. This is the multi-root
correction to the single-root-chain sketch: node0=root broke lease-
exemption (hoisting the lease onto Prebuild) and recovery-reconcile
(root failure cancels all children).
- Spawn / perm-change / power-ops (stop/start/restart) group-rooted the
same way; per-agent power-op subgraphs stay independent roots so a
multi-agent DAG runs them concurrently, each on its own lease.
- insert_group honours the explicit parent axis (no lease hoisting); the
DAG terminal node deps AfterAny on every group root and runs once the
whole op rolls up. Drop the old Graph::add_dep terminal wiring.
36/36 job_queue tests, full hive-c0re suite green, clippy --all-targets.
The boot provisioning pass (orgs, repos, teams, CI-runner token) all hits
the Forgejo API, but ensure_all only checked the container is *present*,
not that its HTTP is *listening*. A nixos-rebuild that restarts hive-forge
and hive-c0re together races: every ensure_* fired at a refused socket and
left a stale 'provisioning failed' banner that never cleared, since the
pass is one-shot. Poll GET /api/v1/version (unauthenticated) until it
answers, bounded at 1 minute, before provisioning; on timeout proceed
anyway so a genuinely-down forge still surfaces its real errors. Mirrors
the readiness-retry the gateway-nginx path already has.
sync_agent() now returns bool (false if any step fails). ensure_all()
collects the names of agents whose sync failed and raises a single
set_boot_warning with the aggregated list:
forge: per-agent sync failed for: alice, bob (see journal for per-step
detail)
The static_kind() leak is already used for per-org boot warnings in the
same file — the leak is bounded (one per hive-c0re boot, not per request)
so reusing it here is appropriate.
The rebuild call site in job_queue/exec.rs discards the bool return and
keeps its existing tracing::warn! lines, which is the right separation:
rebuilds are their own retry loop and don't need to post a persistent boot
warning.
Extends GET /api/journal/{name} to also accept the four hive infra
container names (hive-ci, hive-forge, hive-gateway, hive-matrix —
hive_priv_sock::InfraContainer is the allowlist), reusing the same
journalctl -M / hive-priv delegation path already used for agent
containers. Infra containers don't run the per-agent hive daemons, so
the unit filter is skipped for them — always the full machine journal.
Frontend: the AGENT tab's agent selector now lists infra containers
in a separate optgroup (sourced from /api/state's existing
infra_containers field), and disables the unit-filter select when one
is chosen.
Both warn!-and-forget sites in ensure_ci_runner_registered() now also
call set_boot_warning():
- fetch_registration_token failure → crit banner (forge unreachable or
API error; runner stays with stale/absent creds)
- hive-priv register_ci_runner failure → crit banner (EROFS or priv
socket error; runner token not written)
Both are only ever invoked from ensure_all() at hive-c0re startup (no
periodic retry), so set_boot_warning() is the right API: the banner
persists until the next c0re restart that re-runs the step, which is
exactly when a config/environment fix (e.g. the ReadWritePaths EROFS
fix from 64075107) would take effect.
Journal warn! lines are kept alongside the banner (belt-and-suspenders).
Remaining in scope for #2289: sync_agent() warn! sites (called from
both startup and rebuild paths — needs set_warning RAII or explicit
return value to enable later success to clear the banner; left for a
follow-up).
Forgejo returns team-already-exists as HTTP 422 ValidationFailed, not
409 Conflict, so the 409-only guard in ensure_operators_team missed it
and logged a spurious warning every boot (and skipped the settings
reconcile). Add a lenient discriminator that also treats a 422 whose
message says already-exists as benign.
Adds set_boot_warning() to stats/warnings.rs: a one-shot variant of
set_warning() for steps that run once at hive-c0re boot (no periodic
retry to observe a later success and clear the banner via WarningGuard
drop). It forgets the guard, leaking it for the life of the process —
the banner clears on the next hive-c0re restart, which is also when a
config/environment fix would take effect.
Wires every remaining tracing::warn!()-and-forget site in
forge::ensure_all() to also raise a boot warning: core user/token
provisioning, per-org seeding, the operators-team grant per org, the
core/meta repo, shared docs repo, knowledge repo (+ its local clone),
both avatar uploads, and the final nixos-container list failure.
Split the core_token.is_some() body of ensure_all() into a new
ensure_all_orgs_and_repos() helper to stay under clippy's
function-length limit.
Deliberately out of scope: sync_agent() (shared with rebuild_agent(),
different blast radius) and ci_runner::ensure_ci_runner_registered()
(its own retry logic) — matrix::ensure_all() got the same slice-by-
slice treatment.
Per mara's standing calibration (#2463): extraction > silencing for
too_many_lines. Splits the space + chat-room provisioning tail into its
own fn, bringing ensure_all back under the 100-line threshold without
an #[allow].
