Node gains started_at/finished_at (chrono DateTime<Utc>, serialized
RFC 3339 on the wire per hive_sh4re::wire_time) plus error (String).
Graph::set_state self-stamps started_at on the first Running transition
and finished_at on the first terminal one, via an internal now_utc()
clock (keeps settle/complete signatures stable). Outcome::Failed(String)
carries the failure reason, set on the terminal transition.
hive-c0re complete_node builds Outcome::Failed(msg); its node_rt
side-table stays i64 for now (double-write) until #2637 reads the Node.
Toward #2637: the jobq graph becomes the source of truth for per-node
lifecycle so the queue can be sent to the client as-is.
Replace the in-tree scheduler with the domain-agnostic hive-jobq crate
(merged in #2615): parent-axis grouping + borrow/subtree-reservation
resource model + roll-up completion (State::Finishing).
Host adaptation:
- NodeSpec gains an explicit `parent` axis; templates declare grouping +
sibling ordering directly (deps order execution, parent groups a subtree
whose resource the descendants borrow).
- Rebuild is a nested two-root subtree: Prebuild (root, owns the build slot
for the whole subtree, lease-exempt) -> StopForUpdate (child, owns the
agent lease) -> Swap/PostSwap (children, borrow both); Reconcile is a
separate top-level root (AfterAny Prebuild) so it survives the cancel-
cascade of any failed step (recovery-start invariant) and converges to
the persisted `wanted` on a fresh lease. This is the multi-root
correction to the single-root-chain sketch: node0=root broke lease-
exemption (hoisting the lease onto Prebuild) and recovery-reconcile
(root failure cancels all children).
- Spawn / perm-change / power-ops (stop/start/restart) group-rooted the
same way; per-agent power-op subgraphs stay independent roots so a
multi-agent DAG runs them concurrently, each on its own lease.
- insert_group honours the explicit parent axis (no lease hoisting); the
DAG terminal node deps AfterAny on every group root and runs once the
whole op rolls up. Drop the old Graph::add_dep terminal wiring.
36/36 job_queue tests, full hive-c0re suite green, clippy --all-targets.
The boot provisioning pass (orgs, repos, teams, CI-runner token) all hits
the Forgejo API, but ensure_all only checked the container is *present*,
not that its HTTP is *listening*. A nixos-rebuild that restarts hive-forge
and hive-c0re together races: every ensure_* fired at a refused socket and
left a stale 'provisioning failed' banner that never cleared, since the
pass is one-shot. Poll GET /api/v1/version (unauthenticated) until it
answers, bounded at 1 minute, before provisioning; on timeout proceed
anyway so a genuinely-down forge still surfaces its real errors. Mirrors
the readiness-retry the gateway-nginx path already has.
sync_agent() now returns bool (false if any step fails). ensure_all()
collects the names of agents whose sync failed and raises a single
set_boot_warning with the aggregated list:
forge: per-agent sync failed for: alice, bob (see journal for per-step
detail)
The static_kind() leak is already used for per-org boot warnings in the
same file — the leak is bounded (one per hive-c0re boot, not per request)
so reusing it here is appropriate.
The rebuild call site in job_queue/exec.rs discards the bool return and
keeps its existing tracing::warn! lines, which is the right separation:
rebuilds are their own retry loop and don't need to post a persistent boot
warning.
Extends GET /api/journal/{name} to also accept the four hive infra
container names (hive-ci, hive-forge, hive-gateway, hive-matrix —
hive_priv_sock::InfraContainer is the allowlist), reusing the same
journalctl -M / hive-priv delegation path already used for agent
containers. Infra containers don't run the per-agent hive daemons, so
the unit filter is skipped for them — always the full machine journal.
Frontend: the AGENT tab's agent selector now lists infra containers
in a separate optgroup (sourced from /api/state's existing
infra_containers field), and disables the unit-filter select when one
is chosen.
Both warn!-and-forget sites in ensure_ci_runner_registered() now also
call set_boot_warning():
- fetch_registration_token failure → crit banner (forge unreachable or
API error; runner stays with stale/absent creds)
- hive-priv register_ci_runner failure → crit banner (EROFS or priv
socket error; runner token not written)
Both are only ever invoked from ensure_all() at hive-c0re startup (no
periodic retry), so set_boot_warning() is the right API: the banner
persists until the next c0re restart that re-runs the step, which is
exactly when a config/environment fix (e.g. the ReadWritePaths EROFS
fix from 64075107) would take effect.
Journal warn! lines are kept alongside the banner (belt-and-suspenders).
Remaining in scope for #2289: sync_agent() warn! sites (called from
both startup and rebuild paths — needs set_warning RAII or explicit
return value to enable later success to clear the banner; left for a
follow-up).
Forgejo returns team-already-exists as HTTP 422 ValidationFailed, not
409 Conflict, so the 409-only guard in ensure_operators_team missed it
and logged a spurious warning every boot (and skipped the settings
reconcile). Add a lenient discriminator that also treats a 422 whose
message says already-exists as benign.
Adds set_boot_warning() to stats/warnings.rs: a one-shot variant of
set_warning() for steps that run once at hive-c0re boot (no periodic
retry to observe a later success and clear the banner via WarningGuard
drop). It forgets the guard, leaking it for the life of the process —
the banner clears on the next hive-c0re restart, which is also when a
config/environment fix would take effect.
Wires every remaining tracing::warn!()-and-forget site in
forge::ensure_all() to also raise a boot warning: core user/token
provisioning, per-org seeding, the operators-team grant per org, the
core/meta repo, shared docs repo, knowledge repo (+ its local clone),
both avatar uploads, and the final nixos-container list failure.
Split the core_token.is_some() body of ensure_all() into a new
ensure_all_orgs_and_repos() helper to stay under clippy's
function-length limit.
Deliberately out of scope: sync_agent() (shared with rebuild_agent(),
different blast radius) and ci_runner::ensure_ci_runner_registered()
(its own retry logic) — matrix::ensure_all() got the same slice-by-
slice treatment.
Per mara's standing calibration (#2463): extraction > silencing for
too_many_lines. Splits the space + chat-room provisioning tail into its
own fn, bringing ensure_all back under the 100-line threshold without
an #[allow].
Fast-follow for #2289 — matrix::ensure_all() (admin user, per-agent
sync, hive Space + chat-room provisioning/invites) ran periodically
every 30 minutes but only ever warn!'d to the journal on failure, so a
persistent problem (missing invites, broken admin token, etc.) was
invisible to the operator.
ensure_all() now returns bool (aggregate ok/fail across every
sub-step) instead of (), and both call sites in main.rs feed that into
a debounced SweepHealth("matrix_ensure_all", warn, threshold=2) —
matches the existing knowledge_pull pattern. A lone bad sweep
self-heals silently; two consecutive failures raise a banner that
clears on the next clean sweep.
forge::ensure_all()'s remaining independent steps are still open —
that sweep only runs once at startup (no periodic loop), so the
debounced pattern doesn't map as directly; left for a follow-up.
Per operator review (#2547): remove the never-reached branch. render_flake
read HIVE_FORGE_URL inline with an unwrap_or_else(localhost:3000) fallback
that can't be hit — the daemon always sets HIVE_FORGE_URL (environment.nix)
— and the localhost URL wouldn't match the domain-scoped credential helper
anyway. Thread forge_base as an explicit param instead: the caller
sync_agents passes forge::forge_http_base(), tests pass it explicitly. This
removes the dead branch AND the hidden env-read, and drops the racy env
set_var from the forge-url test. render_flake is pure/param-driven again.
The persistent agent-<n> input url in the meta flake is now the forge
config repo, but the deploy + manual-rebuild re-lock must not follow it:
prepare_deploy and lock_update_for_rebuild override the input back to the
local applied mirror (git+file://<applied_dir>, main head) via
--override-input, which writes the applied rev into flake.lock.
This keeps the forge as the declared, reviewable source while:
- surviving forge unreachability (rebuilds fire on crash-restart and meta
bumps too, not just config PRs -- coupling every rebuild to forge would
be a regression),
- deploying exactly the reviewed head applied/<n>/main was fast-forwarded
to (no TOCTOU on a newer forge head merging mid-deploy),
- reusing verify_commit's local-override pattern so verify and deploy eval
the same source.
New applied_override_url helper + unit test.
