Fast-follow for #2289 — matrix::ensure_all() (admin user, per-agent
sync, hive Space + chat-room provisioning/invites) ran periodically
every 30 minutes but only ever warn!'d to the journal on failure, so a
persistent problem (missing invites, broken admin token, etc.) was
invisible to the operator.
ensure_all() now returns bool (aggregate ok/fail across every
sub-step) instead of (), and both call sites in main.rs feed that into
a debounced SweepHealth("matrix_ensure_all", warn, threshold=2) —
matches the existing knowledge_pull pattern. A lone bad sweep
self-heals silently; two consecutive failures raise a banner that
clears on the next clean sweep.
forge::ensure_all()'s remaining independent steps are still open —
that sweep only runs once at startup (no periodic loop), so the
debounced pattern doesn't map as directly; left for a follow-up.
Per operator review (#2547): remove the never-reached branch. render_flake
read HIVE_FORGE_URL inline with an unwrap_or_else(localhost:3000) fallback
that can't be hit — the daemon always sets HIVE_FORGE_URL (environment.nix)
— and the localhost URL wouldn't match the domain-scoped credential helper
anyway. Thread forge_base as an explicit param instead: the caller
sync_agents passes forge::forge_http_base(), tests pass it explicitly. This
removes the dead branch AND the hidden env-read, and drops the racy env
set_var from the forge-url test. render_flake is pure/param-driven again.
The persistent agent-<n> input url in the meta flake is now the forge
config repo, but the deploy + manual-rebuild re-lock must not follow it:
prepare_deploy and lock_update_for_rebuild override the input back to the
local applied mirror (git+file://<applied_dir>, main head) via
--override-input, which writes the applied rev into flake.lock.
This keeps the forge as the declared, reviewable source while:
- surviving forge unreachability (rebuilds fire on crash-restart and meta
bumps too, not just config PRs -- coupling every rebuild to forge would
be a regression),
- deploying exactly the reviewed head applied/<n>/main was fast-forwarded
to (no TOCTOU on a newer forge head merging mid-deploy),
- reusing verify_commit's local-override pattern so verify and deploy eval
the same source.
New applied_override_url helper + unit test.
Ran nix fmt on the whole change (not just the .nix): rustfmt wraps the long
ci_enabled() is_ok_and line that tripped the treefmt CI gate. Adds the # Errors
doc section on the pub priv_client::register_ci_runner per the HOW-TO rule.
Part B (c0re half). New forge/ci_runner.rs: validate the hive-ci runner's
persisted .runner against the forge (GET /admin/runners/{id}); if absent or
stale, mint a fresh registration token (GET /admin/runners/registration-token,
raw request against the local http forge — forgejo-api 0.11 doesn't wrap it)
and hand it to hive-priv's RegisterCiRunner to write the host env-file +
restart the runner. Gated on HYPERHIVE_FORGE_CI_ENABLED; best-effort (never
aborts the startup sweep). Called from ensure_all after the org/repo seeding.
The nix boot-path change (drop prefetch gating, add runner precond, set the
env var) lands next on this branch.
Part A of moving hive-ci runner registration off the boot-critical path.
Adds a root-side hive-priv op that writes the runner registration token to
the host env-file /run/hive-ci/runner-token (in-place, preserving the inode
the container bind-mounts) and restarts the in-container gitea-runner-hive
unit. The forge admin token stays in hive-c0re; only the registration token
reaches the host env-file the container mounts read-only. The c0re-side
caller (ensure_ci_runner_registered) + the nix boot-path change land next on
this branch.
Bumps the feature (major) versions that update cleanly without breaking the
build: indicatif 0.17->0.18, tower-http 0.6->0.7, hmac 0.12->0.13,
sha2 0.10->0.11. Only adaptation needed: import hmac's KeyInit trait in
webhook_secret (new_from_slice moved from Mac to KeyInit in hmac 0.13).
Held back (require dedicated code-change PRs, out of scope for a
non-breaking bump):
- reqwest 0.13: renames the rustls-tls feature and conflicts with
forgejo-api 0.11 + matrix-sdk 0.14 which pin reqwest 0.12.
- rusqlite 0.40: libsqlite3-sys 0.38 clashes with matrix-sdk-sqlite 0.14's
0.35 (single links=sqlite3) — coupled to the matrix-sdk bump.
- rmcp 2.2, matrix-sdk 0.18: major API rewrites across the MCP/matrix crates.
Extends the SweepHealth/warnings registry (already landed for
knowledge_pull) to two more background sweeps:
- gateway_nginx::reload_gateway_nginx: raises a warn-level banner
immediately on the first failed reload (routing changes silently
not taking effect is user-visible right now, so no debounce).
- forge::repos::ensure_config_repo: raises a crit-level banner
listing every agent whose config-repo branch protection is
currently unapplied (security-relevant — bypasses the deploy
pipeline), clearing agents out of the message as they recover.
Journal warn!/error! logging is left in place; the registry adds a
dashboard-visible signal on top. forge::ensure_all() and
matrix::ensure_all() sweeps are deliberately left for a fast-follow.
A per-agent lifecycle lease gates that agent's container globally across
concurrent DAGs, so it should be held for exactly as long as the agent's
work in a DAG is in flight, no longer. settle() previously freed every
lease a DAG held only at whole-DAG terminal, so a multi-agent DAG (a
hive-wide restart) kept agent A's container locked until B and C also
finished, blocking any other DAG wanting A.
Now free each agent's lease the moment its own subgraph within the DAG is
terminal (no live node still targets it), and drop that agent's dashboard
transient pill on the same edge via a new per-agent release channel. A
single-agent DAG is unaffected: its agent's subgraph goes terminal exactly
when the whole DAG does, so behaviour is identical.
hive-c0re already samples each agent container's cgroup load for the
dashboard (stats/container_stats.rs); this rides those gauges out to the
configured OTLP endpoint, reusing the existing services.hyperhive.otel
config (endpoint + auth header) — no new toggle.
- New stats/otel_metrics.rs: exports via the OpenTelemetry Rust SDK (same
crates as hive-metric) with the semconv container.* metric names +
container.name attribute so off-the-shelf OTel/Grafana dashboards work,
plus the hive agent label. container.cpu.time (counter, s, from cumulative
cpu.stat usage_usec), container.memory.usage, container.memory.usage.limit;
memory peak / on-disk storage / instantaneous cpu percent stay hyperhive.*
custom (no semconv equivalent). Observable instruments read a shared
snapshot an async task refreshes (gather() is async; SDK callbacks sync).
- container_stats: expose cpu_time_usec (cumulative) on ContainerResource.
- The OTLP auth header is loaded onto hive-c0re's own unit via systemd
LoadCredential and read from $CREDENTIALS_DIRECTORY/otel-headers.
- docs/observability.md documents the host-emitted semconv metrics.
Host-side export, so it covers containers even when their agent is idle.
Since the boot sweep (#2450) and meta-update cascade (#2476) became
single DAGs that grow subgraphs in-place, nothing constructs the old
fan-out anchors/parents anymore:
- NodeKind::Noop (the old boot_root grouping anchor) — no constructors.
- Template::StartupSweep / Source::StartupSweep (the old fan-out parent
template + cascade-child source) — replaced by Template::Boot and
Source::AutoUpdate/MetaUpdate respectively.
Drops the three variants + their as_str arms + the Noop executor arm, and
refreshes the stale fan-out/anchor doc comments (Boot/MetaUpdate/Source
docs, coordinator.md, dashboard.md). Frontend: the queue-kind glyph moves
from the dead startup_sweep to boot (which had none), and the dead
rqe-source-startup_sweep style is dropped.
No behaviour change — pure dead-variant removal.
Dashboard AGENT log tab sent unit=hive-ag3nt.service (the crate/dir
name) but the actual systemd unit is hive-agent.service, so every
fetch 400'd against the backend allow-list. Fixed the dropdown value
and, per the same issue's request, added the other per-agent daemons
(hive-mcp-http, hive-bash-daemon, hive-matrix-daemon) as selectable
units, plus hive-priv.service on the SYSTEM tab's host-daemon list.
Extended both backend allow-lists (post_journal / post_journal_host)
to match, and fixed a stale systemd.services.hive-ag3nt doc reference
in agent-hierarchy.md that had the same crate/unit-name confusion.
append_subgraph is the multi-node/multi-agent generalisation of the
single-node append_node, so the two in-DAG-growth channels collapse to
one: the Reconcile planner now emits its mechanical Start/Stop as a
single-node append_subgraph rooted on the reconcile node (stamping
claim.agent on the NodeSpec, which append_node inherited implicitly).
Removes NodeOutput.append_nodes + its scheduler drain loop and
JobQueue::append_node. No behaviour change — a channel unification.
With the meta-update cascade (#2476) and startup sweep (#2450) folded
into single DAGs that grow per-agent subgraphs via append_subgraph,
nothing links parent/child DAGs anymore — parent_id is dead.
hive-c0re: drop parent_id from Dag/DagSpec (+ the DagView copy); delete
append_children and cancel_children (no callers); simplify trim_history
(no more terminal-parent-with-live-children guard — a one-big-DAG is
terminal only when its whole graph settles); drop the rebuild() parent_id
param; QueueDag returns just the polled DAG (no fan-out children to
gather). hive-sh4re: drop the DagView.parent_id wire field.
frontend: a multi-step op is one DAG now, so renderRebuildQueue drops the
childrenOf/orphans cross-DAG grouping and renders each entry flat; its
per-agent subgraphs render as nodes within the one row (split by deps).
Removed the dead rqe-child style + isChild plumbing.
Docs + the child-DAG queue tests updated/removed to match.
MetaLock's non-sweep completion now grows one rebuild subgraph per
affected agent into the same DAG (append_subgraph), replacing the
fan-out-child-DAGs + cancel_children dance. Drops NodeOutput.fanout and
scheduler's fanout_specs. meta_update DAG carries Rebuilding transient so
each cascade agent gets crash-watch suppression at Swap (the property the
old child Rebuild DAGs held via their own transient); MetaLock head needs
no lease so the pseudo-agent gets no pill.
append_children/parent_id and child-DAG tests are intentionally left for
the #2453 capstone.
The boot is now ONE DAG, assembled inline in submit_boot_tree — no boot_root
Noop anchor, no per-agent child DAGs, no display-only parent_id grouping: a
sweep MetaLock root (only when something is stale) that grows one rebuild
subgraph per stale agent into the same DAG (via append_subgraph, previous
commit), plus one Reconcile root per drifted agent (independent — a boot
reconcile needs no lock bump).
- submit_boot_tree builds the DagSpec inline; removed the single-use
templates::{boot_root, startup_sweep} builders (inlined per the operator's
"don't force single-use shapes into templates.rs" steer).
- fanout_specs simplified to the meta-update cascade path only — the startup
sweep no longer fans out child DAGs, so its branch was dead.
- test: append_subgraph_roots_on_emitter_and_rebases_local_deps.
Vestigial after this (deliberately left as follow-ups, flagged in the PR):
NodeKind::Noop is now unconstructed (contained to hive-c0re, removable);
Template::StartupSweep is unconstructed but a hive-sh4re wire type
(frontend-coordinated removal).
First half of making the startup sweep one DAG. Adds the runtime
subgraph-append machinery and switches the sweep MetaLock from fanning out
child Rebuild DAGs to growing one rebuild subgraph per stale agent into the
same DAG:
- JobQueue::append_subgraph(dag_id, nodes, dep_on) — the multi-node,
multi-agent generalisation of append_node: rebases a subgraph's local deps
onto the DAG's id space and roots it on the emitting node.
- NodeOutput.append_subgraph: Vec<Vec<NodeSpec>> — the executor→scheduler
channel for it; scheduler drains it before completing the emitting node
(same ordering as append_nodes).
- run_meta_lock sweep branch returns the stale agents' rebuild_nodes
subgraphs via append_subgraph instead of fanout.
Follow-up commit collapses submit_boot_tree (drop boot_root + per-agent
reconcile child DAGs) so the whole boot is one DAG built inline.
Restart now offers the same graceful-vs-hard choice stop already has:
single-agent menu item and the bulk-select action bar both grow a
'restart gracefully' checkbox that routes through the existing
submit::graceful_restart DAG (signal -> drain -> stop -> reconcile)
instead of a hard restart. Backend gains a ?graceful=true query param
on POST /api/restart/<name>, mirroring post_kill's shape (renamed
KillParams -> GracefulParams since it's now shared).
Hive-wide `stop` / `start` / `restart` emit ONE DAG with a per-agent
subgraph each (concurrent on their own leases) instead of N DAGs — and each
subgraph is now built dynamically from the agent's live running state rather
than a fixed template shape:
- online agent: the full stop→reconcile (restart: stop-for-update→reconcile)
chain; `graceful` prepends signal→drain.
- offline agent: just `SetWanted → Reconcile` (nothing to quiesce/stop; a
restart of a down agent is really a start).
The head `SetWanted` (intent) and tail `Reconcile` (convergence guarantee)
are always present; only the mechanical `Signal`/`Drain`/`StopForUpdate`
nodes are state-conditional. Keeping `Reconcile` in every shape closes the
TOCTOU window — a race-up between the `is_running` read and node exec is
still converged in-DAG (with `StopForUpdate`-noop as the backstop) — with no
reliance on an external reconcile sweep.
The state-aware assembly needs an async `is_running` read, so it moves out
of the pure/sync `templates.rs` into `submit.rs`, layered as pure
`*_chain(running)` → pure `*_spec(targets)` (the unit-test seam) → async
`*_many` (reads live state + submits). `templates.rs` keeps only the shared
pure primitives (`node`/`after_ok`/`rebuild_nodes`).
Callers await the now-async submit fns (server, dashboard, socket_server).
Tests exercise both the online and offline shapes via the pure `*_spec`
seam. docs/coordinator.md shapes updated.
- multi_agent_restart_is_one_dag_with_concurrent_per_agent_subgraphs:
two agents restart in one DAG; both SetWanted heads are roots, each
acquiring its own agent's lease concurrently. Existing single-agent
shape/lease tests updated for the templates::restart(&[..], graceful)
signature.
- coordinator.md + templates.rs module doc: restart takes an agent list;
a hive-wide restart is one DAG with N per-agent subgraphs.
A hive-wide restart was N separate single-agent DAGs (one submit::restart
per agent). Now that agent is per-node (#2445), make it ONE DAG with a
per-agent restart subgraph each.
- templates::restart takes an agent list: each agent gets an independent
subgraph (a head SetWanted(Up) root, then its restart chain), so the N
subgraphs run concurrently on their own leases. One agent = the ordinary
single-agent restart; unifies the old restart + graceful_restart fns.
- submit::restart / graceful_restart stay as single-agent wrappers over
the new submit::restart_many(agents, graceful).
- server.rs handle_restart_all + handle_restart_scoped submit one
restart_many call instead of looping per agent. Infra containers
unchanged (no lease/DAG, synchronous).
Scope: restart + restart-all only. Broad stop+start is the same pattern
(stop/start templates take agent lists) — a follow-up increment.
The durable 'wanted' power intent was written by submit::{start,stop,
restart,graceful_restart,graceful_stop} as a synchronous pre-submit side
effect, then read by the DAG's tail Reconcile. That's not crash-safe
(a crash between the write and the enqueue loses it) and, with agent now
per-node, can't be per-agent in a DAG that spans agents.
Move it into the DAG as a head SetWanted node:
- NodeKind::SetWanted { up } + run_set_wanted executor (fails the node on
a write error, unlike the old warn-and-continue, so a stale intent
never reaches Reconcile).
- LEASE-NEEDING, not lease-exempt: it takes the agent lease so a power-op
DAG's intent-write + reconcile is atomic per-agent. If it were exempt,
two racing ops (restart vs stop) would run both intent-writes up front
and clobber each other before either reconciled — defeating the point
of moving the write into the DAG. (In stale_start the lease is thus held
across the head Prebuild, but that's a no-op there: the agent is down so
prebuild is skipped.)
- templates: explicit SetWanted node 0 on restart/graceful_restart/
graceful_stop, plus dedicated start/stop templates (SetWanted -> Reconcile)
and stale_start (SetWanted(Up) -> rebuild subgraph, reusing rebuild_nodes).
No compose helper / rebuild variant. reconcile_only is now boot-only.
- submit.rs: drop the set_wanted side effect; the stale-rev shape decision
(start vs stale_start) stays submit-side.
All 33 job_queue tests pass (shape/lease tests updated for the head node).
DagView no longer has a DAG-level agent, so consumers derive it from the
per-node agents:
- hivectl dag_progress.rs: dag_agents(d) helper (distinct node agents,
comma-joined) in place of d.agent.
- dashboard builds.js: entryAgents(entry) helper likewise for the
rebuild-queue card + live-log header + cancel confirms.
- docs/coordinator.md: lease prose (node-agent-keyed, global per agent),
wire shape (NodeView.agent, no DagView.agent), and dropped the removed
dedup section.
Agent was a single field on Dag/DagSpec, making a DAG structurally
one-agent — a multi-agent op could only ever be N separate DAGs. Move it
onto Node/NodeSpec (and the NodeView wire type), drop it from Dag/DagSpec
(and DagView): a DAG can now span agents.
- lifecycle lease keys on the node's agent, still globally exclusive per
agent across all DAGs (Inner.leases unchanged in shape). A DAG holds one
lease per distinct agent it touches; settle() frees each at DAG-terminal
(per-agent-subgraph early release is a follow-up, only observable with
multi-agent DAGs).
- transient guard keyed (dag_id, agent); cancel-revert + Rebuilt events
walk TerminalDag.agents.
- submit-time dedup removed (a multi-agent DAG has no single agent to key
on); every submit enqueues a fresh DAG. Whether dedup needs reintroducing
is tracked in a follow-up sub-issue.
- templates gain a node(agent, kind, deps) helper stamping the agent onto
every node; meta templates stamp "hyperhive".
Templates stay single-agent in this PR — behaviour is unchanged, only the
representation + wire shape. Multi-agent DAG emission (restart/restart-all/
broad stop+start as one DAG) and the SetWanted-as-a-node change are
follow-ups off #2439.
argus caught it: the function-level /// comment still described the
old submit-await-submit graceful approach after the code moved to
one atomic GracefulRestart DAG.
mara's review on #2436: no submit-await-submit composition, even
server-side. Adds Template::GracefulRestart (Signal -> Drain ->
StopForUpdate -> Reconcile, wanted=Up) mirroring how Restart already
does StopForUpdate -> Reconcile, plus submit::graceful_restart and
templates::graceful_restart. handle_restart_scoped now submits exactly
one DAG per agent up front for both the graceful and non-graceful
case -- no await_dags in the loop anymore.
hivectl restart --agent NAME previously composed stop() then start() as
two separate client-side daemon calls glued by CLI-process control flow
— not one DAG, and a dropped hivectl connection mid-restart (ssh drop,
Ctrl-C) left the agent stopped with no automatic follow-up. mara flagged
this as the first target for the 'dagify hivectl commands' issue.
New HostRequest::RestartScoped{scope, graceful} handles it server-side:
each targeted agent now rides exactly one atomic Restart-template DAG
(same one hivectl agents restart / restart-all already use) in the
common non-graceful case. --graceful has no single-DAG template yet, so
it submits the graceful-stop DAGs, awaits them server-side, then submits
the start DAGs — still one daemon call end to end, just not yet a single
DAG (noted as a follow-up). Infra containers restart synchronously as
before (no lease/DAG concept for them).
CLI-side restart() now just makes the one call + waits, same output
shape as before via render_lifecycle.
