Hive-wide `stop` / `start` / `restart` emit ONE DAG with a per-agent
subgraph each (concurrent on their own leases) instead of N DAGs — and each
subgraph is now built dynamically from the agent's live running state rather
than a fixed template shape:
- online agent: the full stop→reconcile (restart: stop-for-update→reconcile)
chain; `graceful` prepends signal→drain.
- offline agent: just `SetWanted → Reconcile` (nothing to quiesce/stop; a
restart of a down agent is really a start).
The head `SetWanted` (intent) and tail `Reconcile` (convergence guarantee)
are always present; only the mechanical `Signal`/`Drain`/`StopForUpdate`
nodes are state-conditional. Keeping `Reconcile` in every shape closes the
TOCTOU window — a race-up between the `is_running` read and node exec is
still converged in-DAG (with `StopForUpdate`-noop as the backstop) — with no
reliance on an external reconcile sweep.
The state-aware assembly needs an async `is_running` read, so it moves out
of the pure/sync `templates.rs` into `submit.rs`, layered as pure
`*_chain(running)` → pure `*_spec(targets)` (the unit-test seam) → async
`*_many` (reads live state + submits). `templates.rs` keeps only the shared
pure primitives (`node`/`after_ok`/`rebuild_nodes`).
Callers await the now-async submit fns (server, dashboard, socket_server).
Tests exercise both the online and offline shapes via the pure `*_spec`
seam. docs/coordinator.md shapes updated.
- multi_agent_restart_is_one_dag_with_concurrent_per_agent_subgraphs:
two agents restart in one DAG; both SetWanted heads are roots, each
acquiring its own agent's lease concurrently. Existing single-agent
shape/lease tests updated for the templates::restart(&[..], graceful)
signature.
- coordinator.md + templates.rs module doc: restart takes an agent list;
a hive-wide restart is one DAG with N per-agent subgraphs.
A hive-wide restart was N separate single-agent DAGs (one submit::restart
per agent). Now that agent is per-node (#2445), make it ONE DAG with a
per-agent restart subgraph each.
- templates::restart takes an agent list: each agent gets an independent
subgraph (a head SetWanted(Up) root, then its restart chain), so the N
subgraphs run concurrently on their own leases. One agent = the ordinary
single-agent restart; unifies the old restart + graceful_restart fns.
- submit::restart / graceful_restart stay as single-agent wrappers over
the new submit::restart_many(agents, graceful).
- server.rs handle_restart_all + handle_restart_scoped submit one
restart_many call instead of looping per agent. Infra containers
unchanged (no lease/DAG, synchronous).
Scope: restart + restart-all only. Broad stop+start is the same pattern
(stop/start templates take agent lists) — a follow-up increment.
- builds.js NODE_KIND_LABEL: 'set wanted' (and the previously-missing
'noop') so the new head node renders with a label, not the raw kind.
- docs/coordinator.md + templates.rs module doc: the power-op DAG shapes
now show the head SetWanted node; SetWanted added to the lease-needing
list with the atomicity rationale.
The durable 'wanted' power intent was written by submit::{start,stop,
restart,graceful_restart,graceful_stop} as a synchronous pre-submit side
effect, then read by the DAG's tail Reconcile. That's not crash-safe
(a crash between the write and the enqueue loses it) and, with agent now
per-node, can't be per-agent in a DAG that spans agents.
Move it into the DAG as a head SetWanted node:
- NodeKind::SetWanted { up } + run_set_wanted executor (fails the node on
a write error, unlike the old warn-and-continue, so a stale intent
never reaches Reconcile).
- LEASE-NEEDING, not lease-exempt: it takes the agent lease so a power-op
DAG's intent-write + reconcile is atomic per-agent. If it were exempt,
two racing ops (restart vs stop) would run both intent-writes up front
and clobber each other before either reconciled — defeating the point
of moving the write into the DAG. (In stale_start the lease is thus held
across the head Prebuild, but that's a no-op there: the agent is down so
prebuild is skipped.)
- templates: explicit SetWanted node 0 on restart/graceful_restart/
graceful_stop, plus dedicated start/stop templates (SetWanted -> Reconcile)
and stale_start (SetWanted(Up) -> rebuild subgraph, reusing rebuild_nodes).
No compose helper / rebuild variant. reconcile_only is now boot-only.
- submit.rs: drop the set_wanted side effect; the stale-rev shape decision
(start vs stale_start) stays submit-side.
All 33 job_queue tests pass (shape/lease tests updated for the head node).
Review (iris/argus): the dashboard rebuild live-log header labels one
specific node's log stream (liveNode, keyed by its build_log_id), so it
should show that node's own agent — entryAgents(running) listed every
agent in the DAG, which would mislabel a single agent's log once DAGs
span multiple. The other entryAgents() sites (row label, cancel-confirm,
fingerprint) are correct whole-DAG summaries and unchanged.
DagView no longer has a DAG-level agent, so consumers derive it from the
per-node agents:
- hivectl dag_progress.rs: dag_agents(d) helper (distinct node agents,
comma-joined) in place of d.agent.
- dashboard builds.js: entryAgents(entry) helper likewise for the
rebuild-queue card + live-log header + cancel confirms.
- docs/coordinator.md: lease prose (node-agent-keyed, global per agent),
wire shape (NodeView.agent, no DagView.agent), and dropped the removed
dedup section.
Agent was a single field on Dag/DagSpec, making a DAG structurally
one-agent — a multi-agent op could only ever be N separate DAGs. Move it
onto Node/NodeSpec (and the NodeView wire type), drop it from Dag/DagSpec
(and DagView): a DAG can now span agents.
- lifecycle lease keys on the node's agent, still globally exclusive per
agent across all DAGs (Inner.leases unchanged in shape). A DAG holds one
lease per distinct agent it touches; settle() frees each at DAG-terminal
(per-agent-subgraph early release is a follow-up, only observable with
multi-agent DAGs).
- transient guard keyed (dag_id, agent); cancel-revert + Rebuilt events
walk TerminalDag.agents.
- submit-time dedup removed (a multi-agent DAG has no single agent to key
on); every submit enqueues a fresh DAG. Whether dedup needs reintroducing
is tracked in a follow-up sub-issue.
- templates gain a node(agent, kind, deps) helper stamping the agent onto
every node; meta templates stamp "hyperhive".
Templates stay single-agent in this PR — behaviour is unchanged, only the
representation + wire shape. Multi-agent DAG emission (restart/restart-all/
broad stop+start as one DAG) and the SetWanted-as-a-node change are
follow-ups off #2439.
argus caught it: the function-level /// comment still described the
old submit-await-submit graceful approach after the code moved to
one atomic GracefulRestart DAG.
mara's review on #2436: no submit-await-submit composition, even
server-side. Adds Template::GracefulRestart (Signal -> Drain ->
StopForUpdate -> Reconcile, wanted=Up) mirroring how Restart already
does StopForUpdate -> Reconcile, plus submit::graceful_restart and
templates::graceful_restart. handle_restart_scoped now submits exactly
one DAG per agent up front for both the graceful and non-graceful
case -- no await_dags in the loop anymore.
hivectl restart --agent NAME previously composed stop() then start() as
two separate client-side daemon calls glued by CLI-process control flow
— not one DAG, and a dropped hivectl connection mid-restart (ssh drop,
Ctrl-C) left the agent stopped with no automatic follow-up. mara flagged
this as the first target for the 'dagify hivectl commands' issue.
New HostRequest::RestartScoped{scope, graceful} handles it server-side:
each targeted agent now rides exactly one atomic Restart-template DAG
(same one hivectl agents restart / restart-all already use) in the
common non-graceful case. --graceful has no single-DAG template yet, so
it submits the graceful-stop DAGs, awaits them server-side, then submits
the start DAGs — still one daemon call end to end, just not yet a single
DAG (noted as a follow-up). Infra containers restart synchronously as
before (no lease/DAG concept for them).
CLI-side restart() now just makes the one call + waits, same output
shape as before via render_lifecycle.
Split the priv-socket wire types (PrivRequest/PrivResponse/PrivEvent and
friends) out of hive-sh4re into their own hive-priv-sock crate, mirroring
the existing hive-host-sock split. hive-priv — the root-privileged
helper — now depends on just this narrow protocol crate instead of the
much larger daemon-shared crate, shrinking its dependency surface and
making the privsep boundary easier to audit. No server/client
implementation lives here, only the wire contract; hive-c0re still
depends on hive-sh4re directly for everything else.
argus flagged (approving) that an unvalidated label could build a path
outside the state dir; mara called it out as a usability issue in its
own right, not just a low-risk security nit — a typo'd label should
give a precise 'not a valid label' error, not a confusing file-not-found
or an unexpected traversal.
Reject anything outside the plain lowercase+digits+hyphens charset
dashboard/extra_forges.rs already enforces on write, before touching
the filesystem at all.
Targets a dashboard-provisioned external forge account (FORGES tab)
instead of the internal forge: resolves `forge-<label>-token` for the
token and `forge-<label>.json`'s base_url for the URL, the same two
files dashboard/extra_forges.rs writes, instead of
HIVE_FORGE_URL/forge-token. Falls back to today's behavior when unset.
Orthogonal to -r/--repo.
An unknown label gives a clear error listing the labels actually found
in the state dir instead of a raw file-not-found. The base_url JSON key
is read via a typed sidecar struct pinned to what extra_forges.rs
writes, so the read side can't silently drift from the write side.
- regenerate docs/tools/hivectl-cli.md for the new `subvol snapshot send` verb
- close the TOCTOU on the no-overwrite guard: File::options().create_new(true)
(O_CREAT|O_EXCL) instead of exists()-then-create, so the guarantee is
atomic against a concurrent request racing the same dest filename
- warn (not silently swallow) if cleaning up a partial export after a
failed btrfs send itself fails, so a stuck garbage file masquerading
as a completed export is visible in the log
SendAgentSnapshotToFile priv op: btrfs send [-p <parent>] <snapshot> to a
file under MIGRATE_STAGING_ROOT. Standalone-useful as a point-in-time
snapshot export/backup today; the cross-hive ssh-piped leg (auth/trust
design posted on #1763, awaiting mara/damocles steer) is a later,
separate piece this doesn't block on.
- hive-sh4re: PrivRequest::SendAgentSnapshotToFile + MIGRATE_STAGING_ROOT
- hive-priv: validates names, refuses to overwrite an existing export,
cleans up a partial file on btrfs send failure
- hive-c0re: priv_client::send_agent_snapshot_to_file
- hivectl: `hivectl subvol snapshot send <agent> <label> [--parent <label>] --dest <file>`
nix/modules/ was restructured into nix/host-modules/ + nix/agent-modules/
before the extra-forges branch merged; this doc-only fix from PR #2422
(e53f70ac) landed after the merge went through and got dropped. Reapplying
directly against main.
Per mara: "i would have even disallowed ., we are making up the rules
here lets go strict". validate_credential_name now restricts to
[A-Za-z0-9_-] (no dot at all) instead of [A-Za-z0-9_.-] + a separate
".." substring check — simpler rule, and there's no legitimate need
for a dot in either a systemd credential id or a hive- prefixed
snapshot label. Matching hivectl client-side check + wire-proto doc
comments updated.
Per mara's review: validate_credential_name allowed any [A-Za-z0-9_.-]
byte sequence, which permits a literal ".." substring. Not currently
exploitable (snapshot_path() embeds the label inside a single
format!()'d path component with no "/" in the allowed charset, so
there's no directory to traverse into), but it's a landmine for any
future caller that builds a path via PathBuf::from(name) directly
instead of the current string-embedding. Reject ".." outright in the
shared validator, plus a matching client-side check in hivectl for
fail-fast UX (hive-priv's copy is still the authoritative one).
Per mara's PR review:
- snapshot label is now mandatory (was optional w/ timestamp default)
and must start with "hive-" — hive-priv enforces this as an
allow-list on top of the existing credential-name charset check, so
only hivectl-issued labels can reach the btrfs shellout.
- nest under `subvol snapshot create`/`subvol snapshot delete`
instead of othering delete as a separate top-level `delete-snapshot`
verb.
Per argus's review:
- regenerate docs/tools/hivectl-cli.md (hivectl markdown-docs) to
include the new subcommands — CI's hivectl-docs-fresh check compares
this file against generated output.
Adds the first missing piece from #2391's migration-gaps list: a
read-only btrfs snapshot priv op so hivectl migrate can freeze a
consistent point-in-time copy of an agent's state subvolume for
btrfs send, without stopping the live agent.
- PrivRequest::SnapshotAgentSubvolume / DeleteAgentSnapshot (hive-sh4re)
- hive-priv handlers: btrfs subvolume snapshot -r / delete, sibling
dot-prefixed path (<AGENT_STATE_ROOT>/.<agent>.snapshot.<label>)
- hive-c0re::priv_client wrappers
- hivectl subvol snapshot / delete-snapshot verbs (no agent stop needed
— btrfs snapshots are atomic against a live subvolume)
Does not yet wire actual btrfs send/receive or the hivectl migrate
verb — those stay tracked on #2391 as separate follow-up pieces.
extra.rs (external-forge account minting) was removed in the
dashboard-provisioned redesign; TOKEN_SCOPES only applies to tokens
hive-c0re mints itself on the internal forge. External forge tokens
are pasted by the operator verbatim, so we never mint them and don't
need to know their scope.
Per mara's feedback on PR #2407 ("better: you can also provide url in
dashboard, same as with matrix, no host config"), drops
services.hyperhive.extraForges and the admin-API mint/revoke flow
entirely. The operator now creates a token on the external forge
themselves and pastes a label + base URL + access token into the
dashboard's FORGES tab, the same shape as the GitHub PAT flow plus the
base-URL field from the matrix extra-account flow. hive-c0re only ever
writes/deletes two local files per account (forge-<label>-token,
forge-<label>.json sidecar for the URL) via hive-priv — no remote
account creation, no admin token, no revoke-on-the-remote-side, no nix
config to enumerate.
- nix/host-modules/hive-forge/default.nix: removed the extraForges
option, its label-format assertion, and the HYPERHIVE_EXTRA_FORGES
env forwarding.
- hive-c0re/src/forge/extra.rs: deleted (REST admin-API provisioning,
no longer needed).
- hive-c0re/src/dashboard/extra_forges.rs: GET /api/extra-forges?
agent= lists an agent's stored forges by scanning its state dir
(mirrors matrix_accounts.rs's filename-scan listing), POST
/api/extra-forge-account (agent/label/base_url/token/
action=add|remove) stores or removes an account.
- hive-sh4re/priv_proto.rs + hive-priv/main.rs: new
WriteAgentExtraForgeAccount/DeleteAgentExtraForgeAccount priv
requests (adds base_url, writes/deletes a JSON sidecar alongside the
token).
- hive-c0re/src/priv_client.rs: matching wrapper functions.
- frontend/packages/dashboard/src/credentials.{html,js}: FORGES tab is
a per-agent list + add-account paste form (label/base_url/token), no
grant/revoke-from-catalog UI.
- docs/web-ui/dashboard.md: FORGES tab section rewritten.
Supersedes the design in PR #2407 (already approved+green on the old
admin-API model) — opening as a fresh PR against the same issues
rather than force-pushing over the approved one.
gitea-runner registration (hive-ci-prefetch, host-side) sits on the
boot-critical path -- nspawn readiness is gated on the runner
registering, itself a forge + core-token round trip that can wait up
to 60s for the core token. The default ~60s TimeoutStartSec can trip
mid-register (especially right after a .runner purge, since every
boot re-registers from scratch), killing the half-started container
and triggering a restart loop until the token/forge settle. Bump to
180s so one register attempt has room to finish.
