Destroy was a straight-line async fn with no queue node behind it, so
nothing in the graph could answer "is this container going down on
purpose?". That gap is why an imperative crash-watch suppression guard
existed: an RAII handle held for the operation's duration, a second way
to say what every other lifecycle op already says through its node.
Reuse the existing Stop node rather than teaching a new node to stop
things:
Stop -> DestroyContainer -> (PurgeState) -> DestroyBookkeeping
Stop already declares takes_container_down honestly, so the suppression
is now derived from the graph like every other op's. It also turns the
precondition into an edge: DestroyContainer runs only under a completed
Stop, so it operates on an already-stopped container and carries
takes_container_down = false permanently. A container still alive at
that point is a real bug and stays loud instead of being absorbed by a
flag -- which matters because a wrong true silently swallows a crash
while a wrong false only costs a spurious event.
Removes suppress_crash_watch, CrashWatchSuppression, crash_suppressed,
crash_watch_suppressed and NO_NODE_LABEL. The migration call sites went
with the obsolete startup migrations, so destroy was the last caller and
intent now has exactly one home.
destroy() becomes a submit-and-return, matching every sibling endpoint
(rebuild, kill, restart, start, pause, resume) -- it was the only
lifecycle op that awaited its work. The container rescan moves into the
bookkeeping tail, so ContainerRemoved now arrives after the 200 rather
than before it.
Also drops an orphaned doc-comment in coordinator.rs: two stacked blocks
where only the second described crash_suppressed, the first documenting
a field that no longer exists. Removing the field would have re-pointed
it at recent_transient.
The gateway's nginx + dnsmasq no longer run in their own nspawn container.
`nix/host-modules/hive-gateway/default.nix` loses the
`containers.hive-gateway` wrapper and everything that existed only to punch
holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts,
its own `stateVersion`, `networking.firewall.enable = false`,
`networking.resolvconf.enable = false`, and the `hive-gateway-resolv`
path+service pair. 465 -> 303 lines.
The container never bought isolation here. It shared the host netns by
necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge —
so each of those settings was undoing a boundary the gateway could not
afford in the first place.
Four things made it more than a deletion, none of them visible in the nix
diff:
- The self-signed cert service also imports the hive CA leaf, so removing it
with the container would have left nginx naming a missing cert file, which
it refuses to load at all.
- The nginx reload is a hive-priv verb. It still needs root, but no longer
for the reason its doc gave, and `--machine=` was both transport and
scope — so the unit name is now hard-coded in the helper as the
containment.
- The lifecycle verb named a container that stops existing.
- `journalctl -M hive-gateway` had no machine to enter.
Per the operator's ruling, the operator verb keeps working and agents lose
it. `InfraContainer` answered three questions that used to share an answer;
it now splits into `name()` (identity), `target()` (Container vs HostUnit),
`service_unit()` (the systemd unit), and `agent_restartable()`, which the
MCP restart path checks before the capability so the refusal cannot read as
"ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the
requests that name a container as a string — while `FromStr` still accepts
it, because that answers what a name is, not who may act on it. The
dashboard's gateway journal reads host journald filtered to `nginx.service`.
Prose was corrected where it only named a location, and re-argued where the
container was doing security work: a `0666` per-agent socket was safe
because only the gateway container had the directory bind-mounted. There is
no mount now, so the directory permissions are the whole of the access
control — the constraint holds, its mechanism doesn't.
Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710
tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run
in both TLS shapes at this commit: every delta in the rendered
virtualHosts is one of the three intended path moves, dnsmasq settings are
byte-identical, and the absence probe flips true -> false with bindMounts
emptied.
is_running collapsed every non-active state into false, so a container that
exhausted its bounded restarts read as plain "down" -- indistinguishable
from one an operator stopped deliberately. Bounding the restarts made that
gap sharper: a slow-failing agent used to grind on visibly, now it can stop
quietly.
Adds UnitState + unit_state() beside is_running rather than widening it.
is_running has ~8 call sites and nearly all are reconcile/power logic asking
"is it up? if not, start it" -- a question with two answers. Only the view
builder needs more, and it gets both facts from one systemctl call, since
is-active prints the state when not passed --quiet.
Surfaces as a flat failed flag on ContainerView and AgentStatusRow, matching
the shape those types already document: independent, orthogonally-observed
facts rather than a state machine. serde(default) keeps it order-independent
with the frontend half.
No behaviour change: nothing acts on the flag, per the ruling.
Closes the #3110 split — lib.rs is now just the crate doc comment and
the pub mod list.
journal.rs's new doc comment fixes a pre-existing bug: the old
JournalPriority doc text in lib.rs was actually half Capability's doc
(a leftover from an earlier reorder that moved the code but not the
comment above it).
One attrset describing every hive in the swarm including this one,
identical on every host, with hiveName selecting which entry is us.
"My peers" is derived (swarm.peerHives) rather than declared.
Every field in the old per-host peer list was intrinsic to the hive it
described, never to the pair -- so the list was a directory each host
kept its own copy of. Beyond the deduplication it removes a bug class:
two hosts could hold different endpoints for the same third hive with
nothing to detect the disagreement.
Drops the per-hive caCert. Trust inside a swarm derives from the swarm
root, which every hive chains to. What that genuinely removes is
trusting a hive whose root this swarm does not own -- a cross-swarm
problem that wants a mechanism of its own, not a field that happened to
work.
The matrix container's certificateFiles block goes with it and could
NOT be migrated: that list is read at build time and the swarm root is
a runtime file (its key must never enter the store), so there is no
build-time name to put there. caCert being a nix path was precisely
what made it the build-time distribution channel. Agents are unaffected
-- hive-tls folds the root into the hive trust bundle and the meta
renderer embeds that one file. Tracked separately.
Migration is an assertion plus warnings, not a rename: hives is peers
union {self}, and the set gains a member no existing config has written
down. A rename migrates a name and a default can re-root a meaning;
neither can conjure a new member. The warning explains, the self-entry
assertion stops the build.
Every prose and error-message reference to the moved options, including
two in hive-c0re/src/matrix.rs that omitted the services. prefix and so
read as the per-agent option. An error that names an option the operator
cannot find is worse than no hint.
The last of the DAG-container removal. `tests.rs` navigated by the id
`submit` returned, so removing the container removed the tests' way of
finding what they inserted; they name the roots they assert on now, which
is the same handle production uses.
Three findings the port surfaced, each a behaviour change rather than a
test fix:
- Cancelling a rebuild's head no longer drops the job. `Reconcile`'s edge
accepts a skipped brace, and a cancel-cascade skips rather than cancels,
so the tail stays claimable. Dropping a job means cancelling every id the
insert returned.
- A directly-cancelled group root reads terminal while a spared tail still
runs; the cancel used to land on a node above it, which rolled up
Finishing instead.
- "One DAG per hive-wide op" is not expressible without a container. The
three tests asserting it now assert that every named root is top-level,
which is what makes the per-agent subgraphs concurrent.
Deletes two tests: one asserted only that two containers get distinct ids,
the other re-ran an existing case under a second name.
`Source`, `insert_group` and the stop path's `reason` string went dead with
the container and are removed with it.
`server.rs`'s five sites move to `power::{stop,start,restart}_many` and direct
template inserts. `submit_single` routes through the `*_many` builders with a
one-element slice rather than keeping a parallel single-target shape.
`templates::rebuild` and `templates::reparent` now return the guids of the roots
they declare, so a caller that has to wait on them can name them; previously
only the void-returning form existed and every caller got an empty id list.
Two comments corrected while converting, both contradicted by the code they sit
above:
* `templates::rebuild` said its tail is edged onto "(MetaSync, Prebuild,
Reconcile)" and that "Prebuild's roll-up carries the subtree" — the brace has
been the middle root since the AgentWindow change.
* the restart handler described the per-agent shape as starting with SetWanted,
while `restart_chain`'s own doc says a restart never rewrites `wanted` — that
is the difference between restart and stop/start.
Error handling is no longer swallowed: a failed insert becomes a reported error
rather than a silently-absent id.
Two views of the same graph existed: the typed `DagView`/`NodeView`
(`/api/state.rebuild_queue`, the `QueueDag` socket request, and the
`RebuildQueueChanged` payload) and `hive-jobq-wire`'s generic
`GraphNode` (`/api/jobq/graph`, `QueueNodes`). Every consumer has moved
to the generic one, so the typed pair is deleted rather than kept in
agreement with it.
What that removes, beyond the types: the `QueueDag` request and
`HostResponse::dags`; `Queue::snapshot`; `dag_view`, `visible_dags`,
`shown_on_wire`, `dag_finished_at` and `containers`; and the
`rebuild_queue` field on `/api/state`. `RebuildQueueChanged` keeps its
seq and loses its payload — nothing read it, and shipping the graph
both on an event and on an endpoint is the duplication this issue is
about. It stays an event rather than becoming a poll because
push-on-change is what every other live surface here does.
Two behaviours came out simpler for a structural reason. `await_dags`
needed two rules — settled means "gone from the snapshot" *or* "present
with every node terminal" — because the typed view evicted finished
groups; the generic view doesn't, so pending is just "some node isn't
terminal". And `state_of` in the tests no longer derives a roll-up at
all: a group root's own state is the scheduler's answer.
That second one found a bug. `cancelled_dag_still_runs_its_approval
tail` asserted the group reads `Cancelled` while the tail it exists to
protect was still pending — `rollup_state` flattened the surviving
child away and called the group settled. The root reads `Finishing`,
which is what the scheduler documents: own logic done, children still
running. The test now asserts that, with the reasoning inline so it
doesn't get "fixed" back.
Kept: `Source`, `State`, `PermPayload` and the `NodeId` alias in
`hive-host-sock::jobs` — shared vocabulary, still used by hivectl.
`MATRIX_HTTP` was `http://localhost:8008`, compiled in, used at 18 call
sites. That address is right only while the homeserver happens to share
this daemon's netns, and its doc comment asserted exactly that as a
general fact. A hive whose homeserver lives anywhere else builds fine
and then talks to the wrong machine.
It now reads `HIVE_MATRIX_API_URL`, which `hive-c0re.nix` sets from
`hyperhive.matrix.apiUrl`. The matrix module fills that in with its own
loopback listener when it is the thing running tuwunel — there it is not
a guess but a fact about what it just started — and the operator sets it
by hand otherwise. There is no compiled-in fallback, for the same reason
`forge_http_base()` has none.
`is_present()` follows. It used to scan `nixos-container list` for
`hive-matrix`, which answers "is the homeserver a container on this
host" — a different question, and the reason a remote homeserver would
silently no-op no matter how it was addressed. It now asks whether a URL
is configured. A co-located hive is unaffected: the module supplies the
loopback URL whenever it runs tuwunel itself. It also stops being
`async`, since it no longer does IO, and `require_matrix_present`'s
message names both ways to have a homeserver rather than only the local
container.
Absent a URL, every matrix path no-ops exactly as it did with no
container, and the two accessors make that structural: `Option` for the
callers that fall back to `None`, a `Result` flavour naming the skipped
`is_present()` gate for the ones that propagate.
Follow-up from #2955 (mara: 'make core able to give agent a todo').
First migration slice: Spawned was pure FYI-check-when-convenient
material, not something needing an immediate turn.
Coordinator::push_todo/push_todo_submitter do a best-effort live dial
of the target agent's own hive-agent-sock (hive_host_sock::agent_todo_
socket), sending the exact UpsertTodo request in-container producers
(matrix/bash/forge-notify) already send. Push, not queue: agent
offline (socket absent) or dial failure is a silent no-op, no retry,
no fallback delivery -- matches mara's 'not available if offline'
call exactly.
HelperEvent::Spawned removed entirely (enum variant + all 3 call
sites migrated: handle_spawn's two arms, finish_approval's Spawn
approval-kind arm) rather than kept alongside a translation layer --
per mara's correction on the first design attempt, migrating the
producer means deleting the old path, not bridging it.
Verified: cargo build/clippy/test -p hive-c0re -p hive-host-sock
-p hive-sh4re clean (318 tests), nix fmt clean.
The push side modelled a store per peer hive: a --peer argument, a
swarm.peers.<domain>.snapshotStorePort option, and a swarm_peers module
whose entire job was answering "which peer". A swarm has exactly one
store, so none of that had anything to select between.
The receiver already proved it. It keys destination directories by
agent, not by sending hive, precisely so an agent that migrates keeps
one unbroken incremental chain -- which only makes sense if every hive
pushes to the same place. Per-hive stores would split the chain in two,
the case that keying exists to prevent.
So the destination moves to services.hyperhive.swarm.snapshotStore,
rendered into HYPERHIVE_SNAPSHOT_STORE, and swarm_peers is deleted
rather than adapted. address has no default because it is a
deployment fact this host cannot derive; port defaults because it is a
convention both ends read from the same option docs. An unset or empty
address fails naming the option instead of connecting somewhere
arbitrary, and a test asserts the message suggests no value.
Adds the caller the fd-passing machinery existed for: hivectl agent
<name> subvol snapshot push --peer <hive> resolves the peer, connects
to its snapshot store, writes the agent header, and hands the connected
socket to hive-priv, which runs btrfs send straight into it.
The split keeps the root helper ignorant. Everything that involves
knowing where a peer is, what the wire protocol looks like, and which
hive to trust happens in the unprivileged daemon; hive-priv only ever
receives an already-open descriptor. Once btrfs send starts, neither
process is in the data path, so a multi-gigabyte transfer costs no
per-byte work and survives a hive-c0re restart.
call_with_fd takes the descriptor by value and closes it as soon as the
kernel has it. A socket stays open until every copy closes, so holding
one back would leave the receiver waiting for an EOF that never comes:
btrfs receive blocks and this side reports success for a transfer the
peer never committed. Ownership makes that unrepresentable.
The peer's store port is a new swarm.peers.<domain>.snapshotStorePort
option rather than a constant matching the module default. A pushing
hive cannot read the receiver's configuration, so assuming 51821 would
push at a port nobody promised to listen on; absent, the push fails
naming the option. swarm_peers parses the mesh address the host module
has always rendered into HYPERHIVE_PEERS but nothing read.
`Coordinator::set_paused` wrote the marker directly with `std::fs::write`
from hive-c0re, which runs as the unprivileged `hive-core` user. The
agent's harness dir is chowned to the agent user on every container boot
(`user.nix`'s activation chown), mode 0755 — so hive-core can stat the
marker but gets EACCES creating or unlinking it. Pause therefore only
ever worked on an agent that had never booted; the read side works
because a stat needs traverse, not write, which is why the paused pill
and `is_paused` looked healthy.
Route both directions through hive-priv, the root helper that already
owns the other writes into agent-owned directories:
- `PrivRequest::SetAgentPaused { agent_name, paused }`, with the marker
filename constant moved to hive-priv-sock. That is the narrowest crate
all three sides share (hive-priv deliberately does not depend on
hive-sh4re, which re-exports it for the in-container resolver). A
private copy on any one side would break pause silently, since every
reader just sees "no marker".
- `write_agent_state_file` generalised to `write_agent_dir_file`, taking
the target directory: `state/` and `harness/` are both agent-owned,
which is the same reason both need root.
- resume unlinks via `remove_file`, which acts on the leaf and never
follows a symlink — an agent could otherwise plant a link at the
marker path and have root delete an arbitrary file.
`Coordinator::set_paused` becomes an async round-trip; its three call
sites were already async. Both directions stay idempotent because the
dashboard toggle and `hivectl pause|resume` fire without reading the
current state first.
The agents root is 0700 and owned by the daemon's user, so hivectl's
client-side existence guard hit EACCES on traversal for anyone not root.
It reported that as "this command needs root; re-run with sudo", which
turned three verbs' pre-flight check into a permission error about the
wrong thing: `choom`, `subvol upgrade` and `subvol snapshot create` all
failed at the guard rather than at whatever they actually needed.
The daemon runs as the owning user and already answers this question for
its own provisioning paths, so expose it on the host socket as
`AgentExists` and have hivectl ask. Operators reach that socket through
the `hive-admin` group, so the guard now works without sudo.
`choom` still needs root for `machinectl shell` — we ship no polkit rule
granting those actions — so it now checks the effective uid and says so
directly instead of failing later inside systemd's authorisation.
Both groups only ever act on a single managed agent's state dir, so
they belong in the `agents` namespace rather than as top-level verbs
next to `forge` / `matrix` / `wg`.
Renames `quota limit` -> `quota set`: the enclosing group already
carries the noun, so the bare verb matches the flat `set-parent` /
`set-limits` spelling without stuttering, and it removes the
`set-limits` (cpu/mem) vs `quota limit` (disk) ambiguity. Adds a
cross-pointer from `set-limits` to `agents quota`.
Handlers stay in their own modules; `run_agents` gains the reparenting
glue. Regenerates docs/tools/hivectl-cli.md.
Refs #2724
The hive applies one `agentCpuQuota` / `agentMemoryMax` to every
container. That's the right default and the wrong ceiling: a build-heavy
agent needs headroom the other twelve don't, and raising the hive-wide
value to suit it hands that headroom to everyone.
Adds a per-agent override, persisted host-side and resolved per-field
against the hive defaults.
Follows the existing `meta/*.json` pattern (`capabilities.json`,
`tool-groups.json`): a host-side map read by `hive-c0re`, staged and
committed in the meta repo so every change lands in the audit trail.
```json
{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }
```
Fallback is **per field**, not per agent: an entry with only
`memory_max` leaves that agent on the hive-wide CPU quota. Absent file,
absent agent and absent field all resolve to the hive default, so the
feature is inert until someone opts an agent in.
Unlike the other meta files this one is **not** injected into the
container — a limit is something done *to* an agent, not something it
reads about itself.
```
hivectl agents set-limits sock --cpu-quota 400% --memory-max 8G
hivectl agents set-limits sock --reset
```
Values are validated before they're persisted: they go into a systemd
drop-in verbatim, and a typo there makes the unit fail to *start* —
turning a fat-fingered quota into a container that won't come back.
The command is declarative: each call replaces the agent's whole entry.
That makes a forgotten flag a silent revert, so a bare `set-limits
<name>` is rejected at the clap layer and clearing needs an explicit
`--reset`.
`ContainerView` gains `cpu_quota` / `memory_max`, both always populated:
there's no "unset" state to render, only "same as everyone else". They
reflect what the drop-in *says* — what the next start will enforce — not
a live cgroup reading.
The write goes through `meta::commit_resource_limits` rather than the
bare setter, so it's staged and committed under `META_LOCK`. Writing
without committing would leave the meta working tree dirty for the next
`prepare_deploy` to trip over.
Docs: `persistence.md` (the new meta file, and why it isn't injected),
`tools/hivectl.md` (the prose guide), `tools/hivectl-cli.md`
(regenerated clap dump).
Closes: internal/requests issue 25
A paused agent keeps its container, its claude session and its
dashboard/todo servers up, but stops driving turns. Messages queue
unacked and are drained on resume.
The whole protocol is a single marker file, `<harness>/paused`. That
directory is already a bind-mount shared between host and container, so
both sides just stat the same path: the harness reads it to decide
whether to drive a turn, hive-c0re reads it to render the badge and
writes/removes it for `hivectl pause|resume`. No new wire protocol, no
container round-trip, and it is sticky across restarts by construction.
Not calling `recv_next` while paused *is* the queueing semantic, so
there is no fencing to get wrong: reminders buffer in their unbounded
channel, the todo `Notify` permit coalesces, and a `request_next_turn`
that raced the pause survives because the gate sits above
`self_continue.take()`.
Graceful stop is handled host-side rather than in the harness: a paused
agent provably has no turn in flight, so `run_signal` skips the fence
entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for
a checkpoint turn that will never run.
`paused` is reported on `ContainerView` / `AgentStatusRow` for the
dashboard, orthogonal to `running` and reported for stopped containers
too.
Closes: hyperhive/hyperhive issue 2271
- server::await_dags: a DAG is settled when gone from the snapshot (fully
Done) or present with all nodes terminal; pending only with a non-terminal
node (DagView no longer carries a rolled-up state).
- DagView::rollup_state() added to hive-sh4re — the shared node-set roll-up
derivation every Rust consumer uses.
- JobQueue::build_log_id_of(node_id) — the node_id -> build_logs lookup the
query endpoint will use; tests assert log-id via it now.
- tests: derive roll-up state; drop the off-wire step/build_log_id wire asserts.
With the meta-update cascade (#2476) and startup sweep (#2450) folded
into single DAGs that grow per-agent subgraphs via append_subgraph,
nothing links parent/child DAGs anymore — parent_id is dead.
hive-c0re: drop parent_id from Dag/DagSpec (+ the DagView copy); delete
append_children and cancel_children (no callers); simplify trim_history
(no more terminal-parent-with-live-children guard — a one-big-DAG is
terminal only when its whole graph settles); drop the rebuild() parent_id
param; QueueDag returns just the polled DAG (no fan-out children to
gather). hive-sh4re: drop the DagView.parent_id wire field.
frontend: a multi-step op is one DAG now, so renderRebuildQueue drops the
childrenOf/orphans cross-DAG grouping and renders each entry flat; its
per-agent subgraphs render as nodes within the one row (split by deps).
Removed the dead rqe-child style + isChild plumbing.
Docs + the child-DAG queue tests updated/removed to match.
Hive-wide `stop` / `start` / `restart` emit ONE DAG with a per-agent
subgraph each (concurrent on their own leases) instead of N DAGs — and each
subgraph is now built dynamically from the agent's live running state rather
than a fixed template shape:
- online agent: the full stop→reconcile (restart: stop-for-update→reconcile)
chain; `graceful` prepends signal→drain.
- offline agent: just `SetWanted → Reconcile` (nothing to quiesce/stop; a
restart of a down agent is really a start).
The head `SetWanted` (intent) and tail `Reconcile` (convergence guarantee)
are always present; only the mechanical `Signal`/`Drain`/`StopForUpdate`
nodes are state-conditional. Keeping `Reconcile` in every shape closes the
TOCTOU window — a race-up between the `is_running` read and node exec is
still converged in-DAG (with `StopForUpdate`-noop as the backstop) — with no
reliance on an external reconcile sweep.
The state-aware assembly needs an async `is_running` read, so it moves out
of the pure/sync `templates.rs` into `submit.rs`, layered as pure
`*_chain(running)` → pure `*_spec(targets)` (the unit-test seam) → async
`*_many` (reads live state + submits). `templates.rs` keeps only the shared
pure primitives (`node`/`after_ok`/`rebuild_nodes`).
Callers await the now-async submit fns (server, dashboard, socket_server).
Tests exercise both the online and offline shapes via the pure `*_spec`
seam. docs/coordinator.md shapes updated.
A hive-wide restart was N separate single-agent DAGs (one submit::restart
per agent). Now that agent is per-node (#2445), make it ONE DAG with a
per-agent restart subgraph each.
- templates::restart takes an agent list: each agent gets an independent
subgraph (a head SetWanted(Up) root, then its restart chain), so the N
subgraphs run concurrently on their own leases. One agent = the ordinary
single-agent restart; unifies the old restart + graceful_restart fns.
- submit::restart / graceful_restart stay as single-agent wrappers over
the new submit::restart_many(agents, graceful).
- server.rs handle_restart_all + handle_restart_scoped submit one
restart_many call instead of looping per agent. Infra containers
unchanged (no lease/DAG, synchronous).
Scope: restart + restart-all only. Broad stop+start is the same pattern
(stop/start templates take agent lists) — a follow-up increment.
argus caught it: the function-level /// comment still described the
old submit-await-submit graceful approach after the code moved to
one atomic GracefulRestart DAG.
mara's review on #2436: no submit-await-submit composition, even
server-side. Adds Template::GracefulRestart (Signal -> Drain ->
StopForUpdate -> Reconcile, wanted=Up) mirroring how Restart already
does StopForUpdate -> Reconcile, plus submit::graceful_restart and
templates::graceful_restart. handle_restart_scoped now submits exactly
one DAG per agent up front for both the graceful and non-graceful
case -- no await_dags in the loop anymore.
hivectl restart --agent NAME previously composed stop() then start() as
two separate client-side daemon calls glued by CLI-process control flow
— not one DAG, and a dropped hivectl connection mid-restart (ssh drop,
Ctrl-C) left the agent stopped with no automatic follow-up. mara flagged
this as the first target for the 'dagify hivectl commands' issue.
New HostRequest::RestartScoped{scope, graceful} handles it server-side:
each targeted agent now rides exactly one atomic Restart-template DAG
(same one hivectl agents restart / restart-all already use) in the
common non-graceful case. --graceful has no single-DAG template yet, so
it submits the graceful-stop DAGs, awaits them server-side, then submits
the start DAGs — still one daemon call end to end, just not yet a single
DAG (noted as a follow-up). Infra containers restart synchronously as
before (no lease/DAG concept for them).
CLI-side restart() now just makes the one call + waits, same output
shape as before via render_lifecycle.
