docs(#3034): move the brace rationale out of templates.rs into coordinator.md
The pre-push comment-block lint rejected two 40-line doc blocks, correctly: the module doc and `rebuild_subtree`'s now carry the trigger and a pointer, and the reasoning lives in a new `#### Braces` section. That move surfaced a third doc the resource change had falsified. The scheduler's lease-acquirer list still named `StopForUpdate` / `Swap` / `Signal` / `Drain`, all of which are now exempt. The list now separates container-affecting nodes from braces, and says why the rebuild subtree's members are exempt for a different reason than `MetaSync` / `Prebuild`: they do touch the container, but their brace holds the lease above them.
This commit is contained in:
parent
a0790e4e49
commit
7d26d6017f
2 changed files with 76 additions and 57 deletions
|
|
@ -190,17 +190,56 @@ resources are free. Resources:
|
|||
held by nix-heavy nodes for the node's duration.
|
||||
2. **Per-agent lifecycle lease** — keyed on the **node's** agent (agent is
|
||||
per-node; a DAG can span agents) and globally exclusive per agent across
|
||||
all DAGs: acquired at a container-affecting node (`SetWanted`,
|
||||
`StopForUpdate`, `Swap`, `Signal`, `Drain`, `Reconcile`, `WriteDropin`,
|
||||
`Create`, `DeployWindow`), held by the owning DAG until it's terminal,
|
||||
so two DAGs never interleave container ops on the same agent. A DAG
|
||||
touching several agents holds one lease per agent. (`SetWanted` is a store
|
||||
write, not a container op, but takes the lease anyway so a power-op DAG's
|
||||
intent write + reconcile is atomic — two racing ops can't clobber intent
|
||||
before either reconciles.) **Lease-exempt**: `MetaSync`, `Prebuild`,
|
||||
`MetaLock`, `WritePermFile`, `Reparent` —
|
||||
they touch the store / meta, not the running container, which is exactly
|
||||
why a stop can land while another DAG's prebuild is still building.
|
||||
all DAGs: acquired either at a container-affecting node (`SetWanted`,
|
||||
`Reconcile`, `WriteDropin`, `Create`) or at a **brace** (`AgentWindow`,
|
||||
`DeployWindow`) on behalf of a whole coordinated subtree; held by the owning
|
||||
DAG until it's terminal, so two DAGs never interleave container ops on the
|
||||
same agent. A DAG touching several agents holds one lease per agent.
|
||||
(`SetWanted` is a store write, not a container op, but takes the lease anyway
|
||||
so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
|
||||
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
|
||||
`Prebuild`, `MetaLock`, `WritePermFile`, `Reparent` — they touch the store /
|
||||
meta, not the running container, which is exactly why a stop can land while
|
||||
another DAG's prebuild is still building. Also exempt, for a different
|
||||
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
|
||||
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
|
||||
container, but their `AgentWindow` brace holds the lease above them — see
|
||||
_Braces_ below.
|
||||
|
||||
#### Braces
|
||||
|
||||
Templates otherwise declare a resource on **every** node that needs it, even
|
||||
when a parent already holds it, so the requirement belongs to the node rather
|
||||
than to one DAG shape it happens to appear in. A **brace** is the one sanctioned
|
||||
exception: a pure-resource-holder root that declares on behalf of a subtree
|
||||
coordinated with itself, whose members then declare nothing.
|
||||
|
||||
It is forced rather than stylistic. Declaring a resource means *"I need this
|
||||
exclusively"*, and the agent lease is single-unit — so **two siblings that both
|
||||
declared it could never run concurrently.** For a subtree whose whole point is
|
||||
concurrency (`Prebuild` beside the `Signal` → `Drain` quiesce window), declaring
|
||||
the requirement truthfully on every node and running those nodes in parallel are
|
||||
mutually exclusive. One holder above them speaks for the subtree.
|
||||
|
||||
This is the opposite of the failure the declare-your-own rule exists to prevent,
|
||||
not a relapse into it: there the requirement was *implicit*, inferred from a
|
||||
node's kind and true only by accident of placement. Here it is explicit, on one
|
||||
node, with the omission below it documented on the brace itself.
|
||||
|
||||
Two consequences worth knowing:
|
||||
|
||||
- **Flattening a chain under a brace is safe.** The stop chain used to nest
|
||||
`Signal` over `Drain` over `StopForUpdate` specifically so the lease stayed
|
||||
continuous — as independent siblings each would acquire it separately and
|
||||
leave a gap another DAG could claim the agent in, mid-bounce. A brace supplies
|
||||
that continuity directly, so the nesting is no longer load-bearing.
|
||||
- **Observability is unaffected.** `running_transients` keys off a node's
|
||||
*payload* agent, not off a declared lease edge, so every child still lights its
|
||||
own dashboard pill and still reports its own `takes_container_down` to the
|
||||
crash watcher. A brace itself reports `false`: it parents the stopping nodes
|
||||
but does not stop anything, and claiming otherwise would widen crash
|
||||
suppression across the build and tail, where a vanished container is still a
|
||||
real crash.
|
||||
|
||||
Among simultaneously-ready nodes competing for a resource, DAG-submit order
|
||||
wins (FIFO) so bulk operations drain predictably. The scheduler also owns the
|
||||
|
|
|
|||
|
|
@ -1,25 +1,15 @@
|
|||
//! DAG shape builders — every operation as a template over the shared
|
||||
//! node primitives. Pure (no I/O); each node carries its own `agent` (there is
|
||||
//! no DAG-level agent) and **declares its own resources** with `.needs(…)`
|
||||
//! right where it is constructed, rather than having them derived from its
|
||||
//! kind. Deriving made the requirement a property of the *kind*, so a kind that
|
||||
//! happened to run under an ancestor already holding the resource could get
|
||||
//! away with declaring nothing.
|
||||
//! DAG shape builders — every operation as a template over the shared node
|
||||
//! primitives. Pure (no I/O); each node carries its own `agent` (there is no
|
||||
//! DAG-level agent) and **declares its own resources** with `.needs(…)` right
|
||||
//! where it is constructed, rather than derived from its kind. Deriving made the
|
||||
//! requirement a property of the *kind*, so a kind running under an ancestor
|
||||
//! that already held the resource could get away with declaring nothing.
|
||||
//!
|
||||
//! **The one sanctioned exception is a brace** — a pure-resource-holder root
|
||||
//! ([`NodeKind::AgentWindow`], [`NodeKind::DeployWindow`]) that declares for a
|
||||
//! subtree of nodes coordinated with each other, which then declare nothing.
|
||||
//! This is the opposite of the failure above, not a relapse into it: there the
|
||||
//! requirement was *implicit*, inferred from a kind and true only by accident of
|
||||
//! placement; here it is declared explicitly on one node, and the omission below
|
||||
//! it is deliberate and documented on the brace.
|
||||
//!
|
||||
//! It has to work this way, because declaring a resource means *"I need this
|
||||
//! exclusively"* and the agent lease is single-unit: **two siblings that both
|
||||
//! declared it could never run concurrently.** So for a subtree whose whole
|
||||
//! point is concurrency, declaring the requirement truthfully on every node and
|
||||
//! running those nodes in parallel are mutually exclusive. The brace is how the
|
||||
//! shape says "this subtree is coordinated, one holder speaks for it".
|
||||
//! ([`NodeKind::AgentWindow`], [`NodeKind::DeployWindow`]) declaring for a
|
||||
//! coordinated subtree whose members then declare nothing. Forced by the lease
|
||||
//! being single-unit: two siblings that both declared it could never run
|
||||
//! concurrently. Why this is not a relapse: `docs/coordinator.md`.
|
||||
//!
|
||||
//! ```text
|
||||
//! rebuild(a): MetaSync(a) → AgentWindow(a){ Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) RebuildBookkeeping(a) } →(any) Reconcile(a)
|
||||
|
|
@ -37,7 +27,7 @@
|
|||
//! The hive-wide **power ops** (`stop` / `start` / `restart`) are NOT here:
|
||||
//! their per-agent shape depends on live running state (an async
|
||||
//! `lifecycle::is_running` read), so `submit.rs` assembles them out of the
|
||||
//! primitives this module exports ([`rebuild_nodes`]) over `JobBuilder::node`.
|
||||
//! primitives this module exports ([`rebuild_nodes`]).
|
||||
|
||||
use hive_jobq::TerminalState;
|
||||
|
||||
|
|
@ -174,34 +164,24 @@ impl<'a> RebuildRoots<'a> {
|
|||
/// build slot *and* the agent lease, atomically, and holds both for the whole
|
||||
/// subtree. Everything below it declares **nothing** and re-enters these
|
||||
/// grants.
|
||||
/// - `Prebuild` and the quiesce chain are **siblings under the brace, and run
|
||||
/// concurrently.** That is the point of the brace: they contend for different
|
||||
/// resources (slot vs. agent), so nesting the stop under the build — as this
|
||||
/// template did — hid the entire graceful-stop timeout behind the nix build,
|
||||
/// per agent, on every sweep.
|
||||
/// - the **quiesce chain** (graceful only): `Signal` then `Drain` as its child.
|
||||
/// Asks the agent to checkpoint and waits for it to go quiet; the container
|
||||
/// is still *up* throughout.
|
||||
/// - `Prebuild` and the **quiesce chain** (`Signal` → `Drain`, graceful only)
|
||||
/// are siblings under the brace and run **concurrently** — they contend for
|
||||
/// different resources, so nesting the stop under the build (as this template
|
||||
/// did) only hid the graceful-stop timeout behind the nix build. The container
|
||||
/// is still up throughout the quiesce.
|
||||
/// - `StopForUpdate` (child of the brace): `AfterOk` **both** `Prebuild` and
|
||||
/// `Drain`. Waiting on the build is deliberate — running the drain window
|
||||
/// early is the win, taking the container *down* early would be pure
|
||||
/// downtime. This is the node that actually stops the container.
|
||||
/// - `Swap` (child of `StopForUpdate`), then `RebuildBookkeeping` (`AfterOk` its sibling
|
||||
/// `Swap`): the Ok-only bookkeeping tail (rev marker, forge/matrix sync,
|
||||
/// kick, rescan).
|
||||
/// `Drain`. Waiting on the build is deliberate — running the drain early is
|
||||
/// the win, taking the container *down* early would be pure downtime.
|
||||
/// - `Swap` (child of `StopForUpdate`), then `RebuildBookkeeping` (`AfterOk` its
|
||||
/// sibling `Swap`): the Ok-only bookkeeping tail.
|
||||
/// - `Reconcile` (**last, root**): `AfterAny` `AgentWindow`, which rolls up
|
||||
/// terminal only once its whole subtree has settled — so `Reconcile` runs
|
||||
/// after the swap regardless of outcome, and as a top-level root it survives
|
||||
/// the cancel-cascade of a failed brace (recovery-start invariant, which also
|
||||
/// covers a failed `MetaSync`: that cancel-cascades the brace, i.e. terminal,
|
||||
/// so the tail still runs). It takes a fresh lease; the tiny gap is harmless —
|
||||
/// `Reconcile` converges to the persisted `wanted` idempotently.
|
||||
/// terminal only once its whole subtree has settled — so it runs after the
|
||||
/// swap regardless of outcome, and as a top-level root it survives the
|
||||
/// cancel-cascade of a failed brace (recovery-start invariant). Fresh lease;
|
||||
/// the tiny gap is harmless, `Reconcile` converges idempotently.
|
||||
///
|
||||
/// The lease-continuity argument that used to justify nesting the stop chain
|
||||
/// (*"siblings would each take the lease separately and leave a gap another DAG
|
||||
/// could claim the agent in, mid-bounce"*) is **satisfied by the brace instead**
|
||||
/// — one holder above them all, so siblings share one continuous grant and
|
||||
/// there is no gap to claim. That is what makes flattening them safe.
|
||||
/// Why flattening the stop chain is safe, and why the lease-continuity argument
|
||||
/// that used to justify nesting it is satisfied by the brace: `docs/coordinator.md`.
|
||||
fn rebuild_subtree<'a>(
|
||||
builder: &'a JobBuilder,
|
||||
agent: &str,
|
||||
|
|
|
|||
Loading…
Reference in a new issue