feat(#2476): grow the meta-update cascade in-DAG instead of child DAGs
MetaLock's non-sweep completion now grows one rebuild subgraph per affected agent into the same DAG (append_subgraph), replacing the fan-out-child-DAGs + cancel_children dance. Drops NodeOutput.fanout and scheduler's fanout_specs. meta_update DAG carries Rebuilding transient so each cascade agent gets crash-watch suppression at Swap (the property the old child Rebuild DAGs held via their own transient); MetaLock head needs no lease so the pseudo-agent gets no pill. append_children/parent_id and child-DAG tests are intentionally left for the #2453 capstone.
This commit is contained in:
parent
31752d4089
commit
2b3130f63c
5 changed files with 102 additions and 66 deletions
|
|
@ -104,9 +104,9 @@ restart(a..): online a: SetWanted(a,Up) → [Signal→Drain→ if graceful] Sto
|
||||||
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
|
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
|
||||||
spawn(a): [wanted=Up at approve] Create(a) → WriteDropin(a) → Reconcile(a)
|
spawn(a): [wanted=Up at approve] Create(a) → WriteDropin(a) → Reconcile(a)
|
||||||
perm-change(a): WritePermFile(a) → «rebuild subgraph»
|
perm-change(a): WritePermFile(a) → «rebuild subgraph»
|
||||||
meta-update(inp): MetaLock(inp) → «fan-out rebuild(a) per affected agent»
|
meta-update(inp): MetaLock(inp) →(in-DAG) «rebuild subgraph» per affected agent
|
||||||
boot: (if any rev marker stale) MetaLock(hyperhive) → «fan-out rebuild»;
|
boot: (if any rev marker stale) MetaLock(hyperhive) →(in-DAG) «rebuild subgraph» per stale agent;
|
||||||
plus Reconcile(a) for every drifted agent
|
plus Reconcile(a) for every drifted agent (all ONE DAG)
|
||||||
```
|
```
|
||||||
|
|
||||||
Notable collapses:
|
Notable collapses:
|
||||||
|
|
@ -123,10 +123,14 @@ Notable collapses:
|
||||||
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
|
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
|
||||||
whole-hive graceful stop fires every agent's signal immediately and all
|
whole-hive graceful stop fires every agent's signal immediately and all
|
||||||
drains overlap; each DAG's tail `Reconcile` does the actual stop.
|
drains overlap; each DAG's tail `Reconcile` does the actual stop.
|
||||||
- **The meta-update cascade fans out on completion**: `MetaLock`'s executor
|
- **The meta-update cascade grows in the same DAG on completion**:
|
||||||
computes the affected agent set after the bump lands and appends child
|
`MetaLock`'s executor computes the affected agent set after the bump lands
|
||||||
`rebuild` DAGs (`parent_id` set, `relock = false` so the children don't
|
and grows one `rebuild` subgraph per agent into its *own* DAG via
|
||||||
revert the bump). A failed bump fans out nothing — no cancel-children dance.
|
`append_subgraph` (rooted on the `MetaLock`, `relock = false` so the cascade
|
||||||
|
doesn't revert the bump). Not child DAGs — one DAG, no `parent_id`. A failed
|
||||||
|
bump appends nothing (no cancel-children dance). Same shape as the startup
|
||||||
|
sweep; the meta-update DAG carries the `Rebuilding` transient so each cascade
|
||||||
|
agent keeps crash-watch suppression during its `Swap`.
|
||||||
|
|
||||||
### Desired-state (spec vs status)
|
### Desired-state (spec vs status)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -27,26 +27,24 @@ pub const GRACEFUL_STOP_TIMEOUT: std::time::Duration = std::time::Duration::from
|
||||||
/// success.
|
/// success.
|
||||||
#[derive(Debug, Default)]
|
#[derive(Debug, Default)]
|
||||||
pub struct NodeOutput {
|
pub struct NodeOutput {
|
||||||
/// Agents to fan child `Rebuild` DAGs out for (`MetaLock` only).
|
|
||||||
pub fanout: Vec<String>,
|
|
||||||
/// Mechanical sub-step nodes to append into *this same* DAG at
|
/// Mechanical sub-step nodes to append into *this same* DAG at
|
||||||
/// runtime, each depending `AfterOk` on the emitting node — e.g. a
|
/// runtime, each depending `AfterOk` on the emitting node — e.g. a
|
||||||
/// `Reconcile` planner emitting a `Start` / `Stop`. A separate
|
/// `Reconcile` planner emitting a `Start` / `Stop`. Keeps the sub-step a
|
||||||
/// channel from `fanout` (which appends whole *child* DAGs) so the
|
/// first-class node in the same DAG so the lease-window transient is held
|
||||||
/// sub-step stays a first-class node in the same DAG and the
|
/// across it. The scheduler applies these *before* the emitting node's
|
||||||
/// lease-window transient is held across it. The scheduler applies
|
/// completion so the DAG never rolls terminal with the appended work
|
||||||
/// these *before* the emitting node's completion so the DAG never
|
/// still pending.
|
||||||
/// rolls terminal with the appended work still pending.
|
|
||||||
pub append_nodes: Vec<NodeKind>,
|
pub append_nodes: Vec<NodeKind>,
|
||||||
/// Whole per-agent *subgraphs* to append into *this same* DAG at
|
/// Whole per-agent *subgraphs* to append into *this same* DAG at
|
||||||
/// runtime — the multi-node generalisation of `append_nodes`. Each
|
/// runtime — the multi-node generalisation of `append_nodes`. Each
|
||||||
/// inner `Vec<NodeSpec>` is one independent subgraph whose `deps` are
|
/// inner `Vec<NodeSpec>` is one independent subgraph whose `deps` are
|
||||||
/// local (0-based within that subgraph); the scheduler appends each via
|
/// local (0-based within that subgraph); the scheduler appends each via
|
||||||
/// [`JobQueue::append_subgraph`], which rebases the deps onto the DAG's
|
/// [`JobQueue::append_subgraph`], which rebases the deps onto the DAG's
|
||||||
/// node-id space and roots the subgraph on the emitting node. The
|
/// node-id space and roots the subgraph on the emitting node. Both
|
||||||
/// startup sweep's `MetaLock` uses this to grow one stale-agent rebuild
|
/// `MetaLock` flavours use this to grow one rebuild subgraph per agent
|
||||||
/// subgraph per agent into the same boot DAG instead of fanning out
|
/// into their own DAG (the startup sweep's stale agents; the meta-update
|
||||||
/// child DAGs. Same before-completion ordering as `append_nodes`.
|
/// cascade's affected agents) instead of fanning out child DAGs. Same
|
||||||
|
/// before-completion ordering as `append_nodes`.
|
||||||
pub append_subgraph: Vec<Vec<NodeSpec>>,
|
pub append_subgraph: Vec<Vec<NodeSpec>>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -312,8 +310,18 @@ async fn run_meta_lock(
|
||||||
Some(list) => list,
|
Some(list) => list,
|
||||||
None => meta_update_cascade_agents(&claim.inputs).await,
|
None => meta_update_cascade_agents(&claim.inputs).await,
|
||||||
};
|
};
|
||||||
|
// Grow one rebuild subgraph per affected agent into *this* meta-update
|
||||||
|
// DAG (rooted on this `MetaLock`, so they build against the post-bump
|
||||||
|
// lock), rather than fanning out child DAGs. `relock = false` — the
|
||||||
|
// cascade children must NOT re-lock, which would revert the bump this
|
||||||
|
// node just committed (the property the old `fanout_specs` meta-update
|
||||||
|
// branch encoded).
|
||||||
|
let append_subgraph = cascade
|
||||||
|
.iter()
|
||||||
|
.map(|agent| super::templates::rebuild_nodes(agent, false, 0))
|
||||||
|
.collect();
|
||||||
Ok(NodeOutput {
|
Ok(NodeOutput {
|
||||||
fanout: cascade,
|
append_subgraph,
|
||||||
..Default::default()
|
..Default::default()
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -3,21 +3,20 @@
|
||||||
//! executor task per claim, and on any completion re-evaluate.
|
//! executor task per claim, and on any completion re-evaluate.
|
||||||
//! Concurrency comes from the build-slot count, not multiple workers.
|
//! Concurrency comes from the build-slot count, not multiple workers.
|
||||||
//!
|
//!
|
||||||
//! Also owns the two DAG-lifetime side channels the sync queue core
|
//! Also owns the per-DAG transient guard (dashboard pill + crash-watch
|
||||||
//! can't hold itself:
|
//! suppression) that the sync queue core can't hold itself — created when a
|
||||||
//! - the per-DAG transient guard (dashboard pill + crash-watch
|
//! DAG acquires its agent lease, dropped when the DAG settles terminal.
|
||||||
//! suppression), created when a DAG acquires its agent lease and
|
//!
|
||||||
//! dropped when the DAG settles terminal;
|
//! In-DAG growth (a `MetaLock` growing rebuild subgraphs after the lock
|
||||||
//! - the `MetaLock` fan-out: appending child `Rebuild` DAGs once the
|
//! bump, a `Reconcile` fanning its `Start`/`Stop`) flows through
|
||||||
//! lock bump lands, so children build against the post-bump lock
|
//! `NodeOutput.append_subgraph` / `append_nodes`, applied before the
|
||||||
//! (and a failed bump fans out nothing — replacing the old
|
//! emitting node completes — see `handle_completion`.
|
||||||
//! pre-enqueue + cancel-children dance).
|
|
||||||
|
|
||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use super::Claim;
|
||||||
use super::exec::{self, NodeOutput};
|
use super::exec::{self, NodeOutput};
|
||||||
use super::{Claim, Source, templates};
|
|
||||||
use crate::coordinator::Coordinator;
|
use crate::coordinator::Coordinator;
|
||||||
|
|
||||||
struct NodeDone {
|
struct NodeDone {
|
||||||
|
|
@ -128,10 +127,6 @@ async fn handle_completion(
|
||||||
coord
|
coord
|
||||||
.job_queue
|
.job_queue
|
||||||
.complete_node(claim.dag_id, claim.node_id, Ok(()));
|
.complete_node(claim.dag_id, claim.node_id, Ok(()));
|
||||||
if !output.fanout.is_empty() {
|
|
||||||
let specs = fanout_specs(&claim, output.fanout);
|
|
||||||
coord.job_queue.append_children(specs);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
let msg = format!("{e:#}");
|
let msg = format!("{e:#}");
|
||||||
|
|
@ -165,30 +160,3 @@ async fn process_terminals(
|
||||||
exec::on_dag_terminal(coord, &terminal).await;
|
exec::on_dag_terminal(coord, &terminal).await;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Child `Rebuild` specs for a completed meta-update `MetaLock` fan-out,
|
|
||||||
/// grouped under the parent via `parent_id`. Meta-update children skip the
|
|
||||||
/// per-agent relock (`relock = false`) — it would revert the bump the parent
|
|
||||||
/// just committed. This is now the meta-update cascade path only: the startup
|
|
||||||
/// sweep no longer fans out child DAGs — it grows one rebuild subgraph per
|
|
||||||
/// stale agent into its own DAG via `append_subgraph` (see
|
|
||||||
/// `exec::run_meta_lock`).
|
|
||||||
fn fanout_specs(claim: &Claim, agents: Vec<String>) -> Vec<super::DagSpec> {
|
|
||||||
let reason = if let Some(approval_id) = claim.approval_id {
|
|
||||||
format!("approval #{approval_id} meta input cascade")
|
|
||||||
} else {
|
|
||||||
"meta-update cascade".to_owned()
|
|
||||||
};
|
|
||||||
agents
|
|
||||||
.into_iter()
|
|
||||||
.map(|agent| {
|
|
||||||
templates::rebuild(
|
|
||||||
&agent,
|
|
||||||
Source::MetaUpdate,
|
|
||||||
reason.clone(),
|
|
||||||
Some(claim.dag_id),
|
|
||||||
false,
|
|
||||||
)
|
|
||||||
})
|
|
||||||
.collect()
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -186,10 +186,16 @@ pub fn perm_change(agent: &str, source: Source, reason: String, payload: PermPay
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Meta-input lock bump. Child `Rebuild` DAGs fan out on completion —
|
/// Meta-input lock bump. The `MetaLock` executor grows one rebuild subgraph
|
||||||
/// appended *after* the bump lands so their prebuilds run against the
|
/// per affected agent into *this same* DAG on completion (via
|
||||||
/// post-bump lock (and so a failed bump simply fans out nothing,
|
/// `append_subgraph`) — appended *after* the bump lands so their prebuilds
|
||||||
/// replacing the old pre-enqueue + `cancel_children` dance).
|
/// run against the post-bump lock, and a failed bump appends nothing
|
||||||
|
/// (replacing the old fan-out-child-DAGs + `cancel_children` dance).
|
||||||
|
/// `transient = Rebuilding` because those appended subgraphs are rebuilds:
|
||||||
|
/// it's applied per-agent at claim time (the `MetaLock` head needs no lease,
|
||||||
|
/// so the "hyperhive" pseudo-agent gets no pill), giving each cascade agent
|
||||||
|
/// crash-watch suppression during its `Swap` — the property the old child
|
||||||
|
/// `Rebuild` DAGs carried via their own transient.
|
||||||
pub fn meta_update(
|
pub fn meta_update(
|
||||||
inputs: Vec<String>,
|
inputs: Vec<String>,
|
||||||
source: Source,
|
source: Source,
|
||||||
|
|
@ -204,7 +210,7 @@ pub fn meta_update(
|
||||||
approval_id,
|
approval_id,
|
||||||
inputs,
|
inputs,
|
||||||
perm_payload: None,
|
perm_payload: None,
|
||||||
transient: None,
|
transient: Some(TransientKind::Rebuilding),
|
||||||
nodes: vec![node(
|
nodes: vec![node(
|
||||||
"hyperhive",
|
"hyperhive",
|
||||||
NodeKind::MetaLock {
|
NodeKind::MetaLock {
|
||||||
|
|
|
||||||
|
|
@ -502,6 +502,56 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() {
|
||||||
|
// The meta-update `MetaLock` grows one rebuild subgraph per affected
|
||||||
|
// agent into its OWN DAG (via append_subgraph), not child DAGs.
|
||||||
|
// The DAG carries `Rebuilding` so the folded rebuilds keep crash-watch
|
||||||
|
// suppression (the property the old child Rebuild DAGs had via their own
|
||||||
|
// transient).
|
||||||
|
let spec = templates::meta_update(
|
||||||
|
vec!["nixpkgs".to_owned()],
|
||||||
|
Source::Manual,
|
||||||
|
"bump".to_owned(),
|
||||||
|
None,
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
matches!(
|
||||||
|
spec.transient,
|
||||||
|
Some(crate::coordinator::TransientKind::Rebuilding)
|
||||||
|
),
|
||||||
|
"meta-update DAG must carry Rebuilding so cascade rebuilds get suppression"
|
||||||
|
);
|
||||||
|
let q = JobQueue::new(4);
|
||||||
|
let id = submit(&q, spec);
|
||||||
|
let meta_lock = claim_one(&q);
|
||||||
|
assert_eq!(meta_lock.kind.as_str(), "meta_lock");
|
||||||
|
// Simulate the executor growing the cascade in-DAG (`relock = false` — a
|
||||||
|
// cascade child must not re-lock and revert the parent's bump).
|
||||||
|
for agent in ["alice", "bob"] {
|
||||||
|
q.append_subgraph(
|
||||||
|
id,
|
||||||
|
templates::rebuild_nodes(agent, false, 0),
|
||||||
|
meta_lock.node_id,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
q.complete_node(id, meta_lock.node_id, Ok(()));
|
||||||
|
// Still ONE DAG — no child DAGs — and both cascade rebuild subgraphs root
|
||||||
|
// on the MetaLock, each on its own agent lease.
|
||||||
|
assert_eq!(q.snapshot().len(), 1);
|
||||||
|
let next = q.claim_ready();
|
||||||
|
let mut kinds: Vec<(&str, &str)> = next
|
||||||
|
.iter()
|
||||||
|
.map(|c| (c.agent.as_str(), c.kind.as_str()))
|
||||||
|
.collect();
|
||||||
|
kinds.sort_unstable();
|
||||||
|
assert_eq!(
|
||||||
|
kinds,
|
||||||
|
vec![("alice", "prebuild"), ("bob", "prebuild")],
|
||||||
|
"cascade rebuilds grow in the meta-update DAG, concurrent per agent"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
// ---- failure: cancel-downstream + AfterAny ----
|
// ---- failure: cancel-downstream + AfterAny ----
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue