mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

deploy the closure the swarm declares, read no forge head, drop the update badge

Browse source
This commit is contained in:
atlas 2026-11-01 23:56:40 +01:00 • committed by mara
commit 47c2acb640
1 changed files with 647 additions and 2001 deletions
Download patch file Download diff file Expand all files Collapse all files

240
hive-c0re/src/workers/auto_update.rs
View file

@ -1,30 +1,42 @@
//! Boot reconcile: on `hive-c0re serve` boot, (a) rebuild the agents whose
//! per-agent rev marker differs from the hyperhive commit the swarm's
//! `internal/meta` pins now — one `Boot` DAG holding a `Rebuild` subgraph for
//! each stale agent whose `wanted` power intent is `Up` — and (b) converge
//! every other drifted agent to its persisted `wanted` via `Reconcile` DAGs.
//! This is how a hive that missed the swarm's fan-out catches up. Three rules
//! keep boot-time nix work minimal:
//! Auto-update: rebuild the agents whose swarm-declared closure the hive has
//! not applied. An agent's `agent-closures` record names the store path the
//! swarm wants it to run; the agent's `applied/.<name>.closure` marker holds
//! the one the hive last swapped to ([`crate::attic::closure::pending`]).
//!
//! A pass runs on `hive-c0re serve` boot, then every [`PASS_INTERVAL`]. A boot
//! pass is how a hive that missed the swarm's deploy request catches up; the
//! periodic one covers a request lost while the hive was up. Each pass
//! rebuilds each stale agent whose `wanted` power intent is `Up` as one `Boot`
//! DAG, and the boot pass also converges every other drifted agent to its
//! persisted `wanted` via `Reconcile` DAGs. Rules that keep nix work minimal:
//!
//! 1. **Stale but wanted-offline agents** get no rebuild — their
//! rebuild happens the first time they're started (the start
//! submit path upgrades a stale start to rebuild+start).
//! 2. **Agents whose rev marker matches** the pin are skipped — nothing
//! changed, no nix work to do.
//! 3. **A pin the hive cannot read** (no forge token, forge down, no
//! `internal/meta` yet) rebuilds nothing: a rebuild would have no flake
//! to build from either.
//! 2. **Agents running the declared closure** are skipped.
//! 3. **A declaration the hive cannot read or act on** (no connection to the
//! swarm queue, no record, the closure not in the cache, no marker to
//! compare with) rebuilds nothing.
//! 4. **A declaration is attempted once per process** by the periodic pass, and
//! never while a rebuild of the agent is queued or running, so a closure
//! that fails to deploy does not restart its agent every interval.
//!
//! See `docs/scheduler/coordinator.md::Boot reconcile`.
use std::collections::HashMap;
use std::path::Path;
use std::sync::Arc;
use std::time::Duration;
use anyhow::Result;
use crate::coordinator::Coordinator;
use crate::lifecycle::{self, AGENT_PREFIX};
/// How often the swarm's declarations are compared with what the hive runs
/// after the boot pass.
const PASS_INTERVAL: Duration = Duration::from_mins(5);
/// Resolve the current rev of `hyperhive_flake`. For a path on disk we
/// canonicalize (following symlinks) so a /etc/hyperhive → /nix/store/...
/// update yields a different string. For anything else we return None.
@ -39,47 +51,39 @@ pub fn current_flake_rev(hyperhive_flake: &str) -> Option<String> {
.map(|p| p.display().to_string())
}
/// Returns true when the config commit `internal/meta` pins for the agent's
/// next build (`meta_rev::recorded`) differs from the one its last successful
/// build pinned (`deployed`, the `paths::applied_config_marker` content). This
/// is the semantic the dashboard `needs_update` chip conveys: "there is a
/// config change ready to apply via rebuild." An archetype agent pins no
/// config commit and never has one pending.
///
/// Reads one small file and the marker the caller already holds, no `git`:
/// this runs per agent inside `container_view::build_all`, on the ~10s
/// dashboard sweep and every `AgentStatus` request.
#[must_use]
pub fn agent_config_pending(name: &str, deployed: Option<&str>) -> bool {
let pinned = crate::meta_rev::recorded(name).and_then(|build| build.config);
config_pending(pinned.as_deref(), deployed)
/// What one pass decided.
struct Plan {
/// Stale ∧ wanted up, running agents first.
rebuilds: Vec<String>,
/// Agents whose observed power state drifted from `wanted`; empty after the
/// boot pass.
drifted: Vec<String>,
deferred: usize,
skipped: usize,
}
/// The comparison behind [`agent_config_pending`]: pending only when both
/// commits are known and differ.
fn config_pending(pinned: Option<&str>, deployed: Option<&str>) -> bool {
match (pinned, deployed) {
(Some(pinned), Some(deployed)) => pinned != deployed.trim(),
_ => false,
}
/// Whether an agent needs no rebuild this pass: nothing is pending, a rebuild
/// is already queued, or this process already attempted the declaration
/// (`attempted`) and the periodic pass does not retry it.
fn is_fresh(pending: Option<&str>, attempted: Option<&str>, rebuild_live: bool) -> bool {
pending.is_none() || rebuild_live || pending == attempted
}
/// Boot reconcile (see the module doc): classify every agent by rev
/// freshness + persisted `wanted` intent, and submit one `Boot` DAG holding a
/// graceful rebuild subgraph per stale wanted-up agent and a `Reconcile` per
/// agent whose observed power state drifted from `wanted`. Returns Ok even
/// if some submissions failed.
pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
/// Classify every agent by what the swarm declares + persisted `wanted`
/// intent. `attempted` holds the declaration this process last rebuilt each
/// agent for; the agents it adds to `Plan::rebuilds` are recorded in it.
async fn classify(
coord: &Arc<Coordinator>,
attempted: &mut HashMap<String, String>,
) -> Option<Plan> {
let containers = match lifecycle::list().await {
Ok(c) => c,
Err(e) => {
tracing::warn!(error = ?e, "boot reconcile: nixos-container list failed");
return Ok(());
tracing::warn!(error = ?e, "auto-update: nixos-container list failed");
return None;
}
};
let pin = crate::meta_rev::current_hyperhive_pin().await;
// Resolve container names to logical agent names, then sort. The
// parent field this used to depth-sort by is gone; with no
// hierarchy left to respect, alphabetical is the whole order — and it
@ -91,15 +95,15 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
.collect();
logical_names.sort();
// Classify. `get_or_seed` doubles as the one-time migration: an
// `get_or_seed` doubles as the one-time migration: an
// agent without an `agent_power` row is seeded from its observed
// state (running ⇒ Up), after which the DB is authoritative.
// stale ∧ wanted=Up → sweep rebuild. The bool is the agent's observed
// stale ∧ wanted=Up → rebuild. The bool is the agent's observed
// running state, used to order running agents first before submit.
let mut fanout: Vec<(String, bool)> = Vec::new();
let mut rebuilds: Vec<(String, bool)> = Vec::new();
let mut drifted: Vec<String> = Vec::new(); // fresh ∧ wanted≠observed → reconcile
let mut n_deferred = 0usize;
let mut n_skipped = 0usize;
let mut deferred = 0usize;
let mut skipped = 0usize;
for name in &logical_names {
let running = lifecycle::is_running(name).await;
let wanted = match coord.power.get_or_seed(name, running) {
@ -109,19 +113,28 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
None
}
};
let stored = std::fs::read_to_string(crate::paths::applied_rev_marker(name)).ok();
let fresh = crate::meta_rev::marker_is_fresh(pin.as_deref(), stored.as_deref());
let pending = crate::attic::closure::pending_path(name).await;
let fresh = is_fresh(
pending.as_deref(),
attempted.get(name).map(String::as_str),
coord.job_queue.rebuild_live(name),
);
match boot_action(wanted, fresh, running) {
BootAction::Rebuild => fanout.push((name.clone(), running)),
BootAction::Rebuild => {
if let Some(path) = pending {
attempted.insert(name.clone(), path);
}
rebuilds.push((name.clone(), running));
}
BootAction::Defer { reconcile } => {
n_deferred += 1;
tracing::debug!(%name, "boot reconcile: stale but offline — deferring rebuild to on-start");
deferred += 1;
tracing::debug!(%name, "auto-update: stale but offline — deferring rebuild to on-start");
if reconcile {
drifted.push(name.clone());
}
}
BootAction::Skip { reconcile } => {
n_skipped += 1;
skipped += 1;
if reconcile {
drifted.push(name.clone());
}
@ -130,40 +143,79 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
}
}
tracing::info!(
total = containers.len(),
rebuilds = fanout.len(),
reconciles = drifted.len(),
deferred = n_deferred,
up_to_date = n_skipped,
"boot reconcile"
);
// Rebuild running agents first. All fanout entries are wanted=Up;
// among them, warm the live/serving agents onto the fresh config before the
// Rebuild running agents first. All entries are wanted=Up;
// among them, warm the live/serving agents onto the declared closure before the
// stopped-but-wanted-up ones so the scarce build slots hit uptime-critical
// agents first. Stable sort keeps the alphabetical order within each
// running/stopped group. The `drifted` reconciles aren't sorted
// — they hold no build slot and run concurrently, so their order is moot.
fanout.sort_by_key(|(_, running)| !running);
let fanout: Vec<String> = fanout.into_iter().map(|(name, _)| name).collect();
rebuilds.sort_by_key(|(_, running)| !running);
Some(Plan {
rebuilds: rebuilds.into_iter().map(|(name, _)| name).collect(),
drifted,
deferred,
skipped,
})
}
submit_boot_tree(&coord, &fanout, drifted, n_deferred, n_skipped);
/// Auto-update (see the module doc): the boot pass, then a rebuild-only pass
/// every [`PASS_INTERVAL`] until shutdown. Returns Ok even if some submissions
/// failed.
pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
let mut shutdown = coord.shutdown_rx();
let mut attempted = HashMap::new();
if let Some(plan) = classify(&coord, &mut attempted).await {
tracing::info!(
total = plan.rebuilds.len() + plan.deferred + plan.skipped,
rebuilds = plan.rebuilds.len(),
reconciles = plan.drifted.len(),
deferred = plan.deferred,
up_to_date = plan.skipped,
"boot reconcile"
);
submit_boot_tree(
&coord,
&plan.rebuilds,
plan.drifted,
plan.deferred,
plan.skipped,
);
}
submit_startup_sweep_nodes(&coord);
Ok(())
loop {
tokio::select! {
() = tokio::time::sleep(PASS_INTERVAL) => {}
_ = shutdown.changed() => return Ok(()),
}
if let Some(plan) = classify(&coord, &mut attempted).await
&& !plan.rebuilds.is_empty()
{
tracing::info!(
rebuilds = plan.rebuilds.len(),
"auto-update: the swarm declared new closures"
);
submit_boot_tree(
&coord,
&plan.rebuilds,
Vec::new(),
plan.deferred,
plan.skipped,
);
}
}
}
/// What the boot sweep does with one agent.
#[derive(Debug, PartialEq, Eq)]
enum BootAction {
/// Stale ∧ wanted up: rebuild from the pinned `internal/meta`. The DAG's
/// Stale ∧ wanted up: rebuild to the swarm-declared closure. The DAG's
/// tail `Reconcile` brings the agent (back) up, covering both the
/// running-stale and stopped-but-wanted-up cases.
Rebuild,
/// Stale ∧ wanted offline: no boot-time nix work — the start submit
/// path upgrades a stale start to a rebuild.
Defer { reconcile: bool },
/// Already on the current rev.
/// Already running the declared closure, or nothing to deploy.
Skip { reconcile: bool },
/// The intent could not be read.
Unreadable,
@ -290,7 +342,7 @@ fn submit_boot_tree(
#[cfg(test)]
mod tests {
use super::{BootAction, boot_action, config_pending};
use super::{BootAction, boot_action, is_fresh};
use crate::power::Wanted;
// The regression the `Unreadable` variant exists for: the arm used to
@ -357,14 +409,46 @@ mod tests {
);
}
/// A changed declaration is rebuilt with no operator step, and with no read
/// of the forge: the boot pass reaches `Rebuild` from the record and marker
/// alone.
#[test]
fn config_is_pending_only_when_the_pins_differ() {
assert!(config_pending(Some("bbb"), Some("aaa")));
fn a_changed_declaration_for_a_wanted_up_agent_is_rebuilt_unprompted() {
let fresh = is_fresh(Some("/nix/store/new"), None, false);
assert_eq!(
boot_action(Some(Wanted::Up), fresh, true),
BootAction::Rebuild
);
let fresh = is_fresh(None, None, false);
assert_eq!(
boot_action(Some(Wanted::Up), fresh, true),
BootAction::Skip { reconcile: false },
"control: nothing declared, nothing rebuilt"
);
}
#[test]
fn a_declaration_is_attempted_once_and_never_beside_a_live_rebuild() {
assert!(is_fresh(None, None, false), "nothing pending");
assert!(
!config_pending(Some("aaa"), Some("aaa\n")),
"marker newline"
is_fresh(None, Some("/nix/store/b"), false),
"nothing pending now, whatever was attempted"
);
assert!(
!is_fresh(Some("/nix/store/b"), None, false),
"control: a new declaration"
);
assert!(
is_fresh(Some("/nix/store/b"), None, true),
"a rebuild is queued"
);
assert!(
is_fresh(Some("/nix/store/b"), Some("/nix/store/b"), false),
"already attempted"
);
assert!(
!is_fresh(Some("/nix/store/c"), Some("/nix/store/b"), false),
"a newer declaration is attempted again"
);
assert!(!config_pending(None, Some("aaa")), "archetype agent");
assert!(!config_pending(Some("aaa"), None), "never built");
}
}