feat(#3124): converge the hive onto the agent set the swarm declares
The deploy event is a nudge with no second path: core NATS is at-most-once, so a hive that was down when the controller published simply never learns that an agent is meant to exist here. This adds the repair path — one boot-time DAG node that reads this hive's own key in the `hive-wanted` bucket and converges the agents it names. Two semantics settled on the issue thread, and both are places where a plausible implementation is the wrong one: - **Absence is not a deletion order.** No bucket, no key, or an agent the value does not name all mean the controller has said nothing. Swarm-side lifecycle does not yet cover agents that predate it, so "converge to exactly this set" would tear down every agent the swarm has not adopted. `plan` only ever inspects the agents a declaration names. - **An unrecognised state is inert.** `AgentState` is an open enum: a value this build cannot read deserialises into `Unrecognised` and is left alone. A closed enum would force "not `Up`" onto a state like `paused`, so a controller that learned a new value would take agents down on every hive not yet updated. Divergence is measured against the hive's **stored power intent**, not the container's observed running state — an agent that is down while its intent says `Up` is already the boot reconcile's work, and a loop reading `is_running` would insert a start DAG behind that reconcile's back on every boot. A hive that already agrees with its declaration queues nothing at all. `queue_first_deploy` is extracted from the deploy-event path rather than open-coded here, for the power-intent seed: without it `first_deploy`'s tail `Reconcile` seeds `Wanted` from a container that exists but has not started yet, which locks the agent to `Offline` on its first reconcile. The read is authorised as-is: `store.get` takes async-nats' direct-get arm (the KV bucket is created with `allow_direct`), which is exactly the `$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.<hive>` subject `swarm-nats-auth` grants a hive. The fallback subject is not granted, and a refused NATS request surfaces as a timeout rather than an error. Nothing writes the bucket yet — the controller-side writer is the other half of #3124, so this does not close it.
This commit is contained in:
parent
8cba57e01c
commit
37f3c63eeb
7 changed files with 526 additions and 34 deletions
|
|
@ -152,6 +152,7 @@ pub(super) async fn run_node(
|
||||||
NodeKind::MatrixSweep => run_matrix_sweep().await,
|
NodeKind::MatrixSweep => run_matrix_sweep().await,
|
||||||
NodeKind::WebhookRegister => run_webhook_register().await,
|
NodeKind::WebhookRegister => run_webhook_register().await,
|
||||||
NodeKind::KnowledgePull => run_knowledge_pull(coord).await,
|
NodeKind::KnowledgePull => run_knowledge_pull(coord).await,
|
||||||
|
NodeKind::WantedPull => run_wanted_pull(coord).await,
|
||||||
};
|
};
|
||||||
(builder, result)
|
(builder, result)
|
||||||
}
|
}
|
||||||
|
|
@ -224,6 +225,15 @@ async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
|
||||||
crate::workers::knowledge::pull(coord).await
|
crate::workers::knowledge::pull(coord).await
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Boot-time swarm wanted-state pull as a DAG node — see
|
||||||
|
/// [`NodeKind::WantedPull`]. "The controller has declared nothing" is a
|
||||||
|
/// successful outcome the worker logs, so an error reaching here means the
|
||||||
|
/// read itself failed and this hive is running blind to its declaration —
|
||||||
|
/// which is exactly the state worth seeing on the dashboard.
|
||||||
|
async fn run_wanted_pull(coord: &Arc<Coordinator>) -> Result<()> {
|
||||||
|
crate::workers::wanted::pull(coord).await
|
||||||
|
}
|
||||||
|
|
||||||
/// Resolve the DAG's approval row the way this node's own `outcome` says.
|
/// Resolve the DAG's approval row the way this node's own `outcome` says.
|
||||||
///
|
///
|
||||||
/// Nothing is inspected: a template emits one of these per outcome, each edged to
|
/// Nothing is inspected: a template emits one of these per outcome, each edged to
|
||||||
|
|
|
||||||
|
|
@ -345,6 +345,14 @@ pub enum NodeKind {
|
||||||
/// [`NodeKind::MatrixSweep`]: the periodic hourly re-pull stays a
|
/// [`NodeKind::MatrixSweep`]: the periodic hourly re-pull stays a
|
||||||
/// background loop, only the boot-time instance is a DAG node.
|
/// background loop, only the boot-time instance is a DAG node.
|
||||||
KnowledgePull,
|
KnowledgePull,
|
||||||
|
/// One-shot boot-time pull of the agent set the swarm controller declares
|
||||||
|
/// for this hive (`wanted::pull`), converging the agents it names.
|
||||||
|
///
|
||||||
|
/// Unlike the three above there is **no** background loop behind this one:
|
||||||
|
/// boot is the whole cadence. The per-agent fast path is the deploy event
|
||||||
|
/// (`swarm_status`), and this is what repairs a missed one. Agentless — it
|
||||||
|
/// reads the whole declaration, not one agent.
|
||||||
|
WantedPull,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// How a hive-c0re node describes itself to a generic graph viewer.
|
/// How a hive-c0re node describes itself to a generic graph viewer.
|
||||||
|
|
@ -424,6 +432,7 @@ impl NodeKind {
|
||||||
NodeKind::MatrixSweep => "matrix_sweep",
|
NodeKind::MatrixSweep => "matrix_sweep",
|
||||||
NodeKind::WebhookRegister => "webhook_register",
|
NodeKind::WebhookRegister => "webhook_register",
|
||||||
NodeKind::KnowledgePull => "knowledge_pull",
|
NodeKind::KnowledgePull => "knowledge_pull",
|
||||||
|
NodeKind::WantedPull => "wanted_pull",
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -467,7 +476,8 @@ impl NodeKind {
|
||||||
| NodeKind::ForgeSweep
|
| NodeKind::ForgeSweep
|
||||||
| NodeKind::MatrixSweep
|
| NodeKind::MatrixSweep
|
||||||
| NodeKind::WebhookRegister
|
| NodeKind::WebhookRegister
|
||||||
| NodeKind::KnowledgePull => "",
|
| NodeKind::KnowledgePull
|
||||||
|
| NodeKind::WantedPull => "",
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -141,9 +141,8 @@ pub fn spawn(
|
||||||
/// rebuild insert the operator's own verb makes.
|
/// rebuild insert the operator's own verb makes.
|
||||||
///
|
///
|
||||||
/// Only the deploy event carries a payload, and only the agent name: its
|
/// Only the deploy event carries a payload, and only the agent name: its
|
||||||
/// subject already names the hive, so this one listens on its own rather than
|
/// subject already names the hive, so it listens on its own rather than a
|
||||||
/// filtering a swarm-wide feed. Even then it is a trigger, never the config,
|
/// swarm-wide feed. Even then it is a trigger, never the config git owns.
|
||||||
/// which git owns.
|
|
||||||
///
|
///
|
||||||
/// # A missed message costs the two events very differently
|
/// # A missed message costs the two events very differently
|
||||||
///
|
///
|
||||||
|
|
@ -151,11 +150,12 @@ pub fn spawn(
|
||||||
/// daemon pulls at startup regardless, and the webhook it replaced was lost
|
/// daemon pulls at startup regardless, and the webhook it replaced was lost
|
||||||
/// identically when a hive was down.
|
/// identically when a hive was down.
|
||||||
///
|
///
|
||||||
/// ⚠️ **The deploy event has no such second path** — nothing else would ever
|
/// The deploy event's second path is [`crate::workers::wanted`], which reads
|
||||||
/// tell this hive to build that agent. Its backstop is the hive-side reconcile
|
/// the whole declared set at boot and creates the agents this hive lacks — so
|
||||||
/// loop ("hives pull and self-update"); until that exists this is a nudge with
|
/// a missed **first** deploy repairs itself. A missed **rebuild** does not:
|
||||||
/// no safety net. Not papered over with `JetStream`: durability on one subject
|
/// the declaration names agents, not revisions, so that falls to the boot
|
||||||
/// looks like a fix while the desired state still lives only in a message.
|
/// reconcile noticing drift. Still not `JetStream`: durability on one subject
|
||||||
|
/// looks like a fix while the desired state lives only in a message.
|
||||||
///
|
///
|
||||||
/// ⚠️ **A refused subscription is indistinguishable from a quiet one.** NATS
|
/// ⚠️ **A refused subscription is indistinguishable from a quiet one.** NATS
|
||||||
/// reports an authorization violation asynchronously on the connection, not as
|
/// reports an authorization violation asynchronously on the connection, not as
|
||||||
|
|
@ -275,27 +275,7 @@ async fn handle_deploy_request(
|
||||||
.job_queue
|
.job_queue
|
||||||
.insert_job(|b| crate::job_queue::templates::rebuild(b, &agent, true))
|
.insert_job(|b| crate::job_queue::templates::rebuild(b, &agent, true))
|
||||||
} else {
|
} else {
|
||||||
// First deploy. The swarm has already created the identity, the forge
|
queue_first_deploy(coord, &agent)
|
||||||
// repo and its config; what is left is hive-local, and there is no
|
|
||||||
// approval to wait on because the operator's click at swarm level is
|
|
||||||
// the authorisation.
|
|
||||||
//
|
|
||||||
// Seed the power intent to `Up` before queuing, same as
|
|
||||||
// `actions::approve`'s `ApprovalKind::Spawn` arm does for the
|
|
||||||
// operator-approved path: `first_deploy`'s tail `Reconcile` node
|
|
||||||
// seeds `Wanted` from the container's *currently observed* running
|
|
||||||
// state when no row exists yet (`power::Store::get_or_seed`), and at
|
|
||||||
// that point in a first deploy the container is freshly created but
|
|
||||||
// not yet started — so an unseeded row locks the agent's wanted
|
|
||||||
// state to `Offline` on its very first reconcile, and `Reconcile`
|
|
||||||
// never emits the `Start` node. Setting the row up front here closes
|
|
||||||
// that window the same way the approval path already does.
|
|
||||||
if let Err(e) = coord.power.set(&agent, crate::power::Wanted::Up) {
|
|
||||||
tracing::warn!(%agent, error = ?e, "agent_power: seed on swarm first-deploy failed");
|
|
||||||
}
|
|
||||||
coord
|
|
||||||
.job_queue
|
|
||||||
.insert_job(|b| crate::job_queue::templates::first_deploy(b, &agent))
|
|
||||||
};
|
};
|
||||||
match inserted {
|
match inserted {
|
||||||
Ok(_) => {
|
Ok(_) => {
|
||||||
|
|
@ -306,6 +286,35 @@ async fn handle_deploy_request(
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Queue the first deploy of an agent this hive does not have yet: seed its
|
||||||
|
/// power intent, then insert the DAG.
|
||||||
|
///
|
||||||
|
/// The swarm has already created the identity, the forge repo and its config;
|
||||||
|
/// what is left is hive-local, and there is no approval to wait on because the
|
||||||
|
/// operator's click at swarm level is the authorisation.
|
||||||
|
///
|
||||||
|
/// Shared with [`crate::workers::wanted`] for the **seeding**, not the insert.
|
||||||
|
/// `first_deploy`'s tail `Reconcile` seeds `Wanted` from the container's
|
||||||
|
/// currently observed running state when no row exists yet
|
||||||
|
/// (`power::Store::get_or_seed`), and at that point the container is freshly
|
||||||
|
/// created but not started — so an unseeded row locks the agent to `Offline`
|
||||||
|
/// on its very first reconcile and `Reconcile` never emits the `Start` node.
|
||||||
|
/// Setting the row up front closes that window, the same way
|
||||||
|
/// `actions::approve`'s `ApprovalKind::Spawn` arm does for the
|
||||||
|
/// operator-approved path. A second caller open-coding the insert would lose
|
||||||
|
/// exactly that, and the agent would come up stopped for no visible reason.
|
||||||
|
pub(crate) fn queue_first_deploy(
|
||||||
|
coord: &std::sync::Arc<crate::coordinator::Coordinator>,
|
||||||
|
agent: &str,
|
||||||
|
) -> Result<Vec<hive_jobq::NodeId>> {
|
||||||
|
if let Err(e) = coord.power.set(agent, crate::power::Wanted::Up) {
|
||||||
|
tracing::warn!(%agent, error = ?e, "agent_power: seed on swarm first-deploy failed");
|
||||||
|
}
|
||||||
|
coord
|
||||||
|
.job_queue
|
||||||
|
.insert_job(|b| crate::job_queue::templates::first_deploy(b, agent))
|
||||||
|
}
|
||||||
|
|
||||||
/// Offer one snapshot: this hive's current readiness, under its own key.
|
/// Offer one snapshot: this hive's current readiness, under its own key.
|
||||||
async fn publish(client: &async_nats::Client, hive: &str) -> Result<()> {
|
async fn publish(client: &async_nats::Client, hive: &str) -> Result<()> {
|
||||||
// An unconnected client does not fail a JetStream request, it hangs
|
// An unconnected client does not fail a JetStream request, it hangs
|
||||||
|
|
|
||||||
|
|
@ -317,13 +317,14 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Submit the boot-time forge/matrix/webhook/knowledge sweeps as DAG nodes —
|
/// Submit the boot-time forge/matrix/webhook/knowledge/wanted-state sweeps as
|
||||||
/// `ForgeSweep`, `MatrixSweep`, `WebhookRegister`, `KnowledgePull`. Unlike
|
/// DAG nodes — `ForgeSweep`, `MatrixSweep`, `WebhookRegister`,
|
||||||
|
/// `KnowledgePull`, `WantedPull`. Unlike
|
||||||
/// [`submit_boot_tree`] this runs on **every** boot, quiet or not: these
|
/// [`submit_boot_tree`] this runs on **every** boot, quiet or not: these
|
||||||
/// aren't config-drift work, they're startup housekeeping that always needs
|
/// aren't config-drift work, they're startup housekeeping that always needs
|
||||||
/// to happen, and the point of moving them here is exactly so they show up
|
/// to happen, and the point of moving them here is exactly so they show up
|
||||||
/// as real work on the dashboard instead of an invisible `tokio::spawn` that
|
/// as real work on the dashboard instead of an invisible `tokio::spawn` that
|
||||||
/// only surfaces on failure. Four independent, build-slot- and lease-exempt
|
/// only surfaces on failure. Five independent, build-slot- and lease-exempt
|
||||||
/// roots — no dependency edges between them, matching the existing
|
/// roots — no dependency edges between them, matching the existing
|
||||||
/// `Reconcile`-root pattern in [`boot_nodes`].
|
/// `Reconcile`-root pattern in [`boot_nodes`].
|
||||||
fn submit_startup_sweep_nodes(coord: &Arc<Coordinator>) {
|
fn submit_startup_sweep_nodes(coord: &Arc<Coordinator>) {
|
||||||
|
|
@ -334,6 +335,7 @@ fn submit_startup_sweep_nodes(coord: &Arc<Coordinator>) {
|
||||||
let _ = b.node(NodeKind::MatrixSweep);
|
let _ = b.node(NodeKind::MatrixSweep);
|
||||||
let _ = b.node(NodeKind::WebhookRegister);
|
let _ = b.node(NodeKind::WebhookRegister);
|
||||||
let _ = b.node(NodeKind::KnowledgePull);
|
let _ = b.node(NodeKind::KnowledgePull);
|
||||||
|
let _ = b.node(NodeKind::WantedPull);
|
||||||
Vec::new()
|
Vec::new()
|
||||||
}) {
|
}) {
|
||||||
tracing::warn!(error = ?e, "boot: startup sweep DAG insert failed");
|
tracing::warn!(error = ?e, "boot: startup sweep DAG insert failed");
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,8 @@
|
||||||
//! Background tasks and periodic sweeps: crash/login watcher, the
|
//! Background tasks and periodic sweeps: crash/login watcher, the
|
||||||
//! scheduled-prompt delivery loop, boot-time auto-update
|
//! scheduled-prompt delivery loop, boot-time auto-update
|
||||||
//! reconcile, the agent-sockets.json writer loop, the MCP socket listener
|
//! reconcile, the agent-sockets.json writer loop, the MCP socket listener
|
||||||
//! reconcile loop, and knowledge-repo sync. Each submodule is re-exported
|
//! reconcile loop, knowledge-repo sync, and the swarm wanted-state pull.
|
||||||
|
//! Each submodule is re-exported
|
||||||
//! at the crate root, so `crate::crash_watch::…` etc. keep working unchanged.
|
//! at the crate root, so `crate::crash_watch::…` etc. keep working unchanged.
|
||||||
|
|
||||||
pub mod agent_sockets;
|
pub mod agent_sockets;
|
||||||
|
|
@ -10,3 +11,4 @@ pub mod crash_watch;
|
||||||
pub mod knowledge;
|
pub mod knowledge;
|
||||||
pub mod mcp_sockets;
|
pub mod mcp_sockets;
|
||||||
pub mod scheduled_prompts_worker;
|
pub mod scheduled_prompts_worker;
|
||||||
|
pub mod wanted;
|
||||||
|
|
|
||||||
367
hive-c0re/src/workers/wanted.rs
Normal file
367
hive-c0re/src/workers/wanted.rs
Normal file
|
|
@ -0,0 +1,367 @@
|
||||||
|
//! Converging this hive onto the agent set the swarm controller declares.
|
||||||
|
//!
|
||||||
|
//! The controller writes one key per hive into the `hive-wanted` bucket (see
|
||||||
|
//! [`swarm_queue_client::wanted`]); this reads its own and acts on it. It is
|
||||||
|
//! the **repair** path, not the fast one: a deploy event
|
||||||
|
//! ([`crate::swarm_status`]) is core NATS, so a hive that was down never hears
|
||||||
|
//! it, and until this existed that left the hive on its old shape with nothing
|
||||||
|
//! looking wrong.
|
||||||
|
//!
|
||||||
|
//! # Two things a declaration does not mean
|
||||||
|
//!
|
||||||
|
//! **Absence is not a deletion order.** An agent missing from the value — or
|
||||||
|
//! no value, or no bucket — is one the controller has said nothing about.
|
||||||
|
//! Swarm-side lifecycle does not yet cover agents that predate it, so
|
||||||
|
//! converging "to exactly this set" would tear down every agent the swarm has
|
||||||
|
//! not adopted.
|
||||||
|
//!
|
||||||
|
//! **An unrecognised state is inert**, so a controller that learns a new state
|
||||||
|
//! does not take agents down on hives that have not been updated yet.
|
||||||
|
//!
|
||||||
|
//! # What "diverged" is measured against
|
||||||
|
//!
|
||||||
|
//! The hive's **stored power intent**, not the container's observed running
|
||||||
|
//! state. An agent down while its intent says `Up` is already the boot
|
||||||
|
//! reconcile's work; a loop reading `is_running` would insert a start DAG
|
||||||
|
//! behind that reconcile's back on every boot.
|
||||||
|
//!
|
||||||
|
//! One consequence, deliberate: a declaration outranks a local stop. An agent
|
||||||
|
//! this hive is declared to keep `Up` comes back at the next boot, because the
|
||||||
|
//! controller owns that decision.
|
||||||
|
|
||||||
|
use std::collections::{BTreeMap, BTreeSet};
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
use swarm_queue_client::wanted::{AgentState, HiveWanted};
|
||||||
|
|
||||||
|
use crate::coordinator::Coordinator;
|
||||||
|
use crate::power::Wanted;
|
||||||
|
|
||||||
|
/// Read this hive's declaration and converge the agents it names.
|
||||||
|
///
|
||||||
|
/// `Ok(())` covers three quiet cases that are **not** an empty declaration: no
|
||||||
|
/// hive name, no bucket, no key. Each logs its own line, because a loop that
|
||||||
|
/// silently does nothing and one that is working look identical otherwise.
|
||||||
|
/// Anything else — an unreachable queue, an undecodable value, an agent list
|
||||||
|
/// this hive cannot enumerate — is an error, so the node fails visibly on the
|
||||||
|
/// dashboard instead of reading as "nothing was declared".
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
/// Propagates a failed queue connect, KV read, decode, or agent enumeration.
|
||||||
|
pub async fn pull(coord: &Arc<Coordinator>) -> Result<()> {
|
||||||
|
let Some(hive) = crate::container_view::hive_swarm_names().0 else {
|
||||||
|
tracing::debug!("wanted state: HYPERHIVE_HIVE_NAME unset; no key to read");
|
||||||
|
return Ok(());
|
||||||
|
};
|
||||||
|
let Some(client) = crate::swarm_queue::client().await else {
|
||||||
|
// Absent or failed — either way already reported by the shared
|
||||||
|
// connector (an `info` log or a `swarm_queue_config` banner).
|
||||||
|
return Ok(());
|
||||||
|
};
|
||||||
|
// An unconnected client does not fail a JetStream request, it hangs on it.
|
||||||
|
// A hung node is worse than a failed one: the dashboard shows it running.
|
||||||
|
swarm_queue_client::ensure_connected(&client)?;
|
||||||
|
|
||||||
|
let Some(store) = swarm_queue_client::wanted::open_read_only(&client).await else {
|
||||||
|
tracing::info!(%hive, "wanted state: no bucket yet; nothing has been declared");
|
||||||
|
return Ok(());
|
||||||
|
};
|
||||||
|
let Some(raw) = store
|
||||||
|
.get(&hive)
|
||||||
|
.await
|
||||||
|
.context("reading this hive's wanted state")?
|
||||||
|
else {
|
||||||
|
tracing::info!(%hive, "wanted state: no key for this hive; nothing has been declared");
|
||||||
|
return Ok(());
|
||||||
|
};
|
||||||
|
let declared: HiveWanted =
|
||||||
|
serde_json::from_slice(&raw).context("decoding this hive's wanted state")?;
|
||||||
|
|
||||||
|
converge(coord, &declared).await
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Queue whatever the declaration asks for and this hive is not already doing.
|
||||||
|
async fn converge(coord: &Arc<Coordinator>, declared: &HiveWanted) -> Result<()> {
|
||||||
|
// Fail closed, for the reason the deploy event's own arm gives: without
|
||||||
|
// the list, "this hive does not have that agent" cannot be told from
|
||||||
|
// "this hive cannot see its agents", and acting on the guess creates a
|
||||||
|
// container over one that already exists.
|
||||||
|
let present: BTreeSet<String> = crate::lifecycle::agents_for_meta_listing()
|
||||||
|
.await
|
||||||
|
.context("enumerating this hive's agents")?
|
||||||
|
.into_iter()
|
||||||
|
.map(|spec| spec.name)
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let mut intents = BTreeMap::new();
|
||||||
|
for agent in declared.agents.keys() {
|
||||||
|
match coord.power.get(agent) {
|
||||||
|
Ok(intent) => {
|
||||||
|
intents.insert(agent.clone(), intent);
|
||||||
|
}
|
||||||
|
// Left out of the map, which is what makes `plan` skip it.
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!(%agent, error = ?e, "wanted state: cannot read power intent; skipping");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let plan = plan(declared, &present, &intents);
|
||||||
|
tracing::info!(
|
||||||
|
declared = declared.agents.len(),
|
||||||
|
deploy = plan.deploy.len(),
|
||||||
|
start = plan.start.len(),
|
||||||
|
stop = plan.stop.len(),
|
||||||
|
"wanted state: converging"
|
||||||
|
);
|
||||||
|
|
||||||
|
let mut queued = false;
|
||||||
|
for agent in &plan.deploy {
|
||||||
|
match crate::swarm_status::queue_first_deploy(coord, agent) {
|
||||||
|
Ok(_) => queued = true,
|
||||||
|
// Per agent rather than the whole sweep: one agent that cannot be
|
||||||
|
// queued is not a reason to leave the others undeclared.
|
||||||
|
Err(e) => tracing::warn!(%agent, error = %e, "wanted state: queueing a deploy failed"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !plan.start.is_empty() {
|
||||||
|
crate::job_queue::power::start_many(coord, &plan.start)
|
||||||
|
.await
|
||||||
|
.context("queueing the declared starts")?;
|
||||||
|
}
|
||||||
|
if !plan.stop.is_empty() {
|
||||||
|
crate::job_queue::power::stop_many(coord, &plan.stop, true)
|
||||||
|
.await
|
||||||
|
.context("queueing the declared stops")?;
|
||||||
|
}
|
||||||
|
if queued {
|
||||||
|
// The power ops emit their own; the first deploys do not.
|
||||||
|
coord.emit_rebuild_queue_snapshot();
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Everything one declaration asks this hive to queue.
|
||||||
|
#[derive(Debug, Default, PartialEq, Eq)]
|
||||||
|
struct Plan {
|
||||||
|
deploy: Vec<String>,
|
||||||
|
start: Vec<String>,
|
||||||
|
stop: Vec<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Turn one declaration into that plan — pure, so what the loop does with a
|
||||||
|
/// given declaration is testable without a queue, a container or a store.
|
||||||
|
///
|
||||||
|
/// `intents` is the hive's stored power intent per agent, and an agent
|
||||||
|
/// **missing from the map** is one whose intent could not be read: it is
|
||||||
|
/// skipped. That is not `Some(None)` — "no row yet" is a reading and gets
|
||||||
|
/// converged, "the read failed" is not and would be a guess.
|
||||||
|
///
|
||||||
|
/// An agent the declaration does not name reaches none of the three lists.
|
||||||
|
/// That is the whole of "absence is not a deletion order": the agents this
|
||||||
|
/// hive runs that the swarm has not adopted are never even inspected.
|
||||||
|
fn plan(
|
||||||
|
declared: &HiveWanted,
|
||||||
|
present: &BTreeSet<String>,
|
||||||
|
intents: &BTreeMap<String, Option<Wanted>>,
|
||||||
|
) -> Plan {
|
||||||
|
let mut plan = Plan::default();
|
||||||
|
for (agent, decl) in &declared.agents {
|
||||||
|
let Some(intent) = intents.get(agent) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
match decide(&decl.state, present.contains(agent), *intent) {
|
||||||
|
Converge::Deploy => plan.deploy.push(agent.clone()),
|
||||||
|
Converge::Start => plan.start.push(agent.clone()),
|
||||||
|
Converge::Stop => plan.stop.push(agent.clone()),
|
||||||
|
Converge::Nothing => {}
|
||||||
|
Converge::Inert => {
|
||||||
|
tracing::info!(
|
||||||
|
%agent, state = ?decl.state,
|
||||||
|
"wanted state: state not recognised by this build; leaving the agent alone"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
plan
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What the loop will do about one declared agent.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
enum Converge {
|
||||||
|
/// Declared, and this hive has no container for it — the missed-deploy
|
||||||
|
/// repair.
|
||||||
|
Deploy,
|
||||||
|
Start,
|
||||||
|
Stop,
|
||||||
|
/// The hive already agrees with the declaration.
|
||||||
|
Nothing,
|
||||||
|
/// A state this build does not recognise.
|
||||||
|
Inert,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The whole decision, pure: no queue, no store, no container. The three
|
||||||
|
/// inputs are exactly what [`converge`] reads per agent, so the table below is
|
||||||
|
/// the behaviour rather than a model of it.
|
||||||
|
fn decide(state: &AgentState, present: bool, intent: Option<Wanted>) -> Converge {
|
||||||
|
match state {
|
||||||
|
// First, and whatever else is true of the agent.
|
||||||
|
AgentState::Unrecognised(_) => Converge::Inert,
|
||||||
|
AgentState::Up if !present => Converge::Deploy,
|
||||||
|
AgentState::Up => {
|
||||||
|
if intent == Some(Wanted::Up) {
|
||||||
|
Converge::Nothing
|
||||||
|
} else {
|
||||||
|
Converge::Start
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Declared offline and not here: creating a container in order to
|
||||||
|
// leave it stopped is not what the declaration asks for.
|
||||||
|
AgentState::Offline if !present => Converge::Nothing,
|
||||||
|
AgentState::Offline => {
|
||||||
|
if intent == Some(Wanted::Offline) {
|
||||||
|
Converge::Nothing
|
||||||
|
} else {
|
||||||
|
Converge::Stop
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use std::collections::{BTreeMap, BTreeSet};
|
||||||
|
|
||||||
|
use super::{Converge, Plan, decide, plan};
|
||||||
|
use crate::power::Wanted;
|
||||||
|
use swarm_queue_client::wanted::{AgentState, AgentWanted, HiveWanted};
|
||||||
|
|
||||||
|
fn unknown() -> AgentState {
|
||||||
|
AgentState::Unrecognised("paused".to_owned())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn declaration(agents: &[(&str, AgentState)]) -> HiveWanted {
|
||||||
|
HiveWanted {
|
||||||
|
agents: agents
|
||||||
|
.iter()
|
||||||
|
.map(|(name, state)| {
|
||||||
|
(
|
||||||
|
(*name).to_owned(),
|
||||||
|
AgentWanted {
|
||||||
|
state: state.clone(),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.collect(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn set(names: &[&str]) -> BTreeSet<String> {
|
||||||
|
names.iter().map(|n| (*n).to_owned()).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn intents(entries: &[(&str, Option<Wanted>)]) -> BTreeMap<String, Option<Wanted>> {
|
||||||
|
entries
|
||||||
|
.iter()
|
||||||
|
.map(|(name, intent)| ((*name).to_owned(), *intent))
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The property the whole migration rides on: this hive's own agents are
|
||||||
|
/// not converged, they are not looked at. `adopted` is the control — the
|
||||||
|
/// same call must still act on what the declaration *does* name, or this
|
||||||
|
/// passes on a loop that plans nothing at all.
|
||||||
|
#[test]
|
||||||
|
fn an_agent_the_declaration_does_not_name_is_never_touched() {
|
||||||
|
let declared = declaration(&[("adopted", AgentState::Up)]);
|
||||||
|
let planned = plan(
|
||||||
|
&declared,
|
||||||
|
&set(&["adopted", "legacy"]),
|
||||||
|
&intents(&[("adopted", Some(Wanted::Offline)), ("legacy", None)]),
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
planned,
|
||||||
|
Plan {
|
||||||
|
deploy: vec![],
|
||||||
|
start: vec!["adopted".to_owned()],
|
||||||
|
stop: vec![],
|
||||||
|
}
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// An intent that could not be read is skipped rather than treated as "no
|
||||||
|
/// row". `readable` is the control for the same call.
|
||||||
|
#[test]
|
||||||
|
fn an_unreadable_intent_skips_only_that_agent() {
|
||||||
|
let declared = declaration(&[("unreadable", AgentState::Up), ("readable", AgentState::Up)]);
|
||||||
|
let planned = plan(
|
||||||
|
&declared,
|
||||||
|
&set(&["unreadable", "readable"]),
|
||||||
|
&intents(&[("readable", None)]),
|
||||||
|
);
|
||||||
|
assert_eq!(planned.start, vec!["readable".to_owned()]);
|
||||||
|
assert!(planned.deploy.is_empty() && planned.stop.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_declared_agent_this_hive_does_not_have_is_deployed() {
|
||||||
|
assert_eq!(decide(&AgentState::Up, false, None), Converge::Deploy);
|
||||||
|
// Even holding an intent: a row can outlive its container.
|
||||||
|
assert_eq!(
|
||||||
|
decide(&AgentState::Up, false, Some(Wanted::Up)),
|
||||||
|
Converge::Deploy
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_hive_that_already_agrees_queues_nothing() {
|
||||||
|
assert_eq!(
|
||||||
|
decide(&AgentState::Up, true, Some(Wanted::Up)),
|
||||||
|
Converge::Nothing
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
decide(&AgentState::Offline, true, Some(Wanted::Offline)),
|
||||||
|
Converge::Nothing
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_disagreeing_intent_is_converged_both_ways() {
|
||||||
|
assert_eq!(
|
||||||
|
decide(&AgentState::Up, true, Some(Wanted::Offline)),
|
||||||
|
Converge::Start
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
decide(&AgentState::Offline, true, Some(Wanted::Up)),
|
||||||
|
Converge::Stop
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// No row is not agreement. An agent that predates the power store is
|
||||||
|
/// converged to what it was declared as, not left on whatever it happens
|
||||||
|
/// to be doing.
|
||||||
|
#[test]
|
||||||
|
fn an_agent_with_no_intent_row_is_converged() {
|
||||||
|
assert_eq!(decide(&AgentState::Up, true, None), Converge::Start);
|
||||||
|
assert_eq!(decide(&AgentState::Offline, true, None), Converge::Stop);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Declared offline, no container: the one absent case that must not
|
||||||
|
/// deploy. Its control is the `Up` case above, which must.
|
||||||
|
#[test]
|
||||||
|
fn an_absent_agent_declared_offline_is_left_absent() {
|
||||||
|
assert_eq!(decide(&AgentState::Offline, false, None), Converge::Nothing);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The version-skew arm: an unrecognised state is inert in every
|
||||||
|
/// combination, including the ones where a known state would act.
|
||||||
|
#[test]
|
||||||
|
fn an_unrecognised_state_is_inert_everywhere() {
|
||||||
|
for present in [true, false] {
|
||||||
|
for intent in [None, Some(Wanted::Up), Some(Wanted::Offline)] {
|
||||||
|
assert_eq!(decide(&unknown(), present, intent), Converge::Inert);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -32,6 +32,51 @@ use crate::Error;
|
||||||
/// for two deployments to disagree about which one that is.
|
/// for two deployments to disagree about which one that is.
|
||||||
pub const BUCKET: &str = "hive-wanted";
|
pub const BUCKET: &str = "hive-wanted";
|
||||||
|
|
||||||
|
/// One hive's whole declaration: the agents the controller names, and what it
|
||||||
|
/// wants of each. The value under `BUCKET`/`<hiveName>`.
|
||||||
|
///
|
||||||
|
/// The agents it does **not** name are not part of the declaration at all —
|
||||||
|
/// see the module docs. A hive converges the ones listed here and leaves the
|
||||||
|
/// rest alone.
|
||||||
|
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||||
|
pub struct HiveWanted {
|
||||||
|
/// Agent name → what the controller wants of it.
|
||||||
|
#[serde(default)]
|
||||||
|
pub agents: std::collections::BTreeMap<String, AgentWanted>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One agent's entry in a [`HiveWanted`].
|
||||||
|
///
|
||||||
|
/// A struct around one field rather than the bare state, because this is where
|
||||||
|
/// a later declaration grows — a config revision, a schedule — and a hive
|
||||||
|
/// built before that field existed keeps deserialising the whole document.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||||
|
pub struct AgentWanted {
|
||||||
|
/// The state to converge this agent to.
|
||||||
|
pub state: AgentState,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What the controller wants an agent to be — an **open** enum.
|
||||||
|
///
|
||||||
|
/// A value this build does not know deserialises into
|
||||||
|
/// [`AgentState::Unrecognised`], carried verbatim, rather than failing the
|
||||||
|
/// whole document or collapsing into a known variant. Both alternatives break
|
||||||
|
/// the same way: a closed enum forces "not `Up`" onto a state like `paused`,
|
||||||
|
/// so a controller that learns a new value would take agents down on every
|
||||||
|
/// hive that has not been updated yet.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||||
|
#[serde(rename_all = "snake_case")]
|
||||||
|
pub enum AgentState {
|
||||||
|
/// Exists on the hive and is running.
|
||||||
|
Up,
|
||||||
|
/// Exists on the hive and is not running.
|
||||||
|
Offline,
|
||||||
|
/// Anything else. Kept as written so a hive can say what it declined to
|
||||||
|
/// act on instead of logging that something unspecified was skipped.
|
||||||
|
#[serde(untagged)]
|
||||||
|
Unrecognised(String),
|
||||||
|
}
|
||||||
|
|
||||||
/// Open the wanted-state bucket for writing, creating it if nothing has yet.
|
/// Open the wanted-state bucket for writing, creating it if nothing has yet.
|
||||||
///
|
///
|
||||||
/// **Controller-side only.** `history: 1` because a hive converges to the
|
/// **Controller-side only.** `history: 1` because a hive converges to the
|
||||||
|
|
@ -78,3 +123,50 @@ pub async fn open_read_only(
|
||||||
let js = async_nats::jetstream::new(client.clone());
|
let js = async_nats::jetstream::new(client.clone());
|
||||||
js.get_key_value(BUCKET).await.ok()
|
js.get_key_value(BUCKET).await.ok()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::{AgentState, HiveWanted};
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn known_states_decode_and_an_unknown_one_stays_itself() {
|
||||||
|
let doc = r#"{"agents":{
|
||||||
|
"a":{"state":"up"},
|
||||||
|
"b":{"state":"offline"},
|
||||||
|
"c":{"state":"paused"}
|
||||||
|
}}"#;
|
||||||
|
let decoded: HiveWanted = serde_json::from_str(doc).expect("decodes");
|
||||||
|
assert_eq!(decoded.agents["a"].state, AgentState::Up);
|
||||||
|
assert_eq!(decoded.agents["b"].state, AgentState::Offline);
|
||||||
|
assert_eq!(
|
||||||
|
decoded.agents["c"].state,
|
||||||
|
AgentState::Unrecognised("paused".to_owned())
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_unrecognised_state_survives_a_round_trip() {
|
||||||
|
let doc = r#"{"agents":{"c":{"state":"paused"}}}"#;
|
||||||
|
let decoded: HiveWanted = serde_json::from_str(doc).expect("decodes");
|
||||||
|
let reserialised = serde_json::to_string(&decoded).expect("serialises");
|
||||||
|
assert_eq!(reserialised, r#"{"agents":{"c":{"state":"paused"}}}"#);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A field a newer controller adds must not cost an older hive the whole
|
||||||
|
/// document — that is the version skew the open enum exists for, one level
|
||||||
|
/// up.
|
||||||
|
#[test]
|
||||||
|
fn unknown_fields_are_ignored() {
|
||||||
|
let doc = r#"{"agents":{"a":{"state":"up","config_rev":"deadbeef"}},"epoch":3}"#;
|
||||||
|
let decoded: HiveWanted = serde_json::from_str(doc).expect("decodes");
|
||||||
|
assert_eq!(decoded.agents["a"].state, AgentState::Up);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The control for the two above: openness is about *values and fields*,
|
||||||
|
/// not about accepting anything. A declaration missing the one required
|
||||||
|
/// field is an error, so a malformed document cannot read as "no agents".
|
||||||
|
#[test]
|
||||||
|
fn a_missing_state_is_an_error() {
|
||||||
|
assert!(serde_json::from_str::<HiveWanted>(r#"{"agents":{"a":{}}}"#).is_err());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue