feat(#3124): converge the hive onto the agent set the swarm declares
The deploy event is a nudge with no second path: core NATS is at-most-once, so a hive that was down when the controller published simply never learns that an agent is meant to exist here. This adds the repair path — one boot-time DAG node that reads this hive's own key in the `hive-wanted` bucket and converges the agents it names. Two semantics settled on the issue thread, and both are places where a plausible implementation is the wrong one: - **Absence is not a deletion order.** No bucket, no key, or an agent the value does not name all mean the controller has said nothing. Swarm-side lifecycle does not yet cover agents that predate it, so "converge to exactly this set" would tear down every agent the swarm has not adopted. `plan` only ever inspects the agents a declaration names. - **An unrecognised state is inert.** `AgentState` is an open enum: a value this build cannot read deserialises into `Unrecognised` and is left alone. A closed enum would force "not `Up`" onto a state like `paused`, so a controller that learned a new value would take agents down on every hive not yet updated. Divergence is measured against the hive's **stored power intent**, not the container's observed running state — an agent that is down while its intent says `Up` is already the boot reconcile's work, and a loop reading `is_running` would insert a start DAG behind that reconcile's back on every boot. A hive that already agrees with its declaration queues nothing at all. `queue_first_deploy` is extracted from the deploy-event path rather than open-coded here, for the power-intent seed: without it `first_deploy`'s tail `Reconcile` seeds `Wanted` from a container that exists but has not started yet, which locks the agent to `Offline` on its first reconcile. The read is authorised as-is: `store.get` takes async-nats' direct-get arm (the KV bucket is created with `allow_direct`), which is exactly the `$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.<hive>` subject `swarm-nats-auth` grants a hive. The fallback subject is not granted, and a refused NATS request surfaces as a timeout rather than an error. Nothing writes the bucket yet — the controller-side writer is the other half of #3124, so this does not close it.
This commit is contained in:
atlas
parent
8cba57e01c
commit
37f3c63eeb
7 changed files with 526 additions and 34 deletions
|
|
@ -152,6 +152,7 @@ pub(super) async fn run_node(
|
|||
NodeKind::MatrixSweep => run_matrix_sweep().await,
|
||||
NodeKind::WebhookRegister => run_webhook_register().await,
|
||||
NodeKind::KnowledgePull => run_knowledge_pull(coord).await,
|
||||
NodeKind::WantedPull => run_wanted_pull(coord).await,
|
||||
};
|
||||
(builder, result)
|
||||
}
|
||||
|
|
@ -224,6 +225,15 @@ async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
|
|||
crate::workers::knowledge::pull(coord).await
|
||||
}
|
||||
|
||||
/// Boot-time swarm wanted-state pull as a DAG node — see
|
||||
/// [`NodeKind::WantedPull`]. "The controller has declared nothing" is a
|
||||
/// successful outcome the worker logs, so an error reaching here means the
|
||||
/// read itself failed and this hive is running blind to its declaration —
|
||||
/// which is exactly the state worth seeing on the dashboard.
|
||||
async fn run_wanted_pull(coord: &Arc<Coordinator>) -> Result<()> {
|
||||
crate::workers::wanted::pull(coord).await
|
||||
}
|
||||
|
||||
/// Resolve the DAG's approval row the way this node's own `outcome` says.
|
||||
///
|
||||
/// Nothing is inspected: a template emits one of these per outcome, each edged to
|
||||
|
|
|
|||
|
|
@ -345,6 +345,14 @@ pub enum NodeKind {
|
|||
/// [`NodeKind::MatrixSweep`]: the periodic hourly re-pull stays a
|
||||
/// background loop, only the boot-time instance is a DAG node.
|
||||
KnowledgePull,
|
||||
/// One-shot boot-time pull of the agent set the swarm controller declares
|
||||
/// for this hive (`wanted::pull`), converging the agents it names.
|
||||
///
|
||||
/// Unlike the three above there is **no** background loop behind this one:
|
||||
/// boot is the whole cadence. The per-agent fast path is the deploy event
|
||||
/// (`swarm_status`), and this is what repairs a missed one. Agentless — it
|
||||
/// reads the whole declaration, not one agent.
|
||||
WantedPull,
|
||||
}
|
||||
|
||||
/// How a hive-c0re node describes itself to a generic graph viewer.
|
||||
|
|
@ -424,6 +432,7 @@ impl NodeKind {
|
|||
NodeKind::MatrixSweep => "matrix_sweep",
|
||||
NodeKind::WebhookRegister => "webhook_register",
|
||||
NodeKind::KnowledgePull => "knowledge_pull",
|
||||
NodeKind::WantedPull => "wanted_pull",
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -467,7 +476,8 @@ impl NodeKind {
|
|||
| NodeKind::ForgeSweep
|
||||
| NodeKind::MatrixSweep
|
||||
| NodeKind::WebhookRegister
|
||||
| NodeKind::KnowledgePull => "",
|
||||
| NodeKind::KnowledgePull
|
||||
| NodeKind::WantedPull => "",
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -141,9 +141,8 @@ pub fn spawn(
|
|||
/// rebuild insert the operator's own verb makes.
|
||||
///
|
||||
/// Only the deploy event carries a payload, and only the agent name: its
|
||||
/// subject already names the hive, so this one listens on its own rather than
|
||||
/// filtering a swarm-wide feed. Even then it is a trigger, never the config,
|
||||
/// which git owns.
|
||||
/// subject already names the hive, so it listens on its own rather than a
|
||||
/// swarm-wide feed. Even then it is a trigger, never the config git owns.
|
||||
///
|
||||
/// # A missed message costs the two events very differently
|
||||
///
|
||||
|
|
@ -151,11 +150,12 @@ pub fn spawn(
|
|||
/// daemon pulls at startup regardless, and the webhook it replaced was lost
|
||||
/// identically when a hive was down.
|
||||
///
|
||||
/// ⚠️ **The deploy event has no such second path** — nothing else would ever
|
||||
/// tell this hive to build that agent. Its backstop is the hive-side reconcile
|
||||
/// loop ("hives pull and self-update"); until that exists this is a nudge with
|
||||
/// no safety net. Not papered over with `JetStream`: durability on one subject
|
||||
/// looks like a fix while the desired state still lives only in a message.
|
||||
/// The deploy event's second path is [`crate::workers::wanted`], which reads
|
||||
/// the whole declared set at boot and creates the agents this hive lacks — so
|
||||
/// a missed **first** deploy repairs itself. A missed **rebuild** does not:
|
||||
/// the declaration names agents, not revisions, so that falls to the boot
|
||||
/// reconcile noticing drift. Still not `JetStream`: durability on one subject
|
||||
/// looks like a fix while the desired state lives only in a message.
|
||||
///
|
||||
/// ⚠️ **A refused subscription is indistinguishable from a quiet one.** NATS
|
||||
/// reports an authorization violation asynchronously on the connection, not as
|
||||
|
|
@ -275,27 +275,7 @@ async fn handle_deploy_request(
|
|||
.job_queue
|
||||
.insert_job(|b| crate::job_queue::templates::rebuild(b, &agent, true))
|
||||
} else {
|
||||
// First deploy. The swarm has already created the identity, the forge
|
||||
// repo and its config; what is left is hive-local, and there is no
|
||||
// approval to wait on because the operator's click at swarm level is
|
||||
// the authorisation.
|
||||
//
|
||||
// Seed the power intent to `Up` before queuing, same as
|
||||
// `actions::approve`'s `ApprovalKind::Spawn` arm does for the
|
||||
// operator-approved path: `first_deploy`'s tail `Reconcile` node
|
||||
// seeds `Wanted` from the container's *currently observed* running
|
||||
// state when no row exists yet (`power::Store::get_or_seed`), and at
|
||||
// that point in a first deploy the container is freshly created but
|
||||
// not yet started — so an unseeded row locks the agent's wanted
|
||||
// state to `Offline` on its very first reconcile, and `Reconcile`
|
||||
// never emits the `Start` node. Setting the row up front here closes
|
||||
// that window the same way the approval path already does.
|
||||
if let Err(e) = coord.power.set(&agent, crate::power::Wanted::Up) {
|
||||
tracing::warn!(%agent, error = ?e, "agent_power: seed on swarm first-deploy failed");
|
||||
}
|
||||
coord
|
||||
.job_queue
|
||||
.insert_job(|b| crate::job_queue::templates::first_deploy(b, &agent))
|
||||
queue_first_deploy(coord, &agent)
|
||||
};
|
||||
match inserted {
|
||||
Ok(_) => {
|
||||
|
|
@ -306,6 +286,35 @@ async fn handle_deploy_request(
|
|||
}
|
||||
}
|
||||
|
||||
/// Queue the first deploy of an agent this hive does not have yet: seed its
|
||||
/// power intent, then insert the DAG.
|
||||
///
|
||||
/// The swarm has already created the identity, the forge repo and its config;
|
||||
/// what is left is hive-local, and there is no approval to wait on because the
|
||||
/// operator's click at swarm level is the authorisation.
|
||||
///
|
||||
/// Shared with [`crate::workers::wanted`] for the **seeding**, not the insert.
|
||||
/// `first_deploy`'s tail `Reconcile` seeds `Wanted` from the container's
|
||||
/// currently observed running state when no row exists yet
|
||||
/// (`power::Store::get_or_seed`), and at that point the container is freshly
|
||||
/// created but not started — so an unseeded row locks the agent to `Offline`
|
||||
/// on its very first reconcile and `Reconcile` never emits the `Start` node.
|
||||
/// Setting the row up front closes that window, the same way
|
||||
/// `actions::approve`'s `ApprovalKind::Spawn` arm does for the
|
||||
/// operator-approved path. A second caller open-coding the insert would lose
|
||||
/// exactly that, and the agent would come up stopped for no visible reason.
|
||||
pub(crate) fn queue_first_deploy(
|
||||
coord: &std::sync::Arc<crate::coordinator::Coordinator>,
|
||||
agent: &str,
|
||||
) -> Result<Vec<hive_jobq::NodeId>> {
|
||||
if let Err(e) = coord.power.set(agent, crate::power::Wanted::Up) {
|
||||
tracing::warn!(%agent, error = ?e, "agent_power: seed on swarm first-deploy failed");
|
||||
}
|
||||
coord
|
||||
.job_queue
|
||||
.insert_job(|b| crate::job_queue::templates::first_deploy(b, agent))
|
||||
}
|
||||
|
||||
/// Offer one snapshot: this hive's current readiness, under its own key.
|
||||
async fn publish(client: &async_nats::Client, hive: &str) -> Result<()> {
|
||||
// An unconnected client does not fail a JetStream request, it hangs
|
||||
|
|
|
|||
|
|
@ -317,13 +317,14 @@ pub async fn run(coord: Arc<Coordinator>) -> Result<()> {
|
|||
Ok(())
|
||||
}
|
||||
|
||||
/// Submit the boot-time forge/matrix/webhook/knowledge sweeps as DAG nodes —
|
||||
/// `ForgeSweep`, `MatrixSweep`, `WebhookRegister`, `KnowledgePull`. Unlike
|
||||
/// Submit the boot-time forge/matrix/webhook/knowledge/wanted-state sweeps as
|
||||
/// DAG nodes — `ForgeSweep`, `MatrixSweep`, `WebhookRegister`,
|
||||
/// `KnowledgePull`, `WantedPull`. Unlike
|
||||
/// [`submit_boot_tree`] this runs on **every** boot, quiet or not: these
|
||||
/// aren't config-drift work, they're startup housekeeping that always needs
|
||||
/// to happen, and the point of moving them here is exactly so they show up
|
||||
/// as real work on the dashboard instead of an invisible `tokio::spawn` that
|
||||
/// only surfaces on failure. Four independent, build-slot- and lease-exempt
|
||||
/// only surfaces on failure. Five independent, build-slot- and lease-exempt
|
||||
/// roots — no dependency edges between them, matching the existing
|
||||
/// `Reconcile`-root pattern in [`boot_nodes`].
|
||||
fn submit_startup_sweep_nodes(coord: &Arc<Coordinator>) {
|
||||
|
|
@ -334,6 +335,7 @@ fn submit_startup_sweep_nodes(coord: &Arc<Coordinator>) {
|
|||
let _ = b.node(NodeKind::MatrixSweep);
|
||||
let _ = b.node(NodeKind::WebhookRegister);
|
||||
let _ = b.node(NodeKind::KnowledgePull);
|
||||
let _ = b.node(NodeKind::WantedPull);
|
||||
Vec::new()
|
||||
}) {
|
||||
tracing::warn!(error = ?e, "boot: startup sweep DAG insert failed");
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
//! Background tasks and periodic sweeps: crash/login watcher, the
|
||||
//! scheduled-prompt delivery loop, boot-time auto-update
|
||||
//! reconcile, the agent-sockets.json writer loop, the MCP socket listener
|
||||
//! reconcile loop, and knowledge-repo sync. Each submodule is re-exported
|
||||
//! reconcile loop, knowledge-repo sync, and the swarm wanted-state pull.
|
||||
//! Each submodule is re-exported
|
||||
//! at the crate root, so `crate::crash_watch::…` etc. keep working unchanged.
|
||||
|
||||
pub mod agent_sockets;
|
||||
|
|
@ -10,3 +11,4 @@ pub mod crash_watch;
|
|||
pub mod knowledge;
|
||||
pub mod mcp_sockets;
|
||||
pub mod scheduled_prompts_worker;
|
||||
pub mod wanted;
|
||||
|
|
|
|||
367
hive-c0re/src/workers/wanted.rs
Normal file
367
hive-c0re/src/workers/wanted.rs
Normal file
|
|
@ -0,0 +1,367 @@
|
|||
//! Converging this hive onto the agent set the swarm controller declares.
|
||||
//!
|
||||
//! The controller writes one key per hive into the `hive-wanted` bucket (see
|
||||
//! [`swarm_queue_client::wanted`]); this reads its own and acts on it. It is
|
||||
//! the **repair** path, not the fast one: a deploy event
|
||||
//! ([`crate::swarm_status`]) is core NATS, so a hive that was down never hears
|
||||
//! it, and until this existed that left the hive on its old shape with nothing
|
||||
//! looking wrong.
|
||||
//!
|
||||
//! # Two things a declaration does not mean
|
||||
//!
|
||||
//! **Absence is not a deletion order.** An agent missing from the value — or
|
||||
//! no value, or no bucket — is one the controller has said nothing about.
|
||||
//! Swarm-side lifecycle does not yet cover agents that predate it, so
|
||||
//! converging "to exactly this set" would tear down every agent the swarm has
|
||||
//! not adopted.
|
||||
//!
|
||||
//! **An unrecognised state is inert**, so a controller that learns a new state
|
||||
//! does not take agents down on hives that have not been updated yet.
|
||||
//!
|
||||
//! # What "diverged" is measured against
|
||||
//!
|
||||
//! The hive's **stored power intent**, not the container's observed running
|
||||
//! state. An agent down while its intent says `Up` is already the boot
|
||||
//! reconcile's work; a loop reading `is_running` would insert a start DAG
|
||||
//! behind that reconcile's back on every boot.
|
||||
//!
|
||||
//! One consequence, deliberate: a declaration outranks a local stop. An agent
|
||||
//! this hive is declared to keep `Up` comes back at the next boot, because the
|
||||
//! controller owns that decision.
|
||||
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
use std::sync::Arc;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use swarm_queue_client::wanted::{AgentState, HiveWanted};
|
||||
|
||||
use crate::coordinator::Coordinator;
|
||||
use crate::power::Wanted;
|
||||
|
||||
/// Read this hive's declaration and converge the agents it names.
|
||||
///
|
||||
/// `Ok(())` covers three quiet cases that are **not** an empty declaration: no
|
||||
/// hive name, no bucket, no key. Each logs its own line, because a loop that
|
||||
/// silently does nothing and one that is working look identical otherwise.
|
||||
/// Anything else — an unreachable queue, an undecodable value, an agent list
|
||||
/// this hive cannot enumerate — is an error, so the node fails visibly on the
|
||||
/// dashboard instead of reading as "nothing was declared".
|
||||
///
|
||||
/// # Errors
|
||||
/// Propagates a failed queue connect, KV read, decode, or agent enumeration.
|
||||
pub async fn pull(coord: &Arc<Coordinator>) -> Result<()> {
|
||||
let Some(hive) = crate::container_view::hive_swarm_names().0 else {
|
||||
tracing::debug!("wanted state: HYPERHIVE_HIVE_NAME unset; no key to read");
|
||||
return Ok(());
|
||||
};
|
||||
let Some(client) = crate::swarm_queue::client().await else {
|
||||
// Absent or failed — either way already reported by the shared
|
||||
// connector (an `info` log or a `swarm_queue_config` banner).
|
||||
return Ok(());
|
||||
};
|
||||
// An unconnected client does not fail a JetStream request, it hangs on it.
|
||||
// A hung node is worse than a failed one: the dashboard shows it running.
|
||||
swarm_queue_client::ensure_connected(&client)?;
|
||||
|
||||
let Some(store) = swarm_queue_client::wanted::open_read_only(&client).await else {
|
||||
tracing::info!(%hive, "wanted state: no bucket yet; nothing has been declared");
|
||||
return Ok(());
|
||||
};
|
||||
let Some(raw) = store
|
||||
.get(&hive)
|
||||
.await
|
||||
.context("reading this hive's wanted state")?
|
||||
else {
|
||||
tracing::info!(%hive, "wanted state: no key for this hive; nothing has been declared");
|
||||
return Ok(());
|
||||
};
|
||||
let declared: HiveWanted =
|
||||
serde_json::from_slice(&raw).context("decoding this hive's wanted state")?;
|
||||
|
||||
converge(coord, &declared).await
|
||||
}
|
||||
|
||||
/// Queue whatever the declaration asks for and this hive is not already doing.
|
||||
async fn converge(coord: &Arc<Coordinator>, declared: &HiveWanted) -> Result<()> {
|
||||
// Fail closed, for the reason the deploy event's own arm gives: without
|
||||
// the list, "this hive does not have that agent" cannot be told from
|
||||
// "this hive cannot see its agents", and acting on the guess creates a
|
||||
// container over one that already exists.
|
||||
let present: BTreeSet<String> = crate::lifecycle::agents_for_meta_listing()
|
||||
.await
|
||||
.context("enumerating this hive's agents")?
|
||||
.into_iter()
|
||||
.map(|spec| spec.name)
|
||||
.collect();
|
||||
|
||||
let mut intents = BTreeMap::new();
|
||||
for agent in declared.agents.keys() {
|
||||
match coord.power.get(agent) {
|
||||
Ok(intent) => {
|
||||
intents.insert(agent.clone(), intent);
|
||||
}
|
||||
// Left out of the map, which is what makes `plan` skip it.
|
||||
Err(e) => {
|
||||
tracing::warn!(%agent, error = ?e, "wanted state: cannot read power intent; skipping");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let plan = plan(declared, &present, &intents);
|
||||
tracing::info!(
|
||||
declared = declared.agents.len(),
|
||||
deploy = plan.deploy.len(),
|
||||
start = plan.start.len(),
|
||||
stop = plan.stop.len(),
|
||||
"wanted state: converging"
|
||||
);
|
||||
|
||||
let mut queued = false;
|
||||
for agent in &plan.deploy {
|
||||
match crate::swarm_status::queue_first_deploy(coord, agent) {
|
||||
Ok(_) => queued = true,
|
||||
// Per agent rather than the whole sweep: one agent that cannot be
|
||||
// queued is not a reason to leave the others undeclared.
|
||||
Err(e) => tracing::warn!(%agent, error = %e, "wanted state: queueing a deploy failed"),
|
||||
}
|
||||
}
|
||||
if !plan.start.is_empty() {
|
||||
crate::job_queue::power::start_many(coord, &plan.start)
|
||||
.await
|
||||
.context("queueing the declared starts")?;
|
||||
}
|
||||
if !plan.stop.is_empty() {
|
||||
crate::job_queue::power::stop_many(coord, &plan.stop, true)
|
||||
.await
|
||||
.context("queueing the declared stops")?;
|
||||
}
|
||||
if queued {
|
||||
// The power ops emit their own; the first deploys do not.
|
||||
coord.emit_rebuild_queue_snapshot();
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Everything one declaration asks this hive to queue.
|
||||
#[derive(Debug, Default, PartialEq, Eq)]
|
||||
struct Plan {
|
||||
deploy: Vec<String>,
|
||||
start: Vec<String>,
|
||||
stop: Vec<String>,
|
||||
}
|
||||
|
||||
/// Turn one declaration into that plan — pure, so what the loop does with a
|
||||
/// given declaration is testable without a queue, a container or a store.
|
||||
///
|
||||
/// `intents` is the hive's stored power intent per agent, and an agent
|
||||
/// **missing from the map** is one whose intent could not be read: it is
|
||||
/// skipped. That is not `Some(None)` — "no row yet" is a reading and gets
|
||||
/// converged, "the read failed" is not and would be a guess.
|
||||
///
|
||||
/// An agent the declaration does not name reaches none of the three lists.
|
||||
/// That is the whole of "absence is not a deletion order": the agents this
|
||||
/// hive runs that the swarm has not adopted are never even inspected.
|
||||
fn plan(
|
||||
declared: &HiveWanted,
|
||||
present: &BTreeSet<String>,
|
||||
intents: &BTreeMap<String, Option<Wanted>>,
|
||||
) -> Plan {
|
||||
let mut plan = Plan::default();
|
||||
for (agent, decl) in &declared.agents {
|
||||
let Some(intent) = intents.get(agent) else {
|
||||
continue;
|
||||
};
|
||||
match decide(&decl.state, present.contains(agent), *intent) {
|
||||
Converge::Deploy => plan.deploy.push(agent.clone()),
|
||||
Converge::Start => plan.start.push(agent.clone()),
|
||||
Converge::Stop => plan.stop.push(agent.clone()),
|
||||
Converge::Nothing => {}
|
||||
Converge::Inert => {
|
||||
tracing::info!(
|
||||
%agent, state = ?decl.state,
|
||||
"wanted state: state not recognised by this build; leaving the agent alone"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
plan
|
||||
}
|
||||
|
||||
/// What the loop will do about one declared agent.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum Converge {
|
||||
/// Declared, and this hive has no container for it — the missed-deploy
|
||||
/// repair.
|
||||
Deploy,
|
||||
Start,
|
||||
Stop,
|
||||
/// The hive already agrees with the declaration.
|
||||
Nothing,
|
||||
/// A state this build does not recognise.
|
||||
Inert,
|
||||
}
|
||||
|
||||
/// The whole decision, pure: no queue, no store, no container. The three
|
||||
/// inputs are exactly what [`converge`] reads per agent, so the table below is
|
||||
/// the behaviour rather than a model of it.
|
||||
fn decide(state: &AgentState, present: bool, intent: Option<Wanted>) -> Converge {
|
||||
match state {
|
||||
// First, and whatever else is true of the agent.
|
||||
AgentState::Unrecognised(_) => Converge::Inert,
|
||||
AgentState::Up if !present => Converge::Deploy,
|
||||
AgentState::Up => {
|
||||
if intent == Some(Wanted::Up) {
|
||||
Converge::Nothing
|
||||
} else {
|
||||
Converge::Start
|
||||
}
|
||||
}
|
||||
// Declared offline and not here: creating a container in order to
|
||||
// leave it stopped is not what the declaration asks for.
|
||||
AgentState::Offline if !present => Converge::Nothing,
|
||||
AgentState::Offline => {
|
||||
if intent == Some(Wanted::Offline) {
|
||||
Converge::Nothing
|
||||
} else {
|
||||
Converge::Stop
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
|
||||
use super::{Converge, Plan, decide, plan};
|
||||
use crate::power::Wanted;
|
||||
use swarm_queue_client::wanted::{AgentState, AgentWanted, HiveWanted};
|
||||
|
||||
fn unknown() -> AgentState {
|
||||
AgentState::Unrecognised("paused".to_owned())
|
||||
}
|
||||
|
||||
fn declaration(agents: &[(&str, AgentState)]) -> HiveWanted {
|
||||
HiveWanted {
|
||||
agents: agents
|
||||
.iter()
|
||||
.map(|(name, state)| {
|
||||
(
|
||||
(*name).to_owned(),
|
||||
AgentWanted {
|
||||
state: state.clone(),
|
||||
},
|
||||
)
|
||||
})
|
||||
.collect(),
|
||||
}
|
||||
}
|
||||
|
||||
fn set(names: &[&str]) -> BTreeSet<String> {
|
||||
names.iter().map(|n| (*n).to_owned()).collect()
|
||||
}
|
||||
|
||||
fn intents(entries: &[(&str, Option<Wanted>)]) -> BTreeMap<String, Option<Wanted>> {
|
||||
entries
|
||||
.iter()
|
||||
.map(|(name, intent)| ((*name).to_owned(), *intent))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The property the whole migration rides on: this hive's own agents are
|
||||
/// not converged, they are not looked at. `adopted` is the control — the
|
||||
/// same call must still act on what the declaration *does* name, or this
|
||||
/// passes on a loop that plans nothing at all.
|
||||
#[test]
|
||||
fn an_agent_the_declaration_does_not_name_is_never_touched() {
|
||||
let declared = declaration(&[("adopted", AgentState::Up)]);
|
||||
let planned = plan(
|
||||
&declared,
|
||||
&set(&["adopted", "legacy"]),
|
||||
&intents(&[("adopted", Some(Wanted::Offline)), ("legacy", None)]),
|
||||
);
|
||||
assert_eq!(
|
||||
planned,
|
||||
Plan {
|
||||
deploy: vec![],
|
||||
start: vec!["adopted".to_owned()],
|
||||
stop: vec![],
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
/// An intent that could not be read is skipped rather than treated as "no
|
||||
/// row". `readable` is the control for the same call.
|
||||
#[test]
|
||||
fn an_unreadable_intent_skips_only_that_agent() {
|
||||
let declared = declaration(&[("unreadable", AgentState::Up), ("readable", AgentState::Up)]);
|
||||
let planned = plan(
|
||||
&declared,
|
||||
&set(&["unreadable", "readable"]),
|
||||
&intents(&[("readable", None)]),
|
||||
);
|
||||
assert_eq!(planned.start, vec!["readable".to_owned()]);
|
||||
assert!(planned.deploy.is_empty() && planned.stop.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_declared_agent_this_hive_does_not_have_is_deployed() {
|
||||
assert_eq!(decide(&AgentState::Up, false, None), Converge::Deploy);
|
||||
// Even holding an intent: a row can outlive its container.
|
||||
assert_eq!(
|
||||
decide(&AgentState::Up, false, Some(Wanted::Up)),
|
||||
Converge::Deploy
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_hive_that_already_agrees_queues_nothing() {
|
||||
assert_eq!(
|
||||
decide(&AgentState::Up, true, Some(Wanted::Up)),
|
||||
Converge::Nothing
|
||||
);
|
||||
assert_eq!(
|
||||
decide(&AgentState::Offline, true, Some(Wanted::Offline)),
|
||||
Converge::Nothing
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_disagreeing_intent_is_converged_both_ways() {
|
||||
assert_eq!(
|
||||
decide(&AgentState::Up, true, Some(Wanted::Offline)),
|
||||
Converge::Start
|
||||
);
|
||||
assert_eq!(
|
||||
decide(&AgentState::Offline, true, Some(Wanted::Up)),
|
||||
Converge::Stop
|
||||
);
|
||||
}
|
||||
|
||||
/// No row is not agreement. An agent that predates the power store is
|
||||
/// converged to what it was declared as, not left on whatever it happens
|
||||
/// to be doing.
|
||||
#[test]
|
||||
fn an_agent_with_no_intent_row_is_converged() {
|
||||
assert_eq!(decide(&AgentState::Up, true, None), Converge::Start);
|
||||
assert_eq!(decide(&AgentState::Offline, true, None), Converge::Stop);
|
||||
}
|
||||
|
||||
/// Declared offline, no container: the one absent case that must not
|
||||
/// deploy. Its control is the `Up` case above, which must.
|
||||
#[test]
|
||||
fn an_absent_agent_declared_offline_is_left_absent() {
|
||||
assert_eq!(decide(&AgentState::Offline, false, None), Converge::Nothing);
|
||||
}
|
||||
|
||||
/// The version-skew arm: an unrecognised state is inert in every
|
||||
/// combination, including the ones where a known state would act.
|
||||
#[test]
|
||||
fn an_unrecognised_state_is_inert_everywhere() {
|
||||
for present in [true, false] {
|
||||
for intent in [None, Some(Wanted::Up), Some(Wanted::Offline)] {
|
||||
assert_eq!(decide(&unknown(), present, intent), Converge::Inert);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -32,6 +32,51 @@ use crate::Error;
|
|||
/// for two deployments to disagree about which one that is.
|
||||
pub const BUCKET: &str = "hive-wanted";
|
||||
|
||||
/// One hive's whole declaration: the agents the controller names, and what it
|
||||
/// wants of each. The value under `BUCKET`/`<hiveName>`.
|
||||
///
|
||||
/// The agents it does **not** name are not part of the declaration at all —
|
||||
/// see the module docs. A hive converges the ones listed here and leaves the
|
||||
/// rest alone.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct HiveWanted {
|
||||
/// Agent name → what the controller wants of it.
|
||||
#[serde(default)]
|
||||
pub agents: std::collections::BTreeMap<String, AgentWanted>,
|
||||
}
|
||||
|
||||
/// One agent's entry in a [`HiveWanted`].
|
||||
///
|
||||
/// A struct around one field rather than the bare state, because this is where
|
||||
/// a later declaration grows — a config revision, a schedule — and a hive
|
||||
/// built before that field existed keeps deserialising the whole document.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct AgentWanted {
|
||||
/// The state to converge this agent to.
|
||||
pub state: AgentState,
|
||||
}
|
||||
|
||||
/// What the controller wants an agent to be — an **open** enum.
|
||||
///
|
||||
/// A value this build does not know deserialises into
|
||||
/// [`AgentState::Unrecognised`], carried verbatim, rather than failing the
|
||||
/// whole document or collapsing into a known variant. Both alternatives break
|
||||
/// the same way: a closed enum forces "not `Up`" onto a state like `paused`,
|
||||
/// so a controller that learns a new value would take agents down on every
|
||||
/// hive that has not been updated yet.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum AgentState {
|
||||
/// Exists on the hive and is running.
|
||||
Up,
|
||||
/// Exists on the hive and is not running.
|
||||
Offline,
|
||||
/// Anything else. Kept as written so a hive can say what it declined to
|
||||
/// act on instead of logging that something unspecified was skipped.
|
||||
#[serde(untagged)]
|
||||
Unrecognised(String),
|
||||
}
|
||||
|
||||
/// Open the wanted-state bucket for writing, creating it if nothing has yet.
|
||||
///
|
||||
/// **Controller-side only.** `history: 1` because a hive converges to the
|
||||
|
|
@ -78,3 +123,50 @@ pub async fn open_read_only(
|
|||
let js = async_nats::jetstream::new(client.clone());
|
||||
js.get_key_value(BUCKET).await.ok()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{AgentState, HiveWanted};
|
||||
|
||||
#[test]
|
||||
fn known_states_decode_and_an_unknown_one_stays_itself() {
|
||||
let doc = r#"{"agents":{
|
||||
"a":{"state":"up"},
|
||||
"b":{"state":"offline"},
|
||||
"c":{"state":"paused"}
|
||||
}}"#;
|
||||
let decoded: HiveWanted = serde_json::from_str(doc).expect("decodes");
|
||||
assert_eq!(decoded.agents["a"].state, AgentState::Up);
|
||||
assert_eq!(decoded.agents["b"].state, AgentState::Offline);
|
||||
assert_eq!(
|
||||
decoded.agents["c"].state,
|
||||
AgentState::Unrecognised("paused".to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_unrecognised_state_survives_a_round_trip() {
|
||||
let doc = r#"{"agents":{"c":{"state":"paused"}}}"#;
|
||||
let decoded: HiveWanted = serde_json::from_str(doc).expect("decodes");
|
||||
let reserialised = serde_json::to_string(&decoded).expect("serialises");
|
||||
assert_eq!(reserialised, r#"{"agents":{"c":{"state":"paused"}}}"#);
|
||||
}
|
||||
|
||||
/// A field a newer controller adds must not cost an older hive the whole
|
||||
/// document — that is the version skew the open enum exists for, one level
|
||||
/// up.
|
||||
#[test]
|
||||
fn unknown_fields_are_ignored() {
|
||||
let doc = r#"{"agents":{"a":{"state":"up","config_rev":"deadbeef"}},"epoch":3}"#;
|
||||
let decoded: HiveWanted = serde_json::from_str(doc).expect("decodes");
|
||||
assert_eq!(decoded.agents["a"].state, AgentState::Up);
|
||||
}
|
||||
|
||||
/// The control for the two above: openness is about *values and fields*,
|
||||
/// not about accepting anything. A declaration missing the one required
|
||||
/// field is an error, so a malformed document cannot read as "no agents".
|
||||
#[test]
|
||||
fn a_missing_state_is_an_error() {
|
||||
assert!(serde_json::from_str::<HiveWanted>(r#"{"agents":{"a":{}}}"#).is_err());
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue