feat(#3124): converge the hive onto the agent set the swarm declares
The deploy event is a nudge with no second path: core NATS is at-most-once, so a hive that was down when the controller published simply never learns that an agent is meant to exist here. This adds the repair path — one boot-time DAG node that reads this hive's own key in the `hive-wanted` bucket and converges the agents it names. Two semantics settled on the issue thread, and both are places where a plausible implementation is the wrong one: - **Absence is not a deletion order.** No bucket, no key, or an agent the value does not name all mean the controller has said nothing. Swarm-side lifecycle does not yet cover agents that predate it, so "converge to exactly this set" would tear down every agent the swarm has not adopted. `plan` only ever inspects the agents a declaration names. - **An unrecognised state is inert.** `AgentState` is an open enum: a value this build cannot read deserialises into `Unrecognised` and is left alone. A closed enum would force "not `Up`" onto a state like `paused`, so a controller that learned a new value would take agents down on every hive not yet updated. Divergence is measured against the hive's **stored power intent**, not the container's observed running state — an agent that is down while its intent says `Up` is already the boot reconcile's work, and a loop reading `is_running` would insert a start DAG behind that reconcile's back on every boot. A hive that already agrees with its declaration queues nothing at all. `queue_first_deploy` is extracted from the deploy-event path rather than open-coded here, for the power-intent seed: without it `first_deploy`'s tail `Reconcile` seeds `Wanted` from a container that exists but has not started yet, which locks the agent to `Offline` on its first reconcile. The read is authorised as-is: `store.get` takes async-nats' direct-get arm (the KV bucket is created with `allow_direct`), which is exactly the `$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.<hive>` subject `swarm-nats-auth` grants a hive. The fallback subject is not granted, and a refused NATS request surfaces as a timeout rather than an error. Nothing writes the bucket yet — the controller-side writer is the other half of #3124, so this does not close it.
This commit is contained in:
parent
8cba57e01c
commit
37f3c63eeb
7 changed files with 526 additions and 34 deletions
|
|
@ -141,9 +141,8 @@ pub fn spawn(
|
|||
/// rebuild insert the operator's own verb makes.
|
||||
///
|
||||
/// Only the deploy event carries a payload, and only the agent name: its
|
||||
/// subject already names the hive, so this one listens on its own rather than
|
||||
/// filtering a swarm-wide feed. Even then it is a trigger, never the config,
|
||||
/// which git owns.
|
||||
/// subject already names the hive, so it listens on its own rather than a
|
||||
/// swarm-wide feed. Even then it is a trigger, never the config git owns.
|
||||
///
|
||||
/// # A missed message costs the two events very differently
|
||||
///
|
||||
|
|
@ -151,11 +150,12 @@ pub fn spawn(
|
|||
/// daemon pulls at startup regardless, and the webhook it replaced was lost
|
||||
/// identically when a hive was down.
|
||||
///
|
||||
/// ⚠️ **The deploy event has no such second path** — nothing else would ever
|
||||
/// tell this hive to build that agent. Its backstop is the hive-side reconcile
|
||||
/// loop ("hives pull and self-update"); until that exists this is a nudge with
|
||||
/// no safety net. Not papered over with `JetStream`: durability on one subject
|
||||
/// looks like a fix while the desired state still lives only in a message.
|
||||
/// The deploy event's second path is [`crate::workers::wanted`], which reads
|
||||
/// the whole declared set at boot and creates the agents this hive lacks — so
|
||||
/// a missed **first** deploy repairs itself. A missed **rebuild** does not:
|
||||
/// the declaration names agents, not revisions, so that falls to the boot
|
||||
/// reconcile noticing drift. Still not `JetStream`: durability on one subject
|
||||
/// looks like a fix while the desired state lives only in a message.
|
||||
///
|
||||
/// ⚠️ **A refused subscription is indistinguishable from a quiet one.** NATS
|
||||
/// reports an authorization violation asynchronously on the connection, not as
|
||||
|
|
@ -275,27 +275,7 @@ async fn handle_deploy_request(
|
|||
.job_queue
|
||||
.insert_job(|b| crate::job_queue::templates::rebuild(b, &agent, true))
|
||||
} else {
|
||||
// First deploy. The swarm has already created the identity, the forge
|
||||
// repo and its config; what is left is hive-local, and there is no
|
||||
// approval to wait on because the operator's click at swarm level is
|
||||
// the authorisation.
|
||||
//
|
||||
// Seed the power intent to `Up` before queuing, same as
|
||||
// `actions::approve`'s `ApprovalKind::Spawn` arm does for the
|
||||
// operator-approved path: `first_deploy`'s tail `Reconcile` node
|
||||
// seeds `Wanted` from the container's *currently observed* running
|
||||
// state when no row exists yet (`power::Store::get_or_seed`), and at
|
||||
// that point in a first deploy the container is freshly created but
|
||||
// not yet started — so an unseeded row locks the agent's wanted
|
||||
// state to `Offline` on its very first reconcile, and `Reconcile`
|
||||
// never emits the `Start` node. Setting the row up front here closes
|
||||
// that window the same way the approval path already does.
|
||||
if let Err(e) = coord.power.set(&agent, crate::power::Wanted::Up) {
|
||||
tracing::warn!(%agent, error = ?e, "agent_power: seed on swarm first-deploy failed");
|
||||
}
|
||||
coord
|
||||
.job_queue
|
||||
.insert_job(|b| crate::job_queue::templates::first_deploy(b, &agent))
|
||||
queue_first_deploy(coord, &agent)
|
||||
};
|
||||
match inserted {
|
||||
Ok(_) => {
|
||||
|
|
@ -306,6 +286,35 @@ async fn handle_deploy_request(
|
|||
}
|
||||
}
|
||||
|
||||
/// Queue the first deploy of an agent this hive does not have yet: seed its
|
||||
/// power intent, then insert the DAG.
|
||||
///
|
||||
/// The swarm has already created the identity, the forge repo and its config;
|
||||
/// what is left is hive-local, and there is no approval to wait on because the
|
||||
/// operator's click at swarm level is the authorisation.
|
||||
///
|
||||
/// Shared with [`crate::workers::wanted`] for the **seeding**, not the insert.
|
||||
/// `first_deploy`'s tail `Reconcile` seeds `Wanted` from the container's
|
||||
/// currently observed running state when no row exists yet
|
||||
/// (`power::Store::get_or_seed`), and at that point the container is freshly
|
||||
/// created but not started — so an unseeded row locks the agent to `Offline`
|
||||
/// on its very first reconcile and `Reconcile` never emits the `Start` node.
|
||||
/// Setting the row up front closes that window, the same way
|
||||
/// `actions::approve`'s `ApprovalKind::Spawn` arm does for the
|
||||
/// operator-approved path. A second caller open-coding the insert would lose
|
||||
/// exactly that, and the agent would come up stopped for no visible reason.
|
||||
pub(crate) fn queue_first_deploy(
|
||||
coord: &std::sync::Arc<crate::coordinator::Coordinator>,
|
||||
agent: &str,
|
||||
) -> Result<Vec<hive_jobq::NodeId>> {
|
||||
if let Err(e) = coord.power.set(agent, crate::power::Wanted::Up) {
|
||||
tracing::warn!(%agent, error = ?e, "agent_power: seed on swarm first-deploy failed");
|
||||
}
|
||||
coord
|
||||
.job_queue
|
||||
.insert_job(|b| crate::job_queue::templates::first_deploy(b, agent))
|
||||
}
|
||||
|
||||
/// Offer one snapshot: this hive's current readiness, under its own key.
|
||||
async fn publish(client: &async_nats::Client, hive: &str) -> Result<()> {
|
||||
// An unconnected client does not fail a JetStream request, it hangs
|
||||
|
|
|
|||
Loading…
Reference in a new issue