vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

wanted pull: one deadline bounds both connect waits

Browse source
This commit is contained in:
atlas 2026-10-21 01:18:43 +02:00 • committed by mara
commit de68f175b5
2 changed files with 72 additions and 38 deletions
Download patch file Download diff file Expand all files Collapse all files

48
docs/scheduler/coordinator.md
View file

@ -53,30 +53,30 @@ Cheap — no build slot:
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| Node | Behavior | | Node | Behavior |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build | | `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet | | `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop | | `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down | | `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up | | `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped | | `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace | | `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ | | `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn | | `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way | | `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here | | `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here |
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` | | `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims | | `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step | | `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
| `DestroyBookkeeping` | the post-destroy tail — meta sync, drop the power intent, notify the manager, rescan, re-emit the tombstone. Same split rationale as `RebuildBookkeeping`/`Swap`. The destructive work is `PurgeState`'s | | `DestroyBookkeeping` | the post-destroy tail — meta sync, drop the power intent, notify the manager, rescan, re-emit the tombstone. Same split rationale as `RebuildBookkeeping`/`Swap`. The destructive work is `PurgeState`'s |
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles | | `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
| `EmitRebuilt` | tail of a rebuild — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent _and_ per outcome | | `EmitRebuilt` | tail of a rebuild — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent _and_ per outcome |
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload | | `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
| `ForgeSweep` | one-shot boot-time forge user/token sweep for every container (`forge::ensure_all`) as a first-class node, so it shows as real work on the dashboard instead of running invisibly in a bare `tokio::spawn`. Agentless | | `ForgeSweep` | one-shot boot-time forge user/token sweep for every container (`forge::ensure_all`) as a first-class node, so it shows as real work on the dashboard instead of running invisibly in a bare `tokio::spawn`. Agentless |
| `MatrixSweep` | matrix user/space sweep (`matrix::ensure_all`): the boot-time instance, plus one every 30 min from a loop in `main.rs`. Holds `Resource::MatrixSweep` (capacity 1), so two passes never overlap; each tick queues its own pass, which waits for the resource if one is already live. Agentless | | `MatrixSweep` | matrix user/space sweep (`matrix::ensure_all`): the boot-time instance, plus one every 30 min from a loop in `main.rs`. Holds `Resource::MatrixSweep` (capacity 1), so two passes never overlap; each tick queues its own pass, which waits for the resource if one is already live. Agentless |
| `KnowledgePull` | `/knowledge` pull (`knowledge::pull`): at boot (commits that landed while `hive-c0re` was down), on the swarm knowledge-changed event, and hourly as a fallback. Holds `Resource::KnowledgeTree` (capacity 1), so two pulls never overlap on the working tree; each trigger queues its own pass, which waits for the resource if one is already live. Agentless | | `KnowledgePull` | `/knowledge` pull (`knowledge::pull`): at boot (commits that landed while `hive-c0re` was down), on the swarm knowledge-changed event, and hourly as a fallback. Holds `Resource::KnowledgeTree` (capacity 1), so two pulls never overlap on the working tree; each trigger queues its own pass, which waits for the resource if one is already live. Agentless |
| `WantedPull` | pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. Runs at boot, waiting up to a minute for the queue connection, and again after every attach of the wanted-state watch (which retries with a backoff when the attach fails or the stream ends); the deploy event (`swarm_status`) and the watch are the fast path, this repairs a missed one. A first deploy already queued or running for an agent isn't queued again Agentless | | `WantedPull` | pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. Runs at boot, waiting up to a minute in total for the queue connection, and again after every attach of the wanted-state watch (which retries with a backoff when the attach fails or the stream ends); the deploy event (`swarm_status`) and the watch are the fast path, this repairs a missed one. A first deploy already queued or running for an agent isn't queued again. Agentless |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

62
hive-c0re/src/workers/wanted.rs
View file

@ -39,7 +39,8 @@ use swarm_wire::wanted::{AgentState, HiveWanted};
use crate::coordinator::Coordinator; use crate::coordinator::Coordinator;
use crate::power::Wanted; use crate::power::Wanted;
/// Longest `pull` and each watch attach wait for the swarm queue to connect. /// Longest `pull`, and each watch session's attach and first pull together, wait
/// for the swarm queue to connect.
const CONNECT_WAIT: Duration = Duration::from_mins(1); const CONNECT_WAIT: Duration = Duration::from_mins(1);
/// How often the wait for a connected client looks at the connection state. /// How often the wait for a connected client looks at the connection state.
@ -60,9 +61,9 @@ const MAX_RETRY_DELAY: Duration = Duration::from_mins(5);
/// this hive cannot enumerate — is an error, so the node fails visibly on the /// this hive cannot enumerate — is an error, so the node fails visibly on the
/// dashboard instead of reading as "nothing was declared". /// dashboard instead of reading as "nothing was declared".
/// ///
/// Waits up to [`CONNECT_WAIT`] for the queue connection: at boot the shared /// Waits up to [`CONNECT_WAIT`] in total, for the client and then for its
/// client exists before it is connected, and a pull that did not wait would /// connection: at boot the shared client exists before it is connected, and a
/// fail every boot. /// pull that did not wait would fail every boot.
/// ///
/// # Errors /// # Errors
/// Propagates a failed queue connect, KV read, decode, or agent enumeration. /// Propagates a failed queue connect, KV read, decode, or agent enumeration.
@ -71,24 +72,27 @@ pub async fn pull(coord: &Arc<Coordinator>) -> Result<()> {
tracing::debug!("wanted state: HYPERHIVE_HIVE_NAME unset; no key to read"); tracing::debug!("wanted state: HYPERHIVE_HIVE_NAME unset; no key to read");
return Ok(()); return Ok(());
}; };
let deadline = tokio::time::Instant::now() + CONNECT_WAIT;
let client = let client =
match tokio::time::timeout(CONNECT_WAIT, crate::swarm_queue::wait_for_client()).await { match tokio::time::timeout_at(deadline, crate::swarm_queue::wait_for_client()).await {
// Absent — already reported by the shared connector (an `info` log or // Absent — already reported by the shared connector (an `info` log or
// a `swarm_queue_config` banner). // a `swarm_queue_config` banner).
Ok(None) => return Ok(()), Ok(None) => return Ok(()),
Ok(Some(client)) => client, Ok(Some(client)) => client,
Err(_) => anyhow::bail!("no swarm queue client within {}s", CONNECT_WAIT.as_secs()), Err(_) => anyhow::bail!("no swarm queue client within {}s", CONNECT_WAIT.as_secs()),
}; };
pull_with(coord, &client, &hive).await pull_with(coord, &client, &hive, deadline).await
} }
/// [`pull`] over a client the caller already has. /// [`pull`] over a client the caller already has, waiting for its connection
/// no later than `deadline`.
async fn pull_with( async fn pull_with(
coord: &Arc<Coordinator>, coord: &Arc<Coordinator>,
client: &async_nats::Client, client: &async_nats::Client,
hive: &str, hive: &str,
deadline: tokio::time::Instant,
) -> Result<()> { ) -> Result<()> {
wait_connected(client, CONNECT_WAIT).await?; wait_connected(client, deadline).await?;
let Some(store) = swarm_queue_client::wanted::open_read_only(client, hive) let Some(store) = swarm_queue_client::wanted::open_read_only(client, hive)
.await .await
@ -111,15 +115,14 @@ async fn pull_with(
converge(coord, &declared).await converge(coord, &declared).await
} }
/// Wait up to `limit` for `client` to be connected. /// Wait until `deadline` for `client` to be connected.
/// ///
/// `swarm_queue::wait_for_client` returns a client that may still be /// `swarm_queue::wait_for_client` returns a client that may still be
/// connecting. An unconnected client does not fail a `JetStream` request, it /// connecting. An unconnected client does not fail a `JetStream` request, it
/// hangs on it, and a hung node shows as running on the dashboard, so every /// hangs on it, and a hung node shows as running on the dashboard, so every
/// request here is preceded by this bounded wait and its /// request here is preceded by this bounded wait and its
/// `ensure_connected` verdict. /// `ensure_connected` verdict.
async fn wait_connected(client: &async_nats::Client, limit: Duration) -> Result<()> { async fn wait_connected(client: &async_nats::Client, deadline: tokio::time::Instant) -> Result<()> {
let deadline = tokio::time::Instant::now() + limit;
while swarm_queue_client::ensure_connected(client).is_err() while swarm_queue_client::ensure_connected(client).is_err()
&& tokio::time::Instant::now() < deadline && tokio::time::Instant::now() < deadline
{ {
@ -223,8 +226,9 @@ async fn watch_session(
) -> SessionEnd { ) -> SessionEnd {
use futures_util::StreamExt as _; use futures_util::StreamExt as _;
let deadline = tokio::time::Instant::now() + CONNECT_WAIT;
let attach = async { let attach = async {
if let Err(e) = wait_connected(client, CONNECT_WAIT).await { if let Err(e) = wait_connected(client, deadline).await {
tracing::warn!(%hive, error = ?e, "wanted state: cannot watch this hive's bucket; retrying"); tracing::warn!(%hive, error = ?e, "wanted state: cannot watch this hive's bucket; retrying");
return None; return None;
} }
@ -244,8 +248,11 @@ async fn watch_session(
} }
} }
}; };
let Some(mut updates) = let Some(mut updates) = Box::pin(attach_then_pull(
Box::pin(attach_then_pull(attach, pull_with(coord, client, hive))).await attach,
pull_with(coord, client, hive, deadline),
))
.await
else { else {
return SessionEnd::Lost { attached: false }; return SessionEnd::Lost { attached: false };
}; };
@ -940,6 +947,33 @@ mod tests {
} }
} }
/// A client that never connects: initial-connect retries run in the
/// background and `ensure_connected` keeps failing.
async fn never_connected() -> async_nats::Client {
async_nats::ConnectOptions::new()
.retry_on_initial_connect()
.connect("nats://127.0.0.1:1")
.await
.expect("a client that retries its first connect is returned unconnected")
}
/// One deadline bounds the whole connect wait: time already spent before
/// `wait_connected` counts against it, so `pull`'s client wait and its
/// connection wait together stay inside one `CONNECT_WAIT`.
#[tokio::test(start_paused = true)]
async fn the_connect_wait_is_bounded_by_one_deadline() {
let client = never_connected().await;
let start = tokio::time::Instant::now();
let deadline = start + super::CONNECT_WAIT;
tokio::time::sleep(std::time::Duration::from_secs(59)).await;
assert!(super::wait_connected(&client, deadline).await.is_err());
assert!(
start.elapsed() <= super::CONNECT_WAIT + super::CONNECT_POLL,
"waited {:?} in total",
start.elapsed()
);
}
mod watch_supervision { mod watch_supervision {
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
use std::time::Duration; use std::time::Duration;