job_queue scheduler: observe shutdown every loop iteration, document the no-persistence invariant

run_worker only polled shutdown.changed() inside the select! arm, which the
claim-and-spawn branch skips via continue whenever there's ready work. Under
a sustained stream of ready claims (a boot sweep across agents is the
realistic case) exit was deferred until the queue happened to drain instead
of being observed promptly.

check *shutdown.borrow() explicitly at the top of every loop iteration
instead of relying solely on the select! arm, so a busy loop still sees
shutdown promptly.

also documented why dropping pending Queued DAGs on shutdown is safe: every
NodeKind is idempotent-convergent, which is a property of the node set, not
of the queue, and isn't enforced by the type system. flagged the side-effect
tails (EmitRebuilt, ResolveApproval) as the ones closest to the edge.

fixes hyperhive/hyperhive#2848

hive-c0re/src/job_queue/scheduler.rs

@@ -34,16 +34,37 @@ struct NodeDone {
 /// Scheduler loop. Spawned once at hive-c0re startup from `main.rs`.
 ///
 /// Shutdown semantics: subscribes to `coord.shutdown_rx()`. On a true signal
-/// the loop exits immediately; already-running node tasks ride the runtime down
-/// with the process, and pending `Queued` DAGs are dropped — desired state is
-/// re-derived on next boot (boot sweep + reconcile), so the in-memory queue is
-/// deliberately not durable.
+/// the loop exits — checked explicitly at the top of every iteration (not just
+/// in the `select!` below), so a sustained stream of ready work can't defer
+/// exit until the queue happens to drain. Already-running node tasks ride the
+/// runtime down with the process, and pending `Queued` DAGs are dropped —
+/// desired state is re-derived on next boot (boot sweep + reconcile), so the
+/// in-memory queue is deliberately not durable.
+///
+/// That's safe only because every [`super::NodeKind`] is
+/// idempotent-convergent: re-running a half-applied node converges instead of
+/// double-applying (e.g. a dropped `Swap` re-derives to the same
+/// `nixos-container update`, which is itself declarative). That's a property
+/// of the node set, not of the queue — it holds today but isn't enforced by
+/// the type system, so it's worth re-checking whenever a new `NodeKind` is
+/// added. The side-effect tails (`EmitRebuilt`, `ResolveApproval`) sit
+/// closest to the edge: dropping or re-running one affects something outside
+/// the graph (today, a missed or duplicated notification) rather than
+/// reconverging silently.
 pub async fn run_worker(coord: Arc<Coordinator>) {
     let mut shutdown = coord.shutdown_rx();
     let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel::<NodeDone>();
     // (DAG id, agent) → transient guard held for that agent's lease window.
     let mut transients: HashMap<(u64, String), crate::coordinator::TransientGuard> = HashMap::new();
     loop {
+        // Checked every iteration, not just in the `select!` below — a
+        // continuous stream of ready claims never reaches the `select!`, so
+        // relying on it alone as the only shutdown observation point defers
+        // exit until the queue drains. See the doc comment above.
+        if *shutdown.borrow() {
+            tracing::info!("job_queue: scheduler exiting on shutdown");
+            return;
+        }
         reconcile_transients(&coord, &mut transients);
         let claims = coord.job_queue.claim_ready();
         if !claims.is_empty() {