hive-c0re: serialise the matrix and knowledge sweeps through the job queue

The matrix sweep and the /knowledge pull each had concurrent callers
(#4723 item 4). Two overlapping knowledge pulls fail on .git/index.lock
and the remote-tracking ref lock: 30 of 30 concurrent replays of the
reset/clean/pull sequence in a scratch repo errored, 0 of 10 sequential
ones did. Two overlapping matrix sweeps on a hive with no persisted
Space / chat-room id both miss the by-name lookup and both createRoom
(from reading the code, not reproduced against a homeserver). On every
boot the MatrixSweep DAG node and the main.rs loop's immediate first
call ran at once.

Every sweep now runs as a job node, and each sweep's node holds its own
capacity-1 queue resource (Resource::MatrixSweep,
Resource::KnowledgeTree), the MetaWindow pattern: the scheduler never
starts a second pass of one sweep while the first holds the resource,
and different sweeps still run side by side.

- templates::matrix_sweep / templates::knowledge_pull build the node
  with its resource; boot, the periodic loops and the swarm event all
  use them.
- JobQueue::insert_unless_live folds a submission into a live node of
  the same kind instead of queueing another. Periodic ticks fold into a
  queued or running pass. The swarm knowledge event folds into a queued
  pull only, and queues one behind a running pull, which may have
  fetched before the push.
- The main.rs matrix loop no longer sweeps immediately at startup; the
  boot MatrixSweep node is the startup pass, as KnowledgePull already
  was for knowledge.
- The executors bound each pass (10 min matrix, 5 min knowledge), since
  a hung pass would otherwise hold its resource against every later one,
  and own the sweep-health banners, so every pass reports to them.

Replaces the SweepLock version of this branch, per review.

Refs #4723

hive-c0re/src/job_queue/exec.rs

@@ -6,7 +6,7 @@
 //! `Swap` tail); DAG-level failure handling is cancel-downstream in
 //! the queue.
 
-use std::sync::Arc;
+use std::sync::{Arc, Mutex, OnceLock};
 
 use anyhow::{Context as _, Result};
 
@@ -15,6 +15,7 @@ use hive_jobq::{NodeId, TerminalState};
 use super::model::NodeKind;
 use crate::coordinator::Coordinator;
 use crate::power::{ReconcileAction, reconcile_action};
+use crate::stats::sweep_health::{self, SweepHealth};
 
 /// Max time `Drain` waits for the harness to run its stop-checkpoint
 /// turn before falling back to the hard stop. Generous — a checkpoint
@@ -166,19 +167,57 @@ async fn run_forge_sweep() -> Result<()> {
     Ok(())
 }
 
-/// Boot-time matrix user/space sweep as a DAG node — see
-/// [`NodeKind::MatrixSweep`]. Reports failure as the node's own error so a
-/// failed boot sweep is visible on the dashboard; the debounced
-/// `sweep_health`-driven warning banner is a separate concern owned by the
-/// periodic loop in `main.rs`, unaffected by this node's own outcome.
+/// Matrix user/space sweep as a DAG node — see [`NodeKind::MatrixSweep`].
+/// Every pass reports here, whoever submitted it: as the node's own error on
+/// the dashboard, and into the debounced banner.
 async fn run_matrix_sweep() -> Result<()> {
-    if crate::matrix::ensure_all().await {
+    let ok = tokio::time::timeout(MATRIX_SWEEP_DEADLINE, crate::matrix::ensure_all())
+        .await
+        .unwrap_or_else(|_| {
+            tracing::warn!(deadline = ?MATRIX_SWEEP_DEADLINE, "matrix: sweep timed out");
+            false
+        });
+    let mut health = matrix_sweep_health()
+        .lock()
+        .unwrap_or_else(std::sync::PoisonError::into_inner);
+    if ok {
+        health.record_ok();
         Ok(())
     } else {
+        health.record_err(matrix_sweep_banner);
         anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)")
     }
 }
 
+/// A pass still running holds [`Resource::MatrixSweep`] against every later
+/// one. Each HTTP call has its own 10 s timeout, but the call count grows with
+/// the agent count and `lifecycle::list()` has none. A pass dropped here leaves
+/// at worst a created but unpersisted room, which the next pass adopts by name.
+///
+/// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep
+const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10);
+
+/// Banner for a matrix sweep that keeps failing. Two misses in a row, so one
+/// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the
+/// dashboard; cleared by the next clean pass.
+fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
+    static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new();
+    HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2)))
+}
+
+fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String {
+    let age = ctx.since_last_ok.map_or_else(
+        || "no success this session".to_owned(),
+        |d| format!("last ok {} ago", sweep_health::fmt_age(d)),
+    );
+    format!(
+        "matrix user/space sweep failing ({} consecutive, {age}) \
+         — some agents may be missing matrix accounts, space membership, \
+         or chat-room invites",
+        ctx.consecutive
+    )
+}
+
 /// Boot-time Forgejo webhook management as a DAG node — see
 /// [`NodeKind::WebhookRegister`]. Mirrors the guard chain the
 /// `tokio::spawn` block it replaced used: no-op (not an error) when the
@@ -207,13 +246,55 @@ async fn run_webhook_register() -> Result<()> {
     Ok(())
 }
 
-/// Boot-time `/knowledge` pull as a DAG node — see
-/// [`NodeKind::KnowledgePull`]. Unlike the `main.rs` periodic loop's
-/// startup call, a failure here is *not* swallowed to debug level: the node
-/// exists so a failed boot pull is visible on the dashboard rather than
-/// only in the journal.
+/// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every
+/// pass reports here, whoever submitted it: as the node's own error on the
+/// dashboard, and into the debounced banner.
 async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
-    crate::workers::knowledge::pull(coord).await
+    let result = tokio::time::timeout(
+        KNOWLEDGE_PULL_DEADLINE,
+        crate::workers::knowledge::pull(coord),
+    )
+    .await
+    .unwrap_or_else(|_| {
+        Err(anyhow::anyhow!(
+            "knowledge pull timed out after {KNOWLEDGE_PULL_DEADLINE:?}"
+        ))
+    });
+    let mut health = knowledge_pull_health()
+        .lock()
+        .unwrap_or_else(std::sync::PoisonError::into_inner);
+    match &result {
+        Ok(()) => health.record_ok(),
+        Err(e) => {
+            let err = format!("{e:#}");
+            health.record_err(|ctx| {
+                let age = ctx.since_last_ok.map_or_else(
+                    || "no success this session".to_owned(),
+                    |d| format!("last ok {} ago", sweep_health::fmt_age(d)),
+                );
+                format!(
+                    "knowledge repo pull failing ({} consecutive, {age}) \
+                     — /knowledge is stale until it recovers: {err}",
+                    ctx.consecutive
+                )
+            });
+        }
+    }
+    result
+}
+
+/// A pull still running holds [`Resource::KnowledgeTree`] against every later
+/// one. The `reset` / `clean` / `pull` git children are `kill_on_drop`, so the
+/// deadline kills them.
+///
+/// [`Resource::KnowledgeTree`]: super::resource::Resource::KnowledgeTree
+const KNOWLEDGE_PULL_DEADLINE: std::time::Duration = std::time::Duration::from_mins(5);
+
+/// Banner for a `/knowledge` pull that keeps failing: three misses in a row,
+/// cleared by the next successful pull.
+fn knowledge_pull_health() -> &'static Mutex<SweepHealth> {
+    static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new();
+    HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("knowledge_pull", "warn", 3)))
 }
 
 /// Boot-time swarm wanted-state pull as a DAG node — see