hive-c0re: serialise the matrix and knowledge sweeps through the job queue
The matrix sweep and the /knowledge pull each had concurrent callers (#4723 item 4). Two overlapping knowledge pulls fail on .git/index.lock and the remote-tracking ref lock: 30 of 30 concurrent replays of the reset/clean/pull sequence in a scratch repo errored, 0 of 10 sequential ones did. Two overlapping matrix sweeps on a hive with no persisted Space / chat-room id both miss the by-name lookup and both createRoom (from reading the code, not reproduced against a homeserver). On every boot the MatrixSweep DAG node and the main.rs loop's immediate first call ran at once. Every sweep now runs as a job node, and each sweep's node holds its own capacity-1 queue resource (Resource::MatrixSweep, Resource::KnowledgeTree), the MetaWindow pattern: the scheduler never starts a second pass of one sweep while the first holds the resource, and different sweeps still run side by side. - templates::matrix_sweep / templates::knowledge_pull build the node with its resource; boot, the periodic loops and the swarm event all use them. - JobQueue::insert_unless_live folds a submission into a live node of the same kind instead of queueing another. Periodic ticks fold into a queued or running pass. The swarm knowledge event folds into a queued pull only, and queues one behind a running pull, which may have fetched before the push. - The main.rs matrix loop no longer sweeps immediately at startup; the boot MatrixSweep node is the startup pass, as KnowledgePull already was for knowledge. - The executors bound each pass (10 min matrix, 5 min knowledge), since a hung pass would otherwise hold its resource against every later one, and own the sweep-health banners, so every pass reports to them. Replaces the SweepLock version of this branch, per review. Refs #4723
This commit is contained in:
parent
2252c55df8
commit
1d4c77d2c8
14 changed files with 352 additions and 105 deletions
|
|
@ -192,6 +192,39 @@ impl JobQueue {
|
|||
Ok(named)
|
||||
}
|
||||
|
||||
/// Insert the single node `declare` names, unless a node of `kind` is
|
||||
/// already in one of the `fold_into` states. `None` means the caller's
|
||||
/// request folded into that node and nothing was inserted.
|
||||
///
|
||||
/// For a standalone sweep, where a pass that has not started yet already
|
||||
/// covers "one more pass": it reads whatever is current when it runs. The
|
||||
/// read and the insert share one lock, so two racing callers cannot both
|
||||
/// miss the other.
|
||||
///
|
||||
/// # Errors
|
||||
/// Propagates a graph-insert error.
|
||||
pub fn insert_unless_live(
|
||||
&self,
|
||||
kind: &NodeKind,
|
||||
fold_into: &[State],
|
||||
declare: impl FnOnce(&JobBuilder) -> Handle<'_>,
|
||||
) -> anyhow::Result<Option<NodeId>> {
|
||||
let mut inner = self.lock();
|
||||
let live = inner
|
||||
.graph()
|
||||
.nodes()
|
||||
.any(|n| &n.payload == kind && fold_into.contains(&n.state));
|
||||
if live {
|
||||
return Ok(None);
|
||||
}
|
||||
let named = inner
|
||||
.insert_job(None, |b| vec![declare(b).guid()])
|
||||
.map_err(|e| anyhow::anyhow!("job_queue: graph insert failed: {e}"))?;
|
||||
drop(inner);
|
||||
self.notify.notify_one();
|
||||
Ok(named.first().copied())
|
||||
}
|
||||
|
||||
/// The scheduler itself, for `hive_jobq`'s run-loop seam
|
||||
/// (`Scheduler::claim_next`), which takes exactly this type.
|
||||
///
|
||||
|
|
|
|||
Loading…
Reference in a new issue