1d4c77d2c8
The matrix sweep and the /knowledge pull each had concurrent callers (#4723 item 4). Two overlapping knowledge pulls fail on .git/index.lock and the remote-tracking ref lock: 30 of 30 concurrent replays of the reset/clean/pull sequence in a scratch repo errored, 0 of 10 sequential ones did. Two overlapping matrix sweeps on a hive with no persisted Space / chat-room id both miss the by-name lookup and both createRoom (from reading the code, not reproduced against a homeserver). On every boot the MatrixSweep DAG node and the main.rs loop's immediate first call ran at once. Every sweep now runs as a job node, and each sweep's node holds its own capacity-1 queue resource (Resource::MatrixSweep, Resource::KnowledgeTree), the MetaWindow pattern: the scheduler never starts a second pass of one sweep while the first holds the resource, and different sweeps still run side by side. - templates::matrix_sweep / templates::knowledge_pull build the node with its resource; boot, the periodic loops and the swarm event all use them. - JobQueue::insert_unless_live folds a submission into a live node of the same kind instead of queueing another. Periodic ticks fold into a queued or running pass. The swarm knowledge event folds into a queued pull only, and queues one behind a running pull, which may have fetched before the push. - The main.rs matrix loop no longer sweeps immediately at startup; the boot MatrixSweep node is the startup pass, as KnowledgePull already was for knowledge. - The executors bound each pass (10 min matrix, 5 min knowledge), since a hung pass would otherwise hold its resource against every later one, and own the sweep-health banners, so every pass reports to them. Replaces the SweepLock version of this branch, per review. Refs #4723
61 lines
3 KiB
Rust
61 lines
3 KiB
Rust
//! The concrete resource type the rebuild queue schedules over — the bridge
|
|
//! from hive-c0re's [`super::model::NodeKind`] onto the domain-agnostic `hive-jobq` crate.
|
|
//! `hive-jobq` is generic over a resource type `R: Clone + Eq + Hash` and a node
|
|
//! payload `N`; here `R` is [`Resource`] and `N` is [`super::model::NodeKind`] directly (each
|
|
//! variant carries the agent it targets).
|
|
//!
|
|
//! **A node's resources are declared where the node is constructed** with
|
|
//! `.needs(…)`, not derived from its kind. Deriving them made the requirement a
|
|
//! property of the *kind*, which let a kind that happened to run under a
|
|
//! holding ancestor declare nothing at all.
|
|
//!
|
|
//! All of a node's resource edges are acquired **atomically**
|
|
//! (`try_acquire_all`) — a node never holds one resource while waiting on
|
|
//! another, so the multi-resource nodes (a `MetaLock` wants a build slot *and*
|
|
//! the meta window) cannot deadlock against each other.
|
|
|
|
/// The two resource classes the queue gates concurrency on, as the crate's
|
|
/// generic resource type `R`. What each holds, who declares it, and the
|
|
/// exemptions: see `docs/scheduler/coordinator.md`'s _Scheduler semantics_
|
|
/// (build slots, the per-agent lease) and _Two further layers protect the
|
|
/// meta repo_ (the deploy window) sections — this enum stays the one-line
|
|
/// summary, not a second copy.
|
|
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
|
|
pub enum Resource {
|
|
/// One of the `buildSlots` permits, held by a nix-heavy node for its
|
|
/// duration.
|
|
BuildSlot,
|
|
/// The per-agent lifecycle lease — globally exclusive per agent across
|
|
/// all DAGs.
|
|
Agent(String),
|
|
/// The meta-repo mutation window — a global singleton held by any node
|
|
/// that mutates the meta repo, so two meta mutations never interleave.
|
|
MetaWindow,
|
|
/// The `/knowledge` working tree, held by every `KnowledgePull`. Two
|
|
/// overlapping `reset` / `clean` / `pull` passes there fail on
|
|
/// `.git/index.lock` and the remote-tracking ref lock.
|
|
KnowledgeTree,
|
|
/// The hive's matrix provisioning, held by every `MatrixSweep`. Two
|
|
/// overlapping passes on a hive with no persisted Space / chat-room id both
|
|
/// miss the by-name lookup and both `createRoom`, leaving a duplicate room
|
|
/// with agents invited to both.
|
|
MatrixSweep,
|
|
}
|
|
|
|
/// This resource's name on the generic graph wire.
|
|
///
|
|
/// `hive_jobq` is generic over the resource type, so a viewer that can render
|
|
/// any graph gets a string here rather than this enum. The `agent:` prefix
|
|
/// keeps the per-agent leases from colliding with a hypothetical global
|
|
/// resource that happens to share an agent's name.
|
|
impl hive_jobq_wire::WireResource for Resource {
|
|
fn name(&self) -> String {
|
|
match self {
|
|
Resource::BuildSlot => "build-slot".to_owned(),
|
|
Resource::Agent(agent) => format!("agent:{agent}"),
|
|
Resource::MetaWindow => "meta-window".to_owned(),
|
|
Resource::KnowledgeTree => "knowledge-tree".to_owned(),
|
|
Resource::MatrixSweep => "matrix-sweep".to_owned(),
|
|
}
|
|
}
|
|
}
|