A replay consumer now has a 5s idle heartbeat, and the row stream ends at the first consumer error instead of skipping it. When the server drops the consumer, as on a restart, two missed heartbeats end the SSE response and the browser reconnects with Last-Event-ID. Before, the response stayed open and delivered nothing. A Last-Event-ID above the stream's newest sequence now replays as a fresh reader. The server accepts a start past the end and delivers nothing until the stream reaches it, so a stale id stalled the replay. swarm-controller's stream tests run against a real nats-server on loopback (`test_nats`, binary from HIVE_NATS_SERVER, exported by checks.nix and devshell.nix). They cover: ensuring the stream twice, an existing stream with other limits being opened as it is, the replay arriving before live rows, a reconnect getting every later row exactly once, and a dropped consumer ending the row stream. The limits test now compares against the subagent stream's own config. Two comments that described the agent's terminal route as live-only are corrected.
205 lines
7.5 KiB
Rust
205 lines
7.5 KiB
Rust
//! Subagent terminals: the subject each subagent's rows go to, and the
|
|
//! stream that keeps them.
|
|
//!
|
|
//! An agent's subagent daemon publishes each subagent's classified terminal
|
|
//! rows on `$SWARM.term.<agent>.sub.<subagent>`, under the agent's own queue
|
|
//! credential. The queue grants that credential publish on this subject
|
|
//! family and no `JetStream` subject.
|
|
//!
|
|
//! The stream exists so the swarm can list an agent's subagents: subagents
|
|
//! are spawned on demand under caller-chosen names, and the set of subjects
|
|
//! the stream holds is the list. `swarm-controller` creates one stream per
|
|
//! agent with the fixed config [`crate::subagent_term::open_or_create`]
|
|
//! spells, and takes the list from its subject counts
|
|
//! ([`crate::subagent_term::subagent_names`]).
|
|
//!
|
|
//! The name and the subject live here because three crates must agree on
|
|
//! them: the publisher in `hive-subagent-mcp`, `swarm-controller`, and the
|
|
//! queue grant in `swarm-nats.nix`, whose `{agent}` template spells the same
|
|
//! subjects.
|
|
|
|
use crate::TERM_SUBJECT_PREFIX;
|
|
|
|
/// The token between the agent and the subagent name. A subagent subject has
|
|
/// two more tokens than the agent's own `$SWARM.term.<agent>`, so neither
|
|
/// family's subscriber receives the other's rows.
|
|
const SUBAGENT_TOKEN: &str = "sub";
|
|
|
|
/// How long the stream keeps a row. The subagent list is the subjects with a
|
|
/// row inside this window, so a subagent drops off the list this long after
|
|
/// its last output.
|
|
pub const MAX_AGE: std::time::Duration = std::time::Duration::from_hours(24);
|
|
|
|
/// How large one agent's stream may grow before the discard policy reclaims
|
|
/// space. 64 MiB is enough rows from a day of ordinary subagent chatter to
|
|
/// stay well inside it while still bounding the disk one compromised or
|
|
/// runaway agent can claim for 24h.
|
|
#[cfg(feature = "subagent-term")]
|
|
const MAX_BYTES: i64 = 64 * 1024 * 1024;
|
|
|
|
/// The stream holding `agent`'s subagent rows: `term-sub-<agent>`.
|
|
#[must_use]
|
|
pub fn stream_name(agent: &str) -> String {
|
|
format!("term-sub-{agent}")
|
|
}
|
|
|
|
/// The subject `subagent`'s rows are published on.
|
|
#[must_use]
|
|
pub fn subject(agent: &str, subagent: &str) -> String {
|
|
format!("{TERM_SUBJECT_PREFIX}.{agent}.{SUBAGENT_TOKEN}.{subagent}")
|
|
}
|
|
|
|
/// Every [`subject`] at once: the reader's subscribe grant, which has no
|
|
/// roster to enumerate agents or subagents from.
|
|
#[must_use]
|
|
pub fn subject_wildcard() -> String {
|
|
subject("*", "*")
|
|
}
|
|
|
|
/// Every subject [`stream_name`]'s stream captures, which is also the publish
|
|
/// grant's shape.
|
|
#[must_use]
|
|
pub fn stream_subjects(agent: &str) -> String {
|
|
format!("{TERM_SUBJECT_PREFIX}.{agent}.{SUBAGENT_TOKEN}.>")
|
|
}
|
|
|
|
/// The subagent a stream subject names, or `None` for a subject that is not
|
|
/// exactly one subagent token under `agent`'s family.
|
|
#[must_use]
|
|
pub fn subagent_from_subject<'a>(agent: &str, subject: &'a str) -> Option<&'a str> {
|
|
let name = subject
|
|
.strip_prefix(TERM_SUBJECT_PREFIX)?
|
|
.strip_prefix('.')?
|
|
.strip_prefix(agent)?
|
|
.strip_prefix('.')?
|
|
.strip_prefix(SUBAGENT_TOKEN)?
|
|
.strip_prefix('.')?;
|
|
(!name.is_empty() && !name.contains('.')).then_some(name)
|
|
}
|
|
|
|
/// The subagents named by a stream's subjects, sorted and deduplicated.
|
|
/// Subjects outside `agent`'s family are skipped.
|
|
#[must_use]
|
|
pub fn subagent_names<'a>(agent: &str, subjects: impl IntoIterator<Item = &'a str>) -> Vec<String> {
|
|
let mut names: Vec<String> = subjects
|
|
.into_iter()
|
|
.filter_map(|s| subagent_from_subject(agent, s))
|
|
.map(str::to_owned)
|
|
.collect();
|
|
names.sort_unstable();
|
|
names.dedup();
|
|
names
|
|
}
|
|
|
|
/// The config [`open_or_create`] creates `agent`'s stream with.
|
|
///
|
|
/// `max_message_size` is taken as a parameter rather than read from a
|
|
/// constant here: it tracks the queue's live `max_payload`
|
|
/// ([`crate::max_payload`]), the same value the publisher already sizes its
|
|
/// rows against, so the stream never rejects a row the queue itself
|
|
/// accepted.
|
|
#[cfg(feature = "subagent-term")]
|
|
pub(crate) fn stream_config(
|
|
agent: &str,
|
|
max_message_size: i32,
|
|
) -> async_nats::jetstream::stream::Config {
|
|
async_nats::jetstream::stream::Config {
|
|
name: stream_name(agent),
|
|
description: Some(format!("Terminal rows of {agent}'s subagents")),
|
|
subjects: vec![stream_subjects(agent)],
|
|
max_age: MAX_AGE,
|
|
max_bytes: MAX_BYTES,
|
|
discard: async_nats::jetstream::stream::DiscardPolicy::Old,
|
|
max_message_size,
|
|
..Default::default()
|
|
}
|
|
}
|
|
|
|
/// Open `agent`'s subagent stream, creating it if it does not exist yet. An
|
|
/// existing stream is opened as it is.
|
|
#[cfg(feature = "subagent-term")]
|
|
pub async fn open_or_create(
|
|
client: &async_nats::Client,
|
|
agent: &str,
|
|
) -> Result<async_nats::jetstream::stream::Stream, crate::Error> {
|
|
let js = async_nats::jetstream::new(client.clone());
|
|
let name = stream_name(agent);
|
|
match js.get_stream(&name).await {
|
|
Ok(stream) => Ok(stream),
|
|
Err(e) => {
|
|
tracing::info!(stream = %name, reason = %e, "subagent terminal stream not available, creating it");
|
|
let max_message_size = i32::try_from(crate::max_payload(client)).unwrap_or(i32::MAX);
|
|
js.create_stream(stream_config(agent, max_message_size))
|
|
.await
|
|
.map_err(|source| crate::Error::CreateAgentStream {
|
|
stream: name,
|
|
source,
|
|
})
|
|
}
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
/// The names the grant in `swarm-nats.nix` spells with `{agent}`.
|
|
#[test]
|
|
fn names_match_the_queue_grant() {
|
|
assert_eq!(stream_name("iris"), "term-sub-iris");
|
|
assert_eq!(subject("iris", "scout"), "$SWARM.term.iris.sub.scout");
|
|
assert_eq!(stream_subjects("iris"), "$SWARM.term.iris.sub.>");
|
|
}
|
|
|
|
#[test]
|
|
fn a_subject_reads_back_as_its_subagent() {
|
|
assert_eq!(
|
|
subagent_from_subject("iris", &subject("iris", "scout")),
|
|
Some("scout")
|
|
);
|
|
}
|
|
|
|
/// The agent's own terminal, another agent's subagent, and anything
|
|
/// deeper than one token are not subagents of `iris`.
|
|
#[test]
|
|
fn other_subjects_name_no_subagent() {
|
|
for s in [
|
|
"$SWARM.term.iris",
|
|
"$SWARM.term.h1.iris",
|
|
"$SWARM.term.iris.sub",
|
|
"$SWARM.term.iris.sub.",
|
|
"$SWARM.term.iris.sub.a.b",
|
|
"$SWARM.term.iris-2.sub.scout",
|
|
"$SWARM.term.argus.sub.scout",
|
|
"$SWARM.agent-state.iris.sub.scout",
|
|
] {
|
|
assert_eq!(subagent_from_subject("iris", s), None, "{s}");
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn the_list_is_sorted_unique_and_scoped_to_the_agent() {
|
|
let subjects = [
|
|
"$SWARM.term.iris.sub.zed",
|
|
"$SWARM.term.iris.sub.alpha",
|
|
"$SWARM.term.argus.sub.other",
|
|
"$SWARM.term.iris.sub.alpha",
|
|
];
|
|
assert_eq!(subagent_names("iris", subjects), ["alpha", "zed"]);
|
|
}
|
|
|
|
/// The config the controller creates caps disk per agent and never
|
|
/// fails a publish because of that cap — `discard: Old` reclaims the
|
|
/// oldest rows instead of rejecting the newest.
|
|
#[cfg(feature = "subagent-term")]
|
|
#[test]
|
|
fn stream_config_caps_bytes_and_discards_the_oldest() {
|
|
let cfg = stream_config("iris", 8_388_608);
|
|
assert_eq!(cfg.max_bytes, MAX_BYTES);
|
|
assert_eq!(
|
|
cfg.discard,
|
|
async_nats::jetstream::stream::DiscardPolicy::Old
|
|
);
|
|
assert_eq!(cfg.max_message_size, 8_388_608);
|
|
}
|
|
}
|