swarm-controller: cap a subagent terminal stream's disk use
The controller-created term-sub-<agent> stream had max_age only, so a publishing agent could grow it without bound for 24h. Add a 64 MiB max_bytes cap with discard: Old (oldest rows drop first, publish never fails on the cap), and size max_message_size off the queue's live max_payload rather than a hardcoded guess.
This commit is contained in:
parent
f9feb93ced
commit
e21546d08d
2 changed files with 56 additions and 16 deletions
|
|
@ -174,10 +174,12 @@ subagent daemon: each subagent's rows go to `$SWARM.term.<agent>.sub.<subagent>`
|
||||||
classified the same way. The queue grants that family to the agent's own queue
|
classified the same way. The queue grants that family to the agent's own queue
|
||||||
credential alone, so the daemon reads it from the store under the agent's store
|
credential alone, so the daemon reads it from the store under the agent's store
|
||||||
identity, exactly as the harness does, and publishes nothing without it. The
|
identity, exactly as the harness does, and publishes nothing without it. The
|
||||||
queue keeps these rows in the stream `term-sub-<agent>` for 24 hours, and the
|
queue keeps these rows in the stream `term-sub-<agent>` for 24 hours or 64 MiB,
|
||||||
swarm lists an agent's subagents from that stream's subjects. The swarm
|
whichever comes first, dropping the oldest rows once either limit kicks in so
|
||||||
controller creates that stream for every agent a hive's wanted state names,
|
a publish never fails because of it, and the swarm lists an agent's subagents
|
||||||
within a minute of the agent appearing there. The agent's grant is publish on its own
|
from that stream's subjects. The swarm controller creates that stream for
|
||||||
|
every agent a hive's wanted state names, within a minute of the agent
|
||||||
|
appearing there. The agent's grant is publish on its own
|
||||||
`$SWARM.term.<agent>.sub.>` and no `JetStream` subject, so the stream's
|
`$SWARM.term.<agent>.sub.>` and no `JetStream` subject, so the stream's
|
||||||
subjects and limits are the controller's and never the agent's. Subagents
|
subjects and limits are the controller's and never the agent's. Subagents
|
||||||
publish output only and read nothing.
|
publish output only and read nothing.
|
||||||
|
|
|
||||||
|
|
@ -31,6 +31,13 @@ const SUBAGENT_TOKEN: &str = "sub";
|
||||||
/// its last output.
|
/// its last output.
|
||||||
pub const MAX_AGE: std::time::Duration = std::time::Duration::from_hours(24);
|
pub const MAX_AGE: std::time::Duration = std::time::Duration::from_hours(24);
|
||||||
|
|
||||||
|
/// How large one agent's stream may grow before the discard policy reclaims
|
||||||
|
/// space. 64 MiB is enough rows from a day of ordinary subagent chatter to
|
||||||
|
/// stay well inside it while still bounding the disk one compromised or
|
||||||
|
/// runaway agent can claim for 24h.
|
||||||
|
#[cfg(feature = "subagent-term")]
|
||||||
|
const MAX_BYTES: i64 = 64 * 1024 * 1024;
|
||||||
|
|
||||||
/// The stream holding `agent`'s subagent rows: `term-sub-<agent>`.
|
/// The stream holding `agent`'s subagent rows: `term-sub-<agent>`.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn stream_name(agent: &str) -> String {
|
pub fn stream_name(agent: &str) -> String {
|
||||||
|
|
@ -78,6 +85,27 @@ pub fn subagent_names<'a>(agent: &str, subjects: impl IntoIterator<Item = &'a st
|
||||||
names
|
names
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The config [`open_or_create`] creates `agent`'s stream with.
|
||||||
|
///
|
||||||
|
/// `max_message_size` is taken as a parameter rather than read from a
|
||||||
|
/// constant here: it tracks the queue's live `max_payload`
|
||||||
|
/// ([`crate::max_payload`]), the same value the publisher already sizes its
|
||||||
|
/// rows against, so the stream never rejects a row the queue itself
|
||||||
|
/// accepted.
|
||||||
|
#[cfg(feature = "subagent-term")]
|
||||||
|
fn stream_config(agent: &str, max_message_size: i32) -> async_nats::jetstream::stream::Config {
|
||||||
|
async_nats::jetstream::stream::Config {
|
||||||
|
name: stream_name(agent),
|
||||||
|
description: Some(format!("Terminal rows of {agent}'s subagents")),
|
||||||
|
subjects: vec![stream_subjects(agent)],
|
||||||
|
max_age: MAX_AGE,
|
||||||
|
max_bytes: MAX_BYTES,
|
||||||
|
discard: async_nats::jetstream::stream::DiscardPolicy::Old,
|
||||||
|
max_message_size,
|
||||||
|
..Default::default()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Open `agent`'s subagent stream, creating it if it does not exist yet. An
|
/// Open `agent`'s subagent stream, creating it if it does not exist yet. An
|
||||||
/// existing stream is opened as it is.
|
/// existing stream is opened as it is.
|
||||||
#[cfg(feature = "subagent-term")]
|
#[cfg(feature = "subagent-term")]
|
||||||
|
|
@ -91,13 +119,8 @@ pub async fn open_or_create(
|
||||||
Ok(stream) => Ok(stream),
|
Ok(stream) => Ok(stream),
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
tracing::info!(stream = %name, reason = %e, "subagent terminal stream not available, creating it");
|
tracing::info!(stream = %name, reason = %e, "subagent terminal stream not available, creating it");
|
||||||
js.create_stream(async_nats::jetstream::stream::Config {
|
let max_message_size = i32::try_from(crate::max_payload(client)).unwrap_or(i32::MAX);
|
||||||
name: name.clone(),
|
js.create_stream(stream_config(agent, max_message_size))
|
||||||
description: Some(format!("Terminal rows of {agent}'s subagents")),
|
|
||||||
subjects: vec![stream_subjects(agent)],
|
|
||||||
max_age: MAX_AGE,
|
|
||||||
..Default::default()
|
|
||||||
})
|
|
||||||
.await
|
.await
|
||||||
.map_err(|source| crate::Error::CreateAgentStream {
|
.map_err(|source| crate::Error::CreateAgentStream {
|
||||||
stream: name,
|
stream: name,
|
||||||
|
|
@ -155,4 +178,19 @@ mod tests {
|
||||||
];
|
];
|
||||||
assert_eq!(subagent_names("iris", subjects), ["alpha", "zed"]);
|
assert_eq!(subagent_names("iris", subjects), ["alpha", "zed"]);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The config the controller creates caps disk per agent and never
|
||||||
|
/// fails a publish because of that cap — `discard: Old` reclaims the
|
||||||
|
/// oldest rows instead of rejecting the newest.
|
||||||
|
#[cfg(feature = "subagent-term")]
|
||||||
|
#[test]
|
||||||
|
fn stream_config_caps_bytes_and_discards_the_oldest() {
|
||||||
|
let cfg = stream_config("iris", 8_388_608);
|
||||||
|
assert_eq!(cfg.max_bytes, MAX_BYTES);
|
||||||
|
assert_eq!(
|
||||||
|
cfg.discard,
|
||||||
|
async_nats::jetstream::stream::DiscardPolicy::Old
|
||||||
|
);
|
||||||
|
assert_eq!(cfg.max_message_size, 8_388_608);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue