vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm: keep an agent's terminal rows and replay them on attach

Browse source 
swarm-controller now creates `term-main-<agent>` for every live agent, in
the same one-minute pass and with the same config style as
`term-sub-<agent>` (#4897): subjects `$SWARM.term.<agent>` only, max_age
24h, max_bytes 64 MiB, discard Old, max_message_size from the queue's
max_payload. An existing stream is opened as it is. The agent keeps its
plain publish on `$SWARM.term.<agent>` and gets no JetStream subject.

`GET /api/agents/{name}/term/stream` creates an ephemeral push consumer on
that stream starting 100 rows back, so a swarm-ui terminal opens with
recent history and then follows new rows, in order, from one
subscription. Each SSE event carries the row's stream sequence as its id,
and a reconnect's Last-Event-ID resumes after it. Without the stream the
route falls back to the live core subscription it used before.

The stream is `term-main-<agent>`, not `term-<agent>`: an agent may be
named `sub-x`, whose `term-<agent>` would be agent `x`'s subagent stream.
Its one literal subject cannot match `$SWARM.term.<agent>.sub.>`, which
needs more tokens, so the two streams never overlap.

The reader gains `$JS.API.CONSUMER.CREATE.*.*.$SWARM.term.*`, the
extended create form whose last tokens are the consumer's filter. Checked
against nats-server 2.14.4, the trailing `*` also admits a literal `*` or
`>` filter, so a consumer under it reaches any terminal row, main or
subagent; those are rows the reader already subscribes to and reads with
`STREAM.MSG.GET.*`.

Closes #4826
This commit is contained in:
atlas 2026-10-11 12:35:31 +02:00
commit ae9ca4331e
12 changed files with 533 additions and 38 deletions
Download patch file Download diff file Expand all files Collapse all files

8
swarm-controller/Cargo.toml
View file

@ -92,7 +92,13 @@ swarm-authelia-bridge-sock.workspace = true
#
# `subagent-term`: this daemon creates every agent's subagent stream, with the
# config the crate spells.
swarm-queue-client = { workspace = true, features = ["kv", "subagent-term"] }
# `agent-term`: likewise every agent's terminal stream, plus the consumer it
# replays that stream through.
swarm-queue-client = { workspace = true, features = [
"kv",
"subagent-term",
"agent-term",
] }
# `matrix_account.rs` writes the credential this daemon's route accepts. Same
# crate the hive reads it back with, which is the point: the path, the field
# name and the object's shape are agreements between the two ends, and a

6
swarm-controller/src/main.rs
View file

@ -2621,8 +2621,8 @@ fn spawn_agent_renewal(
});
}
/// Start creating every live agent's subagent stream when a swarm queue is
/// wired up. Lifted out of `main` for `clippy::too_many_lines`.
/// Start creating every live agent's subagent and terminal streams when a swarm
/// queue is wired up. Lifted out of `main` for `clippy::too_many_lines`.
fn spawn_subagent_streams(status: Option<&Arc<status::StatusReader>>, hives: &[HiveEntry]) {
let (Some(status), Some(wanted)) = (status, wanted_writer(status)) else {
return;
@ -2631,7 +2631,7 @@ fn spawn_subagent_streams(status: Option<&Arc<status::StatusReader>>, hives: &[H
subagent_term::spawn(status.queue_client(), wanted, hives);
}
/// The per-agent background passes: subagent streams and schedule keys.
/// The per-agent background passes: terminal streams and schedule keys.
/// Lifted out of `main` for `clippy::too_many_lines`.
fn spawn_per_agent_upkeep(status: Option<&Arc<status::StatusReader>>, hives: &[HiveEntry]) {
spawn_subagent_streams(status, hives);

19
swarm-controller/src/subagent_term.rs
View file

@ -5,7 +5,8 @@
//! `$SWARM.term.{agent}.sub.{subagent}` under the agent's own queue credential,
//! which may publish there and nothing else of these streams. [`spawn`]
//! creates the stream `term-sub-{agent}` for every live agent, with the config
//! `swarm_queue_client::subagent_term` spells. Subagents are spawned on demand
//! `swarm_queue_client::subagent_term` spells, and in the same pass the
//! agent's own terminal stream (`swarm_queue_client::agent_term`). Subagents are spawned on demand
//! under caller-chosen names, so the list is the set of subjects that stream
//! holds rows for, read with `STREAM.INFO`. A missing stream lists nothing.
//! Each subagent carries the stream's own publish time of its newest row, so
@ -30,7 +31,7 @@ use axum::http::StatusCode;
use axum::response::sse::{Event, KeepAlive, Sse};
use futures_util::{Stream, StreamExt as _, TryStreamExt as _};
use serde::Serialize;
use swarm_queue_client::subagent_term;
use swarm_queue_client::{agent_term, subagent_term};
use utoipa::ToSchema;
use crate::AppState;
@ -49,8 +50,9 @@ const LOOKUP_TIMEOUT: Duration = Duration::from_secs(2);
/// How many newest-row lookups [`list_subagents`] has in flight at once.
const LOOKUP_CONCURRENCY: usize = 16;
/// Create the subagent stream of every live agent that has none, now and every
/// [`ENSURE_INTERVAL`] after. An existing stream is left as it is.
/// Create the subagent stream and the terminal stream (`agent_term`) of every
/// live agent that lacks one, now and every [`ENSURE_INTERVAL`] after. An
/// existing stream is left as it is.
///
/// The live agents are the ones some hive's wanted-state declaration names in
/// a state other than `Destroyed` (`agent_renewal::live_agents`). A pass skips
@ -63,7 +65,7 @@ pub(crate) fn spawn(client: async_nats::Client, wanted: Arc<WantedWriter>, hives
loop {
ticker.tick().await;
if swarm_queue_client::ensure_connected(&client).is_err() {
tracing::debug!("subagent streams: queue not connected; next pass");
tracing::debug!("terminal streams: queue not connected; next pass");
continue;
}
for agent in live_agents(&wanted, &hives).await {
@ -74,6 +76,13 @@ pub(crate) fn spawn(client: async_nats::Client, wanted: Arc<WantedWriter>, hives
"subagent streams: creating the agent's stream failed; next pass"
);
}
if let Err(e) = agent_term::open_or_create(&client, &agent).await {
tracing::warn!(
%agent,
error = %swarm_queue_client::chain(&e),
"terminal streams: creating the agent's stream failed; next pass"
);
}
}
}
});

148
swarm-controller/src/term_stream.rs
View file

@ -1,4 +1,4 @@
//! `GET /api/agents/{name}/term/stream` — a live SSE relay of one agent's
//! `GET /api/agents/{name}/term/stream` — an SSE relay of one agent's
//! terminal rows, sourced straight from the swarm queue.
//!
//! **No `hive` in the path, on purpose.** An agent is not pinned to a
@ -6,15 +6,20 @@
//! moment it did. `crate::agent_status`'s bucket is read only to answer
//! 404 for an agent that has never reported.
//!
//! **Live tail only, on purpose.** `hive-agent` publishes each
//! already-classified `TermMsg` row to the core subject
//! `$SWARM.term.{agent}` (see `hive-agent::swarm_term`'s module doc) —
//! a core subject, not `JetStream`, so a subscriber who was not
//! listening missed the row, same as on the agent's own local SSE
//! stream. This handler relays exactly that: no replay, no last-N
//! history. The publish side's own retention half was deliberately
//! never built, and this route's own job — a live tail — never needed
//! it either.
//! **Replay, then the live tail, through one consumer.** `hive-agent`
//! publishes each already-classified `TermMsg` row to the core subject
//! `$SWARM.term.{agent}`, and the stream `term-main-{agent}`
//! (`swarm_queue_client::agent_term`) keeps them. The handler creates an
//! ephemeral consumer on that stream starting at the newest
//! `agent_term::REPLAY_ROWS` rows, so the reader gets those and then every
//! new row, in stream order, with no gap and no duplicate between the two.
//! Each event's `id` is the row's stream sequence; a browser that
//! reconnects sends it back as `Last-Event-ID` and resumes right after it.
//!
//! **No stream, no replay.** While the stream does not exist yet (up to a
//! minute after the agent is declared) or cannot be read, the handler
//! falls back to a core subscription: rows published while the reader is
//! attached, and nothing before.
//!
//! **The payload is passed through opaquely**, the same way
//! `crate::status::HiveStatus::snapshot` stores a hive's status
@ -26,8 +31,11 @@ use std::convert::Infallible;
use std::time::SystemTime;
use axum::extract::{Path, State};
use axum::http::HeaderMap;
use axum::response::sse::{Event, KeepAlive, Sse};
use futures_util::stream::BoxStream;
use futures_util::{Stream, StreamExt as _};
use swarm_queue_client::agent_term;
use crate::AppState;
@ -38,7 +46,9 @@ use crate::AppState;
responses(
(status = 200, description = "server-sent event stream; each event's `data` is \
one already-classified TermMsg row, JSON as hive-agent published it \
(opaque to this daemon) — live only, no replay",
(opaque to this daemon); the newest retained rows first, then live rows. \
Each event's `id` is the row's stream sequence, and a `Last-Event-ID` \
request header resumes right after that row",
body = String, content_type = "text/event-stream"),
(status = 400, description = "the agent name is not shaped like an identifier \
(problem+json)", body = String),
@ -55,6 +65,7 @@ use crate::AppState;
pub(crate) async fn stream_agent_term(
State(state): State<AppState>,
Path(agent): Path<String>,
headers: HeaderMap,
) -> Result<Sse<impl Stream<Item = Result<Event, Infallible>>>, problem_details::ProblemDetails> {
let Some(status) = state.status.as_ref() else {
return Err(crate::error_problem(
@ -105,12 +116,91 @@ pub(crate) async fn stream_agent_term(
)
})?;
let subscriber = subscribe(&client, swarm_queue_client::term_subject(&agent), "term").await?;
let stream =
subscriber.map(|msg| Ok(Event::default().data(String::from_utf8_lossy(&msg.payload))));
let stream = match replay(&client, &agent, last_event_id(&headers)).await {
Ok(stream) => stream,
Err(error) => {
tracing::warn!(%agent, %error, "term stream: no replay, live rows only");
let subscriber =
subscribe(&client, swarm_queue_client::term_subject(&agent), "term").await?;
subscriber
.map(|msg| Ok(Event::default().data(String::from_utf8_lossy(&msg.payload))))
.boxed()
}
};
Ok(Sse::new(stream).keep_alive(KeepAlive::default()))
}
/// The agent's rows from its terminal stream: the replay
/// [`agent_term::replay_start`] picks, then every new row.
async fn replay(
client: &async_nats::Client,
agent: &str,
resume_after: Option<u64>,
) -> Result<BoxStream<'static, Result<Event, Infallible>>, String> {
let js = async_nats::jetstream::new(client.clone());
let name = agent_term::stream_name(agent);
let stream = js
.get_stream(&name)
.await
.map_err(|e| format!("opening {name}: {}", swarm_queue_client::chain(&e)))?;
let start = agent_term::replay_start(stream.cached_info().state.last_sequence, resume_after);
let deliver_subject = client.new_inbox();
// The inbox's last token is a fresh nuid, which is also a valid name.
let consumer_name = format!(
"term-replay-{}",
deliver_subject.rsplit('.').next().unwrap_or_default()
);
let config = agent_term::replay_consumer_config(agent, consumer_name, deliver_subject, start);
let messages = stream
.create_consumer(config)
.await
.map_err(|e| {
format!(
"creating a consumer on {name}: {}",
swarm_queue_client::chain(&e)
)
})?
.messages()
.await
.map_err(|e| format!("subscribing to {name}: {}", swarm_queue_client::chain(&e)))?;
tracing::info!(stream = %name, start, "term stream: client attached");
let agent = agent.to_owned();
Ok(messages
.filter_map(move |msg| {
let event = match msg {
Ok(msg) => Some(Ok(row_event(&msg))),
Err(e) => {
tracing::warn!(%agent, error = %e, "term stream: row skipped");
None
}
};
std::future::ready(event)
})
.boxed())
}
/// One row as an SSE event, carrying its stream sequence as the event `id`
/// when the delivery names one.
fn row_event(msg: &async_nats::jetstream::Message) -> Event {
let event = Event::default().data(String::from_utf8_lossy(&msg.payload));
match msg.info() {
Ok(info) => event.id(info.stream_sequence.to_string()),
Err(_) => event,
}
}
/// The stream sequence a reconnecting browser last received, from its
/// `Last-Event-ID` header. Absent or unparseable means a fresh reader.
fn last_event_id(headers: &HeaderMap) -> Option<u64> {
headers
.get("last-event-id")?
.to_str()
.ok()?
.trim()
.parse()
.ok()
}
/// Subscribe to `subject`. `what` names the relay in logs and errors.
pub(crate) async fn subscribe(
client: &async_nats::Client,
@ -127,3 +217,33 @@ pub(crate) async fn subscribe(
tracing::info!(%subject, "{what} stream: client attached");
Ok(subscriber)
}
#[cfg(test)]
mod tests {
use super::*;
fn with_last_event_id(value: &str) -> HeaderMap {
let mut headers = HeaderMap::new();
headers.insert(
"last-event-id",
value.parse().expect("a valid header value"),
);
headers
}
#[test]
fn a_reconnect_resumes_after_the_sequence_it_sends() {
assert_eq!(last_event_id(&with_last_event_id("917")), Some(917));
assert_eq!(
agent_term::replay_start(1000, last_event_id(&with_last_event_id("917"))),
918
);
}
#[test]
fn no_or_a_foreign_last_event_id_is_a_fresh_reader() {
assert_eq!(last_event_id(&HeaderMap::new()), None);
assert_eq!(last_event_id(&with_last_event_id("r12")), None);
assert_eq!(last_event_id(&with_last_event_id("")), None);
}
}