hyperhive/hive-agent/src/swarm_term.rs
atlas 053340128b term_msg: carry the source event's time on every row
A terminal row published on `$SWARM.term.<hive>.<agent>` goes out bare,
with no envelope around it and no server-side stamp, so a subscriber had
nothing to place the row in time with beyond its own receipt clock —
wrong by the queue's latency and meaningless for anything read later
than live.

`TermMsg` gains `ts`, ISO 8601 UTC. `classify` takes the event's own
unix-seconds stamp and applies it to every row that event expands into,
so a row replayed out of sqlite says when it happened rather than when
it was read, and a row that sat in a lagging subscriber's buffer does
not lie about its time. The oversize degrade keeps it; only the body is
ever spent.

`TermEnvelope` stops duplicating `ts` and keeps `seq`: the dedup counter
is a real transport concern, the event's time is not, now that it rides
on the row. Nothing in the frontend read `envelope.ts` — only the type
declared it.

Refs #4321
2026-09-13 14:23:11 +02:00

340 lines
15 KiB
Rust

//! Publishing this agent's terminal rows onto the swarm queue.
//!
//! The per-agent web UI already classifies the event bus into [`TermMsg`]
//! rows; this offers the same rows upward so a swarm-level terminal can
//! render an agent without reaching into its hive. It publishes the value the
//! web UI is handed rather than a second model, so a new tool or a
//! reclassified event changes both surfaces at once.
//!
//! **Subscribes to the bus, not to a browser.** The SSE handler builds its
//! rows per connected client, so hanging this off it would mean an agent
//! whose terminal nobody has open publishes nothing. This is its own
//! subscriber with its own long-lived [`ClassifyCtx`] — a publisher that
//! restarted its correlation state per reader would lose the `tool_use` →
//! name mapping a `tool_result` needs.
//!
//! **A core subject, not `JetStream`.** A terminal row has no current value a
//! late reader can ask for, and nothing here is worth the durability of the
//! notices stream: a subscriber that was not listening missed the row, the
//! same way it would have missed it on the agent's own SSE stream. That also
//! keeps the agent's grant to a plain publish.
//!
//! **Best-effort in every direction.** The turn loop and the web UI must not
//! notice whether the queue exists, so the bus receiver is the only thing this
//! task blocks on; a failed publish is a log line and the next row is still
//! attempted.
use tokio::sync::broadcast;
use crate::events::BusEvent;
use crate::term_msg::{ClassifyCtx, TermMsg, classify};
/// Subject family carrying agent terminal rows, the swarm-wide agreement this
/// publisher holds up its end of. One leaf subject per agent, so a subscriber
/// can follow one agent without filtering the swarm's whole terminal traffic.
const SUBJECT_PREFIX: &str = "$SWARM.term";
/// What the queue's client id looks like either side of the hive's own name.
///
/// These mirror the auth-callout responder's `--hive-client-prefix` and
/// `--agent-client-suffix`, which is where the authoritative pair lives: the
/// responder parses the hive back out of the presented client id and grants
/// exactly `$SWARM.term.<that hive>.>`. The agent is not told those flags, so
/// it restates their defaults — a deployment that retunes either one has to
/// change them here too, and the symptom of not doing so is every publish
/// refused rather than a wrong subject being accepted.
const CLIENT_ID_PREFIX: &str = "hive-";
const CLIENT_ID_SUFFIX: &str = "-agent";
/// What a row that lost its body to the payload limit carries instead.
const DROPPED_BODY: &str = "[body dropped: over the queue's payload limit]";
/// How much room to leave under the announced limit for everything the
/// publish adds around the payload — subject, headers, protocol framing.
///
/// The server measures the payload alone, so this is not required for
/// correctness; it is here because the cost of being one byte over is not one
/// lost byte but a refused message and a dropped connection, and a kilobyte
/// out of eight megabytes buys that margin for nothing.
const HEADROOM: usize = 1024;
/// The hive a queue client id names.
///
/// The hive in the subject has to be the string the **responder** parses out
/// of this same client id, because that is what its grant is built from. The
/// harness knows a hive display name too, from a different source and with no
/// rule tying the two together — deriving the subject from that one instead
/// produces a publish the broker refuses, which surfaces as a terminal that is
/// simply empty at the swarm.
///
/// `None` for an id that does not have the expected shape: a publisher that
/// guessed at a subject would be asking for a grant it cannot have, so the
/// caller disables itself instead.
fn hive_from_client_id(client_id: &str) -> Option<&str> {
let hive = client_id
.strip_prefix(CLIENT_ID_PREFIX)?
.strip_suffix(CLIENT_ID_SUFFIX)?;
(!hive.is_empty()).then_some(hive)
}
/// Bring `msg` under `limit` serialized bytes, or report that it cannot be.
///
/// The body is the only field that carries arbitrary length — a diff, a whole
/// tool result — so it is the only one worth spending: replacing it keeps the
/// row's identity, level, summary and time, which is what makes the row
/// readable at all, and a reader sees that something was there rather than
/// seeing nothing.
///
/// `None` means even the degraded row does not fit, so the caller reports it
/// instead of publishing. That matters more than it looks: an oversize publish
/// is not truncated by the server, it is refused and the connection is closed,
/// which costs the row *and* every row racing behind it through the reconnect.
fn fit(msg: TermMsg, limit: usize) -> Option<TermMsg> {
if serialized_len(&msg)? <= limit {
return Some(msg);
}
// Rebuilt rather than mutated in place: `body_format` describes the body,
// and a marker left tagged `Diff` renders as a broken diff downstream.
let degraded = TermMsg {
body: Some(DROPPED_BODY.to_owned()),
body_format: None,
..msg
};
(serialized_len(&degraded)? <= limit).then_some(degraded)
}
/// Serialized size of a row, or `None` if it does not serialize at all.
///
/// Measured by serializing rather than estimated from field lengths: the
/// payload is what the server measures, and JSON escaping makes the two differ
/// by an unbounded factor on exactly the rows that are already near the limit.
fn serialized_len(msg: &TermMsg) -> Option<usize> {
serde_json::to_vec(msg).ok().map(|v| v.len())
}
/// Start the publish task, if this agent has both queue coordinates and an
/// identity the subject can be derived from.
///
/// Returns without spawning in every other case — no queue, an unparseable
/// client id, no label — each of which is a legal state for an agent rather
/// than an error, and each logged once here rather than per row.
pub fn spawn(rx: broadcast::Receiver<BusEvent>) {
let Some(cfg) = crate::swarm_queue::config() else {
// `swarm_queue::init` already said why at boot; repeating it here
// would be the same fact logged twice.
return;
};
let Some(hive) = hive_from_client_id(&cfg.client_id) else {
tracing::warn!(
client_id = %cfg.client_id,
expected = format!("{CLIENT_ID_PREFIX}<hive>{CLIENT_ID_SUFFIX}"),
"queue client id does not name a hive; not publishing the terminal upward"
);
return;
};
let agent = crate::identity::label();
if agent.is_empty() {
tracing::warn!("this agent has no label; not publishing the terminal upward");
return;
}
let subject = format!("{SUBJECT_PREFIX}.{hive}.{agent}");
tokio::spawn(run(rx, subject));
}
async fn run(mut rx: broadcast::Receiver<BusEvent>, subject: String) {
let Some(client) = crate::swarm_queue::client().await else {
return;
};
tracing::info!(subject, "publishing the agent terminal to the swarm queue");
// One context for the task's whole life — see the module doc.
let mut ctx = ClassifyCtx::default();
loop {
let event = match rx.recv().await {
Ok(event) => event,
// The bus drops events for a subscriber that falls behind. The
// terminal is a tail rather than a log, so the answer is to say
// how many were missed and keep reading, exactly as the web UI's
// own subscriber does.
Err(broadcast::error::RecvError::Lagged(missed)) => {
tracing::warn!(missed, "swarm terminal: lagged, rows skipped");
continue;
}
// Every sender is gone, so the harness is shutting down.
Err(broadcast::error::RecvError::Closed) => return,
};
for msg in classify(&event.event, event.ts, &mut ctx) {
publish(&client, &subject, msg).await;
}
}
}
/// Offer one row. Every failure is terminal for that row and for nothing else.
async fn publish(client: &async_nats::Client, subject: &str, msg: TermMsg) {
// An unconnected client does not fail a publish, it buffers it — and the
// limit read below is the library's pre-connect default until the server
// has announced its own, which is smaller than any deployment sets and
// would degrade rows that would have fit.
if let Err(e) = swarm_queue_client::ensure_connected(client) {
tracing::warn!(error = %swarm_queue_client::chain(&e), "swarm terminal: publish skipped");
return;
}
let limit = swarm_queue_client::max_payload(client).saturating_sub(HEADROOM);
let summary = msg.summary.clone();
let Some(msg) = fit(msg, limit) else {
tracing::warn!(
summary,
limit,
"swarm terminal: row does not fit even without its body, dropped"
);
return;
};
let payload = match serde_json::to_vec(&msg) {
Ok(payload) => payload,
Err(e) => {
tracing::warn!(error = %e, "swarm terminal: serialising failed, row dropped");
return;
}
};
if let Err(e) = client.publish(subject.to_owned(), payload.into()).await {
tracing::warn!(error = %e, "swarm terminal: publish failed, row dropped");
}
}
#[cfg(test)]
mod tests {
use super::{DROPPED_BODY, fit, hive_from_client_id};
use crate::events::LiveEvent;
use crate::term_msg::{BodyFormat, ClassifyCtx, Level, TermMsg, classify};
/// A row's time as `classify` renders it, for the tests below that need
/// one without classifying an event to get it.
const ROW_TS: &str = "2026-09-13T12:35:03Z";
#[test]
fn a_client_id_names_the_hive_between_the_prefix_and_the_suffix() {
assert_eq!(hive_from_client_id("hive-alpha-agent"), Some("alpha"));
// A hive whose own name contains the suffix still resolves: only the
// trailing one is stripped, so the responder and this agree.
assert_eq!(
hive_from_client_id("hive-alpha-agent-agent"),
Some("alpha-agent")
);
}
/// Each way an id can fail to name a hive. A guessed subject would be
/// refused by the broker, and a refusal reaches an operator as an empty
/// terminal rather than as an error, so none of these may fall back.
#[test]
fn an_id_of_another_shape_names_no_hive() {
for id in [
// The hive's own id, not an agent's.
"hive-alpha",
// Missing the prefix the responder keys on.
"alpha-agent",
// Prefix and suffix but nothing between them: the subject would
// be `$SWARM.term..<agent>`, whose empty token matches no grant.
"hive--agent",
// Prefix and suffix overlapping with no hive at all.
"hive-agent",
"",
] {
assert_eq!(hive_from_client_id(id), None, "{id} must name no hive");
}
}
#[test]
fn a_row_that_already_fits_is_published_unchanged() {
let msg = TermMsg::new(Level::Info, "turn ok")
.icon("")
.body("a short body", Some(BodyFormat::Markdown));
let fitted = fit(msg, 4096).expect("a small row fits");
assert_eq!(fitted.body.as_deref(), Some("a short body"));
assert_eq!(fitted.body_format, Some(BodyFormat::Markdown));
}
/// The case the degrade exists for: a body larger than the limit costs the
/// body and nothing else, and what comes back is actually under the limit
/// rather than merely smaller.
#[test]
fn an_oversize_body_is_replaced_and_the_result_fits() {
let limit = 512;
let msg = TermMsg::new(Level::Info, "Edit(src/main.rs)")
.icon("🔧")
.body("x".repeat(limit * 4), Some(BodyFormat::Diff))
.coalesce("tool-1")
.at(ROW_TS);
let fitted = fit(msg, limit).expect("dropping the body brings this under the limit");
assert_eq!(fitted.body.as_deref(), Some(DROPPED_BODY));
// The tag describes a body that is no longer there; left set, a reader
// renders the marker as a diff.
assert_eq!(fitted.body_format, None);
// The fields that make the row readable survive.
assert_eq!(fitted.summary, "Edit(src/main.rs)");
assert_eq!(fitted.icon.as_deref(), Some("🔧"));
assert_eq!(fitted.coalesce_key.as_deref(), Some("tool-1"));
// Including the time: a degraded row that lost it would be a row a
// subscriber cannot place, and nothing downstream could tell.
assert_eq!(fitted.ts, ROW_TS);
assert!(
serde_json::to_vec(&fitted).expect("serialises").len() <= limit,
"the degraded row must be under the limit, not merely smaller"
);
}
/// A row whose summary alone exceeds the limit cannot be degraded into
/// one, and publishing it anyway would cost the connection rather than the
/// row. Reported by the caller, never sent.
#[test]
fn a_row_too_large_even_without_its_body_is_refused() {
let limit = 256;
let msg = TermMsg::new(Level::Warn, "s".repeat(limit * 4)).body("x".repeat(limit), None);
assert!(fit(msg, limit).is_none());
}
/// Control for the test above: the same oversize summary with a body that
/// would fit still refuses, so the refusal is about total size and not
/// about the body having been present.
#[test]
fn the_refusal_is_about_size_rather_than_the_body_being_present() {
let limit = 256;
let msg = TermMsg::new(Level::Warn, "s".repeat(limit * 4));
assert!(fit(msg, limit).is_none());
}
/// JSON escaping is why the size is measured by serializing: a body of
/// quotes serializes to twice its own length, so a row that fits by
/// character count can still be refused on the wire.
#[test]
fn the_limit_is_measured_on_the_serialized_bytes() {
let body = "\"".repeat(200);
let msg = TermMsg::new(Level::Info, "quotes").body(body.clone(), None);
let limit = body.len() + 64;
// Well under the limit as characters, over it once escaped.
assert!(
serde_json::to_vec(&msg).expect("serialises").len() > limit,
"this fixture must be oversize only after escaping"
);
let fitted = fit(msg, limit).expect("dropping the body fits");
assert_eq!(fitted.body.as_deref(), Some(DROPPED_BODY));
}
/// What a subscriber actually receives: the payload this module hands
/// `publish` is the bare row, so the time has to be *in* it. A `ts` that
/// existed in Rust but never serialized would leave the queue exactly as
/// timeless as it was before.
#[test]
fn the_published_payload_carries_the_events_own_time() {
let ev = LiveEvent::TurnEnd {
ok: true,
note: None,
};
let mut ctx = ClassifyCtx::default();
// 2026-09-13T12:35:03Z, a time that is not the time of this run.
let msgs = classify(&ev, 1_789_302_903, &mut ctx);
let payload = serde_json::to_vec(&fit(msgs[0].clone(), 4096).expect("a small row fits"))
.expect("serialises");
let value: serde_json::Value = serde_json::from_slice(&payload).expect("valid json");
assert_eq!(value["ts"], serde_json::json!(ROW_TS));
}
}