vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-agent: the swarm queue connection is required; retry, then exit

Browse source 
An agent's harness connects to the swarm queue with the agent's own
credential or not at all. The first connect retries after 5s, doubling
(7 attempts, about five minutes), with a warning per failed attempt;
when every attempt fails the harness logs an error and exits 1. Missing
coordinates (HIVE_AGENT_NATS_URL, or the secret store's BAO_ADDR /
HIVE_AGENT_NAME / certificate) fail the boot outright.

Previously the first connect's outcome was memoized for the life of the
process: a store 403 or queue refusal at boot left the agent on the
hive's shared client, or disconnected, until someone restarted it, and
an agent with no queue coordinates ran on without one. Both paths are
gone, with the hive-scoped subjects that only the shared client used.

The harness unit now restarts after 30s instead of 2s, so a swarm-wide
outage doesn't restart every agent every 2s, and has no start limit, so
it keeps coming back however long the outage lasts.

The hive's shared client credential is still imported into the harness
unit for swarm-logs; the two HIVE_AGENT_OIDC_*_FILE variables that
pointed the harness at it are dropped, as is the unset
HIVE_AGENT_OIDC_CA_FILE override.

With the fallback gone, the pieces #4944 added only for it go too:
swarm_input's skip-on-shared-client branch and swarm-queue-client's
connect_with_inbox_prefix. swarm-nats-auth's grant for the hive's shared
agent client is unchanged.

Closes #4909
This commit is contained in:
atlas 2026-10-04 19:05:48 +02:00 • committed by mara
commit 235bc481b3
16 changed files with 392 additions and 964 deletions
Download patch file Download diff file Expand all files Collapse all files

135
hive-agent/src/swarm_term.rs
View file

@ -19,15 +19,13 @@
//! same way it would have missed it on the agent's own SSE stream. That also
//! keeps the agent's grant to a plain publish.
//!
//! **Best-effort in every direction.** The turn loop and the web UI must not
//! notice whether the queue exists, so the bus receiver is the only thing this
//! task blocks on; a failed publish is a log line and the next row is still
//! attempted.
//! **Best-effort per row.** The turn loop and the web UI never wait on the
//! queue, so the bus receiver is the only thing this task blocks on; a failed
//! publish is a log line and the next row is still attempted.
use tokio::sync::broadcast;
use crate::events::BusEvent;
use crate::swarm_queue::{Connection, Presented};
use crate::term_msg::{ClassifyCtx, TermMsg, classify};
use hive_sh4re::term_msg::fit;
@ -36,18 +34,6 @@ use hive_sh4re::term_msg::fit;
/// can follow one agent without filtering the swarm's whole terminal traffic.
const SUBJECT_PREFIX: &str = "$SWARM.term";
/// What the queue's client id looks like either side of the hive's own name.
///
/// These mirror the auth-callout responder's `--hive-client-prefix` and
/// `--agent-client-suffix`, which is where the authoritative pair lives: the
/// responder parses the hive back out of the presented client id and grants
/// exactly `$SWARM.term.<that hive>.>`. The agent is not told those flags, so
/// it restates their defaults — a deployment that retunes either one has to
/// change them here too, and the symptom of not doing so is every publish
/// refused rather than a wrong subject being accepted.
const CLIENT_ID_PREFIX: &str = "hive-";
const CLIENT_ID_SUFFIX: &str = "-agent";
/// How much room to leave under the announced limit for everything the
/// publish adds around the payload — subject, headers, protocol framing.
///
@ -57,56 +43,13 @@ const CLIENT_ID_SUFFIX: &str = "-agent";
/// out of eight megabytes buys that margin for nothing.
const HEADROOM: usize = 1024;
/// The hive a queue client id names.
///
/// The hive in the subject has to be the string the **responder** parses out
/// of this same client id, because that is what its grant is built from. The
/// harness knows a hive display name too, from a different source and with no
/// rule tying the two together — deriving the subject from that one instead
/// produces a publish the broker refuses, which surfaces as a terminal that is
/// simply empty at the swarm.
///
/// `None` for an id that does not have the expected shape: a publisher that
/// guessed at a subject would be asking for a grant it cannot have, so the
/// caller disables itself instead.
fn hive_from_client_id(client_id: &str) -> Option<&str> {
let hive = client_id
.strip_prefix(CLIENT_ID_PREFIX)?
.strip_suffix(CLIENT_ID_SUFFIX)?;
(!hive.is_empty()).then_some(hive)
/// The subject this agent's rows go to.
fn subject(agent: &str) -> String {
format!("{SUBJECT_PREFIX}.{agent}")
}
/// The subject this agent's rows go to under the credential it connected
/// with, or `None` when a hive client id names no hive.
fn subject(presented: &Presented, agent: &str) -> Option<String> {
match presented {
Presented::Agent => Some(format!("{SUBJECT_PREFIX}.{agent}")),
Presented::Hive { client_id } => {
let Some(hive) = hive_from_client_id(client_id) else {
tracing::warn!(
%client_id,
expected = format!("{CLIENT_ID_PREFIX}<hive>{CLIENT_ID_SUFFIX}"),
"queue client id does not name a hive; not publishing the terminal upward"
);
return None;
};
Some(format!("{SUBJECT_PREFIX}.{hive}.{agent}"))
}
}
}
/// Start the publish task, if this agent has a queue credential and a label
/// to name its subject with.
///
/// Returns without spawning in every other case — no queue or no label — each
/// of which is a legal state for an agent rather than an error, and each
/// logged once here rather than per row.
/// Start the publish task, if this agent has a label to name its subject with.
pub fn spawn(rx: broadcast::Receiver<BusEvent>) {
if !crate::swarm_queue::configured() {
// `swarm_queue::init` already said why at boot; repeating it here
// would be the same fact logged twice.
return;
}
let agent = crate::identity::label();
if agent.is_empty() {
tracing::warn!("this agent has no label; not publishing the terminal upward");
@ -116,12 +59,8 @@ pub fn spawn(rx: broadcast::Receiver<BusEvent>) {
}
async fn run(mut rx: broadcast::Receiver<BusEvent>, agent: String) {
let Some(Connection { client, presented }) = crate::swarm_queue::client().await else {
return;
};
let Some(subject) = subject(&presented, &agent) else {
return;
};
let client = crate::swarm_queue::client().await;
let subject = subject(&agent);
tracing::info!(subject, "publishing the agent terminal to the swarm queue");
// One context for the task's whole life — see the module doc.
@ -180,7 +119,7 @@ async fn publish(client: &async_nats::Client, subject: &str, msg: TermMsg) {
#[cfg(test)]
mod tests {
use super::{Presented, hive_from_client_id, subject};
use super::subject;
use crate::events::LiveEvent;
use crate::term_msg::{ClassifyCtx, classify};
use hive_sh4re::term_msg::fit;
@ -188,57 +127,11 @@ mod tests {
/// The time `classify` renders for the event the test below classifies.
const ROW_TS: &str = "2026-09-13T12:35:03Z";
/// The agent's own credential is granted the subject naming the agent
/// alone, with no hive in it.
#[test]
fn a_client_id_names_the_hive_between_the_prefix_and_the_suffix() {
assert_eq!(hive_from_client_id("hive-alpha-agent"), Some("alpha"));
// A hive whose own name contains the suffix still resolves: only the
// trailing one is stripped, so the responder and this agree.
assert_eq!(
hive_from_client_id("hive-alpha-agent-agent"),
Some("alpha-agent")
);
}
/// Each way an id can fail to name a hive. A guessed subject would be
/// refused by the broker, and a refusal reaches an operator as an empty
/// terminal rather than as an error, so none of these may fall back.
#[test]
fn an_id_of_another_shape_names_no_hive() {
for id in [
// The hive's own id, not an agent's.
"hive-alpha",
// Missing the prefix the responder keys on.
"alpha-agent",
// Prefix and suffix but nothing between them: the subject would
// be `$SWARM.term..<agent>`, whose empty token matches no grant.
"hive--agent",
// Prefix and suffix overlapping with no hive at all.
"hive-agent",
"",
] {
assert_eq!(hive_from_client_id(id), None, "{id} must name no hive");
}
}
/// Each credential publishes on the subject it is granted: its own under
/// the agent's credential, its hive's under the shared one.
#[test]
fn the_subject_follows_the_credential_the_agent_connected_with() {
assert_eq!(
subject(&Presented::Agent, "mara").as_deref(),
Some("$SWARM.term.mara")
);
let hive = Presented::Hive {
client_id: "hive-alpha-agent".to_owned(),
};
assert_eq!(
subject(&hive, "mara").as_deref(),
Some("$SWARM.term.alpha.mara")
);
let unparseable = Presented::Hive {
client_id: "hive-alpha".to_owned(),
};
assert_eq!(subject(&unparseable, "mara"), None);
fn the_subject_is_the_prefix_then_the_agent() {
assert_eq!(subject("mara"), "$SWARM.term.mara");
}
/// What a subscriber actually receives: the payload this module hands