An agent's harness connects to the swarm queue with the agent's own credential or not at all. The first connect retries after 5s, doubling (7 attempts, about five minutes), with a warning per failed attempt; when every attempt fails the harness logs an error and exits 1. Missing coordinates (HIVE_AGENT_NATS_URL, or the secret store's BAO_ADDR / HIVE_AGENT_NAME / certificate) fail the boot outright. Previously the first connect's outcome was memoized for the life of the process: a store 403 or queue refusal at boot left the agent on the hive's shared client, or disconnected, until someone restarted it, and an agent with no queue coordinates ran on without one. Both paths are gone, with the hive-scoped subjects that only the shared client used. The harness unit now restarts after 30s instead of 2s, so a swarm-wide outage doesn't restart every agent every 2s, and has no start limit, so it keeps coming back however long the outage lasts. The hive's shared client credential is still imported into the harness unit for swarm-logs; the two HIVE_AGENT_OIDC_*_FILE variables that pointed the harness at it are dropped, as is the unset HIVE_AGENT_OIDC_CA_FILE override. With the fallback gone, the pieces #4944 added only for it go too: swarm_input's skip-on-shared-client branch and swarm-queue-client's connect_with_inbox_prefix. swarm-nats-auth's grant for the hive's shared agent client is unchanged. Closes #4909
139 lines
5 KiB
Rust
139 lines
5 KiB
Rust
//! Operator input from the swarm UI, taken straight off the swarm queue.
|
|
//!
|
|
//! swarm-controller sends each message on
|
|
//! [`swarm_queue_client::agent_input_subject`] for this agent. This task
|
|
//! subscribes to it and pushes each message into the serve loop's local
|
|
//! message channel, where it becomes a turn from the operator it names.
|
|
//! hive-c0re is not in the path, so the message has no broker row: the hive
|
|
//! dashboard's history and `recv` never see it, and `send` has no recipient
|
|
//! by the sender's name.
|
|
//!
|
|
//! **`from` is taken as given.** The controller is the only client granted
|
|
//! publish on the subject (`swarm-nats-auth`'s reader grant), and it names
|
|
//! the sender from the swarm gateway's authelia answer.
|
|
//!
|
|
//! **Own credential only.** The queue grants subscribe on this subject to the
|
|
//! agent's own credential alone.
|
|
//!
|
|
//! **Core NATS, no reply.** A message sent while this task is not subscribed
|
|
//! is gone; the controller sends it as a request so the operator gets a 503
|
|
//! for that instead. No agent grant covers a reply subject, so nothing here
|
|
//! answers the request.
|
|
|
|
use futures_util::{Stream, StreamExt as _};
|
|
use hive_sh4re::inbox::DeliveredMessage;
|
|
use tokio::sync::mpsc::UnboundedSender;
|
|
|
|
/// Start the subscriber, if this agent has a label to name its subject with.
|
|
pub fn spawn(tx: UnboundedSender<DeliveredMessage>) {
|
|
let agent = crate::identity::label();
|
|
if agent.is_empty() {
|
|
tracing::warn!("this agent has no label; not taking input from the swarm queue");
|
|
return;
|
|
}
|
|
tokio::spawn(run(tx, agent));
|
|
}
|
|
|
|
async fn run(tx: UnboundedSender<DeliveredMessage>, agent: String) {
|
|
let client = crate::swarm_queue::client().await;
|
|
let subject = swarm_queue_client::agent_input_subject(&agent);
|
|
let subscriber = match client.subscribe(subject.clone()).await {
|
|
Ok(subscriber) => subscriber,
|
|
Err(e) => {
|
|
tracing::warn!(subject, error = %e, "swarm input: subscribe failed");
|
|
return;
|
|
}
|
|
};
|
|
tracing::info!(subject, "taking operator input from the swarm queue");
|
|
forward(subscriber.map(|msg| msg.payload), &tx).await;
|
|
}
|
|
|
|
/// Push each decodable payload into `tx` until either end closes.
|
|
async fn forward<P: AsRef<[u8]>>(
|
|
payloads: impl Stream<Item = P>,
|
|
tx: &UnboundedSender<DeliveredMessage>,
|
|
) {
|
|
let mut payloads = std::pin::pin!(payloads);
|
|
while let Some(payload) = payloads.next().await {
|
|
let Some(dm) = delivered(payload.as_ref()) else {
|
|
continue;
|
|
};
|
|
if tx.send(dm).is_err() {
|
|
// The serve loop is gone: the harness is shutting down.
|
|
return;
|
|
}
|
|
}
|
|
}
|
|
|
|
/// The message a payload becomes, or `None` (logged) for one that is not an
|
|
/// [`swarm_queue_client::AgentInput`].
|
|
fn delivered(payload: &[u8]) -> Option<DeliveredMessage> {
|
|
match serde_json::from_slice::<swarm_queue_client::AgentInput>(payload) {
|
|
Ok(input) => {
|
|
tracing::info!(from = %input.from, "swarm input: message received");
|
|
Some(DeliveredMessage {
|
|
from: input.from,
|
|
body: input.body,
|
|
id: 0,
|
|
redelivered: false,
|
|
in_reply_to: None,
|
|
})
|
|
}
|
|
Err(e) => {
|
|
tracing::warn!(error = %e, "swarm input: undecodable message dropped");
|
|
None
|
|
}
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::forward;
|
|
use swarm_queue_client::AgentInput;
|
|
|
|
fn payload(from: &str, body: &str) -> Vec<u8> {
|
|
serde_json::to_vec(&AgentInput {
|
|
from: from.to_owned(),
|
|
body: body.to_owned(),
|
|
})
|
|
.expect("encodes")
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn each_message_reaches_the_serve_loop_as_from_its_sender() {
|
|
let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel();
|
|
let payloads = vec![
|
|
payload("mara", "first"),
|
|
b"not json".to_vec(),
|
|
payload("argus", "second"),
|
|
];
|
|
forward(futures_util::stream::iter(payloads), &tx).await;
|
|
drop(tx);
|
|
|
|
let mut got = Vec::new();
|
|
while let Some(dm) = rx.recv().await {
|
|
got.push(dm);
|
|
}
|
|
// The undecodable one is dropped, the ones either side of it are not.
|
|
assert_eq!(got.len(), 2, "{got:?}");
|
|
assert_eq!(
|
|
(got[0].from.as_str(), got[0].body.as_str()),
|
|
("mara", "first")
|
|
);
|
|
assert_eq!(
|
|
(got[1].from.as_str(), got[1].body.as_str()),
|
|
("argus", "second")
|
|
);
|
|
// No broker row behind it, so no `[msg #id]` tag in the prompt.
|
|
assert!(got.iter().all(|dm| dm.id == 0 && dm.in_reply_to.is_none()));
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn a_closed_serve_loop_stops_the_forwarder() {
|
|
let (tx, rx) = tokio::sync::mpsc::unbounded_channel();
|
|
drop(rx);
|
|
// An endless stream: returning at all proves the closed channel ended it.
|
|
let payloads = futures_util::stream::repeat(payload("mara", "hi"));
|
|
forward(payloads, &tx).await;
|
|
}
|
|
}
|