mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

a hive-side pause holds against the swarm's declared state until an operator resumes

Browse source 
the harness writes the reason into the pause marker when it pauses itself and reports it as paused_by_hive in its turn-state header. swarm-controller declares such an agent paused (only if declared up) and shows the reason on the agents status row and the swarm ui card.
This commit is contained in:
atlas 2026-11-09 23:09:14 +01:00 • committed by mara
commit e5c55257f8
17 changed files with 517 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

8
docs/2-handbook/agents/persistence.md
View file

@ -225,7 +225,7 @@ destroy/recreate, gone on `--purge`.
### `/harness/paused` (per agent)
Empty marker file. Its presence parks the agent's turn loop: the
Marker file, empty when an operator paused the agent. Its presence parks the agent's turn loop: the
harness keeps serving its web UI and MCP daemons but drives no turns,
and inbox messages queue unacked until it's removed (see
[turn loop](../../4-implementation/turn-loop/README.md#the-loop)).
@ -243,6 +243,12 @@ pause keeps working when the harness is wedged or the container is
stopped.
<!-- vale write-good.Passive = YES -->
When the harness pauses itself, after three todo wakes in a row that skipped
`get_loose_ends`, it writes the reason into the marker. The agent's turn-state
header reports a non-empty marker as `paused_by_hive`
([swarm](../swarm/README.md)), and the swarm then declares the
agent `paused` so the hive's wanted-state convergence doesn't resume it.
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
is the agent's own space to fill, and this is harness control state.
Survives destroy/recreate, gone on `--purge` — so a paused agent comes

14
docs/2-handbook/swarm/README.md
View file

@ -273,6 +273,17 @@ context budget and the last turn's context and cost token blocks,
session and its turns wait for a login. The harness sets it from its own login
state, so it reaches the swarm without going through the hive.
A header also carries `paused_by_hive` while the harness's own pause is in force:
the reason it wrote into the pause marker when it paused its turn loop. A pause
by an operator (`hivectl agent <name> pause`, the dashboard) leaves the marker
empty and reports nothing here, and so does a harness that predates the field.
The controller reacts by declaring the agent `paused`, but only when the swarm
declares it `up` at that moment. Without that, the hive resumes any paused agent the
swarm declares `up` on its next convergence. It acts once per pause: the entry
stays until a header without the field arrives. Setting the agent `up` again
resumes it, and the hive clears the marker. If the swarm is on a controller that
predates the field, the hive's pause lasts until its next convergence, as before.
`agent_state` reuses the swarm's own wanted-state vocabulary
(`up`/`offline`/`paused`/`destroyed`) so a reader can compare what an agent _is_
against what the swarm declared it should be without translating between two
@ -301,7 +312,8 @@ agent's latest `model`, `effort` and their sources for
agent's latest `needs_login` for the `needs_login` field of
`GET /api/agents/status`. That field is `null` until a header carrying the flag
arrives, so for up to a minute after the controller starts, and for an agent
whose harness doesn't send it.
whose harness doesn't send it. The row's `paused_by_hive` comes from the same
headers and is `null` in the same cases.
The third thing an agent publishes is its **icon**, the same SVG its own
`GET /icon` serves. It goes into the `agent-icons` KV bucket under the key

5
docs/2-handbook/swarm/ui.md
View file

@ -80,8 +80,9 @@ action that asks first.
card per agent, none of them inside a bigger card. A card's
header shows its icon, name, freshness, the status message, a **terminal**
button that opens the agent's terminal tab, the **wanted** menu (up, offline,
paused, destroy), and a **login needed** badge with a
**log in** button when its claude has no session. Select the header to open the
paused, destroy), a **paused by hive** badge when the agent's own harness paused
its turn loop (hover for the reason; set the agent **up** to resume it), and a
**login needed** badge with a **log in** button when its claude has no session. Select the header to open the
card in place; select it again to close it. Several cards can be open at once,
and a card stays open across the page's refresh, but not across a reload. The
filter card, which also holds the **+ agent** button and the refresh picker,

11
frontend/packages/swarm-ui/src/agent/AgentTypes.ts
View file

@ -43,6 +43,10 @@ export interface AgentRow {
// no header since swarm-controller started, or an agent that does not
// send the flag — not known, so not "needs login".
needs_login: boolean | null;
// Why the agent's own harness paused its turn loop. `null`: it did not, or
// the harness does not send the field. The controller declares such an agent
// paused, and only setting it up again resumes it.
paused_by_hive: string | null;
// The archetype the agent is built from; `null` for a custom agent.
archetype: { source: string; name: string } | null;
}
@ -61,6 +65,13 @@ export function showsLoginNeeded(row: Pick<AgentRow, "needs_login">): boolean {
return row.needs_login === true;
}
// The reason to show on the "paused by hive" badge; `null` shows no badge.
export function hivePauseReason(
row: Pick<AgentRow, "paused_by_hive">,
): string | null {
return row.paused_by_hive ? row.paused_by_hive : null;
}
// Same tone pairing as HivesCard — one freshness enum shared by both
// endpoints, so the same tone rule applies to both pages — but
// `unknown`'s label diverges deliberately: an agent outside the

14
frontend/packages/swarm-ui/src/agent/roster/AgentCard.tsx
View file

@ -23,7 +23,12 @@ import { AgentIcon } from "../AgentIcon.js";
import { ClaudeLoginDialog } from "../claude-login/ClaudeLoginDialog.js";
import { AgentDetail } from "../detail/AgentDetail.js";
import { termPaths } from "../terminal/term-paths.js";
import { FRESHNESS, showsLoginNeeded, type AgentRow } from "../AgentTypes.js";
import {
FRESHNESS,
hivePauseReason,
showsLoginNeeded,
type AgentRow,
} from "../AgentTypes.js";
import type { AgentActions } from "./AgentActions.js";
import { WantedMenu } from "./WantedMenu.js";
import { AgentCardLayout } from "./AgentCardLayout.js";
@ -82,6 +87,13 @@ export function AgentCard({
</>
}
/>
{hivePauseReason(row) !== null ? (
<Badge
tone="warning"
value="paused by hive"
title={`the hive paused this agent: ${hivePauseReason(row)}. Set it up again to resume it.`}
/>
) : null}
{showsLoginNeeded(row) ? (
<span
class="ui-agent-card-login"

11
frontend/packages/swarm-ui/test/agent-types.test.mjs
View file

@ -5,6 +5,7 @@ import assert from "node:assert/strict";
import { test } from "node:test";
import {
canTurnIntoCustom,
hivePauseReason,
showsLoginNeeded,
} from "../src/agent/AgentTypes.ts";
@ -42,3 +43,13 @@ test("turn into custom is offered to a placed archetype agent only", () => {
false,
);
});
test("an agent its hive paused shows the reason, any other shows none", () => {
assert.equal(
hivePauseReason({ paused_by_hive: "skipped get_loose_ends" }),
"skipped get_loose_ends",
);
assert.equal(hivePauseReason({ paused_by_hive: null }), null);
assert.equal(hivePauseReason({ paused_by_hive: "" }), null);
assert.equal(hivePauseReason({}), null);
});

45
hive-agent/src/main.rs
View file

@ -945,6 +945,14 @@ async fn drive_turn_and_wake_chain<S: Surface>(
}
}
/// The pause marker's content for an auto-pause after `streak` missed todo
/// wakes. A non-empty marker is a pause by the harness itself, which the
/// turn-state header reports as `paused_by_hive`; the operator's pause writes
/// an empty one.
fn todo_miss_pause_reason(streak: u32) -> String {
format!("skipped get_loose_ends on {streak} consecutive todo wakes")
}
/// Apply a finished turn's `todo_wake_checked` signal to the miss-streak:
/// resets on a checked wake, increments (and auto-pauses at the threshold)
/// on a missed one, no-ops for a non-todo-wake turn. Split out of
@ -963,15 +971,19 @@ fn apply_todo_wake_checked(checked: Option<bool>, todo_miss_streak: &mut u32, bu
);
if streak >= TODO_MISS_PAUSE_THRESHOLD {
tracing::warn!("todo-miss streak hit the threshold — pausing the turn loop");
// Marker first: the Note triggers a turn-state header rebuild,
// which reads the reason back out of this file.
if let Err(e) =
std::fs::write(paths::paused_marker(), todo_miss_pause_reason(streak))
{
tracing::warn!(error = ?e, "failed to write pause marker");
}
bus.emit(LiveEvent::Note {
text: format!(
"auto-paused: skipped get_loose_ends on {streak} \
consecutive todo wakes — an operator needs to resume this agent"
),
});
if let Err(e) = std::fs::write(paths::paused_marker(), "") {
tracing::warn!(error = ?e, "failed to write pause marker");
}
// Fresh slate for whenever this agent gets resumed — see
// `serve_loop`'s `was_paused` reset for the mirror side.
*todo_miss_streak = 0;
@ -1254,4 +1266,31 @@ mod tests {
.await;
assert!(matches!(woke, RecvOutcome::Empty));
}
/// The third consecutive miss parks the agent with its reason in the
/// marker; an earlier miss, or a wake that did call `get_loose_ends`,
/// writes nothing.
#[test]
fn the_threshold_miss_writes_the_reason_into_the_pause_marker() {
crate::paths::with_harness_dir(|| {
let bus = crate::events::Bus::new();
let mut streak = 0;
for _ in 1..super::TODO_MISS_PAUSE_THRESHOLD {
super::apply_todo_wake_checked(Some(false), &mut streak, &bus);
assert!(
!crate::paths::paused_marker().exists(),
"below the threshold"
);
}
super::apply_todo_wake_checked(Some(true), &mut streak, &bus);
assert_eq!(streak, 0);
for _ in 0..super::TODO_MISS_PAUSE_THRESHOLD {
super::apply_todo_wake_checked(Some(false), &mut streak, &bus);
}
assert_eq!(
std::fs::read_to_string(crate::paths::paused_marker()).unwrap(),
super::todo_miss_pause_reason(super::TODO_MISS_PAUSE_THRESHOLD)
);
});
}
}

56
hive-agent/src/swarm_agent_state.rs
View file

@ -35,6 +35,7 @@ use tokio::time::{Interval, MissedTickBehavior};
use swarm_wire::agent_inference::Effective;
use swarm_wire::agent_login::Login;
use swarm_wire::agent_pause::HivePause;
use swarm_wire::hive_status::{Capabilities, TOLERANT_PERMISSIONS};
use swarm_wire::wanted::AgentState;
@ -101,6 +102,10 @@ struct AgentStateMsg {
/// `needs_login`, from this process's own [`LoginState`], so always set.
#[serde(flatten)]
login: Login,
/// `paused_by_hive`, the reason in the pause marker when this harness wrote
/// it itself; absent otherwise.
#[serde(flatten)]
hive_pause: HivePause,
/// What this harness can do, so swarm-controller grants an agent a
/// permission only when the harness running it can decode it.
#[serde(flatten)]
@ -136,6 +141,20 @@ fn observed_state() -> AgentState {
}
}
/// Longest reason reported; the marker is a file the agent can write to.
const PAUSE_REASON_MAX_CHARS: usize = 200;
/// The header's pause-by-harness field: the pause marker's content, when it has
/// any. The operator's pause writes an empty marker, so it reports nothing.
fn hive_pause() -> HivePause {
let reason = std::fs::read_to_string(crate::paths::paused_marker()).unwrap_or_default();
let reason = reason.trim();
HivePause {
paused_by_hive: (!reason.is_empty())
.then(|| reason.chars().take(PAUSE_REASON_MAX_CHARS).collect()),
}
}
/// The header's login fields for `state`.
fn login(state: LoginState) -> Login {
Login {
@ -170,6 +189,7 @@ fn snapshot(bus: &Bus, login_state: &Mutex<LoginState>) -> AgentStateMsg {
ctx_usage: bus.last_ctx_usage(),
cost_usage: bus.last_cost_usage(),
login: login(*login_state.lock().unwrap_or_else(PoisonError::into_inner)),
hive_pause: hive_pause(),
capabilities: capabilities(),
}
}
@ -346,13 +366,14 @@ async fn publish(
#[cfg(test)]
mod tests {
use super::{
AgentStateMsg, REFRESH_INTERVAL, Trigger, capabilities, due, login, next_trigger,
refresh_timer, snapshot,
AgentStateMsg, PAUSE_REASON_MAX_CHARS, REFRESH_INTERVAL, Trigger, capabilities, due,
hive_pause, login, next_trigger, refresh_timer, snapshot,
};
use crate::events::{BusEvent, LiveEvent, TurnState};
use crate::login::LoginState;
use std::sync::Mutex;
use swarm_wire::agent_inference::{Effective, Source};
use swarm_wire::agent_pause::HivePause;
use swarm_wire::hive_status::{Capabilities, TOLERANT_PERMISSIONS};
use swarm_wire::wanted::AgentState;
use tokio::sync::broadcast;
@ -403,6 +424,9 @@ mod tests {
"cache_creation_input_tokens": 88,
}))),
login: login(LoginState::NeedsLogin),
hive_pause: HivePause {
paused_by_hive: Some("skipped get_loose_ends".to_owned()),
},
capabilities: capabilities(),
}
}
@ -432,6 +456,7 @@ mod tests {
"model",
"model_source",
"needs_login",
"paused_by_hive",
"resolved_model",
"turn_state",
"turn_state_since",
@ -523,6 +548,33 @@ mod tests {
}
}
/// The marker's content is the reason; an empty marker (an operator's pause) and
/// no marker report nothing, and swarm-controller reads it back with the wire type.
#[test]
fn the_pause_marker_content_is_the_reported_reason() {
crate::paths::with_harness_dir(|| {
assert_eq!(hive_pause().paused_by_hive, None, "no marker");
std::fs::write(crate::paths::paused_marker(), "").unwrap();
assert_eq!(hive_pause().paused_by_hive, None, "operator pause");
std::fs::write(crate::paths::paused_marker(), "x".repeat(500)).unwrap();
assert_eq!(
hive_pause().paused_by_hive.unwrap().len(),
PAUSE_REASON_MAX_CHARS
);
std::fs::write(crate::paths::paused_marker(), "skipped get_loose_ends\n").unwrap();
let bytes = serde_json::to_vec(&AgentStateMsg {
hive_pause: hive_pause(),
..msg()
})
.expect("serialises");
let decoded: HivePause = serde_json::from_slice(&bytes).expect("decodes");
assert_eq!(
decoded.paused_by_hive.as_deref(),
Some("skipped get_loose_ends")
);
});
}
/// swarm-controller reads the harness's capabilities back out of a header.
#[test]
fn the_header_advertises_tolerant_permissions() {

116
swarm-controller/src/agent_login.rs
View file

@ -7,6 +7,10 @@
//! an agent has no entry for up to a minute after this daemon starts — the
//! same gap [`crate::inference::Reported`] has on the same subject.
//!
//! A header may also carry `paused_by_hive`, the reason the agent's own harness
//! paused its turn loop (`swarm_wire::agent_pause`); [`crate::pause_hold`]
//! acts on it, and `GET /api/agents/status` shows it.
//!
//! The same header carries the harness's `capabilities`
//! (`swarm_wire::hive_status`), which [`Reported::advertises`] answers from:
//! the controller grants `logs` or `metrics` only to an agent whose running
@ -18,14 +22,27 @@ use std::time::Duration;
use futures_util::StreamExt as _;
use swarm_wire::agent_login::Login;
use swarm_wire::agent_pause::HivePause;
use swarm_wire::hive_status::Capabilities;
/// One agent's reported hive pause.
#[derive(Debug)]
struct Pause {
reason: String,
/// Whether [`crate::pause_hold`] has already dealt with this pause, so a
/// header re-sent after an operator resumed the agent does not pause it
/// again.
held: bool,
}
/// The latest login flag each agent reported, by agent.
#[derive(Debug, Default)]
pub(crate) struct Reported {
latest: Mutex<HashMap<String, bool>>,
/// The capabilities each agent's latest header listed.
capabilities: Mutex<HashMap<String, Capabilities>>,
/// Each agent whose latest header carries a hive-pause reason.
pauses: Mutex<HashMap<String, Pause>>,
}
impl Reported {
@ -75,6 +92,7 @@ impl Reported {
capabilities.remove(agent);
}
}
self.record_pause(agent, payload);
let mut latest = self.latest.lock().unwrap_or_else(PoisonError::into_inner);
match serde_json::from_slice::<Login>(payload) {
Ok(Login {
@ -92,6 +110,66 @@ impl Reported {
}
}
/// Keep one header's hive-pause reason. A header without one, or one that
/// does not decode, removes what was kept, which is how a resumed agent
/// is told from one that is still paused.
fn record_pause(&self, agent: &str, payload: &[u8]) {
let mut pauses = self.pauses.lock().unwrap_or_else(PoisonError::into_inner);
match serde_json::from_slice::<HivePause>(payload) {
Ok(HivePause {
paused_by_hive: Some(reason),
}) => {
pauses
.entry(agent.to_owned())
.and_modify(|p| p.reason.clone_from(&reason))
.or_insert(Pause {
reason,
held: false,
});
}
Ok(HivePause {
paused_by_hive: None,
})
| Err(_) => {
pauses.remove(agent);
}
}
}
/// `agent`'s hive-pause reason from its latest header; `None` when that
/// header carried none, or none arrived since this daemon started.
pub(crate) fn paused_by_hive(&self, agent: &str) -> Option<String> {
self.pauses
.lock()
.unwrap_or_else(PoisonError::into_inner)
.get(agent)
.map(|p| p.reason.clone())
}
/// The agents whose reported hive pause has not been held yet.
pub(crate) fn unheld_pauses(&self) -> Vec<String> {
self.pauses
.lock()
.unwrap_or_else(PoisonError::into_inner)
.iter()
.filter(|(_, p)| !p.held)
.map(|(agent, _)| agent.clone())
.collect()
}
/// Note that `agent`'s current hive pause has been dealt with. A no-op
/// once its header has stopped reporting one.
pub(crate) fn mark_pause_held(&self, agent: &str) {
if let Some(p) = self
.pauses
.lock()
.unwrap_or_else(PoisonError::into_inner)
.get_mut(agent)
{
p.held = true;
}
}
/// Whether `agent`'s latest header lists `capability`. `false` for an
/// agent that has sent none since this daemon started, or whose harness
/// predates the field.
@ -162,6 +240,44 @@ mod tests {
assert_eq!(reported.get("iris"), None);
}
fn pause_header(reason: Option<&str>) -> Vec<u8> {
let mut value: serde_json::Value = serde_json::from_slice(&header(None)).unwrap();
if let Some(reason) = reason {
value["paused_by_hive"] = reason.into();
}
serde_json::to_vec(&value).unwrap()
}
/// A header with the reason reports it, one without clears it, and a pause
/// held once stays held until a header without the reason has been seen.
#[test]
fn a_hive_pause_is_reported_until_a_header_clears_it() {
let reported = Reported::default();
let subject = swarm_queue_client::agent_state_subject("iris");
reported.record(&subject, &header(None));
assert_eq!(reported.paused_by_hive("iris"), None, "an old harness");
assert!(reported.unheld_pauses().is_empty());
reported.record(&subject, &pause_header(Some("skipped")));
assert_eq!(reported.paused_by_hive("iris").as_deref(), Some("skipped"));
assert_eq!(reported.unheld_pauses(), ["iris"]);
reported.mark_pause_held("iris");
reported.record(&subject, &pause_header(Some("skipped")));
assert!(reported.unheld_pauses().is_empty(), "held stays held");
assert_eq!(reported.paused_by_hive("iris").as_deref(), Some("skipped"));
reported.record(&subject, &pause_header(None));
assert_eq!(reported.paused_by_hive("iris"), None);
reported.record(&subject, &pause_header(Some("skipped")));
assert_eq!(
reported.unheld_pauses(),
["iris"],
"a new pause is held anew"
);
reported.record(&subject, b"not json");
assert_eq!(reported.paused_by_hive("iris"), None);
}
#[test]
fn a_header_is_kept_under_the_agent_its_subject_names() {
let reported = Reported::default();

7
swarm-controller/src/agent_status.rs
View file

@ -73,6 +73,11 @@ pub struct AgentStatusRow {
/// `AppState::agent_login` — this reader sees the hive's bucket, and the
/// flag comes from the agent itself.
pub needs_login: Option<bool>,
/// Why the agent's own harness paused its turn loop, from its latest
/// turn-state header; `None` when that header named none, none arrived
/// since swarm-controller started, or the harness predates the field.
/// Filled in by the `GET /api/agents/status` handler like `needs_login`.
pub paused_by_hive: Option<String>,
/// The archetype the agent is built from; `None` for a custom agent. Always
/// `None` out of this reader, filled in by the `GET /api/agents/status`
/// handler from the `agent-archetype` bucket.
@ -202,6 +207,7 @@ fn render(
config_pr: None,
wanted: None,
needs_login: None,
paused_by_hive: None,
archetype: None,
},
})
@ -248,6 +254,7 @@ fn row(
config_pr: None,
wanted: None,
needs_login: None,
paused_by_hive: None,
archetype: None,
}
}

3
swarm-controller/src/main.rs
View file

@ -70,6 +70,7 @@ mod notify_flows;
mod notify_router_api;
mod observability_grant;
mod otel_http_client;
mod pause_hold;
mod pr_report;
mod proxy_credential;
mod pulls;
@ -237,6 +238,7 @@ async fn main() -> Result<()> {
read_policy::ensure_agent_policies().await;
let state = AppState {
agent_login: startup::agent_login(status.as_ref(), &hives),
hives: Arc::new(hives),
links: Arc::new(load_links()),
wanted: wanted_writer(status.as_ref()),
@ -244,7 +246,6 @@ async fn main() -> Result<()> {
agent_icons: agent_icon_reader(status.as_ref()),
agents_md: agents_md_reader(status.as_ref()),
reported_inference: reported_inference(status.as_ref()),
agent_login: startup::agent_login(status.as_ref()),
runs: review_classifier::runs::Runs::Live(status.as_ref().map(|s| s.queue_client())),
builds: builder::runs::Builds::Live(status.as_ref().map(|s| s.queue_client())),
status,

133
swarm-controller/src/pause_hold.rs Normal file
View file

@ -0,0 +1,133 @@
//! Holds an agent the hive paused, so the swarm's declaration stops undoing it.
//!
//! An agent whose harness paused its own turn loop says so in its turn-state
//! header (`paused_by_hive`, see [`crate::agent_login::Reported`]). The hive
//! resumes any paused agent that the swarm declares `Up`, so this declares it
//! `Paused` instead, which the hive's pause marker already agrees with. Only an
//! operator declaring the agent `Up` again resumes it.
//!
//! Each pause is held once. A header re-sent after the operator's resume and
//! before the hive cleared the marker still reports it, and must not pause the
//! agent a second time. The agent is only declared `Paused` while it is
//! declared `Up`: an operator's `Offline` or `Destroyed` stays what it is.
use std::sync::Arc;
use std::time::Duration;
use anyhow::Result;
use crate::agent_login::Reported;
use crate::wanted::WantedWriter;
/// How often the reported pauses are looked at.
const POLL: Duration = Duration::from_secs(5);
/// Hold the pauses `login` learns of, for as long as this daemon runs.
pub(crate) fn spawn(login: Arc<Reported>, wanted: Arc<WantedWriter>, hives: Vec<String>) {
tokio::spawn(async move {
loop {
tokio::time::sleep(POLL).await;
hold_pending(&login, &wanted, &hives).await;
}
});
}
/// Hold every reported pause not held yet. A pause that could not be held is
/// tried again on the next call.
pub(crate) async fn hold_pending(login: &Reported, wanted: &WantedWriter, hives: &[String]) {
for agent in login.unheld_pauses() {
match hold(wanted, hives, &agent).await {
Ok(()) => login.mark_pause_held(&agent),
Err(e) => tracing::warn!(
%agent,
error = %format!("{e:#}"),
"holding a hive-paused agent failed; will retry"
),
}
}
}
/// Declare `agent` `Paused` on whichever hive declares it `Up`.
async fn hold(wanted: &WantedWriter, hives: &[String], agent: &str) -> Result<()> {
for hive in hives {
if wanted.pause_if_up(hive, agent).await? {
tracing::info!(%hive, %agent, "the hive paused this agent; declared it paused");
}
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::hold_pending;
use crate::agent_login::Reported;
use crate::test_nats::TestNats;
use crate::wanted::WantedWriter;
use swarm_wire::wanted::AgentState;
fn header(reason: Option<&str>) -> Vec<u8> {
let mut header = serde_json::json!({ "agent_state": "paused", "needs_login": false });
if let Some(reason) = reason {
header["paused_by_hive"] = reason.into();
}
serde_json::to_vec(&header).unwrap()
}
async fn declared(wanted: &WantedWriter, agent: &str) -> Option<AgentState> {
let declaration = wanted.view("pr1ma").await.unwrap()?;
declaration.agents.get(agent).map(|a| a.state)
}
/// The whole fix: a hive-paused agent declared `Up` is declared `Paused`,
/// the operator's `Up` after that sticks, and the next pause is held too.
#[tokio::test]
async fn a_hive_pause_replaces_up_and_an_operator_up_sticks() {
let nats = TestNats::start();
let wanted = WantedWriter::new(nats.client().await);
let hives = vec!["pr1ma".to_owned()];
let login = Reported::default();
let subject = swarm_queue_client::agent_state_subject("atlas");
wanted.set("pr1ma", "atlas", AgentState::Up).await.unwrap();
wanted.set("pr1ma", "iris", AgentState::Up).await.unwrap();
login.record(&subject, &header(Some("skipped get_loose_ends")));
hold_pending(&login, &wanted, &hives).await;
assert_eq!(declared(&wanted, "atlas").await, Some(AgentState::Paused));
assert_eq!(declared(&wanted, "iris").await, Some(AgentState::Up));
// The operator resumes; the hive has not cleared its marker yet and
// re-sends the header.
wanted.set("pr1ma", "atlas", AgentState::Up).await.unwrap();
login.record(&subject, &header(Some("skipped get_loose_ends")));
hold_pending(&login, &wanted, &hives).await;
assert_eq!(declared(&wanted, "atlas").await, Some(AgentState::Up));
// The hive clears the marker, then pauses the agent again.
login.record(&subject, &header(None));
login.record(&subject, &header(Some("skipped get_loose_ends")));
hold_pending(&login, &wanted, &hives).await;
assert_eq!(declared(&wanted, "atlas").await, Some(AgentState::Paused));
}
/// An operator's `Offline` is not overruled by the agent's pause, and an
/// agent no hive declares leaves the pause to be tried again.
#[tokio::test]
async fn only_an_agent_declared_up_is_paused() {
let nats = TestNats::start();
let wanted = WantedWriter::new(nats.client().await);
let hives = vec!["pr1ma".to_owned()];
let login = Reported::default();
wanted
.set("pr1ma", "atlas", AgentState::Offline)
.await
.unwrap();
login.record(
&swarm_queue_client::agent_state_subject("atlas"),
&header(Some("skipped get_loose_ends")),
);
hold_pending(&login, &wanted, &hives).await;
assert_eq!(declared(&wanted, "atlas").await, Some(AgentState::Offline));
assert!(login.unheld_pauses().is_empty(), "nothing left to hold");
}
}

2
swarm-controller/src/routes/agents.rs
View file

@ -471,6 +471,7 @@ pub(crate) async fn get_agents_status(
if let Some(login) = state.agent_login.as_ref() {
for row in &mut rows {
row.needs_login = login.get(&row.name);
row.paused_by_hive = login.paused_by_hive(&row.name);
}
}
// Absent on a failed read, like `wanted`: one unreadable bucket
@ -1816,6 +1817,7 @@ mod tests {
config_pr: None,
wanted: None,
needs_login: None,
paused_by_hive: None,
archetype: None,
}
}

16
swarm-controller/src/startup.rs
View file

@ -31,9 +31,9 @@ use crate::{
agent_permissions, agent_renewal, agent_state_stream, agent_status, agents_md, alerts,
archetypes, auth, backend_env, builder, claude_login, config_drift, config_pr, egress_allow,
egress_deny, forge, forge_account, github_account, gitlab_account, inference, issue_report,
linked_accounts, matrix_account, message_stream, notify_flows, notify_router_api, pr_report,
pulls, queue_identity, review_classifier, schedules, status, subagent_term, term_stream,
wanted, webhook,
linked_accounts, matrix_account, message_stream, notify_flows, notify_router_api, pause_hold,
pr_report, pulls, queue_identity, review_classifier, schedules, status, subagent_term,
term_stream, wanted, webhook,
};
/// Where the daemon binds, overridable via `SWARM_CONTROLLER_SOCKET`.
@ -206,11 +206,17 @@ pub(crate) fn reported_inference(
}
/// Whether each agent last said it needs a login, read on the status
/// reader's connection — same rationale as [`wanted_writer`].
/// reader's connection — same rationale as [`wanted_writer`] — and, from the
/// same headers, which agents their hive paused: those are declared `Paused`
/// (see [`pause_hold`]).
pub(crate) fn agent_login(
status: Option<&Arc<status::StatusReader>>,
hives: &[HiveEntry],
) -> Option<Arc<agent_login::Reported>> {
Some(agent_login::Reported::spawn(status?.queue_client()))
let login = agent_login::Reported::spawn(status?.queue_client());
let hives = hives.iter().map(|h| h.name.clone()).collect();
pause_hold::spawn(Arc::clone(&login), wanted_writer(status)?, hives);
Some(login)
}
/// The homeserver agents' own matrix accounts are minted on. Unset is this

58
swarm-controller/src/wanted.rs
View file

@ -40,17 +40,23 @@ enum Wrote {
/// and a caller that gets an error can ask again with fresh intent.
const MAX_ATTEMPTS: usize = 5;
/// A hive's current declaration; none yet reads as an empty one, an undecodable
/// value as an error.
fn decode(current: Option<&[u8]>) -> Result<HiveWanted> {
match current {
Some(raw) => serde_json::from_slice::<HiveWanted>(raw)
.context("the hive's current declaration is not decodable"),
None => Ok(HiveWanted::default()),
}
}
/// Apply one agent's declared state to a hive's current declaration.
///
/// `current` is the hive's whole agent map — an undecodable value is an
/// **error**, not treated as absent, so a bad read never discards every
/// other agent's declaration under a fresh-start fallback.
fn apply(current: Option<&[u8]>, agent: &str, state: AgentState) -> Result<(HiveWanted, Vec<u8>)> {
let mut declaration = match current {
Some(raw) => serde_json::from_slice::<HiveWanted>(raw)
.context("the hive's current declaration is not decodable")?,
None => HiveWanted::default(),
};
let mut declaration = decode(current)?;
declaration
.agents
.insert(agent.to_owned(), AgentWanted { state });
@ -98,11 +104,37 @@ impl WantedWriter {
/// Declare `agent` on `hive` to be in `state`, unconditionally (see the
/// module doc), and return the whole declaration as published.
pub async fn set(&self, hive: &str, agent: &str, state: AgentState) -> Result<HiveWanted> {
self.write(hive, agent, state, None)
.await
.map(|(declaration, _)| declaration)
}
/// Declare `agent` on `hive` `Paused` if, and only if, the declaration at
/// the moment of the write says `Up`: an operator who has since set the
/// agent offline, destroyed or already paused is not overruled. `true`
/// when the declaration now says `Paused` because of this call.
pub async fn pause_if_up(&self, hive: &str, agent: &str) -> Result<bool> {
self.write(hive, agent, AgentState::Paused, Some(AgentState::Up))
.await
.map(|(_, applied)| applied)
}
/// The write behind [`Self::set`] and [`Self::pause_if_up`]: the
/// declaration as published and whether `state` was written. With
/// `only_if`, the write happens only while the agent is declared in that
/// state.
///
/// Read-modify-write against the entry's revision, not a plain `put`:
/// the value is the hive's whole agent map, so a blind write would drop
/// a concurrent change to a different agent.
pub async fn set(&self, hive: &str, agent: &str, state: AgentState) -> Result<HiveWanted> {
async fn write(
&self,
hive: &str,
agent: &str,
state: AgentState,
only_if: Option<AgentState>,
) -> Result<(HiveWanted, bool)> {
swarm_queue_client::ensure_connected(&self.client)?;
let store = swarm_queue_client::wanted::open_or_create(&self.client, hive).await?;
@ -112,8 +144,14 @@ impl WantedWriter {
.await
.with_context(|| format!("reading the declaration for {hive}"))?;
let revision = entry.as_ref().map(|e| e.revision);
let (declaration, encoded) =
apply(entry.as_ref().map(|e| e.value.as_ref()), agent, state)?;
let current = entry.as_ref().map(|e| e.value.as_ref());
if let Some(from) = only_if {
let declaration = decode(current)?;
if declaration.agents.get(agent).map(|a| a.state) != Some(from) {
return Ok((declaration, false));
}
}
let (declaration, encoded) = apply(current, agent, state)?;
// A redeclare of the identical state re-encodes the declaration
// unchanged (a caller that already sees this state and asks
@ -124,7 +162,7 @@ impl WantedWriter {
// between reads.
if entry.as_ref().map(|e| e.value.as_ref()) == Some(encoded.as_slice()) {
tracing::debug!(hive, agent, "declaration already current, no write needed");
return Ok(declaration);
return Ok((declaration, true));
}
// `update` and `create` have separate error types, and only one
@ -149,7 +187,7 @@ impl WantedWriter {
match written {
Wrote::Ok => {
tracing::info!(hive, agent, ?state, "declared agent state");
return Ok(declaration);
return Ok((declaration, true));
}
// Re-read and re-apply onto the winner's value, not over it.
Wrote::LostRace => {

43
swarm-wire/src/agent_pause.rs Normal file
View file

@ -0,0 +1,43 @@
//! Whether the agent's own harness paused its turn loop, as the agent itself
//! reports it.
//!
//! Part of the turn-state header the agent publishes on
//! `swarm_queue_client::agent_state_subject`, beside [`crate::agent_login::Login`];
//! swarm-controller decodes it out of that header with this same type.
/// The hive-pause fields of an agent's turn-state header.
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub struct HivePause {
/// Why the harness paused its own turn loop; present only while that
/// pause is in force. An operator's pause (`hivectl pause`, the
/// dashboard) is not reported here, and neither is any pause by a harness
/// that predates the field, so absence says nothing about `agent_state`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub paused_by_hive: Option<String>,
}
#[cfg(test)]
mod tests {
use super::HivePause;
#[test]
fn a_header_without_the_field_decodes_as_not_paused_by_the_hive() {
let header = serde_json::json!({ "agent_state": "paused", "needs_login": false });
let decoded: HivePause = serde_json::from_value(header).expect("an old header decodes");
assert_eq!(decoded.paused_by_hive, None);
}
#[test]
fn the_reason_round_trips_and_an_absent_one_is_not_sent() {
let sent = HivePause {
paused_by_hive: Some("skipped get_loose_ends".to_owned()),
};
let value = serde_json::to_value(&sent).expect("serialises");
assert_eq!(value["paused_by_hive"], "skipped get_loose_ends");
let decoded: HivePause = serde_json::from_value(value).expect("decodes");
assert_eq!(decoded, sent);
let none = serde_json::to_value(HivePause::default()).expect("serialises");
assert!(none.as_object().expect("an object").is_empty());
}
}

1
swarm-wire/src/lib.rs
View file

@ -11,6 +11,7 @@ pub mod agent_icon;
pub mod agent_inference;
pub mod agent_limits;
pub mod agent_login;
pub mod agent_pause;
pub mod agent_permissions;
pub mod agent_schedules;
pub mod agent_status;