a hive-side pause holds against the swarm's declared state until an operator resumes
the harness writes the reason into the pause marker when it pauses itself and reports it as paused_by_hive in its turn-state header. swarm-controller declares such an agent paused (only if declared up) and shows the reason on the agents status row and the swarm ui card.
This commit is contained in:
parent
6470e2b109
commit
e5c55257f8
17 changed files with 517 additions and 26 deletions
|
|
@ -225,7 +225,7 @@ destroy/recreate, gone on `--purge`.
|
||||||
|
|
||||||
### `/harness/paused` (per agent)
|
### `/harness/paused` (per agent)
|
||||||
|
|
||||||
Empty marker file. Its presence parks the agent's turn loop: the
|
Marker file, empty when an operator paused the agent. Its presence parks the agent's turn loop: the
|
||||||
harness keeps serving its web UI and MCP daemons but drives no turns,
|
harness keeps serving its web UI and MCP daemons but drives no turns,
|
||||||
and inbox messages queue unacked until it's removed (see
|
and inbox messages queue unacked until it's removed (see
|
||||||
[turn loop](../../4-implementation/turn-loop/README.md#the-loop)).
|
[turn loop](../../4-implementation/turn-loop/README.md#the-loop)).
|
||||||
|
|
@ -243,6 +243,12 @@ pause keeps working when the harness is wedged or the container is
|
||||||
stopped.
|
stopped.
|
||||||
<!-- vale write-good.Passive = YES -->
|
<!-- vale write-good.Passive = YES -->
|
||||||
|
|
||||||
|
When the harness pauses itself, after three todo wakes in a row that skipped
|
||||||
|
`get_loose_ends`, it writes the reason into the marker. The agent's turn-state
|
||||||
|
header reports a non-empty marker as `paused_by_hive`
|
||||||
|
([swarm](../swarm/README.md)), and the swarm then declares the
|
||||||
|
agent `paused` so the hive's wanted-state convergence doesn't resume it.
|
||||||
|
|
||||||
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
|
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
|
||||||
is the agent's own space to fill, and this is harness control state.
|
is the agent's own space to fill, and this is harness control state.
|
||||||
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
|
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
|
||||||
|
|
|
||||||
|
|
@ -273,6 +273,17 @@ context budget and the last turn's context and cost token blocks,
|
||||||
session and its turns wait for a login. The harness sets it from its own login
|
session and its turns wait for a login. The harness sets it from its own login
|
||||||
state, so it reaches the swarm without going through the hive.
|
state, so it reaches the swarm without going through the hive.
|
||||||
|
|
||||||
|
A header also carries `paused_by_hive` while the harness's own pause is in force:
|
||||||
|
the reason it wrote into the pause marker when it paused its turn loop. A pause
|
||||||
|
by an operator (`hivectl agent <name> pause`, the dashboard) leaves the marker
|
||||||
|
empty and reports nothing here, and so does a harness that predates the field.
|
||||||
|
The controller reacts by declaring the agent `paused`, but only when the swarm
|
||||||
|
declares it `up` at that moment. Without that, the hive resumes any paused agent the
|
||||||
|
swarm declares `up` on its next convergence. It acts once per pause: the entry
|
||||||
|
stays until a header without the field arrives. Setting the agent `up` again
|
||||||
|
resumes it, and the hive clears the marker. If the swarm is on a controller that
|
||||||
|
predates the field, the hive's pause lasts until its next convergence, as before.
|
||||||
|
|
||||||
`agent_state` reuses the swarm's own wanted-state vocabulary
|
`agent_state` reuses the swarm's own wanted-state vocabulary
|
||||||
(`up`/`offline`/`paused`/`destroyed`) so a reader can compare what an agent _is_
|
(`up`/`offline`/`paused`/`destroyed`) so a reader can compare what an agent _is_
|
||||||
against what the swarm declared it should be without translating between two
|
against what the swarm declared it should be without translating between two
|
||||||
|
|
@ -301,7 +312,8 @@ agent's latest `model`, `effort` and their sources for
|
||||||
agent's latest `needs_login` for the `needs_login` field of
|
agent's latest `needs_login` for the `needs_login` field of
|
||||||
`GET /api/agents/status`. That field is `null` until a header carrying the flag
|
`GET /api/agents/status`. That field is `null` until a header carrying the flag
|
||||||
arrives, so for up to a minute after the controller starts, and for an agent
|
arrives, so for up to a minute after the controller starts, and for an agent
|
||||||
whose harness doesn't send it.
|
whose harness doesn't send it. The row's `paused_by_hive` comes from the same
|
||||||
|
headers and is `null` in the same cases.
|
||||||
|
|
||||||
The third thing an agent publishes is its **icon**, the same SVG its own
|
The third thing an agent publishes is its **icon**, the same SVG its own
|
||||||
`GET /icon` serves. It goes into the `agent-icons` KV bucket under the key
|
`GET /icon` serves. It goes into the `agent-icons` KV bucket under the key
|
||||||
|
|
|
||||||
|
|
@ -80,8 +80,9 @@ action that asks first.
|
||||||
card per agent, none of them inside a bigger card. A card's
|
card per agent, none of them inside a bigger card. A card's
|
||||||
header shows its icon, name, freshness, the status message, a **terminal**
|
header shows its icon, name, freshness, the status message, a **terminal**
|
||||||
button that opens the agent's terminal tab, the **wanted** menu (up, offline,
|
button that opens the agent's terminal tab, the **wanted** menu (up, offline,
|
||||||
paused, destroy), and a **login needed** badge with a
|
paused, destroy), a **paused by hive** badge when the agent's own harness paused
|
||||||
**log in** button when its claude has no session. Select the header to open the
|
its turn loop (hover for the reason; set the agent **up** to resume it), and a
|
||||||
|
**login needed** badge with a **log in** button when its claude has no session. Select the header to open the
|
||||||
card in place; select it again to close it. Several cards can be open at once,
|
card in place; select it again to close it. Several cards can be open at once,
|
||||||
and a card stays open across the page's refresh, but not across a reload. The
|
and a card stays open across the page's refresh, but not across a reload. The
|
||||||
filter card, which also holds the **+ agent** button and the refresh picker,
|
filter card, which also holds the **+ agent** button and the refresh picker,
|
||||||
|
|
|
||||||
|
|
@ -43,6 +43,10 @@ export interface AgentRow {
|
||||||
// no header since swarm-controller started, or an agent that does not
|
// no header since swarm-controller started, or an agent that does not
|
||||||
// send the flag — not known, so not "needs login".
|
// send the flag — not known, so not "needs login".
|
||||||
needs_login: boolean | null;
|
needs_login: boolean | null;
|
||||||
|
// Why the agent's own harness paused its turn loop. `null`: it did not, or
|
||||||
|
// the harness does not send the field. The controller declares such an agent
|
||||||
|
// paused, and only setting it up again resumes it.
|
||||||
|
paused_by_hive: string | null;
|
||||||
// The archetype the agent is built from; `null` for a custom agent.
|
// The archetype the agent is built from; `null` for a custom agent.
|
||||||
archetype: { source: string; name: string } | null;
|
archetype: { source: string; name: string } | null;
|
||||||
}
|
}
|
||||||
|
|
@ -61,6 +65,13 @@ export function showsLoginNeeded(row: Pick<AgentRow, "needs_login">): boolean {
|
||||||
return row.needs_login === true;
|
return row.needs_login === true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The reason to show on the "paused by hive" badge; `null` shows no badge.
|
||||||
|
export function hivePauseReason(
|
||||||
|
row: Pick<AgentRow, "paused_by_hive">,
|
||||||
|
): string | null {
|
||||||
|
return row.paused_by_hive ? row.paused_by_hive : null;
|
||||||
|
}
|
||||||
|
|
||||||
// Same tone pairing as HivesCard — one freshness enum shared by both
|
// Same tone pairing as HivesCard — one freshness enum shared by both
|
||||||
// endpoints, so the same tone rule applies to both pages — but
|
// endpoints, so the same tone rule applies to both pages — but
|
||||||
// `unknown`'s label diverges deliberately: an agent outside the
|
// `unknown`'s label diverges deliberately: an agent outside the
|
||||||
|
|
|
||||||
|
|
@ -23,7 +23,12 @@ import { AgentIcon } from "../AgentIcon.js";
|
||||||
import { ClaudeLoginDialog } from "../claude-login/ClaudeLoginDialog.js";
|
import { ClaudeLoginDialog } from "../claude-login/ClaudeLoginDialog.js";
|
||||||
import { AgentDetail } from "../detail/AgentDetail.js";
|
import { AgentDetail } from "../detail/AgentDetail.js";
|
||||||
import { termPaths } from "../terminal/term-paths.js";
|
import { termPaths } from "../terminal/term-paths.js";
|
||||||
import { FRESHNESS, showsLoginNeeded, type AgentRow } from "../AgentTypes.js";
|
import {
|
||||||
|
FRESHNESS,
|
||||||
|
hivePauseReason,
|
||||||
|
showsLoginNeeded,
|
||||||
|
type AgentRow,
|
||||||
|
} from "../AgentTypes.js";
|
||||||
import type { AgentActions } from "./AgentActions.js";
|
import type { AgentActions } from "./AgentActions.js";
|
||||||
import { WantedMenu } from "./WantedMenu.js";
|
import { WantedMenu } from "./WantedMenu.js";
|
||||||
import { AgentCardLayout } from "./AgentCardLayout.js";
|
import { AgentCardLayout } from "./AgentCardLayout.js";
|
||||||
|
|
@ -82,6 +87,13 @@ export function AgentCard({
|
||||||
</>
|
</>
|
||||||
}
|
}
|
||||||
/>
|
/>
|
||||||
|
{hivePauseReason(row) !== null ? (
|
||||||
|
<Badge
|
||||||
|
tone="warning"
|
||||||
|
value="paused by hive"
|
||||||
|
title={`the hive paused this agent: ${hivePauseReason(row)}. Set it up again to resume it.`}
|
||||||
|
/>
|
||||||
|
) : null}
|
||||||
{showsLoginNeeded(row) ? (
|
{showsLoginNeeded(row) ? (
|
||||||
<span
|
<span
|
||||||
class="ui-agent-card-login"
|
class="ui-agent-card-login"
|
||||||
|
|
|
||||||
|
|
@ -5,6 +5,7 @@ import assert from "node:assert/strict";
|
||||||
import { test } from "node:test";
|
import { test } from "node:test";
|
||||||
import {
|
import {
|
||||||
canTurnIntoCustom,
|
canTurnIntoCustom,
|
||||||
|
hivePauseReason,
|
||||||
showsLoginNeeded,
|
showsLoginNeeded,
|
||||||
} from "../src/agent/AgentTypes.ts";
|
} from "../src/agent/AgentTypes.ts";
|
||||||
|
|
||||||
|
|
@ -42,3 +43,13 @@ test("turn into custom is offered to a placed archetype agent only", () => {
|
||||||
false,
|
false,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test("an agent its hive paused shows the reason, any other shows none", () => {
|
||||||
|
assert.equal(
|
||||||
|
hivePauseReason({ paused_by_hive: "skipped get_loose_ends" }),
|
||||||
|
"skipped get_loose_ends",
|
||||||
|
);
|
||||||
|
assert.equal(hivePauseReason({ paused_by_hive: null }), null);
|
||||||
|
assert.equal(hivePauseReason({ paused_by_hive: "" }), null);
|
||||||
|
assert.equal(hivePauseReason({}), null);
|
||||||
|
});
|
||||||
|
|
|
||||||
|
|
@ -945,6 +945,14 @@ async fn drive_turn_and_wake_chain<S: Surface>(
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The pause marker's content for an auto-pause after `streak` missed todo
|
||||||
|
/// wakes. A non-empty marker is a pause by the harness itself, which the
|
||||||
|
/// turn-state header reports as `paused_by_hive`; the operator's pause writes
|
||||||
|
/// an empty one.
|
||||||
|
fn todo_miss_pause_reason(streak: u32) -> String {
|
||||||
|
format!("skipped get_loose_ends on {streak} consecutive todo wakes")
|
||||||
|
}
|
||||||
|
|
||||||
/// Apply a finished turn's `todo_wake_checked` signal to the miss-streak:
|
/// Apply a finished turn's `todo_wake_checked` signal to the miss-streak:
|
||||||
/// resets on a checked wake, increments (and auto-pauses at the threshold)
|
/// resets on a checked wake, increments (and auto-pauses at the threshold)
|
||||||
/// on a missed one, no-ops for a non-todo-wake turn. Split out of
|
/// on a missed one, no-ops for a non-todo-wake turn. Split out of
|
||||||
|
|
@ -963,15 +971,19 @@ fn apply_todo_wake_checked(checked: Option<bool>, todo_miss_streak: &mut u32, bu
|
||||||
);
|
);
|
||||||
if streak >= TODO_MISS_PAUSE_THRESHOLD {
|
if streak >= TODO_MISS_PAUSE_THRESHOLD {
|
||||||
tracing::warn!("todo-miss streak hit the threshold — pausing the turn loop");
|
tracing::warn!("todo-miss streak hit the threshold — pausing the turn loop");
|
||||||
|
// Marker first: the Note triggers a turn-state header rebuild,
|
||||||
|
// which reads the reason back out of this file.
|
||||||
|
if let Err(e) =
|
||||||
|
std::fs::write(paths::paused_marker(), todo_miss_pause_reason(streak))
|
||||||
|
{
|
||||||
|
tracing::warn!(error = ?e, "failed to write pause marker");
|
||||||
|
}
|
||||||
bus.emit(LiveEvent::Note {
|
bus.emit(LiveEvent::Note {
|
||||||
text: format!(
|
text: format!(
|
||||||
"auto-paused: skipped get_loose_ends on {streak} \
|
"auto-paused: skipped get_loose_ends on {streak} \
|
||||||
consecutive todo wakes — an operator needs to resume this agent"
|
consecutive todo wakes — an operator needs to resume this agent"
|
||||||
),
|
),
|
||||||
});
|
});
|
||||||
if let Err(e) = std::fs::write(paths::paused_marker(), "") {
|
|
||||||
tracing::warn!(error = ?e, "failed to write pause marker");
|
|
||||||
}
|
|
||||||
// Fresh slate for whenever this agent gets resumed — see
|
// Fresh slate for whenever this agent gets resumed — see
|
||||||
// `serve_loop`'s `was_paused` reset for the mirror side.
|
// `serve_loop`'s `was_paused` reset for the mirror side.
|
||||||
*todo_miss_streak = 0;
|
*todo_miss_streak = 0;
|
||||||
|
|
@ -1254,4 +1266,31 @@ mod tests {
|
||||||
.await;
|
.await;
|
||||||
assert!(matches!(woke, RecvOutcome::Empty));
|
assert!(matches!(woke, RecvOutcome::Empty));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The third consecutive miss parks the agent with its reason in the
|
||||||
|
/// marker; an earlier miss, or a wake that did call `get_loose_ends`,
|
||||||
|
/// writes nothing.
|
||||||
|
#[test]
|
||||||
|
fn the_threshold_miss_writes_the_reason_into_the_pause_marker() {
|
||||||
|
crate::paths::with_harness_dir(|| {
|
||||||
|
let bus = crate::events::Bus::new();
|
||||||
|
let mut streak = 0;
|
||||||
|
for _ in 1..super::TODO_MISS_PAUSE_THRESHOLD {
|
||||||
|
super::apply_todo_wake_checked(Some(false), &mut streak, &bus);
|
||||||
|
assert!(
|
||||||
|
!crate::paths::paused_marker().exists(),
|
||||||
|
"below the threshold"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
super::apply_todo_wake_checked(Some(true), &mut streak, &bus);
|
||||||
|
assert_eq!(streak, 0);
|
||||||
|
for _ in 0..super::TODO_MISS_PAUSE_THRESHOLD {
|
||||||
|
super::apply_todo_wake_checked(Some(false), &mut streak, &bus);
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
std::fs::read_to_string(crate::paths::paused_marker()).unwrap(),
|
||||||
|
super::todo_miss_pause_reason(super::TODO_MISS_PAUSE_THRESHOLD)
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -35,6 +35,7 @@ use tokio::time::{Interval, MissedTickBehavior};
|
||||||
|
|
||||||
use swarm_wire::agent_inference::Effective;
|
use swarm_wire::agent_inference::Effective;
|
||||||
use swarm_wire::agent_login::Login;
|
use swarm_wire::agent_login::Login;
|
||||||
|
use swarm_wire::agent_pause::HivePause;
|
||||||
use swarm_wire::hive_status::{Capabilities, TOLERANT_PERMISSIONS};
|
use swarm_wire::hive_status::{Capabilities, TOLERANT_PERMISSIONS};
|
||||||
use swarm_wire::wanted::AgentState;
|
use swarm_wire::wanted::AgentState;
|
||||||
|
|
||||||
|
|
@ -101,6 +102,10 @@ struct AgentStateMsg {
|
||||||
/// `needs_login`, from this process's own [`LoginState`], so always set.
|
/// `needs_login`, from this process's own [`LoginState`], so always set.
|
||||||
#[serde(flatten)]
|
#[serde(flatten)]
|
||||||
login: Login,
|
login: Login,
|
||||||
|
/// `paused_by_hive`, the reason in the pause marker when this harness wrote
|
||||||
|
/// it itself; absent otherwise.
|
||||||
|
#[serde(flatten)]
|
||||||
|
hive_pause: HivePause,
|
||||||
/// What this harness can do, so swarm-controller grants an agent a
|
/// What this harness can do, so swarm-controller grants an agent a
|
||||||
/// permission only when the harness running it can decode it.
|
/// permission only when the harness running it can decode it.
|
||||||
#[serde(flatten)]
|
#[serde(flatten)]
|
||||||
|
|
@ -136,6 +141,20 @@ fn observed_state() -> AgentState {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Longest reason reported; the marker is a file the agent can write to.
|
||||||
|
const PAUSE_REASON_MAX_CHARS: usize = 200;
|
||||||
|
|
||||||
|
/// The header's pause-by-harness field: the pause marker's content, when it has
|
||||||
|
/// any. The operator's pause writes an empty marker, so it reports nothing.
|
||||||
|
fn hive_pause() -> HivePause {
|
||||||
|
let reason = std::fs::read_to_string(crate::paths::paused_marker()).unwrap_or_default();
|
||||||
|
let reason = reason.trim();
|
||||||
|
HivePause {
|
||||||
|
paused_by_hive: (!reason.is_empty())
|
||||||
|
.then(|| reason.chars().take(PAUSE_REASON_MAX_CHARS).collect()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// The header's login fields for `state`.
|
/// The header's login fields for `state`.
|
||||||
fn login(state: LoginState) -> Login {
|
fn login(state: LoginState) -> Login {
|
||||||
Login {
|
Login {
|
||||||
|
|
@ -170,6 +189,7 @@ fn snapshot(bus: &Bus, login_state: &Mutex<LoginState>) -> AgentStateMsg {
|
||||||
ctx_usage: bus.last_ctx_usage(),
|
ctx_usage: bus.last_ctx_usage(),
|
||||||
cost_usage: bus.last_cost_usage(),
|
cost_usage: bus.last_cost_usage(),
|
||||||
login: login(*login_state.lock().unwrap_or_else(PoisonError::into_inner)),
|
login: login(*login_state.lock().unwrap_or_else(PoisonError::into_inner)),
|
||||||
|
hive_pause: hive_pause(),
|
||||||
capabilities: capabilities(),
|
capabilities: capabilities(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -346,13 +366,14 @@ async fn publish(
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::{
|
use super::{
|
||||||
AgentStateMsg, REFRESH_INTERVAL, Trigger, capabilities, due, login, next_trigger,
|
AgentStateMsg, PAUSE_REASON_MAX_CHARS, REFRESH_INTERVAL, Trigger, capabilities, due,
|
||||||
refresh_timer, snapshot,
|
hive_pause, login, next_trigger, refresh_timer, snapshot,
|
||||||
};
|
};
|
||||||
use crate::events::{BusEvent, LiveEvent, TurnState};
|
use crate::events::{BusEvent, LiveEvent, TurnState};
|
||||||
use crate::login::LoginState;
|
use crate::login::LoginState;
|
||||||
use std::sync::Mutex;
|
use std::sync::Mutex;
|
||||||
use swarm_wire::agent_inference::{Effective, Source};
|
use swarm_wire::agent_inference::{Effective, Source};
|
||||||
|
use swarm_wire::agent_pause::HivePause;
|
||||||
use swarm_wire::hive_status::{Capabilities, TOLERANT_PERMISSIONS};
|
use swarm_wire::hive_status::{Capabilities, TOLERANT_PERMISSIONS};
|
||||||
use swarm_wire::wanted::AgentState;
|
use swarm_wire::wanted::AgentState;
|
||||||
use tokio::sync::broadcast;
|
use tokio::sync::broadcast;
|
||||||
|
|
@ -403,6 +424,9 @@ mod tests {
|
||||||
"cache_creation_input_tokens": 88,
|
"cache_creation_input_tokens": 88,
|
||||||
}))),
|
}))),
|
||||||
login: login(LoginState::NeedsLogin),
|
login: login(LoginState::NeedsLogin),
|
||||||
|
hive_pause: HivePause {
|
||||||
|
paused_by_hive: Some("skipped get_loose_ends".to_owned()),
|
||||||
|
},
|
||||||
capabilities: capabilities(),
|
capabilities: capabilities(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -432,6 +456,7 @@ mod tests {
|
||||||
"model",
|
"model",
|
||||||
"model_source",
|
"model_source",
|
||||||
"needs_login",
|
"needs_login",
|
||||||
|
"paused_by_hive",
|
||||||
"resolved_model",
|
"resolved_model",
|
||||||
"turn_state",
|
"turn_state",
|
||||||
"turn_state_since",
|
"turn_state_since",
|
||||||
|
|
@ -523,6 +548,33 @@ mod tests {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The marker's content is the reason; an empty marker (an operator's pause) and
|
||||||
|
/// no marker report nothing, and swarm-controller reads it back with the wire type.
|
||||||
|
#[test]
|
||||||
|
fn the_pause_marker_content_is_the_reported_reason() {
|
||||||
|
crate::paths::with_harness_dir(|| {
|
||||||
|
assert_eq!(hive_pause().paused_by_hive, None, "no marker");
|
||||||
|
std::fs::write(crate::paths::paused_marker(), "").unwrap();
|
||||||
|
assert_eq!(hive_pause().paused_by_hive, None, "operator pause");
|
||||||
|
std::fs::write(crate::paths::paused_marker(), "x".repeat(500)).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
hive_pause().paused_by_hive.unwrap().len(),
|
||||||
|
PAUSE_REASON_MAX_CHARS
|
||||||
|
);
|
||||||
|
std::fs::write(crate::paths::paused_marker(), "skipped get_loose_ends\n").unwrap();
|
||||||
|
let bytes = serde_json::to_vec(&AgentStateMsg {
|
||||||
|
hive_pause: hive_pause(),
|
||||||
|
..msg()
|
||||||
|
})
|
||||||
|
.expect("serialises");
|
||||||
|
let decoded: HivePause = serde_json::from_slice(&bytes).expect("decodes");
|
||||||
|
assert_eq!(
|
||||||
|
decoded.paused_by_hive.as_deref(),
|
||||||
|
Some("skipped get_loose_ends")
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
/// swarm-controller reads the harness's capabilities back out of a header.
|
/// swarm-controller reads the harness's capabilities back out of a header.
|
||||||
#[test]
|
#[test]
|
||||||
fn the_header_advertises_tolerant_permissions() {
|
fn the_header_advertises_tolerant_permissions() {
|
||||||
|
|
|
||||||
|
|
@ -7,6 +7,10 @@
|
||||||
//! an agent has no entry for up to a minute after this daemon starts — the
|
//! an agent has no entry for up to a minute after this daemon starts — the
|
||||||
//! same gap [`crate::inference::Reported`] has on the same subject.
|
//! same gap [`crate::inference::Reported`] has on the same subject.
|
||||||
//!
|
//!
|
||||||
|
//! A header may also carry `paused_by_hive`, the reason the agent's own harness
|
||||||
|
//! paused its turn loop (`swarm_wire::agent_pause`); [`crate::pause_hold`]
|
||||||
|
//! acts on it, and `GET /api/agents/status` shows it.
|
||||||
|
//!
|
||||||
//! The same header carries the harness's `capabilities`
|
//! The same header carries the harness's `capabilities`
|
||||||
//! (`swarm_wire::hive_status`), which [`Reported::advertises`] answers from:
|
//! (`swarm_wire::hive_status`), which [`Reported::advertises`] answers from:
|
||||||
//! the controller grants `logs` or `metrics` only to an agent whose running
|
//! the controller grants `logs` or `metrics` only to an agent whose running
|
||||||
|
|
@ -18,14 +22,27 @@ use std::time::Duration;
|
||||||
|
|
||||||
use futures_util::StreamExt as _;
|
use futures_util::StreamExt as _;
|
||||||
use swarm_wire::agent_login::Login;
|
use swarm_wire::agent_login::Login;
|
||||||
|
use swarm_wire::agent_pause::HivePause;
|
||||||
use swarm_wire::hive_status::Capabilities;
|
use swarm_wire::hive_status::Capabilities;
|
||||||
|
|
||||||
|
/// One agent's reported hive pause.
|
||||||
|
#[derive(Debug)]
|
||||||
|
struct Pause {
|
||||||
|
reason: String,
|
||||||
|
/// Whether [`crate::pause_hold`] has already dealt with this pause, so a
|
||||||
|
/// header re-sent after an operator resumed the agent does not pause it
|
||||||
|
/// again.
|
||||||
|
held: bool,
|
||||||
|
}
|
||||||
|
|
||||||
/// The latest login flag each agent reported, by agent.
|
/// The latest login flag each agent reported, by agent.
|
||||||
#[derive(Debug, Default)]
|
#[derive(Debug, Default)]
|
||||||
pub(crate) struct Reported {
|
pub(crate) struct Reported {
|
||||||
latest: Mutex<HashMap<String, bool>>,
|
latest: Mutex<HashMap<String, bool>>,
|
||||||
/// The capabilities each agent's latest header listed.
|
/// The capabilities each agent's latest header listed.
|
||||||
capabilities: Mutex<HashMap<String, Capabilities>>,
|
capabilities: Mutex<HashMap<String, Capabilities>>,
|
||||||
|
/// Each agent whose latest header carries a hive-pause reason.
|
||||||
|
pauses: Mutex<HashMap<String, Pause>>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Reported {
|
impl Reported {
|
||||||
|
|
@ -75,6 +92,7 @@ impl Reported {
|
||||||
capabilities.remove(agent);
|
capabilities.remove(agent);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
self.record_pause(agent, payload);
|
||||||
let mut latest = self.latest.lock().unwrap_or_else(PoisonError::into_inner);
|
let mut latest = self.latest.lock().unwrap_or_else(PoisonError::into_inner);
|
||||||
match serde_json::from_slice::<Login>(payload) {
|
match serde_json::from_slice::<Login>(payload) {
|
||||||
Ok(Login {
|
Ok(Login {
|
||||||
|
|
@ -92,6 +110,66 @@ impl Reported {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Keep one header's hive-pause reason. A header without one, or one that
|
||||||
|
/// does not decode, removes what was kept, which is how a resumed agent
|
||||||
|
/// is told from one that is still paused.
|
||||||
|
fn record_pause(&self, agent: &str, payload: &[u8]) {
|
||||||
|
let mut pauses = self.pauses.lock().unwrap_or_else(PoisonError::into_inner);
|
||||||
|
match serde_json::from_slice::<HivePause>(payload) {
|
||||||
|
Ok(HivePause {
|
||||||
|
paused_by_hive: Some(reason),
|
||||||
|
}) => {
|
||||||
|
pauses
|
||||||
|
.entry(agent.to_owned())
|
||||||
|
.and_modify(|p| p.reason.clone_from(&reason))
|
||||||
|
.or_insert(Pause {
|
||||||
|
reason,
|
||||||
|
held: false,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(HivePause {
|
||||||
|
paused_by_hive: None,
|
||||||
|
})
|
||||||
|
| Err(_) => {
|
||||||
|
pauses.remove(agent);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `agent`'s hive-pause reason from its latest header; `None` when that
|
||||||
|
/// header carried none, or none arrived since this daemon started.
|
||||||
|
pub(crate) fn paused_by_hive(&self, agent: &str) -> Option<String> {
|
||||||
|
self.pauses
|
||||||
|
.lock()
|
||||||
|
.unwrap_or_else(PoisonError::into_inner)
|
||||||
|
.get(agent)
|
||||||
|
.map(|p| p.reason.clone())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The agents whose reported hive pause has not been held yet.
|
||||||
|
pub(crate) fn unheld_pauses(&self) -> Vec<String> {
|
||||||
|
self.pauses
|
||||||
|
.lock()
|
||||||
|
.unwrap_or_else(PoisonError::into_inner)
|
||||||
|
.iter()
|
||||||
|
.filter(|(_, p)| !p.held)
|
||||||
|
.map(|(agent, _)| agent.clone())
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Note that `agent`'s current hive pause has been dealt with. A no-op
|
||||||
|
/// once its header has stopped reporting one.
|
||||||
|
pub(crate) fn mark_pause_held(&self, agent: &str) {
|
||||||
|
if let Some(p) = self
|
||||||
|
.pauses
|
||||||
|
.lock()
|
||||||
|
.unwrap_or_else(PoisonError::into_inner)
|
||||||
|
.get_mut(agent)
|
||||||
|
{
|
||||||
|
p.held = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Whether `agent`'s latest header lists `capability`. `false` for an
|
/// Whether `agent`'s latest header lists `capability`. `false` for an
|
||||||
/// agent that has sent none since this daemon started, or whose harness
|
/// agent that has sent none since this daemon started, or whose harness
|
||||||
/// predates the field.
|
/// predates the field.
|
||||||
|
|
@ -162,6 +240,44 @@ mod tests {
|
||||||
assert_eq!(reported.get("iris"), None);
|
assert_eq!(reported.get("iris"), None);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn pause_header(reason: Option<&str>) -> Vec<u8> {
|
||||||
|
let mut value: serde_json::Value = serde_json::from_slice(&header(None)).unwrap();
|
||||||
|
if let Some(reason) = reason {
|
||||||
|
value["paused_by_hive"] = reason.into();
|
||||||
|
}
|
||||||
|
serde_json::to_vec(&value).unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A header with the reason reports it, one without clears it, and a pause
|
||||||
|
/// held once stays held until a header without the reason has been seen.
|
||||||
|
#[test]
|
||||||
|
fn a_hive_pause_is_reported_until_a_header_clears_it() {
|
||||||
|
let reported = Reported::default();
|
||||||
|
let subject = swarm_queue_client::agent_state_subject("iris");
|
||||||
|
reported.record(&subject, &header(None));
|
||||||
|
assert_eq!(reported.paused_by_hive("iris"), None, "an old harness");
|
||||||
|
assert!(reported.unheld_pauses().is_empty());
|
||||||
|
|
||||||
|
reported.record(&subject, &pause_header(Some("skipped")));
|
||||||
|
assert_eq!(reported.paused_by_hive("iris").as_deref(), Some("skipped"));
|
||||||
|
assert_eq!(reported.unheld_pauses(), ["iris"]);
|
||||||
|
reported.mark_pause_held("iris");
|
||||||
|
reported.record(&subject, &pause_header(Some("skipped")));
|
||||||
|
assert!(reported.unheld_pauses().is_empty(), "held stays held");
|
||||||
|
assert_eq!(reported.paused_by_hive("iris").as_deref(), Some("skipped"));
|
||||||
|
|
||||||
|
reported.record(&subject, &pause_header(None));
|
||||||
|
assert_eq!(reported.paused_by_hive("iris"), None);
|
||||||
|
reported.record(&subject, &pause_header(Some("skipped")));
|
||||||
|
assert_eq!(
|
||||||
|
reported.unheld_pauses(),
|
||||||
|
["iris"],
|
||||||
|
"a new pause is held anew"
|
||||||
|
);
|
||||||
|
reported.record(&subject, b"not json");
|
||||||
|
assert_eq!(reported.paused_by_hive("iris"), None);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_header_is_kept_under_the_agent_its_subject_names() {
|
fn a_header_is_kept_under_the_agent_its_subject_names() {
|
||||||
let reported = Reported::default();
|
let reported = Reported::default();
|
||||||
|
|
|
||||||
|
|
@ -73,6 +73,11 @@ pub struct AgentStatusRow {
|
||||||
/// `AppState::agent_login` — this reader sees the hive's bucket, and the
|
/// `AppState::agent_login` — this reader sees the hive's bucket, and the
|
||||||
/// flag comes from the agent itself.
|
/// flag comes from the agent itself.
|
||||||
pub needs_login: Option<bool>,
|
pub needs_login: Option<bool>,
|
||||||
|
/// Why the agent's own harness paused its turn loop, from its latest
|
||||||
|
/// turn-state header; `None` when that header named none, none arrived
|
||||||
|
/// since swarm-controller started, or the harness predates the field.
|
||||||
|
/// Filled in by the `GET /api/agents/status` handler like `needs_login`.
|
||||||
|
pub paused_by_hive: Option<String>,
|
||||||
/// The archetype the agent is built from; `None` for a custom agent. Always
|
/// The archetype the agent is built from; `None` for a custom agent. Always
|
||||||
/// `None` out of this reader, filled in by the `GET /api/agents/status`
|
/// `None` out of this reader, filled in by the `GET /api/agents/status`
|
||||||
/// handler from the `agent-archetype` bucket.
|
/// handler from the `agent-archetype` bucket.
|
||||||
|
|
@ -202,6 +207,7 @@ fn render(
|
||||||
config_pr: None,
|
config_pr: None,
|
||||||
wanted: None,
|
wanted: None,
|
||||||
needs_login: None,
|
needs_login: None,
|
||||||
|
paused_by_hive: None,
|
||||||
archetype: None,
|
archetype: None,
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
|
|
@ -248,6 +254,7 @@ fn row(
|
||||||
config_pr: None,
|
config_pr: None,
|
||||||
wanted: None,
|
wanted: None,
|
||||||
needs_login: None,
|
needs_login: None,
|
||||||
|
paused_by_hive: None,
|
||||||
archetype: None,
|
archetype: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -70,6 +70,7 @@ mod notify_flows;
|
||||||
mod notify_router_api;
|
mod notify_router_api;
|
||||||
mod observability_grant;
|
mod observability_grant;
|
||||||
mod otel_http_client;
|
mod otel_http_client;
|
||||||
|
mod pause_hold;
|
||||||
mod pr_report;
|
mod pr_report;
|
||||||
mod proxy_credential;
|
mod proxy_credential;
|
||||||
mod pulls;
|
mod pulls;
|
||||||
|
|
@ -237,6 +238,7 @@ async fn main() -> Result<()> {
|
||||||
read_policy::ensure_agent_policies().await;
|
read_policy::ensure_agent_policies().await;
|
||||||
|
|
||||||
let state = AppState {
|
let state = AppState {
|
||||||
|
agent_login: startup::agent_login(status.as_ref(), &hives),
|
||||||
hives: Arc::new(hives),
|
hives: Arc::new(hives),
|
||||||
links: Arc::new(load_links()),
|
links: Arc::new(load_links()),
|
||||||
wanted: wanted_writer(status.as_ref()),
|
wanted: wanted_writer(status.as_ref()),
|
||||||
|
|
@ -244,7 +246,6 @@ async fn main() -> Result<()> {
|
||||||
agent_icons: agent_icon_reader(status.as_ref()),
|
agent_icons: agent_icon_reader(status.as_ref()),
|
||||||
agents_md: agents_md_reader(status.as_ref()),
|
agents_md: agents_md_reader(status.as_ref()),
|
||||||
reported_inference: reported_inference(status.as_ref()),
|
reported_inference: reported_inference(status.as_ref()),
|
||||||
agent_login: startup::agent_login(status.as_ref()),
|
|
||||||
runs: review_classifier::runs::Runs::Live(status.as_ref().map(|s| s.queue_client())),
|
runs: review_classifier::runs::Runs::Live(status.as_ref().map(|s| s.queue_client())),
|
||||||
builds: builder::runs::Builds::Live(status.as_ref().map(|s| s.queue_client())),
|
builds: builder::runs::Builds::Live(status.as_ref().map(|s| s.queue_client())),
|
||||||
status,
|
status,
|
||||||
|
|
|
||||||
133
swarm-controller/src/pause_hold.rs
Normal file
133
swarm-controller/src/pause_hold.rs
Normal file
|
|
@ -0,0 +1,133 @@
|
||||||
|
//! Holds an agent the hive paused, so the swarm's declaration stops undoing it.
|
||||||
|
//!
|
||||||
|
//! An agent whose harness paused its own turn loop says so in its turn-state
|
||||||
|
//! header (`paused_by_hive`, see [`crate::agent_login::Reported`]). The hive
|
||||||
|
//! resumes any paused agent that the swarm declares `Up`, so this declares it
|
||||||
|
//! `Paused` instead, which the hive's pause marker already agrees with. Only an
|
||||||
|
//! operator declaring the agent `Up` again resumes it.
|
||||||
|
//!
|
||||||
|
//! Each pause is held once. A header re-sent after the operator's resume and
|
||||||
|
//! before the hive cleared the marker still reports it, and must not pause the
|
||||||
|
//! agent a second time. The agent is only declared `Paused` while it is
|
||||||
|
//! declared `Up`: an operator's `Offline` or `Destroyed` stays what it is.
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use anyhow::Result;
|
||||||
|
|
||||||
|
use crate::agent_login::Reported;
|
||||||
|
use crate::wanted::WantedWriter;
|
||||||
|
|
||||||
|
/// How often the reported pauses are looked at.
|
||||||
|
const POLL: Duration = Duration::from_secs(5);
|
||||||
|
|
||||||
|
/// Hold the pauses `login` learns of, for as long as this daemon runs.
|
||||||
|
pub(crate) fn spawn(login: Arc<Reported>, wanted: Arc<WantedWriter>, hives: Vec<String>) {
|
||||||
|
tokio::spawn(async move {
|
||||||
|
loop {
|
||||||
|
tokio::time::sleep(POLL).await;
|
||||||
|
hold_pending(&login, &wanted, &hives).await;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Hold every reported pause not held yet. A pause that could not be held is
|
||||||
|
/// tried again on the next call.
|
||||||
|
pub(crate) async fn hold_pending(login: &Reported, wanted: &WantedWriter, hives: &[String]) {
|
||||||
|
for agent in login.unheld_pauses() {
|
||||||
|
match hold(wanted, hives, &agent).await {
|
||||||
|
Ok(()) => login.mark_pause_held(&agent),
|
||||||
|
Err(e) => tracing::warn!(
|
||||||
|
%agent,
|
||||||
|
error = %format!("{e:#}"),
|
||||||
|
"holding a hive-paused agent failed; will retry"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Declare `agent` `Paused` on whichever hive declares it `Up`.
|
||||||
|
async fn hold(wanted: &WantedWriter, hives: &[String], agent: &str) -> Result<()> {
|
||||||
|
for hive in hives {
|
||||||
|
if wanted.pause_if_up(hive, agent).await? {
|
||||||
|
tracing::info!(%hive, %agent, "the hive paused this agent; declared it paused");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::hold_pending;
|
||||||
|
use crate::agent_login::Reported;
|
||||||
|
use crate::test_nats::TestNats;
|
||||||
|
use crate::wanted::WantedWriter;
|
||||||
|
use swarm_wire::wanted::AgentState;
|
||||||
|
|
||||||
|
fn header(reason: Option<&str>) -> Vec<u8> {
|
||||||
|
let mut header = serde_json::json!({ "agent_state": "paused", "needs_login": false });
|
||||||
|
if let Some(reason) = reason {
|
||||||
|
header["paused_by_hive"] = reason.into();
|
||||||
|
}
|
||||||
|
serde_json::to_vec(&header).unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn declared(wanted: &WantedWriter, agent: &str) -> Option<AgentState> {
|
||||||
|
let declaration = wanted.view("pr1ma").await.unwrap()?;
|
||||||
|
declaration.agents.get(agent).map(|a| a.state)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The whole fix: a hive-paused agent declared `Up` is declared `Paused`,
|
||||||
|
/// the operator's `Up` after that sticks, and the next pause is held too.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn a_hive_pause_replaces_up_and_an_operator_up_sticks() {
|
||||||
|
let nats = TestNats::start();
|
||||||
|
let wanted = WantedWriter::new(nats.client().await);
|
||||||
|
let hives = vec!["pr1ma".to_owned()];
|
||||||
|
let login = Reported::default();
|
||||||
|
let subject = swarm_queue_client::agent_state_subject("atlas");
|
||||||
|
wanted.set("pr1ma", "atlas", AgentState::Up).await.unwrap();
|
||||||
|
wanted.set("pr1ma", "iris", AgentState::Up).await.unwrap();
|
||||||
|
|
||||||
|
login.record(&subject, &header(Some("skipped get_loose_ends")));
|
||||||
|
hold_pending(&login, &wanted, &hives).await;
|
||||||
|
assert_eq!(declared(&wanted, "atlas").await, Some(AgentState::Paused));
|
||||||
|
assert_eq!(declared(&wanted, "iris").await, Some(AgentState::Up));
|
||||||
|
|
||||||
|
// The operator resumes; the hive has not cleared its marker yet and
|
||||||
|
// re-sends the header.
|
||||||
|
wanted.set("pr1ma", "atlas", AgentState::Up).await.unwrap();
|
||||||
|
login.record(&subject, &header(Some("skipped get_loose_ends")));
|
||||||
|
hold_pending(&login, &wanted, &hives).await;
|
||||||
|
assert_eq!(declared(&wanted, "atlas").await, Some(AgentState::Up));
|
||||||
|
|
||||||
|
// The hive clears the marker, then pauses the agent again.
|
||||||
|
login.record(&subject, &header(None));
|
||||||
|
login.record(&subject, &header(Some("skipped get_loose_ends")));
|
||||||
|
hold_pending(&login, &wanted, &hives).await;
|
||||||
|
assert_eq!(declared(&wanted, "atlas").await, Some(AgentState::Paused));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// An operator's `Offline` is not overruled by the agent's pause, and an
|
||||||
|
/// agent no hive declares leaves the pause to be tried again.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn only_an_agent_declared_up_is_paused() {
|
||||||
|
let nats = TestNats::start();
|
||||||
|
let wanted = WantedWriter::new(nats.client().await);
|
||||||
|
let hives = vec!["pr1ma".to_owned()];
|
||||||
|
let login = Reported::default();
|
||||||
|
wanted
|
||||||
|
.set("pr1ma", "atlas", AgentState::Offline)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
login.record(
|
||||||
|
&swarm_queue_client::agent_state_subject("atlas"),
|
||||||
|
&header(Some("skipped get_loose_ends")),
|
||||||
|
);
|
||||||
|
hold_pending(&login, &wanted, &hives).await;
|
||||||
|
assert_eq!(declared(&wanted, "atlas").await, Some(AgentState::Offline));
|
||||||
|
assert!(login.unheld_pauses().is_empty(), "nothing left to hold");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -471,6 +471,7 @@ pub(crate) async fn get_agents_status(
|
||||||
if let Some(login) = state.agent_login.as_ref() {
|
if let Some(login) = state.agent_login.as_ref() {
|
||||||
for row in &mut rows {
|
for row in &mut rows {
|
||||||
row.needs_login = login.get(&row.name);
|
row.needs_login = login.get(&row.name);
|
||||||
|
row.paused_by_hive = login.paused_by_hive(&row.name);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Absent on a failed read, like `wanted`: one unreadable bucket
|
// Absent on a failed read, like `wanted`: one unreadable bucket
|
||||||
|
|
@ -1816,6 +1817,7 @@ mod tests {
|
||||||
config_pr: None,
|
config_pr: None,
|
||||||
wanted: None,
|
wanted: None,
|
||||||
needs_login: None,
|
needs_login: None,
|
||||||
|
paused_by_hive: None,
|
||||||
archetype: None,
|
archetype: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -31,9 +31,9 @@ use crate::{
|
||||||
agent_permissions, agent_renewal, agent_state_stream, agent_status, agents_md, alerts,
|
agent_permissions, agent_renewal, agent_state_stream, agent_status, agents_md, alerts,
|
||||||
archetypes, auth, backend_env, builder, claude_login, config_drift, config_pr, egress_allow,
|
archetypes, auth, backend_env, builder, claude_login, config_drift, config_pr, egress_allow,
|
||||||
egress_deny, forge, forge_account, github_account, gitlab_account, inference, issue_report,
|
egress_deny, forge, forge_account, github_account, gitlab_account, inference, issue_report,
|
||||||
linked_accounts, matrix_account, message_stream, notify_flows, notify_router_api, pr_report,
|
linked_accounts, matrix_account, message_stream, notify_flows, notify_router_api, pause_hold,
|
||||||
pulls, queue_identity, review_classifier, schedules, status, subagent_term, term_stream,
|
pr_report, pulls, queue_identity, review_classifier, schedules, status, subagent_term,
|
||||||
wanted, webhook,
|
term_stream, wanted, webhook,
|
||||||
};
|
};
|
||||||
|
|
||||||
/// Where the daemon binds, overridable via `SWARM_CONTROLLER_SOCKET`.
|
/// Where the daemon binds, overridable via `SWARM_CONTROLLER_SOCKET`.
|
||||||
|
|
@ -206,11 +206,17 @@ pub(crate) fn reported_inference(
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Whether each agent last said it needs a login, read on the status
|
/// Whether each agent last said it needs a login, read on the status
|
||||||
/// reader's connection — same rationale as [`wanted_writer`].
|
/// reader's connection — same rationale as [`wanted_writer`] — and, from the
|
||||||
|
/// same headers, which agents their hive paused: those are declared `Paused`
|
||||||
|
/// (see [`pause_hold`]).
|
||||||
pub(crate) fn agent_login(
|
pub(crate) fn agent_login(
|
||||||
status: Option<&Arc<status::StatusReader>>,
|
status: Option<&Arc<status::StatusReader>>,
|
||||||
|
hives: &[HiveEntry],
|
||||||
) -> Option<Arc<agent_login::Reported>> {
|
) -> Option<Arc<agent_login::Reported>> {
|
||||||
Some(agent_login::Reported::spawn(status?.queue_client()))
|
let login = agent_login::Reported::spawn(status?.queue_client());
|
||||||
|
let hives = hives.iter().map(|h| h.name.clone()).collect();
|
||||||
|
pause_hold::spawn(Arc::clone(&login), wanted_writer(status)?, hives);
|
||||||
|
Some(login)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The homeserver agents' own matrix accounts are minted on. Unset is this
|
/// The homeserver agents' own matrix accounts are minted on. Unset is this
|
||||||
|
|
|
||||||
|
|
@ -40,17 +40,23 @@ enum Wrote {
|
||||||
/// and a caller that gets an error can ask again with fresh intent.
|
/// and a caller that gets an error can ask again with fresh intent.
|
||||||
const MAX_ATTEMPTS: usize = 5;
|
const MAX_ATTEMPTS: usize = 5;
|
||||||
|
|
||||||
|
/// A hive's current declaration; none yet reads as an empty one, an undecodable
|
||||||
|
/// value as an error.
|
||||||
|
fn decode(current: Option<&[u8]>) -> Result<HiveWanted> {
|
||||||
|
match current {
|
||||||
|
Some(raw) => serde_json::from_slice::<HiveWanted>(raw)
|
||||||
|
.context("the hive's current declaration is not decodable"),
|
||||||
|
None => Ok(HiveWanted::default()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Apply one agent's declared state to a hive's current declaration.
|
/// Apply one agent's declared state to a hive's current declaration.
|
||||||
///
|
///
|
||||||
/// `current` is the hive's whole agent map — an undecodable value is an
|
/// `current` is the hive's whole agent map — an undecodable value is an
|
||||||
/// **error**, not treated as absent, so a bad read never discards every
|
/// **error**, not treated as absent, so a bad read never discards every
|
||||||
/// other agent's declaration under a fresh-start fallback.
|
/// other agent's declaration under a fresh-start fallback.
|
||||||
fn apply(current: Option<&[u8]>, agent: &str, state: AgentState) -> Result<(HiveWanted, Vec<u8>)> {
|
fn apply(current: Option<&[u8]>, agent: &str, state: AgentState) -> Result<(HiveWanted, Vec<u8>)> {
|
||||||
let mut declaration = match current {
|
let mut declaration = decode(current)?;
|
||||||
Some(raw) => serde_json::from_slice::<HiveWanted>(raw)
|
|
||||||
.context("the hive's current declaration is not decodable")?,
|
|
||||||
None => HiveWanted::default(),
|
|
||||||
};
|
|
||||||
declaration
|
declaration
|
||||||
.agents
|
.agents
|
||||||
.insert(agent.to_owned(), AgentWanted { state });
|
.insert(agent.to_owned(), AgentWanted { state });
|
||||||
|
|
@ -98,11 +104,37 @@ impl WantedWriter {
|
||||||
|
|
||||||
/// Declare `agent` on `hive` to be in `state`, unconditionally (see the
|
/// Declare `agent` on `hive` to be in `state`, unconditionally (see the
|
||||||
/// module doc), and return the whole declaration as published.
|
/// module doc), and return the whole declaration as published.
|
||||||
|
pub async fn set(&self, hive: &str, agent: &str, state: AgentState) -> Result<HiveWanted> {
|
||||||
|
self.write(hive, agent, state, None)
|
||||||
|
.await
|
||||||
|
.map(|(declaration, _)| declaration)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Declare `agent` on `hive` `Paused` if, and only if, the declaration at
|
||||||
|
/// the moment of the write says `Up`: an operator who has since set the
|
||||||
|
/// agent offline, destroyed or already paused is not overruled. `true`
|
||||||
|
/// when the declaration now says `Paused` because of this call.
|
||||||
|
pub async fn pause_if_up(&self, hive: &str, agent: &str) -> Result<bool> {
|
||||||
|
self.write(hive, agent, AgentState::Paused, Some(AgentState::Up))
|
||||||
|
.await
|
||||||
|
.map(|(_, applied)| applied)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The write behind [`Self::set`] and [`Self::pause_if_up`]: the
|
||||||
|
/// declaration as published and whether `state` was written. With
|
||||||
|
/// `only_if`, the write happens only while the agent is declared in that
|
||||||
|
/// state.
|
||||||
///
|
///
|
||||||
/// Read-modify-write against the entry's revision, not a plain `put`:
|
/// Read-modify-write against the entry's revision, not a plain `put`:
|
||||||
/// the value is the hive's whole agent map, so a blind write would drop
|
/// the value is the hive's whole agent map, so a blind write would drop
|
||||||
/// a concurrent change to a different agent.
|
/// a concurrent change to a different agent.
|
||||||
pub async fn set(&self, hive: &str, agent: &str, state: AgentState) -> Result<HiveWanted> {
|
async fn write(
|
||||||
|
&self,
|
||||||
|
hive: &str,
|
||||||
|
agent: &str,
|
||||||
|
state: AgentState,
|
||||||
|
only_if: Option<AgentState>,
|
||||||
|
) -> Result<(HiveWanted, bool)> {
|
||||||
swarm_queue_client::ensure_connected(&self.client)?;
|
swarm_queue_client::ensure_connected(&self.client)?;
|
||||||
let store = swarm_queue_client::wanted::open_or_create(&self.client, hive).await?;
|
let store = swarm_queue_client::wanted::open_or_create(&self.client, hive).await?;
|
||||||
|
|
||||||
|
|
@ -112,8 +144,14 @@ impl WantedWriter {
|
||||||
.await
|
.await
|
||||||
.with_context(|| format!("reading the declaration for {hive}"))?;
|
.with_context(|| format!("reading the declaration for {hive}"))?;
|
||||||
let revision = entry.as_ref().map(|e| e.revision);
|
let revision = entry.as_ref().map(|e| e.revision);
|
||||||
let (declaration, encoded) =
|
let current = entry.as_ref().map(|e| e.value.as_ref());
|
||||||
apply(entry.as_ref().map(|e| e.value.as_ref()), agent, state)?;
|
if let Some(from) = only_if {
|
||||||
|
let declaration = decode(current)?;
|
||||||
|
if declaration.agents.get(agent).map(|a| a.state) != Some(from) {
|
||||||
|
return Ok((declaration, false));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let (declaration, encoded) = apply(current, agent, state)?;
|
||||||
|
|
||||||
// A redeclare of the identical state re-encodes the declaration
|
// A redeclare of the identical state re-encodes the declaration
|
||||||
// unchanged (a caller that already sees this state and asks
|
// unchanged (a caller that already sees this state and asks
|
||||||
|
|
@ -124,7 +162,7 @@ impl WantedWriter {
|
||||||
// between reads.
|
// between reads.
|
||||||
if entry.as_ref().map(|e| e.value.as_ref()) == Some(encoded.as_slice()) {
|
if entry.as_ref().map(|e| e.value.as_ref()) == Some(encoded.as_slice()) {
|
||||||
tracing::debug!(hive, agent, "declaration already current, no write needed");
|
tracing::debug!(hive, agent, "declaration already current, no write needed");
|
||||||
return Ok(declaration);
|
return Ok((declaration, true));
|
||||||
}
|
}
|
||||||
|
|
||||||
// `update` and `create` have separate error types, and only one
|
// `update` and `create` have separate error types, and only one
|
||||||
|
|
@ -149,7 +187,7 @@ impl WantedWriter {
|
||||||
match written {
|
match written {
|
||||||
Wrote::Ok => {
|
Wrote::Ok => {
|
||||||
tracing::info!(hive, agent, ?state, "declared agent state");
|
tracing::info!(hive, agent, ?state, "declared agent state");
|
||||||
return Ok(declaration);
|
return Ok((declaration, true));
|
||||||
}
|
}
|
||||||
// Re-read and re-apply onto the winner's value, not over it.
|
// Re-read and re-apply onto the winner's value, not over it.
|
||||||
Wrote::LostRace => {
|
Wrote::LostRace => {
|
||||||
|
|
|
||||||
43
swarm-wire/src/agent_pause.rs
Normal file
43
swarm-wire/src/agent_pause.rs
Normal file
|
|
@ -0,0 +1,43 @@
|
||||||
|
//! Whether the agent's own harness paused its turn loop, as the agent itself
|
||||||
|
//! reports it.
|
||||||
|
//!
|
||||||
|
//! Part of the turn-state header the agent publishes on
|
||||||
|
//! `swarm_queue_client::agent_state_subject`, beside [`crate::agent_login::Login`];
|
||||||
|
//! swarm-controller decodes it out of that header with this same type.
|
||||||
|
|
||||||
|
/// The hive-pause fields of an agent's turn-state header.
|
||||||
|
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||||
|
pub struct HivePause {
|
||||||
|
/// Why the harness paused its own turn loop; present only while that
|
||||||
|
/// pause is in force. An operator's pause (`hivectl pause`, the
|
||||||
|
/// dashboard) is not reported here, and neither is any pause by a harness
|
||||||
|
/// that predates the field, so absence says nothing about `agent_state`.
|
||||||
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
|
pub paused_by_hive: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::HivePause;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_header_without_the_field_decodes_as_not_paused_by_the_hive() {
|
||||||
|
let header = serde_json::json!({ "agent_state": "paused", "needs_login": false });
|
||||||
|
let decoded: HivePause = serde_json::from_value(header).expect("an old header decodes");
|
||||||
|
assert_eq!(decoded.paused_by_hive, None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_reason_round_trips_and_an_absent_one_is_not_sent() {
|
||||||
|
let sent = HivePause {
|
||||||
|
paused_by_hive: Some("skipped get_loose_ends".to_owned()),
|
||||||
|
};
|
||||||
|
let value = serde_json::to_value(&sent).expect("serialises");
|
||||||
|
assert_eq!(value["paused_by_hive"], "skipped get_loose_ends");
|
||||||
|
let decoded: HivePause = serde_json::from_value(value).expect("decodes");
|
||||||
|
assert_eq!(decoded, sent);
|
||||||
|
|
||||||
|
let none = serde_json::to_value(HivePause::default()).expect("serialises");
|
||||||
|
assert!(none.as_object().expect("an object").is_empty());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -11,6 +11,7 @@ pub mod agent_icon;
|
||||||
pub mod agent_inference;
|
pub mod agent_inference;
|
||||||
pub mod agent_limits;
|
pub mod agent_limits;
|
||||||
pub mod agent_login;
|
pub mod agent_login;
|
||||||
|
pub mod agent_pause;
|
||||||
pub mod agent_permissions;
|
pub mod agent_permissions;
|
||||||
pub mod agent_schedules;
|
pub mod agent_schedules;
|
||||||
pub mod agent_status;
|
pub mod agent_status;
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue