vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re stops owning scheduled prompts; agents lose the schedule tools

Browse source 
Scheduled prompts are swarm-level now (swarmctl schedule, #4936). The hive
keeps the rows it has only so the operator can copy them over.

- hive-c0re no longer fires anything: the scheduled-prompts worker is gone,
  and with it every create/edit/cancel/pause/resume/fire path (dashboard
  endpoints, agent-socket verbs, the SchedulePrompt approval kind and its
  approve handler). The store keeps open + list; GET /api/schedules stays.
- A v6 approvals migration drops schedule_prompt rows.
- Nothing submits approvals any more, so the submit path goes too:
  Approvals::submit_kind (now a test fixture), the ApprovalAdded dashboard
  event and its emitter.
- Coordinator::push_todo, whose only caller was the worker, goes, and with
  it hive_host_sock::agent_todo_socket and hive-c0re's hive-agent-sock /
  hive-sock-client deps.
- The schedules_changed dashboard event goes; nothing mutates the rows.
- Agents lose request_schedule_prompt, cancel_schedule, fire_schedule_now,
  edit_schedule and list_schedules, the `scheduling` tool group, their
  system-prompt line and their transcript formatters. `remind` is unchanged.
This commit is contained in:
atlas 2026-10-04 21:23:18 +02:00
commit c3a54bbdd0
31 changed files with 273 additions and 3124 deletions
Download patch file Download diff file Expand all files Collapse all files

46
hive-c0re/src/actions.rs
View file

@ -21,7 +21,7 @@ use crate::lifecycle;
///
/// Every queued kind resolves its approval row via
/// [`resolve_approval_dag`] when the DAG settles terminal.
pub async fn approve(coord: Arc<Coordinator>, id: i64) -> Result<()> {
pub fn approve(coord: &Arc<Coordinator>, id: i64) -> Result<()> {
let approval = coord.approvals.mark_approved(id)?;
tracing::info!(
%approval.id,
@ -50,19 +50,6 @@ pub async fn approve(coord: Arc<Coordinator>, id: i64) -> Result<()> {
coord.emit_rebuild_queue_snapshot();
Ok(())
}
ApprovalKind::SchedulePrompt => {
// No queue card for SchedulePrompt — the work is a single
// sqlite insert, the actual "running" lifetime lives on
// the scheduled-prompts surface itself (worker fires it
// at the scheduled time). Run inline + fire
// `ApprovalResolved` so the approval row leaves Pending
// immediately.
let result = run_approval_schedule_prompt(&coord, approval).await;
if result.is_ok() {
coord.emit_schedules_snapshot();
}
result
}
}
}
@ -162,37 +149,6 @@ pub(crate) async fn post_rev_deploy_failure_to_pr(agent: &str, rev: &str, err: &
}
}
/// Inline (non-queued) handler for `ApprovalKind::SchedulePrompt`.
/// On approve, decode the `SchedulePromptPayload` JSON from the
/// approval's `commit_ref`, insert a row into `scheduled_prompts`
/// (with `source = Approval { id }`), and fire `ApprovalResolved`.
/// The worker takes over from here — fan-out at fire time.
async fn run_approval_schedule_prompt(
coord: &Coordinator,
approval: hive_sh4re::approvals::Approval,
) -> Result<()> {
let result: Result<()> = async {
let payload: hive_sh4re::manager::SchedulePromptPayload =
serde_json::from_str(&approval.commit_ref)
.context("decode SchedulePromptPayload from approval.commit_ref")?;
coord
.scheduled_prompts
.submit(&crate::scheduled_prompts::NewSchedule {
owner: approval.agent.to_string(),
targets: payload.targets,
body: payload.body,
first_fire_at_unix: payload.first_fire_at_unix,
interval_seconds: payload.interval_seconds,
description: payload.description,
source: crate::scheduled_prompts::ScheduleSource::Approval { id: approval.id },
})
.map(|_| ())
.context("insert scheduled prompt")
}
.await;
finish_approval(coord, &approval, result)
}
/// Resolve an approval row from how its DAG's work ended — the body of the
/// [`NodeKind::ResolveApproval`] tail node. Every approval-carrying template
/// resolves here: the pipeline is ordinary queue nodes,

143
hive-c0re/src/coordinator.rs
View file

@ -31,9 +31,9 @@ const LAST_STOPPED_RUNNING_KEY: &str = "last_stopped_running";
pub struct Coordinator {
pub broker: Arc<Broker>,
pub approvals: Arc<Approvals>,
/// Scheduled-prompts queue. One sqlite connection,
/// internal mutex; the worker drains due rows and the manager
/// handlers insert / cancel through the same handle.
/// Scheduled-prompt rows, read-only: the dashboard lists them so the
/// operator can recreate them with `swarmctl schedule`. Nothing on
/// the hive writes or fires them.
pub scheduled_prompts: Arc<crate::scheduled_prompts::ScheduledPrompts>,
/// Full build-log capture. `lifecycle::run` /
/// `lifecycle::prebuild_toplevel` `start()` a row per attempt,
@ -426,17 +426,6 @@ pub struct ApprovalResolved<'a> {
pub description: Option<String>,
}
/// Field-named payload for [`Coordinator::emit_approval_added`].
/// Mirrors the `ApprovalAdded` dashboard-event fields. `agent`
/// borrows from the caller; `approval_kind` is a compile-time
/// constant.
pub struct ApprovalAdded<'a> {
pub id: i64,
pub agent: &'a str,
pub approval_kind: &'static str,
pub description: Option<String>,
}
/// The two ways [`Coordinator::set_paused_by_name`] can fail. See its own
/// doc comment for why this is a distinct type rather than one
/// `anyhow::Error` (`Write` already wraps one, `BadName` never needs to).
@ -587,42 +576,11 @@ impl Coordinator {
});
}
/// Emit a `SchedulesChanged` snapshot event. Called from every
/// schedule mutation site (operator API handlers + the worker
/// after each tick that fires or rearms a row) so the dashboard's
/// scheduled-prompts tab updates live without polling.
pub fn emit_schedules_snapshot(self: &Arc<Self>) {
let mut schedules: Vec<hive_sh4re::schedule::WireSchedule> =
match self.scheduled_prompts.list() {
Ok(rows) => rows
.into_iter()
.map(crate::socket_server::schedule_to_wire_public)
.collect(),
Err(e) => {
tracing::warn!(error = ?e, "emit_schedules_snapshot: list failed");
return;
}
};
// Strip ghost targets (destroyed agents) so the dashboard doesn't
// render dead columns. Best-effort: if the roster cache is
// momentarily contended we emit unfiltered rather than block this
// sync path — the next snapshot / page reload corrects it.
if let Some(live) = self.live_container_names_blocking() {
crate::socket_server::filter_ghost_schedule_targets(&mut schedules, &live);
}
self.emit_dashboard_event(DashboardEvent::SchedulesChanged {
seq: self.next_seq(),
schedules,
});
}
/// Best-effort synchronous snapshot of live container names (the
/// logical agent names from the last `nixos-container list` scan).
/// Returns `None` when the roster cache lock is momentarily
/// contended, so a transient miss is treated as "roster unknown,
/// don't filter" rather than hiding live schedule targets. The async
/// `containers_snapshot` is the reliable path for request handlers;
/// this exists for the sync `emit_schedules_snapshot` SSE emit.
/// contended. The async `containers_snapshot` is the reliable path
/// for request handlers; this exists for the sync SSE snapshot emits.
#[must_use]
pub fn live_container_names_blocking(&self) -> Option<HashSet<String>> {
self.last_containers
@ -772,24 +730,6 @@ impl Coordinator {
self.meta_updates_active.load(Ordering::SeqCst) > 0
}
/// Emit `ApprovalAdded` immediately after the row is inserted in
/// sqlite.
pub fn emit_approval_added(&self, ev: ApprovalAdded<'_>) {
let ApprovalAdded {
id,
agent,
approval_kind,
description,
} = ev;
self.emit_dashboard_event(DashboardEvent::ApprovalAdded {
seq: self.next_seq(),
id,
agent: agent.to_owned(),
approval_kind,
description,
});
}
/// Emit `ApprovalResolved` after `mark_approved` / `mark_denied` /
/// `mark_failed` lands. `resolved_at` is stamped from the system
/// clock here so call sites don't repeat the conversion; if you
@ -904,7 +844,7 @@ impl Coordinator {
.with_context(|| format!("create agent dir {}", agent_dir.display()))?;
let socket_path = Self::socket_path(name);
// Hand the full Coordinator to the per-agent socket — it needs
// more than just the broker (approvals, scheduled_prompts, ...).
// more than just the broker (approvals, ...).
let socket = socket_server::start(name, &socket_path, self.clone())?;
self.agents.lock().unwrap().insert(name.to_owned(), socket);
Ok(agent_dir)
@ -1176,77 +1116,6 @@ impl Coordinator {
.unwrap_or_else(|| hive_sh4re::manager::MANAGER_AGENT.to_owned())
}
/// Push a todo directly into `agent`'s in-container todo store — a
/// best-effort *live* dial of its `hive-agent-sock` socket
/// (`hive_host_sock::agent_todo_socket`), same `UpsertTodo` request
/// shape the built-in in-container producers (matrix/bash/forge-notify)
/// already send — `subsystem` isn't restricted to that set, any
/// producer can use the same shape. This is the migration target for
/// `HelperEvent` variants that are pure "FYI, check when convenient" notices: the
/// event stops being a broker `Message` (which always drives an
/// immediate turn) and becomes a todo instead, with the same
/// dedup-by-key semantics as any other producer.
///
/// Deliberately push, not queue: if the agent's container is down
/// (socket file absent) or the dial otherwise fails, this is a
/// silent no-op (logged at `debug`/`warn`) — no retry, no fallback
/// delivery. An agent that's down doesn't need a todo about
/// something it'll never see appear this way; whatever mechanism
/// resurfaces its state on the next boot is unrelated to this path.
///
/// Returns `Ok(())` on a successful push and `Err(reason)` — a
/// short human-readable string, already logged at `debug`/`warn`
/// by this function — on any of the silent-no-op cases below.
/// Most callers are pure fire-and-forget notices and ignore it
/// (`let _ = coord.push_todo(...).await;`); callers that track a
/// per-target delivery outcome (e.g. the scheduled-prompts worker's
/// `last_result` column) use it instead of assuming success.
/// `reopen_if_acked` forwards to `UpsertTodo` (see its doc comment) —
/// `false` for every caller here except the scheduled-prompts worker.
pub async fn push_todo(
&self,
agent: &str,
subsystem: &str,
key: Option<String>,
summary: String,
source: Option<String>,
reopen_if_acked: bool,
) -> Result<(), String> {
let Ok(ident) = hive_types::Ident::parse(agent) else {
tracing::warn!(%agent, "push_todo: not a valid agent ident, skipping");
return Err(format!("'{agent}' is not a valid agent ident"));
};
let path = hive_host_sock::agent_todo_socket(&ident);
if !path.exists() {
tracing::debug!(%agent, path = %path.display(), "push_todo: agent socket not present (offline?), skipping");
return Err(format!("agent '{agent}' socket not present (offline?)"));
}
let req = hive_agent_sock::Request::UpsertTodo {
subsystem: subsystem.to_owned(),
key,
summary,
source,
reopen_if_acked,
};
match hive_sock_client::request::<_, hive_agent_sock::Response>(
&path,
&req,
hive_sock_client::Retry::None,
)
.await
{
Ok(hive_agent_sock::Response::Err { message }) => {
tracing::warn!(%agent, %message, "push_todo: agent rejected the todo");
Err(format!("agent '{agent}' rejected the todo: {message}"))
}
Err(e) => {
tracing::warn!(%agent, error = ?e, "push_todo: dial failed");
Err(format!("push_todo dial to '{agent}' failed: {e:#}"))
}
Ok(_) => Ok(()),
}
}
/// Push a `HelperEvent` into an arbitrary agent's inbox. Encoded
/// the same way as `notify_manager` (sender = `SYSTEM_SENDER`,
/// body = JSON-encoded event) — e.g. `ContainerCrash`, `NeedsUpdate`.

2
hive-c0re/src/dashboard/approvals.rs
View file

@ -31,7 +31,7 @@ pub(super) async fn post_approve(
State(state): State<AppState>,
AxumPath(id): AxumPath<i64>,
) -> Response {
match actions::approve(state.coord.clone(), id).await {
match actions::approve(&state.coord, id) {
// 200 instead of 303 — `actions::approve` fires
// `ApprovalResolved` (success path) or the eventual failure
// event, both of which the dashboard's derived store applies

25
hive-c0re/src/dashboard/mod.rs
View file

@ -41,7 +41,7 @@ use crate::lifecycle;
(name = "meta_inputs", description = "bulk flake-input update for the meta flake"),
(name = "misc_api", description = "operator inbox, compose, spawn-request, hive stats"),
(name = "permissions", description = "tool-group + capability assignment for agents"),
(name = "schedules", description = "scheduled-prompt + rebuild-queue CRUD"),
(name = "schedules", description = "read-only scheduled-prompt list + rebuild-queue cancel"),
(name = "state_files", description = "proxied reads of allow-listed per-agent state files"),
(name = "state_snapshot", description = "cold-load dashboard snapshot"),
(name = "tombstones", description = "purge of retained state for destroyed agents"),
@ -112,8 +112,7 @@ pub async fn serve(port: u16, coord: Arc<Coordinator>) -> Result<()> {
// GET-vs-GET or POST-vs-POST pairs across different paths do. Each
// call below is therefore scoped to exactly one path — two handlers
// in the same call only when they genuinely share a path with
// different methods (`schedules::api_schedules`/`post_schedule_new`
// on `/api/schedules`) — chained via
// different methods — chained via
// repeated `.routes(...)` calls instead of one giant `routes!(...)`
// with everything in it.
let (router, api) = OpenApiRouter::<AppState>::with_openapi(ApiDoc::openapi())
@ -141,15 +140,7 @@ pub async fn serve(port: u16, coord: Arc<Coordinator>) -> Result<()> {
.routes(routes!(permissions::post_permissions))
.routes(routes!(permissions::get_stale_permissions))
.routes(routes!(permissions::delete_agent_permissions))
.routes(routes!(
schedules::api_schedules,
schedules::post_schedule_new
))
.routes(routes!(schedules::patch_schedule))
.routes(routes!(schedules::post_schedule_cancel))
.routes(routes!(schedules::post_schedule_pause))
.routes(routes!(schedules::post_schedule_resume))
.routes(routes!(schedules::post_schedule_fire_now))
.routes(routes!(schedules::api_schedules))
.routes(routes!(schedules::post_rebuild_queue_cancel))
.routes(routes!(approvals::post_approve))
.routes(routes!(approvals::post_deny))
@ -367,15 +358,7 @@ mod router_build_probe {
.routes(routes!(permissions::post_permissions))
.routes(routes!(permissions::get_stale_permissions))
.routes(routes!(permissions::delete_agent_permissions))
.routes(routes!(
schedules::api_schedules,
schedules::post_schedule_new
))
.routes(routes!(schedules::patch_schedule))
.routes(routes!(schedules::post_schedule_cancel))
.routes(routes!(schedules::post_schedule_pause))
.routes(routes!(schedules::post_schedule_resume))
.routes(routes!(schedules::post_schedule_fire_now))
.routes(routes!(schedules::api_schedules))
.routes(routes!(schedules::post_rebuild_queue_cancel))
.routes(routes!(approvals::post_approve))
.routes(routes!(approvals::post_deny))

422
hive-c0re/src/dashboard/schedules.rs
View file

@ -1,41 +1,31 @@
//! Scheduled-prompt + rebuild-queue endpoints for the dashboard.
//! Scheduled-prompt list + rebuild-queue cancel endpoints for the dashboard.
//!
//! Operator-direct schedule CRUD (`/api/schedules` GET/POST, `{id}` PATCH,
//! `{id}/cancel` + `{id}/fire-now`) — the operator click *is* the approval,
//! so these skip the manager approval gate. Also the co-located
//! `/api/schedules` is read-only: it lists the rows still in the hive's
//! `scheduled_prompts` table so the operator can recreate them with
//! `swarmctl schedule` (`docs/swarm/schedules.md`). Nothing on the hive
//! writes or fires those rows. Also the co-located
//! `/api/rebuild-queue/{id}/cancel` endpoint.
use axum::{
extract::{Path as AxumPath, State},
http::StatusCode,
response::{IntoResponse, Response},
};
use problem_details::ProblemDetails;
use crate::scheduled_prompts::ScheduleNotFoundOrCancelled;
use crate::scheduled_prompts_worker::FireNowReport;
use super::{AppState, error_problem, error_response};
#[derive(serde::Serialize, utoipa::ToSchema)]
pub(super) struct NewScheduleBody {
id: i64,
}
use super::{AppState, error_response};
#[derive(serde::Serialize, utoipa::ToSchema)]
pub(super) struct CancelResultBody {
cancelled: bool,
}
/// Snapshot of every schedule for the
/// Snapshot of every schedule row for the
/// scheduled-prompts tab.
///
/// Returns the wire shape directly so the frontend can render
/// without an extra translation layer.
// `hive_sh4re::schedule::WireSchedule` (the actual body) has no `ToSchema` — adding
// one would pull `utoipa` into the wire-types crate for a single dashboard
// endpoint. `serde_json::Value` placeholder; see the batch report.
// endpoint. `serde_json::Value` placeholder.
#[utoipa::path(
get,
path = "/api/schedules",
@ -48,13 +38,8 @@ pub(super) struct CancelResultBody {
pub(super) async fn api_schedules(State(state): State<AppState>) -> Response {
match state.coord.scheduled_prompts.list() {
Ok(rows) => {
let mut wire: Vec<hive_sh4re::schedule::WireSchedule> = rows
.into_iter()
.map(crate::socket_server::schedule_to_wire_public)
.collect();
// Drop ghost targets (agents that no longer exist) so the
// table never shows dead columns. Uses the reliable async
// roster snapshot here on the request path.
let mut wire: Vec<hive_sh4re::schedule::WireSchedule> =
rows.into_iter().map(schedule_to_wire).collect();
let live: std::collections::HashSet<String> = state
.coord
.containers_snapshot()
@ -62,117 +47,56 @@ pub(super) async fn api_schedules(State(state): State<AppState>) -> Response {
.into_iter()
.map(|c| c.name)
.collect();
crate::socket_server::filter_ghost_schedule_targets(&mut wire, &live);
filter_ghost_schedule_targets(&mut wire, &live);
axum::Json(wire).into_response()
}
Err(e) => error_response(&format!("scheduled_prompts list: {e:#}")),
}
}
/// Operator-direct schedule creation
/// (mara: "user can add them manually").
///
/// Accepts the same `SchedulePromptPayload` shape as the manager
/// request flow but skips the approval gate — the operator click
/// *is* the approval. The schedule lands directly with
/// `source = Operator` and the worker picks it up at fire time.
#[utoipa::path(
post,
path = "/api/schedules",
// `hive_sh4re::manager::SchedulePromptPayload` (the actual body) has no `ToSchema` —
// same reasoning as the `Vec<serde_json::Value>` placeholder on
// `api_schedules` above.
request_body(content = serde_json::Value, description = "SchedulePromptPayload wire shape"),
responses(
(status = 200, description = "created; body carries the new row id", body = NewScheduleBody),
(status = 400, description = "no targets, empty body, or interval_seconds == 0"),
(status = 500, description = "submit failed"),
),
tag = "schedules"
)]
pub(super) async fn post_schedule_new(
State(state): State<AppState>,
axum::Json(payload): axum::Json<hive_sh4re::manager::SchedulePromptPayload>,
) -> Result<Response, ProblemDetails> {
if payload.targets.is_empty() {
return Err(ProblemDetails::from_status_code(StatusCode::BAD_REQUEST)
.with_detail("schedule must have at least one target"));
}
if payload
.targets
.iter()
.any(|t| t == hive_sh4re::manager::OPERATOR_RECIPIENT)
{
return Err(ProblemDetails::from_status_code(StatusCode::BAD_REQUEST)
.with_detail("operator is not a valid schedule target"));
}
if payload.body.trim().is_empty() {
return Err(ProblemDetails::from_status_code(StatusCode::BAD_REQUEST)
.with_detail("schedule body must be non-empty"));
}
if let Some(0) = payload.interval_seconds {
return Err(ProblemDetails::from_status_code(StatusCode::BAD_REQUEST)
.with_detail("interval_seconds must be > 0 (use None for one-shot)"));
}
let new = crate::scheduled_prompts::NewSchedule {
owner: hive_sh4re::manager::OPERATOR_RECIPIENT.to_owned(),
targets: payload.targets,
body: payload.body,
first_fire_at_unix: payload.first_fire_at_unix,
interval_seconds: payload.interval_seconds,
description: payload.description,
source: crate::scheduled_prompts::ScheduleSource::Operator,
};
match state.coord.scheduled_prompts.submit(&new) {
Ok(id) => {
state.coord.emit_schedules_snapshot();
Ok(axum::Json(NewScheduleBody { id }).into_response())
}
Err(e) => Err(error_problem(&format!("schedule submit: {e:#}"))),
/// Drop schedule targets that point at agents which no longer exist, so
/// the schedule table doesn't render ghost columns for destroyed agents.
/// `live` is the set of logical agent names from the last
/// `nixos-container list` scan (stopped agents included, destroyed ones
/// absent). A view filter: the rows keep every target.
fn filter_ghost_schedule_targets(
schedules: &mut [hive_sh4re::schedule::WireSchedule],
live: &std::collections::HashSet<String>,
) {
for s in schedules.iter_mut() {
s.targets.retain(|t| live.contains(&t.target));
}
}
/// Optional JSON body for `fire-now`. Absent / empty body ⇒
/// `reset_timer = false` (back-compat: cadence stays intact).
#[derive(serde::Deserialize, Default, utoipa::ToSchema)]
pub(super) struct FireNowBody {
#[serde(default)]
reset_timer: bool,
}
/// Operator-initiated
/// out-of-band fire of a scheduled prompt.
///
/// Runs the per-target fan-out once immediately and reports
/// per-target outcome counts. One-shot schedules are consumed
/// (cancelled) by a manual fire — the operator's intent is "send
/// this now, the scheduled time was wrong." For recurring schedules
/// the cadence stays intact unless the body carries
/// `{"reset_timer": true}`, in which case the countdown is re-armed
/// from now (`next_fire_at = now + interval`).
#[utoipa::path(
post,
path = "/api/schedules/{id}/fire-now",
params(("id" = i64, Path, description = "schedule row id")),
request_body(content = FireNowBody, description = "optional; absent body means reset_timer = false"),
responses(
(status = 200, description = "fired; per-target outcome counts", body = FireNowReport),
(status = 500, description = "schedule missing, cancelled, or fully drained"),
),
tag = "schedules"
)]
pub(super) async fn post_schedule_fire_now(
State(state): State<AppState>,
AxumPath(id): AxumPath<i64>,
body: Option<axum::Json<FireNowBody>>,
) -> Response {
let reset_timer = body.is_some_and(|axum::Json(b)| b.reset_timer);
match crate::scheduled_prompts_worker::fire_now(&state.coord, id, reset_timer).await {
Ok(report) => {
state.coord.emit_schedules_snapshot();
axum::Json(report).into_response()
}
Err(e) => error_response(&format!("fire schedule {id} now: {e:#}")),
fn schedule_to_wire(s: crate::scheduled_prompts::Schedule) -> hive_sh4re::schedule::WireSchedule {
hive_sh4re::schedule::WireSchedule {
id: s.id,
owner: s.owner,
body: s.body,
interval_seconds: s.interval_seconds,
next_fire_at_unix: s.next_fire_at_unix,
created_at_unix: s.created_at_unix,
source: match s.source {
crate::scheduled_prompts::ScheduleSource::Operator => {
hive_sh4re::schedule::WireScheduleSource::Operator
}
crate::scheduled_prompts::ScheduleSource::Approval { id } => {
hive_sh4re::schedule::WireScheduleSource::Approval { id }
}
},
cancelled_at_unix: s.cancelled_at_unix,
paused_at_unix: s.paused_at_unix,
description: s.description,
targets: s
.targets
.into_iter()
.map(|t| hive_sh4re::schedule::WireScheduleTarget {
target: t.target,
cancelled_at_unix: t.cancelled_at_unix,
last_fired_at_unix: t.last_fired_at_unix,
last_result: t.last_result,
})
.collect(),
}
}
@ -212,213 +136,55 @@ pub(super) async fn post_rebuild_queue_cancel(
}
}
#[derive(serde::Deserialize, Default, utoipa::ToSchema)]
pub(super) struct CancelScheduleForm {
/// `None` / absent / empty array → cancel whole schedule.
#[serde(default)]
targets: Option<Vec<String>>,
}
#[cfg(test)]
mod tests {
use super::*;
#[derive(serde::Deserialize, Default, utoipa::ToSchema)]
#[allow(
clippy::option_option,
reason = "double-Option carries three-state PATCH semantics on the wire \
(missing key = leave alone, JSON null = clear, value = set); \
collapsing to a single Option would lose the 'clear' state"
)]
pub(super) struct EditScheduleForm {
#[serde(default)]
body: Option<String>,
/// Double-`Option` semantics on the wire: missing key = leave
/// alone, explicit `null` = clear, value = set. serde's
/// `deserialize_with` trick to distinguish missing from null:
/// we wrap each editable field in its own helper. Simpler
/// here — keep them plain `Option<Option<_>>` and document
/// that the dashboard caller passes JSON `null` to clear.
#[serde(default, deserialize_with = "deserialize_some")]
description: Option<Option<String>>,
#[serde(default, deserialize_with = "deserialize_some")]
interval_seconds: Option<Option<u64>>,
#[serde(default)]
next_fire_at_unix: Option<i64>,
/// New targets to add. Replace-on-conflict: re-adding a
/// previously-cancelled target drops the tombstone and the
/// target starts fresh (operator intent on re-add = "this
/// target is active again, fresh start").
#[serde(default)]
targets_add: Option<Vec<String>>,
/// Targets to cancel. Same path as `cancel_targets`:
/// tombstones preserve audit and the parent schedule
/// auto-cancels when no active targets remain.
#[serde(default)]
targets_remove: Option<Vec<String>>,
}
/// serde adaptor: turns missing-key into `None`, explicit-null
/// into `Some(None)`, value into `Some(Some(v))`. Standard trick
/// for distinguishing "field absent" from "field set to null" in
/// JSON PATCH bodies.
fn deserialize_some<'de, T, D>(deserializer: D) -> Result<Option<T>, D::Error>
where
T: serde::Deserialize<'de>,
D: serde::Deserializer<'de>,
{
T::deserialize(deserializer).map(Some)
}
/// Partial update of an existing
/// schedule.
///
/// Mutable fields: `body`, `description`, `interval_seconds`,
/// `next_fire_at_unix`, plus the target set via `targets_add` /
/// `targets_remove`. Both target lists are applied in the same
/// transaction as the scalar fields with removes-before-adds;
/// re-adding a previously-removed target resets per-target history
/// (fresh start); draining all targets auto-cancels the parent
/// schedule. JSON body uses missing-key = "leave alone", explicit
/// null = "clear" for `description` + `interval_seconds`. Cancelled
/// schedules are refused — submit a new one instead. Returns the
/// updated `WireSchedule` so the caller's post-edit refresh has the
/// new state inline.
#[utoipa::path(
patch,
path = "/api/schedules/{id}",
params(("id" = i64, Path, description = "schedule row id")),
responses(
(status = 200, description = "updated, wire shape", body = serde_json::Value),
(status = 500, description = "update failed (cancelled, not found, ...)"),
),
tag = "schedules"
)]
pub(super) async fn patch_schedule(
State(state): State<AppState>,
AxumPath(id): AxumPath<i64>,
axum::Json(form): axum::Json<EditScheduleForm>,
) -> Response {
let patch = crate::scheduled_prompts::UpdateSchedule {
body: form.body,
description: form.description,
interval_seconds: form.interval_seconds,
next_fire_at_unix: form.next_fire_at_unix,
targets_add: form.targets_add,
targets_remove: form.targets_remove,
};
if let Err(e) = state.coord.scheduled_prompts.update(id, patch) {
return error_response(&format!("edit schedule {id}: {e:#}"));
}
match state.coord.scheduled_prompts.get(id) {
Ok(Some(s)) => {
let wire = crate::socket_server::schedule_to_wire_public(s);
state.coord.emit_schedules_snapshot();
axum::Json(wire).into_response()
}
Ok(None) => error_response(&format!("edit schedule {id}: row vanished post-update")),
Err(e) => error_response(&format!("re-read schedule {id}: {e:#}")),
}
}
/// Pause a schedule so the worker
/// skips it until explicitly resumed.
///
/// Idempotent; no-op on an already-paused row. Returns 404 when the
/// schedule is cancelled or not found.
#[utoipa::path(
post,
path = "/api/schedules/{id}/pause",
params(("id" = i64, Path, description = "schedule row id")),
responses(
(status = 200, description = "paused", body = String),
(status = 404, description = "schedule cancelled or not found"),
(status = 500, description = "pause failed"),
),
tag = "schedules"
)]
pub(super) async fn post_schedule_pause(
State(state): State<AppState>,
AxumPath(id): AxumPath<i64>,
) -> Response {
match state.coord.scheduled_prompts.pause(id) {
Ok(()) => {
state.coord.emit_schedules_snapshot();
(StatusCode::OK, "ok").into_response()
}
Err(e) => {
if e.downcast_ref::<ScheduleNotFoundOrCancelled>().is_some() {
(StatusCode::NOT_FOUND, format!("{e}")).into_response()
} else {
error_response(&format!("pause schedule {id}: {e:#}"))
}
fn target(name: &str) -> hive_sh4re::schedule::WireScheduleTarget {
hive_sh4re::schedule::WireScheduleTarget {
target: name.to_owned(),
cancelled_at_unix: None,
last_fired_at_unix: None,
last_result: None,
}
}
}
/// Resume a paused schedule.
///
/// Idempotent; no-op on an already-active row. Returns 404 when the
/// schedule is cancelled or not found.
#[utoipa::path(
post,
path = "/api/schedules/{id}/resume",
params(("id" = i64, Path, description = "schedule row id")),
responses(
(status = 200, description = "resumed", body = String),
(status = 404, description = "schedule cancelled or not found"),
(status = 500, description = "resume failed"),
),
tag = "schedules"
)]
pub(super) async fn post_schedule_resume(
State(state): State<AppState>,
AxumPath(id): AxumPath<i64>,
) -> Response {
match state.coord.scheduled_prompts.resume(id) {
Ok(()) => {
state.coord.emit_schedules_snapshot();
(StatusCode::OK, "ok").into_response()
}
Err(e) => {
if e.downcast_ref::<ScheduleNotFoundOrCancelled>().is_some() {
(StatusCode::NOT_FOUND, format!("{e}")).into_response()
} else {
error_response(&format!("resume schedule {id}: {e:#}"))
}
fn schedule(targets: &[&str]) -> hive_sh4re::schedule::WireSchedule {
hive_sh4re::schedule::WireSchedule {
id: 1,
owner: "operator".to_owned(),
body: "ping".to_owned(),
interval_seconds: None,
next_fire_at_unix: hive_sh4re::wire_time::from_secs(0),
created_at_unix: hive_sh4re::wire_time::from_secs(0),
source: hive_sh4re::schedule::WireScheduleSource::Operator,
cancelled_at_unix: None,
paused_at_unix: None,
description: None,
targets: targets.iter().map(|t| target(t)).collect(),
}
}
}
/// Operator-side cancel
/// (whole schedule when no `targets` field, partial when one is
/// provided).
///
/// Operator bypasses the topology check; the manager surface
/// enforces it for agent callers.
#[utoipa::path(
post,
path = "/api/schedules/{id}/cancel",
params(("id" = i64, Path, description = "schedule row id")),
responses(
(status = 200, description = "cancelled (whole or partial)", body = String),
(status = 500, description = "cancel failed"),
),
tag = "schedules"
)]
pub(super) async fn post_schedule_cancel(
State(state): State<AppState>,
AxumPath(id): AxumPath<i64>,
body: Option<axum::Json<CancelScheduleForm>>,
) -> Response {
let targets = body
.and_then(|axum::Json(b)| b.targets)
.filter(|t| !t.is_empty());
let result = match targets.as_deref() {
Some(list) => state.coord.scheduled_prompts.cancel_targets(id, list),
None => state.coord.scheduled_prompts.cancel_all(id),
};
match result {
Ok(()) => {
state.coord.emit_schedules_snapshot();
(StatusCode::OK, "ok").into_response()
}
Err(e) => error_response(&format!("cancel schedule {id}: {e:#}")),
#[test]
fn ghost_filter_drops_dead_agents_keeps_live() {
let live: std::collections::HashSet<String> = ["iris".to_owned(), "damocles".to_owned()]
.into_iter()
.collect();
let mut schedules = vec![schedule(&["iris", "ghost", "damocles"])];
filter_ghost_schedule_targets(&mut schedules, &live);
let kept: Vec<&str> = schedules[0]
.targets
.iter()
.map(|t| t.target.as_str())
.collect();
assert_eq!(kept, vec!["iris", "damocles"]);
}
#[test]
fn ghost_filter_can_empty_targets_when_all_dead() {
let live: std::collections::HashSet<String> = std::collections::HashSet::new();
let mut schedules = vec![schedule(&["gone1", "gone2"])];
filter_ghost_schedule_targets(&mut schedules, &live);
assert!(schedules[0].targets.is_empty());
}
}

3
hive-c0re/src/dashboard/state_snapshot.rs
View file

@ -152,8 +152,7 @@ struct ApprovalView {
#[serde(skip_serializing_if = "Option::is_none")]
description: Option<String>,
/// Raw `commit_ref` payload for `UpdateMetaInputs` (JSON-encoded
/// `Vec<String>` of input names; `"[]"` = all inputs) and
/// `SchedulePrompt` (JSON-encoded `SchedulePromptPayload`). The
/// `Vec<String>` of input names; `"[]"` = all inputs). The
/// frontend parses this to render a human-readable card body.
commit_ref: String,
/// RFC 3339 UTC time the approval was queued. Rendered as a

46
hive-c0re/src/dashboard_events.rs
View file

@ -48,24 +48,14 @@ pub enum DashboardEvent {
#[serde(default, skip_serializing_if = "Vec::is_empty")]
file_refs: Vec<String>,
},
/// A new approval landed in the pending queue. Payload carries
/// enough to render the dashboard row without a `/api/state`
/// refetch.
///
/// The approval's own kind (`"schedule_prompt"` / `"update_meta_inputs"`) lives
/// on `approval_kind` rather than `kind` because the latter is taken
/// by the serde tag identifying which `DashboardEvent` variant
/// this is.
ApprovalAdded {
seq: u64,
id: i64,
agent: String,
approval_kind: &'static str,
description: Option<String>,
},
/// A pending approval transitioned to a terminal state
/// (approved / denied / failed). Clients move the row out of the
/// pending list and into history.
///
/// The approval's own kind (`"update_meta_inputs"`) lives
/// on `approval_kind` rather than `kind` because the latter is taken
/// by the serde tag identifying which `DashboardEvent` variant
/// this is.
ApprovalResolved {
seq: u64,
id: i64,
@ -170,17 +160,6 @@ pub enum DashboardEvent {
/// (transients, container state, approvals), and a poll loop would be
/// slower for the same information.
RebuildQueueChanged { seq: u64 },
/// Full snapshot of all scheduled prompts. Emitted after every
/// operator mutation (new / edit / cancel / fire-now) and after the
/// worker fires or rearms a row. Same snapshot-shape rationale as
/// `TombstonesChanged` / `MetaInputsChanged` — the list is small and the
/// client's
/// per-target `last_result` / `last_fired_at_unix` fields are most
/// naturally re-derived from the full list.
SchedulesChanged {
seq: u64,
schedules: Vec<hive_sh4re::schedule::WireSchedule>,
},
/// Full snapshot of capability grants (per-agent `Vec<cap_name>`).
/// Emitted from the rebuild-queue worker after a `PermChange`
/// `Capabilities` entry commits the JSON file. Lets the P3RM1SS10NS
@ -235,7 +214,6 @@ impl DashboardEvent {
match self {
DashboardEvent::Sent { .. } => "sent",
DashboardEvent::Delivered { .. } => "delivered",
DashboardEvent::ApprovalAdded { .. } => "approval_added",
DashboardEvent::ApprovalResolved { .. } => "approval_resolved",
DashboardEvent::TransientSet { .. } => "transient_set",
DashboardEvent::TransientCleared { .. } => "transient_cleared",
@ -245,7 +223,6 @@ impl DashboardEvent {
DashboardEvent::MetaInputsChanged { .. } => "meta_inputs_changed",
DashboardEvent::MetaUpdateRunning { .. } => "meta_update_running",
DashboardEvent::RebuildQueueChanged { .. } => "rebuild_queue_changed",
DashboardEvent::SchedulesChanged { .. } => "schedules_changed",
DashboardEvent::CapabilitiesChanged { .. } => "capabilities_changed",
DashboardEvent::ToolGroupsChanged { .. } => "tool_groups_changed",
}
@ -294,18 +271,11 @@ mod tests {
in_reply_to: None,
file_refs: Vec::new(),
},
DashboardEvent::ApprovalAdded {
seq: 1,
id: 1,
agent: "x".into(),
approval_kind: "schedule_prompt",
description: None,
},
DashboardEvent::ApprovalResolved {
seq: 1,
id: 1,
agent: "x".into(),
approval_kind: "schedule_prompt",
approval_kind: "update_meta_inputs",
status: "approved",
resolved_at: hive_sh4re::wire_time::from_secs(0),
note: None,
@ -339,10 +309,6 @@ mod tests {
running: false,
},
DashboardEvent::RebuildQueueChanged { seq: 1 },
DashboardEvent::SchedulesChanged {
seq: 1,
schedules: Vec::new(),
},
DashboardEvent::CapabilitiesChanged {
seq: 1,
caps: Vec::new(),

4
hive-c0re/src/job_queue/exec.rs
View file

@ -383,10 +383,6 @@ async fn run_destroy_bookkeeping(coord: &Arc<Coordinator>, agent: &str, purge: b
// (gained one on destroy, lost one on purge — recompute either way).
coord.rescan_containers_and_emit().await;
crate::dashboard::emit_tombstones_snapshot(coord).await;
// Re-emit the schedules snapshot: the rescan above refreshed the live
// roster, so any schedule that still targets the just-destroyed agent now
// drops that ghost column live (no page reload needed).
coord.emit_schedules_snapshot();
}
/// Emit this agent's rebuild-complete todo. `ok` is not computed — it is which

10
hive-c0re/src/main.rs
View file

@ -45,9 +45,7 @@ mod workers;
pub(crate) use agent_config::{capabilities, limits, resource_limits, tool_groups, topology};
pub(crate) use stats::{container_stats, hive_stats, host_stats, otel_metrics, warnings};
pub(crate) use stores::{approvals, broker, build_logs, db, power, scheduled_prompts};
pub(crate) use workers::{
agent_sockets, auto_update, crash_watch, knowledge, mcp_sockets, scheduled_prompts_worker,
};
pub(crate) use workers::{agent_sockets, auto_update, crash_watch, knowledge, mcp_sockets};
use coordinator::{Coordinator, HiveEnv, ServeConfig};
@ -240,7 +238,7 @@ fn submit_periodic_sweep(
}
/// Start the coordinator daemon: open the broker, run migrations, spawn
/// background tasks (auto-update, vacuums, crash-watcher, scheduled-prompts,
/// background tasks (auto-update, vacuums, crash-watcher,
/// dashboard), then serve the admin socket until a signal arrives.
#[allow(
clippy::too_many_lines,
@ -433,10 +431,6 @@ async fn cmd_serve(
// run_reconcile call register_agent on start; kill/destroy call
// unregister_agent. No recurring poll needed — c0re owns the listeners.
mcp_sockets::sync_on_start(coord.clone()).await;
// Scheduled-prompts worker: drains due scheduled_prompts rows
// and fans the body out to each active target's inbox. See
// scheduled_prompts_worker.rs.
scheduled_prompts_worker::spawn(coord.clone());
// Job-queue scheduler: drives the global DAG queue (rebuild /
// meta-update / spawn / power ops). Concurrency comes from the
// build-slot count + per-agent leases inside the queue, not from

2
hive-c0re/src/server.rs
View file

@ -186,7 +186,7 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
HostRequest::Urls => HostResponse::urls(hive_urls()),
HostRequest::Pending => HostResponse::pending(coord.approvals.pending()?),
HostRequest::Approve { id } => {
actions::approve(coord.clone(), *id).await?;
actions::approve(&coord, *id)?;
HostResponse::success()
}
HostRequest::Deny { id } => {

155
hive-c0re/src/socket_server/mod.rs
View file

@ -3,8 +3,8 @@
//! authenticates the caller: connecting to `<.../agents/foo/mcp.sock>` means
//! you are `foo`; the manager socket simply serves as `ruth`. There is no
//! privilege flag — both transports run the same [`serve`] / [`dispatch`]
//! code, and authority derives uniformly from the caller's identity:
//! tool-group membership for the orchestration verbs. An agent-targeting
//! code, and authority derives uniformly from the caller's identity
//! (tool-group membership for cancelling an approval). An agent-targeting
//! verb may name any agent, so `ruth` reaches every agent exactly the way
//! every other agent does, not via any hardcoded name match.
@ -21,16 +21,6 @@ use tokio::task::JoinHandle;
use crate::coordinator::Coordinator;
mod schedules;
pub(crate) use schedules::filter_ghost_schedule_targets;
pub use schedules::schedule_to_wire_public;
use schedules::{
EditSchedulePatch, handle_cancel_schedule, handle_edit_schedule, handle_fire_schedule_now,
handle_list_schedules, handle_request_schedule_prompt,
};
pub struct AgentSocket {
pub path: PathBuf,
pub handle: JoinHandle<()>,
@ -275,21 +265,15 @@ pub(crate) fn recv_timeout(wait_seconds: Option<u64>) -> std::time::Duration {
}
}
/// Handle the subset of `Request` variants that are identical on both
/// the agent socket and the manager socket. Returns `Some(response)` for
/// every variant it handles; returns `None` for the remaining variants —
/// `GetLooseEnds` and the orchestration verbs (schedules), which need
/// per-verb tool-group gating — or for host-admin / unknown requests
/// invalid on either socket.
///
/// The unified `dispatch` calls this first; the remaining arms (which gate
/// on topology / capabilities / tool-groups) are handled there.
pub(crate) async fn dispatch_shared(
/// Unified dispatch for every socket connection — per-agent sockets and the
/// (now pure-transport) manager socket alike. There is no privilege bit;
/// authority derives uniformly from the caller's identity.
async fn dispatch(
req: &hive_core_agent_sock::Request,
agent: &str,
coord: &Arc<Coordinator>,
) -> Option<hive_core_agent_sock::Response> {
Some(match req {
) -> hive_core_agent_sock::Response {
match req {
hive_core_agent_sock::Request::Send {
to,
body,
@ -331,9 +315,8 @@ pub(crate) async fn dispatch_shared(
coord.clear_pause_pending(agent);
hive_core_agent_sock::Response::Ok
}
// Not a shared variant.
_ => return None,
})
hive_core_agent_sock::Request::GetLooseEnds => handle_get_loose_ends(coord, agent),
}
}
/// `Recv` — long-poll the broker for up to `max` messages (capped at
@ -572,104 +555,6 @@ fn handle_requeue_inflight(
}
}
/// Unified dispatch for every socket connection — per-agent sockets and the
/// (now pure-transport) manager socket alike. There is no privilege bit;
/// authority derives uniformly from the caller's identity: hive-wide
/// orchestration verbs (schedules) require the matching tool-group (the
/// grantable capability).
async fn dispatch(req: &Request, agent: &str, coord: &Arc<Coordinator>) -> Response {
if let Some(resp) = dispatch_shared(req, agent, coord).await {
return resp;
}
match req {
Request::GetLooseEnds => handle_get_loose_ends(coord, agent),
// Orchestration verbs — gated per-verb on tool-group membership
// (see `dispatch_orchestration`).
_ => dispatch_orchestration(req, agent, coord).await,
}
}
/// Handle the hive-wide orchestration verbs (scheduling). No blanket socket
/// gate: each verb gates on the grantable capability that authorises it —
/// the matching tool-group (`scheduling`). Any other variant is a
/// host-admin / unknown request invalid on either socket.
async fn dispatch_orchestration(req: &Request, agent: &str, coord: &Arc<Coordinator>) -> Response {
match req {
Request::RequestSchedulePrompt(payload) => {
if let Some(err) = require_group(agent, "scheduling", "schedule a prompt") {
return err;
}
handle_request_schedule_prompt(coord, agent, payload)
}
Request::CancelSchedule { id, targets } => {
if let Some(err) = require_group(agent, "scheduling", "cancel a schedule") {
return err;
}
handle_cancel_schedule(coord, agent, *id, targets.as_deref())
}
Request::EditSchedule {
id,
body,
description,
interval_seconds,
next_fire_at_unix,
targets_add,
targets_remove,
} => {
if let Some(err) = require_group(agent, "scheduling", "edit a schedule") {
return err;
}
handle_edit_schedule(
coord,
agent,
*id,
EditSchedulePatch {
body: body.clone(),
description: description.clone(),
interval_seconds: *interval_seconds,
next_fire_at_unix: *next_fire_at_unix,
targets_add: targets_add.clone(),
targets_remove: targets_remove.clone(),
},
)
}
Request::ListSchedules => {
if let Some(err) = require_group(agent, "scheduling", "list schedules") {
return err;
}
handle_list_schedules(coord, agent)
}
Request::FireScheduleNow { id } => {
if let Some(err) = require_group(agent, "scheduling", "fire a schedule") {
return err;
}
handle_fire_schedule_now(coord, agent, *id).await
}
// Host-admin-only / unknown variants: never valid on either socket.
_ => Response::Err {
message: "request not handled on this socket".to_owned(),
},
}
}
/// Capability guard for the hive-wide orchestration verbs: the caller must
/// hold the given tool-group. The tool-group (c0re-owned `tool_groups.json`,
/// read server-side via [`crate::tool_groups::groups_for`]) is the grantable
/// capability — granting it to an orchestrator (e.g. the root) authorises
/// these verbs without any positional/hardcoded privilege. `action` is the
/// verb phrase for the message.
fn require_group(agent: &str, group: &str, action: &str) -> Option<Response> {
match crate::tool_groups::groups_for(agent) {
Ok(groups) if groups.iter().any(|g| g == group) => None,
Ok(_) => Some(Response::Err {
message: format!("agent `{agent}` cannot {action}: requires the `{group}` tool group"),
}),
Err(e) => Some(Response::Err {
message: format!("agent `{agent}` cannot {action}: {e}"),
}),
}
}
/// `GetLooseEnds` — always the caller's own loose ends.
fn handle_get_loose_ends(coord: &Arc<Coordinator>, agent: &str) -> Response {
match crate::loose_ends::for_agent(coord, agent) {
@ -767,10 +652,9 @@ fn check_can_cancel_approval(canceller: &str) -> Result<(), String> {
}
}
/// Common Send handler shared between dispatch arms. Applies the
/// `Send` handler. Applies the
/// 4 KiB body cap, then routes broadcast (`to == "*"`) / unicast through
/// their respective broker calls.
/// `pub(crate)` so `dispatch_shared` can use it across both socket paths.
pub(crate) fn handle_send(
coord: &Arc<Coordinator>,
agent: &str,
@ -837,6 +721,21 @@ pub(crate) fn handle_send(
mod tests {
use super::*;
/// A real `Coordinator` over a throwaway sqlite dir. `serve` takes an
/// `Arc<Coordinator>`, so there is no lighter way in; `open` touches
/// nothing outside the db path it is handed.
fn coordinator() -> (tempfile::TempDir, Arc<Coordinator>) {
let dir = tempfile::tempdir().expect("tempdir");
let coord = Coordinator::open(
&dir.path().join("broker.sqlite"),
crate::coordinator::HiveEnv::default(),
crate::hive_stats::PriceTable::new(),
1,
)
.expect("open coordinator");
(dir, Arc::new(coord))
}
#[test]
fn approval_cancel_rejects_callers_without_the_approvals_group() {
// A caller that doesn't hold the `approvals` tool-group must not be
@ -869,7 +768,7 @@ mod tests {
#[tokio::test]
async fn serve_answers_an_over_long_line_with_an_error_and_closes() {
let (_dir, coord) = schedules::tests::coordinator();
let (_dir, coord) = coordinator();
let (client, server) = UnixStream::pair().expect("socketpair");
let handler = tokio::spawn(serve(server, "iris".to_owned(), coord));
let (read, mut write) = client.into_split();

485
hive-c0re/src/socket_server/schedules.rs
View file

@ -1,485 +0,0 @@
//! Scheduled-prompt request handlers (`ListSchedules` /
//! `RequestSchedulePrompt` / `CancelSchedule` / `EditSchedule` /
//! `FireScheduleNow`) and the
//! schedule-to-wire mapping reused by the dashboard
//! (`schedule_to_wire_public` / `filter_ghost_schedule_targets`).
use std::sync::Arc;
use hive_core_agent_sock::Response;
use crate::coordinator::Coordinator;
/// `ListSchedules` — snapshot every scheduled prompt. Ownership no longer
/// narrows the view: `CancelSchedule`/`EditSchedule`/`FireScheduleNow` accept
/// any requester, so filtering the listing would only hide rows the requester
/// can act on anyway.
pub(super) fn handle_list_schedules(coord: &Arc<Coordinator>, requester: &str) -> Response {
tracing::debug!(%requester, "list schedules");
match coord.scheduled_prompts.list() {
Ok(schedules) => Response::Schedules {
schedules: schedules.into_iter().map(schedule_to_wire).collect(),
},
Err(e) => Response::Err {
message: format!("list scheduled prompts: {e:#}"),
},
}
}
/// Submit a `RequestSchedulePrompt` payload as an `ApprovalKind::SchedulePrompt`
/// row. Encodes the payload into the approval's `commit_ref` so the
/// approve handler can re-parse it without a side table. Validates
/// inputs (non-empty targets, non-empty body, sane interval) at
/// submit time — the operator should never see a malformed schedule
/// pending approval.
pub(super) fn handle_request_schedule_prompt(
coord: &Arc<Coordinator>,
requester: &str,
payload: &hive_sh4re::manager::SchedulePromptPayload,
) -> Response {
if payload.targets.is_empty() {
return Response::Err {
message: "schedule must have at least one target".into(),
};
}
if payload
.targets
.iter()
.any(|t| t == hive_sh4re::manager::OPERATOR_RECIPIENT)
{
return Response::Err {
message: "operator is not a valid schedule target".into(),
};
}
if payload.body.trim().is_empty() {
return Response::Err {
message: "schedule body must be non-empty".into(),
};
}
if let Some(0) = payload.interval_seconds {
return Response::Err {
message: "interval_seconds must be > 0 (use None for one-shot)".into(),
};
}
let commit_ref = match serde_json::to_string(payload) {
Ok(s) => s,
Err(e) => {
return Response::Err {
message: format!("encode SchedulePromptPayload: {e:#}"),
};
}
};
let id = match coord.approvals.submit_kind(
requester,
hive_sh4re::approvals::ApprovalKind::SchedulePrompt,
&commit_ref,
payload.description.as_deref(),
requester,
) {
Ok(id) => id,
Err(e) => {
return Response::Err {
message: format!("queue schedule_prompt approval: {e:#}"),
};
}
};
tracing::info!(
%id,
requester,
targets = ?payload.targets,
first_fire_at = payload.first_fire_at_unix,
interval = ?payload.interval_seconds,
"schedule_prompt approval queued"
);
coord.emit_approval_added(crate::coordinator::ApprovalAdded {
id,
agent: requester,
approval_kind: "schedule_prompt",
description: payload.description.clone(),
});
Response::Ok
}
/// Cancel a schedule (whole or per-target). Any requester may cancel any
/// schedule, whoever owns it.
pub(super) fn handle_cancel_schedule(
coord: &Arc<Coordinator>,
requester: &str,
schedule_id: i64,
targets: Option<&[String]>,
) -> Response {
// Existence check only — it buys the caller a clean "not found" instead
// of whatever the cancel itself would say about an absent row.
if let Some(err) = require_schedule(coord, schedule_id) {
return err;
}
tracing::info!(%requester, %schedule_id, "cancel schedule");
let result = match targets {
Some(list) if !list.is_empty() => coord
.scheduled_prompts
.cancel_targets(schedule_id, list)
.map_err(|e| format!("cancel targets: {e:#}")),
_ => coord
.scheduled_prompts
.cancel_all(schedule_id)
.map_err(|e| format!("cancel all: {e:#}")),
};
match result {
Ok(()) => {
coord.emit_schedules_snapshot();
Response::Ok
}
Err(message) => Response::Err { message },
}
}
/// Dispatch a `FireScheduleNow` request. Same rule as `CancelSchedule`: any
/// requester, any schedule. The actual fan-out lives in
/// `scheduled_prompts_worker::fire_now`.
pub(super) async fn handle_fire_schedule_now(
coord: &Arc<Coordinator>,
requester: &str,
schedule_id: i64,
) -> Response {
if let Some(err) = require_schedule(coord, schedule_id) {
return err;
}
tracing::info!(%requester, %schedule_id, "fire schedule now");
// MCP fire_schedule_now stays no-reset (cadence intact); the
// reset-timer option is a dashboard-dialog affordance.
match crate::scheduled_prompts_worker::fire_now(coord, schedule_id, false).await {
Ok(_report) => {
coord.emit_schedules_snapshot();
Response::Ok
}
Err(e) => Response::Err {
message: format!("fire schedule {schedule_id} now: {e:#}"),
},
}
}
/// Field-named PATCH payload for [`handle_edit_schedule`]. Every
/// field is "leave alone" when `None`; the double-`Option` fields
/// additionally distinguish clear (`Some(None)`) from set
/// (`Some(Some(v))`).
#[allow(
clippy::option_option,
reason = "double-Option carries three-state PATCH semantics: outer None = \
leave alone, Some(None) = clear, Some(Some(v)) = set"
)]
pub(super) struct EditSchedulePatch {
pub(super) body: Option<String>,
pub(super) description: Option<Option<String>>,
pub(super) interval_seconds: Option<Option<u64>>,
pub(super) next_fire_at_unix: Option<i64>,
pub(super) targets_add: Option<Vec<String>>,
pub(super) targets_remove: Option<Vec<String>>,
}
/// Dispatch an `EditSchedule` patch. Same rule as `CancelSchedule`: any
/// requester, any schedule. Forwards the partial payload to
/// `ScheduledPrompts::update` which enforces the cancelled-row /
/// zero-interval validation. Returns `Ok` on a clean update;
/// `Err` with the underlying message on any validation
/// failure so the dashboard can surface it verbatim.
pub(super) fn handle_edit_schedule(
coord: &Arc<Coordinator>,
requester: &str,
schedule_id: i64,
patch: EditSchedulePatch,
) -> Response {
let EditSchedulePatch {
body,
description,
interval_seconds,
next_fire_at_unix,
targets_add,
targets_remove,
} = patch;
if let Some(err) = require_schedule(coord, schedule_id) {
return err;
}
tracing::info!(%requester, %schedule_id, "edit schedule");
let patch = crate::scheduled_prompts::UpdateSchedule {
body,
description,
interval_seconds,
next_fire_at_unix,
targets_add,
targets_remove,
};
match coord.scheduled_prompts.update(schedule_id, patch) {
Ok(()) => {
coord.emit_schedules_snapshot();
Response::Ok
}
Err(e) => Response::Err {
message: format!("edit schedule {schedule_id}: {e:#}"),
},
}
}
/// Shared existence guard for the three schedule-mutating verbs: `None` when
/// the row is there, `Some(Err)` to short-circuit when it isn't or the read
/// failed. Not a permission check — any requester may act on any schedule.
fn require_schedule(coord: &Arc<Coordinator>, schedule_id: i64) -> Option<Response> {
match coord.scheduled_prompts.get(schedule_id) {
Ok(Some(_)) => None,
Ok(None) => Some(Response::Err {
message: format!("schedule {schedule_id} not found"),
}),
Err(e) => Some(Response::Err {
message: format!("read schedule {schedule_id}: {e:#}"),
}),
}
}
/// Map a `scheduled_prompts::Schedule` to its public wire shape.
/// Field-by-field copy — the two types are intentionally identical;
/// the separation keeps hive-sh4re free of hive-c0re-internal types.
/// Public alias `schedule_to_wire_public` re-exports for
/// `dashboard.rs::api_schedules` without crossing the module
/// boundary into the socket-server file.
pub fn schedule_to_wire_public(
s: crate::scheduled_prompts::Schedule,
) -> hive_sh4re::schedule::WireSchedule {
schedule_to_wire(s)
}
/// Drop schedule targets that point at agents which no longer exist, so
/// the dashboard's schedule table doesn't render ghost columns for
/// destroyed agents. `live` is the set of logical agent names from the
/// last `nixos-container list` scan (stopped agents included, destroyed
/// ones absent). Applied only to the dashboard wire paths
/// (`api_schedules` + the `SchedulesChanged` SSE emit) — `list_schedules`
/// doesn't apply this filter (it still returns every live target, ghost
/// or not, on the schedules the requester is authorized to see) so an
/// agent can still see and cancel its own stale targets. This is a view
/// filter: the underlying schedule rows keep every target, so a
/// re-spawned agent's targets reappear on their own.
pub(crate) fn filter_ghost_schedule_targets(
schedules: &mut [hive_sh4re::schedule::WireSchedule],
live: &std::collections::HashSet<String>,
) {
for s in schedules.iter_mut() {
s.targets.retain(|t| live.contains(&t.target));
}
}
fn schedule_to_wire(s: crate::scheduled_prompts::Schedule) -> hive_sh4re::schedule::WireSchedule {
hive_sh4re::schedule::WireSchedule {
id: s.id,
owner: s.owner,
body: s.body,
interval_seconds: s.interval_seconds,
next_fire_at_unix: s.next_fire_at_unix,
created_at_unix: s.created_at_unix,
source: match s.source {
crate::scheduled_prompts::ScheduleSource::Operator => {
hive_sh4re::schedule::WireScheduleSource::Operator
}
crate::scheduled_prompts::ScheduleSource::Approval { id } => {
hive_sh4re::schedule::WireScheduleSource::Approval { id }
}
},
cancelled_at_unix: s.cancelled_at_unix,
paused_at_unix: s.paused_at_unix,
description: s.description,
targets: s
.targets
.into_iter()
.map(|t| hive_sh4re::schedule::WireScheduleTarget {
target: t.target,
cancelled_at_unix: t.cancelled_at_unix,
last_fired_at_unix: t.last_fired_at_unix,
last_result: t.last_result,
})
.collect(),
}
}
#[cfg(test)]
pub(super) mod tests {
use super::*;
fn target(name: &str) -> hive_sh4re::schedule::WireScheduleTarget {
hive_sh4re::schedule::WireScheduleTarget {
target: name.to_owned(),
cancelled_at_unix: None,
last_fired_at_unix: None,
last_result: None,
}
}
fn schedule(targets: &[&str]) -> hive_sh4re::schedule::WireSchedule {
hive_sh4re::schedule::WireSchedule {
id: 1,
owner: "operator".to_owned(),
body: "ping".to_owned(),
interval_seconds: None,
next_fire_at_unix: hive_sh4re::wire_time::from_secs(0),
created_at_unix: hive_sh4re::wire_time::from_secs(0),
source: hive_sh4re::schedule::WireScheduleSource::Operator,
cancelled_at_unix: None,
paused_at_unix: None,
description: None,
targets: targets.iter().map(|t| target(t)).collect(),
}
}
#[test]
fn ghost_filter_drops_dead_agents_keeps_live() {
let live: std::collections::HashSet<String> = ["iris".to_owned(), "damocles".to_owned()]
.into_iter()
.collect();
let mut schedules = vec![schedule(&["iris", "ghost", "damocles"])];
filter_ghost_schedule_targets(&mut schedules, &live);
let kept: Vec<&str> = schedules[0]
.targets
.iter()
.map(|t| t.target.as_str())
.collect();
// `ghost` (destroyed) dropped; live agents kept.
assert_eq!(kept, vec!["iris", "damocles"]);
}
#[test]
fn ghost_filter_can_empty_targets_when_all_dead() {
let live: std::collections::HashSet<String> = std::collections::HashSet::new();
let mut schedules = vec![schedule(&["gone1", "gone2"])];
filter_ghost_schedule_targets(&mut schedules, &live);
assert!(schedules[0].targets.is_empty());
}
// -----------------------------------------------------------------------
// The scheduling surface permits an unrelated requester
//
// These pin the *decision* the deleted `schedule_authorized` used to make,
// inverted: a requester with no relation to a schedule's owner reaches the
// handler's effect instead of a refusal. They live at the handler level
// because the predicate, its pure `_in` form and all three wrappers are
// gone — the handler is the lowest layer where the decision still exists
// to be observed.
//
// `OWNER` and `STRANGER` are names no hive has, so no `topology.json` on
// any machine running these tests can relate them. Any ancestry check
// reintroduced between the two — disk-reading or pure — therefore answers
// "unrelated", and every assertion below flips from the handler's effect
// to a refusal.
// -----------------------------------------------------------------------
/// Owner of the schedule under test.
const OWNER: &str = "schedule-owner-with-no-relations";
/// Requester acting on it: not the owner, not the operator, and not
/// anybody's ancestor or descendant.
const STRANGER: &str = "requester-with-no-relations";
/// A real `Coordinator` over a throwaway sqlite dir. The socket-server
/// handlers take `&Arc<Coordinator>`, so there is no lighter way in;
/// `open` touches nothing outside the db path it is handed.
pub(crate) fn coordinator() -> (tempfile::TempDir, Arc<Coordinator>) {
let dir = tempfile::tempdir().expect("tempdir");
let coord = Coordinator::open(
&dir.path().join("broker.sqlite"),
crate::coordinator::HiveEnv::default(),
crate::hive_stats::PriceTable::new(),
1,
)
.expect("open coordinator");
(dir, Arc::new(coord))
}
/// Insert a schedule owned by [`OWNER`], returning its id.
fn submit_owned_by_other(coord: &Arc<Coordinator>) -> i64 {
coord
.scheduled_prompts
.submit(&crate::scheduled_prompts::NewSchedule {
owner: OWNER.to_owned(),
targets: vec!["iris".to_owned()],
body: "ping".to_owned(),
first_fire_at_unix: 100,
interval_seconds: None,
description: None,
source: crate::scheduled_prompts::ScheduleSource::Operator,
})
.expect("submit")
}
#[test]
fn cancel_schedule_permits_unrelated_requester() {
let (_dir, coord) = coordinator();
let id = submit_owned_by_other(&coord);
let resp = handle_cancel_schedule(&coord, STRANGER, id, None);
assert!(
matches!(resp, Response::Ok),
"unrelated requester must be allowed to cancel; got {resp:?}"
);
// Ok alone could be a refusal that forgot to say so — check the row.
let row = coord
.scheduled_prompts
.get(id)
.expect("read back")
.expect("row present");
assert!(
row.cancelled_at_unix.is_some(),
"cancel must have taken effect, not just answered Ok"
);
}
#[test]
fn edit_schedule_permits_unrelated_requester() {
let (_dir, coord) = coordinator();
let id = submit_owned_by_other(&coord);
let resp = handle_edit_schedule(
&coord,
STRANGER,
id,
EditSchedulePatch {
body: Some("edited by a stranger".to_owned()),
description: None,
interval_seconds: None,
next_fire_at_unix: None,
targets_add: None,
targets_remove: None,
},
);
assert!(
matches!(resp, Response::Ok),
"unrelated requester must be allowed to edit; got {resp:?}"
);
let row = coord
.scheduled_prompts
.get(id)
.expect("read back")
.expect("row present");
assert_eq!(row.body, "edited by a stranger");
}
#[tokio::test]
async fn fire_schedule_now_permits_unrelated_requester() {
let (_dir, coord) = coordinator();
let id = submit_owned_by_other(&coord);
let resp = handle_fire_schedule_now(&coord, STRANGER, id).await;
// The fan-out itself is allowed to find no live target (this test has
// no containers) — that is reported inside the report, not as an Err.
assert!(
matches!(resp, Response::Ok),
"unrelated requester must be allowed to fire; got {resp:?}"
);
}
#[test]
fn list_schedules_shows_a_schedule_owned_by_someone_else() {
let (_dir, coord) = coordinator();
let id = submit_owned_by_other(&coord);
let Response::Schedules { schedules } = handle_list_schedules(&coord, STRANGER) else {
panic!("expected a Schedules response");
};
// The deleted filter dropped every row the requester was unrelated to,
// which for this requester is all of them.
assert!(
schedules.iter().any(|s| s.id == id && s.owner == OWNER),
"unrelated requester must see another agent's schedule; got {schedules:?}"
);
}
}

80
hive-c0re/src/stores/approvals.rs
View file

@ -1,10 +1,9 @@
//! Approval queue. Requests are submitted by an agent
//! (`RequestSchedulePrompt`); the user approves/denies via the host admin
//! CLI; on approval the host runs the corresponding action.
//! Approval queue. The user approves/denies via the host admin CLI; on
//! approval the host runs the corresponding action.
//!
//! `UpdateMetaInputs` rows are legacy: the MCP tool that queued them was
//! removed and nothing produces the kind any more. The variant and
//! its (de)serialization stay so pre-existing rows still read back.
//! Nothing submits approvals any more. `UpdateMetaInputs`, the only kind
//! left, is legacy: the MCP tool that queued it was removed. The variant
//! and its (de)serialization stay so pre-existing rows still read back.
use std::path::Path;
use std::sync::Mutex;
@ -64,6 +63,13 @@ const MIGRATIONS: &[Migration] = &[
sql: "DELETE FROM approvals WHERE kind = 'merge_config_pr'",
adds_column: None,
},
// v6: drop `schedule_prompt` rows. Scheduled prompts are swarm-level, so
// no approval of that kind can be acted on, and `row_to_approval` rejects
// the kind.
Migration {
sql: "DELETE FROM approvals WHERE kind = 'schedule_prompt'",
adds_column: None,
},
];
pub struct Approvals {
@ -81,7 +87,8 @@ impl Approvals {
})
}
/// Insert a new pending approval row.
/// Test fixture: insert a new pending approval row.
#[cfg(test)]
pub fn submit_kind(
&self,
agent: &str,
@ -354,7 +361,6 @@ fn row_to_approval(row: &rusqlite::Row<'_>) -> rusqlite::Result<Approval> {
let kind: String = row.get(2)?;
let kind = match kind.as_str() {
"update_meta_inputs" => ApprovalKind::UpdateMetaInputs,
"schedule_prompt" => ApprovalKind::SchedulePrompt,
other => {
return Err(rusqlite::Error::FromSqlConversionFailure(
2,
@ -404,7 +410,6 @@ fn row_to_approval(row: &rusqlite::Row<'_>) -> rusqlite::Result<Approval> {
fn kind_from_str(s: &str) -> Result<ApprovalKind> {
Ok(match s {
"update_meta_inputs" => ApprovalKind::UpdateMetaInputs,
"schedule_prompt" => ApprovalKind::SchedulePrompt,
other => bail!("unknown approval kind '{other}'"),
})
}
@ -421,17 +426,6 @@ mod tests {
(dir, path, db)
}
#[test]
fn mixed_kinds_all_listed() {
let (_dir, _path, db) = open_temp();
db.submit_kind("b", ApprovalKind::SchedulePrompt, "", None, "b")
.unwrap();
db.submit_kind("c", ApprovalKind::UpdateMetaInputs, "[]", None, "c")
.unwrap();
let pending = db.pending().expect("pending");
assert_eq!(pending.len(), 2, "both kinds must be visible");
}
#[test]
fn mark_cancelled_transitions_pending_row() {
// Manager withdraws a pending approval. Row leaves pending(),
@ -441,8 +435,8 @@ mod tests {
let id = db
.submit_kind(
"bitburner",
ApprovalKind::SchedulePrompt,
"{}",
ApprovalKind::UpdateMetaInputs,
"[]",
Some("test"),
"bitburner",
)
@ -464,7 +458,7 @@ mod tests {
// final — re-cancelling errors instead of silently overwriting.
let (_dir, _path, db) = open_temp();
let id = db
.submit_kind("a", ApprovalKind::SchedulePrompt, "{}", None, "a")
.submit_kind("a", ApprovalKind::UpdateMetaInputs, "[]", None, "a")
.unwrap();
db.mark_cancelled(id, "manager").expect("first cancel");
let err = db
@ -479,7 +473,7 @@ mod tests {
// whole list — collect_lenient skips it instead of failing.
let (_dir, path, db) = open_temp();
let good = db
.submit_kind("good", ApprovalKind::SchedulePrompt, "{}", None, "good")
.submit_kind("good", ApprovalKind::UpdateMetaInputs, "[]", None, "good")
.unwrap();
let raw = Connection::open(&path).unwrap();
raw.execute(
@ -502,14 +496,20 @@ mod tests {
// fall back to the root agent.
let (_dir, path, db) = open_temp();
let id = db
.submit_kind("child", ApprovalKind::SchedulePrompt, "{}", None, "parent")
.submit_kind(
"child",
ApprovalKind::UpdateMetaInputs,
"[]",
None,
"parent",
)
.unwrap();
assert_eq!(db.submitter_of(id).unwrap().as_deref(), Some("parent"));
let raw = Connection::open(&path).unwrap();
raw.execute(
"INSERT INTO approvals (agent, kind, commit_ref, requested_at, status)
VALUES ('old', 'schedule_prompt', '', 0, 'pending')",
VALUES ('old', 'update_meta_inputs', '', 0, 'pending')",
[],
)
.unwrap();
@ -524,7 +524,7 @@ mod tests {
fn a_pre_v5_merge_config_pr_row_is_dropped_on_open() {
let (_dir, path, db) = open_temp();
let kept = db
.submit_kind("iris", ApprovalKind::SchedulePrompt, "{}", None, "iris")
.submit_kind("iris", ApprovalKind::UpdateMetaInputs, "[]", None, "iris")
.unwrap();
drop(db);
let raw = Connection::open(&path).unwrap();
@ -546,4 +546,30 @@ mod tests {
assert!(db.recent_resolved(10).unwrap().is_empty());
assert!(db.get(old).unwrap().is_none());
}
/// A database written before v6 still holds `schedule_prompt` rows.
/// Opening it drops them and keeps every other row.
#[test]
fn a_pre_v6_schedule_prompt_row_is_dropped_on_open() {
let (_dir, path, db) = open_temp();
let kept = db
.submit_kind("iris", ApprovalKind::UpdateMetaInputs, "[]", None, "iris")
.unwrap();
drop(db);
let raw = Connection::open(&path).unwrap();
raw.execute_batch(
"INSERT INTO approvals (agent, kind, commit_ref, requested_at, status)
VALUES ('iris', 'schedule_prompt', '{}', 0, 'pending');
UPDATE schema_versions SET version = 5 WHERE store = 'approvals';",
)
.unwrap();
let old: i64 = raw.last_insert_rowid();
drop(raw);
let db = Approvals::open(&path).expect("a pre-v6 db opens");
let pending = db.pending().expect("pending");
assert_eq!(pending.len(), 1);
assert_eq!(pending[0].id, kept);
assert!(db.get(old).unwrap().is_none());
}
}

4
hive-c0re/src/stores/mod.rs
View file

@ -1,5 +1,5 @@
//! Sqlite-backed host-side stores (broker, approval / schedule queues,
//! build logs, power intent) plus the shared connection open/migration
//! Sqlite-backed host-side stores (broker, approval queue, read-only
//! scheduled prompts, build logs, power intent) plus the shared connection open/migration
//! helper (`db`). Each submodule is re-exported at the crate root, so
//! `crate::broker::…` etc. keep working unchanged.

1081
hive-c0re/src/stores/scheduled_prompts.rs
View file

File diff suppressed because it is too large Load diff

6
hive-c0re/src/workers/mod.rs
View file

@ -1,6 +1,5 @@
//! Background tasks and periodic sweeps: crash/login watcher, the
//! scheduled-prompt delivery loop, boot-time auto-update
//! reconcile, the agent-sockets.json writer loop, the MCP socket listener
//! Background tasks and periodic sweeps: crash/login watcher, boot-time
//! auto-update reconcile, the agent-sockets.json writer loop, the MCP socket listener
//! reconcile loop, knowledge-repo sync, and the swarm wanted-state pull.
//! Each submodule is re-exported
//! at the crate root, so `crate::crash_watch::…` etc. keep working unchanged.
@ -10,5 +9,4 @@ pub mod auto_update;
pub mod crash_watch;
pub mod knowledge;
pub mod mcp_sockets;
pub mod scheduled_prompts_worker;
pub mod wanted;

386
hive-c0re/src/workers/scheduled_prompts_worker.rs
View file

@ -1,386 +0,0 @@
//! Background loop that drains due `scheduled_prompts` rows and fans
//! the body to each active target. 5s poll cadence, shutdown-aware.
//! Catch-up clamp, missing-target handling, and broker-error retry
//! semantics: `docs/agent-lifecycle/approvals.md::Scheduled prompt worker`.
use std::sync::Arc;
use std::time::Duration;
use chrono::Utc;
use hive_sh4re::inbox::Message;
use crate::coordinator::Coordinator;
use crate::scheduled_prompts::Schedule;
/// Per-tick cap. Each schedule fires once per tick at most;
/// 100/tick × 5s tick = sustained throughput cap of ~20/sec,
/// matching `reminder_scheduler::REMINDER_BATCH_LIMIT`. Bump
/// together if real-world rates push past this.
const SCHEDULE_BATCH_LIMIT: u64 = 100;
/// Poll interval. Same 5s as the reminder scheduler — picking
/// up freshly-due rows within at most one tick keeps the
/// dashboard's "next fire in ..." countdown honest without
/// burning CPU on empty sweeps.
const POLL_INTERVAL: Duration = Duration::from_secs(5);
/// Reap cancelled schedules older than this from the table so
/// the dashboard list view doesn't accrue tombstones forever.
/// Cancelled rows live long enough that the operator can still
/// see what they cancelled in the recent past.
const CANCELLED_REAP_AGE: Duration = Duration::from_hours(1);
pub fn spawn(coord: Arc<Coordinator>) {
let mut shutdown = coord.shutdown_rx();
tokio::spawn(async move {
loop {
tick(&coord).await;
tokio::select! {
() = tokio::time::sleep(POLL_INTERVAL) => {}
_ = shutdown.changed() => {
tracing::info!("scheduled_prompts worker: shutdown signal received");
break;
}
}
}
});
}
async fn tick(coord: &Arc<Coordinator>) {
let now = Utc::now().timestamp();
let due = match coord.scheduled_prompts.due(now, SCHEDULE_BATCH_LIMIT) {
Ok(rows) => rows,
Err(e) => {
tracing::warn!(error = ?e, "scheduled_prompts: query due rows failed");
return;
}
};
if due.is_empty() {
// Periodic reaper still gets a chance even on empty ticks.
let cutoff = now - i64::try_from(CANCELLED_REAP_AGE.as_secs()).unwrap_or(0);
if let Err(e) = coord.scheduled_prompts.reap_cancelled(cutoff) {
tracing::warn!(error = ?e, "scheduled_prompts: reap cancelled failed");
}
return;
}
for schedule in due {
fire_schedule(coord, &schedule, now).await;
}
let cutoff = now - i64::try_from(CANCELLED_REAP_AGE.as_secs()).unwrap_or(0);
if let Err(e) = coord.scheduled_prompts.reap_cancelled(cutoff) {
tracing::warn!(error = ?e, "scheduled_prompts: reap cancelled failed");
}
// Emit after all fires + reaps so the dashboard reflects updated
// last_fired_at_unix, next_fire_at_unix, and any reaped one-shots.
coord.emit_schedules_snapshot();
}
/// Fan out one schedule's body to every active target. Records
/// per-target `last_result`; advances or reaps the parent row at
/// the end depending on whether `interval_seconds` is set.
///
/// Delivery is `push_todo`, not a broker `Message` — a scheduled
/// prompt now wakes its target with a todo instead of driving an
/// immediate turn, by design: mara confirmed on the tracking issue
/// that this is the intended behavior, not an incidental side effect.
/// `key = "schedule:{id}"` per target gives `push_todo`'s own
/// upsert-by-key dedup the same job a now-removed
/// `has_pending_with_body` broker check used to do — collapsing a
/// re-fire of the *same schedule* against a target that hasn't
/// reviewed the last one yet, keyed on schedule identity rather than on
/// the body happening to be byte-identical. `deliver_to_target` also
/// passes `reopen_if_acked = true`, so this collapse only applies while
/// the previous fire is genuinely still sitting there un-reviewed — once
/// the target acks it, the *next* fire reopens the row and wakes again
/// even with an identical body, instead of silently staying quiet
/// forever — a recurring schedule with a static body used to wake its
/// target once, ever, and then look dead.
async fn fire_schedule(coord: &Arc<Coordinator>, schedule: &Schedule, now: i64) {
let known: std::collections::HashSet<String> = known_agents().await;
for target_row in &schedule.targets {
if target_row.cancelled_at_unix.is_some() {
continue;
}
let target = &target_row.target;
if !known.contains(target) {
let reason = format!("no such agent: {target}");
if let Err(e) =
coord
.scheduled_prompts
.record_target_result(schedule.id, target, now, &reason)
{
tracing::warn!(error = ?e, schedule = schedule.id, %target, "record_target_result failed");
}
notify_operator_missing_target(coord, schedule, target);
continue;
}
let result_str = match deliver_to_target(coord, schedule.id, target, &schedule.body).await {
Ok(()) => "ok".to_owned(),
Err(reason) => {
tracing::warn!(
schedule = schedule.id,
%target,
%reason,
"scheduled_prompts: delivery failed (will retry on next interval)"
);
reason
}
};
if let Err(e) =
coord
.scheduled_prompts
.record_target_result(schedule.id, target, now, &result_str)
{
tracing::warn!(error = ?e, schedule = schedule.id, %target, "record_target_result failed");
}
}
// Advance or reap. One-shots delete; recurring re-arm with
// catch-up clamp.
if schedule.interval_seconds.is_some() {
match coord.scheduled_prompts.rearm(schedule.id, now) {
Ok(0) => {}
Ok(skipped) => {
tracing::info!(
schedule = schedule.id,
skipped,
"scheduled_prompts: caught up missed cycles"
);
}
Err(e) => {
tracing::warn!(error = ?e, schedule = schedule.id, "rearm failed");
}
}
} else if let Err(e) = coord.scheduled_prompts.delete(schedule.id) {
tracing::warn!(error = ?e, schedule = schedule.id, "delete one-shot failed");
}
}
/// Deliver `body` to a single already-known-live `target` via
/// `push_todo`, keyed on the schedule's own identity
/// (`schedule:{schedule_id}`) so a re-fire against a target that hasn't
/// reviewed the last one collapses via `push_todo`'s own upsert-by-key
/// dedup — but `reopen_if_acked = true` means that collapse only holds
/// while unreviewed; once acked, the next fire reopens regardless of
/// whether the body changed. Shared by the periodic `fire_schedule`
/// tick and the manual `fire_now` dashboard action — the only
/// difference between them is what each caller does with the `Result`
/// (log/prefix and per-target `last_result`/`FireNowReport`
/// bookkeeping), not the delivery choice itself.
async fn deliver_to_target(
coord: &Arc<Coordinator>,
schedule_id: i64,
target: &str,
body: &str,
) -> Result<(), String> {
coord
.push_todo(
target,
"schedule",
Some(format!("schedule:{schedule_id}")),
body.to_owned(),
Some("scheduled".to_owned()),
// Each fire is a distinct occurrence, not a restatement of
// a persisting condition — an agent that already acked the
// *previous* firing hasn't acked *this* one, so an acked
// row must reopen even when the body is byte-identical
// (the common case: most schedules don't vary their text
// per fire). See `Todos::upsert`'s doc comment for the
// full reconciler-vs-event rationale.
true,
)
.await
}
/// Send the operator a one-line advisory when a schedule fires
/// against an agent that no longer exists. Best-effort — failure
/// to send just gets logged; the schedule continues firing.
fn notify_operator_missing_target(coord: &Coordinator, schedule: &Schedule, target: &str) {
let body = format!(
"scheduled prompt #{id} fired but target `{target}` is not a live agent. \
body was:\n\n{body}",
id = schedule.id,
target = target,
body = schedule.body
);
let msg = Message {
from: hive_sh4re::manager::trusted_sender("scheduled"),
to: hive_sh4re::manager::OPERATOR_RECIPIENT.to_owned(),
body,
in_reply_to: None,
};
if let Err(e) = coord.broker.send(&msg) {
tracing::warn!(error = ?e, schedule = schedule.id, %target, "operator advisory send failed");
}
}
/// Per-target outcome counts for one `fire_now` invocation.
/// Returned to the operator so the dashboard can render
/// "fired to N (M failed, K missing)" without a follow-up GET.
#[derive(Debug, Clone, serde::Serialize, utoipa::ToSchema)]
pub struct FireNowReport {
/// Targets the broker accepted the message for.
pub ok: u32,
/// Targets where broker.send returned an error.
pub failed: u32,
/// Targets that didn't resolve to a known agent (and got the
/// operator-advisory treatment).
pub missing: u32,
/// Whether the one-shot was consumed by this manual fire.
/// `true` only when the schedule was a one-shot (recurring
/// schedules never auto-cancel on manual fire — they keep
/// their cadence).
pub one_shot_consumed: bool,
/// Whether this manual fire re-armed a recurring schedule's timer
/// (`next_fire_at = now + interval`). `true` only when the caller
/// passed `reset_timer` AND the schedule is recurring.
pub timer_reset: bool,
}
/// Manual / out-of-band fire of a scheduled prompt ("fire now"
/// dashboard button). Mirrors the per-target fan-out of `fire_schedule`
/// but skips the rearm step entirely — manual fires don't disturb
/// a recurring schedule's rhythm. For one-shots, a manual fire
/// **consumes** the schedule (operator intent: "send this now,
/// the scheduled time was wrong"); recurring schedules keep their
/// `next_fire_at_unix` unchanged.
///
/// `last_result` is annotated with the `manual fire:` prefix so
/// the dashboard's per-target last-result column can distinguish
/// scheduled fires from operator-initiated ones at a glance.
///
/// `reset_timer` re-arms a *recurring* schedule's countdown from now
/// (`next_fire_at = now + interval`) after the fan-out — the dashboard
/// fire-now dialog's "reset timer" checkbox. It's a no-op for one-shots
/// (still consumed) and when `false` (today's default: cadence intact).
///
/// Returns Err if the schedule is missing, cancelled, or fully
/// drained of active targets — the dashboard can surface those
/// as plain 4xxs instead of pretending to fire a phantom row.
pub async fn fire_now(
coord: &std::sync::Arc<Coordinator>,
schedule_id: i64,
reset_timer: bool,
) -> anyhow::Result<FireNowReport> {
let now = Utc::now().timestamp();
let schedule = coord
.scheduled_prompts
.get(schedule_id)?
.ok_or_else(|| anyhow::anyhow!("schedule {schedule_id} not found"))?;
if schedule.cancelled_at_unix.is_some() {
anyhow::bail!("schedule {schedule_id} is already cancelled");
}
if !schedule
.targets
.iter()
.any(|t| t.cancelled_at_unix.is_none())
{
anyhow::bail!("schedule {schedule_id} has no active targets");
}
let known = known_agents().await;
let mut report = FireNowReport {
ok: 0,
failed: 0,
missing: 0,
one_shot_consumed: false,
timer_reset: false,
};
for target_row in &schedule.targets {
if target_row.cancelled_at_unix.is_some() {
continue;
}
let target = &target_row.target;
if !known.contains(target) {
let reason = format!("manual fire: no such agent: {target}");
if let Err(e) =
coord
.scheduled_prompts
.record_target_result(schedule_id, target, now, &reason)
{
tracing::warn!(error = ?e, schedule = schedule_id, %target, "record_target_result failed");
}
notify_operator_missing_target(coord, &schedule, target);
report.missing += 1;
continue;
}
let result_str = match deliver_to_target(coord, schedule_id, target, &schedule.body).await {
Ok(()) => {
report.ok += 1;
"manual fire: ok".to_owned()
}
Err(reason) => {
report.failed += 1;
tracing::warn!(
schedule = schedule_id,
%target,
%reason,
"fire_now: delivery failed (no retry — manual fires don't loop)"
);
format!("manual fire: {reason}")
}
};
if let Err(e) =
coord
.scheduled_prompts
.record_target_result(schedule_id, target, now, &result_str)
{
tracing::warn!(error = ?e, schedule = schedule_id, %target, "record_target_result failed");
}
}
match schedule.interval_seconds {
None => {
// One-shot is consumed by the manual fire. (reset_timer is
// moot here — there's no recurring cadence to re-arm.)
if let Err(e) = coord.scheduled_prompts.cancel_all(schedule_id) {
tracing::warn!(error = ?e, schedule = schedule_id, "cancel_all after one-shot manual fire failed");
} else {
report.one_shot_consumed = true;
}
}
Some(interval) if reset_timer => {
// Recurring + operator asked to reset: re-arm the countdown
// from now (now + interval), not along the existing cadence.
let next = now.saturating_add(i64::try_from(interval).unwrap_or(i64::MAX));
if let Err(e) = coord.scheduled_prompts.set_next_fire(schedule_id, next) {
tracing::warn!(error = ?e, schedule = schedule_id, "set_next_fire after manual fire reset failed");
} else {
report.timer_reset = true;
}
}
// Recurring without reset: cadence stays intact (additive fire).
Some(_) => {}
}
Ok(report)
}
/// Snapshot of live container names for the missing-target check.
/// Always seeds the manager name (which is always reachable);
/// adds every live nspawn container that matches the `h-` prefix.
/// On `lifecycle::list` failure the set stays at just the manager
/// — fail-CLOSED, meaning every non-manager target looks missing
/// this tick and gets the same treatment as a genuinely-destroyed
/// agent: operator advisory + per-target `last_result` annotation +
/// skipped delivery. Recurring
/// schedules recover automatically on the next tick (the lifecycle
/// listing usually works); one-shots that land on this window
/// lose their single delivery. Logged at `warn`, not propagated.
/// Shared by `fire_schedule` and `fire_now` — both are async now
/// (the sync `fire_schedule` used to need a `block_in_place` variant
/// of this before it started `push_todo`ing, which is itself async).
async fn known_agents() -> std::collections::HashSet<String> {
use std::collections::HashSet;
let mut out: HashSet<String> = HashSet::new();
out.insert(hive_sh4re::manager::MANAGER_AGENT.to_owned());
match crate::lifecycle::list().await {
Ok(list) => {
for raw in list {
if let Some(name) = raw.strip_prefix(crate::lifecycle::AGENT_PREFIX) {
out.insert(name.to_owned());
}
}
}
Err(e) => {
tracing::warn!(error = ?e, "fire_now: container listing failed");
}
}
out
}