From a0b1846006cac64b58b56140aecc672a2cc0fd6c Mon Sep 17 00:00:00 2001 From: atlas Date: Sun, 4 Oct 2026 16:29:44 +0200 Subject: [PATCH] swarm: named scheduled prompts for one or more agents, fired in each agent swarm-controller keeps every scheduled prompt (id, name, agents, body, anchor, interval, paused) as one document under one key of a new `schedule-definitions` KV bucket, so each change is a single compare-and-set and a prompt exists for all of its agents or none. It writes each agent's key of a new `agent-schedules` bucket as the prompts naming that agent, without names or other agents, after every change and on a one-minute reconcile pass. It is the only writer of both, behind GET/POST /api/schedules and PATCH/DELETE /api/schedules/{id}. `swarmctl schedule add|list|edit|pause|resume|rm` drives those routes over the controller socket. An agent's own queue credential is granted one more subject, the direct get of its own key in `agent-schedules`. A KV watch is not granted: its consumer create subject carries no key, so it would cover every agent's. hive-agent's new schedule ticker reads that key every 10 seconds and fires due slots as the same `schedule:` todo hive-c0re's worker pushes. Fire times are anchor + k*interval; the last fired slot per schedule id is kept in the state db, so a body edit or list refresh never moves or refires a slot, downtime fires the latest missed slot once, a restart refires nothing, a one-shot fires once, and paused slots are skipped. Refs #4914 --- Cargo.lock | 1 + docs/README.md | 2 + docs/swarm/README.md | 2 + docs/swarm/schedules.md | 150 ++++ docs/tools/swarmctl-cli.md | 145 ++++ hive-agent/Cargo.toml | 8 +- hive-agent/src/main.rs | 4 + hive-agent/src/schedule_timer.rs | 388 +++++++++ nix/host-modules/swarm-nats.nix | 7 + nix/module-eval/nats-authelia.nix | 3 +- swarm-controller/README.md | 5 + swarm-controller/src/main.rs | 32 +- swarm-controller/src/schedules.rs | 922 ++++++++++++++++++++++ swarm-controller/src/subagent_term.rs | 2 +- swarm-nats-auth/src/main.rs | 4 +- swarm-nats-auth/src/policy.rs | 71 ++ swarm-queue-client/Cargo.toml | 4 + swarm-queue-client/src/agent_schedules.rs | 234 ++++++ swarm-queue-client/src/lib.rs | 5 + swarmctl/Cargo.toml | 1 + swarmctl/README.md | 16 + swarmctl/src/agent.rs | 39 +- swarmctl/src/main.rs | 175 ++++ swarmctl/src/schedule.rs | 416 ++++++++++ 24 files changed, 2617 insertions(+), 19 deletions(-) create mode 100644 docs/swarm/schedules.md create mode 100644 hive-agent/src/schedule_timer.rs create mode 100644 swarm-controller/src/schedules.rs create mode 100644 swarm-queue-client/src/agent_schedules.rs create mode 100644 swarmctl/src/schedule.rs diff --git a/Cargo.lock b/Cargo.lock index 4e71f1ae..1f16b1e2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4909,6 +4909,7 @@ version = "0.1.0" dependencies = [ "anyhow", "bytes", + "chrono", "clap", "clap-markdown", "clap_complete", diff --git a/docs/README.md b/docs/README.md index 196e6571..b7a25bc1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -82,6 +82,8 @@ declarations. [`swarm/secrets.md`](swarm/secrets.md) for where each file lives today. - **How does the secret store come up, who writes its grants, how's it sealed?** → [`swarm/bao.md`](swarm/bao.md). +- **How does an agent get a prompt at set times, and why doesn't it fire + twice after downtime?** → [`swarm/schedules.md`](swarm/schedules.md). ## Scheduler, CI, observability diff --git a/docs/swarm/README.md b/docs/swarm/README.md index eb2fbd3a..bb0a9c7f 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -21,6 +21,8 @@ grafana, the controller, authelia, …) → - **control plane** — `swarm-controller`: hive directory, agent roster, job graph, agent creation. → [below](#swarm-controller) +- **scheduled prompts** — named prompts for one or more agents at set times, + kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md) - **swarm UI** — the operator's day-to-day surface, on the swarm apex, `admins` only. → [`ui.md`](ui.md) - **shared services** — one forge, homeserver, SSO, queue and metrics/logs diff --git a/docs/swarm/schedules.md b/docs/swarm/schedules.md new file mode 100644 index 00000000..fe95a472 --- /dev/null +++ b/docs/swarm/schedules.md @@ -0,0 +1,150 @@ +# Scheduled prompts + +A scheduled prompt hands one or more agents the same text at set times: once, +or every interval from a start time. Each prompt has a name, its title. The +swarm keeps every prompt, and each agent it names fires it in its own +container, so a fire needs neither the hive nor the swarm controller to be up +at that moment, only the agent. + +Operators manage them with `swarmctl schedule` on the swarm controller's +host. Agents can't create, change or remove one. For a one-off wake an agent +sets itself, it uses `remind`, which stays inside its container. + +## Managing them + +```console +# swarmctl schedule add --name standup --agent scribe --agent iris \ + --body "post the standup summary" --at 2026-10-05T09:00:00+02:00 --every 1d +added schedule kqzpmwab +kqzpmwab "standup" every 1d from 2026-10-05T07:00:00Z next 2026-10-05T07:00:00Z agents: iris, scribe | post the standup summary + +# swarmctl schedule list +# swarmctl schedule list --agent scribe +# swarmctl schedule edit kqzpmwab --body "post the standup summary to the team room" +# swarmctl schedule edit kqzpmwab --add-agent atlas --remove-agent iris +# swarmctl schedule pause kqzpmwab +# swarmctl schedule resume kqzpmwab +# swarmctl schedule rm kqzpmwab +``` + +- `--agent` names an agent to fire the prompt in; repeat it for more. Each + must be an agent some hive's wanted state declares. +- `--at` takes an RFC 3339 time with its offset, or unix seconds. Without it + the first fire is now. A time in the past fires its latest past time once, + as soon as each agent reads the prompt. +- `--every` takes a count and a unit, `s`, `m`, `h` or `d`. Without it the + prompt fires once. +- `schedule edit` changes only the flags you pass and keeps the id. `--once` + turns a recurring prompt into a one-shot. `--add-agent` and + `--remove-agent` change who gets it; removing the last agent fails, so + remove the prompt instead. +- `schedule list` shows one row per prompt: id, name, cadence, start, next + fire time, agents, and the first line of the body. + +The full flag reference is [`swarmctl-cli.md`](../tools/swarmctl-cli.md). + +## When a prompt fires + +A prompt's fire times are its start time plus whole intervals. Each agent +works them out from the prompt itself, so: + +- **Editing the text or the name doesn't move them.** Neither does changing + the agents. Only a new `--at` or `--every` does. +- **An agent that was down or disconnected fires once.** When it's back, it + fires the latest time it missed and then carries on at the next one. The + earlier missed times don't fire. +- **A restart refires nothing.** Each agent records the last time it fired + for each prompt in its state database, and fires a time only when it's + later than that. An agent removed from a prompt and added back keeps that + record, so it doesn't refire a time it already fired. +- **A one-shot fires once**, even though it stays in `schedule list` until + you remove it. Moving its `--at` to a later time fires it again then. +- **A paused prompt doesn't fire**, and the times that pass while it's paused + don't fire when it resumes. An agent marks those times as passed while it + watches the paused prompt; if the agent was down for the whole pause, it + fires the latest time it missed once, as above. + +A fire reaches the agent as a `schedule` loose end with the prompt's text, +the same as a hive schedule's. The agent's turn loop wakes for it like any +other loose end. If the agent hasn't dealt with the previous fire yet, the +new fire replaces it rather than adding a second one; once the agent +dismissed the previous one, the next fire opens a new one. + +Each agent reads its prompts every 10 seconds, so a fire can land up to 10 +seconds after its time. A change reaches each agent within the same window, +or within a minute if the controller's first attempt to pass it on failed. + +## When a prompt doesn't fire + +- **The agent has no queue credential of its own.** The agent reads its + prompts with its own swarm-queue credential. An agent connected with its + hive's shared client logs `connected with the hive's shared client, which +has no read grant on the agent-schedules bucket` and fires nothing. +- **The agent's first connect with its own credential failed.** The harness + keeps the outcome of its first connect for the rest of its run: it falls + back to the hive's shared client (the case above) or stays disconnected, + and reads no prompts until it restarts. +- **The agent hasn't read its prompts since it started.** It fires from the + last list it read; until the first read succeeds it fires nothing. A read + that fails later keeps the previous list. + +
Implementation + +swarm-controller is the only client that writes either of the two KV buckets +involved. + +`schedule-definitions` holds every prompt as one JSON document under the key +`all`. Agents hold no grant on this bucket. Each change re-reads the key and +writes against its revision, so a prompt exists for every one of its agents or for +none of them, and two changes at once don't lose one. + +`agent-schedules` holds one key per agent: the prompts naming that agent, +without their names or other agents. + +```json +{ + "schedules": [ + { + "id": "kqzpmwab", + "body": "post the standup summary", + "anchor_unix": 1791183600, + "interval_seconds": 86400, + "paused": false + } + ] +} +``` + +`interval_seconds` is absent for a one-shot. After each change the controller +rewrites the key of every agent the change named before or after, and once a +minute it rewrites every agent's key, so a rewrite that failed converges. Ids +are 8 lowercase letters, the same in every agent's key, never reused, and so +never equal to a hive schedule's numeric id. + +`swarmctl schedule` calls these routes over the controller's socket: + +| method | path | body | +| -------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| `GET` | `/api/schedules` | —; answers every prompt, sorted by name then id | +| `POST` | `/api/schedules` | `name`, `agents`, `body`, and optional `anchor_unix`, `interval_seconds`, `paused`; answers 201 with the prompt | +| `PATCH` | `/api/schedules/` | any of `name`, `body`, `anchor_unix`, `interval_seconds` (`null` for a one-shot), `paused`, `add_agents`, `remove_agents` | +| `DELETE` | `/api/schedules/` | —; answers 204 | + +Each prompt they answer with has `id`, `name`, `agents` (sorted), `body`, +`anchor_unix`, `interval_seconds` (`null` for a one-shot), `paused`, and +`next_fire_unix`: the next time on the prompt's grid, as of the moment the +controller answers, `null` when paused or a one-shot whose time has passed. + +An agent's own queue credential grants exactly one subject in the swarm's +JetStream: `$JS.API.DIRECT.GET.KV_agent-schedules.$KV.agent-schedules.`, +the direct read of its own key. It can't read another agent's key, open or +watch the bucket, or write to it. A KV watch would need a consumer, and a +consumer create subject doesn't name a key, so granting one would let the +agent read every key; the agent reads its key every 10 seconds instead. + +The harness (`hive-agent`'s `schedule_timer`) keeps the last fired time per +prompt id in the `schedule_fired_slots` table of its state database, and +pushes each fire as the todo `schedule`/`schedule:` with source +`scheduled`, reopening an acknowledged one. + +
diff --git a/docs/tools/swarmctl-cli.md b/docs/tools/swarmctl-cli.md index caf28d89..fe16f17a 100644 --- a/docs/tools/swarmctl-cli.md +++ b/docs/tools/swarmctl-cli.md @@ -9,6 +9,13 @@ This document contains the help content for the `swarmctl` command-line program. * [`swarmctl agent create`↴](#swarmctl-agent-create) * [`swarmctl agent mint-identity`↴](#swarmctl-agent-mint-identity) * [`swarmctl agent mint-forge-token`↴](#swarmctl-agent-mint-forge-token) +* [`swarmctl schedule`↴](#swarmctl-schedule) +* [`swarmctl schedule add`↴](#swarmctl-schedule-add) +* [`swarmctl schedule list`↴](#swarmctl-schedule-list) +* [`swarmctl schedule edit`↴](#swarmctl-schedule-edit) +* [`swarmctl schedule pause`↴](#swarmctl-schedule-pause) +* [`swarmctl schedule resume`↴](#swarmctl-schedule-resume) +* [`swarmctl schedule rm`↴](#swarmctl-schedule-rm) * [`swarmctl user`↴](#swarmctl-user) * [`swarmctl user add`↴](#swarmctl-user-add) * [`swarmctl user reset-password`↴](#swarmctl-user-reset-password) @@ -27,6 +34,7 @@ swarm-level operator CLI ###### **Subcommands:** * `agent` — Manage agents across the swarm +* `schedule` — Manage scheduled prompts * `user` — Manage subjects in the swarm's SSO provider * `forge` — Manage human accounts on the swarm's forge * `completions` — Generate a shell completion script for `swarmctl` and print it to stdout @@ -129,6 +137,143 @@ Queues and returns, the same way `agent create` does — watch the swarm UI's jo +## `swarmctl schedule` + +Manage scheduled prompts. + +The swarm keeps each prompt, and every agent it names fires it in its own container. Fire times are the schedule's start time plus whole intervals: changing the body never moves them, and an agent that was down fires the latest missed one once. + +**Usage:** `swarmctl schedule ` + +###### **Subcommands:** + +* `add` — Add a scheduled prompt and print it with its new id +* `list` — List scheduled prompts, one row each, with their agents and next fire time +* `edit` — Change a scheduled prompt +* `pause` — Stop a scheduled prompt firing. Times that pass while it's paused aren't fired when it resumes +* `resume` — Let a paused scheduled prompt fire again, from its next time on +* `rm` — Remove a scheduled prompt from every agent it names + + + +## `swarmctl schedule add` + +Add a scheduled prompt and print it with its new id. + +Without `--every` it fires once. Without `--at` its first fire is now. + +**Usage:** `swarmctl schedule add [OPTIONS] --name --agent --body ` + +###### **Options:** + +* `--name ` — The prompt's title +* `--agent ` — Agent to fire it in. Repeatable; at least one +* `--body ` — The prompt, delivered to each agent verbatim +* `--at