diff --git a/Cargo.lock b/Cargo.lock index 4e71f1ae..1f16b1e2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4909,6 +4909,7 @@ version = "0.1.0" dependencies = [ "anyhow", "bytes", + "chrono", "clap", "clap-markdown", "clap_complete", diff --git a/docs/README.md b/docs/README.md index 196e6571..b7a25bc1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -82,6 +82,8 @@ declarations. [`swarm/secrets.md`](swarm/secrets.md) for where each file lives today. - **How does the secret store come up, who writes its grants, how's it sealed?** → [`swarm/bao.md`](swarm/bao.md). +- **How does an agent get a prompt at set times, and why doesn't it fire + twice after downtime?** → [`swarm/schedules.md`](swarm/schedules.md). ## Scheduler, CI, observability diff --git a/docs/swarm/README.md b/docs/swarm/README.md index eb2fbd3a..bb0a9c7f 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -21,6 +21,8 @@ grafana, the controller, authelia, …) → - **control plane** — `swarm-controller`: hive directory, agent roster, job graph, agent creation. → [below](#swarm-controller) +- **scheduled prompts** — named prompts for one or more agents at set times, + kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md) - **swarm UI** — the operator's day-to-day surface, on the swarm apex, `admins` only. → [`ui.md`](ui.md) - **shared services** — one forge, homeserver, SSO, queue and metrics/logs diff --git a/docs/swarm/schedules.md b/docs/swarm/schedules.md new file mode 100644 index 00000000..fe95a472 --- /dev/null +++ b/docs/swarm/schedules.md @@ -0,0 +1,150 @@ +# Scheduled prompts + +A scheduled prompt hands one or more agents the same text at set times: once, +or every interval from a start time. Each prompt has a name, its title. The +swarm keeps every prompt, and each agent it names fires it in its own +container, so a fire needs neither the hive nor the swarm controller to be up +at that moment, only the agent. + +Operators manage them with `swarmctl schedule` on the swarm controller's +host. Agents can't create, change or remove one. For a one-off wake an agent +sets itself, it uses `remind`, which stays inside its container. + +## Managing them + +```console +# swarmctl schedule add --name standup --agent scribe --agent iris \ + --body "post the standup summary" --at 2026-10-05T09:00:00+02:00 --every 1d +added schedule kqzpmwab +kqzpmwab "standup" every 1d from 2026-10-05T07:00:00Z next 2026-10-05T07:00:00Z agents: iris, scribe | post the standup summary + +# swarmctl schedule list +# swarmctl schedule list --agent scribe +# swarmctl schedule edit kqzpmwab --body "post the standup summary to the team room" +# swarmctl schedule edit kqzpmwab --add-agent atlas --remove-agent iris +# swarmctl schedule pause kqzpmwab +# swarmctl schedule resume kqzpmwab +# swarmctl schedule rm kqzpmwab +``` + +- `--agent` names an agent to fire the prompt in; repeat it for more. Each + must be an agent some hive's wanted state declares. +- `--at` takes an RFC 3339 time with its offset, or unix seconds. Without it + the first fire is now. A time in the past fires its latest past time once, + as soon as each agent reads the prompt. +- `--every` takes a count and a unit, `s`, `m`, `h` or `d`. Without it the + prompt fires once. +- `schedule edit` changes only the flags you pass and keeps the id. `--once` + turns a recurring prompt into a one-shot. `--add-agent` and + `--remove-agent` change who gets it; removing the last agent fails, so + remove the prompt instead. +- `schedule list` shows one row per prompt: id, name, cadence, start, next + fire time, agents, and the first line of the body. + +The full flag reference is [`swarmctl-cli.md`](../tools/swarmctl-cli.md). + +## When a prompt fires + +A prompt's fire times are its start time plus whole intervals. Each agent +works them out from the prompt itself, so: + +- **Editing the text or the name doesn't move them.** Neither does changing + the agents. Only a new `--at` or `--every` does. +- **An agent that was down or disconnected fires once.** When it's back, it + fires the latest time it missed and then carries on at the next one. The + earlier missed times don't fire. +- **A restart refires nothing.** Each agent records the last time it fired + for each prompt in its state database, and fires a time only when it's + later than that. An agent removed from a prompt and added back keeps that + record, so it doesn't refire a time it already fired. +- **A one-shot fires once**, even though it stays in `schedule list` until + you remove it. Moving its `--at` to a later time fires it again then. +- **A paused prompt doesn't fire**, and the times that pass while it's paused + don't fire when it resumes. An agent marks those times as passed while it + watches the paused prompt; if the agent was down for the whole pause, it + fires the latest time it missed once, as above. + +A fire reaches the agent as a `schedule` loose end with the prompt's text, +the same as a hive schedule's. The agent's turn loop wakes for it like any +other loose end. If the agent hasn't dealt with the previous fire yet, the +new fire replaces it rather than adding a second one; once the agent +dismissed the previous one, the next fire opens a new one. + +Each agent reads its prompts every 10 seconds, so a fire can land up to 10 +seconds after its time. A change reaches each agent within the same window, +or within a minute if the controller's first attempt to pass it on failed. + +## When a prompt doesn't fire + +- **The agent has no queue credential of its own.** The agent reads its + prompts with its own swarm-queue credential. An agent connected with its + hive's shared client logs `connected with the hive's shared client, which +has no read grant on the agent-schedules bucket` and fires nothing. +- **The agent's first connect with its own credential failed.** The harness + keeps the outcome of its first connect for the rest of its run: it falls + back to the hive's shared client (the case above) or stays disconnected, + and reads no prompts until it restarts. +- **The agent hasn't read its prompts since it started.** It fires from the + last list it read; until the first read succeeds it fires nothing. A read + that fails later keeps the previous list. + +
Implementation + +swarm-controller is the only client that writes either of the two KV buckets +involved. + +`schedule-definitions` holds every prompt as one JSON document under the key +`all`. Agents hold no grant on this bucket. Each change re-reads the key and +writes against its revision, so a prompt exists for every one of its agents or for +none of them, and two changes at once don't lose one. + +`agent-schedules` holds one key per agent: the prompts naming that agent, +without their names or other agents. + +```json +{ + "schedules": [ + { + "id": "kqzpmwab", + "body": "post the standup summary", + "anchor_unix": 1791183600, + "interval_seconds": 86400, + "paused": false + } + ] +} +``` + +`interval_seconds` is absent for a one-shot. After each change the controller +rewrites the key of every agent the change named before or after, and once a +minute it rewrites every agent's key, so a rewrite that failed converges. Ids +are 8 lowercase letters, the same in every agent's key, never reused, and so +never equal to a hive schedule's numeric id. + +`swarmctl schedule` calls these routes over the controller's socket: + +| method | path | body | +| -------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| `GET` | `/api/schedules` | —; answers every prompt, sorted by name then id | +| `POST` | `/api/schedules` | `name`, `agents`, `body`, and optional `anchor_unix`, `interval_seconds`, `paused`; answers 201 with the prompt | +| `PATCH` | `/api/schedules/` | any of `name`, `body`, `anchor_unix`, `interval_seconds` (`null` for a one-shot), `paused`, `add_agents`, `remove_agents` | +| `DELETE` | `/api/schedules/` | —; answers 204 | + +Each prompt they answer with has `id`, `name`, `agents` (sorted), `body`, +`anchor_unix`, `interval_seconds` (`null` for a one-shot), `paused`, and +`next_fire_unix`: the next time on the prompt's grid, as of the moment the +controller answers, `null` when paused or a one-shot whose time has passed. + +An agent's own queue credential grants exactly one subject in the swarm's +JetStream: `$JS.API.DIRECT.GET.KV_agent-schedules.$KV.agent-schedules.`, +the direct read of its own key. It can't read another agent's key, open or +watch the bucket, or write to it. A KV watch would need a consumer, and a +consumer create subject doesn't name a key, so granting one would let the +agent read every key; the agent reads its key every 10 seconds instead. + +The harness (`hive-agent`'s `schedule_timer`) keeps the last fired time per +prompt id in the `schedule_fired_slots` table of its state database, and +pushes each fire as the todo `schedule`/`schedule:` with source +`scheduled`, reopening an acknowledged one. + +
diff --git a/docs/tools/swarmctl-cli.md b/docs/tools/swarmctl-cli.md index caf28d89..fe16f17a 100644 --- a/docs/tools/swarmctl-cli.md +++ b/docs/tools/swarmctl-cli.md @@ -9,6 +9,13 @@ This document contains the help content for the `swarmctl` command-line program. * [`swarmctl agent create`↴](#swarmctl-agent-create) * [`swarmctl agent mint-identity`↴](#swarmctl-agent-mint-identity) * [`swarmctl agent mint-forge-token`↴](#swarmctl-agent-mint-forge-token) +* [`swarmctl schedule`↴](#swarmctl-schedule) +* [`swarmctl schedule add`↴](#swarmctl-schedule-add) +* [`swarmctl schedule list`↴](#swarmctl-schedule-list) +* [`swarmctl schedule edit`↴](#swarmctl-schedule-edit) +* [`swarmctl schedule pause`↴](#swarmctl-schedule-pause) +* [`swarmctl schedule resume`↴](#swarmctl-schedule-resume) +* [`swarmctl schedule rm`↴](#swarmctl-schedule-rm) * [`swarmctl user`↴](#swarmctl-user) * [`swarmctl user add`↴](#swarmctl-user-add) * [`swarmctl user reset-password`↴](#swarmctl-user-reset-password) @@ -27,6 +34,7 @@ swarm-level operator CLI ###### **Subcommands:** * `agent` — Manage agents across the swarm +* `schedule` — Manage scheduled prompts * `user` — Manage subjects in the swarm's SSO provider * `forge` — Manage human accounts on the swarm's forge * `completions` — Generate a shell completion script for `swarmctl` and print it to stdout @@ -129,6 +137,143 @@ Queues and returns, the same way `agent create` does — watch the swarm UI's jo +## `swarmctl schedule` + +Manage scheduled prompts. + +The swarm keeps each prompt, and every agent it names fires it in its own container. Fire times are the schedule's start time plus whole intervals: changing the body never moves them, and an agent that was down fires the latest missed one once. + +**Usage:** `swarmctl schedule ` + +###### **Subcommands:** + +* `add` — Add a scheduled prompt and print it with its new id +* `list` — List scheduled prompts, one row each, with their agents and next fire time +* `edit` — Change a scheduled prompt +* `pause` — Stop a scheduled prompt firing. Times that pass while it's paused aren't fired when it resumes +* `resume` — Let a paused scheduled prompt fire again, from its next time on +* `rm` — Remove a scheduled prompt from every agent it names + + + +## `swarmctl schedule add` + +Add a scheduled prompt and print it with its new id. + +Without `--every` it fires once. Without `--at` its first fire is now. + +**Usage:** `swarmctl schedule add [OPTIONS] --name --agent --body ` + +###### **Options:** + +* `--name ` — The prompt's title +* `--agent ` — Agent to fire it in. Repeatable; at least one +* `--body ` — The prompt, delivered to each agent verbatim +* `--at