swarm: named scheduled prompts for one or more agents, fired in each agent
swarm-controller keeps every scheduled prompt (id, name, agents, body,
anchor, interval, paused) as one document under one key of a new
`schedule-definitions` KV bucket, so each change is a single
compare-and-set and a prompt exists for all of its agents or none. It
writes each agent's key of a new `agent-schedules` bucket as the prompts
naming that agent, without names or other agents, after every change and
on a one-minute reconcile pass. It is the only writer of both, behind
GET/POST /api/schedules and PATCH/DELETE /api/schedules/{id}.
`swarmctl schedule add|list|edit|pause|resume|rm` drives those routes over
the controller socket.
An agent's own queue credential is granted one more subject, the direct
get of its own key in `agent-schedules`. A KV watch is not granted: its
consumer create subject carries no key, so it would cover every agent's.
hive-agent's new schedule ticker reads that key every 10 seconds and fires
due slots as the same `schedule:<id>` todo hive-c0re's worker pushes. Fire
times are anchor + k*interval; the last fired slot per schedule id is kept
in the state db, so a body edit or list refresh never moves or refires a
slot, downtime fires the latest missed slot once, a restart refires
nothing, a one-shot fires once, and paused slots are skipped.
Refs #4914
This commit is contained in:
atlas
mara
parent
84c89de71c
commit
a0b1846006
24 changed files with 2617 additions and 19 deletions
|
|
@ -82,6 +82,8 @@ declarations.
|
|||
[`swarm/secrets.md`](swarm/secrets.md) for where each file lives today.
|
||||
- **How does the secret store come up, who writes its grants, how's it
|
||||
sealed?** → [`swarm/bao.md`](swarm/bao.md).
|
||||
- **How does an agent get a prompt at set times, and why doesn't it fire
|
||||
twice after downtime?** → [`swarm/schedules.md`](swarm/schedules.md).
|
||||
|
||||
## Scheduler, CI, observability
|
||||
|
||||
|
|
|
|||
|
|
@ -21,6 +21,8 @@ grafana, the controller, authelia, …) →
|
|||
|
||||
- **control plane** — `swarm-controller`: hive directory, agent roster, job
|
||||
graph, agent creation. → [below](#swarm-controller)
|
||||
- **scheduled prompts** — named prompts for one or more agents at set times,
|
||||
kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md)
|
||||
- **swarm UI** — the operator's day-to-day surface, on the swarm apex,
|
||||
`admins` only. → [`ui.md`](ui.md)
|
||||
- **shared services** — one forge, homeserver, SSO, queue and metrics/logs
|
||||
|
|
|
|||
150
docs/swarm/schedules.md
Normal file
150
docs/swarm/schedules.md
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
# Scheduled prompts
|
||||
|
||||
A scheduled prompt hands one or more agents the same text at set times: once,
|
||||
or every interval from a start time. Each prompt has a name, its title. The
|
||||
swarm keeps every prompt, and each agent it names fires it in its own
|
||||
container, so a fire needs neither the hive nor the swarm controller to be up
|
||||
at that moment, only the agent.
|
||||
|
||||
Operators manage them with `swarmctl schedule` on the swarm controller's
|
||||
host. Agents can't create, change or remove one. For a one-off wake an agent
|
||||
sets itself, it uses `remind`, which stays inside its container.
|
||||
|
||||
## Managing them
|
||||
|
||||
```console
|
||||
# swarmctl schedule add --name standup --agent scribe --agent iris \
|
||||
--body "post the standup summary" --at 2026-10-05T09:00:00+02:00 --every 1d
|
||||
added schedule kqzpmwab
|
||||
kqzpmwab "standup" every 1d from 2026-10-05T07:00:00Z next 2026-10-05T07:00:00Z agents: iris, scribe | post the standup summary
|
||||
|
||||
# swarmctl schedule list
|
||||
# swarmctl schedule list --agent scribe
|
||||
# swarmctl schedule edit kqzpmwab --body "post the standup summary to the team room"
|
||||
# swarmctl schedule edit kqzpmwab --add-agent atlas --remove-agent iris
|
||||
# swarmctl schedule pause kqzpmwab
|
||||
# swarmctl schedule resume kqzpmwab
|
||||
# swarmctl schedule rm kqzpmwab
|
||||
```
|
||||
|
||||
- `--agent` names an agent to fire the prompt in; repeat it for more. Each
|
||||
must be an agent some hive's wanted state declares.
|
||||
- `--at` takes an RFC 3339 time with its offset, or unix seconds. Without it
|
||||
the first fire is now. A time in the past fires its latest past time once,
|
||||
as soon as each agent reads the prompt.
|
||||
- `--every` takes a count and a unit, `s`, `m`, `h` or `d`. Without it the
|
||||
prompt fires once.
|
||||
- `schedule edit` changes only the flags you pass and keeps the id. `--once`
|
||||
turns a recurring prompt into a one-shot. `--add-agent` and
|
||||
`--remove-agent` change who gets it; removing the last agent fails, so
|
||||
remove the prompt instead.
|
||||
- `schedule list` shows one row per prompt: id, name, cadence, start, next
|
||||
fire time, agents, and the first line of the body.
|
||||
|
||||
The full flag reference is [`swarmctl-cli.md`](../tools/swarmctl-cli.md).
|
||||
|
||||
## When a prompt fires
|
||||
|
||||
A prompt's fire times are its start time plus whole intervals. Each agent
|
||||
works them out from the prompt itself, so:
|
||||
|
||||
- **Editing the text or the name doesn't move them.** Neither does changing
|
||||
the agents. Only a new `--at` or `--every` does.
|
||||
- **An agent that was down or disconnected fires once.** When it's back, it
|
||||
fires the latest time it missed and then carries on at the next one. The
|
||||
earlier missed times don't fire.
|
||||
- **A restart refires nothing.** Each agent records the last time it fired
|
||||
for each prompt in its state database, and fires a time only when it's
|
||||
later than that. An agent removed from a prompt and added back keeps that
|
||||
record, so it doesn't refire a time it already fired.
|
||||
- **A one-shot fires once**, even though it stays in `schedule list` until
|
||||
you remove it. Moving its `--at` to a later time fires it again then.
|
||||
- **A paused prompt doesn't fire**, and the times that pass while it's paused
|
||||
don't fire when it resumes. An agent marks those times as passed while it
|
||||
watches the paused prompt; if the agent was down for the whole pause, it
|
||||
fires the latest time it missed once, as above.
|
||||
|
||||
A fire reaches the agent as a `schedule` loose end with the prompt's text,
|
||||
the same as a hive schedule's. The agent's turn loop wakes for it like any
|
||||
other loose end. If the agent hasn't dealt with the previous fire yet, the
|
||||
new fire replaces it rather than adding a second one; once the agent
|
||||
dismissed the previous one, the next fire opens a new one.
|
||||
|
||||
Each agent reads its prompts every 10 seconds, so a fire can land up to 10
|
||||
seconds after its time. A change reaches each agent within the same window,
|
||||
or within a minute if the controller's first attempt to pass it on failed.
|
||||
|
||||
## When a prompt doesn't fire
|
||||
|
||||
- **The agent has no queue credential of its own.** The agent reads its
|
||||
prompts with its own swarm-queue credential. An agent connected with its
|
||||
hive's shared client logs `connected with the hive's shared client, which
|
||||
has no read grant on the agent-schedules bucket` and fires nothing.
|
||||
- **The agent's first connect with its own credential failed.** The harness
|
||||
keeps the outcome of its first connect for the rest of its run: it falls
|
||||
back to the hive's shared client (the case above) or stays disconnected,
|
||||
and reads no prompts until it restarts.
|
||||
- **The agent hasn't read its prompts since it started.** It fires from the
|
||||
last list it read; until the first read succeeds it fires nothing. A read
|
||||
that fails later keeps the previous list.
|
||||
|
||||
<details><summary>Implementation</summary>
|
||||
|
||||
swarm-controller is the only client that writes either of the two KV buckets
|
||||
involved.
|
||||
|
||||
`schedule-definitions` holds every prompt as one JSON document under the key
|
||||
`all`. Agents hold no grant on this bucket. Each change re-reads the key and
|
||||
writes against its revision, so a prompt exists for every one of its agents or for
|
||||
none of them, and two changes at once don't lose one.
|
||||
|
||||
`agent-schedules` holds one key per agent: the prompts naming that agent,
|
||||
without their names or other agents.
|
||||
|
||||
```json
|
||||
{
|
||||
"schedules": [
|
||||
{
|
||||
"id": "kqzpmwab",
|
||||
"body": "post the standup summary",
|
||||
"anchor_unix": 1791183600,
|
||||
"interval_seconds": 86400,
|
||||
"paused": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`interval_seconds` is absent for a one-shot. After each change the controller
|
||||
rewrites the key of every agent the change named before or after, and once a
|
||||
minute it rewrites every agent's key, so a rewrite that failed converges. Ids
|
||||
are 8 lowercase letters, the same in every agent's key, never reused, and so
|
||||
never equal to a hive schedule's numeric id.
|
||||
|
||||
`swarmctl schedule` calls these routes over the controller's socket:
|
||||
|
||||
| method | path | body |
|
||||
| -------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `GET` | `/api/schedules` | —; answers every prompt, sorted by name then id |
|
||||
| `POST` | `/api/schedules` | `name`, `agents`, `body`, and optional `anchor_unix`, `interval_seconds`, `paused`; answers 201 with the prompt |
|
||||
| `PATCH` | `/api/schedules/<id>` | any of `name`, `body`, `anchor_unix`, `interval_seconds` (`null` for a one-shot), `paused`, `add_agents`, `remove_agents` |
|
||||
| `DELETE` | `/api/schedules/<id>` | —; answers 204 |
|
||||
|
||||
Each prompt they answer with has `id`, `name`, `agents` (sorted), `body`,
|
||||
`anchor_unix`, `interval_seconds` (`null` for a one-shot), `paused`, and
|
||||
`next_fire_unix`: the next time on the prompt's grid, as of the moment the
|
||||
controller answers, `null` when paused or a one-shot whose time has passed.
|
||||
|
||||
An agent's own queue credential grants exactly one subject in the swarm's
|
||||
JetStream: `$JS.API.DIRECT.GET.KV_agent-schedules.$KV.agent-schedules.<agent>`,
|
||||
the direct read of its own key. It can't read another agent's key, open or
|
||||
watch the bucket, or write to it. A KV watch would need a consumer, and a
|
||||
consumer create subject doesn't name a key, so granting one would let the
|
||||
agent read every key; the agent reads its key every 10 seconds instead.
|
||||
|
||||
The harness (`hive-agent`'s `schedule_timer`) keeps the last fired time per
|
||||
prompt id in the `schedule_fired_slots` table of its state database, and
|
||||
pushes each fire as the todo `schedule`/`schedule:<id>` with source
|
||||
`scheduled`, reopening an acknowledged one.
|
||||
|
||||
</details>
|
||||
|
|
@ -9,6 +9,13 @@ This document contains the help content for the `swarmctl` command-line program.
|
|||
* [`swarmctl agent create`↴](#swarmctl-agent-create)
|
||||
* [`swarmctl agent mint-identity`↴](#swarmctl-agent-mint-identity)
|
||||
* [`swarmctl agent mint-forge-token`↴](#swarmctl-agent-mint-forge-token)
|
||||
* [`swarmctl schedule`↴](#swarmctl-schedule)
|
||||
* [`swarmctl schedule add`↴](#swarmctl-schedule-add)
|
||||
* [`swarmctl schedule list`↴](#swarmctl-schedule-list)
|
||||
* [`swarmctl schedule edit`↴](#swarmctl-schedule-edit)
|
||||
* [`swarmctl schedule pause`↴](#swarmctl-schedule-pause)
|
||||
* [`swarmctl schedule resume`↴](#swarmctl-schedule-resume)
|
||||
* [`swarmctl schedule rm`↴](#swarmctl-schedule-rm)
|
||||
* [`swarmctl user`↴](#swarmctl-user)
|
||||
* [`swarmctl user add`↴](#swarmctl-user-add)
|
||||
* [`swarmctl user reset-password`↴](#swarmctl-user-reset-password)
|
||||
|
|
@ -27,6 +34,7 @@ swarm-level operator CLI
|
|||
###### **Subcommands:**
|
||||
|
||||
* `agent` — Manage agents across the swarm
|
||||
* `schedule` — Manage scheduled prompts
|
||||
* `user` — Manage subjects in the swarm's SSO provider
|
||||
* `forge` — Manage human accounts on the swarm's forge
|
||||
* `completions` — Generate a shell completion script for `swarmctl` and print it to stdout
|
||||
|
|
@ -129,6 +137,143 @@ Queues and returns, the same way `agent create` does — watch the swarm UI's jo
|
|||
|
||||
|
||||
|
||||
## `swarmctl schedule`
|
||||
|
||||
Manage scheduled prompts.
|
||||
|
||||
The swarm keeps each prompt, and every agent it names fires it in its own container. Fire times are the schedule's start time plus whole intervals: changing the body never moves them, and an agent that was down fires the latest missed one once.
|
||||
|
||||
**Usage:** `swarmctl schedule <COMMAND>`
|
||||
|
||||
###### **Subcommands:**
|
||||
|
||||
* `add` — Add a scheduled prompt and print it with its new id
|
||||
* `list` — List scheduled prompts, one row each, with their agents and next fire time
|
||||
* `edit` — Change a scheduled prompt
|
||||
* `pause` — Stop a scheduled prompt firing. Times that pass while it's paused aren't fired when it resumes
|
||||
* `resume` — Let a paused scheduled prompt fire again, from its next time on
|
||||
* `rm` — Remove a scheduled prompt from every agent it names
|
||||
|
||||
|
||||
|
||||
## `swarmctl schedule add`
|
||||
|
||||
Add a scheduled prompt and print it with its new id.
|
||||
|
||||
Without `--every` it fires once. Without `--at` its first fire is now.
|
||||
|
||||
**Usage:** `swarmctl schedule add [OPTIONS] --name <TEXT> --agent <AGENT> --body <TEXT>`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--name <TEXT>` — The prompt's title
|
||||
* `--agent <AGENT>` — Agent to fire it in. Repeatable; at least one
|
||||
* `--body <TEXT>` — The prompt, delivered to each agent verbatim
|
||||
* `--at <TIME>` — First fire time: an RFC 3339 time with its offset, such as `2026-10-05T09:00:00+02:00`, or unix seconds. Defaults to now
|
||||
* `--every <DURATION>` — Time between fires: a count with a unit of `s`, `m`, `h` or `d`, such as `1h` or `7d`. Leave out for a one-shot
|
||||
* `--paused` — Add it paused
|
||||
* `--controller-socket <PATH>` — swarm-controller's unix socket.
|
||||
|
||||
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
|
||||
|
||||
|
||||
|
||||
## `swarmctl schedule list`
|
||||
|
||||
List scheduled prompts, one row each, with their agents and next fire time
|
||||
|
||||
**Usage:** `swarmctl schedule list [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--agent <AGENT>` — Show only the prompts that fire in this agent
|
||||
* `--controller-socket <PATH>` — swarm-controller's unix socket.
|
||||
|
||||
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
|
||||
|
||||
|
||||
|
||||
## `swarmctl schedule edit`
|
||||
|
||||
Change a scheduled prompt.
|
||||
|
||||
Only the flags given change, and the id stays. `--at` and `--every` move the fire times; `--name`, `--body` and the agent flags don't.
|
||||
|
||||
**Usage:** `swarmctl schedule edit [OPTIONS] <ID>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<ID>` — The prompt's id, as `schedule list` shows it
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--name <TEXT>` — New title
|
||||
* `--body <TEXT>` — New prompt text
|
||||
* `--at <TIME>` — New first fire time, in the form `schedule add --at` takes
|
||||
* `--every <DURATION>` — New time between fires, in the form `schedule add --every` takes
|
||||
* `--once` — Make it a one-shot
|
||||
* `--add-agent <AGENT>` — Agent to start firing it in. Repeatable
|
||||
* `--remove-agent <AGENT>` — Agent to stop firing it in. Repeatable. Fails if the agent isn't a target, or if no agent would remain
|
||||
* `--controller-socket <PATH>` — swarm-controller's unix socket.
|
||||
|
||||
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
|
||||
|
||||
|
||||
|
||||
## `swarmctl schedule pause`
|
||||
|
||||
Stop a scheduled prompt firing. Times that pass while it's paused aren't fired when it resumes
|
||||
|
||||
**Usage:** `swarmctl schedule pause [OPTIONS] <ID>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<ID>` — The prompt's id, as `schedule list` shows it
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--controller-socket <PATH>` — swarm-controller's unix socket.
|
||||
|
||||
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
|
||||
|
||||
|
||||
|
||||
## `swarmctl schedule resume`
|
||||
|
||||
Let a paused scheduled prompt fire again, from its next time on
|
||||
|
||||
**Usage:** `swarmctl schedule resume [OPTIONS] <ID>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<ID>` — The prompt's id, as `schedule list` shows it
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--controller-socket <PATH>` — swarm-controller's unix socket.
|
||||
|
||||
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
|
||||
|
||||
|
||||
|
||||
## `swarmctl schedule rm`
|
||||
|
||||
Remove a scheduled prompt from every agent it names
|
||||
|
||||
**Usage:** `swarmctl schedule rm [OPTIONS] <ID>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<ID>` — The prompt's id, as `schedule list` shows it
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--controller-socket <PATH>` — swarm-controller's unix socket.
|
||||
|
||||
Supplied by the nix module that installs this binary, from the same `socketPath` option the daemon binds; falls back to `SWARM_CONTROLLER_SOCKET`.
|
||||
|
||||
|
||||
|
||||
## `swarmctl user`
|
||||
|
||||
Manage subjects in the swarm's SSO provider
|
||||
|
|
|
|||
Loading…
Reference in a new issue