swarm-archive: back up every agent's state to an append-only restic repository per agent
swarm-archive serves restic repositories with rclone, append-only, one login and one repository per agent. swarm-controller writes each placed agent's login and restic password to the store, replaces them on a move and removes them when the agent is destroyed. Every hive backs its agents up hourly under its own store identity. The archive host prunes by retention.
This commit is contained in:
atlas
parent
409e7ee5a0
commit
306aae0378
5 changed files with 2214 additions and 5 deletions
|
|
@ -43,6 +43,8 @@ grafana, the controller, authelia, …) →
|
|||
CLI agents use. → [`forge.md`](forge.md)
|
||||
- **matrix** — the swarm's homeserver, several accounts per agent, and
|
||||
federation. → [`matrix.md`](matrix.md)
|
||||
- **backup archive** — where the hives back up each agent's state, how long it
|
||||
stays and how to check it. → [`archive.md`](archive.md)
|
||||
- **knowledge repo** — the reference docs every agent reads, and how to add
|
||||
one. → [`knowledge.md`](knowledge.md)
|
||||
- **`swarmctl`** — the swarm operator's CLI. → [`swarmctl-cli.md`](swarmctl-cli.md)
|
||||
|
|
|
|||
125
docs/2-handbook/swarm/archive.md
Normal file
125
docs/2-handbook/swarm/archive.md
Normal file
|
|
@ -0,0 +1,125 @@
|
|||
# Backup archive
|
||||
|
||||
The archive keeps a backup of every agent's state. Each hive backs its agents
|
||||
up to it every hour, and it keeps the last few days. This page covers what the
|
||||
hive saves, where it goes, how long it stays and how to check that it works.
|
||||
Restoring an agent from a backup isn't built yet.
|
||||
|
||||
## What's in the backup
|
||||
|
||||
For each agent a hive places, the hive saves the agent's `state/`, `claude/` and
|
||||
`harness/` directories. `claude/` holds the agent's login, so a restored agent
|
||||
keeps its session. Caches and `work/` stay out.
|
||||
|
||||
Two options on the hive change that:
|
||||
|
||||
- `deploy.hive-controller.archive.paths` lists the entries of the agent's
|
||||
directory to save. The hive skips an entry the agent doesn't have.
|
||||
- `deploy.hive-controller.archive.scope = "subvolume"` saves the whole agent
|
||||
directory instead, `work/` included. Add `archive.excludes` patterns to keep
|
||||
build output out.
|
||||
|
||||
On a btrfs hive the hive saves a read-only snapshot of the agent's subvolume, so
|
||||
it copies the databases at one instant. Otherwise it saves the directory in
|
||||
place.
|
||||
|
||||
## Where it goes
|
||||
|
||||
The archive runs as `rclone serve restic` on the host that sets
|
||||
`deploy.archive.enable`, behind the gateway at `archive.<swarm domain>`. Every
|
||||
hive finds it by that name, so which host runs it makes no difference to them.
|
||||
`allSwarmServices` turns it on wherever the host has the archive's store identity,
|
||||
as for the binary cache.
|
||||
|
||||
Each agent has its own [restic](https://restic.net) repository at
|
||||
`https://archive.<swarm domain>/<agent>/`, under `deploy.archive.dataDir`
|
||||
(default `/var/lib/swarm-archive`) on that host. restic encrypts it with a
|
||||
password of its own for that agent.
|
||||
|
||||
Two properties protect the backups from a compromised hive:
|
||||
|
||||
- **One login per agent.** `swarm-controller` creates a login for each agent
|
||||
when it places the agent. The login reaches that agent's repository and no
|
||||
other. A hive holds the logins of the agents it places and no others. When an
|
||||
agent moves, the controller gives the new hive a new password and deletes the
|
||||
old hive's copy, so the old hive can't reach the repository any more.
|
||||
- **Append-only.** A client can add snapshots and nothing else: no delete, no
|
||||
overwrite. The archive host removes old snapshots.
|
||||
|
||||
## How long it stays
|
||||
|
||||
The archive host prunes every repository once a day. A snapshot survives if
|
||||
either rule keeps it:
|
||||
|
||||
| option | default | keeps |
|
||||
| ------------------------------------- | ------- | --------------------------------- |
|
||||
| `deploy.archive.retention.keepWithin` | `3d` | every snapshot younger than this |
|
||||
| `deploy.archive.retention.keepLast` | `3` | the newest this many, however old |
|
||||
|
||||
The age rule limits what a compromised hive can do by flooding the archive with
|
||||
fake snapshots: they can't push the last three days out. Set `keepWithin` to
|
||||
`null` to leave that rule off. `deploy.archive.pruneSchedule` sets when the
|
||||
prune runs (default `daily`).
|
||||
|
||||
Because the archive host prunes, it reads every agent's restic password from
|
||||
the secret store and can decrypt every backup. Treat that host as part of the
|
||||
swarm's trust root.
|
||||
|
||||
## Schedule
|
||||
|
||||
`deploy.hive-controller.archive.schedule` is a systemd calendar expression
|
||||
(default `hourly`). restic sends only what changed, so a run over an idle agent
|
||||
is small. The first backup of a large agent sends everything.
|
||||
|
||||
## Check that it works
|
||||
|
||||
On a hive, `systemctl status hive-archive-backup` shows the last run, and
|
||||
`journalctl -u hive-archive-backup` lists every agent it backed up or failed to
|
||||
back up. `systemctl list-timers hive-archive-backup` shows the next run.
|
||||
|
||||
To see an agent's snapshots, ask the archive with that agent's password, which
|
||||
the secret store holds at `swarm/hives/<hive>/archive/<agent>` (fields
|
||||
`rest-password` and `restic-password`):
|
||||
|
||||
```sh
|
||||
RESTIC_REPOSITORY="rest:https://archive.<swarm domain>/<agent>/" \
|
||||
RESTIC_REST_USERNAME=<agent> RESTIC_REST_PASSWORD=<rest-password> \
|
||||
restic snapshots
|
||||
```
|
||||
|
||||
restic asks for the `restic-password`. On the archive host, `systemctl status
|
||||
swarm-archive swarm-archive-users swarm-archive-prune` shows the server, the
|
||||
users file refresh and the last prune.
|
||||
|
||||
## When it fails
|
||||
|
||||
- **The `archive backup failed` alert.** A hive's run failed for an agent or
|
||||
failed to log in. The journal line `archive backup failed: <agent>` names the
|
||||
agent, and the lines above it say why. The alert waits 30 minutes, so an
|
||||
agent placed a few minutes ago doesn't trip it while its login is on its way.
|
||||
The next hourly run retries.
|
||||
- **`swarm-controller has granted this hive no archive credentials`.** The
|
||||
hive has agents and the store holds no login for any of them. Either
|
||||
`swarm-controller` doesn't have the archive yet (update it), or its archive
|
||||
pass fails: look for `archive:` lines in its journal.
|
||||
- **The `archive prune failed` alert.** The daily prune couldn't apply the
|
||||
retention to one repository, for example because a backup held its lock for
|
||||
longer than ten minutes. The next prune retries.
|
||||
- **A swarm without an archive.** If no host sets `deploy.archive.enable`, every
|
||||
hive's run fails. Set `deploy.hive-controller.archive.enable = false` on the
|
||||
hives until one does.
|
||||
|
||||
## Mixed versions
|
||||
|
||||
A hive that doesn't have the backup unit yet doesn't back up, and nothing else
|
||||
changes. A `swarm-controller` without the archive pass writes no logins, so
|
||||
hives with the unit fail with the message above until it updates. The archive
|
||||
host admits only the users the controller wrote, so until then it admits nobody.
|
||||
|
||||
## What lives where
|
||||
|
||||
Everything secret lives in the swarm secret store, and the archive keeps none
|
||||
on disk. The archive host renders its users file (password hashes only) into
|
||||
`/run/swarm-archive-auth/`, which is tmpfs. Hives read their restic password
|
||||
when restic asks for it. The paths and who writes and reads them are in the
|
||||
[credentials reference](credentials-reference.md).
|
||||
|
|
@ -57,6 +57,9 @@ the rest of the cell says how.
|
|||
| `swarm/services/attic/public-key` | `swarm-bao-attic-tokens` on the cache's host, when none is stored and again whenever the cache reports another key (the key `attic cache info` prints); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (`<name>:<base64>`) | `swarm-controller`, which copies it to each hive | ✅ on the cache's host, the unit's next run replaces any other value, a hand edit included, with the cache's key; with an external cache, replace it yourself | ✅ copied to the hives every 5 minutes |
|
||||
| `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}` | `swarm-controller`, from the four `swarm/services/attic/` leaves, every 5 minutes, and before each deploy to the hive; it skips a leaf whose source holds nothing, and a copy whose source is gone stays | the hive (`hive-c0re`) | ✅ a changed source is written over the copy in the next pass | ✅ read every 5 minutes |
|
||||
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
|
||||
| `swarm/hives/<hive>/archive/<agent>` (fields `rest-password`, `restic-password`) | `swarm-controller`, for every agent a hive places, every 5 minutes: a new random REST password when the agent is new on that hive, the agent's restic password from the leaf below; the copy on a hive that no longer places the agent is deleted | the agent's hive: `hive-archive-backup`, under the hive's own store identity | ✅ a move to another hive writes a new REST password, which locks the old hive out | ✅ read at every backup run |
|
||||
| `swarm/services/archive/users/<agent>` (field `htpasswd`) | `swarm-controller`, with the REST password above: its `{SHA}` hash, never the password; deleted when no hive places the agent | `swarm-archive-users` on the archive's host, which renders the users file into `/run` | ✅ rewritten with the password | ✅ re-read every 2 minutes; the server notices the new file without a restart |
|
||||
| `swarm/services/archive/restic/<agent>` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune |
|
||||
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
|
||||
|
||||
|
|
@ -181,6 +184,7 @@ login minter's, the forge host's and Grafana's, which reach more than one:
|
|||
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
|
||||
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
|
||||
| `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` |
|
||||
| `swarm-archive-users` and `swarm-archive-prune` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*` and `swarm/services/archive/restic/*`, lists `swarm/services/archive/users/` |
|
||||
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
|
||||
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
|
||||
|
||||
|
|
|
|||
|
|
@ -442,7 +442,7 @@ each is enough and their subjects are the flat
|
|||
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
||||
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
|
||||
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
|
||||
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`,
|
||||
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`, `deploy.bao.archiveCommonName`,
|
||||
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
|
||||
The last three are the writers, and the last is the one leaf two units share:
|
||||
the forge's host holds one store identity for the token it publishes and the
|
||||
|
|
|
|||
|
|
@ -119,6 +119,7 @@ look before you switch them on.
|
|||
| CI runner | Runs Forgejo Actions for pull requests. Heavy. | `deploy.ci-runner.enable` | [CI runner](#ci-runner) |
|
||||
| Builder | Evaluates and builds agent config PRs. Heavy. | `deploy.builder.enable` | [Builder](#builder) |
|
||||
| Binary cache | Serves built closures to the hives. | `deploy.attic.enable` | [Binary cache](#binary-cache) |
|
||||
| Backup archive | Keeps every agent's state backups, one repository each. | `deploy.archive.enable` | [Backup archive](archive.md) |
|
||||
| CI metrics | Records CI run times in the metrics store. | `deploy.ci-metrics.enable` | [CI metrics](#ci-metrics) |
|
||||
| Review classifier | Checks PRs against a repo's review rules. | `deploy.review-classifier.enable` | [Review classifier](#review-classifier) |
|
||||
| Claude login minter | Runs the **log in** button on an agent's card. | `deploy.claude-login.enable` | [Claude login minter](#claude-login-minter) |
|
||||
|
|
|
|||
Loading…
Reference in a new issue