306aae0378
swarm-archive serves restic repositories with rclone, append-only, one login and one repository per agent. swarm-controller writes each placed agent's login and restic password to the store, replaces them on a move and removes them when the agent is destroyed. Every hive backs its agents up hourly under its own store identity. The archive host prunes by retention.
125 lines
5.8 KiB
Markdown
125 lines
5.8 KiB
Markdown
# Backup archive
|
|
|
|
The archive keeps a backup of every agent's state. Each hive backs its agents
|
|
up to it every hour, and it keeps the last few days. This page covers what the
|
|
hive saves, where it goes, how long it stays and how to check that it works.
|
|
Restoring an agent from a backup isn't built yet.
|
|
|
|
## What's in the backup
|
|
|
|
For each agent a hive places, the hive saves the agent's `state/`, `claude/` and
|
|
`harness/` directories. `claude/` holds the agent's login, so a restored agent
|
|
keeps its session. Caches and `work/` stay out.
|
|
|
|
Two options on the hive change that:
|
|
|
|
- `deploy.hive-controller.archive.paths` lists the entries of the agent's
|
|
directory to save. The hive skips an entry the agent doesn't have.
|
|
- `deploy.hive-controller.archive.scope = "subvolume"` saves the whole agent
|
|
directory instead, `work/` included. Add `archive.excludes` patterns to keep
|
|
build output out.
|
|
|
|
On a btrfs hive the hive saves a read-only snapshot of the agent's subvolume, so
|
|
it copies the databases at one instant. Otherwise it saves the directory in
|
|
place.
|
|
|
|
## Where it goes
|
|
|
|
The archive runs as `rclone serve restic` on the host that sets
|
|
`deploy.archive.enable`, behind the gateway at `archive.<swarm domain>`. Every
|
|
hive finds it by that name, so which host runs it makes no difference to them.
|
|
`allSwarmServices` turns it on wherever the host has the archive's store identity,
|
|
as for the binary cache.
|
|
|
|
Each agent has its own [restic](https://restic.net) repository at
|
|
`https://archive.<swarm domain>/<agent>/`, under `deploy.archive.dataDir`
|
|
(default `/var/lib/swarm-archive`) on that host. restic encrypts it with a
|
|
password of its own for that agent.
|
|
|
|
Two properties protect the backups from a compromised hive:
|
|
|
|
- **One login per agent.** `swarm-controller` creates a login for each agent
|
|
when it places the agent. The login reaches that agent's repository and no
|
|
other. A hive holds the logins of the agents it places and no others. When an
|
|
agent moves, the controller gives the new hive a new password and deletes the
|
|
old hive's copy, so the old hive can't reach the repository any more.
|
|
- **Append-only.** A client can add snapshots and nothing else: no delete, no
|
|
overwrite. The archive host removes old snapshots.
|
|
|
|
## How long it stays
|
|
|
|
The archive host prunes every repository once a day. A snapshot survives if
|
|
either rule keeps it:
|
|
|
|
| option | default | keeps |
|
|
| ------------------------------------- | ------- | --------------------------------- |
|
|
| `deploy.archive.retention.keepWithin` | `3d` | every snapshot younger than this |
|
|
| `deploy.archive.retention.keepLast` | `3` | the newest this many, however old |
|
|
|
|
The age rule limits what a compromised hive can do by flooding the archive with
|
|
fake snapshots: they can't push the last three days out. Set `keepWithin` to
|
|
`null` to leave that rule off. `deploy.archive.pruneSchedule` sets when the
|
|
prune runs (default `daily`).
|
|
|
|
Because the archive host prunes, it reads every agent's restic password from
|
|
the secret store and can decrypt every backup. Treat that host as part of the
|
|
swarm's trust root.
|
|
|
|
## Schedule
|
|
|
|
`deploy.hive-controller.archive.schedule` is a systemd calendar expression
|
|
(default `hourly`). restic sends only what changed, so a run over an idle agent
|
|
is small. The first backup of a large agent sends everything.
|
|
|
|
## Check that it works
|
|
|
|
On a hive, `systemctl status hive-archive-backup` shows the last run, and
|
|
`journalctl -u hive-archive-backup` lists every agent it backed up or failed to
|
|
back up. `systemctl list-timers hive-archive-backup` shows the next run.
|
|
|
|
To see an agent's snapshots, ask the archive with that agent's password, which
|
|
the secret store holds at `swarm/hives/<hive>/archive/<agent>` (fields
|
|
`rest-password` and `restic-password`):
|
|
|
|
```sh
|
|
RESTIC_REPOSITORY="rest:https://archive.<swarm domain>/<agent>/" \
|
|
RESTIC_REST_USERNAME=<agent> RESTIC_REST_PASSWORD=<rest-password> \
|
|
restic snapshots
|
|
```
|
|
|
|
restic asks for the `restic-password`. On the archive host, `systemctl status
|
|
swarm-archive swarm-archive-users swarm-archive-prune` shows the server, the
|
|
users file refresh and the last prune.
|
|
|
|
## When it fails
|
|
|
|
- **The `archive backup failed` alert.** A hive's run failed for an agent or
|
|
failed to log in. The journal line `archive backup failed: <agent>` names the
|
|
agent, and the lines above it say why. The alert waits 30 minutes, so an
|
|
agent placed a few minutes ago doesn't trip it while its login is on its way.
|
|
The next hourly run retries.
|
|
- **`swarm-controller has granted this hive no archive credentials`.** The
|
|
hive has agents and the store holds no login for any of them. Either
|
|
`swarm-controller` doesn't have the archive yet (update it), or its archive
|
|
pass fails: look for `archive:` lines in its journal.
|
|
- **The `archive prune failed` alert.** The daily prune couldn't apply the
|
|
retention to one repository, for example because a backup held its lock for
|
|
longer than ten minutes. The next prune retries.
|
|
- **A swarm without an archive.** If no host sets `deploy.archive.enable`, every
|
|
hive's run fails. Set `deploy.hive-controller.archive.enable = false` on the
|
|
hives until one does.
|
|
|
|
## Mixed versions
|
|
|
|
A hive that doesn't have the backup unit yet doesn't back up, and nothing else
|
|
changes. A `swarm-controller` without the archive pass writes no logins, so
|
|
hives with the unit fail with the message above until it updates. The archive
|
|
host admits only the users the controller wrote, so until then it admits nobody.
|
|
|
|
## What lives where
|
|
|
|
Everything secret lives in the swarm secret store, and the archive keeps none
|
|
on disk. The archive host renders its users file (password hashes only) into
|
|
`/run/swarm-archive-auth/`, which is tmpfs. Hives read their restic password
|
|
when restic asks for it. The paths and who writes and reads them are in the
|
|
[credentials reference](credentials-reference.md).
|