91f36f39d2
The archive host creates a ready marker in the store and swarm-controller writes no credentials before it exists, so a swarm without an archive, and a mixed-version swarm, leave every hive backup a logged no-op. A hive with no declaration no longer counts as placing nothing: its copies and its agents' user lines stay. The restic password is pinned by tests across a revoke, a re-placement and a move.
132 lines
6.2 KiB
Markdown
132 lines
6.2 KiB
Markdown
# Backup archive
|
|
|
|
The archive keeps a backup of every agent's state. Each hive backs its agents
|
|
up to it every hour, and it keeps the last few days. This page covers what the
|
|
hive saves, where it goes, how long it stays and how to check that it works.
|
|
Restoring an agent from a backup isn't built yet.
|
|
|
|
## What's in the backup
|
|
|
|
For each agent a hive places, the hive saves the agent's `state/`, `claude/` and
|
|
`harness/` directories. `claude/` holds the agent's login, so a restored agent
|
|
keeps its session. Caches and `work/` stay out.
|
|
|
|
Two options on the hive change that:
|
|
|
|
- `deploy.hive-controller.archive.paths` lists the entries of the agent's
|
|
directory to save. The hive skips an entry the agent doesn't have.
|
|
- `deploy.hive-controller.archive.scope = "subvolume"` saves the whole agent
|
|
directory instead, `work/` included. Add `archive.excludes` patterns to keep
|
|
build output out.
|
|
|
|
On a btrfs hive the hive saves a read-only snapshot of the agent's subvolume, so
|
|
it copies the databases at one instant. Otherwise it saves the directory in
|
|
place.
|
|
|
|
## Where it goes
|
|
|
|
The archive runs as `rclone serve restic` on the host that sets
|
|
`deploy.archive.enable`, behind the gateway at `archive.<swarm domain>`. Every
|
|
hive finds it by that name, so which host runs it makes no difference to them.
|
|
`allSwarmServices` turns it on wherever the host has the archive's store identity,
|
|
as for the binary cache.
|
|
|
|
Each agent has its own [restic](https://restic.net) repository at
|
|
`https://archive.<swarm domain>/<agent>/`, under `deploy.archive.dataDir`
|
|
(default `/var/lib/swarm-archive`) on that host. restic encrypts it with a
|
|
password of its own for that agent.
|
|
|
|
Two properties protect the backups from a compromised hive:
|
|
|
|
- **One login per agent.** `swarm-controller` creates a login for each agent
|
|
when it places the agent. The login reaches that agent's repository and no
|
|
other. A hive holds the logins of the agents it places and no others. When an
|
|
agent moves, the controller gives the new hive a new password and deletes the
|
|
old hive's copy, so the old hive can't reach the repository any more.
|
|
Only that login rotates. The controller creates an agent's restic password
|
|
once, when it first places the agent, and a move, a destroy or a re-placement never
|
|
changes it, because it's the key of the repository.
|
|
- **Append-only.** A client can add snapshots and nothing else: no delete, no
|
|
overwrite. The archive host removes old snapshots.
|
|
|
|
## How long it stays
|
|
|
|
The archive host prunes every repository once a day. A snapshot survives if
|
|
either rule keeps it:
|
|
|
|
| option | default | keeps |
|
|
| ------------------------------------- | ------- | --------------------------------- |
|
|
| `deploy.archive.retention.keepWithin` | `3d` | every snapshot younger than this |
|
|
| `deploy.archive.retention.keepLast` | `3` | the newest this many, however old |
|
|
|
|
The age rule limits what a compromised hive can do by flooding the archive with
|
|
fake snapshots: they can't push the last three days out. Set `keepWithin` to
|
|
`null` to leave that rule off. `deploy.archive.pruneSchedule` sets when the
|
|
prune runs (default `daily`).
|
|
|
|
Because the archive host prunes, it reads every agent's restic password from
|
|
the secret store and can decrypt every backup. Treat that host as part of the
|
|
swarm's trust root.
|
|
|
|
## Schedule
|
|
|
|
`deploy.hive-controller.archive.schedule` is a systemd calendar expression
|
|
(default `hourly`). restic sends only what changed, so a run over an idle agent
|
|
is small. The first backup of a large agent sends everything.
|
|
|
|
## Check that it works
|
|
|
|
On a hive, `systemctl status hive-archive-backup` shows the last run, and
|
|
`journalctl -u hive-archive-backup` lists every agent it backed up or failed to
|
|
back up. `systemctl list-timers hive-archive-backup` shows the next run.
|
|
|
|
To see an agent's snapshots, ask the archive with that agent's password, which
|
|
the secret store holds at `swarm/hives/<hive>/archive/<agent>` (fields
|
|
`rest-password` and `restic-password`):
|
|
|
|
```sh
|
|
RESTIC_REPOSITORY="rest:https://archive.<swarm domain>/<agent>/" \
|
|
RESTIC_REST_USERNAME=<agent> RESTIC_REST_PASSWORD=<rest-password> \
|
|
restic snapshots
|
|
```
|
|
|
|
restic asks for the `restic-password`. On the archive host, `systemctl status
|
|
swarm-archive swarm-archive-users swarm-archive-prune` shows the server, the
|
|
users file refresh and the last prune.
|
|
|
|
## When it fails
|
|
|
|
- **The `archive backup failed` alert.** A hive's run failed for an agent or
|
|
failed to log in. The journal line `archive backup failed: <agent>` names the
|
|
agent, and the lines above it say why. The alert waits 30 minutes, so an
|
|
agent placed a few minutes ago doesn't trip it while its login is on its way.
|
|
The next hourly run retries.
|
|
- **`swarm-controller has granted this hive no archive credentials`.** Not a
|
|
failure: the hive logs it and succeeds. It means the store holds no login for
|
|
any agent here, because no archive is running yet, `swarm-controller` is
|
|
older than the archive, or its archive pass fails (look for `archive:` lines
|
|
in its journal). If an archive is running and this persists, check the
|
|
controller.
|
|
- **The `archive prune failed` alert.** The daily prune couldn't apply the
|
|
retention to one repository, for example because a backup held its lock for
|
|
longer than ten minutes. The next prune retries.
|
|
- **A swarm without an archive.** Nothing fails and nothing alerts. The archive
|
|
host announces itself in the secret store when it starts, and
|
|
`swarm-controller` writes no logins before that, so every hive's run finds
|
|
nothing to back up and says so in its journal.
|
|
|
|
## Mixed versions
|
|
|
|
A hive that doesn't have the backup unit yet doesn't back up, and nothing else
|
|
changes. A `swarm-controller` without the archive pass writes no logins, so
|
|
hives with the unit log that they have no credentials and succeed until it
|
|
updates. The archive host admits only the users the controller wrote, so until
|
|
then it admits nobody.
|
|
|
|
## What lives where
|
|
|
|
Everything secret lives in the swarm secret store, and the archive keeps none
|
|
on disk. The archive host renders its users file (password hashes only) into
|
|
`/run/swarm-archive-auth/`, which is tmpfs. Hives read their restic password
|
|
when restic asks for it. The paths and who writes and reads them are in the
|
|
[credentials reference](credentials-reference.md).
|