86c5dac89d
- swarm-archive.nix, lib/archive-server.nix: deploy.archive.backend and deploy.archive.s3.* options; the s3 server and the prune read the key from the store into rclone's environment; the prune lists and addresses repositories through rclone instead of a local directory - swarm-bao.nix: the archive reads swarm/services/archive/s3/* - docs: archive.md "External S3" section, two credentials-reference rows - module-eval: builder.nix cases for local and s3, the missing-setting assertion; bao-grants.nix gains the s3 read grant - vm test: second node with a fake rclone serve s3 whose key is generated at test time
190 lines
9.7 KiB
Markdown
190 lines
9.7 KiB
Markdown
# Backup archive
|
|
|
|
The archive keeps a backup of every agent's state. Each hive backs its agents
|
|
up to it every hour, and it keeps the last few days. This page covers what the
|
|
hive saves, where it goes, how long it stays and how to check that it works.
|
|
Restoring an agent from a backup isn't built yet.
|
|
|
|
## What's in the backup
|
|
|
|
For each agent a hive places, the hive saves the agent's `state/`, `claude/` and
|
|
`harness/` directories. `claude/` holds the agent's login, so a restored agent
|
|
keeps its session. Caches and `work/` stay out.
|
|
|
|
Two options on the hive change that:
|
|
|
|
- `deploy.hive-controller.archive.paths` lists the entries of the agent's
|
|
directory to save. The hive skips an entry the agent doesn't have.
|
|
- `deploy.hive-controller.archive.scope = "subvolume"` saves the whole agent
|
|
directory instead, `work/` included. Add `archive.excludes` patterns to keep
|
|
build output out.
|
|
|
|
On a btrfs hive the hive saves a read-only snapshot of the agent's subvolume, so
|
|
it copies the databases at one instant. Otherwise it saves the directory in
|
|
place.
|
|
|
|
## Where it goes
|
|
|
|
The archive runs as `rclone serve restic` on the host that sets
|
|
`deploy.archive.enable`, behind the gateway at `archive.<swarm domain>`. Every
|
|
hive finds it by that name, so which host runs it makes no difference to them.
|
|
`allSwarmServices` turns it on wherever the host has the archive's store identity,
|
|
as for the binary cache.
|
|
|
|
Each agent has its own [restic](https://restic.net) repository at
|
|
`https://archive.<swarm domain>/<agent>/`, under `deploy.archive.dataDir`
|
|
(default `/var/lib/swarm-archive`) on that host, or in a bucket of an
|
|
[external S3](#external-s3). restic encrypts it with a password of its own for
|
|
that agent.
|
|
|
|
Two properties protect the backups from a compromised hive:
|
|
|
|
- **One login per agent.** `swarm-controller` creates a login for each agent
|
|
when it places the agent. The login reaches that agent's repository and no
|
|
other. A hive holds the logins of the agents it places and no others. When an
|
|
agent moves, the controller gives the new hive a new password and deletes the
|
|
old hive's copy, so the old hive can't reach the repository any more.
|
|
Only that login rotates. The controller creates an agent's restic password
|
|
once, when it first places the agent, and a move, a destroy or a re-placement never
|
|
changes it, because it's the key of the repository.
|
|
- **Append-only.** A client can add snapshots and nothing else: no delete, no
|
|
overwrite. The archive host removes old snapshots.
|
|
|
|
## External S3
|
|
|
|
By default the repositories sit on the archive host's disk. With
|
|
`deploy.archive.backend = "s3"` they sit in a bucket of an S3 service you run
|
|
or rent instead, and the host keeps nothing. Hives don't change: they still
|
|
talk to `archive.<swarm domain>` with their per-agent logins, and the archive
|
|
is still append-only. Only the archive holds the S3 key, so a compromised hive
|
|
can't reach the bucket directly.
|
|
|
|
| option | default | meaning |
|
|
| ---------------------------- | ------- | ----------------------------------------------------------------- |
|
|
| `deploy.archive.backend` | `local` | `local` or `s3` |
|
|
| `deploy.archive.s3.endpoint` | `null` | the service's URL, scheme included; required |
|
|
| `deploy.archive.s3.bucket` | `null` | the bucket; required, and it must exist |
|
|
| `deploy.archive.s3.prefix` | `""` | a path in the bucket; the repositories sit at `<prefix>/<agent>/` |
|
|
| `deploy.archive.s3.region` | `null` | the bucket's region, for services that need one |
|
|
| `deploy.archive.s3.provider` | `Other` | rclone's `--s3-provider`, for example `AWS` |
|
|
|
|
Evaluation fails if `backend` is `s3` and the endpoint or the bucket is unset.
|
|
|
|
The service needs one bucket and one access key that can list, read, write
|
|
and delete objects in it. Scope the key to the bucket. The archive never
|
|
creates a bucket, so the key needs no permission to. The prune removes old
|
|
data, which takes delete rights. Hives can't delete: the archive refuses it
|
|
before the request reaches the bucket.
|
|
|
|
The key lives in the secret store, like every other archive secret. Write it
|
|
before you set `backend = "s3"`; the server doesn't start without it. Read it
|
|
from a file or the terminal so it doesn't land in your shell history:
|
|
|
|
```sh
|
|
bao kv put secret/swarm/services/archive/s3/access-key-id value=-
|
|
bao kv put secret/swarm/services/archive/s3/secret-access-key value=-
|
|
```
|
|
|
|
Each command reads the value from standard input. The store's UI writes the
|
|
same two leaves, field `value`. The archive's host reads them as it starts and
|
|
passes them to rclone in its environment. They never reach a file there, so
|
|
a changed key takes effect at the next restart of `swarm-archive`. The daily
|
|
prune reads them on every run. If the server can't read them, it fails to
|
|
start and systemd retries every five seconds until the store answers.
|
|
|
|
Changing `backend` doesn't move the data. Before you switch an archive that
|
|
holds backups, copy them with `rclone copy <dataDir>/repos :s3:<bucket>/<prefix>`
|
|
using a rclone configured for the bucket, or start with empty repositories and
|
|
let the next hourly runs fill them. The old directory stays on the host until
|
|
you remove it.
|
|
|
|
## How long it stays
|
|
|
|
The archive host prunes every repository once a day. A snapshot survives if
|
|
either rule keeps it:
|
|
|
|
| option | default | keeps |
|
|
| ------------------------------------- | ------- | --------------------------------- |
|
|
| `deploy.archive.retention.keepWithin` | `3d` | every snapshot younger than this |
|
|
| `deploy.archive.retention.keepLast` | `3` | the newest this many, however old |
|
|
|
|
The age rule limits what a compromised hive can do by flooding the archive with
|
|
fake snapshots: they can't push the last three days out. restic ranks snapshots
|
|
by their date, and a client chooses a snapshot's date, so a client could also
|
|
date snapshots years ahead to fill the `keepLast` slots. Before each prune the
|
|
archive host forgets every snapshot dated more than an hour ahead of its own
|
|
clock and logs `archive prune warning: <agent> has snapshots dated after now`.
|
|
A line like that means a hive or its credentials are misbehaving. Set
|
|
`keepWithin` to `null` to leave the age rule off. `deploy.archive.pruneSchedule`
|
|
sets when the prune runs (default `daily`).
|
|
|
|
Because the archive host prunes, it reads every agent's restic password from
|
|
the secret store and can decrypt every backup. Treat that host as part of the
|
|
swarm's trust root.
|
|
|
|
## Schedule
|
|
|
|
`deploy.hive-controller.archive.schedule` is a systemd calendar expression
|
|
(default `hourly`). restic sends only what changed, so a run over an idle agent
|
|
is small. The first backup of a large agent sends everything.
|
|
|
|
## Check that it works
|
|
|
|
On a hive, `systemctl status hive-archive-backup` shows the last run, and
|
|
`journalctl -u hive-archive-backup` lists every agent it backed up or failed to
|
|
back up. `systemctl list-timers hive-archive-backup` shows the next run.
|
|
|
|
To see an agent's snapshots, ask the archive with that agent's password, which
|
|
the secret store holds at `swarm/hives/<hive>/archive/<agent>` (fields
|
|
`rest-password` and `restic-password`):
|
|
|
|
```sh
|
|
RESTIC_REPOSITORY="rest:https://archive.<swarm domain>/<agent>/" \
|
|
RESTIC_REST_USERNAME=<agent> RESTIC_REST_PASSWORD=<rest-password> \
|
|
restic snapshots
|
|
```
|
|
|
|
restic asks for the `restic-password`. On the archive host, `systemctl status
|
|
swarm-archive swarm-archive-users swarm-archive-prune` shows the server, the
|
|
users file refresh and the last prune.
|
|
|
|
## When it fails
|
|
|
|
- **The `archive backup failed` alert.** A hive's run failed for an agent or
|
|
failed to log in. The journal line `archive backup failed: <agent>` names the
|
|
agent, and the lines above it say why. The alert waits 30 minutes, so an
|
|
agent placed a few minutes ago doesn't trip it while its login is on its way.
|
|
The next hourly run retries.
|
|
- **`swarm-controller has granted this hive no archive credentials`.** Not a
|
|
failure: the hive logs it and succeeds. It means the store holds no login for
|
|
any agent here, because no archive is running yet, `swarm-controller` is
|
|
older than the archive, or its archive pass fails (look for `archive:` lines
|
|
in its journal). If an archive is running and this persists, check the
|
|
controller.
|
|
- **The `archive prune failed` alert.** The daily prune couldn't apply the
|
|
retention to one repository, for example because a backup held its lock for
|
|
longer than ten minutes. The next prune retries.
|
|
- **A swarm without an archive.** Nothing fails and nothing alerts. The archive
|
|
host announces itself in the secret store when it starts, and
|
|
`swarm-controller` writes no logins before that, so every hive's run finds
|
|
nothing to back up and says so in its journal.
|
|
|
|
## Mixed versions
|
|
|
|
A hive that doesn't have the backup unit yet doesn't back up, and nothing else
|
|
changes. A `swarm-controller` without the archive pass writes no logins, so
|
|
hives with the unit log that they have no credentials and succeed until it
|
|
updates. The archive host admits only the users the controller wrote, so until
|
|
then it admits nobody.
|
|
|
|
Where the repositories live is a setting of the archive host alone. Hives and `swarm-controller`
|
|
behave the same with either.
|
|
|
|
## What lives where
|
|
|
|
Everything secret lives in the swarm secret store, and the archive keeps none
|
|
on disk. The archive host renders its users file (password hashes only) into
|
|
`/run/swarm-archive-auth/`, which is tmpfs. With an external S3, it reads the
|
|
S3 key from the store when a process starts and passes it to rclone in the
|
|
environment. Hives read their restic password when restic asks for it. The paths and who writes and reads them are in the
|
|
[credentials reference](credentials-reference.md).
|