# Backup archive The archive keeps a backup of every agent's state. Each hive backs its agents up to it every hour, and it keeps the last few days. This page covers what the hive saves, where it goes, how long it stays and how to check that it works. Restoring an agent from a backup isn't built yet. ## What's in the backup For each agent a hive places, the hive saves the agent's `state/`, `claude/` and `harness/` directories. `claude/` holds the agent's login, so a restored agent keeps its session. Caches and `work/` stay out. Two options on the hive change that: - `deploy.hive-controller.archive.paths` lists the entries of the agent's directory to save. The hive skips an entry the agent doesn't have. - `deploy.hive-controller.archive.scope = "subvolume"` saves the whole agent directory instead, `work/` included. Add `archive.excludes` patterns to keep build output out. On a btrfs hive the hive saves a read-only snapshot of the agent's subvolume, so it copies the databases at one instant. Otherwise it saves the directory in place. ## Where it goes The archive runs as `rclone serve restic` on the host that sets `deploy.archive.enable`, behind the gateway at `archive.`. Every hive finds it by that name, so which host runs it makes no difference to them. `allSwarmServices` turns it on wherever the host has the archive's store identity, as for the binary cache. Each agent has its own [restic](https://restic.net) repository at `https://archive.//`, under `deploy.archive.dataDir` (default `/var/lib/swarm-archive`) on that host, or in a bucket of an [external S3](#external-s3). restic encrypts it with a password of its own for that agent. Two properties protect the backups from a compromised hive: - **One login per agent.** `swarm-controller` creates a login for each agent when it places the agent. The login reaches that agent's repository and no other. A hive holds the logins of the agents it places and no others. When an agent moves, the controller gives the new hive a new password and deletes the old hive's copy, so the old hive can't reach the repository any more. Only that login rotates. The controller creates an agent's restic password once, when it first places the agent, and a move, a destroy or a re-placement never changes it, because it's the key of the repository. - **Append-only.** A client can add snapshots and nothing else: no delete, no overwrite. The archive host removes old snapshots. ## External S3 By default the repositories sit on the archive host's disk. With `deploy.archive.backend = "s3"` they sit in a bucket of an S3 service you run or rent instead, and the host keeps nothing. Hives don't change: they still talk to `archive.` with their per-agent logins, and the archive is still append-only. Only the archive holds the S3 key, so a compromised hive can't reach the bucket directly. | option | default | meaning | | ---------------------------- | ------- | ----------------------------------------------------------------- | | `deploy.archive.backend` | `local` | `local` or `s3` | | `deploy.archive.s3.endpoint` | `null` | the service's URL, scheme included; required | | `deploy.archive.s3.bucket` | `null` | the bucket; required, and it must exist | | `deploy.archive.s3.prefix` | `""` | a path in the bucket; the repositories sit at `//` | | `deploy.archive.s3.region` | `null` | the bucket's region, for services that need one | | `deploy.archive.s3.provider` | `Other` | rclone's `--s3-provider`, for example `AWS` | Evaluation fails if `backend` is `s3` and the endpoint or the bucket is unset. The service needs one bucket and one access key that can list, read, write and delete objects in it. Scope the key to the bucket. The archive never creates a bucket, so the key needs no permission to. The prune removes old data, which takes delete rights. Hives can't delete: the archive refuses it before the request reaches the bucket. The key lives in the secret store, like every other archive secret. Write it before you set `backend = "s3"`; the server doesn't start without it. Read it from a file or the terminal so it doesn't land in your shell history: ```sh bao kv put secret/swarm/services/archive/s3/access-key-id value=- bao kv put secret/swarm/services/archive/s3/secret-access-key value=- ``` Each command reads the value from standard input. The store's UI writes the same two leaves, field `value`. The archive's host reads them as it starts and passes them to rclone in its environment. They never reach a file there, so a changed key takes effect at the next restart of `swarm-archive`. The daily prune reads them on every run. If the server can't read them, it fails to start and systemd retries every five seconds until the store answers. Changing `backend` doesn't move the data. Before you switch an archive that holds backups, copy them with `rclone copy /repos :s3:/` using a rclone configured for the bucket, or start with empty repositories and let the next hourly runs fill them. The old directory stays on the host until you remove it. ## How long it stays The archive host prunes every repository once a day. A snapshot survives if either rule keeps it: | option | default | keeps | | ------------------------------------- | ------- | --------------------------------- | | `deploy.archive.retention.keepWithin` | `3d` | every snapshot younger than this | | `deploy.archive.retention.keepLast` | `3` | the newest this many, however old | The age rule limits what a compromised hive can do by flooding the archive with fake snapshots: they can't push the last three days out. restic ranks snapshots by their date, and a client chooses a snapshot's date, so a client could also date snapshots years ahead to fill the `keepLast` slots. Before each prune the archive host forgets every snapshot dated more than an hour ahead of its own clock and logs `archive prune warning: has snapshots dated after now`. A line like that means a hive or its credentials are misbehaving. Set `keepWithin` to `null` to leave the age rule off. `deploy.archive.pruneSchedule` sets when the prune runs (default `daily`). Because the archive host prunes, it reads every agent's restic password from the secret store and can decrypt every backup. Treat that host as part of the swarm's trust root. ## Schedule `deploy.hive-controller.archive.schedule` is a systemd calendar expression (default `hourly`). restic sends only what changed, so a run over an idle agent is small. The first backup of a large agent sends everything. ## Check that it works On a hive, `systemctl status hive-archive-backup` shows the last run, and `journalctl -u hive-archive-backup` lists every agent it backed up or failed to back up. `systemctl list-timers hive-archive-backup` shows the next run. To see an agent's snapshots, ask the archive with that agent's password, which the secret store holds at `swarm/hives//archive/` (fields `rest-password` and `restic-password`): ```sh RESTIC_REPOSITORY="rest:https://archive.//" \ RESTIC_REST_USERNAME= RESTIC_REST_PASSWORD= \ restic snapshots ``` restic asks for the `restic-password`. On the archive host, `systemctl status swarm-archive swarm-archive-users swarm-archive-prune` shows the server, the users file refresh and the last prune. ## When it fails - **The `archive backup failed` alert.** A hive's run failed for an agent or failed to log in. The journal line `archive backup failed: ` names the agent, and the lines above it say why. The alert waits 30 minutes, so an agent placed a few minutes ago doesn't trip it while its login is on its way. The next hourly run retries. - **`swarm-controller has granted this hive no archive credentials`.** Not a failure: the hive logs it and succeeds. It means the store holds no login for any agent here, because no archive is running yet, `swarm-controller` is older than the archive, or its archive pass fails (look for `archive:` lines in its journal). If an archive is running and this persists, check the controller. - **The `archive prune failed` alert.** The daily prune couldn't apply the retention to one repository, for example because a backup held its lock for longer than ten minutes. The next prune retries. - **A swarm without an archive.** Nothing fails and nothing alerts. The archive host announces itself in the secret store when it starts, and `swarm-controller` writes no logins before that, so every hive's run finds nothing to back up and says so in its journal. ## Mixed versions A hive that doesn't have the backup unit yet doesn't back up, and nothing else changes. A `swarm-controller` without the archive pass writes no logins, so hives with the unit log that they have no credentials and succeed until it updates. The archive host admits only the users the controller wrote, so until then it admits nobody. Where the repositories live is a setting of the archive host alone. Hives and `swarm-controller` behave the same with either. ## What lives where Everything secret lives in the swarm secret store, and the archive keeps none on disk. The archive host renders its users file (password hashes only) into `/run/swarm-archive-auth/`, which is tmpfs. With an external S3, it reads the S3 key from the store when a process starts and passes it to rclone in the environment. Hives read their restic password when restic asks for it. The paths and who writes and reads them are in the [credentials reference](../../4-implementation/swarm/credentials-reference.md).