hyperhive/docs/2-handbook/swarm/archive.md

Backup archive

The archive keeps a backup of every agent's state. Each hive backs its agents up to it every hour, and it keeps the last few days. This page covers what the hive saves, where it goes, how long it stays and how to check that it works. Restoring an agent from a backup isn't built yet.

What's in the backup

For each agent a hive places, the hive saves the agent's state/, claude/ and harness/ directories. claude/ holds the agent's login, so a restored agent keeps its session. Caches and work/ stay out.

Two options on the hive change that:

• deploy.hive-controller.archive.paths lists the entries of the agent's directory to save. The hive skips an entry the agent doesn't have.
• deploy.hive-controller.archive.scope = "subvolume" saves the whole agent directory instead, work/ included. Add archive.excludes patterns to keep build output out.

On a btrfs hive the hive saves a read-only snapshot of the agent's subvolume, so it copies the databases at one instant. Otherwise it saves the directory in place.

Where it goes

The archive runs as rclone serve restic on the host that sets deploy.archive.enable, behind the gateway at archive.<swarm domain>. Every hive finds it by that name, so which host runs it makes no difference to them. allSwarmServices turns it on wherever the host has the archive's store identity, as for the binary cache.

Each agent has its own restic repository at https://archive.<swarm domain>/<agent>/, under deploy.archive.dataDir (default /var/lib/swarm-archive) on that host, or in a bucket of an external S3. restic encrypts it with a password of its own for that agent.

Two properties protect the backups from a compromised hive:

• One login per agent. swarm-controller creates a login for each agent when it places the agent. The login reaches that agent's repository and no other. A hive holds the logins of the agents it places and no others. When an agent moves, the controller gives the new hive a new password and deletes the old hive's copy, so the old hive can't reach the repository any more. Only that login rotates. The controller creates an agent's restic password once, when it first places the agent, and a move, a destroy or a re-placement never changes it, because it's the key of the repository.
• Append-only. A client can add snapshots and nothing else: no delete, no overwrite. The archive host removes old snapshots.

External S3

By default the repositories sit on the archive host's disk. With deploy.archive.backend = "s3" they sit in a bucket of an S3 service you run or rent instead, and the host keeps nothing. Hives don't change: they still talk to archive.<swarm domain> with their per-agent logins, and the archive is still append-only. Only the archive holds the S3 key, so a compromised hive can't reach the bucket directly.

option	default	meaning
deploy.archive.backend	local	local or s3
deploy.archive.s3.endpoint	null	the service's URL, scheme included; required
deploy.archive.s3.bucket	null	the bucket; required, and it must exist
deploy.archive.s3.prefix	""	a path in the bucket; the repositories sit at <prefix>/<agent>/
deploy.archive.s3.region	null	the bucket's region, for services that need one
deploy.archive.s3.provider	Other	rclone's --s3-provider, for example AWS

Evaluation fails if backend is s3 and the endpoint or the bucket is unset.

The service needs one bucket and one access key that can list, read, write and delete objects in it. Scope the key to the bucket. The archive never creates a bucket, so the key needs no permission to. The prune removes old data, which takes delete rights. Hives can't delete: the archive refuses it before the request reaches the bucket.

The key lives in the secret store, like every other archive secret. Write it before you set backend = "s3"; the server doesn't start without it. Read it from a file or the terminal so it doesn't land in your shell history:

bao kv put secret/swarm/services/archive/s3/access-key-id value=-
bao kv put secret/swarm/services/archive/s3/secret-access-key value=-

Each command reads the value from standard input. The store's UI writes the same two leaves, field value. The archive's host reads them as it starts and passes them to rclone in its environment. They never reach a file there, so a changed key takes effect at the next restart of swarm-archive. The daily prune reads them on every run. If the server can't read them, it fails to start and systemd retries every five seconds until the store answers.

Changing backend doesn't move the data. Before you switch an archive that holds backups, copy them with rclone copy <dataDir>/repos :s3:<bucket>/<prefix> using a rclone configured for the bucket, or start with empty repositories and let the next hourly runs fill them. The old directory stays on the host until you remove it.

How long it stays

The archive host prunes every repository once a day. A snapshot survives if either rule keeps it:

option	default	keeps
deploy.archive.retention.keepWithin	3d	every snapshot younger than this
deploy.archive.retention.keepLast	3	the newest this many, however old

The age rule limits what a compromised hive can do by flooding the archive with fake snapshots: they can't push the last three days out. restic ranks snapshots by their date, and a client chooses a snapshot's date, so a client could also date snapshots years ahead to fill the keepLast slots. Before each prune the archive host forgets every snapshot dated more than an hour ahead of its own clock and logs archive prune warning: <agent> has snapshots dated after now. A line like that means a hive or its credentials are misbehaving. Set keepWithin to null to leave the age rule off. deploy.archive.pruneSchedule sets when the prune runs (default daily).

Because the archive host prunes, it reads every agent's restic password from the secret store and can decrypt every backup. Treat that host as part of the swarm's trust root.

Schedule

deploy.hive-controller.archive.schedule is a systemd calendar expression (default hourly). restic sends only what changed, so a run over an idle agent is small. The first backup of a large agent sends everything.

Check that it works

On a hive, systemctl status hive-archive-backup shows the last run, and journalctl -u hive-archive-backup lists every agent it backed up or failed to back up. systemctl list-timers hive-archive-backup shows the next run.

To see an agent's snapshots, ask the archive with that agent's password, which the secret store holds at swarm/hives/<hive>/archive/<agent> (fields rest-password and restic-password):

RESTIC_REPOSITORY="rest:https://archive.<swarm domain>/<agent>/" \
RESTIC_REST_USERNAME=<agent> RESTIC_REST_PASSWORD=<rest-password> \
restic snapshots

restic asks for the restic-password. On the archive host, systemctl status swarm-archive swarm-archive-users swarm-archive-prune shows the server, the users file refresh and the last prune.

When it fails

• The archive backup failed alert. A hive's run failed for an agent or failed to log in. The journal line archive backup failed: <agent> names the agent, and the lines above it say why. The alert waits 30 minutes, so an agent placed a few minutes ago doesn't trip it while its login is on its way. The next hourly run retries.
• swarm-controller has granted this hive no archive credentials. Not a failure: the hive logs it and succeeds. It means the store holds no login for any agent here, because no archive is running yet, swarm-controller is older than the archive, or its archive pass fails (look for archive: lines in its journal). If an archive is running and this persists, check the controller.
• The archive prune failed alert. The daily prune couldn't apply the retention to one repository, for example because a backup held its lock for longer than ten minutes. The next prune retries.
• A swarm without an archive. Nothing fails and nothing alerts. The archive host announces itself in the secret store when it starts, and swarm-controller writes no logins before that, so every hive's run finds nothing to back up and says so in its journal.

Mixed versions

A hive that doesn't have the backup unit yet doesn't back up, and nothing else changes. A swarm-controller without the archive pass writes no logins, so hives with the unit log that they have no credentials and succeed until it updates. The archive host admits only the users the controller wrote, so until then it admits nobody.

Where the repositories live is a setting of the archive host alone. Hives and swarm-controller behave the same with either.

What lives where

Everything secret lives in the swarm secret store, and the archive keeps none on disk. The archive host renders its users file (password hashes only) into /run/swarm-archive-auth/, which is tmpfs. With an external S3, it reads the S3 key from the store when a process starts and passes it to rclone in the environment. Hives read their restic password when restic asks for it. The paths and who writes and reads them are in the credentials reference.