hyperhive/docs/2-handbook/swarm/archive.md

Backup archive

The archive keeps a backup of every agent's state. Each hive backs its agents up to it every hour, and it keeps the last few days. This page covers what the hive saves, where it goes, how long it stays and how to check that it works. Restoring an agent from a backup isn't built yet.

What's in the backup

For each agent a hive places, the hive saves the agent's state/, claude/ and harness/ directories. claude/ holds the agent's login, so a restored agent keeps its session. Caches and work/ stay out.

Two options on the hive change that:

• deploy.hive-controller.archive.paths lists the entries of the agent's directory to save. The hive skips an entry the agent doesn't have.
• deploy.hive-controller.archive.scope = "subvolume" saves the whole agent directory instead, work/ included. Add archive.excludes patterns to keep build output out.

On a btrfs hive the hive saves a read-only snapshot of the agent's subvolume, so it copies the databases at one instant. Otherwise it saves the directory in place.

Where it goes

The archive runs as rclone serve restic on the host that sets deploy.archive.enable, behind the gateway at archive.<swarm domain>. Every hive finds it by that name, so which host runs it makes no difference to them. allSwarmServices turns it on wherever the host has the archive's store identity, as for the binary cache.

Each agent has its own restic repository at https://archive.<swarm domain>/<agent>/, under deploy.archive.dataDir (default /var/lib/swarm-archive) on that host. restic encrypts it with a password of its own for that agent.

Two properties protect the backups from a compromised hive:

• One login per agent. swarm-controller creates a login for each agent when it places the agent. The login reaches that agent's repository and no other. A hive holds the logins of the agents it places and no others. When an agent moves, the controller gives the new hive a new password and deletes the old hive's copy, so the old hive can't reach the repository any more. Only that login rotates. The controller creates an agent's restic password once, when it first places the agent, and a move, a destroy or a re-placement never changes it, because it's the key of the repository.
• Append-only. A client can add snapshots and nothing else: no delete, no overwrite. The archive host removes old snapshots.

How long it stays

The archive host prunes every repository once a day. A snapshot survives if either rule keeps it:

option	default	keeps
deploy.archive.retention.keepWithin	3d	every snapshot younger than this
deploy.archive.retention.keepLast	3	the newest this many, however old

The age rule limits what a compromised hive can do by flooding the archive with fake snapshots: they can't push the last three days out. Set keepWithin to null to leave that rule off. deploy.archive.pruneSchedule sets when the prune runs (default daily).

Because the archive host prunes, it reads every agent's restic password from the secret store and can decrypt every backup. Treat that host as part of the swarm's trust root.

Schedule

deploy.hive-controller.archive.schedule is a systemd calendar expression (default hourly). restic sends only what changed, so a run over an idle agent is small. The first backup of a large agent sends everything.

Check that it works

On a hive, systemctl status hive-archive-backup shows the last run, and journalctl -u hive-archive-backup lists every agent it backed up or failed to back up. systemctl list-timers hive-archive-backup shows the next run.

To see an agent's snapshots, ask the archive with that agent's password, which the secret store holds at swarm/hives/<hive>/archive/<agent> (fields rest-password and restic-password):

RESTIC_REPOSITORY="rest:https://archive.<swarm domain>/<agent>/" \
RESTIC_REST_USERNAME=<agent> RESTIC_REST_PASSWORD=<rest-password> \
restic snapshots

restic asks for the restic-password. On the archive host, systemctl status swarm-archive swarm-archive-users swarm-archive-prune shows the server, the users file refresh and the last prune.

When it fails

• The archive backup failed alert. A hive's run failed for an agent or failed to log in. The journal line archive backup failed: <agent> names the agent, and the lines above it say why. The alert waits 30 minutes, so an agent placed a few minutes ago doesn't trip it while its login is on its way. The next hourly run retries.
• swarm-controller has granted this hive no archive credentials. Not a failure: the hive logs it and succeeds. It means the store holds no login for any agent here, because no archive is running yet, swarm-controller is older than the archive, or its archive pass fails (look for archive: lines in its journal). If an archive is running and this persists, check the controller.
• The archive prune failed alert. The daily prune couldn't apply the retention to one repository, for example because a backup held its lock for longer than ten minutes. The next prune retries.
• A swarm without an archive. Nothing fails and nothing alerts. The archive host announces itself in the secret store when it starts, and swarm-controller writes no logins before that, so every hive's run finds nothing to back up and says so in its journal.

Mixed versions

A hive that doesn't have the backup unit yet doesn't back up, and nothing else changes. A swarm-controller without the archive pass writes no logins, so hives with the unit log that they have no credentials and succeed until it updates. The archive host admits only the users the controller wrote, so until then it admits nobody.

What lives where

Everything secret lives in the swarm secret store, and the archive keeps none on disk. The archive host renders its users file (password hashes only) into /run/swarm-archive-auth/, which is tmpfs. Hives read their restic password when restic asks for it. The paths and who writes and reads them are in the credentials reference.