mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-archive: check each agent's restic repository weekly

Browse source 
A weekly swarm-archive-check unit runs restic check --read-data-subset on every repository on the archive host, serialised with the prune by a shared flock. It logs archive check failed per agent, pushes check gauges, and has an alert and dashboard columns.
This commit is contained in:
atlas 2026-11-14 11:51:25 +01:00
commit 5c30985a25
1 changed files with 503 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

25
docs/2-handbook/swarm/archive.md
View file

@ -122,6 +122,22 @@ Because the archive host prunes, it reads every agent's restic password from
the secret store and can decrypt every backup. Treat that host as part of the
swarm's trust root.
## Integrity check
A backup that finishes can still be unreadable: a damaged pack file, a missing
index entry or a repository left half-written would show up only when a
restore fails. Once a week (`deploy.archive.checkSchedule`, default `weekly`)
the archive host runs `restic check` on every repository. It checks the
repository's structure in full and reads a random `deploy.archive.checkSubset`
of its data (default `5%`), so repeated weekly runs cover a repository without
any one run reading all of it. The check runs on the archive host because the
repositories are local there, so it reads no data over the network and needs no
credential beyond the passwords the prune already reads.
The check and the prune share one lock and never run on a repository together;
whichever starts second waits. A run that fails for an agent logs
`archive check failed: <agent>` and carries on with the others.
## Schedule
`deploy.hive-controller.archive.schedule` is a systemd calendar expression
@ -147,8 +163,8 @@ restic snapshots
```
restic asks for the `restic-password`. On the archive host, `systemctl status
swarm-archive swarm-archive-users swarm-archive-prune` shows the server, the
users file refresh and the last prune.
swarm-archive swarm-archive-users swarm-archive-prune swarm-archive-check` shows
the server, the users file refresh, the last prune and the last check.
## When it fails
@ -166,6 +182,11 @@ users file refresh and the last prune.
- **The `archive prune failed` alert.** The daily prune couldn't apply the
retention to one repository, for example because a backup held its lock for
longer than ten minutes. The next prune retries.
- **The `archive check failed` alert.** `restic check` found a problem in one
repository, or couldn't open it. The journal of `swarm-archive-check` names
the agent and carries restic's error. Treat it as a backup that may not
restore: run `restic check --read-data` on that repository by hand before
trusting it. The alert stays up until the failure is older than eight days.
- **A swarm without an archive.** Nothing fails and nothing alerts. The archive
host announces itself in the secret store when it starts, and
`swarm-controller` writes no logins before that, so every hive's run finds