mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-archive: check each agent's restic repository weekly

Browse source 
A weekly swarm-archive-check unit runs restic check --read-data-subset on every repository on the archive host, serialised with the prune by a shared flock. It logs archive check failed per agent, pushes check gauges, and has an alert and dashboard columns.
This commit is contained in:
atlas 2026-11-14 11:51:25 +01:00
commit 5c30985a25
1 changed files with 503 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

31
nix/script-tests/archive-metrics.nix
View file

@ -1,6 +1,6 @@
# `checks.script-test-archive-metrics` — runs the metric functions of the archive's
# backup and prune scripts (../host-modules/lib/archive-backup.nix and
# archive-server.nix) against a stub `curl` and a fake `restic`. The cases are in
# archive-server.nix, including the check script's loop) against a stub `curl` and a fake `restic`. The cases are in
# ./archive-metrics.sh, which cuts each function out of the rendered script.
{
pkgs,
@ -11,6 +11,18 @@ let
# Answers from the files named by FAKE_SNAPSHOTS and FAKE_STATS; an unset
# variable makes that subcommand fail, and so does a call without --no-lock:
# the metrics only read, and must not wait on a lock.
# `check` is the integrity check: it must hold the repository lock, and fails
# for the repositories named in FAKE_CHECK_FAIL (space separated agents).
case " $* " in
*" check "*)
echo "check $*" >>"$FAKE_LOG"
case " $* " in *" --no-lock "*) exit 1 ;; esac
for a in ''${FAKE_CHECK_FAIL:-}; do
case " $* " in *"/$a/ "*) exit 1 ;; esac
done
exit 0
;;
esac
case " $* " in *" --no-lock "*) ;; *) exit 1 ;; esac
case " $* " in
*" snapshots "*) [ -n "''${FAKE_SNAPSHOTS:-}" ] && cat "$FAKE_SNAPSHOTS" ;;
@ -25,6 +37,7 @@ pkgs.runCommand "hyperhive-script-test-archive-metrics"
nativeBuildInputs = [
pkgs.coreutils
pkgs.jq
pkgs.util-linux
];
BACKUP_SCRIPT = pkgs.writeText "archive-backup.sh" (
import ../host-modules/lib/archive-backup.nix { inherit (pkgs) lib; } {
@ -39,6 +52,22 @@ pkgs.runCommand "hyperhive-script-test-archive-metrics"
otlpEndpoint = endpoint;
}
);
CHECK_SCRIPT = pkgs.writeText "archive-check.sh" (
(import ../host-modules/lib/archive-server.nix { inherit (pkgs) lib; } {
rclone = pkgs.rclone;
restic = fakeRestic;
reposDir = "/repos";
addr = "127.0.0.1:1";
authFile = "/auth";
authGroup = "g";
retention = {
keepWithin = null;
keepLast = 1;
};
checkSubset = "7%";
otlpEndpoint = endpoint;
}).checkScript
);
PRUNE_SCRIPT = pkgs.writeText "archive-prune.sh" (
(import ../host-modules/lib/archive-server.nix { inherit (pkgs) lib; } {
rclone = pkgs.rclone;