From 5c30985a25a803eccbf9e8fd68890586e959447b Mon Sep 17 00:00:00 2001 From: atlas Date: Sat, 14 Nov 2026 11:51:25 +0100 Subject: [PATCH] swarm-archive: check each agent's restic repository weekly A weekly swarm-archive-check unit runs restic check --read-data-subset on every repository on the archive host, serialised with the prune by a shared flock. It logs archive check failed per agent, pushes check gauges, and has an alert and dashboard columns. --- docs/2-handbook/swarm/archive.md | 25 ++- .../scheduler/observability.md | 2 +- .../swarm/credentials-reference.md | 8 +- docs/4-implementation/swarm/telemetry.md | 2 +- .../glue-bao-readers-policy-order.nix | 4 + .../glue-bao-readers-store-order.nix | 1 + nix/host-modules/lib/archive-server.nix | 82 ++++++++- nix/host-modules/swarm-archive.nix | 88 +++++++++- nix/host-modules/swarm-grafana.nix | 10 ++ .../swarm-grafana/dashboards/archive.json | 163 +++++++++++++++++- nix/module-eval/builder.nix | 9 + nix/script-tests/archive-metrics.nix | 31 +++- nix/script-tests/archive-metrics.sh | 96 +++++++++++ 13 files changed, 503 insertions(+), 18 deletions(-) diff --git a/docs/2-handbook/swarm/archive.md b/docs/2-handbook/swarm/archive.md index f0bfe0562..5019d3a99 100644 --- a/docs/2-handbook/swarm/archive.md +++ b/docs/2-handbook/swarm/archive.md @@ -122,6 +122,22 @@ Because the archive host prunes, it reads every agent's restic password from the secret store and can decrypt every backup. Treat that host as part of the swarm's trust root. +## Integrity check + +A backup that finishes can still be unreadable: a damaged pack file, a missing +index entry or a repository left half-written would show up only when a +restore fails. Once a week (`deploy.archive.checkSchedule`, default `weekly`) +the archive host runs `restic check` on every repository. It checks the +repository's structure in full and reads a random `deploy.archive.checkSubset` +of its data (default `5%`), so repeated weekly runs cover a repository without +any one run reading all of it. The check runs on the archive host because the +repositories are local there, so it reads no data over the network and needs no +credential beyond the passwords the prune already reads. + +The check and the prune share one lock and never run on a repository together; +whichever starts second waits. A run that fails for an agent logs +`archive check failed: ` and carries on with the others. + ## Schedule `deploy.hive-controller.archive.schedule` is a systemd calendar expression @@ -147,8 +163,8 @@ restic snapshots ``` restic asks for the `restic-password`. On the archive host, `systemctl status -swarm-archive swarm-archive-users swarm-archive-prune` shows the server, the -users file refresh and the last prune. +swarm-archive swarm-archive-users swarm-archive-prune swarm-archive-check` shows +the server, the users file refresh, the last prune and the last check. ## When it fails @@ -166,6 +182,11 @@ users file refresh and the last prune. - **The `archive prune failed` alert.** The daily prune couldn't apply the retention to one repository, for example because a backup held its lock for longer than ten minutes. The next prune retries. +- **The `archive check failed` alert.** `restic check` found a problem in one + repository, or couldn't open it. The journal of `swarm-archive-check` names + the agent and carries restic's error. Treat it as a backup that may not + restore: run `restic check --read-data` on that repository by hand before + trusting it. The alert stays up until the failure is older than eight days. - **A swarm without an archive.** Nothing fails and nothing alerts. The archive host announces itself in the secret store when it starts, and `swarm-controller` writes no logins before that, so every hive's run finds diff --git a/docs/4-implementation/scheduler/observability.md b/docs/4-implementation/scheduler/observability.md index 8bc3b25cf..74e44bf93 100644 --- a/docs/4-implementation/scheduler/observability.md +++ b/docs/4-implementation/scheduler/observability.md @@ -393,7 +393,7 @@ in `services.hyperhive.otel.journalUnits`: `hive-c0re.service`, the `container@* units (the start-time rootfs check lines), plus `squid.service` and `squid-selfcheck.service` (see [Egress proxy](../networking/network.md#egress-proxy-squid)). A host that runs the cache without the swarm collector adds `swarm-bao-attic-secret.service` -and `swarm-bao-attic-tokens.service`; a host that runs the archive adds `swarm-archive.service`, `swarm-archive-users.service` and `swarm-archive-prune.service`, and a hive adds `hive-archive-backup.service`. +and `swarm-bao-attic-tokens.service`; a host that runs the archive adds `swarm-archive.service`, `swarm-archive-users.service`, `swarm-archive-prune.service` and `swarm-archive-check.service`, and a hive adds `hive-archive-backup.service`. A host that also runs the swarm collector names none of these: its whole-host journal receiver ships them already, and the hive collector then has no `journald` receiver. No bind mount is needed. Unlike an agent's journal, this collector already diff --git a/docs/4-implementation/swarm/credentials-reference.md b/docs/4-implementation/swarm/credentials-reference.md index 12d2913b2..b57850880 100644 --- a/docs/4-implementation/swarm/credentials-reference.md +++ b/docs/4-implementation/swarm/credentials-reference.md @@ -58,10 +58,10 @@ the rest of the cell says how. | `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes | | `swarm/hives//archive/` (fields `rest-password`, `restic-password`) | `swarm-controller`, for every agent a hive places, every 5 minutes: a new random REST password when the agent is new on that hive, the agent's restic password from the leaf below; the copy on a hive that no longer places the agent is deleted | the agent's hive: `hive-archive-backup`, under the hive's own store identity | ✅ a move to another hive writes a new REST password, which locks the old hive out | ✅ read at every backup run | | `swarm/services/archive/users/` (field `htpasswd`) | `swarm-controller`, with the REST password above: its `{SHA}` hash, never the password; deleted when no hive places the agent | `swarm-archive-users` on the archive's host, which renders the users file into `/run` | ✅ rewritten with the password | ✅ re-read every 2 minutes; the server notices the new file without a restart | -| `swarm/services/archive/restic/` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune | +| `swarm/services/archive/restic/` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` and `swarm-archive-check` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune | | `swarm/services/archive/ready` | `swarm-archive-users` on the archive's host, once, when none is stored; `swarm-controller` writes no archive credentials before it exists | `swarm-controller`, every 5 minutes | ❌ never rewritten | ✅ read every 5 minutes | -| `swarm/services/archive/s3/access-key-id` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune | -| `swarm/services/archive/s3/secret-access-key` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune | +| `swarm/services/archive/s3/access-key-id` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive`, `swarm-archive-prune` and `swarm-archive-check` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune | +| `swarm/services/archive/s3/secret-access-key` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive`, `swarm-archive-prune` and `swarm-archive-check` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune | | `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass | | _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated | @@ -188,7 +188,7 @@ login minter's, the forge host's and Grafana's, which reach more than one: | `swarm-stt`, in its container | `sttClientCertFile` / `sttClientKeyFile` | `swarm/services//oidc/client` | | `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services//oidc/client` | | `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` | -| `swarm-archive-users`, `swarm-archive-prune` and, with an S3 bucket, `swarm-archive` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*`, `swarm/services/archive/restic/*` and `swarm/services/archive/s3/*`, lists `swarm/services/archive/users/` | +| `swarm-archive-users`, `swarm-archive-prune`, `swarm-archive-check` and, with an S3 bucket, `swarm-archive` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*`, `swarm/services/archive/restic/*` and `swarm/services/archive/s3/*`, lists `swarm/services/archive/users/` | | `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services//oidc/client` | | `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services//oidc/client` and `swarm/services//oidc/client` | diff --git a/docs/4-implementation/swarm/telemetry.md b/docs/4-implementation/swarm/telemetry.md index 97fa08250..2012f97fd 100644 --- a/docs/4-implementation/swarm/telemetry.md +++ b/docs/4-implementation/swarm/telemetry.md @@ -55,7 +55,7 @@ kept. - `agents`: per-agent resource usage and Claude Code activity. The **activity** tab ends with a **Busiest agents** table: turns, input, output and cache-read tokens, and cost for each agent in the range, busiest first. The **turns** tab holds what each agent's own `/stats` page shows about turns: average turn duration, cache hit rate and tokens per turn, turns per hour by agent and by model, the turn duration over time (median, 95th percentile and mean), and the mix of wake sources and results. It also has the tools called, the commands bash tasks ran, reminders per hour, the first-turn context size and fresh sessions per agent: everything the `/stats` page shows, except the per-bucket token bars the **cost** tab covers. The duration percentiles are right from the first deploy that includes the histogram bounds in `hive-agent`; before it, they read a flat 10 seconds. - `attic`: the swarm binary cache. Agents whose latest closure this hive can pull, and why not; the builder's pushes and push failures; closures hive-c0re pulled and deploy steps it refused; the cache's size and store-path count. It reads the closure-pullable gauge, the `swarm-builder` and `hive-c0re` logs, and the `hyperhive.attic.cache_bytes` and `hyperhive.attic.cache_nars` gauges, which a timer in the `swarm-attic` container (`swarm-attic-cache-size`) reads from atticd's database every five minutes and pushes through the container's forwarder. atticd itself exports no metrics and its logs don't reach the log store, so the board has no cache request or hit panel. Filter by hive and agent. -- `archive`: the restic backups of every agent to the swarm archive. Per agent: time since the last successful backup (red over two hours, twice the default hourly schedule), the last run's result, duration and bytes added, then snapshots, repository size and the last prune. Failures and recent runs come from the `hive-archive-backup` and `swarm-archive-prune` logs. The numbers are the `hyperhive.archive.backup.*` gauges, which each hive's backup unit pushes after every agent, and the `hyperhive.archive.prune.*` and `hyperhive.archive.repo.*` gauges, which the prune unit pushes after every repository when its host runs a hive. A hive that predates them shows no data for its agents; the log panels work regardless. Filter by hive and agent. +- `archive`: the restic backups of every agent to the swarm archive. Per agent: time since the last successful backup (red over two hours, twice the default hourly schedule), the last run's result, duration and bytes added, then snapshots, repository size and the last prune and the last check. Failures and recent runs come from the `hive-archive-backup`, `swarm-archive-prune` and `swarm-archive-check` logs. The numbers are the `hyperhive.archive.backup.*` gauges, which each hive's backup unit pushes after every agent, the `hyperhive.archive.prune.*` and `hyperhive.archive.repo.*` gauges, which the prune unit pushes after every repository when its host runs a hive, and the `hyperhive.archive.check.*` gauges from the check unit. A hive that predates them shows no data for its agents; the log panels work regardless. Filter by hive and agent. - `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue. - `claude-usage`: Claude use by model, effort, token type and source. Filter by hive, service (agents or the review classifier) and agent. The service filter separates the classifier's API-key spend from the agents. - `egress`: what agents send through the egress proxy, in three tabs. **overview** has the totals (requests, agents active, refused, credential challenges, data transferred), requests over time by status, the busiest agents and destinations, and data transferred by domain over time (the top 10 domains plus an other line, summed from the proxy's access log in VictoriaLogs; the proxy logs a CONNECT tunnel when it closes, so a long-lived connection shows as one spike at close). **refused** lists the hosts the proxy refused for each agent. **would-deny** lists what agents in `log` mode reached that restricted mode would refuse. The host lists are tables, one row per host (and agent), with a count column you can sort. Filter by agent and by domain, to see which agents queried a domain and when. diff --git a/nix/host-modules/glue-bao-readers-policy-order.nix b/nix/host-modules/glue-bao-readers-policy-order.nix index 5ee53a544..65e83225b 100644 --- a/nix/host-modules/glue-bao-readers-policy-order.nix +++ b/nix/host-modules/glue-bao-readers-policy-order.nix @@ -100,6 +100,10 @@ let here = deployCfg.archive.enable; policies = [ "swarm-bao-archive-policy" ]; }; + swarm-archive-check = { + here = deployCfg.archive.enable; + policies = [ "swarm-bao-archive-policy" ]; + }; # ./hive-matrix.nix, under matrix-ctl's leaf. swarm-bao-matrix-oidc = { here = deployCfg.matrix.enable; diff --git a/nix/host-modules/glue-bao-readers-store-order.nix b/nix/host-modules/glue-bao-readers-store-order.nix index f29b04fd3..f21f40f97 100644 --- a/nix/host-modules/glue-bao-readers-store-order.nix +++ b/nix/host-modules/glue-bao-readers-store-order.nix @@ -72,6 +72,7 @@ let # ./swarm-archive.nix swarm-archive-users = deployCfg.archive.enable; swarm-archive-prune = deployCfg.archive.enable; + swarm-archive-check = deployCfg.archive.enable; # ./swarm-otel.nix swarm-bao-otel-oidc = deployCfg.swarm-otel.enable diff --git a/nix/host-modules/lib/archive-server.nix b/nix/host-modules/lib/archive-server.nix index f17625a5e..241e134d0 100644 --- a/nix/host-modules/lib/archive-server.nix +++ b/nix/host-modules/lib/archive-server.nix @@ -1,6 +1,6 @@ # The archive server's moving parts: the rclone command that serves the # repositories, the script that renders its users file from the secret store, -# and the script that prunes each repository. A pure function, NOT a NixOS +# the script that prunes each repository and the one that checks them. A pure function, NOT a NixOS # module: ../swarm-archive.nix wraps the pieces in units, and the VM test # (nix/vm-tests/swarm-archive.nix) runs the same scripts against a stand-in # `bao`. @@ -33,6 +33,9 @@ retention, # The OTLP/HTTP metrics receiver's base URL (see ./otlp-gauges.nix), or null. otlpEndpoint ? null, + # `restic check --read-data-subset=`: what share of each repository's data the + # check reads, e.g. `5%`. + checkSubset ? "5%", }: let atomicWriteSecret = import ./atomic-write-secret.nix { }; @@ -46,6 +49,16 @@ let s3KeyIdPath = "secret/swarm/services/archive/s3/access-key-id"; s3SecretKeyPath = "secret/swarm/services/archive/s3/secret-access-key"; + # Takes the lock the prune and the check share, when the unit sets + # `ARCHIVE_MAINTENANCE_LOCK` (a path in a directory both units can write); waits + # for the holder. Held until the script exits. Needs `flock` on the unit's `path`. + maintenanceLock = '' + if [ -n "''${ARCHIVE_MAINTENANCE_LOCK:-}" ]; then + exec 9>"$ARCHIVE_MAINTENANCE_LOCK" + flock 9 + fi + ''; + # Slack for the clocks of the hives that stamp snapshots. futureSkewSeconds = 3600; @@ -223,6 +236,7 @@ in pruneScript = '' set -euo pipefail ${storeSeededSecret} + ${maintenanceLock} # Forgets the snapshots of $repo dated after the limit, naming them in the # log; returns non-zero when it cannot read the list or a time. @@ -311,4 +325,70 @@ in done exit "$failed" ''; + + # `restic check` on every repository, reading `checkSubset` of its data: a random + # share each run, so repeated runs cover the repository without any run + # reading all of it. The repository's structure (index, snapshots, trees) is + # always checked in full. One repository's failure does not stop the others; + # the run exits non-zero if any failed. + checkScript = '' + set -euo pipefail + ${storeSeededSecret} + ${maintenanceLock} + + ${otlpGauges} + # Pushes the outcome of one repository's check. Never fails the run. + push_check_metrics() { + local agent="$1" ok="$2" start="$3" end="$4" + otlp_gauges swarm-archive-check "agent=$agent" \ + "hyperhive.archive.check.last_run_success=$ok" \ + "hyperhive.archive.check.last_run_timestamp_seconds=$end" \ + "hyperhive.archive.check.duration_seconds=$(( end - start ))" + } + + # Checks the repository of each agent named in the arguments; returns + # non-zero if any check failed. + check_agents() { + local agent repo start ok failed=0 + for agent in "$@"; do + if [[ ! "$agent" =~ ^[A-Za-z0-9_-]+$ ]]; then + echo "archive check failed: $agent is not an agent name" >&2 + failed=1 + continue + fi + repo=${repoOf} + export RESTIC_PASSWORD_COMMAND="bao kv get -field=password ${resticPrefix}/$agent" + ok=0 + start="$(date +%s)" + if ${restic}/bin/restic -r "$repo" --retry-lock 10m check --read-data-subset=${lib.escapeShellArg checkSubset}; then + echo "checked $agent" + ok=1 + else + echo "archive check failed: $agent" >&2 + failed=1 + fi + push_check_metrics "$agent" "$ok" "$start" "$(date +%s)" + done + return "$failed" + } + + store_init + store_login + if [ "$STORE_UP" != 1 ]; then + echo "archive check failed: could not log in to the swarm secret store" >&2 + exit 1 + fi + + ${lib.optionalString (s3 != null) s3Credentials} + list_agents() { + ${listAgents} + } + # Under `set -e` a failed listing ends the run; inside the `for` it would not. + agents="$(list_agents)" + + rc=0 + # shellcheck disable=SC2086 + check_agents $agents || rc=$? + exit "$rc" + ''; } diff --git a/nix/host-modules/swarm-archive.nix b/nix/host-modules/swarm-archive.nix index e7ad9c571..0149bcdef 100644 --- a/nix/host-modules/swarm-archive.nix +++ b/nix/host-modules/swarm-archive.nix @@ -22,12 +22,12 @@ # `RESTIC_PASSWORD_COMMAND` for its `restic forget --prune`. This host can # therefore decrypt every backup, which is the cost of pruning where clients # cannot delete. +# - `swarm-archive-check` (weekly) runs `restic check` on every repository. # # The units log in as the archive's own store principal (a leaf of the store's -# CA, minted by ./glue-bao-tls.nix). Plain host units rather than a container: -# no secret crosses a mount. The backend is one argument (`reposDir` or `s3`, in -# ./lib/archive-server.nix); with `s3` the server also reads the S3 key from -# `swarm/services/archive/s3/…` as it starts and keeps it in rclone's environment. +# CA, minted by ./glue-bao-tls.nix). Plain host units, no container: no +# secret crosses a mount. The backend is one argument (`reposDir` or `s3`, in +# ./lib/archive-server.nix); `s3` also makes the server read the S3 key as it starts. { pkgs, lib, @@ -56,6 +56,7 @@ let server = import ./lib/archive-server.nix { inherit lib; } ( { inherit (cfg) retention; + checkSubset = cfg.checkSubset; inherit authFile; rclone = cfg.package; restic = cfg.resticPackage; @@ -90,6 +91,9 @@ let BAO_CACERT = "%d/bao-ca"; }; + # The lock file the prune and the check take, so they never run on a repository together. + maintenanceLock = "/run/swarm-archive-maintenance/lock"; + serveScript = pkgs.writeShellScript "swarm-archive-serve" server.s3ServerScript; haveClientIdentity = @@ -246,6 +250,25 @@ in example = "*-*-* 04:00:00"; description = "systemd calendar expression for the prune: the retention rules are applied then."; }; + + checkSchedule = lib.mkOption { + type = lib.types.str; + default = "weekly"; + example = "Sun *-*-* 03:00:00"; + description = "systemd calendar expression for the integrity check of every agent's repository."; + }; + + checkSubset = lib.mkOption { + type = lib.types.strMatching "[0-9]+(\\.[0-9]+)?%"; + default = "5%"; + example = "10%"; + description = '' + The share of each repository's data the integrity check reads (restic's + `--read-data-subset`). The subset is random per run, so over many runs + the checks cover a repository without any one run reading all of it. The + repository's structure is checked in full on every run whatever this is. + ''; + }; }; config = lib.mkIf cfg.enable { @@ -313,6 +336,7 @@ in "swarm-archive.service" "swarm-archive-users.service" "swarm-archive-prune.service" + "swarm-archive-check.service" ]; systemd.tmpfiles.rules = [ @@ -418,6 +442,7 @@ in pkgs.curl pkgs.gnugrep pkgs.jq + pkgs.util-linux ] # restic's rclone backend starts rclone. ++ lib.optional useS3 cfg.package; @@ -425,6 +450,7 @@ in ownerStoreEnvironment // { RESTIC_CACHE_DIR = "%C/swarm-archive-prune"; + ARCHIVE_MAINTENANCE_LOCK = maintenanceLock; } // lib.optionalAttrs useS3 ({ RCLONE_CONFIG = "/dev/null"; } // server.environment); serviceConfig = { @@ -433,6 +459,8 @@ in Group = user; SyslogIdentifier = "swarm-archive-prune"; CacheDirectory = "swarm-archive-prune"; + RuntimeDirectory = "swarm-archive-maintenance"; + RuntimeDirectoryPreserve = "yes"; LoadCredential = ownerStoreCredentials; NoNewPrivileges = true; ProtectSystem = "strict"; @@ -454,5 +482,57 @@ in Persistent = true; }; }; + systemd.services.swarm-archive-check = { + description = "check the integrity of every agent's restic repository"; + after = [ "network-online.target" ]; + wants = [ "network-online.target" ]; + path = [ + baoDeploy.package + cfg.resticPackage + pkgs.coreutils + pkgs.curl + pkgs.gnugrep + pkgs.jq + pkgs.util-linux + ] + # restic's rclone backend starts rclone. + ++ lib.optional useS3 cfg.package; + environment = + ownerStoreEnvironment + // { + RESTIC_CACHE_DIR = "%C/swarm-archive-check"; + ARCHIVE_MAINTENANCE_LOCK = maintenanceLock; + } + // lib.optionalAttrs useS3 ({ RCLONE_CONFIG = "/dev/null"; } // server.environment); + serviceConfig = { + Type = "oneshot"; + User = user; + Group = user; + SyslogIdentifier = "swarm-archive-check"; + CacheDirectory = "swarm-archive-check"; + RuntimeDirectory = "swarm-archive-maintenance"; + RuntimeDirectoryPreserve = "yes"; + LoadCredential = ownerStoreCredentials; + NoNewPrivileges = true; + ProtectSystem = "strict"; + ProtectHome = true; + PrivateTmp = true; + PrivateDevices = true; + # Covers the wait for a running prune (up to its own six hours) and the check. + TimeoutStartSec = "24h"; + } + // lib.optionalAttrs (!useS3) { ReadWritePaths = [ reposDir ]; }; + script = server.checkScript; + }; + + systemd.timers.swarm-archive-check = { + description = "check the archive's repositories"; + wantedBy = [ "timers.target" ]; + timerConfig = { + OnCalendar = cfg.checkSchedule; + RandomizedDelaySec = "6h"; + Persistent = true; + }; + }; }; } diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index 9c712633d..b996116ba 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -1724,6 +1724,16 @@ in for = "0s"; summary = "the archive's daily prune failed for an agent's repository, so its retention is not applied; the unit's journal names the agent"; }) + # `swarm-archive-check` prints `archive check failed: `; it runs + # weekly, so the window spans a missed week. + (logCountRule { + uid = "hh-archive-check-failed"; + title = "archive check failed"; + windowSeconds = 700000; + expr = ''_time:8d _SYSTEMD_UNIT:="swarm-archive-check.service" _msg:"archive check failed" | stats count() as failures''; + for = "0s"; + summary = "the archive's weekly integrity check failed for an agent's repository, so its backups may not restore; the unit's journal names the agent and the restic error"; + }) # Logged once per detected crash, by `emit_crash_transitions` # in hive-c0re's crash_watch.rs (10s poll interval). Replaces # the old per-hive dashboard `agents_crashing` banner — see diff --git a/nix/host-modules/swarm-grafana/dashboards/archive.json b/nix/host-modules/swarm-grafana/dashboards/archive.json index 9b0edb001..a918d5f13 100644 --- a/nix/host-modules/swarm-grafana/dashboards/archive.json +++ b/nix/host-modules/swarm-grafana/dashboards/archive.json @@ -545,8 +545,8 @@ { "id": 9, "type": "table", - "title": "Repository and prune per agent", - "description": "Per agent: snapshots, repository size, newest snapshot age, last prune age and result.", + "title": "Repository, prune and check per agent", + "description": "Per agent: snapshots, size, newest snapshot, last prune and last check.", "datasource": { "type": "prometheus", "uid": "@datasourceUid@" @@ -607,6 +607,36 @@ "expr": "max by (agent) (last_over_time({__name__=\"hyperhive.archive.prune.last_run_success\", agent=~\"$agent\"}[7d]))", "instant": true, "format": "table" + }, + { + "refId": "F", + "datasource": { + "type": "prometheus", + "uid": "@datasourceUid@" + }, + "expr": "time() - max by (agent) (last_over_time({__name__=\"hyperhive.archive.check.last_run_timestamp_seconds\", agent=~\"$agent\"}[30d]))", + "instant": true, + "format": "table" + }, + { + "refId": "G", + "datasource": { + "type": "prometheus", + "uid": "@datasourceUid@" + }, + "expr": "max by (agent) (last_over_time({__name__=\"hyperhive.archive.check.last_run_success\", agent=~\"$agent\"}[30d]))", + "instant": true, + "format": "table" + }, + { + "refId": "H", + "datasource": { + "type": "prometheus", + "uid": "@datasourceUid@" + }, + "expr": "max by (agent) (last_over_time({__name__=\"hyperhive.archive.check.duration_seconds\", agent=~\"$agent\"}[30d]))", + "instant": true, + "format": "table" } ], "transformations": [ @@ -628,7 +658,9 @@ "Time 4": true, "Time 5": true, "Time 6": true, - "Time 7": true + "Time 7": true, + "Time 8": true, + "Time 9": true }, "renameByName": { "agent": "Agent", @@ -636,7 +668,10 @@ "Value #B": "Size", "Value #C": "Newest snapshot age", "Value #D": "Since last prune", - "Value #E": "Prune result" + "Value #E": "Prune result", + "Value #F": "Since last check", + "Value #G": "Check result", + "Value #H": "Check duration" } } } @@ -717,6 +752,64 @@ } } ] + }, + { + "matcher": { + "id": "byName", + "options": "Since last check" + }, + "properties": [ + { + "id": "unit", + "value": "s" + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Check duration" + }, + "properties": [ + { + "id": "unit", + "value": "s" + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Check result" + }, + "properties": [ + { + "id": "mappings", + "value": [ + { + "type": "value", + "options": { + "0": { + "text": "failed", + "color": "red", + "index": 0 + }, + "1": { + "text": "ok", + "color": "green", + "index": 1 + } + } + } + ] + }, + { + "id": "custom.cellOptions", + "value": { + "type": "color-text" + } + } + ] } ] }, @@ -1027,6 +1120,68 @@ "sortOrder": "Descending", "dedupStrategy": "none" } + }, + { + "id": 17, + "type": "stat", + "title": "Oldest last check", + "description": "Age of the least recently checked repository. Source: check last_run_timestamp_seconds.", + "datasource": { + "type": "prometheus", + "uid": "@datasourceUid@" + }, + "gridPos": { + "x": 18, + "y": 1, + "w": 6, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "@datasourceUid@" + }, + "expr": "time() - min(max by (agent) (last_over_time({__name__=\"hyperhive.archive.check.last_run_timestamp_seconds\", agent=~\"$agent\"}[30d])))", + "instant": true + } + ], + "fieldConfig": { + "defaults": { + "unit": "s", + "decimals": 0, + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1209600 + } + ] + }, + "mappings": [] + }, + "overrides": [] + }, + "options": { + "graphMode": "none", + "colorMode": "value", + "textMode": "value", + "justifyMode": "auto", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + } + } } ] } diff --git a/nix/module-eval/builder.nix b/nix/module-eval/builder.nix index 1cb2f38fe..30af736cc 100644 --- a/nix/module-eval/builder.nix +++ b/nix/module-eval/builder.nix @@ -617,6 +617,7 @@ let a = atticHere.systemd.services.swarm-archive; users = atticHere.systemd.services.swarm-archive-users; prune = atticHere.systemd.services.swarm-archive-prune; + check = atticHere.systemd.services.swarm-archive-check; vhost = atticHere.services.hyperhive.gateway.vhosts."archive.t.local"; cfg = atticHere.services.hyperhive.deploy.archive; names = m: m.services.hyperhive.swarm.serviceDomains; @@ -642,6 +643,14 @@ let && !(lib.hasInfix "password" (lib.toLower (builtins.toJSON a.environment))) && atticHere.systemd.timers ? swarm-archive-users && atticHere.systemd.timers.swarm-archive-prune.timerConfig.OnCalendar == "daily" + && atticHere.systemd.timers.swarm-archive-check.timerConfig.OnCalendar == "weekly" + && atticHere.systemd.timers.swarm-archive-check.timerConfig.Persistent + && lib.hasInfix "--read-data-subset=5%" check.script + && check.serviceConfig.User == "swarm-archive" + && check.environment.ARCHIVE_MAINTENANCE_LOCK == prune.environment.ARCHIVE_MAINTENANCE_LOCK + && check.serviceConfig.RuntimeDirectory == prune.serviceConfig.RuntimeDirectory + && lib.hasPrefix "/run/${check.serviceConfig.RuntimeDirectory}/" check.environment.ARCHIVE_MAINTENANCE_LOCK + && lib.elem "swarm-bao-archive-policy.service" check.after && lib.elem "swarm-bao-archive-policy.service" users.after && !(bare.systemd.services ? swarm-archive) && !(bare.systemd.services ? swarm-archive-users); diff --git a/nix/script-tests/archive-metrics.nix b/nix/script-tests/archive-metrics.nix index 459553a5c..991773841 100644 --- a/nix/script-tests/archive-metrics.nix +++ b/nix/script-tests/archive-metrics.nix @@ -1,6 +1,6 @@ # `checks.script-test-archive-metrics` — runs the metric functions of the archive's # backup and prune scripts (../host-modules/lib/archive-backup.nix and -# archive-server.nix) against a stub `curl` and a fake `restic`. The cases are in +# archive-server.nix, including the check script's loop) against a stub `curl` and a fake `restic`. The cases are in # ./archive-metrics.sh, which cuts each function out of the rendered script. { pkgs, @@ -11,6 +11,18 @@ let # Answers from the files named by FAKE_SNAPSHOTS and FAKE_STATS; an unset # variable makes that subcommand fail, and so does a call without --no-lock: # the metrics only read, and must not wait on a lock. + # `check` is the integrity check: it must hold the repository lock, and fails + # for the repositories named in FAKE_CHECK_FAIL (space separated agents). + case " $* " in + *" check "*) + echo "check $*" >>"$FAKE_LOG" + case " $* " in *" --no-lock "*) exit 1 ;; esac + for a in ''${FAKE_CHECK_FAIL:-}; do + case " $* " in *"/$a/ "*) exit 1 ;; esac + done + exit 0 + ;; + esac case " $* " in *" --no-lock "*) ;; *) exit 1 ;; esac case " $* " in *" snapshots "*) [ -n "''${FAKE_SNAPSHOTS:-}" ] && cat "$FAKE_SNAPSHOTS" ;; @@ -25,6 +37,7 @@ pkgs.runCommand "hyperhive-script-test-archive-metrics" nativeBuildInputs = [ pkgs.coreutils pkgs.jq + pkgs.util-linux ]; BACKUP_SCRIPT = pkgs.writeText "archive-backup.sh" ( import ../host-modules/lib/archive-backup.nix { inherit (pkgs) lib; } { @@ -39,6 +52,22 @@ pkgs.runCommand "hyperhive-script-test-archive-metrics" otlpEndpoint = endpoint; } ); + CHECK_SCRIPT = pkgs.writeText "archive-check.sh" ( + (import ../host-modules/lib/archive-server.nix { inherit (pkgs) lib; } { + rclone = pkgs.rclone; + restic = fakeRestic; + reposDir = "/repos"; + addr = "127.0.0.1:1"; + authFile = "/auth"; + authGroup = "g"; + retention = { + keepWithin = null; + keepLast = 1; + }; + checkSubset = "7%"; + otlpEndpoint = endpoint; + }).checkScript + ); PRUNE_SCRIPT = pkgs.writeText "archive-prune.sh" ( (import ../host-modules/lib/archive-server.nix { inherit (pkgs) lib; } { rclone = pkgs.rclone; diff --git a/nix/script-tests/archive-metrics.sh b/nix/script-tests/archive-metrics.sh index 8d7aaae67..151ab8d15 100644 --- a/nix/script-tests/archive-metrics.sh +++ b/nix/script-tests/archive-metrics.sh @@ -16,6 +16,8 @@ mkdir -p "$root/bin" cat >"$root/bin/curl" <"$root/curl.payload" +cat "$root/curl.payload" >>"$root/curl.all" +echo >>"$root/curl.all" STUB chmod +x "$root/bin/curl" @@ -185,5 +187,99 @@ else fail "failed: $(cat "$root/err")" fi +# --- the check side --- +c=hyperhive.archive.check +{ + cut_function "$CHECK_SCRIPT" otlp_gauges + cut_function "$CHECK_SCRIPT" push_check_metrics + cut_function "$CHECK_SCRIPT" check_agents +} >"$root/check-fns.sh" +grep -q 'check_agents()' "$root/check-fns.sh" || { echo "FAILED: check_agents not found"; exit 1; } +grep -q 'push_check_metrics()' "$root/check-fns.sh" || { echo "FAILED: push_check_metrics not found"; exit 1; } + +# The gauges pushed for agent $1 in the last run, as `name=value` lines. +agent_gauges() { + jq -rs --arg a "$1" '.[] | select(.resourceMetrics[0].scopeMetrics[0].metrics[0].gauge.dataPoints[0].attributes[0].value.stringValue == $a) + | .resourceMetrics[0].scopeMetrics[0].metrics[] | "\(.name)=\(.gauge.dataPoints[0].asDouble)"' "$root/curl.all" +} + +case="the check loop goes on after a failure, exits non-zero and names the agent that failed" +rm -f "$root/curl.all" "$root/fake.log" +: >"$root/fake.log" +FAKE_LOG="$root/fake.log" FAKE_CHECK_FAIL=beta run "$root/check-fns.sh" check_agents alpha beta gamma +rc=$? +[ "$rc" != 0 ] || fail "exit status was 0" +grep -qx 'archive check failed: beta' "$root/err" || fail "no failure line: $(cat "$root/err")" +grep -q 'archive check failed: alpha\|archive check failed: gamma' "$root/err" && fail "a passing agent was reported: $(cat "$root/err")" +[ "$(wc -l <"$root/fake.log")" = 3 ] || fail "want a check of every agent: $(cat "$root/fake.log")" +grep -q '/repos/gamma/ .*check --read-data-subset=7%' "$root/fake.log" || fail "gamma not checked with the subset: $(cat "$root/fake.log")" +grep -qx 'checked alpha' "$root/out" || fail "no success line: $(cat "$root/out")" + +case="the check pushes each agent's own result" +[ "$(agent_gauges alpha | grep -c .)" = 3 ] || fail "alpha gauges: $(agent_gauges alpha)" +agent_gauges alpha | grep -qx "$c.last_run_success=1" || fail "alpha result: $(agent_gauges alpha)" +agent_gauges alpha | grep -q "^$c.last_run_timestamp_seconds=[0-9]" || fail "alpha time: $(agent_gauges alpha)" +agent_gauges alpha | grep -q "^$c.duration_seconds=[0-9]" || fail "alpha duration: $(agent_gauges alpha)" +agent_gauges beta | grep -qx "$c.last_run_success=0" || fail "beta result: $(agent_gauges beta)" +agent_gauges gamma | grep -qx "$c.last_run_success=1" || fail "gamma result: $(agent_gauges gamma)" +[ "$(jq -rs '.[0].resourceMetrics[0].resource.attributes[0].value.stringValue' "$root/curl.all")" = swarm-archive-check ] || + fail "service.name" + +case="the check pushes the duration it measured" +if run "$root/check-fns.sh" push_check_metrics alpha 1 100 190; then + want_gauges "$c.last_run_success=1 +$c.last_run_timestamp_seconds=190 +$c.duration_seconds=90" +else + fail "failed: $(cat "$root/err")" +fi + +case="the check exits zero when every repository passes" +if FAKE_LOG="$root/fake.log" run "$root/check-fns.sh" check_agents alpha beta; then + grep -q 'archive check failed' "$root/err" && fail "failure line: $(cat "$root/err")" +else + fail "exit status was non-zero: $(cat "$root/err")" +fi + +case="the check refuses a name that is not an agent name and goes on" +: >"$root/fake.log" +FAKE_LOG="$root/fake.log" run "$root/check-fns.sh" check_agents 'bad name;x' alpha +rc=$? +[ "$rc" != 0 ] || fail "exit status was 0" +grep -qx 'archive check failed: bad name;x is not an agent name' "$root/err" || fail "no refusal: $(cat "$root/err")" +[ "$(wc -l <"$root/fake.log")" = 1 ] || fail "alpha not checked: $(cat "$root/fake.log")" + +# --- the lock the prune and the check share --- +lock_prologue() { + awk '/^if \[ -n "\$\{ARCHIVE_MAINTENANCE_LOCK:-\}" \]; then$/ {on=1} on {print} on && /^fi$/ {exit}' "$1" +} +for s in CHECK_SCRIPT PRUNE_SCRIPT; do + lock_prologue "${!s}" >"$root/lock-$s.sh" + grep -q flock "$root/lock-$s.sh" || { echo "FAILED: $s has no maintenance lock"; exit 1; } +done + +case="the check waits while the prune holds the lock, and runs once it is free" +lockfile="$root/maintenance.lock" +flock "$lockfile" sleep 5 & +holder=$! +sleep 0.5 +ARCHIVE_MAINTENANCE_LOCK="$lockfile" timeout 1 bash "$root/lock-CHECK_SCRIPT.sh" +[ "$?" = 124 ] || fail "did not wait for the held lock" +kill "$holder" 2>/dev/null +wait "$holder" 2>/dev/null +ARCHIVE_MAINTENANCE_LOCK="$lockfile" timeout 5 bash "$root/lock-CHECK_SCRIPT.sh" || fail "did not run on a free lock" + +case="the prune waits on the same lock" +flock "$lockfile" sleep 5 & +holder=$! +sleep 0.5 +ARCHIVE_MAINTENANCE_LOCK="$lockfile" timeout 1 bash "$root/lock-PRUNE_SCRIPT.sh" +[ "$?" = 124 ] || fail "did not wait for the held lock" +kill "$holder" 2>/dev/null +wait "$holder" 2>/dev/null + +case="without the lock variable the scripts take no lock" +timeout 5 bash "$root/lock-CHECK_SCRIPT.sh" || fail "failed without the variable" + [ "$failures" -eq 0 ] || exit 1 echo "all cases passed"