swarm-archive: check each agent's restic repository weekly
A weekly swarm-archive-check unit runs restic check --read-data-subset on every repository on the archive host, serialised with the prune by a shared flock. It logs archive check failed per agent, pushes check gauges, and has an alert and dashboard columns.
This commit is contained in:
atlas
parent
c4919c5945
commit
5c30985a25
2 changed files with 503 additions and 18 deletions
|
|
@ -58,10 +58,10 @@ the rest of the cell says how.
|
|||
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
|
||||
| `swarm/hives/<hive>/archive/<agent>` (fields `rest-password`, `restic-password`) | `swarm-controller`, for every agent a hive places, every 5 minutes: a new random REST password when the agent is new on that hive, the agent's restic password from the leaf below; the copy on a hive that no longer places the agent is deleted | the agent's hive: `hive-archive-backup`, under the hive's own store identity | ✅ a move to another hive writes a new REST password, which locks the old hive out | ✅ read at every backup run |
|
||||
| `swarm/services/archive/users/<agent>` (field `htpasswd`) | `swarm-controller`, with the REST password above: its `{SHA}` hash, never the password; deleted when no hive places the agent | `swarm-archive-users` on the archive's host, which renders the users file into `/run` | ✅ rewritten with the password | ✅ re-read every 2 minutes; the server notices the new file without a restart |
|
||||
| `swarm/services/archive/restic/<agent>` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune |
|
||||
| `swarm/services/archive/restic/<agent>` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` and `swarm-archive-check` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune |
|
||||
| `swarm/services/archive/ready` | `swarm-archive-users` on the archive's host, once, when none is stored; `swarm-controller` writes no archive credentials before it exists | `swarm-controller`, every 5 minutes | ❌ never rewritten | ✅ read every 5 minutes |
|
||||
| `swarm/services/archive/s3/access-key-id` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune |
|
||||
| `swarm/services/archive/s3/secret-access-key` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive` and `swarm-archive-prune` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune |
|
||||
| `swarm/services/archive/s3/access-key-id` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive`, `swarm-archive-prune` and `swarm-archive-check` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune |
|
||||
| `swarm/services/archive/s3/secret-access-key` (field `value`) | you, in the store's UI or with `bao kv put` and `value=-` and the value on stdin, before the archive's `backend` is `s3`; nothing mints it | `swarm-archive`, `swarm-archive-prune` and `swarm-archive-check` on the archive's host, which put it in rclone's environment | ❌ a new key from the provider; the archive can't replace the one it holds | ✅ read when `swarm-archive` starts, so restart it after a change; read at every prune |
|
||||
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
|
||||
|
||||
|
|
@ -188,7 +188,7 @@ login minter's, the forge host's and Grafana's, which reach more than one:
|
|||
| `swarm-stt`, in its container | `sttClientCertFile` / `sttClientKeyFile` | `swarm/services/<service client id>/oidc/client` |
|
||||
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
|
||||
| `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` |
|
||||
| `swarm-archive-users`, `swarm-archive-prune` and, with an S3 bucket, `swarm-archive` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*`, `swarm/services/archive/restic/*` and `swarm/services/archive/s3/*`, lists `swarm/services/archive/users/` |
|
||||
| `swarm-archive-users`, `swarm-archive-prune`, `swarm-archive-check` and, with an S3 bucket, `swarm-archive` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*`, `swarm/services/archive/restic/*` and `swarm/services/archive/s3/*`, lists `swarm/services/archive/users/` |
|
||||
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
|
||||
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
|
||||
|
||||
|
|
|
|||
|
|
@ -55,7 +55,7 @@ kept.
|
|||
|
||||
- `agents`: per-agent resource usage and Claude Code activity. The **activity** tab ends with a **Busiest agents** table: turns, input, output and cache-read tokens, and cost for each agent in the range, busiest first. The **turns** tab holds what each agent's own `/stats` page shows about turns: average turn duration, cache hit rate and tokens per turn, turns per hour by agent and by model, the turn duration over time (median, 95th percentile and mean), and the mix of wake sources and results. It also has the tools called, the commands bash tasks ran, reminders per hour, the first-turn context size and fresh sessions per agent: everything the `/stats` page shows, except the per-bucket token bars the **cost** tab covers. The duration percentiles are right from the first deploy that includes the histogram bounds in `hive-agent`; before it, they read a flat 10 seconds.
|
||||
- `attic`: the swarm binary cache. Agents whose latest closure this hive can pull, and why not; the builder's pushes and push failures; closures hive-c0re pulled and deploy steps it refused; the cache's size and store-path count. It reads the closure-pullable gauge, the `swarm-builder` and `hive-c0re` logs, and the `hyperhive.attic.cache_bytes` and `hyperhive.attic.cache_nars` gauges, which a timer in the `swarm-attic` container (`swarm-attic-cache-size`) reads from atticd's database every five minutes and pushes through the container's forwarder. atticd itself exports no metrics and its logs don't reach the log store, so the board has no cache request or hit panel. Filter by hive and agent.
|
||||
- `archive`: the restic backups of every agent to the swarm archive. Per agent: time since the last successful backup (red over two hours, twice the default hourly schedule), the last run's result, duration and bytes added, then snapshots, repository size and the last prune. Failures and recent runs come from the `hive-archive-backup` and `swarm-archive-prune` logs. The numbers are the `hyperhive.archive.backup.*` gauges, which each hive's backup unit pushes after every agent, and the `hyperhive.archive.prune.*` and `hyperhive.archive.repo.*` gauges, which the prune unit pushes after every repository when its host runs a hive. A hive that predates them shows no data for its agents; the log panels work regardless. Filter by hive and agent.
|
||||
- `archive`: the restic backups of every agent to the swarm archive. Per agent: time since the last successful backup (red over two hours, twice the default hourly schedule), the last run's result, duration and bytes added, then snapshots, repository size and the last prune and the last check. Failures and recent runs come from the `hive-archive-backup`, `swarm-archive-prune` and `swarm-archive-check` logs. The numbers are the `hyperhive.archive.backup.*` gauges, which each hive's backup unit pushes after every agent, the `hyperhive.archive.prune.*` and `hyperhive.archive.repo.*` gauges, which the prune unit pushes after every repository when its host runs a hive, and the `hyperhive.archive.check.*` gauges from the check unit. A hive that predates them shows no data for its agents; the log panels work regardless. Filter by hive and agent.
|
||||
- `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue.
|
||||
- `claude-usage`: Claude use by model, effort, token type and source. Filter by hive, service (agents or the review classifier) and agent. The service filter separates the classifier's API-key spend from the agents.
|
||||
- `egress`: what agents send through the egress proxy, in three tabs. **overview** has the totals (requests, agents active, refused, credential challenges, data transferred), requests over time by status, the busiest agents and destinations, and data transferred by domain over time (the top 10 domains plus an other line, summed from the proxy's access log in VictoriaLogs; the proxy logs a CONNECT tunnel when it closes, so a long-lived connection shows as one spike at close). **refused** lists the hosts the proxy refused for each agent. **would-deny** lists what agents in `log` mode reached that restricted mode would refuse. The host lists are tables, one row per host (and agent), with a count column you can sort. Filter by agent and by domain, to see which agents queried a domain and when.
|
||||
|
|
|
|||
Loading…
Reference in a new issue