mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-archive: back up every agent's state to an append-only restic repository per agent

Browse source 
swarm-archive serves restic repositories with rclone, append-only, one login
and one repository per agent. swarm-controller writes each placed agent's
login and restic password to the store, replaces them on a move and removes
them when the agent is destroyed. Every hive backs its agents up hourly under
its own store identity. The archive host prunes by retention.
This commit is contained in:
atlas 2026-11-10 07:49:49 +01:00
commit 306aae0378
33 changed files with 2214 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

1
Cargo.lock generated
View file

@ -5107,6 +5107,7 @@ dependencies = [
"serde",
"serde_json",
"serde_norway",
"sha1",
"sha2 0.11.1",
"similar",
"strum",

3
Cargo.toml
View file

@ -252,6 +252,9 @@ matrix-sdk = { version = "0.18", default-features = false, features = [
futures-util = "0.3"
hmac = "0.13"
sha2 = "0.11"
# SHA-1 for the `{SHA}` htpasswd lines the archive server checks passwords against
# (`swarm-controller`). Already in the lock through the TLS and websocket stacks.
sha1 = "0.10"
# SHA-512 crypt (`$6$`) for the users file hive-c0re renders; squid's `basic_ncsa_auth`
# hands a `$` hash to the system `crypt()`.
sha-crypt = "0.6"

2
docs/2-handbook/swarm/README.md
View file

@ -43,6 +43,8 @@ grafana, the controller, authelia, …) →
CLI agents use. → [`forge.md`](forge.md)
- **matrix** — the swarm's homeserver, several accounts per agent, and
federation. → [`matrix.md`](matrix.md)
- **backup archive** — where the hives back up each agent's state, how long it
stays and how to check it. → [`archive.md`](archive.md)
- **knowledge repo** — the reference docs every agent reads, and how to add
one. → [`knowledge.md`](knowledge.md)
- **`swarmctl`** — the swarm operator's CLI. → [`swarmctl-cli.md`](swarmctl-cli.md)

125
docs/2-handbook/swarm/archive.md Normal file
View file

@ -0,0 +1,125 @@
# Backup archive
The archive keeps a backup of every agent's state. Each hive backs its agents
up to it every hour, and it keeps the last few days. This page covers what the
hive saves, where it goes, how long it stays and how to check that it works.
Restoring an agent from a backup isn't built yet.
## What's in the backup
For each agent a hive places, the hive saves the agent's `state/`, `claude/` and
`harness/` directories. `claude/` holds the agent's login, so a restored agent
keeps its session. Caches and `work/` stay out.
Two options on the hive change that:
- `deploy.hive-controller.archive.paths` lists the entries of the agent's
directory to save. The hive skips an entry the agent doesn't have.
- `deploy.hive-controller.archive.scope = "subvolume"` saves the whole agent
directory instead, `work/` included. Add `archive.excludes` patterns to keep
build output out.
On a btrfs hive the hive saves a read-only snapshot of the agent's subvolume, so
it copies the databases at one instant. Otherwise it saves the directory in
place.
## Where it goes
The archive runs as `rclone serve restic` on the host that sets
`deploy.archive.enable`, behind the gateway at `archive.<swarm domain>`. Every
hive finds it by that name, so which host runs it makes no difference to them.
`allSwarmServices` turns it on wherever the host has the archive's store identity,
as for the binary cache.
Each agent has its own [restic](https://restic.net) repository at
`https://archive.<swarm domain>/<agent>/`, under `deploy.archive.dataDir`
(default `/var/lib/swarm-archive`) on that host. restic encrypts it with a
password of its own for that agent.
Two properties protect the backups from a compromised hive:
- **One login per agent.** `swarm-controller` creates a login for each agent
when it places the agent. The login reaches that agent's repository and no
other. A hive holds the logins of the agents it places and no others. When an
agent moves, the controller gives the new hive a new password and deletes the
old hive's copy, so the old hive can't reach the repository any more.
- **Append-only.** A client can add snapshots and nothing else: no delete, no
overwrite. The archive host removes old snapshots.
## How long it stays
The archive host prunes every repository once a day. A snapshot survives if
either rule keeps it:
| option | default | keeps |
| ------------------------------------- | ------- | --------------------------------- |
| `deploy.archive.retention.keepWithin` | `3d` | every snapshot younger than this |
| `deploy.archive.retention.keepLast` | `3` | the newest this many, however old |
The age rule limits what a compromised hive can do by flooding the archive with
fake snapshots: they can't push the last three days out. Set `keepWithin` to
`null` to leave that rule off. `deploy.archive.pruneSchedule` sets when the
prune runs (default `daily`).
Because the archive host prunes, it reads every agent's restic password from
the secret store and can decrypt every backup. Treat that host as part of the
swarm's trust root.
## Schedule
`deploy.hive-controller.archive.schedule` is a systemd calendar expression
(default `hourly`). restic sends only what changed, so a run over an idle agent
is small. The first backup of a large agent sends everything.
## Check that it works
On a hive, `systemctl status hive-archive-backup` shows the last run, and
`journalctl -u hive-archive-backup` lists every agent it backed up or failed to
back up. `systemctl list-timers hive-archive-backup` shows the next run.
To see an agent's snapshots, ask the archive with that agent's password, which
the secret store holds at `swarm/hives/<hive>/archive/<agent>` (fields
`rest-password` and `restic-password`):
```sh
RESTIC_REPOSITORY="rest:https://archive.<swarm domain>/<agent>/" \
RESTIC_REST_USERNAME=<agent> RESTIC_REST_PASSWORD=<rest-password> \
restic snapshots
```
restic asks for the `restic-password`. On the archive host, `systemctl status
swarm-archive swarm-archive-users swarm-archive-prune` shows the server, the
users file refresh and the last prune.
## When it fails
- **The `archive backup failed` alert.** A hive's run failed for an agent or
failed to log in. The journal line `archive backup failed: <agent>` names the
agent, and the lines above it say why. The alert waits 30 minutes, so an
agent placed a few minutes ago doesn't trip it while its login is on its way.
The next hourly run retries.
- **`swarm-controller has granted this hive no archive credentials`.** The
hive has agents and the store holds no login for any of them. Either
`swarm-controller` doesn't have the archive yet (update it), or its archive
pass fails: look for `archive:` lines in its journal.
- **The `archive prune failed` alert.** The daily prune couldn't apply the
retention to one repository, for example because a backup held its lock for
longer than ten minutes. The next prune retries.
- **A swarm without an archive.** If no host sets `deploy.archive.enable`, every
hive's run fails. Set `deploy.hive-controller.archive.enable = false` on the
hives until one does.
## Mixed versions
A hive that doesn't have the backup unit yet doesn't back up, and nothing else
changes. A `swarm-controller` without the archive pass writes no logins, so
hives with the unit fail with the message above until it updates. The archive
host admits only the users the controller wrote, so until then it admits nobody.
## What lives where
Everything secret lives in the swarm secret store, and the archive keeps none
on disk. The archive host renders its users file (password hashes only) into
`/run/swarm-archive-auth/`, which is tmpfs. Hives read their restic password
when restic asks for it. The paths and who writes and reads them are in the
[credentials reference](credentials-reference.md).

4
docs/2-handbook/swarm/credentials-reference.md
View file

@ -57,6 +57,9 @@ the rest of the cell says how.
| `swarm/services/attic/public-key` | `swarm-bao-attic-tokens` on the cache's host, when none is stored and again whenever the cache reports another key (the key `attic cache info` prints); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (`<name>:<base64>`) | `swarm-controller`, which copies it to each hive | ✅ on the cache's host, the unit's next run replaces any other value, a hand edit included, with the cache's key; with an external cache, replace it yourself | ✅ copied to the hives every 5 minutes |
| `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}` | `swarm-controller`, from the four `swarm/services/attic/` leaves, every 5 minutes, and before each deploy to the hive; it skips a leaf whose source holds nothing, and a copy whose source is gone stays | the hive (`hive-c0re`) | ✅ a changed source is written over the copy in the next pass | ✅ read every 5 minutes |
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
| `swarm/hives/<hive>/archive/<agent>` (fields `rest-password`, `restic-password`) | `swarm-controller`, for every agent a hive places, every 5 minutes: a new random REST password when the agent is new on that hive, the agent's restic password from the leaf below; the copy on a hive that no longer places the agent is deleted | the agent's hive: `hive-archive-backup`, under the hive's own store identity | ✅ a move to another hive writes a new REST password, which locks the old hive out | ✅ read at every backup run |
| `swarm/services/archive/users/<agent>` (field `htpasswd`) | `swarm-controller`, with the REST password above: its `{SHA}` hash, never the password; deleted when no hive places the agent | `swarm-archive-users` on the archive's host, which renders the users file into `/run` | ✅ rewritten with the password | ✅ re-read every 2 minutes; the server notices the new file without a restart |
| `swarm/services/archive/restic/<agent>` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
@ -181,6 +184,7 @@ login minter's, the forge host's and Grafana's, which reach more than one:
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
| `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` |
| `swarm-archive-users` and `swarm-archive-prune` | `archiveClientCertFile` / `archiveClientKeyFile` | reads `swarm/services/archive/users/*` and `swarm/services/archive/restic/*`, lists `swarm/services/archive/users/` |
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |

2
docs/2-handbook/swarm/credentials.md
View file

@ -442,7 +442,7 @@ each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`,
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`, `deploy.bao.archiveCommonName`,
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
The last three are the writers, and the last is the one leaf two units share:
the forge's host holds one store identity for the token it publishes and the

1
docs/2-handbook/swarm/services.md
View file

@ -119,6 +119,7 @@ look before you switch them on.
| CI runner | Runs Forgejo Actions for pull requests. Heavy. | `deploy.ci-runner.enable` | [CI runner](#ci-runner) |
| Builder | Evaluates and builds agent config PRs. Heavy. | `deploy.builder.enable` | [Builder](#builder) |
| Binary cache | Serves built closures to the hives. | `deploy.attic.enable` | [Binary cache](#binary-cache) |
| Backup archive | Keeps every agent's state backups, one repository each. | `deploy.archive.enable` | [Backup archive](archive.md) |
| CI metrics | Records CI run times in the metrics store. | `deploy.ci-metrics.enable` | [CI metrics](#ci-metrics) |
| Review classifier | Checks PRs against a repo's review rules. | `deploy.review-classifier.enable` | [Review classifier](#review-classifier) |
| Claude login minter | Runs the **log in** button on an agent's card. | `deploy.claude-login.enable` | [Claude login minter](#claude-login-minter) |

2
docs/4-implementation/scheduler/observability.md
View file

@ -390,7 +390,7 @@ in `services.hyperhive.otel.journalUnits`: `hive-c0re.service`, the `container@*
units (the start-time rootfs check lines), plus
`squid.service` and `squid-selfcheck.service` (see [Egress proxy](../networking/network.md#egress-proxy-squid)).
A host that runs the cache without the swarm collector adds `swarm-bao-attic-secret.service`
and `swarm-bao-attic-tokens.service`.
and `swarm-bao-attic-tokens.service`; a host that runs the archive adds `swarm-archive.service`, `swarm-archive-users.service` and `swarm-archive-prune.service`, and a hive adds `hive-archive-backup.service`.
A host that also runs the swarm collector names none of these: its whole-host journal receiver
ships them already, and the hive collector then has no `journald` receiver.
No bind mount is needed. Unlike an agent's journal, this collector already

2
docs/README.md
View file

@ -59,6 +59,8 @@ The services that exist once per swarm. Start at
- **How to log an agent in to claude from the swarm UI, and why doesn't a
stale stored copy log it out again?** →
[`2-handbook/swarm/claude-login.md`](2-handbook/swarm/claude-login.md).
- **Where do the hives back up each agent's state, how long does it stay, and how to
check it?** → [`2-handbook/swarm/archive.md`](2-handbook/swarm/archive.md).
### Hives

2
nix/host-modules/default.nix
View file

@ -22,6 +22,7 @@
./hive-matrix.nix
./hive-network.nix
./hive-priv.nix
./hive-archive-backup.nix
./hive-priv-cache.nix
./hive-squid.nix
./hive-squid-credential.nix
@ -65,6 +66,7 @@
./glue-swarm-otel-oidc-client.nix
./glue-victorialogs-otel-oidc-client.nix
./glue-victoriametrics-otel-oidc-client.nix
./swarm-archive.nix
./swarm-attic-options.nix
./swarm-attic.nix
./swarm-authelia-service.nix

19
nix/host-modules/deploy.nix
View file

@ -734,6 +734,25 @@ in
'';
};
archive.enable = lib.mkOption {
type = lib.types.bool;
default = false;
example = true;
description = ''
Run the swarm's backup archive on this host: `rclone serve restic`
behind the gateway at `archive.<swarm domain>`, append-only, with one
login and one repository per agent, and a daily prune that applies the
retention. Every hive's backup unit finds it by that name, so it runs
on one host and the host does not matter. Run it on one host.
On by default where
{option}`services.hyperhive.deploy.allSwarmServices` is set and this
host has the archive's store identity (the pair
{option}`services.hyperhive.deploy.bao.archiveClientCertFile` and
`archiveClientKeyFile`, minted on the store's host).
'';
};
hive-controller.enable = lib.mkOption {
type = lib.types.bool;
default = false;

10
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -90,6 +90,16 @@ let
here = deployCfg.attic.enable;
policies = [ "swarm-bao-attic-policy" ];
};
# ./swarm-archive.nix, under the archive's leaf, which reads the users and the
# repositories' passwords only once that policy is written.
swarm-archive-users = {
here = deployCfg.archive.enable;
policies = [ "swarm-bao-archive-policy" ];
};
swarm-archive-prune = {
here = deployCfg.archive.enable;
policies = [ "swarm-bao-archive-policy" ];
};
# ./hive-matrix.nix, under matrix-ctl's leaf.
swarm-bao-matrix-oidc = {
here = deployCfg.matrix.enable;

3
nix/host-modules/glue-bao-readers-store-order.nix
View file

@ -69,6 +69,9 @@ let
# ./swarm-attic.nix
swarm-bao-attic-secret = deployCfg.attic.enable;
swarm-bao-attic-tokens = deployCfg.attic.enable;
# ./swarm-archive.nix
swarm-archive-users = deployCfg.archive.enable;
swarm-archive-prune = deployCfg.archive.enable;
# ./swarm-otel.nix
swarm-bao-otel-oidc =
deployCfg.swarm-otel.enable

9
nix/host-modules/glue-bao-tls.nix
View file

@ -126,6 +126,8 @@ in
notifyRouterClientKeyFile = lib.mkDefault "${pkiDir}/notify-router-key.pem";
atticClientCertFile = lib.mkDefault "${pkiDir}/attic.pem";
atticClientKeyFile = lib.mkDefault "${pkiDir}/attic-key.pem";
archiveClientCertFile = lib.mkDefault "${pkiDir}/archive.pem";
archiveClientKeyFile = lib.mkDefault "${pkiDir}/archive-key.pem";
forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem";
forgePublisherClientKeyFile = lib.mkDefault "${pkiDir}/forge-publisher-key.pem";
servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
@ -337,6 +339,13 @@ in
[ -s ${pkiDir}/attic.pem ] || ${signLeaf} ${pkiDir} attic \
${lib.escapeShellArg deployCfg.bao.atticCommonName} "" clientAuth
# The backup archive's, which reads its users and the repositories'
# passwords. Minted here for the matrix-ctl leaf's reason: the archive
# is a swarm service, so on an archive host without the store this is
# the file an operator copies.
[ -s ${pkiDir}/archive.pem ] || ${signLeaf} ${pkiDir} archive \
${lib.escapeShellArg deployCfg.bao.archiveCommonName} "" clientAuth
# The forge's token publisher. Minted here for the matrix-ctl leaf's
# reason: the forge is a swarm singleton, so on a forge host without the
# store this is the file an operator copies.

185
nix/host-modules/hive-archive-backup.nix Normal file
View file

@ -0,0 +1,185 @@
# The hive's backup of its agents' state to the swarm archive
# (./swarm-archive.nix): one restic repository per agent, at
# `https://archive.<swarm domain>/<agent>/`.
#
# A root oneshot on a timer, not hive-c0re and not a hive-priv verb. Root,
# because the agents' files belong to the agents (and, under user namespaces,
# to remapped ids) and hive-c0re cannot read them; a unit of its own, because
# no caller-supplied argument may reach root, and this one takes none.
#
# Each agent backs up under the credentials swarm-controller wrote for it at
# `swarm/hives/<this hive>/archive/<agent>`
# (../../swarm-controller/src/archive_access.rs), read with the hive's own
# store identity: the agent's login to the archive, and its restic password,
# which restic reads through `RESTIC_PASSWORD_COMMAND` and nothing writes to
# disk. An agent the controller has not granted this hive is skipped; a hive
# with agents and no grant at all fails, which is what a swarm-controller
# without the archive looks like.
#
# A failed agent prints `archive backup failed: <agent>` to the journal,
# which the swarm's `archive backup failed` alert reads, and fails the unit
# after the others have been tried.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
hiveController = hyperhiveCfg.deploy.hive-controller;
cfg = hiveController.archive;
tls = hiveController.tls;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = hyperhiveCfg.deploy.bao;
enabled =
cfg.enable
&& hiveController.enable
&& hiveController.hiveName != null
&& tls.baoHiveIdentityCertFile != null
&& tls.baoHiveIdentityKeyFile != null;
# The CA bundle restic trusts: the host's, then the swarm's root, which the
# archive's certificate chains to when the gateway is self-signed. Written by
# the unit at each run; `--cacert` replaces restic's own roots.
runtimeDir = "/run/hive-archive-backup";
caBundle = "${runtimeDir}/ca.pem";
swarmCa = "${tls.stateDir}/swarm-services-root.pem";
storeContainer = lib.optional baoDeploy.enable "container@${baoCfg.machine}.service";
backup = import ./lib/archive-backup.nix { inherit lib; } {
inherit (cfg) scope paths excludes;
restic = cfg.package;
hive = hiveController.hiveName;
agentRoot = "/var/lib/hyperhive/agents";
url = "https://${hyperhiveCfg.swarm.archive.ownDomain}";
inherit caBundle;
};
in
{
options.services.hyperhive.deploy.hive-controller.archive = {
enable = lib.mkOption {
type = lib.types.bool;
default = true;
description = ''
Back this hive's agents up to the swarm archive. On wherever the hive
has its store identity; a swarm that runs no archive
({option}`services.hyperhive.deploy.archive.enable` on one host) sees
every run fail, so turn it off there.
'';
};
schedule = lib.mkOption {
type = lib.types.str;
default = "hourly";
example = "*-*-* 00/6:00:00";
description = ''
systemd calendar expression for the backup. Each run sends only what
changed since the last, so an hourly run on an idle agent is small.
'';
};
scope = lib.mkOption {
type = lib.types.enum [
"paths"
"subvolume"
];
default = "paths";
description = ''
What of each agent is backed up. `paths`: the entries of
{option}`services.hyperhive.deploy.hive-controller.archive.paths`
inside the agent's directory. `subvolume`: the whole directory, which
adds `work/` and everything an agent keeps there.
'';
};
paths = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [
"state"
"claude"
"harness"
];
description = ''
Entries of an agent's directory backed up under `scope = "paths"`: its
state, its claude login and the harness. Caches and `work/` are left
out unless listed. An entry the agent does not have is skipped.
'';
};
excludes = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [ ];
example = [ "*/target" ];
description = "restic `--exclude` patterns, on top of `--exclude-caches`.";
};
package = lib.mkOption {
type = lib.types.package;
default = pkgs.restic;
defaultText = lib.literalExpression "pkgs.restic";
description = "The restic package.";
};
};
config = lib.mkIf enabled {
services.hyperhive.otel.journalUnits = lib.optionals (!hyperhiveCfg.deploy.swarm-otel.enable) [
"hive-archive-backup.service"
];
systemd.services.hive-archive-backup = {
description = "back up this hive's agents to the swarm archive";
wants = [ "network-online.target" ];
after = [ "network-online.target" ] ++ storeContainer;
path = [
baoDeploy.package
pkgs.btrfs-progs
pkgs.coreutils
pkgs.gnugrep
pkgs.jq
];
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = tls.baoHiveIdentityCertFile;
BAO_CLIENT_KEY = tls.baoHiveIdentityKeyFile;
RESTIC_CACHE_DIR = "/var/cache/hive-archive-backup";
}
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
serviceConfig = {
Type = "oneshot";
UMask = "0077";
SyslogIdentifier = "hive-archive-backup";
RuntimeDirectory = "hive-archive-backup";
CacheDirectory = "hive-archive-backup";
# A first backup of a large agent sends all of it.
TimeoutStartSec = "6h";
Nice = 10;
IOSchedulingClass = "idle";
};
script = ''
set -euo pipefail
# A missing file is left out: the bundle is the host's roots alone
# until the swarm's root is there.
cat ${lib.escapeShellArg (toString config.security.pki.caBundle)} > ${caBundle}
if [ -s ${lib.escapeShellArg swarmCa} ]; then
cat ${lib.escapeShellArg swarmCa} >> ${caBundle}
fi
${backup}
'';
};
systemd.timers.hive-archive-backup = {
description = "back up this hive's agents to the swarm archive";
wantedBy = [ "timers.target" ];
timerConfig = {
OnCalendar = cfg.schedule;
RandomizedDelaySec = "5min";
Persistent = true;
};
};
};
}

137
nix/host-modules/lib/archive-backup.nix Normal file
View file

@ -0,0 +1,137 @@
# The hive's backup run: for each agent that has a state directory here and a
# grant from swarm-controller, one `restic backup` into that agent's repository
# on the archive. A pure function, NOT a NixOS module: ../hive-archive-backup.nix
# wraps it in a unit, and the VM test (nix/vm-tests/swarm-archive.nix) runs the
# same script against a stand-in `bao`.
#
# Needs `bao`, `jq`, `coreutils`, `gnugrep`, `btrfs-progs` on the unit's `path`
# and the `BAO_*` client set. Every failure prints a line starting
# `archive backup failed`, which the swarm's alert reads, and ends the run with
# a non-zero status once the other agents have been tried.
{ lib }:
{
restic,
# The hive's name in the store: grants are read from
# `secret/swarm/hives/<hive>/archive/`.
hive,
# The directory holding one directory per agent.
agentRoot,
# The repository server's base URL, no trailing slash.
url,
# "paths": `paths` under each agent's directory. "subvolume": the whole directory.
scope,
paths,
excludes,
# A CA file to trust instead of restic's default, or null.
caBundle,
}:
let
storeSeededSecret = import ./store-seeded-secret.nix { };
grants = hive: "secret/swarm/hives/${hive}/archive";
in
''
set -euo pipefail
${storeSeededSecret}
store_init
store_login
if [ "$STORE_UP" != 1 ]; then
echo "archive backup failed: could not log in to the swarm secret store" >&2
exit 1
fi
${lib.optionalString (caBundle != null) ''
export RESTIC_CACERT=${lib.escapeShellArg caBundle}
''}
root=${lib.escapeShellArg agentRoot}
grants=${lib.escapeShellArg (grants hive)}
# One agent: a read-only snapshot of its subvolume when it is one, so the
# databases are copied at one instant, else its directory in place.
backup_agent() {
local agent="$1" leaf="$grants/$1" rest base snap="" target targets=()
rest="$(store_bao kv get -field=rest-password "$leaf")" || return 1
export RESTIC_REPOSITORY="rest:${url}/$agent/"
export RESTIC_REST_USERNAME="$agent"
export RESTIC_REST_PASSWORD="$rest"
export RESTIC_PASSWORD_COMMAND="bao kv get -field=restic-password $leaf"
if ! ${restic}/bin/restic cat config >/dev/null 2>&1; then
${restic}/bin/restic init || return 1
fi
base="$root/$agent"
if btrfs subvolume show "$base" >/dev/null 2>&1; then
snap="$root/.$agent.snapshot.archive"
btrfs subvolume delete "$snap" >/dev/null 2>&1 || true
btrfs subvolume snapshot -r "$base" "$snap" >/dev/null || return 1
base="$snap"
fi
${
if scope == "subvolume" then
''targets=("$base")''
else
''
for target in ${lib.escapeShellArgs paths}; do
[ -e "$base/$target" ] && targets+=("$base/$target")
done''
}
local rc=0
if [ "''${#targets[@]}" = 0 ]; then
echo "$agent has none of its backup paths yet; nothing to back up."
else
${restic}/bin/restic backup --host "$agent" --tag archive --exclude-caches \
${lib.concatMapStringsSep " " (e: "--exclude ${lib.escapeShellArg e}") excludes} \
-- "''${targets[@]}" || rc=$?
fi
if [ -n "$snap" ]; then
btrfs subvolume delete "$snap" >/dev/null || rc=1
fi
return "$rc"
}
# The agents the controller granted this hive; none yet is "No value found".
rc=0
granted="$(store_bao kv list -format=json "$grants" 2>"$STORE_ERR")" || rc=$?
if [ "$rc" != 0 ]; then
if grep -q 'No value found' "$STORE_ERR"; then
granted='[]'
else
cat "$STORE_ERR" >&2
echo "archive backup failed: could not list $grants" >&2
exit 1
fi
fi
granted="$(printf '%s' "$granted" | jq -r '.[]')"
failed=0
candidates=0
backed_up=0
for dir in "$root"/*/; do
agent="$(basename -- "$dir")"
case "$agent" in .*) continue ;; esac
[[ "$agent" =~ ^[A-Za-z0-9_-]+$ ]] || continue
candidates=$((candidates + 1))
if ! grep -qxF -- "$agent" <<<"$granted"; then
echo "$agent has no archive credentials on this hive; skipping."
continue
fi
if (backup_agent "$agent"); then
backed_up=$((backed_up + 1))
echo "backed up $agent"
else
echo "archive backup failed: $agent" >&2
failed=1
fi
done
# Agents here and not one of them granted: swarm-controller has not written
# this hive's grants, which a controller without the archive never does.
if [ "$candidates" -gt 0 ] && [ -z "$granted" ]; then
echo "archive backup failed: swarm-controller has granted this hive no archive credentials ($candidates agents here)" >&2
failed=1
fi
exit "$failed"
''

156
nix/host-modules/lib/archive-server.nix Normal file
View file

@ -0,0 +1,156 @@
# The archive server's moving parts: the rclone command that serves the
# repositories, the script that renders its users file from the secret store,
# and the script that prunes each repository. A pure function, NOT a NixOS
# module: ../swarm-archive.nix wraps the pieces in units, and the VM test
# (nix/vm-tests/swarm-archive.nix) runs the same scripts against a stand-in
# `bao`.
#
# Both scripts use only `bao login -method=cert -token-only`,
# `bao kv list -format=json` and `bao kv get -field=…`, and need `bao`, `jq`,
# `coreutils`, `gnugrep` on the unit's `path` and the `BAO_*` client set.
{ lib }:
{
rclone,
restic,
# Where the repositories live: `<reposDir>/<agent>/`. The only place that
# names the backend; `rclone serve restic` takes any remote here.
reposDir,
# The address rclone listens on.
addr,
# The users file rclone checks logins against, its group and the user the
# server runs as.
authFile,
authGroup,
# `{ keepWithin, keepLast }`; a null `keepWithin` leaves that flag off.
retention,
}:
let
atomicWriteSecret = import ./atomic-write-secret.nix { };
refreshConsumer = import ./refresh-consumer.nix { };
storeSeededSecret = import ./store-seeded-secret.nix { };
usersPrefix = "secret/swarm/services/archive/users";
resticPrefix = "secret/swarm/services/archive/restic";
keepFlags = lib.concatStringsSep " " (
[ "--keep-last ${toString retention.keepLast}" ]
++ lib.optional (
retention.keepWithin != null
) "--keep-within ${lib.escapeShellArg retention.keepWithin}"
);
in
{
inherit usersPrefix resticPrefix;
# Append-only: a client cannot delete or overwrite a snapshot, a pack or the
# index, so a hive that is compromised cannot erase history. Private repos:
# a login reaches `/<its own name>/` and nothing else.
serverCommand = lib.escapeShellArgs [
"${rclone}/bin/rclone"
"serve"
"restic"
"--addr"
addr
"--append-only"
"--private-repos"
"--htpasswd"
authFile
reposDir
];
# Renders the users file from `usersPrefix`, one leaf per agent holding that
# agent's htpasswd line. The file is written only when the content changed,
# and never from a partial read: any leaf that cannot be read leaves the file
# as it is.
usersScript = ''
set -euo pipefail
${storeSeededSecret}
${atomicWriteSecret}
${refreshConsumer}
file=${lib.escapeShellArg authFile}
# The server will not start without the file; an empty one admits nobody.
if [ ! -e "$file" ]; then
atomic_write_secret 0640 root:${authGroup} "$file" ""
fi
store_init
store_login
if [ "$STORE_UP" != 1 ]; then
echo "could not log in to the swarm secret store; $file stays as it is." >&2
exit 1
fi
# A prefix that holds nothing is "No value found"; it means no agent has a
# user yet, not a failure.
rc=0
names="$(store_bao kv list -format=json ${lib.escapeShellArg usersPrefix} 2>"$STORE_ERR")" || rc=$?
if [ "$rc" != 0 ]; then
if grep -q 'No value found' "$STORE_ERR"; then
names='[]'
else
cat "$STORE_ERR" >&2
echo "could not list ${usersPrefix}; $file stays as it is." >&2
exit 1
fi
fi
content=""
while IFS= read -r name; do
[ -n "$name" ] || continue
case "$name" in */) continue ;; esac
if ! line="$(store_bao kv get -field=htpasswd ${lib.escapeShellArg usersPrefix}/"$name" 2>"$STORE_ERR")"; then
cat "$STORE_ERR" >&2
echo "could not read the user $name; $file stays as it is." >&2
exit 1
fi
# The line must be this agent's own: a leaf cannot add another user.
if [[ ! "$name" =~ ^[A-Za-z0-9_-]+$ ]] || [[ "$line" != "$name:{SHA}"* ]] \
|| [[ ! "$line" =~ ^[A-Za-z0-9_-]+:\{SHA\}[A-Za-z0-9+/=]+$ ]]; then
echo "the user leaf $name does not hold a {SHA} line for $name; leaving it out." >&2
continue
fi
content+="$line"$'\n'
done < <(printf '%s' "$names" | jq -r '.[]')
content="''${content%$'\n'}"
if secret_differs "$file" "$content"; then
atomic_write_secret 0640 root:${authGroup} "$file" "$content"
echo "users file updated."
fi
'';
# `restic forget --prune` on every repository, with its password read from
# the store at run time. The server refuses deletes from clients, so this is
# the only place history is dropped.
pruneScript = ''
set -euo pipefail
${storeSeededSecret}
store_init
store_login
if [ "$STORE_UP" != 1 ]; then
echo "archive prune failed: could not log in to the swarm secret store" >&2
exit 1
fi
failed=0
for repo in ${lib.escapeShellArg reposDir}/*/; do
[ -e "$repo/config" ] || continue
agent="$(basename -- "$repo")"
if [[ ! "$agent" =~ ^[A-Za-z0-9_-]+$ ]]; then
echo "archive prune failed: $agent is not an agent name" >&2
failed=1
continue
fi
export RESTIC_PASSWORD_COMMAND="bao kv get -field=password ${resticPrefix}/$agent"
if ${restic}/bin/restic -r "$repo" --retry-lock 10m forget --group-by host ${keepFlags} --prune; then
echo "pruned $agent"
else
echo "archive prune failed: $agent" >&2
failed=1
fi
done
exit "$failed"
'';
}

348
nix/host-modules/swarm-archive.nix Normal file
View file

@ -0,0 +1,348 @@
# The swarm's backup archive: `rclone serve restic` on the host that sets
# `deploy.archive.enable`, behind the gateway at `archive.<swarm domain>`.
# Every hive's backup unit (./hive-archive-backup.nix) writes one restic
# repository per agent to it; nothing else uses it.
#
# - `--append-only`: a client cannot delete or overwrite a snapshot, a pack or
# the index, so a compromised hive cannot erase history. Dropping history is
# the prune unit's job, on this host, by retention.
# - `--private-repos`: a login reaches `/<its own name>/` only. There is one
# login per agent, created by swarm-controller (../../swarm-controller/src/archive_access.rs)
# when it places the agent, replaced when it moves and removed when it is
# destroyed. An agent's hive can write that agent's repository and no other.
#
# Everything secret lives in the swarm secret store and is read when needed:
#
# - `swarm-archive-users` (root, every two minutes) renders the htpasswd file
# from the users leaves into `/run`, which is tmpfs. rclone re-reads the file
# when it changes. The leaves hold password hashes only.
# - `swarm-archive-prune` (daily) reads each agent's restic password through
# `RESTIC_PASSWORD_COMMAND` for its `restic forget --prune`. This host can
# therefore decrypt every backup, which is the cost of pruning where clients
# cannot delete.
#
# Both units log in as the archive's own store principal (a leaf of the store's
# CA, minted by ./glue-bao-tls.nix). Plain host units rather than a container:
# no secret crosses a mount, and the backend is one string
# (`reposDir`, in ./lib/archive-server.nix) that names where the repositories
# are.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
cfg = hyperhiveCfg.deploy.archive;
archiveCfg = hyperhiveCfg.swarm.archive;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = hyperhiveCfg.deploy.bao;
swarmDomain = hyperhiveCfg.swarm.domain;
# Total on a null swarm domain, as the sibling options modules are: the
# required-domain assertion in hive-network.nix is what an operator sees.
domainBase = if swarmDomain == null then "invalid" else swarmDomain;
user = "swarm-archive";
authDir = "/run/swarm-archive-auth";
authFile = "${authDir}/htpasswd";
reposDir = "${cfg.dataDir}/repos";
server = import ./lib/archive-server.nix { inherit lib; } {
inherit (cfg) retention;
inherit authFile reposDir;
rclone = cfg.package;
restic = cfg.resticPackage;
addr = "127.0.0.1:${toString cfg.port}";
authGroup = user;
};
storeRetry = import ./lib/store-retry.nix { };
haveClientIdentity =
baoDeploy.archiveClientCertFile != null && baoDeploy.archiveClientKeyFile != null;
storeEnvironment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
in
{
options.services.hyperhive.swarm.archive.ownDomain = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "archive.${domainBase}";
defaultText = lib.literalExpression ''"archive.''${services.hyperhive.swarm.domain}"'';
description = ''
Read-only: the name the swarm's backup archive is served under and every
hive's backup unit dials. A sibling of the swarm's other service names, so
the swarm-services sub-CA can issue for it, and in `swarm.serviceDomains`
on every host whether or not one runs the archive.
'';
};
options.services.hyperhive.deploy.archive = {
package = lib.mkOption {
type = lib.types.package;
default = pkgs.rclone;
defaultText = lib.literalExpression "pkgs.rclone";
description = "The rclone package: `rclone serve restic`.";
};
resticPackage = lib.mkOption {
type = lib.types.package;
default = pkgs.restic;
defaultText = lib.literalExpression "pkgs.restic";
description = "The restic package the prune unit runs.";
};
port = lib.mkOption {
type = lib.types.port;
default = 8911;
description = ''
Port rclone listens on, on the host's loopback. The gateway's vhost
proxies to it; every login is checked by rclone itself.
'';
};
dataDir = lib.mkOption {
type = lib.types.path;
default = "/var/lib/swarm-archive";
description = ''
Where the repositories live, under `repos/<agent>/`. Holds restic
ciphertext: every backup of every agent, so size it for that.
'';
};
retention = {
keepWithin = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = "3d";
example = "14d";
description = ''
Every snapshot younger than this is kept (restic's `--keep-within`,
written `1y2m3d4h`). It bounds how fast history can be evicted: a
hive that appends many snapshots cannot push the last few days out.
`null` leaves the flag off.
'';
};
keepLast = lib.mkOption {
type = lib.types.ints.positive;
default = 3;
description = ''
The newest this many snapshots of each agent are kept, however old
(restic's `--keep-last`). Together with
{option}`services.hyperhive.deploy.archive.retention.keepWithin` the
prune keeps a snapshot either rule keeps.
'';
};
};
pruneSchedule = lib.mkOption {
type = lib.types.str;
default = "daily";
example = "*-*-* 04:00:00";
description = "systemd calendar expression for the prune: the retention rules are applied then.";
};
};
config = lib.mkIf cfg.enable {
assertions = [
{
assertion = haveClientIdentity;
message = ''
services.hyperhive.deploy.archive.enable requires the archive's
swarm-secret-store client identity: set both
services.hyperhive.deploy.bao.archiveClientCertFile
services.hyperhive.deploy.bao.archiveClientKeyFile
The archive reads its users and the repositories' passwords in the
store with it.
Both default to the pair glue-bao-tls.nix mints when
services.hyperhive.deploy.bao.enable is true (the host that runs the
store). Otherwise copy archive.pem and archive-key.pem from the store
host's /var/lib/swarm-bao-pki/ and name them in those two options.
'';
}
];
users.users.${user} = {
isSystemUser = true;
group = user;
description = "swarm backup archive";
};
users.groups.${user} = { };
# This host serves the vhost, and the host's own resolver finds the name.
services.hyperhive.gateway.localNames = [ archiveCfg.ownDomain ];
services.hyperhive.gateway.enable = lib.mkDefault true;
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
# `forceSSL`: the logins ride in a header. No `auth_request`: rclone checks
# them, and restic cannot follow a login redirect. The vhost keeps the
# gateway's default `public`: `public = false` needs the gateway's
# `privateListenAddresses` set, and every request is authenticated either
# way. A backup is one request that can run for minutes, so the body is
# neither size-limited nor buffered.
services.hyperhive.gateway.vhosts."${archiveCfg.ownDomain}" = {
forceSSL = true;
locations."/" = {
proxyPass = "http://127.0.0.1:${toString cfg.port}";
extraConfig = ''
client_max_body_size 0;
proxy_request_buffering off;
proxy_read_timeout 1h;
proxy_send_timeout 1h;
'';
};
};
services.hyperhive.otel.journalUnits = lib.optionals (!hyperhiveCfg.deploy.swarm-otel.enable) [
"swarm-archive.service"
"swarm-archive-users.service"
"swarm-archive-prune.service"
];
systemd.tmpfiles.rules = [
"d ${cfg.dataDir} 0750 ${user} ${user} -"
"d ${reposDir} 0750 ${user} ${user} -"
"d ${authDir} 0750 root ${user} -"
];
systemd.services.swarm-archive = {
description = "swarm backup archive (rclone serve restic, append-only, one repository per agent)";
wantedBy = [ "multi-user.target" ];
wants = [ "swarm-archive-users.service" ];
after = [
"network.target"
"swarm-archive-users.service"
"systemd-tmpfiles-setup.service"
];
# rclone writes no config, and reads none.
environment.RCLONE_CONFIG = "/dev/null";
serviceConfig = {
ExecStart = server.serverCommand;
User = user;
Group = user;
Restart = "on-failure";
RestartSec = 5;
SyslogIdentifier = "swarm-archive";
NoNewPrivileges = true;
ProtectSystem = "strict";
ProtectHome = true;
PrivateTmp = true;
PrivateDevices = true;
ProtectKernelTunables = true;
ProtectControlGroups = true;
RestrictAddressFamilies = [
"AF_INET"
"AF_INET6"
];
ReadWritePaths = [ reposDir ];
ReadOnlyPaths = [ authDir ];
};
};
# The users file, from the store. A failed run leaves the file as it is;
# the timer and the retry bring a store that comes back up to date.
systemd.services.swarm-archive-users = {
description = "render the archive's users file from the swarm secret store";
wantedBy = [ "multi-user.target" ];
wants = [ "network-online.target" ];
after = [
"network-online.target"
"systemd-tmpfiles-setup.service"
];
path = [
baoDeploy.package
pkgs.coreutils
pkgs.gnugrep
pkgs.jq
pkgs.systemd
];
environment = storeEnvironment // {
BAO_CLIENT_CERT = baoDeploy.archiveClientCertFile;
BAO_CLIENT_KEY = baoDeploy.archiveClientKeyFile;
};
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
UMask = "0027";
SyslogIdentifier = "swarm-archive-users";
TimeoutStartSec = "5min";
};
script = server.usersScript;
};
# A new grant reaches the server within the controller's pass (five
# minutes) plus this interval.
systemd.timers.swarm-archive-users = {
description = "re-render the archive's users file";
wantedBy = [ "timers.target" ];
timerConfig = {
OnUnitInactiveSec = "2min";
RandomizedDelaySec = "20s";
};
};
systemd.services.swarm-archive-prune = {
description = "apply the archive's retention to every agent's repository";
after = [ "network-online.target" ];
wants = [ "network-online.target" ];
path = [
baoDeploy.package
cfg.resticPackage
pkgs.coreutils
pkgs.gnugrep
pkgs.jq
];
# The unit runs as the repositories' owner, so the store identity comes
# in as credentials, which only it can read.
environment = {
BAO_ADDR = storeEnvironment.BAO_ADDR;
BAO_CLIENT_CERT = "%d/bao-cert";
BAO_CLIENT_KEY = "%d/bao-key";
RESTIC_CACHE_DIR = "%C/swarm-archive-prune";
}
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = "%d/bao-ca";
};
serviceConfig = {
Type = "oneshot";
User = user;
Group = user;
SyslogIdentifier = "swarm-archive-prune";
CacheDirectory = "swarm-archive-prune";
LoadCredential = [
"bao-cert:${baoDeploy.archiveClientCertFile}"
"bao-key:${baoDeploy.archiveClientKeyFile}"
]
++ lib.optional (baoDeploy.serverCaFile != null) "bao-ca:${baoDeploy.serverCaFile}";
NoNewPrivileges = true;
ProtectSystem = "strict";
ProtectHome = true;
PrivateTmp = true;
PrivateDevices = true;
ReadWritePaths = [ reposDir ];
TimeoutStartSec = "6h";
};
script = server.pruneScript;
};
systemd.timers.swarm-archive-prune = {
description = "apply the archive's retention";
wantedBy = [ "timers.target" ];
timerConfig = {
OnCalendar = cfg.pruneSchedule;
RandomizedDelaySec = "30min";
Persistent = true;
};
};
};
}

98
nix/host-modules/swarm-bao.nix
View file

@ -518,7 +518,39 @@ let
capabilities = ["create", "read", "update"]
}
'') atticHiveFactNames}
path "${credentialMountPath}/data/${ciRunnerTokenLeaf}" {
# The archive credentials of the agents each hive places
# (swarm-controller's archive_access.rs): the hive's copy and the archive's
# hash and restic password, listed to find what no placement justifies and
# deleted when a move or a destroy revokes it.
path "${credentialMountPath}/data/swarm/hives/+/archive/*" {
capabilities = ["create", "read", "update"]
}
path "${credentialMountPath}/metadata/swarm/hives/+/archive/" {
capabilities = ["list"]
}
path "${credentialMountPath}/metadata/swarm/hives/+/archive/+" {
capabilities = ["delete"]
}
path "${credentialMountPath}/data/swarm/services/archive/users/*" {
capabilities = ["create", "read", "update"]
}
path "${credentialMountPath}/data/swarm/services/archive/restic/*" {
capabilities = ["create", "read"]
}
path "${credentialMountPath}/metadata/swarm/services/archive/users/" {
capabilities = ["list"]
}
path "${credentialMountPath}/metadata/swarm/services/archive/users/+" {
capabilities = ["delete"]
}
path "${credentialMountPath}/data/${ciRunnerTokenLeaf}" {
capabilities = ["create", "update"]
}
@ -998,7 +1030,11 @@ let
"public-key"
"pull-token"
]
+ readStanza "${credentialMountPath}/data/swarm/hives/${obj.hiveName}/squid-selfcheck";
+ readStanza "${credentialMountPath}/data/swarm/hives/${obj.hiveName}/squid-selfcheck"
# The archive credentials swarm-controller writes for the agents this hive
# places: the backup unit (./hive-archive-backup.nix) lists them and reads each.
+ readStanza "${credentialMountPath}/data/swarm/hives/${obj.hiveName}/archive/*"
+ listStanza "${credentialMountPath}/metadata/swarm/hives/${obj.hiveName}/archive/";
# The PKI mount agent client certificates are issued from. Its root is
# generated inside the store, so the agent CA's key never exists outside it.
@ -1103,6 +1139,12 @@ let
}
'';
listStanza = path: ''
path "${path}" {
capabilities = ["list"]
}
'';
# Singletons, so a list of one rather than a second shape: the granting unit
# below takes a list either way and the units stay identical apart from
# the objects they write.
@ -1339,6 +1381,20 @@ let
}
];
# The backup archive: the users file is rendered from one leaf per agent, and
# the prune reads each agent's restic password. Read and list only; the
# controller writes both.
archiveReaders = [
{
name = "swarm-archive";
cn = baoDeploy.archiveCommonName;
policyText =
readStanza "${credentialMountPath}/data/swarm/services/archive/users/*"
+ readStanza "${credentialMountPath}/data/swarm/services/archive/restic/*"
+ listStanza "${credentialMountPath}/metadata/swarm/services/archive/users/";
}
];
# The CI run poller: its forge token and its own OIDC client secret, both
# read once per pass by the host unit.
ciMetricsReaders = [
@ -2611,6 +2667,42 @@ in
'';
};
archiveCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-archive";
description = ''
Subject the store's `swarm-archive` cert-auth role accepts: the identity
the backup archive's host units present. Its grant is read on
`swarm/services/archive/users/*` and `restic/*`, and list on `users`.
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
'';
};
archiveClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/archive.pem";
description = ''
Certificate the backup archive's host units present to the store. Its
subject must be {option}`services.hyperhive.deploy.bao.archiveCommonName`.
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
On an archive host without the store, it is a file the operator copies
from the store's host. A path, never a value.
'';
};
archiveClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/archive-key.pem";
description = ''
Private key for {option}`services.hyperhive.deploy.bao.archiveClientCertFile`.
A path, never a value.
'';
};
atticClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
@ -3847,6 +3939,8 @@ in
systemd.services.swarm-bao-attic-policy = readerPolicyUnit "write the in-swarm cache's secret-creator bao policy and cert-auth role" atticWriters;
systemd.services.swarm-bao-archive-policy = readerPolicyUnit "write the backup archive's user and password reader bao policy and cert-auth role" archiveReaders;
systemd.services.swarm-bao-notify-router-policy = readerPolicyUnit "write the notification router's client-secret-reader bao policy and cert-auth role" notifyRouterReaders;
systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters;

23
nix/host-modules/swarm-grafana.nix
View file

@ -1684,6 +1684,29 @@ in
for = "0s";
summary = "swarm-controller failed to mint agent {{ $labels.F_AGENT }}'s store identity; the reason is the `F_ERROR` field of the swarm-controller log line";
})
# `hive-archive-backup` (nix/host-modules/hive-archive-backup.nix)
# prints one `archive backup failed: <agent>` line per agent
# it could not back up, and one for a hive swarm-controller
# has granted nothing. The backup runs hourly, so `for`
# rides out the minutes a newly placed agent waits for its
# grant, and a second failed run keeps it firing.
(logCountRule {
uid = "hh-archive-backup-failed";
title = "archive backup failed";
windowSeconds = 7800;
expr = ''_time:130m _SYSTEMD_UNIT:="hive-archive-backup.service" _msg:"archive backup failed" | stats by (_HOSTNAME) count() as failures'';
for = "30m";
summary = "{{ $labels._HOSTNAME }} failed to back up an agent to the archive; the unit's journal names the agent and the cause";
})
# `swarm-archive-prune` prints `archive prune failed: <agent>`.
(logCountRule {
uid = "hh-archive-prune-failed";
title = "archive prune failed";
windowSeconds = 90000;
expr = ''_time:25h _SYSTEMD_UNIT:="swarm-archive-prune.service" _msg:"archive prune failed" | stats count() as failures'';
for = "0s";
summary = "the archive's daily prune failed for an agent's repository, so its retention is not applied; the unit's journal names the agent";
})
# Logged once per detected crash, by `emit_crash_transitions`
# in hive-c0re's crash_watch.rs (10s poll interval). Replaces
# the old per-hive dashboard `agents_crashing` banner — see

Some files were not shown because too many files have changed in this diff Show more