From 458a046b8d76de01cb49bcc137e614bff17b0023 Mon Sep 17 00:00:00 2001 From: atlas Date: Tue, 27 Oct 2026 12:23:28 +0100 Subject: [PATCH] run attic as a swarm service with deploy.attic.enable --- docs/swarm/builder.md | 55 ++- docs/swarm/credentials-reference.md | 67 ++-- docs/swarm/credentials.md | 12 +- docs/swarm/services.md | 19 + nix/checks.nix | 4 + nix/host-modules/default.nix | 1 + nix/host-modules/deploy.nix | 22 ++ .../glue-bao-readers-policy-order.nix | 10 + .../glue-bao-readers-store-order.nix | 3 + nix/host-modules/glue-bao-tls.nix | 9 + nix/host-modules/lib/attic-init.nix | 165 +++++++++ nix/host-modules/swarm-attic-options.nix | 43 ++- nix/host-modules/swarm-attic.nix | 332 ++++++++++++++++++ nix/host-modules/swarm-bao.nix | 72 +++- nix/host-modules/swarm-required-services.nix | 9 + nix/host-modules/swarm.nix | 5 + nix/module-eval/bao-grants.nix | 59 +++- nix/module-eval/builder.nix | 135 ++++++- nix/module-eval/swarm-services-switch.nix | 11 + nix/script-tests/attic-init.nix | 41 +++ nix/script-tests/attic-init.sh | 232 ++++++++++++ swarm-builder/src/push.rs | 2 +- 22 files changed, 1242 insertions(+), 66 deletions(-) create mode 100644 nix/host-modules/lib/attic-init.nix create mode 100644 nix/host-modules/swarm-attic.nix create mode 100644 nix/script-tests/attic-init.nix create mode 100644 nix/script-tests/attic-init.sh diff --git a/docs/swarm/builder.md b/docs/swarm/builder.md index f9e8af86..3f6941dc 100644 --- a/docs/swarm/builder.md +++ b/docs/swarm/builder.md @@ -259,8 +259,8 @@ grant the forge refuses fails the build with no verdict. ## Binary cache With `services.hyperhive.swarm.attic.domain` and `cache` set, the builder pushes -each green meta build's closure to that cache, an attic server outside the swarm, -before it replies. It runs `attic push swarm: ` once per build, +each green meta build's closure to that cache, an attic server inside or outside the +swarm, before it replies. It runs `attic push swarm: ` once per build, not `attic watch`, and the builder pushes again when the controller runs a build again. The push has ten minutes, or less when the reply is due sooner, because it must end inside the 61 minutes the controller waits. `attic` checks the server's certificate @@ -271,8 +271,8 @@ Set both options or neither: evaluation refuses a domain without a cache name. The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`, starting with a letter or digit, and evaluation refuses any other name. -The push token is the operator's to write. The builder reads it on every push -from `secret/swarm/services/attic/push-token` (field `value`), writes it to +The push token is the swarm's own cache's to create, or the operator's to write for an +external one. The builder reads it on every push from `secret/swarm/services/attic/push-token` (field `value`), writes it to a file and an `attic` config naming that file (`token-file`) in the unit's private runtime directory (`/run/swarm-builder`, mode `0700`, both files `0600`), points `attic` at the config with `XDG_CONFIG_HOME` and removes both after the push. @@ -294,6 +294,43 @@ Mixed versions: A record written before the field existed also reads as `false`. +### The swarm's own cache + +`services.hyperhive.deploy.attic.enable` runs an attic server in a `swarm-attic` +container behind the gateway, at `attic.`. It defaults on where +you set `deploy.allSwarmServices` and the host holds the cache's store identity +(`deploy.bao.atticClient{Cert,Key}File`, which the store's host mints). You write +no secret: on first start, two host units log in to the store as the cache's own +principal and create what's missing. + +- `swarm-bao-attic-secret` creates the key atticd signs tokens with at + `secret/swarm/services/attic/jwt-secret` and writes it where the container + reads it. +- `swarm-bao-attic-tokens` creates the cache and mints the push token, which + the builder reads, at `secret/swarm/services/attic/push-token`, and a + read-only pull token for the hives at `secret/swarm/services/attic/pull-token`. + +A unit creates each secret only while the store holds none, so a second start +mints nothing. Both tokens last ten years and nothing renews them. To replace one, +write the new token to its path in the store: nothing mints over a value that +exists. A new signing key invalidates both tokens. + +With the switch on, `swarm.attic.domain` and `swarm.attic.cache` default to +`attic.` and `swarm`, so the builder on that host pushes to it. +Naming either option yourself wins over its default. The defaults apply only on +the host that runs the cache: on any other host, name both options too, with the +same values. + +Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store +path nobody added or fetched for 14 days goes at the next garbage collection, +which runs every 12 hours (`deploy.attic.retentionPeriod`). The next build of +that agent pushes a closure the cache lost. + +The gateway terminates TLS with a swarm services certificate and forwards to +`127.0.0.1:8910` (`deploy.attic.port`) with no login in front: attic checks its +own tokens, and a nix client can't follow a login redirect. The container logs +to its journal only; its journal doesn't reach the swarm log store. + ### External cache The cache can be an attic tenant on an instance outside the swarm. To point @@ -313,10 +350,12 @@ pins nothing, so a Let's Encrypt certificate needs no setup. The builder reads the token again on every push, so a new token takes effect on the next build. -A later change adds the swarm's own attic service: `deploy.attic.enable` will run atticd -in the swarm and fill these two options itself. An operator who sets that option -to false keeps the external cache by setting the two options by hand and -seeding the token as above. +Where you host the shared services, `deploy.attic.enable` defaults on and runs +the swarm's own cache. To use an external one, set it to `false` with +`lib.mkForce`, then set the two options by hand and seed the token as above. +The swarm then renders no container, vhost or unit of its own cache, and +creates no secret in the store. The store's host still mints the cache's +leaf and writes its role, which nothing logs in to. ## Credentials diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index 282250ab..ef3055eb 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -14,36 +14,38 @@ the rest of the cell says how. -| store path | minter | reader | automatic re-mint | automatic re-pull | -| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `swarm/agents//matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection | -| `swarm/agents//matrix/` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection | -| `swarm/agents//matrix-recovery/` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up | -| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl` in `hive-matrix`, once | `swarm-controller` | ❌ minted once | ✅ read every 5 minutes | -| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it | -| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 | -| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 | -| `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read at start | -| `swarm/agents//bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid | -| `swarm/agents//queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) | -| `swarm/agents//forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes | -| `swarm/agents//forge/