vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

copy the cache's pull facts to each hive and create them in the in-swarm cache

Browse source 
the in-swarm cache stores its address, name and public key beside its tokens, create-only. swarm-controller copies the four cache facts to swarm/hives/<hive>/attic/* every five minutes, with read on the sources and write on the copies. part 1 of 3: nothing reads the copies until the hive reader and probe land.
This commit is contained in:
atlas 2026-10-27 15:51:13 +01:00 • committed by mara
commit 1e5070796d
15 changed files with 640 additions and 67 deletions
Download patch file Download diff file Expand all files Collapse all files

25
docs/swarm/builder.md
View file

@ -311,12 +311,21 @@ principal and create what's missing.
- `swarm-bao-attic-tokens` creates the cache and mints the push token, which - `swarm-bao-attic-tokens` creates the cache and mints the push token, which
the builder reads, at `secret/swarm/services/attic/push-token`, and a the builder reads, at `secret/swarm/services/attic/push-token`, and a
read-only pull token for the hives at `secret/swarm/services/attic/pull-token`. read-only pull token for the hives at `secret/swarm/services/attic/pull-token`.
It also stores the facts a hive pulls with: the cache's address at
`secret/swarm/services/attic/url` (`https://<swarm.attic.domain>`), its name at
`secret/swarm/services/attic/cache`, and the public key it signs with at
`secret/swarm/services/attic/public-key` (the key `attic cache info` prints).
A unit creates each secret only while the store holds none, so a second start A unit creates each secret only while the store holds none, so a second start
mints nothing. Both tokens last ten years and nothing renews them. To replace one, mints nothing. Both tokens last ten years and nothing renews them. To replace one,
write the new token to its path in the store: nothing mints over a value that write the new token to its path in the store: nothing mints over a value that
exists. A new signing key invalidates both tokens. exists. A new signing key invalidates both tokens.
The unit creates the address, name and key once and never updates them. After
you change `swarm.attic.domain`, or after the server's database goes and the
cache gets a new key, delete the stale leaf and restart
`swarm-bao-attic-tokens`.
`swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>` `swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
and `swarm` on every host, whether or not it runs the cache, so a builder on any and `swarm` on every host, whether or not it runs the cache, so a builder on any
host pushes to it. host pushes to it.
@ -345,6 +354,14 @@ the swarm at an existing tenant:
or `bao kv put` with `value=-` and the token on stdin, so the token is in no or `bao kv put` with `value=-` and the token on stdin, so the token is in no
command line. The token needs push rights on that cache. Until it command line. The token needs push rights on that cache. Until it
exists, every push fails and builds stay green with `cached` false. exists, every push fails and builds stay green with `cached` false.
Seed the hives' pull token the same way at
`secret/swarm/services/attic/pull-token`, with read rights on that cache.
3. Seed the three facts hives pull with, each as field `value`:
`secret/swarm/services/attic/url` (`https://` and the host, no trailing
slash), `secret/swarm/services/attic/cache` (the cache's name) and
`secret/swarm/services/attic/public-key` (`<name>:<base64>`, from the
cache's page or `attic cache info`). The controller copies the leaves that
exist.
The builder checks the instance's certificate against the system roots and The builder checks the instance's certificate against the system roots and
pins nothing, so a Let's Encrypt certificate needs no setup. The builder pins nothing, so a Let's Encrypt certificate needs no setup. The builder
@ -358,6 +375,14 @@ The swarm then renders no container, vhost or unit of its own cache, and
creates no secret in the store. The store's host still mints the cache's creates no secret in the store. The store's host still mints the cache's
leaf and writes its role, which nothing logs in to. leaf and writes its role, which nothing logs in to.
## What hives do with the cache
Every five minutes `swarm-controller` copies the four `swarm/services/attic/`
leaves (`pull-token`, `url`, `cache`, `public-key`) to
`swarm/hives/<hive>/attic/<leaf>` for each hive, because a hive reads only its
own prefix. It writes a copy only when the value differs, skips a leaf the
store doesn't hold, and leaves a copy whose source disappeared.
## Credentials ## Credentials
| What | Where | Who reads it | | What | Where | Who reads it |

72
docs/swarm/credentials-reference.md
View file

@ -14,40 +14,44 @@ the rest of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader | automatic re-mint | automatic re-pull | | store path | minter | reader | automatic re-mint | automatic re-pull |
| ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up | | `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl appservice sync` in `hive-matrix`, from the registration's `as_token` | `swarm-controller` | ❌ follows the registration below | ✅ read every 5 minutes | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl appservice sync` in `hive-matrix`, from the registration's `as_token` | `swarm-controller` | ❌ follows the registration below | ✅ read every 5 minutes |
| `swarm/controller/swarm-controller/matrix/appservice-registration` | `swarm-matrix-ctl appservice sync` in `hive-matrix`: seeded from the container's own tokens when the store holds none, a new pair only when the container has none either | `swarm-matrix-ctl` in `hive-matrix`; the controller has no grant to it | ❌ minted once; an operator who replaces it makes `sync` write the new pair into the container and restart the homeserver | ✅ `sync` re-reads it every 10 minutes | | `swarm/controller/swarm-controller/matrix/appservice-registration` | `swarm-matrix-ctl appservice sync` in `hive-matrix`: seeded from the container's own tokens when the store holds none, a new pair only when the container has none either | `swarm-matrix-ctl` in `hive-matrix`; the controller has no grant to it | ❌ minted once; an operator who replaces it makes `sync` write the new pair into the container and restart the homeserver | ✅ `sync` re-reads it every 10 minutes |
| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it | | `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 | | `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 | | `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read at start | | `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read at start |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when you link an external forge account in the swarm UI | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when you link an external forge account in the swarm UI | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when you link a GitHub account in the swarm UI | the agent | ❌ your token; delete it and link again | ✅ every 2 minutes | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when you link a GitHub account in the swarm UI | the agent | ❌ your token; delete it and link again | ✅ every 2 minutes |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting | | `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call | | `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
| `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it | | `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute; ❌ Grafana restarts when its copy changes | | `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute; ❌ Grafana restarts when its copy changes |
| `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes | | `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes |
| `swarm/services/swarm-authelia/keys/<name>`, for `jwt`, `session`, `storage-encryption`, `oidc-hmac` and `oidc-issuer` | authelia's `authelia-swarm-secrets` unit in its container, under the secret publisher's identity, from the key file an earlier release generated; a new one only when there is none | authelia | ❌ minted once; a different storage-encryption key leaves authelia's database unreadable and a different issuer key invalidates every issued token | ❌ read when the unit runs; the unit restarts authelia when the file changes | | `swarm/services/swarm-authelia/keys/<name>`, for `jwt`, `session`, `storage-encryption`, `oidc-hmac` and `oidc-issuer` | authelia's `authelia-swarm-secrets` unit in its container, under the secret publisher's identity, from the key file an earlier release generated; a new one only when there is none | authelia | ❌ minted once; a different storage-encryption key leaves authelia's database unreadable and a different issuer key invalidates every issued token | ❌ read when the unit runs; the unit restarts authelia when the file changes |
| `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run | | `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run | | `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review | | `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build | | `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it | | `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years | nothing yet | ❌ write a new token | must be stated | | `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, which copies it to each hive | ❌ write a new token | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes | | `swarm/services/attic/url` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`https://<swarm.attic.domain>`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (scheme included, no trailing slash) | `swarm-controller`, which copies it to each hive | ❌ delete it and restart `swarm-bao-attic-tokens`, or write the new value | ✅ copied to the hives every 5 minutes |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass | | `swarm/services/attic/cache` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`swarm.attic.cache`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, which copies it to each hive | ❌ as the URL | ✅ copied to the hives every 5 minutes |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated | | `swarm/services/attic/public-key` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (the key `attic cache info` prints); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (`<name>:<base64>`) | `swarm-controller`, which copies it to each hive | ❌ as the URL. A cache recreated with a new key leaves the stored one stale: every hive then reports `bad-signature` | ✅ copied to the hives every 5 minutes |
| `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}` | `swarm-controller`, from the four `swarm/services/attic/` leaves, every 5 minutes; it skips a leaf whose source holds nothing, and a copy whose source is gone stays | the hive (`hive-c0re`) | ✅ a changed source is written over the copy in the next pass | ✅ read every 5 minutes |
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

3
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -80,7 +80,8 @@ let
policies = [ "swarm-bao-grafana-oidc-policy" ]; policies = [ "swarm-bao-grafana-oidc-policy" ];
}; };
# ./swarm-attic.nix, under the cache's leaf, which may create the signing # ./swarm-attic.nix, under the cache's leaf, which may create the signing
# key and the two tokens only once that policy is written. # key, the two tokens and the cache's public facts only once that policy is
# written.
swarm-bao-attic-secret = { swarm-bao-attic-secret = {
here = deployCfg.attic.enable; here = deployCfg.attic.enable;
policies = [ "swarm-bao-attic-policy" ]; policies = [ "swarm-bao-attic-policy" ];

2
nix/host-modules/glue-bao-tls.nix
View file

@ -330,7 +330,7 @@ in
[ -s ${pkiDir}/notify-router.pem ] || ${signLeaf} ${pkiDir} notify-router \ [ -s ${pkiDir}/notify-router.pem ] || ${signLeaf} ${pkiDir} notify-router \
${lib.escapeShellArg deployCfg.bao.notifyRouterCommonName} "" clientAuth ${lib.escapeShellArg deployCfg.bao.notifyRouterCommonName} "" clientAuth
# The in-swarm cache's, which creates and reads its three secrets. # The in-swarm cache's, which creates and reads its six secrets.
# Minted here for the matrix-ctl leaf's reason: the cache is a swarm # Minted here for the matrix-ctl leaf's reason: the cache is a swarm
# service, so on a cache host without the store this is the file an # service, so on a cache host without the store this is the file an
# operator copies. # operator copies.

29
nix/host-modules/lib/attic-init.nix
View file

@ -13,14 +13,22 @@
# #
# `prelude` expects `./atomic-write-secret.nix` and `./refresh-consumer.nix` # `prelude` expects `./atomic-write-secret.nix` and `./refresh-consumer.nix`
# above it, and `bao`, `coreutils` and `systemd` on the caller's `path`. # above it, and `bao`, `coreutils` and `systemd` on the caller's `path`.
# `ensureJwt` also wants `openssl` there; `provision` wants `atticadm`, `attic` # `ensureJwt` also wants `openssl` there; `provision` wants `atticadm`, `attic`,
# and `curl`. # `curl` and `sed`.
{ lib }: { lib }:
{ {
# Store paths in the form `bao kv` takes, mount first. # Store paths in the form `bao kv` takes, mount first.
jwtPath, jwtPath,
pullPath, pullPath,
pushPath, pushPath,
# The cache's public facts, created only while absent: the server's address
# (scheme included), the cache's name, and the key its narinfo are signed
# with. Hives pull with them.
urlPath,
cachePath,
publicKeyPath,
# The address `urlPath` holds, which clients reach the server at.
url,
# The cache's name on the server. # The cache's name on the server.
cache, cache,
# Where the host reaches atticd, scheme included. # Where the host reaches atticd, scheme included.
@ -98,9 +106,9 @@ in
refresh_consumer ${q machine} ${q unit} ${q envFile} refresh_consumer ${q machine} ${q unit} ${q envFile}
''; '';
# The cache and the two tokens, once atticd answers. The tokens are signed # The cache, its two tokens and its public facts, once atticd answers. The
# here with the key the store holds; the cache is created through the API by a # tokens are signed here with the key the store holds; the cache is created
# token that lives for five minutes and is never stored. # through the API by a token that lives for five minutes and is never stored.
provision = '' provision = ''
up=0 up=0
for _ in $(seq 1 30); do for _ in $(seq 1 30); do
@ -161,5 +169,16 @@ in
exit 1 exit 1
fi fi
fi fi
# What hives pull with. The key is the one the server reports for the cache;
# `attic` prints it on stderr.
mint_url() { printf '%s' ${q url}; }
mint_cache() { printf '%s' ${q cache}; }
mint_public_key() {
attic cache info ${q "local:${cache}"} 2>&1 | sed -n 's/^ *Public Key: *//p'
}
ensure_secret ${q urlPath} mint_url >/dev/null || exit 1
ensure_secret ${q cachePath} mint_cache >/dev/null || exit 1
ensure_secret ${q publicKeyPath} mint_public_key >/dev/null || exit 1
''; '';
} }

15
nix/host-modules/swarm-attic.nix
View file

@ -6,7 +6,8 @@
# Two host units run under the cache's own store leaf, the shape # Two host units run under the cache's own store leaf, the shape
# ./swarm-grafana.nix gives its admin password. One creates the key atticd signs # ./swarm-grafana.nix gives its admin password. One creates the key atticd signs
# tokens with, if the store holds none, and hands it to the container. The other # tokens with, if the store holds none, and hands it to the container. The other
# creates the cache and the push and pull tokens, each only while absent # creates the cache, the push and pull tokens, and the facts hives pull with
# (its address, name and public key), each only while absent
# (./lib/attic-init.nix). Neither needs a manual step. # (./lib/attic-init.nix). Neither needs a manual step.
# #
# The container shares the host's network namespace, as ./swarm-victorialogs.nix # The container shares the host's network namespace, as ./swarm-victorialogs.nix
@ -73,6 +74,10 @@ let
jwtPath = "secret/swarm/services/attic/jwt-secret"; jwtPath = "secret/swarm/services/attic/jwt-secret";
pullPath = "secret/swarm/services/attic/pull-token"; pullPath = "secret/swarm/services/attic/pull-token";
pushPath = "secret/swarm/services/attic/push-token"; pushPath = "secret/swarm/services/attic/push-token";
urlPath = "secret/swarm/services/attic/url";
cachePath = "secret/swarm/services/attic/cache";
publicKeyPath = "secret/swarm/services/attic/public-key";
url = "https://${atticCfg.domain}";
cache = atticCfg.cache; cache = atticCfg.cache;
inherit inherit
endpoint endpoint
@ -295,10 +300,11 @@ in
''; '';
}; };
# The cache and its two tokens, once atticd answers. The push token is the # The cache, its two tokens and its public facts, once atticd answers. The
# path the swarm builder reads; the pull token is for the hives. # push token is the path the swarm builder reads; the pull token and the
# facts are for the hives, which the controller copies them to.
systemd.services.swarm-bao-attic-tokens = { systemd.services.swarm-bao-attic-tokens = {
description = "create the swarm cache, and its push and pull tokens in the secret store, if absent"; description = "create the swarm cache, and its tokens and public facts in the secret store, if absent";
after = [ after = [
"container@${machine}.service" "container@${machine}.service"
"swarm-bao-attic-secret.service" "swarm-bao-attic-secret.service"
@ -311,6 +317,7 @@ in
cfg.clientPackage cfg.clientPackage
pkgs.coreutils pkgs.coreutils
pkgs.curl pkgs.curl
pkgs.gnused
]; ];
inherit (storeRetry) startLimitBurst startLimitIntervalSec; inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // { serviceConfig = storeRetry.serviceConfig // {

48
nix/host-modules/swarm-bao.nix
View file

@ -482,6 +482,14 @@ let
capabilities = ["create", "read", "update"] capabilities = ["create", "read", "update"]
} }
${lib.concatMapStrings readStanza (
map (leaf: "${credentialMountPath}/data/${leaf}") atticFactLeaves
)}
${lib.concatMapStrings (name: ''
path "${credentialMountPath}/data/swarm/hives/+/attic/${name}" {
capabilities = ["create", "read", "update"]
}
'') atticHiveFactNames}
path "${credentialMountPath}/data/${ciRunnerTokenLeaf}" { path "${credentialMountPath}/data/${ciRunnerTokenLeaf}" {
capabilities = ["create", "update"] capabilities = ["create", "update"]
} }
@ -718,6 +726,28 @@ let
atticJwtLeaf = "swarm/services/attic/jwt-secret"; atticJwtLeaf = "swarm/services/attic/jwt-secret";
atticPullTokenLeaf = "swarm/services/attic/pull-token"; atticPullTokenLeaf = "swarm/services/attic/pull-token";
# The cache's three public facts, beside the pull token: its address, its
# name, and the key its narinfo are signed with. The in-swarm cache's host
# unit creates them; for an external cache the operator writes them.
# `swarm-controller` reads all four and copies each to
# `swarm/hives/<hive>/attic/<name>`
# (`swarm_secret_client::attic::Leaf`).
atticUrlLeaf = "swarm/services/attic/url";
atticCacheLeaf = "swarm/services/attic/cache";
atticPublicKeyLeaf = "swarm/services/attic/public-key";
atticFactLeaves = [
atticPullTokenLeaf
atticUrlLeaf
atticCacheLeaf
atticPublicKeyLeaf
];
atticHiveFactNames = [
"pull-token"
"url"
"cache"
"public-key"
];
# The CI run poller's forge token, the nix half of # The CI run poller's forge token, the nix half of
# `swarm_secret_client::ci_metrics::forge_token_path`. The controller mints it # `swarm_secret_client::ci_metrics::forge_token_path`. The controller mints it
# and reads it back before deciding whether to re-mint; the poller reads it # and reads it back before deciding whether to re-mint; the poller reads it
@ -1199,7 +1229,7 @@ let
} }
]; ];
# The in-swarm cache's three secrets, create and read and nothing more: its # The in-swarm cache's six secrets, create and read and nothing more: its
# host units create each one only while it is absent, so none needs `update`, # host units create each one only while it is absent, so none needs `update`,
# and a rewrite of a signing key or a token an operator holds is refused. The # and a rewrite of a signing key or a token an operator holds is refused. The
# cache container's journal forwarder reads its client secret under the same # cache container's journal forwarder reads its client secret under the same
@ -1215,11 +1245,13 @@ let
capabilities = ["create", "read"] capabilities = ["create", "read"]
} }
'') '')
[ (
atticJwtLeaf [
atticPushTokenLeaf atticJwtLeaf
atticPullTokenLeaf atticPushTokenLeaf
] ]
++ atticFactLeaves
)
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.attic.otel.clientId}/oidc/client"; + readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.attic.otel.clientId}/oidc/client";
} }
]; ];
@ -2474,8 +2506,8 @@ in
description = '' description = ''
Subject the store's `swarm-attic` cert-auth role accepts: the identity Subject the store's `swarm-attic` cert-auth role accepts: the identity
the in-swarm cache's host units present. Its grant is create and read on the in-swarm cache's host units present. Its grant is create and read on
three paths, `swarm/services/attic/jwt-secret`, `push-token` and six paths under `swarm/services/attic/`, `jwt-secret`, `push-token`,
`pull-token`, and nothing else. `pull-token`, `url`, `cache` and `public-key`, and nothing else.
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings. ⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
''; '';

61
nix/module-eval/bao-grants.nix
View file

@ -1722,6 +1722,38 @@ let
&& !(lib.hasInfix "secret/data/swarm/hives/*" s) && !(lib.hasInfix "secret/data/swarm/hives/*" s)
&& !(lib.hasInfix "secret/metadata/swarm/hives" s); && !(lib.hasInfix "secret/metadata/swarm/hives" s);
} }
{
# `attic_copy` reads the cache's four facts and writes each into every
# hive's `attic/` prefix. Pinned as whole stanzas, per leaf: a read of
# the services prefix would reach every service's secrets, and `*` under
# `hives/` every hive's queue credential. The read side has no
# `create`/`update`, so the controller cannot replace the cache's token.
name = "the controller reads the cache's four facts and writes a copy of each into every hive";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
leaves = [
"pull-token"
"url"
"cache"
"public-key"
];
grants = grantsIn s;
in
lib.all (
l:
(grantFor grants "secret/data/swarm/services/attic/${l}").caps == [ "read" ]
&&
(grantFor grants "secret/data/swarm/hives/+/attic/${l}").caps == [
"create"
"read"
"update"
]
) leaves
&& lib.hasInfix "path \"secret/data/swarm/hives/+/attic/pull-token\" {\n capabilities = [\"create\", \"read\", \"update\"]\n}" s
&& (grantFor grants "secret/data/swarm/services/attic/jwt-secret") == null
&& (grantFor grants "secret/data/swarm/services/attic/push-token") == null;
}
{ {
# `forge::runner_token` writes the runner's registration token and keeps # `forge::runner_token` writes the runner's registration token and keeps
# the last value in memory, so its stanza has no `read`. Pinned as the # the last value in memory, so its stanza has no `read`. Pinned as the
@ -1988,11 +2020,11 @@ let
lib.hasInfix "/notify-router.pem ]" s && lib.hasInfix "swarm-notify-router \"\" clientAuth" s; lib.hasInfix "/notify-router.pem ]" s && lib.hasInfix "swarm-notify-router \"\" clientAuth" s;
} }
{ {
# The cache's host units create each of its three secrets and read them # The cache's host units create each of its six secrets and read them
# back, and may not replace one: no `update`. Its container's journal # back, and may not replace one: no `update`. Its container's journal
# forwarder reads its own client secret, read only. The notification # forwarder reads its own client secret, read only. The notification
# router's grant is the control that `grantsIn` reads a one-path grant too. # router's grant is the control that `grantsIn` reads a one-path grant too.
name = "the cache's grant is create and read on its signing key and its two tokens, read on its forwarder client, and nothing else"; name = "the cache's grant is create and read on its signing key, its two tokens and its three public facts, read on its forwarder client, and nothing else";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-attic-policy.script; s = baoGrantHere.systemd.services.swarm-bao-attic-policy.script;
@ -2005,24 +2037,33 @@ let
(grantFor grants "secret/data/swarm/services/attic/jwt-secret").caps == caps (grantFor grants "secret/data/swarm/services/attic/jwt-secret").caps == caps
&& (grantFor grants "secret/data/swarm/services/attic/push-token").caps == caps && (grantFor grants "secret/data/swarm/services/attic/push-token").caps == caps
&& (grantFor grants "secret/data/swarm/services/attic/pull-token").caps == caps && (grantFor grants "secret/data/swarm/services/attic/pull-token").caps == caps
&& (grantFor grants "secret/data/swarm/services/attic/url").caps == caps
&& (grantFor grants "secret/data/swarm/services/attic/cache").caps == caps
&& (grantFor grants "secret/data/swarm/services/attic/public-key").caps == caps
&& &&
(grantFor grants "secret/data/swarm/services/swarm-attic-collector/oidc/client").caps == [ "read" ] (grantFor grants "secret/data/swarm/services/swarm-attic-collector/oidc/client").caps == [ "read" ]
&& lib.length grants == 4 && lib.length grants == 7
&& lib.length (grantsIn baoGrantHere.systemd.services.swarm-bao-notify-router-policy.script) == 1 && lib.length (grantsIn baoGrantHere.systemd.services.swarm-bao-notify-router-policy.script) == 1
&& lib.hasInfix "auth/cert/certs/swarm-attic" s && lib.hasInfix "auth/cert/certs/swarm-attic" s
&& lib.hasInfix "allowed_common_names=swarm-attic" s && lib.hasInfix "allowed_common_names=swarm-attic" s
&& lib.hasInfix "token_policies=swarm-attic" s; && lib.hasInfix "token_policies=swarm-attic" s;
} }
{ {
# The signing key and the pull token are the cache's alone; the push # The signing key is the cache's alone; the push token is also the
# token is also the builder's, and is read there. # builder's, and is read there. The pull token is also the controller's,
name = "no granting unit but the cache's names its signing key or its pull token"; # which only reads it (pinned above).
name = "no granting unit but the cache's names its signing key, and only it and the controller name its pull token";
ok = ok =
lib.filter ( lib.filter (
u: u: lib.hasInfix "services/attic/jwt-secret" baoGrantHere.systemd.services.${u}.script
lib.hasInfix "services/attic/jwt-secret" baoGrantHere.systemd.services.${u}.script ) grantingUnitNames == [ "swarm-bao-attic-policy" ]
|| lib.hasInfix "services/attic/pull-token" baoGrantHere.systemd.services.${u}.script &&
) grantingUnitNames == [ "swarm-bao-attic-policy" ]; lib.filter (
u: lib.hasInfix "services/attic/pull-token" baoGrantHere.systemd.services.${u}.script
) grantingUnitNames == [
"swarm-bao-controller-policy"
"swarm-bao-attic-policy"
];
} }
{ {
name = "the PKI unit signs the cache's leaf under its own subject"; name = "the PKI unit signs the cache's leaf under its own subject";

4
nix/script-tests/attic-init.nix
View file

@ -13,6 +13,10 @@ let
jwtPath = "secret/swarm/services/attic/jwt-secret"; jwtPath = "secret/swarm/services/attic/jwt-secret";
pullPath = "secret/swarm/services/attic/pull-token"; pullPath = "secret/swarm/services/attic/pull-token";
pushPath = "secret/swarm/services/attic/push-token"; pushPath = "secret/swarm/services/attic/push-token";
urlPath = "secret/swarm/services/attic/url";
cachePath = "secret/swarm/services/attic/cache";
publicKeyPath = "secret/swarm/services/attic/public-key";
url = "http://cache.example";
cache = "grid"; cache = "grid";
endpoint = "http://127.0.0.1:8910"; endpoint = "http://127.0.0.1:8910";
configFile = "@DIR@/server.toml"; configFile = "@DIR@/server.toml";

13
nix/script-tests/attic-init.sh
View file

@ -84,7 +84,9 @@ tokenfile=\$(sed -n 's/^token-file = "\(.*\)"$/\1/p' "\$XDG_CONFIG_HOME/attic/co
token=\$(cat "\$tokenfile") token=\$(cat "\$tokenfile")
echo "attic \$* | token=\$token" >>"$calls" echo "attic \$* | token=\$token" >>"$calls"
case "\$1 \$2" in case "\$1 \$2" in
"cache info") [ -f "\$DIR/cache-exists" ] ;; "cache info")
[ -f "\$DIR/cache-exists" ] && echo " Public Key: grid:FAKEKEY" >&2
;;
"cache create") touch "\$DIR/cache-exists" ;; "cache create") touch "\$DIR/cache-exists" ;;
*) exit 1 ;; *) exit 1 ;;
esac esac
@ -123,6 +125,9 @@ stored() { cat "$dir/store/$1" 2>/dev/null || true; }
JWT=secret_swarm_services_attic_jwt-secret JWT=secret_swarm_services_attic_jwt-secret
PULL=secret_swarm_services_attic_pull-token PULL=secret_swarm_services_attic_pull-token
PUSH=secret_swarm_services_attic_push-token PUSH=secret_swarm_services_attic_push-token
URL=secret_swarm_services_attic_url
CACHE=secret_swarm_services_attic_cache
KEY=secret_swarm_services_attic_public-key
ENVFILE=$dir/secrets/server.env ENVFILE=$dir/secrets/server.env
expect() { # <description> <got> <want> expect() { # <description> <got> <want>
@ -179,6 +184,9 @@ expect "exit status" $? 0
expect "pull token stored" "$(stored $PULL)" "tok-swarm-hives-pull-1" expect "pull token stored" "$(stored $PULL)" "tok-swarm-hives-pull-1"
expect "push token stored" "$(stored $PUSH)" "tok-swarm-builder-push-2" expect "push token stored" "$(stored $PUSH)" "tok-swarm-builder-push-2"
expect "cache created once" "$(n '^attic cache create')" 1 expect "cache created once" "$(n '^attic cache create')" 1
expect "the address is stored" "$(stored $URL)" "http://cache.example"
expect "the cache name is stored" "$(stored $CACHE)" "grid"
expect "the cache's public key is stored" "$(stored $KEY)" "grid:FAKEKEY"
expect "cache created by the short-lived token" "$(n '^attic cache create local:grid | token=tok-swarm-attic-init-3')" 1 expect "cache created by the short-lived token" "$(n '^attic cache create local:grid | token=tok-swarm-attic-init-3')" 1
expect "tokens minted: pull, push, the creator" "$(n '^atticadm')" 3 expect "tokens minted: pull, push, the creator" "$(n '^atticadm')" 3
expect "pull token reads only the cache" "$(n 'make-token --sub swarm-hives-pull --validity 10y --pull grid$')" 1 expect "pull token reads only the cache" "$(n 'make-token --sub swarm-hives-pull --validity 10y --pull grid$')" 1
@ -197,6 +205,7 @@ expect "no cache create" "$(n '^attic cache create')" 0
expect "the pull token looked the cache up" "$(n "^attic cache info local:grid | token=$pull")" 1 expect "the pull token looked the cache up" "$(n "^attic cache info local:grid | token=$pull")" 1
expect "pull token unchanged" "$(stored $PULL)" "$pull" expect "pull token unchanged" "$(stored $PULL)" "$pull"
expect "push token unchanged" "$(stored $PUSH)" "$push" expect "push token unchanged" "$(stored $PUSH)" "$push"
expect "address, name and key unchanged" "$(stored $URL)$(stored $CACHE)$(stored $KEY)" "http://cache.examplegridgrid:FAKEKEY"
case="an empty server gets its cache back with the stored tokens" case="an empty server gets its cache back with the stored tokens"
rm -f "$dir/cache-exists" rm -f "$dir/cache-exists"
@ -223,7 +232,7 @@ run "$ENSURE_JWT"
run "$PROVISION" run "$PROVISION"
expect "no token among the client's arguments" "$(grep -c -- '^attic [^|]*tok-' "$calls" || true)" 0 expect "no token among the client's arguments" "$(grep -c -- '^attic [^|]*tok-' "$calls" || true)" 0
# The control: the client did see tokens, through its config file. # The control: the client did see tokens, through its config file.
expect "tokens seen through the config file" "$(grep -c -- '| token=tok-' "$calls" || true)" 3 expect "tokens seen through the config file" "$(grep -c -- '| token=tok-' "$calls" || true)" 4
if [ "$failures" -ne 0 ]; then if [ "$failures" -ne 0 ]; then
echo "$failures of $count checks failed" >&2 echo "$failures of $count checks failed" >&2

323
swarm-controller/src/attic_copy.rs Normal file
View file

@ -0,0 +1,323 @@
//! The cache facts a hive pulls with, copied to where the hive may read them.
//!
//! The cache's operator keeps four facts under `swarm/services/attic/`
//! ([`swarm_secret_client::attic`]). A hive's store identity reads
//! `swarm/hives/<hive>/*` and nothing under `swarm/services/`, so every
//! [`RECONCILE_INTERVAL`] this writes each fact that is stored to every hive's
//! `attic/` prefix, and leaves a copy that already holds the value alone.
//!
//! A fact that is not stored is not copied, and a copy whose source is gone
//! stays: the pass never deletes. A swarm with no cache stores none, so the
//! pass writes nothing.
use std::sync::Arc;
use std::time::Duration;
use anyhow::{Context, Result};
use swarm_secret_client::SecretStore;
use swarm_secret_client::attic::{Fact, Leaf, hive_path, service_path};
/// How often [`spawn`] copies.
const RECONCILE_INTERVAL: Duration = Duration::from_mins(5);
/// A pass that has not finished by now is given up on; the next tick retries.
const PASS_DEADLINE: Duration = Duration::from_mins(2);
/// The two store calls a pass makes, so a test can stand in for the store.
pub(crate) trait FactStore {
/// The fact at `path`; `None` when nothing is stored there.
async fn read(&self, path: &str) -> Result<Option<String>>;
/// Store `value` at `path`, replacing what is there.
async fn write(&self, path: &str, value: &str) -> Result<()>;
}
impl FactStore for SecretStore {
async fn read(&self, path: &str) -> Result<Option<String>> {
Ok(self.read_optional::<Fact>(path).await?.map(|f| f.value))
}
async fn write(&self, path: &str, value: &str) -> Result<()> {
let fact = Fact {
value: value.to_owned(),
};
Ok(SecretStore::write(self, path, &fact).await?)
}
}
/// What one pass did.
#[derive(Debug, Default, PartialEq, Eq)]
pub(crate) struct Copied {
/// Copies written because they were absent or held another value.
pub written: usize,
/// Hives whose copies could not all be checked or written.
pub failed: usize,
}
/// The stored facts, as `(leaf, value)`. An empty value counts as not stored.
async fn sources(store: &impl FactStore) -> Result<Vec<(Leaf, String)>> {
let mut found = Vec::new();
for leaf in Leaf::ALL {
let path = service_path(leaf)?;
let stored = store
.read(&path)
.await
.with_context(|| format!("reading {path}"))?;
if let Some(value) = stored.filter(|v| !v.trim().is_empty()) {
found.push((leaf, value));
}
}
Ok(found)
}
/// Bring `hive`'s copies up to `sources`; how many were written.
async fn copy_hive(
store: &impl FactStore,
hive: &str,
sources: &[(Leaf, String)],
) -> Result<usize> {
let mut written = 0;
for (leaf, value) in sources {
let path = hive_path(hive, *leaf)?;
let current = store
.read(&path)
.await
.with_context(|| format!("reading {path}"))?;
if current.as_deref() == Some(value.as_str()) {
continue;
}
store
.write(&path, value)
.await
.with_context(|| format!("writing {path}"))?;
written += 1;
}
Ok(written)
}
/// One pass over `hives`.
///
/// # Errors
/// When a source cannot be read, which stops the pass before any copy: a
/// failed read is not an absent fact. A hive that fails is counted in
/// [`Copied::failed`] and logged, and the others are still copied.
pub(crate) async fn copy_pass(store: &impl FactStore, hives: &[String]) -> Result<Copied> {
let sources = sources(store).await?;
let mut done = Copied::default();
for hive in hives {
match copy_hive(store, hive, &sources).await {
Ok(n) => done.written += n,
Err(e) => {
tracing::warn!(hive, error = %format!("{e:#}"), "attic facts: copy failed");
done.failed += 1;
}
}
}
Ok(done)
}
/// Copy now and every [`RECONCILE_INTERVAL`] after. A pass that fails is
/// logged and retried on the next tick; it never stops the daemon.
pub fn spawn(hives: Vec<String>) {
let hives = Arc::new(hives);
tokio::spawn(async move {
let mut ticker = tokio::time::interval(RECONCILE_INTERVAL);
loop {
ticker.tick().await;
let pass = async {
let store = crate::store::connect()
.await
.context("logging in to the swarm secret store")?;
copy_pass(&store, &hives).await
};
match tokio::time::timeout(PASS_DEADLINE, pass).await {
Ok(Ok(done)) if done.written > 0 || done.failed > 0 => tracing::info!(
written = done.written,
failed = done.failed,
"attic facts: copied to the hives"
),
Ok(Ok(_)) => tracing::debug!("attic facts: every hive's copy is current"),
Ok(Err(e)) => tracing::warn!(
error = %format!("{e:#}"),
retry_in_s = RECONCILE_INTERVAL.as_secs(),
"attic facts: pass failed; retrying next tick"
),
Err(_) => tracing::warn!(
deadline_s = PASS_DEADLINE.as_secs(),
"attic facts: pass timed out; retrying next tick"
),
}
}
});
}
#[cfg(test)]
mod tests {
use std::collections::BTreeMap;
use std::sync::Mutex;
use super::*;
/// An in-memory store that counts writes and can refuse reads or writes
/// under a prefix.
#[derive(Default)]
struct Memory {
kv: Mutex<BTreeMap<String, String>>,
writes: Mutex<Vec<String>>,
refuse: Option<&'static str>,
}
impl Memory {
fn with(pairs: &[(&str, &str)]) -> Self {
let m = Self::default();
for (path, value) in pairs {
m.kv.lock()
.unwrap()
.insert((*path).to_owned(), (*value).to_owned());
}
m
}
fn get(&self, path: &str) -> Option<String> {
self.kv.lock().unwrap().get(path).cloned()
}
}
impl FactStore for Memory {
async fn read(&self, path: &str) -> Result<Option<String>> {
Ok(self.get(path))
}
async fn write(&self, path: &str, value: &str) -> Result<()> {
if self.refuse.is_some_and(|p| path.starts_with(p)) {
anyhow::bail!("denied");
}
self.writes.lock().unwrap().push(path.to_owned());
self.kv
.lock()
.unwrap()
.insert(path.to_owned(), value.to_owned());
Ok(())
}
}
const SOURCES: [(&str, &str); 4] = [
("swarm/services/attic/pull-token", "tok"),
("swarm/services/attic/url", "https://cache.example"),
("swarm/services/attic/cache", "swarm"),
("swarm/services/attic/public-key", "swarm:AAAA"),
];
fn hives() -> Vec<String> {
vec!["alpha".to_owned(), "beta".to_owned()]
}
#[tokio::test]
async fn present_facts_reach_every_hive() {
let store = Memory::with(&SOURCES);
let done = copy_pass(&store, &hives()).await.expect("pass");
assert_eq!(
done,
Copied {
written: 8,
failed: 0
}
);
for hive in hives() {
assert_eq!(
store
.get(&format!("swarm/hives/{hive}/attic/pull-token"))
.as_deref(),
Some("tok")
);
assert_eq!(
store
.get(&format!("swarm/hives/{hive}/attic/public-key"))
.as_deref(),
Some("swarm:AAAA")
);
}
// Presence control for the absence cases below: copies are the only
// thing a pass writes under a hive.
assert!(store.get("swarm/hives/alpha/forge/token").is_none());
}
#[tokio::test]
async fn absent_facts_write_nothing() {
let store = Memory::default();
let done = copy_pass(&store, &hives()).await.expect("pass");
assert_eq!(done, Copied::default());
assert!(store.writes.lock().unwrap().is_empty());
}
#[tokio::test]
async fn only_the_stored_facts_are_copied_and_an_empty_one_is_not_stored() {
let store = Memory::with(&[
("swarm/services/attic/pull-token", "tok"),
("swarm/services/attic/url", " \n"),
]);
copy_pass(&store, &hives()).await.expect("pass");
assert_eq!(
store.get("swarm/hives/alpha/attic/pull-token").as_deref(),
Some("tok")
);
assert!(store.get("swarm/hives/alpha/attic/url").is_none());
assert!(store.get("swarm/hives/alpha/attic/cache").is_none());
}
#[tokio::test]
async fn a_current_copy_is_not_rewritten_and_a_changed_value_is() {
let store = Memory::with(&SOURCES);
copy_pass(&store, &hives()).await.expect("first");
let again = copy_pass(&store, &hives()).await.expect("second");
assert_eq!(again, Copied::default(), "unchanged values are not written");
store.kv.lock().unwrap().insert(
"swarm/services/attic/pull-token".to_owned(),
"rotated".to_owned(),
);
let rotated = copy_pass(&store, &hives()).await.expect("third");
assert_eq!(
rotated,
Copied {
written: 2,
failed: 0
}
);
assert_eq!(
store.get("swarm/hives/beta/attic/pull-token").as_deref(),
Some("rotated")
);
}
#[tokio::test]
async fn a_removed_source_leaves_the_copies() {
let store = Memory::with(&SOURCES);
copy_pass(&store, &hives()).await.expect("first");
store
.kv
.lock()
.unwrap()
.remove("swarm/services/attic/pull-token");
copy_pass(&store, &hives()).await.expect("second");
assert_eq!(
store.get("swarm/hives/alpha/attic/pull-token").as_deref(),
Some("tok")
);
}
#[tokio::test]
async fn one_hive_failing_does_not_stop_the_others() {
let mut store = Memory::with(&SOURCES);
store.refuse = Some("swarm/hives/alpha/");
let done = copy_pass(&store, &hives()).await.expect("pass");
assert_eq!(
done,
Copied {
written: 4,
failed: 1
}
);
assert!(store.get("swarm/hives/beta/attic/cache").is_some());
assert!(store.get("swarm/hives/alpha/attic/cache").is_none());
}
}

1
swarm-controller/src/main.rs
View file

@ -37,6 +37,7 @@ mod agents_md;
mod agents_md_initial; mod agents_md_initial;
mod alerts; mod alerts;
mod archetypes; mod archetypes;
mod attic_copy;
mod auth; mod auth;
mod builder; mod builder;
mod claude_login; mod claude_login;

4
swarm-controller/src/startup.rs
View file

@ -367,7 +367,8 @@ fn spawn_subagent_streams(status: Option<&Arc<status::StatusReader>>, hives: &[H
/// The per-agent background passes: subagent and terminal streams, and /// The per-agent background passes: subagent and terminal streams, and
/// schedule keys, and the permissions bucket and its import, and the /// schedule keys, and the permissions bucket and its import, and the
/// archetypes agents are created from. /// archetypes agents are created from. Beside them, the
/// copy of the cache facts to each hive.
/// Lifted out of `main` for `clippy::too_many_lines`. /// Lifted out of `main` for `clippy::too_many_lines`.
pub(crate) fn spawn_per_agent_upkeep( pub(crate) fn spawn_per_agent_upkeep(
status: Option<&Arc<status::StatusReader>>, status: Option<&Arc<status::StatusReader>>,
@ -376,6 +377,7 @@ pub(crate) fn spawn_per_agent_upkeep(
spawn_subagent_streams(status, hives); spawn_subagent_streams(status, hives);
spawn_schedule_reconcile(status, hives); spawn_schedule_reconcile(status, hives);
spawn_agent_permissions(status, hives); spawn_agent_permissions(status, hives);
crate::attic_copy::spawn(hives.iter().map(|h| h.name.clone()).collect());
if let Some(status) = status { if let Some(status) = status {
archetypes::spawn(status.queue_client()); archetypes::spawn(status.queue_client());
} }

104
swarm-secret-client/src/attic.rs Normal file
View file

@ -0,0 +1,104 @@
//! The binary cache agreement: the four facts a hive needs to pull from the
//! swarm's cache, where the cache's operator keeps them, and where the
//! controller copies them for each hive.
//!
//! The cache's own unit (or, for an external cache, the operator) writes them
//! under `swarm/services/attic/`. No hive may read that prefix, so
//! `swarm-controller` copies each leaf to `swarm/hives/<hive>/attic/<leaf>`,
//! which the hive's own `swarm/hives/<hive>/*` stanza reads. The nix half of
//! the source paths is `swarm-bao.nix`'s `atticFactLeaves`.
use serde::{Deserialize, Serialize};
use crate::{
Error,
path::{Kind, principal_prefix},
};
/// One of the facts a hive pulls with.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Leaf {
/// The read-only token, as a basic-auth password.
PullToken,
/// The cache server's base URL, scheme included and no trailing slash.
Url,
/// The cache's name on that server.
Cache,
/// The cache's public signing key, `<name>:<base64>`.
PublicKey,
}
impl Leaf {
/// Every leaf, in the order the controller copies them.
pub const ALL: [Leaf; 4] = [Leaf::PullToken, Leaf::Url, Leaf::Cache, Leaf::PublicKey];
/// The last path segment, under both prefixes.
#[must_use]
pub fn name(self) -> &'static str {
match self {
Leaf::PullToken => "pull-token",
Leaf::Url => "url",
Leaf::Cache => "cache",
Leaf::PublicKey => "public-key",
}
}
}
/// Where the cache's operator keeps `leaf`.
///
/// # Errors
/// Never in practice: the name segments are constants. The `Result` is
/// [`principal_prefix`]'s.
pub fn service_path(leaf: Leaf) -> Result<String, Error> {
let prefix = principal_prefix(Kind::Service, "attic")?;
Ok(format!("{prefix}/{}", leaf.name()))
}
/// Where `hive` reads its copy of `leaf`.
///
/// # Errors
/// [`Error::PathSegment`] when `hive` contains anything but `[A-Za-z0-9_-]`,
/// which keeps one hive's name from addressing another hive's secret.
pub fn hive_path(hive: &str, leaf: Leaf) -> Result<String, Error> {
let prefix = principal_prefix(Kind::Hive, hive)?;
Ok(format!("{prefix}/attic/{}", leaf.name()))
}
/// What every leaf holds: one string under the field `bao kv get
/// -field=value` reads.
///
/// No `Debug`: for [`Leaf::PullToken`] `value` reads from the cache.
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct Fact {
/// The fact itself.
pub value: String,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_paths_are_the_ones_the_store_grants() {
// Spelled out: `swarm-bao.nix` grants each literal.
let sources = Leaf::ALL.map(|l| service_path(l).expect("constant segments"));
assert_eq!(
sources,
[
"swarm/services/attic/pull-token",
"swarm/services/attic/url",
"swarm/services/attic/cache",
"swarm/services/attic/public-key",
]
);
assert_eq!(
hive_path("alpha", Leaf::PullToken).expect("a plain name"),
"swarm/hives/alpha/attic/pull-token"
);
}
#[test]
fn a_hive_name_cannot_address_another_hives_copy() {
assert!(hive_path("alpha/../beta", Leaf::Url).is_err());
}
}

3
swarm-secret-client/src/lib.rs
View file

@ -6,7 +6,7 @@
//! deployment's environment into a logged-in client ([`client`]), and, per kind //! deployment's environment into a logged-in client ([`client`]), and, per kind
//! of secret, the path it lives at together with the fields it holds //! of secret, the path it lives at together with the fields it holds
//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`grafana`], //! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`grafana`],
//! [`acp`], [`ci`], [`review_classifier`], [`builder`], [`ci_metrics`], //! [`acp`], [`attic`], [`ci`], [`review_classifier`], [`builder`], [`ci_metrics`],
//! [`claude_login`]). //! [`claude_login`]).
//! Each of those is a thing the controller and a hive must say identically, so //! Each of those is a thing the controller and a hive must say identically, so
//! it is said once here. //! it is said once here.
@ -30,6 +30,7 @@
//! needs only the provider and the store links nothing else. //! needs only the provider and the store links nothing else.
pub mod acp; pub mod acp;
pub mod attic;
pub mod builder; pub mod builder;
pub mod cert; pub mod cert;
#[cfg(feature = "otel")] #[cfg(feature = "otel")]