vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

run attic as a swarm service with deploy.attic.enable

Browse source
This commit is contained in:
atlas 2026-10-27 12:23:28 +01:00
commit 458a046b8d
4 changed files with 1242 additions and 66 deletions
Download patch file Download diff file Expand all files Collapse all files

55
docs/swarm/builder.md
View file

@ -259,8 +259,8 @@ grant the forge refuses fails the build with no verdict.
## Binary cache
With `services.hyperhive.swarm.attic.domain` and `cache` set, the builder pushes
each green meta build's closure to that cache, an attic server outside the swarm,
before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
each green meta build's closure to that cache, an attic server inside or outside the
swarm, before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
not `attic watch`, and the builder pushes again when the controller runs a build again. The
push has ten minutes, or less when the reply is due sooner, because it must end
inside the 61 minutes the controller waits. `attic` checks the server's certificate
@ -271,8 +271,8 @@ Set both options or neither: evaluation refuses a domain without a cache name.
The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`,
starting with a letter or digit, and evaluation refuses any other name.
The push token is the operator's to write. The builder reads it on every push
from `secret/swarm/services/attic/push-token` (field `value`), writes it to
The push token is the swarm's own cache's to create, or the operator's to write for an
external one. The builder reads it on every push from `secret/swarm/services/attic/push-token` (field `value`), writes it to
a file and an `attic` config naming that file (`token-file`) in the unit's private
runtime directory (`/run/swarm-builder`, mode `0700`, both files `0600`), points
`attic` at the config with `XDG_CONFIG_HOME` and removes both after the push.
@ -294,6 +294,43 @@ Mixed versions:
A record written before the field existed also reads as `false`.
### The swarm's own cache
`services.hyperhive.deploy.attic.enable` runs an attic server in a `swarm-attic`
container behind the gateway, at `attic.<swarm domain>`. It defaults on where
you set `deploy.allSwarmServices` and the host holds the cache's store identity
(`deploy.bao.atticClient{Cert,Key}File`, which the store's host mints). You write
no secret: on first start, two host units log in to the store as the cache's own
principal and create what's missing.
- `swarm-bao-attic-secret` creates the key atticd signs tokens with at
`secret/swarm/services/attic/jwt-secret` and writes it where the container
reads it.
- `swarm-bao-attic-tokens` creates the cache and mints the push token, which
the builder reads, at `secret/swarm/services/attic/push-token`, and a
read-only pull token for the hives at `secret/swarm/services/attic/pull-token`.
A unit creates each secret only while the store holds none, so a second start
mints nothing. Both tokens last ten years and nothing renews them. To replace one,
write the new token to its path in the store: nothing mints over a value that
exists. A new signing key invalidates both tokens.
With the switch on, `swarm.attic.domain` and `swarm.attic.cache` default to
`attic.<swarm domain>` and `swarm`, so the builder on that host pushes to it.
Naming either option yourself wins over its default. The defaults apply only on
the host that runs the cache: on any other host, name both options too, with the
same values.
Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store
path nobody added or fetched for 14 days goes at the next garbage collection,
which runs every 12 hours (`deploy.attic.retentionPeriod`). The next build of
that agent pushes a closure the cache lost.
The gateway terminates TLS with a swarm services certificate and forwards to
`127.0.0.1:8910` (`deploy.attic.port`) with no login in front: attic checks its
own tokens, and a nix client can't follow a login redirect. The container logs
to its journal only; its journal doesn't reach the swarm log store.
### External cache
The cache can be an attic tenant on an instance outside the swarm. To point
@ -313,10 +350,12 @@ pins nothing, so a Let's Encrypt certificate needs no setup. The builder
reads the token again on every push, so a new token takes effect on the next
build.
A later change adds the swarm's own attic service: `deploy.attic.enable` will run atticd
in the swarm and fill these two options itself. An operator who sets that option
to false keeps the external cache by setting the two options by hand and
seeding the token as above.
Where you host the shared services, `deploy.attic.enable` defaults on and runs
the swarm's own cache. To use an external one, set it to `false` with
`lib.mkForce`, then set the two options by hand and seed the token as above.
The swarm then renders no container, vhost or unit of its own cache, and
creates no secret in the store. The store's host still mints the cache's
leaf and writes its role, which nothing logs in to.
## Credentials

67
docs/swarm/credentials-reference.md
View file

@ -14,36 +14,38 @@ the rest of the cell says how.
<!-- vale write-good.Passive = NO -->
| store path | minter | reader | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl` in `hive-matrix`, once | `swarm-controller` | ❌ minted once | ✅ read every 5 minutes |
| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read at start |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when you link an external forge account in the swarm UI | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when you link a GitHub account in the swarm UI | the agent | ❌ your token; delete it and link again | ✅ every 2 minutes |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
| `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute; ❌ Grafana restarts when its copy changes |
| `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes |
| `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
| `swarm/services/attic/push-token` | you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with no `swarm.attic` cache set, nothing reads it |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
| store path | minter | reader | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl` in `hive-matrix`, once | `swarm-controller` | ❌ minted once | ✅ read every 5 minutes |
| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read at start |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when you link an external forge account in the swarm UI | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when you link a GitHub account in the swarm UI | the agent | ❌ your token; delete it and link again | ✅ every 2 minutes |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
| `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute; ❌ Grafana restarts when its copy changes |
| `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes |
| `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with no `swarm.attic` cache set, nothing reads it |
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years | nothing yet | ❌ write a new token | must be stated |
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
<!-- vale write-good.Passive = YES -->
@ -145,9 +147,9 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
### Store identities, one per principal
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. Each of these ten leaves has
identity and sharing one merges the identities. Each of these leaves has
a role and a policy of its own, and reaches one path, except the review
classifier's, the CI runner's, the builder's, the CI run poller's, the claude
classifier's, the CI runner's, the builder's, the cache's, the CI run poller's, the claude
login minter's, the forge host's and Grafana's, which reach more than one:
| unit | option pair under `deploy.bao.` | reads, or writes |
@ -160,6 +162,7 @@ login minter's, the forge host's and Grafana's, which reach more than one:
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
| `swarm-bao-attic-secret` and `swarm-bao-attic-tokens` | `atticClientCertFile` / `atticClientKeyFile` | creates and reads `swarm/services/attic/jwt-secret`, `swarm/services/attic/push-token` and `swarm/services/attic/pull-token` |
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |

12
docs/swarm/credentials.md
View file

@ -416,19 +416,19 @@ each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
`deploy.bao.ciMetricsCommonName`, `deploy.bao.claudeLoginCommonName` and
`deploy.bao.forgePublisherCommonName`.
The last two are the writers, and the last is the one leaf two units share:
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`,
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
The last three are the writers, and the last is the one leaf two units share:
the forge's host holds one store identity for the token it publishes and the
OIDC client secret it reads.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all nine and defaults
all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from
On a host that mints its own PKI, `glue-bao-tls.nix` signs all ten and defaults
all twenty options, and there is nothing to do. Elsewhere you copy each leaf from
the store's host and name it here: a service principal's from the top of
`/var/lib/swarm-bao-pki/`, another hive's leaves from
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
directory). That's one file per principal rather than one file
shared by nine, which is the whole of what this buys. `swarm-ci-runner.nix`
shared by ten, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without

19
docs/swarm/services.md
View file

@ -433,6 +433,25 @@ the private config repo with the read-only token of its own `builder` forge
account. On a host without the store, that leaf is the one file to copy, as
for the CI runner. What it builds and why are in [`builder.md`](builder.md).
### Binary cache
The swarm's attic server, in a `swarm-attic` container on the host that sets
`deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has
the cache's store identity, as for the builder: the host that runs the store has
it. Set `deploy.attic.enable = lib.mkForce false` to use an external cache
instead and name `swarm.attic.{domain,cache}` yourself.
The container shares the host's network namespace and atticd listens on loopback
only. The gateway serves it at `swarm.attic.domain` (default
`attic.<swarm domain>`) over TLS, with no login in front, because attic checks
its own tokens. Storage is local disk in the container's state.
It needs no manual secret. Two host units, under the cache's own `attic` leaf,
create its signing key, the cache and the push and pull tokens in the secret
store, each only while absent. On a host without the store, that leaf is the one
file to copy. The builder pushes green builds to it. How it works, and the
external alternative, are in [`builder.md`](builder.md#binary-cache).
### CI metrics
The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host