diff --git a/docs/swarm/builder.md b/docs/swarm/builder.md index 3f6941dc..98ec002c 100644 --- a/docs/swarm/builder.md +++ b/docs/swarm/builder.md @@ -103,10 +103,9 @@ status has no link. hive's. A green check can therefore still fail on a hive, and the reverse. - The agent base is the builder's host's, not the hive's. Hives on another hyperhive or nixpkgs pin build something else. -- The check pushes nothing. The closure stays in the builder +- The builder pushes nothing. The closure stays in the builder host's store until it's collected, so on a host that also runs the hive, the - hive's own build can reuse whatever matches. A [meta build](#deploy-gate) - can push to [a binary cache](#binary-cache). + hive's own build can reuse whatever matches. - The PR check gates nothing. A merge deploys whatever the status says. The [deploy gate](#deploy-gate) is a separate build, from `internal/meta`. - A lost webhook delivery isn't retried: the head isn't built until the next @@ -248,115 +247,13 @@ On a built reply the controller writes the agent's key in the `agent-closures` KV bucket: the `internal/meta` commit, the closure's store path, the builder's platform and the time. Only the controller writes the bucket, and the builder holds no grant on it. A failed build writes nothing and leaves the previous key. -The record's `cached` field says whether the builder pushed the closure to -[the binary cache](#binary-cache). Nothing reads the bucket yet. The closure stays -unrooted in the builder host's store. +Nothing reads the bucket yet. The closure stays unrooted in the builder host's +store. The controller gives the `builder` forge account read on `internal/meta` when it updates the flake, and again before each build, since the repo is private. A grant the forge refuses fails the build with no verdict. -## Binary cache - -With `services.hyperhive.swarm.attic.domain` and `cache` set, the builder pushes -each green meta build's closure to that cache, an attic server inside or outside the -swarm, before it replies. It runs `attic push swarm: ` once per build, -not `attic watch`, and the builder pushes again when the controller runs a build again. The -push has ten minutes, or less when the reply is due sooner, because it must end -inside the 61 minutes the controller waits. `attic` checks the server's certificate -against the system roots and pins nothing, so a Let's Encrypt certificate -works. - -Set both options or neither: evaluation refuses a domain without a cache name. -The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`, -starting with a letter or digit, and evaluation refuses any other name. - -The push token is the swarm's own cache's to create, or the operator's to write for an -external one. The builder reads it on every push from `secret/swarm/services/attic/push-token` (field `value`), writes it to -a file and an `attic` config naming that file (`token-file`) in the unit's private -runtime directory (`/run/swarm-builder`, mode `0700`, both files `0600`), points -`attic` at the config with `XDG_CONFIG_HOME` and removes both after the push. -The token is in no argument or variable, so no process listing shows it. - -The reply's `cached` says whether the push finished. A push that fails, times out -or can't read the token leaves the build green with `cached` false and logs a -warning with the reason; the deploy goes ahead and the next build of the agent -tries again. With no cache set, the builder pushes nothing and `cached` is always -false. - -Mixed versions: - -| controller | builder | what happens | -| ---------- | ------- | ------------------------------------------------------------- | -| new | new | `cached` in the reply and in the `agent-closures` record | -| new | old | the reply has no `cached`, which reads as `false` | -| old | new | the controller ignores `cached`; its record has no such field | - -A record written before the field existed also reads as `false`. - -### The swarm's own cache - -`services.hyperhive.deploy.attic.enable` runs an attic server in a `swarm-attic` -container behind the gateway, at `attic.`. It defaults on where -you set `deploy.allSwarmServices` and the host holds the cache's store identity -(`deploy.bao.atticClient{Cert,Key}File`, which the store's host mints). You write -no secret: on first start, two host units log in to the store as the cache's own -principal and create what's missing. - -- `swarm-bao-attic-secret` creates the key atticd signs tokens with at - `secret/swarm/services/attic/jwt-secret` and writes it where the container - reads it. -- `swarm-bao-attic-tokens` creates the cache and mints the push token, which - the builder reads, at `secret/swarm/services/attic/push-token`, and a - read-only pull token for the hives at `secret/swarm/services/attic/pull-token`. - -A unit creates each secret only while the store holds none, so a second start -mints nothing. Both tokens last ten years and nothing renews them. To replace one, -write the new token to its path in the store: nothing mints over a value that -exists. A new signing key invalidates both tokens. - -With the switch on, `swarm.attic.domain` and `swarm.attic.cache` default to -`attic.` and `swarm`, so the builder on that host pushes to it. -Naming either option yourself wins over its default. The defaults apply only on -the host that runs the cache: on any other host, name both options too, with the -same values. - -Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store -path nobody added or fetched for 14 days goes at the next garbage collection, -which runs every 12 hours (`deploy.attic.retentionPeriod`). The next build of -that agent pushes a closure the cache lost. - -The gateway terminates TLS with a swarm services certificate and forwards to -`127.0.0.1:8910` (`deploy.attic.port`) with no login in front: attic checks its -own tokens, and a nix client can't follow a login redirect. The container logs -to its journal only; its journal doesn't reach the swarm log store. - -### External cache - -The cache can be an attic tenant on an instance outside the swarm. To point -the swarm at an existing tenant: - -1. On the builder host, set `services.hyperhive.swarm.attic.domain` to the - instance's host name (no scheme, `host:port` if it isn't on 443) and - `services.hyperhive.swarm.attic.cache` to the cache's name on it. -2. Seed the push token in the swarm secret store at - `secret/swarm/services/attic/push-token`, field `value`. Use the store's UI, - or `bao kv put` with `value=-` and the token on stdin, so the token is in no - command line. The token needs push rights on that cache. Until it - exists, every push fails and builds stay green with `cached` false. - -The builder checks the instance's certificate against the system roots and -pins nothing, so a Let's Encrypt certificate needs no setup. The builder -reads the token again on every push, so a new token takes effect on the next -build. - -Where you host the shared services, `deploy.attic.enable` defaults on and runs -the swarm's own cache. To use an external one, set it to `false` with -`lib.mkForce`, then set the two options by hand and seed the token as above. -The swarm then renders no container, vhost or unit of its own cache, and -creates no secret in the store. The store's host still mints the cache's -leaf and writes its role, which nothing logs in to. - ## Credentials | What | Where | Who reads it | @@ -364,7 +261,6 @@ leaf and writes its role, which nothing logs in to. | store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential | | queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it | | forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build | -| cache push token | `secret/swarm/services/attic/push-token` | the builder, on every push | The token belongs to the `builder` forge account, which swarm-controller creates, puts in the `hives` team of `agent-configs` (read on every config @@ -374,7 +270,7 @@ gets the token through a credential helper reading the child's environment: it's in no argument, URL or lock, so no line `nix` prints can carry it into a status. The same `git` fetches a config's own inputs, so the builder registers the helper for the forge's URL only, and the helper answers only an `https` -request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the +request for the forge's host: any other host gets nothing. The builder holds no write credential; swarm-controller writes the status with its own forge token. Details per path: [`credentials-reference.md`](credentials-reference.md). diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index ef3055eb..93349050 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -14,38 +14,35 @@ the rest of the cell says how. -| store path | minter | reader | automatic re-mint | automatic re-pull | -| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `swarm/agents//matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection | -| `swarm/agents//matrix/` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection | -| `swarm/agents//matrix-recovery/` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up | -| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl` in `hive-matrix`, once | `swarm-controller` | ❌ minted once | ✅ read every 5 minutes | -| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it | -| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 | -| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 | -| `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read at start | -| `swarm/agents//bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid | -| `swarm/agents//queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) | -| `swarm/agents//forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes | -| `swarm/agents//forge/