d5de1e4943
A new swarm service, `swarm-claude-login`, in its own container on the host
that sets `deploy.claude-login.enable` (defaulted on by `allSwarmServices`
where the host has the minter's store leaf, like the review classifier).
- swarm-ui: an agent card showing "login needed" gets a "log in" button.
Its dialog starts the login, shows the URL claude printed, and takes the
pasted code.
- swarm-controller: `POST/DELETE /api/agents/{name}/claude-login` and
`POST /api/agents/{name}/claude-login/code`, behind the swarm UI's
authelia-gated `/api/`, relay the steps to the minter as requests on
`$SWARM.claude-login.request`.
- swarm-claude-login: runs `claude auth login` over piped stdio with HOME
and CLAUDE_CONFIG_DIR in a per-login scratch dir, scrapes the first
https URL, writes the code to stdin, and stores the `.credentials.json`
at `swarm/agents/<agent>/claude-credentials` with a KV v2 write, a new
version over any earlier one. The scratch dir goes on every exit path; a
login waits 15 minutes for its code.
- hive-agent: while parked in `wait_for_login`, reads that leaf every 30 s
under the agent's own cert and installs it only when its KV version is
newer than the one recorded in `<harness>/claude-credentials-version`.
An equal or older version is never put back, because claude rotates the
refresh token and the stored copy is stale after the first refresh.
- swarm-secret-client: `claude_login` paths and `SecretStore::read_versioned`.
- Credentials: a `claude-login` store leaf minted by glue-bao-tls, a
`swarm-claude-login` policy (create/update on
`swarm/agents/+/claude-credentials`, read on its own queue client
secret), an authelia machine client, and a `--claude-login-client`
grant in the queue's auth-callout responder. The unit runs as a static
system user.
The hive's own web-UI login is unchanged.
Refs #5268
583 lines
74 KiB
Markdown
583 lines
74 KiB
Markdown
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |
|
|
|
|
# Swarm secrets: what exists, and where each one lives
|
|
|
|
A swarm generates its credentials in three different places and reads them
|
|
in a fourth, so "where does this file go" has a different answer per deployment.
|
|
This page is that answer, one row per secret.
|
|
|
|
Two rules run through all of it.
|
|
|
|
**Private key material and access tokens are paths, never values.** Every option
|
|
carrying one takes a file path (`*File`), because a literal written into a nix
|
|
expression ends up in the nix store — world-readable and permanent. No
|
|
option in this tree accepts one inline, and adding one would be a leak rather
|
|
than a convenience.
|
|
|
|
The rule is about what must stay secret, not about credentials generally.
|
|
**Public material is a value**: a certificate, or a public nkey like
|
|
the queue callout's user key, reaches every client that connects,
|
|
so the store is a perfectly good place for it.
|
|
|
|
**The generator and the reader typically live in different containers.** They share
|
|
the host's network namespace, which makes them feel co-located, but their
|
|
filesystem roots are separate. That's why delivery is a **host-side copy rather
|
|
than a bind mount**: `nixos-container` refuses to start when a bind source is
|
|
missing, and a secret minted on another container's first boot doesn't exist
|
|
yet. Binding it would make one container wait on a file that waits on a
|
|
container that starts after it.
|
|
|
|
## Topologies, by who places secrets
|
|
|
|
Read every row below against one of these. This is a different cut
|
|
from the [deployment shapes](services.md#deployment-shapes) --- those
|
|
say _where services run_, these say _who is responsible for a secret
|
|
file being there_ --- the two lists don't line up one-to-one, and
|
|
neither is a renaming of the other.
|
|
|
|
| topology | what it means | who places secrets |
|
|
| ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------- |
|
|
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — the reader itself generates each secret, or a host unit copies it |
|
|
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
|
|
| **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config |
|
|
|
|
## Swarm-level — one of each per swarm
|
|
|
|
| secret | generated by | lives at | hive elsewhere |
|
|
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
|
|
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
|
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
|
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
|
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) |
|
|
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
|
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
|
|
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
|
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/var/lib/grafana-admin/password` inside the `swarm-grafana` container: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. Grafana doesn't start until the file exists, so it stays down rather than falling back to the upstream default `admin` |
|
|
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/var/lib/grafana-metrics/bearer` inside the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. `swarm-metrics-auth` on the metrics store's host reads the stored value itself, under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
|
|
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
|
|
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
|
|
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
|
|
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
|
|
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
|
|
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
|
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
|
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
|
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
|
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
|
| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which reloads nginx if it's active); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place |
|
|
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
|
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
|
|
|
Authelia mints the three keys for itself, in-container, precisely
|
|
because nothing outside that container ever reads them. **That's the test worth
|
|
applying to any secret added here** — and the client secret's plaintext half is
|
|
the one row that fails it, which is the entire reason a delivery step exists.
|
|
|
|
Three telemetry collectors exist, and each one fails that test: each reads its
|
|
own secret out of the store, under its own identity.
|
|
|
|
The **hive's** collector authenticates to the swarm's collector as its own hive,
|
|
so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches
|
|
`swarm/hives/<hive>/oidc/client` under the hive's own leaf
|
|
(`deploy.bao.clientCertFile`), on every topology. A hive with no leaf has no
|
|
default, and the build stops at the collector's identity assertion unless
|
|
`services.hyperhive.otel.clientSecretFile` names a file delivered some other way.
|
|
|
|
The **swarm's** collector reads its own under a leaf of its own, the fourth
|
|
store reader — the same shape as Grafana's, described in full below. In short:
|
|
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
|
|
out of the store on every host that runs the collector and holds a store
|
|
identity, whether authelia is a network away or in the container next door.
|
|
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
|
|
the client it reads for unconditionally: authelia refuses a
|
|
bearer-authz client with no audience, and every host publishes at least
|
|
forgejo's `/metrics` as a scrape target, so the audience is never empty.
|
|
|
|
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
|
|
where an absent OIDC block is a container with no way in at all, a collector
|
|
with nothing to authenticate with still receives every hive's telemetry —
|
|
its pushes to the stores go out unauthenticated and get refused there, and its
|
|
scrape jobs for the published targets read a credential file nothing
|
|
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
|
|
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice
|
|
`glue-queue-agent-credential.nix` makes for its own reader, rather than the hard assertion `swarm-grafana.nix`
|
|
uses for its non-optional one. The operator can still deliver the secret by
|
|
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
|
|
collector this unit never reaches.
|
|
|
|
The **secret store's own** collector — the forwarder inside the `swarm-bao`
|
|
container — needs a delivery step too, and it takes the same route with one
|
|
principal of its own: `swarm-bao-forwarder-oidc.service` reads
|
|
`swarm/services/<client-id>/oidc/client` under a leaf of its own and
|
|
lands it in the container's tree, where `LoadCredential` hands it to the
|
|
collector. The client id is its own
|
|
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
|
|
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
|
|
collector admits it on a receiver of its own — one identity per principal, so
|
|
this forwarder borrows neither the swarm collector's client nor
|
|
`swarm-controller`'s.
|
|
|
|
⚠️ **This one has no degraded mode, and the difference is structural.** The
|
|
route it exports over accepts an authenticated request and nothing else, so a
|
|
forwarder without the secret has nothing to fall back to: the unit fails and
|
|
retries, and the collector refuses to start until the file arrives rather than
|
|
exporting into a 401 while reporting healthy. Its ordering differs from the
|
|
unit above it for the same structural reason — it reads the store inside the
|
|
container it delivers into, so it runs **after** that container rather than
|
|
before it, and only a first boot ever waits.
|
|
|
|
### The queue's callout nkeys
|
|
|
|
The responder holds two nkeys besides the callout xkey below. It signs every
|
|
user JWT it issues with the **issuer** key, an account key, so possession of
|
|
the issuer seed is the authority to admit anyone to the queue. It connects to
|
|
the queue as the **user** key, the one identity the server exempts from
|
|
callout. You configure nothing about either: both live in the swarm secret
|
|
store beside the xkey, and come into being on the queue's first boot.
|
|
|
|
- **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`,
|
|
**public half** at `secret/swarm/public/swarm-nats/callout-issuer`.
|
|
- **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`,
|
|
**public half** at `secret/swarm/public/swarm-nats/callout-user`.
|
|
|
|
The unit that creates the xkey creates these too, under the same rules: only
|
|
the `swarm-nats-auth` role reads or creates them, nothing replaces a stored
|
|
seed, and the policy of `swarm-secret-publisher` denies it both seed paths.
|
|
The server renders both public halves into its `auth_callout` block as it
|
|
starts, and doesn't start without them.
|
|
|
|
A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or
|
|
`/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what
|
|
`swarm-nats-auth ensure-callout-keys` stores for that key when the store holds
|
|
none, so the key keeps its public half. Once a read of the store returns a
|
|
seed, the verb removes the file. When the store already holds a different
|
|
seed, the verb keeps the store's seed, logs a warning naming the file, then
|
|
removes it.
|
|
|
|
A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`,
|
|
`calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`,
|
|
under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the
|
|
option and where the keys live now.
|
|
|
|
To replace either key, delete both of its paths the way the xkey's section
|
|
below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`.
|
|
|
|
One consequence worth knowing before you debug it: the queue assembles its
|
|
config each time the server starts rather than at build time, in every mode,
|
|
because the callout keys exist only at runtime. A malformed config
|
|
surfaces when the server starts instead of when the system builds. The server
|
|
names the offending file and refuses to run.
|
|
|
|
### The queue's callout xkey
|
|
|
|
The server seals every auth-callout request to a curve key, the callout
|
|
xkey, and the responder seals its reply back, so nothing else in the callout
|
|
account or on the wire can read the request or the user JWT minted for it.
|
|
You configure nothing about the key: it lives in the swarm secret store and
|
|
comes into being on the queue's first boot.
|
|
|
|
- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`.
|
|
- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`.
|
|
|
|
Only the `swarm-nats-auth` role reads either path, and it may create each but
|
|
never replace it. The policy of `swarm-secret-publisher`, which writes the rest
|
|
of `swarm/services/`, denies it the seed's path.
|
|
|
|
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
|
|
`swarm-nats` container, under the responder's store leaf:
|
|
|
|
1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no
|
|
seed, and otherwise only reads it. It needs no queue, so on a first boot
|
|
the key exists before the server that needs it.
|
|
2. It reads the public half back from the store into
|
|
`/var/lib/nats-callout/callout-xkey.pub` inside the container.
|
|
3. Each time `nats-server` starts, it renders that key into
|
|
`auth_callout.xkey`. With no file, the server doesn't start.
|
|
4. The responder starts after the server and reads the seed from the store.
|
|
A responder that can't read it doesn't start, rather than answering
|
|
requests it can't open.
|
|
|
|
The store has to be reachable each time the responder starts. When it's down,
|
|
the fetch unit fails and retries for a day. The server starts with the key
|
|
fetched last time, or not at all on a first boot. A responder that starts
|
|
while the store is down exits and retries every 5 seconds, and until one
|
|
start reads the seed the queue refuses every client, OIDC ones included. The
|
|
retry that lands the file starts or restarts the server, and the responder
|
|
comes up with it.
|
|
|
|
The responder's store leaf is `deploy.nats.authBaoClientCertFile` /
|
|
`authBaoClientKeyFile`. On the store's host both default to the leaf
|
|
`glue-bao-tls.nix` mints there. A queue host without the store gets no
|
|
default, even when it holds the hive's own reader `deploy.bao.clientCertFile`:
|
|
issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and
|
|
name it in both options. Until you do, evaluation fails naming both.
|
|
|
|
Where the queue and the store run on different hosts, deploy the store's host
|
|
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the
|
|
keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of
|
|
the seeds' paths; where the queue runs on that host too, the fetch waits for
|
|
both. A queue host deployed before it can't read or create the keys, so its
|
|
fetch fails, and the queue refuses every client until you deploy the store's
|
|
host: the server starts only on public keys fetched before, and the responder
|
|
can't load its seeds. Seed files the import is waiting for stay where they
|
|
are. If that takes longer than the fetch's day of retries, run
|
|
`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
|
|
|
|
Two starts that create the key at once can't both win: each write is a
|
|
create that fails once the path holds anything, and the start that lost reads
|
|
the winner's key back.
|
|
|
|
To replace the key, remove both paths with their metadata and every version,
|
|
then restart the fetch unit on the queue's host. Its restart of `nats.service`
|
|
restarts the responder too, which then loads the new seed:
|
|
|
|
```sh
|
|
bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
|
|
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
|
|
systemctl restart swarm-bao-nats-callout-xkey
|
|
```
|
|
|
|
`bao kv delete` isn't enough. It deletes the latest version but keeps the
|
|
path's metadata, so the store still counts the path as existing and refuses
|
|
the create that makes a new key. No principal hyperhive configures may
|
|
delete either path's metadata, so run the two deletes with a token that may,
|
|
such as the root token.
|
|
|
|
## Hive-level — one of each per hive
|
|
|
|
| secret | generated by | lives at |
|
|
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
|
|
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
|
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
|
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
|
|
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
|
|
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
|
|
|
|
The forge and the homeserver read their copies out of the store, the way
|
|
Grafana does below: the forge under the forge publisher's leaf, the
|
|
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
|
|
publisher on authelia's host has written the secret, rather than skipping. A
|
|
silent skip produces a service whose login button always fails, which is a
|
|
symptom many layers from its cause.
|
|
|
|
The store's **first reader** is the agent containers' queue credential:
|
|
`glue-queue-agent-credential.nix` lands it as two files, the client secret and
|
|
the client id it authenticates, because that's how a queue client reads them —
|
|
the secret by path, the id by value. No local fallback exists, and none is
|
|
possible, so absent files mean an agent built before `swarm-logs` read its own
|
|
queue secret has no working `swarm-logs`, which is what a swarm looks like
|
|
before the publisher on the authelia host has run.
|
|
The reader runs before `hive-c0re.service`, which wants it (doesn't require
|
|
it), so an agent container never renders ahead of the credential; an
|
|
unreachable store delays the daemon's start rather than failing it.
|
|
|
|
That credential still has one hop left, because the reader of it lives inside
|
|
an agent container. **It crosses as a systemd credential, not as a bind
|
|
mount**, and the mode decides that: the secret is `root:0600` and a harness
|
|
runs as its own unprivileged agent user, so a bind would deliver a file that
|
|
user can't open. `hive-c0re` stats the two files — the directory is `0755`, so
|
|
it needs no read access to either — and hands them to
|
|
`systemd-nspawn --load-credential` through `hive-priv`, which runs as root.
|
|
Inside, `hive-agent.service` names the same two ids in `LoadCredential=`, which
|
|
lands them in its own `$CREDENTIALS_DIRECTORY`, owned by the agent user and by
|
|
nobody else, where a `swarm-logs` built before it read the agent's own queue
|
|
secret reads them. The harness itself doesn't: it connects to the queue with the agent's own credential. `hive-c0re`
|
|
never reads the bytes at any point: it runs as `hive-core`, which is the reason
|
|
a copy wasn't an option either. When the files aren't there the daemon forwards
|
|
nothing and says so in its journal, and the container starts without them.
|
|
|
|
The **second reader** is Grafana's OIDC client secret, and it's the first one
|
|
that's a _swarm service's_ own credential rather than a hive's. It has exactly
|
|
one delivery route: `swarm-bao-grafana-oidc.service` reads
|
|
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
|
|
that runs Grafana. Where authelia is doesn't enter into it — the publisher on
|
|
authelia's host writes that path whether the reader is a network away or in the
|
|
container next door.
|
|
|
|
That's a deliberate refusal of the shortcut. Copying the plaintext
|
|
straight out of authelia's tree when the two happen to share a host is one
|
|
round trip cheaper and costs a second delivery unit, a second way for the file
|
|
to be wrong, and a gate deciding between them — and the gate is the expensive
|
|
part, because whatever it's wrong about is an outage nobody can read. The store
|
|
exists so the only secrets a host holds out of band are **certificates**, and
|
|
a host reads everything else with them.
|
|
|
|
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
|
|
running Grafana requires `swarm.authelia.url` (this module disables its local
|
|
login form unconditionally, so SSO isn't a feature of some topologies — it's the
|
|
only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
|
|
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
|
|
a null URL dropped the OIDC block, a missing leaf produced a warning, and
|
|
either one left a Grafana with no SSO and no password box — nothing failed, so
|
|
nothing said anything. Registration is separate and stays where authelia is
|
|
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
|
|
so it's declared on the host running authelia, which isn't necessarily the host
|
|
running Grafana. The swarm controller, the queue, the homeserver and the forge
|
|
register the same way (`glue-controller-oidc-client.nix`,
|
|
`glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
|
|
`glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
|
|
forge read their secrets from the store too: the responder itself under its own
|
|
leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
|
|
`swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
|
|
the option pair, when that leaf is unset — neither the responder nor the
|
|
homeserver starts without its secret, and the forge registers no login source
|
|
without its.
|
|
|
|
Grafana's admin password rides the same leaf and the same store. Where Grafana
|
|
and the store run on different hosts, deploy the store's host first:
|
|
`swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create
|
|
the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
|
|
the Grafana host, which retries, while Grafana stays down because its config
|
|
reads that file.
|
|
|
|
The **third reader** is the swarm collector's own OIDC client secret, the
|
|
same shape one level down: `swarm-bao-otel-oidc.service` reads
|
|
`swarm/services/<client-id>/oidc/client` out of the store on every host that
|
|
runs the collector, and registration lives separately in
|
|
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
|
|
differs is what an absent credential means: the collector still receives
|
|
telemetry with none, so `swarm-otel.nix` doesn't assert
|
|
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
|
|
does — it
|
|
doesn't render the reading unit without them, the shape
|
|
`glue-queue-agent-credential.nix` uses for its own read.
|
|
|
|
The **fourth reader** is the secret store forwarder's OIDC client secret:
|
|
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
|
|
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
|
|
third in both directions. It renders unconditionally, because the export it
|
|
authenticates has no unauthenticated mode to degrade into — and it orders
|
|
itself **after** `container@swarm-bao`, because the store it reads runs in the
|
|
container it delivers into. Nothing circular sits behind that: the identity it
|
|
logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not
|
|
anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix`
|
|
refuses the build and names `deploy.bao.forwarderOidcClientCertFile` /
|
|
`forwarderOidcClientKeyFile`.
|
|
|
|
A service's secret is one value for the whole swarm rather than one per hive, so
|
|
it lives under the `services` prefix, and a hive's read policy grants that prefix
|
|
whole. That's because nothing in a swarm's configuration records which hive runs
|
|
a given service — placement is a `deploy.*` fact, per-host by definition — so
|
|
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
|
|
every swarm service's client secret, and that's stated in
|
|
`swarm-secret-client`'s `policy` module beside the grant itself.
|
|
|
|
⚠️ **The readers no longer present it.** Five units used to log in with
|
|
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
|
|
principal by the subject of the certificate it presents — so five readers behind
|
|
one leaf were one principal holding the union of their needs. Each now holds a
|
|
leaf of its own, and a policy naming only the path that unit reads. See
|
|
[per-principal identities](#per-principal-identities) below.
|
|
|
|
Absence behaves
|
|
as it does for the other readers: a store that says "not here" leaves the
|
|
file alone and says so, since there is no such thing as a locally valid OIDC
|
|
client secret to stand in.
|
|
|
|
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
|
|
credential you must already hold to authenticate from the thing it
|
|
authenticates you to, so the store's identity can't come from an authority the store
|
|
distributes — which excludes the store's `pki` root, and has nothing to do
|
|
with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
|
|
mints a CA that signs a fixed, short list of leaves — the store's server
|
|
certificate, the hive's own reader certificate, and one per service principal —
|
|
and distributes nothing. A deployment with a real
|
|
internal CA deletes that file and names its own paths in
|
|
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
|
|
A hive that reads from a store on **another** machine names the reader's half —
|
|
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
|
|
plus one leaf per principal it runs (the table below names the options). Those are the
|
|
credentials that can't come out of the store, being what opens it; everything
|
|
else a hive needs does.
|
|
|
|
Agents are the one principal whose leaf the store issues. Its `pki-agents`
|
|
mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the
|
|
store, and `swarm-controller`, already logged in under its own host leaf, asks
|
|
that mount's one role for a `hive-agent-<agent>` client certificate at agent
|
|
creation. Host roles never pin that CA and agent roles pin only it, so an
|
|
agent's certificate opens that agent's own `swarm/agents/<agent>/*` and
|
|
nothing else: `read` on the values there and `list` on their names.
|
|
`swarm-controller` writes that policy at agent creation and rewrites every
|
|
agent's at its own start, so a change to it reaches existing agents with the
|
|
next controller restart.
|
|
|
|
### Per-principal identities
|
|
|
|
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
|
identity and sharing one merges the identities. Each of these nine leaves has
|
|
a role and a policy of its own, and reaches one path, except the review
|
|
classifier's, the builder's, the claude login minter's, the forge host's and
|
|
Grafana's, which reach two each:
|
|
|
|
| unit | option pair under `deploy.bao.` | reads, or writes |
|
|
| --------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
|
|
| `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` |
|
|
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
|
|
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
|
|
| `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` |
|
|
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
|
|
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
|
|
| `swarm-bao-forge-publisher`, `swarm-bao-forge-oidc` | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` |
|
|
|
|
The first exists **per hive**, because the path it reads carries a hive name
|
|
and every hive runs its own reader. Its subject is
|
|
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
|
|
composed spelling as a hive name, so nobody can name a hive into another hive's
|
|
role. The other eight reach a path that names a swarm service rather than a hive,
|
|
so one role each is enough and their subjects are the flat
|
|
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
|
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
|
|
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
|
|
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
|
|
The last two are the writers, and the last is the one leaf two units share:
|
|
the forge's host holds one store identity for the token it publishes and the
|
|
OIDC client secret it reads.
|
|
|
|
On a host that mints its own PKI, `glue-bao-tls.nix` signs all eight and defaults
|
|
all sixteen options, and there is nothing to do. Elsewhere you copy each leaf from
|
|
the store's host and name it here: a service principal's from the top of
|
|
`/var/lib/swarm-bao-pki/`, another hive's queue-agent pair from
|
|
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
|
|
directory). That's one file per principal rather than one file
|
|
shared by eight, which is the whole of what this buys. `swarm-ci-runner.nix`
|
|
refuses the build wherever the runner runs without its pair, as
|
|
`swarm-review-classifier.nix` does for the classifier and `swarm-grafana.nix`
|
|
for Grafana. The forge's pair refuses nothing: without
|
|
it `swarm-bao-forge-publisher` and `swarm-bao-forge-oidc` render, fail at start
|
|
naming both options, and the controller runs without forge access and the forge
|
|
without a login source until an operator places the leaf.
|
|
|
|
On a host without the store, where the forwarder never runs, forgetting one of
|
|
the other eight isn't a quiet degrade. A hand-configured remote-store hive that
|
|
names the hive's own `clientCertFile` and misses a principal's pair has the
|
|
build refused, by the module of the reader it missed, naming that pair — see
|
|
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
|
|
|
|
### What a missing leaf costs
|
|
|
|
Each of the three modules asserts its own pair, on the same condition: **this host
|
|
already reads the store** — it holds `deploy.bao.clientCertFile` and
|
|
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
|
|
The refusal names the two options, so an operator acts on the message without
|
|
opening the nix.
|
|
|
|
That condition fires on exactly one shape, the hand-configured remote-store hive
|
|
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
|
|
that matters on its own:
|
|
|
|
- **A host with no store identity at all** holds no `clientCertFile` either, so
|
|
the first clause is false. That's the supported no-store deployment, and also
|
|
the one an operator passes through while bringing a hive up — neither should
|
|
fail to evaluate.
|
|
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
|
|
defaults, so the third clause is false. Nothing to set, nothing to refuse.
|
|
- **A consumer that's off** — no collector, no Grafana — has no
|
|
unit to skip, so the second clause is false. Only the queue-credential reader
|
|
has no toggle of its own to check: every hive runs one for its own agents, so
|
|
reading the store at all is what asks for it.
|
|
|
|
The collector keeps a separate degrade underneath this, unchanged: a host with no
|
|
store identity runs a collector with no OIDC client secret and still receives
|
|
telemetry. The assertion doesn't touch that shape, because that host has no
|
|
`clientCertFile` either.
|
|
|
|
## How a reader reaches the store
|
|
|
|
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
|
|
the host that runs the store that name resolves twice. `/etc/hosts` answers a
|
|
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
|
|
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
|
|
and the network namespace that asked decides which half of it answers.
|
|
|
|
The container's half goes through nginx and **still doesn't terminate TLS**. It
|
|
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
|
|
ClientHello and splices the rest of the connection through byte for byte, so
|
|
openbao completes the handshake itself. A vhost would decrypt here instead, and
|
|
the client certificate — the store's whole authentication — would stop at nginx,
|
|
leaving openbao seeing one client for every hive in the swarm. That's why the
|
|
store has no vhost and gets a passthrough: not an exception to the no-vhost
|
|
rule, the one shape that keeps it.
|
|
|
|
The stream listener binds the bridge IP rather than every address, because
|
|
openbao already holds loopback on that port in the same network namespace and a
|
|
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
|
|
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
|
|
on the bridge firewall and nowhere else.
|
|
|
|
Reaching the port grants nothing by itself: openbao answers nothing without a
|
|
client certificate signed by `deploy.bao.clientCaFile` or by the store's own
|
|
agent CA. The listener reads both from `listener-client-ca.pem`, which no
|
|
cert-auth role pins. The passthrough carries
|
|
whichever certificate the reader presents, unchanged.
|
|
|
|
## The browser UI
|
|
|
|
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
|
|
(`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in
|
|
with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost
|
|
checks the session with authelia and proxies to an nginx inside the store's
|
|
container on
|
|
`127.0.0.1:<deploy.bao.uiProxyPort>`. That nginx forwards `/ui/` and `/v1/` to
|
|
a second openbao listener on `127.0.0.1:<deploy.bao.uiPort>`, answers 403 on
|
|
the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on
|
|
everything else. ⚠️ That listener asks for **no client certificate**, because
|
|
a browser has none to present, so on this one door a bao token is the whole
|
|
credential. Anything on the store's host that can dial loopback, and any
|
|
`admins` session through the vhost, needs only a token to use the API. Unseal
|
|
from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed
|
|
gateway where the UI is the only swarm name its host fronts, that host's nginx
|
|
doesn't wait for the store: it serves the hive leaf, or on a host that runs no
|
|
hive a throwaway self-signed certificate, on the UI's name (a browser warning)
|
|
until the unsealed store issues the services leaf, so the
|
|
stream passthrough readers use on that host comes up with the store sealed.
|
|
|
|
### OIDC login
|
|
|
|
The OIDC tab logs in through authelia as the client `swarm-bao-ui`
|
|
(`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under
|
|
`swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`, and
|
|
`create` and `update` on
|
|
`secret/data/swarm/services/review-classifier/anthropic-api-key` and
|
|
`secret/data/swarm/services/review-classifier/acp-provider`. That shows the key
|
|
tree and each key's versions and timestamps, and no value: the store refuses
|
|
every `secret/data/` read. The two writes are the review classifier's keys,
|
|
which the operator fills in by hand. Anything more needs a token.
|
|
|
|
| piece | written by |
|
|
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
|
| authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs |
|
|
| its secret at `swarm/services/swarm-bao-ui/oidc/client` | `swarm-secret-publish`, like every other service's |
|
|
| the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token |
|
|
| `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config |
|
|
|
|
The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token
|
|
step. A store set up before the mount existed needs the
|
|
[one-time granter step](../getting-started/setup.md) again: it re-writes
|
|
`bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own
|
|
policy, which covers `auth/oidc/`. Until then
|
|
`swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and
|
|
exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs
|
|
it each time the step succeeds, so OIDC login needs no further restart.
|
|
|
|
## The constraint that decides where the root lives
|
|
|
|
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
|
|
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
|
|
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
|
|
That role can't issue a certificate for a swarm service; the store refuses it.
|
|
|
|
The store's `swarm-services` role issues them instead, from the root inside its
|
|
`pki` mount. That root's private key never leaves the mount, and its
|
|
certificate is the only anchor the swarm has. Each hive fetches the certificate
|
|
from the store at boot, so nothing build-time can name it — the system reads
|
|
`security.pki.certificateFiles`' value at build time, which makes it the wrong
|
|
tool here. Trust reaches containers through a bind-mounted bundle assembled at
|
|
boot instead.
|
|
|
|
## Adding a secret
|
|
|
|
State three things, in the row you add above: **who mints it**, **which
|
|
container reads it**, and **what happens when they differ**. If they differ, it
|
|
needs a delivery unit, and the unit copies — it doesn't bind.
|