swarm-nats: take the callout issuer and user keys from the store
`swarm-bao-nats-callout-xkey` runs `ensure-callout-keys`, importing
/var/lib/swarm-nats-callout/{issuer,callout-user}.seed when present so the
keys keep their public halves, then fetches all three public halves from
the store. The verb removes each seed file once a read of the store returns
a seed; when the store's seed differs, it is kept and a warning names the
file. No tmpfiles rule touches those files, since one could run before the
import with the store sealed and leave the next run to mint a new key.
The server renders the issuer and user public halves from the fetched files,
checked for shape as the xkey's is, and does not start without them. The
responder reads both seeds from the store, so the copies into the container
and their LoadCredential entries are gone, and a tmpfiles rule removes the
copies a container still holds. `swarm-nats-callout-keys` is removed.
On the store's host the fetch unit is also ordered after
`swarm-bao-secret-publisher-policy`, so the publisher's denial of the seed
paths is written before the unit creates or keeps a seed there.
`deploy.nats.autoGenerateCallout`, `calloutUserPublicKey`,
`calloutIssuerPublicKey`, `calloutUserSeedFile` and `calloutIssuerSeedFile`
are removed with `mkRemovedOptionModule`, under `swarm.nats` too, each
message saying the keys live in the store.
This commit is contained in:
atlas
mara
parent
66ac034949
commit
c10cc0e018
12 changed files with 427 additions and 556 deletions
|
|
@ -14,7 +14,7 @@ than a convenience.
|
|||
|
||||
The rule is about what must stay secret, not about credentials generally.
|
||||
**Public material is a value**: a certificate, or a public nkey like
|
||||
`deploy.nats.calloutUserPublicKey`, reaches every client that connects,
|
||||
the queue callout's user key, reaches every client that connects,
|
||||
so the store is a perfectly good place for it.
|
||||
|
||||
**The generator and the reader typically live in different containers.** They share
|
||||
|
|
@ -41,24 +41,24 @@ neither is a renaming of the other.
|
|||
|
||||
## Swarm-level — one of each per swarm
|
||||
|
||||
| secret | generated by | lives at | hive elsewhere |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
|
||||
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
||||
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
||||
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
||||
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
|
||||
| the queue responder's copy of its OIDC secret | `swarm-bao-nats-auth-oidc.service` reads it out of the swarm secret store, on the host that runs the queue, under the responder's own leaf | `/var/lib/swarm-nats-auth/oidc-client.secret` inside the `swarm-nats` container, handed to the responder through `LoadCredential` | same unit, same path, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when the operator sets `deploy.nats.autoGenerateCallout` | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
|
||||
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-xkey`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
||||
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
||||
| secret | generated by | lives at | hive elsewhere |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
|
||||
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
||||
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
||||
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
||||
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
|
||||
| the queue responder's copy of its OIDC secret | `swarm-bao-nats-auth-oidc.service` reads it out of the swarm secret store, on the host that runs the queue, under the responder's own leaf | `/var/lib/swarm-nats-auth/oidc-client.secret` inside the `swarm-nats` container, handed to the responder through `LoadCredential` | same unit, same path, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
||||
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
||||
|
||||
Authelia mints the three keys for itself, in-container, precisely
|
||||
because nothing outside that container ever reads them. **That's the test worth
|
||||
|
|
@ -116,40 +116,45 @@ unit above it for the same structural reason — it reads the store inside the
|
|||
container it delivers into, so it runs **after** that container rather than
|
||||
before it, and only a first boot ever waits.
|
||||
|
||||
### Minting the queue's callout nkeys
|
||||
### The queue's callout nkeys
|
||||
|
||||
`deploy.nats.autoGenerateCallout` mints both keypairs on the host before the queue
|
||||
starts. It's on by default only under `singleHostSwarm` — the one
|
||||
topology where the queue, its responder and the operator are the same person. On
|
||||
every other topology, mint them yourself:
|
||||
The responder holds two nkeys besides the callout xkey below. It signs every
|
||||
user JWT it issues with the **issuer** key, an account key, so possession of
|
||||
the issuer seed is the authority to admit anyone to the queue. It connects to
|
||||
the queue as the **user** key, the one identity the server exempts from
|
||||
callout. You configure nothing about either: both live in the swarm secret
|
||||
store beside the xkey, and come into being on the queue's first boot.
|
||||
|
||||
```
|
||||
nk -gen user > callout-user.seed # the responder's own identity
|
||||
nk -gen account > issuer.seed # signs the user JWTs it hands out
|
||||
nk -inkey callout-user.seed -pubout # → calloutUserPublicKey
|
||||
nk -inkey issuer.seed -pubout # → calloutIssuerPublicKey
|
||||
```
|
||||
- **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`,
|
||||
**public half** at `secret/swarm/public/swarm-nats/callout-issuer`.
|
||||
- **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`,
|
||||
**public half** at `secret/swarm/public/swarm-nats/callout-user`.
|
||||
|
||||
Keep both seeds at `0600` and name them in `deploy.nats.calloutUserSeedFile` /
|
||||
`deploy.nats.calloutIssuerSeedFile`. Possession of the **issuer** seed is the authority to
|
||||
admit anyone to the queue, so it belongs wherever the responder runs and nowhere
|
||||
else.
|
||||
The unit that creates the xkey creates these too, under the same rules: only
|
||||
the `swarm-nats-auth` role reads or creates them, nothing replaces a stored
|
||||
seed, and the policy of `swarm-secret-publisher` denies it both seed paths.
|
||||
The server renders both public halves into its `auth_callout` block as it
|
||||
starts, and doesn't start without them.
|
||||
|
||||
A hive that sets neither the public keys nor `autoGenerateCallout` fails at
|
||||
eval, naming the option it wants. That's deliberate: a queue that started
|
||||
without them would accept `CONNECT {"user":"auth"}` from anyone sharing the
|
||||
host's network namespace, and nothing would look wrong until somebody connected.
|
||||
A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or
|
||||
`/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what
|
||||
`swarm-nats-auth ensure-callout-keys` stores for that key when the store holds
|
||||
none, so the key keeps its public half. Once a read of the store returns a
|
||||
seed, the verb removes the file. When the store already holds a different
|
||||
seed, the verb keeps the store's seed, logs a warning naming the file, then
|
||||
removes it.
|
||||
|
||||
**All four or none** — you must supply the seed paths too, not just the public
|
||||
keys. They're two halves of the same pair: the server verifies with the public
|
||||
half, the responder signs with the private one. Supplying only the public keys
|
||||
used to pass eval and leave the queue with an auth-callout nobody answers, which
|
||||
refuses every client rather than degrading — and a refusal reaches the client as
|
||||
a timeout, so the symptom is every consumer hanging with nothing logged.
|
||||
A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`,
|
||||
`calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`,
|
||||
under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the
|
||||
option and where the keys live now.
|
||||
|
||||
To replace either key, delete both of its paths the way the xkey's section
|
||||
below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`.
|
||||
|
||||
One consequence worth knowing before you debug it: the queue assembles its
|
||||
config each time the server starts rather than at build time, in every mode,
|
||||
because the callout xkey below exists only at runtime. A malformed config
|
||||
because the callout keys exist only at runtime. A malformed config
|
||||
surfaces when the server starts instead of when the system builds. The server
|
||||
names the offending file and refuses to run.
|
||||
|
||||
|
|
@ -171,7 +176,7 @@ of `swarm/services/`, denies it the seed's path.
|
|||
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
|
||||
`swarm-nats` container, under the responder's store leaf:
|
||||
|
||||
1. `swarm-nats-auth ensure-xkey` creates the pair when the store holds no
|
||||
1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no
|
||||
seed, and otherwise only reads it. It needs no queue, so on a first boot
|
||||
the key exists before the server that needs it.
|
||||
2. It reads the public half back from the store into
|
||||
|
|
@ -198,10 +203,14 @@ issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and
|
|||
name it in both options. Until you do, evaluation fails naming both.
|
||||
|
||||
Where the queue and the store run on different hosts, deploy the store's host
|
||||
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the two
|
||||
paths. A queue host deployed before it can't read or create the key, so its
|
||||
fetch fails and the server stays down until you deploy the store's host. If
|
||||
that takes longer than the fetch's day of retries, run
|
||||
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the
|
||||
keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of
|
||||
the seeds' paths; where the queue runs on that host too, the fetch waits for
|
||||
both. A queue host deployed before it can't read or create the keys, so its
|
||||
fetch fails, and the queue refuses every client until you deploy the store's
|
||||
host: the server starts only on public keys fetched before, and the responder
|
||||
can't load its seeds. Seed files the import is waiting for stay where they
|
||||
are. If that takes longer than the fetch's day of retries, run
|
||||
`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
|
||||
|
||||
Two starts that create the key at once can't both win: each write is a
|
||||
|
|
|
|||
Loading…
Reference in a new issue