docs: the swarm README and credentials page keep the overview, the rest moves to its own pages
The hive directory and the WireGuard mesh become handbook pages of their own, the controller's mint-failure handling moves to controller.md, and the credential-by-credential mechanics and the secret store UI's proxy move to a new implementation page.
This commit is contained in:
iris
mara
parent
7beb1193f4
commit
0e99662710
1 changed files with 559 additions and 514 deletions
302
docs/4-implementation/swarm/credential-delivery.md
Normal file
302
docs/4-implementation/swarm/credential-delivery.md
Normal file
|
|
@ -0,0 +1,302 @@
|
|||
# How each credential reaches its reader
|
||||
|
||||
The swarm-level and hive-level secrets, one by one: which unit mints each, where
|
||||
it lands, and what a missing identity costs. The rules a credential follows are
|
||||
on [the handbook's credentials page](../../2-handbook/swarm/credentials.md); the
|
||||
table of every credential is in
|
||||
[`credentials-reference.md`](credentials-reference.md), and the store's own
|
||||
mechanics are in [`credentials.md`](credentials.md).
|
||||
|
||||
## Swarm-level secrets
|
||||
|
||||
The rows are in the [reference](credentials-reference.md#swarm-level--one-of-each-per-swarm).
|
||||
|
||||
Nothing outside authelia's container reads its keys (session, JWT, storage encryption,
|
||||
OIDC HMAC and the OIDC issuer), which is why no delivery step exists for them. **That's the
|
||||
test worth applying to any secret added here** — and the client secret's plaintext half is
|
||||
the one row that fails it, which is the entire reason a delivery step exists.
|
||||
|
||||
The store still holds those five keys, at `swarm/services/swarm-authelia/keys/<name>`,
|
||||
so that losing the host doesn't lose them. `authelia-swarm-secrets` in the container reads
|
||||
each with the secret publisher's identity and writes it to the file authelia loads. A store
|
||||
with no value takes the file's, so the key keeps its value; the unit mints a new key only when
|
||||
there is neither. Nothing rotates a key: the storage-encryption key encrypts authelia's
|
||||
database, and a new issuer key invalidates every token issued. A container without the
|
||||
publisher's identity generates the files as before.
|
||||
|
||||
The same unit mints the client secrets, straight into the store. It reads
|
||||
`swarm/hives/<hive>/oidc/client` or `swarm/services/<client id>/oidc/client` first, and
|
||||
only when the store answers and holds none does it run `authelia crypto hash generate`,
|
||||
create the leaf and write the digest. Authelia keeps only the digest on disk, in
|
||||
`oidc-clients/<id>.digest`, and starts from it while the store is down. A store that
|
||||
can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries after 15 seconds, then at growing intervals up to two minutes, so authelia starts meanwhile and `swarm-authelia-bridge` finds its secret once the store answers. A client with a
|
||||
secret in the store but no digest gets the digest of that secret and keeps the secret; `authelia crypto hash generate` computes the digest,
|
||||
with the password typed into its prompt through a pty, never in a command line.
|
||||
`swarm-authelia-bridge` fetches its own secret from
|
||||
`swarm/services/swarm-authelia-bridge/oidc/client` into
|
||||
`/run/swarm-authelia-bridge/client-secret` before each start, and a store that doesn't
|
||||
answer keeps the bridge down while it retries. A container without the publisher's
|
||||
identity keeps the plaintext in `oidc-clients/<id>.secret`, as before.
|
||||
|
||||
The mint reads an `oidc-clients/<id>.secret` file an earlier release left once: when the store holds
|
||||
no value for that client, it stores the file's value with `-cas=0`, so the secret keeps its value.
|
||||
A store that holds a value wins, whatever the file says, and the file stays on disk, unused.
|
||||
|
||||
Three telemetry collectors exist, and each one fails that test: each reads its
|
||||
own secret out of the store, under its own identity.
|
||||
|
||||
The **hive's** collector authenticates to the swarm's collector as its own hive,
|
||||
so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches
|
||||
`swarm/hives/<hive>/oidc/client` under the hive's own leaf
|
||||
(`deploy.bao.clientCertFile`), on every topology, into `/run/hive-otel-oidc` on
|
||||
the host's tmpfs. The file never reaches persistent disk and a reboot fetches
|
||||
it again; until the store answers, the collector runs on an empty file and the
|
||||
swarm refuses its exports. A hive with no leaf has no
|
||||
default, and the build stops at the collector's identity assertion unless
|
||||
`services.hyperhive.otel.clientSecretFile` names a file delivered some other way.
|
||||
|
||||
The **swarm's** collector reads its own under a leaf of its own, the fourth
|
||||
store reader — the same shape as Grafana's, described in full below. In short:
|
||||
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
|
||||
out of the store on every host that runs the collector and holds a store
|
||||
identity, whether authelia is a network away or in the container next door.
|
||||
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
|
||||
the client it reads for unconditionally: authelia refuses a
|
||||
bearer-authz client with no audience, and every host publishes at least
|
||||
forgejo's `/metrics` as a scrape target, so the audience is never empty.
|
||||
|
||||
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
|
||||
where an absent OIDC block is a container with no way in at all, a collector
|
||||
with nothing to authenticate with still receives every hive's telemetry —
|
||||
its pushes to the stores go out unauthenticated and get refused there, and its
|
||||
scrape jobs for the published targets read a credential file nothing
|
||||
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
|
||||
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
|
||||
`swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
|
||||
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
|
||||
collector this unit never reaches.
|
||||
|
||||
The **secret store's own** collector — the forwarder inside the `swarm-bao`
|
||||
container — needs a delivery step too, and it takes the same route with one
|
||||
principal of its own: `swarm-bao-forwarder-oidc.service` reads
|
||||
`swarm/services/<client-id>/oidc/client` under a leaf of its own and
|
||||
writes it to `/run/swarm-bao-forwarder-oidc` on the host's tmpfs, which the container bind-mounts read-only and where `LoadCredential` hands it to the
|
||||
collector. The client id is its own
|
||||
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
|
||||
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
|
||||
collector admits it on a receiver of its own — one identity per principal, so
|
||||
this forwarder borrows neither the swarm collector's client nor
|
||||
`swarm-controller`'s.
|
||||
|
||||
⚠️ **This one has no degraded mode, and the difference is structural.** The
|
||||
route it exports over accepts an authenticated request and nothing else, so a
|
||||
forwarder without the secret has nothing to fall back to: the unit fails and
|
||||
retries, and the collector refuses to start until the file arrives, after every boot, rather than
|
||||
exporting into a 401 while reporting healthy. Its ordering differs from the
|
||||
unit above it for the same structural reason — it reads the store inside the
|
||||
container it delivers into, so it runs **after** that container rather than
|
||||
before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.
|
||||
|
||||
## Hive-level secrets
|
||||
|
||||
The rows are in the [reference](credentials-reference.md#hive-level--one-of-each-per-hive).
|
||||
|
||||
The forge and the homeserver read their copies out of the store, the way
|
||||
Grafana does below: the forge under the forge publisher's leaf, the
|
||||
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
|
||||
mint in authelia's container has stored the secret, rather than skipping.
|
||||
A silent skip produces a service whose login button always fails, which is a
|
||||
symptom many layers from its cause.
|
||||
|
||||
The store's **first reader** is Grafana's OIDC client secret, a _swarm
|
||||
service's_ own credential rather than a hive's. It has exactly
|
||||
one delivery route: `swarm-bao-grafana-oidc.service` reads
|
||||
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
|
||||
that runs Grafana. Where authelia is doesn't enter into it — the mint in
|
||||
authelia's container writes that path whether the reader is a network away or in the
|
||||
container next door.
|
||||
|
||||
That's a deliberate refusal of the shortcut. Copying the plaintext
|
||||
straight out of authelia's tree when the two happen to share a host is one
|
||||
round trip cheaper and costs a second delivery unit, a second way for the file
|
||||
to be wrong, and a gate deciding between them — and the gate is the expensive
|
||||
part, because whatever it's wrong about is an outage nobody can read. The store
|
||||
exists so the only secrets a host holds out of band are **certificates**, and
|
||||
a host reads everything else with them.
|
||||
|
||||
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
|
||||
running Grafana requires `swarm.authelia.url` (this module disables its local
|
||||
login form unconditionally, so SSO isn't a feature of some topologies — it's the
|
||||
only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
|
||||
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
|
||||
a null URL dropped the OIDC block, a missing leaf produced a warning, and
|
||||
either one left a Grafana with no SSO and no password box — nothing failed, so
|
||||
nothing said anything. Registration is separate and stays where authelia is
|
||||
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
|
||||
so it's declared on the host running authelia, which isn't necessarily the host
|
||||
running Grafana. The swarm controller, the queue, the homeserver and the forge
|
||||
register the same way (`glue-controller-oidc-client.nix`,
|
||||
`glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
|
||||
`glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
|
||||
forge read their secrets from the store too: the responder itself under its own
|
||||
leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
|
||||
`swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
|
||||
the option pair, when that leaf is unset — neither the responder nor the
|
||||
homeserver starts without its secret, and the forge registers no login source
|
||||
without its.
|
||||
|
||||
Grafana's admin password rides the same leaf and the same store. Where Grafana
|
||||
and the store run on different hosts, deploy the store's host first:
|
||||
`swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create
|
||||
the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
|
||||
the Grafana host, which retries, while Grafana stays down because its config
|
||||
reads that file.
|
||||
|
||||
The **second reader** is the swarm collector's own OIDC client secret, the
|
||||
same shape one level down: `swarm-bao-otel-oidc.service` reads
|
||||
`swarm/services/<client-id>/oidc/client` out of the store on every host that
|
||||
runs the collector, and registration lives separately in
|
||||
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
|
||||
differs is what an absent credential means: the collector still receives
|
||||
telemetry with none, so `swarm-otel.nix` doesn't assert
|
||||
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
|
||||
does — it
|
||||
doesn't render the reading unit without them.
|
||||
|
||||
The **third reader** is the secret store forwarder's OIDC client secret:
|
||||
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
|
||||
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
|
||||
second in both directions. It renders unconditionally, because the export it
|
||||
authenticates has no unauthenticated mode to degrade into — and it orders
|
||||
itself **after** `container@swarm-bao`, because the store it reads runs in the
|
||||
container it delivers into. Nothing circular sits behind that: the identity it
|
||||
logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not
|
||||
anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix`
|
||||
refuses the build and names `deploy.bao.forwarderOidcClientCertFile` /
|
||||
`forwarderOidcClientKeyFile`.
|
||||
|
||||
A service's secret is one value for the whole swarm rather than one per hive, so
|
||||
it lives under the `services` prefix, and a hive's read policy grants that prefix
|
||||
whole. That's because nothing in a swarm's configuration records which hive runs
|
||||
a given service — placement is a `deploy.*` fact, per-host by definition — so
|
||||
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
|
||||
every swarm service's client secret, and that's stated in
|
||||
`swarm-secret-client`'s `policy` module beside the grant itself.
|
||||
|
||||
⚠️ **The readers no longer present it.** Five units used to log in with
|
||||
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
|
||||
principal by the subject of the certificate it presents — so five readers behind
|
||||
one leaf were one principal holding the union of their needs. Each now holds a
|
||||
leaf of its own, and a policy naming only the path that unit reads. See
|
||||
[per-principal identities](credentials.md#per-principal-identities).
|
||||
|
||||
Absence behaves
|
||||
as it does for the other readers: a store that says "not here" leaves the
|
||||
file alone and says so, since there is no such thing as a locally valid OIDC
|
||||
client secret to stand in.
|
||||
|
||||
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
|
||||
credential you must already hold to authenticate from the thing it
|
||||
authenticates you to, so the store's identity can't come from an authority the store
|
||||
distributes — which excludes the store's `pki` root, and has nothing to do
|
||||
with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
|
||||
mints a CA that signs a fixed, short list of leaves — the store's server
|
||||
certificate, the hive's own reader certificate, and one per service principal —
|
||||
and distributes nothing. A deployment with a real
|
||||
internal CA deletes that file and names its own paths in
|
||||
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
|
||||
A hive that reads from a store on **another** machine names the reader's half —
|
||||
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
|
||||
plus one leaf per principal it runs (the [table in the reference](credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
|
||||
credentials that can't come out of the store, being what opens it; everything
|
||||
else a hive needs does.
|
||||
|
||||
Agents are the one principal whose leaf the store issues. Its `pki-agents`
|
||||
mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the
|
||||
store, and `swarm-controller`, already logged in under its own host leaf, asks
|
||||
that mount's one role for a `hive-agent-<agent>` client certificate at agent
|
||||
creation. Host roles never pin that CA and agent roles pin only it, so an
|
||||
agent's certificate opens that agent's own `swarm/agents/<agent>/*` and
|
||||
nothing else: `read` on the values there and `list` on their names, and `create` (never `update`) on one leaf, `backend-env`, which the agent seeds from its environment file.
|
||||
|
||||
All agents share one policy, `hive-agent`, which the agent's cert-auth role
|
||||
attaches and no other. It states the rule with `{{identity.entity.name}}`
|
||||
where an agent's name would go, so the rule is one document and a change to it
|
||||
is one write. `swarm-controller` writes it at agent creation and at its own
|
||||
start.
|
||||
|
||||
For that the controller names each agent's identity entity after the agent: at
|
||||
each certificate mint or renewal, before it writes the agent's cert-auth role,
|
||||
and for every agent at its own start. It reads the cert mount's accessor and
|
||||
creates the entity `<agent>` with the alias `hive-agent-<agent>`, or renames the
|
||||
entity a first login created under a generated name. At a mint or renewal this is
|
||||
a precondition: when the store refuses a call, or the alias sits on any other
|
||||
entity, the mint fails and the role stays as it was, so a live agent keeps
|
||||
working until an operator resolves it. The start pass only logs the same
|
||||
failures, because it writes no role. The controller needs the `identity/` and
|
||||
`sys/auth/cert` grants in `swarm-bao.nix`'s `controllerPolicyText` for that.
|
||||
|
||||
A cert-auth role that predates the shared policy may still name a policy
|
||||
`hive-agent-<agent>`. The controller no longer writes or attaches these. The
|
||||
objects stay in the store and keep granting what they did, to a token that
|
||||
already carries them and to a role not yet rewritten. A live agent's role is
|
||||
rewritten at its next certificate renewal.
|
||||
|
||||
A hive's cert-auth role `hive-<hive>` attaches one policy, `hive-hives`, shared
|
||||
by all hives. A hive's certificate carries its bare name as
|
||||
the common name, which is the login's alias name, so hives need no entity:
|
||||
the policy states the hive's own path as
|
||||
`swarm/hives/{{identity.entity.aliases.<accessor>.name}}/*`, where `<accessor>`
|
||||
is the cert mount's accessor. The controller reads the accessor at its own start,
|
||||
writes the policy, and rewrites the roles to attach it. No hive name can
|
||||
spell `hive-hives`: it would be a hive called `hives`, and
|
||||
`reserved-hive-fragments.nix` refuses the substring `hive`. When the controller
|
||||
can't read the accessor or write the policy it logs a warning and leaves the
|
||||
roles untouched, so a role that names a `hive-<hive>` policy keeps it.
|
||||
|
||||
### What a missing leaf costs
|
||||
|
||||
Each of the four modules asserts its own pair, on the same condition: **this host
|
||||
already reads the store** — it holds `deploy.bao.clientCertFile` and
|
||||
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
|
||||
The refusal names the two options, so an operator acts on the message without
|
||||
opening the nix.
|
||||
|
||||
That condition fires on exactly one shape, the hand-configured remote-store hive
|
||||
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
|
||||
that matters on its own:
|
||||
|
||||
- **A host with no store identity at all** holds no `clientCertFile` either, so
|
||||
the first clause is false. That's the supported no-store deployment, and also
|
||||
the one an operator passes through while bringing a hive up — neither should
|
||||
fail to evaluate.
|
||||
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
|
||||
defaults, so the third clause is false. Nothing to set, nothing to refuse.
|
||||
- **A consumer that's off** — no collector, no Grafana — has no
|
||||
unit to skip, so the second clause is false.
|
||||
|
||||
The collector keeps a separate degrade underneath this, unchanged: a host with no
|
||||
store identity runs a collector with no OIDC client secret and still receives
|
||||
telemetry. The assertion doesn't touch that shape, because that host has no
|
||||
`clientCertFile` either.
|
||||
|
||||
## The secret store browser UI
|
||||
|
||||
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
|
||||
(`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in
|
||||
with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost
|
||||
checks the session with authelia and proxies to an nginx inside the store's
|
||||
container on
|
||||
`127.0.0.1:<deploy.bao.uiProxyPort>`. That nginx forwards `/ui/` and `/v1/` to
|
||||
a second openbao listener on `127.0.0.1:<deploy.bao.uiPort>`, answers 403 on
|
||||
the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on
|
||||
everything else. ⚠️ That listener asks for **no client certificate**, because
|
||||
a browser has none to present, so on this one door a bao token is the whole
|
||||
credential. Anything on the store's host that can dial loopback, and any
|
||||
`admins` session through the vhost, needs only a token to use the API. Unseal
|
||||
from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed
|
||||
gateway where the UI is the only swarm name its host fronts, that host's nginx
|
||||
doesn't wait for the store: it serves the hive leaf, or on a host that runs no
|
||||
hive a throwaway self-signed certificate, on the UI's name (a browser warning)
|
||||
until the unsealed store issues the services leaf, so the
|
||||
stream passthrough readers use on that host comes up with the store sealed.
|
||||
Loading…
Reference in a new issue