swarm-otel: deliver the OIDC client secret through the secret store
The swarm collector's OIDC client secret only existed where authelia did: `swarm-otel-oidc-secret.service` copied the minted plaintext out of authelia's container tree, reachable only because the two share a host's network namespace. A swarm that placed authelia elsewhere delivered nothing, and the option's own description said so — "a deployment that places authelia elsewhere points this at a file it delivers itself." Same gap as #3853 and #4234, and this is the swarm-otel twin of #4234's fix for Grafana. Mirrors PR #4361 (Grafana) almost exactly: - `swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client` out of the store, in every deployment, replacing the co-located copy unit outright — one delivery route, not two, per the ruling that landed under #4234. - Client registration moved out of `swarm-otel.nix`'s own `config` block (gated on this host running the collector) into `glue-swarm-otel-oidc-client.nix` (gated on this host running authelia), the same split `glue-grafana-oidc-client.nix` made. It was broken the same way: a split deployment registered the client nowhere at all, so authelia never minted a secret for the publisher to send on. - The publisher's `services` prefix (write grant in `swarm-bao.nix`, hive read grant in `policy::render`) already covers any service's path — nothing to add there. `swarm-secret-publisher.nix` only grew `serviceClientIds` by one entry. One judgement call, stated rather than buried: the store-reading unit renders only where this host holds a client identity (`deploy.bao.clientCertFile`/`clientKeyFile`), rather than asserting it the way `swarm-grafana.nix` does. Grafana's local login form is disabled unconditionally, so a Grafana with no OIDC secret has no way in at all — that earns a hard refusal. This collector without a credential still receives every hive's telemetry; only its own pushes to the stores go out unauthenticated and get refused there, an already-supported degrade the module's own `haveCollectorSecret` flag named before this change. So the reading unit follows the shape `glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for their own optional readers: no unit when the identity is absent, not a build refusal. Fixtures mirror #4361's: `otelBaoWithAuthelia`/`otelBaoRemoteAuthelia` are the positive pair (co-located and split, both reading through the store), `otelNoIdentity` is the negative — no reading unit, no assertion firing, `clientSecretFile` left null. Refs #4258
This commit is contained in:
atlas
mara
parent
baab0f393e
commit
0ff5c8110b
7 changed files with 409 additions and 148 deletions
|
|
@ -41,23 +41,23 @@ neither is a renaming of the other.
|
|||
|
||||
## Swarm-level — one of each per swarm
|
||||
|
||||
| secret | generated by | lives at | hive elsewhere |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
|
||||
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
|
||||
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
|
||||
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
||||
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
||||
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||
| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
|
||||
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
||||
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
||||
| secret | generated by | lives at | hive elsewhere |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
|
||||
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
|
||||
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
|
||||
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
||||
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
||||
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity is left with `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
|
||||
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
||||
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
||||
|
||||
Authelia mints the three keys for itself, in-container, precisely
|
||||
because nothing outside that container ever reads them. **That's the test worth
|
||||
|
|
@ -72,27 +72,27 @@ an all-local swarm it reads authelia's file where it lies and no second copy is
|
|||
made. On any other topology it's an ordinary "operator provides the file"
|
||||
case — see `services.hyperhive.otel.clientSecretFile`.
|
||||
|
||||
The **swarm's** collector does need one. It runs in a container, so
|
||||
`swarm-otel-oidc-secret.service` places its copy, landing at
|
||||
`/var/lib/swarm-otel-oidc/<client-id>.secret` — the same shape as the forge and
|
||||
homeserver rows below, and for the same reason: the container that mints the
|
||||
secret isn't the container that reads it.
|
||||
The **swarm's** collector does need one, and its delivery route is the fourth
|
||||
store reader — the same shape as Grafana's, described in full below. In short:
|
||||
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
|
||||
out of the store on every host that runs the collector and holds a store
|
||||
identity, whether authelia is a network away or in the container next door.
|
||||
The client it reads for is registered unconditionally, the same call
|
||||
`glue-grafana-oidc-client.nix` makes for Grafana's: authelia refuses a
|
||||
bearer-authz client with no audience, and the push audiences supply one even
|
||||
for a collector with nothing published to scrape.
|
||||
|
||||
The copy is made whenever authelia is enabled **on this host**, whatever this
|
||||
collector scrapes. The client it delivers for is registered unconditionally:
|
||||
authelia refuses a bearer-authz client with no audience, and the push audiences
|
||||
supply one even for a collector with nothing published to scrape. Like the
|
||||
hive-level delivery units below, this one waits a bounded 120s for authelia's
|
||||
mint and then fails rather than skipping, so a secret that never arrives takes
|
||||
the collector's container start with it instead of surfacing later as a 401.
|
||||
|
||||
⚠️ **Don't read that delivery unit as the only way this collector is fed.**
|
||||
Whether it authenticates follows the _credential_, never another service's
|
||||
placement: a swarm collector may run on a host that holds neither store and no
|
||||
authelia, and then the secret is an ordinary operator-provided file named in
|
||||
`services.hyperhive.deploy.swarm-otel.clientSecretFile` — the same shape as the hive
|
||||
collector's row above. The copy unit is the convenience for the co-located case,
|
||||
not the definition of the case.
|
||||
⚠️ **A collector with no store identity is not refused.** Unlike Grafana,
|
||||
where an absent OIDC block is a container with no way in at all, a collector
|
||||
with nothing to authenticate with still receives every hive's telemetry —
|
||||
only its own pushes to the stores go out unauthenticated and get refused
|
||||
there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
|
||||
`deploy.bao.clientCertFile` / `clientKeyFile`, the same choice
|
||||
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for
|
||||
their own optional readers, rather than the hard assertion `swarm-grafana.nix`
|
||||
uses for its non-optional one. The secret can still be delivered by hand, at
|
||||
`services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a collector this
|
||||
unit never reaches.
|
||||
|
||||
### Minting the queue's callout nkeys
|
||||
|
||||
|
|
@ -212,6 +212,18 @@ nothing said anything. Registration is separate and stays where authelia is
|
|||
so it's declared on the host running authelia, which isn't necessarily the host
|
||||
running Grafana.
|
||||
|
||||
The **fourth reader** is the swarm collector's own OIDC client secret, the
|
||||
same shape one level down: `swarm-bao-otel-oidc.service` reads
|
||||
`swarm/services/<client-id>/oidc/client` out of the store on every host that
|
||||
runs the collector, and registration lives separately in
|
||||
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
|
||||
differs is what an absent credential means: the collector still receives
|
||||
telemetry with none, so `swarm-otel.nix` doesn't assert
|
||||
`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it simply
|
||||
doesn't render the reading unit without them, the shape
|
||||
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for
|
||||
their own optional reads.
|
||||
|
||||
A service's secret is one value for the whole swarm rather than one per hive, so
|
||||
it lives under the `services` prefix, and a hive's read policy grants that prefix
|
||||
whole. That's because nothing in a swarm's configuration records which hive runs
|
||||
|
|
@ -220,7 +232,7 @@ there is nothing to scope the grant to. Every hive can therefore read every swar
|
|||
service's client secret; that's the cost of a shape where the service's host has
|
||||
no store identity of its own to present, and it's stated in
|
||||
`swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves
|
||||
as it does for the other two readers: a store that says "not here" leaves the
|
||||
as it does for the other readers: a store that says "not here" leaves the
|
||||
file alone and says so, since there is no such thing as a locally valid OIDC
|
||||
client secret to stand in.
|
||||
|
||||
|
|
|
|||
|
|
@ -66,16 +66,17 @@ _own_ entry under the hive namespace, which names the hive asking and so widens
|
|||
nothing between them, and every **swarm service's** OIDC client secret.
|
||||
|
||||
That third prefix has the same shape of reason as the first, and the same honest
|
||||
cost. A swarm service (Grafana is the one there today) registers **one** client
|
||||
for the whole swarm, so its credential's path names the service and never the
|
||||
host — and which hive runs a given service is a `deploy.*` fact, per-host by
|
||||
definition, so nothing swarm-wide exists to scope the grant to. The host running
|
||||
such a service has no store identity of its own either; it reads with the
|
||||
certificate of the hive it is. **Any hive can therefore read any swarm service's
|
||||
client secret**, which is worth what it buys: every swarm service gets its
|
||||
credential the same way from anywhere, instead of only where the identity
|
||||
provider happens to sit. Giving such a service its own store identity is what
|
||||
would remove this rather than re-scope it.
|
||||
cost. A swarm service (Grafana and the swarm collector are the two there today)
|
||||
registers **one** client for the whole swarm, so its credential's path names
|
||||
the service and never the host — and which hive runs a given service is a
|
||||
`deploy.*` fact, per-host by definition, so nothing swarm-wide exists to scope
|
||||
the grant to. The host running such a service has no store identity of its own
|
||||
either; it reads with the certificate of the hive it is. **Any hive can
|
||||
therefore read any swarm service's client secret**, which is worth what it
|
||||
buys: every swarm service gets its credential the same way from anywhere,
|
||||
instead of only where the identity provider happens to sit. Giving such a
|
||||
service its own store identity is what would remove this rather than re-scope
|
||||
it.
|
||||
|
||||
**A tracked follow-up** narrows this, with the two candidate directions: scope
|
||||
the grant per hive (and pay for the re-emission), or give each agent container
|
||||
|
|
|
|||
Loading…
Reference in a new issue