swarm-otel: deliver the OIDC client secret through the secret store
The swarm collector's OIDC client secret only existed where authelia did: `swarm-otel-oidc-secret.service` copied the minted plaintext out of authelia's container tree, reachable only because the two share a host's network namespace. A swarm that placed authelia elsewhere delivered nothing, and the option's own description said so — "a deployment that places authelia elsewhere points this at a file it delivers itself." Same gap as #3853 and #4234, and this is the swarm-otel twin of #4234's fix for Grafana. Mirrors PR #4361 (Grafana) almost exactly: - `swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client` out of the store, in every deployment, replacing the co-located copy unit outright — one delivery route, not two, per the ruling that landed under #4234. - Client registration moved out of `swarm-otel.nix`'s own `config` block (gated on this host running the collector) into `glue-swarm-otel-oidc-client.nix` (gated on this host running authelia), the same split `glue-grafana-oidc-client.nix` made. It was broken the same way: a split deployment registered the client nowhere at all, so authelia never minted a secret for the publisher to send on. - The publisher's `services` prefix (write grant in `swarm-bao.nix`, hive read grant in `policy::render`) already covers any service's path — nothing to add there. `swarm-secret-publisher.nix` only grew `serviceClientIds` by one entry. One judgement call, stated rather than buried: the store-reading unit renders only where this host holds a client identity (`deploy.bao.clientCertFile`/`clientKeyFile`), rather than asserting it the way `swarm-grafana.nix` does. Grafana's local login form is disabled unconditionally, so a Grafana with no OIDC secret has no way in at all — that earns a hard refusal. This collector without a credential still receives every hive's telemetry; only its own pushes to the stores go out unauthenticated and get refused there, an already-supported degrade the module's own `haveCollectorSecret` flag named before this change. So the reading unit follows the shape `glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for their own optional readers: no unit when the identity is absent, not a build refusal. Fixtures mirror #4361's: `otelBaoWithAuthelia`/`otelBaoRemoteAuthelia` are the positive pair (co-located and split, both reading through the store), `otelNoIdentity` is the negative — no reading unit, no assertion firing, `clientSecretFile` left null. Refs #4258
This commit is contained in:
atlas
mara
parent
baab0f393e
commit
0ff5c8110b
7 changed files with 409 additions and 148 deletions
|
|
@ -41,23 +41,23 @@ neither is a renaming of the other.
|
|||
|
||||
## Swarm-level — one of each per swarm
|
||||
|
||||
| secret | generated by | lives at | hive elsewhere |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
|
||||
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
|
||||
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
|
||||
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
||||
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
||||
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||
| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
|
||||
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
||||
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
||||
| secret | generated by | lives at | hive elsewhere |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
|
||||
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
|
||||
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
|
||||
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
||||
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
||||
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity is left with `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
|
||||
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
||||
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
||||
|
||||
Authelia mints the three keys for itself, in-container, precisely
|
||||
because nothing outside that container ever reads them. **That's the test worth
|
||||
|
|
@ -72,27 +72,27 @@ an all-local swarm it reads authelia's file where it lies and no second copy is
|
|||
made. On any other topology it's an ordinary "operator provides the file"
|
||||
case — see `services.hyperhive.otel.clientSecretFile`.
|
||||
|
||||
The **swarm's** collector does need one. It runs in a container, so
|
||||
`swarm-otel-oidc-secret.service` places its copy, landing at
|
||||
`/var/lib/swarm-otel-oidc/<client-id>.secret` — the same shape as the forge and
|
||||
homeserver rows below, and for the same reason: the container that mints the
|
||||
secret isn't the container that reads it.
|
||||
The **swarm's** collector does need one, and its delivery route is the fourth
|
||||
store reader — the same shape as Grafana's, described in full below. In short:
|
||||
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
|
||||
out of the store on every host that runs the collector and holds a store
|
||||
identity, whether authelia is a network away or in the container next door.
|
||||
The client it reads for is registered unconditionally, the same call
|
||||
`glue-grafana-oidc-client.nix` makes for Grafana's: authelia refuses a
|
||||
bearer-authz client with no audience, and the push audiences supply one even
|
||||
for a collector with nothing published to scrape.
|
||||
|
||||
The copy is made whenever authelia is enabled **on this host**, whatever this
|
||||
collector scrapes. The client it delivers for is registered unconditionally:
|
||||
authelia refuses a bearer-authz client with no audience, and the push audiences
|
||||
supply one even for a collector with nothing published to scrape. Like the
|
||||
hive-level delivery units below, this one waits a bounded 120s for authelia's
|
||||
mint and then fails rather than skipping, so a secret that never arrives takes
|
||||
the collector's container start with it instead of surfacing later as a 401.
|
||||
|
||||
⚠️ **Don't read that delivery unit as the only way this collector is fed.**
|
||||
Whether it authenticates follows the _credential_, never another service's
|
||||
placement: a swarm collector may run on a host that holds neither store and no
|
||||
authelia, and then the secret is an ordinary operator-provided file named in
|
||||
`services.hyperhive.deploy.swarm-otel.clientSecretFile` — the same shape as the hive
|
||||
collector's row above. The copy unit is the convenience for the co-located case,
|
||||
not the definition of the case.
|
||||
⚠️ **A collector with no store identity is not refused.** Unlike Grafana,
|
||||
where an absent OIDC block is a container with no way in at all, a collector
|
||||
with nothing to authenticate with still receives every hive's telemetry —
|
||||
only its own pushes to the stores go out unauthenticated and get refused
|
||||
there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
|
||||
`deploy.bao.clientCertFile` / `clientKeyFile`, the same choice
|
||||
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for
|
||||
their own optional readers, rather than the hard assertion `swarm-grafana.nix`
|
||||
uses for its non-optional one. The secret can still be delivered by hand, at
|
||||
`services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a collector this
|
||||
unit never reaches.
|
||||
|
||||
### Minting the queue's callout nkeys
|
||||
|
||||
|
|
@ -212,6 +212,18 @@ nothing said anything. Registration is separate and stays where authelia is
|
|||
so it's declared on the host running authelia, which isn't necessarily the host
|
||||
running Grafana.
|
||||
|
||||
The **fourth reader** is the swarm collector's own OIDC client secret, the
|
||||
same shape one level down: `swarm-bao-otel-oidc.service` reads
|
||||
`swarm/services/<client-id>/oidc/client` out of the store on every host that
|
||||
runs the collector, and registration lives separately in
|
||||
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
|
||||
differs is what an absent credential means: the collector still receives
|
||||
telemetry with none, so `swarm-otel.nix` doesn't assert
|
||||
`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it simply
|
||||
doesn't render the reading unit without them, the shape
|
||||
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for
|
||||
their own optional reads.
|
||||
|
||||
A service's secret is one value for the whole swarm rather than one per hive, so
|
||||
it lives under the `services` prefix, and a hive's read policy grants that prefix
|
||||
whole. That's because nothing in a swarm's configuration records which hive runs
|
||||
|
|
@ -220,7 +232,7 @@ there is nothing to scope the grant to. Every hive can therefore read every swar
|
|||
service's client secret; that's the cost of a shape where the service's host has
|
||||
no store identity of its own to present, and it's stated in
|
||||
`swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves
|
||||
as it does for the other two readers: a store that says "not here" leaves the
|
||||
as it does for the other readers: a store that says "not here" leaves the
|
||||
file alone and says so, since there is no such thing as a locally valid OIDC
|
||||
client secret to stand in.
|
||||
|
||||
|
|
|
|||
|
|
@ -66,16 +66,17 @@ _own_ entry under the hive namespace, which names the hive asking and so widens
|
|||
nothing between them, and every **swarm service's** OIDC client secret.
|
||||
|
||||
That third prefix has the same shape of reason as the first, and the same honest
|
||||
cost. A swarm service (Grafana is the one there today) registers **one** client
|
||||
for the whole swarm, so its credential's path names the service and never the
|
||||
host — and which hive runs a given service is a `deploy.*` fact, per-host by
|
||||
definition, so nothing swarm-wide exists to scope the grant to. The host running
|
||||
such a service has no store identity of its own either; it reads with the
|
||||
certificate of the hive it is. **Any hive can therefore read any swarm service's
|
||||
client secret**, which is worth what it buys: every swarm service gets its
|
||||
credential the same way from anywhere, instead of only where the identity
|
||||
provider happens to sit. Giving such a service its own store identity is what
|
||||
would remove this rather than re-scope it.
|
||||
cost. A swarm service (Grafana and the swarm collector are the two there today)
|
||||
registers **one** client for the whole swarm, so its credential's path names
|
||||
the service and never the host — and which hive runs a given service is a
|
||||
`deploy.*` fact, per-host by definition, so nothing swarm-wide exists to scope
|
||||
the grant to. The host running such a service has no store identity of its own
|
||||
either; it reads with the certificate of the hive it is. **Any hive can
|
||||
therefore read any swarm service's client secret**, which is worth what it
|
||||
buys: every swarm service gets its credential the same way from anywhere,
|
||||
instead of only where the identity provider happens to sit. Giving such a
|
||||
service its own store identity is what would remove this rather than re-scope
|
||||
it.
|
||||
|
||||
**A tracked follow-up** narrows this, with the two candidate directions: scope
|
||||
the grant per hive (and pay for the re-emission), or give each agent container
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@
|
|||
./glue-matrix-bao-token.nix
|
||||
./glue-queue-agent-credential.nix
|
||||
./glue-secret-publisher-bao-identity.nix
|
||||
./glue-swarm-otel-oidc-client.nix
|
||||
./swarm-authelia.nix
|
||||
./swarm-bao.nix
|
||||
./swarm-ca.nix
|
||||
|
|
|
|||
58
nix/host-modules/glue-swarm-otel-oidc-client.nix
Normal file
58
nix/host-modules/glue-swarm-otel-oidc-client.nix
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
# Glue: register the swarm's collector as an OIDC client wherever authelia
|
||||
# runs.
|
||||
#
|
||||
# ONE PAIRING PER FILE — swarm-otel ← authelia, and nothing else. Deleting
|
||||
# this leaves a swarm whose collector is not a client authelia has ever heard
|
||||
# of, so no token it presents is ever accepted and authelia mints no secret
|
||||
# for the publisher to send on.
|
||||
#
|
||||
# ⚠️ Gated on authelia being HERE, and deliberately NOT on this host running
|
||||
# the collector. A client is a row in THIS host's provider config, so it can
|
||||
# only be declared where that config is rendered — and ./swarm-otel.nix's
|
||||
# whole `config` block hangs off `deploy.swarm-otel.enable`, so a swarm with
|
||||
# the collector and authelia on different hosts registered the client
|
||||
# nowhere at all. Same bug, same fix, as ./glue-grafana-oidc-client.nix one
|
||||
# module over — read that file's own comment for the property this one
|
||||
# shares with it.
|
||||
#
|
||||
# ⚠️ Registered whether or not the swarm has a collector, for the same
|
||||
# reason as Grafana's: nothing in `swarm.*` records that one exists,
|
||||
# `deploy.swarm-otel.enable` only answers "does THIS host run it". The cost
|
||||
# is one unused client and one unused minted secret in a swarm with no
|
||||
# collector — the same trade ./swarm-otel.nix already made for this exact
|
||||
# client when it dropped the published-scrape-target guard on registering it.
|
||||
{
|
||||
lib,
|
||||
config,
|
||||
...
|
||||
}:
|
||||
let
|
||||
hyperhiveCfg = config.services.hyperhive;
|
||||
deployCfg = hyperhiveCfg.deploy;
|
||||
otelCfg = hyperhiveCfg.swarm.otel;
|
||||
in
|
||||
{
|
||||
config = lib.mkIf (hyperhiveCfg.enable && deployCfg.authelia.enable) {
|
||||
# One declaration, two readers: `clientId` and `audience` are read-only
|
||||
# options ./swarm-otel.nix derives from the scrape/push targets it owns,
|
||||
# so this file states neither formula a second time.
|
||||
services.hyperhive.swarm.authelia.oidc.clients = [
|
||||
{
|
||||
id = otelCfg.clientId;
|
||||
description = "HyperHive swarm collector";
|
||||
kind = "machine";
|
||||
# Grants `authelia.bearer.authz`, without which the authz endpoint
|
||||
# refuses an otherwise valid token and blames the token rather than
|
||||
# the missing grant.
|
||||
bearerAuthz = true;
|
||||
audience = otelCfg.audience;
|
||||
# Stated rather than left on authelia's default, because the two
|
||||
# agreeing today is not the same as this being the required value:
|
||||
# authelia permits only basic / JWT methods for a confidential
|
||||
# client holding this scope, and enforces it in the startup
|
||||
# validator.
|
||||
tokenEndpointAuthMethod = "client_secret_basic";
|
||||
}
|
||||
];
|
||||
};
|
||||
}
|
||||
|
|
@ -23,6 +23,8 @@ let
|
|||
vlCfg = config.services.hyperhive.swarm.victorialogs;
|
||||
hyperhiveCfg = config.services.hyperhive;
|
||||
gatewayCfg = hyperhiveCfg.gateway;
|
||||
baoCfg = hyperhiveCfg.swarm.bao;
|
||||
baoDeploy = deployCfg.bao;
|
||||
swarmDomain = hyperhiveCfg.swarm.domain;
|
||||
|
||||
# Total on a null swarm domain for the same reason every sibling module is:
|
||||
|
|
@ -132,6 +134,13 @@ let
|
|||
# delivery unit on a later boot, and a secret that evaporates on reboot
|
||||
# turns a working scrape into an intermittent one.
|
||||
collectorSecretInContainer = "/var/lib/swarm-otel-oidc/${cfg.clientId}.secret";
|
||||
# The same file seen from the host, which is where the delivery unit
|
||||
# writes it — spelled once for the same reason ./swarm-grafana.nix spells
|
||||
# its own pair once: the unit that writes it and the option that names it
|
||||
# are a few hundred lines apart, and a collector reading a path nothing
|
||||
# writes is an export that fails with nothing in any log about the file.
|
||||
collectorHostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}";
|
||||
collectorHostSecretDir = builtins.dirOf collectorHostSecretPath;
|
||||
collectorCredentialId = "oidc-client-secret";
|
||||
# What the scrape config points at. ⚠️ This path and the `LoadCredential`
|
||||
# id below are one fact spelled twice by systemd's design — both derive from
|
||||
|
|
@ -206,11 +215,32 @@ let
|
|||
pushAuthenticator = name: "oauth2client/${name}";
|
||||
|
||||
# Whether this collector holds a credential — a property of the credential,
|
||||
# not of where any other service runs. Not an assertion: a collector on a
|
||||
# host of its own is a supported shape, and refusing to build it would make
|
||||
# this fix illegal where the bug bites hardest.
|
||||
# not of where any other service runs. Not an assertion: a collector that
|
||||
# pushes nowhere authenticated still receives from every hive, so refusing
|
||||
# to build one would make a supported shape illegal for want of a value
|
||||
# that only degrades what it can push.
|
||||
haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null;
|
||||
|
||||
# A reader of the store is defined by holding a certificate the store
|
||||
# accepts, never by standing next to it — the rule
|
||||
# ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical-
|
||||
# looking flag, this one stays outside `assertions`: the store-reading unit
|
||||
# below simply does not render without it, the same choice
|
||||
# ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for
|
||||
# their own optional readers, because a collector with no client identity is
|
||||
# `haveCollectorSecret = false` above, and that is already a supported,
|
||||
# merely degraded shape rather than a service with no way in at all.
|
||||
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
|
||||
|
||||
# Where the publisher on authelia's host leaves this client's secret —
|
||||
# composed from the same swarm-wide `clientId` the registration in
|
||||
# ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of
|
||||
# them behind. The `services` segment is `swarm-secret-client`'s
|
||||
# `path::Kind::Service`, the same prefix ./swarm-grafana.nix reads its own
|
||||
# client secret under — one write grant in ./swarm-bao.nix and one hive read
|
||||
# grant in `policy::render` already cover it.
|
||||
storeSecretPath = "secret/swarm/services/${cfg.clientId}/oidc/client";
|
||||
|
||||
# The operator-configured upstream, named once: the same exporter carries
|
||||
# every signal, so metrics and logs both reach it without a second
|
||||
# definition.
|
||||
|
|
@ -493,6 +523,21 @@ in
|
|||
after the change that caused it evaluated cleanly.
|
||||
'';
|
||||
};
|
||||
|
||||
audience = lib.mkOption {
|
||||
type = lib.types.listOf lib.types.str;
|
||||
readOnly = true;
|
||||
default = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences;
|
||||
description = ''
|
||||
Every audience this collector's OAuth2 client is permitted to
|
||||
present a token for — the scrape targets it reads with a credential,
|
||||
plus the stores it pushes to. Published read-only so
|
||||
./glue-swarm-otel-oidc-client.nix can register the client wherever
|
||||
authelia runs without restating the derivation: this option and
|
||||
that registration are the same fact seen from two hosts, and a
|
||||
second formula for it would be free to drift from this one.
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
# What stays above is what the collector IS to the swarm — the client it is
|
||||
|
|
@ -529,7 +574,10 @@ in
|
|||
# delivering says so in the store it stopped delivering to. That is
|
||||
# less circular than it sounds: the failure that matters here is a
|
||||
# single hive's receiver refusing pushes, not the process dying.
|
||||
services.hyperhive.swarm.otel.journaldUnits = [ "opentelemetry-collector" ];
|
||||
services.hyperhive.swarm.otel.journaldUnits = [
|
||||
"opentelemetry-collector"
|
||||
"swarm-bao-otel-oidc"
|
||||
];
|
||||
|
||||
# The metrics counterpart to the journal line above, closing the same
|
||||
# gap from the other side: the journal says the process is alive, these
|
||||
|
|
@ -584,109 +632,152 @@ in
|
|||
# this is how it authenticates *itself* to a service published behind
|
||||
# the gateway.
|
||||
#
|
||||
# Only where authelia is co-located. A client is a row in this host's
|
||||
# provider config, so declaring one against a remote provider would
|
||||
# render nothing while reading as done; a swarm whose authelia lives
|
||||
# elsewhere registers it there.
|
||||
#
|
||||
# ⚠️ NO LONGER conditional on a published scrape target. Authelia refuses
|
||||
# a bearer-authz client with no audience, which is what that guard was
|
||||
# for — and the push audiences below are unconditional, so there is now
|
||||
# always at least one. Keeping the old guard would have left a collector
|
||||
# that scrapes nothing pushing to the stores with no client to get a
|
||||
# token from.
|
||||
services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf deployCfg.authelia.enable [
|
||||
{
|
||||
id = cfg.clientId;
|
||||
description = "HyperHive swarm collector";
|
||||
kind = "machine";
|
||||
# Grants `authelia.bearer.authz`, without which the authz
|
||||
# endpoint refuses an otherwise valid token and blames the
|
||||
# token rather than the missing grant.
|
||||
bearerAuthz = true;
|
||||
# DERIVED from the targets rather than contributed alongside
|
||||
# them. A service declares a URL once and this is the
|
||||
# permission to reach it; two lists that had to agree would be
|
||||
# a drift to maintain, and the failure mode is the quiet one —
|
||||
# a target whose audience was forgotten authenticates against
|
||||
# nothing and looks like a broken scrape.
|
||||
#
|
||||
# Both directions land in one list because authelia has one: what
|
||||
# this collector may SCRAPE and what it may PUSH TO are the same
|
||||
# kind of permission, differing only in who initiates.
|
||||
audience = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences;
|
||||
# Stated rather than left on authelia's default, because the
|
||||
# two agreeing today is not the same as this being the
|
||||
# required value: authelia permits only basic / JWT methods
|
||||
# for a confidential client holding that scope, and enforces
|
||||
# it in the startup validator.
|
||||
tokenEndpointAuthMethod = "client_secret_basic";
|
||||
}
|
||||
];
|
||||
# Registering the client is NOT here any more: it has to happen on the
|
||||
# host that runs authelia, and this whole block is gated on the host that
|
||||
# runs the collector. ./glue-swarm-otel-oidc-client.nix is where it moved
|
||||
# to, the same split ./swarm-grafana.nix made for its own client.
|
||||
|
||||
# Deliver the collector's client secret from authelia's container into
|
||||
# this one. On the HOST because that is the only place both container
|
||||
# trees are addressable: they share this host's network namespace, which
|
||||
# makes them feel co-located, but their filesystem roots are separate.
|
||||
# THE delivery unit — one route, in every deployment. The secret authelia
|
||||
# minted arrives out of the swarm secret store, which the publisher on
|
||||
# authelia's host wrote it into, whether authelia is a network away or in
|
||||
# the container next door.
|
||||
#
|
||||
# ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container
|
||||
# refuses to start when a bind source is missing, and this secret does not
|
||||
# exist until authelia's first boot has minted it — so binding it would
|
||||
# make the collector wait on a file that waits on a container that starts
|
||||
# after it. On a fresh swarm that is a permanent stall presenting as
|
||||
# "metrics are broken", several layers from its cause.
|
||||
# The delivery unit below is the one thing here that may know where
|
||||
# authelia runs — it copies out of its container — so it is also what
|
||||
# names the file. `mkDefault`, so a deployment that delivers the secret
|
||||
# some other way just sets the option.
|
||||
services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable (
|
||||
lib.mkDefault collectorSecretInContainer
|
||||
);
|
||||
|
||||
systemd.services.swarm-otel-oidc-secret = lib.mkIf deployCfg.authelia.enable {
|
||||
description = "deliver the swarm collector's OIDC client secret from authelia";
|
||||
after = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ];
|
||||
requires = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ];
|
||||
# 🩸 A unit here used to copy the plaintext directly out of authelia's
|
||||
# host tree, reachable only because they share this host's network
|
||||
# namespace — which stopped working the moment authelia moved to another
|
||||
# host, which is the defect this whole change exists to fix. The ruling
|
||||
# that deleted it rather than gave it a remote sibling: the store exists
|
||||
# so a host holds ONE out-of-band secret — its client certificate — and
|
||||
# reads everything else with it. Recorded in docs/swarm/secrets.md.
|
||||
#
|
||||
# Shaped after ./glue-queue-agent-credential.nix: a cert login that fails
|
||||
# LOUDLY, since every state it fails on is one a retry fixes, then a read
|
||||
# that degrades QUIETLY, since no retry turns "no value there" into a
|
||||
# value.
|
||||
#
|
||||
# ⚠️ Renders only where `haveClientIdentity` holds, unlike
|
||||
# ./swarm-grafana.nix's equivalent unit. That module asserts the identity
|
||||
# because a Grafana with none has no way in at all; this collector without
|
||||
# one is `haveCollectorSecret = false` above — already a supported,
|
||||
# merely degraded shape, so the unit that would fetch a credential simply
|
||||
# does not exist rather than refusing the build for want of one.
|
||||
systemd.services.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity {
|
||||
description = "fetch the swarm collector's OIDC client secret from the swarm secret store";
|
||||
# Every one of these names a unit that exists only where the store runs.
|
||||
# `Requires=` on an absent unit fails the job outright, so the ordering
|
||||
# is conditional even though the read is not: off-host there is nothing
|
||||
# local to wait for, and the timeout below bounds the attempt instead.
|
||||
after = lib.optionals baoDeploy.enable [
|
||||
"swarm-bao-pki.service"
|
||||
"container@${baoCfg.machine}.service"
|
||||
];
|
||||
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
|
||||
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
|
||||
before = [ "container@${cfg.machine}.service" ];
|
||||
wantedBy = [ "container@${cfg.machine}.service" ];
|
||||
wantedBy = [
|
||||
"multi-user.target"
|
||||
"container@${cfg.machine}.service"
|
||||
];
|
||||
path = [
|
||||
baoDeploy.package
|
||||
pkgs.coreutils
|
||||
];
|
||||
# Sized for the race this loses, not for an unseal: `swarm-bao` comes up
|
||||
# seconds before this unit asks, and the cert-auth role it logs in
|
||||
# against is written seconds after, so a few short attempts cover it. An
|
||||
# hours-long window would be a bet on a sealed store, and the degrade
|
||||
# below is already correct for that.
|
||||
#
|
||||
# `StartLimit*` are `[Unit]` settings, so they go here and not in
|
||||
# `serviceConfig` — systemd ignores them under `[Service]`. The window
|
||||
# has to exceed `RestartSec × burst`.
|
||||
startLimitBurst = 4;
|
||||
startLimitIntervalSec = 300;
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
SyslogIdentifier = "swarm-otel-oidc-secret";
|
||||
# Longer than the bounded wait below, and that is the point:
|
||||
# `DefaultTimeoutStartSec` is 90s, so without this systemd kills
|
||||
# the unit before it can emit the message naming the file it was
|
||||
# waiting for — the failure then reads as a timeout with no cause.
|
||||
TimeoutStartSec = "180s";
|
||||
SyslogIdentifier = "swarm-bao-otel-oidc";
|
||||
# What actually bounds the read below. Stated here rather than left to
|
||||
# systemd's default, so the number a boot waits on is in the file that
|
||||
# waits.
|
||||
TimeoutStartSec = 30;
|
||||
Restart = "on-failure";
|
||||
RestartSec = 15;
|
||||
};
|
||||
environment = {
|
||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||
BAO_CLIENT_CERT = baoDeploy.clientCertFile;
|
||||
BAO_CLIENT_KEY = baoDeploy.clientKeyFile;
|
||||
}
|
||||
# Absent means the system trust store, which is what a deployment with a
|
||||
# real CA wants and what a self-signed one must not be left with.
|
||||
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
|
||||
BAO_CACERT = baoDeploy.serverCaFile;
|
||||
};
|
||||
path = [ pkgs.coreutils ];
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
|
||||
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.clientId}.secret"}
|
||||
dst=${lib.escapeShellArg "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}"}
|
||||
# `bao`'s own message is the only thing separating a missing value
|
||||
# from a refused identity from an unreachable host. This unit's
|
||||
# degraded mode is correct for all three, so it reports which one
|
||||
# rather than asserting all three in a sentence of ours.
|
||||
err="$(mktemp)"
|
||||
trap 'rm -f "$err"' EXIT
|
||||
|
||||
# authelia's container is up, but its first-boot generator may
|
||||
# still be minting. Bounded wait, then fail: skipping silently
|
||||
# produces a collector whose scrape gets a 401 forever, which is
|
||||
# the failure this whole design exists to make impossible.
|
||||
for _ in $(seq 1 60); do
|
||||
[ -s "$src" ] && break
|
||||
sleep 2
|
||||
done
|
||||
if [ ! -s "$src" ]; then
|
||||
echo "authelia has not minted $src after 120s" >&2
|
||||
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
|
||||
# variables above only decide which certificate the TLS handshake
|
||||
# presents; without a token `bao` asks its token helper instead, and
|
||||
# that is a `sh` this unit's `path` does not carry. `-token-only`
|
||||
# answers on stdout and skips the helper on both sides.
|
||||
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
|
||||
echo "could not log in to swarm-bao with this host's certificate; leaving the collector's OIDC client secret as it is." >&2
|
||||
if [ -s "$err" ]; then
|
||||
cat "$err" >&2
|
||||
else
|
||||
echo "bao failed without writing a diagnostic." >&2
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
export BAO_TOKEN
|
||||
|
||||
# root-owned 0400. The collector runs under `DynamicUser`, so
|
||||
# there is no uid to give it to — `LoadCredential` reads this as
|
||||
# root before the sandbox exists and re-exposes it to whichever
|
||||
# uid the unit got.
|
||||
install -D -m 0400 -o root -g root "$src" "$dst"
|
||||
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
|
||||
echo "swarm-bao did not return ${storeSecretPath}; the collector has no OIDC client secret yet." >&2
|
||||
if [ -s "$err" ]; then
|
||||
cat "$err" >&2
|
||||
else
|
||||
echo "bao failed without writing a diagnostic." >&2
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ -z "$secret" ]; then
|
||||
echo "swarm-bao returned an empty ${storeSecretPath}; leaving the file as it is." >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# root-owned 0400, written with a shell builtin and never handed to a
|
||||
# program: `printf` is bash's own, so the plaintext never becomes an
|
||||
# argument in /proc the way `install <<<"$secret"` or an `echo` from
|
||||
# `path` would. The collector runs under `DynamicUser`, so there is
|
||||
# no uid to give this to — `LoadCredential` reads it as root before
|
||||
# the sandbox exists and re-exposes it to whichever uid the unit got.
|
||||
install -d -m 0755 ${lib.escapeShellArg collectorHostSecretDir}
|
||||
umask 077
|
||||
printf '%s\n' "$secret" > ${lib.escapeShellArg collectorHostSecretPath}
|
||||
chown root:root ${lib.escapeShellArg collectorHostSecretPath}
|
||||
chmod 0400 ${lib.escapeShellArg collectorHostSecretPath}
|
||||
'';
|
||||
};
|
||||
|
||||
# Where the delivery unit above lands the secret. `mkDefault`, so a
|
||||
# deployment delivering it some other way just sets the option — and
|
||||
# `mkIf haveClientIdentity` so a host with no store identity is left with
|
||||
# `clientSecretFile == null`, the already-supported degrade rather than a
|
||||
# path nothing ever writes.
|
||||
services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf haveClientIdentity (
|
||||
lib.mkDefault collectorSecretInContainer
|
||||
);
|
||||
|
||||
# The CA bind source is written at runtime by a host unit, so the
|
||||
# container has to start after it — otherwise nspawn sets up a mount
|
||||
# over a file that does not exist yet.
|
||||
|
|
|
|||
|
|
@ -64,10 +64,13 @@ let
|
|||
#
|
||||
# The ids come from `swarm.*`, which is identical on every host — that is what
|
||||
# lets this host name a service's client while running none of them, and it is
|
||||
# the same read the service's own module registers the client with. A swarm
|
||||
# that runs no Grafana mints no secret for it, so its entry skips below rather
|
||||
# than needing a condition here.
|
||||
serviceClientIds = [ hyperhiveCfg.swarm.grafana.oidc.clientId ];
|
||||
# the same read each service's own module registers its client with. A swarm
|
||||
# that runs no Grafana, or no collector, mints no secret for the one it lacks,
|
||||
# so its entry skips below rather than needing a condition here.
|
||||
serviceClientIds = [
|
||||
hyperhiveCfg.swarm.grafana.oidc.clientId
|
||||
hyperhiveCfg.swarm.otel.clientId
|
||||
];
|
||||
in
|
||||
{
|
||||
options.services.hyperhive.deploy.swarm-secret-publisher = {
|
||||
|
|
@ -92,10 +95,11 @@ in
|
|||
identity's job (see `baoClientCertFile`), not this option's.
|
||||
|
||||
Turning it off leaves every hive but this one without its agents'
|
||||
credential, and the swarm's Grafana without any login at all — its
|
||||
secret has exactly one route and this is the producer's end of it. So
|
||||
the honest reason to set it false is a deployment delivering those
|
||||
secrets by some other mechanism it owns.
|
||||
credential, the swarm's Grafana without any login at all, and its
|
||||
collector pushing unauthenticated — each secret has exactly one route
|
||||
and this is the producer's end of it. So the honest reason to set it
|
||||
false is a deployment delivering those secrets by some other
|
||||
mechanism it owns.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -504,6 +504,35 @@ let
|
|||
otelSettings =
|
||||
machine: machine.containers.swarm-otel.config.services.opentelemetry-collector.settings;
|
||||
|
||||
# The collector beside authelia, reading its own OIDC secret out of the
|
||||
# store like every other collector — the cert pair here is not scenery, it
|
||||
# is the arm that would catch the deleted co-located copy unit coming back.
|
||||
otelBaoWithAuthelia = hive {
|
||||
deploy.swarm-otel.enable = true;
|
||||
deploy.authelia.enable = true;
|
||||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||
};
|
||||
# The same collector with the IdP on ANOTHER host and a store leaf placed by
|
||||
# hand. Identical to the fixture above in everything the delivery path
|
||||
# reads, which is the point.
|
||||
otelBaoRemoteAuthelia = hive {
|
||||
deploy.swarm-otel.enable = true;
|
||||
swarm.authelia.url = "https://auth.example.invalid";
|
||||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||
};
|
||||
# A collector holding no store identity at all. Unlike Grafana's mirror
|
||||
# image, this is not a refused shape: the collector still receives every
|
||||
# hive's telemetry with nothing to push authenticated with, which is the
|
||||
# already-supported degrade `haveCollectorSecret` names above the module's
|
||||
# `let`. What this fixture is for is checking the reading unit itself does
|
||||
# not render, rather than rendering with an env var nothing filled in.
|
||||
otelNoIdentity = hive {
|
||||
deploy.swarm-otel.enable = true;
|
||||
swarm.authelia.url = "https://auth.example.invalid";
|
||||
};
|
||||
|
||||
# authelia somewhere else, the credential delivered by hand. Whether this
|
||||
# collector authenticates must follow the credential, never another
|
||||
# service's placement.
|
||||
|
|
@ -1031,6 +1060,71 @@ let
|
|||
lib.elem "swarm-grafana" (clients secretPublisherHere)
|
||||
&& !(lib.elem "swarm-grafana" (clients grafanaRemoteAuthelia));
|
||||
}
|
||||
{
|
||||
# The collector's half of the same defect and the same fix: this used to
|
||||
# be gated on `deploy.swarm-otel.enable`, so a split deployment
|
||||
# registered the client nowhere and authelia minted nothing to publish.
|
||||
name = "the swarm's collector client is registered wherever authelia runs";
|
||||
ok =
|
||||
let
|
||||
clients = m: map (c: c.id) m.services.hyperhive.swarm.authelia.oidc.clients;
|
||||
in
|
||||
lib.elem "swarm-collector" (clients secretPublisherHere)
|
||||
&& !(lib.elem "swarm-collector" (clients otelBaoRemoteAuthelia));
|
||||
}
|
||||
{
|
||||
# 🩸 The arm that guards the ruling this slice landed under, the
|
||||
# collector's half of ./swarm-grafana.nix's own. There is ONE delivery
|
||||
# route: the store reader, on every host that runs the collector and
|
||||
# holds a store identity. The negative names the deleted unit rather
|
||||
# than a generic absence, because the way this regresses is someone
|
||||
# re-adding the co-located copy as an optimisation.
|
||||
name = "the collector's OIDC secret has exactly one delivery unit, the store reader, in both topologies";
|
||||
ok =
|
||||
let
|
||||
local = otelBaoWithAuthelia.systemd.services;
|
||||
remote = otelBaoRemoteAuthelia.systemd.services;
|
||||
in
|
||||
local ? swarm-bao-otel-oidc
|
||||
&& remote ? swarm-bao-otel-oidc
|
||||
&& !(local ? swarm-otel-oidc-secret)
|
||||
&& !(remote ? swarm-otel-oidc-secret);
|
||||
}
|
||||
{
|
||||
# Same 403-not-a-miss reason as grafana's arm above: the reader's grant
|
||||
# covers the `services` prefix, so a path outside it is refused rather
|
||||
# than empty, however correct it reads.
|
||||
name = "the collector's OIDC secret is read from the prefix the publisher writes";
|
||||
ok =
|
||||
let
|
||||
s = otelBaoRemoteAuthelia.systemd.services.swarm-bao-otel-oidc.script;
|
||||
in
|
||||
lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" s
|
||||
&& !(lib.hasInfix "secret/swarm/hives/" s);
|
||||
}
|
||||
{
|
||||
# Both ends of a wire nothing at eval time carries end to end: the
|
||||
# publisher on authelia's host writes the path the reader on the
|
||||
# collector's host reads, and the two files agree only because both
|
||||
# compose it from the same swarm-wide client id. `secretPublisherHere`
|
||||
# already grew this client when `serviceClientIds` did.
|
||||
name = "the publisher writes the swarm service path the collector reads";
|
||||
ok = lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" (
|
||||
secretPublisherHere.systemd.services.swarm-secret-publish.script
|
||||
);
|
||||
}
|
||||
{
|
||||
# The collector's non-assertion, the deliberate mirror of Grafana's
|
||||
# assertion two cases up: a host with no store identity is a supported,
|
||||
# merely degraded shape here, so the reading unit simply does not exist
|
||||
# rather than refusing the build. `haveCollectorSecret` is what the
|
||||
# degrade already reads, unchanged by this slice.
|
||||
name = "a collector with no store identity renders no reading unit, and is not refused";
|
||||
ok =
|
||||
!(otelNoIdentity.systemd.services ? swarm-bao-otel-oidc)
|
||||
&& otelNoIdentity.services.hyperhive.deploy.swarm-otel.clientSecretFile == null
|
||||
&& !(lib.any (a: !a.assertion) otelNoIdentity.assertions);
|
||||
}
|
||||
{
|
||||
# Reads the daemon's rendered unit, not the options: the queue address
|
||||
# arrives as an env var whose whole attrset is guarded on `natsUrl`, and
|
||||
|
|
|
|||
Loading…
Reference in a new issue