vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm-otel: deliver the OIDC client secret through the secret store

Browse source 
The swarm collector's OIDC client secret only existed where authelia
did: `swarm-otel-oidc-secret.service` copied the minted plaintext out
of authelia's container tree, reachable only because the two share a
host's network namespace. A swarm that placed authelia elsewhere
delivered nothing, and the option's own description said so —
"a deployment that places authelia elsewhere points this at a file it
delivers itself." Same gap as #3853 and #4234, and this is the
swarm-otel twin of #4234's fix for Grafana.

Mirrors PR #4361 (Grafana) almost exactly:

- `swarm-bao-otel-oidc.service` reads
  `swarm/services/<client-id>/oidc/client` out of the store, in every
  deployment, replacing the co-located copy unit outright — one
  delivery route, not two, per the ruling that landed under #4234.
- Client registration moved out of `swarm-otel.nix`'s own `config`
  block (gated on this host running the collector) into
  `glue-swarm-otel-oidc-client.nix` (gated on this host running
  authelia), the same split `glue-grafana-oidc-client.nix` made. It
  was broken the same way: a split deployment registered the client
  nowhere at all, so authelia never minted a secret for the publisher
  to send on.
- The publisher's `services` prefix (write grant in `swarm-bao.nix`,
  hive read grant in `policy::render`) already covers any service's
  path — nothing to add there. `swarm-secret-publisher.nix` only grew
  `serviceClientIds` by one entry.

One judgement call, stated rather than buried: the store-reading unit
renders only where this host holds a client identity
(`deploy.bao.clientCertFile`/`clientKeyFile`), rather than asserting
it the way `swarm-grafana.nix` does. Grafana's local login form is
disabled unconditionally, so a Grafana with no OIDC secret has no way
in at all — that earns a hard refusal. This collector without a
credential still receives every hive's telemetry; only its own pushes
to the stores go out unauthenticated and get refused there, an
already-supported degrade the module's own `haveCollectorSecret` flag
named before this change. So the reading unit follows the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use
for their own optional readers: no unit when the identity is absent,
not a build refusal.

Fixtures mirror #4361's: `otelBaoWithAuthelia`/`otelBaoRemoteAuthelia`
are the positive pair (co-located and split, both reading through the
store), `otelNoIdentity` is the negative — no reading unit, no
assertion firing, `clientSecretFile` left null.

Refs #4258
This commit is contained in:
atlas 2026-09-13 20:23:39 +02:00 • committed by mara
commit 0ff5c8110b
7 changed files with 409 additions and 148 deletions
Download patch file Download diff file Expand all files Collapse all files

58
docs/swarm/secrets.md
View file

@ -42,7 +42,7 @@ neither is a renaming of the other.
## Swarm-level — one of each per swarm ## Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere | | secret | generated by | lives at | hive elsewhere |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public | | swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below | | swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives | | swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
@ -51,7 +51,7 @@ neither is a renaming of the other.
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` | | OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity is left with `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
@ -72,27 +72,27 @@ an all-local swarm it reads authelia's file where it lies and no second copy is
made. On any other topology it's an ordinary "operator provides the file" made. On any other topology it's an ordinary "operator provides the file"
case — see `services.hyperhive.otel.clientSecretFile`. case — see `services.hyperhive.otel.clientSecretFile`.
The **swarm's** collector does need one. It runs in a container, so The **swarm's** collector does need one, and its delivery route is the fourth
`swarm-otel-oidc-secret.service` places its copy, landing at store reader — the same shape as Grafana's, described in full below. In short:
`/var/lib/swarm-otel-oidc/<client-id>.secret` — the same shape as the forge and `swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
homeserver rows below, and for the same reason: the container that mints the out of the store on every host that runs the collector and holds a store
secret isn't the container that reads it. identity, whether authelia is a network away or in the container next door.
The client it reads for is registered unconditionally, the same call
`glue-grafana-oidc-client.nix` makes for Grafana's: authelia refuses a
bearer-authz client with no audience, and the push audiences supply one even
for a collector with nothing published to scrape.
The copy is made whenever authelia is enabled **on this host**, whatever this ⚠️ **A collector with no store identity is not refused.** Unlike Grafana,
collector scrapes. The client it delivers for is registered unconditionally: where an absent OIDC block is a container with no way in at all, a collector
authelia refuses a bearer-authz client with no audience, and the push audiences with nothing to authenticate with still receives every hive's telemetry —
supply one even for a collector with nothing published to scrape. Like the only its own pushes to the stores go out unauthenticated and get refused
hive-level delivery units below, this one waits a bounded 120s for authelia's there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
mint and then fails rather than skipping, so a secret that never arrives takes `deploy.bao.clientCertFile` / `clientKeyFile`, the same choice
the collector's container start with it instead of surfacing later as a 401. `glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for
their own optional readers, rather than the hard assertion `swarm-grafana.nix`
⚠️ **Don't read that delivery unit as the only way this collector is fed.** uses for its non-optional one. The secret can still be delivered by hand, at
Whether it authenticates follows the _credential_, never another service's `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a collector this
placement: a swarm collector may run on a host that holds neither store and no unit never reaches.
authelia, and then the secret is an ordinary operator-provided file named in
`services.hyperhive.deploy.swarm-otel.clientSecretFile` — the same shape as the hive
collector's row above. The copy unit is the convenience for the co-located case,
not the definition of the case.
### Minting the queue's callout nkeys ### Minting the queue's callout nkeys
@ -212,6 +212,18 @@ nothing said anything. Registration is separate and stays where authelia is
so it's declared on the host running authelia, which isn't necessarily the host so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. running Grafana.
The **fourth reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it simply
doesn't render the reading unit without them, the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for
their own optional reads.
A service's secret is one value for the whole swarm rather than one per hive, so A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs whole. That's because nothing in a swarm's configuration records which hive runs
@ -220,7 +232,7 @@ there is nothing to scope the grant to. Every hive can therefore read every swar
service's client secret; that's the cost of a shape where the service's host has service's client secret; that's the cost of a shape where the service's host has
no store identity of its own to present, and it's stated in no store identity of its own to present, and it's stated in
`swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves `swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves
as it does for the other two readers: a store that says "not here" leaves the as it does for the other readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in. client secret to stand in.

21
docs/trust-boundary/security.md
View file

@ -66,16 +66,17 @@ _own_ entry under the hive namespace, which names the hive asking and so widens
nothing between them, and every **swarm service's** OIDC client secret. nothing between them, and every **swarm service's** OIDC client secret.
That third prefix has the same shape of reason as the first, and the same honest That third prefix has the same shape of reason as the first, and the same honest
cost. A swarm service (Grafana is the one there today) registers **one** client cost. A swarm service (Grafana and the swarm collector are the two there today)
for the whole swarm, so its credential's path names the service and never the registers **one** client for the whole swarm, so its credential's path names
host — and which hive runs a given service is a `deploy.*` fact, per-host by the service and never the host — and which hive runs a given service is a
definition, so nothing swarm-wide exists to scope the grant to. The host running `deploy.*` fact, per-host by definition, so nothing swarm-wide exists to scope
such a service has no store identity of its own either; it reads with the the grant to. The host running such a service has no store identity of its own
certificate of the hive it is. **Any hive can therefore read any swarm service's either; it reads with the certificate of the hive it is. **Any hive can
client secret**, which is worth what it buys: every swarm service gets its therefore read any swarm service's client secret**, which is worth what it
credential the same way from anywhere, instead of only where the identity buys: every swarm service gets its credential the same way from anywhere,
provider happens to sit. Giving such a service its own store identity is what instead of only where the identity provider happens to sit. Giving such a
would remove this rather than re-scope it. service its own store identity is what would remove this rather than re-scope
it.
**A tracked follow-up** narrows this, with the two candidate directions: scope **A tracked follow-up** narrows this, with the two candidate directions: scope
the grant per hive (and pay for the re-emission), or give each agent container the grant per hive (and pay for the re-emission), or give each agent container

1
nix/host-modules/default.nix
View file

@ -29,6 +29,7 @@
./glue-matrix-bao-token.nix ./glue-matrix-bao-token.nix
./glue-queue-agent-credential.nix ./glue-queue-agent-credential.nix
./glue-secret-publisher-bao-identity.nix ./glue-secret-publisher-bao-identity.nix
./glue-swarm-otel-oidc-client.nix
./swarm-authelia.nix ./swarm-authelia.nix
./swarm-bao.nix ./swarm-bao.nix
./swarm-ca.nix ./swarm-ca.nix

58
nix/host-modules/glue-swarm-otel-oidc-client.nix Normal file
View file

@ -0,0 +1,58 @@
# Glue: register the swarm's collector as an OIDC client wherever authelia
# runs.
#
# ONE PAIRING PER FILE — swarm-otel ← authelia, and nothing else. Deleting
# this leaves a swarm whose collector is not a client authelia has ever heard
# of, so no token it presents is ever accepted and authelia mints no secret
# for the publisher to send on.
#
# ⚠️ Gated on authelia being HERE, and deliberately NOT on this host running
# the collector. A client is a row in THIS host's provider config, so it can
# only be declared where that config is rendered — and ./swarm-otel.nix's
# whole `config` block hangs off `deploy.swarm-otel.enable`, so a swarm with
# the collector and authelia on different hosts registered the client
# nowhere at all. Same bug, same fix, as ./glue-grafana-oidc-client.nix one
# module over — read that file's own comment for the property this one
# shares with it.
#
# ⚠️ Registered whether or not the swarm has a collector, for the same
# reason as Grafana's: nothing in `swarm.*` records that one exists,
# `deploy.swarm-otel.enable` only answers "does THIS host run it". The cost
# is one unused client and one unused minted secret in a swarm with no
# collector — the same trade ./swarm-otel.nix already made for this exact
# client when it dropped the published-scrape-target guard on registering it.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
otelCfg = hyperhiveCfg.swarm.otel;
in
{
config = lib.mkIf (hyperhiveCfg.enable && deployCfg.authelia.enable) {
# One declaration, two readers: `clientId` and `audience` are read-only
# options ./swarm-otel.nix derives from the scrape/push targets it owns,
# so this file states neither formula a second time.
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = otelCfg.clientId;
description = "HyperHive swarm collector";
kind = "machine";
# Grants `authelia.bearer.authz`, without which the authz endpoint
# refuses an otherwise valid token and blames the token rather than
# the missing grant.
bearerAuthz = true;
audience = otelCfg.audience;
# Stated rather than left on authelia's default, because the two
# agreeing today is not the same as this being the required value:
# authelia permits only basic / JWT methods for a confidential
# client holding this scope, and enforces it in the startup
# validator.
tokenEndpointAuthMethod = "client_secret_basic";
}
];
};
}

269
nix/host-modules/swarm-otel.nix
View file

@ -23,6 +23,8 @@ let
vlCfg = config.services.hyperhive.swarm.victorialogs; vlCfg = config.services.hyperhive.swarm.victorialogs;
hyperhiveCfg = config.services.hyperhive; hyperhiveCfg = config.services.hyperhive;
gatewayCfg = hyperhiveCfg.gateway; gatewayCfg = hyperhiveCfg.gateway;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
swarmDomain = hyperhiveCfg.swarm.domain; swarmDomain = hyperhiveCfg.swarm.domain;
# Total on a null swarm domain for the same reason every sibling module is: # Total on a null swarm domain for the same reason every sibling module is:
@ -132,6 +134,13 @@ let
# delivery unit on a later boot, and a secret that evaporates on reboot # delivery unit on a later boot, and a secret that evaporates on reboot
# turns a working scrape into an intermittent one. # turns a working scrape into an intermittent one.
collectorSecretInContainer = "/var/lib/swarm-otel-oidc/${cfg.clientId}.secret"; collectorSecretInContainer = "/var/lib/swarm-otel-oidc/${cfg.clientId}.secret";
# The same file seen from the host, which is where the delivery unit
# writes it — spelled once for the same reason ./swarm-grafana.nix spells
# its own pair once: the unit that writes it and the option that names it
# are a few hundred lines apart, and a collector reading a path nothing
# writes is an export that fails with nothing in any log about the file.
collectorHostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}";
collectorHostSecretDir = builtins.dirOf collectorHostSecretPath;
collectorCredentialId = "oidc-client-secret"; collectorCredentialId = "oidc-client-secret";
# What the scrape config points at. ⚠️ This path and the `LoadCredential` # What the scrape config points at. ⚠️ This path and the `LoadCredential`
# id below are one fact spelled twice by systemd's design — both derive from # id below are one fact spelled twice by systemd's design — both derive from
@ -206,11 +215,32 @@ let
pushAuthenticator = name: "oauth2client/${name}"; pushAuthenticator = name: "oauth2client/${name}";
# Whether this collector holds a credential — a property of the credential, # Whether this collector holds a credential — a property of the credential,
# not of where any other service runs. Not an assertion: a collector on a # not of where any other service runs. Not an assertion: a collector that
# host of its own is a supported shape, and refusing to build it would make # pushes nowhere authenticated still receives from every hive, so refusing
# this fix illegal where the bug bites hardest. # to build one would make a supported shape illegal for want of a value
# that only degrades what it can push.
haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null; haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null;
# A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical-
# looking flag, this one stays outside `assertions`: the store-reading unit
# below simply does not render without it, the same choice
# ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for
# their own optional readers, because a collector with no client identity is
# `haveCollectorSecret = false` above, and that is already a supported,
# merely degraded shape rather than a service with no way in at all.
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Where the publisher on authelia's host leaves this client's secret —
# composed from the same swarm-wide `clientId` the registration in
# ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of
# them behind. The `services` segment is `swarm-secret-client`'s
# `path::Kind::Service`, the same prefix ./swarm-grafana.nix reads its own
# client secret under — one write grant in ./swarm-bao.nix and one hive read
# grant in `policy::render` already cover it.
storeSecretPath = "secret/swarm/services/${cfg.clientId}/oidc/client";
# The operator-configured upstream, named once: the same exporter carries # The operator-configured upstream, named once: the same exporter carries
# every signal, so metrics and logs both reach it without a second # every signal, so metrics and logs both reach it without a second
# definition. # definition.
@ -493,6 +523,21 @@ in
after the change that caused it evaluated cleanly. after the change that caused it evaluated cleanly.
''; '';
}; };
audience = lib.mkOption {
type = lib.types.listOf lib.types.str;
readOnly = true;
default = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences;
description = ''
Every audience this collector's OAuth2 client is permitted to
present a token for — the scrape targets it reads with a credential,
plus the stores it pushes to. Published read-only so
./glue-swarm-otel-oidc-client.nix can register the client wherever
authelia runs without restating the derivation: this option and
that registration are the same fact seen from two hosts, and a
second formula for it would be free to drift from this one.
'';
};
}; };
# What stays above is what the collector IS to the swarm — the client it is # What stays above is what the collector IS to the swarm — the client it is
@ -529,7 +574,10 @@ in
# delivering says so in the store it stopped delivering to. That is # delivering says so in the store it stopped delivering to. That is
# less circular than it sounds: the failure that matters here is a # less circular than it sounds: the failure that matters here is a
# single hive's receiver refusing pushes, not the process dying. # single hive's receiver refusing pushes, not the process dying.
services.hyperhive.swarm.otel.journaldUnits = [ "opentelemetry-collector" ]; services.hyperhive.swarm.otel.journaldUnits = [
"opentelemetry-collector"
"swarm-bao-otel-oidc"
];
# The metrics counterpart to the journal line above, closing the same # The metrics counterpart to the journal line above, closing the same
# gap from the other side: the journal says the process is alive, these # gap from the other side: the journal says the process is alive, these
@ -584,109 +632,152 @@ in
# this is how it authenticates *itself* to a service published behind # this is how it authenticates *itself* to a service published behind
# the gateway. # the gateway.
# #
# Only where authelia is co-located. A client is a row in this host's # Registering the client is NOT here any more: it has to happen on the
# provider config, so declaring one against a remote provider would # host that runs authelia, and this whole block is gated on the host that
# render nothing while reading as done; a swarm whose authelia lives # runs the collector. ./glue-swarm-otel-oidc-client.nix is where it moved
# elsewhere registers it there. # to, the same split ./swarm-grafana.nix made for its own client.
# THE delivery unit — one route, in every deployment. The secret authelia
# minted arrives out of the swarm secret store, which the publisher on
# authelia's host wrote it into, whether authelia is a network away or in
# the container next door.
# #
# ⚠️ NO LONGER conditional on a published scrape target. Authelia refuses # 🩸 A unit here used to copy the plaintext directly out of authelia's
# a bearer-authz client with no audience, which is what that guard was # host tree, reachable only because they share this host's network
# for — and the push audiences below are unconditional, so there is now # namespace — which stopped working the moment authelia moved to another
# always at least one. Keeping the old guard would have left a collector # host, which is the defect this whole change exists to fix. The ruling
# that scrapes nothing pushing to the stores with no client to get a # that deleted it rather than gave it a remote sibling: the store exists
# token from. # so a host holds ONE out-of-band secret — its client certificate — and
services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf deployCfg.authelia.enable [ # reads everything else with it. Recorded in docs/swarm/secrets.md.
{
id = cfg.clientId;
description = "HyperHive swarm collector";
kind = "machine";
# Grants `authelia.bearer.authz`, without which the authz
# endpoint refuses an otherwise valid token and blames the
# token rather than the missing grant.
bearerAuthz = true;
# DERIVED from the targets rather than contributed alongside
# them. A service declares a URL once and this is the
# permission to reach it; two lists that had to agree would be
# a drift to maintain, and the failure mode is the quiet one —
# a target whose audience was forgotten authenticates against
# nothing and looks like a broken scrape.
# #
# Both directions land in one list because authelia has one: what # Shaped after ./glue-queue-agent-credential.nix: a cert login that fails
# this collector may SCRAPE and what it may PUSH TO are the same # LOUDLY, since every state it fails on is one a retry fixes, then a read
# kind of permission, differing only in who initiates. # that degrades QUIETLY, since no retry turns "no value there" into a
audience = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences; # value.
# Stated rather than left on authelia's default, because the #
# two agreeing today is not the same as this being the # ⚠️ Renders only where `haveClientIdentity` holds, unlike
# required value: authelia permits only basic / JWT methods # ./swarm-grafana.nix's equivalent unit. That module asserts the identity
# for a confidential client holding that scope, and enforces # because a Grafana with none has no way in at all; this collector without
# it in the startup validator. # one is `haveCollectorSecret = false` above — already a supported,
tokenEndpointAuthMethod = "client_secret_basic"; # merely degraded shape, so the unit that would fetch a credential simply
} # does not exist rather than refusing the build for want of one.
systemd.services.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity {
description = "fetch the swarm collector's OIDC client secret from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering
# is conditional even though the read is not: off-host there is nothing
# local to wait for, and the timeout below bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
]; ];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
# Deliver the collector's client secret from authelia's container into requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
# this one. On the HOST because that is the only place both container
# trees are addressable: they share this host's network namespace, which
# makes them feel co-located, but their filesystem roots are separate.
#
# ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container
# refuses to start when a bind source is missing, and this secret does not
# exist until authelia's first boot has minted it — so binding it would
# make the collector wait on a file that waits on a container that starts
# after it. On a fresh swarm that is a permanent stall presenting as
# "metrics are broken", several layers from its cause.
# The delivery unit below is the one thing here that may know where
# authelia runs — it copies out of its container — so it is also what
# names the file. `mkDefault`, so a deployment that delivers the secret
# some other way just sets the option.
services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable (
lib.mkDefault collectorSecretInContainer
);
systemd.services.swarm-otel-oidc-secret = lib.mkIf deployCfg.authelia.enable {
description = "deliver the swarm collector's OIDC client secret from authelia";
after = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ];
requires = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ];
before = [ "container@${cfg.machine}.service" ]; before = [ "container@${cfg.machine}.service" ];
wantedBy = [ "container@${cfg.machine}.service" ]; wantedBy = [
"multi-user.target"
"container@${cfg.machine}.service"
];
path = [
baoDeploy.package
pkgs.coreutils
];
# Sized for the race this loses, not for an unseal: `swarm-bao` comes up
# seconds before this unit asks, and the cert-auth role it logs in
# against is written seconds after, so a few short attempts cover it. An
# hours-long window would be a bet on a sealed store, and the degrade
# below is already correct for that.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec × burst`.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = { serviceConfig = {
Type = "oneshot"; Type = "oneshot";
RemainAfterExit = true; RemainAfterExit = true;
SyslogIdentifier = "swarm-otel-oidc-secret"; SyslogIdentifier = "swarm-bao-otel-oidc";
# Longer than the bounded wait below, and that is the point: # What actually bounds the read below. Stated here rather than left to
# `DefaultTimeoutStartSec` is 90s, so without this systemd kills # systemd's default, so the number a boot waits on is in the file that
# the unit before it can emit the message naming the file it was # waits.
# waiting for — the failure then reads as a timeout with no cause. TimeoutStartSec = 30;
TimeoutStartSec = "180s"; Restart = "on-failure";
RestartSec = 15;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.clientCertFile;
BAO_CLIENT_KEY = baoDeploy.clientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
}; };
path = [ pkgs.coreutils ];
script = '' script = ''
set -euo pipefail set -euo pipefail
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.clientId}.secret"} # `bao`'s own message is the only thing separating a missing value
dst=${lib.escapeShellArg "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}"} # from a refused identity from an unreachable host. This unit's
# degraded mode is correct for all three, so it reports which one
# rather than asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# authelia's container is up, but its first-boot generator may # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# still be minting. Bounded wait, then fail: skipping silently # variables above only decide which certificate the TLS handshake
# produces a collector whose scrape gets a 401 forever, which is # presents; without a token `bao` asks its token helper instead, and
# the failure this whole design exists to make impossible. # that is a `sh` this unit's `path` does not carry. `-token-only`
for _ in $(seq 1 60); do # answers on stdout and skips the helper on both sides.
[ -s "$src" ] && break if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
sleep 2 echo "could not log in to swarm-bao with this host's certificate; leaving the collector's OIDC client secret as it is." >&2
done if [ -s "$err" ]; then
if [ ! -s "$src" ]; then cat "$err" >&2
echo "authelia has not minted $src after 120s" >&2 else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1 exit 1
fi fi
export BAO_TOKEN
# root-owned 0400. The collector runs under `DynamicUser`, so if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
# there is no uid to give it to — `LoadCredential` reads this as echo "swarm-bao did not return ${storeSecretPath}; the collector has no OIDC client secret yet." >&2
# root before the sandbox exists and re-exposes it to whichever if [ -s "$err" ]; then
# uid the unit got. cat "$err" >&2
install -D -m 0400 -o root -g root "$src" "$dst" else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; leaving the file as it is." >&2
exit 0
fi
# root-owned 0400, written with a shell builtin and never handed to a
# program: `printf` is bash's own, so the plaintext never becomes an
# argument in /proc the way `install <<<"$secret"` or an `echo` from
# `path` would. The collector runs under `DynamicUser`, so there is
# no uid to give this to — `LoadCredential` reads it as root before
# the sandbox exists and re-exposes it to whichever uid the unit got.
install -d -m 0755 ${lib.escapeShellArg collectorHostSecretDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg collectorHostSecretPath}
chown root:root ${lib.escapeShellArg collectorHostSecretPath}
chmod 0400 ${lib.escapeShellArg collectorHostSecretPath}
''; '';
}; };
# Where the delivery unit above lands the secret. `mkDefault`, so a
# deployment delivering it some other way just sets the option — and
# `mkIf haveClientIdentity` so a host with no store identity is left with
# `clientSecretFile == null`, the already-supported degrade rather than a
# path nothing ever writes.
services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf haveClientIdentity (
lib.mkDefault collectorSecretInContainer
);
# The CA bind source is written at runtime by a host unit, so the # The CA bind source is written at runtime by a host unit, so the
# container has to start after it — otherwise nspawn sets up a mount # container has to start after it — otherwise nspawn sets up a mount
# over a file that does not exist yet. # over a file that does not exist yet.

20
nix/host-modules/swarm-secret-publisher.nix
View file

@ -64,10 +64,13 @@ let
# #
# The ids come from `swarm.*`, which is identical on every host — that is what # The ids come from `swarm.*`, which is identical on every host — that is what
# lets this host name a service's client while running none of them, and it is # lets this host name a service's client while running none of them, and it is
# the same read the service's own module registers the client with. A swarm # the same read each service's own module registers its client with. A swarm
# that runs no Grafana mints no secret for it, so its entry skips below rather # that runs no Grafana, or no collector, mints no secret for the one it lacks,
# than needing a condition here. # so its entry skips below rather than needing a condition here.
serviceClientIds = [ hyperhiveCfg.swarm.grafana.oidc.clientId ]; serviceClientIds = [
hyperhiveCfg.swarm.grafana.oidc.clientId
hyperhiveCfg.swarm.otel.clientId
];
in in
{ {
options.services.hyperhive.deploy.swarm-secret-publisher = { options.services.hyperhive.deploy.swarm-secret-publisher = {
@ -92,10 +95,11 @@ in
identity's job (see `baoClientCertFile`), not this option's. identity's job (see `baoClientCertFile`), not this option's.
Turning it off leaves every hive but this one without its agents' Turning it off leaves every hive but this one without its agents'
credential, and the swarm's Grafana without any login at all — its credential, the swarm's Grafana without any login at all, and its
secret has exactly one route and this is the producer's end of it. So collector pushing unauthenticated — each secret has exactly one route
the honest reason to set it false is a deployment delivering those and this is the producer's end of it. So the honest reason to set it
secrets by some other mechanism it owns. false is a deployment delivering those secrets by some other
mechanism it owns.
''; '';
}; };

94
nix/module-eval.nix
View file

@ -504,6 +504,35 @@ let
otelSettings = otelSettings =
machine: machine.containers.swarm-otel.config.services.opentelemetry-collector.settings; machine: machine.containers.swarm-otel.config.services.opentelemetry-collector.settings;
# The collector beside authelia, reading its own OIDC secret out of the
# store like every other collector — the cert pair here is not scenery, it
# is the arm that would catch the deleted co-located copy unit coming back.
otelBaoWithAuthelia = hive {
deploy.swarm-otel.enable = true;
deploy.authelia.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# The same collector with the IdP on ANOTHER host and a store leaf placed by
# hand. Identical to the fixture above in everything the delivery path
# reads, which is the point.
otelBaoRemoteAuthelia = hive {
deploy.swarm-otel.enable = true;
swarm.authelia.url = "https://auth.example.invalid";
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# A collector holding no store identity at all. Unlike Grafana's mirror
# image, this is not a refused shape: the collector still receives every
# hive's telemetry with nothing to push authenticated with, which is the
# already-supported degrade `haveCollectorSecret` names above the module's
# `let`. What this fixture is for is checking the reading unit itself does
# not render, rather than rendering with an env var nothing filled in.
otelNoIdentity = hive {
deploy.swarm-otel.enable = true;
swarm.authelia.url = "https://auth.example.invalid";
};
# authelia somewhere else, the credential delivered by hand. Whether this # authelia somewhere else, the credential delivered by hand. Whether this
# collector authenticates must follow the credential, never another # collector authenticates must follow the credential, never another
# service's placement. # service's placement.
@ -1031,6 +1060,71 @@ let
lib.elem "swarm-grafana" (clients secretPublisherHere) lib.elem "swarm-grafana" (clients secretPublisherHere)
&& !(lib.elem "swarm-grafana" (clients grafanaRemoteAuthelia)); && !(lib.elem "swarm-grafana" (clients grafanaRemoteAuthelia));
} }
{
# The collector's half of the same defect and the same fix: this used to
# be gated on `deploy.swarm-otel.enable`, so a split deployment
# registered the client nowhere and authelia minted nothing to publish.
name = "the swarm's collector client is registered wherever authelia runs";
ok =
let
clients = m: map (c: c.id) m.services.hyperhive.swarm.authelia.oidc.clients;
in
lib.elem "swarm-collector" (clients secretPublisherHere)
&& !(lib.elem "swarm-collector" (clients otelBaoRemoteAuthelia));
}
{
# 🩸 The arm that guards the ruling this slice landed under, the
# collector's half of ./swarm-grafana.nix's own. There is ONE delivery
# route: the store reader, on every host that runs the collector and
# holds a store identity. The negative names the deleted unit rather
# than a generic absence, because the way this regresses is someone
# re-adding the co-located copy as an optimisation.
name = "the collector's OIDC secret has exactly one delivery unit, the store reader, in both topologies";
ok =
let
local = otelBaoWithAuthelia.systemd.services;
remote = otelBaoRemoteAuthelia.systemd.services;
in
local ? swarm-bao-otel-oidc
&& remote ? swarm-bao-otel-oidc
&& !(local ? swarm-otel-oidc-secret)
&& !(remote ? swarm-otel-oidc-secret);
}
{
# Same 403-not-a-miss reason as grafana's arm above: the reader's grant
# covers the `services` prefix, so a path outside it is refused rather
# than empty, however correct it reads.
name = "the collector's OIDC secret is read from the prefix the publisher writes";
ok =
let
s = otelBaoRemoteAuthelia.systemd.services.swarm-bao-otel-oidc.script;
in
lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" s
&& !(lib.hasInfix "secret/swarm/hives/" s);
}
{
# Both ends of a wire nothing at eval time carries end to end: the
# publisher on authelia's host writes the path the reader on the
# collector's host reads, and the two files agree only because both
# compose it from the same swarm-wide client id. `secretPublisherHere`
# already grew this client when `serviceClientIds` did.
name = "the publisher writes the swarm service path the collector reads";
ok = lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" (
secretPublisherHere.systemd.services.swarm-secret-publish.script
);
}
{
# The collector's non-assertion, the deliberate mirror of Grafana's
# assertion two cases up: a host with no store identity is a supported,
# merely degraded shape here, so the reading unit simply does not exist
# rather than refusing the build. `haveCollectorSecret` is what the
# degrade already reads, unchanged by this slice.
name = "a collector with no store identity renders no reading unit, and is not refused";
ok =
!(otelNoIdentity.systemd.services ? swarm-bao-otel-oidc)
&& otelNoIdentity.services.hyperhive.deploy.swarm-otel.clientSecretFile == null
&& !(lib.any (a: !a.assertion) otelNoIdentity.assertions);
}
{ {
# Reads the daemon's rendered unit, not the options: the queue address # Reads the daemon's rendered unit, not the options: the queue address
# arrives as an env var whose whole attrset is guarded on `natsUrl`, and # arrives as an env var whose whole attrset is guarded on `natsUrl`, and