Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

..
7 changed files with 148 additions and 412 deletions

View file

@ -42,7 +42,7 @@ neither is a renaming of the other.
## Swarm-level — one of each per swarm ## Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere | | secret | generated by | lives at | hive elsewhere |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public | | swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below | | swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives | | swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
@ -51,7 +51,7 @@ neither is a renaming of the other.
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector | | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` | | OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity is left with `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
@ -72,27 +72,27 @@ an all-local swarm it reads authelia's file where it lies and no second copy is
made. On any other topology it's an ordinary "operator provides the file" made. On any other topology it's an ordinary "operator provides the file"
case — see `services.hyperhive.otel.clientSecretFile`. case — see `services.hyperhive.otel.clientSecretFile`.
The **swarm's** collector does need one, and its delivery route is the fourth The **swarm's** collector does need one. It runs in a container, so
store reader — the same shape as Grafana's, described in full below. In short: `swarm-otel-oidc-secret.service` places its copy, landing at
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client` `/var/lib/swarm-otel-oidc/<client-id>.secret` — the same shape as the forge and
out of the store on every host that runs the collector and holds a store homeserver rows below, and for the same reason: the container that mints the
identity, whether authelia is a network away or in the container next door. secret isn't the container that reads it.
The client it reads for is registered unconditionally, the same call
`glue-grafana-oidc-client.nix` makes for Grafana's: authelia refuses a
bearer-authz client with no audience, and the push audiences supply one even
for a collector with nothing published to scrape.
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana, The copy is made whenever authelia is enabled **on this host**, whatever this
where an absent OIDC block is a container with no way in at all, a collector collector scrapes. The client it delivers for is registered unconditionally:
with nothing to authenticate with still receives every hive's telemetry — authelia refuses a bearer-authz client with no audience, and the push audiences
only its own pushes to the stores go out unauthenticated and get refused supply one even for a collector with nothing published to scrape. Like the
there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without hive-level delivery units below, this one waits a bounded 120s for authelia's
`deploy.bao.clientCertFile` / `clientKeyFile`, the same choice mint and then fails rather than skipping, so a secret that never arrives takes
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for the collector's container start with it instead of surfacing later as a 401.
their own optional readers, rather than the hard assertion `swarm-grafana.nix`
uses for its non-optional one. The secret can still be delivered by hand, at ⚠️ **Don't read that delivery unit as the only way this collector is fed.**
`services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a collector this Whether it authenticates follows the _credential_, never another service's
unit never reaches. placement: a swarm collector may run on a host that holds neither store and no
authelia, and then the secret is an ordinary operator-provided file named in
`services.hyperhive.deploy.swarm-otel.clientSecretFile` — the same shape as the hive
collector's row above. The copy unit is the convenience for the co-located case,
not the definition of the case.
### Minting the queue's callout nkeys ### Minting the queue's callout nkeys
@ -212,18 +212,6 @@ nothing said anything. Registration is separate and stays where authelia is
so it's declared on the host running authelia, which isn't necessarily the host so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. running Grafana.
The **fourth reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it
doesn't render the reading unit without them, the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for
their own optional reads.
A service's secret is one value for the whole swarm rather than one per hive, so A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs whole. That's because nothing in a swarm's configuration records which hive runs
@ -232,7 +220,7 @@ there is nothing to scope the grant to. Every hive can therefore read every swar
service's client secret; that's the cost of a shape where the service's host has service's client secret; that's the cost of a shape where the service's host has
no store identity of its own to present, and it's stated in no store identity of its own to present, and it's stated in
`swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves `swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves
as it does for the other readers: a store that says "not here" leaves the as it does for the other two readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in. client secret to stand in.

View file

@ -66,17 +66,16 @@ _own_ entry under the hive namespace, which names the hive asking and so widens
nothing between them, and every **swarm service's** OIDC client secret. nothing between them, and every **swarm service's** OIDC client secret.
That third prefix has the same shape of reason as the first, and the same honest That third prefix has the same shape of reason as the first, and the same honest
cost. A swarm service (Grafana and the swarm collector are the two there today) cost. A swarm service (Grafana is the one there today) registers **one** client
registers **one** client for the whole swarm, so its credential's path names for the whole swarm, so its credential's path names the service and never the
the service and never the host — and which hive runs a given service is a host — and which hive runs a given service is a `deploy.*` fact, per-host by
`deploy.*` fact, per-host by definition, so nothing swarm-wide exists to scope definition, so nothing swarm-wide exists to scope the grant to. The host running
the grant to. The host running such a service has no store identity of its own such a service has no store identity of its own either; it reads with the
either; it reads with the certificate of the hive it is. **Any hive can certificate of the hive it is. **Any hive can therefore read any swarm service's
therefore read any swarm service's client secret**, which is worth what it client secret**, which is worth what it buys: every swarm service gets its
buys: every swarm service gets its credential the same way from anywhere, credential the same way from anywhere, instead of only where the identity
instead of only where the identity provider happens to sit. Giving such a provider happens to sit. Giving such a service its own store identity is what
service its own store identity is what would remove this rather than re-scope would remove this rather than re-scope it.
it.
**A tracked follow-up** narrows this, with the two candidate directions: scope **A tracked follow-up** narrows this, with the two candidate directions: scope
the grant per hive (and pay for the re-emission), or give each agent container the grant per hive (and pay for the re-emission), or give each agent container

View file

@ -29,7 +29,6 @@
./glue-matrix-bao-token.nix ./glue-matrix-bao-token.nix
./glue-queue-agent-credential.nix ./glue-queue-agent-credential.nix
./glue-secret-publisher-bao-identity.nix ./glue-secret-publisher-bao-identity.nix
./glue-swarm-otel-oidc-client.nix
./swarm-authelia.nix ./swarm-authelia.nix
./swarm-bao.nix ./swarm-bao.nix
./swarm-ca.nix ./swarm-ca.nix

View file

@ -1,58 +0,0 @@
# Glue: register the swarm's collector as an OIDC client wherever authelia
# runs.
#
# ONE PAIRING PER FILE — swarm-otel ← authelia, and nothing else. Deleting
# this leaves a swarm whose collector is not a client authelia has ever heard
# of, so no token it presents is ever accepted and authelia mints no secret
# for the publisher to send on.
#
# ⚠️ Gated on authelia being HERE, and deliberately NOT on this host running
# the collector. A client is a row in THIS host's provider config, so it can
# only be declared where that config is rendered — and ./swarm-otel.nix's
# whole `config` block hangs off `deploy.swarm-otel.enable`, so a swarm with
# the collector and authelia on different hosts registered the client
# nowhere at all. Same bug, same fix, as ./glue-grafana-oidc-client.nix one
# module over — read that file's own comment for the property this one
# shares with it.
#
# ⚠️ Registered whether or not the swarm has a collector, for the same
# reason as Grafana's: nothing in `swarm.*` records that one exists,
# `deploy.swarm-otel.enable` only answers "does THIS host run it". The cost
# is one unused client and one unused minted secret in a swarm with no
# collector — the same trade ./swarm-otel.nix already made for this exact
# client when it dropped the published-scrape-target guard on registering it.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
otelCfg = hyperhiveCfg.swarm.otel;
in
{
config = lib.mkIf (hyperhiveCfg.enable && deployCfg.authelia.enable) {
# One declaration, two readers: `clientId` and `audience` are read-only
# options ./swarm-otel.nix derives from the scrape/push targets it owns,
# so this file states neither formula a second time.
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = otelCfg.clientId;
description = "HyperHive swarm collector";
kind = "machine";
# Grants `authelia.bearer.authz`, without which the authz endpoint
# refuses an otherwise valid token and blames the token rather than
# the missing grant.
bearerAuthz = true;
audience = otelCfg.audience;
# Stated rather than left on authelia's default, because the two
# agreeing today is not the same as this being the required value:
# authelia permits only basic / JWT methods for a confidential
# client holding this scope, and enforces it in the startup
# validator.
tokenEndpointAuthMethod = "client_secret_basic";
}
];
};
}

View file

@ -23,8 +23,6 @@ let
vlCfg = config.services.hyperhive.swarm.victorialogs; vlCfg = config.services.hyperhive.swarm.victorialogs;
hyperhiveCfg = config.services.hyperhive; hyperhiveCfg = config.services.hyperhive;
gatewayCfg = hyperhiveCfg.gateway; gatewayCfg = hyperhiveCfg.gateway;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
swarmDomain = hyperhiveCfg.swarm.domain; swarmDomain = hyperhiveCfg.swarm.domain;
# Total on a null swarm domain for the same reason every sibling module is: # Total on a null swarm domain for the same reason every sibling module is:
@ -134,13 +132,6 @@ let
# delivery unit on a later boot, and a secret that evaporates on reboot # delivery unit on a later boot, and a secret that evaporates on reboot
# turns a working scrape into an intermittent one. # turns a working scrape into an intermittent one.
collectorSecretInContainer = "/var/lib/swarm-otel-oidc/${cfg.clientId}.secret"; collectorSecretInContainer = "/var/lib/swarm-otel-oidc/${cfg.clientId}.secret";
# The same file seen from the host, which is where the delivery unit
# writes it — spelled once for the same reason ./swarm-grafana.nix spells
# its own pair once: the unit that writes it and the option that names it
# are a few hundred lines apart, and a collector reading a path nothing
# writes is an export that fails with nothing in any log about the file.
collectorHostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}";
collectorHostSecretDir = builtins.dirOf collectorHostSecretPath;
collectorCredentialId = "oidc-client-secret"; collectorCredentialId = "oidc-client-secret";
# What the scrape config points at. ⚠️ This path and the `LoadCredential` # What the scrape config points at. ⚠️ This path and the `LoadCredential`
# id below are one fact spelled twice by systemd's design — both derive from # id below are one fact spelled twice by systemd's design — both derive from
@ -215,32 +206,11 @@ let
pushAuthenticator = name: "oauth2client/${name}"; pushAuthenticator = name: "oauth2client/${name}";
# Whether this collector holds a credential — a property of the credential, # Whether this collector holds a credential — a property of the credential,
# not of where any other service runs. Not an assertion: a collector that # not of where any other service runs. Not an assertion: a collector on a
# pushes nowhere authenticated still receives from every hive, so refusing # host of its own is a supported shape, and refusing to build it would make
# to build one would make a supported shape illegal for want of a value # this fix illegal where the bug bites hardest.
# that only degrades what it can push.
haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null; haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null;
# A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical-
# looking flag, this one stays outside `assertions`: the store-reading unit
# below simply does not render without it, the same choice
# ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for
# their own optional readers, because a collector with no client identity is
# `haveCollectorSecret = false` above, and that is already a supported,
# merely degraded shape rather than a service with no way in at all.
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Where the publisher on authelia's host leaves this client's secret —
# composed from the same swarm-wide `clientId` the registration in
# ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of
# them behind. The `services` segment is `swarm-secret-client`'s
# `path::Kind::Service`, the same prefix ./swarm-grafana.nix reads its own
# client secret under — one write grant in ./swarm-bao.nix and one hive read
# grant in `policy::render` already cover it.
storeSecretPath = "secret/swarm/services/${cfg.clientId}/oidc/client";
# The operator-configured upstream, named once: the same exporter carries # The operator-configured upstream, named once: the same exporter carries
# every signal, so metrics and logs both reach it without a second # every signal, so metrics and logs both reach it without a second
# definition. # definition.
@ -523,21 +493,6 @@ in
after the change that caused it evaluated cleanly. after the change that caused it evaluated cleanly.
''; '';
}; };
audience = lib.mkOption {
type = lib.types.listOf lib.types.str;
readOnly = true;
default = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences;
description = ''
Every audience this collector's OAuth2 client is permitted to
present a token for — the scrape targets it reads with a credential,
plus the stores it pushes to. Published read-only so
./glue-swarm-otel-oidc-client.nix can register the client wherever
authelia runs without restating the derivation: this option and
that registration are the same fact seen from two hosts, and a
second formula for it would be free to drift from this one.
'';
};
}; };
# What stays above is what the collector IS to the swarm — the client it is # What stays above is what the collector IS to the swarm — the client it is
@ -574,10 +529,7 @@ in
# delivering says so in the store it stopped delivering to. That is # delivering says so in the store it stopped delivering to. That is
# less circular than it sounds: the failure that matters here is a # less circular than it sounds: the failure that matters here is a
# single hive's receiver refusing pushes, not the process dying. # single hive's receiver refusing pushes, not the process dying.
services.hyperhive.swarm.otel.journaldUnits = [ services.hyperhive.swarm.otel.journaldUnits = [ "opentelemetry-collector" ];
"opentelemetry-collector"
"swarm-bao-otel-oidc"
];
# The metrics counterpart to the journal line above, closing the same # The metrics counterpart to the journal line above, closing the same
# gap from the other side: the journal says the process is alive, these # gap from the other side: the journal says the process is alive, these
@ -632,152 +584,109 @@ in
# this is how it authenticates *itself* to a service published behind # this is how it authenticates *itself* to a service published behind
# the gateway. # the gateway.
# #
# Registering the client is NOT here any more: it has to happen on the # Only where authelia is co-located. A client is a row in this host's
# host that runs authelia, and this whole block is gated on the host that # provider config, so declaring one against a remote provider would
# runs the collector. ./glue-swarm-otel-oidc-client.nix is where it moved # render nothing while reading as done; a swarm whose authelia lives
# to, the same split ./swarm-grafana.nix made for its own client. # elsewhere registers it there.
#
# ⚠️ NO LONGER conditional on a published scrape target. Authelia refuses
# a bearer-authz client with no audience, which is what that guard was
# for — and the push audiences below are unconditional, so there is now
# always at least one. Keeping the old guard would have left a collector
# that scrapes nothing pushing to the stores with no client to get a
# token from.
services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf deployCfg.authelia.enable [
{
id = cfg.clientId;
description = "HyperHive swarm collector";
kind = "machine";
# Grants `authelia.bearer.authz`, without which the authz
# endpoint refuses an otherwise valid token and blames the
# token rather than the missing grant.
bearerAuthz = true;
# DERIVED from the targets rather than contributed alongside
# them. A service declares a URL once and this is the
# permission to reach it; two lists that had to agree would be
# a drift to maintain, and the failure mode is the quiet one —
# a target whose audience was forgotten authenticates against
# nothing and looks like a broken scrape.
#
# Both directions land in one list because authelia has one: what
# this collector may SCRAPE and what it may PUSH TO are the same
# kind of permission, differing only in who initiates.
audience = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences;
# Stated rather than left on authelia's default, because the
# two agreeing today is not the same as this being the
# required value: authelia permits only basic / JWT methods
# for a confidential client holding that scope, and enforces
# it in the startup validator.
tokenEndpointAuthMethod = "client_secret_basic";
}
];
# THE delivery unit — one route, in every deployment. The secret authelia # Deliver the collector's client secret from authelia's container into
# minted arrives out of the swarm secret store, which the publisher on # this one. On the HOST because that is the only place both container
# authelia's host wrote it into, whether authelia is a network away or in # trees are addressable: they share this host's network namespace, which
# the container next door. # makes them feel co-located, but their filesystem roots are separate.
# #
# 🩸 A unit here used to copy the plaintext directly out of authelia's # ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container
# host tree, reachable only because they share this host's network # refuses to start when a bind source is missing, and this secret does not
# namespace — which stopped working the moment authelia moved to another # exist until authelia's first boot has minted it — so binding it would
# host, which is the defect this whole change exists to fix. The ruling # make the collector wait on a file that waits on a container that starts
# that deleted it rather than gave it a remote sibling: the store exists # after it. On a fresh swarm that is a permanent stall presenting as
# so a host holds ONE out-of-band secret — its client certificate — and # "metrics are broken", several layers from its cause.
# reads everything else with it. Recorded in docs/swarm/secrets.md. # The delivery unit below is the one thing here that may know where
# # authelia runs — it copies out of its container — so it is also what
# Shaped after ./glue-queue-agent-credential.nix: a cert login that fails # names the file. `mkDefault`, so a deployment that delivers the secret
# LOUDLY, since every state it fails on is one a retry fixes, then a read # some other way just sets the option.
# that degrades QUIETLY, since no retry turns "no value there" into a services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable (
# value. lib.mkDefault collectorSecretInContainer
# );
# ⚠️ Renders only where `haveClientIdentity` holds, unlike
# ./swarm-grafana.nix's equivalent unit. That module asserts the identity systemd.services.swarm-otel-oidc-secret = lib.mkIf deployCfg.authelia.enable {
# because a Grafana with none has no way in at all; this collector without description = "deliver the swarm collector's OIDC client secret from authelia";
# one is `haveCollectorSecret = false` above — already a supported, after = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ];
# merely degraded shape, so the unit that would fetch a credential simply requires = [ "container@${hyperhiveCfg.swarm.authelia.machine}.service" ];
# does not exist rather than refusing the build for want of one.
systemd.services.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity {
description = "fetch the swarm collector's OIDC client secret from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering
# is conditional even though the read is not: off-host there is nothing
# local to wait for, and the timeout below bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
before = [ "container@${cfg.machine}.service" ]; before = [ "container@${cfg.machine}.service" ];
wantedBy = [ wantedBy = [ "container@${cfg.machine}.service" ];
"multi-user.target"
"container@${cfg.machine}.service"
];
path = [
baoDeploy.package
pkgs.coreutils
];
# Sized for the race this loses, not for an unseal: `swarm-bao` comes up
# seconds before this unit asks, and the cert-auth role it logs in
# against is written seconds after, so a few short attempts cover it. An
# hours-long window would be a bet on a sealed store, and the degrade
# below is already correct for that.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec × burst`.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = { serviceConfig = {
Type = "oneshot"; Type = "oneshot";
RemainAfterExit = true; RemainAfterExit = true;
SyslogIdentifier = "swarm-bao-otel-oidc"; SyslogIdentifier = "swarm-otel-oidc-secret";
# What actually bounds the read below. Stated here rather than left to # Longer than the bounded wait below, and that is the point:
# systemd's default, so the number a boot waits on is in the file that # `DefaultTimeoutStartSec` is 90s, so without this systemd kills
# waits. # the unit before it can emit the message naming the file it was
TimeoutStartSec = 30; # waiting for — the failure then reads as a timeout with no cause.
Restart = "on-failure"; TimeoutStartSec = "180s";
RestartSec = 15;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.clientCertFile;
BAO_CLIENT_KEY = baoDeploy.clientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
}; };
path = [ pkgs.coreutils ];
script = '' script = ''
set -euo pipefail set -euo pipefail
# `bao`'s own message is the only thing separating a missing value src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.clientId}.secret"}
# from a refused identity from an unreachable host. This unit's dst=${lib.escapeShellArg "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}"}
# degraded mode is correct for all three, so it reports which one
# rather than asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` # authelia's container is up, but its first-boot generator may
# variables above only decide which certificate the TLS handshake # still be minting. Bounded wait, then fail: skipping silently
# presents; without a token `bao` asks its token helper instead, and # produces a collector whose scrape gets a 401 forever, which is
# that is a `sh` this unit's `path` does not carry. `-token-only` # the failure this whole design exists to make impossible.
# answers on stdout and skips the helper on both sides. for _ in $(seq 1 60); do
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then [ -s "$src" ] && break
echo "could not log in to swarm-bao with this host's certificate; leaving the collector's OIDC client secret as it is." >&2 sleep 2
if [ -s "$err" ]; then done
cat "$err" >&2 if [ ! -s "$src" ]; then
else echo "authelia has not minted $src after 120s" >&2
echo "bao failed without writing a diagnostic." >&2
fi
exit 1 exit 1
fi fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then # root-owned 0400. The collector runs under `DynamicUser`, so
echo "swarm-bao did not return ${storeSecretPath}; the collector has no OIDC client secret yet." >&2 # there is no uid to give it to — `LoadCredential` reads this as
if [ -s "$err" ]; then # root before the sandbox exists and re-exposes it to whichever
cat "$err" >&2 # uid the unit got.
else install -D -m 0400 -o root -g root "$src" "$dst"
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; leaving the file as it is." >&2
exit 0
fi
# root-owned 0400, written with a shell builtin and never handed to a
# program: `printf` is bash's own, so the plaintext never becomes an
# argument in /proc the way `install <<<"$secret"` or an `echo` from
# `path` would. The collector runs under `DynamicUser`, so there is
# no uid to give this to — `LoadCredential` reads it as root before
# the sandbox exists and re-exposes it to whichever uid the unit got.
install -d -m 0755 ${lib.escapeShellArg collectorHostSecretDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg collectorHostSecretPath}
chown root:root ${lib.escapeShellArg collectorHostSecretPath}
chmod 0400 ${lib.escapeShellArg collectorHostSecretPath}
''; '';
}; };
# Where the delivery unit above lands the secret. `mkDefault`, so a
# deployment delivering it some other way just sets the option — and
# `mkIf haveClientIdentity` so a host with no store identity is left with
# `clientSecretFile == null`, the already-supported degrade rather than a
# path nothing ever writes.
services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf haveClientIdentity (
lib.mkDefault collectorSecretInContainer
);
# The CA bind source is written at runtime by a host unit, so the # The CA bind source is written at runtime by a host unit, so the
# container has to start after it — otherwise nspawn sets up a mount # container has to start after it — otherwise nspawn sets up a mount
# over a file that does not exist yet. # over a file that does not exist yet.

View file

@ -64,13 +64,10 @@ let
# #
# The ids come from `swarm.*`, which is identical on every host — that is what # The ids come from `swarm.*`, which is identical on every host — that is what
# lets this host name a service's client while running none of them, and it is # lets this host name a service's client while running none of them, and it is
# the same read each service's own module registers its client with. A swarm # the same read the service's own module registers the client with. A swarm
# that runs no Grafana, or no collector, mints no secret for the one it lacks, # that runs no Grafana mints no secret for it, so its entry skips below rather
# so its entry skips below rather than needing a condition here. # than needing a condition here.
serviceClientIds = [ serviceClientIds = [ hyperhiveCfg.swarm.grafana.oidc.clientId ];
hyperhiveCfg.swarm.grafana.oidc.clientId
hyperhiveCfg.swarm.otel.clientId
];
in in
{ {
options.services.hyperhive.deploy.swarm-secret-publisher = { options.services.hyperhive.deploy.swarm-secret-publisher = {
@ -95,11 +92,10 @@ in
identity's job (see `baoClientCertFile`), not this option's. identity's job (see `baoClientCertFile`), not this option's.
Turning it off leaves every hive but this one without its agents' Turning it off leaves every hive but this one without its agents'
credential, the swarm's Grafana without any login at all, and its credential, and the swarm's Grafana without any login at all — its
collector pushing unauthenticated — each secret has exactly one route secret has exactly one route and this is the producer's end of it. So
and this is the producer's end of it. So the honest reason to set it the honest reason to set it false is a deployment delivering those
false is a deployment delivering those secrets by some other secrets by some other mechanism it owns.
mechanism it owns.
''; '';
}; };

View file

@ -498,44 +498,12 @@ let
otelNoStores = hive { otelNoStores = hive {
deploy.swarm-otel.enable = true; deploy.swarm-otel.enable = true;
deploy.authelia.enable = true; deploy.authelia.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.victoriametrics.enable = false; deploy.victoriametrics.enable = false;
deploy.victorialogs.enable = false; deploy.victorialogs.enable = false;
}; };
otelSettings = otelSettings =
machine: machine.containers.swarm-otel.config.services.opentelemetry-collector.settings; machine: machine.containers.swarm-otel.config.services.opentelemetry-collector.settings;
# The collector beside authelia, reading its own OIDC secret out of the
# store like every other collector — the cert pair here is not scenery, it
# is the arm that would catch the deleted co-located copy unit coming back.
otelBaoWithAuthelia = hive {
deploy.swarm-otel.enable = true;
deploy.authelia.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# The same collector with the IdP on ANOTHER host and a store leaf placed by
# hand. Identical to the fixture above in everything the delivery path
# reads, which is the point.
otelBaoRemoteAuthelia = hive {
deploy.swarm-otel.enable = true;
swarm.authelia.url = "https://auth.example.invalid";
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# A collector holding no store identity at all. Unlike Grafana's mirror
# image, this is not a refused shape: the collector still receives every
# hive's telemetry with nothing to push authenticated with, which is the
# already-supported degrade `haveCollectorSecret` names above the module's
# `let`. What this fixture is for is checking the reading unit itself does
# not render, rather than rendering with an env var nothing filled in.
otelNoIdentity = hive {
deploy.swarm-otel.enable = true;
swarm.authelia.url = "https://auth.example.invalid";
deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret";
};
# authelia somewhere else, the credential delivered by hand. Whether this # authelia somewhere else, the credential delivered by hand. Whether this
# collector authenticates must follow the credential, never another # collector authenticates must follow the credential, never another
# service's placement. # service's placement.
@ -1063,71 +1031,6 @@ let
lib.elem "swarm-grafana" (clients secretPublisherHere) lib.elem "swarm-grafana" (clients secretPublisherHere)
&& !(lib.elem "swarm-grafana" (clients grafanaRemoteAuthelia)); && !(lib.elem "swarm-grafana" (clients grafanaRemoteAuthelia));
} }
{
# The collector's half of the same defect and the same fix: this used to
# be gated on `deploy.swarm-otel.enable`, so a split deployment
# registered the client nowhere and authelia minted nothing to publish.
name = "the swarm's collector client is registered wherever authelia runs";
ok =
let
clients = m: map (c: c.id) m.services.hyperhive.swarm.authelia.oidc.clients;
in
lib.elem "swarm-collector" (clients secretPublisherHere)
&& !(lib.elem "swarm-collector" (clients otelBaoRemoteAuthelia));
}
{
# 🩸 The arm that guards the ruling this slice landed under, the
# collector's half of ./swarm-grafana.nix's own. There is ONE delivery
# route: the store reader, on every host that runs the collector and
# holds a store identity. The negative names the deleted unit rather
# than a generic absence, because the way this regresses is someone
# re-adding the co-located copy as an optimisation.
name = "the collector's OIDC secret has exactly one delivery unit, the store reader, in both topologies";
ok =
let
local = otelBaoWithAuthelia.systemd.services;
remote = otelBaoRemoteAuthelia.systemd.services;
in
local ? swarm-bao-otel-oidc
&& remote ? swarm-bao-otel-oidc
&& !(local ? swarm-otel-oidc-secret)
&& !(remote ? swarm-otel-oidc-secret);
}
{
# Same 403-not-a-miss reason as grafana's arm above: the reader's grant
# covers the `services` prefix, so a path outside it is refused rather
# than empty, however correct it reads.
name = "the collector's OIDC secret is read from the prefix the publisher writes";
ok =
let
s = otelBaoRemoteAuthelia.systemd.services.swarm-bao-otel-oidc.script;
in
lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" s
&& !(lib.hasInfix "secret/swarm/hives/" s);
}
{
# Both ends of a wire nothing at eval time carries end to end: the
# publisher on authelia's host writes the path the reader on the
# collector's host reads, and the two files agree only because both
# compose it from the same swarm-wide client id. `secretPublisherHere`
# already grew this client when `serviceClientIds` did.
name = "the publisher writes the swarm service path the collector reads";
ok = lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" (
secretPublisherHere.systemd.services.swarm-secret-publish.script
);
}
{
# The collector's non-assertion, the deliberate mirror of Grafana's
# assertion two cases up: a host with no store identity is a supported,
# merely degraded shape here, so the reading unit simply does not exist
# rather than refusing the build. `haveCollectorSecret` is what the
# degrade already reads, unchanged by this slice.
name = "a collector with no store identity renders no reading unit, and is not refused";
ok =
!(otelNoIdentity.systemd.services ? swarm-bao-otel-oidc)
&& otelNoIdentity.services.hyperhive.deploy.swarm-otel.clientSecretFile == null
&& !(lib.any (a: !a.assertion) otelNoIdentity.assertions);
}
{ {
# Reads the daemon's rendered unit, not the options: the queue address # Reads the daemon's rendered unit, not the options: the queue address
# arrives as an env var whose whole attrset is guarded on `natsUrl`, and # arrives as an env var whose whole attrset is guarded on `natsUrl`, and