vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

nix: split statusPublish and the otel secret into deploy.*

Browse source 
Slices 8 and 9 of the swarm/deploy split, and the last two.

statusPublish had three coordinates under one namespace. Two of them
are this machine's — where the queue listens *as seen from here*, and
where its client secret sits on this disk — so they move to
`deploy.hive-controller.statusPublish.*`, the namespace of the daemon
that is their only reader. `tokenEndpoint` is the swarm's one address,
so it stays. That leaves `swarm.statusPublish` holding a single option:
a legitimate split, not a botched move.

The all-or-nothing assertion now spans both namespaces. It is repointed
in both its condition and its message, and the message spells all three
paths in full so an operator is never told to set two options under a
path that only has one. `environment.nix`'s guard and the value beside
it likewise read different namespaces on purpose.

The collector's secret moves the same way, for the same reason, to
`deploy.swarm-otel.*` — `enable` already lives there. That also retires
one of the eight cross-namespace assignments tracked in #4048: the
delivery unit set a `swarm.*` value under a `deploy.*` gate, and now
sets a `deploy.*` value under one.

module-eval gets a fixture per slice. `otelRemoteAuthelia` already set
the collector secret through its pre-rename path, so it becomes slice
9's old-path case as it stands — left spelled that way deliberately,
with a comment, so it is not read later as a missed site.

That fixture also turned out to be describing an impossible hive: it
said authelia lives elsewhere without saying where, so the
authenticator interpolated a null `swarm.authelia.url` into its
`token_url`. Nothing to do with the rename, and invisible to the
existing case over the same fixture, which reads `? auth` and `elem` —
both stop at names and never force the extension's value. Given the
address a remote-IdP deployment has.

Verified: 49 -> 51 properties, all holding.
This commit is contained in:
atlas 2026-09-07 16:20:46 +02:00 • committed by mara
commit 01ce968fb6
8 changed files with 154 additions and 48 deletions
Download patch file Download diff file Expand all files Collapse all files

19
docs/swarm/README.md
View file

@ -301,8 +301,8 @@ it said. Hives publish upward through the swarm queue; the controller
never reaches down to collect, so a hive that can't reach the swarm
still knows its own state — you just can't see it from here.
A hive publishes only once it has been given the three
`swarm.statusPublish` coordinates below. A hive that has not reads
A hive publishes only once it holds all three status-publish coordinates
below. A hive without them reads
`never_reported` — it's not broken, it just has nothing to say upward.
| freshness | what to do about it |
@ -322,17 +322,18 @@ between reports. Hives publish once a minute, so the default tolerates
one missed report and flags two. It takes effect on the next request;
nothing has to re-publish.
### Making a hive report (`swarm.statusPublish`)
### Making a hive report
Three options, on the **hive**, set together or not at all — a
half-configured hive is an eval error rather than one that quietly never
reports:
reports. They sit in two namespaces, because two of them are facts about
_this machine_ and one is the swarm's single address:
| option | what to set it to |
| ------------------ | ------------------------------------------------------ |
| `natsUrl` | where the swarm queue listens, as this hive reaches it |
| `tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
| `clientSecretFile` | path to this hive's client secret, plaintext |
| option | what to set it to |
| ------------------------------------------------------- | ------------------------------------------------------ |
| `deploy.hive-controller.statusPublish.natsUrl` | where the swarm queue listens, as this hive reaches it |
| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
| `deploy.hive-controller.statusPublish.clientSecretFile` | path to this hive's client secret, plaintext |
On a host that runs the queue and the IdP itself, all three default to
the local ones and there is nothing to set. Any other hive needs them

4
docs/swarm/secrets.md
View file

@ -51,7 +51,7 @@ neither is a renaming of the other.
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `swarm.otel.clientSecretFile` — the collector need not share a host with authelia |
| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
@ -84,7 +84,7 @@ published is being scraped; otherwise no secret is needed and none is placed.
Whether it authenticates follows the _credential_, never another service's
placement: a swarm collector may run on a host that holds neither store and no
authelia, and then the secret is an ordinary operator-provided file named in
`services.hyperhive.swarm.otel.clientSecretFile` — the same shape as the hive
`services.hyperhive.deploy.swarm-otel.clientSecretFile` — the same shape as the hive
collector's row above. The copy unit is the convenience for the co-located case,
not the definition of the case.

20
nix/host-modules/deploy.nix
View file

@ -110,10 +110,30 @@ in
[ "services" "hyperhive" "swarm" "nats" "enable" ]
[ "services" "hyperhive" "deploy" "nats" "enable" ]
)
# Two of statusPublish's three coordinates. `tokenEndpoint` stays under
# `swarm.*`: it is the swarm IdP's one address, identical on every hive,
# where these two are "the queue as seen from here" and "the secret on
# this disk". The all-or-nothing assertion in ./swarm.nix spans the split
# deliberately and names all three paths in its message.
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "statusPublish" "natsUrl" ]
[ "services" "hyperhive" "deploy" "hive-controller" "statusPublish" "natsUrl" ]
)
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "statusPublish" "clientSecretFile" ]
[ "services" "hyperhive" "deploy" "hive-controller" "statusPublish" "clientSecretFile" ]
)
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "otel" "enable" ]
[ "services" "hyperhive" "deploy" "swarm-otel" "enable" ]
)
# ⚠️ Not to be confused with `services.hyperhive.otel.clientSecretFile`,
# a different option on the hive tier that is not part of this move. A
# bare `clientSecretFile` matches five distinct options across the tree.
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "otel" "clientSecretFile" ]
[ "services" "hyperhive" "deploy" "swarm-otel" "clientSecretFile" ]
)
# The CI runner, and the only entry here that renames more than an
# `enable`: every knob under it describes the runner THIS host would run,

9
nix/host-modules/hive-c0re/default.nix
View file

@ -123,7 +123,8 @@ in
# plain http or loopback — so this changes nothing on an all-local
# hive. It matters for the split-host shape the options invite:
# `swarm.matrix.apiUrl`'s own example is `https://matrix.example.com`,
# and pointing it (or `statusPublish.natsUrl`) at another hive's
# and pointing it (or `deploy.hive-controller.statusPublish.natsUrl`)
# at another hive's
# gateway means verifying a leaf signed by a CA generated at runtime,
# which no build-time trust store can contain.
#
@ -314,9 +315,9 @@ in
# and the daemon never needs read access to wherever it lives.
# (The callout responder copies instead only because it
# delivers into a container, across a filesystem boundary.)
lib.optional (
config.services.hyperhive.swarm.statusPublish.clientSecretFile != null
) "swarm-status-client.secret:${config.services.hyperhive.swarm.statusPublish.clientSecretFile}"
lib.optional
(config.services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile != null)
"swarm-status-client.secret:${config.services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile}"
# The secret store's client identity, on the same reasoning one
# paragraph up — with a sharper edge: ./glue-bao-tls.nix mints the
# key `0600` inside a `0700` directory, so hive-core cannot read it

9
nix/host-modules/hive-c0re/environment.nix
View file

@ -259,8 +259,13 @@ in
# produce is a hive that comes up fine and silently never reports.
# The three-option version of that same rule is asserted at eval in
# ./../swarm.nix, so this can only ever emit a complete set.
lib.optionalAttrs (config.services.hyperhive.swarm.statusPublish.natsUrl != null) {
HIVE_C0RE_NATS_URL = config.services.hyperhive.swarm.statusPublish.natsUrl;
#
# ⚠️ The guard and the value beside it read different namespaces on
# purpose: where the queue is and where its secret sits are this
# machine's, the token endpoint is the swarm's one address. The
# assertion covers all three, which is what keeps the guard honest.
lib.optionalAttrs (config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl != null) {
HIVE_C0RE_NATS_URL = config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl;
HIVE_C0RE_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
# The identity swarm-authelia.nix already declares for every entry in
# `swarm.hives` — the hive does not choose its own name here, it uses

12
nix/host-modules/swarm-otel.nix
View file

@ -213,7 +213,7 @@ let
# not of where any other service runs. Not an assertion: a collector on a
# host of its own is a supported shape, and refusing to build it would make
# this fix illegal where the bug bites hardest.
haveCollectorSecret = cfg.clientSecretFile != null;
haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null;
# The operator-configured upstream, named once: the same exporter carries
# every signal, so metrics and logs both reach it without a second
@ -497,7 +497,13 @@ in
after the change that caused it evaluated cleanly.
'';
};
};
# What stays above is what the collector IS to the swarm — the client it is
# registered as, where it exports. The secret is a path on the machine that
# runs it, so it hangs off the deployment. `enable` already lives in
# ./deploy.nix, which also carries the rename.
options.services.hyperhive.deploy.swarm-otel = {
clientSecretFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
@ -630,7 +636,7 @@ in
# authelia runs — it copies out of its container — so it is also what
# names the file. `mkDefault`, so a deployment that delivers the secret
# some other way just sets the option.
services.hyperhive.swarm.otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable (
services.hyperhive.deploy.swarm-otel.clientSecretFile = lib.mkIf deployCfg.authelia.enable (
lib.mkDefault collectorSecretInContainer
);
@ -1428,7 +1434,7 @@ in
# no authenticator is rendered either, so the collector starts and
# is refused by the stores rather than failing to start.
// lib.optionalAttrs haveCollectorSecret {
LoadCredential = [ "${collectorCredentialId}:${cfg.clientSecretFile}" ];
LoadCredential = [ "${collectorCredentialId}:${deployCfg.swarm-otel.clientSecretFile}" ];
};
};
};

75
nix/host-modules/swarm.nix
View file

@ -356,24 +356,32 @@ in
# genuinely cannot publish with two of three coordinates — which
# is the distinction the `serviceDomains'` comment at the top of
# this file was written about.
#
# The three coordinates live in two namespaces now: the token
# endpoint is the swarm's one address, the other two are this
# machine's. The message spells all three paths out, because an
# operator told only the option names would look for them under
# one prefix and find one of them.
assertion =
let
set = lib.filter (v: v != null) [
swarmCfg.statusPublish.natsUrl
deployCfg.hive-controller.statusPublish.natsUrl
swarmCfg.statusPublish.tokenEndpoint
swarmCfg.statusPublish.clientSecretFile
deployCfg.hive-controller.statusPublish.clientSecretFile
];
in
builtins.length set == 0 || builtins.length set == 3;
message = ''
services.hyperhive.swarm.statusPublish needs natsUrl,
tokenEndpoint and clientSecretFile set together or not at all
— this hive has only some of them.
This hive's status-publishing coordinates have to be set
together or not at all — it has only some of them.
Currently:
natsUrl = ${toString swarmCfg.statusPublish.natsUrl}
tokenEndpoint = ${toString swarmCfg.statusPublish.tokenEndpoint}
clientSecretFile = ${toString swarmCfg.statusPublish.clientSecretFile}
deploy.hive-controller.statusPublish.natsUrl
= ${toString deployCfg.hive-controller.statusPublish.natsUrl}
swarm.statusPublish.tokenEndpoint
= ${toString swarmCfg.statusPublish.tokenEndpoint}
deploy.hive-controller.statusPublish.clientSecretFile
= ${toString deployCfg.hive-controller.statusPublish.clientSecretFile}
Set the missing ones to publish this hive's status to the
swarm, or set all three to null to turn publishing off.
@ -440,6 +448,38 @@ in
# enable. An extra flag would let a hive be configured-but-off, which
# is one more state to explain and one more way to be silently quiet.
options.services.hyperhive.swarm.statusPublish = {
tokenEndpoint = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =
if queueLocal && swarmCfg.authelia.url != null then
"${swarmCfg.authelia.url}/api/oidc/token"
else
null;
defaultText = lib.literalExpression ''"''${swarm.authelia.url}/api/oidc/token" when this host runs both the queue and the IdP, else null'';
example = "https://auth.example.com/api/oidc/token";
description = ''
The swarm IdP's OAuth2 token endpoint. This hive mints a
`client_credentials` access token there and presents it when
connecting to the queue, which authenticates it as
`hive-<hiveName>` — the client
{file}`nix/host-modules/swarm-authelia.nix` already declares for
every entry in {option}`services.hyperhive.swarm.hives`.
'';
};
};
# What stays above is the one thing every hive shares: the address of the
# swarm's token endpoint. The other two coordinates are this machine's —
# where the queue is *as seen from here*, and where its secret sits on this
# disk — so they hang off the daemon that uses them. `hive-controller` is
# hive-c0re's new name, which is why the namespace is the daemon's rather
# than a `deploy.statusPublish` of its own.
#
# ⚠️ The all-or-nothing assertion above now spans both namespaces. That is
# one service's own options either side of the split, not a service reaching
# into a foreign `deploy.*`; its message names all three paths so an operator
# is not told to set two options under a path that has one.
options.services.hyperhive.deploy.hive-controller.statusPublish = {
natsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = if queueLocal then "nats://127.0.0.1:${toString swarmCfg.nats.port}" else null;
@ -461,25 +501,6 @@ in
'';
};
tokenEndpoint = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =
if queueLocal && swarmCfg.authelia.url != null then
"${swarmCfg.authelia.url}/api/oidc/token"
else
null;
defaultText = lib.literalExpression ''"''${swarm.authelia.url}/api/oidc/token" when this host runs both the queue and the IdP, else null'';
example = "https://auth.example.com/api/oidc/token";
description = ''
The swarm IdP's OAuth2 token endpoint. This hive mints a
`client_credentials` access token there and presents it when
connecting to the queue, which authenticates it as
`hive-<hiveName>` — the client
{file}`nix/host-modules/swarm-authelia.nix` already declares for
every entry in {option}`services.hyperhive.swarm.hives`.
'';
};
clientSecretFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =

52
nix/module-eval.nix
View file

@ -159,6 +159,18 @@ let
swarm.grafana.plugins = [ ];
};
# The first slice to leave options on BOTH sides of the split, so the
# fixture sets all three of them through the paths an existing config uses:
# the two movers via their rename entries, `tokenEndpoint` via the path it
# kept. That is also what the all-or-nothing assertion wants, so this hive
# is a valid one rather than one that only evaluates because nothing forced
# the assertion.
statusPublishOldPath = hive {
swarm.statusPublish.natsUrl = "nats://10.0.0.9:4222";
swarm.statusPublish.tokenEndpoint = "https://auth.example.invalid/api/oidc/token";
swarm.statusPublish.clientSecretFile = "/run/secrets/status-client.secret";
};
baoPkcs11 = hive {
deploy.bao.enable = true;
deploy.bao.seal = "pkcs11";
@ -249,9 +261,19 @@ let
# authelia somewhere else, the credential delivered by hand. Whether this
# collector authenticates must follow the credential, never another
# service's placement.
#
# The `swarm.otel.clientSecretFile` below is the PRE-RENAME path. It predates
# the split and is deliberately left spelled that way: it makes this fixture
# the old-path case for that option too, so dropping its rename entry fails
# the eval here rather than only in a real operator's config.
otelRemoteAuthelia = hive {
deploy.swarm-otel.enable = true;
deploy.authelia.enable = false;
# Where that elsewhere IS. Running no IdP does not mean knowing no IdP:
# the authenticator this fixture exists to render puts this address in its
# `token_url`, so a hive with a secret and no URL has a credential it can
# present nowhere.
swarm.authelia.url = "https://auth.example.invalid";
swarm.otel.clientSecretFile = "/var/lib/swarm-otel-oidc/by-hand.secret";
};
@ -376,6 +398,36 @@ let
rule: lib.hasInfix "/run/test-grafana-sock" rule
) grafanaOldPath.systemd.tmpfiles.rules;
}
{
# Reads the daemon's rendered unit, not the options: the queue address
# arrives as an env var whose whole attrset is guarded on `natsUrl`, and
# the secret as a systemd credential. A rename that resolved but stopped
# reaching the module leaves hive-c0re coming up perfectly and reporting
# to nobody, which is the one failure this option set exists to prevent.
# The third arm is the option that did NOT move, read out of the same
# attrset: the guard and the value beside it now come from different
# namespaces, so a hive that renders one and drops the other is exactly
# what a split can silently produce.
name = "a config written against the pre-rename statusPublish paths still reaches the daemon";
ok =
let
u = statusPublishOldPath.systemd.services.hive-c0re;
in
u.environment.HIVE_C0RE_NATS_URL == "nats://10.0.0.9:4222"
&& u.environment.HIVE_C0RE_OIDC_TOKEN_ENDPOINT == "https://auth.example.invalid/api/oidc/token"
&& lib.any (c: lib.hasInfix "/run/secrets/status-client.secret" c) u.serviceConfig.LoadCredential;
}
{
# The collector's half of the same split, and a different arm from the
# authenticator case below: this one reads the PATH the unit loads, so a
# reader left on a source that is non-null but wrong still fails. The
# fixture spells the option its pre-rename way, so it covers the rename
# entry at the same time.
name = "a config written against the pre-rename otel secret path still loads it as a credential";
ok =
lib.any (c: lib.hasInfix "/var/lib/swarm-otel-oidc/by-hand.secret" c)
otelRemoteAuthelia.containers.swarm-otel.config.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential;
}
{
name = "a config written against the pre-rename authelia usersFile still reaches the bridge";
ok =