vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive status publishing: read the hive's OIDC secret from the store, retry until it is there

Browse source 
hive-c0re publishes this hive's status to the swarm queue as the OIDC
client `hive-<name>`. Its secret came from
`deploy.hive-controller.statusPublish.clientSecretFile`, which defaulted
to authelia's minted file only on a host running both authelia and the
queue. Every other hive had no secret and never published, so the
controller showed it `never_reported`.

Now `swarm-secret-publish`, beside authelia, also copies each hive's
`hive-<name>` secret to `swarm/hives/<hive>/oidc/client`. Its policy
already allows `swarm/hives/*`. hive-c0re reads that path under the
hive's own store leaf, which the existing `swarm/hives/<hive>/*` grant
covers, and holds the value in memory.

The shared queue client no longer remembers a failed first connect for
the life of the process. A failed attempt is retried with a backoff
from 5s, doubling to a cap of 5 minutes; the dashboard banner stays up
until an attempt succeeds. The two status publishers wait for the
client instead of giving up on the first `None`, so a secret published
after boot is picked up without a restart.

Removed: the `statusPublish.clientSecretFile` option (both spellings,
now removed options with a message), its same-host default, its
LoadCredential, the assertion tying it to the other two coordinates,
and `swarm_queue_client::QueueConfig::from_env`, whose only caller was
this read. Publishing is now on wherever the queue URL and token
endpoint are set and the hive holds a store leaf.

Refs #5015
This commit is contained in:
atlas 2026-10-09 23:12:28 +02:00 • committed by mara
commit c2c61b6146
19 changed files with 643 additions and 394 deletions
Download patch file Download diff file Expand all files Collapse all files

35
nix/host-modules/deploy.nix
View file

@ -32,6 +32,14 @@
let
deployCfg = config.services.hyperhive.deploy;
# Shared by the two removed spellings of the status-publishing secret below.
statusSecretRemoved = ''
hive-c0re reads this hive's status-publishing client secret from the
swarm's secret store, under the hive's own certificate
(deploy.bao.clientCertFile), and swarm-secret-publish on the host that runs
authelia puts it there. Remove this definition; the file it named is read
by nothing now and can be deleted.
'';
in
{
imports = [
@ -115,19 +123,28 @@ in
[ "services" "hyperhive" "swarm" "nats" "enable" ]
[ "services" "hyperhive" "deploy" "nats" "enable" ]
)
# Two of statusPublish's three coordinates. `tokenEndpoint` stays under
# `swarm.*`: it is the swarm IdP's one address, identical on every hive,
# where these two are "the queue as seen from here" and "the secret on
# this disk". The all-or-nothing assertion in ./swarm.nix spans the split
# deliberately and names all three paths in its message.
# `tokenEndpoint` stays under `swarm.*`: it is the swarm IdP's one
# address, identical on every hive, where this is "the queue as seen from
# here".
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "statusPublish" "natsUrl" ]
[ "services" "hyperhive" "deploy" "hive-controller" "statusPublish" "natsUrl" ]
)
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "statusPublish" "clientSecretFile" ]
[ "services" "hyperhive" "deploy" "hive-controller" "statusPublish" "clientSecretFile" ]
)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"swarm"
"statusPublish"
"clientSecretFile"
] statusSecretRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"hive-controller"
"statusPublish"
"clientSecretFile"
] statusSecretRemoved)
# Both old spellings of "run a hive here" forward to the one switch.
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "enable" ]

13
nix/host-modules/glue-queue-agent-credential.nix
View file

@ -5,10 +5,10 @@
# value there" into a value.
#
# ⚠️ Two files, not one, and that is the consumer's shape rather than a
# preference. `swarm_queue_client::QueueConfig::from_env` takes the secret as
# a PATH (`<prefix>_OIDC_CLIENT_SECRET_FILE`) and the client id as a VALUE
# (`<prefix>_OIDC_CLIENT_ID`), so splitting them here is what lets the next
# slice hand both to an agent without parsing anything.
# preference. `swarm-logs` inside the agent reads the secret and the client id
# each from its own file (`<prefix>_OIDC_CLIENT_SECRET_FILE`,
# `<prefix>_OIDC_CLIENT_ID_FILE`), so splitting them here is what lets
# hive-c0re hand both to an agent without parsing anything.
#
# ⚠️ An ABSENT file means this hive's agents do not connect to the queue, and
# that is correct rather than degraded. The publisher runs on the authelia
@ -259,9 +259,8 @@ in
exit 0
fi
# Both or neither, for the reason `QueueConfig::from_env` refuses a
# half-set environment: a client that finds one of the two comes up
# "fine" and never connects.
# Both or neither: a client that finds one of the two comes up "fine"
# and never connects.
if [ -z "$secret" ] || [ -z "$client_id" ]; then
echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2
exit 0

24
nix/host-modules/hive-c0re/default.nix
View file

@ -316,26 +316,18 @@ in
# the daemon as well would put a secret on a process that has nowhere
# to present it.
LoadCredential =
# The swarm-queue client secret this hive authenticates with to
# publish its own status. `LoadCredential` and not a copy: root
# reads the plaintext at unit start and hive-core sees it 0400
# under `%d`, so the secret never gains a second on-disk copy
# and the daemon never needs read access to wherever it lives.
# (The callout responder copies instead only because it
# delivers into a container, across a filesystem boundary.)
lib.optional
(config.services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile != null)
"swarm-status-client.secret:${config.services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile}"
# The secret store's client identity, on the same reasoning one
# paragraph up — with a sharper edge: ./glue-bao-tls.nix mints the
# key `0600` inside a `0700` directory, so hive-core cannot read it
# at all. Loading it as a credential is what makes the store
# reachable from an unprivileged daemon without widening either.
# The secret store's client identity. `LoadCredential` and not a
# copy: root reads it at unit start and hive-core sees it 0400
# under `%d`, so it never gains a second on-disk copy. And
# ./glue-bao-tls.nix mints the key `0600` inside a `0700`
# directory, so hive-core cannot read the original at all.
# Loading it as a credential is what makes the store reachable
# from an unprivileged daemon without widening either.
# The gate is the identity, never `deploy.bao.enable`: a hive that
# reads a store on another machine holds one of these and runs no
# store. ./glue-queue-agent-credential.nix gates its own reader the
# same way.
++ lib.optionals baoClientIdentity [
lib.optionals baoClientIdentity [
"bao-client.pem:${baoDeploy.clientCertFile}"
"bao-client-key.pem:${baoDeploy.clientKeyFile}"
]

19
nix/host-modules/hive-c0re/environment.nix
View file

@ -223,22 +223,20 @@ in
}
//
# Swarm-queue coordinates for offering this hive's status upward
# (hive-c0re::swarm_status). All four together or none: a half-set
# (hive-c0re::swarm_status). All three together or none: a half-set
# environment is a deployment bug the daemon refuses to treat as
# "no queue configured", because the failure it would otherwise
# produce is a hive that comes up fine and silently never reports.
# The rule that a secret needs the other two options is asserted at
# eval in ./../swarm.nix, so this can only ever emit a complete set.
#
# ⚠️ Gated on the SECRET as well as the URL. The URL defaults to the
# queue's name on every hive, so on its own it says where the queue is,
# not that this hive publishes; the secret is that claim. Gated on the
# URL alone, a hive with no secret would be handed a secret path that
# `LoadCredential` in ./default.nix never fills.
# The client secret is not among them: the daemon reads it from the
# swarm's secret store under this hive's own certificate
# (`swarm_secret_client::queue::hive_client_path`), so a hive with no
# store identity has no way to publish and is handed nothing.
lib.optionalAttrs
(
config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl != null
&& config.services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile != null
&& config.services.hyperhive.swarm.statusPublish.tokenEndpoint != null
&& haveBaoClientIdentity
)
{
HIVE_C0RE_NATS_URL = config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl;
@ -247,9 +245,6 @@ in
# `swarm.hives` — the hive does not choose its own name here, it uses
# the one the roster gave it.
HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}";
# `%d` is systemd's credentials directory — see the LoadCredential in
# ./default.nix. The daemon reads a path, never a value.
HIVE_C0RE_OIDC_CLIENT_SECRET_FILE = "%d/swarm-status-client.secret";
}
// {
# Where ../glue-queue-agent-credential.nix lands the AGENTS' queue

35
nix/host-modules/swarm-secret-publisher.nix
View file

@ -1,7 +1,7 @@
# The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its agents' credential, a
# service or the controller its own. It mints nothing: every value is an OIDC
# client secret authelia minted, and this unit copies it.
# whoever needs one can read it there: a hive its own and its agents'
# credentials, a service or the controller its own. It mints nothing: every
# value is an OIDC client secret authelia minted, and this unit copies it.
#
# ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
# into the store even when the service runs beside authelia, because its
@ -57,6 +57,10 @@ let
hive:
"${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}${hyperhiveCfg.swarm.authelia.agentClientSuffix}";
# The hive's own client id, which hive-c0re publishes the hive's status with.
# Composed as ./swarm-authelia.nix composes it, for the reason above.
hiveClientId = hive: "${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}";
# The swarm's own services, as opposed to its hives. One client for the whole
# swarm rather than one per hive, so one value in the store rather than a copy
# each: `swarm/services/<id>/oidc/client`.
@ -100,8 +104,9 @@ in
description = ''
Publish the OIDC client secrets this host mints into the swarm's
secret store, so a hive that does not run authelia can read its
agents' credential and a swarm service or the controller can read its
own — from wherever it runs, this host included.
own and its agents' credentials and a swarm service or the
controller can read its own — from wherever it runs, this host
included.
Defaults to whether this host mints the OIDC secrets, which is the
only half of the question that is a property of *this* host.
@ -114,8 +119,9 @@ in
identity's job (see `baoClientCertFile`), not this option's.
Turning it off leaves every hive but this one without its agents'
credential, the swarm controller unable to start, the swarm's Grafana
without any login at all, its collector pushing unauthenticated, the
credential, every hive unable to publish its status, the swarm
controller unable to start, the swarm's Grafana without any login at
all, its collector pushing unauthenticated, the
queue's responder unable to start, and the homeserver unable to start.
Each secret has exactly one route and this is the producer's
end of it, so the honest reason to set it false is a deployment
@ -223,6 +229,21 @@ in
fi
'') hiveNames}
# Each hive's own client, which that hive's hive-c0re reads under its
# own certificate. The nix half of
# `swarm_secret_client::queue::hive_client_path`.
${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${hiveClientId hive}.secret"}
if [ -s "$src" ]; then
bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/oidc/client"} \
value=@"$src"
published=$((published + 1))
else
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
'') hiveNames}
${lib.concatMapStringsSep "\n" (id: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${id}.secret"}
if [ -s "$src" ]; then

128
nix/host-modules/swarm.nix
View file

@ -112,17 +112,6 @@ let
pinnedHives = lib.attrNames (
lib.filterAttrs (_: hive: hive.certFingerprint != null) swarmCfg.hives
);
# Whether this hive's minted status secret is on THIS host, which is the
# one status-publishing coordinate that is a statement about this
# machine's disk.
#
# ⚠️ Neither the token endpoint nor the queue's URL is gated on it. Both
# are the swarm's one address: the endpoint derived from
# `swarm.authelia.url`, the queue from its name, which resolves on every
# hive. So they are correct wherever the queue and the IdP run, and do
# not ask where anything runs.
queueLocal = deployCfg.nats.enable && deployCfg.authelia.enable && cfg.hiveName != null;
in
{
options.services.hyperhive.swarm.hives = lib.mkOption {
@ -379,53 +368,6 @@ in
designed for it.
'';
}
{
# Deliberately an assertion and not a silent "then publish
# nothing": a secret without the coordinates it is presented at is
# a config an operator believes is working, and its runtime failure
# mode is the expensive one — the daemon comes up fine, never
# connects, and the hive reads `never_reported` on a dashboard
# nobody is watching yet.
#
# It encodes a property of the code rather than an intended shape —
# hive-c0re genuinely cannot publish with two of three coordinates —
# which is the distinction the `serviceDomains'` comment at the top
# of this file was written about.
#
# The three coordinates live in two namespaces now: the token
# endpoint is the swarm's one address, the other two are this
# machine's. The message spells all three paths out, because an
# operator told only the option names would look for them under
# one prefix and find one of them.
#
# ⚠️ Asymmetric on purpose. The token endpoint and the queue's URL
# both default to the swarm's one address on every hive: they say
# where the IdP and the queue are, not that this hive publishes, so
# either without the secret is publishing off rather than a half-set
# trio. Only the client secret is a claim to be publishing, and it
# needs the other two to mean anything.
assertion =
deployCfg.hive-controller.statusPublish.clientSecretFile == null
|| (
deployCfg.hive-controller.statusPublish.natsUrl != null
&& swarmCfg.statusPublish.tokenEndpoint != null
);
message = ''
This hive has a status-publishing client secret but not the
coordinates to publish with it.
Currently:
deploy.hive-controller.statusPublish.natsUrl
= ${toString deployCfg.hive-controller.statusPublish.natsUrl}
swarm.statusPublish.tokenEndpoint
= ${toString swarmCfg.statusPublish.tokenEndpoint}
deploy.hive-controller.statusPublish.clientSecretFile
= ${toString deployCfg.hive-controller.statusPublish.clientSecretFile}
Set the missing ones to publish this hive's status to the
swarm, or set clientSecretFile to null to turn publishing off.
'';
}
(nameGuards.mustNotEqual {
option = "services.hyperhive.swarm.hives";
names = hiveNames;
@ -508,13 +450,14 @@ in
};
# How this hive reaches the swarm queue to offer its own status
# (hive-c0re's `swarm_status`). Three coordinates: the token endpoint
# below follows the swarm's IdP address wherever that is, and the two
# under `deploy.hive-controller` default from the local swarm services
# when this host runs them and are set by hand otherwise.
# (hive-c0re's `swarm_status`): the token endpoint below, which follows
# the swarm's IdP address wherever that is, and the queue's URL under
# `deploy.hive-controller`. The client secret is not an option: hive-c0re
# reads it from the swarm's secret store under this hive's own
# certificate (`deploy.bao.clientCertFile`).
#
# There is no `enable`: three coordinates that are all set is the
# enable. An extra flag would let a hive be configured-but-off, which
# There is no `enable`: both coordinates set, plus a store identity, is
# the enable. An extra flag would let a hive be configured-but-off, which
# is one more state to explain and one more way to be silently quiet.
options.services.hyperhive.swarm.statusPublish = {
tokenEndpoint = lib.mkOption {
@ -534,16 +477,10 @@ in
};
# What stays above is the one thing every hive shares: the address of the
# swarm's token endpoint. The other two coordinates are this machine's —
# where the queue is *as seen from here*, and where its secret sits on this
# disk — so they hang off the daemon that uses them. `hive-controller` is
# the hive-c0re daemon's namespace, hence these live there rather than
# under a `deploy.statusPublish` of their own.
#
# ⚠️ The all-or-nothing assertion above now spans both namespaces. That is
# one service's own options either side of the split, not a service reaching
# into a foreign `deploy.*`; its message names all three paths so an operator
# is not told to set two options under a path that has one.
# swarm's token endpoint. The queue's URL is this machine's — where the
# queue is *as seen from here* — so it hangs off the daemon that uses it.
# `hive-controller` is the hive-c0re daemon's namespace, hence it lives
# there rather than under a `deploy.statusPublish` of its own.
options.services.hyperhive.deploy.hive-controller.statusPublish = {
natsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
@ -558,41 +495,14 @@ in
DNS. TLS only, and by name, since the queue's certificate carries
the name and no address.
A URL alone does not publish.
{option}`services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile`
is what turns publishing on. Without it this hive computes its own
readiness as always and offers it to nobody, and the swarm
controller reports it `never_reported`, which is the honest
reading.
'';
};
clientSecretFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =
if queueLocal then
"${deployCfg.authelia.hostClientSecretDir}/hive-${cfg.hiveName}.secret"
else
null;
defaultText = lib.literalExpression ''"''${deploy.authelia.hostClientSecretDir}/hive-''${hiveName}.secret" when this host runs both the queue and the IdP, else null'';
example = "/var/lib/secrets/swarm-queue-client.secret";
description = ''
Path to a file holding the plaintext client secret for this
hive's `hive-<hiveName>` identity.
A path and not a value: a secret in the Nix store is world
readable, and one in the environment is readable by anything
that can open {file}`/proc/<pid>/environ`.
Setting it is what makes this hive publish its status; null
turns publishing off.
Defaults to authelia's own minted secret when the IdP runs on
this host. On any other hive the secret has to get here somehow,
and the swarm does not distribute it — copy it out of the swarm
host's
{option}`services.hyperhive.deploy.authelia.hostClientSecretDir`
with whatever secret management this deployment already uses.
A URL alone does not publish. This hive also needs its own store
identity ({option}`services.hyperhive.deploy.bao.clientCertFile`),
under which hive-c0re reads its `hive-<hiveName>` client secret
from the swarm's secret store; `swarm-secret-publish`, on the host
that runs authelia, puts it there. Without an identity, or with
this set to null, this hive computes its own readiness as always
and offers it to nobody, and the swarm controller reports it
`never_reported`, which is the honest reading.
'';
};
};