vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive status publishing: read the hive's OIDC secret from the store, retry until it is there

Browse source 
hive-c0re publishes this hive's status to the swarm queue as the OIDC
client `hive-<name>`. Its secret came from
`deploy.hive-controller.statusPublish.clientSecretFile`, which defaulted
to authelia's minted file only on a host running both authelia and the
queue. Every other hive had no secret and never published, so the
controller showed it `never_reported`.

Now `swarm-secret-publish`, beside authelia, also copies each hive's
`hive-<name>` secret to `swarm/hives/<hive>/oidc/client`. Its policy
already allows `swarm/hives/*`. hive-c0re reads that path under the
hive's own store leaf, which the existing `swarm/hives/<hive>/*` grant
covers, and holds the value in memory.

The shared queue client no longer remembers a failed first connect for
the life of the process. A failed attempt is retried with a backoff
from 5s, doubling to a cap of 5 minutes; the dashboard banner stays up
until an attempt succeeds. The two status publishers wait for the
client instead of giving up on the first `None`, so a secret published
after boot is picked up without a restart.

Removed: the `statusPublish.clientSecretFile` option (both spellings,
now removed options with a message), its same-host default, its
LoadCredential, the assertion tying it to the other two coordinates,
and `swarm_queue_client::QueueConfig::from_env`, whose only caller was
this read. Publishing is now on wherever the queue URL and token
endpoint are set and the hive holds a store leaf.

Refs #5015
This commit is contained in:
atlas 2026-10-09 23:12:28 +02:00 • committed by mara
commit c2c61b6146
19 changed files with 643 additions and 394 deletions
Download patch file Download diff file Expand all files Collapse all files

128
nix/host-modules/swarm.nix
View file

@ -112,17 +112,6 @@ let
pinnedHives = lib.attrNames (
lib.filterAttrs (_: hive: hive.certFingerprint != null) swarmCfg.hives
);
# Whether this hive's minted status secret is on THIS host, which is the
# one status-publishing coordinate that is a statement about this
# machine's disk.
#
# ⚠️ Neither the token endpoint nor the queue's URL is gated on it. Both
# are the swarm's one address: the endpoint derived from
# `swarm.authelia.url`, the queue from its name, which resolves on every
# hive. So they are correct wherever the queue and the IdP run, and do
# not ask where anything runs.
queueLocal = deployCfg.nats.enable && deployCfg.authelia.enable && cfg.hiveName != null;
in
{
options.services.hyperhive.swarm.hives = lib.mkOption {
@ -379,53 +368,6 @@ in
designed for it.
'';
}
{
# Deliberately an assertion and not a silent "then publish
# nothing": a secret without the coordinates it is presented at is
# a config an operator believes is working, and its runtime failure
# mode is the expensive one — the daemon comes up fine, never
# connects, and the hive reads `never_reported` on a dashboard
# nobody is watching yet.
#
# It encodes a property of the code rather than an intended shape —
# hive-c0re genuinely cannot publish with two of three coordinates —
# which is the distinction the `serviceDomains'` comment at the top
# of this file was written about.
#
# The three coordinates live in two namespaces now: the token
# endpoint is the swarm's one address, the other two are this
# machine's. The message spells all three paths out, because an
# operator told only the option names would look for them under
# one prefix and find one of them.
#
# ⚠️ Asymmetric on purpose. The token endpoint and the queue's URL
# both default to the swarm's one address on every hive: they say
# where the IdP and the queue are, not that this hive publishes, so
# either without the secret is publishing off rather than a half-set
# trio. Only the client secret is a claim to be publishing, and it
# needs the other two to mean anything.
assertion =
deployCfg.hive-controller.statusPublish.clientSecretFile == null
|| (
deployCfg.hive-controller.statusPublish.natsUrl != null
&& swarmCfg.statusPublish.tokenEndpoint != null
);
message = ''
This hive has a status-publishing client secret but not the
coordinates to publish with it.
Currently:
deploy.hive-controller.statusPublish.natsUrl
= ${toString deployCfg.hive-controller.statusPublish.natsUrl}
swarm.statusPublish.tokenEndpoint
= ${toString swarmCfg.statusPublish.tokenEndpoint}
deploy.hive-controller.statusPublish.clientSecretFile
= ${toString deployCfg.hive-controller.statusPublish.clientSecretFile}
Set the missing ones to publish this hive's status to the
swarm, or set clientSecretFile to null to turn publishing off.
'';
}
(nameGuards.mustNotEqual {
option = "services.hyperhive.swarm.hives";
names = hiveNames;
@ -508,13 +450,14 @@ in
};
# How this hive reaches the swarm queue to offer its own status
# (hive-c0re's `swarm_status`). Three coordinates: the token endpoint
# below follows the swarm's IdP address wherever that is, and the two
# under `deploy.hive-controller` default from the local swarm services
# when this host runs them and are set by hand otherwise.
# (hive-c0re's `swarm_status`): the token endpoint below, which follows
# the swarm's IdP address wherever that is, and the queue's URL under
# `deploy.hive-controller`. The client secret is not an option: hive-c0re
# reads it from the swarm's secret store under this hive's own
# certificate (`deploy.bao.clientCertFile`).
#
# There is no `enable`: three coordinates that are all set is the
# enable. An extra flag would let a hive be configured-but-off, which
# There is no `enable`: both coordinates set, plus a store identity, is
# the enable. An extra flag would let a hive be configured-but-off, which
# is one more state to explain and one more way to be silently quiet.
options.services.hyperhive.swarm.statusPublish = {
tokenEndpoint = lib.mkOption {
@ -534,16 +477,10 @@ in
};
# What stays above is the one thing every hive shares: the address of the
# swarm's token endpoint. The other two coordinates are this machine's —
# where the queue is *as seen from here*, and where its secret sits on this
# disk — so they hang off the daemon that uses them. `hive-controller` is
# the hive-c0re daemon's namespace, hence these live there rather than
# under a `deploy.statusPublish` of their own.
#
# ⚠️ The all-or-nothing assertion above now spans both namespaces. That is
# one service's own options either side of the split, not a service reaching
# into a foreign `deploy.*`; its message names all three paths so an operator
# is not told to set two options under a path that has one.
# swarm's token endpoint. The queue's URL is this machine's — where the
# queue is *as seen from here* — so it hangs off the daemon that uses it.
# `hive-controller` is the hive-c0re daemon's namespace, hence it lives
# there rather than under a `deploy.statusPublish` of its own.
options.services.hyperhive.deploy.hive-controller.statusPublish = {
natsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
@ -558,41 +495,14 @@ in
DNS. TLS only, and by name, since the queue's certificate carries
the name and no address.
A URL alone does not publish.
{option}`services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile`
is what turns publishing on. Without it this hive computes its own
readiness as always and offers it to nobody, and the swarm
controller reports it `never_reported`, which is the honest
reading.
'';
};
clientSecretFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =
if queueLocal then
"${deployCfg.authelia.hostClientSecretDir}/hive-${cfg.hiveName}.secret"
else
null;
defaultText = lib.literalExpression ''"''${deploy.authelia.hostClientSecretDir}/hive-''${hiveName}.secret" when this host runs both the queue and the IdP, else null'';
example = "/var/lib/secrets/swarm-queue-client.secret";
description = ''
Path to a file holding the plaintext client secret for this
hive's `hive-<hiveName>` identity.
A path and not a value: a secret in the Nix store is world
readable, and one in the environment is readable by anything
that can open {file}`/proc/<pid>/environ`.
Setting it is what makes this hive publish its status; null
turns publishing off.
Defaults to authelia's own minted secret when the IdP runs on
this host. On any other hive the secret has to get here somehow,
and the swarm does not distribute it — copy it out of the swarm
host's
{option}`services.hyperhive.deploy.authelia.hostClientSecretDir`
with whatever secret management this deployment already uses.
A URL alone does not publish. This hive also needs its own store
identity ({option}`services.hyperhive.deploy.bao.clientCertFile`),
under which hive-c0re reads its `hive-<hiveName>` client secret
from the swarm's secret store; `swarm-secret-publish`, on the host
that runs authelia, puts it there. Without an identity, or with
this set to null, this hive computes its own readiness as always
and offers it to nobody, and the swarm controller reports it
`never_reported`, which is the honest reading.
'';
};
};