vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm: default every queue URL to the queue's name on every hive

Browse source 
A remote hive dialled nothing until an operator copied the queue's URL
into it, though the URL is the same string everywhere. statusPublish.natsUrl,
queue.agentNatsUrl and controller.queue.natsUrl now default to
tls://<swarm.nats.domain>:<port> unconditionally.

The statusPublish assertion treated a URL without a secret as a half
config. With the URL a default on every hive, only the secret claims
publishing: the assertion now refuses a secret without a URL or token
endpoint, and hive-c0re's status environment is gated on the secret too,
so a hive without one publishes nothing instead of reading a missing
credential.
This commit is contained in:
atlas 2026-09-24 16:40:26 +02:00 • committed by mara
commit a5259146dc
8 changed files with 220 additions and 129 deletions
Download patch file Download diff file Expand all files Collapse all files

110
nix/host-modules/swarm.nix
View file

@ -115,16 +115,15 @@ let
lib.filterAttrs (_: hive: hive.certFingerprint != null) swarmCfg.hives
);
# Whether the queue and its minted secret are on THIS host — the two
# coordinates that are a statement about this machine's disk and
# netns, from one condition so a partial set is unrepresentable rather
# than merely detected.
# Whether this hive's minted status secret is on THIS host, which is the
# one status-publishing coordinate that is a statement about this
# machine's disk.
#
# ⚠️ The token endpoint is NOT one of them any more. It is the swarm's
# one address, derived from `swarm.authelia.url` like
# ./swarm-controller.nix's own `queue.tokenEndpoint` already is, so it
# is correct for a remote provider and does not ask where anything
# runs.
# ⚠️ Neither the token endpoint nor the queue's URL is gated on it. Both
# are the swarm's one address: the endpoint derived from
# `swarm.authelia.url`, the queue from its name, which resolves on every
# hive. So they are correct wherever the queue and the IdP run, and do
# not ask where anything runs.
queueLocal = deployCfg.nats.enable && deployCfg.authelia.enable && cfg.hiveName != null;
in
{
@ -373,18 +372,16 @@ in
}
{
# Deliberately an assertion and not a silent "then publish
# nothing": a half-set trio is a config an operator believes is
# working, and its runtime failure mode is the expensive one —
# the daemon comes up fine, never connects, and the hive reads
# `never_reported` on a dashboard nobody is watching yet.
# nothing": a secret without the coordinates it is presented at is
# a config an operator believes is working, and its runtime failure
# mode is the expensive one — the daemon comes up fine, never
# connects, and the hive reads `never_reported` on a dashboard
# nobody is watching yet.
#
# Safe to add to an existing deployment: every `statusPublish`
# default is either all-local or all-null, so no config that
# evaluates today can be caught by this. It also encodes a
# property of the code rather than an intended shape — hive-c0re
# genuinely cannot publish with two of three coordinates — which
# is the distinction the `serviceDomains'` comment at the top of
# this file was written about.
# It encodes a property of the code rather than an intended shape —
# hive-c0re genuinely cannot publish with two of three coordinates —
# which is the distinction the `serviceDomains'` comment at the top
# of this file was written about.
#
# The three coordinates live in two namespaces now: the token
# endpoint is the swarm's one address, the other two are this
@ -392,23 +389,21 @@ in
# operator told only the option names would look for them under
# one prefix and find one of them.
#
# ⚠️ Asymmetric on purpose. A token endpoint on its own is the
# normal state of every hive in a swarm that has an IdP — it says
# where the IdP is, not that this hive publishes. Only the two
# per-host coordinates are a claim to be publishing, and they
# need the endpoint to mean anything.
# ⚠️ Asymmetric on purpose. The token endpoint and the queue's URL
# both default to the swarm's one address on every hive: they say
# where the IdP and the queue are, not that this hive publishes, so
# either without the secret is publishing off rather than a half-set
# trio. Only the client secret is a claim to be publishing, and it
# needs the other two to mean anything.
assertion =
let
local = lib.filter (v: v != null) [
deployCfg.hive-controller.statusPublish.natsUrl
deployCfg.hive-controller.statusPublish.clientSecretFile
];
in
builtins.length local == 0
|| (builtins.length local == 2 && swarmCfg.statusPublish.tokenEndpoint != null);
deployCfg.hive-controller.statusPublish.clientSecretFile == null
|| (
deployCfg.hive-controller.statusPublish.natsUrl != null
&& swarmCfg.statusPublish.tokenEndpoint != null
);
message = ''
This hive's status-publishing coordinates have to be set
together or not at all — it has only some of them.
This hive has a status-publishing client secret but not the
coordinates to publish with it.
Currently:
deploy.hive-controller.statusPublish.natsUrl
@ -419,7 +414,7 @@ in
= ${toString deployCfg.hive-controller.statusPublish.clientSecretFile}
Set the missing ones to publish this hive's status to the
swarm, or set all three to null to turn publishing off.
swarm, or set clientSecretFile to null to turn publishing off.
'';
}
(nameGuards.mustNotEqual {
@ -543,21 +538,22 @@ in
options.services.hyperhive.deploy.hive-controller.statusPublish = {
natsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =
if queueLocal then "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}" else null;
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}" when this host runs the queue and the IdP, else null'';
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}"'';
example = "tls://nats.example.com:4222";
description = ''
Where the swarm queue listens, as seen from *this* hive.
Defaults to the queue's name when this host runs the queue and the
IdP. A hive that is not the swarm host sets the same URL: the name
resolves through the operator's DNS there. TLS only, and by name,
since the queue's certificate carries the name and no address.
Defaults to the queue's name on every hive. The queue's own host
resolves it locally, and any other hive through the operator's
DNS. TLS only, and by name, since the queue's certificate carries
the name and no address.
Null disables status publishing: this hive computes its own
readiness as always, and simply offers it to nobody. The swarm
controller then reports it `never_reported`, which is the honest
A URL alone does not publish.
{option}`services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile`
is what turns publishing on. Without it this hive computes its own
readiness as always and offers it to nobody, and the swarm
controller reports it `never_reported`, which is the honest
reading.
'';
};
@ -579,6 +575,9 @@ in
readable, and one in the environment is readable by anything
that can open {file}`/proc/<pid>/environ`.
Setting it is what makes this hive publish its status; null
turns publishing off.
Defaults to authelia's own minted secret when the IdP runs on
this host. On any other hive the secret has to get here somehow,
and the swarm does not distribute it — copy it out of the swarm
@ -598,25 +597,24 @@ in
# owns `queue.agentCredentialDir` in the same namespace.
options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =
if queueLocal then "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}" else null;
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}" when this host runs the queue and the IdP, else null'';
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}"'';
example = "tls://nats.example.com:4222";
description = ''
Where the swarm queue listens, as an agent *container* on this host
reaches it.
Defaults to the queue's name when this host runs the queue. The agent
resolves it through the bridge, where dnsmasq answers it with the
bridge address. ⚠️ Never a loopback address: inside an agent's network
namespace `127.0.0.1` is the agent, not this host, and the queue's
certificate names no address anyway.
Defaults to the queue's name on every hive. The agent resolves it
through the bridge: on the queue's own host dnsmasq answers it with
the bridge address, and elsewhere it forwards to the operator's DNS,
as for the store's name. ⚠️ Never a loopback address: inside an
agent's network namespace `127.0.0.1` is the agent, not this host,
and the queue's certificate names no address anyway.
Null means this hive's agents have not been given the queue's address.
Together with
{option}`services.hyperhive.swarm.statusPublish.tokenEndpoint` it is
what decides whether the harness is handed queue coordinates at all; a
hive whose queue is elsewhere names the address its containers route to.
what decides whether the harness is handed queue coordinates at all.
'';
};