vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm: default every queue URL to the queue's name on every hive

Browse source 
A remote hive dialled nothing until an operator copied the queue's URL
into it, though the URL is the same string everywhere. statusPublish.natsUrl,
queue.agentNatsUrl and controller.queue.natsUrl now default to
tls://<swarm.nats.domain>:<port> unconditionally.

The statusPublish assertion treated a URL without a secret as a half
config. With the URL a default on every hive, only the secret claims
publishing: the assertion now refuses a secret without a URL or token
endpoint, and hive-c0re's status environment is gated on the secret too,
so a hive without one publishes nothing instead of reading a missing
credential.
This commit is contained in:
atlas 2026-09-24 16:40:26 +02:00 • committed by mara
commit a5259146dc
8 changed files with 220 additions and 129 deletions
Download patch file Download diff file Expand all files Collapse all files

12
nix/host-modules/hive-c0re/default.nix
View file

@ -133,12 +133,12 @@ in
./options.nix
./theme.nix
# Hive-CA trust for this daemon's outbound TLS. Nothing it is given by
# default is https — the forge, matrix and queue URLs all resolve to
# plain http or loopback — so this changes nothing on an all-local
# hive. It matters for the split-host shape the options invite:
# `swarm.matrix.apiUrl`'s own example is `https://matrix.example.com`,
# and pointing it (or `deploy.hive-controller.statusPublish.natsUrl`)
# at another hive's
# default is https — the forge and matrix URLs resolve to plain http or
# loopback, and the queue's TLS is verified against
# `HIVE_C0RE_OIDC_CA_FILE` (../hive-tls.nix), not this bundle — so this
# changes nothing on an all-local hive. It matters for the split-host
# shape the options invite: `swarm.matrix.apiUrl`'s own example is
# `https://matrix.example.com`, and pointing it at another hive's
# gateway means verifying a leaf signed by a CA generated at runtime,
# which no build-time trust store can contain.
#

40
nix/host-modules/hive-c0re/environment.nix
View file

@ -257,24 +257,30 @@ in
# environment is a deployment bug the daemon refuses to treat as
# "no queue configured", because the failure it would otherwise
# produce is a hive that comes up fine and silently never reports.
# The three-option version of that same rule is asserted at eval in
# ./../swarm.nix, so this can only ever emit a complete set.
# The rule that a secret needs the other two options is asserted at
# eval in ./../swarm.nix, so this can only ever emit a complete set.
#
# ⚠️ The guard and the value beside it read different namespaces on
# purpose: where the queue is and where its secret sits are this
# machine's, the token endpoint is the swarm's one address. The
# assertion covers all three, which is what keeps the guard honest.
lib.optionalAttrs (config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl != null) {
HIVE_C0RE_NATS_URL = config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl;
HIVE_C0RE_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
# The identity swarm-authelia.nix already declares for every entry in
# `swarm.hives` — the hive does not choose its own name here, it uses
# the one the roster gave it.
HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}";
# `%d` is systemd's credentials directory — see the LoadCredential in
# ./default.nix. The daemon reads a path, never a value.
HIVE_C0RE_OIDC_CLIENT_SECRET_FILE = "%d/swarm-status-client.secret";
}
# ⚠️ Gated on the SECRET as well as the URL. The URL defaults to the
# queue's name on every hive, so on its own it says where the queue is,
# not that this hive publishes; the secret is that claim. Gated on the
# URL alone, a hive with no secret would be handed a secret path that
# `LoadCredential` in ./default.nix never fills.
lib.optionalAttrs
(
config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl != null
&& config.services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile != null
)
{
HIVE_C0RE_NATS_URL = config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl;
HIVE_C0RE_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
# The identity swarm-authelia.nix already declares for every entry in
# `swarm.hives` — the hive does not choose its own name here, it uses
# the one the roster gave it.
HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}";
# `%d` is systemd's credentials directory — see the LoadCredential in
# ./default.nix. The daemon reads a path, never a value.
HIVE_C0RE_OIDC_CLIENT_SECRET_FILE = "%d/swarm-status-client.secret";
}
// {
# Where ../glue-queue-agent-credential.nix lands the AGENTS' queue
# credential. Read by `hive_c0re::lifecycle::host_config`, which stats the

28
nix/host-modules/local-defaults.nix
View file

@ -86,31 +86,13 @@ in
config.services.hyperhive.deploy.nats.autoGenerateCallout =
lib.mkDefault cfg.deploy.singleHostSwarm;
# ⚠️ Every `swarm.*` default this mode sets lives INSIDE this attrset, not
# as a second `config.services.hyperhive.swarm.…` path beside it — written
# that way the two definitions of `swarm` collide and the nested one is
# silently lost. The gate caught exactly that on the controller's queue URL,
# when this mode still set it: mode on, `natsUrl` still "".
config.services.hyperhive.swarm = {
ca.autoConfigure = lib.mkDefault cfg.deploy.singleHostSwarm;
# The controller's queue coordinates. Kept with the mode, not in the
# options' own `default`, because the controller's minted client secret
# only exists on the host authelia ran its first boot on — so they belong
# to the mode that asserts this box is the whole deployment.
#
# Deriving them from `deploy.nats` / `deploy.authelia`
# inside those defaults is the mixing this file exists to prevent: the
# option would be describing a deployment shape instead of describing
# itself, and "what does all-local turn on?" would stop having one
# answer.
#
# ⚠️ Must live INSIDE this attrset, not as a second
# `config.services.hyperhive.swarm.…` path beside it — written that
# way the two definitions of `swarm` collide and the nested one is
# silently lost. The gate caught exactly that: mode on, `natsUrl`
# still "".
#
# The *requirement* stays in `swarm-controller.nix` as an assertion:
# needing a queue is the controller's own property in every topology,
# and only the convenience is local.
controller.queue.natsUrl = lib.mkIf cfg.deploy.singleHostSwarm (
lib.mkDefault "tls://${config.services.hyperhive.swarm.nats.domain}:${toString config.services.hyperhive.swarm.nats.port}"
);
};
# The controller is asserted by the MODE and by nothing else. Its own

17
nix/host-modules/swarm-controller.nix
View file

@ -17,6 +17,7 @@ let
cfg = config.services.hyperhive.swarm.controller;
deployCfg = config.services.hyperhive.deploy;
autheliaCfg = config.services.hyperhive.swarm.authelia;
natsCfg = config.services.hyperhive.swarm.nats;
# Where the secret store is, and whether this host holds the controller's
# own leaf for it. ⚠️ The controller's pair, NOT `deploy.bao.clientCertFile`
@ -364,18 +365,18 @@ in
queue = {
natsUrl = lib.mkOption {
type = lib.types.str;
default = "";
default = "tls://${natsCfg.domain}:${toString natsCfg.port}";
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}"'';
example = "tls://nats.example.com:4222";
description = ''
Where the controller reaches the swarm queue.
Defaults to the queue's name, which is the same URL on every
host: the queue's own host resolves it locally, and any other
through the operator's DNS.
Empty means unset, which the assertion below refuses — a
controller with no queue is not a lighter controller.
`singleHostSwarm` fills this in with
`tls://<swarm.nats.domain>:<port>`. That derivation lives with the
mode rather than here, so this option describes itself rather than
a deployment shape.
'';
};
@ -764,8 +765,8 @@ in
message = ''
services.hyperhive.swarm.controller.queue.natsUrl is unset.
`singleHostSwarm` fills it in with the queue's name. A controller
in any other deployment has to be told the URL.
It defaults to the queue's name; something in this configuration
set it to "". Remove that setting, or set the URL.
'';
}
{

110
nix/host-modules/swarm.nix
View file

@ -115,16 +115,15 @@ let
lib.filterAttrs (_: hive: hive.certFingerprint != null) swarmCfg.hives
);
# Whether the queue and its minted secret are on THIS host — the two
# coordinates that are a statement about this machine's disk and
# netns, from one condition so a partial set is unrepresentable rather
# than merely detected.
# Whether this hive's minted status secret is on THIS host, which is the
# one status-publishing coordinate that is a statement about this
# machine's disk.
#
# ⚠️ The token endpoint is NOT one of them any more. It is the swarm's
# one address, derived from `swarm.authelia.url` like
# ./swarm-controller.nix's own `queue.tokenEndpoint` already is, so it
# is correct for a remote provider and does not ask where anything
# runs.
# ⚠️ Neither the token endpoint nor the queue's URL is gated on it. Both
# are the swarm's one address: the endpoint derived from
# `swarm.authelia.url`, the queue from its name, which resolves on every
# hive. So they are correct wherever the queue and the IdP run, and do
# not ask where anything runs.
queueLocal = deployCfg.nats.enable && deployCfg.authelia.enable && cfg.hiveName != null;
in
{
@ -373,18 +372,16 @@ in
}
{
# Deliberately an assertion and not a silent "then publish
# nothing": a half-set trio is a config an operator believes is
# working, and its runtime failure mode is the expensive one —
# the daemon comes up fine, never connects, and the hive reads
# `never_reported` on a dashboard nobody is watching yet.
# nothing": a secret without the coordinates it is presented at is
# a config an operator believes is working, and its runtime failure
# mode is the expensive one — the daemon comes up fine, never
# connects, and the hive reads `never_reported` on a dashboard
# nobody is watching yet.
#
# Safe to add to an existing deployment: every `statusPublish`
# default is either all-local or all-null, so no config that
# evaluates today can be caught by this. It also encodes a
# property of the code rather than an intended shape — hive-c0re
# genuinely cannot publish with two of three coordinates — which
# is the distinction the `serviceDomains'` comment at the top of
# this file was written about.
# It encodes a property of the code rather than an intended shape —
# hive-c0re genuinely cannot publish with two of three coordinates —
# which is the distinction the `serviceDomains'` comment at the top
# of this file was written about.
#
# The three coordinates live in two namespaces now: the token
# endpoint is the swarm's one address, the other two are this
@ -392,23 +389,21 @@ in
# operator told only the option names would look for them under
# one prefix and find one of them.
#
# ⚠️ Asymmetric on purpose. A token endpoint on its own is the
# normal state of every hive in a swarm that has an IdP — it says
# where the IdP is, not that this hive publishes. Only the two
# per-host coordinates are a claim to be publishing, and they
# need the endpoint to mean anything.
# ⚠️ Asymmetric on purpose. The token endpoint and the queue's URL
# both default to the swarm's one address on every hive: they say
# where the IdP and the queue are, not that this hive publishes, so
# either without the secret is publishing off rather than a half-set
# trio. Only the client secret is a claim to be publishing, and it
# needs the other two to mean anything.
assertion =
let
local = lib.filter (v: v != null) [
deployCfg.hive-controller.statusPublish.natsUrl
deployCfg.hive-controller.statusPublish.clientSecretFile
];
in
builtins.length local == 0
|| (builtins.length local == 2 && swarmCfg.statusPublish.tokenEndpoint != null);
deployCfg.hive-controller.statusPublish.clientSecretFile == null
|| (
deployCfg.hive-controller.statusPublish.natsUrl != null
&& swarmCfg.statusPublish.tokenEndpoint != null
);
message = ''
This hive's status-publishing coordinates have to be set
together or not at all — it has only some of them.
This hive has a status-publishing client secret but not the
coordinates to publish with it.
Currently:
deploy.hive-controller.statusPublish.natsUrl
@ -419,7 +414,7 @@ in
= ${toString deployCfg.hive-controller.statusPublish.clientSecretFile}
Set the missing ones to publish this hive's status to the
swarm, or set all three to null to turn publishing off.
swarm, or set clientSecretFile to null to turn publishing off.
'';
}
(nameGuards.mustNotEqual {
@ -543,21 +538,22 @@ in
options.services.hyperhive.deploy.hive-controller.statusPublish = {
natsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =
if queueLocal then "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}" else null;
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}" when this host runs the queue and the IdP, else null'';
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}"'';
example = "tls://nats.example.com:4222";
description = ''
Where the swarm queue listens, as seen from *this* hive.
Defaults to the queue's name when this host runs the queue and the
IdP. A hive that is not the swarm host sets the same URL: the name
resolves through the operator's DNS there. TLS only, and by name,
since the queue's certificate carries the name and no address.
Defaults to the queue's name on every hive. The queue's own host
resolves it locally, and any other hive through the operator's
DNS. TLS only, and by name, since the queue's certificate carries
the name and no address.
Null disables status publishing: this hive computes its own
readiness as always, and simply offers it to nobody. The swarm
controller then reports it `never_reported`, which is the honest
A URL alone does not publish.
{option}`services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile`
is what turns publishing on. Without it this hive computes its own
readiness as always and offers it to nobody, and the swarm
controller reports it `never_reported`, which is the honest
reading.
'';
};
@ -579,6 +575,9 @@ in
readable, and one in the environment is readable by anything
that can open {file}`/proc/<pid>/environ`.
Setting it is what makes this hive publish its status; null
turns publishing off.
Defaults to authelia's own minted secret when the IdP runs on
this host. On any other hive the secret has to get here somehow,
and the swarm does not distribute it — copy it out of the swarm
@ -598,25 +597,24 @@ in
# owns `queue.agentCredentialDir` in the same namespace.
options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default =
if queueLocal then "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}" else null;
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}" when this host runs the queue and the IdP, else null'';
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}"'';
example = "tls://nats.example.com:4222";
description = ''
Where the swarm queue listens, as an agent *container* on this host
reaches it.
Defaults to the queue's name when this host runs the queue. The agent
resolves it through the bridge, where dnsmasq answers it with the
bridge address. ⚠️ Never a loopback address: inside an agent's network
namespace `127.0.0.1` is the agent, not this host, and the queue's
certificate names no address anyway.
Defaults to the queue's name on every hive. The agent resolves it
through the bridge: on the queue's own host dnsmasq answers it with
the bridge address, and elsewhere it forwards to the operator's DNS,
as for the store's name. ⚠️ Never a loopback address: inside an
agent's network namespace `127.0.0.1` is the agent, not this host,
and the queue's certificate names no address anyway.
Null means this hive's agents have not been given the queue's address.
Together with
{option}`services.hyperhive.swarm.statusPublish.tokenEndpoint` it is
what decides whether the harness is handed queue coordinates at all; a
hive whose queue is elsewhere names the address its containers route to.
what decides whether the harness is handed queue coordinates at all.
'';
};