vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(#3343): let the swarm controller reach a queue it is not co-located with

Browse source 
The controller's queue coordinates were gated on `autheliaCfg.enable &&
natsCfg.enable` -- i.e. on this host running both. Split the hosts and
the whole env block vanished, which the daemon's own rule reads as "no
queue configured": it started cleanly, served /api/hives, and silently
never connected or published.

Make the coordinates options instead, defaulted to the co-located values
so nobody types a path they didn't need, and assert each one so a split
deployment fails at eval naming the option rather than running blind.
The queue is required for a controller -- /api/hives/status reads the KV
-- but co-location with it is not.
This commit is contained in:
atlas 2026-08-16 19:19:26 +02:00
commit 92025e01de
1 changed files with 148 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

182
nix/host-modules/swarm-controller.nix
View file

@ -43,39 +43,43 @@ let
# the swarm has one.
queueClientId = "swarm-controller";
# Both halves have to be here: authelia to have minted the secret, and the
# queue to connect to. Same guard, and the same reasoning, as `autheliaEnv`
# above — a value set on a host that runs neither would point at a file
# that does not exist and produce a daemon that retries forever.
queueLocal = autheliaCfg.enable && natsCfg.enable;
# `LoadCredential` and not a copy-oneshot, which is where this deliberately
# differs from the callout responder: that one delivers INTO a container,
# so it has to copy across a filesystem boundary. The controller is a plain
# host unit, so systemd can hand it the file directly — fewer moving parts,
# and the secret never gains a second on-disk copy to forget about.
queueEnv = lib.optionalAttrs queueLocal {
# The queue container shares the host netns, so loopback is correct here
# and is not the `localhost`-means-the-wrong-thing trap that applies
# inside agent containers.
SWARM_CONTROLLER_NATS_URL = "nats://127.0.0.1:${toString natsCfg.port}";
SWARM_CONTROLLER_OIDC_TOKEN_ENDPOINT = "${autheliaCfg.url}/api/oidc/token";
#
# Every value here comes from an option rather than from what happens to
# run on this host. The queue is not optional for a controller, but
# *co-location with it* is — "controller needs queue, but it may not run on
# same host if configured properly by operator" is the rule. The
# defaults below cover the co-located case so nobody types a path they
# didn't need to; the assertions cover the split case so nobody gets a
# daemon that silently never connects.
queueEnv = {
SWARM_CONTROLLER_NATS_URL = cfg.queue.natsUrl;
SWARM_CONTROLLER_OIDC_TOKEN_ENDPOINT = cfg.queue.tokenEndpoint;
SWARM_CONTROLLER_OIDC_CLIENT_ID = queueClientId;
# `%d` is systemd's credentials directory: root reads the plaintext at
# unit start and the daemon's own user sees it 0400, without the unit
# ever being able to read the rest of authelia's state dir.
# ever being able to read the rest of whatever directory the secret
# came from.
SWARM_CONTROLLER_OIDC_CLIENT_SECRET_FILE = "%d/queue-client.secret";
};
# Independent of `queueEnv`/`queueLocal` on purpose — the queue
# coordinates' own co-location guard is narrower than it looks (it
# silently drops the controller's identity on a host that splits
# authelia/NATS from swarm-controller; a known gap tracked separately).
# Forge access has nothing to do with whether authelia+NATS happen to be
# local, only with whether `cfg.forgeTokenFile` resolves to a real file —
# which `forgeTokenFile`'s own default already handles (forge-local vs.
# operator-copied). Gating a second time here would just repeat that
# option's own logic under a different name.
# Independent of `queueEnv` on purpose, and now for a simpler reason
# than when this was written: the queue's own co-location guard
# (`queueLocal`) is gone — its coordinates are options with assertions,
# so a split-host controller fails at eval instead of silently losing
# its identity. Forge access still has nothing to do with where authelia
# and NATS run, only with whether `cfg.forgeTokenFile` resolves to a real
# file — which `forgeTokenFile`'s own default already handles
# (forge-local vs. operator-copied). Gating a second time here would
# just repeat that option's own logic under a different name.
#
# ⚠️ The asymmetry is deliberate: the queue is REQUIRED for a controller
# and the forge is not, so absence is an error for one and a supported
# shape for the other.
forgeEnv = lib.optionalAttrs (cfg.forgeTokenFile != null) {
SWARM_CONTROLLER_FORGE_URL = "https://${forgeCfg.domain}";
# Same `%d` shape as the queue secret above — root reads the plaintext
@ -182,6 +186,63 @@ in
'';
};
queue = {
natsUrl = lib.mkOption {
type = lib.types.str;
default = lib.optionalString natsCfg.enable "nats://127.0.0.1:${toString natsCfg.port}";
defaultText = lib.literalExpression ''
if swarm.nats.enable then "nats://127.0.0.1:''${toString swarm.nats.port}" else ""
'';
example = "nats://queue.example.com:4222";
description = ''
Where the controller reaches the swarm queue.
Defaults to loopback when this host runs the queue — the queue
container shares the host netns, so that address is correct
here and is not the `localhost`-means-the-agent trap that
applies inside agent containers. Set it explicitly when the
queue lives elsewhere.
'';
};
tokenEndpoint = lib.mkOption {
type = lib.types.str;
default = lib.optionalString (autheliaCfg.url != null) "${autheliaCfg.url}/api/oidc/token";
defaultText = lib.literalExpression ''"''${swarm.authelia.url}/api/oidc/token"'';
description = ''
The OIDC token endpoint the controller mints its own access
token from.
Derived from `swarm.authelia.url`, which is the half of the
authelia module that exists on every host — so this is already
correct for a remote provider as long as that URL is set.
'';
};
clientSecretFile = lib.mkOption {
type = lib.types.str;
default = lib.optionalString autheliaCfg.enable "${autheliaCfg.hostClientSecretDir}/${queueClientId}.secret";
defaultText = lib.literalExpression ''
"''${swarm.authelia.hostClientSecretDir}/swarm-controller.secret"
'';
example = "/var/lib/secrets/swarm-controller-queue.secret";
description = ''
Path on **this** host holding the plaintext of the controller's
OAuth2 client secret. A path, never a value: the secret would
otherwise land in the world-readable nix store.
Defaults to the file authelia's first-boot generator mints when
authelia runs here. When it doesn't, the operator places the
secret and names it — the controller cannot mint its own,
because minting happens inside authelia's state directory.
Read by `LoadCredential`, so it needs to be readable by root at
unit start and nothing more; the daemon's own user never sees
the original path.
'';
};
};
links = lib.mkOption {
type = lib.types.listOf (
lib.types.submodule {
@ -298,6 +359,53 @@ in
}
];
# A controller without a queue is not a lighter controller, it is a
# broken one: `/api/hives/status` reads the KV, so its whole aggregate
# half is off. The operator's rule is that the controller needs a queue
# but may not run on the same host as one, given it is configured
# properly — so the queue is required and the co-location is not.
#
# ⚠️ These fail at EVAL on purpose. The state they replace is worse than
# an error: with the coordinates absent the daemon started cleanly,
# served `/api/hives`, and silently never connected — nothing logged,
# because on an all-local deployment nothing was wrong.
assertions = [
{
assertion = cfg.queue.natsUrl != "";
message = ''
services.hyperhive.swarm.controller.queue.natsUrl is unset.
It defaults to loopback only when this host also runs the queue
(`services.hyperhive.swarm.nats.enable`). A controller on its
own host has to be told where the queue is.
'';
}
{
assertion = cfg.queue.tokenEndpoint != "";
message = ''
services.hyperhive.swarm.controller.queue.tokenEndpoint is unset,
which means services.hyperhive.swarm.authelia.url is null.
The controller mints its own access token before it may connect
to the queue, so it needs to know where the swarm's identity
provider lives — set that URL, or set this endpoint directly.
'';
}
{
assertion = cfg.queue.clientSecretFile != "";
message = ''
services.hyperhive.swarm.controller.queue.clientSecretFile is
unset.
It defaults to the file authelia's first-boot generator mints,
which only exists when authelia runs on this host. Elsewhere the
operator places the secret and names it here — the controller
cannot mint its own, because minting happens inside authelia's
state directory.
'';
}
];
systemd.services.swarm-controller = {
description = "hyperhive swarm-level controller daemon";
wantedBy = [ "multi-user.target" ];
@ -306,14 +414,16 @@ in
serviceConfig = {
ExecStart = "${cfg.package}/bin/swarm-controller";
# Only the credentials that actually resolve on this host. An
# absent credential is not a failure for either: the daemon logs
# that no queue / no forge is configured and serves its HTTP
# surface regardless, which is the correct behaviour on a host
# that doesn't run one.
LoadCredential =
lib.optional queueLocal "queue-client.secret:${autheliaCfg.hostClientSecretDir}/${queueClientId}.secret"
++ lib.optional (cfg.forgeTokenFile != null) "forge-token:${cfg.forgeTokenFile}";
# The two differ on purpose. The queue credential is
# unconditional — the assertions above make its path a value that
# always exists by the time this renders, so there is no "queue is
# off here" case left for a `mkIf` to express. The forge token
# stays optional: a controller with no forge access still serves
# its HTTP surface, and that IS a supported shape.
LoadCredential = [
"queue-client.secret:${cfg.queue.clientSecretFile}"
]
++ lib.optional (cfg.forgeTokenFile != null) "forge-token:${cfg.forgeTokenFile}";
User = "swarm-controller";
Group = "swarm-controller";
Restart = "on-failure";
@ -352,10 +462,14 @@ in
};
# Queue coordinates (`queueEnv`) and forge coordinates (`forgeEnv`)
# merge in last, each present only when its own resolution actually
# succeeded on this host. Both refuse a PARTIAL set rather than
# treating it as absent, which is why each is built as one attrset
# and never assigned individually.
# merge in last. The daemon refuses a PARTIAL set of either rather
# than treating it as absent, which is why each is built as one
# attrset and never assigned individually.
#
# They differ in how absence is prevented: the queue's is checked by
# the assertions above, because a controller without a queue is
# broken rather than lighter; the forge's is genuinely optional and
# stays gated on `forgeTokenFile` resolving.
environment = {
SWARM_CONTROLLER_SOCKET = cfg.socketPath;
# The swarm's hive directory, JSON-encoded — the full directory