feat(swarm): wire a hive's queue coordinates for status publishing
Three options, all three derived from ONE predicate — this host runs both the queue and the IdP — so a defaulted set is all or nothing. Deriving them per-service looks equivalent and is not: `enableRequiredServices` turns on matrix and authelia but not nats, so an ordinary all-local hive would resolve two of three and trip the assertion below. Making the partial state unrepresentable is what keeps that assertion honest. Deliberately not the shape swarm-controller uses. That module emits its queue coordinates only when authelia and NATS are local, which is right for a service that *is* a swarm-host service — but a hive is the one thing in a swarm that routinely is not on the swarm host, so the same rule would make status publishing work on exactly the deployment that needs it least. There is no `enable`: three coordinates that are all set is the enable. An extra flag would allow configured-but-off, which is one more state to explain and one more way to be silently quiet. A half-set trio is an eval error rather than a silent no-op, because its runtime failure mode is the expensive kind — the daemon comes up fine, never connects, and the hive reads never_reported on a dashboard nobody is watching yet. With the defaults all-or-nothing, the assertion only ever judges what an operator typed by hand. The secret arrives by LoadCredential, not a copy: hive-c0re is a host unit, so systemd hands it the file directly and the secret never gains a second on-disk copy. The client id is not chosen here either — it is `hive-<hiveName>`, the identity swarm-authelia.nix already declares for every entry in the roster.
This commit is contained in:
parent
dc394b459d
commit
48f69fcdea
3 changed files with 167 additions and 3 deletions
|
|
@ -251,9 +251,20 @@ in
|
||||||
# path. Same secret the agent containers get (forwarded there via
|
# path. Same secret the agent containers get (forwarded there via
|
||||||
# nspawn --load-credential); this just also hands it to c0re itself.
|
# nspawn --load-credential); this just also hands it to c0re itself.
|
||||||
# Empty list (no credential) when otel is off or no header is set.
|
# Empty list (no credential) when otel is off or no header is set.
|
||||||
LoadCredential = lib.optional (
|
LoadCredential =
|
||||||
config.services.hyperhive.otel.enable && config.services.hyperhive.otel.headersCredential != null
|
lib.optional (
|
||||||
) "otel-headers:${config.services.hyperhive.otel.headersCredential}";
|
config.services.hyperhive.otel.enable && config.services.hyperhive.otel.headersCredential != null
|
||||||
|
) "otel-headers:${config.services.hyperhive.otel.headersCredential}"
|
||||||
|
# The swarm-queue client secret this hive authenticates with to
|
||||||
|
# publish its own status. `LoadCredential` and not a copy: root
|
||||||
|
# reads the plaintext at unit start and hive-core sees it 0400
|
||||||
|
# under `%d`, so the secret never gains a second on-disk copy
|
||||||
|
# and the daemon never needs read access to wherever it lives.
|
||||||
|
# (The callout responder copies instead only because it
|
||||||
|
# delivers into a container, across a filesystem boundary.)
|
||||||
|
++ lib.optional (
|
||||||
|
config.services.hyperhive.swarm.statusPublish.clientSecretFile != null
|
||||||
|
) "swarm-status-client.secret:${config.services.hyperhive.swarm.statusPublish.clientSecretFile}";
|
||||||
# Sandboxing. hive-c0re is unprivileged (runs as hive-core, never
|
# Sandboxing. hive-c0re is unprivileged (runs as hive-core, never
|
||||||
# setuid), makes HTTP requests to forge/matrix/Anthropic (keeps INET),
|
# setuid), makes HTTP requests to forge/matrix/Anthropic (keeps INET),
|
||||||
# and delegates all privileged ops to hive-priv via a Unix socket.
|
# and delegates all privileged ops to hive-priv via a Unix socket.
|
||||||
|
|
|
||||||
|
|
@ -209,3 +209,22 @@ in
|
||||||
in
|
in
|
||||||
"${s.address}:${toString s.port}";
|
"${s.address}:${toString s.port}";
|
||||||
}
|
}
|
||||||
|
//
|
||||||
|
# Swarm-queue coordinates for offering this hive's status upward
|
||||||
|
# (hive-c0re::swarm_status). All four together or none: a half-set
|
||||||
|
# environment is a deployment bug the daemon refuses to treat as
|
||||||
|
# "no queue configured", because the failure it would otherwise
|
||||||
|
# produce is a hive that comes up fine and silently never reports.
|
||||||
|
# The three-option version of that same rule is asserted at eval in
|
||||||
|
# ./../swarm.nix, so this can only ever emit a complete set.
|
||||||
|
lib.optionalAttrs (config.services.hyperhive.swarm.statusPublish.natsUrl != null) {
|
||||||
|
HIVE_C0RE_NATS_URL = config.services.hyperhive.swarm.statusPublish.natsUrl;
|
||||||
|
HIVE_C0RE_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
|
||||||
|
# The identity swarm-authelia.nix already declares for every entry in
|
||||||
|
# `swarm.hives` — the hive does not choose its own name here, it uses
|
||||||
|
# the one the roster gave it.
|
||||||
|
HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}";
|
||||||
|
# `%d` is systemd's credentials directory — see the LoadCredential in
|
||||||
|
# ./default.nix. The daemon reads a path, never a value.
|
||||||
|
HIVE_C0RE_OIDC_CLIENT_SECRET_FILE = "%d/swarm-status-client.secret";
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -53,6 +53,23 @@ let
|
||||||
pinnedHives = lib.attrNames (
|
pinnedHives = lib.attrNames (
|
||||||
lib.filterAttrs (_: hive: hive.certFingerprint != null) swarmCfg.hives
|
lib.filterAttrs (_: hive: hive.certFingerprint != null) swarmCfg.hives
|
||||||
);
|
);
|
||||||
|
|
||||||
|
# Whether this host can derive its own status-publishing coordinates —
|
||||||
|
# ONE condition for all three of them, deliberately.
|
||||||
|
#
|
||||||
|
# 🩸 They were three independent conditions first, and that was wrong in
|
||||||
|
# a way only an eval gate finds: `enableRequiredServices` turns on
|
||||||
|
# matrix and authelia but NOT nats (nats has no mode that enables it —
|
||||||
|
# see the auto-deploy question on the swarm-queue issue), so an ordinary
|
||||||
|
# all-local hive resolved authelia's two coordinates and not the queue
|
||||||
|
# URL. Two of three set is exactly what the assertion below rejects, so
|
||||||
|
# every `enableAllLocalDefaults` hive would have stopped evaluating.
|
||||||
|
#
|
||||||
|
# Deriving all three from one predicate makes the partial state
|
||||||
|
# unrepresentable rather than merely detected: a default set is all or
|
||||||
|
# nothing, and the assertion is then only ever about what an operator
|
||||||
|
# typed.
|
||||||
|
queueLocal = swarmCfg.nats.enable && swarmCfg.authelia.enable && cfg.hiveName != null;
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
options.services.hyperhive.swarm.hives = lib.mkOption {
|
options.services.hyperhive.swarm.hives = lib.mkOption {
|
||||||
|
|
@ -298,6 +315,43 @@ in
|
||||||
designed for it.
|
designed for it.
|
||||||
'';
|
'';
|
||||||
}
|
}
|
||||||
|
{
|
||||||
|
# Deliberately an assertion and not a silent "then publish
|
||||||
|
# nothing": a half-set trio is a config an operator believes is
|
||||||
|
# working, and its runtime failure mode is the expensive one —
|
||||||
|
# the daemon comes up fine, never connects, and the hive reads
|
||||||
|
# `never_reported` on a dashboard nobody is watching yet.
|
||||||
|
#
|
||||||
|
# Safe to add to an existing deployment: every `statusPublish`
|
||||||
|
# default is either all-local or all-null, so no config that
|
||||||
|
# evaluates today can be caught by this. It also encodes a
|
||||||
|
# property of the code rather than an intended shape — hive-c0re
|
||||||
|
# genuinely cannot publish with two of three coordinates — which
|
||||||
|
# is the distinction the `serviceDomains'` comment at the top of
|
||||||
|
# this file was written about.
|
||||||
|
assertion =
|
||||||
|
let
|
||||||
|
set = lib.filter (v: v != null) [
|
||||||
|
swarmCfg.statusPublish.natsUrl
|
||||||
|
swarmCfg.statusPublish.tokenEndpoint
|
||||||
|
swarmCfg.statusPublish.clientSecretFile
|
||||||
|
];
|
||||||
|
in
|
||||||
|
builtins.length set == 0 || builtins.length set == 3;
|
||||||
|
message = ''
|
||||||
|
services.hyperhive.swarm.statusPublish needs natsUrl,
|
||||||
|
tokenEndpoint and clientSecretFile set together or not at all
|
||||||
|
— this hive has only some of them.
|
||||||
|
|
||||||
|
Currently:
|
||||||
|
natsUrl = ${toString swarmCfg.statusPublish.natsUrl}
|
||||||
|
tokenEndpoint = ${toString swarmCfg.statusPublish.tokenEndpoint}
|
||||||
|
clientSecretFile = ${toString swarmCfg.statusPublish.clientSecretFile}
|
||||||
|
|
||||||
|
Set the missing ones to publish this hive's status to the
|
||||||
|
swarm, or set all three to null to turn publishing off.
|
||||||
|
'';
|
||||||
|
}
|
||||||
];
|
];
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -343,4 +397,84 @@ in
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# How this hive reaches the swarm queue to offer its own status
|
||||||
|
# (hive-c0re's `swarm_status`). Three coordinates, defaulted from the
|
||||||
|
# local swarm services when this host runs them, and set by hand
|
||||||
|
# otherwise — one code path for both deployments.
|
||||||
|
#
|
||||||
|
# The alternative was the shape swarm-controller uses: emit the
|
||||||
|
# coordinates only when authelia and NATS are local, and nothing
|
||||||
|
# otherwise. That is right for the controller, which *is* a swarm-host
|
||||||
|
# service — but a hive is the one thing in a swarm that routinely is
|
||||||
|
# not on the swarm host, so the same rule would mean status publishing
|
||||||
|
# works on exactly the deployment that needs it least.
|
||||||
|
#
|
||||||
|
# There is no `enable`: three coordinates that are all set is the
|
||||||
|
# enable. An extra flag would let a hive be configured-but-off, which
|
||||||
|
# is one more state to explain and one more way to be silently quiet.
|
||||||
|
options.services.hyperhive.swarm.statusPublish = {
|
||||||
|
natsUrl = lib.mkOption {
|
||||||
|
type = lib.types.nullOr lib.types.str;
|
||||||
|
default = if queueLocal then "nats://127.0.0.1:${toString swarmCfg.nats.port}" else null;
|
||||||
|
defaultText = lib.literalExpression ''"nats://127.0.0.1:''${swarm.nats.port}" when this host runs the queue and the IdP, else null'';
|
||||||
|
example = "nats://10.100.0.1:4222";
|
||||||
|
description = ''
|
||||||
|
Where the swarm queue listens, as seen from *this* hive.
|
||||||
|
|
||||||
|
Defaults to loopback when this host runs the queue container
|
||||||
|
itself (it shares the host netns, so loopback is correct there
|
||||||
|
and is not the "localhost means the wrong thing" trap that
|
||||||
|
applies inside agent containers). A hive that is not the swarm
|
||||||
|
host has to name the swarm's mesh address.
|
||||||
|
|
||||||
|
Null disables status publishing: this hive computes its own
|
||||||
|
readiness as always, and simply offers it to nobody. The swarm
|
||||||
|
controller then reports it `never_reported`, which is the honest
|
||||||
|
reading.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
tokenEndpoint = lib.mkOption {
|
||||||
|
type = lib.types.nullOr lib.types.str;
|
||||||
|
default =
|
||||||
|
if queueLocal && swarmCfg.authelia.url != null then
|
||||||
|
"${swarmCfg.authelia.url}/api/oidc/token"
|
||||||
|
else
|
||||||
|
null;
|
||||||
|
defaultText = lib.literalExpression ''"''${swarm.authelia.url}/api/oidc/token" when this host runs both the queue and the IdP, else null'';
|
||||||
|
example = "https://auth.example.com/api/oidc/token";
|
||||||
|
description = ''
|
||||||
|
The swarm IdP's OAuth2 token endpoint. This hive mints a
|
||||||
|
`client_credentials` access token there and presents it when
|
||||||
|
connecting to the queue, which authenticates it as
|
||||||
|
`hive-<hiveName>` — the client
|
||||||
|
{file}`nix/host-modules/swarm-authelia.nix` already declares for
|
||||||
|
every entry in {option}`services.hyperhive.swarm.hives`.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
clientSecretFile = lib.mkOption {
|
||||||
|
type = lib.types.nullOr lib.types.str;
|
||||||
|
default =
|
||||||
|
if queueLocal then "${swarmCfg.authelia.hostClientSecretDir}/hive-${cfg.hiveName}.secret" else null;
|
||||||
|
defaultText = lib.literalExpression ''"''${swarm.authelia.hostClientSecretDir}/hive-''${hiveName}.secret" when this host runs both the queue and the IdP, else null'';
|
||||||
|
example = "/var/lib/secrets/swarm-queue-client.secret";
|
||||||
|
description = ''
|
||||||
|
Path to a file holding the plaintext client secret for this
|
||||||
|
hive's `hive-<hiveName>` identity.
|
||||||
|
|
||||||
|
A path and not a value: a secret in the Nix store is world
|
||||||
|
readable, and one in the environment is readable by anything
|
||||||
|
that can open {file}`/proc/<pid>/environ`.
|
||||||
|
|
||||||
|
Defaults to authelia's own minted secret when the IdP runs on
|
||||||
|
this host. On any other hive the secret has to get here somehow,
|
||||||
|
and the swarm does not distribute it — copy it out of the swarm
|
||||||
|
host's
|
||||||
|
{option}`services.hyperhive.swarm.authelia.hostClientSecretDir`
|
||||||
|
with whatever secret management this deployment already uses.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue