vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-authelia: run the secret publisher and a journal forwarder inside the container

Browse source 
swarm-secret-publish and its path unit now run inside authelia's container,
beside the mint, and read each client secret at the path authelia writes it.
The host unit and host path unit, which read the secrets through the
container's root on the host, are gone, and so is
deploy.authelia.hostClientSecretDir, which only they read. The publisher keeps
its store identity, the secret-publisher leaf. The host hands the leaf and the
store's CA into the container as nspawn system credentials, and the unit loads
them; where this host mints the leaf, the container waits for swarm-bao-pki.
Authelia does not wait for the publisher, which retries every 30 seconds while
the store is down.

The container also runs the journal forwarder the other service containers
run, pushing to a receiver of its own on the swarm collector (serviceProducers
swarm-authelia) as a new machine OIDC client (swarm.authelia.otel.clientId)
that authelia registers for itself and the publisher carries to the store. The
fetch unit reads it back under the publisher's leaf, also handed to it as a
system credential, so the swarm-secret-publisher policy gains read on that one
leaf; its stanza keeps create and update, which the exact path would otherwise
take from the services/* prefix. Without the leaf the forwarder is off.

The container drops --link-journal=host, and the host collector stops reading
it. The services' "nothing at <path>" errors name the publisher in authelia's
container.
This commit is contained in:
atlas 2026-10-16 15:05:29 +02:00 • committed by mara
commit b68a8ef290
36 changed files with 753 additions and 363 deletions
Download patch file Download diff file Expand all files Collapse all files

236
nix/host-modules/swarm-secret-publisher.nix
View file

@ -1,55 +1,15 @@
# The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its own and its agents'
# credentials, a service or the controller its own. It mints nothing: every
# value is an OIDC client secret authelia minted, and this unit copies it.
#
# ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
# into the store even when the service runs beside authelia, because its
# reader fetches it from the store in every deployment — ./swarm-grafana.nix
# states the ruling that made that the only route. Publishing "only when the
# reader is elsewhere" would be a second shape of this unit, gated on a fact
# about another host, to save a round trip on the one host that can afford it.
#
# ⚠️ IT RUNS WHERE AUTHELIA DOES, and that is the whole reason it exists as a
# separate thing. `deploy.authelia.hostClientSecretDir`'s own description says
# why: the plaintext's other reader lives in a **different container**, and
# containers that share this host's network namespace still have separate
# filesystem roots, so "the host is the only place both trees are addressable".
# A delivery step therefore runs on the minting host, not the store's or the
# reading hive's.
#
# ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite
# every hive's policy and login role; a unit that writes a handful of secrets
# has no business holding it. ./swarm-bao.nix grants this one `create`/`update`
# under the hive prefix and nothing else — no `read`.
#
# ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every
# argument is world-readable in /proc for the life of the call — the value is
# passed as `@<path>` and read by bao itself.
# What a host decides about the swarm's secret publisher: whether it runs, and
# the store identity it presents, plus the list of secrets it copies. The unit
# itself is ../container-modules/swarm-secret-publisher.nix, inside authelia's
# container beside the mint whose secrets it copies; ./swarm-authelia.nix
# imports it there and hands the identity in as system credentials.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoDeploy = deployCfg.bao;
cfg = deployCfg.swarm-secret-publisher;
# A reader is defined by holding a certificate the store accepts, never by
# standing next to it — the rule ./glue-queue-agent-credential.nix states in
# full.
# `deploy.bao.enable` here would be the co-location assumption itself.
haveClientIdentity = cfg.baoClientCertFile != null && cfg.baoClientKeyFile != null;
# Both halves have to be here: the mint (authelia, for the plaintext) and an
# identity (for the store). Neither implies the other. Authelia here is a
# requirement, not a co-location toggle: every file the unit copies is under
# its `hostClientSecretDir`.
active = cfg.enable && deployCfg.authelia.enable && haveClientIdentity;
hiveNames = lib.attrNames hyperhiveCfg.swarm.hives;
# The client id agent containers present, per hive — composed exactly as
@ -110,12 +70,46 @@ let
hyperhiveCfg.swarm.victoriametrics.otel.clientId
# The CI run poller, which reads it itself on its host.
hyperhiveCfg.swarm.ci-metrics.clientId
# Authelia's container's journal forwarder, which reads it back under the
# publisher's own identity.
hyperhiveCfg.swarm.authelia.otel.clientId
];
# The swarm controller's OIDC client, which it reads back from the store
# under its own certificate.
controllerClientId = hyperhiveCfg.swarm.controller.queueClientId;
controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client";
secrets =
# `client_id` beside the value: an agent's id is derived per hive and has
# to be reconstructable from the store alone.
map (hive: {
clientId = agentClientId hive;
path = "secret/swarm/hives/${hive}/queue/agent";
withClientId = true;
}) hiveNames
# Each hive's own client, which that hive's hive-c0re reads under its own
# certificate. The nix half of
# `swarm_secret_client::queue::hive_client_path`.
++ map (hive: {
clientId = hiveClientId hive;
path = "secret/swarm/hives/${hive}/oidc/client";
}) hiveNames
# No `client_id`: a service's client id is the swarm-wide option both
# ends already read.
++ map (id: {
clientId = id;
path = "secret/swarm/services/${id}/oidc/client";
}) serviceClientIds
# Under `controller/`, which no hive reads: this client may read every
# hive's status. The nix half of
# `swarm_secret_client::queue::controller_client_path`.
++ [
{
clientId = controllerClientId;
path = controllerSecretPath;
}
];
in
{
options.services.hyperhive.deploy.swarm-secret-publisher = {
@ -127,7 +121,8 @@ in
secret store, so a hive that does not run authelia can read its
own and its agents' credentials and a swarm service or the
controller can read its own — from wherever it runs, this host
included.
included. The unit, `swarm-secret-publish`, runs inside authelia's
container, so it exists only where authelia runs.
Defaults to whether this host mints the OIDC secrets (set by
`glue-secret-publisher-authelia.nix`), which is the only half of the
@ -160,6 +155,11 @@ in
must be {option}`services.hyperhive.deploy.bao.secretPublisherCommonName`
— cert auth matches on the CN, and the role accepts nothing else.
A path on the host. ./swarm-authelia.nix hands it into authelia's
container as an nspawn system credential, and nspawn refuses to
start the container while a credential's source is missing: set,
the file has to exist before authelia's container starts.
No default: a module that guessed would be holding the CA opinion
./swarm-bao.nix deliberately does not hold.
./glue-secret-publisher-bao-identity.nix points it at the leaf
@ -172,141 +172,21 @@ in
default = null;
description = ''
Private key for {option}`services.hyperhive.deploy.swarm-secret-publisher.baoClientCertFile`.
Both or neither — the unit does not exist unless each is set.
Both or neither — the unit does not exist unless each is set. A host
path, handed in like the certificate.
'';
};
};
config = lib.mkIf active {
# Re-publish when authelia rotates a secret. The mint writes the file, so
# the file is the event — there is no signal from authelia to subscribe to.
systemd.paths.swarm-secret-publish = {
description = "watch for minted OIDC client secrets to publish";
wantedBy = [ "multi-user.target" ];
pathConfig = {
PathChanged = deployCfg.authelia.hostClientSecretDir;
Unit = "swarm-secret-publish.service";
};
};
systemd.services.swarm-secret-publish = {
description = "publish the swarm's minted secrets to the swarm secret store";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" ];
path = [
baoDeploy.package
pkgs.coreutils
];
# ⚠️ No `RemainAfterExit`. A oneshot that stays `active (exited)` turns
# every later start from the path unit above into a no-op.
serviceConfig = {
Type = "oneshot";
# Bounded here rather than left to systemd's default, so the number a
# boot waits on is in the file that waits. A sealed store answers on
# the port and never answers the write.
TimeoutStartSec = 60;
Restart = "on-failure";
RestartSec = 30;
};
environment = {
BAO_ADDR = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}";
BAO_CLIENT_CERT = cfg.baoClientCertFile;
BAO_CLIENT_KEY = cfg.baoClientKeyFile;
}
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Unhandled on purpose: `Restart=on-failure` above is what a store that
# cannot authenticate this host should get. Degrading here would report
# "published 0" as an ordinary quiet day.
BAO_TOKEN="$(bao login -method=cert -token-only)"
export BAO_TOKEN
published=0
skipped=0
${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${agentClientId hive}.secret"}
if [ -s "$src" ]; then
# `value=@$src` hands bao the PATH: bao opens the file itself, so
# the plaintext is never an argument of this process. Writing it as
# `value="$(cat "$src")"` would publish it to /proc for anyone on
# the host to read.
bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/queue/agent"} \
value=@"$src" \
client_id=${lib.escapeShellArg (agentClientId hive)}
published=$((published + 1))
else
# Not an error: authelia mints on its FIRST BOOT, so an absent file
# is "not yet", and the path unit above re-runs this when it lands.
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
'') hiveNames}
# Each hive's own client, which that hive's hive-c0re reads under its
# own certificate. The nix half of
# `swarm_secret_client::queue::hive_client_path`.
${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${hiveClientId hive}.secret"}
if [ -s "$src" ]; then
bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/oidc/client"} \
value=@"$src"
published=$((published + 1))
else
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
'') hiveNames}
${lib.concatMapStringsSep "\n" (id: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${id}.secret"}
if [ -s "$src" ]; then
# `value=@$src` for the same reason as the hive loop above: bao
# opens the file itself, so the plaintext is never an argument of
# this process.
#
# No `client_id` field beside it, unlike a hive's credential: that
# one is derived per hive and has to be reconstructable from the
# store alone, whereas a service's client id is the swarm-wide
# option both ends already read.
bao kv put ${lib.escapeShellArg "secret/swarm/services/${id}/oidc/client"} \
value=@"$src"
published=$((published + 1))
else
# Not an error, and the ordinary state of a swarm that runs this
# service nowhere: nothing registered the client, so authelia minted
# nothing to publish.
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
'') serviceClientIds}
# The controller's own client, under `controller/`, which no hive
# reads: this client may read every hive's status. The nix half of
# `swarm_secret_client::queue::controller_client_path`.
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${controllerClientId}.secret"}
if [ -s "$src" ]; then
bao kv put ${lib.escapeShellArg controllerSecretPath} \
value=@"$src"
published=$((published + 1))
else
# Not an error: authelia mints on its first boot, and the path unit
# above re-runs this when it lands.
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
echo "published $published client secret(s), skipped $skipped"
secrets = lib.mkOption {
type = lib.types.listOf lib.types.attrs;
readOnly = true;
internal = true;
default = secrets;
description = ''
Every secret the publisher copies: the minted client's id, the store
path its reader fetches, and whether `client_id` is stored beside the
value. Derived from `swarm.*` alone, so it is the same list on every
host; ./swarm-authelia.nix hands it to the unit in its container.
'';
};
};