vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

swarm: read the agent queue credential out of the store onto the hive host

Browse source 
The publisher on the authelia host has been writing
`secret/swarm/hives/<hive>/queue/agent` — the OIDC client secret agent
containers present to the swarm queue, plus the client id it belongs to —
and nothing read it. This is the reader: a oneshot `swarm-bao-queue-agent`
that logs in with the host's certificate and lands the two fields as two
files under `deploy.hive-controller.queue.agentCredentialDir`, the secret
`0600` and the client id `0644`.

Two files rather than one because that is the consumer's shape:
`swarm_queue_client::QueueConfig::from_env` takes the secret as a path and
the client id as a value, so the split here is what keeps the next slice
from parsing anything.

Same shape as the store's first reader, `glue-matrix-bao-token.nix` — a
cert login that fails loudly under `Restart=on-failure` because every state
it fails on is one a retry fixes, then reads that degrade quietly because no
retry turns "no value there" into a value. Unlike the matrix token there is
no local fallback and none is possible, so absent files mean this hive's
agents do not connect, which is the ordinary state of a swarm before the
publisher has run.

Nothing consumes the files yet and this unit is ordered `Before=` nothing.
The next slice bind-mounts them into agent containers through hive-c0re and
adds the ordering edge along with them.

Refs #3805
This commit is contained in:
atlas 2026-09-12 21:05:52 +02:00
commit b8157cb08e
4 changed files with 289 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

22
docs/swarm/secrets.md
View file

@ -131,13 +131,14 @@ when the system builds. The server names the offending file and refuses to run.
## Hive-level — one of each per hive
| secret | generated by | lives at |
| ---------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| hive CA cert + key | `hive-tls.nix` first-boot unit | `<deploy.hive-controller.tls.stateDir>/ca.pem`, `ca-key.pem` (`0600`) |
| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| matrix registration token | a host activation script, on first boot | `/var/lib/hyperhive/matrix-register-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| secret | generated by | lives at |
| ---------------------------------------- | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| hive CA cert + key | `hive-tls.nix` first-boot unit | `<deploy.hive-controller.tls.stateDir>/ca.pem`, `ca-key.pem` (`0600`) |
| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| matrix registration token | a host activation script, on first boot | `/var/lib/hyperhive/matrix-register-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
Both delivery units wait for authelia's first boot to mint the secret — a
bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip
@ -157,6 +158,13 @@ path — no such key, sealed store, unreachable store, empty value — leaves th
locally minted token in place, so a hive with no store behaves exactly as it
did before.
The **second reader** is the agent containers' queue credential:
`glue-queue-agent-credential.nix` lands it as two files, the client secret and
the client id it authenticates, because that is how a queue client reads them —
the secret by path, the id by value. There is no local fallback and none is
possible, so absent files mean this hive's agents do not connect, which is what
a swarm looks like before the publisher on the authelia host has run.
⚠️ **Service↔store mTLS is its own trust domain.** A credential you must
already hold to authenticate can't be fetched from the thing it authenticates
you to, so the store's identity can't come from an authority the store

1
nix/host-modules/default.nix
View file

@ -26,6 +26,7 @@
./glue-bao-tls.nix
./glue-controller-bao-identity.nix
./glue-matrix-bao-token.nix
./glue-queue-agent-credential.nix
./glue-secret-publisher-bao-identity.nix
./swarm-authelia.nix
./swarm-bao.nix

222
nix/host-modules/glue-queue-agent-credential.nix Normal file
View file

@ -0,0 +1,222 @@
# Glue: this hive's agent queue credential comes out of the secret store.
#
# The store's second reader, and deliberately the same shape as its first
# (./glue-matrix-bao-token.nix): a cert login that fails LOUDLY because every
# state it fails on is one a retry fixes, then a read that degrades QUIETLY
# because no retry turns "no value there" into a value.
#
# ⚠️ Two files, not one, and that is the consumer's shape rather than a
# preference. `swarm_queue_client::QueueConfig::from_env` takes the secret as
# a PATH (`<prefix>_OIDC_CLIENT_SECRET_FILE`) and the client id as a VALUE
# (`<prefix>_OIDC_CLIENT_ID`), so splitting them here is what lets the next
# slice hand both to an agent without parsing anything.
#
# ⚠️ An ABSENT file means this hive's agents do not connect to the queue, and
# that is correct rather than degraded. The publisher runs on the authelia
# host and authelia mints on its first boot, so "nothing at that path yet" is
# the ordinary early state of a swarm. Nothing here writes a local stand-in:
# unlike a matrix registration token there is no such thing as a locally valid
# OIDC client secret, so a placeholder would turn a hive that cannot connect
# into one that is refused, which reaches the agent as a timeout.
#
# 📌 This runs wherever a client identity is configured, NOT only where the
# store is — the rule ./glue-matrix-bao-token.nix states in full. There is no
# "this hive runs agents" condition to gate it on as well: agent containers
# are created at runtime by hive-c0re, so every hyperhive host is a host that
# may run one.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
# What decides whether this unit exists at all. A reader is defined by holding
# a certificate the store accepts, and that is true on the store's own host
# and on a hive three networks away for exactly the same reason.
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir;
secretFile = "${credentialDir}/secret";
clientIdFile = "${credentialDir}/client_id";
# Where the credential lives in the store, spelled from the same pieces the
# writer uses. Whoever writes it and whoever reads it must agree, and the
# agreement belongs in one visible place.
#
# ⚠️ The `hives/<name>` segment is not decoration — it is what the reader's
# own grant covers, so a path outside it is a 403 rather than a miss, however
# correct it looks. `swarm-secret-client`'s `queue::agent_client_path` builds
# the same string from the same pieces; this literal is the nix half of that
# one agreement.
#
# `hiveName` has no fallback here for the reason ./glue-bao-tls.nix gives at
# its own use of it: it is asserted set for every hyperhive host.
credentialPath = "secret/swarm/hives/${hyperhiveCfg.hiveName}/queue/agent";
in
{
options.services.hyperhive.deploy.hive-controller.queue = {
agentCredentialDir = lib.mkOption {
type = lib.types.path;
default = "/var/lib/hyperhive/queue-agent";
description = ''
Host directory holding this hive's agent queue credential: `secret`
(the OIDC client secret, `0600`) and `client_id` (the client that
secret authenticates, `0644` — it is sent to the token endpoint on
every connection and is not itself a secret).
Both files are written by `swarm-bao-queue-agent.service`, which also
creates this directory. Named here because nothing else creates it:
the parent `/var/lib/hyperhive` belongs to `hive-c0re.service`'s
`StateDirectory=`, which re-applies its own mode on every start, and a
second declaration of *that* path is the two-mechanisms-one-path trap
./hive-gateway/default.nix records — so this directory is the unit's
to make and the parent stays c0re's.
Absent files mean this hive has no queue credential yet, which is what
a swarm looks like before the publisher on the authelia host has run.
'';
};
};
config = lib.mkIf (hyperhiveCfg.enable && haveClientIdentity) {
# Same rule as the unit's own gate: this reader exists on any host holding
# a client identity, which is not every host that runs the store, so the
# store's module cannot name it.
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-queue-agent" ];
systemd.services.swarm-bao-queue-agent = {
description = "fetch this hive's agent queue credential from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering is
# conditional even though the read is not: off-host there is nothing local
# to wait for, and the timeout below is what bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
# Ordered `Before=` nothing, because the consumer does not exist yet.
# Agent containers are created at runtime, so no static unit name can be
# named here anyway; the slice that bind-mounts these files in adds the
# edge through hive-c0re's `container@h-<agent>.service` drop-in.
wantedBy = [ "multi-user.target" ];
path = [
baoDeploy.package
pkgs.coreutils
];
# Sized for the race this loses, not for an unseal: `swarm-bao` comes up
# seconds before this unit asks, and the cert-auth role it logs in
# against is written seconds after, so a few short attempts cover it.
# An hours-long window would be a bet on a store that is sealed, and the
# degrade below is already correct for that.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec × burst`.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
# What actually bounds the reads below. Stated here rather than left to
# systemd's default, so the number a boot waits on is in the file that
# waits.
TimeoutStartSec = 30;
Restart = "on-failure";
RestartSec = 15;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.clientCertFile;
BAO_CLIENT_KEY = baoDeploy.clientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
# `bao`'s own message is the only thing separating a missing value from
# a refused identity from an unreachable host. This unit's degraded
# mode is correct for all three, so it reports which one rather than
# asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Fails LOUDLY, unlike the reads below: the three states a login
# failure covers — store not up, sealed, role not written yet — are all
# things a retry fixes, and `Restart=on-failure` above is what retries.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
# Two reads of one object rather than one `-format=json` parsed with
# `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot
# actually disagree — the client id is derived from this hive's name, so
# a rotation landing between these two calls changes the secret and
# rewrites the same id.
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
# Both or neither, for the reason `QueueConfig::from_env` refuses a
# half-set environment: a client that finds one of the two comes up
# "fine" and never connects.
if [ -z "$secret" ] || [ -z "$client_id" ]; then
echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2
exit 0
fi
install -d -m 0755 ${lib.escapeShellArg credentialDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile}
chmod 0600 ${lib.escapeShellArg secretFile}
# `0644` on purpose: an OIDC client id is presented to the token
# endpoint on every connection and is public by construction.
printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile}
chmod 0644 ${lib.escapeShellArg clientIdFile}
'';
};
};
}

51
nix/module-eval.nix
View file

@ -1164,6 +1164,57 @@ let
# which no grant covers.
&& !(lib.hasInfix "secret/swarm/matrix/" s);
}
{
# The store's second reader, and the gate that decides it exists is the
# certificate rather than anything about agents: containers are created
# at runtime, so there is no static "this hive runs agents" fact to ask.
name = "a hive that names a client identity reads its agent queue credential";
ok = baoRemoteReader.systemd.services ? swarm-bao-queue-agent;
}
{
# Absence arm, and what makes the one above able to fail: with no leaf
# this unit would fail a TLS handshake on every boot, so it must not
# exist at all rather than retry its way through the start limit.
name = "a hive with no store identity renders no queue credential reader";
ok = !(matrixNoBaoIdentity.systemd.services ? swarm-bao-queue-agent);
}
{
# Same 403-not-a-miss reason as the matrix arm above, against the path
# `swarm_secret_client::queue::agent_client_path` builds from the same
# pieces. The negative arm is the rename this one is exposed to: a
# credential named for the queue rather than for the hive that presents
# it reads as correct and is refused on every boot.
name = "the agent queue credential path sits inside the prefix the reader is granted";
ok =
let
s = baoRemoteReader.systemd.services.swarm-bao-queue-agent.script;
in
lib.hasInfix "secret/swarm/hives/h1/queue/agent" s && !(lib.hasInfix "secret/swarm/queue/" s);
}
{
# The unit's output is the option's value, not a literal that agrees with
# it today: an operator moving the directory has to move both files. The
# prefix is asserted too because `hasInfix ""` is true — an option
# renamed out from under this arm would otherwise read empty and pass.
name = "the queue credential reader writes both files under the directory its option names";
ok =
let
m = baoRemoteReader;
dir = toString m.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
s = m.systemd.services.swarm-bao-queue-agent.script;
in
lib.hasPrefix "/var/lib/" dir
&& lib.hasInfix "${dir}/secret" s
&& lib.hasInfix "${dir}/client_id" s;
}
{
# A reader off the store's host is a reader whose journal is the only
# record of why a hive's agents never connected, so the collector has to
# be told the unit exists. Nothing else can say it: the store's module
# does not know who holds a certificate.
name = "the queue credential reader's journal reaches the collector";
ok = builtins.elem "swarm-bao-queue-agent" baoRemoteReader.services.hyperhive.swarm.otel.journaldUnits;
}
{
# The doctrine three glue files state, as a property a rewrite has to
# keep: a client is defined by holding a certificate the store accepts,