vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-nats-auth: read its own OIDC client secret from the store

Browse source 
The responder reads its OIDC client secret at
`swarm/services/<client-id>/oidc/client` with the store login it already
opens for the callout keys, and holds it in memory. `--client-secret-file`
is removed.

When introspection answers non-2xx, the responder re-reads the secret from
the store once and, if it changed, retries the introspection once. Both fit
inside the request's 1.5s introspection budget, under the server's 2s
callout timeout: the budget is now one deadline for the whole request. A
re-read that does not finish by then keeps running in the background, the
request is denied, and the next request introspects with what it read. One
re-read runs at a time; a refusal while one runs waits on it.

In swarm-nats.nix the `swarm-bao-nats-auth-oidc` fetch unit, the flag and
the `oidc-client.secret` LoadCredential entry are gone, along with the
unit's policy ordering in glue-bao-readers-policy-order.nix. A tmpfiles `r`
rule removes the copy a container still holds at
/var/lib/nixos-containers/swarm-nats/var/lib/swarm-nats-auth/oidc-client.secret;
nothing reads it.

No policy change: the `swarm-nats-auth` role already reads this path
(`natsAuthReaders` in swarm-bao.nix), which is the role the removed fetch
unit logged in as.

Deploy order: none, and no operator step. The store's grant already exists,
so the queue host can be deployed before, after or without the store's host
and authelia's. Rolling back re-creates the fetch unit and the file; the
store leaf is untouched.
This commit is contained in:
atlas 2026-10-06 13:43:14 +02:00
commit 6646329da5
12 changed files with 557 additions and 253 deletions
Download patch file Download diff file Expand all files Collapse all files

13
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -53,15 +53,10 @@ let
# Readers that log in under another unit's principal: reader unit → the
# gate its own module defines it under, and the policy units it waits for.
readersOfOtherPolicy = {
# ./swarm-nats.nix, under the queue responder's leaf.
swarm-bao-nats-auth-oidc = {
here = deployCfg.nats.enable;
policies = [ "swarm-bao-nats-auth-policy" ];
};
# ./swarm-nats.nix, under the same leaf. Also after the publisher's
# policy, which denies the publisher the callout seeds' paths: until it
# is written the publisher may create them, and this unit keeps a seed it
# finds there.
# ./swarm-nats.nix, under the queue responder's leaf. Also after the
# publisher's policy, which denies the publisher the callout seeds' paths:
# until it is written the publisher may create them, and this unit keeps
# a seed it finds there.
swarm-bao-nats-callout-xkey = {
here = deployCfg.nats.enable;
policies = [

2
nix/host-modules/swarm-bao.nix
View file

@ -866,7 +866,7 @@ let
# presents against the one stored for it. One leaf under every agent: `+` is
# one path segment, where `*` globs only at the end and would reach every
# other credential an agent holds. Plus the responder's own OIDC client
# secret, which `swarm-bao-nats-auth-oidc` reads under this same leaf.
# secret, which it reads as it starts and again when the IdP refuses it.
#
# And the two leaves of each callout key, which `ensure-callout-keys`
# creates when the store holds no seed. `create` without `update`, so a

146
nix/host-modules/swarm-nats.nix
View file

@ -48,10 +48,6 @@ let
secretDir = "/var/lib/nixos-containers/${machine}${secretDirInContainer}";
hostPath = name: "${secretDir}/${name}";
# Where swarm-secret-publish on authelia's host leaves this client's secret,
# composed from the same swarm-wide client id it publishes under. The
# responder's store role grants read on this one path.
storeSecretPath = "secret/swarm/services/${cfg.clientId}/oidc/client";
introspectionUrl = "${toString autheliaUrl}/api/oidc/introspection";
# Host seed files `ensure-callout-keys` stores when the secret store holds
@ -402,10 +398,9 @@ in
example = "/var/lib/swarm-bao-pki/nats-auth.pem";
description = ''
Client certificate the auth-callout responder presents to the swarm's
secret store to read the callout xkey and agent queue credentials, and
that `swarm-bao-nats-auth-oidc` and `swarm-bao-nats-callout-xkey`
present to read the responder's own OIDC client secret and the xkey's
public half. Its subject must be
secret store to read the callout keys, its own OIDC client secret and
agent queue credentials, and that `swarm-bao-nats-callout-xkey`
presents to read the callout keys' public halves. Its subject must be
{option}`services.hyperhive.deploy.bao.natsAuthCommonName`.
Required when `deploy.nats.enable` is set; unset fails at eval.
@ -415,9 +410,8 @@ in
host it is a file the operator places. Never derived from
{option}`services.hyperhive.deploy.bao.clientCertFile`, which is the
hive's own reader. Set to a file that does
not exist, `swarm-nats-auth-secrets`, `swarm-bao-nats-auth-oidc` and
`swarm-bao-nats-callout-xkey` fail, and the queue's server does not
start without the xkey.
not exist, `swarm-nats-auth-secrets` and `swarm-bao-nats-callout-xkey`
fail, and the queue's server does not start without the xkey.
A path, never a value.
'';
@ -519,7 +513,13 @@ in
# The bind source has to exist before the container starts, leaf or not:
# nixos-container refuses to start over a missing one, which would take
# the whole queue down rather than only its TLS.
systemd.tmpfiles.rules = [ "d ${tlsDir} 0755 root root -" ];
#
# The `r` removes a copy of the responder's OIDC client secret a container
# may still hold: the responder reads it from the store and names no file.
systemd.tmpfiles.rules = [
"d ${tlsDir} 0755 root root -"
"r ${hostPath "oidc-client.secret"}"
];
containers.swarm-nats = {
autoStart = true;
@ -697,7 +697,6 @@ in
# container's resolver goes through the bridge, where dnsmasq
# answers it.
"--nats-url tls://${cfg.domain}:${toString cfg.port}"
"--client-secret-file \${CREDENTIALS_DIRECTORY}/oidc-client.secret"
"--client-id ${lib.escapeShellArg cfg.clientId}"
# The account admitted clients land in, by NAME: in
# server-config mode the server resolves `aud` against its
@ -748,14 +747,12 @@ in
"--agent-token-publish-subject ${lib.escapeShellArg "\$\$JS.API.DIRECT.GET.KV_agent-schedules.\$\$KV.agent-schedules.{agent}"}"
"--store-cert-role ${lib.escapeShellArg authCertRole}"
];
# Every credential arrives by `LoadCredential` and is named
# on the command line only as a **path** — `argv` is
# world-readable via /proc/<pid>/cmdline, so a value there
# would be readable by every process on the host netns.
LoadCredential = [
"oidc-client.secret:${inContainer "oidc-client.secret"}"
]
++ lib.mapAttrsToList (id: _: "${id}:${inContainer id}") authStoreFiles;
# The store identity arrives by `LoadCredential` and is named
# only as a **path**: `argv` is world-readable via
# /proc/<pid>/cmdline. Every secret the responder holds beyond
# it, its own OIDC client secret included, it reads from the
# store with that identity.
LoadCredential = lib.mapAttrsToList (id: _: "${id}:${inContainer id}") authStoreFiles;
DynamicUser = true;
Restart = "on-failure";
RestartSec = "5s";
@ -815,108 +812,6 @@ in
);
};
# The responder's OIDC client secret, out of the swarm secret store under
# the responder's own leaf — the one route, wherever authelia runs. Same
# shape as ./swarm-bao.nix's `swarm-bao-forwarder-oidc`: the login and the
# read both fail loudly, and `storeRetry` retries until the publisher on
# authelia's host has written the secret, because the responder has no
# mode that runs without it.
#
# ⚠️ Without the secret the responder does not start (its `LoadCredential`
# source is missing), and `auth_callout` with no responder refuses every
# client.
systemd.services.swarm-bao-nats-auth-oidc = {
description = "fetch the swarm queue responder's OIDC client secret from the swarm secret store";
# The store's units exist only where the store runs; `Requires=` on an
# absent unit fails the job, so off-host the timeout below bounds the
# attempt instead.
after = [
"swarm-nats-auth-secrets.service"
]
++ lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
before = [ "container@${machine}.service" ];
wantedBy = [
"multi-user.target"
"container@${machine}.service"
];
path = [
baoDeploy.package
pkgs.coreutils
pkgs.systemd
];
# ./lib/store-retry.nix. The container waits for one attempt; a secret a
# later attempt lands restarts the responder (below).
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "swarm-bao-nats-auth-oidc";
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = deployCfg.nats.authBaoClientCertFile;
BAO_CLIENT_KEY = deployCfg.nats.authBaoClientKeyFile;
}
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
${atomicWriteSecret}
${refreshConsumer}
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# `-token-only` answers on stdout and skips the token helper, a
# `sh` this unit's `path` does not carry.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with the queue responder's certificate; leaving its OIDC client secret as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
echo "swarm-bao did not return ${storeSecretPath}; the queue responder has no OIDC client secret yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; the queue responder has no OIDC client secret yet." >&2
exit 1
fi
# Root-owned 0400: `LoadCredential` reads it as root before the
# responder's `DynamicUser` exists. `printf` inside
# `atomic_write_secret` is a builtin, so the plaintext never
# becomes an argument in /proc.
install -d -m 0700 ${lib.escapeShellArg secretDir}
if secret_differs ${lib.escapeShellArg (hostPath "oidc-client.secret")} "$secret"; then
atomic_write_secret 0400 "" ${lib.escapeShellArg (hostPath "oidc-client.secret")} "$secret"
fi
# `LoadCredential` copies the file once, at the responder's start.
refresh_consumer ${lib.escapeShellArg machine} swarm-nats-auth.service ${lib.escapeShellArg (hostPath "oidc-client.secret")}
'';
};
# The callout's three public keys, from the store into the files
# `renderConfig` reads at every server start. `ensure-callout-keys` first
# creates each pair the store holds none of; it needs no queue, so on a
@ -936,8 +831,9 @@ in
# responder's own leaf, the one principal that may create the keys' paths.
systemd.services.swarm-bao-nats-callout-xkey = {
description = "create the swarm queue's callout keys if absent and fetch their public halves for the server";
# As `swarm-bao-nats-auth-oidc`: the store's units exist only where it
# runs.
# The store's units exist only where the store runs; `Requires=` on an
# absent unit fails the job, so off-host the timeout below bounds the
# attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"

4
nix/host-modules/swarm-secret-publisher.nix
View file

@ -79,8 +79,8 @@ let
# The store's browser UI, whose `oidc` auth method the store's host
# configures with this secret.
hyperhiveCfg.swarm.bao.ui.oidc.clientId
# The queue's auth-callout responder, read by `swarm-bao-nats-auth-oidc`
# on the queue's host.
# The queue's auth-callout responder, which reads it itself on the
# queue's host.
hyperhiveCfg.swarm.nats.clientId
# The homeserver, read by `swarm-bao-matrix-oidc` on its host.
hyperhiveCfg.swarm.matrix.sso.clientId

2
nix/module-eval/bao-grants.nix
View file

@ -823,7 +823,7 @@ let
&& lib.elem "swarm-bao-matrix-ctl-policy.service" u.wants
&& !(lib.elem "swarm-bao-matrix-ctl-policy.service" u.requires)
&& !(noReaders ? swarm-bao-matrix-oidc)
&& !(noReaders ? swarm-bao-nats-auth-oidc);
&& !(noReaders ? swarm-bao-nats-callout-xkey);
}
{
# The ordering is set apart from each reader's own definition, so it can

54
nix/module-eval/nats-authelia.nix
View file

@ -347,8 +347,8 @@ let
}
{
# The responder names no seed file, by flag or by credential, and the
# copies a container may still hold are removed. The OIDC secret is the
# control that the credential list is read at all.
# copies a container may still hold are removed. The store identity is
# the control that the credential list is read at all.
name = "the responder takes no seed from a file, and the container's seed copies are removed";
ok =
let
@ -357,7 +357,7 @@ let
dir = "/var/lib/nixos-containers/swarm-nats/var/lib/swarm-nats-auth";
rules = natsWithStoreIdentity.systemd.tmpfiles.rules;
in
lib.elem "oidc-client.secret:/var/lib/swarm-nats-auth/oidc-client.secret" creds
lib.elem "bao-client.pem:/var/lib/swarm-nats-auth/bao-client.pem" creds
&& !(lib.any (lib.hasInfix ".seed") creds)
&& !(lib.hasInfix "seed-file" u.serviceConfig.ExecStart)
&& lib.elem "r ${dir}/issuer.seed" rules
@ -404,41 +404,31 @@ let
&& !(lib.any (path: refused natsWithStoreIdentity path) (lib.attrNames setting));
}
{
# The responder's OIDC secret has one route, the store, under the
# responder's own leaf. The copy unit's arms are the control: on a queue
# host without authelia nothing reads authelia's tree, and nothing
# copies the secret from any host file.
name = "a queue host without authelia reads the responder's OIDC secret from the store, and copies none from a host file";
# The responder reads its OIDC secret from the store itself, so no host
# unit fetches it, no credential or flag carries it, and a copy a
# container still holds is removed. On or off the store's host alike;
# the callout key fetch beside it is the control that the unit set is
# read at all, and the store cert role the control on the flags.
name = "the responder's OIDC secret is fetched by no unit and carried by no credential or flag, and the container's copy is removed";
ok =
let
services = natsWithStoreIdentity.systemd.services;
fetch = services.swarm-bao-nats-auth-oidc;
copy = services.swarm-nats-auth-secrets.script;
fetchedNowhere =
h:
!(h.systemd.services ? swarm-bao-nats-auth-oidc)
&& h.systemd.services ? swarm-bao-nats-callout-xkey;
u = responderOf natsWithStoreIdentity;
copy = natsWithStoreIdentity.systemd.services.swarm-nats-auth-secrets.script;
rules = natsWithStoreIdentity.systemd.tmpfiles.rules;
in
!natsWithStoreIdentity.services.hyperhive.deploy.authelia.enable
&& fetch.environment.BAO_CLIENT_CERT == "/etc/pki/nats-auth.pem"
&& fetch.environment.BAO_CLIENT_KEY == "/etc/pki/nats-auth-key.pem"
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/swarm-nats/oidc/client" fetch.script
&& lib.hasInfix "/var/lib/nixos-containers/swarm-nats/var/lib/swarm-nats-auth/oidc-client.secret" fetch.script
&& lib.hasInfix "refresh_consumer swarm-nats swarm-nats-auth.service" fetch.script
# An unpublished or empty secret fails the unit, so `storeRetry` retries it.
&& !(lib.hasInfix "exit 0" fetch.script)
&& lib.elem "container@swarm-nats.service" fetch.before
fetchedNowhere natsWithStoreIdentity
&& fetchedNowhere natsOnStoreHost
&& !(lib.any (lib.hasInfix "oidc-client") u.serviceConfig.LoadCredential)
&& !(lib.hasInfix "client-secret" u.serviceConfig.ExecStart)
&& lib.hasInfix "--store-cert-role swarm-nats-auth" u.serviceConfig.ExecStart
&& lib.elem "r /var/lib/nixos-containers/swarm-nats/var/lib/swarm-nats-auth/oidc-client.secret" rules
&& !(lib.hasInfix "oidc-client" copy)
&& !(lib.hasInfix "authelia" copy);
}
{
# On the store's host the fetch logs in under the responder's leaf, so it
# waits for the unit writing that role.
name = "on the store's host the responder's OIDC secret fetch uses its leaf and waits for its role";
ok =
let
u = natsOnStoreHost.systemd.services.swarm-bao-nats-auth-oidc;
in
u.environment.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/nats-auth.pem"
&& lib.elem "swarm-bao-nats-auth-policy.service" u.after
&& lib.elem "swarm-bao-nats-auth-policy.service" u.wants;
}
{
# On the store's host the glue pairs the responder with a leaf of its
# own, never the queue's TLS-issuing one or the hive's.