vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

forge: fetch the OIDC client secret from bao instead of a host-side copy

Browse source 
The forge's login source read its OIDC client secret from a file that
`hive-forge-oidc-secret` copied out of authelia's container tree. That
unit only existed where authelia ran on the forge's host (`ssoLocal`),
so a forge anywhere else needed the file placed by hand.

The publisher on authelia's host now carries the forge's client secret
to `secret/swarm/services/<forge client id>/oidc/client`, like every
other swarm service's. On the forge's host `swarm-bao-forge-oidc`, a copy
of `swarm-bao-matrix-oidc`, reads it under the forge publisher's leaf
(the forge host's one store identity, whose policy gains read on that
one path) and writes the file `forgejo-sso-source` reads, re-running the
login source when the value changes. It retries on the shared
store-retry shape while the store is down or the secret is not yet
published.

`hive-forge-oidc-secret` is removed, `clientSecretFile` defaults to the
in-container path wherever the forge runs, and the `ssoLocal` gate on
the login source's resolver ordering goes with it: the login source
fetches the issuer's discovery document on every forge host.

Refs #5015
This commit is contained in:
atlas 2026-10-09 23:30:17 +02:00 • committed by mara
commit f6ca5f1433
10 changed files with 332 additions and 159 deletions
Download patch file Download diff file Expand all files Collapse all files

218
nix/host-modules/hive-forge/default.nix
View file

@ -28,18 +28,21 @@ let
autheliaUrl = autheliaCfg.url;
autheliaDiscoveryUrl = "${toString autheliaUrl}/.well-known/openid-configuration";
# The all-local case: this host runs BOTH the forge and the swarm's
# authelia, so the secret can be moved without an operator. The other
# two cases (swarm side via swarmctl, remote hive) leave
# `clientSecretFile` to be set explicitly — see docs/swarm/.
ssoLocal = deployCfg.authelia.enable;
# Where the plaintext lands inside the forge container. Under
# /var/lib rather than /run: the forge may start before the delivery
# unit on a later boot, and a secret that evaporates on reboot turns a
# working login into an intermittent one.
forgeSecretPath = "/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret";
# The same file seen from the host, where `swarm-bao-forge-oidc` writes it.
# Follows `clientSecretFile`, so an operator-chosen path is where the
# fetched secret lands.
hostSecretPath = "/var/lib/nixos-containers/hive-forge${toString deployCfg.forgejo.sso.clientSecretFile}";
# Where swarm-secret-publish on authelia's host leaves this client's secret.
# The forge publisher's store role grants read on this one path.
oidcStorePath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";
# The swarm-controller's forge account. Same name as
# `swarm-controller.nix`'s existing `queueClientId` — the account is
# provisioned to *match* that identity, not invented independently —
@ -66,6 +69,7 @@ let
baoCfg = config.services.hyperhive.swarm.bao;
storeRetry = import ../lib/store-retry.nix { };
atomicWriteSecret = import ../lib/atomic-write-secret.nix { };
refreshConsumer = import ../lib/refresh-consumer.nix { };
havePublisherIdentity =
deployCfg.bao.forgePublisherClientCertFile != null
&& deployCfg.bao.forgePublisherClientKeyFile != null;
@ -251,10 +255,13 @@ in
plaintext), and a literal written here would be rendered into
the world-readable nix store.
Required when `enable` is set — deliberately no fallback. A
forge that boots with SSO half-configured presents as a login
button that always fails, which is harder to diagnose than an
eval error.
Set to `/var/lib/forgejo-oidc/<client id>.secret` wherever the
forge runs. `swarm-bao-forge-oidc` writes the secret here from the
swarm secret store, whichever path this names.
Null is refused at eval: a forge that boots with SSO
half-configured presents as a login button that always fails,
which is harder to diagnose than an eval error.
'';
};
};
@ -363,9 +370,9 @@ in
(inside the forge container) holding the OIDC client secret's
plaintext.
On a hive that also runs the swarm's authelia this is wired up
for you. Set it explicitly when authelia lives on another
host: see docs/swarm/ for which secret goes where.
It is set by default wherever the forge runs, and
swarm-bao-forge-oidc writes the secret there from the swarm
secret store. Remove the definition that set it to null.
'';
}
{
@ -483,8 +490,8 @@ in
inherit (networkCfg) bridgeIp;
dnsConsumers = [
"forgejo.service"
]
++ lib.optional ssoLocal "forgejo-sso-source.service";
"forgejo-sso-source.service"
];
})
# Forgejo is Go, and `SSL_CERT_FILE` *replaces* the default store
@ -495,10 +502,7 @@ in
# BOTH units that make an outbound HTTPS call are consumers, not
# just the obvious one: `forgejo-sso-source` fetches the issuer's
# `.well-known/openid-configuration` under the store's root, and it
# once shipped without the trust its sibling had. `optional`
# rather than a flat list because that unit only exists when SSO
# is on — naming an absent unit would order nothing and quietly
# define a serviceless one.
# once shipped without the trust its sibling had.
(caTrust.trustBundle {
inherit pkgs;
name = "hive-forge";
@ -946,12 +950,10 @@ in
token_path=${lib.escapeShellArg swarmControllerTokenPath}
# Token minting is idempotent by the token FILE's existence,
# not a separate stamp — same reasoning `hive-forge-oidc-secret`
# (below) uses for its own delivery: the file is both the
# record of "already done" and the thing that has to survive,
# so a state wipe that deletes it correctly triggers a
# re-mint instead of being silently masked by a stamp that
# outlived what it claims exists.
# not a separate stamp: the file is both the record of "already
# done" and the thing that has to survive, so a state wipe that
# deletes it correctly triggers a re-mint instead of being
# silently masked by a stamp that outlived what it claims exists.
if [ ! -s "$token_path" ]; then
# `write:admin` added for `swarm-controller::forge::Client::
# ensure_agent_user` (the `CreateForgeUser` jobq node) —
@ -994,76 +996,128 @@ in
};
# The client this login source authenticates as is registered by
# ../glue-forge-oidc-client.nix, wherever authelia runs. When authelia
# is here too, this host minted the secret, so it can say where the
# forge will find it.
services.hyperhive.deploy.forgejo.sso.clientSecretFile = lib.mkIf ssoLocal (
lib.mkDefault forgeSecretPath
);
# ../glue-forge-oidc-client.nix, wherever authelia runs.
services.hyperhive.deploy.forgejo.sso.clientSecretFile = lib.mkDefault forgeSecretPath;
# The delivery. It runs on the HOST because that is the only place
# both container trees are addressable: they share this host's
# network namespace, which makes them feel co-located, but their
# filesystem roots are separate — the forge cannot open a path inside
# authelia's tree however local the port looks.
# The forge's OIDC client secret, out of the swarm secret store under the
# forge publisher's leaf — the one route, wherever authelia runs. Same
# shape as ../hive-matrix.nix's `swarm-bao-matrix-oidc`: the login and the
# read both fail loudly, and `storeRetry` retries until the publisher on
# authelia's host has written the secret.
#
# ⚠️ Deliberately a copy and not a `bindMounts` entry.
# nixos-container refuses to start when a bind source is missing, and
# this secret does not exist until authelia's first boot has minted
# it — so binding it would make the forge wait on a file that waits
# on a container that starts after it. On a fresh hive that is a
# permanent stall presenting as "the forge is broken", several layers
# from its cause.
systemd.services.hive-forge-oidc-secret = lib.mkIf ssoLocal {
description = "deliver the forge's OIDC client secret from authelia";
after = [ "container@${autheliaCfg.machine}.service" ];
requires = [ "container@${autheliaCfg.machine}.service" ];
# On the host, where that leaf is: the container holds no store identity.
# A host with no publisher leaf fails here, naming the options, rather than
# rendering no unit.
systemd.services.swarm-bao-forge-oidc = {
description = "fetch the forge's OIDC client secret from the swarm secret store";
# The store's units exist only where the store runs; `Requires=` on an
# absent unit fails the job, so off-host the timeout below bounds the
# attempt instead.
after = lib.optionals deployCfg.bao.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals deployCfg.bao.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals deployCfg.bao.enable [ "swarm-bao-pki.service" ];
before = [ "container@hive-forge.service" ];
wantedBy = [ "container@hive-forge.service" ];
serviceConfig = {
wantedBy = [
"multi-user.target"
"container@hive-forge.service"
];
path = [
deployCfg.bao.package
pkgs.coreutils
pkgs.systemd
];
# ../lib/store-retry.nix. The container waits for one attempt; a secret a
# later attempt lands re-runs `forgejo-sso-source` (below).
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "hive-forge-oidc-secret";
# Longer than the 120s bounded wait below. `DefaultTimeoutStartSec`
# is 90s, so without this systemd kills the unit at 90 — before it
# can emit the message naming the file it was waiting for.
TimeoutStartSec = "180s";
SyslogIdentifier = "swarm-bao-forge-oidc";
TimeoutStartSec = 30;
};
path = [ pkgs.coreutils ];
script = ''
set -euo pipefail
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs havePublisherIdentity {
BAO_CLIENT_CERT = deployCfg.bao.forgePublisherClientCertFile;
BAO_CLIENT_KEY = deployCfg.bao.forgePublisherClientKeyFile;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
};
script =
if !havePublisherIdentity then
''
echo "the forge's host has no forge-publisher store identity, so it cannot read the forge's OIDC client secret: set services.hyperhive.deploy.bao.forgePublisherClientCertFile and forgePublisherClientKeyFile (they default to the leaf the store mints only where services.hyperhive.deploy.bao.enable is set on this host)" >&2
exit 1
''
else
''
set -euo pipefail
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.sso.clientId}.secret"}
dst=${lib.escapeShellArg "/var/lib/nixos-containers/hive-forge${forgeSecretPath}"}
${atomicWriteSecret}
${refreshConsumer}
# authelia's container is up, but its first-boot generator may
# still be minting. Bounded wait, then fail: a silent skip here
# produces a forge whose SSO button 401s, which is the failure
# this whole design is trying not to ship.
for _ in $(seq 1 60); do
[ -s "$src" ] && break
sleep 2
done
if [ ! -s "$src" ]; then
echo "authelia has not minted $src after 120s" >&2
exit 1
fi
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# The owning uid is DISCOVERED, not assumed: read it off the
# forge container's own state dir. Whatever uid maps to forgejo
# inside that container is by definition the one that owns the
# directory it was created with, and hardcoding a number here
# would be a second place for it to be wrong.
uid=$(stat -c %u /var/lib/nixos-containers/hive-forge/var/lib/forgejo)
install -D -m 0400 -o "$uid" -g "$uid" "$src" "$dst"
'';
# `-token-only` answers on stdout and skips the token helper, a
# `sh` this unit's `path` does not carry.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with the forge publisher's certificate; leaving the forge's OIDC client secret as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg oidcStorePath} 2>"$err")"; then
echo "swarm-bao did not return ${oidcStorePath}; the forge has no OIDC client secret yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${oidcStorePath}; the forge has no OIDC client secret yet." >&2
exit 1
fi
# `forgejo-sso-source` reads the file as forgejo. The owning uid is
# read off the container's forgejo state dir rather than assumed,
# which exists only once the container has started.
state=/var/lib/nixos-containers/hive-forge/var/lib/forgejo
if [ ! -d "$state" ]; then
echo "the forge container has not created $state yet, so the forge's OIDC client secret has no owner to be written for." >&2
exit 1
fi
uid="$(stat -c %u "$state")"
mkdir -p ${lib.escapeShellArg (builtins.dirOf hostSecretPath)}
if secret_differs ${lib.escapeShellArg hostSecretPath} "$secret"; then
atomic_write_secret 0400 "$uid:$uid" ${lib.escapeShellArg hostSecretPath} "$secret"
fi
# The login source is a row `forgejo-sso-source` writes from the
# file each time it runs.
refresh_consumer hive-forge forgejo-sso-source.service ${lib.escapeShellArg hostSecretPath}
'';
};
# The reverse direction of the unit above: publish the swarm-controller's
# freshly-minted forge token OUT of the container into the swarm secret
# store, where the controller reads it wherever it runs. On the host for the
# reason `hive-forge-oidc-secret` gives, and under the forge's own store
# identity, so nothing here asks where the store or the controller is.
# store, where the controller reads it wherever it runs. On the host and
# under the forge's own store identity for the reason the unit above gives,
# so nothing here asks where the store or the controller is.
#
# It runs at boot and on a switch that adds or changes it
# (`multi-user.target`), each time the container starts, and each time the