forge: fetch the OIDC client secret from bao instead of a host-side copy
The forge's login source read its OIDC client secret from a file that `hive-forge-oidc-secret` copied out of authelia's container tree. That unit only existed where authelia ran on the forge's host (`ssoLocal`), so a forge anywhere else needed the file placed by hand. The publisher on authelia's host now carries the forge's client secret to `secret/swarm/services/<forge client id>/oidc/client`, like every other swarm service's. On the forge's host `swarm-bao-forge-oidc`, a copy of `swarm-bao-matrix-oidc`, reads it under the forge publisher's leaf (the forge host's one store identity, whose policy gains read on that one path) and writes the file `forgejo-sso-source` reads, re-running the login source when the value changes. It retries on the shared store-retry shape while the store is down or the secret is not yet published. `hive-forge-oidc-secret` is removed, `clientSecretFile` defaults to the in-container path wherever the forge runs, and the `ssoLocal` gate on the login source's resolver ordering goes with it: the login source fetches the issuer's discovery document on every forge host. Refs #5015
This commit is contained in:
atlas
mara
parent
6bf4792d61
commit
f6ca5f1433
10 changed files with 332 additions and 159 deletions
|
|
@ -71,6 +71,11 @@ let
|
|||
here = deployCfg.matrix.enable;
|
||||
policies = [ "swarm-bao-matrix-ctl-policy" ];
|
||||
};
|
||||
# ./hive-forge/default.nix, under the forge publisher's leaf.
|
||||
swarm-bao-forge-oidc = {
|
||||
here = deployCfg.forgejo.enable;
|
||||
policies = [ "swarm-bao-forge-publisher-policy" ];
|
||||
};
|
||||
};
|
||||
|
||||
orderAfterPolicies =
|
||||
|
|
|
|||
|
|
@ -28,18 +28,21 @@ let
|
|||
autheliaUrl = autheliaCfg.url;
|
||||
autheliaDiscoveryUrl = "${toString autheliaUrl}/.well-known/openid-configuration";
|
||||
|
||||
# The all-local case: this host runs BOTH the forge and the swarm's
|
||||
# authelia, so the secret can be moved without an operator. The other
|
||||
# two cases (swarm side via swarmctl, remote hive) leave
|
||||
# `clientSecretFile` to be set explicitly — see docs/swarm/.
|
||||
ssoLocal = deployCfg.authelia.enable;
|
||||
|
||||
# Where the plaintext lands inside the forge container. Under
|
||||
# /var/lib rather than /run: the forge may start before the delivery
|
||||
# unit on a later boot, and a secret that evaporates on reboot turns a
|
||||
# working login into an intermittent one.
|
||||
forgeSecretPath = "/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret";
|
||||
|
||||
# The same file seen from the host, where `swarm-bao-forge-oidc` writes it.
|
||||
# Follows `clientSecretFile`, so an operator-chosen path is where the
|
||||
# fetched secret lands.
|
||||
hostSecretPath = "/var/lib/nixos-containers/hive-forge${toString deployCfg.forgejo.sso.clientSecretFile}";
|
||||
|
||||
# Where swarm-secret-publish on authelia's host leaves this client's secret.
|
||||
# The forge publisher's store role grants read on this one path.
|
||||
oidcStorePath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";
|
||||
|
||||
# The swarm-controller's forge account. Same name as
|
||||
# `swarm-controller.nix`'s existing `queueClientId` — the account is
|
||||
# provisioned to *match* that identity, not invented independently —
|
||||
|
|
@ -66,6 +69,7 @@ let
|
|||
baoCfg = config.services.hyperhive.swarm.bao;
|
||||
storeRetry = import ../lib/store-retry.nix { };
|
||||
atomicWriteSecret = import ../lib/atomic-write-secret.nix { };
|
||||
refreshConsumer = import ../lib/refresh-consumer.nix { };
|
||||
havePublisherIdentity =
|
||||
deployCfg.bao.forgePublisherClientCertFile != null
|
||||
&& deployCfg.bao.forgePublisherClientKeyFile != null;
|
||||
|
|
@ -251,10 +255,13 @@ in
|
|||
plaintext), and a literal written here would be rendered into
|
||||
the world-readable nix store.
|
||||
|
||||
Required when `enable` is set — deliberately no fallback. A
|
||||
forge that boots with SSO half-configured presents as a login
|
||||
button that always fails, which is harder to diagnose than an
|
||||
eval error.
|
||||
Set to `/var/lib/forgejo-oidc/<client id>.secret` wherever the
|
||||
forge runs. `swarm-bao-forge-oidc` writes the secret here from the
|
||||
swarm secret store, whichever path this names.
|
||||
|
||||
Null is refused at eval: a forge that boots with SSO
|
||||
half-configured presents as a login button that always fails,
|
||||
which is harder to diagnose than an eval error.
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
|
@ -363,9 +370,9 @@ in
|
|||
(inside the forge container) holding the OIDC client secret's
|
||||
plaintext.
|
||||
|
||||
On a hive that also runs the swarm's authelia this is wired up
|
||||
for you. Set it explicitly when authelia lives on another
|
||||
host: see docs/swarm/ for which secret goes where.
|
||||
It is set by default wherever the forge runs, and
|
||||
swarm-bao-forge-oidc writes the secret there from the swarm
|
||||
secret store. Remove the definition that set it to null.
|
||||
'';
|
||||
}
|
||||
{
|
||||
|
|
@ -483,8 +490,8 @@ in
|
|||
inherit (networkCfg) bridgeIp;
|
||||
dnsConsumers = [
|
||||
"forgejo.service"
|
||||
]
|
||||
++ lib.optional ssoLocal "forgejo-sso-source.service";
|
||||
"forgejo-sso-source.service"
|
||||
];
|
||||
})
|
||||
|
||||
# Forgejo is Go, and `SSL_CERT_FILE` *replaces* the default store
|
||||
|
|
@ -495,10 +502,7 @@ in
|
|||
# BOTH units that make an outbound HTTPS call are consumers, not
|
||||
# just the obvious one: `forgejo-sso-source` fetches the issuer's
|
||||
# `.well-known/openid-configuration` under the store's root, and it
|
||||
# once shipped without the trust its sibling had. `optional`
|
||||
# rather than a flat list because that unit only exists when SSO
|
||||
# is on — naming an absent unit would order nothing and quietly
|
||||
# define a serviceless one.
|
||||
# once shipped without the trust its sibling had.
|
||||
(caTrust.trustBundle {
|
||||
inherit pkgs;
|
||||
name = "hive-forge";
|
||||
|
|
@ -946,12 +950,10 @@ in
|
|||
token_path=${lib.escapeShellArg swarmControllerTokenPath}
|
||||
|
||||
# Token minting is idempotent by the token FILE's existence,
|
||||
# not a separate stamp — same reasoning `hive-forge-oidc-secret`
|
||||
# (below) uses for its own delivery: the file is both the
|
||||
# record of "already done" and the thing that has to survive,
|
||||
# so a state wipe that deletes it correctly triggers a
|
||||
# re-mint instead of being silently masked by a stamp that
|
||||
# outlived what it claims exists.
|
||||
# not a separate stamp: the file is both the record of "already
|
||||
# done" and the thing that has to survive, so a state wipe that
|
||||
# deletes it correctly triggers a re-mint instead of being
|
||||
# silently masked by a stamp that outlived what it claims exists.
|
||||
if [ ! -s "$token_path" ]; then
|
||||
# `write:admin` added for `swarm-controller::forge::Client::
|
||||
# ensure_agent_user` (the `CreateForgeUser` jobq node) —
|
||||
|
|
@ -994,76 +996,128 @@ in
|
|||
};
|
||||
|
||||
# The client this login source authenticates as is registered by
|
||||
# ../glue-forge-oidc-client.nix, wherever authelia runs. When authelia
|
||||
# is here too, this host minted the secret, so it can say where the
|
||||
# forge will find it.
|
||||
services.hyperhive.deploy.forgejo.sso.clientSecretFile = lib.mkIf ssoLocal (
|
||||
lib.mkDefault forgeSecretPath
|
||||
);
|
||||
# ../glue-forge-oidc-client.nix, wherever authelia runs.
|
||||
services.hyperhive.deploy.forgejo.sso.clientSecretFile = lib.mkDefault forgeSecretPath;
|
||||
|
||||
# The delivery. It runs on the HOST because that is the only place
|
||||
# both container trees are addressable: they share this host's
|
||||
# network namespace, which makes them feel co-located, but their
|
||||
# filesystem roots are separate — the forge cannot open a path inside
|
||||
# authelia's tree however local the port looks.
|
||||
# The forge's OIDC client secret, out of the swarm secret store under the
|
||||
# forge publisher's leaf — the one route, wherever authelia runs. Same
|
||||
# shape as ../hive-matrix.nix's `swarm-bao-matrix-oidc`: the login and the
|
||||
# read both fail loudly, and `storeRetry` retries until the publisher on
|
||||
# authelia's host has written the secret.
|
||||
#
|
||||
# ⚠️ Deliberately a copy and not a `bindMounts` entry.
|
||||
# nixos-container refuses to start when a bind source is missing, and
|
||||
# this secret does not exist until authelia's first boot has minted
|
||||
# it — so binding it would make the forge wait on a file that waits
|
||||
# on a container that starts after it. On a fresh hive that is a
|
||||
# permanent stall presenting as "the forge is broken", several layers
|
||||
# from its cause.
|
||||
systemd.services.hive-forge-oidc-secret = lib.mkIf ssoLocal {
|
||||
description = "deliver the forge's OIDC client secret from authelia";
|
||||
after = [ "container@${autheliaCfg.machine}.service" ];
|
||||
requires = [ "container@${autheliaCfg.machine}.service" ];
|
||||
# On the host, where that leaf is: the container holds no store identity.
|
||||
# A host with no publisher leaf fails here, naming the options, rather than
|
||||
# rendering no unit.
|
||||
systemd.services.swarm-bao-forge-oidc = {
|
||||
description = "fetch the forge's OIDC client secret from the swarm secret store";
|
||||
# The store's units exist only where the store runs; `Requires=` on an
|
||||
# absent unit fails the job, so off-host the timeout below bounds the
|
||||
# attempt instead.
|
||||
after = lib.optionals deployCfg.bao.enable [
|
||||
"swarm-bao-pki.service"
|
||||
"container@${baoCfg.machine}.service"
|
||||
];
|
||||
wants = lib.optionals deployCfg.bao.enable [ "container@${baoCfg.machine}.service" ];
|
||||
requires = lib.optionals deployCfg.bao.enable [ "swarm-bao-pki.service" ];
|
||||
before = [ "container@hive-forge.service" ];
|
||||
wantedBy = [ "container@hive-forge.service" ];
|
||||
serviceConfig = {
|
||||
wantedBy = [
|
||||
"multi-user.target"
|
||||
"container@hive-forge.service"
|
||||
];
|
||||
path = [
|
||||
deployCfg.bao.package
|
||||
pkgs.coreutils
|
||||
pkgs.systemd
|
||||
];
|
||||
# ../lib/store-retry.nix. The container waits for one attempt; a secret a
|
||||
# later attempt lands re-runs `forgejo-sso-source` (below).
|
||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||
serviceConfig = storeRetry.serviceConfig // {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
SyslogIdentifier = "hive-forge-oidc-secret";
|
||||
# Longer than the 120s bounded wait below. `DefaultTimeoutStartSec`
|
||||
# is 90s, so without this systemd kills the unit at 90 — before it
|
||||
# can emit the message naming the file it was waiting for.
|
||||
TimeoutStartSec = "180s";
|
||||
SyslogIdentifier = "swarm-bao-forge-oidc";
|
||||
TimeoutStartSec = 30;
|
||||
};
|
||||
path = [ pkgs.coreutils ];
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
environment = {
|
||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||
}
|
||||
// lib.optionalAttrs havePublisherIdentity {
|
||||
BAO_CLIENT_CERT = deployCfg.bao.forgePublisherClientCertFile;
|
||||
BAO_CLIENT_KEY = deployCfg.bao.forgePublisherClientKeyFile;
|
||||
}
|
||||
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
|
||||
BAO_CACERT = deployCfg.bao.serverCaFile;
|
||||
};
|
||||
script =
|
||||
if !havePublisherIdentity then
|
||||
''
|
||||
echo "the forge's host has no forge-publisher store identity, so it cannot read the forge's OIDC client secret: set services.hyperhive.deploy.bao.forgePublisherClientCertFile and forgePublisherClientKeyFile (they default to the leaf the store mints only where services.hyperhive.deploy.bao.enable is set on this host)" >&2
|
||||
exit 1
|
||||
''
|
||||
else
|
||||
''
|
||||
set -euo pipefail
|
||||
|
||||
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.sso.clientId}.secret"}
|
||||
dst=${lib.escapeShellArg "/var/lib/nixos-containers/hive-forge${forgeSecretPath}"}
|
||||
${atomicWriteSecret}
|
||||
${refreshConsumer}
|
||||
|
||||
# authelia's container is up, but its first-boot generator may
|
||||
# still be minting. Bounded wait, then fail: a silent skip here
|
||||
# produces a forge whose SSO button 401s, which is the failure
|
||||
# this whole design is trying not to ship.
|
||||
for _ in $(seq 1 60); do
|
||||
[ -s "$src" ] && break
|
||||
sleep 2
|
||||
done
|
||||
if [ ! -s "$src" ]; then
|
||||
echo "authelia has not minted $src after 120s" >&2
|
||||
exit 1
|
||||
fi
|
||||
err="$(mktemp)"
|
||||
trap 'rm -f "$err"' EXIT
|
||||
|
||||
# The owning uid is DISCOVERED, not assumed: read it off the
|
||||
# forge container's own state dir. Whatever uid maps to forgejo
|
||||
# inside that container is by definition the one that owns the
|
||||
# directory it was created with, and hardcoding a number here
|
||||
# would be a second place for it to be wrong.
|
||||
uid=$(stat -c %u /var/lib/nixos-containers/hive-forge/var/lib/forgejo)
|
||||
install -D -m 0400 -o "$uid" -g "$uid" "$src" "$dst"
|
||||
'';
|
||||
# `-token-only` answers on stdout and skips the token helper, a
|
||||
# `sh` this unit's `path` does not carry.
|
||||
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
|
||||
echo "could not log in to swarm-bao with the forge publisher's certificate; leaving the forge's OIDC client secret as it is." >&2
|
||||
if [ -s "$err" ]; then
|
||||
cat "$err" >&2
|
||||
else
|
||||
echo "bao failed without writing a diagnostic." >&2
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
export BAO_TOKEN
|
||||
|
||||
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg oidcStorePath} 2>"$err")"; then
|
||||
echo "swarm-bao did not return ${oidcStorePath}; the forge has no OIDC client secret yet." >&2
|
||||
if [ -s "$err" ]; then
|
||||
cat "$err" >&2
|
||||
else
|
||||
echo "bao failed without writing a diagnostic." >&2
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -z "$secret" ]; then
|
||||
echo "swarm-bao returned an empty ${oidcStorePath}; the forge has no OIDC client secret yet." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# `forgejo-sso-source` reads the file as forgejo. The owning uid is
|
||||
# read off the container's forgejo state dir rather than assumed,
|
||||
# which exists only once the container has started.
|
||||
state=/var/lib/nixos-containers/hive-forge/var/lib/forgejo
|
||||
if [ ! -d "$state" ]; then
|
||||
echo "the forge container has not created $state yet, so the forge's OIDC client secret has no owner to be written for." >&2
|
||||
exit 1
|
||||
fi
|
||||
uid="$(stat -c %u "$state")"
|
||||
|
||||
mkdir -p ${lib.escapeShellArg (builtins.dirOf hostSecretPath)}
|
||||
if secret_differs ${lib.escapeShellArg hostSecretPath} "$secret"; then
|
||||
atomic_write_secret 0400 "$uid:$uid" ${lib.escapeShellArg hostSecretPath} "$secret"
|
||||
fi
|
||||
|
||||
# The login source is a row `forgejo-sso-source` writes from the
|
||||
# file each time it runs.
|
||||
refresh_consumer hive-forge forgejo-sso-source.service ${lib.escapeShellArg hostSecretPath}
|
||||
'';
|
||||
};
|
||||
|
||||
# The reverse direction of the unit above: publish the swarm-controller's
|
||||
# freshly-minted forge token OUT of the container into the swarm secret
|
||||
# store, where the controller reads it wherever it runs. On the host for the
|
||||
# reason `hive-forge-oidc-secret` gives, and under the forge's own store
|
||||
# identity, so nothing here asks where the store or the controller is.
|
||||
# store, where the controller reads it wherever it runs. On the host and
|
||||
# under the forge's own store identity for the reason the unit above gives,
|
||||
# so nothing here asks where the store or the controller is.
|
||||
#
|
||||
# It runs at boot and on a switch that adds or changes it
|
||||
# (`multi-user.target`), each time the container starts, and each time the
|
||||
|
|
|
|||
|
|
@ -692,9 +692,6 @@ in
|
|||
# exist. This oneshot converts the fast failure into a slow one, which
|
||||
# is what that restart policy is actually good at.
|
||||
#
|
||||
# Copied from `hive-forge-oidc-secret.service`, which solves exactly
|
||||
# this for the forge — see `docs/swarm/secrets.md`.
|
||||
#
|
||||
# Not rendered beside the store fetch below, which restarts the
|
||||
# collector itself when the file lands.
|
||||
systemd.services.${secretWaitUnit} = lib.mkIf waits {
|
||||
|
|
|
|||
|
|
@ -931,9 +931,11 @@ let
|
|||
}
|
||||
];
|
||||
|
||||
# The forge's host, publishing the controller's forge token. The one writer
|
||||
# of the path, and write-only like the secret publisher: it copies a file in
|
||||
# and never reads one back.
|
||||
# The forge host's one store identity. `swarm-bao-forge-publisher` writes
|
||||
# the controller's forge token under it, the one writer of that path and
|
||||
# write-only: it copies a file in and never reads one back.
|
||||
# `swarm-bao-forge-oidc` reads the forge's own OIDC client secret, that one
|
||||
# leaf and no other service's.
|
||||
forgePublisherWriters = [
|
||||
{
|
||||
name = "swarm-forge-publisher";
|
||||
|
|
@ -942,7 +944,8 @@ let
|
|||
path "${credentialMountPath}/data/${controllerForgeTokenLeaf}" {
|
||||
capabilities = ["create", "update"]
|
||||
}
|
||||
'';
|
||||
''
|
||||
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.forge.sso.clientId}/oidc/client";
|
||||
}
|
||||
];
|
||||
|
||||
|
|
@ -1754,9 +1757,11 @@ in
|
|||
default = "swarm-forge-publisher";
|
||||
description = ''
|
||||
Subject the store's `swarm-forge-publisher` cert-auth role accepts: the
|
||||
identity `swarm-bao-forge-publisher`, the unit on the forge's host that
|
||||
publishes the swarm-controller's forge token, presents. Its grant is
|
||||
one path, `swarm/controller/swarm-controller/forge/token`, and write
|
||||
forge host's one store identity, presented by `swarm-bao-forge-publisher`,
|
||||
which publishes the swarm-controller's forge token, and by
|
||||
`swarm-bao-forge-oidc`, which fetches the forge's OIDC client secret.
|
||||
Its grant is two paths: `swarm/controller/swarm-controller/forge/token`,
|
||||
write only, and `swarm/services/<forge client id>/oidc/client`, read
|
||||
only.
|
||||
|
||||
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
|
||||
|
|
@ -2001,8 +2006,9 @@ in
|
|||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/forge-publisher.pem";
|
||||
description = ''
|
||||
Certificate the unit that publishes the swarm-controller's forge token
|
||||
presents to the store. Its subject must be
|
||||
Certificate the forge's host presents to the store, both to publish the
|
||||
swarm-controller's forge token and to fetch the forge's OIDC client
|
||||
secret. Its subject must be
|
||||
{option}`services.hyperhive.deploy.bao.forgePublisherCommonName`.
|
||||
|
||||
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
|
||||
|
|
|
|||
|
|
@ -88,6 +88,8 @@ let
|
|||
hyperhiveCfg.swarm.nats.clientId
|
||||
# The homeserver, read by `swarm-bao-matrix-oidc` on its host.
|
||||
hyperhiveCfg.swarm.matrix.sso.clientId
|
||||
# The forge's login source, read by `swarm-bao-forge-oidc` on its host.
|
||||
hyperhiveCfg.swarm.forge.sso.clientId
|
||||
];
|
||||
|
||||
# The swarm controller's OIDC client, which it reads back from the store
|
||||
|
|
@ -121,8 +123,9 @@ in
|
|||
Turning it off leaves every hive but this one without its agents'
|
||||
credential, every hive unable to publish its status, the swarm
|
||||
controller unable to start, the swarm's Grafana without any login at
|
||||
all, its collector pushing unauthenticated, the
|
||||
queue's responder unable to start, and the homeserver unable to start.
|
||||
all, its collector pushing unauthenticated, the queue's responder
|
||||
unable to start, the homeserver unable to start, and the forge's login
|
||||
source without its secret.
|
||||
Each secret has exactly one route and this is the producer's
|
||||
end of it, so the honest reason to set it false is a deployment
|
||||
delivering those secrets by some other mechanism it owns.
|
||||
|
|
|
|||
Loading…
Reference in a new issue