forge: fetch the OIDC client secret from bao instead of a host-side copy
The forge's login source read its OIDC client secret from a file that `hive-forge-oidc-secret` copied out of authelia's container tree. That unit only existed where authelia ran on the forge's host (`ssoLocal`), so a forge anywhere else needed the file placed by hand. The publisher on authelia's host now carries the forge's client secret to `secret/swarm/services/<forge client id>/oidc/client`, like every other swarm service's. On the forge's host `swarm-bao-forge-oidc`, a copy of `swarm-bao-matrix-oidc`, reads it under the forge publisher's leaf (the forge host's one store identity, whose policy gains read on that one path) and writes the file `forgejo-sso-source` reads, re-running the login source when the value changes. It retries on the shared store-retry shape while the store is down or the secret is not yet published. `hive-forge-oidc-secret` is removed, `clientSecretFile` defaults to the in-container path wherever the forge runs, and the `ssoLocal` gate on the login source's resolver ordering goes with it: the login source fetches the issuer's discovery document on every forge host. Refs #5015
This commit is contained in:
atlas
mara
parent
6bf4792d61
commit
f6ca5f1433
10 changed files with 332 additions and 159 deletions
|
|
@ -56,7 +56,7 @@ of the cell says how.
|
|||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
|
||||
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
|
||||
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
|
||||
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
|
||||
|
|
@ -71,7 +71,7 @@ of the cell says how.
|
|||
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
|
||||
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
||||
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held until hive-c0re restarts |
|
||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
||||
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ❌ `swarm-bao-ci-runner` reads it when it starts, at boot or deploy, and restarts the runner when the value changed |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
||||
|
||||
|
|
|
|||
|
|
@ -243,15 +243,16 @@ such as the root token.
|
|||
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
|
||||
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
|
||||
|
||||
The forge's delivery unit waits for authelia's first boot to mint the secret — a
|
||||
bounded wait, 120s — and then **fails loudly** rather than skipping. A silent skip
|
||||
produces a service whose login button always fails, which is a symptom many
|
||||
layers from its cause. The homeserver's reads the store under matrix-ctl's leaf,
|
||||
the way Grafana's does below.
|
||||
The forge and the homeserver read their copies out of the store, the way
|
||||
Grafana does below: the forge under the forge publisher's leaf, the
|
||||
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
|
||||
publisher on authelia's host has written the secret, rather than skipping. A
|
||||
silent skip produces a service whose login button always fails, which is a
|
||||
symptom many layers from its cause.
|
||||
|
||||
The store's **first reader** is the agent containers' queue credential:
|
||||
`glue-queue-agent-credential.nix` lands it as two files, the client secret and
|
||||
|
|
@ -304,13 +305,16 @@ either one left a Grafana with no SSO and no password box — nothing failed, so
|
|||
nothing said anything. Registration is separate and stays where authelia is
|
||||
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
|
||||
so it's declared on the host running authelia, which isn't necessarily the host
|
||||
running Grafana. The swarm controller, the queue and the homeserver register the
|
||||
same way (`glue-controller-oidc-client.nix`, `glue-nats-oidc-client.nix`,
|
||||
`glue-matrix-oidc-client.nix`), and the queue's responder and the homeserver read
|
||||
their secrets from the store too: the responder itself under its own leaf,
|
||||
`swarm-bao-matrix-oidc.service` under matrix-ctl's. Each fails, naming the
|
||||
option pair, when that leaf is unset — neither consumer starts without its
|
||||
secret.
|
||||
running Grafana. The swarm controller, the queue, the homeserver and the forge
|
||||
register the same way (`glue-controller-oidc-client.nix`,
|
||||
`glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
|
||||
`glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
|
||||
forge read their secrets from the store too: the responder itself under its own
|
||||
leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
|
||||
`swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
|
||||
the option pair, when that leaf is unset — neither the responder nor the
|
||||
homeserver starts without its secret, and the forge registers no login source
|
||||
without its.
|
||||
|
||||
The **third reader** is the swarm collector's own OIDC client secret, the
|
||||
same shape one level down: `swarm-bao-otel-oidc.service` reads
|
||||
|
|
@ -386,17 +390,18 @@ next controller restart.
|
|||
### Per-principal identities
|
||||
|
||||
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
||||
identity and sharing one merges the identities. These six units reach one path
|
||||
each and each holds a leaf, a role and a policy of its own:
|
||||
identity and sharing one merges the identities. Each of these six leaves has a
|
||||
role and a policy of its own, and reaches one path, except the forge host's,
|
||||
which reaches two:
|
||||
|
||||
| unit | option pair under `deploy.bao.` | reads, or writes |
|
||||
| --------------------------- | -------------------------------------------------------------- | -------------------------------------------------------- |
|
||||
| --------------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
|
||||
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
|
||||
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
|
||||
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
|
||||
| `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` |
|
||||
| `swarm-bao-forge-publisher` | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token` |
|
||||
| `swarm-bao-forge-publisher`, `swarm-bao-forge-oidc` | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` |
|
||||
|
||||
The first exists **per hive**, because the path it reads carries a hive name
|
||||
and every hive runs its own reader. Its subject is
|
||||
|
|
@ -406,17 +411,19 @@ role. The other five reach a path that names a swarm service rather than a hive,
|
|||
so one role each is enough and their subjects are the flat
|
||||
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
||||
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName` and
|
||||
`deploy.bao.forgePublisherCommonName`. The last is the only writer: the other
|
||||
five read.
|
||||
`deploy.bao.forgePublisherCommonName`. The last is the only writer, and the one
|
||||
leaf two units share: the forge's host holds one store identity for the token it
|
||||
publishes and the OIDC client secret it reads.
|
||||
|
||||
On a host that mints its own PKI, `glue-bao-tls.nix` signs all six and defaults
|
||||
all twelve options, and there is nothing to do. Elsewhere you issue each leaf from
|
||||
that CA out of band and name it here — one file per principal rather than one file
|
||||
shared by six, which is the whole of what this buys. `swarm-ci-runner.nix`
|
||||
refuses the build wherever the runner runs without its pair, as
|
||||
`swarm-grafana.nix` does for Grafana. `swarm-bao-forge-publisher` refuses
|
||||
nothing: without its pair it renders, fails at start naming both options, and
|
||||
the controller runs without forge access until an operator places the leaf.
|
||||
`swarm-grafana.nix` does for Grafana. The forge's pair refuses nothing: without
|
||||
it `swarm-bao-forge-publisher` and `swarm-bao-forge-oidc` render, fail at start
|
||||
naming both options, and the controller runs without forge access and the forge
|
||||
without a login source until an operator places the leaf.
|
||||
|
||||
On a host without the store, where the forwarder never runs, forgetting one of
|
||||
the other six isn't a quiet degrade. A hand-configured remote-store hive that
|
||||
|
|
|
|||
|
|
@ -71,6 +71,11 @@ let
|
|||
here = deployCfg.matrix.enable;
|
||||
policies = [ "swarm-bao-matrix-ctl-policy" ];
|
||||
};
|
||||
# ./hive-forge/default.nix, under the forge publisher's leaf.
|
||||
swarm-bao-forge-oidc = {
|
||||
here = deployCfg.forgejo.enable;
|
||||
policies = [ "swarm-bao-forge-publisher-policy" ];
|
||||
};
|
||||
};
|
||||
|
||||
orderAfterPolicies =
|
||||
|
|
|
|||
|
|
@ -28,18 +28,21 @@ let
|
|||
autheliaUrl = autheliaCfg.url;
|
||||
autheliaDiscoveryUrl = "${toString autheliaUrl}/.well-known/openid-configuration";
|
||||
|
||||
# The all-local case: this host runs BOTH the forge and the swarm's
|
||||
# authelia, so the secret can be moved without an operator. The other
|
||||
# two cases (swarm side via swarmctl, remote hive) leave
|
||||
# `clientSecretFile` to be set explicitly — see docs/swarm/.
|
||||
ssoLocal = deployCfg.authelia.enable;
|
||||
|
||||
# Where the plaintext lands inside the forge container. Under
|
||||
# /var/lib rather than /run: the forge may start before the delivery
|
||||
# unit on a later boot, and a secret that evaporates on reboot turns a
|
||||
# working login into an intermittent one.
|
||||
forgeSecretPath = "/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret";
|
||||
|
||||
# The same file seen from the host, where `swarm-bao-forge-oidc` writes it.
|
||||
# Follows `clientSecretFile`, so an operator-chosen path is where the
|
||||
# fetched secret lands.
|
||||
hostSecretPath = "/var/lib/nixos-containers/hive-forge${toString deployCfg.forgejo.sso.clientSecretFile}";
|
||||
|
||||
# Where swarm-secret-publish on authelia's host leaves this client's secret.
|
||||
# The forge publisher's store role grants read on this one path.
|
||||
oidcStorePath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";
|
||||
|
||||
# The swarm-controller's forge account. Same name as
|
||||
# `swarm-controller.nix`'s existing `queueClientId` — the account is
|
||||
# provisioned to *match* that identity, not invented independently —
|
||||
|
|
@ -66,6 +69,7 @@ let
|
|||
baoCfg = config.services.hyperhive.swarm.bao;
|
||||
storeRetry = import ../lib/store-retry.nix { };
|
||||
atomicWriteSecret = import ../lib/atomic-write-secret.nix { };
|
||||
refreshConsumer = import ../lib/refresh-consumer.nix { };
|
||||
havePublisherIdentity =
|
||||
deployCfg.bao.forgePublisherClientCertFile != null
|
||||
&& deployCfg.bao.forgePublisherClientKeyFile != null;
|
||||
|
|
@ -251,10 +255,13 @@ in
|
|||
plaintext), and a literal written here would be rendered into
|
||||
the world-readable nix store.
|
||||
|
||||
Required when `enable` is set — deliberately no fallback. A
|
||||
forge that boots with SSO half-configured presents as a login
|
||||
button that always fails, which is harder to diagnose than an
|
||||
eval error.
|
||||
Set to `/var/lib/forgejo-oidc/<client id>.secret` wherever the
|
||||
forge runs. `swarm-bao-forge-oidc` writes the secret here from the
|
||||
swarm secret store, whichever path this names.
|
||||
|
||||
Null is refused at eval: a forge that boots with SSO
|
||||
half-configured presents as a login button that always fails,
|
||||
which is harder to diagnose than an eval error.
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
|
@ -363,9 +370,9 @@ in
|
|||
(inside the forge container) holding the OIDC client secret's
|
||||
plaintext.
|
||||
|
||||
On a hive that also runs the swarm's authelia this is wired up
|
||||
for you. Set it explicitly when authelia lives on another
|
||||
host: see docs/swarm/ for which secret goes where.
|
||||
It is set by default wherever the forge runs, and
|
||||
swarm-bao-forge-oidc writes the secret there from the swarm
|
||||
secret store. Remove the definition that set it to null.
|
||||
'';
|
||||
}
|
||||
{
|
||||
|
|
@ -483,8 +490,8 @@ in
|
|||
inherit (networkCfg) bridgeIp;
|
||||
dnsConsumers = [
|
||||
"forgejo.service"
|
||||
]
|
||||
++ lib.optional ssoLocal "forgejo-sso-source.service";
|
||||
"forgejo-sso-source.service"
|
||||
];
|
||||
})
|
||||
|
||||
# Forgejo is Go, and `SSL_CERT_FILE` *replaces* the default store
|
||||
|
|
@ -495,10 +502,7 @@ in
|
|||
# BOTH units that make an outbound HTTPS call are consumers, not
|
||||
# just the obvious one: `forgejo-sso-source` fetches the issuer's
|
||||
# `.well-known/openid-configuration` under the store's root, and it
|
||||
# once shipped without the trust its sibling had. `optional`
|
||||
# rather than a flat list because that unit only exists when SSO
|
||||
# is on — naming an absent unit would order nothing and quietly
|
||||
# define a serviceless one.
|
||||
# once shipped without the trust its sibling had.
|
||||
(caTrust.trustBundle {
|
||||
inherit pkgs;
|
||||
name = "hive-forge";
|
||||
|
|
@ -946,12 +950,10 @@ in
|
|||
token_path=${lib.escapeShellArg swarmControllerTokenPath}
|
||||
|
||||
# Token minting is idempotent by the token FILE's existence,
|
||||
# not a separate stamp — same reasoning `hive-forge-oidc-secret`
|
||||
# (below) uses for its own delivery: the file is both the
|
||||
# record of "already done" and the thing that has to survive,
|
||||
# so a state wipe that deletes it correctly triggers a
|
||||
# re-mint instead of being silently masked by a stamp that
|
||||
# outlived what it claims exists.
|
||||
# not a separate stamp: the file is both the record of "already
|
||||
# done" and the thing that has to survive, so a state wipe that
|
||||
# deletes it correctly triggers a re-mint instead of being
|
||||
# silently masked by a stamp that outlived what it claims exists.
|
||||
if [ ! -s "$token_path" ]; then
|
||||
# `write:admin` added for `swarm-controller::forge::Client::
|
||||
# ensure_agent_user` (the `CreateForgeUser` jobq node) —
|
||||
|
|
@ -994,76 +996,128 @@ in
|
|||
};
|
||||
|
||||
# The client this login source authenticates as is registered by
|
||||
# ../glue-forge-oidc-client.nix, wherever authelia runs. When authelia
|
||||
# is here too, this host minted the secret, so it can say where the
|
||||
# forge will find it.
|
||||
services.hyperhive.deploy.forgejo.sso.clientSecretFile = lib.mkIf ssoLocal (
|
||||
lib.mkDefault forgeSecretPath
|
||||
);
|
||||
# ../glue-forge-oidc-client.nix, wherever authelia runs.
|
||||
services.hyperhive.deploy.forgejo.sso.clientSecretFile = lib.mkDefault forgeSecretPath;
|
||||
|
||||
# The delivery. It runs on the HOST because that is the only place
|
||||
# both container trees are addressable: they share this host's
|
||||
# network namespace, which makes them feel co-located, but their
|
||||
# filesystem roots are separate — the forge cannot open a path inside
|
||||
# authelia's tree however local the port looks.
|
||||
# The forge's OIDC client secret, out of the swarm secret store under the
|
||||
# forge publisher's leaf — the one route, wherever authelia runs. Same
|
||||
# shape as ../hive-matrix.nix's `swarm-bao-matrix-oidc`: the login and the
|
||||
# read both fail loudly, and `storeRetry` retries until the publisher on
|
||||
# authelia's host has written the secret.
|
||||
#
|
||||
# ⚠️ Deliberately a copy and not a `bindMounts` entry.
|
||||
# nixos-container refuses to start when a bind source is missing, and
|
||||
# this secret does not exist until authelia's first boot has minted
|
||||
# it — so binding it would make the forge wait on a file that waits
|
||||
# on a container that starts after it. On a fresh hive that is a
|
||||
# permanent stall presenting as "the forge is broken", several layers
|
||||
# from its cause.
|
||||
systemd.services.hive-forge-oidc-secret = lib.mkIf ssoLocal {
|
||||
description = "deliver the forge's OIDC client secret from authelia";
|
||||
after = [ "container@${autheliaCfg.machine}.service" ];
|
||||
requires = [ "container@${autheliaCfg.machine}.service" ];
|
||||
# On the host, where that leaf is: the container holds no store identity.
|
||||
# A host with no publisher leaf fails here, naming the options, rather than
|
||||
# rendering no unit.
|
||||
systemd.services.swarm-bao-forge-oidc = {
|
||||
description = "fetch the forge's OIDC client secret from the swarm secret store";
|
||||
# The store's units exist only where the store runs; `Requires=` on an
|
||||
# absent unit fails the job, so off-host the timeout below bounds the
|
||||
# attempt instead.
|
||||
after = lib.optionals deployCfg.bao.enable [
|
||||
"swarm-bao-pki.service"
|
||||
"container@${baoCfg.machine}.service"
|
||||
];
|
||||
wants = lib.optionals deployCfg.bao.enable [ "container@${baoCfg.machine}.service" ];
|
||||
requires = lib.optionals deployCfg.bao.enable [ "swarm-bao-pki.service" ];
|
||||
before = [ "container@hive-forge.service" ];
|
||||
wantedBy = [ "container@hive-forge.service" ];
|
||||
serviceConfig = {
|
||||
wantedBy = [
|
||||
"multi-user.target"
|
||||
"container@hive-forge.service"
|
||||
];
|
||||
path = [
|
||||
deployCfg.bao.package
|
||||
pkgs.coreutils
|
||||
pkgs.systemd
|
||||
];
|
||||
# ../lib/store-retry.nix. The container waits for one attempt; a secret a
|
||||
# later attempt lands re-runs `forgejo-sso-source` (below).
|
||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||
serviceConfig = storeRetry.serviceConfig // {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
SyslogIdentifier = "hive-forge-oidc-secret";
|
||||
# Longer than the 120s bounded wait below. `DefaultTimeoutStartSec`
|
||||
# is 90s, so without this systemd kills the unit at 90 — before it
|
||||
# can emit the message naming the file it was waiting for.
|
||||
TimeoutStartSec = "180s";
|
||||
SyslogIdentifier = "swarm-bao-forge-oidc";
|
||||
TimeoutStartSec = 30;
|
||||
};
|
||||
path = [ pkgs.coreutils ];
|
||||
script = ''
|
||||
environment = {
|
||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||
}
|
||||
// lib.optionalAttrs havePublisherIdentity {
|
||||
BAO_CLIENT_CERT = deployCfg.bao.forgePublisherClientCertFile;
|
||||
BAO_CLIENT_KEY = deployCfg.bao.forgePublisherClientKeyFile;
|
||||
}
|
||||
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
|
||||
BAO_CACERT = deployCfg.bao.serverCaFile;
|
||||
};
|
||||
script =
|
||||
if !havePublisherIdentity then
|
||||
''
|
||||
echo "the forge's host has no forge-publisher store identity, so it cannot read the forge's OIDC client secret: set services.hyperhive.deploy.bao.forgePublisherClientCertFile and forgePublisherClientKeyFile (they default to the leaf the store mints only where services.hyperhive.deploy.bao.enable is set on this host)" >&2
|
||||
exit 1
|
||||
''
|
||||
else
|
||||
''
|
||||
set -euo pipefail
|
||||
|
||||
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.sso.clientId}.secret"}
|
||||
dst=${lib.escapeShellArg "/var/lib/nixos-containers/hive-forge${forgeSecretPath}"}
|
||||
${atomicWriteSecret}
|
||||
${refreshConsumer}
|
||||
|
||||
# authelia's container is up, but its first-boot generator may
|
||||
# still be minting. Bounded wait, then fail: a silent skip here
|
||||
# produces a forge whose SSO button 401s, which is the failure
|
||||
# this whole design is trying not to ship.
|
||||
for _ in $(seq 1 60); do
|
||||
[ -s "$src" ] && break
|
||||
sleep 2
|
||||
done
|
||||
if [ ! -s "$src" ]; then
|
||||
echo "authelia has not minted $src after 120s" >&2
|
||||
err="$(mktemp)"
|
||||
trap 'rm -f "$err"' EXIT
|
||||
|
||||
# `-token-only` answers on stdout and skips the token helper, a
|
||||
# `sh` this unit's `path` does not carry.
|
||||
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
|
||||
echo "could not log in to swarm-bao with the forge publisher's certificate; leaving the forge's OIDC client secret as it is." >&2
|
||||
if [ -s "$err" ]; then
|
||||
cat "$err" >&2
|
||||
else
|
||||
echo "bao failed without writing a diagnostic." >&2
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
export BAO_TOKEN
|
||||
|
||||
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg oidcStorePath} 2>"$err")"; then
|
||||
echo "swarm-bao did not return ${oidcStorePath}; the forge has no OIDC client secret yet." >&2
|
||||
if [ -s "$err" ]; then
|
||||
cat "$err" >&2
|
||||
else
|
||||
echo "bao failed without writing a diagnostic." >&2
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The owning uid is DISCOVERED, not assumed: read it off the
|
||||
# forge container's own state dir. Whatever uid maps to forgejo
|
||||
# inside that container is by definition the one that owns the
|
||||
# directory it was created with, and hardcoding a number here
|
||||
# would be a second place for it to be wrong.
|
||||
uid=$(stat -c %u /var/lib/nixos-containers/hive-forge/var/lib/forgejo)
|
||||
install -D -m 0400 -o "$uid" -g "$uid" "$src" "$dst"
|
||||
if [ -z "$secret" ]; then
|
||||
echo "swarm-bao returned an empty ${oidcStorePath}; the forge has no OIDC client secret yet." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# `forgejo-sso-source` reads the file as forgejo. The owning uid is
|
||||
# read off the container's forgejo state dir rather than assumed,
|
||||
# which exists only once the container has started.
|
||||
state=/var/lib/nixos-containers/hive-forge/var/lib/forgejo
|
||||
if [ ! -d "$state" ]; then
|
||||
echo "the forge container has not created $state yet, so the forge's OIDC client secret has no owner to be written for." >&2
|
||||
exit 1
|
||||
fi
|
||||
uid="$(stat -c %u "$state")"
|
||||
|
||||
mkdir -p ${lib.escapeShellArg (builtins.dirOf hostSecretPath)}
|
||||
if secret_differs ${lib.escapeShellArg hostSecretPath} "$secret"; then
|
||||
atomic_write_secret 0400 "$uid:$uid" ${lib.escapeShellArg hostSecretPath} "$secret"
|
||||
fi
|
||||
|
||||
# The login source is a row `forgejo-sso-source` writes from the
|
||||
# file each time it runs.
|
||||
refresh_consumer hive-forge forgejo-sso-source.service ${lib.escapeShellArg hostSecretPath}
|
||||
'';
|
||||
};
|
||||
|
||||
# The reverse direction of the unit above: publish the swarm-controller's
|
||||
# freshly-minted forge token OUT of the container into the swarm secret
|
||||
# store, where the controller reads it wherever it runs. On the host for the
|
||||
# reason `hive-forge-oidc-secret` gives, and under the forge's own store
|
||||
# identity, so nothing here asks where the store or the controller is.
|
||||
# store, where the controller reads it wherever it runs. On the host and
|
||||
# under the forge's own store identity for the reason the unit above gives,
|
||||
# so nothing here asks where the store or the controller is.
|
||||
#
|
||||
# It runs at boot and on a switch that adds or changes it
|
||||
# (`multi-user.target`), each time the container starts, and each time the
|
||||
|
|
|
|||
|
|
@ -692,9 +692,6 @@ in
|
|||
# exist. This oneshot converts the fast failure into a slow one, which
|
||||
# is what that restart policy is actually good at.
|
||||
#
|
||||
# Copied from `hive-forge-oidc-secret.service`, which solves exactly
|
||||
# this for the forge — see `docs/swarm/secrets.md`.
|
||||
#
|
||||
# Not rendered beside the store fetch below, which restarts the
|
||||
# collector itself when the file lands.
|
||||
systemd.services.${secretWaitUnit} = lib.mkIf waits {
|
||||
|
|
|
|||
|
|
@ -931,9 +931,11 @@ let
|
|||
}
|
||||
];
|
||||
|
||||
# The forge's host, publishing the controller's forge token. The one writer
|
||||
# of the path, and write-only like the secret publisher: it copies a file in
|
||||
# and never reads one back.
|
||||
# The forge host's one store identity. `swarm-bao-forge-publisher` writes
|
||||
# the controller's forge token under it, the one writer of that path and
|
||||
# write-only: it copies a file in and never reads one back.
|
||||
# `swarm-bao-forge-oidc` reads the forge's own OIDC client secret, that one
|
||||
# leaf and no other service's.
|
||||
forgePublisherWriters = [
|
||||
{
|
||||
name = "swarm-forge-publisher";
|
||||
|
|
@ -942,7 +944,8 @@ let
|
|||
path "${credentialMountPath}/data/${controllerForgeTokenLeaf}" {
|
||||
capabilities = ["create", "update"]
|
||||
}
|
||||
'';
|
||||
''
|
||||
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.forge.sso.clientId}/oidc/client";
|
||||
}
|
||||
];
|
||||
|
||||
|
|
@ -1754,9 +1757,11 @@ in
|
|||
default = "swarm-forge-publisher";
|
||||
description = ''
|
||||
Subject the store's `swarm-forge-publisher` cert-auth role accepts: the
|
||||
identity `swarm-bao-forge-publisher`, the unit on the forge's host that
|
||||
publishes the swarm-controller's forge token, presents. Its grant is
|
||||
one path, `swarm/controller/swarm-controller/forge/token`, and write
|
||||
forge host's one store identity, presented by `swarm-bao-forge-publisher`,
|
||||
which publishes the swarm-controller's forge token, and by
|
||||
`swarm-bao-forge-oidc`, which fetches the forge's OIDC client secret.
|
||||
Its grant is two paths: `swarm/controller/swarm-controller/forge/token`,
|
||||
write only, and `swarm/services/<forge client id>/oidc/client`, read
|
||||
only.
|
||||
|
||||
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
|
||||
|
|
@ -2001,8 +2006,9 @@ in
|
|||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/forge-publisher.pem";
|
||||
description = ''
|
||||
Certificate the unit that publishes the swarm-controller's forge token
|
||||
presents to the store. Its subject must be
|
||||
Certificate the forge's host presents to the store, both to publish the
|
||||
swarm-controller's forge token and to fetch the forge's OIDC client
|
||||
secret. Its subject must be
|
||||
{option}`services.hyperhive.deploy.bao.forgePublisherCommonName`.
|
||||
|
||||
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
|
||||
|
|
|
|||
|
|
@ -88,6 +88,8 @@ let
|
|||
hyperhiveCfg.swarm.nats.clientId
|
||||
# The homeserver, read by `swarm-bao-matrix-oidc` on its host.
|
||||
hyperhiveCfg.swarm.matrix.sso.clientId
|
||||
# The forge's login source, read by `swarm-bao-forge-oidc` on its host.
|
||||
hyperhiveCfg.swarm.forge.sso.clientId
|
||||
];
|
||||
|
||||
# The swarm controller's OIDC client, which it reads back from the store
|
||||
|
|
@ -121,8 +123,9 @@ in
|
|||
Turning it off leaves every hive but this one without its agents'
|
||||
credential, every hive unable to publish its status, the swarm
|
||||
controller unable to start, the swarm's Grafana without any login at
|
||||
all, its collector pushing unauthenticated, the
|
||||
queue's responder unable to start, and the homeserver unable to start.
|
||||
all, its collector pushing unauthenticated, the queue's responder
|
||||
unable to start, the homeserver unable to start, and the forge's login
|
||||
source without its secret.
|
||||
Each secret has exactly one route and this is the producer's
|
||||
end of it, so the honest reason to set it false is a deployment
|
||||
delivering those secrets by some other mechanism it owns.
|
||||
|
|
|
|||
|
|
@ -1373,16 +1373,19 @@ let
|
|||
lib.hasInfix "/ci-runner.pem ]" s && lib.hasInfix "swarm-ci-runner \"\" clientAuth" s;
|
||||
}
|
||||
{
|
||||
# The forge's host is the one writer of the controller's forge token.
|
||||
# Pinned as the whole stanza, so an added `read` fails: it copies a file
|
||||
# in and never reads one back.
|
||||
name = "the forge publisher's grant is writing the controller's forge token and nothing else";
|
||||
# The forge's host is the one writer of the controller's forge token,
|
||||
# and reads the forge's own OIDC client secret under the same leaf.
|
||||
# Pinned as two whole stanzas, so an added `read` on the token or a
|
||||
# write or wildcard on the secret fails.
|
||||
name = "the forge publisher's grant is writing the controller's forge token and reading the forge's OIDC secret, and nothing else";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-forge-publisher-policy.script;
|
||||
in
|
||||
lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/forge/token\" {\n capabilities = [\"create\", \"update\"]\n}" s
|
||||
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 1
|
||||
&& lib.hasInfix "path \"secret/data/swarm/services/forgejo/oidc/client\" {\n capabilities = [\"read\"]\n}" s
|
||||
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 2
|
||||
&& !(lib.hasInfix "secret/data/swarm/services/*" s)
|
||||
&& lib.hasInfix "auth/cert/certs/swarm-forge-publisher" s
|
||||
&& lib.hasInfix "allowed_common_names=swarm-forge-publisher" s
|
||||
&& lib.hasInfix "token_policies=swarm-forge-publisher" s;
|
||||
|
|
@ -1413,6 +1416,16 @@ let
|
|||
"swarm-bao-forge-publisher-policy"
|
||||
];
|
||||
}
|
||||
{
|
||||
# The forge's OIDC client secret by name: the forge publisher reads it,
|
||||
# and the secret publisher writes it under its `swarm/services/*` grant,
|
||||
# which names no service.
|
||||
name = "no granting unit but the forge publisher's names the forge's OIDC secret";
|
||||
ok =
|
||||
lib.filter (
|
||||
u: lib.hasInfix "swarm/services/forgejo/oidc/client" baoGrantHere.systemd.services.${u}.script
|
||||
) grantingUnitNames == [ "swarm-bao-forge-publisher-policy" ];
|
||||
}
|
||||
{
|
||||
name = "the PKI unit signs the forge publisher's leaf under its own subject";
|
||||
ok =
|
||||
|
|
|
|||
|
|
@ -40,10 +40,21 @@ let
|
|||
deploy.forgejo.enable = false;
|
||||
};
|
||||
|
||||
# The forge without authelia: the other half of the split.
|
||||
# The forge without authelia: the other half of the split, with its secret's
|
||||
# path and its store identity chosen by hand.
|
||||
forgeNoAuthelia = hive {
|
||||
deploy.forgejo.enable = true;
|
||||
deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret";
|
||||
deploy.bao.forgePublisherClientCertFile = "/etc/pki/forge-publisher.pem";
|
||||
deploy.bao.forgePublisherClientKeyFile = "/etc/pki/forge-publisher-key.pem";
|
||||
};
|
||||
|
||||
# The forge with neither authelia nor the store, its store identity placed by
|
||||
# hand: the deployment the store fetch exists for.
|
||||
forgeAlone = hive {
|
||||
deploy.forgejo.enable = true;
|
||||
deploy.bao.forgePublisherClientCertFile = "/etc/pki/forge-publisher.pem";
|
||||
deploy.bao.forgePublisherClientKeyFile = "/etc/pki/forge-publisher-key.pem";
|
||||
};
|
||||
|
||||
# The runner alone: no hive, no forge, no store, its store identity placed
|
||||
|
|
@ -105,6 +116,7 @@ let
|
|||
refusedOver = m: needle: lib.any (a: !a.assertion && lib.hasInfix needle a.message) m.assertions;
|
||||
|
||||
publisher = m: m.systemd.services.swarm-bao-forge-publisher;
|
||||
oidcFetch = m: m.systemd.services.swarm-bao-forge-oidc;
|
||||
|
||||
forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings;
|
||||
|
||||
|
|
@ -329,6 +341,79 @@ let
|
|||
&& lib.hasInfix "exit 1" s
|
||||
&& !(lib.hasInfix "bao kv put" s);
|
||||
}
|
||||
{
|
||||
# The forge's OIDC secret has one route, the store, under the forge
|
||||
# publisher's leaf, landing where `clientSecretFile` points inside the
|
||||
# container. No unit copies it out of authelia's tree, even on a host
|
||||
# that runs authelia, and a forge host without authelia needs no path
|
||||
# set by hand.
|
||||
name = "the forge reads its OIDC secret from the store under the forge publisher's leaf";
|
||||
ok =
|
||||
let
|
||||
u = oidcFetch forgeAlone;
|
||||
d = forgeAlone.services.hyperhive.deploy;
|
||||
in
|
||||
!d.authelia.enable
|
||||
&& !d.bao.enable
|
||||
&& d.forgejo.sso.clientSecretFile == "/var/lib/forgejo-oidc/forgejo.secret"
|
||||
&& !(refusedOver forgeAlone "deploy.forgejo.sso.clientSecretFile")
|
||||
&& u.environment.BAO_CLIENT_CERT == "/etc/pki/forge-publisher.pem"
|
||||
&& u.environment.BAO_CLIENT_KEY == "/etc/pki/forge-publisher-key.pem"
|
||||
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/forgejo/oidc/client" u.script
|
||||
&& lib.hasInfix "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/forgejo.secret" u.script
|
||||
&& lib.hasInfix "refresh_consumer hive-forge forgejo-sso-source.service" u.script
|
||||
# An unpublished or empty secret fails the unit, so `storeRetry` retries it.
|
||||
&& !(lib.hasInfix "exit 0" u.script)
|
||||
&& u.serviceConfig.Restart == "on-failure"
|
||||
&& lib.elem "container@hive-forge.service" u.before
|
||||
&& lib.elem "multi-user.target" u.wantedBy
|
||||
&& (oidcFetch allLocal).environment.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/forge-publisher.pem"
|
||||
&& !(forgeAlone.systemd.services ? hive-forge-oidc-secret)
|
||||
&& !(allLocal.systemd.services ? hive-forge-oidc-secret)
|
||||
&& !(bare.systemd.services ? swarm-bao-forge-oidc);
|
||||
}
|
||||
{
|
||||
# An operator-chosen path is where the fetched secret lands.
|
||||
name = "the forge's OIDC secret lands where clientSecretFile points";
|
||||
ok =
|
||||
let
|
||||
s = (oidcFetch forgeNoAuthelia).script;
|
||||
in
|
||||
lib.hasInfix "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/by-hand.secret" s;
|
||||
}
|
||||
{
|
||||
# Beside the store, the fetch waits for the unit writing the forge
|
||||
# publisher's role, the leaf it logs in with.
|
||||
name = "the forge's OIDC secret fetch is ordered after the forge publisher's policy unit";
|
||||
ok =
|
||||
let
|
||||
u = oidcFetch allLocal;
|
||||
in
|
||||
lib.elem "swarm-bao-forge-publisher-policy.service" u.after
|
||||
&& lib.elem "swarm-bao-forge-publisher-policy.service" u.wants;
|
||||
}
|
||||
{
|
||||
# The absence arm: no publisher leaf, no secret, and the unit says which
|
||||
# options to set.
|
||||
name = "a forge host with no publisher leaf renders an OIDC secret fetch that fails naming the options";
|
||||
ok =
|
||||
let
|
||||
u = oidcFetch forgeHere;
|
||||
in
|
||||
!(u.environment ? BAO_CLIENT_CERT)
|
||||
&& lib.hasInfix "forgePublisherClientCertFile" u.script
|
||||
&& lib.hasInfix "exit 1" u.script
|
||||
&& !(lib.hasInfix "bao login" u.script);
|
||||
}
|
||||
{
|
||||
# The login source fetches the issuer's discovery document by name
|
||||
# wherever the forge runs, so it waits for the resolver on a forge host
|
||||
# without authelia too.
|
||||
name = "the login source waits for the container's resolver on a forge host without authelia";
|
||||
ok = lib.elem "forgejo-sso-source.service" (
|
||||
forgeAlone.containers.hive-forge.config.systemd.services.swarm-bridge-dns.before
|
||||
);
|
||||
}
|
||||
{
|
||||
# The host copy is removed where the forge runs: nothing reads it.
|
||||
name = "the forge's host removes its old copy of the controller's forge token";
|
||||
|
|
|
|||
|
|
@ -189,10 +189,10 @@ let
|
|||
]);
|
||||
}
|
||||
{
|
||||
# The other half of the queue's and the homeserver's one route: the
|
||||
# publisher on an IdP host that runs neither carries both secrets to the
|
||||
# paths their readers fetch.
|
||||
name = "the publisher on an authelia host carries the queue's and homeserver's secrets";
|
||||
# The other half of the queue's, the homeserver's and the forge's one
|
||||
# route: the publisher on an IdP host that runs none of them carries each
|
||||
# secret to the path its reader fetches.
|
||||
name = "the publisher on an authelia host carries the queue's, homeserver's and forge's secrets";
|
||||
ok =
|
||||
let
|
||||
s = secretPublisherRemote.systemd.services.swarm-secret-publish.script;
|
||||
|
|
@ -200,7 +200,10 @@ let
|
|||
lib.hasInfix "secret/swarm/services/swarm-nats/oidc/client" s
|
||||
&& lib.hasInfix "/swarm-nats.secret" s
|
||||
&& lib.hasInfix "secret/swarm/services/tuwunel/oidc/client" s
|
||||
&& lib.hasInfix "/tuwunel.secret" s;
|
||||
&& lib.hasInfix "/tuwunel.secret" s
|
||||
&& !secretPublisherRemote.services.hyperhive.deploy.forgejo.enable
|
||||
&& lib.hasInfix "secret/swarm/services/forgejo/oidc/client" s
|
||||
&& lib.hasInfix "/forgejo.secret" s;
|
||||
}
|
||||
{
|
||||
# Both ends of a wire nothing at eval time carries end to end: the
|
||||
|
|
|
|||
Loading…
Reference in a new issue