swarm-authelia: run the secret publisher and a journal forwarder inside the container
swarm-secret-publish and its path unit now run inside authelia's container, beside the mint, and read each client secret at the path authelia writes it. The host unit and host path unit, which read the secrets through the container's root on the host, are gone, and so is deploy.authelia.hostClientSecretDir, which only they read. The publisher keeps its store identity, the secret-publisher leaf. The host hands the leaf and the store's CA into the container as nspawn system credentials, and the unit loads them; where this host mints the leaf, the container waits for swarm-bao-pki. Authelia does not wait for the publisher, which retries every 30 seconds while the store is down. The container also runs the journal forwarder the other service containers run, pushing to a receiver of its own on the swarm collector (serviceProducers swarm-authelia) as a new machine OIDC client (swarm.authelia.otel.clientId) that authelia registers for itself and the publisher carries to the store. The fetch unit reads it back under the publisher's leaf, also handed to it as a system credential, so the swarm-secret-publisher policy gains read on that one leaf; its stanza keeps create and update, which the exact path would otherwise take from the services/* prefix. Without the leaf the forwarder is off. The container drops --link-journal=host, and the host collector stops reading it. The services' "nothing at <path>" errors name the publisher in authelia's container.
This commit is contained in:
atlas
mara
parent
a2dca32bb8
commit
b68a8ef290
36 changed files with 753 additions and 363 deletions
|
|
@ -49,9 +49,9 @@ let
|
|||
statusSecretRemoved = ''
|
||||
hive-c0re reads this hive's status-publishing client secret from the
|
||||
swarm's secret store, under the hive's own certificate
|
||||
(deploy.bao.clientCertFile), and swarm-secret-publish on the host that runs
|
||||
authelia puts it there. Remove this definition; the file it named is read
|
||||
by nothing now and can be deleted.
|
||||
(deploy.bao.clientCertFile), and swarm-secret-publish in authelia's container
|
||||
puts it there. Remove this definition; the file it named is read by
|
||||
nothing now and can be deleted.
|
||||
'';
|
||||
in
|
||||
{
|
||||
|
|
|
|||
|
|
@ -197,8 +197,8 @@ in
|
|||
|
||||
# The secret publisher's, minted here for the reason the controller's
|
||||
# line above gives — and it serves that case more often, not less: the
|
||||
# publisher runs beside AUTHELIA, which is the one host guaranteed not
|
||||
# to be this one whenever the store has a host of its own.
|
||||
# publisher runs in AUTHELIA's container, on the one host
|
||||
# guaranteed not to be this one once the store has a host of its own.
|
||||
[ -s ${pkiDir}/secret-publisher.pem ] || ${signLeaf} ${pkiDir} secret-publisher \
|
||||
${lib.escapeShellArg deployCfg.bao.secretPublisherCommonName} "" clientAuth
|
||||
|
||||
|
|
|
|||
|
|
@ -102,7 +102,7 @@ in
|
|||
to make and the parent stays c0re's.
|
||||
|
||||
Absent files mean this hive has no queue credential yet, which is what
|
||||
a swarm looks like before the publisher on the authelia host has run.
|
||||
a swarm looks like before the publisher in authelia's container has run.
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
#
|
||||
# ONE PAIRING PER FILE — publisher ← authelia, and nothing else. Deleting this
|
||||
# leaves the publisher off unless an operator enables it, so every OIDC client
|
||||
# secret authelia mints stays on this host and no reader gets one, this
|
||||
# secret authelia mints stays in its container and no reader gets one, this
|
||||
# host's included.
|
||||
#
|
||||
# ⚠️ Gated on authelia alone, NOT on `deploy.bao.enable`: a publisher beside a
|
||||
|
|
|
|||
|
|
@ -6,7 +6,8 @@
|
|||
#
|
||||
# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the
|
||||
# leaf, because the thing that owns a private key owns issuing from it. What
|
||||
# belongs here is the pairing: which paths this host's publisher reads.
|
||||
# belongs here is the pairing: which host paths go to the publisher in
|
||||
# authelia's container.
|
||||
#
|
||||
# ⚠️ Gated on the leaf existing, not on the store being enabled — the same rule
|
||||
# ./glue-controller-bao-identity.nix states. A publisher on the store's own
|
||||
|
|
|
|||
|
|
@ -40,7 +40,8 @@ let
|
|||
# root. Removed by the tmpfiles rule below.
|
||||
oldOidcSecretPath = "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret";
|
||||
|
||||
# Where swarm-secret-publish on authelia's host leaves this client's secret.
|
||||
# Where swarm-secret-publish in authelia's container leaves this client's
|
||||
# secret.
|
||||
# The forge publisher's store role grants read on this path.
|
||||
oidcStorePath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";
|
||||
|
||||
|
|
@ -1196,8 +1197,8 @@ in
|
|||
# The forge's OIDC client secret, out of the swarm secret store under the
|
||||
# forge publisher's leaf — the one route, wherever authelia runs. Same
|
||||
# shape as ../hive-matrix.nix's `swarm-bao-matrix-oidc`: the login and the
|
||||
# read both fail loudly, and `storeRetry` retries until the publisher on
|
||||
# authelia's host has written the secret.
|
||||
# read both fail loudly, and `storeRetry` retries until the publisher in
|
||||
# authelia's container has written the secret.
|
||||
#
|
||||
# On the host, which reads that leaf from its own files. A host with no
|
||||
# publisher leaf fails here, naming the options, rather than rendering no
|
||||
|
|
|
|||
|
|
@ -48,7 +48,8 @@ let
|
|||
# Where the secret sat on persistent disk before it moved to /run.
|
||||
legacyHostSecretPath = "/var/lib/nixos-containers/hive-matrix/var/lib/tuwunel-oidc/${cfg.sso.clientId}.secret";
|
||||
|
||||
# Where swarm-secret-publish on authelia's host leaves this client's secret.
|
||||
# Where swarm-secret-publish in authelia's container leaves this client's
|
||||
# secret.
|
||||
# matrix-ctl's store role grants read on this path.
|
||||
storeSecretPath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";
|
||||
|
||||
|
|
|
|||
|
|
@ -369,7 +369,7 @@ in
|
|||
# ./glue-hive-otel-authelia.nix restates this gate.
|
||||
waits = senderAuth && !fetches;
|
||||
secretWaitUnit = "hive-otel-client-secret";
|
||||
# Written by swarm-secret-publish on authelia's host. The nix half of
|
||||
# Written by swarm-secret-publish in authelia's container. The nix half of
|
||||
# `swarm_secret_client::queue::hive_client_path`.
|
||||
storeSecretPath = "secret/swarm/hives/${toString hiveName}/oidc/client";
|
||||
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
|
||||
|
|
@ -829,8 +829,8 @@ in
|
|||
# This hive's OIDC client secret, out of the swarm secret store under the
|
||||
# hive's own leaf. Same shape as ./hive-matrix.nix's
|
||||
# `swarm-bao-matrix-oidc`: the login and the read both fail loudly, and
|
||||
# `storeRetry` retries until the publisher on authelia's host has written
|
||||
# the secret, because the collector does not start without it.
|
||||
# `storeRetry` retries until the publisher in authelia's container has
|
||||
# written the secret, because the collector does not start without it.
|
||||
#
|
||||
# The collector re-reads `client_secret_file` on every token fetch, but
|
||||
# the file it reads is `LoadCredential`'s copy, made at start: a changed
|
||||
|
|
|
|||
|
|
@ -440,5 +440,21 @@ in
|
|||
`swarm-controller`'s OIDC client.
|
||||
'';
|
||||
};
|
||||
|
||||
otel.clientId = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
readOnly = true;
|
||||
default = "swarm-authelia-collector";
|
||||
description = ''
|
||||
OAuth2 client id the journal forwarder inside authelia's container
|
||||
authenticates as, and the audience it asks its token for.
|
||||
|
||||
Read-only because three ends have to agree on it: authelia registers
|
||||
the client, the swarm collector admits it on the container's receiver
|
||||
(`swarm.otel.serviceProducers`), and the secret publisher carries its
|
||||
secret to the store, where the forwarder reads it back under the
|
||||
publisher's identity.
|
||||
'';
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -181,9 +181,82 @@ let
|
|||
caBundleModule = caTrust.trustBundle {
|
||||
inherit pkgs;
|
||||
name = cfg.machine;
|
||||
consumers = [ "swarm-authelia-bridge" ];
|
||||
consumers = [
|
||||
"swarm-authelia-bridge"
|
||||
]
|
||||
# Go, for the swarm collector's name and authelia's own. Only where the
|
||||
# forwarder runs: naming it would otherwise define an empty unit.
|
||||
++ lib.optional publishes "opentelemetry-collector";
|
||||
};
|
||||
|
||||
# The container's journal forwarder pushes to its own receiver on the swarm
|
||||
# collector, the route `serviceProducers.swarm-authelia` in
|
||||
# ./swarm-otel-service.nix declares, as a client of its own. The port suffix
|
||||
# is elided on 443, as in ./swarm-bao.nix.
|
||||
forwarderClientId = cfg.otel.clientId;
|
||||
gatewayPortSuffix = lib.optionalString (
|
||||
gatewayCfg.httpsPort != 443
|
||||
) ":${toString gatewayCfg.httpsPort}";
|
||||
forwarderEndpoint = "https://${hyperhiveCfg.swarm.otel.domain}${gatewayPortSuffix}/${cfg.machine}";
|
||||
# In the host netns: distinct from every other collector's there, which the
|
||||
# assertion below checks.
|
||||
forwarderTelemetryPort = 8901;
|
||||
# A client this container registers and mints for itself. Its secret goes
|
||||
# through the store like every other forwarder's: the publisher carries it
|
||||
# there and the forwarder reads it back under the same identity.
|
||||
forwarderClient = {
|
||||
id = forwarderClientId;
|
||||
description = "HyperHive authelia journal forwarder";
|
||||
kind = "machine";
|
||||
redirectUris = [ ];
|
||||
# Its own id as its own permitted audience, the one the swarm collector's
|
||||
# receiver for this container checks.
|
||||
audience = [ forwarderClientId ];
|
||||
# The receiver verifies the token offline against `/jwks.json`;
|
||||
# authelia's default opaque handle fails there.
|
||||
accessTokenSignedResponseAlg = "RS256";
|
||||
};
|
||||
|
||||
# The secret publisher runs in this container, beside the mint whose secrets
|
||||
# it copies. It needs a store identity, and a reader is defined by holding a
|
||||
# certificate the store accepts, never by standing next to it — the rule
|
||||
# ./glue-queue-agent-credential.nix states in full. `deploy.bao.enable` here
|
||||
# would be the co-location assumption itself.
|
||||
publisherCfg = deployCfg.swarm-secret-publisher;
|
||||
publishes =
|
||||
publisherCfg.enable
|
||||
&& publisherCfg.baoClientCertFile != null
|
||||
&& publisherCfg.baoClientKeyFile != null;
|
||||
publisherUnit = "swarm-secret-publish";
|
||||
forwarderFetchUnit = "swarm-journal-forwarder-oidc";
|
||||
# The publisher's store identity, handed into the container as system
|
||||
# credentials (credential id → host source). The forwarder's fetch presents
|
||||
# it too: one identity for the whole container.
|
||||
storeFiles = lib.optionalAttrs publishes (
|
||||
lib.filterAttrs (_: source: source != null) {
|
||||
publisher-bao-cert = publisherCfg.baoClientCertFile;
|
||||
publisher-bao-key = publisherCfg.baoClientKeyFile;
|
||||
publisher-bao-server-ca = deployCfg.bao.serverCaFile;
|
||||
}
|
||||
);
|
||||
# nixos-container expands `extraFlags` unquoted, so a source path must be
|
||||
# one word.
|
||||
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
|
||||
# The store options both units take, with the credentials as each loads them.
|
||||
storeIdentityIn =
|
||||
unit:
|
||||
let
|
||||
credential = id: "/run/credentials/${unit}.service/${id}";
|
||||
in
|
||||
{
|
||||
package = deployCfg.bao.package;
|
||||
addr = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}";
|
||||
clientCertFile = credential "publisher-bao-cert";
|
||||
clientKeyFile = credential "publisher-bao-key";
|
||||
caFile =
|
||||
if storeFiles ? publisher-bao-server-ca then credential "publisher-bao-server-ca" else null;
|
||||
};
|
||||
|
||||
# The SWARM's domain, because that is where the protected apps now live
|
||||
# (`forge.<swarm>`, `chat.<swarm>`, `auth.<swarm>`). It moves in the
|
||||
# same commit as `domain` (./swarm-authelia-service.nix) and cannot lag
|
||||
|
|
@ -529,7 +602,7 @@ in
|
|||
# register every service checks itself against and its port, is
|
||||
# `swarm.authelia` in ./swarm-authelia-service.nix. What the host running
|
||||
# the container decides is here — which two builds it
|
||||
# runs, and three filesystem paths that only exist on the machine running
|
||||
# runs, and two filesystem paths that only exist on the machine running
|
||||
# `swarm-authelia`. A hive that does not run it has nothing at any of those
|
||||
# paths. `enable` already lives in ./deploy.nix, which also carries the
|
||||
# renames.
|
||||
|
|
@ -578,28 +651,6 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
hostClientSecretDir = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
readOnly = true;
|
||||
default = "/var/lib/nixos-containers/${cfg.machine}${clientsDir}";
|
||||
description = ''
|
||||
Where the minted client secrets sit **as seen from the host** —
|
||||
`<id>.secret` holds a plaintext, `<id>.digest` the hash authelia
|
||||
itself reads.
|
||||
|
||||
Published for the same reason as `hostUsersFile`: the plaintext's
|
||||
other reader lives in a **different container**, and containers
|
||||
that share this host's network namespace still have separate
|
||||
filesystem roots. The host is the only place both trees are
|
||||
addressable, so the host is where a delivery step has to run.
|
||||
|
||||
⚠️ Nothing here exists until authelia's **first boot** has run.
|
||||
A consumer must wait for it — it cannot be a `bindMounts` source,
|
||||
because nixos-container refuses to start when a bind source is
|
||||
missing, and that turns a fresh hive into a boot-order deadlock.
|
||||
'';
|
||||
};
|
||||
|
||||
hostUsersFile = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
readOnly = true;
|
||||
|
|
@ -628,7 +679,10 @@ in
|
|||
# — see `oidcEnabled`'s comment above.
|
||||
services.hyperhive.swarm.authelia.oidc.clients =
|
||||
lib.optionals cfg.oidc.hiveIdentities (hiveClients ++ agentClients)
|
||||
++ [ bridgeClient ]
|
||||
++ [
|
||||
bridgeClient
|
||||
forwarderClient
|
||||
]
|
||||
++ lib.optional (swarmDomain != null) swarmctlClient;
|
||||
|
||||
# A redirect URI on a machine client is not harmless-but-unused: it
|
||||
|
|
@ -772,6 +826,39 @@ in
|
|||
+ "evaluating and deploying cleanly. A browser client has a user "
|
||||
+ "to authorise and does not need it.";
|
||||
}
|
||||
{
|
||||
assertion = lib.all oneWord (lib.attrValues storeFiles);
|
||||
message = ''
|
||||
The secret publisher's store identity paths must not contain
|
||||
whitespace: nspawn's --load-credential receives them through
|
||||
nixos-container's unquoted extra flags.
|
||||
'';
|
||||
}
|
||||
{
|
||||
assertion =
|
||||
(hyperhiveCfg.swarm.otel.serviceProducers.${cfg.machine}.clientId or null) == forwarderClientId;
|
||||
message = ''
|
||||
services.hyperhive.swarm.otel.serviceProducers has no entry
|
||||
"${cfg.machine}" with clientId "${forwarderClientId}", so the swarm
|
||||
collector has no receiver for authelia's journal forwarder. Declare
|
||||
it in swarm-otel-service.nix.
|
||||
'';
|
||||
}
|
||||
{
|
||||
assertion =
|
||||
!(lib.elem forwarderTelemetryPort [
|
||||
hyperhiveCfg.otel.telemetryPort
|
||||
hyperhiveCfg.swarm.otel.telemetryPort
|
||||
hyperhiveCfg.swarm.bao.otel.telemetryPort
|
||||
hyperhiveCfg.swarm.nats.otel.telemetryPort
|
||||
]);
|
||||
message = ''
|
||||
authelia's journal forwarder's telemetry port (${toString forwarderTelemetryPort})
|
||||
collides with another collector's in the host netns: the hive
|
||||
collector's, the swarm collector's, the store forwarder's or the
|
||||
queue forwarder's.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
# Authelia's own gateway surface: the vhost that fronts it and the
|
||||
|
|
@ -899,15 +986,25 @@ in
|
|||
};
|
||||
|
||||
# Order the container after the host CA generator, so the bind source
|
||||
# exists before nspawn sets the mount up.
|
||||
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;
|
||||
# exists before nspawn sets the mount up. nspawn also fails to start it
|
||||
# when a credential source is missing, so where this host mints the
|
||||
# publisher's leaf the container pulls the minting unit in and waits for
|
||||
# it, as ./swarm-grafana.nix's does. The minting is local: the store need
|
||||
# not be up.
|
||||
systemd.services."container@${cfg.machine}" = lib.mkMerge [
|
||||
caTrust.containerOrdering
|
||||
(lib.mkIf publishes {
|
||||
after = [ "swarm-bao-pki.service" ];
|
||||
wants = [ "swarm-bao-pki.service" ];
|
||||
})
|
||||
];
|
||||
|
||||
containers.${cfg.machine} = {
|
||||
autoStart = true;
|
||||
ephemeral = false;
|
||||
# Journal files on the host, not inside the container: nixpkgs hardcodes
|
||||
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
|
||||
extraFlags = [ "--link-journal=host" ];
|
||||
# No `--link-journal=host`: the journal stays in the guest, where the
|
||||
# container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix).
|
||||
extraFlags = lib.mapAttrsToList (id: source: "--load-credential=${id}:${source}") storeFiles;
|
||||
inherit privateNetwork;
|
||||
# Public trust bundle only, read-only. Empty when the gateway is not
|
||||
# self-signed, so the whole trust path drops out cleanly.
|
||||
|
|
@ -918,15 +1015,61 @@ in
|
|||
{
|
||||
imports = [
|
||||
../container-modules/swarm-container.nix
|
||||
../container-modules/swarm-journal-forwarder.nix
|
||||
../container-modules/swarm-secret-publisher.nix
|
||||
(import ../container-modules/swarm-container-resolver.nix {
|
||||
inherit (networkCfg) bridgeIp;
|
||||
dnsConsumers = [ cfg.unit ];
|
||||
# The forwarder dials the swarm collector and authelia by name,
|
||||
# and its fetch and the publisher the store.
|
||||
dnsConsumers = [
|
||||
cfg.unit
|
||||
]
|
||||
++ lib.optionals publishes [
|
||||
"opentelemetry-collector.service"
|
||||
"${forwarderFetchUnit}.service"
|
||||
"${publisherUnit}.service"
|
||||
];
|
||||
})
|
||||
caBundleModule
|
||||
];
|
||||
|
||||
services.hyperhive.swarmContainer = { inherit privateNetwork; };
|
||||
|
||||
# Authenticated as the forwarder's own OIDC client, its secret read
|
||||
# from the store with the publisher's identity: one identity for the
|
||||
# whole container. With no identity there is nothing to read it with,
|
||||
# and the forwarder is off.
|
||||
services.hyperhive.swarmJournalForwarder = lib.mkIf publishes {
|
||||
enable = true;
|
||||
serviceName = "authelia";
|
||||
endpoint = forwarderEndpoint;
|
||||
clientId = forwarderClientId;
|
||||
tokenUrl = "${toString cfg.url}/api/oidc/token";
|
||||
telemetryPort = forwarderTelemetryPort;
|
||||
store = storeIdentityIn forwarderFetchUnit // {
|
||||
secretPath = "secret/swarm/services/${forwarderClientId}/oidc/client";
|
||||
};
|
||||
};
|
||||
systemd.services.${forwarderFetchUnit} = lib.mkIf publishes {
|
||||
serviceConfig.LoadCredential = lib.attrNames storeFiles;
|
||||
};
|
||||
|
||||
# Copies what the mint below leaves in `clientsDir` into the store,
|
||||
# under the publisher's own identity, a system credential. Nothing
|
||||
# orders authelia after it.
|
||||
services.hyperhive.swarmSecretPublisher = lib.mkIf publishes {
|
||||
enable = true;
|
||||
clientSecretDir = clientsDir;
|
||||
inherit (publisherCfg) secrets;
|
||||
store = storeIdentityIn publisherUnit;
|
||||
};
|
||||
systemd.services.${publisherUnit} = lib.mkIf publishes {
|
||||
# A run before the mint would only skip every secret and wait
|
||||
# for the path unit.
|
||||
after = [ "${unitName}-secrets.service" ];
|
||||
serviceConfig.LoadCredential = lib.attrNames storeFiles;
|
||||
};
|
||||
|
||||
# The authelia binary itself, so an operator who gets a shell
|
||||
# in here can run `authelia crypto hash generate` to make a
|
||||
# password for the users file. Without it the container runs
|
||||
|
|
|
|||
|
|
@ -481,7 +481,7 @@ let
|
|||
secretPublisherPolicyName = "swarm-secret-publisher";
|
||||
secretPublisherCn = baoDeploy.secretPublisherCommonName;
|
||||
|
||||
# Three grants and one denial, and every narrowing in each is load-bearing.
|
||||
# Four grants and six denials, and every narrowing in each is load-bearing.
|
||||
#
|
||||
# `secret/data/` is KV v2's ACL prefix, inserted by the engine rather than
|
||||
# written by the caller — same trap as the controller's grant above.
|
||||
|
|
@ -494,8 +494,9 @@ let
|
|||
# here too — a grant widens when a path gains a WRITER, not when a kind is
|
||||
# declared.
|
||||
#
|
||||
# Write-only. It copies secrets in and never reads one back; a read
|
||||
# capability would let a file-copier recover every hive's credentials.
|
||||
# Write-only but for one leaf, the client secret of its container's forwarder:
|
||||
# a read on a prefix would let a file-copier recover every hive's credentials.
|
||||
# That stanza repeats `create`/`update`, which its exact path would drop.
|
||||
#
|
||||
# The queue callout's three seeds sit under `swarm/services/` and are no
|
||||
# secret this principal copies, so their exact paths, which outrank the
|
||||
|
|
@ -550,6 +551,10 @@ let
|
|||
path "${credentialMountPath}/data/${controllerClientLeaf}" {
|
||||
capabilities = ["create", "update"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.authelia.otel.clientId}/oidc/client" {
|
||||
capabilities = ["create", "update", "read"]
|
||||
}
|
||||
'';
|
||||
|
||||
# The identity the matrix container's `swarm-matrix-ctl` presents. Named outside `hive-*`
|
||||
|
|
@ -1448,10 +1453,10 @@ let
|
|||
# own and lands it in a collector's container — because that route is
|
||||
# proven and there is no second one worth inventing.
|
||||
#
|
||||
# Where the publisher on authelia's host leaves it. A swarm service's client
|
||||
# is registered once for the whole swarm, so its secret is one value. The
|
||||
# `swarm-forwarder-oidc` role's grant is this one path, so any other answers
|
||||
# 403 however correct it looks.
|
||||
# Where the publisher in authelia's container leaves it. A swarm service's
|
||||
# client is registered once for the whole swarm, so its secret is one value.
|
||||
# The `swarm-forwarder-oidc` role's grant is this one path, so any other
|
||||
# answers 403 however correct it looks.
|
||||
forwarderStoreSecretPath = "secret/swarm/services/${cfg.otel.clientId}/oidc/client";
|
||||
# On the host's /run (tmpfs), so the secret never reaches persistent disk:
|
||||
# after a reboot `swarm-bao-forwarder-oidc` fetches it again. The container
|
||||
|
|
@ -1770,7 +1775,9 @@ in
|
|||
controller's grant includes rewriting every hive's policy and login
|
||||
role, which is not something a unit whose whole job is copying one file
|
||||
should be able to do. This one's grant is `create`/`update` under the
|
||||
hive prefix and nothing else.
|
||||
hive and service prefixes and on the controller's client, and `read` on
|
||||
one leaf: the client secret of the journal forwarder in authelia's
|
||||
container, which presents this identity too.
|
||||
|
||||
⚠️ Same collision as its sibling above, and the same answer: ./swarm.nix
|
||||
feeds this value into the guard on
|
||||
|
|
@ -2967,9 +2974,8 @@ in
|
|||
# matters most and is the least obvious:
|
||||
# 1. `nixos-container` refuses to start when a bind source is missing,
|
||||
# and a certificate minted on this same boot does not exist yet when
|
||||
# the container is ordered. Same trap `hostClientSecretDir`
|
||||
# documents in ./swarm-authelia.nix. Ordering against whatever
|
||||
# mints it belongs with whatever named the path, not here.
|
||||
# the container is ordered. Ordering against whatever mints it
|
||||
# belongs with whatever named the path, not here.
|
||||
# 2. `tlsDir` is already mounted at the same path inside, so a copy
|
||||
# needs no second mount.
|
||||
# 3. A directory holding a leaf usually holds the CA's private key
|
||||
|
|
@ -3797,8 +3803,8 @@ in
|
|||
;;
|
||||
esac
|
||||
|
||||
# Published by ./swarm-secret-publisher.nix on authelia's host,
|
||||
# which this boot does not wait on, so absence is retried.
|
||||
# Published by the secret publisher in authelia's container, which
|
||||
# this boot does not wait on, so absence is retried.
|
||||
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg "${credentialMountPath}/${baoUiClientLeaf}"} 2>"$err")"; then
|
||||
echo ${lib.escapeShellArg "the store did not return ${credentialMountPath}/${baoUiClientLeaf}: the UI's OIDC client secret is not published yet."} >&2
|
||||
cat "$err" >&2
|
||||
|
|
|
|||
|
|
@ -177,7 +177,7 @@ let
|
|||
|
||||
# No secret here. The daemon reads its client secret from the store at
|
||||
# start, under `baoEnv`'s identity, from the path `swarm-secret-publish`
|
||||
# writes it to on authelia's host (`queue_identity.rs`).
|
||||
# writes it to in authelia's container (`queue_identity.rs`).
|
||||
#
|
||||
# Every value here comes from an option rather than from what happens to
|
||||
# run on this host. The queue is not optional for a controller, but
|
||||
|
|
|
|||
|
|
@ -278,7 +278,7 @@ let
|
|||
"/var/lib/grafana-metrics/bearer"
|
||||
];
|
||||
|
||||
# Where the publisher on authelia's host leaves this client's secret. A
|
||||
# Where the publisher in authelia's container leaves this client's secret. A
|
||||
# swarm service's client is registered once for the whole swarm, so its
|
||||
# secret is one value and not one per hive. Both ends compose this from the
|
||||
# same swarm-wide client id, so a rename cannot leave one of them behind.
|
||||
|
|
@ -649,11 +649,11 @@ in
|
|||
++ map (p: "r ${p}") legacyHostSecretPaths;
|
||||
|
||||
# THE delivery unit — one route, in every deployment. The secret authelia
|
||||
# minted arrives out of the swarm secret store, which the publisher on
|
||||
# authelia's host wrote it into, and that is true whether authelia is a
|
||||
# network away or in the container next door.
|
||||
# minted arrives out of the swarm secret store, which the publisher in
|
||||
# authelia's container wrote it into, and that is true whether authelia is
|
||||
# a network away or in the container next door.
|
||||
#
|
||||
# 🩸 No second unit copies the plaintext out of authelia's host tree
|
||||
# 🩸 No second unit copies the plaintext out of authelia's tree
|
||||
# where the two are co-located. The store exists so a host holds ONE
|
||||
# out-of-band secret — its client certificate — and reads everything else
|
||||
# with it, so skipping the store when the producer is local would buy a
|
||||
|
|
@ -667,8 +667,9 @@ in
|
|||
# that degrades QUIETLY, since no retry turns "no value there" into a value.
|
||||
#
|
||||
# ⚠️ An ABSENT secret is "not published yet", not a fault of this host's.
|
||||
# The publisher runs on the authelia host and authelia mints on its first
|
||||
# boot, so early in a swarm's life there is legitimately nothing to read.
|
||||
# The publisher runs in authelia's container and authelia mints on its
|
||||
# first boot, so early in a swarm's life there is legitimately nothing to
|
||||
# read.
|
||||
# Nothing here writes a stand-in: there is no such thing as a locally valid
|
||||
# OIDC client secret, and a placeholder would turn a Grafana that cannot
|
||||
# log anyone in into one whose token exchange is refused.
|
||||
|
|
|
|||
|
|
@ -442,6 +442,11 @@ in
|
|||
clientId = hyperhiveCfg.swarm.ci-metrics.clientId;
|
||||
port = lib.mkDefault 4403;
|
||||
};
|
||||
# The forwarder in ./swarm-authelia.nix.
|
||||
swarm-authelia = {
|
||||
clientId = hyperhiveCfg.swarm.authelia.otel.clientId;
|
||||
port = lib.mkDefault 4402;
|
||||
};
|
||||
};
|
||||
|
||||
# Not under any `enable`: see `publishedScrapeTargets` for why every host
|
||||
|
|
|
|||
|
|
@ -216,9 +216,9 @@ let
|
|||
# The `serviceProducers` containers (`swarm-bao`, `swarm-nats`,
|
||||
# `swarm-builder`, `swarm-claude-login`, `swarm-review-classifier`,
|
||||
# `swarm-ci-runner`, `swarm-grafana`, `swarm-victorialogs`, `hive-forge`,
|
||||
# `hive-matrix`, `swarm-victoriametrics`) are the exceptions on purpose: their
|
||||
# journals stay inside the container for its own collector instead, so this
|
||||
# receiver never sees their units.
|
||||
# `hive-matrix`, `swarm-victoriametrics`, `swarm-authelia`) are the exceptions
|
||||
# on purpose: their journals stay inside the container for its own collector
|
||||
# instead, so this receiver never sees their units.
|
||||
hostJournalDir = "/var/log/journal";
|
||||
|
||||
# Each store's OTLP route, by domain. One binding because the same string is
|
||||
|
|
@ -274,7 +274,7 @@ let
|
|||
# regression rather than a choice anyone made.
|
||||
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
|
||||
|
||||
# Where the publisher on authelia's host leaves this client's secret —
|
||||
# Where the publisher in authelia's container leaves this client's secret —
|
||||
# composed from the same swarm-wide `clientId` the registration in
|
||||
# ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of
|
||||
# them behind. The same prefix ./swarm-grafana.nix reads its own client
|
||||
|
|
@ -409,11 +409,11 @@ in
|
|||
# ./swarm-grafana.nix has for its own client.
|
||||
|
||||
# THE delivery unit — one route, in every deployment. The secret authelia
|
||||
# minted arrives out of the swarm secret store, which the publisher on
|
||||
# authelia's host wrote it into, whether authelia is a network away or in
|
||||
# the container next door.
|
||||
# minted arrives out of the swarm secret store, which the publisher in
|
||||
# authelia's container wrote it into, whether authelia is a network away or
|
||||
# in the container next door.
|
||||
#
|
||||
# 🩸 No unit here copies the plaintext directly out of authelia's host
|
||||
# 🩸 No unit here copies the plaintext directly out of authelia's
|
||||
# tree: that is reachable only where authelia shares this host's network
|
||||
# namespace, and breaks the moment authelia is on another host. The store
|
||||
# exists so a host holds ONE out-of-band secret — its client certificate —
|
||||
|
|
|
|||
|
|
@ -1,55 +1,15 @@
|
|||
# The unit that puts swarm-level secrets into the swarm's secret store, so
|
||||
# whoever needs one can read it there: a hive its own and its agents'
|
||||
# credentials, a service or the controller its own. It mints nothing: every
|
||||
# value is an OIDC client secret authelia minted, and this unit copies it.
|
||||
#
|
||||
# ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
|
||||
# into the store even when the service runs beside authelia, because its
|
||||
# reader fetches it from the store in every deployment — ./swarm-grafana.nix
|
||||
# states the ruling that made that the only route. Publishing "only when the
|
||||
# reader is elsewhere" would be a second shape of this unit, gated on a fact
|
||||
# about another host, to save a round trip on the one host that can afford it.
|
||||
#
|
||||
# ⚠️ IT RUNS WHERE AUTHELIA DOES, and that is the whole reason it exists as a
|
||||
# separate thing. `deploy.authelia.hostClientSecretDir`'s own description says
|
||||
# why: the plaintext's other reader lives in a **different container**, and
|
||||
# containers that share this host's network namespace still have separate
|
||||
# filesystem roots, so "the host is the only place both trees are addressable".
|
||||
# A delivery step therefore runs on the minting host, not the store's or the
|
||||
# reading hive's.
|
||||
#
|
||||
# ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite
|
||||
# every hive's policy and login role; a unit that writes a handful of secrets
|
||||
# has no business holding it. ./swarm-bao.nix grants this one `create`/`update`
|
||||
# under the hive prefix and nothing else — no `read`.
|
||||
#
|
||||
# ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every
|
||||
# argument is world-readable in /proc for the life of the call — the value is
|
||||
# passed as `@<path>` and read by bao itself.
|
||||
# What a host decides about the swarm's secret publisher: whether it runs, and
|
||||
# the store identity it presents, plus the list of secrets it copies. The unit
|
||||
# itself is ../container-modules/swarm-secret-publisher.nix, inside authelia's
|
||||
# container beside the mint whose secrets it copies; ./swarm-authelia.nix
|
||||
# imports it there and hands the identity in as system credentials.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
config,
|
||||
...
|
||||
}:
|
||||
let
|
||||
hyperhiveCfg = config.services.hyperhive;
|
||||
deployCfg = hyperhiveCfg.deploy;
|
||||
baoDeploy = deployCfg.bao;
|
||||
cfg = deployCfg.swarm-secret-publisher;
|
||||
|
||||
# A reader is defined by holding a certificate the store accepts, never by
|
||||
# standing next to it — the rule ./glue-queue-agent-credential.nix states in
|
||||
# full.
|
||||
# `deploy.bao.enable` here would be the co-location assumption itself.
|
||||
haveClientIdentity = cfg.baoClientCertFile != null && cfg.baoClientKeyFile != null;
|
||||
|
||||
# Both halves have to be here: the mint (authelia, for the plaintext) and an
|
||||
# identity (for the store). Neither implies the other. Authelia here is a
|
||||
# requirement, not a co-location toggle: every file the unit copies is under
|
||||
# its `hostClientSecretDir`.
|
||||
active = cfg.enable && deployCfg.authelia.enable && haveClientIdentity;
|
||||
|
||||
hiveNames = lib.attrNames hyperhiveCfg.swarm.hives;
|
||||
|
||||
# The client id agent containers present, per hive — composed exactly as
|
||||
|
|
@ -110,12 +70,46 @@ let
|
|||
hyperhiveCfg.swarm.victoriametrics.otel.clientId
|
||||
# The CI run poller, which reads it itself on its host.
|
||||
hyperhiveCfg.swarm.ci-metrics.clientId
|
||||
# Authelia's container's journal forwarder, which reads it back under the
|
||||
# publisher's own identity.
|
||||
hyperhiveCfg.swarm.authelia.otel.clientId
|
||||
];
|
||||
|
||||
# The swarm controller's OIDC client, which it reads back from the store
|
||||
# under its own certificate.
|
||||
controllerClientId = hyperhiveCfg.swarm.controller.queueClientId;
|
||||
controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client";
|
||||
|
||||
secrets =
|
||||
# `client_id` beside the value: an agent's id is derived per hive and has
|
||||
# to be reconstructable from the store alone.
|
||||
map (hive: {
|
||||
clientId = agentClientId hive;
|
||||
path = "secret/swarm/hives/${hive}/queue/agent";
|
||||
withClientId = true;
|
||||
}) hiveNames
|
||||
# Each hive's own client, which that hive's hive-c0re reads under its own
|
||||
# certificate. The nix half of
|
||||
# `swarm_secret_client::queue::hive_client_path`.
|
||||
++ map (hive: {
|
||||
clientId = hiveClientId hive;
|
||||
path = "secret/swarm/hives/${hive}/oidc/client";
|
||||
}) hiveNames
|
||||
# No `client_id`: a service's client id is the swarm-wide option both
|
||||
# ends already read.
|
||||
++ map (id: {
|
||||
clientId = id;
|
||||
path = "secret/swarm/services/${id}/oidc/client";
|
||||
}) serviceClientIds
|
||||
# Under `controller/`, which no hive reads: this client may read every
|
||||
# hive's status. The nix half of
|
||||
# `swarm_secret_client::queue::controller_client_path`.
|
||||
++ [
|
||||
{
|
||||
clientId = controllerClientId;
|
||||
path = controllerSecretPath;
|
||||
}
|
||||
];
|
||||
in
|
||||
{
|
||||
options.services.hyperhive.deploy.swarm-secret-publisher = {
|
||||
|
|
@ -127,7 +121,8 @@ in
|
|||
secret store, so a hive that does not run authelia can read its
|
||||
own and its agents' credentials and a swarm service or the
|
||||
controller can read its own — from wherever it runs, this host
|
||||
included.
|
||||
included. The unit, `swarm-secret-publish`, runs inside authelia's
|
||||
container, so it exists only where authelia runs.
|
||||
|
||||
Defaults to whether this host mints the OIDC secrets (set by
|
||||
`glue-secret-publisher-authelia.nix`), which is the only half of the
|
||||
|
|
@ -160,6 +155,11 @@ in
|
|||
must be {option}`services.hyperhive.deploy.bao.secretPublisherCommonName`
|
||||
— cert auth matches on the CN, and the role accepts nothing else.
|
||||
|
||||
A path on the host. ./swarm-authelia.nix hands it into authelia's
|
||||
container as an nspawn system credential, and nspawn refuses to
|
||||
start the container while a credential's source is missing: set,
|
||||
the file has to exist before authelia's container starts.
|
||||
|
||||
No default: a module that guessed would be holding the CA opinion
|
||||
./swarm-bao.nix deliberately does not hold.
|
||||
./glue-secret-publisher-bao-identity.nix points it at the leaf
|
||||
|
|
@ -172,141 +172,21 @@ in
|
|||
default = null;
|
||||
description = ''
|
||||
Private key for {option}`services.hyperhive.deploy.swarm-secret-publisher.baoClientCertFile`.
|
||||
Both or neither — the unit does not exist unless each is set.
|
||||
Both or neither — the unit does not exist unless each is set. A host
|
||||
path, handed in like the certificate.
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
config = lib.mkIf active {
|
||||
# Re-publish when authelia rotates a secret. The mint writes the file, so
|
||||
# the file is the event — there is no signal from authelia to subscribe to.
|
||||
systemd.paths.swarm-secret-publish = {
|
||||
description = "watch for minted OIDC client secrets to publish";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
pathConfig = {
|
||||
PathChanged = deployCfg.authelia.hostClientSecretDir;
|
||||
Unit = "swarm-secret-publish.service";
|
||||
};
|
||||
};
|
||||
|
||||
systemd.services.swarm-secret-publish = {
|
||||
description = "publish the swarm's minted secrets to the swarm secret store";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network.target" ];
|
||||
path = [
|
||||
baoDeploy.package
|
||||
pkgs.coreutils
|
||||
];
|
||||
# ⚠️ No `RemainAfterExit`. A oneshot that stays `active (exited)` turns
|
||||
# every later start from the path unit above into a no-op.
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
# Bounded here rather than left to systemd's default, so the number a
|
||||
# boot waits on is in the file that waits. A sealed store answers on
|
||||
# the port and never answers the write.
|
||||
TimeoutStartSec = 60;
|
||||
Restart = "on-failure";
|
||||
RestartSec = 30;
|
||||
};
|
||||
environment = {
|
||||
BAO_ADDR = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}";
|
||||
BAO_CLIENT_CERT = cfg.baoClientCertFile;
|
||||
BAO_CLIENT_KEY = cfg.baoClientKeyFile;
|
||||
}
|
||||
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
|
||||
BAO_CACERT = baoDeploy.serverCaFile;
|
||||
};
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
|
||||
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
|
||||
# variables above only decide which certificate the TLS handshake
|
||||
# presents; without a token `bao` asks its token helper instead, and
|
||||
# that is a `sh` this unit's `path` does not carry. `-token-only`
|
||||
# answers on stdout and skips the helper on both sides.
|
||||
#
|
||||
# Unhandled on purpose: `Restart=on-failure` above is what a store that
|
||||
# cannot authenticate this host should get. Degrading here would report
|
||||
# "published 0" as an ordinary quiet day.
|
||||
BAO_TOKEN="$(bao login -method=cert -token-only)"
|
||||
export BAO_TOKEN
|
||||
|
||||
published=0
|
||||
skipped=0
|
||||
|
||||
${lib.concatMapStringsSep "\n" (hive: ''
|
||||
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${agentClientId hive}.secret"}
|
||||
if [ -s "$src" ]; then
|
||||
# `value=@$src` hands bao the PATH: bao opens the file itself, so
|
||||
# the plaintext is never an argument of this process. Writing it as
|
||||
# `value="$(cat "$src")"` would publish it to /proc for anyone on
|
||||
# the host to read.
|
||||
bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/queue/agent"} \
|
||||
value=@"$src" \
|
||||
client_id=${lib.escapeShellArg (agentClientId hive)}
|
||||
published=$((published + 1))
|
||||
else
|
||||
# Not an error: authelia mints on its FIRST BOOT, so an absent file
|
||||
# is "not yet", and the path unit above re-runs this when it lands.
|
||||
echo "no minted secret at $src yet; the path unit will re-run this" >&2
|
||||
skipped=$((skipped + 1))
|
||||
fi
|
||||
'') hiveNames}
|
||||
|
||||
# Each hive's own client, which that hive's hive-c0re reads under its
|
||||
# own certificate. The nix half of
|
||||
# `swarm_secret_client::queue::hive_client_path`.
|
||||
${lib.concatMapStringsSep "\n" (hive: ''
|
||||
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${hiveClientId hive}.secret"}
|
||||
if [ -s "$src" ]; then
|
||||
bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/oidc/client"} \
|
||||
value=@"$src"
|
||||
published=$((published + 1))
|
||||
else
|
||||
echo "no minted secret at $src yet; the path unit will re-run this" >&2
|
||||
skipped=$((skipped + 1))
|
||||
fi
|
||||
'') hiveNames}
|
||||
|
||||
${lib.concatMapStringsSep "\n" (id: ''
|
||||
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${id}.secret"}
|
||||
if [ -s "$src" ]; then
|
||||
# `value=@$src` for the same reason as the hive loop above: bao
|
||||
# opens the file itself, so the plaintext is never an argument of
|
||||
# this process.
|
||||
#
|
||||
# No `client_id` field beside it, unlike a hive's credential: that
|
||||
# one is derived per hive and has to be reconstructable from the
|
||||
# store alone, whereas a service's client id is the swarm-wide
|
||||
# option both ends already read.
|
||||
bao kv put ${lib.escapeShellArg "secret/swarm/services/${id}/oidc/client"} \
|
||||
value=@"$src"
|
||||
published=$((published + 1))
|
||||
else
|
||||
# Not an error, and the ordinary state of a swarm that runs this
|
||||
# service nowhere: nothing registered the client, so authelia minted
|
||||
# nothing to publish.
|
||||
echo "no minted secret at $src yet; the path unit will re-run this" >&2
|
||||
skipped=$((skipped + 1))
|
||||
fi
|
||||
'') serviceClientIds}
|
||||
|
||||
# The controller's own client, under `controller/`, which no hive
|
||||
# reads: this client may read every hive's status. The nix half of
|
||||
# `swarm_secret_client::queue::controller_client_path`.
|
||||
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${controllerClientId}.secret"}
|
||||
if [ -s "$src" ]; then
|
||||
bao kv put ${lib.escapeShellArg controllerSecretPath} \
|
||||
value=@"$src"
|
||||
published=$((published + 1))
|
||||
else
|
||||
# Not an error: authelia mints on its first boot, and the path unit
|
||||
# above re-runs this when it lands.
|
||||
echo "no minted secret at $src yet; the path unit will re-run this" >&2
|
||||
skipped=$((skipped + 1))
|
||||
fi
|
||||
|
||||
echo "published $published client secret(s), skipped $skipped"
|
||||
secrets = lib.mkOption {
|
||||
type = lib.types.listOf lib.types.attrs;
|
||||
readOnly = true;
|
||||
internal = true;
|
||||
default = secrets;
|
||||
description = ''
|
||||
Every secret the publisher copies: the minted client's id, the store
|
||||
path its reader fetches, and whether `client_id` is stored beside the
|
||||
value. Derived from `swarm.*` alone, so it is the same list on every
|
||||
host; ./swarm-authelia.nix hands it to the unit in its container.
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
|
|
|||
|
|
@ -488,8 +488,8 @@ in
|
|||
A URL alone does not publish. This hive also needs its own store
|
||||
identity ({option}`services.hyperhive.deploy.bao.clientCertFile`),
|
||||
under which hive-c0re reads its `hive-<hiveName>` client secret
|
||||
from the swarm's secret store; `swarm-secret-publish`, on the host
|
||||
that runs authelia, puts it there. Without an identity, or with
|
||||
from the swarm's secret store; `swarm-secret-publish`, in authelia's
|
||||
container, puts it there. Without an identity, or with
|
||||
this set to null, this hive computes its own readiness as always
|
||||
and offers it to nobody, and the swarm controller reports it
|
||||
`never_reported`, which is the honest reading.
|
||||
|
|
|
|||
Loading…
Reference in a new issue