vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

Compare commits

base: vinzenz:9d339b1b585851766310e63fa922da9e6eea71a8
Branches Tags
vinzenz:main
..
compare: vinzenz:ded5b08258ce7a9f776a0da0bf4fcd48726f2b38
Branches Tags
vinzenz:main
15 changed files with 309 additions and 1477 deletions
Download patch file Download diff file Expand all files Collapse all files

15
docs/getting-started/setup.md
View file

@ -170,19 +170,16 @@ What else you need depends on
restart, with the keys `init` printed.
The store serves TLS, and on a hive that deploys it you need do nothing: a
first-boot unit mints a CA of the store's own plus the leaves it signs — the
store's server certificate, this host's client certificate, and one per service
principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and
`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and
`.serverCaFile` at the reader's, and each principal's own pair at its own leaf.
first-boot unit mints a CA of the store's own plus the two leaves it signs —
the store's server certificate and this host's client certificate — and points
`deploy.bao.serverCertFile`, `.serverKeyFile` and `.clientCaFile` at the store's
half, `.clientCertFile`, `.clientKeyFile` and `.serverCaFile` at the reader's.
Those are `mkDefault`s, so naming your own paths wins. Do that when your
certificates come from a real internal CA; the store has no opinion about
which. A hive that does **not** deploy the store names the reader's three
itself, plus a pair for every principal it runs — see
[per-principal identities](../swarm/secrets.md#per-principal-identities) for the
list. The operator issues those leaves out of band; they're the credentials the
store can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is
itself: the operator issues that leaf out of band, and it's the one credential
the store can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is
**mTLS between services and the store**, a separate trust domain, because a
store that took its identity from an authority it itself distributes could
never come up before that authority.

103
docs/swarm/secrets.md
View file

@ -88,7 +88,7 @@ where an absent OIDC block is a container with no way in at all, a collector
with nothing to authenticate with still receives every hive's telemetry —
only its own pushes to the stores go out unauthenticated and get refused
there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice
`deploy.bao.clientCertFile` / `clientKeyFile`, the same choice
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for
their own optional readers, rather than the hard assertion `swarm-grafana.nix`
uses for its non-optional one. The operator can still deliver the secret by
@ -249,14 +249,14 @@ straight out of authelia's tree when the two happen to share a host is one
round trip cheaper and costs a second delivery unit, a second way for the file
to be wrong, and a gate deciding between them — and the gate is the expensive
part, because whatever it's wrong about is an outage nobody can read. The store
exists so the only secrets a host holds out of band are **certificates**, and
a host reads everything else with them.
exists so a host holds **one** out-of-band secret, its client certificate, and
reads everything else with it.
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (this module disables its local
login form unconditionally, so SSO isn't a feature of some topologies — it's the
only way in), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
only way in), and it requires this host's `deploy.bao.clientCertFile` /
`clientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is
@ -271,8 +271,7 @@ runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it
doesn't render the reading unit without them, the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for
their own optional reads.
@ -291,18 +290,10 @@ A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs
a given service — placement is a `deploy.*` fact, per-host by definition — so
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
every swarm service's client secret, and that's stated in
`swarm-secret-client`'s `policy` module beside the grant itself.
⚠️ **The readers no longer present it.** Four units used to log in with
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
principal by the subject of the certificate it presents — so four readers behind
one leaf were one principal holding the union of their needs. Each now holds a
leaf of its own, and a policy naming only the path that unit reads. See
[per-principal identities](#per-principal-identities) below.
Absence behaves
there is nothing to scope the grant to. Every hive can therefore read every swarm
service's client secret; that's the cost of a shape where the service's host has
no store identity of its own to present, and it's stated in
`swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves
as it does for the other readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in.
@ -312,78 +303,14 @@ credential you must already hold to authenticate from the thing it
authenticates you to, so the store's identity can't come from an authority the store
distributes — which excludes the hive CA and the swarm CA both, and has nothing
to do with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
mints a CA that signs a fixed, short list of leaves — the store's server
certificate, the hive's own reader certificate, and one per service principal —
and distributes nothing. A deployment with a real
mints a CA that signs exactly two things, the store's server certificate and a
reader's client certificate, and distributes nothing. A deployment with a real
internal CA deletes that file and names its own paths in
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
A hive that reads from a store on **another** machine names the reader's half —
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
plus one leaf per principal it runs (the table below names the options). Those are the
credentials that can't come out of the store, being what opens it; everything
else a hive needs does.
### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. These four units read one path
each and each holds a leaf, a role and a policy of its own:
| unit | option pair under `deploy.bao.` | reads |
| ------------------------ | -------------------------------------------------------- | -------------------------------------------------- |
| `swarm-bao-matrix-token` | `matrixTokenClientCertFile` / `matrixTokenClientKeyFile` | `swarm/hives/<hive>/matrix/appservice-token` |
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
The first two exist **per hive**, because the path they read carries a hive
name and every hive runs its own reader. Their subjects are
`<deploy.bao.matrixTokenCommonNamePrefix>-<hive>` and
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves both
composed spellings as hive names, so nobody can name a hive into another hive's
role. The other two read a path that names a swarm service rather than a hive, so
one role each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName` and `deploy.bao.otelOidcCommonName`.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults
all eight options, and there is nothing to do. Elsewhere you issue each leaf from
that CA out of band and name it here — one file per principal rather than one file
shared by four, which is the whole of what this buys.
Forgetting one of the eight elsewhere isn't a quiet degrade. Three of the four
readers used to render only where their leaf existed, so a hand-configured
remote-store hive that named the hive's own `clientCertFile` and missed a
principal's pair just lost that unit; `swarm-grafana.nix` was alone in refusing
the build. All four now refuse it, each naming its own option pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
opening the nix.
That condition fires on exactly one shape, the hand-configured remote-store hive
that named seven of the eight. It stays silent everywhere else, and each half of
that matters on its own:
- **A host with no store identity at all** holds no `clientCertFile` either, so
the first clause is false. That's the supported no-store deployment, and also
the one an operator passes through while bringing a hive up — neither should
fail to evaluate.
- **A host that runs the store** gets all eight from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no homeserver, no collector, no Grafana — has no
unit to skip, so the second clause is false. Only the queue-credential reader
has no toggle of its own to check: every hive runs one for its own agents, so
reading the store at all is what asks for it.
The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives
telemetry. The assertion doesn't touch that shape, because that host has no
`clientCertFile` either.
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand.
It's the one credential that can't come out of the store, being what opens it;
everything else a hive needs does.
## How a reader reaches the store

48
nix/host-modules/glue-bao-tls.nix
View file

@ -84,25 +84,6 @@ in
clientCertFile = lib.mkDefault "${pkiDir}/client.pem";
clientKeyFile = lib.mkDefault "${pkiDir}/client-key.pem";
serverCaFile = lib.mkDefault "${pkiDir}/ca.pem";
# 🩸 Four readers that used to present `client.pem` above, each now
# pointed at a leaf of its own. The leaf is what bao sees, so this pairing
# is the whole of what turns "one principal with the union of four grants"
# into four principals with one grant each — see ./swarm-bao.nix's
# `perHiveReaders` block for the grants themselves.
#
# Defaulted here rather than in four `glue-<consumer>-bao-identity.nix`
# files: those exist where the consumer is a CONTAINER with a
# `deploy.<service>.*` namespace of its own to point at. These four are
# host units reading the store, which is the pairing this file already is.
matrixTokenClientCertFile = lib.mkDefault "${pkiDir}/matrix-token.pem";
matrixTokenClientKeyFile = lib.mkDefault "${pkiDir}/matrix-token-key.pem";
queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem";
queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem";
grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem";
grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem";
otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem";
otelOidcClientKeyFile = lib.mkDefault "${pkiDir}/otel-oidc-key.pem";
};
# Idempotent on ABSENCE, never on content. Re-issuing the CA invalidates
@ -179,35 +160,6 @@ in
# container an identity instead of lending it the hive's.
[ -s ${pkiDir}/matrix-ctl.pem ] || ${signLeaf} ${pkiDir} matrix-ctl \
${lib.escapeShellArg deployCfg.bao.matrixCtlCommonName} "" clientAuth
# 🩸 The four readers that used to present `client.pem` above. Each
# carries its own subject, which is the entire mechanism: bao matches a
# cert-auth role on the CN, so four units sharing one leaf were one
# principal holding the union of four grants, and the union included
# every agent credential in the swarm.
#
# ⚠️ The first two carry THIS hive's name in the subject, unlike the
# three service leaves above. Their grants name one hive's path, because
# a matrix token and a queue credential live under `swarm/hives/<name>/`
# and every hive runs a reader for its own; ./swarm-bao.nix writes one
# role per hive in the swarm directory to match. The two OIDC readers
# need no such segment — a client is registered once per swarm.
#
# Minted whether or not the consumer runs here, for the reason the three
# leaves above give: on a hive that does not run the store this is the
# file an operator copies, and a leaf that only appears where its
# consumer does is one nobody can copy from anywhere.
[ -s ${pkiDir}/matrix-token.pem ] || ${signLeaf} ${pkiDir} matrix-token \
${lib.escapeShellArg "${deployCfg.bao.matrixTokenCommonNamePrefix}-${clientCn}"} "" clientAuth
[ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth
[ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \
${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth
[ -s ${pkiDir}/otel-oidc.pem ] || ${signLeaf} ${pkiDir} otel-oidc \
${lib.escapeShellArg deployCfg.bao.otelOidcCommonName} "" clientAuth
'';
};
};

317
nix/host-modules/glue-matrix-bao-token.nix
View file

@ -38,24 +38,7 @@ let
# What decides whether this unit exists at all. A reader is defined by holding
# a certificate the store accepts, and that is true on the store's own host
# and on a hive three networks away for exactly the same reason.
#
# 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four
# units used to share. Bao matches a cert-auth role on the CN, so one leaf for
# four readers was ONE principal holding the union of four grants: read on
# `swarm/agents/*` AND `swarm/hives/<hive>/*` AND `swarm/services/*`, when
# this unit reads one appservice token and nothing else. Its own leaf carries
# `<matrixTokenCommonNamePrefix>-<hive>` and its role grants the single path
# below.
haveClientIdentity =
baoDeploy.matrixTokenClientCertFile != null && baoDeploy.matrixTokenClientKeyFile != null;
# Does this host read the store at all — the hive's own leaf, which is the
# one thing a remote-store deployment has always had to place by hand. Only
# used to decide whether a missing per-principal leaf is a mistake or a
# deployment that has no store: a host holding neither is the supported
# no-store shape, and one holding this pair but not the pair above named
# seven of the eight options and stopped.
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Where the token lives in the store. A path, not a convention to guess at:
# whoever writes it and whoever reads it must agree, and the agreement
@ -81,178 +64,134 @@ let
matrixMachine = "hive-matrix";
in
{
config = lib.mkMerge [
# ⚠️ A SEPARATE arm from the unit below, and that separation is the whole
# mechanism: the unit's arm is gated on `haveClientIdentity`, so an
# assertion written inside it could never be reached in the state it
# exists to report.
#
# Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the
# same refusal, named to this principal's own pair. What differs is the
# gate. Grafana asserts wherever Grafana runs, because a Grafana with no
# store identity has no way in at all; a homeserver with no store identity
# is a hive that has no store, which is supported. So this one additionally
# requires `hiveReaderIdentity`: the host demonstrably reads the store, and
# named every option but this pair.
(lib.mkIf (hyperhiveCfg.enable && deployCfg.matrix.enable && hiveReaderIdentity) {
assertions = [
{
assertion = haveClientIdentity;
message = ''
This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile
is set) and runs a homeserver, so it needs the matrix appservice
token reader's own client identity: set both
config = lib.mkIf (hyperhiveCfg.enable && haveClientIdentity && deployCfg.matrix.enable) {
# Same rule as the unit's own gate: this reader exists on a host that has a
# client identity and a homeserver, which is not every host that runs the
# store, so the store's module cannot name it.
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ];
services.hyperhive.deploy.bao.matrixTokenClientCertFile
services.hyperhive.deploy.bao.matrixTokenClientKeyFile
swarm-bao-matrix-token.service fetches this hive's appservice token
out of the store, and without these it is not rendered at all —
leaving the homeserver authenticating hive-c0re against whatever is
already on disk, which is a 401 on every request naming nothing.
⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one is
the hive's, and its grant reads every secret in the store; this role
reads the one appservice-token path. Pointing this option at the
hive's leaf would evaluate, deploy and log in — and undo the split.
On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere the leaf is issued
from that CA out of band and named here — see docs/swarm/secrets.md.
'';
}
systemd.services.swarm-bao-matrix-token = {
description = "fetch the matrix appservice token from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering is
# conditional even though the read is not: off-host there is nothing local
# to wait for, and the timeout below is what bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
})
(lib.mkIf (hyperhiveCfg.enable && haveClientIdentity && deployCfg.matrix.enable) {
# Same rule as the unit's own gate: this reader exists on a host that has a
# client identity and a homeserver, which is not every host that runs the
# store, so the store's module cannot name it.
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ];
systemd.services.swarm-bao-matrix-token = {
description = "fetch the matrix appservice token from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering is
# conditional even though the read is not: off-host there is nothing local
# to wait for, and the timeout below is what bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
before = [ "container@${matrixMachine}.service" ];
wantedBy = [ "container@${matrixMachine}.service" ];
path = [
deployCfg.bao.package
pkgs.coreutils
];
# Sized for the race this loses, not for an unseal. `swarm-bao` comes up
# seconds before this unit asks, and the cert-auth role it logs in
# against is written seconds after — so a few short attempts cover it.
# ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy.
# That unit blocks nothing; this one is `Before=` the homeserver's
# container, and whether that ordering waits across an auto-restart is
# unverified — so an hours-long window would be a bet on an unknown,
# where a minute is not. A store still sealed after it keeps the degrade
# below, as today.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec × burst`.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
# What actually bounds the read below. Stated here rather than
# left to systemd's default, so the number a boot waits on is in
# the file that waits.
TimeoutStartSec = 30;
Restart = "on-failure";
RestartSec = 15;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile;
BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
# A sealed or uninitialised store answers on the port and never
# answers the read, so "the store is up" is not the same as "the
# store can answer". `TimeoutStartSec` above is the bound; the
# homeserver only `Wants=` this unit, so hitting it degrades to
# keeping the local token rather than holding up the container.
# `bao`'s own message is the only thing separating a missing value
# from a refused identity from an unreachable host. This unit's
# degraded mode is correct for all three, so it reports which one
# rather than asserting all three in a sentence of ours — a reader
# that cannot say why it read nothing is indistinguishable from a
# broken one.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Fails LOUDLY, unlike the read below: the three states a login failure
# covers — store not up, sealed, role not written yet — are all things
# a retry fixes, and `Restart=on-failure` above is what retries. Exiting
# 0 here spends the whole boot on a condition that was seconds old.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then
echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if [ -z "$token" ]; then
echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2
exit 0
fi
umask 077
printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)}
chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)}
# Re-stamp the registration file from the token just written. The
# token is half an agreement — the registration the homeserver loads
# has to carry the same value — so writing the file and stopping
# would leave the homeserver authenticating hive-c0re against
# whatever activation put there: a 401 on every request, naming
# nothing. Unconditional rather than on-change, because this unit
# has no way to know what the registration currently says.
#
# hive-matrix's own renderer rather than a `printf` here, so the
# registration's shape has one home.
${deployCfg.matrix.appserviceRegistrationScript}
'';
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
before = [ "container@${matrixMachine}.service" ];
wantedBy = [ "container@${matrixMachine}.service" ];
path = [
deployCfg.bao.package
pkgs.coreutils
];
# Sized for the race this loses, not for an unseal. `swarm-bao` comes up
# seconds before this unit asks, and the cert-auth role it logs in
# against is written seconds after — so a few short attempts cover it.
# ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy.
# That unit blocks nothing; this one is `Before=` the homeserver's
# container, and whether that ordering waits across an auto-restart is
# unverified — so an hours-long window would be a bet on an unknown,
# where a minute is not. A store still sealed after it keeps the degrade
# below, as today.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec × burst`.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
# What actually bounds the read below. Stated here rather than
# left to systemd's default, so the number a boot waits on is in
# the file that waits.
TimeoutStartSec = 30;
Restart = "on-failure";
RestartSec = 15;
};
})
];
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.clientCertFile;
BAO_CLIENT_KEY = baoDeploy.clientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
# A sealed or uninitialised store answers on the port and never
# answers the read, so "the store is up" is not the same as "the
# store can answer". `TimeoutStartSec` above is the bound; the
# homeserver only `Wants=` this unit, so hitting it degrades to
# keeping the local token rather than holding up the container.
# `bao`'s own message is the only thing separating a missing value
# from a refused identity from an unreachable host. This unit's
# degraded mode is correct for all three, so it reports which one
# rather than asserting all three in a sentence of ours — a reader
# that cannot say why it read nothing is indistinguishable from a
# broken one.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Fails LOUDLY, unlike the read below: the three states a login failure
# covers — store not up, sealed, role not written yet — are all things
# a retry fixes, and `Restart=on-failure` above is what retries. Exiting
# 0 here spends the whole boot on a condition that was seconds old.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then
echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if [ -z "$token" ]; then
echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2
exit 0
fi
umask 077
printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)}
chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)}
# Re-stamp the registration file from the token just written. The
# token is half an agreement — the registration the homeserver loads
# has to carry the same value — so writing the file and stopping
# would leave the homeserver authenticating hive-c0re against
# whatever activation put there: a 401 on every request, naming
# nothing. Unconditional rather than on-change, because this unit
# has no way to know what the registration currently says.
#
# hive-matrix's own renderer rather than a `printf` here, so the
# registration's shape has one home.
${deployCfg.matrix.appserviceRegistrationScript}
'';
};
};
}

341
nix/host-modules/glue-queue-agent-credential.nix
View file

@ -40,24 +40,7 @@ let
# What decides whether this unit exists at all. A reader is defined by holding
# a certificate the store accepts, and that is true on the store's own host
# and on a hive three networks away for exactly the same reason.
#
# 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four
# units used to share. Bao matches a cert-auth role on the CN, so one leaf for
# four readers was ONE principal holding the union of four grants: read on
# `swarm/agents/*` AND `swarm/hives/<hive>/*` AND `swarm/services/*`, when
# this unit reads one queue credential and nothing else. Its own leaf carries
# `<queueAgentCommonNamePrefix>-<hive>` and its role grants the single path
# below — still this hive's own, so the narrowing costs no reach.
haveClientIdentity =
baoDeploy.queueAgentClientCertFile != null && baoDeploy.queueAgentClientKeyFile != null;
# Does this host read the store at all — the hive's own leaf, which is the
# one thing a remote-store deployment has always had to place by hand. Only
# used to decide whether a missing per-principal leaf is a mistake or a
# deployment that has no store: a host holding neither is the supported
# no-store shape, and one holding this pair but not the pair above named
# seven of the eight options and stopped.
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir;
secretFile = "${credentialDir}/secret";
@ -102,192 +85,144 @@ in
};
};
config = lib.mkMerge [
# ⚠️ A SEPARATE arm from the unit below, and that separation is the whole
# mechanism: the unit's arm is gated on `haveClientIdentity`, so an
# assertion written inside it could never be reached in the state it
# exists to report.
#
# Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the
# same refusal, named to this principal's own pair. Where Grafana's gate
# is `deploy.grafana.enable`, this reader has no toggle of its own to
# check: every hive runs one for its own agents, so reading the store at
# all is what asks for it. Hence `hiveReaderIdentity` alone — a host
# holding no hive leaf has no store to read and nothing is missing.
(lib.mkIf (hyperhiveCfg.enable && hiveReaderIdentity) {
assertions = [
{
assertion = haveClientIdentity;
message = ''
This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile
is set), so it needs the agent queue credential reader's own client
identity: set both
config = lib.mkIf (hyperhiveCfg.enable && haveClientIdentity) {
# Same rule as the unit's own gate: this reader exists on any host holding
# a client identity, which is not every host that runs the store, so the
# store's module cannot name it.
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-queue-agent" ];
services.hyperhive.deploy.bao.queueAgentClientCertFile
services.hyperhive.deploy.bao.queueAgentClientKeyFile
swarm-bao-queue-agent.service fetches this hive's agent queue
credential out of the store, and without these it is not rendered
at all — leaving hive-c0re with no queue credential and no unit
that would ever have written one.
Every hive runs this reader for its own agents, so unlike the other
three principals there is no per-service toggle that turns it off:
a hive that reads the store at all is a hive that needs this pair.
⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one
is the hive's, and its grant reads every secret in the store; this
role reads the one queue-credential path. Pointing this option at
the hive's leaf would evaluate, deploy and log in — and undo the
split.
On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere the leaf is issued
from that CA out of band and named here — see docs/swarm/secrets.md.
'';
}
systemd.services.swarm-bao-queue-agent = {
description = "fetch this hive's agent queue credential from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering is
# conditional even though the read is not: off-host there is nothing local
# to wait for, and the timeout below is what bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
})
(lib.mkIf (hyperhiveCfg.enable && haveClientIdentity) {
# Same rule as the unit's own gate: this reader exists on any host holding
# a client identity, which is not every host that runs the store, so the
# store's module cannot name it.
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-queue-agent" ];
systemd.services.swarm-bao-queue-agent = {
description = "fetch this hive's agent queue credential from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering is
# conditional even though the read is not: off-host there is nothing local
# to wait for, and the timeout below is what bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
# Ordered before hive-c0re, so no agent container renders ahead of an
# attempt at its credential. `Wants=`, not `Requires=`: a store this
# unit can't reach delays hive-c0re's start by its own start-limit
# window (`TimeoutStartSec`, retried up to `startLimitBurst` times
# below) rather than failing it — hive-c0re starts once that window
# elapses, whatever credential is or isn't on disk by then.
before = [ "hive-c0re.service" ];
wantedBy = [
"multi-user.target"
"hive-c0re.service"
];
path = [
baoDeploy.package
pkgs.coreutils
];
# Sized for the race this loses, not for an unseal: `swarm-bao` comes up
# seconds before this unit asks, and the cert-auth role it logs in
# against is written seconds after, so a few short attempts cover it.
# An hours-long window would be a bet on a store that is sealed, and the
# degrade below is already correct for that.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec × burst`.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
# What actually bounds the reads below. Stated here rather than left to
# systemd's default, so the number a boot waits on is in the file that
# waits.
TimeoutStartSec = 30;
Restart = "on-failure";
RestartSec = 15;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile;
BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
# `bao`'s own message is the only thing separating a missing value from
# a refused identity from an unreachable host. This unit's degraded
# mode is correct for all three, so it reports which one rather than
# asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Fails LOUDLY, unlike the reads below: the three states a login
# failure covers — store not up, sealed, role not written yet — are all
# things a retry fixes, and `Restart=on-failure` above is what retries.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
# Two reads of one object rather than one `-format=json` parsed with
# `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot
# actually disagree — the client id is derived from this hive's name, so
# a rotation landing between these two calls changes the secret and
# rewrites the same id.
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
# Both or neither, for the reason `QueueConfig::from_env` refuses a
# half-set environment: a client that finds one of the two comes up
# "fine" and never connects.
if [ -z "$secret" ] || [ -z "$client_id" ]; then
echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2
exit 0
fi
install -d -m 0755 ${lib.escapeShellArg credentialDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile}
chmod 0600 ${lib.escapeShellArg secretFile}
# `0644` on purpose: an OIDC client id is presented to the token
# endpoint on every connection and is public by construction.
printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile}
chmod 0644 ${lib.escapeShellArg clientIdFile}
'';
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
# Ordered before hive-c0re, so no agent container renders ahead of an
# attempt at its credential. `Wants=`, not `Requires=`: a store this
# unit can't reach delays hive-c0re's start by its own start-limit
# window (`TimeoutStartSec`, retried up to `startLimitBurst` times
# below) rather than failing it — hive-c0re starts once that window
# elapses, whatever credential is or isn't on disk by then.
before = [ "hive-c0re.service" ];
wantedBy = [
"multi-user.target"
"hive-c0re.service"
];
path = [
baoDeploy.package
pkgs.coreutils
];
# Sized for the race this loses, not for an unseal: `swarm-bao` comes up
# seconds before this unit asks, and the cert-auth role it logs in
# against is written seconds after, so a few short attempts cover it.
# An hours-long window would be a bet on a store that is sealed, and the
# degrade below is already correct for that.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec × burst`.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
# What actually bounds the reads below. Stated here rather than left to
# systemd's default, so the number a boot waits on is in the file that
# waits.
TimeoutStartSec = 30;
Restart = "on-failure";
RestartSec = 15;
};
})
];
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.clientCertFile;
BAO_CLIENT_KEY = baoDeploy.clientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
# `bao`'s own message is the only thing separating a missing value from
# a refused identity from an unreachable host. This unit's degraded
# mode is correct for all three, so it reports which one rather than
# asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Fails LOUDLY, unlike the reads below: the three states a login
# failure covers — store not up, sealed, role not written yet — are all
# things a retry fixes, and `Restart=on-failure` above is what retries.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
# Two reads of one object rather than one `-format=json` parsed with
# `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot
# actually disagree — the client id is derived from this hive's name, so
# a rotation landing between these two calls changes the secret and
# rewrites the same id.
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
# Both or neither, for the reason `QueueConfig::from_env` refuses a
# half-set environment: a client that finds one of the two comes up
# "fine" and never connects.
if [ -z "$secret" ] || [ -z "$client_id" ]; then
echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2
exit 0
fi
install -d -m 0755 ${lib.escapeShellArg credentialDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile}
chmod 0600 ${lib.escapeShellArg secretFile}
# `0644` on purpose: an OIDC client id is presented to the token
# endpoint on every connection and is public by construction.
printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile}
chmod 0644 ${lib.escapeShellArg clientIdFile}
'';
};
};
}

342
nix/host-modules/swarm-bao.nix
View file

@ -340,156 +340,6 @@ let
}
'';
# ── the four principals that used to share the hive's own leaf ─────────────
#
# 🩸 Each of the four reads exactly ONE path in the store, and until this
# split each did it holding `deploy.bao.clientCertFile` — the hive's own leaf,
# whose policy (`swarm-secret-client`'s `policy::render`) grants read on
# `swarm/agents/*`, `swarm/hives/<hive>/*` AND `swarm/services/*`. Four
# principals presenting one certificate are ONE principal to bao, so the union
# of what the four need was the only grant expressible: the unit that fetches
# Grafana's OIDC client secret could fetch every agent credential in the
# swarm, and the unit that fetches this hive's matrix token could fetch
# Grafana's. Least privilege was not misconfigured, it was unrepresentable.
#
# Each now gets its own leaf, its own cert-auth role and a policy holding the
# one `secret/data/…` path that principal's script actually names — spelled to
# the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText` above
# gives for doing the same.
#
# ⚠️ Two of the four are PER HIVE and two are not, and that asymmetry is the
# shape of the paths rather than a preference. A matrix appservice token and a
# queue credential live under `swarm/hives/<name>/`, and every hive runs a
# reader for its own; an OIDC client secret lives under
# `swarm/services/<client-id>/` and a swarm registers each of those exactly
# once. A single role for a per-hive reader would have to name `hives/*` to
# serve every hive — which would let one hive read another's matrix token, a
# reach no hive has today and one this change exists to remove rather than
# create.
readStanza = path: ''
path "${path}" {
capabilities = ["read"]
}
'';
# One reader per hive in the swarm directory. Written from the directory
# rather than from this host's own name because the policy is written where
# the STORE is and the reader runs where its hive is — the same split
# `matrixCtlHiveName` above exists to paper over, answered here by naming
# every hive instead of asking the operator which one.
perHiveReaders =
{ rolePrefix, cnPrefix, ... }@spec:
lib.mapAttrsToList (hiveName: _: {
name = "${rolePrefix}-${hiveName}";
cn = "${cnPrefix}-${hiveName}";
policyText = readStanza (spec.path hiveName);
}) hyperhiveCfg.swarm.hives;
matrixTokenReaders = perHiveReaders {
rolePrefix = "swarm-matrix-token";
cnPrefix = baoDeploy.matrixTokenCommonNamePrefix;
path = h: "${credentialMountPath}/data/swarm/hives/${h}/matrix/appservice-token";
};
queueAgentReaders = perHiveReaders {
rolePrefix = "swarm-queue-agent";
cnPrefix = baoDeploy.queueAgentCommonNamePrefix;
path = h: "${credentialMountPath}/data/swarm/hives/${h}/queue/agent";
};
# Singletons, so a list of one rather than a second shape: the granting unit
# below takes a list either way and the four units stay identical apart from
# the objects they write.
grafanaOidcReaders = [
{
name = "swarm-grafana-oidc";
cn = baoDeploy.grafanaOidcCommonName;
policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client";
}
];
otelOidcReaders = [
{
name = "swarm-otel-oidc";
cn = baoDeploy.otelOidcCommonName;
policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.otel.clientId}/oidc/client";
}
];
# The role name IS the policy name, as for the three service principals
# above: the role attaches the policy by spelling it identically, and one
# string for both objects removes the way they drift apart.
readerPolicyWrite = obj: ''
printf '%s' ${lib.escapeShellArg obj.policyText} |
bao policy write ${lib.escapeShellArg obj.name} -
'';
# ⚠️ Names are `swarm-<principal>[-<hive>]`, outside the `hive-*` namespace the
# controller may rewrite, for the reason the three service principals above
# state. A per-hive CN cannot collide with a hive's own leaf either — a hive's
# CN is its bare name and ../reserved-hive-fragments.nix forbids the substring
# `swarm` in one — and ./swarm.nix reserves the concrete per-hive spellings
# anyway, because the prefixes are operator-settable and an operator may spell
# one without that substring.
readerRoleWrite = obj: ''
bao write auth/cert/certs/${lib.escapeShellArg obj.name} \
certificate=@${tlsDir}/client-ca.pem \
allowed_common_names=${lib.escapeShellArg obj.cn} \
token_policies=${lib.escapeShellArg obj.name} \
display_name=${lib.escapeShellArg obj.cn}
'';
# One unit per PRINCIPAL, not one unit for the four: these names are
# operator-facing strings, the same reason `swarm-bao-secret-publisher-policy`
# above gives for not being more script inside the controller's unit. A hive
# whose matrix token never arrives is diagnosed by the state of the unit named
# after it.
#
# `after` and not `requires`, for the reason the publisher's unit states: the
# controller's unit creates the KV and cert-auth mounts this one writes into,
# but a failed oneshot still counts as finished, so ordering plus this unit's
# own retry is what converges.
#
# The role write is inside the client-CA branch and the policy write is not,
# exactly as the three above: with no CA there is no trust anchor for a login
# role, but the policy it would attach is still worth asserting.
readerPolicyUnit =
description: objects:
lib.mkIf haveBootstrapToken {
inherit description;
after = [
"container@${cfg.machine}.service"
"swarm-bao-controller-policy.service"
];
wantedBy = [ "multi-user.target" ];
path = [
baoCli
pkgs.coreutils
];
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
# Same unseal wait as its siblings above, for the reason stated there:
# under `seal = "shamir"` a human unseals by hand.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
Restart = "on-failure";
RestartSec = 30;
};
script = ''
set -euo pipefail
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
export BAO_TOKEN
''
+ lib.concatMapStrings readerPolicyWrite objects
+ lib.optionalString (baoDeploy.clientCaFile != null) (
"\n" + lib.concatMapStrings readerRoleWrite objects
);
};
# Every listener serves the same identity: they differ in which address
# they answer on, not in who they are. Client verification is separate and
# optional — a store with no `clientCaFile` still serves TLS, it just does
@ -996,185 +846,6 @@ in
'';
};
matrixTokenCommonNamePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-matrix-token";
example = "swarm-bao-matrix-token.svc";
description = ''
Prefix of the subject the store's per-hive matrix-token roles accept.
The role for hive `h` accepts `<prefix>-h` and nothing else, and grants
read on exactly `swarm/hives/h/matrix/appservice-token`.
A **prefix** rather than one subject because every hive runs its own
reader and the path it reads carries that hive's name. One subject for
all of them would need a `hives/*` grant, which would let one hive read
another hive's appservice token — a reach no hive has today.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its
fixed-subject siblings are: cert auth trusts the CA and
`allowed_common_names` is the whole narrowing, so a hive whose own leaf
carried one of these spellings would receive that grant.
'';
};
queueAgentCommonNamePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-queue-agent";
example = "swarm-bao-queue-agent.svc";
description = ''
Prefix of the subject the store's per-hive queue-credential roles
accept. The role for hive `h` accepts `<prefix>-h` and grants read on
exactly `swarm/hives/h/queue/agent` — the client credential that hive's
agent containers authenticate to the swarm queue with.
Per hive for the same reason
{option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is,
and more sharply: every hive in the swarm runs this reader, so a single
subject would have to be granted `hives/*`.
'';
};
grafanaOidcCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-grafana-oidc";
example = "swarm-bao-grafana-oidc.svc";
description = ''
Subject the store's `swarm-grafana-oidc` cert-auth role accepts — the
identity the unit that fetches Grafana's OIDC client secret presents.
Its grant is one path,
`swarm/services/<grafana client id>/oidc/client`, and read only. Not
per hive, unlike the two prefixes above: an OIDC client is registered
once per swarm, so the path names the service and never a hive.
⚠️ Same collision as its siblings, and the same answer: ./swarm.nix
feeds this value into the guard on
{option}`services.hyperhive.swarm.hives`.
'';
};
otelOidcCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-otel-oidc";
example = "swarm-bao-otel-oidc.svc";
description = ''
Subject the store's `swarm-otel-oidc` cert-auth role accepts — the
identity the unit that fetches the collector's OIDC client secret
presents. Its grant is one path,
`swarm/services/<collector client id>/oidc/client`, and read only.
A **fourth** identity rather than reuse of
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`: the two
read different services' client secrets, and a collector is not
entitled to Grafana's.
'';
};
matrixTokenClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/matrix-token.pem";
description = ''
Certificate the unit that fetches this hive's matrix appservice token
presents to the store. Its subject must be
`<matrixTokenCommonNamePrefix>-<this hive's name>`; cert auth matches on
the CN and the role accepts nothing else.
⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
Pointing this at the hive's own leaf would evaluate, deploy and log in —
and hand a unit that reads one token a credential that reads every
secret in the store. That is what this option exists to end.
No default here: ./glue-bao-tls.nix points it at the leaf it mints,
wherever this host mints its own PKI. A deployment that reads a store on
another host names it by hand, alongside the leaf itself — one file per
principal placed out of band instead of one shared by four.
'';
};
matrixTokenClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/matrix-token-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile`. Both
or neither — a certificate with no key authenticates nothing.
'';
};
queueAgentClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/queue-agent.pem";
description = ''
Certificate the unit that fetches this hive's queue client credential
presents to the store. Its subject must be
`<queueAgentCommonNamePrefix>-<this hive's name>`.
⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives.
'';
};
queueAgentClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/queue-agent-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile`.
'';
};
grafanaOidcClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/grafana-oidc.pem";
description = ''
Certificate the unit that fetches Grafana's OIDC client secret presents
to the store. Its subject must be
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives.
'';
};
grafanaOidcClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/grafana-oidc-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile`.
'';
};
otelOidcClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/otel-oidc.pem";
description = ''
Certificate the unit that fetches the collector's OIDC client secret
presents to the store. Its subject must be
{option}`services.hyperhive.deploy.bao.otelOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives.
'';
};
otelOidcClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/otel-oidc-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.otelOidcClientCertFile`.
'';
};
serverCaFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
@ -1922,19 +1593,6 @@ in
'';
};
# 🩸 The four principals that used to log in as the hive. See the block
# around `perHiveReaders` for what each grant is and why the first two are
# per hive; the four units are identical apart from the objects they
# write, so the shape lives in `readerPolicyUnit` and the difference is
# visible here in one line each.
systemd.services.swarm-bao-matrix-token-policy = readerPolicyUnit "write the per-hive matrix-token bao policies and cert-auth roles" matrixTokenReaders;
systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders;
systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders;
systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders;
# The CA bind source is written at runtime by a host unit, so the
# container has to start after it — otherwise nspawn sets up a mount
# over a file that does not exist yet.

28
nix/host-modules/swarm-grafana.nix
View file

@ -119,14 +119,7 @@ let
# other secret is read with it. A Grafana host without it has not been given
# its identity yet, which is a thing to say out loud rather than to route
# around by reaching into authelia's tree whenever it happens to be local.
#
# 🩸 Grafana's OWN leaf, not `clientCertFile` — the hive's, which four units
# used to share. Bao matches a cert-auth role on the CN, so one leaf for four
# readers was ONE principal holding the union of four grants: this unit could
# read every agent credential in the swarm and every other service's OIDC
# client secret, when what it needs is the one path `storeSecretPath` names.
haveClientIdentity =
baoDeploy.grafanaOidcClientCertFile != null && baoDeploy.grafanaOidcClientKeyFile != null;
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
autheliaUrl = toString hyperhiveCfg.swarm.authelia.url;
@ -458,20 +451,15 @@ in
services.hyperhive.deploy.grafana.enable requires this host to hold a
swarm-secret-store client identity: set both
services.hyperhive.deploy.bao.grafanaOidcClientCertFile
services.hyperhive.deploy.bao.grafanaOidcClientKeyFile
services.hyperhive.deploy.bao.clientCertFile
services.hyperhive.deploy.bao.clientKeyFile
Grafana's OIDC client secret is minted by authelia and read out of
the store, on every host that runs Grafana — including the host that
runs authelia. That is one delivery route rather than two, and it is
what the store is for: a certificate is the credential placed out of
band, and every other secret comes from the store with it.
⚠️ Grafana's OWN leaf, not deploy.bao.clientCertFile. That one is the
hive's, and its grant reads every secret in the store; this role
reads the one path Grafana's client secret lives at. Pointing this
option at the hive's leaf would evaluate, deploy and log in — and
undo the split.
what the store is for: this certificate is the single credential
placed out of band, and every other secret comes from the store with
it.
On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere the leaf is issued
@ -622,8 +610,8 @@ in
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.grafanaOidcClientCertFile;
BAO_CLIENT_KEY = baoDeploy.grafanaOidcClientKeyFile;
BAO_CLIENT_CERT = baoDeploy.clientCertFile;
BAO_CLIENT_KEY = baoDeploy.clientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.

85
nix/host-modules/swarm-otel.nix
View file

@ -242,32 +242,13 @@ let
# A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical-
# looking flag, this one does not gate an assertion on its own: the
# store-reading unit below simply does not render without it, the same choice
# looking flag, this one stays outside `assertions`: the store-reading unit
# below simply does not render without it, the same choice
# ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for
# their own readers, because a collector with no client identity is
# their own optional readers, because a collector with no client identity is
# `haveCollectorSecret = false` above, and that is already a supported,
# merely degraded shape rather than a service with no way in at all. What
# IS asserted is narrower and lives in `assertions` below — see
# `hiveReaderIdentity`.
#
# 🩸 The collector's OWN leaf, not `clientCertFile` — the hive's, which four
# units used to share. Bao matches a cert-auth role on the CN, so one leaf for
# four readers was ONE principal holding the union of four grants: this unit
# could read every agent credential in the swarm and Grafana's OIDC client
# secret, when what it needs is the one path `storeSecretPath` names.
haveClientIdentity =
baoDeploy.otelOidcClientCertFile != null && baoDeploy.otelOidcClientKeyFile != null;
# Does this host read the store at all — the hive's own leaf, which is the
# one thing a remote-store deployment has always had to place by hand. It is
# what separates the degrade the comment above describes from the mistake the
# assertion below reports: a collector on a host holding NO store identity is
# the supported shape, and one on a host that demonstrably reads the store
# named seven of the eight options and stopped. Before the four-way split
# that second host rendered this unit off `clientCertFile`, so it is a silent
# regression rather than a choice anyone made.
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# merely degraded shape rather than a service with no way in at all.
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Where the publisher on authelia's host leaves this client's secret —
# composed from the same swarm-wide `clientId` the registration in
@ -744,14 +725,11 @@ in
# value.
#
# ⚠️ Renders only where `haveClientIdentity` holds, unlike
# ./swarm-grafana.nix's equivalent unit. That module refuses any host that
# runs Grafana without the identity, because a Grafana with none has no way
# in at all; this collector without one is `haveCollectorSecret = false`
# above — already a supported, merely degraded shape, so the unit that would
# fetch a credential simply does not exist rather than refusing the build
# for want of one. The one case that IS refused is the host that already
# holds the hive's leaf and is missing only this pair, which is a silent
# regression rather than that degrade — `hiveReaderIdentity` above.
# ./swarm-grafana.nix's equivalent unit. That module asserts the identity
# because a Grafana with none has no way in at all; this collector without
# one is `haveCollectorSecret = false` above — already a supported,
# merely degraded shape, so the unit that would fetch a credential simply
# does not exist rather than refusing the build for want of one.
systemd.services.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity {
description = "fetch the swarm collector's OIDC client secret from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
@ -797,8 +775,8 @@ in
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.otelOidcClientCertFile;
BAO_CLIENT_KEY = baoDeploy.otelOidcClientKeyFile;
BAO_CLIENT_CERT = baoDeploy.clientCertFile;
BAO_CLIENT_KEY = baoDeploy.clientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
@ -875,45 +853,6 @@ in
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;
assertions = [
{
# Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion —
# the same refusal, named to this principal's own pair. What differs is
# the `!hiveReaderIdentity ||` guard, and it is what keeps the degrade
# the flag's own comment describes intact: Grafana refuses any host
# that runs it without a leaf, because a Grafana with no SSO has no way
# in at all, while a collector with none still receives telemetry. So
# this fires only where the host already reads the store and is missing
# this one pair — which before the four-way split rendered the unit off
# `clientCertFile`, and now silently does not.
assertion = !hiveReaderIdentity || haveClientIdentity;
message = ''
This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile
is set) and runs the swarm collector, so it needs the collector's own
client identity: set both
services.hyperhive.deploy.bao.otelOidcClientCertFile
services.hyperhive.deploy.bao.otelOidcClientKeyFile
swarm-bao-otel-oidc.service fetches the collector's OIDC client
secret out of the store, and without these it is not rendered at all
— so this collector would authenticate to nothing, quietly, on a host
that has everything it needs to fetch the secret.
A collector on a host holding NO store identity is a different and
supported shape: it runs without a client secret and still receives
telemetry. That is not this host.
⚠️ The collector's OWN leaf, not deploy.bao.clientCertFile. That one
is the hive's, and its grant reads every secret in the store; this
role reads the one path this collector's client secret lives at.
Pointing this option at the hive's leaf would evaluate, deploy and
log in — and undo the split.
On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere the leaf is issued
from that CA out of band and named here — see docs/swarm/secrets.md.
'';
}
# ⚠️ An assertion that this collector has "somewhere to send" was REMOVED
# rather than relaxed: it read the store's *per-host* enable, so it
# rejected at eval the very deployment the stores are reached by domain

15
nix/host-modules/swarm.nix
View file

@ -47,20 +47,7 @@ let
deployCfg.bao.controllerCommonName
deployCfg.bao.secretPublisherCommonName
deployCfg.bao.matrixCtlCommonName
deployCfg.bao.grafanaOidcCommonName
deployCfg.bao.otelOidcCommonName
]
# The two per-hive readers' subjects, spelled out per hive rather than as the
# prefix. The prefix alone would reserve the wrong string: the role for hive
# `h` accepts `<prefix>-h`, so the name a hive must not BE is that composed
# spelling, and a hive named it would present a leaf that role accepts.
#
# Both spellings are reserved for every declared hive, including the one the
# CN belongs to — a hive cannot be named after its own reader either.
++ lib.concatMap (hive: [
"${deployCfg.bao.matrixTokenCommonNamePrefix}-${hive}"
"${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}"
]) hiveNames;
];
# Public hostnames of the swarm's own services, in declaration order.
# `serviceDomains` below is this set sorted + deduplicated.

256
nix/module-eval/bao-grants.nix
View file

@ -45,18 +45,6 @@ let
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
deploy.bao.clientCaFile = lib.mkForce null;
};
# The store plus every one of the four readers that used to log in as the
# hive. One fixture rather than four: the claim they are four *separate*
# principals is only testable where all four render at once — that is the
# deployment in which two of them sharing a leaf would be invisible.
baoGrantWithConsumers = hive {
deploy.bao.enable = true;
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
deploy.matrix.enable = true;
deploy.grafana.enable = true;
deploy.swarm-otel.enable = true;
};
cases = [
{
# Reads the rendered unit on the HOST, which is where the write happens:
@ -208,250 +196,6 @@ let
)
&& !(baoGrantHere.containers.swarm-bao.config.systemd.services ? swarm-bao-matrix-ctl-policy);
}
# ── the four readers that used to share the hive's own leaf ──────────────
#
# 🩸 Until this split all four presented `deploy.bao.clientCertFile`, whose
# policy grants read on `swarm/agents/*`, `swarm/hives/<hive>/*` AND
# `swarm/services/*`. Four principals behind one certificate are one
# principal to bao, so the only expressible grant was the union: the unit
# fetching Grafana's OIDC secret could fetch every agent credential in the
# swarm.
#
# Every one of these cases carries the same three negative arms, and they
# are the deliverable rather than decoration — a positive arm alone passes
# just as well when the other two stanzas are still there beside it. The
# arms pin what each principal must NOT reach, so a later widening fails
# here instead of being noticed in a store.
{
name = "the matrix-token reader's grant is one hive's appservice token and nothing else";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-matrix-token-policy.script;
in
lib.hasInfix "path \"secret/data/swarm/hives/h1/matrix/appservice-token\" {" s
&& lib.hasInfix "capabilities = [\"read\"]" s
# The three stanzas the hive's own leaf carried, none of which this
# principal needs: every agent's credential, every service's OIDC
# client, and the rest of its own hive's tree — including the queue
# credential its sibling reader fetches.
&& !(lib.hasInfix "secret/data/swarm/agents" s)
&& !(lib.hasInfix "secret/data/swarm/services" s)
&& !(lib.hasInfix "secret/data/swarm/hives/h1/*" s)
&& !(lib.hasInfix "secret/data/swarm/hives/h1/queue" s)
# A `hives/*` wildcard would serve every hive from one role and let any
# hive read any other's token — the reach this split exists to remove,
# not to create.
&& !(lib.hasInfix "secret/data/swarm/hives/*" s)
# Nothing may rewrite the policy constraining it, for the reason the
# controller's own `hive-*` narrowing above gives.
&& !(lib.hasInfix "sys/policies/acl" s);
}
{
name = "the queue-credential reader's grant is one hive's queue credential and nothing else";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-queue-agent-policy.script;
in
lib.hasInfix "path \"secret/data/swarm/hives/h1/queue/agent\" {" s
&& lib.hasInfix "capabilities = [\"read\"]" s
&& !(lib.hasInfix "secret/data/swarm/agents" s)
&& !(lib.hasInfix "secret/data/swarm/services" s)
&& !(lib.hasInfix "secret/data/swarm/hives/h1/*" s)
&& !(lib.hasInfix "secret/data/swarm/hives/h1/matrix" s)
&& !(lib.hasInfix "secret/data/swarm/hives/*" s)
&& !(lib.hasInfix "sys/policies/acl" s);
}
{
# ⚠️ The client id is the path segment, so the negative arm that matters
# for this one is the OTHER service's: `services/*` would have granted
# both, and the two are separate principals precisely because a
# dashboard is not entitled to a collector's credential.
name = "the Grafana OIDC reader's grant is Grafana's own client secret and nothing else";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-grafana-oidc-policy.script;
in
lib.hasInfix "path \"secret/data/swarm/services/swarm-grafana/oidc/client\" {" s
&& lib.hasInfix "capabilities = [\"read\"]" s
&& !(lib.hasInfix "secret/data/swarm/agents" s)
&& !(lib.hasInfix "secret/data/swarm/hives" s)
&& !(lib.hasInfix "secret/data/swarm/services/*" s)
&& !(lib.hasInfix "swarm-collector" s)
&& !(lib.hasInfix "sys/policies/acl" s);
}
{
# The mirror of the case above, and the arm naming `swarm-grafana` is why
# these are two principals rather than one `services/*` grant shared.
name = "the collector OIDC reader's grant is the collector's own client secret and nothing else";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-otel-oidc-policy.script;
in
lib.hasInfix "path \"secret/data/swarm/services/swarm-collector/oidc/client\" {" s
&& lib.hasInfix "capabilities = [\"read\"]" s
&& !(lib.hasInfix "secret/data/swarm/agents" s)
&& !(lib.hasInfix "secret/data/swarm/hives" s)
&& !(lib.hasInfix "secret/data/swarm/services/*" s)
&& !(lib.hasInfix "swarm-grafana" s)
&& !(lib.hasInfix "sys/policies/acl" s);
}
{
# 🩸 The half that makes the policies above bind: a policy grants only
# through a token that carries it, and a token is minted by a cert-auth
# role matching a CN. Four distinct subjects is the whole mechanism — one
# subject for four readers is one principal however the policies read.
#
# The per-hive subjects carry the hive name because their paths do; the
# two service subjects do not, because an OIDC client is registered once
# per swarm. Pinned so neither shape is tidied into the other.
name = "each of the four readers logs in under a subject of its own";
ok =
let
subjectOf =
unit: role: cn:
let
s = baoGrantHere.systemd.services.${unit}.script;
in
lib.hasInfix "auth/cert/certs/${role}" s
&& lib.hasInfix "allowed_common_names=${cn}" s
&& lib.hasInfix "token_policies=${role}" s
# Outside the `hive-*` namespace the controller may rewrite, for
# the reason the three service principals above state.
&& !(lib.hasInfix "auth/cert/certs/hive-" s);
in
subjectOf "swarm-bao-matrix-token-policy" "swarm-matrix-token-h1" "swarm-bao-matrix-token-h1"
&& subjectOf "swarm-bao-queue-agent-policy" "swarm-queue-agent-h1" "swarm-bao-queue-agent-h1"
&& subjectOf "swarm-bao-grafana-oidc-policy" "swarm-grafana-oidc" "swarm-bao-grafana-oidc"
&& subjectOf "swarm-bao-otel-oidc-policy" "swarm-otel-oidc" "swarm-bao-otel-oidc";
}
{
# 🩸 The consuming side, and the arm that would catch the regression that
# costs the most: a unit repointed back at `deploy.bao.clientCertFile`
# evaluates, deploys and logs in — and silently restores the union grant,
# because bao would again see one principal. Nothing about the policies
# above would look wrong.
#
# Each pair is asserted whole: a certificate with no key authenticates
# nothing, so a half-set pair is a reader that does not render.
name = "each of the four readers presents its own leaf, never the hive's";
ok =
let
b = baoGrantWithConsumers.services.hyperhive.deploy.bao;
hiveLeaf = [
b.clientCertFile
b.clientKeyFile
];
own = [
b.matrixTokenClientCertFile
b.matrixTokenClientKeyFile
b.queueAgentClientCertFile
b.queueAgentClientKeyFile
b.grafanaOidcClientCertFile
b.grafanaOidcClientKeyFile
b.otelOidcClientCertFile
b.otelOidcClientKeyFile
];
envOf = unit: baoGrantWithConsumers.systemd.services.${unit}.environment;
presents =
unit: cert: key:
(envOf unit).BAO_CLIENT_CERT == cert && (envOf unit).BAO_CLIENT_KEY == key;
in
lib.all (p: p != null) own
&& !(lib.any (p: lib.elem p hiveLeaf) own)
&& lib.length (lib.unique own) == lib.length own
&& presents "swarm-bao-matrix-token" b.matrixTokenClientCertFile b.matrixTokenClientKeyFile
&& presents "swarm-bao-queue-agent" b.queueAgentClientCertFile b.queueAgentClientKeyFile
&& presents "swarm-bao-grafana-oidc" b.grafanaOidcClientCertFile b.grafanaOidcClientKeyFile
&& presents "swarm-bao-otel-oidc" b.otelOidcClientCertFile b.otelOidcClientKeyFile;
}
{
# The minting side of the same claim. A role matching a subject nothing
# signs is a reader that cannot log in, so the leaves and the roles have
# to be asserted against each other — and the two per-hive leaves carry
# THIS host's hive name, which is what makes one hive's leaf useless
# against another hive's role.
name = "the PKI unit signs a leaf per reader, each under that reader's own subject";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-pki.script;
in
# The basename and the subject are matched separately: `signLeaf` takes
# them as consecutive arguments across a `\` continuation, so one
# literal spanning both would pin this file's line wrapping rather than
# the pairing it means to.
lib.all (lib.flip lib.hasInfix s) [
"/matrix-token.pem ]"
"swarm-bao-matrix-token-h1 \"\" clientAuth"
"/queue-agent.pem ]"
"swarm-bao-queue-agent-h1 \"\" clientAuth"
"/grafana-oidc.pem ]"
"swarm-bao-grafana-oidc \"\" clientAuth"
"/otel-oidc.pem ]"
"swarm-bao-otel-oidc \"\" clientAuth"
];
}
{
# The absence arm: with no client CA there is no trust anchor, so the
# login roles cannot be written — but the policies they would attach are
# still asserted, exactly as the three service principals above behave in
# this deployment. A unit that vanished here would take the policy with
# it and leave nothing to diagnose.
name = "with no client CA the four readers get policies but no login roles";
ok =
let
units = [
"swarm-bao-matrix-token-policy"
"swarm-bao-queue-agent-policy"
"swarm-bao-grafana-oidc-policy"
"swarm-bao-otel-oidc-policy"
];
scriptOf = unit: baoGrantNoClientCa.systemd.services.${unit}.script;
in
lib.all (
unit:
(baoGrantNoClientCa.systemd.services ? ${unit})
&& lib.hasInfix "bao policy write" (scriptOf unit)
&& !(lib.hasInfix "auth/cert/certs" (scriptOf unit))
) units;
}
{
# Same control the three service principals carry: the write needs a
# client certificate and the host is the side that has one, so a unit
# rendered inside the store's container would have neither an identity
# nor a route. Plus the ordering that makes the mounts exist first.
name = "the four readers' granting units are ordered after the mounts and rendered on the host";
ok =
let
units = [
"swarm-bao-matrix-token-policy"
"swarm-bao-queue-agent-policy"
"swarm-bao-grafana-oidc-policy"
"swarm-bao-otel-oidc-policy"
];
in
lib.all (
unit:
lib.elem "swarm-bao-controller-policy.service" baoGrantHere.systemd.services.${unit}.after
&& !(baoGrantHere.containers.swarm-bao.config.systemd.services ? ${unit})
) units;
}
{
# A store host that has not placed a bootstrap token can write no grant at
# all, so none of the four units may exist — the same claim
# `baoGrantNoStore` makes for the controller's, one file over. Without
# this arm `lib.mkIf haveBootstrapToken` could be dropped from the shared
# builder and every other case here would still pass.
name = "without a bootstrap token none of the four readers' granting units render";
ok =
let
s = baoGrantNoStore.systemd.services;
in
!(s ? swarm-bao-matrix-token-policy)
&& !(s ? swarm-bao-queue-agent-policy)
&& !(s ? swarm-bao-grafana-oidc-policy)
&& !(s ? swarm-bao-otel-oidc-policy);
}
{
# The policy authorising this route lives in another file, and nothing
# else relates the grants to the paths the code actually writes.

99
nix/module-eval/bao-matrix-reader.nix
View file

@ -32,20 +32,10 @@ let
# nothing here mints a leaf and the operator names one placed by hand. The
# deployment this pairing exists to serve, and the one that was previously
# inexpressible — the gate asked whether the store was a neighbour.
#
# 🩸 One leaf PER READER, and this fixture is where that cost is visible: the
# two units below identify themselves to the store separately, so an operator
# placing leaves by hand places one for each rather than one for both. Naming
# only `clientCertFile` here would leave neither unit rendered — which is what
# the arms below would then be asserting about.
baoRemoteReader = hive {
deploy.matrix.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.matrixTokenClientCertFile = "/etc/pki/bao-matrix-token.pem";
deploy.bao.matrixTokenClientKeyFile = "/etc/pki/bao-matrix-token-key.pem";
deploy.bao.queueAgentClientCertFile = "/etc/pki/bao-queue-agent.pem";
deploy.bao.queueAgentClientKeyFile = "/etc/pki/bao-queue-agent-key.pem";
};
# A homeserver on a hive with NO store identity at all — neither a local
@ -53,38 +43,6 @@ let
# needs it, and defining it here rather than importing keeps each group's
# fixture set its own, as ./lib.nix asks.
matrixNoBaoIdentity = hive { deploy.matrix.enable = true; };
# 🩸 The hand-configured remote reader that named seven of the eight options.
# `baoRemoteReader` above is the same deployment done right; this one holds
# the hive's own leaf, so it demonstrably reads the store, and is missing both
# per-principal pairs. Before the four-way split this host rendered both units
# off `clientCertFile` alone, so what it has now is a silent regression rather
# than any shape an operator chose — which is what the refusal arms below are
# about. Kept as one fixture rather than two because both refusals fire on it
# and each arm names which.
baoRemoteReaderMissingLeaves = hive {
deploy.matrix.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# The same omission on a hive that does NOT run a homeserver. Separates the
# matrix refusal's `deploy.matrix.enable` clause from the queue refusal, which
# has no toggle to check — an arm below reads exactly one refusal off it.
remoteReaderNoMatrixMissingLeaves = hive {
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# Did a module refuse this host, and over which option. An assertion is a
# config VALUE until something forces it — `.config` never throws — so a
# fixture in a state the module refuses stays evaluable and the refusal reads
# back as data. Matched on the option name the message names rather than on
# its prose, because the option name is the part an operator has to act on
# and a message that stopped naming it would be the actual defect. Same shape
# as ./grafana.nix's `grafanaRefusedFor`, which reads `swarm-grafana.nix`'s
# own refusal.
refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions;
cases = [
{
# A login failure is the store being unreachable, sealed, or not yet
@ -405,63 +363,6 @@ let
in
!(g ? admin_execute) || g.admin_execute == [ ];
}
{
# 🩸 The arm the whole refusal exists for. Both readers are gated on their
# own leaf, so the way this regresses is the build going green and three
# units rendering where four should — which no presence check on a
# rendered unit can see, because the unit that is missing is the evidence.
# Read as a refusal naming each option, so an operator acts on the message
# without opening the nix.
name = "a remote reader missing the per-principal leaves is refused, naming both options";
ok =
refusedOver baoRemoteReaderMissingLeaves "matrixTokenClientCertFile"
&& refusedOver baoRemoteReaderMissingLeaves "matrixTokenClientKeyFile"
&& refusedOver baoRemoteReaderMissingLeaves "queueAgentClientCertFile"
&& refusedOver baoRemoteReaderMissingLeaves "queueAgentClientKeyFile";
}
{
# The matrix refusal's own gate, which the queue refusal does not have.
# Without this arm the two are indistinguishable on the fixture above.
name = "the queue refusal needs no homeserver, and the matrix refusal stays quiet without one";
ok =
refusedOver remoteReaderNoMatrixMissingLeaves "queueAgentClientCertFile"
&& !(refusedOver remoteReaderNoMatrixMissingLeaves "matrixTokenClientCertFile");
}
{
# ⚠️ The arm that keeps the refusal from being worse than the silence it
# replaced. A hive with NO store identity is the supported no-store
# deployment and also the state an operator passes through bringing a hive
# up — neither may fail to evaluate. Asserted as "no refusal names any of
# the four options", not as "this one fixture is fine", because the way
# this breaks is a gate widened to the principal's leaf alone.
name = "a hive with no store identity at all is refused over none of the per-principal leaves";
ok = lib.all (option: !(refusedOver matrixNoBaoIdentity option)) [
"matrixTokenClientCertFile"
"matrixTokenClientKeyFile"
"queueAgentClientCertFile"
"queueAgentClientKeyFile"
];
}
{
# The other half of the same guard, and the one an operator meets far more
# often: on the store's own host ./host-modules/glue-bao-tls.nix mkDefaults
# all eight, so there is nothing to name and nothing to refuse. Paired with
# the fully-named remote reader, which is the same deployment done by hand.
name = "neither a store host nor a correctly-named remote reader is refused";
ok =
lib.all
(
m:
lib.all (option: !(refusedOver m option)) [
"matrixTokenClientCertFile"
"queueAgentClientCertFile"
]
)
[
baoWithMatrix
baoRemoteReader
];
}
];
in
runGroup "bao-matrix-reader" cases

12
nix/module-eval/grafana.nix
View file

@ -41,8 +41,6 @@ let
deploy.authelia.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem";
deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem";
};
# The same UI with the IdP on ANOTHER host and a store leaf placed by hand.
@ -56,8 +54,6 @@ let
swarm.authelia.url = "https://auth.example.invalid";
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem";
deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem";
};
# A Grafana host holding no store identity. This used to be the shape the
@ -152,8 +148,8 @@ let
# mean the two assertions had been collapsed into one conjunction.
name = "a grafana host with no store identity is refused, naming the options to set";
ok =
grafanaRefusedFor grafanaNoIdentity "deploy.bao.grafanaOidcClientCertFile"
&& grafanaRefusedFor grafanaNoIdentity "deploy.bao.grafanaOidcClientKeyFile"
grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientCertFile"
&& grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientKeyFile"
&& !(grafanaRefusedFor grafanaNoIdentity "swarm.authelia.url");
}
{
@ -165,9 +161,9 @@ let
name = "neither grafana refusal fires on a correctly configured host, co-located or not";
ok =
!(grafanaRefusedFor grafanaWithAuthelia "services.hyperhive.swarm.authelia.url")
&& !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.grafanaOidcClientCertFile")
&& !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.clientCertFile")
&& !(grafanaRefusedFor grafanaRemoteAuthelia "services.hyperhive.swarm.authelia.url")
&& !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.grafanaOidcClientCertFile");
&& !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.clientCertFile");
}
{
# Same 403-not-a-miss reason as the matrix and queue arms below: the

59
nix/module-eval/name-guards.nix
View file

@ -63,35 +63,6 @@ let
swarm.hives.mintctl.domain = "m.t.local";
};
# The two OIDC-secret readers' subjects, fixed strings like the three above.
hiveNamedAfterGrafanaOidcSubject = hive {
deploy.swarm-otel.enable = false;
deploy.bao.grafanaOidcCommonName = "gfctl";
swarm.hives.gfctl.domain = "g.t.local";
};
hiveNamedAfterOtelOidcSubject = hive {
deploy.swarm-otel.enable = false;
deploy.bao.otelOidcCommonName = "otctl";
swarm.hives.otctl.domain = "o.t.local";
};
# 🩸 A different shape from every fixture above: the matrix-token and
# queue-credential roles are written PER HIVE, so the subject a hive must not
# be is `<prefix>-<some hive's name>` rather than the prefix itself. Reserving
# only the prefix would leave the composed spelling free, and a hive taking it
# would present a leaf the other hive's role accepts — which is a hive reading
# another hive's queue credential, the exact widening the split exists to
# avoid.
#
# Two hives here, not one: the collision is with the OTHER hive's role.
hiveNamedAfterPerHiveReaderSubject = hive {
deploy.swarm-otel.enable = false;
deploy.bao.queueAgentCommonNamePrefix = "qr";
swarm.hives.other.domain = "o.t.local";
swarm.hives.qr-other.domain = "q.t.local";
};
hiveNameWithComposedWord = hive {
deploy.swarm-otel.enable = false;
swarm.hives."h1-agent".domain = "a.t.local";
@ -140,36 +111,6 @@ let
a: !a.assertion && lib.hasInfix "'mintctl'" a.message
) hiveNamedAfterMatrixCtlSubject.assertions;
}
{
# The fourth and fifth, for the reason the case above gives: `certAuthCns`
# is where a role added beside the others registers itself, and nothing
# but a case per element notices when one forgets. These two are the
# subjects of the readers that fetch Grafana's and the collector's OIDC
# client secrets.
name = "a hive named after either OIDC-secret reader's subject is refused too";
ok =
equalityGuardFired hiveNamedAfterGrafanaOidcSubject
&& lib.any (
a: !a.assertion && lib.hasInfix "'gfctl'" a.message
) hiveNamedAfterGrafanaOidcSubject.assertions
&& equalityGuardFired hiveNamedAfterOtelOidcSubject
&& lib.any (
a: !a.assertion && lib.hasInfix "'otctl'" a.message
) hiveNamedAfterOtelOidcSubject.assertions;
}
{
# 🩸 The per-hive half, and the one a prefix-only reservation would miss:
# the role is `<prefix>-<hive>`, so the reserved string has to be composed
# against every declared hive. Here hive `qr-other` collides with the role
# written for hive `other` — a leaf that reads a credential belonging to a
# hive that is not it.
name = "a hive named after another hive's per-hive reader subject is refused";
ok =
equalityGuardFired hiveNamedAfterPerHiveReaderSubject
&& lib.any (
a: !a.assertion && lib.hasInfix "'qr-other'" a.message
) hiveNamedAfterPerHiveReaderSubject.assertions;
}
{
# Without this the case above proves nothing: an arm that fires for every
# roster is not a guard, and `hives` is non-empty in both fixtures.

6
nix/module-eval/swarm-otel-core.nix
View file

@ -32,8 +32,6 @@ let
deploy.authelia.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem";
deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem";
deploy.victoriametrics.enable = false;
deploy.victorialogs.enable = false;
};
@ -46,8 +44,6 @@ let
deploy.authelia.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem";
deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem";
};
# The same collector with the IdP on ANOTHER host and a store leaf placed by
@ -58,8 +54,6 @@ let
swarm.authelia.url = "https://auth.example.invalid";
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem";
deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem";
};
cases = [
{

66
nix/module-eval/swarm-otel-identity.nix
View file

@ -34,40 +34,6 @@ let
deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret";
};
# 🩸 The shape the degrade above must not swallow: a collector on a host that
# demonstrably READS the store — it holds the hive's own leaf — and is missing
# only the collector's own pair. Before the four-way split this host rendered
# the reading unit off `clientCertFile` alone, so the silence it gets now is a
# regression rather than the supported degrade `otelNoIdentity` stands for. The
# two fixtures differ in exactly that one pair, which is what lets the arms
# below separate a refusal from a degrade.
otelReaderMissingLeaf = hive {
deploy.swarm-otel.enable = true;
swarm.authelia.url = "https://auth.example.invalid";
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# The same deployment named in full, which must stay buildable. Pairs with the
# fixture above so the refusal is pinned to the omission and not to reading a
# remote store at all.
otelReaderNamedInFull = hive {
deploy.swarm-otel.enable = true;
swarm.authelia.url = "https://auth.example.invalid";
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem";
deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem";
};
# Did the module refuse this host, and over which option. An assertion is a
# config VALUE until something forces it — `.config` never throws — so a
# fixture in a refused state stays evaluable and the refusal reads back as
# data. Matched on the option name the message names rather than on its prose:
# the option name is the part an operator has to act on, and a message that
# stopped naming it would be the actual defect.
refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions;
# authelia somewhere else, the credential delivered by hand. Whether this
# collector authenticates must follow the credential, never another
# service's placement.
@ -196,38 +162,6 @@ let
&& !(builtins.elem "otlphttp/victoriametrics" used)
&& lib.all (e: s.exporters ? ${e}) used;
}
{
# 🩸 The refusal, and the only way this module's omission is visible at
# all: the reading unit is gated on its own leaf, so the regression is a
# green build with the unit absent — and the missing unit is the evidence.
# Read as a refusal naming both options, so an operator acts on the
# message without opening the nix.
name = "a store reader missing the collector's own leaf is refused, naming both options";
ok =
refusedOver otelReaderMissingLeaf "otelOidcClientCertFile"
&& refusedOver otelReaderMissingLeaf "otelOidcClientKeyFile";
}
{
# ⚠️ The arm that keeps the refusal from eating the degrade beside it. A
# collector on a host with NO store identity still receives every hive's
# telemetry and is a supported deployment — `otelNoIdentity` is that
# fixture, and it differs from the refused one only in the hive's own
# leaf. Widening the gate to the collector's leaf alone would reject it,
# which is what this pins.
name = "a collector with no store identity at all is not refused over the collector's leaf";
ok =
!(refusedOver otelNoIdentity "otelOidcClientCertFile")
&& !(refusedOver otelNoIdentity "otelOidcClientKeyFile");
}
{
# The other two shapes that must stay buildable: the same remote reader
# named in full, and a host that mints its own PKI and therefore has all
# eight as defaults from ./host-modules/glue-bao-tls.nix.
name = "neither a fully-named remote reader nor a store host is refused";
ok =
!(refusedOver otelReaderNamedInFull "otelOidcClientCertFile")
&& !(refusedOver otelRemoteAuthelia "otelOidcClientCertFile");
}
];
in
runGroup "swarm-otel-identity" cases