vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

drop the shared hive agent login

Browse source 
Every agent now connects to the queue and reads logs with its own secret
from the store (#5181), so the per-hive `hive-<h>-agent` OIDC client has
no reader left. Remove it end to end:

- hive-c0re no longer forwards `hive-queue-agent-{secret,client-id}` into
  containers, and no longer forwards HIVE_AGENT_OIDC_TOKEN_ENDPOINT; the
  agent module that loaded them (nix/agent-modules/queue.nix) is gone, its
  options refused with a removal message.
- The host reader (glue-queue-agent-credential.nix, swarm-bao-queue-agent),
  its store role and policy unit (swarm-bao-queue-agent-policy), its PKI
  leaf and the deploy.bao.queueAgent* / queue.agentCredentialDir options
  are gone; the options are refused with a removal message.
- authelia mints no `hive-<h>-agent` client, and swarm-secret-publish no
  longer writes swarm/hives/<h>/queue/agent.
- swarm-secret-client drops queue::agent_client_path / queue::Credential.
- HIVE_AGENT_NATS_URL is now gated on the agents' queue address alone.

The log store's authelia check stays for operators. swarm-nats-auth still
refuses a `-agent` client id; that refusal and `agentClientSuffix` are left
for a follow-up.

Refs #4459
Refs #4765
This commit is contained in:
atlas 2026-10-13 09:49:32 +02:00 • committed by mara
commit 868c6cf96c
41 changed files with 286 additions and 1410 deletions
Download patch file Download diff file Expand all files Collapse all files

10
nix/agent-modules/bao.nix
View file

@ -14,8 +14,7 @@
# is why: the host file is `0600` to the hive daemon and this unit runs as the
# unprivileged agent user. nspawn's `--load-credential` is read by the
# container manager as root and re-exposed under this unit's own `User=`; a
# bind would deliver a private key this user cannot open. Same answer, same
# reason, as ./queue.nix's credential pair.
# bind would deliver a private key this user cannot open.
#
# This is the courier's other end: the hive-side collector has no purpose
# without it, so the two ship together and neither is reachable alone.
@ -63,12 +62,11 @@ let
# Where the identity lives in the store, spelled from the same pieces the
# publisher uses. `swarm_secret_client::mtls::identity_path` builds
# `swarm/agents/<agent>/bao-mtls` and `path::MOUNT` is `secret`; this
# literal is the nix half of that one agreement, exactly as
# ../host-modules/glue-queue-agent-credential.nix spells its own.
# literal is the nix half of that one agreement.
identityPath = "secret/swarm/agents/${agentName}/bao-mtls";
# The address is the whole switch — no separate `enable`, the same shape
# ./queue.nix and ./logs.nix gate themselves with. A hive that has a store
# ./logs.nix gates itself with. A hive that has a store
# forwards its address and every agent on it gets the check; a hive that has
# none forwards nothing and no agent does. An `enable` beside it would be a
# knob whose only correct setting is whatever the address already says, and
@ -153,7 +151,7 @@ in
UMask = "0377";
# Bare ids, no paths: the terse `LoadCredential=` form that inherits a
# credential the service *manager* received, which is what the
# container manager passed in. ./queue.nix states the same shape.
# container manager passed in.
LoadCredential = [
certCredential
keyCredential

9
nix/agent-modules/default.nix
View file

@ -34,6 +34,12 @@ let
sets it. Pin it host-side with
services.hyperhive.deploy.hive-controller.claudeCodePackage.
'';
queueRemoved = ''
The harness reads the queue's address at runtime from HIVE_AGENT_NATS_URL,
which the hive forwards into every container, and authenticates with the
agent's own secret from the swarm secret store, so no agent build sets a
queue option. Remove the setting.
'';
# `claude` on PATH. A hive that pins a claude-code
# (`services.hyperhive.deploy.hive-controller.claudeCodePackage`) hands its
@ -81,7 +87,6 @@ in
./otel.nix
./otel-removed.nix
./packages.nix
./queue.nix
./queue-identity.nix
./user.nix
./screen.nix
@ -101,6 +106,8 @@ in
"agent"
"claudeCodePath"
] claudeCodePathRemoved)
(lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "queue" "natsUrl" ] queueRemoved)
(lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "queue" "tokenEndpoint" ] queueRemoved)
];
options.services.hyperhive.agent.icon = lib.mkOption {

20
nix/agent-modules/queue-identity.nix
View file

@ -1,13 +1,11 @@
# This agent's own credential for the swarm queue, read from the store by the
# harness itself.
#
# The credential beside it in ./queue.nix is keyed per **hive**: one OIDC
# client minted at deploy time and handed to every agent container on the
# hive, so at the queue's auth callout one agent is indistinguishable from its
# co-hived neighbours. This one is keyed per **agent** — `swarm-controller`
# mints it at agent creation into `swarm/agents/<agent>/queue`
# (`swarm_secret_client::queue::agent_queue_path`), at swarm level, with no
# hive anywhere in the chain.
# Keyed per **agent**, so at the queue's auth callout one agent is
# distinguishable from its co-hived neighbours: `swarm-controller` mints it at
# agent creation into `swarm/agents/<agent>/queue`
# (`swarm_secret_client::queue::agent_queue_path`), at swarm level, with no hive
# anywhere in the chain.
#
# The agent authenticates to the store as itself, with the certificate
# ./bao.nix already proves it can log in with, and reads its own path. A hive
@ -42,11 +40,7 @@ let
# too.
serverCaCredential = "hive-agent-bao-server-ca";
# The store's address is the whole switch, exactly as in ./bao.nix — and
# deliberately *not* the queue coordinates in ./queue.nix. Those are the
# hive's, and gating a swarm-minted per-agent credential on a hive-level
# option would put the hive back in a path whose entire purpose is not
# having one.
# The store's address is the whole switch, exactly as in ./bao.nix.
configured = cfg.addr != null;
in
{
@ -54,7 +48,7 @@ in
systemd.services.hive-agent = {
# A bare id, no path: the terse `LoadCredential=` form that inherits a
# credential the service *manager* received, which is what the container
# manager passed in. ./bao.nix and ./queue.nix state the same shape.
# manager passed in. ./bao.nix states the same shape.
serviceConfig.LoadCredential = [ serverCaCredential ];
# The live certificate pair exists once ./bao.nix's unit has run.
after = [ "hive-agent-bao-identity.service" ];

117
nix/agent-modules/queue.nix
View file

@ -1,117 +0,0 @@
# The hive's shared swarm-queue client credential, imported into the harness
# unit's credentials directory for `swarm-logs` (./logs.nix) to authenticate
# with. The harness itself connects to the queue with the agent's own
# credential (./queue-identity.nix) and reads neither file.
#
# The credential is two files, the secret and the client id beside it, so a
# reader never spells `hive-<name>-agent` a second time.
#
# ⚠️ The credential arrives as a systemd credential and NOT as a bind mount,
# and the mode is why: the host file is `root:0600` and the reader runs as the
# unprivileged agent user. nspawn's `--load-credential` (written by
# `hive_c0re::lifecycle::host_config`) is read by the container manager as
# root and re-exposed under this unit's own `User=`; a bind would deliver a
# file the agent user cannot open.
{
lib,
config,
...
}:
let
cfg = config.services.hyperhive.agent.queue;
configured = cfg.natsUrl != null && cfg.tokenEndpoint != null;
# The two ids `hive_c0re::lifecycle::host_config` forwards under. Neither
# side can discover the other's spelling, so a rename is a rename there too.
secretCredential = "hive-queue-agent-secret";
clientIdCredential = "hive-queue-agent-client-id";
# Where systemd materialises this unit's credentials. `%d` only expands
# *inside* a unit, so `swarm-logs` (./logs.nix), which is not one, needs the
# literal; deriving it from the unit name here is what keeps the two from
# disagreeing about which unit's credentials they mean.
credentialsDir = "/run/credentials/hive-agent.service";
in
{
options.services.hyperhive.agent.queue = {
natsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "tls://nats.example.com:4222";
description = ''
Where the swarm queue listens, as this container reaches it.
Set by the generated meta flake from the host's
{option}`services.hyperhive.deploy.hive-controller.queue.agentNatsUrl`,
which names the queue rather than a loopback address — inside this
container `127.0.0.1` is the agent itself.
`null` means this hive has not been given the queue's address for its
agents: the harness unit imports no credential, and the harness fails
at boot, since it cannot run without a queue connection. It is
deliberately not defaulted to anything: a guessed address builds fine
and talks to the wrong machine.
'';
};
tokenEndpoint = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "https://auth.example.com/api/oidc/token";
description = ''
The swarm IdP's OAuth2 token endpoint, where `swarm-logs` mints a
`client_credentials` token for the client named in the delivered
credential.
The harness unit imports that credential only when this and
{option}`services.hyperhive.agent.queue.natsUrl` are both set.
'';
};
# One declaration, two readers. The credential ids are this module's —
# `hive_c0re::lifecycle::host_config` and the `LoadCredential=` below are
# the pair that has to agree on them — and a second consumer that spelled
# them again would be a second source of truth for a string whose
# mismatch is a file that is simply not there.
clientIdFile = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "${credentialsDir}/${clientIdCredential}";
description = ''
Path the agent's OIDC client id is delivered at, for a consumer
outside the harness unit. Read-only: it is a fact about where the
credential lands, not a knob — see {option}`services.hyperhive.agent.logs.queryUrl`
for the consumer this exists for.
'';
};
clientSecretFile = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "${credentialsDir}/${secretCredential}";
description = ''
Path the agent's OIDC client secret is delivered at. Read-only for
the same reason as {option}`services.hyperhive.agent.queue.clientIdFile`.
🩸 A PATH and never a value. The file is `0400` to the agent user and
is read at the moment of a token request; nothing in this tree puts
its contents in an environment variable, where `/proc/<pid>/environ`
would publish them to every process in the container.
'';
};
};
config = lib.mkIf configured {
systemd.services.hive-agent = {
# Bare ids, no paths: this is the terse `LoadCredential=` form that
# inherits a credential the service *manager* received, which is what
# the container manager passed in. `man systemd.exec` also makes that
# form non-fatal when the credential is absent, which is exactly the
# behaviour a hive whose publisher has not run yet needs.
serviceConfig.LoadCredential = [
secretCredential
clientIdCredential
];
};
};
}

14
nix/container-modules/swarm-secret-publisher.nix
View file

@ -1,6 +1,6 @@
# The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its own and its agents'
# credentials, a service or the controller its own. It mints nothing: every
# whoever needs one can read it there: a hive, a service or the controller its
# own. It mints nothing: every
# value is an OIDC client secret authelia minted, and this unit copies it.
#
# Imported inside authelia's container, beside the mint: it reads each secret
@ -61,14 +61,6 @@ in
example = "secret/swarm/services/swarm-grafana/oidc/client";
description = "KV path its reader fetches the value from.";
};
withClientId = lib.mkOption {
type = lib.types.bool;
default = false;
description = ''
Also store the client id beside the value, as `client_id`, for
a reader that has to reconstruct it from the store alone.
'';
};
};
}
);
@ -170,7 +162,7 @@ in
# `value="$(cat "$src")"` would publish it to /proc for anyone in
# the container to read.
bao kv put ${lib.escapeShellArg s.path} \
value=@"$src"${lib.optionalString s.withClientId " client_id=${lib.escapeShellArg s.clientId}"}
value=@"$src"
published=$((published + 1))
else
# Not an error: authelia mints on its FIRST BOOT, so an absent file

1
nix/host-modules/default.nix
View file

@ -47,7 +47,6 @@
./glue-nats-bao-identity.nix
./glue-nats-oidc-client.nix
./glue-nats-wireguard.nix
./glue-queue-agent-credential.nix
./glue-builder-oidc-client.nix
./glue-ci-metrics-otel-oidc-client.nix
./glue-ci-runner-otel-oidc-client.nix

37
nix/host-modules/deploy.nix
View file

@ -53,6 +53,14 @@ let
puts it there. Remove this definition; the file it named is read by
nothing now and can be deleted.
'';
# Shared by the removed options of the hive's agent queue credential reader.
queueAgentCredentialRemoved = ''
A hive no longer fetches a shared queue credential for its agents: each
agent reads its own secret from the swarm's secret store under its own
certificate. Remove this definition; the leaf or directory it named is read
by nothing now and can be deleted.
'';
in
{
imports = [
@ -95,6 +103,35 @@ in
"statusPublish"
"clientSecretFile"
] statusSecretRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"bao"
"queueAgentCommonNamePrefix"
] queueAgentCredentialRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"bao"
"queueAgentClientCertFile"
] queueAgentCredentialRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"bao"
"queueAgentClientKeyFile"
] queueAgentCredentialRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"hive-controller"
"queue"
"agentCredentialDir"
] queueAgentCredentialRemoved)
# Same type, same meaning, new path — so a rename carries it exactly
# and existing configs keep evaluating with one warning naming both

4
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -31,10 +31,6 @@ let
# Reader unit → the gate its own module defines it under.
readersHere = {
# ./glue-queue-agent-credential.nix
swarm-bao-queue-agent =
deployCfg.hive-controller.enable
&& havePair baoDeploy.queueAgentClientCertFile baoDeploy.queueAgentClientKeyFile;
# ./swarm-grafana.nix
swarm-bao-grafana-oidc = deployCfg.grafana.enable;
# ./swarm-otel.nix

26
nix/host-modules/glue-bao-tls.nix
View file

@ -95,15 +95,13 @@ in
# 🩸 The readers below each present a leaf of their own rather than
# `client.pem` above. The leaf is what bao sees, so this pairing is the
# whole of what turns "one principal with the union of their grants" into
# one principal per grant — see ./swarm-bao.nix's `perHiveReaders` block
# for the grants themselves.
# one principal per grant — see ./swarm-bao.nix's `readStanza` block for
# the grants themselves.
#
# Defaulted here rather than in `glue-<consumer>-bao-identity.nix` files:
# those exist where the consumer is a CONTAINER with a
# `deploy.<service>.*` namespace of its own to point at. These are host
# units reading the store, which is the pairing this file already is.
queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem";
queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem";
grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem";
grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem";
otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem";
@ -223,20 +221,10 @@ in
# principal holding the union of their grants, and the union would
# include every agent credential in the swarm.
#
# ⚠️ The first carries THIS hive's name in the subject, unlike the three
# service leaves above. Its grant names one hive's path, because a queue
# credential lives under `swarm/hives/<name>/` and every hive runs a
# reader for its own; ./swarm-bao.nix writes one role per hive in the
# swarm directory to match. The OIDC readers need no such segment — a
# client is registered once per swarm.
#
# Minted whether or not the consumer runs here, for the reason the three
# leaves above give: on a hive that does not run the store this is the
# file an operator copies, and a leaf that only appears where its
# consumer does is one nobody can copy from anywhere.
[ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth
[ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \
${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth
@ -344,14 +332,14 @@ in
${lib.optionalString deployCfg.hive-controller.enable ''
# This hive's gateway-leaf identity, which opens
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its
# subject carries this hive's name, like the queue-agent leaf
# above, because ./swarm-bao.nix writes one role
# per hive. Only where a hive runs; the other hives' are below.
# subject carries this hive's name, because ./swarm-bao.nix writes
# one role per hive. Only where a hive runs; the other hives' are
# below.
[ -s ${pkiDir}/hive-leaf.pem ] || ${signLeaf} ${pkiDir} hive-leaf \
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${clientCn}"} "" clientAuth
''}
# The reader, queue-agent and gateway-leaf leaves of every other hive in
# The reader and gateway-leaf leaves of every other hive in
# the swarm directory: the files an operator copies to that hive's
# host. One directory per hive, with the basenames this hive's leaves
# use, so a hive's set is one directory to copy.
@ -364,8 +352,6 @@ in
install -d -m 0700 ${d}
[ -s ${d}/client.pem ] || ${signLeaf} ${d} client \
${lib.escapeShellArg h} "" clientAuth
[ -s ${d}/queue-agent.pem ] || ${signLeaf} ${d} queue-agent \
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${h}"} "" clientAuth
[ -s ${d}/hive-leaf.pem ] || ${signLeaf} ${d} hive-leaf \
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${h}"} "" clientAuth
''

284
nix/host-modules/glue-queue-agent-credential.nix
View file

@ -1,284 +0,0 @@
# Glue: this hive's agent queue credential comes out of the secret store.
#
# A cert login that fails LOUDLY because every state it fails on is one a
# retry fixes, then a read that degrades QUIETLY because no retry turns "no
# value there" into a value.
#
# ⚠️ Two files, not one, and that is the consumer's shape rather than a
# preference. `swarm-logs` inside the agent reads the secret and the client id
# each from its own file (`<prefix>_OIDC_CLIENT_SECRET_FILE`,
# `<prefix>_OIDC_CLIENT_ID_FILE`), so splitting them here is what lets
# hive-c0re hand both to an agent without parsing anything.
#
# ⚠️ An ABSENT file means this hive's agents do not connect to the queue, and
# that is correct rather than degraded. The publisher runs on the authelia
# host and authelia mints on its first boot, so "nothing at that path yet" is
# the ordinary early state of a swarm. Nothing here writes a local stand-in:
# there is no such thing as a locally valid OIDC client secret, so a placeholder would turn a hive that cannot connect
# into one that is refused, which reaches the agent as a timeout.
#
# 📌 This runs wherever a client identity is configured, NOT only where the
# store is. `deploy.bao.enable` would have been the co-location assumption
# itself; the reader needs a certificate, not a neighbour. On the store's own
# host ./glue-bao-tls.nix supplies one as a `mkDefault`; elsewhere an operator
# places the leaf and names it, and the same unit works. It also needs this
# host to run a hive (`deploy.hive-controller.enable`): the credential is for
# the agents hive-c0re creates here at runtime, and the unit writes under
# hive-c0re's own state directory.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
# What decides whether this unit exists at all. A reader is defined by holding
# a certificate the store accepts, and that is true on the store's own host
# and on a hive three networks away for exactly the same reason.
#
# 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's. Bao
# matches a cert-auth role on the CN, so one leaf shared by four readers
# would be ONE principal holding the union of four grants: read on
# `swarm/agents/*` AND `swarm/hives/<hive>/*`, when this unit reads one
# queue credential and nothing else. Its own leaf carries
# `<queueAgentCommonNamePrefix>-<hive>` and its role grants the single path
# below — still this hive's own, so the narrowing costs no reach.
haveClientIdentity =
baoDeploy.queueAgentClientCertFile != null && baoDeploy.queueAgentClientKeyFile != null;
# Does this host read the store at all — the hive's own leaf, which is the
# one thing a remote-store deployment has to place by hand. Only used to
# decide whether a missing per-principal leaf is a mistake or a deployment
# that has no store: a host holding neither is the supported no-store
# shape, and one holding this pair but not the pair above reads the store
# without a reader identity of its own.
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir;
secretFile = "${credentialDir}/secret";
clientIdFile = "${credentialDir}/client_id";
# Where the credential lives in the store, spelled from the same pieces the
# writer uses. Whoever writes it and whoever reads it must agree, and the
# agreement belongs in one visible place.
#
# ⚠️ The `hives/<name>` segment is not decoration — it is what the reader's
# own grant covers, so a path outside it is a 403 rather than a miss, however
# correct it looks. `swarm-secret-client`'s `queue::agent_client_path` builds
# the same string from the same pieces; this literal is the nix half of that
# one agreement.
#
# `hiveName` has no fallback here for the reason ./glue-bao-tls.nix gives at
# its own use of it: it is asserted set wherever
# `deploy.hive-controller.enable` is.
credentialPath = "secret/swarm/hives/${hyperhiveCfg.hiveName}/queue/agent";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
storeRetry = import ./lib/store-retry.nix { };
in
{
options.services.hyperhive.deploy.hive-controller.queue = {
agentCredentialDir = lib.mkOption {
type = lib.types.path;
default = "/var/lib/hyperhive/queue-agent";
description = ''
Host directory holding this hive's agent queue credential: `secret`
(the OIDC client secret, `0600`) and `client_id` (the client that
secret authenticates, `0644` — it is sent to the token endpoint on
every connection and is not itself a secret).
Both files are written by `swarm-bao-queue-agent.service`, which also
creates this directory. Named here because nothing else creates it:
the parent `/var/lib/hyperhive` belongs to `hive-c0re.service`'s
`StateDirectory=`, which re-applies its own mode on every start, and a
second declaration of *that* path is the two-mechanisms-one-path trap
./hive-gateway/default.nix records — so this directory is the unit's
to make and the parent stays c0re's.
Absent files mean this hive has no queue credential yet, which is what
a swarm looks like before the publisher in authelia's container has run.
'';
};
};
config = lib.mkMerge [
# ⚠️ A SEPARATE arm from the unit below, and that separation is the whole
# mechanism: the unit's arm is gated on `haveClientIdentity`, so an
# assertion written inside it could never be reached in the state it
# exists to report.
#
# Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the
# same refusal, named to this principal's own pair. Where Grafana's gate
# is `deploy.grafana.enable`, this reader's is the hive's own: every hive
# runs one for its own agents, so on a hive, reading the store at all is
# what asks for it. Hence `hiveReaderIdentity` — a host holding no hive
# leaf has no store to read and nothing is missing.
(lib.mkIf (deployCfg.hive-controller.enable && hiveReaderIdentity) {
assertions = [
{
assertion = haveClientIdentity;
message = ''
This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile
is set), so it needs the agent queue credential reader's own client
identity: set both
services.hyperhive.deploy.bao.queueAgentClientCertFile
services.hyperhive.deploy.bao.queueAgentClientKeyFile
swarm-bao-queue-agent.service fetches this hive's agent queue
credential out of the store, and without these it is not rendered
at all — leaving hive-c0re with no queue credential and no unit
that would ever have written one.
Every hive runs this reader for its own agents, so unlike the other
three principals there is no per-service toggle that turns it off:
a hive that reads the store at all is a hive that needs this pair.
⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one
is the hive's, and its grant reads every agent credential in the
swarm and this hive's own secrets; this role reads the one
queue-credential path. Pointing this option at the hive's leaf
would evaluate, deploy and log in — and undo the split.
On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere copy the pair from
/var/lib/swarm-bao-pki/hives/<hive>/ on the store's host and name it
here — see docs/swarm/bao.md.
'';
}
];
})
(lib.mkIf (deployCfg.hive-controller.enable && haveClientIdentity) {
systemd.services.swarm-bao-queue-agent = {
description = "fetch this hive's agent queue credential from the swarm secret store";
# Every one of these names a unit that exists only where the store runs,
# so the ordering is conditional even though the read is not: off-host
# there is nothing local to wait for, and the timeout below is what
# bounds the attempt instead. `Wants=` pki, for
# ./glue-bao-readers-store-order.nix's reason.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
# Ordered before hive-c0re, so no agent container renders ahead of an
# attempt at its credential. `Wants=`, not `Requires=`: a store this
# unit can't reach delays hive-c0re's start by one attempt
# (`TimeoutStartSec` below) rather than failing it — hive-c0re starts
# after that attempt with whatever credential is on disk, while
# ./lib/store-retry.nix keeps retrying in the background.
before = [ "hive-c0re.service" ];
wantedBy = [
"multi-user.target"
"hive-c0re.service"
];
path = [
baoDeploy.package
pkgs.coreutils
];
# ./lib/store-retry.nix.
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
# What actually bounds each attempt at the reads below. Stated here
# rather than left to systemd's default, so the number a boot waits on
# is in the file that waits.
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile;
BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
${atomicWriteSecret}
# `bao`'s own message is the only thing separating a missing value from
# a refused identity from an unreachable host. This unit's degraded
# mode is correct for all three, so it reports which one rather than
# asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Fails LOUDLY, unlike the reads below: the three states a login
# failure covers — store not up, sealed, role not written yet — are all
# things a retry fixes, and `Restart=on-failure` above is what retries.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
# Two reads of one object rather than one `-format=json` parsed with
# `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot
# actually disagree — the client id is derived from this hive's name, so
# a rotation landing between these two calls changes the secret and
# rewrites the same id.
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
# Both or neither: a client that finds one of the two comes up "fine"
# and never connects.
if [ -z "$secret" ] || [ -z "$client_id" ]; then
echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2
exit 0
fi
install -d -m 0755 ${lib.escapeShellArg credentialDir}
atomic_write_secret 0600 "" ${lib.escapeShellArg secretFile} "$secret"
# `0644` on purpose: an OIDC client id is presented to the token
# endpoint on every connection and is public by construction.
atomic_write_secret 0644 "" ${lib.escapeShellArg clientIdFile} "$client_id"
'';
};
})
];
}

3
nix/host-modules/hive-c0re/default.nix
View file

@ -333,8 +333,7 @@ in
# from an unprivileged daemon without widening either.
# The gate is the identity, never `deploy.bao.enable`: a hive that
# reads a store on another machine holds one of these and runs no
# store. ./glue-queue-agent-credential.nix gates its own reader the
# same way.
# store. ../swarm-otel.nix gates its own reader the same way.
lib.optionals baoClientIdentity [
"bao-client.pem:${baoDeploy.clientCertFile}"
"bao-client-key.pem:${baoDeploy.clientKeyFile}"

29
nix/host-modules/hive-c0re/environment.nix
View file

@ -11,9 +11,8 @@ let
cfg = config.services.hyperhive.c0re;
baoDeploy = config.services.hyperhive.deploy.bao;
baoCfg = config.services.hyperhive.swarm.bao;
# The same test ../glue-queue-agent-credential.nix applies, and for the same
# reason: a reader is defined by holding a certificate the store accepts,
# not by sharing a host with the store. Deriving this from
# A reader is defined by holding a certificate the store accepts, not by
# sharing a host with the store. Deriving this from
# `deploy.bao.enable` would be the co-location assumption itself.
haveBaoClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
in
@ -222,31 +221,15 @@ in
# the one the roster gave it.
HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}";
}
// {
# Where ../glue-queue-agent-credential.nix lands the AGENTS' queue
# credential. Read by `hive_c0re::lifecycle::host_config`, which stats the
# two files and forwards them into each container as systemd credentials.
# Never read for its contents here: the secret is `0600` root-owned and
# this daemon is `hive-core`, which is exactly why the transport is a
# credential rather than a bind mount.
HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR = toString config.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
}
//
# The agents' half of the same queue, forwarded by `hive_c0re::meta` into
# every container. A pair, gated together, because an agent that got one of
# them would report a half-configured queue instead of none.
# every container.
#
# ⚠️ By name, like `HIVE_C0RE_NATS_URL` above, and never a loopback
# address: inside a container `127.0.0.1` is the agent itself.
lib.optionalAttrs
(
config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null
&& config.services.hyperhive.swarm.statusPublish.tokenEndpoint != null
)
{
HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl;
HIVE_AGENT_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
}
lib.optionalAttrs (config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null) {
HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl;
}
//
# Where an agent reads the swarm's logs. Gated on the store identity below,
# because `swarm-logs` authenticates with the agent's own queue secret, read

22
nix/host-modules/swarm-authelia-service.nix
View file

@ -352,22 +352,14 @@ in
readOnly = true;
default = "-agent";
description = ''
Suffix appended to a hive's own client id to name the client its
*agent containers* present — agents on hive `alpha` all present
`${config.services.hyperhive.swarm.authelia.hiveClientPrefix}alpha${config.services.hyperhive.swarm.authelia.agentClientSuffix}`.
Read-only for the same reason as `hiveClientPrefix`, and read by the
same consumer with the same failure mode: a split spelling denies
every agent as a timeout.
Suffix the queue's auth-callout responder refuses on a hive's client
id: `${config.services.hyperhive.swarm.authelia.hiveClientPrefix}alpha${config.services.hyperhive.swarm.authelia.agentClientSuffix}`
is denied, never read as hive `alpha-agent`. No client is minted with
it; an agent authenticates to the queue with its own secret.
A suffix on the hive's id rather than a prefix of its own, because
`agent-alpha` reads as *the agent named alpha* — which is precisely
what this identity does not say.
One id per hive rather than per agent, because agents are created at
runtime and a per-agent client would make creating one a config
change plus a reload. The consequence is that this identity says
*which hive* an agent belongs to and never *which agent* — the
broker cannot tell two agents on one hive apart.
Read-only for the same reason as `hiveClientPrefix`. A hive named with
this suffix is refused at evaluation, since the queue would refuse its
own client id.
'';
};

91
nix/host-modules/swarm-authelia.nix
View file

@ -230,8 +230,8 @@ let
# The secret publisher runs in this container, beside the mint whose secrets
# it copies. It needs a store identity, and a reader is defined by holding a
# certificate the store accepts, never by standing next to it — the rule
# ./glue-queue-agent-credential.nix states in full. `deploy.bao.enable` here
# would be the co-location assumption itself.
# ./swarm-bao.nix's `deploy.bao` options block states in full.
# `deploy.bao.enable` here would be the co-location assumption itself.
publisherCfg = deployCfg.swarm-secret-publisher;
publishes =
publisherCfg.enable
@ -300,71 +300,6 @@ let
accessTokenSignedResponseAlg = "RS256";
}) hyperhiveCfg.swarm.hives;
# The one URL an agent's token has to be authorised at beyond the queue:
# the log store's machine query route. Read off `swarm.victorialogs.domain`
# — an unconditional option, declared whether or not this host runs the
# store — rather than restated, because it is also the string `swarm-logs`
# requests as its audience and the string `hive-c0re/environment.nix` hands
# an agent as its endpoint. Three readers, one formula; two spellings
# present as a valid token refused at the store.
logsQueryUrl = "https://${hyperhiveCfg.swarm.victorialogs.domain}/select/logsql/query";
# The identity an agent container presents to the swarm's queue. One per
# HIVE, not per agent, and that is the load-bearing choice rather than a
# shortcut: agents are created at **runtime**, so anything minted per agent
# would make creating one a config change plus an authelia reload. Keyed on
# the hive, this list's length tracks the roster above it — deploy-time, like
# every other entry here.
#
# ⚠️ The cost, stated rather than discovered later: every agent on a hive
# presents the SAME client id, so the broker can tell *hives* apart and not
# *agents*. Deliberate and deferred — closing it requires the client list to
# stop being static, which is the same problem the users-database writer
# already solves for identities.
#
# 🏷️ The id EXTENDS the hive's own (`hive-<name>-agent`) rather than taking a
# prefix of its own (`agent-<name>`), because that reads as *the agent called
# `<name>`* — which is the one thing this identity does not carry. The cost is
# that the two ids are not distinguishable by prefix, so the responder's
# agent rule must be tried BEFORE its hive rule; `policy.rs` says so at the
# match site, and the assertion below covers the case that ordering cannot.
#
# Mirrors `hiveClients` field for field so the two stay comparable. The
# signing algorithm is not load-bearing here — this client's consumers are
# the queue's auth-callout responder, which *introspects* rather than
# verifying offline, and authelia's own authz endpoint — but it matches its
# sibling rather than inventing a second answer to a question nobody asked.
agentClients = lib.mapAttrsToList (name: _: {
id = "${cfg.hiveClientPrefix}${name}${cfg.agentClientSuffix}";
description = "HyperHive agents on hive ${name}";
kind = "machine";
redirectUris = [ ];
# ⚠️ **Two** audiences, where `hiveClients` has one, because an agent
# presents this token at two different kinds of consumer. The queue
# introspects it and reads the client id; the log store's gateway
# location runs `auth_request` against authelia, which authorises a
# bearer token **by the URL being requested** and refuses one whose
# audience does not name it. Measured, not assumed: without the URL
# registered here the token endpoint answers `invalid_target` for it,
# and a token minted with no audience at all is answered by the gateway
# with a bare 401 that names no cause.
audience = [
"${cfg.hiveClientPrefix}${name}${cfg.agentClientSuffix}"
logsQueryUrl
];
# Without it the authz endpoint refuses an otherwise valid token and
# blames the token rather than the missing grant — the same note
# `glue-swarm-otel-oidc-client.nix` carries over the collector's client.
bearerAuthz = true;
# Stated rather than left on authelia's default, for
# `glue-swarm-otel-oidc-client.nix`'s reason: authelia permits only
# basic / JWT methods for a confidential client holding this scope, and
# enforces it in the startup validator. `swarm-queue-client` sends the
# credential as basic auth, so this is also what the minter already does.
tokenEndpointAuthMethod = "client_secret_basic";
accessTokenSignedResponseAlg = "RS256";
}) hyperhiveCfg.swarm.hives;
# `swarm-authelia-bridge`'s own identity — distinct from
# `swarm-controller`'s (`swarm-controller.nix`'s `queueClientId`). A
# resource server introspecting a token proves its OWN identity to the
@ -706,7 +641,7 @@ in
# wherever this module is, so `oidc.clients` is never actually empty
# — see `oidcEnabled`'s comment above.
services.hyperhive.swarm.authelia.oidc.clients =
lib.optionals cfg.oidc.hiveIdentities (hiveClients ++ agentClients)
lib.optionals cfg.oidc.hiveIdentities hiveClients
++ [
bridgeClient
forwarderClient
@ -749,18 +684,11 @@ in
}
{
# A hive whose name ENDS with the agent suffix mints an id the
# queue's responder reads as somebody else's agents:
# queue's responder reads as a hive's agent client, which it refuses:
# `hive-foo-agent` parses as *the agents of hive foo* before it
# parses as *the hive foo-agent*, because the agent rule is the
# more specific one and therefore runs first.
#
# ⚠️ NOT covered by the duplicate-id assertion above, and the gap is
# the interesting half. That one fires only when BOTH `foo` and
# `foo-agent` are on the roster, because only then are two clients
# actually named the same string. With `foo-agent` alone there is no
# duplicate and nothing to see — the hive simply receives its
# agents' grant instead of its own, and a NATS denial arrives as a
# timeout, so the symptom names nothing.
# more specific one and therefore runs first. A NATS denial arrives
# as a timeout, so the symptom names nothing.
#
# Checkable here and nowhere downstream: the responder runs in a
# container with no view of the roster (`policy.rs`'s `hive_name`
@ -774,10 +702,9 @@ in
+ lib.concatMapStringsSep ", " (h: "'${h}'") (
lib.filter (h: lib.hasSuffix cfg.agentClientSuffix h) (lib.attrNames hyperhiveCfg.swarm.hives)
)
+ " — names ending with '${cfg.agentClientSuffix}', the suffix that "
+ "marks a hive's agent containers. Such a hive's own client id is "
+ "indistinguishable from another hive's agent client, and the "
+ "queue resolves it as the agents. Rename the hive.";
+ " — names ending with '${cfg.agentClientSuffix}', the suffix the "
+ "queue refuses as a hive's agent client. Such a hive's own client "
+ "id reads as one, and the queue refuses it. Rename the hive.";
}
# The two obligations `authelia.bearer.authz` carries. Authelia
# enforces both itself — but in its `preStart` validator, so a

130
nix/host-modules/swarm-bao.nix
View file

@ -808,9 +808,9 @@ let
}
'';
# One gateway-leaf issuer per hive in the swarm directory, for the reason
# `perHiveReaders` below is per hive. One name for the pki role, the policy
# and the cert-auth role, as for the readers.
# One gateway-leaf issuer per hive in the swarm directory: one subject for
# every hive could ask for any hive's certificate. One name for the pki role,
# the policy and the cert-auth role, as for the readers below.
hiveLeafIssuers = lib.mapAttrsToList (hiveName: hive: {
name = "${baoDeploy.hiveLeafPkiRolePrefix}-${hiveName}";
cn = "${baoDeploy.hiveLeafCommonNamePrefix}-${hiveName}";
@ -897,48 +897,21 @@ let
# `swarm/agents/*` AND `swarm/hives/<hive>/*`, and nothing under
# `swarm/services/`.
# Principals presenting one certificate are ONE principal to bao, so sharing
# the hive's leaf would give each of them everything that leaf reads: the
# unit that fetches this hive's queue credential could fetch every agent
# credential in the swarm, and a unit that fetches a service's OIDC client
# secret could not fetch it at all, as that path is not the hive's to read.
# the hive's leaf would give each of them everything that leaf reads: a unit
# that fetches a service's OIDC client secret could fetch every agent
# credential in the swarm, and could not fetch its own secret at all, as
# that path is not the hive's to read.
#
# Each gets its own leaf, its own cert-auth role and a policy holding the
# one `secret/data/…` path that principal's script actually names — spelled
# to the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText`
# above gives for doing the same.
#
# ⚠️ The queue-credential reader is PER HIVE and the OIDC readers are not,
# and that asymmetry is the shape of the paths rather than a preference. A
# queue credential lives under `swarm/hives/<name>/`, and every hive runs a
# reader for its own; an OIDC client secret lives under
# `swarm/services/<client-id>/` and a swarm registers each of those exactly
# once. A single role for a per-hive reader would have to name `hives/*` to
# serve every hive — which would let one hive read another's queue
# credential, a reach no hive needs.
readStanza = path: ''
path "${path}" {
capabilities = ["read"]
}
'';
# One reader per hive in the swarm directory. Written from the directory
# rather than from this host's own name because the policy is written where
# the STORE is and the reader runs where its hive is, so this host's name is
# not the reader's.
perHiveReaders =
{ rolePrefix, cnPrefix, ... }@spec:
lib.mapAttrsToList (hiveName: _: {
name = "${rolePrefix}-${hiveName}";
cn = "${cnPrefix}-${hiveName}";
policyText = readStanza (spec.path hiveName);
}) hyperhiveCfg.swarm.hives;
queueAgentReaders = perHiveReaders {
rolePrefix = "swarm-queue-agent";
cnPrefix = baoDeploy.queueAgentCommonNamePrefix;
path = h: "${credentialMountPath}/data/swarm/hives/${h}/queue/agent";
};
# Singletons, so a list of one rather than a second shape: the granting unit
# below takes a list either way and the units stay identical apart from
# the objects they write.
@ -1257,9 +1230,9 @@ let
# One unit per PRINCIPAL, not one unit for all of them: these names are
# operator-facing strings, the same reason `swarm-bao-secret-publisher-policy`
# below gives for not being more script inside the controller's unit. A hive
# whose queue credential never arrives is diagnosed by the state of the unit named
# after it.
# below gives for not being more script inside the controller's unit. A
# reader whose secret never arrives is diagnosed by the state of the unit
# named after it.
readerPolicyUnit =
description: objects:
granterPolicyUnit description (
@ -1589,7 +1562,7 @@ in
# `serverCaFile` live here too, and they are set by whoever authenticates TO
# the store — which includes a host that runs none of it. A reader is defined
# by holding a certificate the store accepts, never by sharing a host with
# one; ./glue-queue-agent-credential.nix gates on exactly that and not on
# one; ./swarm-otel.nix's store reader gates on exactly that and not on
# `enable`.
# Reading this block as store-runner-only is what makes an off-host reader
# look inexpressible when it is already supported.
@ -2102,28 +2075,6 @@ in
'';
};
queueAgentCommonNamePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-queue-agent";
example = "swarm-bao-queue-agent.svc";
description = ''
Prefix of the subject the store's per-hive queue-credential roles
accept. The role for hive `h` accepts `<prefix>-h` and grants read on
exactly `swarm/hives/h/queue/agent` — the client credential that hive's
agent containers authenticate to the swarm queue with.
A **prefix** rather than one subject because every hive runs its own
reader and the path it reads carries that hive's name. One subject for
all of them would need a `hives/*` grant, which would let one hive read
another hive's queue credential.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its
fixed-subject siblings are: cert auth trusts the CA and
`allowed_common_names` is the whole narrowing, so a hive whose own leaf
carried one of these spellings would receive that grant.
'';
};
hiveLeafCommonNamePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-hive-leaf";
@ -2134,9 +2085,8 @@ in
`<servicesPkiMountPath>/issue/<hiveLeafPkiRolePrefix>-h`, which issues
for `h`'s domain and `*.` that domain and nothing else.
Per hive for the reason
{option}`services.hyperhive.deploy.bao.queueAgentCommonNamePrefix` is:
one subject for every hive could ask for any hive's certificate.
Per hive because one subject for every hive could ask for any hive's
certificate.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, like its siblings.
'';
@ -2163,7 +2113,7 @@ in
Its grant is one path,
`swarm/services/<grafana client id>/oidc/client`, and read only. Not
per hive, unlike the two prefixes above: an OIDC client is registered
per hive, unlike the prefix above: an OIDC client is registered
once per swarm, so the path names the service and never a hive.
⚠️ Same collision as its siblings, and the same answer: ./swarm.nix
@ -2206,37 +2156,6 @@ in
'';
};
queueAgentClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/queue-agent.pem";
description = ''
Certificate the unit that fetches this hive's queue client credential
presents to the store. Its subject must be
`<queueAgentCommonNamePrefix>-<this hive's name>`; cert auth matches on
the CN and the role accepts nothing else.
⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
Pointing this at the hive's own leaf would evaluate, deploy and log in —
and hand a unit that reads one credential a certificate that reads every
agent credential in the swarm and this hive's own secrets.
No default here: ./glue-bao-tls.nix points it at the leaf it mints,
wherever this host mints its own PKI. A deployment that reads a store on
another host names it by hand, alongside the leaf itself.
'';
};
queueAgentClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/queue-agent-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile`.
'';
};
grafanaOidcClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
@ -2246,8 +2165,10 @@ in
to the store. Its subject must be
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
Pointing this at the hive's own leaf would evaluate, deploy and log in —
and hand a unit that reads one secret a certificate that reads every
agent credential in the swarm and this hive's own secrets.
'';
};
@ -2271,7 +2192,7 @@ in
{option}`services.hyperhive.deploy.bao.otelOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
{option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile` gives.
'';
};
@ -2295,7 +2216,7 @@ in
{option}`services.hyperhive.deploy.bao.forwarderOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
{option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile` gives.
'';
};
@ -3543,12 +3464,10 @@ in
'';
# 🩸 The principals that each log in with a leaf of their own, not as the
# hive. See the block around `perHiveReaders` for what each grant is and
# why the first is per hive; the units are identical apart from the
# objects they write, so the shape lives in `readerPolicyUnit` and the
# difference is visible here in one line each.
systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders;
# hive. See the block around `readStanza` for what each grant is; the
# units are identical apart from the objects they write, so the shape
# lives in `readerPolicyUnit` and the difference is visible here in one
# line each.
systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders;
systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders;
@ -3693,7 +3612,6 @@ in
"swarm-bao-controller-policy.service"
"swarm-bao-secret-publisher-policy.service"
"swarm-bao-matrix-ctl-policy.service"
"swarm-bao-queue-agent-policy.service"
"swarm-bao-grafana-oidc-policy.service"
"swarm-bao-otel-oidc-policy.service"
"swarm-bao-forwarder-oidc-policy.service"

12
nix/host-modules/swarm-grafana.nix
View file

@ -282,8 +282,8 @@ let
);
# A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule
# ./glue-queue-agent-credential.nix states in full.
# accepts, never by standing next to it — the rule ./swarm-bao.nix's
# `deploy.bao` options block states in full.
#
# Also asserted rather than gating anything, and that is the ruling this
# module's second delivery route was deleted under: the store exists so that
@ -722,10 +722,10 @@ in
# to be wrong, and a gate whose false arm is a Grafana with no login.
# Recorded in docs/swarm/secrets.md.
#
# Shaped after ./glue-queue-agent-credential.nix, because it is a store
# reader and this is what the store's readers do: a cert login that fails
# LOUDLY, since every state it fails on is one a retry fixes, then a read
# that degrades QUIETLY, since no retry turns "no value there" into a value.
# It is a store reader, and this is what the store's readers do: a cert
# login that fails LOUDLY, since every state it fails on is one a retry
# fixes, then a read that degrades QUIETLY, since no retry turns "no value
# there" into a value.
#
# ⚠️ An ABSENT secret is "not published yet", not a fault of this host's.
# The publisher runs in authelia's container and authelia mints on its

2
nix/host-modules/swarm-nats.nix
View file

@ -873,7 +873,7 @@ in
# terminal rows and its turn-state header
# (`hive-agent::swarm_agent_state`), two subject families so a
# subscriber following one agent's terminal is not also handed
# every header. The hive's shared agent client gets nothing.
# every header.
#
# `$$`, not `$`: systemd substitutes `$NAME` in `ExecStart`
# whether or not it is quoted, so a single dollar reaches the

11
nix/host-modules/swarm-otel.nix
View file

@ -232,12 +232,11 @@ let
haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null;
# A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule
# ./glue-queue-agent-credential.nix states in full. Unlike Grafana's identical-
# accepts, never by standing next to it — the rule ./swarm-bao.nix's
# `deploy.bao` options block states in full. Unlike Grafana's identical-
# looking flag, this one does not gate an assertion on its own: the
# store-reading unit below simply does not render without it, the same choice
# ./glue-queue-agent-credential.nix makes for its own reader, because a collector with no client identity is
# `haveCollectorSecret = false` above, and that is already a supported,
# store-reading unit below simply does not render without it, because a
# collector with no client identity is `haveCollectorSecret = false` above, and that is already a supported,
# merely degraded shape rather than a service with no way in at all. What
# IS asserted is narrower and lives in `assertions` below — see
# `hiveReaderIdentity`.
@ -539,7 +538,7 @@ in
# exists so a host holds ONE out-of-band secret — its client certificate —
# and reads everything else with it. Recorded in docs/swarm/secrets.md.
#
# Shaped after ./glue-queue-agent-credential.nix: a cert login that fails
# Shaped after ./swarm-grafana.nix's store reader: a cert login that fails
# LOUDLY, since every state it fails on is one a retry fixes, then a read
# that degrades QUIETLY, since no retry turns "no value there" into a
# value.

39
nix/host-modules/swarm-secret-publisher.nix
View file

@ -12,15 +12,9 @@ let
hyperhiveCfg = config.services.hyperhive;
hiveNames = lib.attrNames hyperhiveCfg.swarm.hives;
# The client id agent containers present, per hive — composed exactly as
# ./swarm-authelia.nix composes it, from the same two read-only options, so a
# rename there cannot leave this spelling behind.
agentClientId =
hive:
"${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}${hyperhiveCfg.swarm.authelia.agentClientSuffix}";
# The hive's own client id, which hive-c0re publishes the hive's status with.
# Composed as ./swarm-authelia.nix composes it, for the reason above.
# Composed exactly as ./swarm-authelia.nix composes it, from the same
# read-only option, so a rename there cannot leave this spelling behind.
hiveClientId = hive: "${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}";
# The swarm's own services, as opposed to its hives. One client for the whole
@ -83,22 +77,14 @@ let
controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client";
secrets =
# `client_id` beside the value: an agent's id is derived per hive and has
# to be reconstructable from the store alone.
map (hive: {
clientId = agentClientId hive;
path = "secret/swarm/hives/${hive}/queue/agent";
withClientId = true;
}) hiveNames
# Each hive's own client, which that hive's hive-c0re reads under its own
# certificate. The nix half of
# `swarm_secret_client::queue::hive_client_path`.
++ map (hive: {
map (hive: {
clientId = hiveClientId hive;
path = "secret/swarm/hives/${hive}/oidc/client";
}) hiveNames
# No `client_id`: a service's client id is the swarm-wide option both
# ends already read.
# A service's client id is the swarm-wide option both ends already read.
++ map (id: {
clientId = id;
path = "secret/swarm/services/${id}/oidc/client";
@ -120,9 +106,8 @@ in
default = false;
description = ''
Publish the OIDC client secrets this host mints into the swarm's
secret store, so a hive that does not run authelia can read its
own and its agents' credentials and a swarm service or the
controller can read its own — from wherever it runs, this host
secret store, so a hive that does not run authelia, a swarm service
or the controller can read its own — from wherever it runs, this host
included. The unit, `swarm-secret-publish`, runs inside authelia's
container, so it exists only where authelia runs.
@ -137,10 +122,9 @@ in
publishing nothing. Whether the wiring is complete is the client
identity's job (see `baoClientCertFile`), not this option's.
Turning it off leaves every hive but this one without its agents'
credential, every hive unable to publish its status, the swarm
controller unable to start, the swarm's Grafana without any login at
all, its collector pushing unauthenticated, the queue's responder
Turning it off leaves every hive unable to publish its status, the
swarm controller unable to start, the swarm's Grafana without any login
at all, its collector pushing unauthenticated, the queue's responder
unable to start, the homeserver unable to start, and the forge's login
source without its secret.
Each secret has exactly one route and this is the producer's
@ -185,9 +169,8 @@ in
internal = true;
default = secrets;
description = ''
Every secret the publisher copies: the minted client's id, the store
path its reader fetches, and whether `client_id` is stored beside the
value. Derived from `swarm.*` alone, so it is the same list on every
Every secret the publisher copies: the minted client's id and the store
path its reader fetches. Derived from `swarm.*` alone, so it is the same list on every
host; ./swarm-authelia.nix hands it to the unit in its container.
'';
};

20
nix/host-modules/swarm.nix
View file

@ -64,17 +64,14 @@ let
deployCfg.bao.notifyRouterCommonName
deployCfg.bao.forgePublisherCommonName
]
# The per-hive subjects, spelled out per hive rather than as the prefix.
# The per-hive subject, spelled out per hive rather than as the prefix.
# The prefix alone would reserve the wrong string: the role for hive
# `h` accepts `<prefix>-h`, so the name a hive must not BE is that composed
# spelling, and a hive named it would present a leaf that role accepts.
#
# Every spelling is reserved for every declared hive, including the one the
# CN belongs to — a hive cannot be named after its own reader either.
++ lib.concatMap (hive: [
"${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}"
"${deployCfg.bao.hiveLeafCommonNamePrefix}-${hive}"
]) hiveNames;
# CN belongs to — a hive cannot be named after its own leaf issuer either.
++ map (hive: "${deployCfg.bao.hiveLeafCommonNamePrefix}-${hive}") hiveNames;
# Public hostnames of the swarm's own services, in declaration order.
# `serviceDomains` below is this set sorted + deduplicated.
@ -477,9 +474,8 @@ in
# its own network namespace and resolver, so this is its own option, even
# though by name it defaults to the same URL as `statusPublish.natsUrl`.
#
# Only the address lives here. The credential does not: it is published per
# hive and read out of the store by ./glue-queue-agent-credential.nix, which
# owns `queue.agentCredentialDir` in the same namespace.
# Only the address lives here. Each agent reads its own credential from the
# store (`nix/agent-modules/queue-identity.nix`).
options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
@ -496,10 +492,8 @@ in
agent's network namespace `127.0.0.1` is the agent, not this host,
and the queue's certificate names no address anyway.
Null means this hive's agents have not been given the queue's address.
Together with
{option}`services.hyperhive.swarm.statusPublish.tokenEndpoint` it is
what decides whether the harness is handed queue coordinates at all.
Null means this hive's agents have not been given the queue's address,
and the harness is handed none.
'';
};

30
nix/module-eval/agent-queue-bao.nix
View file

@ -24,16 +24,13 @@ let
agentHarness
;
# The agent side of the swarm queue. Both coordinates set is the only state
# in which the harness unit declares a credential at all, so the pair and
# the empty fixture beside it are the two arms worth having.
# An `agent.nix` still setting the queue coordinates the agent module no
# longer declares.
agentQueue = agent {
queue.natsUrl = "nats://10.42.0.1:4222";
queue.tokenEndpoint = "https://auth.t.local/api/oidc/token";
};
agentNoQueue = shared.agentBare;
# The agent side of the swarm secret store. The address is the whole switch —
# it is both what generates the login check and what that check points at —
# so it and the empty fixture beside it are the two arms worth having.
@ -106,21 +103,24 @@ let
ok = swarmMetricsIn agentLogsNoBao == null && swarmMetricsIn agentLogs != null;
}
{
# Both ids or neither: the secret authenticates nobody without the id it
# belongs to, and `swarm-logs` reads both.
name = "an agent with queue coordinates imports both halves of its credential";
# The hive forwards no shared queue credential, so the harness unit
# names none, whatever the agent's config says.
name = "an agent imports no hive-shared queue credential";
ok =
let
c = (agentHarness agentQueue).serviceConfig.LoadCredential;
credentials = m: (agentHarness m).serviceConfig.LoadCredential or [ ];
in
builtins.elem "hive-queue-agent-secret" c && builtins.elem "hive-queue-agent-client-id" c;
!lib.any (lib.hasPrefix "hive-queue-agent-") (credentials agentQueue ++ credentials agentBao)
# Control: the lookup finds the store identity's credential.
&& lib.elem "hive-agent-bao-server-ca" (credentials agentBao);
}
{
# An agent built before its hive was handed the queue's address. It
# must declare nothing rather than name a credential that never
# arrives.
name = "an agent with no queue coordinates declares no credential";
ok = !((agentHarness agentNoQueue).serviceConfig ? LoadCredential);
# Both options are refused, naming where the address
# comes from now, rather than dropped silently.
name = "an agent setting the removed queue options is refused";
ok =
lib.count (a: !a.assertion && lib.hasInfix "HIVE_AGENT_NATS_URL" a.message) agentQueue.assertions
== 2;
}
{
# The harness exits when it cannot reach the queue. It has to come back

60
nix/module-eval/bao-grants.nix
View file

@ -106,14 +106,11 @@ let
deploy.matrix.enable = true;
};
# The store with none of the readers beside it. Grafana and the collector are
# simply off; the queue reader renders on any host holding
# its leaf, which ./glue-bao-tls.nix mints here, so that leaf is taken away.
# The store with none of the readers beside it: Grafana and the collector are
# off.
baoGrantNoReaders = hiveWithController {
deploy.bao.enable = true;
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
deploy.bao.queueAgentClientCertFile = lib.mkForce null;
deploy.bao.queueAgentClientKeyFile = lib.mkForce null;
};
# The store with the forwarder's own pair taken away. The forwarder renders
@ -131,8 +128,6 @@ let
deploy.swarm-otel.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.queueAgentClientCertFile = "/etc/pki/bao-queue-agent.pem";
deploy.bao.queueAgentClientKeyFile = "/etc/pki/bao-queue-agent-key.pem";
deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem";
deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem";
deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem";
@ -142,7 +137,6 @@ let
# The readers ./glue-bao-readers-policy-order.nix orders after their policy
# units.
policyReaders = [
"swarm-bao-queue-agent"
"swarm-bao-grafana-oidc"
"swarm-bao-otel-oidc"
];
@ -175,7 +169,6 @@ let
"swarm-bao-controller-policy"
"swarm-bao-secret-publisher-policy"
"swarm-bao-matrix-ctl-policy"
"swarm-bao-queue-agent-policy"
"swarm-bao-grafana-oidc-policy"
"swarm-bao-otel-oidc-policy"
"swarm-bao-forwarder-oidc-policy"
@ -599,19 +592,18 @@ let
# arms pin what each principal must NOT reach, so a later widening fails
# here instead of being noticed in a store.
{
name = "the queue-credential reader's grant is one hive's queue credential and nothing else";
# A hive holds no shared queue credential for its agents: no unit writes
# a role or policy for one, none reads one, and no granting unit names
# its path. The control: the same scan finds a reader's grant.
name = "no unit grants, reads or writes a hive-shared agent queue credential";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-queue-agent-policy.script;
scripts = lib.mapAttrsToList (_: u: u.script or "") baoGrantWithConsumers.systemd.services;
in
lib.hasInfix "path \"secret/data/swarm/hives/h1/queue/agent\" {" s
&& lib.hasInfix "capabilities = [\"read\"]" s
&& !(lib.hasInfix "secret/data/swarm/agents" s)
&& !(lib.hasInfix "secret/data/swarm/services" s)
&& !(lib.hasInfix "secret/data/swarm/hives/h1/*" s)
&& !(lib.hasInfix "secret/data/swarm/hives/h1/matrix" s)
&& !(lib.hasInfix "secret/data/swarm/hives/*" s)
&& !(lib.hasInfix "sys/policies/acl" s);
!(baoGrantWithConsumers.systemd.services ? swarm-bao-queue-agent-policy)
&& !(baoGrantWithConsumers.systemd.services ? swarm-bao-queue-agent)
&& !(lib.any (lib.hasInfix "/queue/agent") scripts)
&& lib.any (lib.hasInfix "/oidc/client") scripts;
}
{
# ⚠️ The client id is the path segment, so the negative arm that matters
@ -856,11 +848,7 @@ let
# through a token that carries it, and a token is minted by a cert-auth
# role matching a CN. Distinct subjects is the whole mechanism — one
# subject for every reader is one principal however the policies read.
#
# The per-hive subject carries the hive name because its path does; the
# service subjects do not, because an OIDC client is registered once
# per swarm. Pinned so neither shape is tidied into the other.
name = "each of the four readers logs in under a subject of its own";
name = "each of the three readers logs in under a subject of its own";
ok =
let
subjectOf =
@ -875,8 +863,7 @@ let
# the reason the three service principals above state.
&& !(lib.hasInfix "auth/cert/certs/hive-" s);
in
subjectOf "swarm-bao-queue-agent-policy" "swarm-queue-agent-h1" "swarm-bao-queue-agent-h1"
&& subjectOf "swarm-bao-grafana-oidc-policy" "swarm-grafana-oidc" "swarm-bao-grafana-oidc"
subjectOf "swarm-bao-grafana-oidc-policy" "swarm-grafana-oidc" "swarm-bao-grafana-oidc"
&& subjectOf "swarm-bao-otel-oidc-policy" "swarm-otel-oidc" "swarm-bao-otel-oidc"
&& subjectOf "swarm-bao-forwarder-oidc-policy" "swarm-forwarder-oidc" "swarm-bao-forwarder-oidc";
}
@ -889,7 +876,7 @@ let
#
# Each pair is asserted whole: a certificate with no key authenticates
# nothing, so a half-set pair is a reader that does not render.
name = "each of the four readers presents its own leaf, never the hive's";
name = "each of the three readers presents its own leaf, never the hive's";
ok =
let
b = baoGrantWithConsumers.services.hyperhive.deploy.bao;
@ -898,8 +885,6 @@ let
b.clientKeyFile
];
own = [
b.queueAgentClientCertFile
b.queueAgentClientKeyFile
b.grafanaOidcClientCertFile
b.grafanaOidcClientKeyFile
b.otelOidcClientCertFile
@ -915,7 +900,6 @@ let
lib.all (p: p != null) own
&& !(lib.any (p: lib.elem p hiveLeaf) own)
&& lib.length (lib.unique own) == lib.length own
&& presents "swarm-bao-queue-agent" b.queueAgentClientCertFile b.queueAgentClientKeyFile
&& presents "swarm-bao-grafana-oidc" b.grafanaOidcClientCertFile b.grafanaOidcClientKeyFile
&& presents "swarm-bao-otel-oidc" b.otelOidcClientCertFile b.otelOidcClientKeyFile
&& presents "swarm-bao-forwarder-oidc" b.forwarderOidcClientCertFile b.forwarderOidcClientKeyFile;
@ -923,9 +907,7 @@ let
{
# The minting side of the same claim. A role matching a subject nothing
# signs is a reader that cannot log in, so the leaves and the roles have
# to be asserted against each other — and the per-hive leaf carries
# THIS host's hive name, which is what makes one hive's leaf useless
# against another hive's role.
# to be asserted against each other.
name = "the PKI unit signs a leaf per reader, each under that reader's own subject";
ok =
let
@ -936,8 +918,6 @@ let
# literal spanning both would pin this file's line wrapping rather than
# the pairing it means to.
lib.all (lib.flip lib.hasInfix s) [
"/queue-agent.pem ]"
"swarm-bao-queue-agent-h1 \"\" clientAuth"
"/grafana-oidc.pem ]"
"swarm-bao-grafana-oidc \"\" clientAuth"
"/otel-oidc.pem ]"
@ -965,11 +945,10 @@ let
# client certificate and the host is the side that has one, so a unit
# rendered inside the store's container would have neither an identity
# nor a route. Plus the ordering that makes the mounts exist first.
name = "the four readers' granting units are ordered after the mounts and rendered on the host";
name = "the three readers' granting units are ordered after the mounts and rendered on the host";
ok =
let
units = [
"swarm-bao-queue-agent-policy"
"swarm-bao-grafana-oidc-policy"
"swarm-bao-otel-oidc-policy"
"swarm-bao-forwarder-oidc-policy"
@ -990,7 +969,7 @@ let
# The forwarder is listed apart from `policyReaders`: it renders wherever
# the store does, so it is never absent on a store host and never present
# on a remote one, and the two cases below would fail on it for that.
name = "each of the four readers is ordered after the unit writing its role";
name = "each of the three readers is ordered after the unit writing its role";
ok =
let
s = baoGrantWithConsumers.systemd.services;
@ -1483,9 +1462,9 @@ let
}
{
# The other hives' side: every hive in the directory but this one gets
# the same three leaves, under its own name, in a directory of its own
# the same two leaves, under its own name, in a directory of its own
# that the operator copies to its host.
name = "the PKI unit signs every other hive's reader, queue-agent and gateway-leaf identities";
name = "the PKI unit signs every other hive's reader and gateway-leaf identities";
ok =
let
s = baoGrantTwoHives.systemd.services.swarm-bao-pki.script;
@ -1494,7 +1473,6 @@ let
file: cn: lib.hasInfix "/hives/h2/${file}.pem ]" s && lib.hasInfix " ${cn} \"\" clientAuth" s;
in
signs "client" "h2"
&& signs "queue-agent" "${b.queueAgentCommonNamePrefix}-h2"
&& signs "hive-leaf" "${b.hiveLeafCommonNamePrefix}-h2"
# This hive's own leaves stay at the top of the directory.
&& !(lib.hasInfix "/hives/h1/" s);

184
nix/module-eval/bao-matrix-reader.nix
View file

@ -31,18 +31,10 @@ let
# nothing here mints a leaf and the operator names one placed by hand. The
# deployment this pairing exists to serve, and the one that was previously
# inexpressible — the gate asked whether the store was a neighbour.
#
# 🩸 One leaf PER READER, and this fixture is where that cost is visible: the
# queue-credential unit identifies itself to the store separately from the
# hive, so an operator placing leaves by hand places one for it as well as
# the hive's own. Naming only `clientCertFile` here would leave the unit
# unrendered — which is what the arms below would then be asserting about.
baoRemoteReader = hiveWithController {
deploy.matrix.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.queueAgentClientCertFile = "/etc/pki/bao-queue-agent.pem";
deploy.bao.queueAgentClientKeyFile = "/etc/pki/bao-queue-agent-key.pem";
};
# A homeserver on a hive with NO store identity at all — neither a local
@ -51,24 +43,15 @@ let
# fixture set its own, as ./lib.nix asks.
inherit (shared) matrixNoBaoIdentity;
# 🩸 The hand-configured remote reader that named the hive's leaf and stopped.
# `baoRemoteReader` above is the same deployment done right; this one holds
# the hive's own leaf, so it demonstrably reads the store, and is missing the
# queue reader's pair. Without the refusal it would render no queue reader
# and say nothing — which is what the refusal arms below are about.
# 🩸 The hand-configured remote reader that named the hive's leaf and stopped:
# it holds the hive's own leaf, so it demonstrably reads the store, and is
# missing matrix-ctl's pair.
baoRemoteReaderMissingLeaves = hiveWithController {
deploy.matrix.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# The same omission on a hive that does NOT run a homeserver: the queue
# refusal checks only the hive's own toggle.
remoteReaderNoMatrixMissingLeaves = hiveWithController {
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# The store's host with the hive's own reader pointed somewhere else.
matrixOnStoreHostReaderMoved = hiveWithController {
deploy.bao.enable = true;
@ -77,122 +60,15 @@ let
deploy.bao.clientKeyFile = "/etc/pki/h1-key.pem";
};
# Did a module refuse this host, and over which option. An assertion is a
# config VALUE until something forces it — `.config` never throws — so a
# fixture in a state the module refuses stays evaluable and the refusal reads
# back as data. Matched on the option name the message names rather than on
# its prose, because the option name is the part an operator has to act on
# and a message that stopped naming it would be the actual defect. Same shape
# as ./grafana.nix's `grafanaRefusedFor`, which reads `swarm-grafana.nix`'s
# own refusal.
refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions;
cases = [
{
# A login failure is the store being unreachable, sealed, or not yet
# holding this host's role — all of which a retry fixes. A read that
# answers "nothing there" is not, so only the first is allowed to fail
# the unit.
name = "the queue credential reader retries a failed login and still degrades on an empty read";
ok =
let
u = baoWithMatrix.systemd.services.swarm-bao-queue-agent;
# Everything between the login's failure branch and the read's, which
# is where the exit that decides "retry or give up" lives.
afterLogin = lib.last (lib.splitString "bao login" u.script);
loginBranch = lib.head (lib.splitString "bao kv get" afterLogin);
in
u.serviceConfig.Restart or null == "on-failure"
&& u.startLimitBurst or 0 > 0
# The window has to outlast every attempt, or the burst is unreachable.
&& u.startLimitIntervalSec or 0 > (u.serviceConfig.RestartSec or 0) * (u.startLimitBurst or 0)
&& lib.hasInfix "exit 1" loginBranch
&& lib.hasInfix "exit 0" (lib.last (lib.splitString "bao kv get" u.script));
}
{
# The reader's own grant covers `swarm/hives/<this hive>/*`; a path
# outside it answers 403, not "no such key". This is the path
# `swarm_secret_client::queue::agent_client_path` builds from the same
# pieces. The negative arm is the rename this one is exposed to: a
# credential named for the queue rather than for the hive that presents
# it reads as correct and is refused on every boot.
name = "the agent queue credential path sits inside the prefix the reader is granted";
ok =
let
s = baoRemoteReader.systemd.services.swarm-bao-queue-agent.script;
in
lib.hasInfix "secret/swarm/hives/h1/queue/agent" s && !(lib.hasInfix "secret/swarm/queue/" s);
}
{
# The unit's output is the option's value, not a literal that agrees with
# it today: an operator moving the directory has to move both files. The
# prefix is asserted too because `hasInfix ""` is true — an option
# renamed out from under this arm would otherwise read empty and pass.
name = "the queue credential reader writes both files under the directory its option names";
ok =
let
m = baoRemoteReader;
dir = toString m.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
s = m.systemd.services.swarm-bao-queue-agent.script;
in
lib.hasPrefix "/var/lib/" dir
&& lib.hasInfix "${dir}/secret" s
&& lib.hasInfix "${dir}/client_id" s;
}
{
# No agent container may render before this unit has had its attempts,
# and the edge that guarantees it must delay hive-c0re rather than sink
# it: an unreachable store is this unit's `Restart=on-failure` window,
# not a reason for the daemon that renders every agent to fail its own
# start.
name = "the queue credential reader orders before hive-c0re and is wanted, not required, by it";
ok =
let
u = baoRemoteReader.systemd.services.swarm-bao-queue-agent;
in
builtins.elem "hive-c0re.service" (u.before or [ ])
&& builtins.elem "hive-c0re.service" (u.wantedBy or [ ])
&& !(builtins.elem "hive-c0re.service" (u.requiredBy or [ ]))
&& !(builtins.elem "hive-c0re.service" (u.requires or [ ]));
}
{
# Nothing local mints certificates off-host, so this orders against
# nothing there.
name = "an off-host reader names no unit the store's host would have provided";
# Membership first, then the value: indexing a missing unit throws, and a
# table that reports which property broke must not be the thing that dies.
ok =
let
s = baoRemoteReader.systemd.services;
u = s.swarm-bao-queue-agent;
in
s ? swarm-bao-queue-agent && !(lib.elem "swarm-bao-pki.service" (u.after ++ u.wants ++ u.requires));
}
{
# Presence control for the case above: the list is conditional, not gone.
# Wanted, never required: a refused pki start fails a requirer's job with
# `dependency`, which its `Restart=on-failure` never retries.
name = "a co-located reader still orders after and wants the local pki unit, and does not require it";
ok =
let
s = baoWithMatrix.systemd.services;
u = s.swarm-bao-queue-agent;
in
s ? swarm-bao-queue-agent
&& lib.elem "swarm-bao-pki.service" u.after
&& lib.elem "swarm-bao-pki.service" u.wants
&& u.requires == [ ];
}
{
# Nothing asserted this script before, which is how it kept a branch that
# named three states and threw away the only thing telling them apart. A
# missing value, a refused identity and an unreachable host all end in the
# same degraded mode here, correctly — what must survive is which one.
name = "the queue credential reader carries the store's own diagnostic into the journal";
ok =
let
s = baoWithMatrix.systemd.services.swarm-bao-queue-agent.script;
in
!(lib.hasInfix "2>/dev/null" s) && lib.hasInfix ''cat "''$err"'' s;
# A hive holds no shared queue credential for its agents, wherever the
# store runs. The control: hive-c0re renders on both hosts.
name = "no hive renders a reader for a hive-shared agent queue credential";
ok = lib.all (m: !(m.systemd.services ? swarm-bao-queue-agent) && m.systemd.services ? hive-c0re) [
baoWithMatrix
baoRemoteReader
];
}
{
# hive-c0re runs as hive-core and the client key is `0600` root-owned
@ -404,46 +280,6 @@ let
&& !(c.systemd.services ? swarm-matrix-appservice-publish)
&& !(lib.any (lib.hasPrefix "swarm-appservice.yaml") c.systemd.services.tuwunel.serviceConfig.LoadCredential);
}
{
# 🩸 The arm the whole refusal exists for. The reader is gated on its own
# leaf, so the way this regresses is the build going green and the unit
# not rendering — which no presence check on a rendered unit can see,
# because the unit that is missing is the evidence. Read as a refusal
# naming each option, so an operator acts on the message without opening
# the nix.
name = "a remote reader missing the queue reader's leaf is refused, naming both options";
ok =
refusedOver baoRemoteReaderMissingLeaves "queueAgentClientCertFile"
&& refusedOver baoRemoteReaderMissingLeaves "queueAgentClientKeyFile";
}
{
name = "the queue refusal needs no homeserver";
ok = refusedOver remoteReaderNoMatrixMissingLeaves "queueAgentClientCertFile";
}
{
# ⚠️ The arm that keeps the refusal from being worse than the silence it
# replaced. A hive with NO store identity is the supported no-store
# deployment and also the state an operator passes through bringing a hive
# up — neither may fail to evaluate. Asserted as "no refusal names either
# option", not as "this one fixture is fine", because the way this breaks
# is a gate widened to the principal's leaf alone.
name = "a hive with no store identity at all is refused over none of the per-principal leaves";
ok = lib.all (option: !(refusedOver matrixNoBaoIdentity option)) [
"queueAgentClientCertFile"
"queueAgentClientKeyFile"
];
}
{
# The other half of the same guard, and the one an operator meets far more
# often: on the store's own host ./host-modules/glue-bao-tls.nix mkDefaults
# every pair, so there is nothing to name and nothing to refuse. Paired with
# the fully-named remote reader, which is the same deployment done by hand.
name = "neither a store host nor a correctly-named remote reader is refused";
ok = lib.all (m: !(refusedOver m "queueAgentClientCertFile")) [
baoWithMatrix
baoRemoteReader
];
}
];
in
runGroup "bao-matrix-reader" cases

42
nix/module-eval/core-toggle.nix
View file

@ -216,13 +216,17 @@ let
&& env shared.controllerAlone == { };
}
{
# Where the reader puts the files and where the daemon looks for them is
# one agreement spanning two modules. Asserted against the option rather
# than the literal so moving the directory moves both ends.
name = "hive-c0re is told where the agents' queue credential lands";
# An agent connects to the queue with its own secret, so the hive hands
# its agents no shared credential and no token endpoint to mint one at.
# The control: the same environment carries the agents' queue address.
name = "hive-c0re forwards no shared agent queue credential";
ok =
allLocal.systemd.services.hive-c0re.environment.HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR
== toString allLocal.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
let
e = allLocal.systemd.services.hive-c0re.environment;
in
!(e ? HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR)
&& !(e ? HIVE_AGENT_OIDC_TOKEN_ENDPOINT)
&& e ? HIVE_AGENT_NATS_URL;
}
{
# Inside an agent container loopback is the agent itself, so the agents'
@ -256,28 +260,10 @@ let
&& lib.elem "/hive-collector.hive/${bridge}" allLocal.services.dnsmasq.settings.address;
}
{
# The agents mint against the swarm's IdP, the same endpoint the hive's
# own client uses — a hive-local guess would produce a token the queue
# would not accept.
name = "the agents' token endpoint is the swarm IdP's";
ok =
let
e = allLocal.systemd.services.hive-c0re.environment;
in
lib.hasSuffix "/api/oidc/token" e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT
&& e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT == e.HIVE_C0RE_OIDC_TOKEN_ENDPOINT;
}
{
# The absence arm, and what makes the two above able to fail: a hive
# with no queue address for its agents must forward neither coordinate,
# because half a pair reaches the harness as a partial configuration
# rather than as none.
name = "a hive with no agent queue address forwards no agent queue coordinates";
ok =
let
e = noAgentQueue.systemd.services.hive-c0re.environment;
in
!(e ? HIVE_AGENT_NATS_URL) && !(e ? HIVE_AGENT_OIDC_TOKEN_ENDPOINT);
# The absence arm, and what makes the agents' queue address cases above able
# to fail.
name = "a hive with no agent queue address forwards none";
ok = !(noAgentQueue.systemd.services.hive-c0re.environment ? HIVE_AGENT_NATS_URL);
}
{
# Control: `bare` sets none of them, and its serve.json carries the

7
nix/module-eval/matrix-core.nix
View file

@ -37,13 +37,6 @@ let
# homeserver across the network.
matrixElsewhere = shared.bareWithController;
cases = [
{
# Absence arm, and what makes the one above able to fail: with no leaf
# this unit would fail a TLS handshake on every boot, so it must not
# exist at all rather than retry its way through the start limit.
name = "a hive with no store identity renders no queue credential reader";
ok = !(matrixNoBaoIdentity.systemd.services ? swarm-bao-queue-agent);
}
{
# Two limits governed a matrix upload and nothing kept them in agreement,
# so raising the documented one past the gateway's hardcoded cap changed

29
nix/module-eval/name-guards.nix
View file

@ -43,14 +43,12 @@ let
# leaves, not a whole prefix, so a hive that inherited a dead entry there
# would silently gain the ability to overwrite the swarm's matrix credential.
#
# 🩸 The two per-hive subjects have a different shape: those roles are
# written PER HIVE, so the subject a hive must not be is
# `<prefix>-<some hive's name>` rather than the prefix itself. Reserving only
# the prefix would leave the composed spelling free, and a hive taking it
# would present a leaf the other hive's role accepts — a hive reading another
# hive's queue credential, the exact widening the split exists to avoid. The
# collision is with the OTHER hive's role, hence `other` beside `qr-other`
# and `hl-other`.
# 🩸 The per-hive subject has a different shape: that role is written PER
# HIVE, so the subject a hive must not be is `<prefix>-<some hive's name>`
# rather than the prefix itself. Reserving only the prefix would leave the
# composed spelling free, and a hive taking it would present a leaf the other
# hive's role accepts. The collision is with the OTHER hive's role, hence
# `other` beside `hl-other`.
hiveNamedAfterEverySubject = hive {
deploy.swarm-otel.enable = false;
deploy.bao = {
@ -62,7 +60,6 @@ let
forwarderOidcCommonName = "fwctl";
forgePublisherCommonName = "fpctl";
granterCommonName = "grctl";
queueAgentCommonNamePrefix = "qr";
hiveLeafCommonNamePrefix = "hl";
};
swarm.hives = {
@ -75,7 +72,6 @@ let
fpctl.domain = "fp.t.local";
grctl.domain = "gr.t.local";
other.domain = "other.t.local";
qr-other.domain = "q.t.local";
hl-other.domain = "hl.t.local";
};
};
@ -216,16 +212,9 @@ let
{
# 🩸 The per-hive half, and the one a prefix-only reservation would miss:
# the role is `<prefix>-<hive>`, so the reserved string has to be composed
# against every declared hive. Here hive `qr-other` collides with the role
# written for hive `other` — a leaf that reads a credential belonging to a
# hive that is not it.
name = "a hive named after another hive's per-hive reader subject is refused";
ok = equalityGuardRefuses hiveNamedAfterEverySubject "qr-other";
}
{
# A hive taking this spelling would present a leaf hive `other`'s
# gateway-leaf role accepts, and could ask the store for that hive's
# certificate. The control is the legal roster below.
# against every declared hive. A hive taking this spelling would present
# a leaf hive `other`'s gateway-leaf role accepts, and could ask the store
# for that hive's certificate. The control is the legal roster below.
name = "a hive named after another hive's gateway-leaf subject is refused";
ok = equalityGuardRefuses hiveNamedAfterEverySubject "hl-other";
}

5
nix/module-eval/nats-tls.nix
View file

@ -77,13 +77,10 @@ let
# The same hive given its own store leaf by hand, which is what turns
# publishing on away from the IdP's host: hive-c0re reads the status secret
# from the store under it. The queue-credential reader's pair comes with it,
# because a hive that reads the store is refused at eval without one.
# from the store under it.
hiveLeaf = {
deploy.bao.clientCertFile = "/etc/pki/h1.pem";
deploy.bao.clientKeyFile = "/etc/pki/h1-key.pem";
deploy.bao.queueAgentClientCertFile = "/etc/pki/h1-queue-agent.pem";
deploy.bao.queueAgentClientKeyFile = "/etc/pki/h1-queue-agent-key.pem";
};
remotePublishing = hiveWithController (lib.recursiveUpdate remoteSecrets hiveLeaf);

20
nix/module-eval/secret-publisher.nix
View file

@ -80,7 +80,6 @@ let
# Each hive's hive-c0re reads this exact path
# (`swarm_secret_client::queue::hive_client_path`, pinned by that crate's
# own test), for every hive in the roster, not only the one this host is.
# The agents' client beside it stays a separate path.
name = "the publisher writes every hive's own client under that hive's prefix";
ok =
let
@ -91,8 +90,7 @@ let
[
"h1"
"h2"
]
&& lib.hasInfix "secret/swarm/hives/h1/queue/agent" s;
];
}
{
# The controller reads this exact path
@ -333,17 +331,17 @@ let
lib.hasInfix "value=@" code && !(lib.hasInfix "$(cat" code);
}
{
# Two ends of a wire nothing at eval time carries end to end: this is the
# path `swarm_secret_client::queue` resolves for the reader. Both hives
# are asserted, so a publisher that knew one name rather than the roster
# fails here rather than on the second hive ever added to a swarm.
name = "the publisher writes every hive in the roster to that hive's own queue path";
# No hive holds a shared queue credential for its agents, so authelia
# mints no agent client and the publisher writes no such path. The
# control: the same script writes each hive's own client.
name = "the publisher writes no hive-shared agent queue credential";
ok =
let
s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script;
in
lib.hasInfix "secret/swarm/hives/h1/queue/agent" s
&& lib.hasInfix "secret/swarm/hives/h2/queue/agent" s;
!(lib.hasInfix "/queue/agent" s)
&& !(lib.hasInfix "-agent.secret" s)
&& lib.hasInfix "secret/swarm/hives/h1/oidc/client" s;
}
{
# A property of the SET, not of one unit: both of these authenticate by
@ -373,7 +371,7 @@ let
&& !(lib.hasInfix "bao kv" (lib.head (lib.splitString "bao login" c)));
in
holdsTokenFirst (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script
&& holdsTokenFirst baoWithMatrix.systemd.services.swarm-bao-queue-agent.script
&& holdsTokenFirst baoWithMatrix.systemd.services.swarm-bao-matrix-oidc.script
# Controls, so a clean verdict above means something. In order: a bare
# read is refused, a read placed before the login is refused, and a
# login that exists only in a comment is refused — that last one is the