vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-controller: read the queue client secret from the store, drop the file
Some checks were skipped
public bin cache / build + push to preem:grid (push) Has been skipped
Details

Browse source 
The controller's OIDC client secret (client `swarm-controller`, used for
the queue connection, the auth-bridge bearer and the OTLP push) came from
an operator-placed file, `deploy.swarm-controller.queue.clientSecretFile`,
handed in by `LoadCredential=`.

Now `swarm-secret-publish`, which already copies authelia's minted OIDC
secrets into the store, also publishes this one, to
`swarm/controller/swarm-controller/oidc/client`. That path sits under
`controller/`, which no hive's policy reads. The controller reads it once
at start with its existing store certificate and holds it in memory, as
`swarm_queue_client::ClientSecret::Value`. If the store is down, it
retries for about a minute and then fails the start, so `Restart=` tries
again.

Policy delta: the controller gets `read` on that leaf, and the publisher
gets `create`/`update` on that leaf.

Removed: the `queue.clientSecretFile` option (both spellings, now removed
options with a message), its singleHostSwarm default, the credential and
placeholder, and the path watcher plus its restart oneshot. A controller
without a store identity is now an eval error, because it has no other
way to get the secret.
This commit is contained in:
atlas 2026-09-28 19:00:31 +02:00
commit e94406cdb9
22 changed files with 595 additions and 247 deletions
Download patch file Download diff file Expand all files Collapse all files

10
nix/host-modules/deploy.nix
View file

@ -77,9 +77,15 @@ in
[ "services" "hyperhive" "swarm" "controller" "authBridgeUrl" ]
[ "services" "hyperhive" "deploy" "swarm-controller" "authBridgeUrl" ]
)
(lib.mkRenamedOptionModule
(lib.mkRemovedOptionModule
[ "services" "hyperhive" "swarm" "controller" "queue" "clientSecretFile" ]
[ "services" "hyperhive" "deploy" "swarm-controller" "queue" "clientSecretFile" ]
''
The controller reads its queue client secret from the swarm's secret
store, under its own certificate (deploy.swarm-controller.baoClientCertFile),
and swarm-secret-publish on the host that runs authelia puts it there.
Remove this definition; the file it named is read by nothing now and can
be deleted.
''
)
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "ui" "enable" ]

11
nix/host-modules/local-defaults.nix
View file

@ -127,15 +127,4 @@ in
config.services.hyperhive.deploy.bao.bootstrapTokenFile = lib.mkIf cfg.deploy.singleHostSwarm (
lib.mkDefault "/var/lib/swarm-bao-bootstrap/grant.token"
);
# Same reason, one option later: the queue secret is a path on THIS host,
# so it moved to `deploy.*` with the rest of the controller's credentials.
# It has to sit out here rather than in the `swarm` attrset above — a bare
# `controller.` prefix in there means `swarm.controller`, which is now only
# a rename shim, so the definition would still resolve and warn on every
# evaluation of a single-host swarm.
config.services.hyperhive.deploy.swarm-controller.queue.clientSecretFile =
lib.mkIf cfg.deploy.singleHostSwarm (
lib.mkDefault "${config.services.hyperhive.deploy.authelia.hostClientSecretDir}/swarm-controller.secret"
);
}

34
nix/host-modules/swarm-bao.nix
View file

@ -360,9 +360,9 @@ let
# re-run keeps the value a live agent already holds instead of rotating it —
# the read is required, not incidental.
#
# The swarm appservice token, read-only: the controller creates agents'
# matrix accounts with it and never writes it. matrix-ctl mints and publishes
# it (`matrixCtlPolicyText` below).
# The swarm appservice token and its own OIDC client secret, read-only: it
# uses both and writes neither. matrix-ctl publishes the token
# (`matrixCtlPolicyText` below), the publisher the secret.
#
# The agent PKI grant is its only path on that mount: it can ask the one role
# for a certificate, not write that role or reach the issuer.
@ -391,6 +391,10 @@ let
capabilities = ["read"]
}
path "${credentialMountPath}/data/${controllerClientLeaf}" {
capabilities = ["read"]
}
path "${agentPkiMountPath}/issue/${agentPkiRoleName}" {
capabilities = ["update"]
}
@ -403,17 +407,18 @@ let
secretPublisherPolicyName = "swarm-secret-publisher";
secretPublisherCn = baoDeploy.secretPublisherCommonName;
# Two grants, and every narrowing in each is load-bearing.
# Three grants, and every narrowing in each is load-bearing.
#
# `secret/data/` is KV v2's ACL prefix, inserted by the engine rather than
# written by the caller — same trap as the controller's grant above.
#
# Two prefixes and not `swarm/*`: this principal has no business with an
# agent's credentials or the controller's, and these two are the only paths
# it produces. It grew the `services/` one when the publisher gained a swarm
# service's OIDC secret to copy, which is the rule ../module-eval.nix states
# for the controller's side of the same wall — a grant widens when a path
# gains a WRITER, not when a kind is declared.
# Two prefixes and one leaf, not `swarm/*`: this principal has no business
# with an agent's credentials, and these are the only paths it produces.
# Under `controller/` it writes the controller's OIDC client alone, spelled
# to the leaf so the appservice token beside it stays out of reach. The rule
# ../module-eval.nix states for the controller's side of the same wall holds
# here too — a grant widens when a path gains a WRITER, not when a kind is
# declared.
#
# Write-only. It copies secrets in and never reads one back; a read
# capability would let a file-copier recover every hive's credentials.
@ -425,6 +430,10 @@ let
path "${credentialMountPath}/data/swarm/services/*" {
capabilities = ["create", "update"]
}
path "${credentialMountPath}/data/${controllerClientLeaf}" {
capabilities = ["create", "update"]
}
'';
# The identity the matrix container's `swarm-matrix-ctl` presents. Named outside `hive-*`
@ -489,6 +498,11 @@ let
# homeserver's admin.
swarmAppserviceTokenLeaf = "swarm/controller/swarm-controller/matrix/appservice-token";
# The controller's own OIDC client secret, the nix half of
# `swarm_secret_client::queue::controller_client_path`. The publisher writes
# it; the controller reads it and holds no other copy.
controllerClientLeaf = "swarm/controller/swarm-controller/oidc/client";
# The KV v2 engine the controller writes agent credentials through. Named
# once because the grant above and the `secrets enable` in the bootstrap unit
# have to agree: a policy pointing at a mount nobody created is precisely the

193
nix/host-modules/swarm-controller.nix
View file

@ -121,11 +121,9 @@ let
# the swarm has one.
queueClientId = cfg.queueClientId;
# `LoadCredential` and not a copy-oneshot, which is where this deliberately
# differs from the callout responder: that one delivers INTO a container,
# so it has to copy across a filesystem boundary. The controller is a plain
# host unit, so systemd can hand it the file directly — fewer moving parts,
# and the secret never gains a second on-disk copy to forget about.
# No secret here. The daemon reads its client secret from the store at
# start, under `baoEnv`'s identity, from the path `swarm-secret-publish`
# writes it to on authelia's host (`queue_identity.rs`).
#
# Every value here comes from an option rather than from what happens to
# run on this host. The queue is not optional for a controller, but
@ -138,11 +136,6 @@ let
SWARM_CONTROLLER_NATS_URL = cfg.queue.natsUrl;
SWARM_CONTROLLER_OIDC_TOKEN_ENDPOINT = cfg.queue.tokenEndpoint;
SWARM_CONTROLLER_OIDC_CLIENT_ID = queueClientId;
# `%d` is systemd's credentials directory: root reads the plaintext at
# unit start and the daemon's own user sees it 0400, without the unit
# ever being able to read the rest of whatever directory the secret
# came from.
SWARM_CONTROLLER_OIDC_CLIENT_SECRET_FILE = "%d/queue-client.secret";
};
# Independent of `queueEnv` on purpose, and now for a simpler reason
@ -160,8 +153,8 @@ let
# shape for the other.
forgeEnv = lib.optionalAttrs (deployCfg.swarm-controller.forgeTokenFile != null) {
SWARM_CONTROLLER_FORGE_URL = "https://${forgeCfg.domain}";
# Same `%d` shape as the queue secret above — root reads the plaintext
# at unit start, the daemon's own user sees a 0400 copy.
# `%d` is systemd's credentials directory: root reads the plaintext at
# unit start, the daemon's own user sees a 0400 copy.
SWARM_CONTROLLER_FORGE_TOKEN_FILE = "%d/forge-token";
# `agent-configs` org avatar for the forge-objects pass
# (`forge/objects.rs`). Only meaningful with forge access, hence here.
@ -297,6 +290,16 @@ in
[ "services" "hyperhive" "c0re" "orgAvatarPng" ]
[ "services" "hyperhive" "deploy" "swarm-controller" "configOrgAvatarPng" ]
)
(lib.mkRemovedOptionModule
[ "services" "hyperhive" "deploy" "swarm-controller" "queue" "clientSecretFile" ]
''
The controller reads its queue client secret from the swarm's secret
store, under its own certificate (deploy.swarm-controller.baoClientCertFile),
and swarm-secret-publish on the host that runs authelia puts it there.
Remove this definition; the file it named is read by nothing now and can
be deleted.
''
)
];
options.services.hyperhive.swarm.controller = {
@ -549,7 +552,7 @@ in
No new credential to configure: the bearer token presented to
the bridge is minted from THIS daemon's own existing queue OIDC
identity — its endpoints stay in `swarm.controller.queue`, its secret is
`queue.clientSecretFile` below — "one identity per principal"
read from the swarm's secret store — "one identity per principal"
already covers it. `null` means no agent-identity support:
`CreateIdentity` jobs fail with a clear "no auth bridge
configured here" error rather than the daemon refusing to
@ -637,38 +640,10 @@ in
before anyone has onboarded a hive.
'';
};
queue = {
clientSecretFile = lib.mkOption {
type = lib.types.str;
default = "";
example = "/var/lib/secrets/swarm-controller-queue.secret";
description = ''
Path on **this** host holding the plaintext of the controller's
OAuth2 client secret. A path, never a value: the secret would
otherwise land in the world-readable nix store.
The controller cannot mint its own — minting happens inside
authelia's state directory during its first boot — so away from
that host the operator places the secret and names it here.
`singleHostSwarm` points this at the minted file, which
is exactly the case where one exists locally.
Read by `LoadCredential`, so it needs to be readable by root at
unit start and nothing more; the daemon's own user never sees
the original path.
'';
};
};
};
config = lib.mkIf deployCfg.swarm-controller.enable {
# The daemon and the oneshot that mints its credential — the second one
# failing leaves the first running and unable to authenticate anywhere.
services.hyperhive.swarm.otel.journaldUnits = [
"swarm-controller"
"swarm-controller-credential"
];
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-controller" ];
users.users.swarm-controller = {
isSystemUser = true;
@ -743,16 +718,17 @@ in
'';
}
{
assertion = deployCfg.swarm-controller.queue.clientSecretFile != "";
assertion = haveBaoIdentity;
message = ''
services.hyperhive.deploy.swarm-controller.queue.clientSecretFile
is unset.
services.hyperhive.deploy.swarm-controller.baoClientCertFile and
baoClientKeyFile must both be set.
It defaults to the file authelia's first-boot generator mints,
which only exists when authelia runs on this host. Elsewhere the
operator places the secret and names it here — the controller
cannot mint its own, because minting happens inside authelia's
state directory.
The controller reads its queue client secret from the swarm's
secret store, logging in with this certificate. They default to
the leaf the store mints when it runs on this host; elsewhere,
issue a leaf whose CN is
services.hyperhive.deploy.bao.controllerCommonName and name it
in both options.
'';
}
];
@ -765,63 +741,28 @@ in
serviceConfig = {
ExecStart = "${deployCfg.swarm-controller.package}/bin/swarm-controller";
# The two differ on purpose. The queue credential is
# unconditional — the assertions above make its path a value that
# always exists by the time this renders, so there is no "queue is
# off here" case left for a `mkIf` to express. The forge token
# stays optional: a controller with no forge access still serves
# its HTTP surface, and that IS a supported shape.
#
# ⚠️ "the path is a value" is not "the file is on disk". The
# co-located secret is minted by authelia's FIRST BOOT, in another
# container, and `hostClientSecretDir`'s own description says a
# consumer has to wait for it. A `LoadCredential=` naming an
# absolute path that is not there yet is fatal (`243/CREDENTIALS`),
# so the daemon spent three of systemd's five default starts losing
# that race on a real boot — two seconds more and it would have hit
# `start-limit-hit`, which does not self-heal.
LoadCredential = [
"queue-client.secret:${deployCfg.swarm-controller.queue.clientSecretFile}"
]
++ lib.optional (
deployCfg.swarm-controller.forgeTokenFile != null
) "forge-token:${deployCfg.swarm-controller.forgeTokenFile}"
# The store identity, same shape and same reason as hive-c0re's: the
# key is root-owned `0600` and this daemon runs as `swarm-controller`,
# so it never gets read access to the original file.
++ lib.optionals haveBaoIdentity [
"bao-client.pem:${deployCfg.swarm-controller.baoClientCertFile}"
"bao-client-key.pem:${deployCfg.swarm-controller.baoClientKeyFile}"
]
++ lib.optional (
haveBaoIdentity && deployCfg.bao.serverCaFile != null
) "bao-ca.pem:${deployCfg.bao.serverCaFile}"
++ lib.optional haveHiveClientCa "hive-client-ca.pem:${deployCfg.swarm-controller.hiveClientCaFile}";
# The forge token is optional: a controller with no forge access
# still serves its HTTP surface, and that IS a supported shape.
LoadCredential =
lib.optional (
deployCfg.swarm-controller.forgeTokenFile != null
) "forge-token:${deployCfg.swarm-controller.forgeTokenFile}"
# The store identity, same shape and same reason as hive-c0re's: the
# key is root-owned `0600` and this daemon runs as `swarm-controller`,
# so it never gets read access to the original file.
++ lib.optionals haveBaoIdentity [
"bao-client.pem:${deployCfg.swarm-controller.baoClientCertFile}"
"bao-client-key.pem:${deployCfg.swarm-controller.baoClientKeyFile}"
]
++ lib.optional (
haveBaoIdentity && deployCfg.bao.serverCaFile != null
) "bao-ca.pem:${deployCfg.bao.serverCaFile}"
++ lib.optional haveHiveClientCa "hive-client-ca.pem:${deployCfg.swarm-controller.hiveClientCaFile}";
# The placeholder default that makes the above non-fatal.
# `LoadCredential=` takes priority over `SetCredential=`, so this is
# only ever seen when the file is missing — and in that case systemd
# starts the unit instead of refusing to. The controller then serves
# its HTTP surface with the queue unconfigured, which is a supported
# shape it already knows how to report.
#
# ⚠️ THE VALUE MUST BE NON-EMPTY. `SetCredential=<id>:` with an empty
# value is rejected by systemd's parser — *"Invalid syntax, ignoring"*
# — so the whole line is dropped and the fail-soft above silently does
# not exist. Measured with `systemd-analyze verify`: empty is refused,
# any non-empty value is accepted. This shipped broken and only looked
# fine because the credential file happened to be present.
#
# The word is deliberate rather than arbitrary: it reaches the token
# request as the client secret, so authelia refuses it and the journal
# says so in terms an operator can act on.
#
# Safe in a unit file precisely because it is not a secret:
# `SetCredential=` values are readable by unprivileged processes over
# IPC, so real key material must never appear here.
SetCredential = [ "queue-client.secret:placeholder-no-secret-file" ];
User = "swarm-controller";
Group = "swarm-controller";
# Also what retries a start that gave up waiting for the secret store
# (`queue_identity.rs`).
Restart = "on-failure";
RestartSec = "5s";
@ -925,49 +866,5 @@ in
// baoEnv
// otelEnv;
};
# A systemd credential is a SNAPSHOT: it is materialised into `%d` once,
# at unit start, and never re-read. That is invisible until the file
# underneath it changes — and two ordinary things change it.
#
# - it ARRIVES LATE. The co-located secret is minted by authelia's
# first boot, in another container, which a host unit cannot order
# against. Before this, the unit died at `243/CREDENTIALS` and was
# rescued only by burning restarts until the file showed up.
# - it is ROTATED. `mint_token` deliberately reads the secret file on
# every call so a rotation takes effect without a restart — a
# snapshot in `%d` quietly defeats that, and nothing reports it.
#
# Watching the file closes both: on close-after-write, restart the
# daemon so it re-snapshots. `PathChanged=` and not `PathExists=`,
# measured against the semantics rather than guessed — `PathExists=`
# activates immediately whenever the file is *already there* at unit
# start, which would restart a perfectly healthy daemon on every boot.
# `PathChanged=` requires a write, so it cannot do that and cannot spin.
#
# ⚠️ Known gap, stated rather than papered over: a secret appearing in
# the sub-second window between the daemon starting and this unit
# watching is missed until the next write. Closing it needs
# `PathExists=`, whose cost is the spurious per-boot restart above.
systemd.paths.swarm-controller-credential = {
description = "watch the swarm controller's queue credential";
wantedBy = [ "multi-user.target" ];
pathConfig = {
PathChanged = deployCfg.swarm-controller.queue.clientSecretFile;
Unit = "swarm-controller-credential.service";
};
};
# `try-restart`, not `restart`: if the daemon is stopped — masked,
# disabled, or deliberately down — a secret rotation is not a reason to
# start it. Rotating a credential should never be how a service comes
# back to life.
systemd.services.swarm-controller-credential = {
description = "restart the swarm controller after its queue credential changed";
serviceConfig = {
Type = "oneshot";
ExecStart = "${pkgs.systemd}/bin/systemctl try-restart swarm-controller.service";
};
};
};
}

39
nix/host-modules/swarm-secret-publisher.nix
View file

@ -1,7 +1,7 @@
# The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its agents' credential, a swarm
# service its own, a hive its matrix appservice token. The OIDC secrets it only
# copies; the appservice token it MINTS, having had no swarm-side producer.
# whoever needs one can read it there: a hive its agents' credential, a
# service or the controller its own, a hive its appservice token. OIDC secrets
# it only copies; the appservice token it MINTS, having no swarm-side producer.
#
# ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
# into the store even when the service runs beside authelia, because its
@ -80,6 +80,11 @@ let
hyperhiveCfg.swarm.bao.otel.clientId
];
# The swarm controller's OIDC client, which it reads back from the store
# under its own certificate.
controllerClientId = hyperhiveCfg.swarm.controller.queueClientId;
controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client";
# Where this unit keeps the appservice tokens it minted, and the whole reason
# a re-publish is idempotent. The store cannot be that record: ./swarm-bao.nix
# grants this principal `create`/`update` and deliberately no `read`, so "does
@ -104,8 +109,8 @@ in
description = ''
Publish the OIDC client secrets this host mints into the swarm's
secret store, so a hive that does not run authelia can read its
agents' credential and a swarm service can read its own — from
wherever it runs, this host included. Also mints each hive's matrix
agents' credential and a swarm service or the controller can read its
own — from wherever it runs, this host included. Also mints each hive's matrix
appservice token, which has no other swarm-side producer, and
publishes it the same way.
@ -120,10 +125,10 @@ in
identity's job (see `baoClientCertFile`), not this option's.
Turning it off leaves every hive but this one without its agents'
credential, the swarm's Grafana without any login at all, its
collector pushing unauthenticated, and every hive falling back to the
appservice token its own first boot minted — so two hives never agree
on one. Each secret has exactly one route and this is the producer's
credential, the swarm controller unable to start, the swarm's Grafana
without any login at all, its collector pushing unauthenticated, and
every hive falling back to the appservice token its own first boot
minted — so two hives never agree on one. Each secret has exactly one route and this is the producer's
end of it, so the honest reason to set it false is a deployment
delivering those secrets by some other mechanism it owns.
'';
@ -261,6 +266,22 @@ in
fi
'') serviceClientIds}
# The controller's own client, under `controller/` rather than
# `services/`: every hive reads `services/*`, and this client may read
# every hive's status. The nix half of
# `swarm_secret_client::queue::controller_client_path`.
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${controllerClientId}.secret"}
if [ -s "$src" ]; then
bao kv put ${lib.escapeShellArg controllerSecretPath} \
value=@"$src"
published=$((published + 1))
else
# Authelia mints it only where the controller registers its client,
# which is where the controller runs beside it.
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
# The matrix appservice token, per hive. Unlike the two loops above
# there is nothing to copy: authelia never minted this one, and each
# homeserver's own host minted its own — which is exactly why two hives