swarm-controller: read the queue client secret from the store, drop the file
Some checks were skipped
public bin cache / build + push to preem:grid (push) Has been skipped
Some checks were skipped
public bin cache / build + push to preem:grid (push) Has been skipped
The controller's OIDC client secret (client `swarm-controller`, used for the queue connection, the auth-bridge bearer and the OTLP push) came from an operator-placed file, `deploy.swarm-controller.queue.clientSecretFile`, handed in by `LoadCredential=`. Now `swarm-secret-publish`, which already copies authelia's minted OIDC secrets into the store, also publishes this one, to `swarm/controller/swarm-controller/oidc/client`. That path sits under `controller/`, which no hive's policy reads. The controller reads it once at start with its existing store certificate and holds it in memory, as `swarm_queue_client::ClientSecret::Value`. If the store is down, it retries for about a minute and then fails the start, so `Restart=` tries again. Policy delta: the controller gets `read` on that leaf, and the publisher gets `create`/`update` on that leaf. Removed: the `queue.clientSecretFile` option (both spellings, now removed options with a message), its singleHostSwarm default, the credential and placeholder, and the path watcher plus its restart oneshot. A controller without a store identity is now an eval error, because it has no other way to get the secret.
This commit is contained in:
atlas
parent
97c4771514
commit
e94406cdb9
22 changed files with 595 additions and 247 deletions
|
|
@ -121,11 +121,9 @@ let
|
|||
# the swarm has one.
|
||||
queueClientId = cfg.queueClientId;
|
||||
|
||||
# `LoadCredential` and not a copy-oneshot, which is where this deliberately
|
||||
# differs from the callout responder: that one delivers INTO a container,
|
||||
# so it has to copy across a filesystem boundary. The controller is a plain
|
||||
# host unit, so systemd can hand it the file directly — fewer moving parts,
|
||||
# and the secret never gains a second on-disk copy to forget about.
|
||||
# No secret here. The daemon reads its client secret from the store at
|
||||
# start, under `baoEnv`'s identity, from the path `swarm-secret-publish`
|
||||
# writes it to on authelia's host (`queue_identity.rs`).
|
||||
#
|
||||
# Every value here comes from an option rather than from what happens to
|
||||
# run on this host. The queue is not optional for a controller, but
|
||||
|
|
@ -138,11 +136,6 @@ let
|
|||
SWARM_CONTROLLER_NATS_URL = cfg.queue.natsUrl;
|
||||
SWARM_CONTROLLER_OIDC_TOKEN_ENDPOINT = cfg.queue.tokenEndpoint;
|
||||
SWARM_CONTROLLER_OIDC_CLIENT_ID = queueClientId;
|
||||
# `%d` is systemd's credentials directory: root reads the plaintext at
|
||||
# unit start and the daemon's own user sees it 0400, without the unit
|
||||
# ever being able to read the rest of whatever directory the secret
|
||||
# came from.
|
||||
SWARM_CONTROLLER_OIDC_CLIENT_SECRET_FILE = "%d/queue-client.secret";
|
||||
};
|
||||
|
||||
# Independent of `queueEnv` on purpose, and now for a simpler reason
|
||||
|
|
@ -160,8 +153,8 @@ let
|
|||
# shape for the other.
|
||||
forgeEnv = lib.optionalAttrs (deployCfg.swarm-controller.forgeTokenFile != null) {
|
||||
SWARM_CONTROLLER_FORGE_URL = "https://${forgeCfg.domain}";
|
||||
# Same `%d` shape as the queue secret above — root reads the plaintext
|
||||
# at unit start, the daemon's own user sees a 0400 copy.
|
||||
# `%d` is systemd's credentials directory: root reads the plaintext at
|
||||
# unit start, the daemon's own user sees a 0400 copy.
|
||||
SWARM_CONTROLLER_FORGE_TOKEN_FILE = "%d/forge-token";
|
||||
# `agent-configs` org avatar for the forge-objects pass
|
||||
# (`forge/objects.rs`). Only meaningful with forge access, hence here.
|
||||
|
|
@ -297,6 +290,16 @@ in
|
|||
[ "services" "hyperhive" "c0re" "orgAvatarPng" ]
|
||||
[ "services" "hyperhive" "deploy" "swarm-controller" "configOrgAvatarPng" ]
|
||||
)
|
||||
(lib.mkRemovedOptionModule
|
||||
[ "services" "hyperhive" "deploy" "swarm-controller" "queue" "clientSecretFile" ]
|
||||
''
|
||||
The controller reads its queue client secret from the swarm's secret
|
||||
store, under its own certificate (deploy.swarm-controller.baoClientCertFile),
|
||||
and swarm-secret-publish on the host that runs authelia puts it there.
|
||||
Remove this definition; the file it named is read by nothing now and can
|
||||
be deleted.
|
||||
''
|
||||
)
|
||||
];
|
||||
|
||||
options.services.hyperhive.swarm.controller = {
|
||||
|
|
@ -549,7 +552,7 @@ in
|
|||
No new credential to configure: the bearer token presented to
|
||||
the bridge is minted from THIS daemon's own existing queue OIDC
|
||||
identity — its endpoints stay in `swarm.controller.queue`, its secret is
|
||||
`queue.clientSecretFile` below — "one identity per principal"
|
||||
read from the swarm's secret store — "one identity per principal"
|
||||
already covers it. `null` means no agent-identity support:
|
||||
`CreateIdentity` jobs fail with a clear "no auth bridge
|
||||
configured here" error rather than the daemon refusing to
|
||||
|
|
@ -637,38 +640,10 @@ in
|
|||
before anyone has onboarded a hive.
|
||||
'';
|
||||
};
|
||||
|
||||
queue = {
|
||||
clientSecretFile = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "";
|
||||
example = "/var/lib/secrets/swarm-controller-queue.secret";
|
||||
description = ''
|
||||
Path on **this** host holding the plaintext of the controller's
|
||||
OAuth2 client secret. A path, never a value: the secret would
|
||||
otherwise land in the world-readable nix store.
|
||||
|
||||
The controller cannot mint its own — minting happens inside
|
||||
authelia's state directory during its first boot — so away from
|
||||
that host the operator places the secret and names it here.
|
||||
`singleHostSwarm` points this at the minted file, which
|
||||
is exactly the case where one exists locally.
|
||||
|
||||
Read by `LoadCredential`, so it needs to be readable by root at
|
||||
unit start and nothing more; the daemon's own user never sees
|
||||
the original path.
|
||||
'';
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
config = lib.mkIf deployCfg.swarm-controller.enable {
|
||||
# The daemon and the oneshot that mints its credential — the second one
|
||||
# failing leaves the first running and unable to authenticate anywhere.
|
||||
services.hyperhive.swarm.otel.journaldUnits = [
|
||||
"swarm-controller"
|
||||
"swarm-controller-credential"
|
||||
];
|
||||
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-controller" ];
|
||||
|
||||
users.users.swarm-controller = {
|
||||
isSystemUser = true;
|
||||
|
|
@ -743,16 +718,17 @@ in
|
|||
'';
|
||||
}
|
||||
{
|
||||
assertion = deployCfg.swarm-controller.queue.clientSecretFile != "";
|
||||
assertion = haveBaoIdentity;
|
||||
message = ''
|
||||
services.hyperhive.deploy.swarm-controller.queue.clientSecretFile
|
||||
is unset.
|
||||
services.hyperhive.deploy.swarm-controller.baoClientCertFile and
|
||||
baoClientKeyFile must both be set.
|
||||
|
||||
It defaults to the file authelia's first-boot generator mints,
|
||||
which only exists when authelia runs on this host. Elsewhere the
|
||||
operator places the secret and names it here — the controller
|
||||
cannot mint its own, because minting happens inside authelia's
|
||||
state directory.
|
||||
The controller reads its queue client secret from the swarm's
|
||||
secret store, logging in with this certificate. They default to
|
||||
the leaf the store mints when it runs on this host; elsewhere,
|
||||
issue a leaf whose CN is
|
||||
services.hyperhive.deploy.bao.controllerCommonName and name it
|
||||
in both options.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
|
@ -765,63 +741,28 @@ in
|
|||
serviceConfig = {
|
||||
ExecStart = "${deployCfg.swarm-controller.package}/bin/swarm-controller";
|
||||
|
||||
# The two differ on purpose. The queue credential is
|
||||
# unconditional — the assertions above make its path a value that
|
||||
# always exists by the time this renders, so there is no "queue is
|
||||
# off here" case left for a `mkIf` to express. The forge token
|
||||
# stays optional: a controller with no forge access still serves
|
||||
# its HTTP surface, and that IS a supported shape.
|
||||
#
|
||||
# ⚠️ "the path is a value" is not "the file is on disk". The
|
||||
# co-located secret is minted by authelia's FIRST BOOT, in another
|
||||
# container, and `hostClientSecretDir`'s own description says a
|
||||
# consumer has to wait for it. A `LoadCredential=` naming an
|
||||
# absolute path that is not there yet is fatal (`243/CREDENTIALS`),
|
||||
# so the daemon spent three of systemd's five default starts losing
|
||||
# that race on a real boot — two seconds more and it would have hit
|
||||
# `start-limit-hit`, which does not self-heal.
|
||||
LoadCredential = [
|
||||
"queue-client.secret:${deployCfg.swarm-controller.queue.clientSecretFile}"
|
||||
]
|
||||
++ lib.optional (
|
||||
deployCfg.swarm-controller.forgeTokenFile != null
|
||||
) "forge-token:${deployCfg.swarm-controller.forgeTokenFile}"
|
||||
# The store identity, same shape and same reason as hive-c0re's: the
|
||||
# key is root-owned `0600` and this daemon runs as `swarm-controller`,
|
||||
# so it never gets read access to the original file.
|
||||
++ lib.optionals haveBaoIdentity [
|
||||
"bao-client.pem:${deployCfg.swarm-controller.baoClientCertFile}"
|
||||
"bao-client-key.pem:${deployCfg.swarm-controller.baoClientKeyFile}"
|
||||
]
|
||||
++ lib.optional (
|
||||
haveBaoIdentity && deployCfg.bao.serverCaFile != null
|
||||
) "bao-ca.pem:${deployCfg.bao.serverCaFile}"
|
||||
++ lib.optional haveHiveClientCa "hive-client-ca.pem:${deployCfg.swarm-controller.hiveClientCaFile}";
|
||||
# The forge token is optional: a controller with no forge access
|
||||
# still serves its HTTP surface, and that IS a supported shape.
|
||||
LoadCredential =
|
||||
lib.optional (
|
||||
deployCfg.swarm-controller.forgeTokenFile != null
|
||||
) "forge-token:${deployCfg.swarm-controller.forgeTokenFile}"
|
||||
# The store identity, same shape and same reason as hive-c0re's: the
|
||||
# key is root-owned `0600` and this daemon runs as `swarm-controller`,
|
||||
# so it never gets read access to the original file.
|
||||
++ lib.optionals haveBaoIdentity [
|
||||
"bao-client.pem:${deployCfg.swarm-controller.baoClientCertFile}"
|
||||
"bao-client-key.pem:${deployCfg.swarm-controller.baoClientKeyFile}"
|
||||
]
|
||||
++ lib.optional (
|
||||
haveBaoIdentity && deployCfg.bao.serverCaFile != null
|
||||
) "bao-ca.pem:${deployCfg.bao.serverCaFile}"
|
||||
++ lib.optional haveHiveClientCa "hive-client-ca.pem:${deployCfg.swarm-controller.hiveClientCaFile}";
|
||||
|
||||
# The placeholder default that makes the above non-fatal.
|
||||
# `LoadCredential=` takes priority over `SetCredential=`, so this is
|
||||
# only ever seen when the file is missing — and in that case systemd
|
||||
# starts the unit instead of refusing to. The controller then serves
|
||||
# its HTTP surface with the queue unconfigured, which is a supported
|
||||
# shape it already knows how to report.
|
||||
#
|
||||
# ⚠️ THE VALUE MUST BE NON-EMPTY. `SetCredential=<id>:` with an empty
|
||||
# value is rejected by systemd's parser — *"Invalid syntax, ignoring"*
|
||||
# — so the whole line is dropped and the fail-soft above silently does
|
||||
# not exist. Measured with `systemd-analyze verify`: empty is refused,
|
||||
# any non-empty value is accepted. This shipped broken and only looked
|
||||
# fine because the credential file happened to be present.
|
||||
#
|
||||
# The word is deliberate rather than arbitrary: it reaches the token
|
||||
# request as the client secret, so authelia refuses it and the journal
|
||||
# says so in terms an operator can act on.
|
||||
#
|
||||
# Safe in a unit file precisely because it is not a secret:
|
||||
# `SetCredential=` values are readable by unprivileged processes over
|
||||
# IPC, so real key material must never appear here.
|
||||
SetCredential = [ "queue-client.secret:placeholder-no-secret-file" ];
|
||||
User = "swarm-controller";
|
||||
Group = "swarm-controller";
|
||||
# Also what retries a start that gave up waiting for the secret store
|
||||
# (`queue_identity.rs`).
|
||||
Restart = "on-failure";
|
||||
RestartSec = "5s";
|
||||
|
||||
|
|
@ -925,49 +866,5 @@ in
|
|||
// baoEnv
|
||||
// otelEnv;
|
||||
};
|
||||
|
||||
# A systemd credential is a SNAPSHOT: it is materialised into `%d` once,
|
||||
# at unit start, and never re-read. That is invisible until the file
|
||||
# underneath it changes — and two ordinary things change it.
|
||||
#
|
||||
# - it ARRIVES LATE. The co-located secret is minted by authelia's
|
||||
# first boot, in another container, which a host unit cannot order
|
||||
# against. Before this, the unit died at `243/CREDENTIALS` and was
|
||||
# rescued only by burning restarts until the file showed up.
|
||||
# - it is ROTATED. `mint_token` deliberately reads the secret file on
|
||||
# every call so a rotation takes effect without a restart — a
|
||||
# snapshot in `%d` quietly defeats that, and nothing reports it.
|
||||
#
|
||||
# Watching the file closes both: on close-after-write, restart the
|
||||
# daemon so it re-snapshots. `PathChanged=` and not `PathExists=`,
|
||||
# measured against the semantics rather than guessed — `PathExists=`
|
||||
# activates immediately whenever the file is *already there* at unit
|
||||
# start, which would restart a perfectly healthy daemon on every boot.
|
||||
# `PathChanged=` requires a write, so it cannot do that and cannot spin.
|
||||
#
|
||||
# ⚠️ Known gap, stated rather than papered over: a secret appearing in
|
||||
# the sub-second window between the daemon starting and this unit
|
||||
# watching is missed until the next write. Closing it needs
|
||||
# `PathExists=`, whose cost is the spurious per-boot restart above.
|
||||
systemd.paths.swarm-controller-credential = {
|
||||
description = "watch the swarm controller's queue credential";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
pathConfig = {
|
||||
PathChanged = deployCfg.swarm-controller.queue.clientSecretFile;
|
||||
Unit = "swarm-controller-credential.service";
|
||||
};
|
||||
};
|
||||
|
||||
# `try-restart`, not `restart`: if the daemon is stopped — masked,
|
||||
# disabled, or deliberately down — a secret rotation is not a reason to
|
||||
# start it. Rotating a credential should never be how a service comes
|
||||
# back to life.
|
||||
systemd.services.swarm-controller-credential = {
|
||||
description = "restart the swarm controller after its queue credential changed";
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
ExecStart = "${pkgs.systemd}/bin/systemctl try-restart swarm-controller.service";
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue