swarm-controller reads its forge token from the store, not a host file
The controller's forge token never entered bao: the forge minted it in its container, hive-forge-swarm-controller-token copied it to a host file, and the controller LoadCredential'ed that file. forgeTokenFile defaulted to that path only where the forge ran, so a controller on any other host had no forge access unless an operator copied the file by hand. Now the forge's host publishes it and the controller reads it: - swarm-bao-forge-publisher (replaces hive-forge-swarm-controller-token) waits for the in-container token, logs in with its own forge-publisher leaf and writes it to swarm/controller/swarm-controller/forge/token. It runs on every forge container start, the old unit's trigger. - New principal swarm-forge-publisher: leaf minted by glue-bao-tls.nix, policy create/update on that one path, written by swarm-bao-forge-publisher-policy, CN reserved as a hive name. - The controller policy gains read on that path. swarm-controller reads it at start under its own certificate with the OIDC secret's retry, and runs without forge access if it still cannot. - forgeTokenFile, its LoadCredential and the forgeEnv gate are gone; SWARM_CONTROLLER_FORGE_URL is always set. hostSwarmControllerTokenFile is removed, and a tmpfiles rule deletes the old host copy on the forge host. The in-container file stays: the mint unit's only-once guard reads it, and it is the publisher's source. Refs #4998
This commit is contained in:
atlas
parent
c433bdef92
commit
121c4678aa
18 changed files with 580 additions and 277 deletions
|
|
@ -189,24 +189,11 @@ let
|
|||
SWARM_CONTROLLER_OIDC_CLIENT_ID = queueClientId;
|
||||
};
|
||||
|
||||
# Independent of `queueEnv` on purpose, and now for a simpler reason
|
||||
# than when this was written: the queue's own co-location guard
|
||||
# (`queueLocal`) is gone — its coordinates are options with assertions,
|
||||
# so a split-host controller fails at eval instead of silently losing
|
||||
# its identity. Forge access still has nothing to do with where authelia
|
||||
# and NATS run, only with whether `forgeTokenFile` resolves to a real
|
||||
# file — which that option's own default already handles
|
||||
# (forge-local vs. operator-copied). Gating a second time here would
|
||||
# just repeat that option's own logic under a different name.
|
||||
#
|
||||
# ⚠️ The asymmetry is deliberate: the queue is REQUIRED for a controller
|
||||
# and the forge is not, so absence is an error for one and a supported
|
||||
# shape for the other.
|
||||
forgeEnv = lib.optionalAttrs (deployCfg.swarm-controller.forgeTokenFile != null) {
|
||||
# Unconditional: every swarm has a forge, reached by name wherever it runs.
|
||||
# The token is not here: the daemon reads it from the swarm's secret store at
|
||||
# start (`forge.rs`), and runs without forge access when it cannot.
|
||||
forgeEnv = {
|
||||
SWARM_CONTROLLER_FORGE_URL = "https://${forgeCfg.domain}";
|
||||
# `%d` is systemd's credentials directory: root reads the plaintext at
|
||||
# unit start, the daemon's own user sees a 0400 copy.
|
||||
SWARM_CONTROLLER_FORGE_TOKEN_FILE = "%d/forge-token";
|
||||
# `agent-configs` org avatar for the forge-objects pass
|
||||
# (`forge/objects.rs`). Only meaningful with forge access, hence here.
|
||||
SWARM_CONTROLLER_CONFIG_ORG_AVATAR_PNG =
|
||||
|
|
@ -268,8 +255,8 @@ let
|
|||
|
||||
# `vcs_metrics`/`hive_jobq_metrics` (this daemon's OTLP push exporters)
|
||||
# read the standard `OTEL_EXPORTER_OTLP_ENDPOINT` var and are silent,
|
||||
# graceful no-ops without it — same shape as `forgeEnv`/`authBridgeEnv`
|
||||
# above, genuinely optional and gated on the option resolving rather
|
||||
# graceful no-ops without it — same shape as `authBridgeEnv` above,
|
||||
# genuinely optional and gated on the option resolving rather
|
||||
# than assumed.
|
||||
#
|
||||
# By NAME through the gateway, never a loopback literal or a bare
|
||||
|
|
@ -294,8 +281,7 @@ let
|
|||
# HERE"): the collector is reachable by name wherever it runs, so a
|
||||
# controller not co-located with it now exports over https instead of
|
||||
# exporting nothing — the graceful-absence case left is "no swarm-otel
|
||||
# anywhere in this swarm at all," which `forgeEnv` already models the
|
||||
# same way.
|
||||
# anywhere in this swarm at all."
|
||||
otelSwarmCfg = config.services.hyperhive.swarm.otel;
|
||||
otelEnv = lib.optionalAttrs deployCfg.swarm-otel.enable {
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT = "https://${otelSwarmCfg.domain}/${otelSwarmCfg.producerName}";
|
||||
|
|
@ -352,6 +338,14 @@ in
|
|||
be deleted.
|
||||
''
|
||||
)
|
||||
(lib.mkRemovedOptionModule [ "services" "hyperhive" "deploy" "swarm-controller" "forgeTokenFile" ]
|
||||
''
|
||||
The controller reads its forge token from the swarm's secret store, under
|
||||
its own certificate (deploy.swarm-controller.baoClientCertFile), and
|
||||
swarm-bao-forge-publisher on the forge's host puts it there. Remove this
|
||||
definition; the file it named is read by nothing now and can be deleted.
|
||||
''
|
||||
)
|
||||
];
|
||||
|
||||
options.services.hyperhive.swarm.controller = {
|
||||
|
|
@ -554,40 +548,6 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
forgeTokenFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
# Forge's own delivery path when the forge runs here, since nothing
|
||||
# writes that path anywhere else. `null` otherwise, so a controller
|
||||
# away from the forge logs that it has no forge access instead of
|
||||
# waiting on a file that never appears. The controller's units are
|
||||
# the thing that decides whether the file is ever read: they only
|
||||
# exist under `deploy.swarm-controller.enable`.
|
||||
default = if deployCfg.forgejo.enable then deployCfg.forgejo.hostSwarmControllerTokenFile else null;
|
||||
defaultText = lib.literalExpression ''
|
||||
if config.services.hyperhive.deploy.forgejo.enable
|
||||
then config.services.hyperhive.deploy.forgejo.hostSwarmControllerTokenFile
|
||||
else null
|
||||
'';
|
||||
example = "/var/lib/secrets/swarm-controller-forge.token";
|
||||
description = ''
|
||||
Path to this host's copy of the swarm-controller's forge access
|
||||
token (see `hive-forge/default.nix`'s
|
||||
`forgejo-swarm-controller-account` + `hive-forge-swarm-controller-token`
|
||||
units, which mint and collect it onto forge's own host).
|
||||
|
||||
Defaults to forge's own delivery path when
|
||||
{option}`services.hyperhive.deploy.forgejo.enable` is set on this
|
||||
host, and to `null` otherwise. Set it explicitly when the forge
|
||||
runs on another host — copy the token out of forge's
|
||||
{option}`services.hyperhive.deploy.forgejo.hostSwarmControllerTokenFile`
|
||||
with whatever secret management this deployment already uses,
|
||||
the same shape `swarm.nix`'s `clientSecretFile` documents for
|
||||
its own cross-host case. `null` means no forge access — the
|
||||
daemon logs that and continues without it, the same
|
||||
graceful-absence shape the queue coordinates already use.
|
||||
'';
|
||||
};
|
||||
|
||||
authBridgeUrl = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = "${autheliaCfg.url}/bridge";
|
||||
|
|
@ -606,7 +566,7 @@ in
|
|||
already covers it. `null` means no agent-identity support:
|
||||
`CreateIdentity` jobs fail with a clear "no auth bridge
|
||||
configured here" error rather than the daemon refusing to
|
||||
start, the same graceful-absence shape `forgeTokenFile` uses.
|
||||
start, the same graceful-absence shape forge access has.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -758,21 +718,21 @@ in
|
|||
systemd.services.swarm-controller = {
|
||||
description = "hyperhive swarm-level controller daemon";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network.target" ];
|
||||
# Where the forge runs here too, the first read waits out one publish
|
||||
# attempt. `After=` on a unit this host does not define is a no-op.
|
||||
after = [
|
||||
"network.target"
|
||||
"swarm-bao-forge-publisher.service"
|
||||
];
|
||||
|
||||
serviceConfig = {
|
||||
ExecStart = "${deployCfg.swarm-controller.package}/bin/swarm-controller";
|
||||
|
||||
# The forge token is optional: a controller with no forge access
|
||||
# still serves its HTTP surface, and that IS a supported shape.
|
||||
# The store identity, same shape and same reason as hive-c0re's: the
|
||||
# key is root-owned `0600` and this daemon runs as `swarm-controller`,
|
||||
# so it never gets read access to the original file.
|
||||
LoadCredential =
|
||||
lib.optional (
|
||||
deployCfg.swarm-controller.forgeTokenFile != null
|
||||
) "forge-token:${deployCfg.swarm-controller.forgeTokenFile}"
|
||||
# The store identity, same shape and same reason as hive-c0re's: the
|
||||
# key is root-owned `0600` and this daemon runs as `swarm-controller`,
|
||||
# so it never gets read access to the original file.
|
||||
++ lib.optionals haveBaoIdentity [
|
||||
lib.optionals haveBaoIdentity [
|
||||
"bao-client.pem:${deployCfg.swarm-controller.baoClientCertFile}"
|
||||
"bao-client-key.pem:${deployCfg.swarm-controller.baoClientKeyFile}"
|
||||
]
|
||||
|
|
@ -846,8 +806,8 @@ in
|
|||
#
|
||||
# They differ in how absence is prevented: the queue's is checked by
|
||||
# the assertions above, because a controller without a queue is
|
||||
# broken rather than lighter; forge's and the auth bridge's are
|
||||
# genuinely optional and stay gated on their own option resolving.
|
||||
# broken rather than lighter; the auth bridge's is genuinely optional
|
||||
# and stays gated on its own option resolving. Forge's is always set.
|
||||
environment = {
|
||||
SWARM_CONTROLLER_SOCKET = deployCfg.swarm-controller.socketPath;
|
||||
# The swarm's hive directory, JSON-encoded — the full directory
|
||||
|
|
|
|||
Loading…
Reference in a new issue