swarm-bao-forge-publisher re-runs on container start and token writes
The publisher was a oneshot with RemainAfterExit=true, started only through `wantedBy = container@hive-forge.service`. Once it had run it stayed `active (exited)`, so a later container start made its start a no-op: a token re-minted without a host reboot or a unit change never reached the store. - RemainAfterExit is dropped, so each start runs the script again. - systemd.paths.swarm-bao-forge-publisher watches the token file in the container (PathChanged) and starts the publisher when the mint writes it, the same shape as swarm-secret-publish's path unit. - The code comment, secrets.md and credentials.md now name both triggers; forge-placement pins the missing RemainAfterExit and the path unit. Refs #4998
This commit is contained in:
atlas
parent
c45f7a5330
commit
2d2d7529fc
4 changed files with 41 additions and 6 deletions
|
|
@ -61,7 +61,7 @@ of the cell says how.
|
|||
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
|
||||
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
|
||||
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts |
|
||||
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
|
||||
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
|
||||
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires |
|
||||
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff |
|
||||
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
|
||||
|
|
|
|||
|
|
@ -52,7 +52,7 @@ neither is a renaming of the other.
|
|||
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
||||
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
|
||||
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
|
||||
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
|
||||
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
|
|
|
|||
|
|
@ -56,6 +56,9 @@ let
|
|||
# `forgeSecretPath` above.
|
||||
swarmControllerTokenPath = "/var/lib/forgejo/swarm-controller-token";
|
||||
|
||||
# The same file from the host, which is where its publisher reads and watches it.
|
||||
swarmControllerTokenHostPath = "/var/lib/nixos-containers/hive-forge${swarmControllerTokenPath}";
|
||||
|
||||
# Where `swarm-bao-forge-publisher` puts it: `controllerForgeTokenLeaf` in
|
||||
# ../swarm-bao.nix, under the KV mount.
|
||||
controllerTokenStorePath = "secret/swarm/controller/swarm-controller/forge/token";
|
||||
|
|
@ -1057,8 +1060,12 @@ in
|
|||
# It can only depend on the container being up, not on the in-container
|
||||
# oneshot that mints the token (containers run their own systemd instance,
|
||||
# invisible to this one by unit name), so it polls the same bounded way
|
||||
# `hive-forge-oidc-secret` does. `wantedBy` the container, so every
|
||||
# container start re-publishes whatever the mint unit left there.
|
||||
# `hive-forge-oidc-secret` does. It runs whenever the token can have
|
||||
# changed: each time the container starts (`wantedBy`), and each time the
|
||||
# file is written (`systemd.paths.swarm-bao-forge-publisher` below).
|
||||
#
|
||||
# ⚠️ No `RemainAfterExit`. A oneshot that stays `active (exited)` turns
|
||||
# every later start into a no-op, so neither trigger would run it again.
|
||||
#
|
||||
# Unconditional, like the in-container unit it publishes from: forge is a
|
||||
# swarm-wide singleton, so the token always gets minted and published,
|
||||
|
|
@ -1088,7 +1095,6 @@ in
|
|||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||
serviceConfig = storeRetry.serviceConfig // {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
SyslogIdentifier = "swarm-bao-forge-publisher";
|
||||
# Longer than the 120s bounded wait below. `DefaultTimeoutStartSec`
|
||||
# is 90s, so without this systemd kills the unit at 90 — before it
|
||||
|
|
@ -1115,7 +1121,7 @@ in
|
|||
''
|
||||
set -euo pipefail
|
||||
|
||||
src=${lib.escapeShellArg "/var/lib/nixos-containers/hive-forge${swarmControllerTokenPath}"}
|
||||
src=${lib.escapeShellArg swarmControllerTokenHostPath}
|
||||
|
||||
for _ in $(seq 1 60); do
|
||||
[ -s "$src" ] && break
|
||||
|
|
@ -1157,6 +1163,18 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
# Re-publish when the mint unit writes a token, which it can do without the
|
||||
# container restarting. The file is the event: the mint runs in the
|
||||
# container's own systemd, which this one cannot subscribe to.
|
||||
systemd.paths.swarm-bao-forge-publisher = {
|
||||
description = "watch the swarm-controller's forge token for a publish";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
pathConfig = {
|
||||
PathChanged = swarmControllerTokenHostPath;
|
||||
Unit = "swarm-bao-forge-publisher.service";
|
||||
};
|
||||
};
|
||||
|
||||
# Unused: nothing writes or reads this host path, and it may hold a copy of
|
||||
# the controller's forge token.
|
||||
systemd.tmpfiles.rules = [ "r /var/lib/hyperhive-forge/swarm-controller.token" ];
|
||||
|
|
|
|||
|
|
@ -294,6 +294,23 @@ let
|
|||
&& lib.elem "container@hive-forge.service" u.wantedBy
|
||||
&& !(allLocal.systemd.services ? hive-forge-swarm-controller-token);
|
||||
}
|
||||
{
|
||||
# What makes it re-run. A oneshot left `active (exited)` makes every later
|
||||
# start a no-op, so `RemainAfterExit` would silence both triggers: the
|
||||
# container's start (`wantedBy`, above) and a write of the token file.
|
||||
name = "the forge publisher re-runs on each container start and each write of the token file";
|
||||
ok =
|
||||
let
|
||||
p = allLocal.systemd.paths.swarm-bao-forge-publisher;
|
||||
in
|
||||
!((publisher allLocal).serviceConfig.RemainAfterExit or false)
|
||||
&&
|
||||
p.pathConfig.PathChanged
|
||||
== "/var/lib/nixos-containers/hive-forge/var/lib/forgejo/swarm-controller-token"
|
||||
&& p.pathConfig.Unit == "swarm-bao-forge-publisher.service"
|
||||
&& lib.elem "multi-user.target" p.wantedBy
|
||||
&& !(bare.systemd.paths ? swarm-bao-forge-publisher);
|
||||
}
|
||||
{
|
||||
# The absence arm: a forge host without the store has no leaf unless one
|
||||
# is placed, and the unit says which options to set.
|
||||
|
|
|
|||
Loading…
Reference in a new issue