vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-bao-forge-publisher re-runs on container start and token writes

Browse source 
The publisher was a oneshot with RemainAfterExit=true, started only through
`wantedBy = container@hive-forge.service`. Once it had run it stayed
`active (exited)`, so a later container start made its start a no-op: a token
re-minted without a host reboot or a unit change never reached the store.

- RemainAfterExit is dropped, so each start runs the script again.
- systemd.paths.swarm-bao-forge-publisher watches the token file in the
  container (PathChanged) and starts the publisher when the mint writes it,
  the same shape as swarm-secret-publish's path unit.
- The code comment, secrets.md and credentials.md now name both triggers;
  forge-placement pins the missing RemainAfterExit and the path unit.

Refs #4998
This commit is contained in:
atlas 2026-10-08 13:45:56 +02:00
commit 2d2d7529fc
4 changed files with 41 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/swarm/credentials.md
View file

@ -61,7 +61,7 @@ of the cell says how.
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts |
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart | | `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |

2
docs/swarm/secrets.md
View file

@ -52,7 +52,7 @@ neither is a renaming of the other.
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | | the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |

26
nix/host-modules/hive-forge/default.nix
View file

@ -56,6 +56,9 @@ let
# `forgeSecretPath` above. # `forgeSecretPath` above.
swarmControllerTokenPath = "/var/lib/forgejo/swarm-controller-token"; swarmControllerTokenPath = "/var/lib/forgejo/swarm-controller-token";
# The same file from the host, which is where its publisher reads and watches it.
swarmControllerTokenHostPath = "/var/lib/nixos-containers/hive-forge${swarmControllerTokenPath}";
# Where `swarm-bao-forge-publisher` puts it: `controllerForgeTokenLeaf` in # Where `swarm-bao-forge-publisher` puts it: `controllerForgeTokenLeaf` in
# ../swarm-bao.nix, under the KV mount. # ../swarm-bao.nix, under the KV mount.
controllerTokenStorePath = "secret/swarm/controller/swarm-controller/forge/token"; controllerTokenStorePath = "secret/swarm/controller/swarm-controller/forge/token";
@ -1057,8 +1060,12 @@ in
# It can only depend on the container being up, not on the in-container # It can only depend on the container being up, not on the in-container
# oneshot that mints the token (containers run their own systemd instance, # oneshot that mints the token (containers run their own systemd instance,
# invisible to this one by unit name), so it polls the same bounded way # invisible to this one by unit name), so it polls the same bounded way
# `hive-forge-oidc-secret` does. `wantedBy` the container, so every # `hive-forge-oidc-secret` does. It runs whenever the token can have
# container start re-publishes whatever the mint unit left there. # changed: each time the container starts (`wantedBy`), and each time the
# file is written (`systemd.paths.swarm-bao-forge-publisher` below).
#
# ⚠️ No `RemainAfterExit`. A oneshot that stays `active (exited)` turns
# every later start into a no-op, so neither trigger would run it again.
# #
# Unconditional, like the in-container unit it publishes from: forge is a # Unconditional, like the in-container unit it publishes from: forge is a
# swarm-wide singleton, so the token always gets minted and published, # swarm-wide singleton, so the token always gets minted and published,
@ -1088,7 +1095,6 @@ in
inherit (storeRetry) startLimitBurst startLimitIntervalSec; inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // { serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot"; Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "swarm-bao-forge-publisher"; SyslogIdentifier = "swarm-bao-forge-publisher";
# Longer than the 120s bounded wait below. `DefaultTimeoutStartSec` # Longer than the 120s bounded wait below. `DefaultTimeoutStartSec`
# is 90s, so without this systemd kills the unit at 90 — before it # is 90s, so without this systemd kills the unit at 90 — before it
@ -1115,7 +1121,7 @@ in
'' ''
set -euo pipefail set -euo pipefail
src=${lib.escapeShellArg "/var/lib/nixos-containers/hive-forge${swarmControllerTokenPath}"} src=${lib.escapeShellArg swarmControllerTokenHostPath}
for _ in $(seq 1 60); do for _ in $(seq 1 60); do
[ -s "$src" ] && break [ -s "$src" ] && break
@ -1157,6 +1163,18 @@ in
''; '';
}; };
# Re-publish when the mint unit writes a token, which it can do without the
# container restarting. The file is the event: the mint runs in the
# container's own systemd, which this one cannot subscribe to.
systemd.paths.swarm-bao-forge-publisher = {
description = "watch the swarm-controller's forge token for a publish";
wantedBy = [ "multi-user.target" ];
pathConfig = {
PathChanged = swarmControllerTokenHostPath;
Unit = "swarm-bao-forge-publisher.service";
};
};
# Unused: nothing writes or reads this host path, and it may hold a copy of # Unused: nothing writes or reads this host path, and it may hold a copy of
# the controller's forge token. # the controller's forge token.
systemd.tmpfiles.rules = [ "r /var/lib/hyperhive-forge/swarm-controller.token" ]; systemd.tmpfiles.rules = [ "r /var/lib/hyperhive-forge/swarm-controller.token" ];

17
nix/module-eval/forge-placement.nix
View file

@ -294,6 +294,23 @@ let
&& lib.elem "container@hive-forge.service" u.wantedBy && lib.elem "container@hive-forge.service" u.wantedBy
&& !(allLocal.systemd.services ? hive-forge-swarm-controller-token); && !(allLocal.systemd.services ? hive-forge-swarm-controller-token);
} }
{
# What makes it re-run. A oneshot left `active (exited)` makes every later
# start a no-op, so `RemainAfterExit` would silence both triggers: the
# container's start (`wantedBy`, above) and a write of the token file.
name = "the forge publisher re-runs on each container start and each write of the token file";
ok =
let
p = allLocal.systemd.paths.swarm-bao-forge-publisher;
in
!((publisher allLocal).serviceConfig.RemainAfterExit or false)
&&
p.pathConfig.PathChanged
== "/var/lib/nixos-containers/hive-forge/var/lib/forgejo/swarm-controller-token"
&& p.pathConfig.Unit == "swarm-bao-forge-publisher.service"
&& lib.elem "multi-user.target" p.wantedBy
&& !(bare.systemd.paths ? swarm-bao-forge-publisher);
}
{ {
# The absence arm: a forge host without the store has no leaf unless one # The absence arm: a forge host without the store has no leaf unless one
# is placed, and the unit says which options to set. # is placed, and the unit says which options to set.