warn when a hive host without a leaf names the otel oidc secret file
This commit is contained in:
atlas
mara
parent
144fae3919
commit
c79c41e004
3 changed files with 44 additions and 8 deletions
|
|
@ -117,7 +117,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
|
|||
| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on authelia's forwarder's client secret and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `swarm-secret-publish` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` |
|
||||
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
|
||||
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
|
||||
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/run/hive-otel-oidc/hive-<name>.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
|
||||
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-<name>.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives/<hive>/` and drop the setting |
|
||||
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
|
||||
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
|
||||
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
|
||||
|
|
|
|||
|
|
@ -51,6 +51,8 @@ let
|
|||
inherit (config.services.hyperhive.deploy.hive-controller) hiveName;
|
||||
};
|
||||
fetchedSecretPath = hiveOtelSecret.path;
|
||||
|
||||
pkiDir = import ./lib/bao-pki-dir.nix;
|
||||
in
|
||||
{
|
||||
imports = [
|
||||
|
|
@ -883,13 +885,22 @@ in
|
|||
|
||||
# The fetch unit writes `fetchedSecretPath` now, so a file named at the
|
||||
# old default is delivered by nobody and never refreshed.
|
||||
warnings = lib.optional (hiveName != null && otel.clientSecretFile == hiveOtelSecret.legacyPath) ''
|
||||
services.hyperhive.otel.clientSecretFile is ${hiveOtelSecret.legacyPath}, the path
|
||||
swarm-bao-hive-otel-oidc.service used to write. That unit now writes
|
||||
${fetchedSecretPath} and, with the option set to anything else, does
|
||||
not exist, so nothing refreshes the file. Remove the setting to fetch
|
||||
from the swarm secret store, or deliver the file yourself.
|
||||
'';
|
||||
warnings =
|
||||
lib.optional (hiveName != null && otel.clientSecretFile == hiveOtelSecret.legacyPath) ''
|
||||
services.hyperhive.otel.clientSecretFile is ${hiveOtelSecret.legacyPath}, the path
|
||||
swarm-bao-hive-otel-oidc.service used to write. That unit now writes
|
||||
${fetchedSecretPath} and, with the option set to anything else, does
|
||||
not exist, so nothing refreshes the file. Remove the setting to fetch
|
||||
from the swarm secret store, or deliver the file yourself.
|
||||
''
|
||||
++ lib.optional (hiveName != null && !hiveLeaf && otel.clientSecretFile != null) ''
|
||||
services.hyperhive.otel.clientSecretFile is ${otel.clientSecretFile} and this
|
||||
host holds no store identity (services.hyperhive.deploy.bao.clientCertFile
|
||||
and clientKeyFile). Place the hive's leaf from ${pkiDir}/hives/${hiveName}/
|
||||
on the host that runs the store, set those two options, then drop
|
||||
services.hyperhive.otel.clientSecretFile: the collector then fetches its
|
||||
secret from the swarm secret store under that leaf.
|
||||
'';
|
||||
|
||||
assertions = [
|
||||
{
|
||||
|
|
|
|||
|
|
@ -49,6 +49,10 @@ let
|
|||
deploy.authelia.enable = true;
|
||||
};
|
||||
|
||||
# Neither a leaf nor a secret file: the one shape the collector has no
|
||||
# credential for.
|
||||
hiveOtelBare = hiveWithController { };
|
||||
|
||||
hiveOtelFetch = machine: machine.systemd.services.swarm-bao-hive-otel-oidc;
|
||||
|
||||
hiveOtelPipelines = hiveOtel.services.opentelemetry-collector.settings.service.pipelines;
|
||||
|
|
@ -262,6 +266,27 @@ let
|
|||
&& !(builtins.elem "f ${run} 0400 root root -" (rules hiveOtel))
|
||||
&& warned hiveOtel;
|
||||
}
|
||||
{
|
||||
# Without a store identity, a hand-named secret file is warned to place
|
||||
# the leaf; with the leaf nothing is warned and the fetch unit exists;
|
||||
# with neither, the assertion still refuses the build. `hiveOtelBare`
|
||||
# and `hiveOtelLeaf` are the controls: a lookup that never matches
|
||||
# would pass the negated arms.
|
||||
name = "a hive host with no leaf and an operator secret file is warned to place the leaf";
|
||||
ok =
|
||||
let
|
||||
warned = m: lib.any (lib.hasInfix "holds no store identity") m.warnings;
|
||||
refused = m: lib.any (a: !a.assertion && lib.hasInfix "identity to present" a.message) m.assertions;
|
||||
in
|
||||
warned hiveOtel
|
||||
&& lib.any (lib.hasInfix "/var/lib/swarm-bao-pki/hives/h1/") hiveOtel.warnings
|
||||
&& !(warned hiveOtelLeaf)
|
||||
&& hiveOtelLeaf.systemd.services ? swarm-bao-hive-otel-oidc
|
||||
&& !(warned hiveOtelBare)
|
||||
&& refused hiveOtelBare
|
||||
&& !(refused hiveOtel)
|
||||
&& !(refused hiveOtelLeaf);
|
||||
}
|
||||
{
|
||||
# `LoadCredential` copies the file at the collector's start, so a value
|
||||
# landed later reaches it only through a restart. Retries every 30s
|
||||
|
|
|
|||
Loading…
Reference in a new issue