From a5091c58c0b77530f79dc838346d1771e85d568f Mon Sep 17 00:00:00 2001 From: atlas Date: Sun, 30 Aug 2026 18:54:35 +0200 Subject: [PATCH 1/6] deploy: which host runs the secret store is its own decision A swarm has one store, so running it is a fact about this machine and belongs beside the other deploy.* toggles rather than in the namespace every host agrees on. Attrset with an enable, not a bare bool, so a second deployment decision has somewhere to go. Asserted from swarm.enableRequiredServices alongside the other once-per-swarm services, and asserted as a mkDefault -- which is what keeps *where* the store runs a separate question from *that* it runs. Set it directly to put the store on a host of its own; a hive that does not run it is a client either way, reaching it by name. Two descriptions in the file still narrated the old swarm.* layout as something that used to be true, and the queue's claimed it belongs on the shared-services host. Both now say what holds. --- nix/host-modules/deploy.nix | 37 +++++++++++++++++--- nix/host-modules/swarm-required-services.nix | 7 ++++ 2 files changed, 39 insertions(+), 5 deletions(-) diff --git a/nix/host-modules/deploy.nix b/nix/host-modules/deploy.nix index 1645f3c7..ebc9f7fd 100644 --- a/nix/host-modules/deploy.nix +++ b/nix/host-modules/deploy.nix @@ -10,9 +10,9 @@ # deploy.* — this machine's deployment decisions. Necessarily different # on every host, because that is what a deployment is. # -# The `enable` toggles used to live under `swarm.*`, which made the -# namespace that is supposed to be identical everywhere carry the one -# thing that must differ. +# The `enable` toggles live here rather than under `swarm.*` so the +# namespace that is identical everywhere does not carry the one thing +# that must differ per host. # # Flat and named for the thing deployed — `deploy.forgejo`, not # `deploy.swarmServices.forgejo`: grouping by "swarm service" re-encodes @@ -161,6 +161,32 @@ in ''; }; + bao.enable = lib.mkOption { + type = lib.types.bool; + default = false; + example = true; + description = '' + Run the swarm's secret store in a `swarm-bao` container on this + host. A swarm has one store and it has to exist somewhere, so + this is asserted from + {option}`services.hyperhive.swarm.enableRequiredServices` + alongside the other once-per-swarm services. + + That assertion is a `mkDefault`, which is what keeps *where* the + store runs a separate question from *that* it runs: set this + directly to put the store on a host of its own, and clients + still reach it by name at + {option}`services.hyperhive.swarm.bao.domain` rather than at a + local address. + + With it off, this hive is a *client*: it still reads its own + secrets from whoever runs the store, authenticating with its own + client certificate. Every hive needs the client half; only one + runs the server half, which is why the two live in different + namespaces. + ''; + }; + authelia.enable = lib.mkOption { type = lib.types.bool; default = false; @@ -223,8 +249,9 @@ in default = false; description = '' Run the swarm's message queue in a `swarm-nats` container on this - host. A swarm has one queue, so this belongs on the same host as - the rest of the shared services. + host. A swarm has one queue, so at most one host turns this on — + but *which* host is its own decision, not necessarily the one + running the swarm's other shared services. Off by default, and off means *absent*: no container is created and nothing else in the evaluated config changes. diff --git a/nix/host-modules/swarm-required-services.nix b/nix/host-modules/swarm-required-services.nix index c69eeaff..a3c649f5 100644 --- a/nix/host-modules/swarm-required-services.nix +++ b/nix/host-modules/swarm-required-services.nix @@ -90,4 +90,11 @@ in # hive that is not the service host is a *client* of it, not a second one. config.services.hyperhive.deploy.victorialogs.enable = lib.mkDefault swarmCfg.enableRequiredServices; + + # The secret store. Once per swarm and optional, so it belongs to the + # same switch: a hive that does not run it is a *client*, reading its + # own secrets from whoever does. `mkDefault` is what keeps the store + # placeable on a host of its own — it can be set directly here and + # turned off wherever this switch happens to be on. + config.services.hyperhive.deploy.bao.enable = lib.mkDefault swarmCfg.enableRequiredServices; } From fa540ca1ec90b76332a776fa5acd0f0f49e3c779 Mon Sep 17 00:00:00 2001 From: atlas Date: Sun, 30 Aug 2026 18:54:36 +0200 Subject: [PATCH 2/6] swarm-bao: run the swarm's secret store in a container OpenBao in a swarm-bao nixos-container, unsealed by the host TPM through the built-in pkcs11 seal, with shamir as the documented opt-out for hardware that has no TPM. The seal guard asks the package whether it was built with an HSM rather than reading its version, and falls back to the version cliff only when a package advertises no tags -- it fails closed. Every listener serves TLS: loopback unconditionally, because the host running the store is always one of its readers, plus whatever extraListenAddresses names, because which network the other hives share is a deployment fact. Client-certificate verification turns on only when clientCaFile says what to trust. The store's identity is an input, and nothing in THIS module fills it in. Service-to-store mTLS is a separate trust domain from the gateway's HTTPS certificates and from both CAs in this tree: a store must not take its identity from an authority it will itself distribute, or it cannot come up before the thing that issues it. The module asserts when the paths are unset rather than reaching for whichever CA happens to be wired; what supplies them on a self-contained deploy is the next commit. The leaf is copied to the container's state dir by a host unit rather than bind-mounted: nixos-container refuses to start when a bind source is missing, and a directory holding a leaf usually holds the CA's key beside it. --- nix/host-modules/default.nix | 1 + nix/host-modules/swarm-bao.nix | 542 +++++++++++++++++++++++++++++++++ 2 files changed, 543 insertions(+) create mode 100644 nix/host-modules/swarm-bao.nix diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index 8c180f4f..7683d730 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -24,6 +24,7 @@ ./hive-tls.nix ./otel.nix ./swarm-authelia.nix + ./swarm-bao.nix ./swarm-ca.nix ./swarm-nats.nix ./swarm-controller.nix diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix new file mode 100644 index 00000000..37802f39 --- /dev/null +++ b/nix/host-modules/swarm-bao.nix @@ -0,0 +1,542 @@ +# The swarm's secret store: one OpenBao for the whole swarm, in a +# `swarm-bao` nixos-container. +# +# Today every credential in ./swarm-*.nix is minted where it is read or copied +# there by a delivery unit (docs/swarm/secrets.md), which ties each secret's +# lifetime to its container's. +# +# ⚠️ It authenticates hive clients with a CLIENT CERTIFICATE, not with the +# swarm's SSO — a boot-order fact rather than a preference. This store holds +# authelia's own OIDC client secret, so a client that had to obtain an authelia +# token first could never start from cold. `swarm-nats` can lean on authelia +# precisely because it does not store authelia's credentials. +# +# ⚠️ NO GATEWAY VHOST, and unlike `swarm-nats` that is not because this speaks +# a non-HTTP protocol. It speaks HTTPS, so nginx *could* front it: **the client +# certificate IS the authentication**, and a terminating proxy strips it, +# leaving bao seeing nginx as the client for every hive in the swarm — one +# identity where there must be many. Reach is loopback plus whatever +# `deploy.bao.extraListenAddresses` names. +# +# ⚠️ THIS MODULE HAS NO OPINION ABOUT WHERE THE STORE'S IDENTITY COMES FROM. +# A store must not take its certificates from an authority it will itself +# distribute: reach the store to get the CA material, need a cert from that CA +# to reach the store. Service↔store mTLS is therefore its own trust domain, +# separate from the gateway's HTTPS certificates and from both CAs in this +# tree. The cert paths are inputs with no defaults and nothing here fills them +# in; whatever comes to mint that identity is what they will point at. +{ + pkgs, + lib, + config, + ... +}: +let + cfg = config.services.hyperhive.swarm.bao; + hyperhiveCfg = config.services.hyperhive; + deployCfg = hyperhiveCfg.deploy; + baoDeploy = deployCfg.bao; + networkCfg = hyperhiveCfg.network; + swarmDomain = hyperhiveCfg.swarm.domain; + + # Two names for one location. `stateDir` is where openbao writes inside the + # container — upstream's own default, kept so its documentation matches. The + # host path is a sibling of the other swarm services' state rather than a + # path inside the container's tree, so `nixos-container destroy` cannot take + # the swarm's secrets with it. + stateDir = "/var/lib/openbao"; + hostStateDir = "/var/lib/swarm-bao"; + + # The PIN is deliberately absent here. It arrives as `BAO_HSM_PIN` from an + # EnvironmentFile the provisioning unit writes, because a value interpolated + # into a nix expression renders world-readable into the store. + # + # With no seal stanza openbao falls back to Shamir, so this attrset being + # empty is the difference between a store that unseals itself and one that + # needs a human after every restart. + # The PKCS11 token store and its PINs live on the HOST and are bind-mounted + # in. Losing them loses the sealed store, so they are a host-level fact an + # operator can back up — the same reasoning as the raft data below. + tokenStoreDir = "/var/lib/swarm-bao-token"; + pinEnvFile = "${tokenStoreDir}/pin.env"; + + sealSettings = lib.optionalAttrs (baoDeploy.seal == "pkcs11") { + seal.pkcs11 = { + lib = "${pkgs.tpm2-pkcs11}/lib/libtpm2_pkcs11.so"; + token_label = "swarm-bao"; + key_label = "swarm-bao-seal"; + }; + }; + + # Total on a null swarm domain for the same reason every sibling module is: + # the required-domain assertion in hive-network.nix should be what an operator + # sees, not a coercion error from here. + domainBase = if swarmDomain == null then "invalid" else swarmDomain; + + # Where the leaf lands for openbao to read. Inside the container, because + # `hostStateDir` is already bind-mounted at `stateDir` — so the delivery + # below needs no second mount, and nothing has to bind `tls.stateDir`, + # which holds the hive CA's private key. + serverCertPath = "${stateDir}/server.pem"; + serverKeyPath = "${stateDir}/server-key.pem"; + + # The host-side sources, verbatim from the options — no fallback, because a + # fallback is exactly the CA opinion this module must not hold. The units + # below only exist when both are set (see `haveServerTls`), so these are + # never forced while null. + serverCertSrc = baoDeploy.serverCertFile; + serverKeySrc = baoDeploy.serverKeyFile; + + # Both or neither: a certificate without its key configures a listener that + # cannot start, and the failure would surface as openbao refusing to boot + # rather than as the missing setting it is. + haveServerTls = baoDeploy.serverCertFile != null && baoDeploy.serverKeyFile != null; + + # Every listener serves the same identity: they differ in which address + # they answer on, not in who they are. Client verification is separate and + # optional — a store with no `clientCaFile` still serves TLS, it just does + # not authenticate the far end, which is the honest rendering of "nobody + # has said what to trust yet". + listenerTls = { + tls_cert_file = serverCertPath; + tls_key_file = serverKeyPath; + } + // lib.optionalAttrs (baoDeploy.clientCaFile != null) { + tls_client_ca_file = clientCaPath; + tls_require_and_verify_client_cert = true; + }; + + clientCaPath = "${stateDir}/client-ca.pem"; + + extraListeners = lib.listToAttrs ( + lib.imap1 ( + i: addr: + lib.nameValuePair "extra-${toString i}" ( + { + type = "tcp"; + address = "${addr}:${toString cfg.port}"; + } + // listenerTls + ) + ) baoDeploy.extraListenAddresses + ); + + # Loopback is unconditional and everything else is declared, which is not + # symmetry for its own sake: + # + # A reader on this host reaches the store through loopback, and the host + # running the store is always one of its readers — so loopback is a property + # of what the store IS, not of where it sits. Every other address depends on + # which network the hives that read it share, and that is a deployment fact. + # Bind only loopback and no remote hive can reach the store; bind only a + # shared-network address and an all-local swarm cannot reach its own. + # + # Neither is a superset of the other, which is why this is not one address + # with a conditional value. + listeners = { + loopback = { + type = "tcp"; + address = "127.0.0.1:${toString cfg.port}"; + } + // listenerTls; + } + // extraListeners; +in +{ + # One service, two namespaces, and the split decides who may set what. + # + # `deploy.bao.*` is what the host RUNNING the store decides: whether to run + # it (`enable`, declared in ./deploy.nix with its siblings), which build, + # how the root key is sealed, what it listens on. None of it means anything + # on a host that only reads secrets. + # + # `swarm.bao.*` below is what every host in the swarm has to agree on — the + # name the store answers to, its port, its container. A host that is purely + # a *client* needs all of that, because it is how the client finds the store. + options.services.hyperhive.deploy.bao = { + package = lib.mkOption { + type = lib.types.package; + default = pkgs.openbao; + defaultText = lib.literalExpression "pkgs.openbao"; + description = '' + OpenBao package to run. + + ⚠️ An assertion below refuses 2.7.0 or newer, which drops the + built-in PKCS11 seal. + ''; + }; + + seal = lib.mkOption { + type = lib.types.enum [ + "pkcs11" + "shamir" + ]; + default = "pkcs11"; + example = "shamir"; + description = '' + How the store's root key is sealed. + + `pkcs11` is the default and binds the key to the host's TPM: the store + unseals itself at boot, and an attacker with the disk does not get the + secrets. `shamir` is openbao's own default — unseal keys held by + whoever ran `bao operator init`, entered by hand after every restart — + and is the honest choice for a host with no TPM. + + ⚠️ This is a **declaration**, and nothing at evaluation time can check + it: nix runs on the build machine and cannot see the target's TPM. + Saying `pkcs11` on a host without one fails at activation, when the + provisioning unit cannot create the token. That is deliberate — a + store that comes up sealed by software while the config says hardware + is weaker than it reads, and silently so. + ''; + }; + + serverCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao/server.pem"; + description = '' + Certificate the store serves, covering + {option}`services.hyperhive.swarm.bao.domain`. + + No default, and this module deliberately does not know what could + provide one — for the same reason + {option}`services.hyperhive.deploy.bao.clientCaFile` doesn't. The + deployment names the file; the store never reaches for an authority. + + A path, never a value. + ''; + }; + + serverKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao/server-key.pem"; + description = '' + Private key for {option}`services.hyperhive.deploy.bao.serverCertFile`. + Both or neither — a certificate with no key is a listener that cannot + start. + ''; + }; + + clientCaFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-ca/root.pem"; + description = '' + Authority the store validates hive **client** certificates against. + + Deliberately has no default, and does not reach for the hive CA: + the hive CA is a future *consumer* of the store, so a store that + authenticated against it could not come up before the thing it + issues. It is a value someone points at — the swarm root for a + swarm that runs one, an operator's own CA otherwise. + + `null` leaves client-certificate verification off, which is only + appropriate where something else authenticates the connection. + ''; + }; + + extraListenAddresses = lib.mkOption { + type = lib.types.listOf lib.types.str; + default = [ ]; + example = [ "10.100.0.1" ]; + description = '' + Addresses the store listens on **in addition to loopback**, each on + {option}`services.hyperhive.swarm.bao.port`. + + Loopback is unconditional and not listed here: the host running the + store is always one of its readers. Every other address depends on + which network the reading hives share with this one, and that is a + deployment fact no other module's config can be read to infer — a + swarm meshed over wireguard names its mesh address, one on a trusted + LAN names that interface, and an all-local swarm names nothing at all. + + Addresses only, no port: a store reachable on two ports is a + misconfiguration rather than a topology. + ''; + }; + }; + + options.services.hyperhive.swarm.bao = { + machine = lib.mkOption { + type = lib.types.str; + readOnly = true; + default = "swarm-bao"; + description = '' + Container name. Read-only: the name appears in host paths and in + `machinectl`, so it is a fact other modules may read rather than a knob. + ''; + }; + + domain = lib.mkOption { + type = lib.types.str; + default = "bao.${domainBase}"; + defaultText = lib.literalExpression ''"bao.''${services.hyperhive.swarm.domain}"''; + description = '' + Name the store is reached on. A **sibling** of the swarm's other + service names, not a child of any hive domain: an authority whose + `nameConstraints` permit one hive's domain cannot issue for a sibling + of it, so the shape of this name decides which authorities could ever + sign for the store. That is a property of the name, not a choice of + issuer — this module makes no such choice. + ''; + }; + + port = lib.mkOption { + type = lib.types.port; + default = 8200; + description = '' + TCP port the store listens on. Upstream's own default, kept so an + operator reading OpenBao documentation finds what they expect. + + Swarm-wide because a client has to know it to reach the store, and + the same port on every listener: which *addresses* the store answers + on is the running host's business + ({option}`services.hyperhive.deploy.bao.extraListenAddresses`), but + which port it answers on is something the whole swarm agrees. + ''; + }; + + }; + + # ⚠️ Gated on `deploy.bao.enable`, and that is load-bearing rather than + # tidiness: an unconditional `config` block would evaluate the seal + # assertion on EVERY hive, so a hive that runs no secret store at all + # would fail to build the day nixpkgs moves openbao past 2.7.0. A check + # about running this service has no business firing where it is not run. + config = lib.mkMerge [ + # Assertions sit in their own arm, gated only on running the store, so + # they still fire when the cert paths are unset — the arm below is not + # evaluated in that case, and an assertion that disappears exactly when + # its subject is broken would be worse than none. + (lib.mkIf (hyperhiveCfg.enable && deployCfg.bao.enable) { + assertions = [ + { + assertion = lib.versionOlder baoDeploy.package.version "2.7.0"; + message = '' + The swarm secret store needs openbao older than 2.7.0 (this is + ${baoDeploy.package.version}). 2.7.0 moves the PKCS11 seal out of the + distribution into a plugin nixpkgs does not package, so the store + would come up sealed by software without saying so. + See https://openbao.org/community/deprecation/ + ''; + } + { + assertion = haveServerTls; + message = '' + The swarm secret store has no server certificate: set both + services.hyperhive.deploy.bao.serverCertFile and .serverKeyFile. + + Nothing defaults them, on purpose — a store must not take its + identity from an authority it will itself distribute, and + service-to-store mTLS is a separate trust domain from the + gateway's certificates and from either CA in this tree. + ''; + } + ]; + }) + + (lib.mkIf (hyperhiveCfg.enable && deployCfg.bao.enable && haveServerTls) { + # Provisions the TPM-backed token the seal above names. One-shot and + # idempotent on ABSENCE, never on content: regenerating a PIN would + # orphan an already-sealed store, so a rebuild must not rotate it. + # + # ⚠️ Untestable without a TPM, and the failure is deliberately at + # activation — nix evaluates on the build machine and cannot see the + # target's hardware, so `seal = "pkcs11"` is a declaration this unit + # either makes true or fails on. + # The store's server certificate, delivered rather than bind-mounted. + # + # ⚠️ A copy, for three separate reasons — the last one is the one that + # matters most and is the least obvious: + # 1. `nixos-container` refuses to start when a bind source is missing, + # and a certificate minted on this same boot does not exist yet when + # the container is ordered. Same trap `hostClientSecretDir` + # documents in ./swarm-authelia.nix. Ordering against whatever + # mints it belongs with whatever named the path, not here. + # 2. `hostStateDir` is already mounted at `stateDir`, so a copy needs + # no second mount. + # 3. A directory holding a leaf usually holds the CA's private key + # beside it. Binding that directory to reach one file inside it + # would hand the container authority to mint any name that CA can — + # which is why this takes a path to a FILE and copies it. + systemd.services.swarm-bao-certs = { + description = "deliver the swarm secret store's server certificate"; + before = [ "container@${cfg.machine}.service" ]; + requiredBy = [ "container@${cfg.machine}.service" ]; + path = [ pkgs.coreutils ]; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + }; + script = '' + set -euo pipefail + install -d -m 0700 ${hostStateDir} + + # Fail loudly rather than start a store that cannot serve. The path + # is configured, so a missing file means whatever was supposed to + # produce it did not run or failed — either way this is where it is + # cheapest to notice. Otherwise it surfaces at the TLS handshake, + # several layers from the setting that caused it. + for f in ${lib.escapeShellArg serverCertSrc} ${lib.escapeShellArg serverKeySrc}; do + if [ ! -s "$f" ]; then + echo "swarm-bao has no server certificate: $f is missing or empty." >&2 + echo "That path comes from deploy.bao.serverCertFile/serverKeyFile." >&2 + exit 1 + fi + done + + install -m 0644 ${lib.escapeShellArg serverCertSrc} ${hostStateDir}/server.pem + install -m 0600 ${lib.escapeShellArg serverKeySrc} ${hostStateDir}/server-key.pem + '' + + lib.optionalString (baoDeploy.clientCaFile != null) '' + + if [ ! -s ${lib.escapeShellArg baoDeploy.clientCaFile} ]; then + echo "deploy.bao.clientCaFile names ${baoDeploy.clientCaFile}, which is missing or empty." >&2 + exit 1 + fi + install -m 0644 ${lib.escapeShellArg baoDeploy.clientCaFile} ${hostStateDir}/client-ca.pem + ''; + }; + + systemd.services.swarm-bao-token = lib.mkIf (baoDeploy.seal == "pkcs11") { + description = "provision the swarm secret store's TPM-backed PKCS11 token"; + before = [ "container@${cfg.machine}.service" ]; + requiredBy = [ "container@${cfg.machine}.service" ]; + path = [ + pkgs.openssl + pkgs.tpm2-pkcs11 + pkgs.tpm2-tools + ]; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + }; + script = '' + set -euo pipefail + install -d -m 0700 ${tokenStoreDir} + # Required whenever the store is not at its default location, or the + # library cannot find the token the seal asks for. + export TPM2_PKCS11_STORE=${tokenStoreDir} + + # Absence is the only trigger. `openssl rand` is the same generator + # the grafana admin key uses; the value never passes through a nix + # expression, which would render it world-readable into the store. + for p in so-pin user-pin; do + if [ ! -e ${tokenStoreDir}/$p ]; then + ( umask 077; openssl rand -hex 16 > ${tokenStoreDir}/$p ) + chmod 0400 ${tokenStoreDir}/$p + fi + done + + if [ ! -e ${tokenStoreDir}/tpm2_pkcs11.sqlite3 ]; then + pid=$(tpm2_ptool init --path ${tokenStoreDir} | sed -n 's/.*id: //p') + tpm2_ptool addtoken --path ${tokenStoreDir} --pid="$pid" \ + --label=swarm-bao \ + --sopin="$(cat ${tokenStoreDir}/so-pin)" \ + --userpin="$(cat ${tokenStoreDir}/user-pin)" + # AES rather than RSA on purpose: openbao discussion 1826 + # reports an RSA keypair here yielding duplicate labels, so + # `bao operator init` fails with "got more than 1 key for the + # label" and then CKR_GENERAL_ERROR. The seal supports AES-GCM. + tpm2_ptool addkey --path ${tokenStoreDir} --label=swarm-bao \ + --userpin="$(cat ${tokenStoreDir}/user-pin)" \ + --algorithm=aes256 --key-label=swarm-bao-seal + fi + + ( umask 077; printf 'BAO_HSM_PIN=%s\n' "$(cat ${tokenStoreDir}/user-pin)" > ${pinEnvFile} ) + chmod 0400 ${pinEnvFile} + ''; + }; + + containers.${cfg.machine} = { + autoStart = true; + ephemeral = false; + # Shared host netns, like every sibling swarm container. Unlike them the + # gateway is NOT the client here (see the no-vhost note at the top), so + # sharing the netns is what lets the store bind the host's own addresses + # rather than a convenience for nginx. + privateNetwork = false; + + # Raft state outlives the container. `ephemeral = false` keeps the + # container's own /var, but a bind makes the store's data a host-level + # fact an operator can back up and a `nixos-container destroy` cannot + # take with it — which for the swarm's secrets is the difference between + # a rebuild and an outage. + bindMounts = { + ${stateDir} = { + hostPath = hostStateDir; + isReadOnly = false; + }; + } + // lib.optionalAttrs (baoDeploy.seal == "pkcs11") { + # Writable: the library keeps its sqlite store here, and the seal + # reads the token through it on every unseal. + ${tokenStoreDir} = { + hostPath = tokenStoreDir; + isReadOnly = false; + }; + }; + + # The seal talks to the TPM through the kernel's resource manager, so + # the device has to cross the container boundary or the store cannot + # unseal itself — which is the whole point of pkcs11 over shamir. + allowedDevices = lib.optionals (baoDeploy.seal == "pkcs11") [ + { + node = "/dev/tpmrm0"; + modifier = "rw"; + } + ]; + + config = + { ... }: + { + imports = [ + (import ./swarm-container-resolver.nix { + inherit (networkCfg) bridgeIp; + dnsConsumers = [ "openbao.service" ]; + }) + ]; + + system.stateVersion = "26.05"; + + # Shares the host netns, so its own firewall.service would rewrite + # the HOST ruleset at every boot. The host firewall owns filtering. + networking.firewall.enable = false; + # The resolver unit imported above owns /etc/resolv.conf; leaving + # resolvconf on would let host-tracking regenerate it empty. + networking.resolvconf.enable = lib.mkForce false; + + services.openbao = { + enable = true; + package = baoDeploy.package; + settings = { + listener = listeners; + storage.raft.path = stateDir; + } + // sealSettings; + }; + + # The PIN reaches openbao as an environment variable read from a + # 0400 file the provisioning unit wrote — never as a value in this + # expression, which would render it world-readable into the store. + # `TPM2_PKCS11_STORE` is required because the store is not at the + # library's default location. + systemd.services.openbao.serviceConfig = lib.mkIf (baoDeploy.seal == "pkcs11") { + EnvironmentFile = pinEnvFile; + Environment = [ "TPM2_PKCS11_STORE=${tokenStoreDir}" ]; + }; + + # ⚠️ Upstream sets `restartIfChanged = false` on this unit, on + # purpose: a restart SEALS the store and disconnects every client. + # So a change to the settings above does NOT take effect on + # `nixos-rebuild switch` — it lands in the config file and waits. + # Restarting is an operator action with an unseal on the far side of + # it, which is why nothing here tries to be clever about it. + }; + }; + }) + + ]; +} From 9bbc74ee517e6c8b665726b46bda79d848e20943 Mon Sep 17 00:00:00 2001 From: atlas Date: Sun, 30 Aug 2026 18:54:36 +0200 Subject: [PATCH 3/6] glue-bao-tls: mint the store an identity it can hold before it is up A CA that signs exactly two things -- the store's server certificate and the client certificate of whoever reads from it -- and distributes nothing. Not the hive CA, not the swarm CA: the store will distribute both, and an authority you must already hold a certificate from cannot be one the store hands out. Not the gateway's HTTPS material either, self-signed or ACME; that is a different trust domain with a different audience. The minting lives here rather than in swarm-bao.nix because it is an opinion about where the store's identity comes from -- the most consequential one available. The service serves what it is handed. A deployment with a real internal CA drops this file and names its own paths in serverCertFile / clientCaFile, and nothing in the store changes. Ordering simplifies too: with the unit and its consumer in one module, before/requiredBy is internal rather than a cross-module fact. Idempotent on ABSENCE only. Re-issuing the CA invalidates every client certificate already trusting it, so a rebuild that refreshed it would lock out every reader in the swarm at once. --- nix/host-modules/default.nix | 1 + nix/host-modules/glue-bao-tls.nix | 119 ++++++++++++++++++++++++++++++ 2 files changed, 120 insertions(+) create mode 100644 nix/host-modules/glue-bao-tls.nix diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index 7683d730..bf7c2fa0 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -23,6 +23,7 @@ ./hive-priv.nix ./hive-tls.nix ./otel.nix + ./glue-bao-tls.nix ./swarm-authelia.nix ./swarm-bao.nix ./swarm-ca.nix diff --git a/nix/host-modules/glue-bao-tls.nix b/nix/host-modules/glue-bao-tls.nix new file mode 100644 index 00000000..eab900b2 --- /dev/null +++ b/nix/host-modules/glue-bao-tls.nix @@ -0,0 +1,119 @@ +# Glue: give the secret store a PKI of its own, and point it at it. +# +# ONE PAIRING PER FILE — `glue--.nix`. A single module holding +# every co-location default becomes the file nobody dares change, because a +# reader cannot tell which of its rules their deployment is subject to. Each +# of these should be deletable on its own, and deleting this one leaves a +# store that takes operator-provided certificates and nothing else. +# +# ⚠️ Why the PKI lives HERE and not in ./swarm-bao.nix: the store must have no +# opinion about where its identity comes from. Minting is an opinion — the +# most consequential one available — so it belongs to the glue that decides +# this deployment self-signs, not to the service that merely serves what it is +# handed. A deployment with a real internal CA drops this file and names its +# own paths; nothing in the store changes. +# +# ⚠️ Not the hive CA and not the swarm CA. The store will eventually +# distribute both, and an authority you must already hold a certificate from +# cannot be one the store hands out — reach the store to get the CA material, +# need a cert from that CA to reach the store. This CA signs exactly two +# things and distributes nothing, so it cannot enter that cycle. +# +# ⚠️ Files like this are the only place a `deploy.` value may derive from +# a `deploy..enable`. Everywhere else that is forbidden. The exception +# earns itself: the derivation happens either way, and the alternative is +# having it spread through the service modules where it is invisible. +# +# Everything is `mkDefault`. An operator naming their own paths wins. +{ + pkgs, + lib, + config, + ... +}: +let + hyperhiveCfg = config.services.hyperhive; + deployCfg = hyperhiveCfg.deploy; + cfg = hyperhiveCfg.swarm.bao; + + # Host-side, outside the container's tree, for the same reason the raft data + # is: `nixos-container destroy` must not take it. Losing the CA key means + # re-issuing every client certificate in the swarm. + pkiDir = "/var/lib/swarm-bao-pki"; + + # What a reader calls itself to the store. The hive's name, because a bao + # cert-auth role matches on the CN — this is an interface, not a label. + clientCn = if hyperhiveCfg.hiveName != null then hyperhiveCfg.hiveName else cfg.domain; + + # $1 dir $2 basename $3 CN $4 SAN or "" $5 EKU + signLeaf = pkgs.writeShellScript "swarm-bao-sign-leaf" '' + set -euo pipefail + d="$1"; base="$2"; cn="$3"; sans="$4"; eku="$5" + csr="$(mktemp "$d/$base.csr.XXXXXX")" + ext="$(mktemp "$d/$base.ext.XXXXXX")" + trap 'rm -f "$csr" "$ext"' EXIT + + openssl req -newkey rsa:4096 -nodes -sha256 \ + -keyout "$d/$base-key.pem" -out "$csr" -subj "/CN=$cn" + { + [ -n "$sans" ] && printf 'subjectAltName=%s\n' "$sans" + printf 'basicConstraints=critical,CA:FALSE\n' + printf 'keyUsage=critical,digitalSignature,keyEncipherment\n' + printf 'extendedKeyUsage=%s\n' "$eku" + } > "$ext" + openssl x509 -req -in "$csr" -CA "$d/ca.pem" -CAkey "$d/ca-key.pem" \ + -CAcreateserial -days 3650 -sha256 -extfile "$ext" -out "$d/$base.pem" + chmod 0600 "$d/$base-key.pem" + chmod 0644 "$d/$base.pem" + ''; +in +{ + config = lib.mkIf (hyperhiveCfg.enable && deployCfg.bao.enable) { + services.hyperhive.deploy.bao = { + serverCertFile = lib.mkDefault "${pkiDir}/server.pem"; + serverKeyFile = lib.mkDefault "${pkiDir}/server-key.pem"; + clientCaFile = lib.mkDefault "${pkiDir}/ca.pem"; + }; + + # Idempotent on ABSENCE, never on content. Re-issuing the CA invalidates + # every client certificate already trusting it, so a rebuild that + # "refreshed" it would lock every reader in the swarm out at once — the + # same rule the store's TPM PIN unit follows, for a sharper reason. + systemd.services.swarm-bao-pki = { + description = "mint the swarm secret store's own CA and leaves"; + before = [ "swarm-bao-certs.service" ]; + requiredBy = [ "swarm-bao-certs.service" ]; + path = [ + pkgs.openssl + pkgs.coreutils + ]; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + }; + script = '' + set -euo pipefail + install -d -m 0700 ${pkiDir} + + if [ ! -s ${pkiDir}/ca.pem ]; then + openssl req -x509 -newkey rsa:4096 -nodes -sha256 -days 3650 \ + -keyout ${pkiDir}/ca-key.pem -out ${pkiDir}/ca.pem \ + -subj "/CN=swarm-bao-ca ${cfg.domain}" \ + -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \ + -addext "keyUsage=critical,keyCertSign,cRLSign" + chmod 0600 ${pkiDir}/ca-key.pem + chmod 0644 ${pkiDir}/ca.pem + fi + + # The store's own identity, and the identity of a reader on this host. + # A reader elsewhere gets its leaf from this CA out of band — that is + # what makes the store reachable from another machine at all, and why + # the CA is a file rather than a service. + [ -s ${pkiDir}/server.pem ] || ${signLeaf} ${pkiDir} server \ + ${lib.escapeShellArg cfg.domain} ${lib.escapeShellArg "DNS:${cfg.domain}"} serverAuth + [ -s ${pkiDir}/client.pem ] || ${signLeaf} ${pkiDir} client \ + ${lib.escapeShellArg clientCn} "" clientAuth + ''; + }; + }; +} From 0e5eb1e8d61e05256eea5e647b4de179fa51ca25 Mon Sep 17 00:00:00 2001 From: atlas Date: Sun, 30 Aug 2026 18:54:36 +0200 Subject: [PATCH 4/6] glue-matrix-bao-token: the store's first reader Fetches an opaque 32-byte value and writes it where hive-matrix.nix already looks, so the homeserver never learns the store exists. Chosen over authelia's OIDC secret deliberately: that one needs a .secret AND a matching .digest, so shipping it first would debug 'can a reader authenticate and get bytes back' and 'did we write authelia's file format right' at once, with an SSO outage as the failure mode. Here the failure is narrow -- new agent accounts cannot be provisioned, existing ones untouched. Every failure path keeps the local token: no such key, sealed store, unreachable store, empty value. The activation script's mint-if-absent is untouched, so a hive with no store behaves exactly as it does today. matrixMachine is a literal because hive-matrix.nix declares no `machine` option -- `matrixCfg.machine` parses cleanly and fails at module-system resolution, which is the kind of error only reading the target module catches. --- nix/host-modules/default.nix | 1 + nix/host-modules/glue-matrix-bao-token.nix | 103 +++++++++++++++++++++ 2 files changed, 104 insertions(+) create mode 100644 nix/host-modules/glue-matrix-bao-token.nix diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index bf7c2fa0..0ff92c8e 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -24,6 +24,7 @@ ./hive-tls.nix ./otel.nix ./glue-bao-tls.nix + ./glue-matrix-bao-token.nix ./swarm-authelia.nix ./swarm-bao.nix ./swarm-ca.nix diff --git a/nix/host-modules/glue-matrix-bao-token.nix b/nix/host-modules/glue-matrix-bao-token.nix new file mode 100644 index 00000000..a89db5b3 --- /dev/null +++ b/nix/host-modules/glue-matrix-bao-token.nix @@ -0,0 +1,103 @@ +# Glue: the matrix registration token comes from the secret store. +# +# The store's first reader, and deliberately a small one. It fetches an opaque +# 32-byte value and writes it where ./hive-matrix.nix already looks — the +# homeserver never learns the store exists, and its config is unchanged. +# +# ⚠️ Why this credential first. It has no second file and no format: authelia's +# OIDC secret needs a `.secret` *and* a matching `.digest`, so shipping that +# one first would debug "can a reader authenticate and get bytes back" and +# "did we write authelia's file format right" at the same time, with an SSO +# outage as the failure mode. Here the failure is narrow — new agent accounts +# cannot be provisioned, existing ones are untouched, nothing crash-loops. +# +# ⚠️ The fallback is today's behaviour, not a new one. `hive-matrix.nix`'s +# activation script still mints a token when the file is absent; this unit +# overwrites it with the swarm's copy when the store has one. A store that is +# empty or unreachable leaves a working hive with a local token. +# +# 📌 LIMIT, stated rather than hidden: this gates on the store running HERE. +# A hive reading from a store on another machine needs the same unit with that +# machine's address and a client leaf issued out of band — the mechanism is +# identical, only `-address` and the cert's provenance differ. Deferred until +# there is a second hive to test it against, rather than shipped untested. +{ + pkgs, + lib, + config, + ... +}: +let + hyperhiveCfg = config.services.hyperhive; + deployCfg = hyperhiveCfg.deploy; + baoCfg = hyperhiveCfg.swarm.bao; + matrixCfg = hyperhiveCfg.swarm.matrix; + + # Owned by ./glue-bao-tls.nix, which mints them. Named here rather than + # shared through a `let`: a cross-module binding would make these two files + # one file with a gap in the middle, and the whole point of a glue module is + # that it can be deleted on its own. + pkiDir = "/var/lib/swarm-bao-pki"; + + # Where the token lives in the store. A path, not a convention to guess at: + # whoever writes it and whoever reads it must agree, and the agreement + # belongs in one visible place. + tokenPath = "secret/swarm/matrix/registration-token"; + + # A literal, not an option — ./hive-matrix.nix names its container + # `containers.hive-matrix` directly and declares no `machine` to derive it + # from, which the trust-bundle call in that file already says out loud. + # ⚠️ `matrixCfg.machine` parses fine and fails at module-system resolution, + # so this is the kind of mistake only reading the target module catches. + matrixMachine = "hive-matrix"; +in +{ + config = lib.mkIf (hyperhiveCfg.enable && deployCfg.bao.enable && deployCfg.matrix.enable) { + systemd.services.swarm-bao-matrix-token = { + description = "fetch the matrix registration token from the swarm secret store"; + after = [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" + ]; + wants = [ "container@${baoCfg.machine}.service" ]; + requires = [ "swarm-bao-pki.service" ]; + before = [ "container@${matrixMachine}.service" ]; + wantedBy = [ "container@${matrixMachine}.service" ]; + path = [ + deployCfg.bao.package + pkgs.coreutils + ]; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CACERT = "${pkiDir}/ca.pem"; + BAO_CLIENT_CERT = "${pkiDir}/client.pem"; + BAO_CLIENT_KEY = "${pkiDir}/client-key.pem"; + }; + script = '' + set -euo pipefail + + # A sealed or uninitialised store answers on the port and times out on + # every read, so "the store is up" is not the same as "the store can + # answer" -- bound the wait rather than hanging the boot behind it. + if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>/dev/null)"; then + echo "swarm-bao holds no ${tokenPath}, or is sealed/unreachable." >&2 + echo "Keeping the token hive-matrix already has." >&2 + exit 0 + fi + + if [ -z "$token" ]; then + echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2 + exit 0 + fi + + umask 077 + printf '%s\n' "$token" > ${lib.escapeShellArg (toString matrixCfg.registrationTokenFile)} + chmod 0600 ${lib.escapeShellArg (toString matrixCfg.registrationTokenFile)} + ''; + }; + }; +} From aa784a746d24e8582bfb52c06db2a7c9e4958060 Mon Sep 17 00:00:00 2001 From: atlas Date: Sun, 30 Aug 2026 18:54:36 +0200 Subject: [PATCH 5/6] module-eval: the seal, the store's identity, and the reader pairing The seal choice gates the TPM machinery, in both directions. The store's three certificate paths are a function of the glue: present when the store is deployed, and an operator's own path beats the mkDefault. The absence arm this replaces asserted the store defaults NO path -- true while nothing supplied one, false the moment something did. The reader unit is a function of the PAIRING: present with store and homeserver together, absent with the store alone. That pair is what makes it glue rather than a feature of either side, and it is the case that fails if a later change wires the reader to one of them. --- nix/module-eval.nix | 71 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/nix/module-eval.nix b/nix/module-eval.nix index b07b80da..67b8b5e5 100644 --- a/nix/module-eval.nix +++ b/nix/module-eval.nix @@ -71,6 +71,26 @@ let bare = hive { }; withCi = hive { deploy.forgejo.ci.enable = true; }; + baoPkcs11 = hive { + deploy.bao.enable = true; + deploy.bao.seal = "pkcs11"; + }; + baoShamir = hive { + deploy.bao.enable = true; + deploy.bao.seal = "shamir"; + }; + baoExplicitCerts = hive { + deploy.bao.enable = true; + deploy.bao.serverCertFile = "/etc/pki/bao.pem"; + deploy.bao.serverKeyFile = "/etc/pki/bao-key.pem"; + }; + # The store and a service that reads from it, versus the store alone. The + # pair is what makes the reader's absence arm mean anything. + baoWithMatrix = hive { + deploy.bao.enable = true; + deploy.matrix.enable = true; + }; + # A priority collision is a property of the *option*, not # of the merged value's interior — nix throws the moment the value is # demanded at all, so `seq`-ing each `serviceConfig` value to WHNF is @@ -134,6 +154,57 @@ let name = "the CI container's unit definitions merge without a priority collision"; ok = forceCiServiceConfigs; } + { + # The store's seal is spread over five gates — the stanza, the + # provisioning unit, a bind mount, a device and an EnvironmentFile. + # Rendering only some of them is the dangerous state: a store that + # says hardware-backed and seals with a software key, which no + # assertion can catch because every value is individually valid. + name = "a shamir store renders no TPM provisioning unit"; + ok = !(baoShamir.systemd.services ? swarm-bao-token); + } + { + # Presence control for the case above. Without it, a typo in the + # option name would satisfy the absence arm forever. + name = "a pkcs11 store renders the TPM provisioning unit"; + ok = baoPkcs11.systemd.services ? swarm-bao-token; + } + { + # The store's mTLS identity is a separate trust domain from both CAs in + # this tree, because it must not come from an authority the store will + # itself distribute. What supplies it is the glue, which mints a CA of + # the store's own — so an enabled store has all three paths, and if this + # ever reads null again the store stops coming up on its own. + name = "a deployed store is given its own certificate, key and client CA"; + ok = + let + b = baoPkcs11.services.hyperhive.deploy.bao; + in + b.serverCertFile != null && b.serverKeyFile != null && b.clientCaFile != null; + } + { + # Everything the glue sets is `mkDefault`, and this is the case that + # says so: a deployment whose certificates come from somewhere the glue + # has never heard of must win. Also the presence control for the case + # above — a renamed option would read `null` on both and satisfy + # neither, but only this one names a value. + name = "an operator's own certificate path beats the glue's default"; + ok = baoExplicitCerts.services.hyperhive.deploy.bao.serverCertFile == "/etc/pki/bao.pem"; + } + { + # The store's first reader. Its unit belongs to the pairing, not to + # either service: matrix must not learn the store exists, and the store + # must not know who reads it. + name = "a store deployed beside the homeserver fetches its registration token"; + ok = baoWithMatrix.systemd.services ? swarm-bao-matrix-token; + } + { + # Absence arm. A store with nothing to serve renders no reader, so the + # unit is a function of the PAIRING rather than of the store — which is + # the property that makes it glue instead of a feature of either side. + name = "a store with no homeserver beside it renders no token reader"; + ok = !(baoPkcs11.systemd.services ? swarm-bao-matrix-token); + } ]; bad = builtins.filter (c: !c.ok) cases; From c1b7be1135127d890e431eaa45840a9a1bb2e6d4 Mon Sep 17 00:00:00 2001 From: atlas Date: Sun, 30 Aug 2026 18:54:36 +0200 Subject: [PATCH 6/6] docs/swarm: the store's init step, its identity, and its first reader bao operator init stays manual: it emits recovery keys and a root token, so automating it would put the disaster-recovery material on the host it is the recovery path for. The unseal note is split by seal mode, since pkcs11 unseals itself and only shamir needs an operator per boot. secrets.md said nothing reads the store and nothing mints its identity; setup.md said the deployment must supply the certificate. All three are false now. They name the first reader, why that credential and not authelia's OIDC secret, that every failure path leaves the local token in place, and that the minted paths are mkDefaults an operator's own paths beat. Delta sweep clean: 0 hits for useSelfSigned / acme / swarm-services / 'nothing reads' / 'issued and unused'; control returns 12 for serverCertFile. The single 'no reader' hit is a module-eval comment describing the absence arm. --- docs/setup.md | 44 ++++++++++++++++++++++++++++++++++++++----- docs/swarm/secrets.md | 25 ++++++++++++++++++++++++ 2 files changed, 64 insertions(+), 5 deletions(-) diff --git a/docs/setup.md b/docs/setup.md index 4af6a569..da3d3d44 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -44,7 +44,41 @@ echo "hunter2" | hivectl gateway create-user mara --password-stdin hivectl gateway list-users ``` -### 3 · Swarm SSO (only when `deploy.authelia`) +### 3 · Secret store (only when `deploy.bao`) + +⚠️ **A sealed store still answers.** OpenBao starts uninitialised and +sealed, so the container is up and the port responds while every read +times out — the failure looks like a hang, not like a store that was +never initialised. Do this before anything is pointed at it. + +```bash +# On the host that RUNS the store, once. +bao operator init # keep the keys it prints and the root token OFF this host +``` + +Whether anything more is needed depends on +`services.hyperhive.deploy.bao.seal`: + +- **`pkcs11`** (the default) — the key is bound to the host's TPM and the + store unseals itself on every restart. `init` is the only manual step. +- **`shamir`** — no TPM, so `bao operator unseal` is needed again after + every restart, with the keys `init` printed. + +The store serves TLS, and on a hive that deploys it you need do nothing: a +first-boot unit mints a CA of the store's own plus the two leaves it signs — +the store's server certificate and this host's client certificate — and points +`deploy.bao.serverCertFile`, `.serverKeyFile` and `.clientCaFile` at them. + +Those are `mkDefault`s, so naming your own paths wins. Do that when your +certificates come from a real internal CA; the store has no opinion about +which. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is +**mTLS between services and the store**, a separate trust domain, because a +store that took its identity from an authority it will itself distribute could +never come up before that authority. + +Making even the `init` unnecessary is tracked in issue #3768. + +### 4 · Swarm SSO (only when `deploy.authelia`) ⚠️ **Required to finish the install, not optional.** Authelia treats an empty user store as a fatal startup error, so until this runs the @@ -70,7 +104,7 @@ than amend — adding the group afterwards is `swarmctl user update mara Detail, including what the password is and why this stays manual: [`swarm/sso.md`](swarm/sso.md). -### 4 · Swarm UI (only when `deploy.swarm-ui`, on by default with the controller) +### 5 · Swarm UI (only when `deploy.swarm-ui`, on by default with the controller) Nothing to run — it is served on the swarm apex (`https:///`) as soon as the host rebuilds. Two things @@ -86,7 +120,7 @@ decide whether you can actually open it: Detail, including why reachability is deliberately not the access control: [`swarm/ui.md`](swarm/ui.md). -### 5 · Matrix +### 6 · Matrix ```bash # 5a. Ensure the hive-internal admin account exists first @@ -107,7 +141,7 @@ hivectl matrix promote-user mara The human operator's own matrix account is created via swarm SSO instead of a manual `hivectl` step — see step 3 (`swarmctl user add`). -### 6 · Spawn sub-agents +### 7 · Spawn sub-agents Sub-agent creation goes through the approval queue — ruth proposes, the operator approves, the container builds. From ruth's own turn (inside @@ -128,7 +162,7 @@ request_init_config(name: "iris") See [`approvals.md`](approvals.md) for the full flow. -### 7 · Useful host commands +### 8 · Useful host commands ```bash # Roster: all agents, status, rev, parent, pending reminders diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index a44b52ac..d43012f9 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -55,6 +55,8 @@ neither is a renaming of the other. | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | wireguard private key | **the operator** — `wg genkey` | whatever `swarm.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `nats.calloutUserSeedFile` / `nats.calloutIssuerSeedFile` | +| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | `/var/lib/swarm-bao` on the host of whoever runs the store, bind-mounted into the `swarm-bao` container | n/a — there is one store; a hive elsewhere is a *client* of it and holds none of this | +| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it is stated rather than inferred | n/a — only the host running the store seals anything | The three keys authelia mints for itself are generated in-container precisely because nothing outside that container ever reads them. **That is the test worth @@ -132,6 +134,29 @@ bounded wait, 120s — and then **fail loudly** rather than skipping. A silent s produces a service whose login button always fails, which is a symptom several layers from its cause. +The store's **first reader** is the matrix registration token, and it is worth +saying why that one: it is an opaque 32-byte value with no second file and no +format. Authelia's OIDC secret needs a `.secret` *and* a matching `.digest`, so +starting there would have meant debugging "can a reader authenticate and get +bytes back" and "did we write authelia's file format right" at once, with an +SSO outage as the failure mode. + +`glue-matrix-bao-token.nix` fetches it and writes the file `hive-matrix.nix` +already reads, so the homeserver never learns the store exists. Every failure +path — no such key, sealed store, unreachable store, empty value — leaves the +locally minted token in place, so a hive with no store behaves exactly as it +did before. + +⚠️ **Service↔store mTLS is its own trust domain.** A credential you must +already hold to authenticate cannot be fetched from the thing it authenticates +you to, so the store's identity cannot come from an authority the store +distributes — which excludes the hive CA and the swarm CA both, and has nothing +to do with the gateway's HTTPS certificates either way. `glue-bao-tls.nix` +mints a CA that signs exactly two things, the store's server certificate and a +reader's client certificate, and distributes nothing. A deployment with a real +internal CA deletes that file and names its own paths in +`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion. + ## The constraint that decides where the root lives A hive CA carries `nameConstraints=permitted;DNS:`, and **a swarm