docs(swarm): state the secrets rule precisely instead of carving an exception

Review: the public keys are not an exception, the paragraph was
incomplete. It is private key material and access tokens that are
file-based so they stay out of the nix store; public material in a
public store is fine.

Fixes the rule itself, which removes the need for the special case the
previous commit added.

docs/swarm/secrets.md

@@ -6,11 +6,16 @@ This page is that answer, one row per secret.
 
 Two rules run through all of it.
 
-**A secret is a path, never a value.** Every option that carries a credential
+**Private key material and access tokens are paths, never values.** Every option
-takes a file path (`*File`), because a literal written into a nix expression is
+carrying one takes a file path (`*File`), because a literal written into a nix
-rendered into the nix store — which is world-readable and permanent. There is no
+expression is rendered into the nix store — world-readable and permanent. No
-option anywhere in this tree that accepts a secret inline, and adding one would
+option in this tree accepts one inline, and adding one would be a leak rather
-be a leak rather than a convenience.
+than a convenience.
+
+The rule is about what must stay secret, not about credentials generally.
+**Public material is a value**: a certificate, or a public nkey like
+`swarm.nats.calloutUserPublicKey`, is published to every client that connects,
+so the store is a perfectly good place for it.
 
 **The generator and the reader are usually in different containers.** They share
 the host's network namespace, which makes them feel co-located, but their
@@ -44,12 +49,42 @@ Every row below is read against one of these.
 | OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
 | authelia subject store | `swarmctl` | `users.json` (canonical) → `users.yml` (rendered) | `swarmctl`, on the host that runs authelia |
 | wireguard private key | **the operator** — `wg genkey` | whatever `swarm.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
+| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `nats.calloutUserSeedFile` / `nats.calloutIssuerSeedFile` |
 
 The three keys authelia mints for itself are generated in-container precisely
 because nothing outside that container ever reads them. **That is the test worth
 applying to any secret added here** — and the client secret's plaintext half is
 the one row that fails it, which is the entire reason a delivery step exists.
 
+### Minting the queue's callout nkeys
+
+`nats.autoGenerateCallout` mints both keypairs on the host before the queue
+starts. It is on by default only under `enableAllLocalDefaults` — the one
+topology where the queue, its responder and the operator are the same person. On
+every other topology, mint them yourself:
+
+```
+nk -gen user    > callout-user.seed     # the responder's own identity
+nk -gen account > issuer.seed           # signs the user JWTs it hands out
+nk -inkey callout-user.seed -pubout     # → calloutUserPublicKey
+nk -inkey issuer.seed       -pubout     # → calloutIssuerPublicKey
+```
+
+Keep both seeds at `0600` and name them in `calloutUserSeedFile` /
+`calloutIssuerSeedFile`. Possession of the **issuer** seed is the authority to
+admit anyone to the queue, so it belongs wherever the responder runs and nowhere
+else.
+
+A hive that sets neither the public keys nor `autoGenerateCallout` fails at
+eval, naming the option it wants. That is deliberate: a queue that started
+without them would accept `CONNECT {"user":"auth"}` from anyone sharing the
+host's network namespace, and nothing would look wrong until somebody connected.
+
+One consequence of the generated path worth knowing before you debug it: with
+`autoGenerateCallout` set, the queue's config is assembled at boot rather than at
+build time, so a malformed one surfaces when the container starts instead of
+when the system builds. The server names the offending file and refuses to run.
+
 ## Hive-level — one of each per hive
 
 | secret | generated by | lives at |

nix/host-modules/local-defaults.nix

@@ -74,6 +74,12 @@ in
   config.services.hyperhive.swarm = {
     enableRequiredServices = lib.mkDefault cfg.enableAllLocalDefaults;
     ca.autoConfigure = lib.mkDefault cfg.enableAllLocalDefaults;
+    # The queue's auth-callout nkeys. Generating them is safe exactly
+    # when one operator owns both the queue and its responder, which is
+    # what this mode asserts. On any other topology the seeds have to
+    # reach whoever runs the responder, and minting them here would move
+    # that hand-off somewhere less visible rather than removing it.
+    nats.autoGenerateCallout = lib.mkDefault cfg.enableAllLocalDefaults;
     # The controller is asserted by the MODE and by nothing else. Its own
     # option stays `default = false` precisely because running it is a
     # statement about swarm topology — but "this box is the whole

nix/host-modules/swarm-nats.nix

@@ -29,9 +29,153 @@ let
   # The responder needs all three credentials. Gating on them rather than
   # on `cfg.enable` keeps a half-configured hive at "queue up, denying
   # everyone" instead of "unit crash-looping on a missing file".
-  responderConfigured = cfg.calloutUserSeedFile != "" && cfg.calloutIssuerSeedFile != "";
+  #
+  # In auto mode the seeds are minted on this host before the container
+  # starts, so they are configured by construction.
+  responderConfigured =
+    cfg.autoGenerateCallout || (cfg.calloutUserSeedFile != "" && cfg.calloutIssuerSeedFile != "");
   clientSecretSource = "${autheliaCfg.hostClientSecretDir}/${cfg.clientId}.secret";
   introspectionUrl = "${toString autheliaUrl}/api/oidc/introspection";
+
+  # Where the responder's seeds actually come from. One name for two
+  # origins, so everything downstream stops caring which mode it is in.
+  userSeedFile = if cfg.autoGenerateCallout then autoUserSeed else cfg.calloutUserSeedFile;
+  issuerSeedFile = if cfg.autoGenerateCallout then autoIssuerSeed else cfg.calloutIssuerSeedFile;
+
+  # Seeds stay on the host at 0600 and never enter the container or the
+  # store: only the responder needs them, and it reads them by
+  # `LoadCredential` from here.
+  autoSeedDir = "/var/lib/swarm-nats-callout";
+  autoUserSeed = "${autoSeedDir}/callout-user.seed";
+  autoIssuerSeed = "${autoSeedDir}/issuer.seed";
+
+  # The runtime config directory: wrapper, settings symlink and fragment.
+  # World-readable is correct (everything in it is public) and it is not in
+  # the 0700 responder secret dir, which the `nats` user cannot traverse.
+  #
+  # ⚠️ ONE DIRECTORY IS FORCED, NOT TIDINESS. NATS resolves an `include` with
+  # `filepath.Join(configDir, path)`, which strips a leading slash, so an
+  # absolute include silently becomes relative and is never found. The
+  # includes must therefore be bare filenames, i.e. siblings. Invisible to
+  # eval: the wrapper renders perfectly and the server refuses to start.
+  runtimeDir = "/var/lib/nats-callout";
+  runtimeWrapper = "${runtimeDir}/nats.conf";
+  hostRuntimeDir = "/var/lib/nixos-containers/${machine}${runtimeDir}";
+
+  natsFormat = pkgs.formats.json { };
+
+  # Upstream's rendered settings, re-rendered with the same generator on the
+  # same value — the artifact `services.nats` would have used, not a
+  # transcription. ⚠️ This reference is also the only thing keeping it alive:
+  # `ExecStart` names a runtime path, so nothing else in the closure names
+  # the store file, and its string context is what stops it being
+  # garbage-collected out from under a running server.
+  renderedSettings = natsFormat.generate "nats.conf" (
+    config.containers.${machine}.config.services.nats.settings
+  );
+
+  # Defined once, rendered twice: into `settings` with the operator's values,
+  # and into the runtime fragment with placeholders the generator fills in.
+  #
+  # 🪤 Do not inline these and hand-write the fragment instead. Both files are
+  # loaded and the fragment is the LATER definition, so it wins — a
+  # hand-written copy would make a future edit to `settings` silently
+  # ineffective on exactly the hives that use auto mode.
+  calloutBlocks =
+    {
+      userKey,
+      issuerKey,
+    }:
+    {
+      # Two accounts, and the callout user lives in neither of the accounts
+      # it authorizes into.
+      accounts = {
+        # ⚠️ The nkey is not decoration and its absence was a real hole: a
+        # `users` entry carrying only a `user` name has no credential, and
+        # `CONNECT {"user":"auth"}` is then accepted with no password at
+        # all. Since the name is a literal in this public module, that made
+        # the callout-exempt identity walk-in-able from every container on
+        # the shared netns — the same class of hole this module exists to
+        # close, moved rather than fixed. Caught in review on the first
+        # version of this file. An nkey and NOTHING else, both halves
+        # measured against a running server rather than reasoned about:
+        #
+        #   { user = "auth"; }               → `CONNECT {"user":"auth"}`
+        #                                      is accepted with no
+        #                                      credential at all
+        #   { user = "auth"; nkey = "U…"; }  → refuses to START:
+        #                                      "Nkey users do not take
+        #                                      usernames or passwords"
+        #   { nkey = "U…"; }                 → what this is
+        #
+        # A malformed key is fail-closed too: the server exits with
+        # "Not a valid public nkey for a user" rather than starting with a
+        # hole. So the only way to get a live server here is a real key
+        # whose seed nobody but the responder holds.
+        #
+        # 🔒 That property is what makes auto mode safe: it renders as ""
+        # there, so any field the fragment fails to override keeps a value
+        # the server refuses to start on. An incomplete merge cannot leave
+        # a walk-in-able server.
+        ${calloutAccount}.users = [ { nkey = userKey; } ];
+        # ⚠️ `services.nats.jetstream = true` gives the SERVER JetStream;
+        # an account gets it only from its own grant. Measured against a
+        # running 2.14.1 with this exact two-account shape, because the
+        # failure is invisible to any config-rendering check:
+        #
+        #   global jetstream only → `nats kv add` from this account fails
+        #     `code=503 err_code=10039 jetstream not enabled for account`,
+        #     while the server starts cleanly and logs "Starting JetStream"
+        #   + this line            → the same command succeeds
+        #
+        # The grant is per-account by design, and that is worth keeping:
+        # the callout account above deliberately does NOT get it. The
+        # responder mints credentials; it has no business holding stream
+        # state.
+        #
+        # ⚠️ Also why the fragment renders the COMPLETE accounts block: if a
+        # later definition replaced rather than merged, a partial one would
+        # drop this grant and every KV op would fail on a healthy server.
+        ${clientAccount} = {
+          jetstream = "enabled";
+        };
+      };
+
+      authorization = {
+        timeout = "2s";
+        # 🔒 THIS BLOCK IS THE FAIL-CLOSED STATE, and it is the measured
+        # one rather than the obvious one.
+        #
+        # Measured on the pinned nats-server 2.14.1: both
+        # `authorization { }` and `authorization { users: [] }` accept an
+        # anonymous client and answer PONG — they read like "authorize
+        # nobody" and are wide open. An auth_callout block sets
+        # `auth_required` and refuses every client whose credential no
+        # responder has approved, so a config whose responder does not
+        # exist yet denies everyone.
+        #
+        # `nats-server -t` calls all three valid; it parses, it does not
+        # authenticate. Only running them tells the difference.
+        #
+        # ⇒ this is both the safe interim state and the final shape.
+        # Nothing here has to be swapped out when the responder lands
+        # beside it — it only starts being able to say yes.
+        auth_callout = {
+          issuer = issuerKey;
+          auth_users = [ userKey ];
+          account = calloutAccount;
+        };
+      };
+    };
+
+  # The fragment as nix renders it, with placeholders where the runtime
+  # values go. Rendered by the same JSON generator upstream uses, so the
+  # fragment is generated rather than transcribed — NATS' config parser
+  # accepts JSON, and an `include` of it merges (measured).
+  calloutTemplate = natsFormat.generate "swarm-nats-callout-template.conf" (calloutBlocks {
+    userKey = "@USER_PUBKEY@";
+    issuerKey = "@ISSUER_PUBKEY@";
+  });
 in
 {
   # The swarm's message queue: one NATS server, reached by every hive.
@@ -85,6 +229,26 @@ in
       '';
     };
 
+    autoGenerateCallout = lib.mkOption {
+      type = lib.types.bool;
+      default = false;
+      example = true;
+      description = ''
+        Generate the auth-callout nkeys on this host instead of taking
+        them from `calloutUserPublicKey` / `calloutIssuerPublicKey`.
+
+        A first-boot unit mints both keypairs if absent, keeps the seeds
+        host-side at `0600`, and writes only the public halves into a
+        fragment the server reads. Nothing secret is evaluated, so
+        nothing secret reaches the nix store.
+
+        Leave it off wherever the queue and its clients are not the same
+        operator's problem: the seeds must reach whoever runs the
+        responder, and minting them here only moves that distribution
+        somewhere less visible. `enableAllLocalDefaults` turns it on.
+      '';
+    };
+
     calloutUserPublicKey = lib.mkOption {
       type = lib.types.str;
       default = "";
@@ -189,7 +353,7 @@ in
         # Fail at EVAL, not at boot: a queue that comes up unable to
         # authenticate anyone presents as every client hanging, which is
         # several layers from "the operator never set the issuer".
-        assertion = cfg.calloutIssuerPublicKey != "";
+        assertion = cfg.autoGenerateCallout || cfg.calloutIssuerPublicKey != "";
         message = ''
           services.hyperhive.swarm.nats.enable requires
           nats.calloutIssuerPublicKey — the public half of the account
@@ -206,7 +370,7 @@ in
         # failure is the only place to catch that: the rendered config is
         # valid, the server starts, and the hole is invisible until
         # somebody connects.
-        assertion = cfg.calloutUserPublicKey != "";
+        assertion = cfg.autoGenerateCallout || cfg.calloutUserPublicKey != "";
         message = ''
           services.hyperhive.swarm.nats.enable requires
           nats.calloutUserPublicKey — the public half of the user nkey
@@ -300,86 +464,31 @@ in
 
             serverName = "swarm-nats";
             port = cfg.port;
-            settings = {
+            # In auto mode the keys are empty until the generator runs,
-              # Two accounts, and the callout user lives in neither of
+            # and `nats-server -t` rejects that ("Expected callout user to
-              # the accounts it authorizes into.
+            # be a valid public account nkey, got \"\""), so leaving this
-              accounts = {
+            # on fails the BUILD of every all-local hive. Upstream's own
-                # ⚠️ The nkey is not decoration and its absence was a real
+            # description names the case: disable it when the config
-                # hole: a `users` entry carrying only a `user` name has no
+            # includes other files. The check moves to server start.
-                # credential, and `CONNECT {"user":"auth"}` is then
+            validateConfig = !cfg.autoGenerateCallout;
-                # accepted with no password at all. Since the name is a
+
-                # literal in this public module, that made the
+            settings = calloutBlocks {
-                # callout-exempt identity walk-in-able from every
+              userKey = cfg.calloutUserPublicKey;
-                # container on the shared netns — the same class of hole
+              issuerKey = cfg.calloutIssuerPublicKey;
-                # this module exists to close, moved rather than fixed.
-                # Caught in review on the first version of this file.
-                # An nkey and NOTHING else, both halves measured against a
-                # running server rather than reasoned about:
-                #
-                #   { user = "auth"; }               → `CONNECT {"user":"auth"}`
-                #                                      is accepted with no
-                #                                      credential at all
-                #   { user = "auth"; nkey = "U…"; }  → refuses to START:
-                #                                      "Nkey users do not take
-                #                                      usernames or passwords"
-                #   { nkey = "U…"; }                 → what this is
-                #
-                # A malformed key is fail-closed too: the server exits with
-                # "Not a valid public nkey for a user" rather than starting
-                # with a hole. So the only way to get a live server here is
-                # a real key whose seed nobody but the responder holds.
-                ${calloutAccount}.users = [ { nkey = cfg.calloutUserPublicKey; } ];
-                # ⚠️ `services.nats.jetstream = true` gives the SERVER
-                # JetStream; an account gets it only from its own grant.
-                # Measured against a running 2.14.1 with this exact
-                # two-account shape, because the failure is invisible to
-                # any config-rendering check:
-                #
-                #   global jetstream only → `nats kv add` from this
-                #     account fails `code=503 err_code=10039 jetstream
-                #     not enabled for account`, while the server starts
-                #     cleanly and logs "Starting JetStream"
-                #   + this line            → the same command succeeds
-                #
-                # The grant is per-account by design, and that is worth
-                # keeping: the callout account above deliberately does
-                # NOT get it. The responder mints credentials; it has no
-                # business holding stream state.
-                ${clientAccount} = {
-                  jetstream = "enabled";
             };
           };
 
-              authorization = {
+          # A wrapper that includes upstream's rendered settings verbatim
-                timeout = "2s";
+          # plus the runtime fragment; rendering the config ourselves
-                # 🔒 THIS BLOCK IS THE FAIL-CLOSED STATE, and it is the
+          # instead would throw away upstream's `settings`, where the
-                # measured one rather than the obvious one.
+          # reviewed reasoning lives. The generator writes it, because the
+          # includes must be siblings of the fragment (see `runtimeDir`).
           #
-                # Measured on the pinned nats-server 2.14.1: both
+          # `mkForce`: upstream defines ExecStart inside an `mkMerge`, so a
-                # `authorization { }` and `authorization { users: [] }`
+          # plain override conflicts rather than wins.
-                # accept an anonymous client and answer PONG — they read
+          systemd.services.nats.serviceConfig.ExecStart = lib.mkIf cfg.autoGenerateCallout (
-                # like "authorize nobody" and are wide open. An
+            lib.mkForce "${pkgs.nats-server}/bin/nats-server -c ${runtimeWrapper}"
-                # auth_callout block sets `auth_required` and refuses
+          );
-                # every client whose credential no responder has
-                # approved, so a config whose responder does not exist
-                # yet denies everyone.
-                #
-                # `nats-server -t` calls all three valid; it parses, it
-                # does not authenticate. Only running them tells the
-                # difference.
-                #
-                # ⇒ this is both the safe interim state and the final
-                # shape. Nothing here has to be swapped out when the
-                # responder lands beside it — it only starts being able
-                # to say yes.
-                auth_callout = {
-                  issuer = cfg.calloutIssuerPublicKey;
-                  auth_users = [ cfg.calloutUserPublicKey ];
-                  account = calloutAccount;
-                };
-              };
-            };
-          };
 
           # The auth-callout responder: the half that lets the server
           # above say *yes*. Without it the `auth_callout` block is a
@@ -444,6 +553,12 @@ in
       description = "deliver the swarm queue responder's credentials";
       before = [ "container@swarm-nats.service" ];
       wantedBy = [ "container@swarm-nats.service" ];
+      # In auto mode the seeds this copies do not exist until the generator
+      # has run. `requires` as well as `after`: if minting fails there is
+      # nothing to deliver, and a copy that silently succeeds with a stale
+      # or absent seed is worse than not running.
+      after = lib.optional cfg.autoGenerateCallout "swarm-nats-callout-keys.service";
+      requires = lib.optional cfg.autoGenerateCallout "swarm-nats-callout-keys.service";
       serviceConfig = {
         Type = "oneshot";
         RemainAfterExit = true;
@@ -453,13 +568,83 @@ in
       script = ''
         set -euo pipefail
         install -d -m 0700 ${lib.escapeShellArg secretDir}
-        install -m 0400 ${lib.escapeShellArg cfg.calloutUserSeedFile} \
+        install -m 0400 ${lib.escapeShellArg userSeedFile} \
           ${lib.escapeShellArg (hostPath "callout-user.seed")}
-        install -m 0400 ${lib.escapeShellArg cfg.calloutIssuerSeedFile} \
+        install -m 0400 ${lib.escapeShellArg issuerSeedFile} \
           ${lib.escapeShellArg (hostPath "issuer.seed")}
         install -m 0400 ${lib.escapeShellArg clientSecretSource} \
           ${lib.escapeShellArg (hostPath "oidc-client.secret")}
       '';
     };
+
+    # ⚠️ Minted on the HOST, not in the container, because the responder is
+    # a separate unit that needs the user seed: generating it inside would
+    # trap it there and require a secret-export path back out — the exact
+    # mechanism this is meant to avoid inventing. Only public halves cross.
+    systemd.services.swarm-nats-callout-keys = lib.mkIf cfg.autoGenerateCallout {
+      description = "mint the swarm queue's auth-callout nkeys";
+      before = [ "container@swarm-nats.service" ];
+      wantedBy = [ "container@swarm-nats.service" ];
+      serviceConfig = {
+        Type = "oneshot";
+        RemainAfterExit = true;
+        SyslogIdentifier = "swarm-nats-callout-keys";
+      };
+      path = [
+        pkgs.coreutils
+        pkgs.nkeys
+      ];
+      script = ''
+        set -euo pipefail
+        umask 077
+        install -d -m 0700 ${lib.escapeShellArg autoSeedDir}
+
+        # Mint iff absent. Idempotence is the whole contract: this runs on
+        # every boot, and regenerating would silently invalidate every
+        # credential the responder has already issued against the old
+        # issuer.
+        if [ ! -s ${lib.escapeShellArg autoUserSeed} ]; then
+          nk -gen user > ${lib.escapeShellArg autoUserSeed}.tmp
+          mv ${lib.escapeShellArg autoUserSeed}.tmp ${lib.escapeShellArg autoUserSeed}
+        fi
+        if [ ! -s ${lib.escapeShellArg autoIssuerSeed} ]; then
+          nk -gen account > ${lib.escapeShellArg autoIssuerSeed}.tmp
+          mv ${lib.escapeShellArg autoIssuerSeed}.tmp ${lib.escapeShellArg autoIssuerSeed}
+        fi
+        chmod 0600 ${lib.escapeShellArg autoUserSeed} ${lib.escapeShellArg autoIssuerSeed}
+
+        user_pub="$(nk -inkey ${lib.escapeShellArg autoUserSeed} -pubout)"
+        issuer_pub="$(nk -inkey ${lib.escapeShellArg autoIssuerSeed} -pubout)"
+
+        # Public halves only — world-readable on purpose, since the server
+        # publishes them to every client that connects.
+        install -d -m 0755 ${lib.escapeShellArg hostRuntimeDir}
+        sed -e "s|@USER_PUBKEY@|$user_pub|g" \
+            -e "s|@ISSUER_PUBKEY@|$issuer_pub|g" \
+            ${calloutTemplate} > ${lib.escapeShellArg hostRuntimeDir}/callout.conf.tmp
+        # Mode BEFORE the rename: a rename publishes whatever the file
+        # already is, so setting it afterwards leaves a window where the
+        # live path has the wrong mode. Same trap as the gateway's
+        # atomic-publish path.
+        chmod 0444 ${lib.escapeShellArg hostRuntimeDir}/callout.conf.tmp
+        mv ${lib.escapeShellArg hostRuntimeDir}/callout.conf.tmp \
+           ${lib.escapeShellArg hostRuntimeDir}/callout.conf
+
+        # Upstream's rendered settings, as a sibling the wrapper can name
+        # without a leading slash. Refreshed unconditionally — unlike the
+        # seeds, this one MUST track the current system, and a stale copy
+        # would silently run yesterday's config.
+        ln -sfn ${renderedSettings} ${lib.escapeShellArg hostRuntimeDir}/settings.conf
+
+        # The wrapper. Bare filenames — see `runtimeDir`. printf rather than
+        # a heredoc, whose terminator would depend on nix's indentation
+        # stripping and break the next time `nix fmt` touched this block.
+        printf 'include "settings.conf"\ninclude "callout.conf"\n' \
+          > ${lib.escapeShellArg hostRuntimeDir}/nats.conf.tmp
+        chmod 0444 ${lib.escapeShellArg hostRuntimeDir}/nats.conf.tmp
+        mv ${lib.escapeShellArg hostRuntimeDir}/nats.conf.tmp \
+           ${lib.escapeShellArg hostRuntimeDir}/nats.conf
+      '';
+    };
   };
 }