forge: hand the OIDC client secret to forgejo from /run, not from the container's disk

swarm-bao-forge-oidc fetched the forge's OIDC client secret from bao and wrote
it to /var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/<id>.secret,
on the container's persistent root, where it outlived a bao rotation and was
in every backup of that tree.

Now the fetch writes it to /run/hive-forge-oidc/<id>.secret on the host
(tmpfs, root 0400). The directory is bound read-only into the container at
the same path, and forgejo-sso-source loads it through LoadCredential. A
tmpfiles rule creates the directory at boot, so the bind source always
exists, and removes the old persistent copy.

After a reboot the file is absent until the fetch succeeds, and
forgejo-sso-source fails to start until then; the fetch starts it again once
it writes the file. Logins keep working meanwhile: forgejo holds the secret
in its login-source row.

deploy.forgejo.sso.clientSecretFile (and its old swarm.forge.sso name) is
removed: it only chose where in the container the fetched file landed. The
fetch no longer needs the container's forgejo uid, so it no longer fails on a
fresh host before the container has created its state dir.

Part of #4998.

docs/swarm/secrets.md

@@ -249,10 +249,10 @@ such as the root token.
 ## Hive-level — one of each per hive
 
 | secret                                   | generated by                                                                                                                               | lives at                                                                                                                                        |
-| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
+| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
 | hive leaf certs                          | `hive-tls.nix`, issued by the secret store's `pki` mount                                                                                   | `<deploy.hive-controller.tls.stateDir>/<name>.pem`                                                                                              |
 | the hive's own store identity            | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile`                                                                 |
-| the forge's copy of its OIDC secret      | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs             | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container                                |
+| the forge's copy of its OIDC secret      | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs             | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
 | the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs                     | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential`                                                                 |
 | the agent containers' queue credential   | authelia, published to the store by `swarm-secret-publish`                                                                                 | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`)                                                   |
 

docs/swarm/sso.md

@@ -126,8 +126,8 @@ first boot, and `swarm-secret-publish` on its host writes it to the swarm
 secret store at `swarm/services/<client id>/oidc/client`. A host-side unit
 then reads it from the store and writes it into the service's container:
 `swarm-bao-forge-oidc` for the forge, under the forge publisher's store
-leaf, into `/var/lib/forgejo-oidc/<client id>.secret` (the file
+leaf, into `/run/hive-forge-oidc/<client id>.secret` on the host, which
-`forgejo-sso-source` reads); `swarm-bao-matrix-oidc` for matrix, under
+`forgejo-sso-source` loads as a credential; `swarm-bao-matrix-oidc` for matrix, under
 matrix-ctl's. A unit that can't read the secret yet fails and retries
 every 30 seconds for a day, so there is no start order to get right. The
 service's own module contributes its client entry — callback URL
@@ -141,10 +141,14 @@ rejected login with no error text worth reading.
 
 <!-- vale write-good.Passive = NO -->
 
-⚠️ The secret is written into the container's state directory, not bound
+⚠️ nixos-container refuses to start a container whose bind source is
-in with `bindMounts`: nixos-container refuses to start a container whose
+missing, and this secret doesn't exist until authelia's first boot has run
-bind source is missing, and this secret doesn't exist until authelia's
+and the publisher has written it. The forge therefore binds the **directory**
-first boot has run and the publisher has written it.
+`/run/hive-forge-oidc`, which a tmpfiles rule creates at boot, never the
+file. The directory is tmpfs, so the secret is gone after a reboot until
+the fetch runs again; forgejo keeps it in its login-source row, so logins
+still work meanwhile. Matrix's secret is written into the container's state
+directory instead.
 
 <!-- vale write-good.Passive = YES -->
 

nix/host-modules/deploy.nix

@@ -244,11 +244,10 @@ in
 
     # The rest of the forge split. What stays under `swarm.forge` is what the
     # forge IS from any hive's point of view — the names and ports it answers
-    # on, the URLs it advertises, the client id it is registered under; these
+    # on, the URLs it advertises, the client id it is registered under; the
-    # four are what the host running it decides. Its package is renamed
+    # two renamed here are what the host running it decides. Its package is
-    # further down with the other `*.package` entries. `sso` splits
+    # renamed further down with the other `*.package` entries. The two removed
-    # because its two halves are different facts: the client id must match the
+    # here named host files that the swarm secret store now replaces.
-    # entry in authelia's register, the secret is a path on this machine.
     #
     # `mirrors` renames in one entry rather than one per field: it is a single
     # option of a list-of-submodule type, so the rename carries its whole
@@ -261,10 +260,11 @@ in
       [ "services" "hyperhive" "swarm" "forge" "mirrors" ]
       [ "services" "hyperhive" "deploy" "forgejo" "mirrors" ]
     )
-    (lib.mkRenamedOptionModule
+    (lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "forge" "sso" "clientSecretFile" ] ''
-      [ "services" "hyperhive" "swarm" "forge" "sso" "clientSecretFile" ]
+      The forge's OIDC client secret is fetched from the swarm secret store
-      [ "services" "hyperhive" "deploy" "forgejo" "sso" "clientSecretFile" ]
+      (swarm-bao-forge-oidc) into a fixed runtime path. Remove this
-    )
+      definition.
+    '')
     (lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "forge" "hostSwarmControllerTokenFile" ]
       ''
         The forge publishes the swarm-controller's forge token to the swarm

nix/host-modules/hive-forge/default.nix

@@ -28,16 +28,17 @@ let
   autheliaUrl = autheliaCfg.url;
   autheliaDiscoveryUrl = "${toString autheliaUrl}/.well-known/openid-configuration";
 
-  # Where the plaintext lands inside the forge container. Under
+  # The forge's OIDC client secret, in a host tmpfs directory bound read-only
-  # /var/lib rather than /run: the forge may start before the delivery
+  # into the container at the same path: `swarm-bao-forge-oidc` writes it,
-  # unit on a later boot, and a secret that evaporates on reboot turns a
+  # `forgejo-sso-source` loads it as a credential. After a reboot it is absent
-  # working login into an intermittent one.
+  # until the fetch succeeds; logins still work then, because forgejo keeps
-  forgeSecretPath = "/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret";
+  # the secret in its login-source row.
+  oidcSecretDir = "/run/hive-forge-oidc";
+  oidcSecretPath = "${oidcSecretDir}/${cfg.sso.clientId}.secret";
 
-  # The same file seen from the host, where `swarm-bao-forge-oidc` writes it.
+  # Where the fetched secret used to be written, on the container's persistent
-  # Follows `clientSecretFile`, so an operator-chosen path is where the
+  # root. Removed by the tmpfiles rule below.
-  # fetched secret lands.
+  oldOidcSecretPath = "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret";
-  hostSecretPath = "/var/lib/nixos-containers/hive-forge${toString deployCfg.forgejo.sso.clientSecretFile}";
 
   # Where swarm-secret-publish on authelia's host leaves this client's secret.
   # The forge publisher's store role grants read on this one path.
@@ -55,8 +56,8 @@ let
   swarmControllerForgeUser = "swarm-controller";
 
   # Where the minted token lands inside the forge container — under
-  # forgejo's own state dir for the same "survives a reboot" reason as
+  # forgejo's own state dir, because the file is the mint's record that it
-  # `forgeSecretPath` above.
+  # has minted and has to survive a reboot.
   swarmControllerTokenPath = "/var/lib/forgejo/swarm-controller-token";
 
   # The same file from the host, which is where its publisher reads and watches it.
@@ -128,6 +129,11 @@ in
 
   imports = [
     ./service.nix
+    (lib.mkRemovedOptionModule [ "services" "hyperhive" "deploy" "forgejo" "sso" "clientSecretFile" ] ''
+      The forge's OIDC client secret is fetched from the swarm secret store
+      (swarm-bao-forge-oidc) into a fixed runtime path. Remove this
+      definition.
+    '')
     (lib.mkRemovedOptionModule
       [ "services" "hyperhive" "deploy" "forgejo" "hostSwarmControllerTokenFile" ]
       ''
@@ -249,32 +255,6 @@ in
         adds; what `glue-forge-controller.nix` forwards to the controller.
       '';
     };
-
-    # The other half of the SSO pair: the client *id* is swarm-wide (it has
-    # to match authelia's register), the secret is a path on this host. The
-    # design note for the login source itself is with the id.
-    sso.clientSecretFile = lib.mkOption {
-      type = lib.types.nullOr lib.types.str;
-      default = null;
-      example = "/var/lib/forgejo-oidc/forgejo.secret";
-      description = ''
-        Path **inside the forge container** holding the client
-        secret's plaintext.
-
-        A path, never a value: an OIDC client secret has two holders
-        in two containers (authelia keeps a hash, this forge needs the
-        plaintext), and a literal written here would be rendered into
-        the world-readable nix store.
-
-        Set to `/var/lib/forgejo-oidc/<client id>.secret` wherever the
-        forge runs. `swarm-bao-forge-oidc` writes the secret here from the
-        swarm secret store, whichever path this names.
-
-        Null is refused at eval: a forge that boots with SSO
-        half-configured presents as a login button that always fails,
-        which is harder to diagnose than an eval error.
-      '';
-    };
   };
 
   config = lib.mkIf deployCfg.forgejo.enable {
@@ -371,21 +351,6 @@ in
     };
 
     assertions = [
-      {
-        # Fail at EVAL, not at boot. The alternative failure is a login
-        # button that always 401s, three layers from the missing file.
-        assertion = deployCfg.forgejo.sso.clientSecretFile != null;
-        message = ''
-          The forge's SSO login source requires
-          services.hyperhive.deploy.forgejo.sso.clientSecretFile — the path
-          (inside the forge container) holding the OIDC client secret's
-          plaintext.
-
-          It is set by default wherever the forge runs, and
-          swarm-bao-forge-oidc writes the secret there from the swarm
-          secret store. Remove the definition that set it to null.
-        '';
-      }
       {
         assertion = cfg.rootUrl == null || lib.hasSuffix "/" cfg.rootUrl;
         message = ''
@@ -460,7 +425,16 @@ in
       # delivery. The bind-mount, the `container@` ordering and the bundle
       # the container's consumers read all come from the hive-ca-trust
       # helper — see the container's `imports` for the consumption half.
-      bindMounts = caTrust.bindMount;
+      #
+      # The OIDC secret's directory: a container's `/run` is an nspawn tmpfs
+      # the host cannot see into. The host directory comes from the tmpfiles
+      # rule below, so it exists before the container starts.
+      bindMounts = caTrust.bindMount // {
+        ${oidcSecretDir} = {
+          hostPath = oidcSecretDir;
+          isReadOnly = true;
+        };
+      };
       config =
         { pkgs, ... }:
         let
@@ -817,6 +791,10 @@ in
               User = "forgejo";
               Group = "forgejo";
               SyslogIdentifier = "forgejo-sso-source";
+              # Absent until `swarm-bao-forge-oidc` on the host has fetched
+              # it, and this unit fails to start until then; that unit starts
+              # it again once the file is written.
+              LoadCredential = [ "oidc-client-secret:${oidcSecretPath}" ];
             };
             # `FORGEJO_CUSTOM` (the forgejo-prefixed name, not `GITEA_CUSTOM`)
             # is how the CLI finds the app.ini upstream's module wrote.
@@ -842,9 +820,9 @@ in
             script = ''
               set -euo pipefail
 
-              secret=$(cat ${lib.escapeShellArg deployCfg.forgejo.sso.clientSecretFile})
+              secret=$(cat "$CREDENTIALS_DIRECTORY/oidc-client-secret")
               if [ -z "$secret" ]; then
-                echo "empty OIDC client secret at ${deployCfg.forgejo.sso.clientSecretFile}" >&2
+                echo "empty OIDC client secret at ${oidcSecretPath}" >&2
                 exit 1
               fi
 
@@ -1008,8 +986,7 @@ in
 
     # The client this login source authenticates as is registered by
     # ../glue-forge-oidc-client.nix, wherever authelia runs.
-    services.hyperhive.deploy.forgejo.sso.clientSecretFile = lib.mkDefault forgeSecretPath;
+    #
-
     # The forge's OIDC client secret, out of the swarm secret store under the
     # forge publisher's leaf — the one route, wherever authelia runs. Same
     # shape as ../hive-matrix.nix's `swarm-bao-matrix-oidc`: the login and the
@@ -1098,24 +1075,15 @@ in
               exit 1
             fi
 
-            # `forgejo-sso-source` reads the file as forgejo. The owning uid is
+            # Root-owned: the container's systemd reads it for `LoadCredential`.
-            # read off the container's forgejo state dir rather than assumed,
+            install -d -m 0700 ${lib.escapeShellArg oidcSecretDir}
-            # which exists only once the container has started.
+            if secret_differs ${lib.escapeShellArg oidcSecretPath} "$secret"; then
-            state=/var/lib/nixos-containers/hive-forge/var/lib/forgejo
+              atomic_write_secret 0400 "" ${lib.escapeShellArg oidcSecretPath} "$secret"
-            if [ ! -d "$state" ]; then
-              echo "the forge container has not created $state yet, so the forge's OIDC client secret has no owner to be written for." >&2
-              exit 1
-            fi
-            uid="$(stat -c %u "$state")"
-
-            mkdir -p ${lib.escapeShellArg (builtins.dirOf hostSecretPath)}
-            if secret_differs ${lib.escapeShellArg hostSecretPath} "$secret"; then
-              atomic_write_secret 0400 "$uid:$uid" ${lib.escapeShellArg hostSecretPath} "$secret"
             fi
 
             # The login source is a row `forgejo-sso-source` writes from the
             # file each time it runs.
-            refresh_consumer hive-forge forgejo-sso-source.service ${lib.escapeShellArg hostSecretPath}
+            refresh_consumer hive-forge forgejo-sso-source.service ${lib.escapeShellArg oidcSecretPath}
           '';
     };
 
@@ -1237,9 +1205,14 @@ in
       };
     };
 
-    # Unused: nothing writes or reads this host path, and it may hold a copy of
+    systemd.tmpfiles.rules = [
-    # the controller's forge token.
+      # The OIDC secret's directory, which the container binds at start.
-    systemd.tmpfiles.rules = [ "r /var/lib/hyperhive-forge/swarm-controller.token" ];
+      "d ${oidcSecretDir} 0700 root root -"
+      # Unused: nothing writes or reads these host paths, and they may hold a
+      # copy of the controller's forge token or of the forge's OIDC secret.
+      "r /var/lib/hyperhive-forge/swarm-controller.token"
+      "r ${oldOidcSecretPath}"
+    ];
 
     networking.firewall = lib.mkIf deployCfg.forgejo.openFirewall {
       allowedTCPPorts = [

nix/host-modules/hive-forge/service.nix

@@ -46,8 +46,8 @@ in
       Removed rather than defaulted to true so a config that turned it
       OFF fails here, where the line is, instead of silently gaining a
       login provider on the next rebuild. Drop the line; if it was
-      false, set services.hyperhive.deploy.forgejo.sso.clientSecretFile and
+      false, set services.hyperhive.swarm.authelia.url as the assertion
-      services.hyperhive.swarm.authelia.url as the assertions describe.
+      describes.
     '')
   ];
 
@@ -188,8 +188,8 @@ in
           if they do not.
         '';
       };
-      # The secret half is a path on the host that runs the forge, so it
+      # The secret half is fetched from the swarm secret store on the host
-      # lives under `deploy.forgejo.sso`, in ./default.nix.
+      # that runs the forge: `swarm-bao-forge-oidc`, in ./default.nix.
     };
   };
 }

nix/module-eval/forge-placement.nix

@@ -40,17 +40,9 @@ let
     deploy.forgejo.enable = false;
   };
 
-  # The forge without authelia: the other half of the split, with its secret's
-  # path and its store identity chosen by hand.
-  forgeNoAuthelia = hive {
-    deploy.forgejo.enable = true;
-    deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret";
-    deploy.bao.forgePublisherClientCertFile = "/etc/pki/forge-publisher.pem";
-    deploy.bao.forgePublisherClientKeyFile = "/etc/pki/forge-publisher-key.pem";
-  };
-
   # The forge with neither authelia nor the store, its store identity placed by
-  # hand: the deployment the store fetch exists for.
+  # hand: the other half of the split, and the deployment the store fetch
+  # exists for.
   forgeAlone = hive {
     deploy.forgejo.enable = true;
     deploy.bao.forgePublisherClientCertFile = "/etc/pki/forge-publisher.pem";
@@ -134,8 +126,7 @@ let
         !bare.services.hyperhive.deploy.forgejo.enable
         && !(bare.containers ? hive-forge)
         && !(bare.systemd.services ? swarm-bao-forge-publisher)
-        && !(lib.elem "forge.t.local" bare.services.hyperhive.gateway.localNames)
+        && !(lib.elem "forge.t.local" bare.services.hyperhive.gateway.localNames);
-        && !(refusedOver bare "deploy.forgejo.sso.clientSecretFile");
     }
     {
       # All three are published targets; ./swarm-otel-identity.nix covers them.
@@ -178,7 +169,7 @@ let
       ok =
         forgeClientIds servicesForgeElsewhere == [ "forgejo" ]
         && forgeClientIds allLocal == [ "forgejo" ]
-        && forgeClientIds forgeNoAuthelia == [ ];
+        && forgeClientIds forgeAlone == [ ];
     }
     {
       name = "the forge's OIDC callback is the one forgejo sends";
@@ -370,10 +361,8 @@ let
     }
     {
       # The forge's OIDC secret has one route, the store, under the forge
-      # publisher's leaf, landing where `clientSecretFile` points inside the
+      # publisher's leaf. No unit copies it out of authelia's tree, even on a
-      # container. No unit copies it out of authelia's tree, even on a host
+      # host that runs authelia, and nothing defines the removed path option.
-      # that runs authelia, and a forge host without authelia needs no path
-      # set by hand.
       name = "the forge reads its OIDC secret from the store under the forge publisher's leaf";
       ok =
         let
@@ -382,12 +371,11 @@ let
         in
         !d.authelia.enable
         && !d.bao.enable
-        && d.forgejo.sso.clientSecretFile == "/var/lib/forgejo-oidc/forgejo.secret"
         && !(refusedOver forgeAlone "deploy.forgejo.sso.clientSecretFile")
+        && !(refusedOver allLocal "deploy.forgejo.sso.clientSecretFile")
         && u.environment.BAO_CLIENT_CERT == "/etc/pki/forge-publisher.pem"
         && u.environment.BAO_CLIENT_KEY == "/etc/pki/forge-publisher-key.pem"
         && lib.hasInfix "bao kv get -field=value secret/swarm/services/forgejo/oidc/client" u.script
-        && lib.hasInfix "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/forgejo.secret" u.script
         && lib.hasInfix "refresh_consumer hive-forge forgejo-sso-source.service" u.script
         # An unpublished or empty secret fails the unit, so `storeRetry` retries it.
         && !(lib.hasInfix "exit 0" u.script)
@@ -400,13 +388,27 @@ let
         && !(bare.systemd.services ? swarm-bao-forge-oidc);
     }
     {
-      # An operator-chosen path is where the fetched secret lands.
+      # The fetched secret lives in a host tmpfs directory bound read-only into
-      name = "the forge's OIDC secret lands where clientSecretFile points";
+      # the container, and reaches the login source as a credential. Nothing
+      # writes it under the container's persistent root, and the copy an
+      # earlier fetch left there is removed.
+      name = "the forge's OIDC secret reaches its login source from /run, not from persistent disk";
       ok =
         let
-          s = (oidcFetch forgeNoAuthelia).script;
+          s = (oidcFetch forgeAlone).script;
+          c = forgeAlone.containers.hive-forge;
+          src = c.config.systemd.services.forgejo-sso-source;
+          rules = forgeAlone.systemd.tmpfiles.rules;
         in
-        lib.hasInfix "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/by-hand.secret" s;
+        lib.hasInfix "atomic_write_secret 0400" s
+        && lib.hasInfix "/run/hive-forge-oidc/forgejo.secret" s
+        && !(lib.hasInfix "/var/lib/nixos-containers/hive-forge" s)
+        && c.bindMounts."/run/hive-forge-oidc".hostPath == "/run/hive-forge-oidc"
+        && c.bindMounts."/run/hive-forge-oidc".isReadOnly
+        && lib.elem "oidc-client-secret:/run/hive-forge-oidc/forgejo.secret" src.serviceConfig.LoadCredential
+        && lib.hasInfix "$CREDENTIALS_DIRECTORY/oidc-client-secret" src.script
+        && lib.elem "d /run/hive-forge-oidc 0700 root root -" rules
+        && lib.elem "r /var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/forgejo.secret" rules;
     }
     {
       # Beside the store, the fetch waits for the unit writing the forge

nix/module-eval/nats-tls.nix

@@ -66,12 +66,10 @@ let
   # address set by hand: what every hive but one in a multi-host swarm looks
   # like. The controller is on too, since it may run away from the queue.
   #
-  # The forge's secret and the controller's store identity are what a hive
+  # The controller's store identity is what a hive away from the store already
-  # away from authelia and the store already has to be handed, and none is an
+  # has to be handed, and it is not an address; without it this hive would
-  # address; without them this hive would fail assertions that have nothing to
+  # fail assertions that have nothing to do with the queue.
-  # do with the queue.
   remoteSecrets = {
-    deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret";
     deploy.swarm-controller.baoClientCertFile = "/var/lib/swarm-controller/bao-client.pem";
     deploy.swarm-controller.baoClientKeyFile = "/var/lib/swarm-controller/bao-client-key.pem";
   };

nix/module-eval/swarm-otel-identity.nix

@@ -34,7 +34,6 @@ let
   otelNoIdentity = hive {
     deploy.swarm-otel.enable = true;
     swarm.authelia.url = "https://auth.example.invalid";
-    deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret";
   };
 
   # 🩸 The shape the degrade above must not swallow: a collector on a host that