From 20da00735151c8842ccfe9405de12cbfc039cbe9 Mon Sep 17 00:00:00 2001 From: atlas Date: Thu, 10 Sep 2026 20:11:14 +0200 Subject: [PATCH 1/3] swarm-bao: bound the granting unit's restarts for real MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `StartLimitBurst` sat in `serviceConfig`, so it rendered into `[Service]`, where systemd silently ignores it — the unit retried every 30s forever. Measured on a live store: the journal reports `restart counter is at 18` against a burst of 10. This repo already states the rule and pins it with a test: `hive-priv/src/main.rs` renders its drop-in with `StartLimit*` under `[Unit]` and says why — "systemd silently ignores them under `[Service]`, so a bound that moved sections would look configured and do nothing". That is exactly what happened here, in another module. Moving the burst alone would not have fixed it. systemd's default window is 10s while `RestartSec = 30`, so at most one restart falls inside it and a burst of 10 is unreachable; the interval has to exceed `RestartSec` times the burst. 600 matches the value hive-priv already uses. Uses the NixOS service-level options rather than a hand-written `unitConfig`: nixpkgs renders `startLimitBurst` / `startLimitIntervalSec` into `unitConfig` itself (`nixos/lib/systemd-lib.nix`), and `hive-ci.nix` already sets `startLimitIntervalSec` that way. The module-eval case asserts placement where nixpkgs puts it, and that `serviceConfig` does not carry it — so moving it back fails the build. --- nix/host-modules/swarm-bao.nix | 18 ++++++++++++------ nix/module-eval.nix | 15 +++++++++++++++ 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index 83f77383..282f9687 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -1092,17 +1092,23 @@ in # after `bao operator init`. Skipping rather than failing is also # what makes deleting the token at the end of that procedure safe. unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile; + # A store that is up is not necessarily unsealed — under + # `seal = "shamir"` an operator unseals by hand after every + # restart — so early attempts legitimately fail. Bounded, + # because a token that is wrong rather than early would + # otherwise retry forever. + # + # `StartLimit*` are `[Unit]` settings that systemd ignores under + # `[Service]`, which is what these two options render to. The + # interval also has to exceed `RestartSec × burst`, or the window + # closes between attempts and the burst is never reached. + startLimitBurst = 10; + startLimitIntervalSec = 600; serviceConfig = { Type = "oneshot"; RemainAfterExit = true; - # A store that is up is not necessarily unsealed — under - # `seal = "shamir"` an operator unseals by hand after every - # restart — so early attempts legitimately fail. Bounded, - # because a token that is wrong rather than early would - # otherwise retry forever. Restart = "on-failure"; RestartSec = 30; - StartLimitBurst = 10; }; environment.BAO_ADDR = "https://${cfg.domain}:${toString cfg.port}"; script = '' diff --git a/nix/module-eval.nix b/nix/module-eval.nix index d51f92fe..625fc875 100644 --- a/nix/module-eval.nix +++ b/nix/module-eval.nix @@ -536,6 +536,21 @@ let lib.hasInfix "/run/secrets/bao-bootstrap.token" u.script && u.unitConfig.ConditionPathExists == "/run/secrets/bao-bootstrap.token"; } + { + # `StartLimit*` are `[Unit]` settings that systemd ignores under + # `[Service]`, so a bound written into `serviceConfig` renders, deploys + # and does nothing. Asserted where nixpkgs puts it rather than where it + # was written, and the interval is part of the bound: it has to exceed + # `RestartSec × burst` or the window closes between attempts. + name = "the granting unit's start limit lands in [Unit], not [Service]"; + ok = + let + u = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy; + in + toString u.unitConfig.StartLimitBurst == "10" + && toString u.unitConfig.StartLimitIntervalSec == "600" + && !(u.serviceConfig ? StartLimitBurst); + } { # The grants themselves, and the `hive-` prefix is the whole point: # without it the controller can rewrite the policy that constrains it, From 16182c670eed0c595bc2f88eac713a9d354fb4c4 Mon Sep 17 00:00:00 2001 From: atlas Date: Thu, 10 Sep 2026 21:14:23 +0200 Subject: [PATCH 2/3] swarm-bao: write the first grant from the host, not the container MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `swarm-bao-controller-policy` creates the `swarm-controller` policy and cert-auth role — the credential every hive logs in with. It has never succeeded on any deployment, and the reason is where it ran. Inside the container it had neither of the two things the store demands. Its `BAO_ADDR` was the public DNS name, which from that netns resolves to the hive bridge: `dial tcp 10.42.0.1:8200: connect: connection refused`. And every API listener carries `tls_require_and_verify_client_cert`, while `tlsDir` holds the server's leaf and the CA that signs clients — no client identity at all. Fixing only the address moves the failure one hop. The comment above the unit asserted the opposite — that in there the store is "reachable without a client certificate at all, which is the point". The listener config decides that, and says otherwise. That belief is what put the unit in the container, so it goes with it. On the host all four coordinates already exist: `baoCli` carries the address, the CA, the certificate and the key, so the unit needs no `environment` block at all. `bootstrapTokenFile` was always a host path — the container only saw it through a bind mount. Nothing new crosses the boundary; the mount gets no wider. The retry bound is resized with it. 10 attempts at 30s is five minutes, and under `seal = "shamir"` an operator unseals by hand, so it would give up before a human arrived — permanently, because `start-limit-hit` does not self-heal. That is the same silent no-bootstrap this issue is about. 2880 × 30s covers a day, inside a 25h window. module-eval follows the unit to the host and gains an arm asserting it is NOT rendered inside the container: the move is the fix, so the side it landed on is worth pinning. --- nix/host-modules/swarm-bao.nix | 160 +++++++++++++++++---------------- nix/module-eval.nix | 41 +++++---- 2 files changed, 106 insertions(+), 95 deletions(-) diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index 282f9687..9d538a65 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -856,6 +856,88 @@ in ''; }; + # The swarm's first grant, written from the HOST. Every API listener sets + # `tls_require_and_verify_client_cert`, so a client needs an identity + # wherever it runs — and only the host has one. The bootstrap token is a + # host path too; the container saw it through a bind mount. + systemd.services.swarm-bao-controller-policy = lib.mkIf haveBootstrapToken { + description = "write the swarm controller's bao policy and cert-auth role"; + after = [ "container@${cfg.machine}.service" ]; + wantedBy = [ "multi-user.target" ]; + # The wrapper rather than the package: it carries the address, the CA + # and this host's certificate, which is what makes running here cheaper + # than shipping an identity the other way. + path = [ + baoCli + pkgs.coreutils + ]; + # Named but not placed is a legitimate state: all-local supplies the + # path as a default and the operator drops the file there after + # `bao operator init`. Skipping rather than failing is also what makes + # deleting the token at the end of that procedure safe. + unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile; + # A store that is up is not necessarily unsealed — under + # `seal = "shamir"` an operator unseals BY HAND, so early attempts fail + # for as long as that takes, which can be a day. + # + # `StartLimit*` are `[Unit]` settings that systemd ignores under + # `[Service]`, which is what these two options render to. The window has + # to exceed `RestartSec × burst`, or it closes between attempts and the + # burst is never reached: 2880 × 30s is 24h, inside a 25h window. + # Bounding it tighter would give up before a human could unseal, and + # `start-limit-hit` does not self-heal. + startLimitBurst = 2880; + startLimitIntervalSec = 90000; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + Restart = "on-failure"; + RestartSec = 30; + }; + script = '' + set -euo pipefail + + BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})" + export BAO_TOKEN + + # Idempotent on purpose: a rebuild re-asserts the policy rather + # than failing on one that already exists. + printf '%s' ${lib.escapeShellArg controllerPolicyText} | + bao policy write ${lib.escapeShellArg controllerPolicyName} - + '' + + lib.optionalString (baoDeploy.clientCaFile != null) '' + + # The policy above grants paths under `auth/cert/`, and nothing + # in this tree creates that mount. Without this, the grant names + # a location that does not exist and every certificate login + # fails — the controller's own, and the per-hive ones it later + # issues against the same mount. + # + # Asked rather than attempted: `auth enable` errors on a mount + # that already exists, and recognising that would tie a rebuild + # to an error string we have never seen this store emit. + mounted="$(bao auth list -format=json)" + case "$mounted" in + *'"cert/"'*) ;; + *) bao auth enable cert ;; + esac + + # `certificate=` is the CA, so this role trusts every leaf that + # CA signed and `allowed_common_names` is the whole of what + # narrows it to one identity. ⚠️ The same CA signs each hive's + # reader leaf with CN = the hive's name, so a hive named + # `${controllerCn}` would satisfy this role. + # + # Named outside the `hive-*` namespace the policy grants, so the + # controller cannot rewrite the role that constrains it. + bao write auth/cert/certs/${lib.escapeShellArg controllerPolicyName} \ + certificate=@${tlsDir}/client-ca.pem \ + allowed_common_names=${lib.escapeShellArg controllerCn} \ + token_policies=${lib.escapeShellArg controllerPolicyName} \ + display_name=${lib.escapeShellArg controllerCn} + ''; + }; + containers.${cfg.machine} = { autoStart = true; ephemeral = false; @@ -1076,84 +1158,6 @@ in # Restarting is an operator action with an unseal on the far side of # it, which is why nothing here tries to be clever about it. - # The swarm's first grant, written from in here because this is - # where the store is reachable without a client certificate — which - # is the point, since no role exists yet to issue one against. - systemd.services.swarm-bao-controller-policy = lib.mkIf haveBootstrapToken { - description = "write the swarm controller's bao policy and cert-auth role"; - after = [ "openbao.service" ]; - wantedBy = [ "multi-user.target" ]; - path = [ - baoDeploy.package - pkgs.coreutils - ]; - # Named but not placed is a legitimate state: all-local supplies - # the path as a default and the operator drops the file there - # after `bao operator init`. Skipping rather than failing is also - # what makes deleting the token at the end of that procedure safe. - unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile; - # A store that is up is not necessarily unsealed — under - # `seal = "shamir"` an operator unseals by hand after every - # restart — so early attempts legitimately fail. Bounded, - # because a token that is wrong rather than early would - # otherwise retry forever. - # - # `StartLimit*` are `[Unit]` settings that systemd ignores under - # `[Service]`, which is what these two options render to. The - # interval also has to exceed `RestartSec × burst`, or the window - # closes between attempts and the burst is never reached. - startLimitBurst = 10; - startLimitIntervalSec = 600; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - Restart = "on-failure"; - RestartSec = 30; - }; - environment.BAO_ADDR = "https://${cfg.domain}:${toString cfg.port}"; - script = '' - set -euo pipefail - - BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})" - export BAO_TOKEN - - # Idempotent on purpose: a rebuild re-asserts the policy rather - # than failing on one that already exists. - printf '%s' ${lib.escapeShellArg controllerPolicyText} | - bao policy write ${lib.escapeShellArg controllerPolicyName} - - '' - + lib.optionalString (baoDeploy.clientCaFile != null) '' - - # The policy above grants paths under `auth/cert/`, and nothing - # in this tree creates that mount. Without this, the grant names - # a location that does not exist and every certificate login - # fails — the controller's own, and the per-hive ones it later - # issues against the same mount. - # - # Asked rather than attempted: `auth enable` errors on a mount - # that already exists, and recognising that would tie a rebuild - # to an error string we have never seen this store emit. - mounted="$(bao auth list -format=json)" - case "$mounted" in - *'"cert/"'*) ;; - *) bao auth enable cert ;; - esac - - # `certificate=` is the CA, so this role trusts every leaf that - # CA signed and `allowed_common_names` is the whole of what - # narrows it to one identity. ⚠️ The same CA signs each hive's - # reader leaf with CN = the hive's name, so a hive named - # `${controllerCn}` would satisfy this role. - # - # Named outside the `hive-*` namespace the policy grants, so the - # controller cannot rewrite the role that constrains it. - bao write auth/cert/certs/${lib.escapeShellArg controllerPolicyName} \ - certificate=@${tlsDir}/client-ca.pem \ - allowed_common_names=${lib.escapeShellArg controllerCn} \ - token_policies=${lib.escapeShellArg controllerPolicyName} \ - display_name=${lib.escapeShellArg controllerCn} - ''; - }; }; }; }) diff --git a/nix/module-eval.nix b/nix/module-eval.nix index 625fc875..7a48f032 100644 --- a/nix/module-eval.nix +++ b/nix/module-eval.nix @@ -525,30 +525,38 @@ let otelRemoteAuthelia.containers.swarm-otel.config.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential; } { - # Reads the rendered unit INSIDE the container, which is where the write - # happens: reaching the store locally is what lets the grant be written - # without a client certificate at all. - name = "a store host with a placed bootstrap token renders the granting unit inside the container"; + # Reads the rendered unit on the HOST, which is where the write happens: + # every API listener demands a client certificate, and the host is the + # side that has one. + name = "a store host with a placed bootstrap token renders the granting unit on the host"; ok = let - u = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy; + u = baoGrantHere.systemd.services.swarm-bao-controller-policy; in lib.hasInfix "/run/secrets/bao-bootstrap.token" u.script && u.unitConfig.ConditionPathExists == "/run/secrets/bao-bootstrap.token"; } + { + # The move is the fix, so pin the side it landed on: in the container it + # had no identity to open a connection with, and no address that resolved + # to the store from its own netns. + name = "the granting unit is not rendered inside the store's container"; + ok = !(baoGrantHere.containers.swarm-bao.config.systemd.services ? swarm-bao-controller-policy); + } { # `StartLimit*` are `[Unit]` settings that systemd ignores under # `[Service]`, so a bound written into `serviceConfig` renders, deploys # and does nothing. Asserted where nixpkgs puts it rather than where it - # was written, and the interval is part of the bound: it has to exceed - # `RestartSec × burst` or the window closes between attempts. + # was written. The values are pinned because they are the bound: under + # `shamir` a human unseals by hand, and anything shorter than a day gives + # up first — `start-limit-hit` does not self-heal. name = "the granting unit's start limit lands in [Unit], not [Service]"; ok = let - u = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy; + u = baoGrantHere.systemd.services.swarm-bao-controller-policy; in - toString u.unitConfig.StartLimitBurst == "10" - && toString u.unitConfig.StartLimitIntervalSec == "600" + toString u.unitConfig.StartLimitBurst == "2880" + && toString u.unitConfig.StartLimitIntervalSec == "90000" && !(u.serviceConfig ? StartLimitBurst); } { @@ -560,7 +568,7 @@ let name = "the controller's bao grants cannot reach the policy that constrains it"; ok = let - s = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script; + s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script; in lib.hasInfix "sys/policies/acl/hive-*" s && !(lib.hasInfix "sys/policies/acl/*" s); } @@ -574,7 +582,7 @@ let name = "the controller may write agent credentials, and only under the agent prefix"; ok = let - s = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script; + s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script; in lib.hasInfix "secret/data/swarm/agents/*" s && !(lib.hasInfix "secret/data/*" s) @@ -588,7 +596,7 @@ let name = "the controller's grant on agent credentials is write-only"; ok = let - s = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script; + s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script; in lib.hasInfix "path \"secret/data/swarm/agents/*\" {\n capabilities = [\"create\", \"update\"]" s; } @@ -599,7 +607,7 @@ let name = "the granting unit creates the cert auth mount and the controller's role"; ok = let - s = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script; + s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script; in lib.hasInfix "bao auth enable cert" s && lib.hasInfix "auth/cert/certs/swarm-controller" s @@ -752,7 +760,7 @@ let ok = let m = baoControllerHere; - role = m.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script; + role = m.systemd.services.swarm-bao-controller-policy.script; pki = m.systemd.services.swarm-bao-pki.script; in lib.hasInfix "cn-marker-not-a-default" role && lib.hasInfix "cn-marker-not-a-default" pki; @@ -768,8 +776,7 @@ let name = "with no client CA the unit still writes the policy and skips the role"; ok = let - s = - baoGrantNoClientCa.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script; + s = baoGrantNoClientCa.systemd.services.swarm-bao-controller-policy.script; in lib.hasInfix "bao policy write" s && !(lib.hasInfix "bao auth enable cert" s) From bf544369374eeb529d2e9049af24d569e4ae4a2d Mon Sep 17 00:00:00 2001 From: atlas Date: Thu, 10 Sep 2026 21:22:57 +0200 Subject: [PATCH 3/3] docs/setup: the granting unit runs on the host now MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bootstrap block still described a one-shot unit "inside the store's container". That stopped being true when the unit moved, and a merged change that leaves the docs asserting the old topology is worse than one that never moved it. Also retires the "only run once, didn't go all the way through" note. It has run twice against a live store now and failed both times — but for its own reasons, not the grants': it could not reach the store from where it ran, so neither attempt exercised a single one of them. The grants are still derived from `-output-policy` rather than proven, and the warning now says that directly instead of implying a partial test. --- docs/getting-started/setup.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/getting-started/setup.md b/docs/getting-started/setup.md index 84fbabcc..cd0bdacb 100644 --- a/docs/getting-started/setup.md +++ b/docs/getting-started/setup.md @@ -103,16 +103,17 @@ bao token create -policy=swarm-bootstrap -ttl=24h -orphan -display-name=swarm-bo ``` Put the token's value at `services.hyperhive.deploy.bao.bootstrapTokenFile` -(all-local names that path for you), then rebuild. A one-shot unit inside the -store's container reads it, writes the `swarm-controller` policy, enables the -cert auth method, and creates the `swarm-controller` role that attaches the two. +(all-local names that path for you), then rebuild. A one-shot unit **on the +host** reads it, writes the `swarm-controller` policy, enables the cert auth +method, and creates the `swarm-controller` role that attaches the two. It runs +there because every API listener demands a client certificate, and the host is +the side that has one. -⚠️ **This has only run once, and it didn't go all the way through.** The -first real provision against a live store returned 403, so -treat the block above as derived, then only partly tested — not proven: the -grants still come from `-output-policy`, not from a swarm that came up on them. -What that 403 does _not_ tell you is whether the login or the write was -refused — `bao read auth/cert/certs/swarm-controller` separates the two. +⚠️ **The grants above are derived, not proven.** They come from +`-output-policy`, not from a swarm that came up on them. The unit has run twice +against a live store and failed both times for reasons of its own — it could +not reach the store from where it then ran — so neither run exercised these +grants at all. **Delete the file only once `bao read auth/cert/certs/swarm-controller` returns the role.** The unit skips when the token is absent, so a host that has finished