vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

Compare commits

base: vinzenz:a95a81bb12ff61887c07708cf43d557b3c0511bb
Branches Tags
vinzenz:main
...
compare: vinzenz:bf544369374eeb529d2e9049af24d569e4ae4a2d
Branches Tags
vinzenz:main
Author SHA1 Message Date
atlas
bf54436937 docs/setup: the granting unit runs on the host now 
The bootstrap block still described a one-shot unit "inside the store's
container". That stopped being true when the unit moved, and a merged
change that leaves the docs asserting the old topology is worse than one
that never moved it.

Also retires the "only run once, didn't go all the way through" note. It
has run twice against a live store now and failed both times — but for
its own reasons, not the grants': it could not reach the store from where
it ran, so neither attempt exercised a single one of them. The grants are
still derived from `-output-policy` rather than proven, and the warning
now says that directly instead of implying a partial test.
 2026-09-10 22:19:48 +02:00
atlas
16182c670e swarm-bao: write the first grant from the host, not the container 
`swarm-bao-controller-policy` creates the `swarm-controller` policy and
cert-auth role — the credential every hive logs in with. It has never
succeeded on any deployment, and the reason is where it ran.

Inside the container it had neither of the two things the store demands.
Its `BAO_ADDR` was the public DNS name, which from that netns resolves to
the hive bridge: `dial tcp 10.42.0.1:8200: connect: connection refused`.
And every API listener carries `tls_require_and_verify_client_cert`, while
`tlsDir` holds the server's leaf and the CA that signs clients — no client
identity at all. Fixing only the address moves the failure one hop.

The comment above the unit asserted the opposite — that in there the store
is "reachable without a client certificate at all, which is the point". The
listener config decides that, and says otherwise. That belief is what put
the unit in the container, so it goes with it.

On the host all four coordinates already exist: `baoCli` carries the
address, the CA, the certificate and the key, so the unit needs no
`environment` block at all. `bootstrapTokenFile` was always a host path —
the container only saw it through a bind mount. Nothing new crosses the
boundary; the mount gets no wider.

The retry bound is resized with it. 10 attempts at 30s is five minutes,
and under `seal = "shamir"` an operator unseals by hand, so it would give
up before a human arrived — permanently, because `start-limit-hit` does
not self-heal. That is the same silent no-bootstrap this issue is about.
2880 × 30s covers a day, inside a 25h window.

module-eval follows the unit to the host and gains an arm asserting it is
NOT rendered inside the container: the move is the fix, so the side it
landed on is worth pinning.
 2026-09-10 22:19:48 +02:00
atlas
20da007351 swarm-bao: bound the granting unit's restarts for real 
`StartLimitBurst` sat in `serviceConfig`, so it rendered into `[Service]`,
where systemd silently ignores it — the unit retried every 30s forever.
Measured on a live store: the journal reports `restart counter is at 18`
against a burst of 10.

This repo already states the rule and pins it with a test:
`hive-priv/src/main.rs` renders its drop-in with `StartLimit*` under
`[Unit]` and says why — "systemd silently ignores them under `[Service]`,
so a bound that moved sections would look configured and do nothing".
That is exactly what happened here, in another module.

Moving the burst alone would not have fixed it. systemd's default window
is 10s while `RestartSec = 30`, so at most one restart falls inside it and
a burst of 10 is unreachable; the interval has to exceed `RestartSec` times
the burst. 600 matches the value hive-priv already uses.

Uses the NixOS service-level options rather than a hand-written
`unitConfig`: nixpkgs renders `startLimitBurst` / `startLimitIntervalSec`
into `unitConfig` itself (`nixos/lib/systemd-lib.nix`), and `hive-ci.nix`
already sets `startLimitIntervalSec` that way.

The module-eval case asserts placement where nixpkgs puts it, and that
`serviceConfig` does not carry it — so moving it back fails the build.
 2026-09-10 22:19:48 +02:00
3 changed files with 127 additions and 94 deletions
Download patch file Download diff file Expand all files Collapse all files

19
docs/getting-started/setup.md
View file

@ -103,16 +103,17 @@ bao token create -policy=swarm-bootstrap -ttl=24h -orphan -display-name=swarm-bo
``` ```
Put the token's value at `services.hyperhive.deploy.bao.bootstrapTokenFile` Put the token's value at `services.hyperhive.deploy.bao.bootstrapTokenFile`
(all-local names that path for you), then rebuild. A one-shot unit inside the (all-local names that path for you), then rebuild. A one-shot unit **on the
store's container reads it, writes the `swarm-controller` policy, enables the host** reads it, writes the `swarm-controller` policy, enables the cert auth
cert auth method, and creates the `swarm-controller` role that attaches the two. method, and creates the `swarm-controller` role that attaches the two. It runs
there because every API listener demands a client certificate, and the host is
the side that has one.
⚠️ **This has only run once, and it didn't go all the way through.** The ⚠️ **The grants above are derived, not proven.** They come from
first real provision against a live store returned 403, so `-output-policy`, not from a swarm that came up on them. The unit has run twice
treat the block above as derived, then only partly tested — not proven: the against a live store and failed both times for reasons of its own — it could
grants still come from `-output-policy`, not from a swarm that came up on them. not reach the store from where it then ran — so neither run exercised these
What that 403 does _not_ tell you is whether the login or the write was grants at all.
refused — `bao read auth/cert/certs/swarm-controller` separates the two.
**Delete the file only once `bao read auth/cert/certs/swarm-controller` returns **Delete the file only once `bao read auth/cert/certs/swarm-controller` returns
the role.** The unit skips when the token is absent, so a host that has finished the role.** The unit skips when the token is absent, so a host that has finished

154
nix/host-modules/swarm-bao.nix
View file

@ -856,6 +856,88 @@ in
''; '';
}; };
# The swarm's first grant, written from the HOST. Every API listener sets
# `tls_require_and_verify_client_cert`, so a client needs an identity
# wherever it runs — and only the host has one. The bootstrap token is a
# host path too; the container saw it through a bind mount.
systemd.services.swarm-bao-controller-policy = lib.mkIf haveBootstrapToken {
description = "write the swarm controller's bao policy and cert-auth role";
after = [ "container@${cfg.machine}.service" ];
wantedBy = [ "multi-user.target" ];
# The wrapper rather than the package: it carries the address, the CA
# and this host's certificate, which is what makes running here cheaper
# than shipping an identity the other way.
path = [
baoCli
pkgs.coreutils
];
# Named but not placed is a legitimate state: all-local supplies the
# path as a default and the operator drops the file there after
# `bao operator init`. Skipping rather than failing is also what makes
# deleting the token at the end of that procedure safe.
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
# A store that is up is not necessarily unsealed — under
# `seal = "shamir"` an operator unseals BY HAND, so early attempts fail
# for as long as that takes, which can be a day.
#
# `StartLimit*` are `[Unit]` settings that systemd ignores under
# `[Service]`, which is what these two options render to. The window has
# to exceed `RestartSec × burst`, or it closes between attempts and the
# burst is never reached: 2880 × 30s is 24h, inside a 25h window.
# Bounding it tighter would give up before a human could unseal, and
# `start-limit-hit` does not self-heal.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
Restart = "on-failure";
RestartSec = 30;
};
script = ''
set -euo pipefail
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
export BAO_TOKEN
# Idempotent on purpose: a rebuild re-asserts the policy rather
# than failing on one that already exists.
printf '%s' ${lib.escapeShellArg controllerPolicyText} |
bao policy write ${lib.escapeShellArg controllerPolicyName} -
''
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
# The policy above grants paths under `auth/cert/`, and nothing
# in this tree creates that mount. Without this, the grant names
# a location that does not exist and every certificate login
# fails — the controller's own, and the per-hive ones it later
# issues against the same mount.
#
# Asked rather than attempted: `auth enable` errors on a mount
# that already exists, and recognising that would tie a rebuild
# to an error string we have never seen this store emit.
mounted="$(bao auth list -format=json)"
case "$mounted" in
*'"cert/"'*) ;;
*) bao auth enable cert ;;
esac
# `certificate=` is the CA, so this role trusts every leaf that
# CA signed and `allowed_common_names` is the whole of what
# narrows it to one identity. ⚠️ The same CA signs each hive's
# reader leaf with CN = the hive's name, so a hive named
# `${controllerCn}` would satisfy this role.
#
# Named outside the `hive-*` namespace the policy grants, so the
# controller cannot rewrite the role that constrains it.
bao write auth/cert/certs/${lib.escapeShellArg controllerPolicyName} \
certificate=@${tlsDir}/client-ca.pem \
allowed_common_names=${lib.escapeShellArg controllerCn} \
token_policies=${lib.escapeShellArg controllerPolicyName} \
display_name=${lib.escapeShellArg controllerCn}
'';
};
containers.${cfg.machine} = { containers.${cfg.machine} = {
autoStart = true; autoStart = true;
ephemeral = false; ephemeral = false;
@ -1076,78 +1158,6 @@ in
# Restarting is an operator action with an unseal on the far side of # Restarting is an operator action with an unseal on the far side of
# it, which is why nothing here tries to be clever about it. # it, which is why nothing here tries to be clever about it.
# The swarm's first grant, written from in here because this is
# where the store is reachable without a client certificate — which
# is the point, since no role exists yet to issue one against.
systemd.services.swarm-bao-controller-policy = lib.mkIf haveBootstrapToken {
description = "write the swarm controller's bao policy and cert-auth role";
after = [ "openbao.service" ];
wantedBy = [ "multi-user.target" ];
path = [
baoDeploy.package
pkgs.coreutils
];
# Named but not placed is a legitimate state: all-local supplies
# the path as a default and the operator drops the file there
# after `bao operator init`. Skipping rather than failing is also
# what makes deleting the token at the end of that procedure safe.
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
# A store that is up is not necessarily unsealed — under
# `seal = "shamir"` an operator unseals by hand after every
# restart — so early attempts legitimately fail. Bounded,
# because a token that is wrong rather than early would
# otherwise retry forever.
Restart = "on-failure";
RestartSec = 30;
StartLimitBurst = 10;
};
environment.BAO_ADDR = "https://${cfg.domain}:${toString cfg.port}";
script = ''
set -euo pipefail
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
export BAO_TOKEN
# Idempotent on purpose: a rebuild re-asserts the policy rather
# than failing on one that already exists.
printf '%s' ${lib.escapeShellArg controllerPolicyText} |
bao policy write ${lib.escapeShellArg controllerPolicyName} -
''
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
# The policy above grants paths under `auth/cert/`, and nothing
# in this tree creates that mount. Without this, the grant names
# a location that does not exist and every certificate login
# fails — the controller's own, and the per-hive ones it later
# issues against the same mount.
#
# Asked rather than attempted: `auth enable` errors on a mount
# that already exists, and recognising that would tie a rebuild
# to an error string we have never seen this store emit.
mounted="$(bao auth list -format=json)"
case "$mounted" in
*'"cert/"'*) ;;
*) bao auth enable cert ;;
esac
# `certificate=` is the CA, so this role trusts every leaf that
# CA signed and `allowed_common_names` is the whole of what
# narrows it to one identity. ⚠️ The same CA signs each hive's
# reader leaf with CN = the hive's name, so a hive named
# `${controllerCn}` would satisfy this role.
#
# Named outside the `hive-*` namespace the policy grants, so the
# controller cannot rewrite the role that constrains it.
bao write auth/cert/certs/${lib.escapeShellArg controllerPolicyName} \
certificate=@${tlsDir}/client-ca.pem \
allowed_common_names=${lib.escapeShellArg controllerCn} \
token_policies=${lib.escapeShellArg controllerPolicyName} \
display_name=${lib.escapeShellArg controllerCn}
'';
};
}; };
}; };
}) })

46
nix/module-eval.nix
View file

@ -525,17 +525,40 @@ let
otelRemoteAuthelia.containers.swarm-otel.config.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential; otelRemoteAuthelia.containers.swarm-otel.config.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential;
} }
{ {
# Reads the rendered unit INSIDE the container, which is where the write # Reads the rendered unit on the HOST, which is where the write happens:
# happens: reaching the store locally is what lets the grant be written # every API listener demands a client certificate, and the host is the
# without a client certificate at all. # side that has one.
name = "a store host with a placed bootstrap token renders the granting unit inside the container"; name = "a store host with a placed bootstrap token renders the granting unit on the host";
ok = ok =
let let
u = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy; u = baoGrantHere.systemd.services.swarm-bao-controller-policy;
in in
lib.hasInfix "/run/secrets/bao-bootstrap.token" u.script lib.hasInfix "/run/secrets/bao-bootstrap.token" u.script
&& u.unitConfig.ConditionPathExists == "/run/secrets/bao-bootstrap.token"; && u.unitConfig.ConditionPathExists == "/run/secrets/bao-bootstrap.token";
} }
{
# The move is the fix, so pin the side it landed on: in the container it
# had no identity to open a connection with, and no address that resolved
# to the store from its own netns.
name = "the granting unit is not rendered inside the store's container";
ok = !(baoGrantHere.containers.swarm-bao.config.systemd.services ? swarm-bao-controller-policy);
}
{
# `StartLimit*` are `[Unit]` settings that systemd ignores under
# `[Service]`, so a bound written into `serviceConfig` renders, deploys
# and does nothing. Asserted where nixpkgs puts it rather than where it
# was written. The values are pinned because they are the bound: under
# `shamir` a human unseals by hand, and anything shorter than a day gives
# up first — `start-limit-hit` does not self-heal.
name = "the granting unit's start limit lands in [Unit], not [Service]";
ok =
let
u = baoGrantHere.systemd.services.swarm-bao-controller-policy;
in
toString u.unitConfig.StartLimitBurst == "2880"
&& toString u.unitConfig.StartLimitIntervalSec == "90000"
&& !(u.serviceConfig ? StartLimitBurst);
}
{ {
# The grants themselves, and the `hive-` prefix is the whole point: # The grants themselves, and the `hive-` prefix is the whole point:
# without it the controller can rewrite the policy that constrains it, # without it the controller can rewrite the policy that constrains it,
@ -545,7 +568,7 @@ let
name = "the controller's bao grants cannot reach the policy that constrains it"; name = "the controller's bao grants cannot reach the policy that constrains it";
ok = ok =
let let
s = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script; s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
in in
lib.hasInfix "sys/policies/acl/hive-*" s && !(lib.hasInfix "sys/policies/acl/*" s); lib.hasInfix "sys/policies/acl/hive-*" s && !(lib.hasInfix "sys/policies/acl/*" s);
} }
@ -559,7 +582,7 @@ let
name = "the controller may write agent credentials, and only under the agent prefix"; name = "the controller may write agent credentials, and only under the agent prefix";
ok = ok =
let let
s = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script; s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
in in
lib.hasInfix "secret/data/swarm/agents/*" s lib.hasInfix "secret/data/swarm/agents/*" s
&& !(lib.hasInfix "secret/data/*" s) && !(lib.hasInfix "secret/data/*" s)
@ -573,7 +596,7 @@ let
name = "the controller's grant on agent credentials is write-only"; name = "the controller's grant on agent credentials is write-only";
ok = ok =
let let
s = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script; s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
in in
lib.hasInfix "path \"secret/data/swarm/agents/*\" {\n capabilities = [\"create\", \"update\"]" s; lib.hasInfix "path \"secret/data/swarm/agents/*\" {\n capabilities = [\"create\", \"update\"]" s;
} }
@ -584,7 +607,7 @@ let
name = "the granting unit creates the cert auth mount and the controller's role"; name = "the granting unit creates the cert auth mount and the controller's role";
ok = ok =
let let
s = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script; s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
in in
lib.hasInfix "bao auth enable cert" s lib.hasInfix "bao auth enable cert" s
&& lib.hasInfix "auth/cert/certs/swarm-controller" s && lib.hasInfix "auth/cert/certs/swarm-controller" s
@ -737,7 +760,7 @@ let
ok = ok =
let let
m = baoControllerHere; m = baoControllerHere;
role = m.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script; role = m.systemd.services.swarm-bao-controller-policy.script;
pki = m.systemd.services.swarm-bao-pki.script; pki = m.systemd.services.swarm-bao-pki.script;
in in
lib.hasInfix "cn-marker-not-a-default" role && lib.hasInfix "cn-marker-not-a-default" pki; lib.hasInfix "cn-marker-not-a-default" role && lib.hasInfix "cn-marker-not-a-default" pki;
@ -753,8 +776,7 @@ let
name = "with no client CA the unit still writes the policy and skips the role"; name = "with no client CA the unit still writes the policy and skips the role";
ok = ok =
let let
s = s = baoGrantNoClientCa.systemd.services.swarm-bao-controller-policy.script;
baoGrantNoClientCa.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script;
in in
lib.hasInfix "bao policy write" s lib.hasInfix "bao policy write" s
&& !(lib.hasInfix "bao auth enable cert" s) && !(lib.hasInfix "bao auth enable cert" s)