vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

bao: write the swarm controller's policy from inside the store

Browse source 
Cert auth answers a role, so nothing in the swarm can authenticate to
the store until some role exists. Creating the first one therefore
cannot itself use a certificate — the credential has to come from
outside that cycle, and an operator places it.

`deploy.bao.bootstrapTokenFile` names that token. A one-shot unit inside
the store's container reads it and writes the `swarm-controller` policy;
`local-defaults.nix` supplies the path (never the file) on an all-local
deploy, since co-location makes only the location derivable. `bao
operator init` stays an operator action in every shape.

The unit runs in the container rather than on the host because writing
the first grant is a store-side operation: it reaches the store locally
and needs no client certificate. Gating it on the store being here is
therefore not the co-location assumption glue-matrix-bao-token.nix
warns about — a reader has to work from anywhere, the first write never
does.

Policy only, deliberately: a cert-auth role binds a certificate and the
controller has no bao identity yet — it holds no leaf and contains no
bao code at all. The two certificates that do exist are both wrong to
bind. `clientCertFile` is the store host's own reader leaf rather than
the controller's, and the CA that signed it also signs every other
reader's, so binding that would let any reader authenticate as the
controller. Whoever gives the controller an identity writes the role.

The grants are scoped to `hive-*`. Without the prefix the controller
could rewrite the policy that constrains it.

Two things the module-eval arms pin: the unit renders inside the
container with the token path in both its script and its
ConditionPathExists, and a host that names a token while running no
store gets neither the unit nor the host-side directory.

The capabilities were derived with `bao write -output-policy` rather
than written from memory, and the setup.md commands were run against a
real binary for shape before being written down.

Refs #3726.
This commit is contained in:
atlas 2026-09-07 17:36:24 +02:00 • committed by mara
commit 2f7d3e02e9
4 changed files with 223 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

18
nix/host-modules/local-defaults.nix
View file

@ -129,6 +129,24 @@ in
# itself, and this is a different top-level path.
config.services.hyperhive.deploy.swarm-controller.enable = lib.mkDefault cfg.deploy.singleHostSwarm;
# Where the operator drops the token that writes the swarm's first bao
# grants. All-local supplies the path, never the file: `bao operator init`
# stays a human step in every shape (see ./swarm-bao.nix), so what
# co-location makes derivable is only *where* it goes. The granting unit
# skips until the file appears, so naming the path early costs a boot
# nothing.
#
# ⚠️ Deliberately NOT under `/var/lib/swarm-bao-token`, which already holds
# the PKCS11 seal material. Those are both "a bao token" and neither can
# substitute for the other: one unseals the store, this one authorises the
# first write.
#
# Outside the `swarm` attrset above for the same reason the controller's
# options are — a path on this host is a `deploy.*` fact.
config.services.hyperhive.deploy.bao.bootstrapTokenFile = lib.mkIf cfg.deploy.singleHostSwarm (
lib.mkDefault "/var/lib/swarm-bao-bootstrap/grant.token"
);
# Same reason, one option later: the queue secret is a path on THIS host,
# so it moved to `deploy.*` with the rest of the controller's credentials.
# It has to sit out here rather than in the `swarm` attrset above — a bare

133
nix/host-modules/swarm-bao.nix
View file

@ -138,6 +138,50 @@ let
# rather than as the missing setting it is.
haveServerTls = baoDeploy.serverCertFile != null && baoDeploy.serverKeyFile != null;
# The credential that writes the swarm's first grant. A token and not a
# certificate: cert auth answers a *role*, so nothing can authenticate here
# until some role exists, and whatever creates the first one cannot itself
# use one. An operator places it — ../../docs/getting-started/setup.md.
haveBootstrapToken = baoDeploy.bootstrapTokenFile != null;
# Bind the directory, never the file, for the reason `tokenStoreDir` gives
# above: nothing creates a `hostPath`, so naming a file an operator has not
# placed yet would stop the container from starting.
bootstrapTokenDir =
if haveBootstrapToken then builtins.dirOf baoDeploy.bootstrapTokenFile else null;
# The name both ends must agree on: whoever later gives the controller a
# cert-auth role attaches this policy by spelling it the same way.
controllerPolicyName = "swarm-controller";
# Derived with `bao write -output-policy`, which short-circuits the request
# and needs no server, rather than written from memory.
#
# The `hive-` prefix is load-bearing — without it the controller could
# rewrite the policy that constrains it.
#
# Piped to `bao policy write -` as a shell-quoted argument, not a
# `writeText`: a store path puts the grants behind a hash where
# ../module-eval.nix cannot read them, and a heredoc would make the HCL's
# indentation a function of this file's.
controllerPolicyText = ''
path "auth/cert/certs/hive-*" {
capabilities = ["create", "update", "read", "delete"]
}
path "auth/cert/certs" {
capabilities = ["list"]
}
path "sys/policies/acl/hive-*" {
capabilities = ["create", "update", "read", "delete"]
}
path "sys/policies/acl" {
capabilities = ["list"]
}
'';
# Every listener serves the same identity: they differ in which address
# they answer on, not in who they are. Client verification is separate and
# optional — a store with no `clientCaFile` still serves TLS, it just does
@ -396,6 +440,29 @@ in
'';
};
bootstrapTokenFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-bootstrap/grant.token";
description = ''
Token used **once per swarm** to write the first authorisation grants,
after which every client authenticates with a certificate instead.
Cert auth answers a *role*, so no client can authenticate until some
role exists — and creating that first one is what this token is for.
It has to come from outside that cycle, which is why an operator places
it rather than the deployment minting it.
Produce it from the root token `bao operator init` printed, scoped to
that one policy write and nothing else, then delete it once the swarm
has come up — {file}`docs/getting-started/setup.md` has the commands.
Setting this is what enables the granting unit; leaving it null means
the deployment writes those grants some other way.
A path, never a value.
'';
};
clientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
@ -692,6 +759,22 @@ in
'';
};
# Same shape and the same create-only reason as the token directory
# above: the operator owns what they drop here, so a rebuild must not
# re-impose ownership on it.
systemd.services.swarm-bao-bootstrap-dir = lib.mkIf haveBootstrapToken {
description = "create the directory the swarm's bao bootstrap token goes in";
before = [ "container@${cfg.machine}.service" ];
requiredBy = [ "container@${cfg.machine}.service" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
};
script = ''
test -d ${bootstrapTokenDir} || install -d -m 0700 ${bootstrapTokenDir}
'';
};
containers.${cfg.machine} = {
autoStart = true;
ephemeral = false;
@ -733,6 +816,14 @@ in
hostPath = "/dev/tpmrm0";
isReadOnly = false;
};
}
// lib.optionalAttrs haveBootstrapToken {
# Read-only: the grant unit only reads it, and removing the file
# once the swarm is up is an operator action on the host.
${bootstrapTokenDir} = {
hostPath = bootstrapTokenDir;
isReadOnly = true;
};
};
# The seal talks to the TPM through the kernel's resource manager, so
@ -903,6 +994,48 @@ in
# `nixos-rebuild switch` — it lands in the config file and waits.
# Restarting is an operator action with an unseal on the far side of
# it, which is why nothing here tries to be clever about it.
# The swarm's first grant, written from in here because this is
# where the store is reachable without a client certificate — which
# is the point, since no role exists yet to issue one against.
systemd.services.swarm-bao-controller-policy = lib.mkIf haveBootstrapToken {
description = "write the swarm controller's bao policy";
after = [ "openbao.service" ];
wantedBy = [ "multi-user.target" ];
path = [
baoDeploy.package
pkgs.coreutils
];
# Named but not placed is a legitimate state: all-local supplies
# the path as a default and the operator drops the file there
# after `bao operator init`. Skipping rather than failing is also
# what makes deleting the token at the end of that procedure safe.
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
# A store that is up is not necessarily unsealed — under
# `seal = "shamir"` an operator unseals by hand after every
# restart — so early attempts legitimately fail. Bounded,
# because a token that is wrong rather than early would
# otherwise retry forever.
Restart = "on-failure";
RestartSec = 30;
StartLimitBurst = 10;
};
environment.BAO_ADDR = "https://${cfg.domain}:${toString cfg.port}";
script = ''
set -euo pipefail
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
export BAO_TOKEN
# Idempotent on purpose: a rebuild re-asserts the policy rather
# than failing on one that already exists.
printf '%s' ${lib.escapeShellArg controllerPolicyText} |
bao policy write ${lib.escapeShellArg controllerPolicyName} -
'';
};
};
};
})