vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-bao: drop the default-policy grant; swarm/public/ is read by name

Browse source 
Writing bao's `default` policy needed a new stanza in the granter's own
policy, which only the bootstrap token can upload, so deploying it took
a manual root-token step. Nothing needed it: the one reader of the
callout xkey's public half is `swarm-bao-nats-callout-xkey`, which logs
in as the `swarm-nats-auth` role, and that role's policy already names
the path. `swarm-bao-nats-auth-policy` writes that policy as the granter
on every deploy, under the granter's existing `sys/policies/acl/swarm-*`
grant.

Removed: the `swarm-bao-public-read-policy` unit, its marker block, the
granter's `sys/policies/acl/default` stanza, the granter-role restart of
that unit, the bao.md section and the module-eval cases pinning them.
This commit is contained in:
atlas 2026-10-05 22:13:58 +02:00 • committed by mara
commit 06784e55fa
4 changed files with 31 additions and 224 deletions
Download patch file Download diff file Expand all files Collapse all files

21
docs/swarm/bao.md
View file

@ -55,9 +55,9 @@ the unit's journal instead.
### Residual risk ### Residual risk
The granter is root-equivalent. It may write any `swarm-*` policy, and bao's The granter is root-equivalent. It may write any `swarm-*` policy with any
`default` policy, with any content, and attach a policy to a role that accepts content, and attach it to a role that accepts any certificate; no bao ACL can
any certificate; no bao ACL can constrain what a policy says. What bounds it: constrain what a policy says. What bounds it:
- `nix/host-modules/swarm-bao.nix` renders every policy it writes, and - `nix/host-modules/swarm-bao.nix` renders every policy it writes, and
module-eval pins each principal's grants. **Merging a change to that policy module-eval pins each principal's grants. **Merging a change to that policy
@ -76,21 +76,6 @@ any certificate; no bao ACL can constrain what a policy says. What bounds it:
restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old
one. one.
### What every token may read
Anything under `secret/swarm/public/` is readable by every token the store
issues. `swarm-bao-public-read-policy` puts that one read into bao's `default`
policy, which every token carries, between two marker lines. It reads the
policy, replaces only its own block, and writes the result back when it
differs, so the store's own stanzas stay as the store wrote them. It never
writes a policy it read back empty. The callout xkey's public half lives
there ([`secrets.md`](secrets.md#the-queues-callout-xkey)).
Writing `default` is a grant on the granter's own policy, so on a store
whose granter predates it, the unit says so and exits 0 until the bootstrap
step re-runs. Until then only principals whose policy names a `swarm/public/`
path read it; the queue's responder is one, so the queue doesn't wait on this.
## Sealing ## Sealing
`services.hyperhive.deploy.bao.seal`: `services.hyperhive.deploy.bao.seal`:

119
nix/host-modules/swarm-bao.nix
View file

@ -185,11 +185,9 @@ let
# the services root. The next six set up the agent PKI mount: the mount, its # the services root. The next six set up the agent PKI mount: the mount, its
# root and the `swarm-*` role agent certificates are issued through. No # root and the `swarm-*` role agent certificates are issued through. No
# `root` delete there: every agent's cert-auth role pins that root by value, # `root` delete there: every agent's cert-auth role pins that root by value,
# so replacing it would lock every agent out. The next three configure the # so replacing it would lock every agent out. The last three configure the
# `oidc` auth method: its config, its `swarm-*` roles, and a read on the one # `oidc` auth method: its config, its `swarm-*` roles, and a read on the one
# client secret that config carries. The last is bao's `default` policy, # client secret that config carries. No `sys/auth`: the auth mounts are
# which every token carries: `swarm-bao-public-read-policy` adds
# the `swarm/public/` read to it. No `sys/auth`: the auth mounts are
# created with the bootstrap token by `swarm-bao-granter-role`. # created with the bootstrap token by `swarm-bao-granter-role`.
# #
# Piped as a shell-quoted argument like `controllerPolicyText`, so # Piped as a shell-quoted argument like `controllerPolicyText`, so
@ -274,10 +272,6 @@ let
path "${credentialMountPath}/data/${baoUiClientLeaf}" { path "${credentialMountPath}/data/${baoUiClientLeaf}" {
capabilities = ["read"] capabilities = ["read"]
} }
path "sys/policies/acl/default" {
capabilities = ["create", "read", "update"]
}
''; '';
# What a granting unit prints when the store refuses the granter: the # What a granting unit prints when the store refuses the granter: the
@ -873,8 +867,9 @@ let
]; ];
# The callout xkey, the nix half of `swarm-nats-auth`'s `xkey::SEED_PATH` # The callout xkey, the nix half of `swarm-nats-auth`'s `xkey::SEED_PATH`
# and `xkey::PUBLIC_PATH`. The seed is the responder's alone; the public half # and `xkey::PUBLIC_PATH`. Both are readable by the `swarm-nats-auth` role
# is under `swarm/public/`, which `publicReadBlock` lets every token read. # alone: `swarm/public/` marks the half that is no secret, and no policy
# opens it to every token.
xkeySeedLeaf = "swarm/services/swarm-nats/callout/xkey-seed"; xkeySeedLeaf = "swarm/services/swarm-nats/callout/xkey-seed";
xkeyPublicLeaf = "swarm/public/swarm-nats/callout-xkey"; xkeyPublicLeaf = "swarm/public/swarm-nats/callout-xkey";
@ -884,20 +879,6 @@ let
} }
''; '';
# What `swarm/public/` is: readable by every token the store issues, through
# bao's `default` policy, which every token carries. A block between two
# marker lines, so `swarm-bao-public-read-policy` can replace it and leave
# the rest of that policy, the store's own, as it found it.
publicReadBegin = "# hyperhive: swarm/public begin";
publicReadEnd = "# hyperhive: swarm/public end";
publicReadBlock = lib.concatStringsSep "\n" [
publicReadBegin
''path "${credentialMountPath}/data/swarm/public/*" {''
" capabilities = [\"read\"]"
"}"
publicReadEnd
];
# The CI runner's token fetch. The one reader of the path, which the # The CI runner's token fetch. The one reader of the path, which the
# controller writes and cannot read back. # controller writes and cannot read back.
ciRunnerReaders = [ ciRunnerReaders = [
@ -2574,12 +2555,12 @@ in
Restart = "on-failure"; Restart = "on-failure";
RestartSec = 30; RestartSec = 30;
# Runs only once the script above exits 0, which is when the # Runs only once the script above exits 0, which is when the
# granter may first configure `auth/oidc` and write bao's `default` # granter may first configure `auth/oidc`; the viewer unit exits 0
# policy; the viewer and public-read units exit 0 without doing so # without doing so before then, and nothing else re-runs it. A
# before then, and nothing else re-runs them. A restart because # restart because it is `RemainAfterExit`, where a start is a
# they are `RemainAfterExit`, where a start is a no-op. # no-op.
# #
# `--no-block`: both units are ordered after this one, so a # `--no-block`: the viewer unit is ordered after this one, so a
# blocking restart waits on a job that cannot run until this unit # blocking restart waits on a job that cannot run until this unit
# is active, which it is not until this command returns. # is active, which it is not until this command returns.
# #
@ -2588,7 +2569,7 @@ in
# already running (or a `systemctl` that itself fails to enqueue # already running (or a `systemctl` that itself fails to enqueue
# the job) would mark the granter failed and trigger its own # the job) would mark the granter failed and trigger its own
# `Restart=on-failure` for a problem that isn't the granter's. # `Restart=on-failure` for a problem that isn't the granter's.
ExecStartPost = "-${pkgs.systemd}/bin/systemctl restart --no-block swarm-bao-operator-viewer-policy.service swarm-bao-public-read-policy.service"; ExecStartPost = "-${pkgs.systemd}/bin/systemctl restart --no-block swarm-bao-operator-viewer-policy.service";
}; };
script = '' script = ''
set -euo pipefail set -euo pipefail
@ -3419,84 +3400,6 @@ in
''; '';
}; };
# `publicReadBlock` in bao's `default` policy. Read, the old block
# dropped, the current one appended, and written back only when that
# differs, so the store's own stanzas stay as the store wrote them.
#
# The granter may write `default` only once `swarm-bao-granter-role` has
# run with a bootstrap token since that grant was added. Until then this
# says so and exits 0, as the viewer unit does: principals granted a
# `swarm/public/` path by name still read it, and a day of retries would
# change nothing. `swarm-bao-granter-role` restarts this unit each time
# it succeeds.
systemd.services.swarm-bao-public-read-policy = lib.mkIf haveGranter {
description = "let every bao token read swarm/public/ through the default policy";
after = [
"container@${cfg.machine}.service"
"swarm-bao-controller-policy.service"
]
++ granterAfter;
requires = [ "swarm-bao-pki.service" ];
wantedBy = [ "multi-user.target" ];
path = [
baoCli
pkgs.coreutils
pkgs.gawk
];
environment = granterEnv;
# Same unseal wait as its siblings above, for the reason stated there.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
Restart = "on-failure";
RestartSec = 30;
};
script = ''
set -euo pipefail
${granterLogin}
caps="$(bao token capabilities sys/policies/acl/default)"
case "$caps" in
*update*) ;;
*)
echo "the granter may not write the store's default policy yet (capabilities: $caps), so swarm/public/ is readable only where a policy names it." >&2
echo "one-time step, as root on this host (docs/getting-started/setup.md):" >&2
${lib.concatMapStringsSep "\n" (l: "echo ${lib.escapeShellArg " ${l}"} >&2") granterSetupSteps}
echo "swarm-bao-granter-role re-runs this unit once it succeeds." >&2
exit 0
;;
esac
if ! current="$(bao read -field=policy sys/policies/acl/default 2>"$err")"; then
echo "could not read the store's default policy; leaving it as it is." >&2
cat "$err" >&2
exit 1
fi
# Never write a default policy built from nothing: it would drop
# every stanza the store's tokens rely on for their own lookups.
if [ -z "$current" ]; then
echo "the store returned an empty default policy; leaving it as it is." >&2
exit 1
fi
kept="$(printf '%s\n' "$current" |
awk -v b=${lib.escapeShellArg publicReadBegin} -v e=${lib.escapeShellArg publicReadEnd} \
'$0 == b { skip = 1 } !skip { print } $0 == e { skip = 0 }')"
wanted="$kept
${publicReadBlock}"
if [ "$wanted" = "$current" ]; then
echo "the default policy already lets every token read swarm/public/"
exit 0
fi
printf '%s\n' "$wanted" |
bao policy write default -
echo "the default policy now lets every token read swarm/public/"
'';
};
# The CA bind source is written at runtime by a host unit, so the # The CA bind source is written at runtime by a host unit, so the
# container has to start after it — otherwise nspawn sets up a mount # container has to start after it — otherwise nspawn sets up a mount
# over a file that does not exist yet. # over a file that does not exist yet.

107
nix/module-eval/bao-grants.nix
View file

@ -170,8 +170,8 @@ let
_: u: (u.environment.BAO_CLIENT_CERT or null) == granterCertFile _: u: (u.environment.BAO_CLIENT_CERT or null) == granterCertFile
) baoGrantWithConsumers.systemd.services; ) baoGrantWithConsumers.systemd.services;
# The fifteen units that write a grant as the granter, by name, for the # The fourteen units that write a `swarm-*` grant, by name, for the discovery
# discovery control below. # control below.
grantingUnitNames = [ grantingUnitNames = [
"swarm-bao-controller-policy" "swarm-bao-controller-policy"
"swarm-bao-secret-publisher-policy" "swarm-bao-secret-publisher-policy"
@ -187,7 +187,6 @@ let
"swarm-bao-operator-viewer-policy" "swarm-bao-operator-viewer-policy"
"swarm-bao-hive-leaf-policy" "swarm-bao-hive-leaf-policy"
"swarm-bao-ci-runner-policy" "swarm-bao-ci-runner-policy"
"swarm-bao-public-read-policy"
]; ];
# Comment lines dropped first: both the HCL and the scripts explain # Comment lines dropped first: both the HCL and the scripts explain
@ -813,24 +812,24 @@ let
} }
{ {
# A store host without the granter's pair writes its grants some other # A store host without the granter's pair writes its grants some other
# way, so none of the fifteen units may exist. Without this arm # way, so none of the fourteen units may exist. Without this arm
# `lib.mkIf haveGranter` could be dropped from any of them and every other # `lib.mkIf haveGranter` could be dropped from any of them and every other
# case here would still pass. # case here would still pass.
name = "without the granter's pair none of the fifteen granting units render"; name = "without the granter's pair none of the fourteen granting units render";
ok = ok =
let let
s = baoGranterOptOut.systemd.services; s = baoGranterOptOut.systemd.services;
in in
lib.all (unit: !(s ? ${unit})) (grantingUnitNames ++ [ "swarm-bao-granter-role" ]) lib.all (unit: !(s ? ${unit})) (grantingUnitNames ++ [ "swarm-bao-granter-role" ])
# The control: the same store with the pair renders all fifteen. # The control: the same store with the pair renders all fourteen.
&& lib.all (unit: baoGrantHere.systemd.services ? ${unit}) grantingUnitNames; && lib.all (unit: baoGrantHere.systemd.services ? ${unit}) grantingUnitNames;
} }
{ {
# 🩸 What replaced the silent skip. With no bootstrap token the fifteen still # 🩸 What replaced the silent skip. With no bootstrap token the fourteen still
# render, and a refused granter fails them with the step that fixes it. # render, and a refused granter fails them with the step that fixes it.
# A store host that never named a token is told to name one, since the # A store host that never named a token is told to name one, since the
# unit that sets the granter up renders only where it has. # unit that sets the granter up renders only where it has.
name = "a store host without a bootstrap token renders the fifteen, each failing loudly with the one-time step"; name = "a store host without a bootstrap token renders the fourteen, each failing loudly with the one-time step";
ok = ok =
let let
s = baoGranterNoToken.systemd.services; s = baoGranterNoToken.systemd.services;
@ -937,7 +936,7 @@ let
{ {
# The granter's grants, whole. Pinned as the full list, because an added # The granter's grants, whole. Pinned as the full list, because an added
# path or capability is exactly what a presence check misses. # path or capability is exactly what a presence check misses.
name = "the granter's policy is exactly these twenty-one stanzas"; name = "the granter's policy is exactly these twenty stanzas";
ok = ok =
let let
cu = [ cu = [
@ -1029,14 +1028,6 @@ let
path = "secret/data/swarm/services/swarm-bao-ui/oidc/client"; path = "secret/data/swarm/services/swarm-bao-ui/oidc/client";
caps = [ "read" ]; caps = [ "read" ];
} }
{
path = "sys/policies/acl/default";
caps = [
"create"
"read"
"update"
];
}
]; ];
} }
{ {
@ -1051,10 +1042,9 @@ let
]; ];
} }
{ {
# Outside `swarm-*`, the store's own mounts and the `default` policy it # Outside `swarm-*` and the store's own mounts it holds nothing: no
# holds nothing: no hive's policy or role, no auth mount, no token, no # hive's policy or role, no auth mount, no token, no secret.
# secret. name = "the granter grants nothing outside swarm-* and the store's own mounts";
name = "the granter grants nothing outside swarm-*, the store's own mounts and the default policy";
ok = ok =
lib.all (p: grantFor granterGrants p == null) [ lib.all (p: grantFor granterGrants p == null) [
"sys/policies/acl/hive-x" "sys/policies/acl/hive-x"
@ -1470,8 +1460,8 @@ let
} }
{ {
# What makes the case above mean something: discovery by the granter's # What makes the case above mean something: discovery by the granter's
# certificate reaches all fifteen units, and each yields calls. # certificate reaches all fourteen units, and each yields calls.
name = "the granter-policy check sees all fifteen granting units, and parses calls from each"; name = "the granter-policy check sees all fourteen granting units, and parses calls from each";
ok = ok =
lib.sort lib.lessThan (lib.attrNames granterUnits) == lib.sort lib.lessThan grantingUnitNames lib.sort lib.lessThan (lib.attrNames granterUnits) == lib.sort lib.lessThan grantingUnitNames
&& lib.all (u: baoCalls u.script != [ ]) (lib.attrValues granterUnits) && lib.all (u: baoCalls u.script != [ ]) (lib.attrValues granterUnits)
@ -1733,77 +1723,6 @@ let
&& !(lib.elem "swarm-bao-operator-viewer-policy.service" g.after) && !(lib.elem "swarm-bao-operator-viewer-policy.service" g.after)
&& !(v ? postStart && lib.hasInfix "swarm-bao-granter-role" v.postStart); && !(v ? postStart && lib.hasInfix "swarm-bao-granter-role" v.postStart);
} }
{
# Every token carries bao's `default` policy, so a read added there is
# a read for every authenticated client. Exactly one stanza, a read on
# `swarm/public/` and nothing wider; the markers are what a later run
# replaces it by.
name = "the default policy gains one read, on swarm/public/, between markers";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-public-read-policy.script;
block = lib.head (
lib.tail (
lib.splitString "# hyperhive: swarm/public begin\n" (builtins.unsafeDiscardStringContext s)
)
);
stanzas = grantsIn (lib.head (lib.splitString "# hyperhive: swarm/public end" block));
in
stanzas == [
{
path = "secret/data/swarm/public/*";
caps = [ "read" ];
}
]
&& lib.hasInfix "-v e='# hyperhive: swarm/public end'" s
&& lib.hasInfix "printf '%s\\n' \"$wanted\" |\n bao policy write default -" s;
}
{
# A read-modify-write of a policy every token depends on: the read
# comes first, an empty one stops the unit before anything is written,
# and the capability probe comes before both, so a granter without the
# grant writes nothing.
name = "the default policy is read before it is written, and never written from an empty read";
ok =
let
s = builtins.unsafeDiscardStringContext baoGrantHere.systemd.services.swarm-bao-public-read-policy.script;
at = needle: lib.stringLength (lib.head (lib.splitString needle s));
probe = "caps=\"$(bao token capabilities sys/policies/acl/default)\"";
read = "bao read -field=policy sys/policies/acl/default";
emptyGuard = "if [ -z \"$current\" ]; then";
write = "bao policy write default -";
in
lib.hasInfix probe s
&& at probe < at "exit 0"
&& at "exit 0" < at read
&& at read < at emptyGuard
&& at emptyGuard < at write
&& lib.hasInfix "swarm-bao-granter-role re-runs this unit once it succeeds." s;
}
{
# The granter's own policy is what lets that unit write `default`, and a
# changed granter policy lands only with the bootstrap step, so the
# step's success restarts the unit as it does the viewer's.
name = "the granter may write the default policy, and its step restarts the public-read unit";
ok =
let
s = baoGrantHere.systemd.services;
g = s.swarm-bao-granter-role;
p = s.swarm-bao-public-read-policy;
in
lib.any (
gr:
gr.path == "sys/policies/acl/default"
&&
gr.caps == [
"create"
"read"
"update"
]
) granterGrants
&& lib.hasInfix "swarm-bao-public-read-policy.service" g.serviceConfig.ExecStartPost
&& lib.elem "swarm-bao-granter-role.service" p.after;
}
{ {
# The client secret is on stdin, never an argument in /proc; the rest is # The client secret is on stdin, never an argument in /proc; the rest is
# the swarm's authelia and the UI's own client id. # the swarm's authelia and the UI's own client id.

8
swarm-nats-auth/src/xkey.rs
View file

@ -1,10 +1,10 @@
//! The callout xkey: the curve keypair `nats-server` seals every auth request //! The callout xkey: the curve keypair `nats-server` seals every auth request
//! to, and that this responder seals every reply back with. //! to, and that this responder seals every reply back with.
//! //!
//! Both halves live in the secret store. The seed is at [`SEED_PATH`], which //! Both halves live in the secret store, and only the `swarm-nats-auth` role
//! only the `swarm-nats-auth` role reads. The public half is at //! reads either. The seed is at [`SEED_PATH`]; the public half is at
//! [`PUBLIC_PATH`], under `swarm/public/`, which every token the store issues //! [`PUBLIC_PATH`], and the queue's host renders it into `auth_callout.xkey`
//! may read; the queue's host renders it into `auth_callout.xkey` from there. //! from there.
//! //!
//! [`ensure`] creates the pair when the seed is absent and otherwise only //! [`ensure`] creates the pair when the seed is absent and otherwise only
//! reads it. [`load`] only ever reads, and is what the responder uses: a //! reads it. [`load`] only ever reads, and is what the responder uses: a