Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

..
3 changed files with 16 additions and 96 deletions

View file

@ -75,7 +75,7 @@ authenticate until some role exists — this token is what breaks that cycle,
and it's the only step that needs the root token. and it's the only step that needs the root token.
```bash ```bash
# Exactly the six grants the bootstrap unit needs, and nothing else. Each was # Exactly the four grants the bootstrap unit needs, and nothing else. Each was
# derived with `bao <cmd> -output-policy`, which prints what a command requires # derived with `bao <cmd> -output-policy`, which prints what a command requires
# without running it. # without running it.
bao policy write swarm-bootstrap - <<'EOF' bao policy write swarm-bootstrap - <<'EOF'
@ -96,17 +96,6 @@ path "sys/auth/cert" {
path "auth/cert/certs/swarm-controller" { path "auth/cert/certs/swarm-controller" {
capabilities = ["create", "update"] capabilities = ["create", "update"]
} }
# The KV engine the controller writes agent credentials through — also absent
# on a fresh store, and checked the same way. No `sudo` here, unlike the auth
# mount above: enabling a secrets engine does not ask for it.
path "sys/mounts" {
capabilities = ["read"]
}
path "sys/mounts/secret" {
capabilities = ["create", "update"]
}
EOF EOF
# A token holding it. `-orphan` so it outlives the session that made it. # A token holding it. `-orphan` so it outlives the session that made it.
@ -116,47 +105,23 @@ bao token create -policy=swarm-bootstrap -ttl=24h -orphan -display-name=swarm-bo
Put the token's value at `services.hyperhive.deploy.bao.bootstrapTokenFile` Put the token's value at `services.hyperhive.deploy.bao.bootstrapTokenFile`
(all-local names that path for you), then rebuild. A one-shot unit **on the (all-local names that path for you), then rebuild. A one-shot unit **on the
host** reads it, writes the `swarm-controller` policy, enables the cert auth host** reads it, writes the `swarm-controller` policy, enables the cert auth
method, mounts the KV engine the controller stores credentials in, and creates method, and creates the `swarm-controller` role that attaches the two. It runs
the `swarm-controller` role that attaches policy to certificate. It runs there there because every API listener demands a client certificate, and the host is
because every API listener demands a client certificate, and the host is the the side that has one.
side that has one.
⏱️ **Expect the first attempt to fail if you rebuilt into this.** A rebuild ⚠️ **The grants above are derived, not proven.** They come from
restarts the store, and the unit races it — the store answers `local node not `-output-policy`, not from a swarm that came up on them. The unit has run twice
active` until it finishes coming up. It retries every 30s and the second against a live store and failed both times for reasons of its own — it could
attempt is the one that usually lands. Nothing to do. not reach the store from where it then ran — so neither run exercised these
grants at all.
**Confirm with `systemctl status swarm-bao-controller-policy`**, which wants no **Delete the file only once `bao read auth/cert/certs/swarm-controller` returns
token — a successful run logs `Uploaded policy`, `Enabled cert auth method` and the role.** The unit skips when the token is absent, so a host that has finished
`Data written to: auth/cert/certs/swarm-controller`. ⚠️ Do _not_ reach for `bao bootstrapping stops carrying the credential — but deleting it before the role
read auth/cert/…` to check: the host's `bao` wrapper carries an address, a CA
and a client certificate but deliberately **no token**, so that read answers
`403` whether or not the role exists.
**Delete the token file only once that unit has succeeded.** It skips when the
token is absent, so a host that has finished bootstrapping stops carrying the
credential — but deleting it before the role
exists leaves the unit skipping forever with nothing to show for it, and looks exists leaves the unit skipping forever with nothing to show for it, and looks
exactly like a store that was never bootstrapped. The TTL above means a exactly like a store that was never bootstrapped. The TTL above means a
forgotten one expires rather than lingering. forgotten one expires rather than lingering.
<details><summary>Already bootstrapped before the KV mount existed?</summary>
A store bootstrapped by an earlier version has the policy, the auth method and
the role, but no `secret/` engine — the controller's first credential write
answers `no handler for route "secret/data/…"`. The bootstrap token can't fix it
either: the policy it was minted from names nothing under `sys/mounts`. Mount it
once with the root token from `init`:
```bash
sudo bash -c 'BAO_TOKEN="<root token>" bao secrets enable -path=secret kv-v2'
```
No rebuild needed — the unit's own check finds the mount on its next run and
leaves it alone.
</details>
Whether anything more is needed depends on Whether anything more is needed depends on
`services.hyperhive.deploy.bao.seal`: `services.hyperhive.deploy.bao.seal`:

View file

@ -198,18 +198,11 @@ let
capabilities = ["list"] capabilities = ["list"]
} }
path "${credentialMountPath}/data/swarm/agents/*" { path "secret/data/swarm/agents/*" {
capabilities = ["create", "update"] capabilities = ["create", "update"]
} }
''; '';
# The KV v2 engine the controller writes agent credentials through. Named
# once because the grant above and the `secrets enable` in the bootstrap unit
# have to agree: a policy pointing at a mount nobody created is precisely the
# failure this pair exists to avoid. `swarm-secret-client` pins the same
# value on the reading side.
credentialMountPath = "secret";
# Every listener serves the same identity: they differ in which address # Every listener serves the same identity: they differ in which address
# they answer on, not in who they are. Client verification is separate and # they answer on, not in who they are. Client verification is separate and
# optional — a store with no `clientCaFile` still serves TLS, it just does # optional — a store with no `clientCaFile` still serves TLS, it just does
@ -911,23 +904,6 @@ in
# than failing on one that already exists. # than failing on one that already exists.
printf '%s' ${lib.escapeShellArg controllerPolicyText} | printf '%s' ${lib.escapeShellArg controllerPolicyText} |
bao policy write ${lib.escapeShellArg controllerPolicyName} - bao policy write ${lib.escapeShellArg controllerPolicyName} -
# The policy grants paths under `${credentialMountPath}/`, and a
# fresh store has no such engine — only a dev-mode one does. Writing
# to an unmounted path answers "route entry not found", so the
# controller's first credential write fails against a grant that
# reads as correct.
#
# Outside the client-CA block below on purpose: this mount is what
# the controller writes *through*, independent of who may log in.
#
# Asked rather than attempted, same as the auth mount: `secrets
# enable` errors on a path already in use.
mounts="$(bao secrets list -format=json)"
case "$mounts" in
*'"${credentialMountPath}/"'*) ;;
*) bao secrets enable -path=${credentialMountPath} kv-v2 ;;
esac
'' ''
+ lib.optionalString (baoDeploy.clientCaFile != null) '' + lib.optionalString (baoDeploy.clientCaFile != null) ''

View file

@ -613,22 +613,6 @@ let
&& lib.hasInfix "auth/cert/certs/swarm-controller" s && lib.hasInfix "auth/cert/certs/swarm-controller" s
&& lib.hasInfix "/var/lib/swarm-bao-tls/client-ca.pem" s; && lib.hasInfix "/var/lib/swarm-bao-tls/client-ca.pem" s;
} }
{
# Same shape as the cert mount above, for the engine the controller
# writes credentials through: a fresh store has no `secret/`, so the
# grant would name a mount nobody created and the first write would 404.
#
# ⚠️ Matched on the COMMAND, for the reason the no-client-CA case below
# spells out: the policy text is embedded in this same script and grants
# `secret/data/...`, so any arm keyed on the *path* is satisfied either
# way and could never fail.
name = "the granting unit creates the KV mount the controller writes through";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
in
lib.hasInfix "bao secrets enable -path=secret kv-v2" s;
}
{ {
# Nothing asserted the PKI script before this, so a third leaf could be # Nothing asserted the PKI script before this, so a third leaf could be
# added to it and every case still passed — measured, not assumed: the # added to it and every case still passed — measured, not assumed: the
@ -796,12 +780,7 @@ let
in in
lib.hasInfix "bao policy write" s lib.hasInfix "bao policy write" s
&& !(lib.hasInfix "bao auth enable cert" s) && !(lib.hasInfix "bao auth enable cert" s)
&& !(lib.hasInfix "client-ca.pem" s) && !(lib.hasInfix "client-ca.pem" s);
# The KV mount is NOT part of what a missing client CA switches off:
# the controller writes through it whether or not anything can log in
# by certificate. Asserted here rather than trusted, because both
# steps live in the same script and one indentation level decides it.
&& lib.hasInfix "bao secrets enable -path=secret kv-v2" s;
} }
{ {
# Reads the vhost's rendered `root`, not the option: the UI is served # Reads the vhost's rendered `root`, not the option: the UI is served