Compare commits
12 changed files with 443 additions and 975 deletions
|
|
@ -88,108 +88,66 @@ its DNS name resolves to the bridge from in there: export
|
||||||
domain>` to verify the name while connecting on loopback. The host is the
|
domain>` to verify the name while connecting on loopback. The host is the
|
||||||
shorter path.
|
shorter path.
|
||||||
|
|
||||||
While you still hold that root token, set up the **granter**: the one principal
|
While you still hold that root token, mint the one credential the swarm needs
|
||||||
that writes every `swarm-*` policy and role from then on. Cert auth answers a
|
to grant itself anything. Cert auth answers a _role_, so nothing can
|
||||||
_role_, so nothing can authenticate until some role exists. A short-lived
|
authenticate until some role exists — this token is what breaks that cycle,
|
||||||
bootstrap token breaks that cycle once, and it's the only step that needs the
|
and it's the only step that needs the root token.
|
||||||
root token.
|
|
||||||
|
|
||||||
The policy it carries is `nix/host-modules/bao-bootstrap-policy.hcl`, shipped
|
The policy is `nix/host-modules/swarm-bao-bootstrap-policy.hcl` in this
|
||||||
on the store's host at `/etc/hyperhive/bao-bootstrap-policy.hcl`. It covers
|
repository, and CI fails when a unit using the token needs a path it lacks.
|
||||||
the auth mounts and the granter's own policy and role, and nothing else. CI
|
|
||||||
fails when the unit using the token needs a path it lacks.
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo -i
|
# The policy file, copied to wherever you run `bao`.
|
||||||
read -rs BAO_TOKEN && export BAO_TOKEN # paste the root token from `bao operator init`
|
bao policy write swarm-bootstrap swarm-bao-bootstrap-policy.hcl
|
||||||
bao policy write bao-bootstrap /etc/hyperhive/bao-bootstrap-policy.hcl
|
|
||||||
bao token create -policy=bao-bootstrap -ttl=24h -orphan -display-name=bao-bootstrap -field=token \
|
# A token holding it. `-orphan` so it outlives the session that made it.
|
||||||
| install -D -m 0600 /dev/stdin /var/lib/swarm-bao-bootstrap/grant.token
|
bao token create -policy=swarm-bootstrap -ttl=24h -orphan -display-name=swarm-bootstrap
|
||||||
unset BAO_TOKEN
|
|
||||||
systemctl restart swarm-bao-granter-role
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The token file is `services.hyperhive.deploy.bao.bootstrapTokenFile`, which
|
Put the token's value at `services.hyperhive.deploy.bao.bootstrapTokenFile`
|
||||||
all-local names for you. On a store host that isn't all-local, set it and
|
(all-local names that path for you), then rebuild. A one-shot unit **on the
|
||||||
rebuild first.
|
host** reads it, writes the `swarm-controller` policy, enables the cert auth
|
||||||
|
method, mounts the KV engine the controller stores credentials in, and creates
|
||||||
`swarm-bao-granter-role` runs **on the host**. It enables the cert auth
|
the `swarm-controller` role that attaches policy to certificate. It runs there
|
||||||
method, writes the `bao-granter` policy, and creates the `bao-granter` role,
|
because every API listener demands a client certificate, and the host is the
|
||||||
which accepts the leaf `/var/lib/swarm-bao-pki/granter.pem`. Every
|
side that has one.
|
||||||
`swarm-bao-*-policy` unit then logs in with that leaf. The controller's unit
|
|
||||||
mounts the KV and pki engines and writes the `swarm-controller` role, and each
|
|
||||||
sibling unit writes its own principal's policy and role. Every one runs on the
|
|
||||||
host, because every API listener demands a client certificate and the host is
|
|
||||||
the side that has one.
|
|
||||||
|
|
||||||
**Confirm with `systemctl status swarm-bao-granter-role`**, which should log
|
|
||||||
`Uploaded policy: bao-granter` and `Data written to: auth/cert/certs/bao-granter`.
|
|
||||||
Then restart the granting units that failed while they waited:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl reset-failed 'swarm-bao-*-policy.service'
|
|
||||||
systemctl restart 'swarm-bao-*-policy.service'
|
|
||||||
systemctl status swarm-bao-controller-policy # Uploaded policy, Data written to: auth/cert/certs/swarm-controller
|
|
||||||
rm /var/lib/swarm-bao-bootstrap/grant.token
|
|
||||||
```
|
|
||||||
|
|
||||||
⚠️ Don't reach for `bao read auth/cert/…` to check. The host's `bao`
|
|
||||||
wrapper carries an address, a CA and a client certificate but deliberately
|
|
||||||
**no token**, so that read answers `403` whether or not the role exists.
|
|
||||||
|
|
||||||
⏱️ **Expect the first attempt to fail if you rebuilt into this.** A rebuild
|
⏱️ **Expect the first attempt to fail if you rebuilt into this.** A rebuild
|
||||||
restarts the store, and the units race it: the store answers `local node not
|
restarts the store, and the unit races it — the store answers `local node not
|
||||||
active` until it finishes coming up. They retry every 30s for a day, so a
|
active` until it finishes coming up. It retries every 30s and the second
|
||||||
sealed or late store heals itself.
|
attempt is the one that usually lands. Nothing to do.
|
||||||
|
|
||||||
Until you set up the granter, each `swarm-bao-*-policy` unit **fails** and logs
|
**Confirm with `systemctl status swarm-bao-controller-policy`**, which wants no
|
||||||
the commands above. It never skips. Delete the token file only once
|
token — a successful run logs `Uploaded policy`, `Enabled cert auth method` and
|
||||||
`swarm-bao-granter-role` has succeeded. That unit skips while the file is
|
`Data written to: auth/cert/certs/swarm-controller`. ⚠️ Do _not_ reach for `bao
|
||||||
absent, which is the steady state afterwards. The TTL above means a forgotten
|
read auth/cert/…` to check: the host's `bao` wrapper carries an address, a CA
|
||||||
token expires rather than lingering.
|
and a client certificate but deliberately **no token**, so that read answers
|
||||||
|
`403` whether or not the role exists.
|
||||||
|
|
||||||
After that, a new or changed `swarm-*` grant needs no operator step: the unit
|
**Delete the token file only once that unit has succeeded.** It skips when the
|
||||||
that writes it changes, and the deploy restarts it. A root step comes back only
|
token is absent, so a host that has finished bootstrapping stops carrying the
|
||||||
when the granter itself needs a path it lacks, such as a new mount.
|
credential — but deleting it before the role
|
||||||
|
exists leaves the unit skipping forever with nothing to show for it, and looks
|
||||||
|
exactly like a store that was never bootstrapped. The TTL above means a
|
||||||
|
forgotten one expires rather than lingering.
|
||||||
|
|
||||||
⚠️ The granting units re-run on **boot** and whenever a deploy **changes**
|
<details><summary>Already bootstrapped before the KV mount existed?</summary>
|
||||||
them, not on every deploy. When a grant drifts in the store and its unit stays the
|
|
||||||
same, the next boot re-asserts it, not the next switch.
|
|
||||||
|
|
||||||
<details><summary>Upgrading a swarm set up with the older swarm-bootstrap policy</summary>
|
A store bootstrapped by an earlier version has the policy, the auth method and
|
||||||
|
the role, but no `secret/` engine — the controller's first credential write
|
||||||
A store set up before the granter existed has every grant, but no `bao-granter`
|
answers `no handler for route "secret/data/…"`. The bootstrap token can't fix it
|
||||||
policy or role. After the deploy that introduces it, each `swarm-bao-*-policy`
|
either: the policy that minted it names nothing under `sys/mounts`. Mount it
|
||||||
unit fails and logs the one-time step. Run the two blocks above as they stand.
|
once with the root token from `init`:
|
||||||
The old policy can go, with the root token again:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bao policy delete swarm-bootstrap
|
sudo bash -c 'BAO_TOKEN="<root token>" bao secrets enable -path=secret kv-v2'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
No rebuild needed — the unit's own check finds the mount on its next run and
|
||||||
|
leaves it alone.
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
**Residual risk, stated plainly.** The granter is root-equivalent. It may write
|
|
||||||
any `swarm-*` policy with any content, and attach it to a role that accepts any
|
|
||||||
certificate; no bao ACL can constrain what a policy says. What bounds it:
|
|
||||||
|
|
||||||
- `nix/host-modules/swarm-bao.nix` renders every policy it writes, and
|
|
||||||
module-eval pins each principal's grants. **Merging a change to that policy
|
|
||||||
text is granting it**: it takes effect on the next deploy with no bao step,
|
|
||||||
so code review is the only gate.
|
|
||||||
- Its key sits permanently at `/var/lib/swarm-bao-pki/granter-key.pem`, `0600`
|
|
||||||
root in a `0700` directory, readable only by root units on the store's host.
|
|
||||||
That host already holds `ca-key.pem`, which can mint a leaf with any subject,
|
|
||||||
and `controller-key.pem`, whose policy is already root-equivalent. Root on
|
|
||||||
that host gains nothing new.
|
|
||||||
- **Never copy `granter-key.pem` off the host** the way operators copy the
|
|
||||||
other leaves in that directory. That hands out root-equivalence.
|
|
||||||
- Nothing revokes a stolen leaf on its own: the role trusts the CA plus the
|
|
||||||
subject. Rotate the store's CA, or have root point the `bao-granter` role at
|
|
||||||
a new `deploy.bao.granterCommonName`. Deleting `granter{,-key}.pem` and
|
|
||||||
restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old
|
|
||||||
one.
|
|
||||||
|
|
||||||
What else you need depends on
|
What else you need depends on
|
||||||
`services.hyperhive.deploy.bao.seal`:
|
`services.hyperhive.deploy.bao.seal`:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,37 +0,0 @@
|
||||||
# The `bao-bootstrap` policy: what the 24h bootstrap token may do, and nothing
|
|
||||||
# else. ../../docs/getting-started/setup.md has the operator write it with the
|
|
||||||
# root token, from the copy ./swarm-bao.nix ships at
|
|
||||||
# /etc/hyperhive/bao-bootstrap-policy.hcl; `swarm-bao-granter-role` then acts
|
|
||||||
# with it. Every other grant is written by the `bao-granter` principal this
|
|
||||||
# creates.
|
|
||||||
#
|
|
||||||
# Named outside `swarm-*`, so the granter cannot rewrite the policy the next
|
|
||||||
# bootstrap token carries.
|
|
||||||
#
|
|
||||||
# Each stanza was derived with `bao <cmd> -output-policy`, which prints what a
|
|
||||||
# command requires without sending it. ../module-eval/bao-grants.nix reads
|
|
||||||
# this file and fails when the unit that uses the token calls a path it does
|
|
||||||
# not grant.
|
|
||||||
|
|
||||||
# The auth mounts. Reading `sys/auth` is how the unit checks, and `sudo` is
|
|
||||||
# what enabling one costs.
|
|
||||||
path "sys/auth" {
|
|
||||||
capabilities = ["read"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "sys/auth/cert" {
|
|
||||||
capabilities = ["create", "update", "sudo"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "sys/auth/approle" {
|
|
||||||
capabilities = ["create", "update", "sudo"]
|
|
||||||
}
|
|
||||||
|
|
||||||
# The granter's own policy and role, and nothing it may write.
|
|
||||||
path "sys/policies/acl/bao-granter" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "auth/cert/certs/bao-granter" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
@ -12,10 +12,9 @@
|
||||||
# with no ExecStart, so each gate below restates the one the reader's own
|
# with no ExecStart, so each gate below restates the one the reader's own
|
||||||
# module puts on it. A reader whose gate changes must change here too.
|
# module puts on it. A reader whose gate changes must change here too.
|
||||||
#
|
#
|
||||||
# Ordering, never a requirement: a policy unit that failed still counts as
|
# Ordering, never a requirement: a policy unit skips once the bootstrap token
|
||||||
# done, and the reader's own retries carry it past that. `wants` as well as
|
# is gone, and a skipped unit counts as done. `wants` as well as `after`, so a
|
||||||
# `after`, so a reader started on its own pulls its policy unit into the same
|
# reader started on its own pulls its policy unit into the same transaction.
|
||||||
# transaction.
|
|
||||||
{
|
{
|
||||||
lib,
|
lib,
|
||||||
config,
|
config,
|
||||||
|
|
|
||||||
|
|
@ -108,12 +108,6 @@ in
|
||||||
# on `client.pem`.
|
# on `client.pem`.
|
||||||
forwarderOidcClientCertFile = lib.mkDefault "${pkiDir}/forwarder-oidc.pem";
|
forwarderOidcClientCertFile = lib.mkDefault "${pkiDir}/forwarder-oidc.pem";
|
||||||
forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem";
|
forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem";
|
||||||
|
|
||||||
# The granter: every `swarm-bao-*-policy` unit on this host logs in with
|
|
||||||
# it. ⚠️ Unlike every other leaf here, never the file an operator copies:
|
|
||||||
# its policy is root-equivalent and its only reader is this host.
|
|
||||||
granterClientCertFile = lib.mkDefault "${pkiDir}/granter.pem";
|
|
||||||
granterClientKeyFile = lib.mkDefault "${pkiDir}/granter-key.pem";
|
|
||||||
};
|
};
|
||||||
|
|
||||||
# Idempotent on ABSENCE, never on content. Re-issuing the CA invalidates
|
# Idempotent on ABSENCE, never on content. Re-issuing the CA invalidates
|
||||||
|
|
@ -254,12 +248,6 @@ in
|
||||||
# the file an operator copies.
|
# the file an operator copies.
|
||||||
[ -s ${pkiDir}/nats.pem ] || ${signLeaf} ${pkiDir} nats \
|
[ -s ${pkiDir}/nats.pem ] || ${signLeaf} ${pkiDir} nats \
|
||||||
${lib.escapeShellArg deployCfg.bao.natsCommonName} "" clientAuth
|
${lib.escapeShellArg deployCfg.bao.natsCommonName} "" clientAuth
|
||||||
|
|
||||||
# The granter's, which writes every `swarm-*` grant. Minted here because
|
|
||||||
# it opens the store for the units that create the roles every other
|
|
||||||
# leaf logs in with. Stays on this host; see its default above.
|
|
||||||
[ -s ${pkiDir}/granter.pem ] || ${signLeaf} ${pkiDir} granter \
|
|
||||||
${lib.escapeShellArg deployCfg.bao.granterCommonName} "" clientAuth
|
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
|
||||||
159
nix/host-modules/swarm-bao-bootstrap-policy.hcl
Normal file
159
nix/host-modules/swarm-bao-bootstrap-policy.hcl
Normal file
|
|
@ -0,0 +1,159 @@
|
||||||
|
# The `swarm-bootstrap` policy: what the 24h bootstrap token may do, and
|
||||||
|
# nothing else. ../../docs/getting-started/setup.md has the operator write it
|
||||||
|
# with the root token; ./swarm-bao.nix's granting units then act with it.
|
||||||
|
#
|
||||||
|
# Each stanza was derived with `bao <cmd> -output-policy`, which prints what a
|
||||||
|
# command requires without sending it. ../module-eval/bao-grants.nix reads
|
||||||
|
# this file and fails when a unit that uses the token calls a path it does not
|
||||||
|
# grant. The pki paths assume the default `servicesPkiMountPath` (`pki`),
|
||||||
|
# `servicesPkiRoleName` (`swarm-services`) and `natsPkiRoleName` (`swarm-nats`).
|
||||||
|
|
||||||
|
# swarm-bao-controller-policy: the controller's own policy and role.
|
||||||
|
path "sys/policies/acl/swarm-controller" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "auth/cert/certs/swarm-controller" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# The auth mounts it creates. Reading `sys/auth` is how the unit checks, and
|
||||||
|
# `sudo` is what enabling one costs.
|
||||||
|
path "sys/auth" {
|
||||||
|
capabilities = ["read"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "sys/auth/cert" {
|
||||||
|
capabilities = ["create", "update", "sudo"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "sys/auth/approle" {
|
||||||
|
capabilities = ["create", "update", "sudo"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# The KV and PKI engines, checked the same way. Enabling a secrets engine does
|
||||||
|
# not ask for `sudo`.
|
||||||
|
path "sys/mounts" {
|
||||||
|
capabilities = ["read"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "sys/mounts/secret" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "sys/mounts/pki" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "sys/mounts/pki/tune" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# The services root: generated once, read back on every run, and replaced
|
||||||
|
# only when it can no longer outlive a leaf.
|
||||||
|
path "pki/issuers" {
|
||||||
|
capabilities = ["list"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "pki/cert/ca" {
|
||||||
|
capabilities = ["read"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "pki/root" {
|
||||||
|
capabilities = ["delete", "sudo"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "pki/root/generate/internal" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "pki/roles/swarm-services" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# swarm-bao-secret-publisher-policy
|
||||||
|
path "sys/policies/acl/swarm-secret-publisher" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "auth/cert/certs/swarm-secret-publisher" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# swarm-bao-matrix-ctl-policy
|
||||||
|
path "sys/policies/acl/swarm-matrix-ctl" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "auth/cert/certs/swarm-matrix-ctl" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# swarm-bao-services-issuer-policy
|
||||||
|
path "sys/policies/acl/swarm-services-issuer" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "auth/cert/certs/swarm-services-issuer" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# swarm-bao-grafana-oidc-policy
|
||||||
|
path "sys/policies/acl/swarm-grafana-oidc" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "auth/cert/certs/swarm-grafana-oidc" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# swarm-bao-otel-oidc-policy
|
||||||
|
path "sys/policies/acl/swarm-otel-oidc" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "auth/cert/certs/swarm-otel-oidc" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# swarm-bao-forwarder-oidc-policy
|
||||||
|
path "sys/policies/acl/swarm-forwarder-oidc" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "auth/cert/certs/swarm-forwarder-oidc" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# swarm-bao-nats-tls-policy: the queue's own pki role, beside
|
||||||
|
# `swarm-services` above, and its policy and login role.
|
||||||
|
path "pki/roles/swarm-nats" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "sys/policies/acl/swarm-nats" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "auth/cert/certs/swarm-nats" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# swarm-bao-matrix-token-policy and swarm-bao-queue-agent-policy write one
|
||||||
|
# policy and role per hive, `<prefix>-<hive>`, so these two are globs. Each
|
||||||
|
# stops at its own prefix.
|
||||||
|
path "sys/policies/acl/swarm-matrix-token-*" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "auth/cert/certs/swarm-matrix-token-*" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "sys/policies/acl/swarm-queue-agent-*" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "auth/cert/certs/swarm-queue-agent-*" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
@ -151,7 +151,7 @@ let
|
||||||
# rather than as the missing setting it is.
|
# rather than as the missing setting it is.
|
||||||
haveServerTls = baoDeploy.serverCertFile != null && baoDeploy.serverKeyFile != null;
|
haveServerTls = baoDeploy.serverCertFile != null && baoDeploy.serverKeyFile != null;
|
||||||
|
|
||||||
# The credential that writes the granter's role below. A token and not a
|
# The credential that writes the swarm's first grant. A token and not a
|
||||||
# certificate: cert auth answers a *role*, so nothing can authenticate here
|
# certificate: cert auth answers a *role*, so nothing can authenticate here
|
||||||
# until some role exists, and whatever creates the first one cannot itself
|
# until some role exists, and whatever creates the first one cannot itself
|
||||||
# use one. An operator places it — ../../docs/getting-started/setup.md.
|
# use one. An operator places it — ../../docs/getting-started/setup.md.
|
||||||
|
|
@ -163,164 +163,6 @@ let
|
||||||
bootstrapTokenDir =
|
bootstrapTokenDir =
|
||||||
if haveBootstrapToken then builtins.dirOf baoDeploy.bootstrapTokenFile else null;
|
if haveBootstrapToken then builtins.dirOf baoDeploy.bootstrapTokenFile else null;
|
||||||
|
|
||||||
# The principal every `swarm-bao-*-policy` unit logs in as, so a new or
|
|
||||||
# changed `swarm-*` grant applies on deploy with no operator step. Policy and
|
|
||||||
# role share one name, outside both `swarm-*` and `hive-*`: neither the
|
|
||||||
# granter's globs nor the controller's reach the objects that constrain it.
|
|
||||||
granterPolicyName = "bao-granter";
|
|
||||||
granterCn = baoDeploy.granterCommonName;
|
|
||||||
|
|
||||||
# No CA means no login role can be written, so nothing could log in as the
|
|
||||||
# granter; the units that need it do not render.
|
|
||||||
haveGranter =
|
|
||||||
baoDeploy.granterClientCertFile != null
|
|
||||||
&& baoDeploy.granterClientKeyFile != null
|
|
||||||
&& baoDeploy.clientCaFile != null;
|
|
||||||
|
|
||||||
# ⚠️ ROOT-EQUIVALENT BY CONSTRUCTION. No ACL constrains the body of a policy,
|
|
||||||
# so a principal that may write `swarm-*` policies and the roles attaching
|
|
||||||
# them may grant itself anything. What bounds it is that every policy it
|
|
||||||
# writes is rendered from this file, and that its key never leaves this host.
|
|
||||||
#
|
|
||||||
# A trailing `*` in a bao ACL path is a pure string-prefix match, and an
|
|
||||||
# exact path wins over any prefix.
|
|
||||||
#
|
|
||||||
# The first three are the per-principal grants. The next eight are what
|
|
||||||
# `swarm-bao-controller-policy` does besides grants: the KV and pki mounts and
|
|
||||||
# the services root. The last six set up the agent PKI mount: the mount, its
|
|
||||||
# root and the `swarm-*` role agent certificates are issued through. No
|
|
||||||
# `root` delete there: every agent's cert-auth role pins that root by value,
|
|
||||||
# so replacing it would lock every agent out. No `sys/auth`: the auth mounts
|
|
||||||
# are created with the bootstrap token by `swarm-bao-granter-role`.
|
|
||||||
#
|
|
||||||
# Piped as a shell-quoted argument like `controllerPolicyText`, so
|
|
||||||
# ../module-eval/bao-grants.nix can read it out of the unit script.
|
|
||||||
granterPolicyText = ''
|
|
||||||
path "sys/policies/acl/swarm-*" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "auth/cert/certs/swarm-*" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "${servicesPkiMountPath}/roles/swarm-*" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "sys/mounts" {
|
|
||||||
capabilities = ["read"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "sys/mounts/${credentialMountPath}" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "sys/mounts/${servicesPkiMountPath}" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "sys/mounts/${servicesPkiMountPath}/tune" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "${servicesPkiMountPath}/issuers" {
|
|
||||||
capabilities = ["list"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "${servicesPkiMountPath}/cert/ca" {
|
|
||||||
capabilities = ["read"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "${servicesPkiMountPath}/root" {
|
|
||||||
capabilities = ["delete", "sudo"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "${servicesPkiMountPath}/root/generate/internal" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "sys/mounts/${agentPkiMountPath}" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "sys/mounts/${agentPkiMountPath}/tune" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "${agentPkiMountPath}/issuers" {
|
|
||||||
capabilities = ["list"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "${agentPkiMountPath}/cert/ca" {
|
|
||||||
capabilities = ["read"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "${agentPkiMountPath}/root/generate/internal" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "${agentPkiMountPath}/roles/swarm-*" {
|
|
||||||
capabilities = ["create", "update"]
|
|
||||||
}
|
|
||||||
'';
|
|
||||||
|
|
||||||
# What a granting unit prints when the store refuses the granter: the
|
|
||||||
# one-time step, runnable as root on this host.
|
|
||||||
granterSetupSteps = [
|
|
||||||
"read -rs BAO_TOKEN && export BAO_TOKEN # the root token from 'bao operator init'"
|
|
||||||
"bao policy write bao-bootstrap /etc/hyperhive/bao-bootstrap-policy.hcl"
|
|
||||||
]
|
|
||||||
++ (
|
|
||||||
if haveBootstrapToken then
|
|
||||||
[
|
|
||||||
"bao token create -policy=bao-bootstrap -ttl=24h -orphan -display-name=bao-bootstrap -field=token | install -D -m 0600 /dev/stdin ${baoDeploy.bootstrapTokenFile}"
|
|
||||||
"unset BAO_TOKEN"
|
|
||||||
"systemctl restart swarm-bao-granter-role"
|
|
||||||
]
|
|
||||||
else
|
|
||||||
[
|
|
||||||
"# then set services.hyperhive.deploy.bao.bootstrapTokenFile on this host, deploy, and place a token there:"
|
|
||||||
"bao token create -policy=bao-bootstrap -ttl=24h -orphan -display-name=bao-bootstrap -field=token"
|
|
||||||
]
|
|
||||||
);
|
|
||||||
|
|
||||||
# The login every granting unit starts with. It FAILS rather than skips: a
|
|
||||||
# grant that was not written is otherwise invisible until whatever needs it
|
|
||||||
# fails somewhere else. `bao status` exits 0 only on a reachable, unsealed
|
|
||||||
# store, which separates "the granter is not set up" from "retry later";
|
|
||||||
# either way bao's own message follows.
|
|
||||||
granterLogin = ''
|
|
||||||
err="$(mktemp)"
|
|
||||||
trap 'rm -f "$err"' EXIT
|
|
||||||
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
|
|
||||||
if bao status >/dev/null 2>&1; then
|
|
||||||
echo ${lib.escapeShellArg "the store is unsealed but refused the granter's certificate (CN ${granterCn}): the ${granterPolicyName} role is not set up."} >&2
|
|
||||||
echo "one-time step, as root on this host (docs/getting-started/setup.md):" >&2
|
|
||||||
${lib.concatMapStringsSep "\n" (l: "echo ${lib.escapeShellArg " ${l}"} >&2") granterSetupSteps}
|
|
||||||
else
|
|
||||||
echo "the store is sealed or unreachable; retrying." >&2
|
|
||||||
fi
|
|
||||||
cat "$err" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
export BAO_TOKEN
|
|
||||||
'';
|
|
||||||
|
|
||||||
# The granter's certificate for the granting units. The `baoCli` wrapper
|
|
||||||
# only defaults these, so the unit's environment wins.
|
|
||||||
granterEnv = {
|
|
||||||
BAO_CLIENT_CERT = baoDeploy.granterClientCertFile;
|
|
||||||
BAO_CLIENT_KEY = baoDeploy.granterClientKeyFile;
|
|
||||||
};
|
|
||||||
|
|
||||||
# `swarm-bao-pki` mints the granter's leaf; the granter's role is written by
|
|
||||||
# `swarm-bao-granter-role`, which normally skips, hence ordering only there.
|
|
||||||
granterAfter = [
|
|
||||||
"swarm-bao-pki.service"
|
|
||||||
"swarm-bao-granter-role.service"
|
|
||||||
];
|
|
||||||
|
|
||||||
# The name both ends must agree on: the cert-auth role below attaches this
|
# The name both ends must agree on: the cert-auth role below attaches this
|
||||||
# policy by spelling it the same way, and is itself named after it.
|
# policy by spelling it the same way, and is itself named after it.
|
||||||
controllerPolicyName = "swarm-controller";
|
controllerPolicyName = "swarm-controller";
|
||||||
|
|
@ -500,13 +342,6 @@ let
|
||||||
# and has to spell it the same way.
|
# and has to spell it the same way.
|
||||||
servicesPkiMountPath = baoDeploy.servicesPkiMountPath;
|
servicesPkiMountPath = baoDeploy.servicesPkiMountPath;
|
||||||
|
|
||||||
# The PKI mount agent client certificates are issued from. Its root is
|
|
||||||
# generated inside the store, so the agent CA's key never exists outside it.
|
|
||||||
# A mount of its own because the services mount holds exactly one issuer; a
|
|
||||||
# root apart from ./glue-bao-tls.nix's CA because that is what keeps an
|
|
||||||
# agent's certificate from satisfying any host role.
|
|
||||||
agentPkiMountPath = baoDeploy.agentPkiMountPath;
|
|
||||||
|
|
||||||
# Subject of the root generated into that mount. A label for a human reading
|
# Subject of the root generated into that mount. A label for a human reading
|
||||||
# a chain, not an identity anything authenticates against — same fall-through
|
# a chain, not an identity anything authenticates against — same fall-through
|
||||||
# ./swarm-ca.nix:29-37 uses, and for the same reason: a hive that has set
|
# ./swarm-ca.nix:29-37 uses, and for the same reason: a hive that has set
|
||||||
|
|
@ -711,25 +546,27 @@ let
|
||||||
# after it.
|
# after it.
|
||||||
#
|
#
|
||||||
# `after` and not `requires`, for the reason the publisher's unit states: the
|
# `after` and not `requires`, for the reason the publisher's unit states: the
|
||||||
# controller's and the granter's units create the mounts this one writes
|
# controller's unit creates the KV and cert-auth mounts this one writes into,
|
||||||
# into, but a failed oneshot still counts as finished, so ordering plus this
|
# but a failed oneshot still counts as finished, so ordering plus this unit's
|
||||||
# unit's own retry is what converges.
|
# own retry is what converges.
|
||||||
|
#
|
||||||
|
# The role write is inside the client-CA branch and the policy write is not,
|
||||||
|
# exactly as the three above: with no CA there is no trust anchor for a login
|
||||||
|
# role, but the policy it would attach is still worth asserting.
|
||||||
readerPolicyUnit =
|
readerPolicyUnit =
|
||||||
description: objects:
|
description: objects:
|
||||||
lib.mkIf haveGranter {
|
lib.mkIf haveBootstrapToken {
|
||||||
inherit description;
|
inherit description;
|
||||||
after = [
|
after = [
|
||||||
"container@${cfg.machine}.service"
|
"container@${cfg.machine}.service"
|
||||||
"swarm-bao-controller-policy.service"
|
"swarm-bao-controller-policy.service"
|
||||||
]
|
];
|
||||||
++ granterAfter;
|
|
||||||
requires = [ "swarm-bao-pki.service" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
wantedBy = [ "multi-user.target" ];
|
||||||
path = [
|
path = [
|
||||||
baoCli
|
baoCli
|
||||||
pkgs.coreutils
|
pkgs.coreutils
|
||||||
];
|
];
|
||||||
environment = granterEnv;
|
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
|
||||||
# Same unseal wait as its siblings above, for the reason stated there:
|
# Same unseal wait as its siblings above, for the reason stated there:
|
||||||
# under `seal = "shamir"` a human unseals by hand.
|
# under `seal = "shamir"` a human unseals by hand.
|
||||||
startLimitBurst = 2880;
|
startLimitBurst = 2880;
|
||||||
|
|
@ -743,11 +580,14 @@ let
|
||||||
script = ''
|
script = ''
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
${granterLogin}
|
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
|
||||||
|
export BAO_TOKEN
|
||||||
|
|
||||||
''
|
''
|
||||||
+ lib.concatMapStrings readerPolicyWrite objects
|
+ lib.concatMapStrings readerPolicyWrite objects
|
||||||
+ "\n"
|
+ lib.optionalString (baoDeploy.clientCaFile != null) (
|
||||||
+ lib.concatMapStrings readerRoleWrite objects;
|
"\n" + lib.concatMapStrings readerRoleWrite objects
|
||||||
|
);
|
||||||
};
|
};
|
||||||
|
|
||||||
# Every listener serves the same identity: they differ in which address
|
# Every listener serves the same identity: they differ in which address
|
||||||
|
|
@ -1139,22 +979,19 @@ in
|
||||||
default = null;
|
default = null;
|
||||||
example = "/var/lib/swarm-bao-bootstrap/grant.token";
|
example = "/var/lib/swarm-bao-bootstrap/grant.token";
|
||||||
description = ''
|
description = ''
|
||||||
Token used **once per swarm** to create the store's cert-auth mount and
|
Token used **once per swarm** to write the first authorisation grants,
|
||||||
the `bao-granter` policy and role, after which every granting unit
|
after which every client authenticates with a certificate instead.
|
||||||
logs in as the granter with a certificate instead.
|
|
||||||
|
|
||||||
Cert auth answers a *role*, so no client can authenticate until some
|
Cert auth answers a *role*, so no client can authenticate until some
|
||||||
role exists — and creating the granter's is what this token is for.
|
role exists — and creating that first one is what this token is for.
|
||||||
It has to come from outside that cycle, which is why an operator places
|
It has to come from outside that cycle, which is why an operator places
|
||||||
it rather than the deployment minting it.
|
it rather than the deployment minting it.
|
||||||
|
|
||||||
Produce it from the root token `bao operator init` printed, under the
|
Produce it from the root token `bao operator init` printed, scoped to
|
||||||
`bao-bootstrap` policy shipped at
|
that one policy write and nothing else, then delete it once the swarm
|
||||||
{file}`/etc/hyperhive/bao-bootstrap-policy.hcl`, then delete it once
|
has come up — {file}`docs/getting-started/setup.md` has the commands.
|
||||||
`swarm-bao-granter-role` has run —
|
Setting this is what enables the granting unit; leaving it null means
|
||||||
{file}`docs/getting-started/setup.md` has the commands. Setting this is
|
the deployment writes those grants some other way.
|
||||||
what renders `swarm-bao-granter-role`; while it is null, a store whose
|
|
||||||
granter is not set up has no way to set it up.
|
|
||||||
|
|
||||||
A path, never a value.
|
A path, never a value.
|
||||||
'';
|
'';
|
||||||
|
|
@ -1255,20 +1092,6 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
agentPkiMountPath = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "pki-agents";
|
|
||||||
description = ''
|
|
||||||
Mount path of the PKI engine agent client certificates are issued
|
|
||||||
from. Its root is generated inside the store and its key never leaves
|
|
||||||
it.
|
|
||||||
|
|
||||||
An option rather than a literal because the store host sets the mount
|
|
||||||
up while swarm-controller, possibly on another host, issues through
|
|
||||||
it: both have to spell it identically.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
servicesPkiRoleName = lib.mkOption {
|
servicesPkiRoleName = lib.mkOption {
|
||||||
type = lib.types.str;
|
type = lib.types.str;
|
||||||
default = "swarm-services";
|
default = "swarm-services";
|
||||||
|
|
@ -1597,48 +1420,6 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
granterCommonName = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "bao-granter";
|
|
||||||
description = ''
|
|
||||||
Subject the store's `bao-granter` cert-auth role accepts: the identity
|
|
||||||
every `swarm-bao-*-policy` unit on the store's host logs in as to write
|
|
||||||
the `swarm-*` policies, cert-auth roles and pki roles.
|
|
||||||
|
|
||||||
⚠️ Root-equivalent: it may write a `swarm-*` policy with any content.
|
|
||||||
Reserved as a hive name by ./swarm.nix, like its siblings.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
granterClientCertFile = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.str;
|
|
||||||
default = null;
|
|
||||||
example = "/var/lib/swarm-bao-pki/granter.pem";
|
|
||||||
description = ''
|
|
||||||
Certificate the store's granting units present to the store. Its
|
|
||||||
subject must be
|
|
||||||
{option}`services.hyperhive.deploy.bao.granterCommonName`.
|
|
||||||
|
|
||||||
Null, or a null
|
|
||||||
{option}`services.hyperhive.deploy.bao.clientCaFile`, means this
|
|
||||||
deployment writes those grants some other way: no granting unit
|
|
||||||
renders.
|
|
||||||
|
|
||||||
⚠️ Unlike every other leaf the store's host mints, this one is never
|
|
||||||
copied to another host; its only reader is that host.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
granterClientKeyFile = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.str;
|
|
||||||
default = null;
|
|
||||||
example = "/var/lib/swarm-bao-pki/granter-key.pem";
|
|
||||||
description = ''
|
|
||||||
Private key for
|
|
||||||
{option}`services.hyperhive.deploy.bao.granterClientCertFile`.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
serverCaFile = lib.mkOption {
|
serverCaFile = lib.mkOption {
|
||||||
type = lib.types.nullOr lib.types.str;
|
type = lib.types.nullOr lib.types.str;
|
||||||
default = null;
|
default = null;
|
||||||
|
|
@ -1854,29 +1635,8 @@ in
|
||||||
grant reads every secret in the store; this role reads one path.
|
grant reads every secret in the store; this role reads one path.
|
||||||
'';
|
'';
|
||||||
}
|
}
|
||||||
{
|
|
||||||
# The granter writes pki roles through `roles/swarm-*` and nothing
|
|
||||||
# else, so a role named otherwise is a 403 at deploy time.
|
|
||||||
assertion =
|
|
||||||
!haveGranter
|
|
||||||
|| (lib.hasPrefix "swarm-" servicesPkiRoleName && lib.hasPrefix "swarm-" natsPkiRoleName);
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.deploy.bao.servicesPkiRoleName
|
|
||||||
(${servicesPkiRoleName}) and
|
|
||||||
services.hyperhive.deploy.bao.natsPkiRoleName (${natsPkiRoleName})
|
|
||||||
must both start with `swarm-`: the bao granter that writes them may
|
|
||||||
write pki roles under that prefix only.
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
];
|
];
|
||||||
|
|
||||||
warnings = lib.optional (haveServerTls && baoDeploy.clientCaFile == null) ''
|
|
||||||
services.hyperhive.deploy.bao.clientCaFile is null, so no cert-auth
|
|
||||||
role can be written and no client can log in to the swarm secret store.
|
|
||||||
None of the swarm-bao-*-policy units render: this deployment writes no
|
|
||||||
bao policy or role.
|
|
||||||
'';
|
|
||||||
|
|
||||||
# The name every reader dials, made resolvable where the store runs.
|
# The name every reader dials, made resolvable where the store runs.
|
||||||
# Cross-hive traffic always goes via the domain; only what it resolves
|
# Cross-hive traffic always goes via the domain; only what it resolves
|
||||||
# to varies, and a multi-host swarm is the operator's upstream DNS. This
|
# to varies, and a multi-host swarm is the operator's upstream DNS. This
|
||||||
|
|
@ -1904,10 +1664,6 @@ in
|
||||||
# addresses on every command.
|
# addresses on every command.
|
||||||
environment.systemPackages = [ baoCli ];
|
environment.systemPackages = [ baoCli ];
|
||||||
|
|
||||||
# The policy the operator writes with the root token for the one-time
|
|
||||||
# granter step, on the host where that step runs.
|
|
||||||
environment.etc."hyperhive/bao-bootstrap-policy.hcl".source = ./bao-bootstrap-policy.hcl;
|
|
||||||
|
|
||||||
# The in-container unit plus the host-side ones this module defines.
|
# The in-container unit plus the host-side ones this module defines.
|
||||||
# `swarm-bao-pki` and `swarm-bao-matrix-token` are declared by the glue
|
# `swarm-bao-pki` and `swarm-bao-matrix-token` are declared by the glue
|
||||||
# modules that create them, per the option's own rule — and a name
|
# modules that create them, per the option's own rule — and a name
|
||||||
|
|
@ -1918,7 +1674,6 @@ in
|
||||||
"swarm-bao-certs"
|
"swarm-bao-certs"
|
||||||
"swarm-bao-token"
|
"swarm-bao-token"
|
||||||
"swarm-bao-forwarder-oidc"
|
"swarm-bao-forwarder-oidc"
|
||||||
"swarm-bao-granter-role"
|
|
||||||
"swarm-bao-controller-policy"
|
"swarm-bao-controller-policy"
|
||||||
"swarm-bao-secret-publisher-policy"
|
"swarm-bao-secret-publisher-policy"
|
||||||
"swarm-bao-matrix-ctl-policy"
|
"swarm-bao-matrix-ctl-policy"
|
||||||
|
|
@ -2210,85 +1965,17 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
# The one unit that still acts with the bootstrap token: it creates the
|
|
||||||
# auth mounts and the granter's own policy and role, which nothing the
|
|
||||||
# granter holds may write. Skipped while the token is absent, which is
|
|
||||||
# the steady state once it has run; the granting units below are the ones
|
|
||||||
# that fail loudly when it has never run.
|
|
||||||
#
|
|
||||||
# Ordering only toward them, never a requirement, for that same reason.
|
|
||||||
systemd.services.swarm-bao-granter-role = lib.mkIf (haveBootstrapToken && haveGranter) {
|
|
||||||
description = "write the bao granter's policy and cert-auth role with the bootstrap token";
|
|
||||||
after = [ "container@${cfg.machine}.service" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
path = [
|
|
||||||
baoCli
|
|
||||||
pkgs.coreutils
|
|
||||||
];
|
|
||||||
# Named but not placed is a legitimate state: all-local supplies the
|
|
||||||
# path as a default and the operator drops the file there after
|
|
||||||
# `bao operator init`. Skipping rather than failing is also what makes
|
|
||||||
# deleting the token at the end of that procedure safe.
|
|
||||||
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
|
|
||||||
# Same unseal wait as the controller's unit below, for the reason
|
|
||||||
# stated there.
|
|
||||||
startLimitBurst = 2880;
|
|
||||||
startLimitIntervalSec = 90000;
|
|
||||||
serviceConfig = {
|
|
||||||
Type = "oneshot";
|
|
||||||
RemainAfterExit = true;
|
|
||||||
Restart = "on-failure";
|
|
||||||
RestartSec = 30;
|
|
||||||
};
|
|
||||||
script = ''
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
|
|
||||||
export BAO_TOKEN
|
|
||||||
|
|
||||||
# Every cert-auth role in this file lives under `auth/cert/`, and
|
|
||||||
# nothing else creates that mount.
|
|
||||||
#
|
|
||||||
# Asked rather than attempted: `auth enable` errors on a mount
|
|
||||||
# that already exists, and recognising that would tie a rebuild
|
|
||||||
# to an error string we have never seen this store emit.
|
|
||||||
mounted="$(bao auth list -format=json)"
|
|
||||||
case "$mounted" in
|
|
||||||
*'"cert/"'*) ;;
|
|
||||||
*) bao auth enable cert ;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
case "$mounted" in
|
|
||||||
*'"approle/"'*) ;;
|
|
||||||
*) bao auth enable approle ;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
printf '%s' ${lib.escapeShellArg granterPolicyText} |
|
|
||||||
bao policy write ${lib.escapeShellArg granterPolicyName} -
|
|
||||||
|
|
||||||
# The TTL bounds a leaked login token to minutes; the leaf is what
|
|
||||||
# lives long.
|
|
||||||
bao write auth/cert/certs/${lib.escapeShellArg granterPolicyName} \
|
|
||||||
certificate=@${tlsDir}/client-ca.pem \
|
|
||||||
allowed_common_names=${lib.escapeShellArg granterCn} \
|
|
||||||
token_policies=${lib.escapeShellArg granterPolicyName} \
|
|
||||||
display_name=${lib.escapeShellArg granterCn} \
|
|
||||||
token_ttl=15m \
|
|
||||||
token_max_ttl=15m
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
# The swarm's first grant, written from the HOST. Every API listener sets
|
# The swarm's first grant, written from the HOST. Every API listener sets
|
||||||
# `tls_require_and_verify_client_cert`, so a client needs an identity
|
# `tls_require_and_verify_client_cert`, so a client needs an identity
|
||||||
# wherever it runs — and only the host has one: the granter's leaf.
|
# wherever it runs — and only the host has one. The bootstrap token is a
|
||||||
systemd.services.swarm-bao-controller-policy = lib.mkIf haveGranter {
|
# host path too; the container saw it through a bind mount.
|
||||||
|
systemd.services.swarm-bao-controller-policy = lib.mkIf haveBootstrapToken {
|
||||||
description = "write the swarm controller's bao policy and cert-auth role";
|
description = "write the swarm controller's bao policy and cert-auth role";
|
||||||
after = [ "container@${cfg.machine}.service" ] ++ granterAfter;
|
after = [ "container@${cfg.machine}.service" ];
|
||||||
requires = [ "swarm-bao-pki.service" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
wantedBy = [ "multi-user.target" ];
|
||||||
# The wrapper rather than the package: it carries the address and the
|
# The wrapper rather than the package: it carries the address, the CA
|
||||||
# CA, which is what makes running here cheaper than shipping an
|
# and this host's certificate, which is what makes running here cheaper
|
||||||
# identity the other way.
|
# than shipping an identity the other way.
|
||||||
path = [
|
path = [
|
||||||
baoCli
|
baoCli
|
||||||
pkgs.coreutils
|
pkgs.coreutils
|
||||||
|
|
@ -2296,7 +1983,11 @@ in
|
||||||
# regeneration guard below turns that into a decision.
|
# regeneration guard below turns that into a decision.
|
||||||
pkgs.openssl
|
pkgs.openssl
|
||||||
];
|
];
|
||||||
environment = granterEnv;
|
# Named but not placed is a legitimate state: all-local supplies the
|
||||||
|
# path as a default and the operator drops the file there after
|
||||||
|
# `bao operator init`. Skipping rather than failing is also what makes
|
||||||
|
# deleting the token at the end of that procedure safe.
|
||||||
|
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
|
||||||
# A store that is up is not necessarily unsealed — under
|
# A store that is up is not necessarily unsealed — under
|
||||||
# `seal = "shamir"` an operator unseals BY HAND, so early attempts fail
|
# `seal = "shamir"` an operator unseals BY HAND, so early attempts fail
|
||||||
# for as long as that takes, which can be a day.
|
# for as long as that takes, which can be a day.
|
||||||
|
|
@ -2318,7 +2009,8 @@ in
|
||||||
script = ''
|
script = ''
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
${granterLogin}
|
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
|
||||||
|
export BAO_TOKEN
|
||||||
|
|
||||||
# Idempotent on purpose: a rebuild re-asserts the policy rather
|
# Idempotent on purpose: a rebuild re-asserts the policy rather
|
||||||
# than failing on one that already exists.
|
# than failing on one that already exists.
|
||||||
|
|
@ -2331,9 +2023,11 @@ in
|
||||||
# controller's first credential write fails against a grant that
|
# controller's first credential write fails against a grant that
|
||||||
# reads as correct.
|
# reads as correct.
|
||||||
#
|
#
|
||||||
# Asked rather than attempted, same as the auth mounts in
|
# Outside the client-CA block below on purpose: this mount is what
|
||||||
# `swarm-bao-granter-role`: `secrets enable` errors on a path
|
# the controller writes *through*, independent of who may log in.
|
||||||
# already in use.
|
#
|
||||||
|
# Asked rather than attempted, same as the auth mount: `secrets
|
||||||
|
# enable` errors on a path already in use.
|
||||||
mounts="$(bao secrets list -format=json)"
|
mounts="$(bao secrets list -format=json)"
|
||||||
case "$mounts" in
|
case "$mounts" in
|
||||||
*'"${credentialMountPath}/"'*) ;;
|
*'"${credentialMountPath}/"'*) ;;
|
||||||
|
|
@ -2492,6 +2186,28 @@ in
|
||||||
key_bits=4096 \
|
key_bits=4096 \
|
||||||
ttl=${servicesPkiLeafTtl} \
|
ttl=${servicesPkiLeafTtl} \
|
||||||
max_ttl=${servicesPkiLeafTtl}
|
max_ttl=${servicesPkiLeafTtl}
|
||||||
|
''
|
||||||
|
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
|
||||||
|
|
||||||
|
# The policy above grants paths under `auth/cert/`, and nothing
|
||||||
|
# in this tree creates that mount. Without this, the grant names
|
||||||
|
# a location that does not exist and every certificate login
|
||||||
|
# fails — the controller's own, and the per-hive ones it later
|
||||||
|
# issues against the same mount.
|
||||||
|
#
|
||||||
|
# Asked rather than attempted: `auth enable` errors on a mount
|
||||||
|
# that already exists, and recognising that would tie a rebuild
|
||||||
|
# to an error string we have never seen this store emit.
|
||||||
|
mounted="$(bao auth list -format=json)"
|
||||||
|
case "$mounted" in
|
||||||
|
*'"cert/"'*) ;;
|
||||||
|
*) bao auth enable cert ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
case "$mounted" in
|
||||||
|
*'"approle/"'*) ;;
|
||||||
|
*) bao auth enable approle ;;
|
||||||
|
esac
|
||||||
|
|
||||||
# `certificate=` is the CA, so this role trusts every leaf that
|
# `certificate=` is the CA, so this role trusts every leaf that
|
||||||
# CA signed and `allowed_common_names` is the whole narrowing —
|
# CA signed and `allowed_common_names` is the whole narrowing —
|
||||||
|
|
@ -2514,25 +2230,23 @@ in
|
||||||
# Widening it to two principals would make the name wrong, and renaming it
|
# Widening it to two principals would make the name wrong, and renaming it
|
||||||
# would make that instruction wrong.
|
# would make that instruction wrong.
|
||||||
#
|
#
|
||||||
# `after` and not `requires`: the unit above and the granter's create the
|
# `after` and not `requires`: the unit above creates the KV and cert-auth
|
||||||
# mounts this one writes into, but a failed oneshot still counts as
|
# mounts this one writes into, but a failed oneshot still counts as
|
||||||
# finished, so `requires` would neither wait for its success nor re-run
|
# finished, so `requires` would neither wait for its success nor re-run
|
||||||
# this one when its own retry eventually lands. Ordering plus this unit's
|
# this one when its own retry eventually lands. Ordering plus this unit's
|
||||||
# own retry is what actually converges.
|
# own retry is what actually converges.
|
||||||
systemd.services.swarm-bao-secret-publisher-policy = lib.mkIf haveGranter {
|
systemd.services.swarm-bao-secret-publisher-policy = lib.mkIf haveBootstrapToken {
|
||||||
description = "write the swarm secret publisher's bao policy and cert-auth role";
|
description = "write the swarm secret publisher's bao policy and cert-auth role";
|
||||||
after = [
|
after = [
|
||||||
"container@${cfg.machine}.service"
|
"container@${cfg.machine}.service"
|
||||||
"swarm-bao-controller-policy.service"
|
"swarm-bao-controller-policy.service"
|
||||||
]
|
];
|
||||||
++ granterAfter;
|
|
||||||
requires = [ "swarm-bao-pki.service" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
wantedBy = [ "multi-user.target" ];
|
||||||
path = [
|
path = [
|
||||||
baoCli
|
baoCli
|
||||||
pkgs.coreutils
|
pkgs.coreutils
|
||||||
];
|
];
|
||||||
environment = granterEnv;
|
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
|
||||||
# Same unseal wait as its sibling above, for the reason stated there:
|
# Same unseal wait as its sibling above, for the reason stated there:
|
||||||
# under `seal = "shamir"` a human unseals by hand, which can take a day.
|
# under `seal = "shamir"` a human unseals by hand, which can take a day.
|
||||||
startLimitBurst = 2880;
|
startLimitBurst = 2880;
|
||||||
|
|
@ -2546,10 +2260,13 @@ in
|
||||||
script = ''
|
script = ''
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
${granterLogin}
|
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
|
||||||
|
export BAO_TOKEN
|
||||||
|
|
||||||
printf '%s' ${lib.escapeShellArg secretPublisherPolicyText} |
|
printf '%s' ${lib.escapeShellArg secretPublisherPolicyText} |
|
||||||
bao policy write ${lib.escapeShellArg secretPublisherPolicyName} -
|
bao policy write ${lib.escapeShellArg secretPublisherPolicyName} -
|
||||||
|
''
|
||||||
|
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
|
||||||
|
|
||||||
bao write auth/cert/certs/${lib.escapeShellArg secretPublisherPolicyName} \
|
bao write auth/cert/certs/${lib.escapeShellArg secretPublisherPolicyName} \
|
||||||
certificate=@${tlsDir}/client-ca.pem \
|
certificate=@${tlsDir}/client-ca.pem \
|
||||||
|
|
@ -2566,20 +2283,18 @@ in
|
||||||
# creates the mounts this one writes into, but a failed oneshot still
|
# creates the mounts this one writes into, but a failed oneshot still
|
||||||
# counts as finished, so only ordering plus this unit's own retry
|
# counts as finished, so only ordering plus this unit's own retry
|
||||||
# converges.
|
# converges.
|
||||||
systemd.services.swarm-bao-matrix-ctl-policy = lib.mkIf haveGranter {
|
systemd.services.swarm-bao-matrix-ctl-policy = lib.mkIf haveBootstrapToken {
|
||||||
description = "write swarm-matrix-ctl's bao policy and cert-auth role";
|
description = "write swarm-matrix-ctl's bao policy and cert-auth role";
|
||||||
after = [
|
after = [
|
||||||
"container@${cfg.machine}.service"
|
"container@${cfg.machine}.service"
|
||||||
"swarm-bao-controller-policy.service"
|
"swarm-bao-controller-policy.service"
|
||||||
]
|
];
|
||||||
++ granterAfter;
|
|
||||||
requires = [ "swarm-bao-pki.service" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
wantedBy = [ "multi-user.target" ];
|
||||||
path = [
|
path = [
|
||||||
baoCli
|
baoCli
|
||||||
pkgs.coreutils
|
pkgs.coreutils
|
||||||
];
|
];
|
||||||
environment = granterEnv;
|
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
|
||||||
# Same unseal wait as its two siblings above, for the reason stated
|
# Same unseal wait as its two siblings above, for the reason stated
|
||||||
# there: under `seal = "shamir"` a human unseals by hand.
|
# there: under `seal = "shamir"` a human unseals by hand.
|
||||||
startLimitBurst = 2880;
|
startLimitBurst = 2880;
|
||||||
|
|
@ -2593,10 +2308,13 @@ in
|
||||||
script = ''
|
script = ''
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
${granterLogin}
|
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
|
||||||
|
export BAO_TOKEN
|
||||||
|
|
||||||
printf '%s' ${lib.escapeShellArg matrixCtlPolicyText} |
|
printf '%s' ${lib.escapeShellArg matrixCtlPolicyText} |
|
||||||
bao policy write ${lib.escapeShellArg matrixCtlPolicyName} -
|
bao policy write ${lib.escapeShellArg matrixCtlPolicyName} -
|
||||||
|
''
|
||||||
|
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
|
||||||
|
|
||||||
bao write auth/cert/certs/${lib.escapeShellArg matrixCtlPolicyName} \
|
bao write auth/cert/certs/${lib.escapeShellArg matrixCtlPolicyName} \
|
||||||
certificate=@${tlsDir}/client-ca.pem \
|
certificate=@${tlsDir}/client-ca.pem \
|
||||||
|
|
@ -2631,20 +2349,18 @@ in
|
||||||
# The policy text moved here from the controller's unit, where it sat
|
# The policy text moved here from the controller's unit, where it sat
|
||||||
# while it attached to nothing — a policy and the role that carries it
|
# while it attached to nothing — a policy and the role that carries it
|
||||||
# belong in one place, and now there is a principal to put them with.
|
# belong in one place, and now there is a principal to put them with.
|
||||||
systemd.services.swarm-bao-services-issuer-policy = lib.mkIf haveGranter {
|
systemd.services.swarm-bao-services-issuer-policy = lib.mkIf haveBootstrapToken {
|
||||||
description = "write the swarm services issuer's bao policy and cert-auth role";
|
description = "write the swarm services issuer's bao policy and cert-auth role";
|
||||||
after = [
|
after = [
|
||||||
"container@${cfg.machine}.service"
|
"container@${cfg.machine}.service"
|
||||||
"swarm-bao-controller-policy.service"
|
"swarm-bao-controller-policy.service"
|
||||||
]
|
];
|
||||||
++ granterAfter;
|
|
||||||
requires = [ "swarm-bao-pki.service" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
wantedBy = [ "multi-user.target" ];
|
||||||
path = [
|
path = [
|
||||||
baoCli
|
baoCli
|
||||||
pkgs.coreutils
|
pkgs.coreutils
|
||||||
];
|
];
|
||||||
environment = granterEnv;
|
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
|
||||||
# Same unseal wait as its three siblings above, for the reason stated
|
# Same unseal wait as its three siblings above, for the reason stated
|
||||||
# there: under `seal = "shamir"` a human unseals by hand.
|
# there: under `seal = "shamir"` a human unseals by hand.
|
||||||
startLimitBurst = 2880;
|
startLimitBurst = 2880;
|
||||||
|
|
@ -2658,10 +2374,13 @@ in
|
||||||
script = ''
|
script = ''
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
${granterLogin}
|
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
|
||||||
|
export BAO_TOKEN
|
||||||
|
|
||||||
printf '%s' ${lib.escapeShellArg servicesIssuerPolicyText} |
|
printf '%s' ${lib.escapeShellArg servicesIssuerPolicyText} |
|
||||||
bao policy write ${lib.escapeShellArg servicesIssuerPolicyName} -
|
bao policy write ${lib.escapeShellArg servicesIssuerPolicyName} -
|
||||||
|
''
|
||||||
|
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
|
||||||
|
|
||||||
bao write auth/cert/certs/${lib.escapeShellArg servicesIssuerPolicyName} \
|
bao write auth/cert/certs/${lib.escapeShellArg servicesIssuerPolicyName} \
|
||||||
certificate=@${tlsDir}/client-ca.pem \
|
certificate=@${tlsDir}/client-ca.pem \
|
||||||
|
|
@ -2679,20 +2398,18 @@ in
|
||||||
# The role narrows exactly as `swarm-services` does, to one name. It is
|
# The role narrows exactly as `swarm-services` does, to one name. It is
|
||||||
# the queue's domain alone, since the same name reaches it from every
|
# the queue's domain alone, since the same name reaches it from every
|
||||||
# hive; no IP SANs, since nothing dials an address.
|
# hive; no IP SANs, since nothing dials an address.
|
||||||
systemd.services.swarm-bao-nats-tls-policy = lib.mkIf haveGranter {
|
systemd.services.swarm-bao-nats-tls-policy = lib.mkIf haveBootstrapToken {
|
||||||
description = "write the swarm queue's pki role, bao policy and cert-auth role";
|
description = "write the swarm queue's pki role, bao policy and cert-auth role";
|
||||||
after = [
|
after = [
|
||||||
"container@${cfg.machine}.service"
|
"container@${cfg.machine}.service"
|
||||||
"swarm-bao-controller-policy.service"
|
"swarm-bao-controller-policy.service"
|
||||||
]
|
];
|
||||||
++ granterAfter;
|
|
||||||
requires = [ "swarm-bao-pki.service" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
wantedBy = [ "multi-user.target" ];
|
||||||
path = [
|
path = [
|
||||||
baoCli
|
baoCli
|
||||||
pkgs.coreutils
|
pkgs.coreutils
|
||||||
];
|
];
|
||||||
environment = granterEnv;
|
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
|
||||||
# Same unseal wait as its siblings above.
|
# Same unseal wait as its siblings above.
|
||||||
startLimitBurst = 2880;
|
startLimitBurst = 2880;
|
||||||
startLimitIntervalSec = 90000;
|
startLimitIntervalSec = 90000;
|
||||||
|
|
@ -2705,7 +2422,8 @@ in
|
||||||
script = ''
|
script = ''
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
${granterLogin}
|
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
|
||||||
|
export BAO_TOKEN
|
||||||
|
|
||||||
bao write ${lib.escapeShellArg "${servicesPkiMountPath}/roles/${natsPkiRoleName}"} \
|
bao write ${lib.escapeShellArg "${servicesPkiMountPath}/roles/${natsPkiRoleName}"} \
|
||||||
allowed_domains=${lib.escapeShellArg hyperhiveCfg.swarm.nats.domain} \
|
allowed_domains=${lib.escapeShellArg hyperhiveCfg.swarm.nats.domain} \
|
||||||
|
|
@ -2725,6 +2443,8 @@ in
|
||||||
|
|
||||||
printf '%s' ${lib.escapeShellArg natsPolicyText} |
|
printf '%s' ${lib.escapeShellArg natsPolicyText} |
|
||||||
bao policy write ${lib.escapeShellArg natsPolicyName} -
|
bao policy write ${lib.escapeShellArg natsPolicyName} -
|
||||||
|
''
|
||||||
|
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
|
||||||
|
|
||||||
bao write auth/cert/certs/${lib.escapeShellArg natsPolicyName} \
|
bao write auth/cert/certs/${lib.escapeShellArg natsPolicyName} \
|
||||||
certificate=@${tlsDir}/client-ca.pem \
|
certificate=@${tlsDir}/client-ca.pem \
|
||||||
|
|
|
||||||
|
|
@ -52,7 +52,6 @@ let
|
||||||
deployCfg.bao.forwarderOidcCommonName
|
deployCfg.bao.forwarderOidcCommonName
|
||||||
deployCfg.bao.servicesIssuerCommonName
|
deployCfg.bao.servicesIssuerCommonName
|
||||||
deployCfg.bao.natsCommonName
|
deployCfg.bao.natsCommonName
|
||||||
deployCfg.bao.granterCommonName
|
|
||||||
]
|
]
|
||||||
# The two per-hive readers' subjects, spelled out per hive rather than as the
|
# The two per-hive readers' subjects, spelled out per hive rather than as the
|
||||||
# prefix. The prefix alone would reserve the wrong string: the role for hive
|
# prefix. The prefix alone would reserve the wrong string: the role for hive
|
||||||
|
|
|
||||||
|
|
@ -22,7 +22,7 @@ let
|
||||||
;
|
;
|
||||||
|
|
||||||
# The store, plus a placed bootstrap token: the only shape in which the
|
# The store, plus a placed bootstrap token: the only shape in which the
|
||||||
# granter's own role can be written at all.
|
# swarm's first grant can be written at all.
|
||||||
baoGrantHere = hive {
|
baoGrantHere = hive {
|
||||||
deploy.bao.enable = true;
|
deploy.bao.enable = true;
|
||||||
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
||||||
|
|
@ -36,31 +36,10 @@ let
|
||||||
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
||||||
};
|
};
|
||||||
|
|
||||||
# The store with no bootstrap token: the steady state once the granter is set
|
|
||||||
# up, and the state of a store host that has never named one.
|
|
||||||
baoGranterNoToken = hive {
|
|
||||||
deploy.bao.enable = true;
|
|
||||||
};
|
|
||||||
|
|
||||||
# The store with the granter's pair taken away: the deployment that writes
|
|
||||||
# its grants some other way.
|
|
||||||
baoGranterOptOut = hive {
|
|
||||||
deploy.bao.enable = true;
|
|
||||||
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
|
||||||
deploy.bao.granterClientCertFile = lib.mkForce null;
|
|
||||||
deploy.bao.granterClientKeyFile = lib.mkForce null;
|
|
||||||
};
|
|
||||||
|
|
||||||
# A pki role the granter's `roles/swarm-*` does not reach.
|
|
||||||
baoGranterOddPkiRole = hive {
|
|
||||||
deploy.bao.enable = true;
|
|
||||||
deploy.bao.natsPkiRoleName = "queue";
|
|
||||||
};
|
|
||||||
|
|
||||||
# The store and the token, with no CA to trust. `mkForce` because the PKI
|
# The store and the token, with no CA to trust. `mkForce` because the PKI
|
||||||
# glue supplies one by default here — this is the deployment that brings its
|
# glue supplies one by default here — this is the deployment that brings its
|
||||||
# own certificates and has not named the authority yet, in which nothing can
|
# own certificates and has not named the authority yet, and it separates
|
||||||
# log in as the granter.
|
# "the grant unit runs" from "cert auth can be set up".
|
||||||
baoGrantNoClientCa = hive {
|
baoGrantNoClientCa = hive {
|
||||||
deploy.bao.enable = true;
|
deploy.bao.enable = true;
|
||||||
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
||||||
|
|
@ -123,71 +102,38 @@ let
|
||||||
"swarm-bao-otel-oidc"
|
"swarm-bao-otel-oidc"
|
||||||
];
|
];
|
||||||
|
|
||||||
# Two credentials write grants, and each is checked against what the units
|
# What the bootstrap token may do, read from the file the operator writes it
|
||||||
# holding it actually call. The bootstrap token's policy is read from the
|
# from (../../docs/getting-started/setup.md points there), against what the
|
||||||
# file the operator writes it from (../../docs/getting-started/setup.md
|
# units holding that token actually call. The units are found by the token
|
||||||
# points there); the granter's from the unit that writes it. Units are found
|
# path in their script rather than by name, so a new one is checked without
|
||||||
# by the credential they read rather than by name, so a new one is checked
|
# anyone listing it here.
|
||||||
# without anyone listing it here.
|
|
||||||
bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
||||||
|
|
||||||
# The READ, not the path: every granting unit prints the path in the
|
|
||||||
# one-time step it shows when the granter is refused.
|
|
||||||
bootstrapUnits = lib.filterAttrs (
|
bootstrapUnits = lib.filterAttrs (
|
||||||
_: u: lib.hasInfix "cat ${lib.escapeShellArg bootstrapTokenFile}" u.script
|
_: u: lib.hasInfix bootstrapTokenFile u.script
|
||||||
) baoGrantWithConsumers.systemd.services;
|
) baoGrantWithConsumers.systemd.services;
|
||||||
|
|
||||||
# The pair ./glue-bao-tls.nix defaults on a store host.
|
|
||||||
granterCertFile = "/var/lib/swarm-bao-pki/granter.pem";
|
|
||||||
granterKeyFile = "/var/lib/swarm-bao-pki/granter-key.pem";
|
|
||||||
|
|
||||||
granterUnits = lib.filterAttrs (
|
|
||||||
_: u: (u.environment.BAO_CLIENT_CERT or null) == granterCertFile
|
|
||||||
) baoGrantWithConsumers.systemd.services;
|
|
||||||
|
|
||||||
# The ten units that write a `swarm-*` grant, by name, for the discovery
|
|
||||||
# control below.
|
|
||||||
grantingUnitNames = [
|
|
||||||
"swarm-bao-controller-policy"
|
|
||||||
"swarm-bao-secret-publisher-policy"
|
|
||||||
"swarm-bao-matrix-ctl-policy"
|
|
||||||
"swarm-bao-matrix-token-policy"
|
|
||||||
"swarm-bao-queue-agent-policy"
|
|
||||||
"swarm-bao-grafana-oidc-policy"
|
|
||||||
"swarm-bao-otel-oidc-policy"
|
|
||||||
"swarm-bao-forwarder-oidc-policy"
|
|
||||||
"swarm-bao-services-issuer-policy"
|
|
||||||
"swarm-bao-nats-tls-policy"
|
|
||||||
];
|
|
||||||
|
|
||||||
# Comment lines dropped first: both the HCL and the scripts explain
|
# Comment lines dropped first: both the HCL and the scripts explain
|
||||||
# themselves in prose that names paths and `bao` commands.
|
# themselves in prose that names paths and `bao` commands.
|
||||||
codeLines =
|
codeLines =
|
||||||
text: lib.filter (l: builtins.match "[[:space:]]*#.*" l == null) (lib.splitString "\n" text);
|
text: lib.filter (l: builtins.match "[[:space:]]*#.*" l == null) (lib.splitString "\n" text);
|
||||||
|
|
||||||
bootstrapPolicyText = lib.concatStringsSep "\n" (
|
bootstrapPolicyText = lib.concatStringsSep "\n" (
|
||||||
codeLines (builtins.readFile ../host-modules/bao-bootstrap-policy.hcl)
|
codeLines (builtins.readFile ../host-modules/swarm-bao-bootstrap-policy.hcl)
|
||||||
);
|
);
|
||||||
|
|
||||||
# The granter's HCL is the only policy text in the unit that writes it.
|
|
||||||
granterPolicyText = baoGrantHere.systemd.services.swarm-bao-granter-role.script;
|
|
||||||
|
|
||||||
matches = re: text: lib.filter lib.isList (builtins.split re text);
|
matches = re: text: lib.filter lib.isList (builtins.split re text);
|
||||||
|
|
||||||
grantsIn =
|
bootstrapGrants =
|
||||||
text:
|
|
||||||
map
|
map
|
||||||
(m: {
|
(m: {
|
||||||
path = lib.elemAt m 0;
|
path = lib.elemAt m 0;
|
||||||
caps = map lib.head (matches ''"([a-z]+)"'' (lib.elemAt m 1));
|
caps = map lib.head (matches ''"([a-z]+)"'' (lib.elemAt m 1));
|
||||||
})
|
})
|
||||||
(
|
(
|
||||||
matches ''path "([^"]+)"[[:space:]]*[{][[:space:]]*capabilities[[:space:]]*=[[:space:]]*[[]([a-z", ]*)'' text
|
matches ''path "([^"]+)"[[:space:]]*[{][[:space:]]*capabilities[[:space:]]*=[[:space:]]*[[]([a-z", ]*)'' bootstrapPolicyText
|
||||||
);
|
);
|
||||||
|
|
||||||
bootstrapGrants = grantsIn bootstrapPolicyText;
|
|
||||||
granterGrants = grantsIn granterPolicyText;
|
|
||||||
|
|
||||||
# One `bao …` invocation → the path and capabilities it needs, as
|
# One `bao …` invocation → the path and capabilities it needs, as
|
||||||
# `bao <cmd> -output-policy` reports them. Path-specific `sudo` (bao's
|
# `bao <cmd> -output-policy` reports them. Path-specific `sudo` (bao's
|
||||||
# root-protected paths, e.g. `pki/root` for a delete) does not follow from
|
# root-protected paths, e.g. `pki/root` for a delete) does not follow from
|
||||||
|
|
@ -208,10 +154,7 @@ let
|
||||||
"update"
|
"update"
|
||||||
];
|
];
|
||||||
in
|
in
|
||||||
# A login and a seal-status check are unauthenticated: no policy grants them.
|
if a 0 == "policy" && a 1 == "write" then
|
||||||
if a 0 == "login" || a 0 == "status" then
|
|
||||||
null
|
|
||||||
else if a 0 == "policy" && a 1 == "write" then
|
|
||||||
need "sys/policies/acl/${a 2}" cu
|
need "sys/policies/acl/${a 2}" cu
|
||||||
else if a 0 == "secrets" && a 1 == "list" then
|
else if a 0 == "secrets" && a 1 == "list" then
|
||||||
need "sys/mounts" [ "read" ]
|
need "sys/mounts" [ "read" ]
|
||||||
|
|
@ -236,7 +179,6 @@ let
|
||||||
|
|
||||||
baoCalls =
|
baoCalls =
|
||||||
script:
|
script:
|
||||||
lib.filter (n: n != null) (
|
|
||||||
map
|
map
|
||||||
(
|
(
|
||||||
inv:
|
inv:
|
||||||
|
|
@ -246,18 +188,16 @@ let
|
||||||
lib.concatMap (l: map (m: lib.elemAt m 1) (matches "(^[[:space:]]*|[$][(]|[)] )bao ([^|;)]*)" l)) (
|
lib.concatMap (l: map (m: lib.elemAt m 1) (matches "(^[[:space:]]*|[$][(]|[)] )bao ([^|;)]*)" l)) (
|
||||||
codeLines script
|
codeLines script
|
||||||
)
|
)
|
||||||
)
|
|
||||||
);
|
);
|
||||||
|
|
||||||
# bao's own rule (vault/policy/acl.go): an exact path wins, otherwise the
|
# bao's own rule: an exact path wins, otherwise the longest glob prefix.
|
||||||
# longest glob prefix, and a trailing `*` is a plain string prefix.
|
bootstrapGrantFor =
|
||||||
grantFor =
|
path:
|
||||||
grants: path:
|
|
||||||
let
|
let
|
||||||
exact = lib.filter (g: g.path == path) grants;
|
exact = lib.filter (g: g.path == path) bootstrapGrants;
|
||||||
globs = lib.filter (
|
globs = lib.filter (
|
||||||
g: lib.hasSuffix "*" g.path && lib.hasPrefix (lib.removeSuffix "*" g.path) path
|
g: lib.hasSuffix "*" g.path && lib.hasPrefix (lib.removeSuffix "*" g.path) path
|
||||||
) grants;
|
) bootstrapGrants;
|
||||||
in
|
in
|
||||||
if exact != [ ] then
|
if exact != [ ] then
|
||||||
lib.head exact
|
lib.head exact
|
||||||
|
|
@ -266,40 +206,33 @@ let
|
||||||
best: g: if best == null || lib.stringLength g.path > lib.stringLength best.path then g else best
|
best: g: if best == null || lib.stringLength g.path > lib.stringLength best.path then g else best
|
||||||
) null globs;
|
) null globs;
|
||||||
|
|
||||||
ungranted =
|
bootstrapUngranted = lib.concatLists (
|
||||||
grants: units:
|
|
||||||
lib.concatLists (
|
|
||||||
lib.mapAttrsToList (
|
lib.mapAttrsToList (
|
||||||
unit: u:
|
unit: u:
|
||||||
map (n: "${unit}: `bao ${n.call}` needs ${n.path} [${toString n.caps}]") (
|
map (n: "${unit}: `bao ${n.call}` needs ${n.path} [${toString n.caps}]") (
|
||||||
lib.filter (
|
lib.filter (
|
||||||
n:
|
n:
|
||||||
let
|
let
|
||||||
g = grantFor grants n.path;
|
g = bootstrapGrantFor n.path;
|
||||||
in
|
in
|
||||||
g == null || !(lib.all (c: lib.elem c g.caps) n.caps)
|
g == null || !(lib.all (c: lib.elem c g.caps) n.caps)
|
||||||
) (baoCalls u.script)
|
) (baoCalls u.script)
|
||||||
)
|
)
|
||||||
) units
|
) bootstrapUnits
|
||||||
);
|
);
|
||||||
|
|
||||||
bootstrapUngranted = ungranted bootstrapGrants bootstrapUnits;
|
|
||||||
granterUngranted = ungranted granterGrants granterUnits;
|
|
||||||
|
|
||||||
cases = [
|
cases = [
|
||||||
{
|
{
|
||||||
# Reads the rendered unit on the HOST, which is where the write happens:
|
# Reads the rendered unit on the HOST, which is where the write happens:
|
||||||
# every API listener demands a client certificate, and the host is the
|
# every API listener demands a client certificate, and the host is the
|
||||||
# side that has one.
|
# side that has one.
|
||||||
name = "a store host renders the granting unit on the host, logging in as the granter";
|
name = "a store host with a placed bootstrap token renders the granting unit on the host";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
u = baoGrantHere.systemd.services.swarm-bao-controller-policy;
|
u = baoGrantHere.systemd.services.swarm-bao-controller-policy;
|
||||||
in
|
in
|
||||||
u.environment.BAO_CLIENT_CERT == granterCertFile
|
lib.hasInfix "/run/secrets/bao-bootstrap.token" u.script
|
||||||
&& u.environment.BAO_CLIENT_KEY == granterKeyFile
|
&& u.unitConfig.ConditionPathExists == "/run/secrets/bao-bootstrap.token";
|
||||||
&& lib.hasInfix "bao login -method=cert -token-only" u.script
|
|
||||||
&& !(u.unitConfig ? ConditionPathExists);
|
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The move is the fix, so pin the side it landed on: in the container it
|
# The move is the fix, so pin the side it landed on: in the container it
|
||||||
|
|
@ -340,12 +273,13 @@ let
|
||||||
{
|
{
|
||||||
# Same host-side reasoning as the controller's granting unit above: the
|
# Same host-side reasoning as the controller's granting unit above: the
|
||||||
# write needs a client certificate and the host is the side that has one.
|
# write needs a client certificate and the host is the side that has one.
|
||||||
name = "a store host renders the publisher's granting unit too, logging in as the granter";
|
name = "a store host with a placed bootstrap token renders the publisher's granting unit too";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
u = baoGrantHere.systemd.services.swarm-bao-secret-publisher-policy;
|
u = baoGrantHere.systemd.services.swarm-bao-secret-publisher-policy;
|
||||||
in
|
in
|
||||||
u.environment.BAO_CLIENT_CERT == granterCertFile && lib.hasInfix "swarm-secret-publisher" u.script;
|
u.unitConfig.ConditionPathExists == "/run/secrets/bao-bootstrap.token"
|
||||||
|
&& lib.hasInfix "swarm-secret-publisher" u.script;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The control for the case above, and the same one the controller's unit
|
# The control for the case above, and the same one the controller's unit
|
||||||
|
|
@ -656,18 +590,29 @@ let
|
||||||
];
|
];
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The absence arm: with no client CA there is no trust anchor, so no
|
# The absence arm: with no client CA there is no trust anchor, so the
|
||||||
# role can be written and nothing can log in as the granter. The units
|
# login roles cannot be written — but the policies they would attach are
|
||||||
# are gone, so the deployment has to say so itself.
|
# still asserted, exactly as the three service principals above behave in
|
||||||
name = "with no client CA no granting unit renders, and the deployment warns";
|
# this deployment. A unit that vanished here would take the policy with
|
||||||
|
# it and leave nothing to diagnose.
|
||||||
|
name = "with no client CA the five readers get policies but no login roles";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
s = baoGrantNoClientCa.systemd.services;
|
units = [
|
||||||
|
"swarm-bao-matrix-token-policy"
|
||||||
|
"swarm-bao-queue-agent-policy"
|
||||||
|
"swarm-bao-grafana-oidc-policy"
|
||||||
|
"swarm-bao-otel-oidc-policy"
|
||||||
|
"swarm-bao-forwarder-oidc-policy"
|
||||||
|
];
|
||||||
|
scriptOf = unit: baoGrantNoClientCa.systemd.services.${unit}.script;
|
||||||
in
|
in
|
||||||
lib.all (unit: !(s ? ${unit})) (grantingUnitNames ++ [ "swarm-bao-granter-role" ])
|
lib.all (
|
||||||
&& lib.any (lib.hasInfix "services.hyperhive.deploy.bao.clientCaFile is null") baoGrantNoClientCa.warnings
|
unit:
|
||||||
# The control: a store with a CA does not warn.
|
(baoGrantNoClientCa.systemd.services ? ${unit})
|
||||||
&& !(lib.any (lib.hasInfix "clientCaFile is null") baoGrantHere.warnings);
|
&& lib.hasInfix "bao policy write" (scriptOf unit)
|
||||||
|
&& !(lib.hasInfix "auth/cert/certs" (scriptOf unit))
|
||||||
|
) units;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# Same control the three service principals carry: the write needs a
|
# Same control the three service principals carry: the write needs a
|
||||||
|
|
@ -694,8 +639,7 @@ let
|
||||||
{
|
{
|
||||||
# The other end of those units: each reader logs in against the role its
|
# The other end of those units: each reader logs in against the role its
|
||||||
# own policy unit writes, so it has to wait for that unit. Ordering and
|
# own policy unit writes, so it has to wait for that unit. Ordering and
|
||||||
# never a requirement: a failed policy unit still counts as done, and the
|
# never a requirement, since the policy unit skips once the token is gone.
|
||||||
# reader's own retries carry it past that.
|
|
||||||
#
|
#
|
||||||
# The forwarder is listed apart from `policyReaders`: it renders wherever
|
# The forwarder is listed apart from `policyReaders`: it renders wherever
|
||||||
# the store does, so it is never absent on a store host and never present
|
# the store does, so it is never absent on a store host and never present
|
||||||
|
|
@ -740,266 +684,21 @@ let
|
||||||
lib.all unordered policyReaders;
|
lib.all unordered policyReaders;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# A store host without the granter's pair writes its grants some other
|
# A store host that has not placed a bootstrap token can write no grant at
|
||||||
# way, so none of the ten units may exist. Without this arm
|
# all, so none of the four units may exist — the same claim
|
||||||
# `lib.mkIf haveGranter` could be dropped from any of them and every other
|
# `baoGrantNoStore` makes for the controller's, one file over. Without
|
||||||
# case here would still pass.
|
# this arm `lib.mkIf haveBootstrapToken` could be dropped from the shared
|
||||||
name = "without the granter's pair none of the ten granting units render";
|
# builder and every other case here would still pass.
|
||||||
|
name = "without a bootstrap token none of the five readers' granting units render";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
s = baoGranterOptOut.systemd.services;
|
s = baoGrantNoStore.systemd.services;
|
||||||
in
|
in
|
||||||
lib.all (unit: !(s ? ${unit})) (grantingUnitNames ++ [ "swarm-bao-granter-role" ])
|
!(s ? swarm-bao-matrix-token-policy)
|
||||||
# The control: the same store with the pair renders all ten.
|
&& !(s ? swarm-bao-queue-agent-policy)
|
||||||
&& lib.all (unit: baoGrantHere.systemd.services ? ${unit}) grantingUnitNames;
|
&& !(s ? swarm-bao-grafana-oidc-policy)
|
||||||
}
|
&& !(s ? swarm-bao-otel-oidc-policy)
|
||||||
{
|
&& !(s ? swarm-bao-forwarder-oidc-policy);
|
||||||
# 🩸 What replaced the silent skip. With no bootstrap token the ten still
|
|
||||||
# render, and a refused granter fails them with the step that fixes it.
|
|
||||||
# A store host that never named a token is told to name one, since the
|
|
||||||
# unit that sets the granter up renders only where it has.
|
|
||||||
name = "a store host without a bootstrap token renders the ten, each failing loudly with the one-time step";
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
s = baoGranterNoToken.systemd.services;
|
|
||||||
loud =
|
|
||||||
unit:
|
|
||||||
s ? ${unit}
|
|
||||||
&&
|
|
||||||
lib.hasInfix "bao policy write bao-bootstrap /etc/hyperhive/bao-bootstrap-policy.hcl"
|
|
||||||
s.${unit}.script
|
|
||||||
&& lib.hasInfix "set services.hyperhive.deploy.bao.bootstrapTokenFile" s.${unit}.script
|
|
||||||
&& lib.hasInfix "exit 1" s.${unit}.script;
|
|
||||||
in
|
|
||||||
lib.all loud grantingUnitNames && !(s ? swarm-bao-granter-role);
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Where the token is named, the step names the file to put it in and the
|
|
||||||
# unit to restart.
|
|
||||||
name = "with a bootstrap token named, the one-time step places it and restarts the granter's unit";
|
|
||||||
ok = lib.all (
|
|
||||||
unit:
|
|
||||||
let
|
|
||||||
sc = baoGrantHere.systemd.services.${unit}.script;
|
|
||||||
in
|
|
||||||
lib.hasInfix "install -D -m 0600 /dev/stdin /run/secrets/bao-bootstrap.token" sc
|
|
||||||
&& lib.hasInfix "systemctl restart swarm-bao-granter-role" sc
|
|
||||||
) grantingUnitNames;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Every granting unit retries a sealed or late store for a day, in the
|
|
||||||
# `[Unit]` section systemd reads it from, and waits for the unit that
|
|
||||||
# mints the granter's leaf.
|
|
||||||
name = "each granting unit requires the PKI unit and retries 2880 times at 30s";
|
|
||||||
ok = lib.all (
|
|
||||||
unit:
|
|
||||||
let
|
|
||||||
u = baoGrantHere.systemd.services.${unit};
|
|
||||||
in
|
|
||||||
lib.elem "swarm-bao-pki.service" u.requires
|
|
||||||
&& lib.elem "swarm-bao-pki.service" u.after
|
|
||||||
&& lib.elem "swarm-bao-granter-role.service" u.after
|
|
||||||
&& !(lib.elem "swarm-bao-granter-role.service" (u.requires ++ u.wants))
|
|
||||||
&& toString u.unitConfig.StartLimitBurst == "2880"
|
|
||||||
&& toString u.unitConfig.StartLimitIntervalSec == "90000"
|
|
||||||
&& toString u.serviceConfig.RestartSec == "30"
|
|
||||||
&& u.serviceConfig.Restart == "on-failure"
|
|
||||||
) grantingUnitNames;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# The only unit left acting with the token, so the only one that may
|
|
||||||
# skip on it.
|
|
||||||
name = "no unit but the granter's role reads the bootstrap token or skips on it";
|
|
||||||
ok =
|
|
||||||
lib.attrNames bootstrapUnits == [ "swarm-bao-granter-role" ]
|
|
||||||
&& lib.all (u: !(u.unitConfig ? ConditionPathExists)) (lib.attrValues granterUnits)
|
|
||||||
&&
|
|
||||||
baoGrantHere.systemd.services.swarm-bao-granter-role.unitConfig.ConditionPathExists
|
|
||||||
== bootstrapTokenFile;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# The granter's grants, whole. Pinned as the full list, because an added
|
|
||||||
# path or capability is exactly what a presence check misses.
|
|
||||||
name = "the granter's policy is exactly these seventeen stanzas";
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
cu = [
|
|
||||||
"create"
|
|
||||||
"update"
|
|
||||||
];
|
|
||||||
in
|
|
||||||
granterGrants == [
|
|
||||||
{
|
|
||||||
path = "sys/policies/acl/swarm-*";
|
|
||||||
caps = cu;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "auth/cert/certs/swarm-*";
|
|
||||||
caps = cu;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "pki/roles/swarm-*";
|
|
||||||
caps = cu;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "sys/mounts";
|
|
||||||
caps = [ "read" ];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "sys/mounts/secret";
|
|
||||||
caps = cu;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "sys/mounts/pki";
|
|
||||||
caps = cu;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "sys/mounts/pki/tune";
|
|
||||||
caps = cu;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "pki/issuers";
|
|
||||||
caps = [ "list" ];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "pki/cert/ca";
|
|
||||||
caps = [ "read" ];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "pki/root";
|
|
||||||
caps = [
|
|
||||||
"delete"
|
|
||||||
"sudo"
|
|
||||||
];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "pki/root/generate/internal";
|
|
||||||
caps = cu;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "sys/mounts/pki-agents";
|
|
||||||
caps = cu;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "sys/mounts/pki-agents/tune";
|
|
||||||
caps = cu;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "pki-agents/issuers";
|
|
||||||
caps = [ "list" ];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "pki-agents/cert/ca";
|
|
||||||
caps = [ "read" ];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "pki-agents/root/generate/internal";
|
|
||||||
caps = cu;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
path = "pki-agents/roles/swarm-*";
|
|
||||||
caps = cu;
|
|
||||||
}
|
|
||||||
];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Neither its own policy and role nor the bootstrap policy may be
|
|
||||||
# reachable, or the granter could rewrite what constrains it and what the
|
|
||||||
# next bootstrap token carries.
|
|
||||||
name = "the granter cannot reach the policy or role that constrains it, nor the bootstrap policy";
|
|
||||||
ok = lib.all (p: grantFor granterGrants p == null) [
|
|
||||||
"sys/policies/acl/bao-granter"
|
|
||||||
"auth/cert/certs/bao-granter"
|
|
||||||
"sys/policies/acl/bao-bootstrap"
|
|
||||||
];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Outside `swarm-*` and the store's own mounts it holds nothing: no
|
|
||||||
# hive's policy or role, no auth mount, no token, no secret.
|
|
||||||
name = "the granter grants nothing outside swarm-* and the store's own mounts";
|
|
||||||
ok =
|
|
||||||
lib.all (p: grantFor granterGrants p == null) [
|
|
||||||
"sys/policies/acl/hive-x"
|
|
||||||
"auth/cert/certs/hive-x"
|
|
||||||
"sys/auth"
|
|
||||||
"sys/auth/cert"
|
|
||||||
"sys/auth/x"
|
|
||||||
"auth/token/create"
|
|
||||||
"auth/token/create-orphan"
|
|
||||||
"secret/data/x"
|
|
||||||
"secret/data/swarm/agents/x/queue"
|
|
||||||
"sys/policies/acl/x"
|
|
||||||
"sys/policies/acl/root"
|
|
||||||
"pki/issue/swarm-services"
|
|
||||||
"pki/sign/swarm-services"
|
|
||||||
"pki-agents/root"
|
|
||||||
"pki-agents/issue/swarm-agent"
|
|
||||||
"pki-agents/sign/swarm-agent"
|
|
||||||
"pki-agents/sign-verbatim"
|
|
||||||
"*"
|
|
||||||
]
|
|
||||||
&& !(lib.any (
|
|
||||||
g:
|
|
||||||
lib.elem g.path [
|
|
||||||
"*"
|
|
||||||
"sys/policies/acl/*"
|
|
||||||
"auth/cert/certs/*"
|
|
||||||
"pki/roles/*"
|
|
||||||
"pki-agents/roles/*"
|
|
||||||
]
|
|
||||||
) granterGrants);
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Its names sit outside both globs that write grants — its own
|
|
||||||
# `swarm-*` and the controller's `hive-*`.
|
|
||||||
name = "the granter's own names are outside swarm-* and hive-*";
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
sc = baoGrantHere.systemd.services.swarm-bao-granter-role.script;
|
|
||||||
cn = baoGrantHere.services.hyperhive.deploy.bao.granterCommonName;
|
|
||||||
in
|
|
||||||
lib.hasInfix "bao policy write bao-granter -" sc
|
|
||||||
&& lib.hasInfix "auth/cert/certs/bao-granter" sc
|
|
||||||
&& lib.hasInfix "token_policies=bao-granter" sc
|
|
||||||
&& lib.hasInfix "token_ttl=15m" sc
|
|
||||||
&& !(lib.hasPrefix "swarm-" cn)
|
|
||||||
&& !(lib.hasPrefix "hive-" cn);
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# The other principals are what they were: no unit but the granter's own
|
|
||||||
# hands its policy to a role, and none of them logs in as it.
|
|
||||||
name = "no other principal gains the granter's policy";
|
|
||||||
ok =
|
|
||||||
lib.all (u: !(lib.hasInfix "token_policies=bao-granter" u.script)) (
|
|
||||||
lib.attrValues (lib.removeAttrs baoGrantWithConsumers.systemd.services [ "swarm-bao-granter-role" ])
|
|
||||||
)
|
|
||||||
&& lib.all (u: (u.environment.BAO_CLIENT_CERT or null) != granterCertFile) (
|
|
||||||
lib.attrValues (lib.removeAttrs baoGrantWithConsumers.systemd.services grantingUnitNames)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# The minting side: a role matching a subject nothing signs is a
|
|
||||||
# granter that cannot log in.
|
|
||||||
name = "the PKI unit signs the granter's leaf under its own subject";
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
s = baoGrantHere.systemd.services.swarm-bao-pki.script;
|
|
||||||
in
|
|
||||||
lib.hasInfix "/granter.pem ]" s && lib.hasInfix "bao-granter \"\" clientAuth" s;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# The granter writes pki roles through `roles/swarm-*` only, so a role
|
|
||||||
# named otherwise is refused at eval rather than 403'd at deploy.
|
|
||||||
name = "a pki role name outside swarm-* is refused, naming both options";
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
names =
|
|
||||||
a:
|
|
||||||
lib.hasInfix "services.hyperhive.deploy.bao.servicesPkiRoleName" a.message
|
|
||||||
&& lib.hasInfix "services.hyperhive.deploy.bao.natsPkiRoleName" a.message;
|
|
||||||
in
|
|
||||||
lib.any (a: !a.assertion && names a) baoGranterOddPkiRole.assertions
|
|
||||||
&& !(lib.any (a: !a.assertion && names a) baoGrantHere.assertions);
|
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# 🩸 The refusal half of the forwarder's own leaf. It renders wherever the
|
# 🩸 The refusal half of the forwarder's own leaf. It renders wherever the
|
||||||
|
|
@ -1063,17 +762,15 @@ let
|
||||||
ok = lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/matrix/appservice-token\" {\n capabilities = [\"read\"]\n}" baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
|
ok = lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/matrix/appservice-token\" {\n capabilities = [\"read\"]\n}" baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# Every role lives under a mount nothing else creates, and the granter
|
# The policy above grants paths under a mount nothing else creates, so
|
||||||
# holds no `sys/auth`, so the token-holding unit creates it — otherwise
|
# the unit that writes the policy has to create it too — otherwise every
|
||||||
# every certificate login fails against a path that is not there.
|
# certificate login fails against a path that is not there.
|
||||||
name = "the granter's role unit creates the cert auth mount, and the controller's unit writes its role";
|
name = "the granting unit creates the cert auth mount and the controller's role";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
|
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
|
||||||
g = baoGrantHere.systemd.services.swarm-bao-granter-role.script;
|
|
||||||
in
|
in
|
||||||
lib.hasInfix "bao auth enable cert" g
|
lib.hasInfix "bao auth enable cert" s
|
||||||
&& !(lib.hasInfix "bao auth enable" s)
|
|
||||||
&& lib.hasInfix "auth/cert/certs/swarm-controller" s
|
&& lib.hasInfix "auth/cert/certs/swarm-controller" s
|
||||||
&& lib.hasInfix "/var/lib/swarm-bao-tls/client-ca.pem" s;
|
&& lib.hasInfix "/var/lib/swarm-bao-tls/client-ca.pem" s;
|
||||||
}
|
}
|
||||||
|
|
@ -1093,6 +790,28 @@ let
|
||||||
in
|
in
|
||||||
lib.hasInfix "bao secrets enable -path=secret kv-v2" s;
|
lib.hasInfix "bao secrets enable -path=secret kv-v2" s;
|
||||||
}
|
}
|
||||||
|
{
|
||||||
|
# The arm that makes the one above mean something. A role's trust anchor
|
||||||
|
# is the CA, so with none named there is nothing to write — and the
|
||||||
|
# policy write, which needs no CA, must survive that.
|
||||||
|
#
|
||||||
|
# ⚠️ Matched on the COMMANDS, not on `auth/cert/certs`: the policy text is
|
||||||
|
# embedded in this same script and grants that very path, so the shorter
|
||||||
|
# infix is present either way and the arm could never fail.
|
||||||
|
name = "with no client CA the unit still writes the policy and skips the role";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
s = baoGrantNoClientCa.systemd.services.swarm-bao-controller-policy.script;
|
||||||
|
in
|
||||||
|
lib.hasInfix "bao policy write" s
|
||||||
|
&& !(lib.hasInfix "bao auth enable cert" s)
|
||||||
|
&& !(lib.hasInfix "client-ca.pem" s)
|
||||||
|
# The KV mount is NOT part of what a missing client CA switches off:
|
||||||
|
# the controller writes through it whether or not anything can log in
|
||||||
|
# by certificate. Asserted here rather than trusted, because both
|
||||||
|
# steps live in the same script and one indentation level decides it.
|
||||||
|
&& lib.hasInfix "bao secrets enable -path=secret kv-v2" s;
|
||||||
|
}
|
||||||
{
|
{
|
||||||
# What makes the granting-unit cases mean something, and the property
|
# What makes the granting-unit cases mean something, and the property
|
||||||
# the host-side half depends on: no store here, so no bind mount and no
|
# the host-side half depends on: no store here, so no bind mount and no
|
||||||
|
|
@ -1102,66 +821,43 @@ let
|
||||||
ok = !(baoGrantNoStore.systemd.services ? swarm-bao-bootstrap-dir);
|
ok = !(baoGrantNoStore.systemd.services ? swarm-bao-bootstrap-dir);
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The operator writes this policy by hand, so a call the token-holding
|
# The drift this case exists to stop: setup.md's copy of the policy
|
||||||
# unit makes and the file does not grant is a one-time step that fails.
|
# stayed at the controller's first six grants while seven more units
|
||||||
# Failing names every ungranted call.
|
# started using the token. Failing names every ungranted call.
|
||||||
name =
|
name =
|
||||||
"every bao call the bootstrap-token unit makes is granted by bao-bootstrap-policy.hcl"
|
"every bao call a bootstrap-token unit makes is granted by swarm-bao-bootstrap-policy.hcl"
|
||||||
+ lib.optionalString (bootstrapUngranted != [ ]) (
|
+ lib.optionalString (bootstrapUngranted != [ ]) (
|
||||||
": " + lib.concatStringsSep "; " bootstrapUngranted
|
": " + lib.concatStringsSep "; " bootstrapUngranted
|
||||||
);
|
);
|
||||||
ok = bootstrapUngranted == [ ];
|
ok = bootstrapUngranted == [ ];
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The same check for the granter: a grant a unit writes outside its
|
# What makes the case above mean something: discovery by token path
|
||||||
# globs is a 403 on deploy. Failing names every ungranted call.
|
# reaches every unit that uses the token today, and each yields calls.
|
||||||
name =
|
name = "the bootstrap-policy check sees all ten units that use the token, and parses calls from each";
|
||||||
"every bao call a granting unit makes is granted by the granter's policy"
|
|
||||||
+ lib.optionalString (granterUngranted != [ ]) (": " + lib.concatStringsSep "; " granterUngranted);
|
|
||||||
ok = granterUngranted == [ ];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# What makes the case above mean something: discovery by the granter's
|
|
||||||
# certificate reaches all ten units, and each yields calls.
|
|
||||||
name = "the granter-policy check sees all ten granting units, and parses calls from each";
|
|
||||||
ok =
|
ok =
|
||||||
lib.sort lib.lessThan (lib.attrNames granterUnits) == lib.sort lib.lessThan grantingUnitNames
|
lib.all (n: bootstrapUnits ? ${n}) [
|
||||||
&& lib.all (u: baoCalls u.script != [ ]) (lib.attrValues granterUnits)
|
"swarm-bao-controller-policy"
|
||||||
|
"swarm-bao-secret-publisher-policy"
|
||||||
|
"swarm-bao-matrix-ctl-policy"
|
||||||
|
"swarm-bao-matrix-token-policy"
|
||||||
|
"swarm-bao-queue-agent-policy"
|
||||||
|
"swarm-bao-grafana-oidc-policy"
|
||||||
|
"swarm-bao-otel-oidc-policy"
|
||||||
|
"swarm-bao-forwarder-oidc-policy"
|
||||||
|
"swarm-bao-services-issuer-policy"
|
||||||
|
"swarm-bao-nats-tls-policy"
|
||||||
|
]
|
||||||
&& lib.all (u: baoCalls u.script != [ ]) (lib.attrValues bootstrapUnits);
|
&& lib.all (u: baoCalls u.script != [ ]) (lib.attrValues bootstrapUnits);
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# And the grants side: a stanza the parser skipped would read as a
|
# And the grants side: a stanza the parser skipped would read as a
|
||||||
# grant that is not there.
|
# grant that is not there.
|
||||||
name = "every path stanza in bao-bootstrap-policy.hcl and the granter's policy parses";
|
name = "every path stanza in swarm-bao-bootstrap-policy.hcl parses";
|
||||||
ok =
|
ok =
|
||||||
lib.all
|
bootstrapGrants != [ ]
|
||||||
(
|
&& lib.length bootstrapGrants == lib.length (matches ''path "'' bootstrapPolicyText)
|
||||||
t:
|
&& lib.all (g: g.caps != [ ]) bootstrapGrants;
|
||||||
let
|
|
||||||
grants = grantsIn t;
|
|
||||||
in
|
|
||||||
grants != [ ]
|
|
||||||
&& lib.length grants == lib.length (matches ''path "'' t)
|
|
||||||
&& lib.all (g: g.caps != [ ]) grants
|
|
||||||
)
|
|
||||||
[
|
|
||||||
bootstrapPolicyText
|
|
||||||
granterPolicyText
|
|
||||||
];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# The bootstrap policy, whole: the auth mounts and the granter's own two
|
|
||||||
# objects, and nothing a `swarm-*` grant lives at.
|
|
||||||
name = "the bootstrap policy is exactly the auth mounts and the granter's policy and role";
|
|
||||||
ok =
|
|
||||||
lib.map (g: g.path) bootstrapGrants == [
|
|
||||||
"sys/auth"
|
|
||||||
"sys/auth/cert"
|
|
||||||
"sys/auth/approle"
|
|
||||||
"sys/policies/acl/bao-granter"
|
|
||||||
"auth/cert/certs/bao-granter"
|
|
||||||
]
|
|
||||||
&& grantFor bootstrapGrants "sys/policies/acl/swarm-controller" == null;
|
|
||||||
}
|
}
|
||||||
];
|
];
|
||||||
in
|
in
|
||||||
|
|
|
||||||
|
|
@ -23,16 +23,18 @@ let
|
||||||
runGroup
|
runGroup
|
||||||
;
|
;
|
||||||
|
|
||||||
# Every service on one host, so the store's granting unit renders the role
|
# Every service on one host, with a bootstrap token so the store's granting
|
||||||
# this leaf is issued through.
|
# unit renders the role this leaf is issued through.
|
||||||
allLocal = hive {
|
allLocal = hive {
|
||||||
deploy.singleHostSwarm = true;
|
deploy.singleHostSwarm = true;
|
||||||
|
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
||||||
};
|
};
|
||||||
|
|
||||||
# The same host with the role's lifetime moved, so a threshold that is a
|
# The same host with the role's lifetime moved, so a threshold that is a
|
||||||
# number of its own shows up as one that did not move with it.
|
# number of its own shows up as one that did not move with it.
|
||||||
shortTtl = hive {
|
shortTtl = hive {
|
||||||
deploy.singleHostSwarm = true;
|
deploy.singleHostSwarm = true;
|
||||||
|
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
||||||
deploy.bao.servicesPkiLeafTtlHours = 48;
|
deploy.bao.servicesPkiLeafTtlHours = 48;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -83,13 +83,6 @@ let
|
||||||
swarm.hives.fwctl.domain = "f.t.local";
|
swarm.hives.fwctl.domain = "f.t.local";
|
||||||
};
|
};
|
||||||
|
|
||||||
# The granter's, the one subject whose role may write every `swarm-*` grant.
|
|
||||||
hiveNamedAfterGranterSubject = hive {
|
|
||||||
deploy.swarm-otel.enable = false;
|
|
||||||
deploy.bao.granterCommonName = "grctl";
|
|
||||||
swarm.hives.grctl.domain = "gr.t.local";
|
|
||||||
};
|
|
||||||
|
|
||||||
# 🩸 A different shape from every fixture above: the matrix-token and
|
# 🩸 A different shape from every fixture above: the matrix-token and
|
||||||
# queue-credential roles are written PER HIVE, so the subject a hive must not
|
# queue-credential roles are written PER HIVE, so the subject a hive must not
|
||||||
# be is `<prefix>-<some hive's name>` rather than the prefix itself. Reserving
|
# be is `<prefix>-<some hive's name>` rather than the prefix itself. Reserving
|
||||||
|
|
@ -180,16 +173,6 @@ let
|
||||||
a: !a.assertion && lib.hasInfix "'fwctl'" a.message
|
a: !a.assertion && lib.hasInfix "'fwctl'" a.message
|
||||||
) hiveNamedAfterForwarderOidcSubject.assertions;
|
) hiveNamedAfterForwarderOidcSubject.assertions;
|
||||||
}
|
}
|
||||||
{
|
|
||||||
# And the granter's, whose role is root-equivalent: a hive holding a leaf
|
|
||||||
# it accepts could grant itself anything.
|
|
||||||
name = "a hive named after the bao granter's subject is refused too";
|
|
||||||
ok =
|
|
||||||
equalityGuardFired hiveNamedAfterGranterSubject
|
|
||||||
&& lib.any (
|
|
||||||
a: !a.assertion && lib.hasInfix "'grctl'" a.message
|
|
||||||
) hiveNamedAfterGranterSubject.assertions;
|
|
||||||
}
|
|
||||||
{
|
{
|
||||||
# 🩸 The per-hive half, and the one a prefix-only reservation would miss:
|
# 🩸 The per-hive half, and the one a prefix-only reservation would miss:
|
||||||
# the role is `<prefix>-<hive>`, so the reserved string has to be composed
|
# the role is `<prefix>-<hive>`, so the reserved string has to be composed
|
||||||
|
|
|
||||||
|
|
@ -26,10 +26,12 @@ let
|
||||||
natsName = "nats.t.local";
|
natsName = "nats.t.local";
|
||||||
natsUrl = "tls://${natsName}:4222";
|
natsUrl = "tls://${natsName}:4222";
|
||||||
|
|
||||||
# Every service on one host. The queue, the store and every in-tree client of the queue
|
# Every service on one host, with a bootstrap token so the store's granting
|
||||||
|
# units render. The queue, the store and every in-tree client of the queue
|
||||||
# are all here, so the scan below reads each of them.
|
# are all here, so the scan below reads each of them.
|
||||||
allLocal = hive {
|
allLocal = hive {
|
||||||
deploy.singleHostSwarm = true;
|
deploy.singleHostSwarm = true;
|
||||||
|
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
||||||
};
|
};
|
||||||
|
|
||||||
# The same host on the mesh.
|
# The same host on the mesh.
|
||||||
|
|
@ -230,8 +232,8 @@ let
|
||||||
&& !(lib.elem 4222 allLocal.networking.firewall.allowedTCPPorts);
|
&& !(lib.elem 4222 allLocal.networking.firewall.allowedTCPPorts);
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# Ordering, never a requirement: a policy unit that failed still counts
|
# Ordering, never a requirement: the policy unit skips once the bootstrap
|
||||||
# as done, and the leaf unit's own retries carry it past that.
|
# token is gone, and a skipped unit counts as done.
|
||||||
name = "the leaf unit is ordered after its policy unit, with no requires";
|
name = "the leaf unit is ordered after its policy unit, with no requires";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
|
|
|
||||||
|
|
@ -80,7 +80,6 @@ let
|
||||||
"hive-tls-ca"
|
"hive-tls-ca"
|
||||||
"swarm-services-cert"
|
"swarm-services-cert"
|
||||||
"hive-gateway-self-signed-cert"
|
"hive-gateway-self-signed-cert"
|
||||||
"swarm-bao-granter-role"
|
|
||||||
"swarm-bao-controller-policy"
|
"swarm-bao-controller-policy"
|
||||||
"swarm-bao-secret-publisher-policy"
|
"swarm-bao-secret-publisher-policy"
|
||||||
"swarm-bao-matrix-ctl-policy"
|
"swarm-bao-matrix-ctl-policy"
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue