swarm-bao: give each hive-cert consumer its own bao identity
Four units read one path each out of the store, and all four logged in holding `deploy.bao.clientCertFile` — the hive's own leaf. Bao identifies a principal by the subject of the certificate it presents, so four readers behind one certificate were ONE principal, and the only grant expressible was the union of what the four need: read on `swarm/agents/*`, `swarm/hives/<hive>/*` and `swarm/services/*`. The unit fetching Grafana's OIDC client secret could fetch every agent credential in the swarm; the one fetching this hive's matrix token could fetch Grafana's. Least privilege was not misconfigured here, it was unrepresentable. Each now holds a leaf, a cert-auth role and a policy of its own, and each policy is the single `secret/data/…` path that unit's own script names — spelled to the leaf, not to a prefix, the way matrix-ctl's already is. Following the four exemplars in-tree rather than building a mechanism: `signLeaf` mints the leaves, `swarm-bao.nix` writes the roles from the bootstrap token, the consumers name their own pair. Two of the four are written PER HIVE and two are not, which is the shape of the paths rather than a preference. A matrix appservice token and a queue credential live under `swarm/hives/<name>/` and every hive runs a reader for its own, so one role for all of them would have to be granted `hives/*` — letting one hive read another's, a reach no hive has today. An OIDC client secret lives under `swarm/services/<client-id>/` and a swarm registers each exactly once, so one role each is enough. The per-hive subjects are `<prefix>-<hive>` and swarm.nix reserves every composed spelling as a hive name, so a hive cannot be named into another hive's role. The shared leaf stays: hive-c0re still passes it into its container, the `bao` CLI wrapper still defaults to it, and the three `glue-*-bao-identity.nix` files derive the PKI directory from it. module-eval-bao-grants gains a negative arm per principal — each pins the three stanzas the hive's leaf carried and the two wildcards a later widening would reach for, so a policy that grows fails here rather than in a store. Plus the consuming side: repointing a unit back at the hive's leaf would evaluate, deploy and log in, and silently restore the union. A hive that reads a store on another machine now places one leaf per principal instead of one shared by four. That cost is the point, and docs/swarm/secrets.md lists the pairs.
This commit is contained in:
atlas
mara
parent
ded5b08258
commit
f1445b4c8b
14 changed files with 859 additions and 43 deletions
|
|
@ -340,6 +340,156 @@ let
|
|||
}
|
||||
'';
|
||||
|
||||
# ── the four principals that used to share the hive's own leaf ─────────────
|
||||
#
|
||||
# 🩸 Each of the four reads exactly ONE path in the store, and until this
|
||||
# split each did it holding `deploy.bao.clientCertFile` — the hive's own leaf,
|
||||
# whose policy (`swarm-secret-client`'s `policy::render`) grants read on
|
||||
# `swarm/agents/*`, `swarm/hives/<hive>/*` AND `swarm/services/*`. Four
|
||||
# principals presenting one certificate are ONE principal to bao, so the union
|
||||
# of what the four need was the only grant expressible: the unit that fetches
|
||||
# Grafana's OIDC client secret could fetch every agent credential in the
|
||||
# swarm, and the unit that fetches this hive's matrix token could fetch
|
||||
# Grafana's. Least privilege was not misconfigured, it was unrepresentable.
|
||||
#
|
||||
# Each now gets its own leaf, its own cert-auth role and a policy holding the
|
||||
# one `secret/data/…` path that principal's script actually names — spelled to
|
||||
# the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText` above
|
||||
# gives for doing the same.
|
||||
#
|
||||
# ⚠️ Two of the four are PER HIVE and two are not, and that asymmetry is the
|
||||
# shape of the paths rather than a preference. A matrix appservice token and a
|
||||
# queue credential live under `swarm/hives/<name>/`, and every hive runs a
|
||||
# reader for its own; an OIDC client secret lives under
|
||||
# `swarm/services/<client-id>/` and a swarm registers each of those exactly
|
||||
# once. A single role for a per-hive reader would have to name `hives/*` to
|
||||
# serve every hive — which would let one hive read another's matrix token, a
|
||||
# reach no hive has today and one this change exists to remove rather than
|
||||
# create.
|
||||
readStanza = path: ''
|
||||
path "${path}" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
'';
|
||||
|
||||
# One reader per hive in the swarm directory. Written from the directory
|
||||
# rather than from this host's own name because the policy is written where
|
||||
# the STORE is and the reader runs where its hive is — the same split
|
||||
# `matrixCtlHiveName` above exists to paper over, answered here by naming
|
||||
# every hive instead of asking the operator which one.
|
||||
perHiveReaders =
|
||||
{ rolePrefix, cnPrefix, ... }@spec:
|
||||
lib.mapAttrsToList (hiveName: _: {
|
||||
name = "${rolePrefix}-${hiveName}";
|
||||
cn = "${cnPrefix}-${hiveName}";
|
||||
policyText = readStanza (spec.path hiveName);
|
||||
}) hyperhiveCfg.swarm.hives;
|
||||
|
||||
matrixTokenReaders = perHiveReaders {
|
||||
rolePrefix = "swarm-matrix-token";
|
||||
cnPrefix = baoDeploy.matrixTokenCommonNamePrefix;
|
||||
path = h: "${credentialMountPath}/data/swarm/hives/${h}/matrix/appservice-token";
|
||||
};
|
||||
|
||||
queueAgentReaders = perHiveReaders {
|
||||
rolePrefix = "swarm-queue-agent";
|
||||
cnPrefix = baoDeploy.queueAgentCommonNamePrefix;
|
||||
path = h: "${credentialMountPath}/data/swarm/hives/${h}/queue/agent";
|
||||
};
|
||||
|
||||
# Singletons, so a list of one rather than a second shape: the granting unit
|
||||
# below takes a list either way and the four units stay identical apart from
|
||||
# the objects they write.
|
||||
grafanaOidcReaders = [
|
||||
{
|
||||
name = "swarm-grafana-oidc";
|
||||
cn = baoDeploy.grafanaOidcCommonName;
|
||||
policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client";
|
||||
}
|
||||
];
|
||||
|
||||
otelOidcReaders = [
|
||||
{
|
||||
name = "swarm-otel-oidc";
|
||||
cn = baoDeploy.otelOidcCommonName;
|
||||
policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.otel.clientId}/oidc/client";
|
||||
}
|
||||
];
|
||||
|
||||
# The role name IS the policy name, as for the three service principals
|
||||
# above: the role attaches the policy by spelling it identically, and one
|
||||
# string for both objects removes the way they drift apart.
|
||||
readerPolicyWrite = obj: ''
|
||||
printf '%s' ${lib.escapeShellArg obj.policyText} |
|
||||
bao policy write ${lib.escapeShellArg obj.name} -
|
||||
'';
|
||||
|
||||
# ⚠️ Names are `swarm-<principal>[-<hive>]`, outside the `hive-*` namespace the
|
||||
# controller may rewrite, for the reason the three service principals above
|
||||
# state. A per-hive CN cannot collide with a hive's own leaf either — a hive's
|
||||
# CN is its bare name and ../reserved-hive-fragments.nix forbids the substring
|
||||
# `swarm` in one — and ./swarm.nix reserves the concrete per-hive spellings
|
||||
# anyway, because the prefixes are operator-settable and an operator may spell
|
||||
# one without that substring.
|
||||
readerRoleWrite = obj: ''
|
||||
bao write auth/cert/certs/${lib.escapeShellArg obj.name} \
|
||||
certificate=@${tlsDir}/client-ca.pem \
|
||||
allowed_common_names=${lib.escapeShellArg obj.cn} \
|
||||
token_policies=${lib.escapeShellArg obj.name} \
|
||||
display_name=${lib.escapeShellArg obj.cn}
|
||||
'';
|
||||
|
||||
# One unit per PRINCIPAL, not one unit for the four: these names are
|
||||
# operator-facing strings, the same reason `swarm-bao-secret-publisher-policy`
|
||||
# above gives for not being more script inside the controller's unit. A hive
|
||||
# whose matrix token never arrives is diagnosed by the state of the unit named
|
||||
# after it.
|
||||
#
|
||||
# `after` and not `requires`, for the reason the publisher's unit states: the
|
||||
# controller's unit creates the KV and cert-auth mounts this one writes into,
|
||||
# but a failed oneshot still counts as finished, so ordering plus this unit's
|
||||
# own retry is what converges.
|
||||
#
|
||||
# The role write is inside the client-CA branch and the policy write is not,
|
||||
# exactly as the three above: with no CA there is no trust anchor for a login
|
||||
# role, but the policy it would attach is still worth asserting.
|
||||
readerPolicyUnit =
|
||||
description: objects:
|
||||
lib.mkIf haveBootstrapToken {
|
||||
inherit description;
|
||||
after = [
|
||||
"container@${cfg.machine}.service"
|
||||
"swarm-bao-controller-policy.service"
|
||||
];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
path = [
|
||||
baoCli
|
||||
pkgs.coreutils
|
||||
];
|
||||
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
|
||||
# Same unseal wait as its siblings above, for the reason stated there:
|
||||
# under `seal = "shamir"` a human unseals by hand.
|
||||
startLimitBurst = 2880;
|
||||
startLimitIntervalSec = 90000;
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
Restart = "on-failure";
|
||||
RestartSec = 30;
|
||||
};
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
|
||||
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
|
||||
export BAO_TOKEN
|
||||
|
||||
''
|
||||
+ lib.concatMapStrings readerPolicyWrite objects
|
||||
+ lib.optionalString (baoDeploy.clientCaFile != null) (
|
||||
"\n" + lib.concatMapStrings readerRoleWrite objects
|
||||
);
|
||||
};
|
||||
|
||||
# Every listener serves the same identity: they differ in which address
|
||||
# they answer on, not in who they are. Client verification is separate and
|
||||
# optional — a store with no `clientCaFile` still serves TLS, it just does
|
||||
|
|
@ -846,6 +996,185 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
matrixTokenCommonNamePrefix = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "swarm-bao-matrix-token";
|
||||
example = "swarm-bao-matrix-token.svc";
|
||||
description = ''
|
||||
Prefix of the subject the store's per-hive matrix-token roles accept.
|
||||
The role for hive `h` accepts `<prefix>-h` and nothing else, and grants
|
||||
read on exactly `swarm/hives/h/matrix/appservice-token`.
|
||||
|
||||
A **prefix** rather than one subject because every hive runs its own
|
||||
reader and the path it reads carries that hive's name. One subject for
|
||||
all of them would need a `hives/*` grant, which would let one hive read
|
||||
another hive's appservice token — a reach no hive has today.
|
||||
|
||||
⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its
|
||||
fixed-subject siblings are: cert auth trusts the CA and
|
||||
`allowed_common_names` is the whole narrowing, so a hive whose own leaf
|
||||
carried one of these spellings would receive that grant.
|
||||
'';
|
||||
};
|
||||
|
||||
queueAgentCommonNamePrefix = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "swarm-bao-queue-agent";
|
||||
example = "swarm-bao-queue-agent.svc";
|
||||
description = ''
|
||||
Prefix of the subject the store's per-hive queue-credential roles
|
||||
accept. The role for hive `h` accepts `<prefix>-h` and grants read on
|
||||
exactly `swarm/hives/h/queue/agent` — the client credential that hive's
|
||||
agent containers authenticate to the swarm queue with.
|
||||
|
||||
Per hive for the same reason
|
||||
{option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is,
|
||||
and more sharply: every hive in the swarm runs this reader, so a single
|
||||
subject would have to be granted `hives/*`.
|
||||
'';
|
||||
};
|
||||
|
||||
grafanaOidcCommonName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "swarm-bao-grafana-oidc";
|
||||
example = "swarm-bao-grafana-oidc.svc";
|
||||
description = ''
|
||||
Subject the store's `swarm-grafana-oidc` cert-auth role accepts — the
|
||||
identity the unit that fetches Grafana's OIDC client secret presents.
|
||||
|
||||
Its grant is one path,
|
||||
`swarm/services/<grafana client id>/oidc/client`, and read only. Not
|
||||
per hive, unlike the two prefixes above: an OIDC client is registered
|
||||
once per swarm, so the path names the service and never a hive.
|
||||
|
||||
⚠️ Same collision as its siblings, and the same answer: ./swarm.nix
|
||||
feeds this value into the guard on
|
||||
{option}`services.hyperhive.swarm.hives`.
|
||||
'';
|
||||
};
|
||||
|
||||
otelOidcCommonName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "swarm-bao-otel-oidc";
|
||||
example = "swarm-bao-otel-oidc.svc";
|
||||
description = ''
|
||||
Subject the store's `swarm-otel-oidc` cert-auth role accepts — the
|
||||
identity the unit that fetches the collector's OIDC client secret
|
||||
presents. Its grant is one path,
|
||||
`swarm/services/<collector client id>/oidc/client`, and read only.
|
||||
|
||||
A **fourth** identity rather than reuse of
|
||||
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`: the two
|
||||
read different services' client secrets, and a collector is not
|
||||
entitled to Grafana's.
|
||||
'';
|
||||
};
|
||||
|
||||
matrixTokenClientCertFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/matrix-token.pem";
|
||||
description = ''
|
||||
Certificate the unit that fetches this hive's matrix appservice token
|
||||
presents to the store. Its subject must be
|
||||
`<matrixTokenCommonNamePrefix>-<this hive's name>`; cert auth matches on
|
||||
the CN and the role accepts nothing else.
|
||||
|
||||
⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
|
||||
Pointing this at the hive's own leaf would evaluate, deploy and log in —
|
||||
and hand a unit that reads one token a credential that reads every
|
||||
secret in the store. That is what this option exists to end.
|
||||
|
||||
No default here: ./glue-bao-tls.nix points it at the leaf it mints,
|
||||
wherever this host mints its own PKI. A deployment that reads a store on
|
||||
another host names it by hand, alongside the leaf itself — one file per
|
||||
principal placed out of band instead of one shared by four.
|
||||
'';
|
||||
};
|
||||
|
||||
matrixTokenClientKeyFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/matrix-token-key.pem";
|
||||
description = ''
|
||||
Private key for
|
||||
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile`. Both
|
||||
or neither — a certificate with no key authenticates nothing.
|
||||
'';
|
||||
};
|
||||
|
||||
queueAgentClientCertFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/queue-agent.pem";
|
||||
description = ''
|
||||
Certificate the unit that fetches this hive's queue client credential
|
||||
presents to the store. Its subject must be
|
||||
`<queueAgentCommonNamePrefix>-<this hive's name>`.
|
||||
|
||||
⚠️ Not the hive's own leaf, for the reason
|
||||
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives.
|
||||
'';
|
||||
};
|
||||
|
||||
queueAgentClientKeyFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/queue-agent-key.pem";
|
||||
description = ''
|
||||
Private key for
|
||||
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile`.
|
||||
'';
|
||||
};
|
||||
|
||||
grafanaOidcClientCertFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/grafana-oidc.pem";
|
||||
description = ''
|
||||
Certificate the unit that fetches Grafana's OIDC client secret presents
|
||||
to the store. Its subject must be
|
||||
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`.
|
||||
|
||||
⚠️ Not the hive's own leaf, for the reason
|
||||
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives.
|
||||
'';
|
||||
};
|
||||
|
||||
grafanaOidcClientKeyFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/grafana-oidc-key.pem";
|
||||
description = ''
|
||||
Private key for
|
||||
{option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile`.
|
||||
'';
|
||||
};
|
||||
|
||||
otelOidcClientCertFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/otel-oidc.pem";
|
||||
description = ''
|
||||
Certificate the unit that fetches the collector's OIDC client secret
|
||||
presents to the store. Its subject must be
|
||||
{option}`services.hyperhive.deploy.bao.otelOidcCommonName`.
|
||||
|
||||
⚠️ Not the hive's own leaf, for the reason
|
||||
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives.
|
||||
'';
|
||||
};
|
||||
|
||||
otelOidcClientKeyFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/otel-oidc-key.pem";
|
||||
description = ''
|
||||
Private key for
|
||||
{option}`services.hyperhive.deploy.bao.otelOidcClientCertFile`.
|
||||
'';
|
||||
};
|
||||
|
||||
serverCaFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
|
|
@ -1593,6 +1922,19 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
# 🩸 The four principals that used to log in as the hive. See the block
|
||||
# around `perHiveReaders` for what each grant is and why the first two are
|
||||
# per hive; the four units are identical apart from the objects they
|
||||
# write, so the shape lives in `readerPolicyUnit` and the difference is
|
||||
# visible here in one line each.
|
||||
systemd.services.swarm-bao-matrix-token-policy = readerPolicyUnit "write the per-hive matrix-token bao policies and cert-auth roles" matrixTokenReaders;
|
||||
|
||||
systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders;
|
||||
|
||||
systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders;
|
||||
|
||||
systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders;
|
||||
|
||||
# The CA bind source is written at runtime by a host unit, so the
|
||||
# container has to start after it — otherwise nspawn sets up a mount
|
||||
# over a file that does not exist yet.
|
||||
|
|
|
|||
Loading…
Reference in a new issue