swarm-bao: agent certificates issued by a store-generated agent CA
An agent's store identity was signed in swarm-controller's memory by a CA a controller-host unit generated on disk, and the listener never trusted that CA. Agent leaves now come from the store itself: a `pki-agents` PKI mount whose root openbao generates internally, so the agent CA's key never exists outside the store. - swarm-bao-agent-pki (new, store host, as the bao granter): enables and tunes the mount, generates the root once (guarded on an empty issuer list, no replace branch), upserts the `swarm-agent` role (client certificates named `hive-agent-*` only, 90 days), caches the CA at /var/lib/swarm-bao-tls/agent-ca.pem and composes the listener bundle. - The listener's tls_client_ca_file is a new listener-client-ca.pem (client-ca.pem, then the agent CA). Host cert-auth roles still pin client-ca.pem, so an agent leaf satisfies no host role. swarm-bao-certs composes the same bundle before openbao starts. - openbao reads tls_client_ca_file only at start, so when the bundle changed after openbao started, swarm-bao-agent-pki restarts openbao.service in the container; under `seal = "shamir"` it prints the step instead. Once swarm-bao-certs has a cached CA, later boots start openbao with it and do not restart. - The controller policy gains exactly `update` on pki-agents/issue/swarm-agent. mint_and_verify now asks that role for the leaf (the store generates the key), writes the agent's cert-auth role pinning the issuing CA bao returned, and writes the agent's policy as render_agent alone: the hive-shared queue credential stanza is gone. - deploy.bao.agentPkiRoleName (must start `swarm-`, asserted with the other pki role names); swarm-controller gets SWARM_CONTROLLER_AGENT_PKI_MOUNT/_ROLE from the deploy.bao options. Deleted: swarm-controller-agent-ca and its options (agentCaFile, agentCaKeyFile), env, LoadCredential entries and assertion; agent_identity's Authority, rcgen signing and validity window; the rcgen and time dependencies of swarm-controller (rcgen leaves the workspace); policy::render_agent_with_queue and its tests. The CN-prefix assertion policy.rs said was owed is not: agent and host roles pin different CAs. Migration is re-creating each agent after deploy; that overwrites the stale role and policy. Closes #4756
This commit is contained in:
atlas
parent
5cd7f866f4
commit
6170e74a31
16 changed files with 894 additions and 925 deletions
|
|
@ -356,6 +356,9 @@ let
|
|||
# The swarm appservice token, read-only: the controller creates agents'
|
||||
# matrix accounts with it and never writes it. matrix-ctl mints and publishes
|
||||
# it (`matrixCtlPolicyText` below).
|
||||
#
|
||||
# The agent PKI grant is its only path on that mount: it can ask the one role
|
||||
# for a certificate, not write that role or reach the issuer.
|
||||
controllerPolicyText = ''
|
||||
path "auth/cert/certs/hive-*" {
|
||||
capabilities = ["create", "update", "read", "delete"]
|
||||
|
|
@ -380,6 +383,10 @@ let
|
|||
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
|
||||
path "${agentPkiMountPath}/issue/${agentPkiRoleName}" {
|
||||
capabilities = ["update"]
|
||||
}
|
||||
'';
|
||||
|
||||
# The identity that copies authelia's minted OIDC client secrets into the
|
||||
|
|
@ -489,7 +496,9 @@ let
|
|||
#
|
||||
# ⚠️ NOT bao's own client-auth PKI. That one is ./glue-bao-tls.nix's
|
||||
# self-signed CA under `/var/lib/swarm-bao-pki`, and it stays outside the
|
||||
# store permanently: bao cannot issue the credential that opens bao.
|
||||
# store permanently: bao cannot issue the credential a host opens bao with.
|
||||
# Agent leaves come from `agentPkiMountPath`, requested by a principal that
|
||||
# has already logged in.
|
||||
#
|
||||
# The mount's own root is generated into it by the bootstrap unit below and
|
||||
# its private key never leaves — `root/generate/internal` keeps it inside
|
||||
|
|
@ -500,13 +509,6 @@ let
|
|||
# and has to spell it the same way.
|
||||
servicesPkiMountPath = baoDeploy.servicesPkiMountPath;
|
||||
|
||||
# The PKI mount agent client certificates are issued from. Its root is
|
||||
# generated inside the store, so the agent CA's key never exists outside it.
|
||||
# A mount of its own because the services mount holds exactly one issuer; a
|
||||
# root apart from ./glue-bao-tls.nix's CA because that is what keeps an
|
||||
# agent's certificate from satisfying any host role.
|
||||
agentPkiMountPath = baoDeploy.agentPkiMountPath;
|
||||
|
||||
# Subject of the root generated into that mount. A label for a human reading
|
||||
# a chain, not an identity anything authenticates against — same fall-through
|
||||
# ./swarm-ca.nix:29-37 uses, and for the same reason: a hive that has set
|
||||
|
|
@ -594,6 +596,77 @@ let
|
|||
}
|
||||
'';
|
||||
|
||||
# The PKI mount agent client certificates are issued from. Its root is
|
||||
# generated inside the store, so the agent CA's key never exists outside it.
|
||||
# A mount of its own because the services mount holds exactly one issuer; a
|
||||
# root apart from ./glue-bao-tls.nix's CA because that is what keeps an
|
||||
# agent's certificate from satisfying any host role.
|
||||
agentPkiMountPath = baoDeploy.agentPkiMountPath;
|
||||
agentPkiRoleName = baoDeploy.agentPkiRoleName;
|
||||
|
||||
# Every common name the agent role issues for: `swarm_secret_client`'s
|
||||
# `policy::AGENT_PREFIX`, which names each agent's cert-auth role and
|
||||
# policy too. Outside it the role refuses, so the controller cannot obtain
|
||||
# a certificate for any other name.
|
||||
agentCnGlob = "hive-agent-*";
|
||||
|
||||
# The anchor every agent's cert-auth role pins by value, so it is never
|
||||
# replaced by a deploy. 262800h like `servicesPkiRootTtl`, and for the same
|
||||
# reason: the mount is tuned to it before generation, or bao clamps it.
|
||||
agentPkiRootTtl = "262800h";
|
||||
|
||||
# The agent leaf's window, pinned on the role. Nothing re-issues a leaf
|
||||
# before it ends; an operator re-runs agent creation.
|
||||
agentPkiLeafTtl = "2160h";
|
||||
|
||||
# The agent CA's certificate, cached on this host by `swarm-bao-agent-pki`,
|
||||
# so `swarm-bao-certs` can compose the listener's bundle before the store
|
||||
# is up. Public material.
|
||||
agentCaCachePath = "${tlsDir}/agent-ca.pem";
|
||||
|
||||
# What the listener verifies client certificates against: `client-ca.pem`
|
||||
# first, then the agent CA. A file of its own because every host cert-auth
|
||||
# role pins `client-ca.pem`: with the agent CA in that file, every leaf the
|
||||
# controller can request would satisfy every host role.
|
||||
listenerClientCaPath = "${tlsDir}/listener-client-ca.pem";
|
||||
|
||||
# Writes `listenerClientCaPath` from `clientCaPath` and, when it parses, the
|
||||
# cached agent CA. The file is replaced only when its bytes change, so its
|
||||
# mtime says when the listener's trust last changed; `swarm-bao-agent-pki`
|
||||
# restarts openbao on exactly that. Refuses, leaving the current file, when
|
||||
# the result would not begin with `client-ca.pem`: host logins depend on it.
|
||||
composeListenerBundle = ''
|
||||
compose_listener_bundle() {
|
||||
if [ ! -s ${clientCaPath} ]; then
|
||||
echo "${clientCaPath} is missing or empty; not composing the listener bundle" >&2
|
||||
return 1
|
||||
fi
|
||||
local tmp
|
||||
tmp="$(mktemp -p ${tlsDir} .listener-client-ca.XXXXXX)"
|
||||
cat ${clientCaPath} > "$tmp"
|
||||
if [ -s ${agentCaCachePath} ]; then
|
||||
if openssl x509 -noout -in ${agentCaCachePath} >/dev/null 2>&1; then
|
||||
printf '\n' >> "$tmp"
|
||||
cat ${agentCaCachePath} >> "$tmp"
|
||||
else
|
||||
echo "${agentCaCachePath} does not parse; the listener will not trust agent certificates" >&2
|
||||
fi
|
||||
fi
|
||||
if ! cmp -s -n "$(stat -c %s ${clientCaPath})" ${clientCaPath} "$tmp"; then
|
||||
rm -f "$tmp"
|
||||
echo "the composed listener bundle does not begin with ${clientCaPath}; leaving the current one" >&2
|
||||
return 1
|
||||
fi
|
||||
if cmp -s "$tmp" ${listenerClientCaPath}; then
|
||||
rm -f "$tmp"
|
||||
else
|
||||
chmod 0644 "$tmp"
|
||||
mv -f "$tmp" ${listenerClientCaPath}
|
||||
echo "wrote ${listenerClientCaPath}"
|
||||
fi
|
||||
}
|
||||
'';
|
||||
|
||||
# ── the four principals that used to share the hive's own leaf ─────────────
|
||||
#
|
||||
# 🩸 Each of the four reads exactly ONE path in the store, and until this
|
||||
|
|
@ -760,7 +833,7 @@ let
|
|||
tls_key_file = serverKeyCredentialPath;
|
||||
}
|
||||
// lib.optionalAttrs (baoDeploy.clientCaFile != null) {
|
||||
tls_client_ca_file = clientCaPath;
|
||||
tls_client_ca_file = listenerClientCaPath;
|
||||
tls_require_and_verify_client_cert = true;
|
||||
};
|
||||
|
||||
|
|
@ -1251,7 +1324,9 @@ in
|
|||
|
||||
⚠️ NOT the store's own client-auth PKI. That one is
|
||||
./glue-bao-tls.nix's self-signed CA on disk, and it stays outside the
|
||||
store permanently — bao cannot issue the credential that opens bao.
|
||||
store permanently — bao cannot issue the credential a host opens bao
|
||||
with. Agent certificates come from
|
||||
{option}`services.hyperhive.deploy.bao.agentPkiMountPath`.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -1269,6 +1344,17 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
agentPkiRoleName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "swarm-agent";
|
||||
description = ''
|
||||
Role on {option}`services.hyperhive.deploy.bao.agentPkiMountPath`
|
||||
agent client certificates are issued through. It issues client
|
||||
certificates named `hive-agent-*` and nothing else, and swarm-controller
|
||||
may call its `issue` endpoint and no other path on the mount.
|
||||
'';
|
||||
};
|
||||
|
||||
servicesPkiRoleName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "swarm-services";
|
||||
|
|
@ -1859,13 +1945,18 @@ in
|
|||
# else, so a role named otherwise is a 403 at deploy time.
|
||||
assertion =
|
||||
!haveGranter
|
||||
|| (lib.hasPrefix "swarm-" servicesPkiRoleName && lib.hasPrefix "swarm-" natsPkiRoleName);
|
||||
|| (
|
||||
lib.hasPrefix "swarm-" servicesPkiRoleName
|
||||
&& lib.hasPrefix "swarm-" natsPkiRoleName
|
||||
&& lib.hasPrefix "swarm-" agentPkiRoleName
|
||||
);
|
||||
message = ''
|
||||
services.hyperhive.deploy.bao.servicesPkiRoleName
|
||||
(${servicesPkiRoleName}) and
|
||||
(${servicesPkiRoleName}),
|
||||
services.hyperhive.deploy.bao.natsPkiRoleName (${natsPkiRoleName})
|
||||
must both start with `swarm-`: the bao granter that writes them may
|
||||
write pki roles under that prefix only.
|
||||
and services.hyperhive.deploy.bao.agentPkiRoleName
|
||||
(${agentPkiRoleName}) must all start with `swarm-`: the bao granter
|
||||
that writes them may write pki roles under that prefix only.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
|
@ -1929,6 +2020,7 @@ in
|
|||
"swarm-bao-forwarder-oidc-policy"
|
||||
"swarm-bao-services-issuer-policy"
|
||||
"swarm-bao-nats-tls-policy"
|
||||
"swarm-bao-agent-pki"
|
||||
];
|
||||
|
||||
# 🚫 No `swarm.otel.scrapeTargets.bao` entry any more, and its absence is
|
||||
|
|
@ -2119,7 +2211,11 @@ in
|
|||
description = "deliver the swarm secret store's server certificate";
|
||||
before = [ "container@${cfg.machine}.service" ];
|
||||
requiredBy = [ "container@${cfg.machine}.service" ];
|
||||
path = [ pkgs.coreutils ];
|
||||
path = [
|
||||
pkgs.coreutils
|
||||
pkgs.diffutils
|
||||
pkgs.openssl
|
||||
];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
|
|
@ -2155,6 +2251,12 @@ in
|
|||
exit 1
|
||||
fi
|
||||
install -m 0644 ${lib.escapeShellArg baoDeploy.clientCaFile} ${tlsDir}/client-ca.pem
|
||||
|
||||
# Composed here as well as by `swarm-bao-agent-pki` so that openbao
|
||||
# starts already trusting the cached agent CA, and so the file it
|
||||
# refuses to start without exists before the store's first run.
|
||||
${composeListenerBundle}
|
||||
compose_listener_bundle
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -2734,6 +2836,170 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
# The agent PKI mount: its root, its one role, the host's copy of the
|
||||
# root's certificate, and the listener's trust in it.
|
||||
#
|
||||
# ⚠️ This unit RESTARTS openbao. openbao reads `tls_client_ca_file` only
|
||||
# when a listener is built, at start: neither SIGHUP nor a reload re-reads
|
||||
# it. So when the listener bundle changed after openbao last started,
|
||||
# openbao is restarted here, which drops every client for the restart and
|
||||
# the unseal. `swarm-bao-certs` composes the same bundle before every
|
||||
# later start, so this happens when the agent CA first appears (or is
|
||||
# replaced by a re-initialised store), not per boot. Under `shamir` a
|
||||
# restart needs a human unseal, so there it prints the step instead.
|
||||
#
|
||||
# `after` the granting units so a restart does not cut their writes off.
|
||||
systemd.services.swarm-bao-agent-pki = lib.mkIf haveGranter {
|
||||
description = "set up the swarm agent PKI mount and make the store's listener trust it";
|
||||
after = [
|
||||
"container@${cfg.machine}.service"
|
||||
"swarm-bao-controller-policy.service"
|
||||
"swarm-bao-secret-publisher-policy.service"
|
||||
"swarm-bao-matrix-ctl-policy.service"
|
||||
"swarm-bao-matrix-token-policy.service"
|
||||
"swarm-bao-queue-agent-policy.service"
|
||||
"swarm-bao-grafana-oidc-policy.service"
|
||||
"swarm-bao-otel-oidc-policy.service"
|
||||
"swarm-bao-forwarder-oidc-policy.service"
|
||||
"swarm-bao-services-issuer-policy.service"
|
||||
"swarm-bao-nats-tls-policy.service"
|
||||
]
|
||||
++ granterAfter;
|
||||
requires = [ "swarm-bao-pki.service" ];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
path = [
|
||||
baoCli
|
||||
pkgs.coreutils
|
||||
pkgs.diffutils
|
||||
pkgs.openssl
|
||||
];
|
||||
environment = granterEnv;
|
||||
# Same unseal wait as its siblings above.
|
||||
startLimitBurst = 2880;
|
||||
startLimitIntervalSec = 90000;
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
Restart = "on-failure";
|
||||
RestartSec = 30;
|
||||
};
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
|
||||
${granterLogin}
|
||||
|
||||
# Asked rather than attempted: `secrets enable` errors on a path
|
||||
# already in use.
|
||||
mounts="$(bao secrets list -format=json)"
|
||||
case "$mounts" in
|
||||
*'"${agentPkiMountPath}/"'*) ;;
|
||||
*) bao secrets enable -path=${agentPkiMountPath} pki ;;
|
||||
esac
|
||||
|
||||
# Before generation, on every run: an untuned mount silently clamps
|
||||
# the root to 768h (see the services mount above).
|
||||
bao secrets tune -max-lease-ttl=${agentPkiRootTtl} ${agentPkiMountPath}
|
||||
|
||||
# Generated once, ever. Every agent's cert-auth role pins this root
|
||||
# by value, so a second one locks every agent out; there is no
|
||||
# replace branch, and the granter holds no delete on the root. The
|
||||
# mount's own issuer list is the guard, for the reason the services
|
||||
# root gives: `{}` is the only answer that means "none".
|
||||
if issuers="$(bao list -format=json ${lib.escapeShellArg "${agentPkiMountPath}/issuers"})"; then
|
||||
echo "the ${agentPkiMountPath} mount already has an issuer — leaving it alone"
|
||||
elif [ "$issuers" = '{}' ]; then
|
||||
echo "generating the swarm agent CA into the ${agentPkiMountPath} mount"
|
||||
# `max_path_length=0`: this CA signs leaves only.
|
||||
bao write -field=issuing_ca ${lib.escapeShellArg "${agentPkiMountPath}/root/generate/internal"} \
|
||||
common_name=${lib.escapeShellArg "swarm-bao-agent-ca ${servicesPkiRootLabel}"} \
|
||||
issuer_name=swarm-agent-ca \
|
||||
ttl=${agentPkiRootTtl} \
|
||||
max_path_length=0 \
|
||||
key_type=rsa \
|
||||
key_bits=4096 >/dev/null
|
||||
else
|
||||
echo "could not list the ${agentPkiMountPath} issuers — not generating a root over one that may exist" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Upserted every run. The whole narrowing of what the controller can
|
||||
# obtain: client certificates, named `hive-agent-*`, nothing else.
|
||||
# `allow_localhost` and `server_flag` are on by default in bao, so
|
||||
# they are turned off here, not left out. No `issuer_ref`: the mount
|
||||
# holds one issuer, which is its default.
|
||||
bao write ${lib.escapeShellArg "${agentPkiMountPath}/roles/${agentPkiRoleName}"} \
|
||||
allowed_domains=${lib.escapeShellArg agentCnGlob} \
|
||||
allow_glob_domains=true \
|
||||
allow_bare_domains=false \
|
||||
allow_subdomains=false \
|
||||
allow_wildcard_certificates=false \
|
||||
allow_localhost=false \
|
||||
allow_any_name=false \
|
||||
allow_ip_sans=false \
|
||||
enforce_hostnames=true \
|
||||
server_flag=false \
|
||||
client_flag=true \
|
||||
code_signing_flag=false \
|
||||
email_protection_flag=false \
|
||||
key_usage=DigitalSignature \
|
||||
key_type=ec \
|
||||
key_bits=256 \
|
||||
ttl=${agentPkiLeafTtl} \
|
||||
max_ttl=${agentPkiLeafTtl}
|
||||
|
||||
ca="$(bao read -field=certificate ${lib.escapeShellArg "${agentPkiMountPath}/cert/ca"})"
|
||||
if ! openssl x509 -noout <<<"$ca" >/dev/null 2>&1; then
|
||||
echo "the ${agentPkiMountPath} CA that bao returned does not parse — not caching it" >&2
|
||||
exit 1
|
||||
fi
|
||||
tmp="$(mktemp -p ${tlsDir} .agent-ca.XXXXXX)"
|
||||
printf '%s\n' "$ca" > "$tmp"
|
||||
if cmp -s "$tmp" ${agentCaCachePath}; then
|
||||
rm -f "$tmp"
|
||||
else
|
||||
chmod 0644 "$tmp"
|
||||
mv -f "$tmp" ${agentCaCachePath}
|
||||
echo "wrote ${agentCaCachePath}"
|
||||
fi
|
||||
|
||||
${composeListenerBundle}
|
||||
compose_listener_bundle
|
||||
|
||||
machine=${lib.escapeShellArg cfg.machine}
|
||||
if ! systemctl --machine="$machine" is-active --quiet openbao.service; then
|
||||
echo "openbao in $machine is not running; it reads ${listenerClientCaPath} when it starts"
|
||||
exit 0
|
||||
fi
|
||||
started="$(systemctl --machine="$machine" show --timestamp=us+utc -p ActiveEnterTimestamp --value openbao.service)"
|
||||
started_us="$(date -u -d "$started" +%s%6N)"
|
||||
written_us="$(stat -c %.6Y ${listenerClientCaPath} | tr -d .)"
|
||||
if [ "$written_us" -le "$started_us" ]; then
|
||||
echo "openbao started after ${listenerClientCaPath} last changed; nothing to pick up"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Fail closed: host logins go through this file too.
|
||||
if [ ! -s ${listenerClientCaPath} ] \
|
||||
|| ! cmp -s -n "$(stat -c %s ${clientCaPath})" ${clientCaPath} ${listenerClientCaPath} \
|
||||
|| ! openssl x509 -noout -in ${listenerClientCaPath} >/dev/null 2>&1; then
|
||||
echo "${listenerClientCaPath} is empty, unparseable or does not begin with ${clientCaPath}; not restarting openbao" >&2
|
||||
exit 1
|
||||
fi
|
||||
''
|
||||
+ (
|
||||
if baoDeploy.seal == "pkcs11" then
|
||||
''
|
||||
echo "${listenerClientCaPath} changed after openbao started; restarting openbao in $machine (it unseals itself)"
|
||||
systemctl --machine="$machine" restart openbao.service
|
||||
''
|
||||
else
|
||||
''
|
||||
echo "${listenerClientCaPath} changed after openbao started. A restart seals a ${baoDeploy.seal} store, so it is left to you, as root on this host:" >&2
|
||||
echo " systemctl --machine=$machine restart openbao.service # then unseal" >&2
|
||||
''
|
||||
);
|
||||
};
|
||||
|
||||
# The CA bind source is written at runtime by a host unit, so the
|
||||
# container has to start after it — otherwise nspawn sets up a mount
|
||||
# over a file that does not exist yet.
|
||||
|
|
@ -2754,7 +3020,8 @@ in
|
|||
# /var, systemd owns `${stateDir}` through `StateDirectory=`, and
|
||||
# binding over it is what breaks the unit.
|
||||
bindMounts = {
|
||||
# Read-only: `swarm-bao-certs` on the host is the only writer, and
|
||||
# Read-only: host units write it (`swarm-bao-certs`, and
|
||||
# `swarm-bao-agent-pki` for the agent CA and the listener bundle), and
|
||||
# the store has no reason to modify its own identity.
|
||||
${tlsDir} = {
|
||||
hostPath = tlsDir;
|
||||
|
|
@ -2985,12 +3252,15 @@ in
|
|||
];
|
||||
};
|
||||
|
||||
# ⚠️ Upstream sets `restartIfChanged = false` on this unit, on
|
||||
# purpose: a restart SEALS the store and disconnects every client.
|
||||
# So a change to the settings above does NOT take effect on
|
||||
# `nixos-rebuild switch` — it lands in the config file and waits.
|
||||
# Restarting is an operator action with an unseal on the far side of
|
||||
# it, which is why nothing here tries to be clever about it.
|
||||
# ⚠️ Upstream sets `restartIfChanged = false` on this unit: a restart
|
||||
# SEALS the store and disconnects every client. The container around
|
||||
# it does restart on `nixos-rebuild switch` whenever its config
|
||||
# (these settings included) changes: nixos-containers sets
|
||||
# `restartTriggers` on `container@${cfg.machine}` and nothing here
|
||||
# overrides its `restartIfChanged`. The only in-place restart of this
|
||||
# unit is `swarm-bao-agent-pki` on the host, once, when the listener's
|
||||
# client-CA bundle changed after openbao started, and only under the
|
||||
# self-unsealing `pkcs11` seal.
|
||||
|
||||
# This container's own journal forwarder, copied from an agent
|
||||
# container's (nix/agent-modules/otel.nix) because every container
|
||||
|
|
|
|||
Loading…
Reference in a new issue