swarm-bao: agent certificates issued by a store-generated agent CA
An agent's store identity was signed in swarm-controller's memory by a CA a controller-host unit generated on disk, and the listener never trusted that CA. Agent leaves now come from the store itself: a `pki-agents` PKI mount whose root openbao generates internally, so the agent CA's key never exists outside the store. - swarm-bao-agent-pki (new, store host, as the bao granter): enables and tunes the mount, generates the root once (guarded on an empty issuer list, no replace branch), upserts the `swarm-agent` role (client certificates named `hive-agent-*` only, 90 days), caches the CA at /var/lib/swarm-bao-tls/agent-ca.pem and composes the listener bundle. - The listener's tls_client_ca_file is a new listener-client-ca.pem (client-ca.pem, then the agent CA). Host cert-auth roles still pin client-ca.pem, so an agent leaf satisfies no host role. swarm-bao-certs composes the same bundle before openbao starts. - openbao reads tls_client_ca_file only at start, so when the bundle changed after openbao started, swarm-bao-agent-pki restarts openbao.service in the container; under `seal = "shamir"` it prints the step instead. Once swarm-bao-certs has a cached CA, later boots start openbao with it and do not restart. - The controller policy gains exactly `update` on pki-agents/issue/swarm-agent. mint_and_verify now asks that role for the leaf (the store generates the key), writes the agent's cert-auth role pinning the issuing CA bao returned, and writes the agent's policy as render_agent alone: the hive-shared queue credential stanza is gone. - deploy.bao.agentPkiRoleName (must start `swarm-`, asserted with the other pki role names); swarm-controller gets SWARM_CONTROLLER_AGENT_PKI_MOUNT/_ROLE from the deploy.bao options. Deleted: swarm-controller-agent-ca and its options (agentCaFile, agentCaKeyFile), env, LoadCredential entries and assertion; agent_identity's Authority, rcgen signing and validity window; the rcgen and time dependencies of swarm-controller (rcgen leaves the workspace); policy::render_agent_with_queue and its tests. The CN-prefix assertion policy.rs said was owed is not: agent and host roles pin different CAs. Migration is re-creating each agent after deploy; that overwrites the stale role and policy. Closes #4756
This commit is contained in:
atlas
parent
5cd7f866f4
commit
6170e74a31
16 changed files with 894 additions and 925 deletions
|
|
@ -38,33 +38,6 @@ let
|
|||
# file would be handed to a daemon that cannot use it.
|
||||
haveHiveClientCa = haveBaoIdentity && deployCfg.swarm-controller.hiveClientCaFile != null;
|
||||
|
||||
# The authority this daemon issues AGENT client leaves from — a different
|
||||
# question from `hiveClientCaFile` above, which is the authority it *trusts*
|
||||
# hives by. This one it signs with, so it needs the private key too.
|
||||
#
|
||||
# ⚠️ Deliberately NOT the store's own PKI (`glue-bao-tls.nix`'s
|
||||
# `/var/lib/swarm-bao-pki`). A cert-auth role pins its authority by value,
|
||||
# per role, so a role this daemon writes carries whatever authority this
|
||||
# daemon hands it — which is what lets the controller mint from its own CA
|
||||
# on its own host without anything being co-located and without any
|
||||
# existing role changing. `swarm-controller/src/agent_identity.rs`'s module
|
||||
# doc is the long form.
|
||||
agentCaDir = "/var/lib/swarm-controller-agent-ca";
|
||||
|
||||
# No authority named means mint one here. The alternative — leaving agent
|
||||
# identities off until an operator places a CA by hand — is the state where
|
||||
# the whole path is configured and silently does nothing, which is the
|
||||
# failure mode `glue-bao-tls.nix` avoids the same way.
|
||||
selfSignAgentCa = deployCfg.swarm-controller.agentCaFile == null;
|
||||
agentCaCert =
|
||||
if selfSignAgentCa then "${agentCaDir}/ca.pem" else deployCfg.swarm-controller.agentCaFile;
|
||||
agentCaKey =
|
||||
if selfSignAgentCa then "${agentCaDir}/ca-key.pem" else deployCfg.swarm-controller.agentCaKeyFile;
|
||||
|
||||
# Minting an agent's identity means publishing it to the store, so the
|
||||
# authority alone is not enough — same rule `haveHiveClientCa` states.
|
||||
haveAgentCa = haveBaoIdentity && agentCaKey != null;
|
||||
|
||||
# `swarm_secret_client` reads these spellings explicitly rather than
|
||||
# vaultrs's `VAULT_*` defaults — falling through to those builds a client
|
||||
# with no identity and fails at the TLS handshake, naming neither. `%d` and
|
||||
|
|
@ -88,12 +61,11 @@ let
|
|||
# to put in each hive's cert-auth role.
|
||||
SWARM_CONTROLLER_HIVE_CLIENT_CA_FILE = "%d/hive-client-ca.pem";
|
||||
}
|
||||
// lib.optionalAttrs haveAgentCa {
|
||||
# The authority agent leaves are ISSUED FROM, so unlike every other
|
||||
# `*_CA_FILE` here it comes with a key. `%d` for both: the key is
|
||||
# `0600` and root-owned, and this daemon runs unprivileged.
|
||||
SWARM_CONTROLLER_AGENT_CA_FILE = "%d/agent-ca.pem";
|
||||
SWARM_CONTROLLER_AGENT_CA_KEY_FILE = "%d/agent-ca-key.pem";
|
||||
// lib.optionalAttrs haveBaoIdentity {
|
||||
# Where agent leaves are issued. The store host sets that mount and
|
||||
# role up from the same two options, which keeps the spellings equal.
|
||||
SWARM_CONTROLLER_AGENT_PKI_MOUNT = deployCfg.bao.agentPkiMountPath;
|
||||
SWARM_CONTROLLER_AGENT_PKI_ROLE = deployCfg.bao.agentPkiRoleName;
|
||||
};
|
||||
|
||||
# What `swarmctl` needs in order to act on authelia from the host.
|
||||
|
|
@ -663,47 +635,6 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
agentCaFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-agent-ca/ca.pem";
|
||||
description = ''
|
||||
Authority this daemon **issues** agent client certificates from, so
|
||||
that an agent container can authenticate to the swarm secret store
|
||||
under its own name. Read together with
|
||||
{option}`services.hyperhive.deploy.swarm-controller.agentCaKeyFile`,
|
||||
which is the private key it signs with.
|
||||
|
||||
The mirror image of
|
||||
{option}`services.hyperhive.deploy.swarm-controller.hiveClientCaFile`:
|
||||
that one is an authority this daemon only *trusts by value*, so it is
|
||||
public material and needs no key. This one signs, so it does.
|
||||
|
||||
Leaving this `null` — the default — makes the module mint a
|
||||
self-signed authority in `${agentCaDir}` on first boot and use that.
|
||||
That is the ordinary shape: the store pins an authority per cert-auth
|
||||
role, by value, so the authority agents are issued from does not have
|
||||
to be the store's own PKI and does not have to live on the store's
|
||||
host. Name a file here only when an operator issues agent leaves from
|
||||
somewhere else; doing so turns the self-signing unit off.
|
||||
'';
|
||||
};
|
||||
|
||||
agentCaKeyFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-agent-ca/ca-key.pem";
|
||||
description = ''
|
||||
Private key for
|
||||
{option}`services.hyperhive.deploy.swarm-controller.agentCaFile`.
|
||||
Both or neither — an authority with no key signs nothing, and the
|
||||
daemon refuses to start half-configured rather than looking ready.
|
||||
|
||||
A path, never a value: the key's bytes in a nix expression land in
|
||||
the world-readable nix store, permanently.
|
||||
'';
|
||||
};
|
||||
|
||||
queue = {
|
||||
clientSecretFile = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
|
|
@ -734,10 +665,7 @@ in
|
|||
services.hyperhive.swarm.otel.journaldUnits = [
|
||||
"swarm-controller"
|
||||
"swarm-controller-credential"
|
||||
]
|
||||
# Declared only where the unit exists — an entry for a unit that was
|
||||
# never defined is a collector waiting on a journal that never speaks.
|
||||
++ lib.optional (haveAgentCa && selfSignAgentCa) "swarm-controller-agent-ca";
|
||||
];
|
||||
|
||||
users.users.swarm-controller = {
|
||||
isSystemUser = true;
|
||||
|
|
@ -824,18 +752,6 @@ in
|
|||
state directory.
|
||||
'';
|
||||
}
|
||||
{
|
||||
assertion =
|
||||
deployCfg.swarm-controller.agentCaFile == null || deployCfg.swarm-controller.agentCaKeyFile != null;
|
||||
message = ''
|
||||
services.hyperhive.deploy.swarm-controller.agentCaFile names an
|
||||
authority but agentCaKeyFile is unset.
|
||||
|
||||
The controller does not merely trust this authority, it issues
|
||||
agent client certificates from it, so it needs the private key.
|
||||
Set both, or set neither and let the module mint its own.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
systemd.services.swarm-controller = {
|
||||
|
|
@ -877,16 +793,7 @@ in
|
|||
++ lib.optional (
|
||||
haveBaoIdentity && deployCfg.bao.serverCaFile != null
|
||||
) "bao-ca.pem:${deployCfg.bao.serverCaFile}"
|
||||
++ lib.optional haveHiveClientCa "hive-client-ca.pem:${deployCfg.swarm-controller.hiveClientCaFile}"
|
||||
# The agent authority, key included — same shape and same reason as
|
||||
# the store identity above: the key is root-owned `0600` and this
|
||||
# daemon runs as `swarm-controller`. `swarm-controller-agent-ca`
|
||||
# below is `requiredBy` this unit, so the files exist by the time
|
||||
# systemd resolves these.
|
||||
++ lib.optionals haveAgentCa [
|
||||
"agent-ca.pem:${agentCaCert}"
|
||||
"agent-ca-key.pem:${agentCaKey}"
|
||||
];
|
||||
++ lib.optional haveHiveClientCa "hive-client-ca.pem:${deployCfg.swarm-controller.hiveClientCaFile}";
|
||||
|
||||
# The placeholder default that makes the above non-fatal.
|
||||
# `LoadCredential=` takes priority over `SetCredential=`, so this is
|
||||
|
|
@ -1059,50 +966,5 @@ in
|
|||
ExecStart = "${pkgs.systemd}/bin/systemctl try-restart swarm-controller.service";
|
||||
};
|
||||
};
|
||||
|
||||
# The authority agent client leaves are issued from, minted here when the
|
||||
# operator named none. Shape copied from ./glue-bao-tls.nix's
|
||||
# `swarm-bao-pki`, including the rule that matters most:
|
||||
#
|
||||
# 🩸 Idempotent on ABSENCE, never on content. Re-issuing this CA would
|
||||
# invalidate every agent leaf already published to the store AND every
|
||||
# cert-auth role that pinned it by value, locking every agent container
|
||||
# in the swarm out at once — on a rebuild that changed nothing an
|
||||
# operator asked for.
|
||||
#
|
||||
# `before` + `requiredBy` rather than `after`: the daemon's
|
||||
# `LoadCredential=` names these files by absolute path, and a
|
||||
# `LoadCredential=` pointing at a file that is not there yet is fatal
|
||||
# (`243/CREDENTIALS`), not a slow start.
|
||||
systemd.services.swarm-controller-agent-ca = lib.mkIf (haveAgentCa && selfSignAgentCa) {
|
||||
description = "mint the authority swarm agents' store certificates are issued from";
|
||||
before = [ "swarm-controller.service" ];
|
||||
requiredBy = [ "swarm-controller.service" ];
|
||||
path = [
|
||||
pkgs.openssl
|
||||
pkgs.coreutils
|
||||
];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
};
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
install -d -m 0700 ${agentCaDir}
|
||||
|
||||
if [ ! -s ${agentCaCert} ]; then
|
||||
# `pathlen:0` — this authority signs leaves and nothing else. An
|
||||
# intermediate under it would be a second issuer for the one name
|
||||
# space the store matches agents by.
|
||||
openssl req -x509 -newkey rsa:4096 -nodes -sha256 -days 3650 \
|
||||
-keyout ${agentCaKey} -out ${agentCaCert} \
|
||||
-subj "/CN=swarm-agent-ca ${swarmDomain}" \
|
||||
-addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
|
||||
-addext "keyUsage=critical,keyCertSign,cRLSign"
|
||||
chmod 0600 ${agentCaKey}
|
||||
chmod 0644 ${agentCaCert}
|
||||
fi
|
||||
'';
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue