vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

agents: re-pull the agent's own bao mTLS leaf on a timer

Browse source 
swarm-controller re-issues an agent's store leaf at half its validity, but
the running container kept presenting the copy its hive delivered at
start, so a re-issued leaf reached the agent only on its next restart.

hive-agent-bao-identity now seeds a live copy of the pair in its own
runtime directory from the delivered credentials, and re-runs every 10
minutes: it logs in with the live leaf, reads its own
swarm/agents/<agent>/bao-mtls (a read render_agent already grants), and
when the leaf changed, logs in with the new pair and swaps it in only if
the store accepts it.

Every unit in the container that logs in to the store presents the live
pair: hive-agent (queue-identity.nix), hive-matrix-daemon,
hive-subagent-daemon, hive-agent-forge-token, hive-agent-forge-accounts
and hive-agent-github-token. They keep loading the server CA as a
credential. The long-running daemons are ordered after the refresher so
the live pair exists when they start.

It renews only while the held leaf is still valid: the login that reads
the new leaf is made with the current one.

Refs #5017
This commit is contained in:
atlas 2026-10-11 03:05:59 +02:00 • committed by mara
commit c6717bccb0
17 changed files with 308 additions and 166 deletions
Download patch file Download diff file Expand all files Collapse all files

134
nix/agent-modules/bao.nix
View file

@ -1,11 +1,14 @@
# This agent's own identity at the swarm secret store, and the check that
# proves it works.
# This agent's own identity at the swarm secret store: the check that proves
# it works, and the live copy every store reader in the container presents.
#
# `swarm-controller` mints the leaf at agent creation and publishes it; this
# agent's hive collects it under the hive's own certificate and hands it in as
# systemd credentials (`hive_c0re::lifecycle::agent_identity`). Nothing here
# fetches anything from the store — by the time this container boots, its
# identity is already inside it.
# systemd credentials (`hive_c0re::lifecycle::agent_identity`). That copy is
# fixed for the life of the container, and the controller re-issues the leaf
# at half its validity, so this unit seeds a live copy from it and, on a timer,
# reads the current leaf back from the store under the one it holds. Because
# that login needs the held leaf to still be valid, a container whose leaf
# expired while every pull failed recovers only at its next start.
#
# ⚠️ The identity arrives as credentials and NOT as a bind mount, and the mode
# is why: the host file is `0600` to the hive daemon and this unit runs as the
@ -46,6 +49,17 @@ let
unitName = "hive-agent-bao-identity";
runtimeDir = unitName;
certFile = "/run/${runtimeDir}/cert";
keyFile = "/run/${runtimeDir}/key";
# Beside the live pair, so each rename that replaces it stays in one
# directory.
stagedCert = "/run/${runtimeDir}/cert.new";
stagedKey = "/run/${runtimeDir}/key.new";
# The store's whole answer, key included, in the unit's own `0700` directory.
rawFile = "/run/${runtimeDir}/bao-mtls.json";
errFile = "/run/${runtimeDir}/bao.err";
# Where the identity lives in the store, spelled from the same pieces the
# publisher uses. `swarm_secret_client::mtls::identity_path` builds
# `swarm/agents/<agent>/bao-mtls` and `path::MOUNT` is `secret`; this
@ -87,16 +101,39 @@ in
loudest possible way about a deployment that never asked for it.
'';
};
clientCertFile = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = certFile;
description = ''
This agent's current store certificate, kept by `${unitName}.service`.
Read-only: it is where that unit writes, not a knob. Every unit that
logs in to the store as this agent points `BAO_CLIENT_CERT` here.
'';
};
clientKeyFile = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = keyFile;
description = ''
The private key for `clientCertFile`, `0400` to the agent user.
🩸 A PATH and never a value.
'';
};
};
config = lib.mkIf configured {
systemd.services.${unitName} = {
description = "prove this agent can authenticate to the swarm secret store as itself";
description = "keep this agent's swarm secret store identity current, and prove it works";
after = [ "network.target" ];
wantedBy = [ "multi-user.target" ];
path = [
pkgs.openbao
pkgs.coreutils
pkgs.diffutils
pkgs.jq
];
# ../host-modules/lib/store-retry.nix. From an agent the store is reached
# through the gateway's stream passthrough, so either being down fails
@ -104,10 +141,16 @@ in
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
# Not `RemainAfterExit`, so the timer below can start it again.
# `RuntimeDirectoryPreserve` keeps the live pair between runs.
RemainAfterExit = false;
TimeoutStartSec = 30;
User = agentName;
Group = agentName;
RuntimeDirectory = runtimeDir;
RuntimeDirectoryMode = "0700";
RuntimeDirectoryPreserve = "yes";
UMask = "0377";
# Bare ids, no paths: the terse `LoadCredential=` form that inherits a
# credential the service *manager* received, which is what the
# container manager passed in. ./queue.nix states the same shape.
@ -119,9 +162,8 @@ in
};
environment = {
BAO_ADDR = cfg.addr;
# `%d` is `$CREDENTIALS_DIRECTORY`, per-unit and owned by `User=`.
BAO_CLIENT_CERT = "%d/${certCredential}";
BAO_CLIENT_KEY = "%d/${keyCredential}";
BAO_CLIENT_CERT = certFile;
BAO_CLIENT_KEY = keyFile;
};
script = ''
set -euo pipefail
@ -150,8 +192,19 @@ in
export BAO_CACERT="$CREDENTIALS_DIRECTORY/${serverCaCredential}"
fi
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# `UMask=0377` makes every file this script creates `0400`, so each one
# is removed before the command that creates it.
err=${lib.escapeShellArg errFile}
raw=${lib.escapeShellArg rawFile}
trap 'rm -f "$err" "$raw" ${lib.escapeShellArg stagedCert} ${lib.escapeShellArg stagedKey}' EXIT
# The delivered pair is the start of every container's life: /run is
# empty at boot and kept across this unit's runs.
if [ ! -s "$BAO_CLIENT_CERT" ] || [ ! -s "$BAO_CLIENT_KEY" ]; then
rm -f "$BAO_CLIENT_CERT" "$BAO_CLIENT_KEY"
cp "$CREDENTIALS_DIRECTORY/${keyCredential}" "$BAO_CLIENT_KEY"
cp "$CREDENTIALS_DIRECTORY/${certCredential}" "$BAO_CLIENT_CERT"
fi
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
@ -163,6 +216,7 @@ in
# it accepts, so this agent's leaf matches its own role and no other.
# Naming the role here would be a third copy of a string
# `swarm_secret_client::policy::agent_object_name` already owns.
rm -f "$err"
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "this agent's certificate was refused by the swarm secret store at $BAO_ADDR." >&2
if [ -s "$err" ]; then
@ -175,16 +229,13 @@ in
export BAO_TOKEN
# A token is not yet an answer: the login proves the certificate, this
# proves the policy attached to it. Reading this agent's own identity
# back is the smallest read its document grants, and it is the same
# check `swarm-controller` runs against the leaf before it reports the
# creation done — so a policy that drifted apart from the path fails
# here rather than in whichever pull needed it first.
# proves the policy attached to it, and the answer is the current
# leaf. One read, so the certificate and key come from one version.
#
# ⚠️ Output discarded, not printed: the field is a certificate and the
# object beside it is a private key. Nothing about this check needs a
# value, only whether the read succeeded.
if ! bao kv get -field=cert ${lib.escapeShellArg identityPath} >/dev/null 2>"$err"; then
# ⚠️ Written by redirect, never printed: the object holds a private
# key.
rm -f "$err" "$raw"
if ! bao kv get -format=json ${lib.escapeShellArg identityPath} >"$raw" 2>"$err"; then
echo "this agent logged in to the swarm secret store but cannot read ${identityPath}, so its policy does not cover its own path." >&2
if [ -s "$err" ]; then
cat "$err" >&2
@ -194,8 +245,47 @@ in
exit 1
fi
echo "authenticated to the swarm secret store at $BAO_ADDR as ${agentName} and read ${identityPath}."
# `-j`: the bytes as the store holds them, which is what the hive
# delivered, so an unchanged leaf compares equal to the seeded copy.
rm -f ${lib.escapeShellArg stagedCert} ${lib.escapeShellArg stagedKey}
if ! jq -ej '.data.data.cert | strings' "$raw" >${lib.escapeShellArg stagedCert} \
|| ! jq -ej '.data.data.key | strings' "$raw" >${lib.escapeShellArg stagedKey}; then
echo "${identityPath} holds no certificate and key; this agent keeps the one it has." >&2
exit 1
fi
if cmp -s ${lib.escapeShellArg stagedCert} "$BAO_CLIENT_CERT" \
&& cmp -s ${lib.escapeShellArg stagedKey} "$BAO_CLIENT_KEY"; then
echo "authenticated to the swarm secret store at $BAO_ADDR as ${agentName}; its certificate at ${identityPath} is unchanged."
else
# This unit's next login uses whatever is swapped in, so a pair the
# store refuses must never replace the one it accepts.
rm -f "$err"
if ! BAO_CLIENT_CERT=${lib.escapeShellArg stagedCert} BAO_CLIENT_KEY=${lib.escapeShellArg stagedKey} \
bao login -method=cert -token-only >/dev/null 2>"$err"; then
echo "the swarm secret store refused the certificate at ${identityPath}; this agent keeps the one it has." >&2
if [ -s "$err" ]; then cat "$err" >&2; fi
exit 1
fi
# Two renames: a reader that opens the pair between them gets a
# mismatched cert and key and fails that one login.
mv -f ${lib.escapeShellArg stagedKey} "$BAO_CLIENT_KEY"
mv -f ${lib.escapeShellArg stagedCert} "$BAO_CLIENT_CERT"
echo "replaced this agent's store certificate with the one at ${identityPath}."
fi
'';
};
# The controller re-issues a leaf at half its validity, so ten minutes is
# small against the time the old one stays valid.
systemd.timers.${unitName} = {
description = "re-read this agent's store certificate from the secret store";
wantedBy = [ "timers.target" ];
timerConfig = {
OnUnitInactiveSec = "10min";
RandomizedDelaySec = "1min";
};
};
};
}

18
nix/agent-modules/forge-accounts.nix
View file

@ -23,9 +23,7 @@ let
agentName = config.services.hyperhive.agent.user.name;
stateDir = "/agents/${agentName}/state";
# The same three ids ./bao.nix and ./forge-token.nix load.
certCredential = "hive-agent-bao-cert";
keyCredential = "hive-agent-bao-key";
# The server CA id ./bao.nix loads.
serverCaCredential = "hive-agent-bao-server-ca";
unitName = "hive-agent-forge-accounts";
@ -73,23 +71,19 @@ in
RuntimeDirectoryMode = "0700";
# `0600`, the mode the files in the state dir have always had.
UMask = "0077";
LoadCredential = [
certCredential
keyCredential
serverCaCredential
];
LoadCredential = [ serverCaCredential ];
};
environment = {
BAO_ADDR = cfg.addr;
BAO_CLIENT_CERT = "%d/${certCredential}";
BAO_CLIENT_KEY = "%d/${keyCredential}";
BAO_CLIENT_CERT = cfg.clientCertFile;
BAO_CLIENT_KEY = cfg.clientKeyFile;
};
script = ''
set -euo pipefail
# No identity delivered: ./bao.nix's check reports that.
for id in ${lib.escapeShellArg certCredential} ${lib.escapeShellArg keyCredential}; do
if [ ! -s "$CREDENTIALS_DIRECTORY/$id" ]; then
for f in "$BAO_CLIENT_CERT" "$BAO_CLIENT_KEY"; do
if [ ! -s "$f" ]; then
echo "this agent has no store identity, so it cannot fetch its external forge accounts." >&2
exit 0
fi

22
nix/agent-modules/forge-token.nix
View file

@ -28,9 +28,7 @@ let
# The name `swarm-controller` minted the token under — see ./queue-identity.nix.
agentName = config.services.hyperhive.agent.user.name;
# The same three ids ./bao.nix and ./queue-identity.nix load.
certCredential = "hive-agent-bao-cert";
keyCredential = "hive-agent-bao-key";
# The server CA id ./bao.nix loads.
serverCaCredential = "hive-agent-bao-server-ca";
unitName = "hive-agent-forge-token";
@ -108,29 +106,25 @@ in
RuntimeDirectoryMode = "0700";
RuntimeDirectoryPreserve = "yes";
UMask = "0377";
LoadCredential = [
certCredential
keyCredential
serverCaCredential
];
LoadCredential = [ serverCaCredential ];
};
environment = {
BAO_ADDR = cfg.addr;
BAO_CLIENT_CERT = "%d/${certCredential}";
BAO_CLIENT_KEY = "%d/${keyCredential}";
BAO_CLIENT_CERT = cfg.clientCertFile;
BAO_CLIENT_KEY = cfg.clientKeyFile;
};
script = ''
set -euo pipefail
# No identity delivered: ./bao.nix's check reports that; saying it
# twice adds nothing.
for id in ${lib.escapeShellArg certCredential} ${lib.escapeShellArg keyCredential}; do
if [ ! -s "$CREDENTIALS_DIRECTORY/$id" ]; then
for f in "$BAO_CLIENT_CERT" "$BAO_CLIENT_KEY"; do
if [ ! -s "$f" ]; then
echo "this agent has no store identity, so it cannot fetch its own forge token." >&2
exit 0
fi
if [ ! -r "$CREDENTIALS_DIRECTORY/$id" ]; then
echo "cannot read $CREDENTIALS_DIRECTORY/$id, so this agent cannot present its store identity." >&2
if [ ! -r "$f" ]; then
echo "cannot read $f, so this agent cannot present its store identity." >&2
exit 1
fi
done

18
nix/agent-modules/github-token.nix
View file

@ -22,9 +22,7 @@ let
agentName = config.services.hyperhive.agent.user.name;
stateDir = "/agents/${agentName}/state";
# The same three ids ./bao.nix and ./forge-accounts.nix load.
certCredential = "hive-agent-bao-cert";
keyCredential = "hive-agent-bao-key";
# The server CA id ./bao.nix loads.
serverCaCredential = "hive-agent-bao-server-ca";
unitName = "hive-agent-github-token";
@ -76,23 +74,19 @@ in
RuntimeDirectoryMode = "0700";
# `0600`, the mode `github-token` has.
UMask = "0077";
LoadCredential = [
certCredential
keyCredential
serverCaCredential
];
LoadCredential = [ serverCaCredential ];
};
environment = {
BAO_ADDR = cfg.addr;
BAO_CLIENT_CERT = "%d/${certCredential}";
BAO_CLIENT_KEY = "%d/${keyCredential}";
BAO_CLIENT_CERT = cfg.clientCertFile;
BAO_CLIENT_KEY = cfg.clientKeyFile;
};
script = ''
set -euo pipefail
# No identity delivered: ./bao.nix's check reports that.
for id in ${lib.escapeShellArg certCredential} ${lib.escapeShellArg keyCredential}; do
if [ ! -s "$CREDENTIALS_DIRECTORY/$id" ]; then
for f in "$BAO_CLIENT_CERT" "$BAO_CLIENT_KEY"; do
if [ ! -s "$f" ]; then
echo "this agent has no store identity, so it cannot fetch its GitHub token." >&2
exit 0
fi

26
nix/agent-modules/matrix.nix
View file

@ -21,12 +21,10 @@ let
accounts = config.services.hyperhive.agent.matrixAccounts;
# This agent's own identity at the swarm secret store (./bao.nix). The daemon
# reads each account's token from the store itself, `main` included (the
# swarm mints that one), so the store's coordinates belong on its unit. The
# same three credential ids ./bao.nix and ./forge-token.nix load.
# swarm mints that one), so the store's coordinates belong on its unit: the
# live certificate pair ./bao.nix keeps, and the server CA id it loads.
baoCfg = config.services.hyperhive.agent.bao;
storeConfigured = baoCfg.addr != null;
certCredential = "hive-agent-bao-cert";
keyCredential = "hive-agent-bao-key";
serverCaCredential = "hive-agent-bao-server-ca";
# **The enable signal.** Matrix is on for this agent exactly when it has at
# least one account, because an account is the only thing the daemon has to
@ -308,7 +306,11 @@ in
description = "long-running matrix-sdk Client + MCP daemon";
wantedBy = [ "multi-user.target" ];
before = [ "hive-agent.service" ];
after = [ "network-online.target" ];
# The live certificate pair exists once ./bao.nix's unit has run.
after = [
"network-online.target"
]
++ lib.optional storeConfigured "hive-agent-bao-identity.service";
wants = [ "network-online.target" ];
environment = {
# In-agent todo socket the harness serves (loose-ends v2): the
@ -328,8 +330,8 @@ in
// lib.optionalAttrs storeConfigured {
HIVE_AGENT_NAME = userName;
BAO_ADDR = baoCfg.addr;
BAO_CLIENT_CERT = "%d/${certCredential}";
BAO_CLIENT_KEY = "%d/${keyCredential}";
BAO_CLIENT_CERT = baoCfg.clientCertFile;
BAO_CLIENT_KEY = baoCfg.clientKeyFile;
# Named even when no CA was delivered: a bare `LoadCredential=` is
# non-fatal when absent, and the daemon treats a missing or empty file
# as "use the container's own trust store".
@ -400,14 +402,10 @@ in
User = userName;
Group = userName;
}
# This agent's own certificate, imported by id so systemd materialises it
# under this unit's `User=` — the same bare form ./forge-token.nix uses.
# The store's server CA, imported by id so systemd materialises it under
# this unit's `User=` — the same bare form ./forge-token.nix uses.
// lib.optionalAttrs storeConfigured {
LoadCredential = [
certCredential
keyCredential
serverCaCredential
];
LoadCredential = [ serverCaCredential ];
};
};

14
nix/agent-modules/mcp.nix
View file

@ -381,6 +381,8 @@ in
description = "subagent task runner + MCP daemon for hive-subagent";
wantedBy = [ "multi-user.target" ];
before = [ "hive-agent.service" ];
# The live certificate pair exists once ./bao.nix's unit has run.
after = lib.optional subagentReadsStore "hive-agent-bao-identity.service";
# A subagent task runs its own nested `claude` invocation (via
# `hive-claude`), which needs to resolve `claude` itself off PATH —
# same reasoning as `hive-bash-daemon`'s `path` above, even though
@ -442,8 +444,8 @@ in
# Paths and a name only. `%d` is this unit's own credentials directory.
HIVE_AGENT_NAME = userName;
BAO_ADDR = config.services.hyperhive.agent.bao.addr;
BAO_CLIENT_CERT = "%d/hive-agent-bao-cert";
BAO_CLIENT_KEY = "%d/hive-agent-bao-key";
BAO_CLIENT_CERT = config.services.hyperhive.agent.bao.clientCertFile;
BAO_CLIENT_KEY = config.services.hyperhive.agent.bao.clientKeyFile;
BAO_CACERT = "%d/hive-agent-bao-server-ca";
};
serviceConfig = {
@ -481,13 +483,9 @@ in
MemoryHigh = toString subagentMemoryHigh;
}
// lib.optionalAttrs subagentReadsStore {
# Bare ids: inherits the credentials the container manager passed in,
# Bare id: inherits the credential the container manager passed in,
# the same form ./queue-identity.nix uses.
LoadCredential = [
"hive-agent-bao-cert"
"hive-agent-bao-key"
"hive-agent-bao-server-ca"
];
LoadCredential = [ "hive-agent-bao-server-ca" ];
}
//
lib.optionalAttrs

24
nix/agent-modules/queue-identity.nix
View file

@ -18,7 +18,7 @@
# before its first queue connection and again on every reconnect attempt
# (`hive-agent`'s `swarm_queue`), so a re-minted secret is picked up without a
# restart. What this module hands the harness is the store's address and the
# paths of its certificate.
# paths of the live certificate pair ./bao.nix keeps.
#
# ⛔ The store certificate is for reaching the store and nothing else. It is
# never presented to the queue: what goes to the queue is the secret read
@ -37,11 +37,9 @@ let
# role from it.
agentName = config.services.hyperhive.agent.user.name;
# The same three ids ./bao.nix loads, because there is one store identity
# per agent; the ids are `hive_c0re::lifecycle::agent_identity`'s and a
# rename is a rename there too.
certCredential = "hive-agent-bao-cert";
keyCredential = "hive-agent-bao-key";
# The server CA id ./bao.nix loads; the id is
# `hive_c0re::lifecycle::agent_identity`'s and a rename is a rename there
# too.
serverCaCredential = "hive-agent-bao-server-ca";
# The store's address is the whole switch, exactly as in ./bao.nix — and
@ -54,20 +52,18 @@ in
{
config = lib.mkIf configured {
systemd.services.hive-agent = {
# Bare ids, no paths: the terse `LoadCredential=` form that inherits a
# A bare id, no path: the terse `LoadCredential=` form that inherits a
# credential the service *manager* received, which is what the container
# manager passed in. ./bao.nix and ./queue.nix state the same shape.
serviceConfig.LoadCredential = [
certCredential
keyCredential
serverCaCredential
];
serviceConfig.LoadCredential = [ serverCaCredential ];
# The live certificate pair exists once ./bao.nix's unit has run.
after = [ "hive-agent-bao-identity.service" ];
# Paths and a name only. `%d` is this unit's own credentials directory.
environment = {
HIVE_AGENT_NAME = agentName;
BAO_ADDR = cfg.addr;
BAO_CLIENT_CERT = "%d/${certCredential}";
BAO_CLIENT_KEY = "%d/${keyCredential}";
BAO_CLIENT_CERT = cfg.clientCertFile;
BAO_CLIENT_KEY = cfg.clientKeyFile;
# Named even when no CA was delivered: a bare `LoadCredential=` is
# non-fatal when absent, and the harness treats a missing or empty file
# as "use the container's own trust store".