vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm: courier an agent's store identity into its container, and log in with it

Browse source 
`swarm-controller` mints an agent's mTLS leaf at creation and publishes it
at `swarm/agents/<agent>/bao-mtls`. Nothing read it back. This adds the
hop that carries it the rest of the way, and the in-container consumer
that proves the hop works.

Host side, `lifecycle::agent_identity` reads the row under *this hive's*
own certificate — the hive is a principal the store already knows — and
stages the leaf and its key `0600` under a new `agent-identity/<name>`
state dir, deliberately outside every bind-mounted tree. Both files go in
as systemd credentials rather than binds, the same answer and the same
mode reason as the queue secret beside it: the staged key is unreadable
to the unprivileged agent user, and the container manager reads a
`--load-credential` source as root before re-exposing it under the
consuming unit's own `User=`. The agent is never asked to authenticate in
order to obtain the thing it authenticates with.

Container side, `hive-agent-bao-identity.service` logs in with that
certificate and reads the agent's own path back, failing the unit when
either step does not succeed. It fails loudly where the hive-side readers
degrade quietly, because a refused certificate means an agent that
believes it reaches the store and never does — a cause only the login
itself can name.

The address is the whole switch, no separate `enable`, matching how
`queue.nix` and `logs.nix` already gate themselves. A hive with a store
forwards `HIVE_AGENT_BAO_ADDR` and every agent on it gets the check; a
hive without one forwards nothing and no agent does. That is what keeps
the delivery from landing in a container with nothing to read it.

The hive can now reach an agent's identity, so hive privilege covers
agent privilege. Accepted, not mitigated: the alternative is an agent
fetching its own credential with a credential it does not yet have.

Refs #4137
This commit is contained in:
atlas 2026-09-18 19:31:25 +02:00
commit 837e658d4a
10 changed files with 716 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

206
nix/agent-modules/bao.nix Normal file
View file

@ -0,0 +1,206 @@
# This agent's own identity at the swarm secret store, and the check that
# proves it works.
#
# `swarm-controller` mints the leaf at agent creation and publishes it; this
# agent's hive collects it under the hive's own certificate and hands it in as
# systemd credentials (`hive_c0re::lifecycle::agent_identity`). Nothing here
# fetches anything from the store — by the time this container boots, its
# identity is already inside it.
#
# ⚠️ The identity arrives as credentials and NOT as a bind mount, and the mode
# is why: the host file is `0600` to the hive daemon and this unit runs as the
# unprivileged agent user. nspawn's `--load-credential` is read by the
# container manager as root and re-exposed under this unit's own `User=`; a
# bind would deliver a private key this user cannot open. Same answer, same
# reason, as ./queue.nix's credential pair.
#
# This is the courier's other end: the hive-side collector has no purpose
# without it, so the two ship together and neither is reachable alone.
#
# 🩸 This unit fails LOUDLY where the hive-side readers degrade quietly, and
# that is the opposite default on purpose. A missing OIDC client secret means
# a hive whose publisher has yet to run; a missing or refused certificate
# means an agent that believes it can reach the store and cannot, which every
# later pull would report as its own unrelated failure. The one place that
# knows the real cause is the login itself, so this is where it is said.
{
pkgs,
lib,
config,
...
}:
let
cfg = config.services.hyperhive.agent.bao;
# This container's agent name. The same string the hive published the
# identity under, because the agent's unix user is named for the agent —
# see ./user.nix.
agentName = config.services.hyperhive.agent.user.name;
# The three ids `hive_c0re::lifecycle::agent_identity` forwards under.
# Neither side can discover the other's spelling, so a rename is a rename
# there too.
certCredential = "hive-agent-bao-cert";
keyCredential = "hive-agent-bao-key";
serverCaCredential = "hive-agent-bao-server-ca";
unitName = "hive-agent-bao-identity";
# Where the identity lives in the store, spelled from the same pieces the
# publisher uses. `swarm_secret_client::mtls::identity_path` builds
# `swarm/agents/<agent>/bao-mtls` and `path::MOUNT` is `secret`; this
# literal is the nix half of that one agreement, exactly as
# ../host-modules/glue-queue-agent-credential.nix spells its own.
identityPath = "secret/swarm/agents/${agentName}/bao-mtls";
# The address is the whole switch — no separate `enable`, the same shape
# ./queue.nix and ./logs.nix gate themselves with. A hive that has a store
# forwards its address and every agent on it gets the check; a hive that has
# none forwards nothing and no agent does. An `enable` beside it would be a
# knob whose only correct setting is whatever the address already says, and
# its default would decide whether the hive-side courier delivers into a
# container that reads what it is given or into one that never looks.
configured = cfg.addr != null;
in
{
options.services.hyperhive.agent.bao = {
addr = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "https://bao.example.com:8200";
description = ''
Where the swarm secret store listens, as this container reaches it.
Set by the generated meta flake from the host's own `BAO_ADDR`, which
is the address this hive already uses, and setting it is what generates
`${unitName}.service`: at boot that unit logs in with the certificate
its hive delivered and reads this agent's own path back, failing if
either step does not succeed. `systemctl status ${unitName}` inside the
container is then the answer to "can this agent reach the store as
itself", which nothing else in the tree reports.
`null` means the hive was given no store. No unit is generated then,
because an agent whose swarm never minted an identity has nothing to
log in with, and a failed unit at every boot would say that in the
loudest possible way about a deployment that never asked for it.
'';
};
};
config = lib.mkIf configured {
systemd.services.${unitName} = {
description = "prove this agent can authenticate to the swarm secret store as itself";
after = [ "network.target" ];
wantedBy = [ "multi-user.target" ];
path = [
pkgs.openbao
pkgs.coreutils
];
# Sized for a store that comes up around the same time this container
# does, not for one that is sealed: a few short attempts cover the race,
# and a longer window would only delay the report of a real failure.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec` times the burst.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
TimeoutStartSec = 30;
Restart = "on-failure";
RestartSec = 15;
User = agentName;
Group = agentName;
# Bare ids, no paths: the terse `LoadCredential=` form that inherits a
# credential the service *manager* received, which is what the
# container manager passed in. ./queue.nix states the same shape.
LoadCredential = [
certCredential
keyCredential
serverCaCredential
];
};
environment = {
BAO_ADDR = cfg.addr;
# `%d` is `$CREDENTIALS_DIRECTORY`, per-unit and owned by `User=`.
BAO_CLIENT_CERT = "%d/${certCredential}";
BAO_CLIENT_KEY = "%d/${keyCredential}";
};
script = ''
set -euo pipefail
# Absent credentials are the first thing checked, because every later
# message would blame the store for a delivery that never happened.
# 🩸 The files are named and never read here: what went wrong is a
# property of the path, and the bytes at it are a private key.
missing=
for id in ${lib.escapeShellArg certCredential} ${lib.escapeShellArg keyCredential}; do
if [ ! -s "$CREDENTIALS_DIRECTORY/$id" ]; then
missing="$missing $id"
fi
done
if [ -n "$missing" ]; then
echo "this agent has no store identity: the hive delivered no$missing." >&2
echo "swarm-controller publishes it at ${identityPath} when the agent is created, and the hive collects it from there." >&2
exit 1
fi
# Only when one was delivered. Absent means the container verifies the
# store's listener against its own trust store, which is what a
# deployment with a real CA wants; pointing BAO_CACERT at a file that
# is not there would fail the handshake and name the wrong cause.
if [ -s "$CREDENTIALS_DIRECTORY/${serverCaCredential}" ]; then
export BAO_CACERT="$CREDENTIALS_DIRECTORY/${serverCaCredential}"
fi
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# No `name=`: a cert role pins both its authority and the common name
# it accepts, so this agent's leaf matches its own role and no other.
# Naming the role here would be a third copy of a string
# `swarm_secret_client::policy::agent_object_name` already owns.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "this agent's certificate was refused by the swarm secret store at $BAO_ADDR." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
# A token is not yet an answer: the login proves the certificate, this
# proves the policy attached to it. Reading this agent's own identity
# back is the smallest read its document grants, and it is the same
# check `swarm-controller` runs against the leaf before it reports the
# creation done — so a policy that drifted apart from the path fails
# here rather than in whichever pull needed it first.
#
# ⚠️ Output discarded, not printed: the field is a certificate and the
# object beside it is a private key. Nothing about this check needs a
# value, only whether the read succeeded.
if ! bao kv get -field=cert ${lib.escapeShellArg identityPath} >/dev/null 2>"$err"; then
echo "this agent logged in to the swarm secret store but cannot read ${identityPath}, so its policy does not cover its own path." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
echo "authenticated to the swarm secret store at $BAO_ADDR as ${agentName} and read ${identityPath}."
'';
};
};
}

1
nix/agent-modules/default.nix
View file

@ -40,6 +40,7 @@ in
{
imports = [
./agent-service.nix
./bao.nix
./bash-env.nix
./claude-settings.nix
./dashboard-links.nix

6
nix/host-modules/hive-c0re/environment.nix
View file

@ -334,6 +334,12 @@ in
# handshake, naming neither.
lib.optionalAttrs haveBaoClientIdentity {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
# The same address again under an agent-facing name, forwarded into every
# container's module block by `meta::render_flake`. Deliberately not the
# `BAO_ADDR` line itself: a container that inherited this hive's store
# environment wholesale would inherit the hive's certificate paths with it,
# and those name a credential no agent may present.
HIVE_AGENT_BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
# `%d`, not the paths themselves: the key is `0600` root-owned and this
# daemon runs as hive-core, so it never gets read access to the original.
# See the LoadCredential in ./default.nix.

134
nix/module-eval.nix
View file

@ -64,10 +64,12 @@ let
# the meta flake hands a container, so what this evaluates is what an
# agent gets.
#
# Note `hyperhive`, not `services.hyperhive`: an agent container's options
# live at the top level.
agent =
extra:
# Takes a whole module rather than a settings attrset, so a fixture can
# reach either spelling of the agent tier. `user.name` is the one option
# with no usable default, set here at its real path and at `mkDefault` so a
# fixture naming its own agent still wins.
agentWith =
module:
(nixosSystem {
system = pkgs.stdenv.hostPlatform.system;
modules = [
@ -79,11 +81,20 @@ let
};
boot.loader.grub.enable = false;
system.stateVersion = "25.11";
hyperhive = lib.recursiveUpdate { user.name = "a1"; } extra;
services.hyperhive.agent.user.name = lib.mkDefault "a1";
}
module
];
}).config;
# Note `hyperhive`, not `services.hyperhive`: the agent tier's options moved
# under `services.hyperhive.agent`, and ../agent-modules/renamed-options.nix
# keeps the top-level spelling reaching them. ⚠️ That shim covers the
# options that existed when the tier moved and nothing since, so a fixture
# for an option added afterwards has to go through [`agentWith`] and name
# the real path.
agent = extra: agentWith { hyperhive = extra; };
allLocal = hive { deploy.singleHostSwarm = true; };
bare = hive { };
withCi = hive { deploy.forgejo.ci.enable = true; };
@ -605,6 +616,12 @@ let
};
agentNoQueue = agent { };
# The agent side of the swarm secret store. The address is the whole switch —
# it is both what generates the login check and what that check points at —
# so it and the empty fixture beside it are the two arms worth having.
agentBao = agentWith { services.hyperhive.agent.bao.addr = "https://bao.t.local:8200"; };
agentNoBao = agentWith { };
# The memory-pressure pair. `claudeMemoryMaxBytes` is the container's own
# cap, rendered per agent by meta.rs — the capped arm is the one every
# real deploy gets, the uncapped arm is a hive that set `infinity` or a
@ -636,6 +653,7 @@ let
};
agentHarness = machine: machine.systemd.services.hive-agent;
agentSubagentDaemon = machine: machine.systemd.services.hive-subagent-daemon;
agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity;
agentSettings = machine: machine.services.opentelemetry-collector.settings;
# This hive's own collector, which is a HOST service — unlike the swarm
@ -1766,6 +1784,112 @@ let
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_SECRET_FILE)
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_ID_FILE);
}
{
# The three ids `hive_c0re::lifecycle::agent_identity` forwards under.
# Neither end can discover the other's spelling, and a mismatch is a
# credential that is simply not there — which this unit then reports as
# a hive that delivered nothing.
name = "an agent with the store enabled imports every half of its identity";
ok =
let
c = (agentBaoIdentity agentBao).serviceConfig.LoadCredential;
in
builtins.elem "hive-agent-bao-cert" c
&& builtins.elem "hive-agent-bao-key" c
&& builtins.elem "hive-agent-bao-server-ca" c;
}
{
# `%d` and not a path under the agent's state dir, for the reason the
# queue arm above gives: the host file is `0600` to the hive daemon, so
# the only copy this unprivileged unit can open is the one systemd puts
# in its own credentials directory. The address is the option's value
# rather than a literal that agrees with it today.
name = "the identity check presents its certificate out of the credentials directory";
ok =
let
e = (agentBaoIdentity agentBao).environment;
in
e.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert"
&& e.BAO_CLIENT_KEY == "%d/hive-agent-bao-key"
&& e.BAO_ADDR == agentBao.services.hyperhive.agent.bao.addr;
}
{
# Same 403-not-a-miss reason as the hive-side readers: the path
# `swarm_secret_client::mtls::identity_path` builds is the one this
# agent's own policy stanza covers, and a path outside it is refused
# however correct it looks. Built from the agent's own name rather than
# from a literal, because the name is what makes it this agent's path
# and not some other agent's.
name = "the identity check reads the agent's own path";
ok =
let
m = agentBao;
name = m.services.hyperhive.agent.user.name;
in
lib.hasInfix "secret/swarm/agents/${name}/bao-mtls" (agentBaoIdentity m).script;
}
{
# The whole point of the unit, and the thing a quieter default would
# undo: every arm of the check ends the unit non-zero, so an agent that
# cannot authenticate as itself says so at boot instead of at whichever
# pull needed the store first.
name = "the identity check fails the unit rather than degrading";
ok =
let
u = agentBaoIdentity agentBao;
in
lib.hasInfix "exit 1" u.script
&& !(lib.hasInfix "exit 0" u.script)
&& u.serviceConfig.Restart == "on-failure";
}
{
# Nothing about the identity may be printed, and the read-back is where
# that could slip: `bao kv get` on this path answers with certificate
# material, and the object beside it is a private key. The check needs
# only whether the read succeeded.
#
# The path goes through `lib.escapeShellArg` here for the same reason the
# module passes it through one — that helper decides whether an argument
# needs quotes at all, and this one (only `[a-z0-9/-]`) comes back bare.
# Spelling the quotes in by hand asserts a rendering nixpkgs chooses
# rather than the redirect this property is about.
name = "the identity check discards what it reads back";
ok =
let
m = agentBao;
name = m.services.hyperhive.agent.user.name;
arg = lib.escapeShellArg "secret/swarm/agents/${name}/bao-mtls";
in
lib.hasInfix "bao kv get -field=cert ${arg} >/dev/null" (agentBaoIdentity m).script;
}
{
# The absence arm, and what makes the four above able to fail. An agent
# whose swarm never minted an identity has nothing to log in with, and a
# failed unit at every boot would be the loudest possible statement
# about a deployment that never asked for one.
name = "an agent told no store address runs no identity check";
ok = !(agentNoBao.systemd.services ? hive-agent-bao-identity);
}
{
# Where the store is and where the agent is told it is, one agreement
# spanning two modules. Asserted against the hive's own `BAO_ADDR`
# rather than a literal, because an agent pointed at a different
# spelling of the same store presents a certificate to a listener whose
# name it cannot verify.
name = "an agent is told the same store address its hive uses";
ok =
let
e = allLocal.systemd.services.hive-c0re.environment;
in
e.HIVE_AGENT_BAO_ADDR == e.BAO_ADDR;
}
{
# A hive with no certificate of its own can collect no agent's identity,
# so forwarding an address would name a store nothing in the container
# can reach. The same gate the `BAO_*` pair beside it sits behind.
name = "a hive with no store identity forwards no store address to its agents";
ok = !(bare.systemd.services.hive-c0re.environment ? HIVE_AGENT_BAO_ADDR);
}
{
# Two thirds of the container's cap, and a SOFT ceiling: the daemon's
# cgroup holds every nested claude, so `MemoryHigh=` throttles the