vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm: courier an agent's store identity into its container, and log in with it

Browse source 
`swarm-controller` mints an agent's mTLS leaf at creation and publishes it
at `swarm/agents/<agent>/bao-mtls`. Nothing read it back. This adds the
hop that carries it the rest of the way, and the in-container consumer
that proves the hop works.

Host side, `lifecycle::agent_identity` reads the row under *this hive's*
own certificate — the hive is a principal the store already knows — and
stages the leaf and its key `0600` under a new `agent-identity/<name>`
state dir, deliberately outside every bind-mounted tree. Both files go in
as systemd credentials rather than binds, the same answer and the same
mode reason as the queue secret beside it: the staged key is unreadable
to the unprivileged agent user, and the container manager reads a
`--load-credential` source as root before re-exposing it under the
consuming unit's own `User=`. The agent is never asked to authenticate in
order to obtain the thing it authenticates with.

Container side, `hive-agent-bao-identity.service` logs in with that
certificate and reads the agent's own path back, failing the unit when
either step does not succeed. It fails loudly where the hive-side readers
degrade quietly, because a refused certificate means an agent that
believes it reaches the store and never does — a cause only the login
itself can name.

The address is the whole switch, no separate `enable`, matching how
`queue.nix` and `logs.nix` already gate themselves. A hive with a store
forwards `HIVE_AGENT_BAO_ADDR` and every agent on it gets the check; a
hive without one forwards nothing and no agent does. That is what keeps
the delivery from landing in a container with nothing to read it.

The hive can now reach an agent's identity, so hive privilege covers
agent privilege. Accepted, not mitigated: the alternative is an agent
fetching its own credential with a credential it does not yet have.

Refs #4137
This commit is contained in:
atlas 2026-09-18 19:31:25 +02:00
commit 837e658d4a
10 changed files with 716 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

23
docs/swarm/credentials.md
View file

@ -51,7 +51,7 @@ strategy for every credential, including the mTLS leaf.
| store path | minter | reader — pulls at runtime, holds in memory | renewal |
| -------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- | -------------- |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated |
| `swarm/agents/<agent>/bao-mtls` | `swarm-controller`, at agent creation | the agent's **hive**, under the hive's own certificate, which hands it into the container | must be stated |
| `swarm/agents/<agent>/bao-mtls` | `swarm-controller`, at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | must be stated |
| `swarm/hives/<hive>/matrix/appservice-token` | one minter, on the authelia host | the hive process that presents the token to its homeserver, under the hive's own certificate | must be stated |
| `swarm/hives/<hive>/queue/agent` | authelia | the agent container presenting the OIDC client to the swarm queue, under its own certificate | must be stated |
| `swarm/services/<clientId>/oidc/client` | authelia | the service process that presents the client secret, under the certificate of the host it runs on | must be stated |
@ -67,6 +67,27 @@ carries it, and no hive ever needs the capability to mint an identity.
`swarm-controller` proves the leaf it publishes before the creation job
reports success, by logging in with it and reading the row back.
**Who reads that row, and what happens to it.** `hive-c0re` reads it every
time it writes an agent's container configuration
(`lifecycle::agent_identity`), stages the certificate and its key `0600`
outside every bind-mounted tree, and passes both to the container as systemd
credentials — the same mechanism, and for the same mode reason, as the
per-hive queue secret. A bind mount would hand the agent's unprivileged user
a file it lacks the rights to open; the container manager reads a credential
as root and re-exposes it under the consuming unit's own user.
Inside the container, `hive-agent-bao-identity.service` logs in with that
certificate and reads this row back before reporting success, so an agent
locked out of its own identity says so at boot rather than at whichever pull
needed the store first. The unit exists whenever
`services.hyperhive.agent.bao.addr` has a value, which the hive's meta flake
sets from its own store address — the same all-or-nothing gate the per-hive
queue credential beside it uses, and the reason the delivery above never lands
in a container with nothing to read it. It fails loudly where the hive-side
readers degrade quietly, which is deliberate: a missing queue secret means a
swarm whose publisher has yet to run, while a refused certificate means an
agent that believes it reaches the store and never does.
## Progressive enhancement
New functionality has to match this shape immediately — no PR introducing a

306
hive-c0re/src/lifecycle/agent_identity.rs Normal file
View file

@ -0,0 +1,306 @@
//! The courier hop: this hive collects an agent's own store identity and
//! stages it for delivery into that agent's container.
//!
//! `swarm-controller` mints the leaf at agent-create and publishes it at
//! `swarm/agents/<agent>/bao-mtls`; this reads it back under **the hive's**
//! certificate — the hive is a principal the store already knows — and lands
//! the two files `systemd-nspawn` forwards as credentials. The agent is never
//! asked to authenticate in order to obtain the thing it authenticates with:
//! by the time it runs, its identity is already inside it.
//!
//! **A credential and not a bind mount**, the same answer and the same reason
//! as the queue secret beside it in [`super::host_config`]: the files are
//! `0600` to `hive-core` and the units that use them run as the unprivileged
//! agent user, so a bind would deliver a private key that user cannot open.
//! The container manager reads a `--load-credential` source as root and
//! re-exposes it under the consuming unit's own `User=`.
//!
//! 🩸 Nothing here ever formats the credential's fields. The value moves
//! store → memory → file and is never an argument to a log macro; the
//! [`mtls::Credential`] type's hand-written `Debug` redacts the key for the
//! one place a value could still reach a line — an error context chain.
//!
//! The consumer sits inside the container: `nix/agent-modules/bao.nix`'s
//! `hive-agent-bao-identity.service` logs in with what lands here and fails
//! loudly when it cannot.
use std::path::{Path, PathBuf};
use hive_priv_sock::CredentialMount;
use swarm_secret_client::{SecretStore, client, mtls};
/// systemd credential ids the agent's identity arrives under inside the
/// container. `nix/agent-modules/bao.nix` spells the same three in its
/// `LoadCredential=`; neither side can discover the other's, so a rename here
/// is a rename there.
///
/// No `.pem` suffix: `hive-priv` refuses a credential name containing a dot,
/// since the name is interpolated into `--load-credential=<name>:<path>`.
const CERT_CREDENTIAL: &str = "hive-agent-bao-cert";
const KEY_CREDENTIAL: &str = "hive-agent-bao-key";
/// The authority the **store's own listener** is verified against — not the
/// authority that issued the agent's leaf, which no reader in the container
/// needs and which is therefore not delivered at all.
///
/// Copied into the staging dir rather than forwarded from the hive's own
/// `BAO_CACERT`: that path is this daemon's per-unit credential directory,
/// which systemd removes when `hive-c0re` stops. An nspawn config pointing at
/// it would leave a container unable to start whenever the daemon was down.
const SERVER_CA_CREDENTIAL: &str = "hive-agent-bao-server-ca";
/// Collect `agent`'s store identity and return the credentials to forward.
///
/// Empty whenever this hive cannot answer the question — no store configured,
/// no hive name to log in as, nothing published for this agent yet. That is
/// the ordinary state of an agent created before the swarm minted identities,
/// and an empty list keeps its container starting exactly as it does today.
/// The loud failure belongs inside the container, where the option that asked
/// for an identity is set; refusing to write an nspawn config here would take
/// down agents that never wanted one.
///
/// Already-staged files survive a failed collection. A store that is briefly
/// unreachable then delivers the identity from the last successful read rather
/// than none at all — the same call as the queue reader's, which leaves its
/// files alone when the store says nothing.
pub async fn stage(agent: &str) -> Vec<CredentialMount> {
let dir = crate::paths::agent_identity_dir(agent);
match collect(agent, &dir).await {
Ok(()) => {}
Err(Skipped::NotConfigured(why)) => {
tracing::info!(%agent, "no swarm store identity for this agent: {why}");
}
Err(Skipped::Failed(e)) => {
// `warn`, not `error`: whatever is already staged still goes in,
// and the container's own check is what decides whether that is
// good enough.
tracing::warn!(
%agent, error = ?e,
"collecting this agent's store identity failed; delivering whatever was staged before"
);
}
}
mounts(&dir)
}
/// Why a collection produced nothing, split by whether an operator has
/// anything to fix: a hive with no store wired up says so once at `info`,
/// a hive that has one and could not read says so at `warn`.
enum Skipped {
NotConfigured(&'static str),
Failed(anyhow::Error),
}
/// Read the published identity and write it into `dir`.
async fn collect(agent: &str, dir: &Path) -> Result<(), Skipped> {
// The hive's name is the cert-auth role it logs in as: `glue-bao-tls.nix`
// mints this host's client certificate with the hive name as its CN and a
// bao cert role matches on CN, so the two share a name by construction —
// the same reasoning `swarm_status::handle_credential_notice` records for
// the other read this hive does on an agent's behalf.
let Some(hive) = crate::container_view::hive_swarm_names().0 else {
return Err(Skipped::NotConfigured(
"HYPERHIVE_HIVE_NAME is unset, so this hive has no role to log in as",
));
};
// Checked before connecting so a host with no store at all costs a lookup
// rather than a timeout, and reports the cause instead of the symptom.
if std::env::var_os(client::ENV_ADDR).is_none_or(|v| v.is_empty()) {
return Err(Skipped::NotConfigured(
"this hive has no swarm secret store configured",
));
}
let path = mtls::identity_path(agent).map_err(|e| Skipped::Failed(e.into()))?;
let store = SecretStore::from_env(&hive)
.await
.map_err(|e| Skipped::Failed(anyhow::Error::new(e).context("connecting to the store")))?;
let credential: mtls::Credential = store.read(&path).await.map_err(|e| {
Skipped::Failed(anyhow::Error::new(e).context(format!("reading {path} from the store")))
})?;
write_identity(dir, &credential).map_err(Skipped::Failed)
}
/// Write the two private files, replacing whatever was there.
///
/// `0600` from the moment each file exists rather than after a `chmod`: the
/// key is written through a handle opened with that mode, so there is no
/// instant where it sits on disk world-readable.
fn write_identity(dir: &Path, credential: &mtls::Credential) -> anyhow::Result<()> {
use anyhow::Context as _;
use std::io::Write as _;
use std::os::unix::fs::{OpenOptionsExt as _, PermissionsExt as _};
std::fs::create_dir_all(dir).with_context(|| format!("create {}", dir.display()))?;
// The directory too: its own mode is what stops anything but `hive-core`
// and root from reaching a file inside it by name.
std::fs::set_permissions(dir, std::fs::Permissions::from_mode(0o700))
.with_context(|| format!("chmod 0700 {}", dir.display()))?;
for (name, bytes) in [
(CERT_CREDENTIAL, credential.cert.as_bytes()),
(KEY_CREDENTIAL, credential.key.as_bytes()),
] {
let file = dir.join(name);
let mut handle = std::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.mode(0o600)
.open(&file)
.with_context(|| format!("open {}", file.display()))?;
// The error carries the path and never the bytes — the one line in
// this function where a value could otherwise be interpolated.
handle
.write_all(bytes)
.with_context(|| format!("write {}", file.display()))?;
}
// Public material, and `0644` says so — it is the same bundle every host
// unit in the tree already points `BAO_CACERT` at. Absent means the
// container falls back to its own trust store, which is what a deployment
// with a real CA wants.
let server_ca = std::env::var_os(client::ENV_CACERT)
.map(PathBuf::from)
.filter(|p| p.is_file());
if let Some(source) = server_ca {
let staged = dir.join(SERVER_CA_CREDENTIAL);
std::fs::copy(&source, &staged)
.with_context(|| format!("copy {} to {}", source.display(), staged.display()))?;
std::fs::set_permissions(&staged, std::fs::Permissions::from_mode(0o644))
.with_context(|| format!("chmod 0644 {}", staged.display()))?;
}
Ok(())
}
/// The credentials to forward, given what is actually staged.
///
/// Both private files or neither: a container that received a certificate
/// without its key would start a login check that cannot pass, and the
/// resulting failure would name the wrong problem.
fn mounts(dir: &Path) -> Vec<CredentialMount> {
let cert = dir.join(CERT_CREDENTIAL);
let key = dir.join(KEY_CREDENTIAL);
if !cert.is_file() || !key.is_file() {
return Vec::new();
}
let mut out = vec![
CredentialMount {
name: CERT_CREDENTIAL.to_owned(),
host_path: cert.to_string_lossy().into_owned(),
},
CredentialMount {
name: KEY_CREDENTIAL.to_owned(),
host_path: key.to_string_lossy().into_owned(),
},
];
// Absent means the container verifies the store's listener against its own
// trust store, which is what a deployment with a real CA wants — the same
// optional shape every `BAO_CACERT` site in the nix tree already has.
let ca = dir.join(SERVER_CA_CREDENTIAL);
if ca.is_file() {
out.push(CredentialMount {
name: SERVER_CA_CREDENTIAL.to_owned(),
host_path: ca.to_string_lossy().into_owned(),
});
}
out
}
#[cfg(test)]
mod tests {
use super::{CERT_CREDENTIAL, KEY_CREDENTIAL, SERVER_CA_CREDENTIAL, mounts, write_identity};
use swarm_secret_client::mtls;
fn credential() -> mtls::Credential {
mtls::Credential {
cert: "-----BEGIN CERTIFICATE-----\nleaf\n".to_owned(),
key: "-----BEGIN PRIVATE KEY-----\nSUPER-SECRET\n".to_owned(),
ca: "-----BEGIN CERTIFICATE-----\nauthority\n".to_owned(),
}
}
/// The property the whole delivery rests on: the key lands unreadable to
/// anything but its owner, so the only way into the container is the
/// credential mechanism.
#[test]
fn the_staged_key_is_private_the_moment_it_exists() {
use std::os::unix::fs::PermissionsExt as _;
let dir = tempfile::tempdir().expect("tempdir");
let staged = dir.path().join("iris");
write_identity(&staged, &credential()).expect("writes");
for name in [CERT_CREDENTIAL, KEY_CREDENTIAL] {
let mode = std::fs::metadata(staged.join(name))
.expect("staged")
.permissions()
.mode();
assert_eq!(mode & 0o777, 0o600, "{name} must be 0600, got {mode:o}");
}
let dir_mode = std::fs::metadata(&staged)
.expect("staged dir")
.permissions()
.mode();
assert_eq!(dir_mode & 0o777, 0o700, "got {dir_mode:o}");
}
/// Re-staging is what happens on every rebuild, so a second write must
/// leave the file holding only the new value — a shorter certificate
/// written over a longer one without truncation is a file that parses as
/// neither.
#[test]
fn re_staging_replaces_rather_than_overwrites_in_place() {
let dir = tempfile::tempdir().expect("tempdir");
let staged = dir.path().join("iris");
let mut long = credential();
long.cert = "-".repeat(400);
write_identity(&staged, &long).expect("writes");
write_identity(&staged, &credential()).expect("re-writes");
let on_disk = std::fs::read_to_string(staged.join(CERT_CREDENTIAL)).expect("read back");
assert_eq!(on_disk, credential().cert);
}
/// Half a delivery is worse than none: the container's check would fail
/// on the missing key and report a login problem instead of a delivery
/// one. Same both-or-neither rule the queue credential states.
#[test]
fn a_certificate_without_its_key_forwards_nothing() {
let dir = tempfile::tempdir().expect("tempdir");
assert!(mounts(dir.path()).is_empty());
std::fs::write(dir.path().join(CERT_CREDENTIAL), "leaf").expect("write cert");
assert!(mounts(dir.path()).is_empty());
}
/// The store's own authority rides along when there is one, so a
/// container verifying a self-signed listener has something to verify
/// against — and the absence of it is not a reason to withhold the
/// identity.
#[test]
fn a_staged_server_authority_is_forwarded_beside_the_identity() {
let dir = tempfile::tempdir().expect("tempdir");
std::fs::write(dir.path().join(CERT_CREDENTIAL), "leaf").expect("write cert");
std::fs::write(dir.path().join(KEY_CREDENTIAL), "private").expect("write key");
let without: Vec<String> = mounts(dir.path()).into_iter().map(|c| c.name).collect();
assert_eq!(without, [CERT_CREDENTIAL, KEY_CREDENTIAL]);
std::fs::write(dir.path().join(SERVER_CA_CREDENTIAL), "authority").expect("write ca");
let with: Vec<String> = mounts(dir.path()).into_iter().map(|c| c.name).collect();
assert_eq!(
with,
[CERT_CREDENTIAL, KEY_CREDENTIAL, SERVER_CA_CREDENTIAL]
);
}
/// The names are the agreement with `nix/agent-modules/bao.nix`, and
/// `hive-priv` refuses any of them that could close the flag it is
/// interpolated into.
#[test]
fn every_credential_id_is_one_hive_priv_will_accept() {
for name in [CERT_CREDENTIAL, KEY_CREDENTIAL, SERVER_CA_CREDENTIAL] {
assert!(
name.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
"{name} must be [A-Za-z0-9_-] only"
);
}
}
}

9
hive-c0re/src/lifecycle/host_config.rs
View file

@ -253,7 +253,14 @@ async fn set_nspawn_flags(
// here, and forwarded *as a credential* rather than a bind for the
// reason `queue_agent_credentials` records.
let queue_credential_dir = std::env::var_os(QUEUE_CREDENTIAL_DIR_ENV).map(PathBuf::from);
let load_creds = queue_agent_credentials(agent_name, queue_credential_dir.as_deref());
let mut load_creds = queue_agent_credentials(agent_name, queue_credential_dir.as_deref());
// The agent's own identity *at* the swarm secret store, collected under
// this hive's certificate and forwarded the same way and for the same
// reason. Staged here rather than at spawn because this is the one place
// that writes the container's credential list — see
// `super::agent_identity` for the whole hop.
load_creds.extend(super::agent_identity::stage(agent_name).await);
let mut binds: Vec<BindMount> = vec![
BindMount {

1
hive-c0re/src/lifecycle/mod.rs
View file

@ -1,5 +1,6 @@
//! `nixos-container` lifecycle + per-agent config flake generation.
mod agent_identity;
mod git;
mod host_config;
mod setup;

11
hive-c0re/src/meta.rs
View file

@ -706,6 +706,11 @@ const FORWARDED_VARS: &[&str] = &[
// address, which inside a container is the agent itself.
"HIVE_AGENT_NATS_URL",
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
// Where the swarm secret store listens. The hive's own `BAO_ADDR` value,
// forwarded under a separate name so the container never inherits the
// hive's store environment wholesale — the address is the only part of it
// an agent may have, the certificate beside it being the hive's own.
"HIVE_AGENT_BAO_ADDR",
// Where an agent's `swarm-logs` queries, computed by the host for the
// same reason as the two above: it is a gateway address, and a container
// cannot derive it from anything it holds.
@ -747,6 +752,12 @@ const FORWARDED_VAR_OPTIONS: &[(&str, &str)] = &[
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
"services.hyperhive.agent.queue.tokenEndpoint",
),
// Read at build time to decide whether the in-container identity check is
// generated at all — see `nix/agent-modules/bao.nix`. Both halves wired
// for this map's own reason: the unit bakes the address into its own
// environment, so an agent whose option says no store and whose env says
// otherwise would have a check that is simply not there.
("HIVE_AGENT_BAO_ADDR", "services.hyperhive.agent.bao.addr"),
// Read at build time to decide whether `swarm-logs` is installed at all —
// see `nix/agent-modules/logs.nix`. Both halves wired for this map's own
// reason: an agent whose option says no log store and whose env says

26
hive-c0re/src/paths.rs
View file

@ -102,6 +102,32 @@ pub fn forge_repo_creation_disabled_marker(name: &str) -> PathBuf {
forge_dir().join(format!("repo-creation-disabled-{name}"))
}
/// `agent-identity/` — one subdir per agent holding the store identity this
/// hive collected for it. Staging only: the files exist so `systemd-nspawn`
/// has something to `--load-credential` from, and nothing on the host ever
/// reads them back.
///
/// Under the state root rather than `/run` deliberately. The forward happens
/// when a container's nspawn config is (re)written, and a container can be
/// restarted long after that; material that vanished with `/run` would leave
/// a boot where the agent's identity is simply absent and the only symptom is
/// a unit inside the container refusing to start.
#[must_use]
pub fn agent_identity_root() -> PathBuf {
state_root().join("agent-identity")
}
/// `agent-identity/<name>` — one agent's staged store identity.
///
/// ⚠️ Deliberately **not** under `agents/<name>`: that tree is bind-mounted
/// into the container (and a parent's, for a child), so a private key placed
/// there would be readable by the agent as a plain file, bypassing the
/// credential mechanism that exists to control exactly that.
#[must_use]
pub fn agent_identity_dir(name: &str) -> PathBuf {
agent_identity_root().join(name)
}
/// `matrix/` — host-side matrix provisioning state (admin token, hive
/// Space room id, per-agent password creds). The shared registration
/// token is bind-mounted into the tuwunel container via nix and stays

206
nix/agent-modules/bao.nix Normal file
View file

@ -0,0 +1,206 @@
# This agent's own identity at the swarm secret store, and the check that
# proves it works.
#
# `swarm-controller` mints the leaf at agent creation and publishes it; this
# agent's hive collects it under the hive's own certificate and hands it in as
# systemd credentials (`hive_c0re::lifecycle::agent_identity`). Nothing here
# fetches anything from the store — by the time this container boots, its
# identity is already inside it.
#
# ⚠️ The identity arrives as credentials and NOT as a bind mount, and the mode
# is why: the host file is `0600` to the hive daemon and this unit runs as the
# unprivileged agent user. nspawn's `--load-credential` is read by the
# container manager as root and re-exposed under this unit's own `User=`; a
# bind would deliver a private key this user cannot open. Same answer, same
# reason, as ./queue.nix's credential pair.
#
# This is the courier's other end: the hive-side collector has no purpose
# without it, so the two ship together and neither is reachable alone.
#
# 🩸 This unit fails LOUDLY where the hive-side readers degrade quietly, and
# that is the opposite default on purpose. A missing OIDC client secret means
# a hive whose publisher has yet to run; a missing or refused certificate
# means an agent that believes it can reach the store and cannot, which every
# later pull would report as its own unrelated failure. The one place that
# knows the real cause is the login itself, so this is where it is said.
{
pkgs,
lib,
config,
...
}:
let
cfg = config.services.hyperhive.agent.bao;
# This container's agent name. The same string the hive published the
# identity under, because the agent's unix user is named for the agent —
# see ./user.nix.
agentName = config.services.hyperhive.agent.user.name;
# The three ids `hive_c0re::lifecycle::agent_identity` forwards under.
# Neither side can discover the other's spelling, so a rename is a rename
# there too.
certCredential = "hive-agent-bao-cert";
keyCredential = "hive-agent-bao-key";
serverCaCredential = "hive-agent-bao-server-ca";
unitName = "hive-agent-bao-identity";
# Where the identity lives in the store, spelled from the same pieces the
# publisher uses. `swarm_secret_client::mtls::identity_path` builds
# `swarm/agents/<agent>/bao-mtls` and `path::MOUNT` is `secret`; this
# literal is the nix half of that one agreement, exactly as
# ../host-modules/glue-queue-agent-credential.nix spells its own.
identityPath = "secret/swarm/agents/${agentName}/bao-mtls";
# The address is the whole switch — no separate `enable`, the same shape
# ./queue.nix and ./logs.nix gate themselves with. A hive that has a store
# forwards its address and every agent on it gets the check; a hive that has
# none forwards nothing and no agent does. An `enable` beside it would be a
# knob whose only correct setting is whatever the address already says, and
# its default would decide whether the hive-side courier delivers into a
# container that reads what it is given or into one that never looks.
configured = cfg.addr != null;
in
{
options.services.hyperhive.agent.bao = {
addr = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "https://bao.example.com:8200";
description = ''
Where the swarm secret store listens, as this container reaches it.
Set by the generated meta flake from the host's own `BAO_ADDR`, which
is the address this hive already uses, and setting it is what generates
`${unitName}.service`: at boot that unit logs in with the certificate
its hive delivered and reads this agent's own path back, failing if
either step does not succeed. `systemctl status ${unitName}` inside the
container is then the answer to "can this agent reach the store as
itself", which nothing else in the tree reports.
`null` means the hive was given no store. No unit is generated then,
because an agent whose swarm never minted an identity has nothing to
log in with, and a failed unit at every boot would say that in the
loudest possible way about a deployment that never asked for it.
'';
};
};
config = lib.mkIf configured {
systemd.services.${unitName} = {
description = "prove this agent can authenticate to the swarm secret store as itself";
after = [ "network.target" ];
wantedBy = [ "multi-user.target" ];
path = [
pkgs.openbao
pkgs.coreutils
];
# Sized for a store that comes up around the same time this container
# does, not for one that is sealed: a few short attempts cover the race,
# and a longer window would only delay the report of a real failure.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec` times the burst.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
TimeoutStartSec = 30;
Restart = "on-failure";
RestartSec = 15;
User = agentName;
Group = agentName;
# Bare ids, no paths: the terse `LoadCredential=` form that inherits a
# credential the service *manager* received, which is what the
# container manager passed in. ./queue.nix states the same shape.
LoadCredential = [
certCredential
keyCredential
serverCaCredential
];
};
environment = {
BAO_ADDR = cfg.addr;
# `%d` is `$CREDENTIALS_DIRECTORY`, per-unit and owned by `User=`.
BAO_CLIENT_CERT = "%d/${certCredential}";
BAO_CLIENT_KEY = "%d/${keyCredential}";
};
script = ''
set -euo pipefail
# Absent credentials are the first thing checked, because every later
# message would blame the store for a delivery that never happened.
# 🩸 The files are named and never read here: what went wrong is a
# property of the path, and the bytes at it are a private key.
missing=
for id in ${lib.escapeShellArg certCredential} ${lib.escapeShellArg keyCredential}; do
if [ ! -s "$CREDENTIALS_DIRECTORY/$id" ]; then
missing="$missing $id"
fi
done
if [ -n "$missing" ]; then
echo "this agent has no store identity: the hive delivered no$missing." >&2
echo "swarm-controller publishes it at ${identityPath} when the agent is created, and the hive collects it from there." >&2
exit 1
fi
# Only when one was delivered. Absent means the container verifies the
# store's listener against its own trust store, which is what a
# deployment with a real CA wants; pointing BAO_CACERT at a file that
# is not there would fail the handshake and name the wrong cause.
if [ -s "$CREDENTIALS_DIRECTORY/${serverCaCredential}" ]; then
export BAO_CACERT="$CREDENTIALS_DIRECTORY/${serverCaCredential}"
fi
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# No `name=`: a cert role pins both its authority and the common name
# it accepts, so this agent's leaf matches its own role and no other.
# Naming the role here would be a third copy of a string
# `swarm_secret_client::policy::agent_object_name` already owns.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "this agent's certificate was refused by the swarm secret store at $BAO_ADDR." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
# A token is not yet an answer: the login proves the certificate, this
# proves the policy attached to it. Reading this agent's own identity
# back is the smallest read its document grants, and it is the same
# check `swarm-controller` runs against the leaf before it reports the
# creation done — so a policy that drifted apart from the path fails
# here rather than in whichever pull needed it first.
#
# ⚠️ Output discarded, not printed: the field is a certificate and the
# object beside it is a private key. Nothing about this check needs a
# value, only whether the read succeeded.
if ! bao kv get -field=cert ${lib.escapeShellArg identityPath} >/dev/null 2>"$err"; then
echo "this agent logged in to the swarm secret store but cannot read ${identityPath}, so its policy does not cover its own path." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
echo "authenticated to the swarm secret store at $BAO_ADDR as ${agentName} and read ${identityPath}."
'';
};
};
}

1
nix/agent-modules/default.nix
View file

@ -40,6 +40,7 @@ in
{
imports = [
./agent-service.nix
./bao.nix
./bash-env.nix
./claude-settings.nix
./dashboard-links.nix

6
nix/host-modules/hive-c0re/environment.nix
View file

@ -334,6 +334,12 @@ in
# handshake, naming neither.
lib.optionalAttrs haveBaoClientIdentity {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
# The same address again under an agent-facing name, forwarded into every
# container's module block by `meta::render_flake`. Deliberately not the
# `BAO_ADDR` line itself: a container that inherited this hive's store
# environment wholesale would inherit the hive's certificate paths with it,
# and those name a credential no agent may present.
HIVE_AGENT_BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
# `%d`, not the paths themselves: the key is `0600` root-owned and this
# daemon runs as hive-core, so it never gets read access to the original.
# See the LoadCredential in ./default.nix.

134
nix/module-eval.nix
View file

@ -64,10 +64,12 @@ let
# the meta flake hands a container, so what this evaluates is what an
# agent gets.
#
# Note `hyperhive`, not `services.hyperhive`: an agent container's options
# live at the top level.
agent =
extra:
# Takes a whole module rather than a settings attrset, so a fixture can
# reach either spelling of the agent tier. `user.name` is the one option
# with no usable default, set here at its real path and at `mkDefault` so a
# fixture naming its own agent still wins.
agentWith =
module:
(nixosSystem {
system = pkgs.stdenv.hostPlatform.system;
modules = [
@ -79,11 +81,20 @@ let
};
boot.loader.grub.enable = false;
system.stateVersion = "25.11";
hyperhive = lib.recursiveUpdate { user.name = "a1"; } extra;
services.hyperhive.agent.user.name = lib.mkDefault "a1";
}
module
];
}).config;
# Note `hyperhive`, not `services.hyperhive`: the agent tier's options moved
# under `services.hyperhive.agent`, and ../agent-modules/renamed-options.nix
# keeps the top-level spelling reaching them. ⚠️ That shim covers the
# options that existed when the tier moved and nothing since, so a fixture
# for an option added afterwards has to go through [`agentWith`] and name
# the real path.
agent = extra: agentWith { hyperhive = extra; };
allLocal = hive { deploy.singleHostSwarm = true; };
bare = hive { };
withCi = hive { deploy.forgejo.ci.enable = true; };
@ -605,6 +616,12 @@ let
};
agentNoQueue = agent { };
# The agent side of the swarm secret store. The address is the whole switch —
# it is both what generates the login check and what that check points at —
# so it and the empty fixture beside it are the two arms worth having.
agentBao = agentWith { services.hyperhive.agent.bao.addr = "https://bao.t.local:8200"; };
agentNoBao = agentWith { };
# The memory-pressure pair. `claudeMemoryMaxBytes` is the container's own
# cap, rendered per agent by meta.rs — the capped arm is the one every
# real deploy gets, the uncapped arm is a hive that set `infinity` or a
@ -636,6 +653,7 @@ let
};
agentHarness = machine: machine.systemd.services.hive-agent;
agentSubagentDaemon = machine: machine.systemd.services.hive-subagent-daemon;
agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity;
agentSettings = machine: machine.services.opentelemetry-collector.settings;
# This hive's own collector, which is a HOST service — unlike the swarm
@ -1766,6 +1784,112 @@ let
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_SECRET_FILE)
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_ID_FILE);
}
{
# The three ids `hive_c0re::lifecycle::agent_identity` forwards under.
# Neither end can discover the other's spelling, and a mismatch is a
# credential that is simply not there — which this unit then reports as
# a hive that delivered nothing.
name = "an agent with the store enabled imports every half of its identity";
ok =
let
c = (agentBaoIdentity agentBao).serviceConfig.LoadCredential;
in
builtins.elem "hive-agent-bao-cert" c
&& builtins.elem "hive-agent-bao-key" c
&& builtins.elem "hive-agent-bao-server-ca" c;
}
{
# `%d` and not a path under the agent's state dir, for the reason the
# queue arm above gives: the host file is `0600` to the hive daemon, so
# the only copy this unprivileged unit can open is the one systemd puts
# in its own credentials directory. The address is the option's value
# rather than a literal that agrees with it today.
name = "the identity check presents its certificate out of the credentials directory";
ok =
let
e = (agentBaoIdentity agentBao).environment;
in
e.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert"
&& e.BAO_CLIENT_KEY == "%d/hive-agent-bao-key"
&& e.BAO_ADDR == agentBao.services.hyperhive.agent.bao.addr;
}
{
# Same 403-not-a-miss reason as the hive-side readers: the path
# `swarm_secret_client::mtls::identity_path` builds is the one this
# agent's own policy stanza covers, and a path outside it is refused
# however correct it looks. Built from the agent's own name rather than
# from a literal, because the name is what makes it this agent's path
# and not some other agent's.
name = "the identity check reads the agent's own path";
ok =
let
m = agentBao;
name = m.services.hyperhive.agent.user.name;
in
lib.hasInfix "secret/swarm/agents/${name}/bao-mtls" (agentBaoIdentity m).script;
}
{
# The whole point of the unit, and the thing a quieter default would
# undo: every arm of the check ends the unit non-zero, so an agent that
# cannot authenticate as itself says so at boot instead of at whichever
# pull needed the store first.
name = "the identity check fails the unit rather than degrading";
ok =
let
u = agentBaoIdentity agentBao;
in
lib.hasInfix "exit 1" u.script
&& !(lib.hasInfix "exit 0" u.script)
&& u.serviceConfig.Restart == "on-failure";
}
{
# Nothing about the identity may be printed, and the read-back is where
# that could slip: `bao kv get` on this path answers with certificate
# material, and the object beside it is a private key. The check needs
# only whether the read succeeded.
#
# The path goes through `lib.escapeShellArg` here for the same reason the
# module passes it through one — that helper decides whether an argument
# needs quotes at all, and this one (only `[a-z0-9/-]`) comes back bare.
# Spelling the quotes in by hand asserts a rendering nixpkgs chooses
# rather than the redirect this property is about.
name = "the identity check discards what it reads back";
ok =
let
m = agentBao;
name = m.services.hyperhive.agent.user.name;
arg = lib.escapeShellArg "secret/swarm/agents/${name}/bao-mtls";
in
lib.hasInfix "bao kv get -field=cert ${arg} >/dev/null" (agentBaoIdentity m).script;
}
{
# The absence arm, and what makes the four above able to fail. An agent
# whose swarm never minted an identity has nothing to log in with, and a
# failed unit at every boot would be the loudest possible statement
# about a deployment that never asked for one.
name = "an agent told no store address runs no identity check";
ok = !(agentNoBao.systemd.services ? hive-agent-bao-identity);
}
{
# Where the store is and where the agent is told it is, one agreement
# spanning two modules. Asserted against the hive's own `BAO_ADDR`
# rather than a literal, because an agent pointed at a different
# spelling of the same store presents a certificate to a listener whose
# name it cannot verify.
name = "an agent is told the same store address its hive uses";
ok =
let
e = allLocal.systemd.services.hive-c0re.environment;
in
e.HIVE_AGENT_BAO_ADDR == e.BAO_ADDR;
}
{
# A hive with no certificate of its own can collect no agent's identity,
# so forwarding an address would name a store nothing in the container
# can reach. The same gate the `BAO_*` pair beside it sits behind.
name = "a hive with no store identity forwards no store address to its agents";
ok = !(bare.systemd.services.hive-c0re.environment ? HIVE_AGENT_BAO_ADDR);
}
{
# Two thirds of the container's cap, and a SOFT ceiling: the daemon's
# cgroup holds every nested claude, so `MemoryHigh=` throttles the