diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index 1d4b8c9d..9aa73a70 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -51,7 +51,7 @@ strategy for every credential, including the mTLS leaf. | store path | minter | reader — pulls at runtime, holds in memory | renewal | | -------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- | -------------- | | `swarm/agents//matrix/` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | -| `swarm/agents//bao-mtls` | `swarm-controller`, at agent creation | the agent's **hive**, under the hive's own certificate, which hands it into the container | must be stated | +| `swarm/agents//bao-mtls` | `swarm-controller`, at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | must be stated | | `swarm/hives//matrix/appservice-token` | one minter, on the authelia host | the hive process that presents the token to its homeserver, under the hive's own certificate | must be stated | | `swarm/hives//queue/agent` | authelia | the agent container presenting the OIDC client to the swarm queue, under its own certificate | must be stated | | `swarm/services//oidc/client` | authelia | the service process that presents the client secret, under the certificate of the host it runs on | must be stated | @@ -67,6 +67,27 @@ carries it, and no hive ever needs the capability to mint an identity. `swarm-controller` proves the leaf it publishes before the creation job reports success, by logging in with it and reading the row back. +**Who reads that row, and what happens to it.** `hive-c0re` reads it every +time it writes an agent's container configuration +(`lifecycle::agent_identity`), stages the certificate and its key `0600` +outside every bind-mounted tree, and passes both to the container as systemd +credentials — the same mechanism, and for the same mode reason, as the +per-hive queue secret. A bind mount would hand the agent's unprivileged user +a file it lacks the rights to open; the container manager reads a credential +as root and re-exposes it under the consuming unit's own user. + +Inside the container, `hive-agent-bao-identity.service` logs in with that +certificate and reads this row back before reporting success, so an agent +locked out of its own identity says so at boot rather than at whichever pull +needed the store first. The unit exists whenever +`services.hyperhive.agent.bao.addr` has a value, which the hive's meta flake +sets from its own store address — the same all-or-nothing gate the per-hive +queue credential beside it uses, and the reason the delivery above never lands +in a container with nothing to read it. It fails loudly where the hive-side +readers degrade quietly, which is deliberate: a missing queue secret means a +swarm whose publisher has yet to run, while a refused certificate means an +agent that believes it reaches the store and never does. + ## Progressive enhancement New functionality has to match this shape immediately — no PR introducing a diff --git a/hive-c0re/src/lifecycle/agent_identity.rs b/hive-c0re/src/lifecycle/agent_identity.rs new file mode 100644 index 00000000..9fff4044 --- /dev/null +++ b/hive-c0re/src/lifecycle/agent_identity.rs @@ -0,0 +1,306 @@ +//! The courier hop: this hive collects an agent's own store identity and +//! stages it for delivery into that agent's container. +//! +//! `swarm-controller` mints the leaf at agent-create and publishes it at +//! `swarm/agents//bao-mtls`; this reads it back under **the hive's** +//! certificate — the hive is a principal the store already knows — and lands +//! the two files `systemd-nspawn` forwards as credentials. The agent is never +//! asked to authenticate in order to obtain the thing it authenticates with: +//! by the time it runs, its identity is already inside it. +//! +//! **A credential and not a bind mount**, the same answer and the same reason +//! as the queue secret beside it in [`super::host_config`]: the files are +//! `0600` to `hive-core` and the units that use them run as the unprivileged +//! agent user, so a bind would deliver a private key that user cannot open. +//! The container manager reads a `--load-credential` source as root and +//! re-exposes it under the consuming unit's own `User=`. +//! +//! 🩸 Nothing here ever formats the credential's fields. The value moves +//! store → memory → file and is never an argument to a log macro; the +//! [`mtls::Credential`] type's hand-written `Debug` redacts the key for the +//! one place a value could still reach a line — an error context chain. +//! +//! The consumer sits inside the container: `nix/agent-modules/bao.nix`'s +//! `hive-agent-bao-identity.service` logs in with what lands here and fails +//! loudly when it cannot. + +use std::path::{Path, PathBuf}; + +use hive_priv_sock::CredentialMount; +use swarm_secret_client::{SecretStore, client, mtls}; + +/// systemd credential ids the agent's identity arrives under inside the +/// container. `nix/agent-modules/bao.nix` spells the same three in its +/// `LoadCredential=`; neither side can discover the other's, so a rename here +/// is a rename there. +/// +/// No `.pem` suffix: `hive-priv` refuses a credential name containing a dot, +/// since the name is interpolated into `--load-credential=:`. +const CERT_CREDENTIAL: &str = "hive-agent-bao-cert"; +const KEY_CREDENTIAL: &str = "hive-agent-bao-key"; +/// The authority the **store's own listener** is verified against — not the +/// authority that issued the agent's leaf, which no reader in the container +/// needs and which is therefore not delivered at all. +/// +/// Copied into the staging dir rather than forwarded from the hive's own +/// `BAO_CACERT`: that path is this daemon's per-unit credential directory, +/// which systemd removes when `hive-c0re` stops. An nspawn config pointing at +/// it would leave a container unable to start whenever the daemon was down. +const SERVER_CA_CREDENTIAL: &str = "hive-agent-bao-server-ca"; + +/// Collect `agent`'s store identity and return the credentials to forward. +/// +/// Empty whenever this hive cannot answer the question — no store configured, +/// no hive name to log in as, nothing published for this agent yet. That is +/// the ordinary state of an agent created before the swarm minted identities, +/// and an empty list keeps its container starting exactly as it does today. +/// The loud failure belongs inside the container, where the option that asked +/// for an identity is set; refusing to write an nspawn config here would take +/// down agents that never wanted one. +/// +/// Already-staged files survive a failed collection. A store that is briefly +/// unreachable then delivers the identity from the last successful read rather +/// than none at all — the same call as the queue reader's, which leaves its +/// files alone when the store says nothing. +pub async fn stage(agent: &str) -> Vec { + let dir = crate::paths::agent_identity_dir(agent); + match collect(agent, &dir).await { + Ok(()) => {} + Err(Skipped::NotConfigured(why)) => { + tracing::info!(%agent, "no swarm store identity for this agent: {why}"); + } + Err(Skipped::Failed(e)) => { + // `warn`, not `error`: whatever is already staged still goes in, + // and the container's own check is what decides whether that is + // good enough. + tracing::warn!( + %agent, error = ?e, + "collecting this agent's store identity failed; delivering whatever was staged before" + ); + } + } + mounts(&dir) +} + +/// Why a collection produced nothing, split by whether an operator has +/// anything to fix: a hive with no store wired up says so once at `info`, +/// a hive that has one and could not read says so at `warn`. +enum Skipped { + NotConfigured(&'static str), + Failed(anyhow::Error), +} + +/// Read the published identity and write it into `dir`. +async fn collect(agent: &str, dir: &Path) -> Result<(), Skipped> { + // The hive's name is the cert-auth role it logs in as: `glue-bao-tls.nix` + // mints this host's client certificate with the hive name as its CN and a + // bao cert role matches on CN, so the two share a name by construction — + // the same reasoning `swarm_status::handle_credential_notice` records for + // the other read this hive does on an agent's behalf. + let Some(hive) = crate::container_view::hive_swarm_names().0 else { + return Err(Skipped::NotConfigured( + "HYPERHIVE_HIVE_NAME is unset, so this hive has no role to log in as", + )); + }; + // Checked before connecting so a host with no store at all costs a lookup + // rather than a timeout, and reports the cause instead of the symptom. + if std::env::var_os(client::ENV_ADDR).is_none_or(|v| v.is_empty()) { + return Err(Skipped::NotConfigured( + "this hive has no swarm secret store configured", + )); + } + + let path = mtls::identity_path(agent).map_err(|e| Skipped::Failed(e.into()))?; + let store = SecretStore::from_env(&hive) + .await + .map_err(|e| Skipped::Failed(anyhow::Error::new(e).context("connecting to the store")))?; + let credential: mtls::Credential = store.read(&path).await.map_err(|e| { + Skipped::Failed(anyhow::Error::new(e).context(format!("reading {path} from the store"))) + })?; + + write_identity(dir, &credential).map_err(Skipped::Failed) +} + +/// Write the two private files, replacing whatever was there. +/// +/// `0600` from the moment each file exists rather than after a `chmod`: the +/// key is written through a handle opened with that mode, so there is no +/// instant where it sits on disk world-readable. +fn write_identity(dir: &Path, credential: &mtls::Credential) -> anyhow::Result<()> { + use anyhow::Context as _; + use std::io::Write as _; + use std::os::unix::fs::{OpenOptionsExt as _, PermissionsExt as _}; + + std::fs::create_dir_all(dir).with_context(|| format!("create {}", dir.display()))?; + // The directory too: its own mode is what stops anything but `hive-core` + // and root from reaching a file inside it by name. + std::fs::set_permissions(dir, std::fs::Permissions::from_mode(0o700)) + .with_context(|| format!("chmod 0700 {}", dir.display()))?; + + for (name, bytes) in [ + (CERT_CREDENTIAL, credential.cert.as_bytes()), + (KEY_CREDENTIAL, credential.key.as_bytes()), + ] { + let file = dir.join(name); + let mut handle = std::fs::OpenOptions::new() + .write(true) + .create(true) + .truncate(true) + .mode(0o600) + .open(&file) + .with_context(|| format!("open {}", file.display()))?; + // The error carries the path and never the bytes — the one line in + // this function where a value could otherwise be interpolated. + handle + .write_all(bytes) + .with_context(|| format!("write {}", file.display()))?; + } + + // Public material, and `0644` says so — it is the same bundle every host + // unit in the tree already points `BAO_CACERT` at. Absent means the + // container falls back to its own trust store, which is what a deployment + // with a real CA wants. + let server_ca = std::env::var_os(client::ENV_CACERT) + .map(PathBuf::from) + .filter(|p| p.is_file()); + if let Some(source) = server_ca { + let staged = dir.join(SERVER_CA_CREDENTIAL); + std::fs::copy(&source, &staged) + .with_context(|| format!("copy {} to {}", source.display(), staged.display()))?; + std::fs::set_permissions(&staged, std::fs::Permissions::from_mode(0o644)) + .with_context(|| format!("chmod 0644 {}", staged.display()))?; + } + Ok(()) +} + +/// The credentials to forward, given what is actually staged. +/// +/// Both private files or neither: a container that received a certificate +/// without its key would start a login check that cannot pass, and the +/// resulting failure would name the wrong problem. +fn mounts(dir: &Path) -> Vec { + let cert = dir.join(CERT_CREDENTIAL); + let key = dir.join(KEY_CREDENTIAL); + if !cert.is_file() || !key.is_file() { + return Vec::new(); + } + let mut out = vec![ + CredentialMount { + name: CERT_CREDENTIAL.to_owned(), + host_path: cert.to_string_lossy().into_owned(), + }, + CredentialMount { + name: KEY_CREDENTIAL.to_owned(), + host_path: key.to_string_lossy().into_owned(), + }, + ]; + // Absent means the container verifies the store's listener against its own + // trust store, which is what a deployment with a real CA wants — the same + // optional shape every `BAO_CACERT` site in the nix tree already has. + let ca = dir.join(SERVER_CA_CREDENTIAL); + if ca.is_file() { + out.push(CredentialMount { + name: SERVER_CA_CREDENTIAL.to_owned(), + host_path: ca.to_string_lossy().into_owned(), + }); + } + out +} + +#[cfg(test)] +mod tests { + use super::{CERT_CREDENTIAL, KEY_CREDENTIAL, SERVER_CA_CREDENTIAL, mounts, write_identity}; + use swarm_secret_client::mtls; + + fn credential() -> mtls::Credential { + mtls::Credential { + cert: "-----BEGIN CERTIFICATE-----\nleaf\n".to_owned(), + key: "-----BEGIN PRIVATE KEY-----\nSUPER-SECRET\n".to_owned(), + ca: "-----BEGIN CERTIFICATE-----\nauthority\n".to_owned(), + } + } + + /// The property the whole delivery rests on: the key lands unreadable to + /// anything but its owner, so the only way into the container is the + /// credential mechanism. + #[test] + fn the_staged_key_is_private_the_moment_it_exists() { + use std::os::unix::fs::PermissionsExt as _; + let dir = tempfile::tempdir().expect("tempdir"); + let staged = dir.path().join("iris"); + write_identity(&staged, &credential()).expect("writes"); + for name in [CERT_CREDENTIAL, KEY_CREDENTIAL] { + let mode = std::fs::metadata(staged.join(name)) + .expect("staged") + .permissions() + .mode(); + assert_eq!(mode & 0o777, 0o600, "{name} must be 0600, got {mode:o}"); + } + let dir_mode = std::fs::metadata(&staged) + .expect("staged dir") + .permissions() + .mode(); + assert_eq!(dir_mode & 0o777, 0o700, "got {dir_mode:o}"); + } + + /// Re-staging is what happens on every rebuild, so a second write must + /// leave the file holding only the new value — a shorter certificate + /// written over a longer one without truncation is a file that parses as + /// neither. + #[test] + fn re_staging_replaces_rather_than_overwrites_in_place() { + let dir = tempfile::tempdir().expect("tempdir"); + let staged = dir.path().join("iris"); + let mut long = credential(); + long.cert = "-".repeat(400); + write_identity(&staged, &long).expect("writes"); + write_identity(&staged, &credential()).expect("re-writes"); + let on_disk = std::fs::read_to_string(staged.join(CERT_CREDENTIAL)).expect("read back"); + assert_eq!(on_disk, credential().cert); + } + + /// Half a delivery is worse than none: the container's check would fail + /// on the missing key and report a login problem instead of a delivery + /// one. Same both-or-neither rule the queue credential states. + #[test] + fn a_certificate_without_its_key_forwards_nothing() { + let dir = tempfile::tempdir().expect("tempdir"); + assert!(mounts(dir.path()).is_empty()); + std::fs::write(dir.path().join(CERT_CREDENTIAL), "leaf").expect("write cert"); + assert!(mounts(dir.path()).is_empty()); + } + + /// The store's own authority rides along when there is one, so a + /// container verifying a self-signed listener has something to verify + /// against — and the absence of it is not a reason to withhold the + /// identity. + #[test] + fn a_staged_server_authority_is_forwarded_beside_the_identity() { + let dir = tempfile::tempdir().expect("tempdir"); + std::fs::write(dir.path().join(CERT_CREDENTIAL), "leaf").expect("write cert"); + std::fs::write(dir.path().join(KEY_CREDENTIAL), "private").expect("write key"); + let without: Vec = mounts(dir.path()).into_iter().map(|c| c.name).collect(); + assert_eq!(without, [CERT_CREDENTIAL, KEY_CREDENTIAL]); + + std::fs::write(dir.path().join(SERVER_CA_CREDENTIAL), "authority").expect("write ca"); + let with: Vec = mounts(dir.path()).into_iter().map(|c| c.name).collect(); + assert_eq!( + with, + [CERT_CREDENTIAL, KEY_CREDENTIAL, SERVER_CA_CREDENTIAL] + ); + } + + /// The names are the agreement with `nix/agent-modules/bao.nix`, and + /// `hive-priv` refuses any of them that could close the flag it is + /// interpolated into. + #[test] + fn every_credential_id_is_one_hive_priv_will_accept() { + for name in [CERT_CREDENTIAL, KEY_CREDENTIAL, SERVER_CA_CREDENTIAL] { + assert!( + name.chars() + .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'), + "{name} must be [A-Za-z0-9_-] only" + ); + } + } +} diff --git a/hive-c0re/src/lifecycle/host_config.rs b/hive-c0re/src/lifecycle/host_config.rs index b1592dbe..faedfa1a 100644 --- a/hive-c0re/src/lifecycle/host_config.rs +++ b/hive-c0re/src/lifecycle/host_config.rs @@ -253,7 +253,14 @@ async fn set_nspawn_flags( // here, and forwarded *as a credential* rather than a bind for the // reason `queue_agent_credentials` records. let queue_credential_dir = std::env::var_os(QUEUE_CREDENTIAL_DIR_ENV).map(PathBuf::from); - let load_creds = queue_agent_credentials(agent_name, queue_credential_dir.as_deref()); + let mut load_creds = queue_agent_credentials(agent_name, queue_credential_dir.as_deref()); + + // The agent's own identity *at* the swarm secret store, collected under + // this hive's certificate and forwarded the same way and for the same + // reason. Staged here rather than at spawn because this is the one place + // that writes the container's credential list — see + // `super::agent_identity` for the whole hop. + load_creds.extend(super::agent_identity::stage(agent_name).await); let mut binds: Vec = vec![ BindMount { diff --git a/hive-c0re/src/lifecycle/mod.rs b/hive-c0re/src/lifecycle/mod.rs index 3b6b69e1..4a83734d 100644 --- a/hive-c0re/src/lifecycle/mod.rs +++ b/hive-c0re/src/lifecycle/mod.rs @@ -1,5 +1,6 @@ //! `nixos-container` lifecycle + per-agent config flake generation. +mod agent_identity; mod git; mod host_config; mod setup; diff --git a/hive-c0re/src/meta.rs b/hive-c0re/src/meta.rs index f5988faa..4f71ec35 100644 --- a/hive-c0re/src/meta.rs +++ b/hive-c0re/src/meta.rs @@ -706,6 +706,11 @@ const FORWARDED_VARS: &[&str] = &[ // address, which inside a container is the agent itself. "HIVE_AGENT_NATS_URL", "HIVE_AGENT_OIDC_TOKEN_ENDPOINT", + // Where the swarm secret store listens. The hive's own `BAO_ADDR` value, + // forwarded under a separate name so the container never inherits the + // hive's store environment wholesale — the address is the only part of it + // an agent may have, the certificate beside it being the hive's own. + "HIVE_AGENT_BAO_ADDR", // Where an agent's `swarm-logs` queries, computed by the host for the // same reason as the two above: it is a gateway address, and a container // cannot derive it from anything it holds. @@ -747,6 +752,12 @@ const FORWARDED_VAR_OPTIONS: &[(&str, &str)] = &[ "HIVE_AGENT_OIDC_TOKEN_ENDPOINT", "services.hyperhive.agent.queue.tokenEndpoint", ), + // Read at build time to decide whether the in-container identity check is + // generated at all — see `nix/agent-modules/bao.nix`. Both halves wired + // for this map's own reason: the unit bakes the address into its own + // environment, so an agent whose option says no store and whose env says + // otherwise would have a check that is simply not there. + ("HIVE_AGENT_BAO_ADDR", "services.hyperhive.agent.bao.addr"), // Read at build time to decide whether `swarm-logs` is installed at all — // see `nix/agent-modules/logs.nix`. Both halves wired for this map's own // reason: an agent whose option says no log store and whose env says diff --git a/hive-c0re/src/paths.rs b/hive-c0re/src/paths.rs index dcf16ef4..5ad8066b 100644 --- a/hive-c0re/src/paths.rs +++ b/hive-c0re/src/paths.rs @@ -102,6 +102,32 @@ pub fn forge_repo_creation_disabled_marker(name: &str) -> PathBuf { forge_dir().join(format!("repo-creation-disabled-{name}")) } +/// `agent-identity/` — one subdir per agent holding the store identity this +/// hive collected for it. Staging only: the files exist so `systemd-nspawn` +/// has something to `--load-credential` from, and nothing on the host ever +/// reads them back. +/// +/// Under the state root rather than `/run` deliberately. The forward happens +/// when a container's nspawn config is (re)written, and a container can be +/// restarted long after that; material that vanished with `/run` would leave +/// a boot where the agent's identity is simply absent and the only symptom is +/// a unit inside the container refusing to start. +#[must_use] +pub fn agent_identity_root() -> PathBuf { + state_root().join("agent-identity") +} + +/// `agent-identity/` — one agent's staged store identity. +/// +/// ⚠️ Deliberately **not** under `agents/`: that tree is bind-mounted +/// into the container (and a parent's, for a child), so a private key placed +/// there would be readable by the agent as a plain file, bypassing the +/// credential mechanism that exists to control exactly that. +#[must_use] +pub fn agent_identity_dir(name: &str) -> PathBuf { + agent_identity_root().join(name) +} + /// `matrix/` — host-side matrix provisioning state (admin token, hive /// Space room id, per-agent password creds). The shared registration /// token is bind-mounted into the tuwunel container via nix and stays diff --git a/nix/agent-modules/bao.nix b/nix/agent-modules/bao.nix new file mode 100644 index 00000000..f484f87d --- /dev/null +++ b/nix/agent-modules/bao.nix @@ -0,0 +1,206 @@ +# This agent's own identity at the swarm secret store, and the check that +# proves it works. +# +# `swarm-controller` mints the leaf at agent creation and publishes it; this +# agent's hive collects it under the hive's own certificate and hands it in as +# systemd credentials (`hive_c0re::lifecycle::agent_identity`). Nothing here +# fetches anything from the store — by the time this container boots, its +# identity is already inside it. +# +# ⚠️ The identity arrives as credentials and NOT as a bind mount, and the mode +# is why: the host file is `0600` to the hive daemon and this unit runs as the +# unprivileged agent user. nspawn's `--load-credential` is read by the +# container manager as root and re-exposed under this unit's own `User=`; a +# bind would deliver a private key this user cannot open. Same answer, same +# reason, as ./queue.nix's credential pair. +# +# This is the courier's other end: the hive-side collector has no purpose +# without it, so the two ship together and neither is reachable alone. +# +# 🩸 This unit fails LOUDLY where the hive-side readers degrade quietly, and +# that is the opposite default on purpose. A missing OIDC client secret means +# a hive whose publisher has yet to run; a missing or refused certificate +# means an agent that believes it can reach the store and cannot, which every +# later pull would report as its own unrelated failure. The one place that +# knows the real cause is the login itself, so this is where it is said. +{ + pkgs, + lib, + config, + ... +}: +let + cfg = config.services.hyperhive.agent.bao; + + # This container's agent name. The same string the hive published the + # identity under, because the agent's unix user is named for the agent — + # see ./user.nix. + agentName = config.services.hyperhive.agent.user.name; + + # The three ids `hive_c0re::lifecycle::agent_identity` forwards under. + # Neither side can discover the other's spelling, so a rename is a rename + # there too. + certCredential = "hive-agent-bao-cert"; + keyCredential = "hive-agent-bao-key"; + serverCaCredential = "hive-agent-bao-server-ca"; + + unitName = "hive-agent-bao-identity"; + + # Where the identity lives in the store, spelled from the same pieces the + # publisher uses. `swarm_secret_client::mtls::identity_path` builds + # `swarm/agents//bao-mtls` and `path::MOUNT` is `secret`; this + # literal is the nix half of that one agreement, exactly as + # ../host-modules/glue-queue-agent-credential.nix spells its own. + identityPath = "secret/swarm/agents/${agentName}/bao-mtls"; + + # The address is the whole switch — no separate `enable`, the same shape + # ./queue.nix and ./logs.nix gate themselves with. A hive that has a store + # forwards its address and every agent on it gets the check; a hive that has + # none forwards nothing and no agent does. An `enable` beside it would be a + # knob whose only correct setting is whatever the address already says, and + # its default would decide whether the hive-side courier delivers into a + # container that reads what it is given or into one that never looks. + configured = cfg.addr != null; +in +{ + options.services.hyperhive.agent.bao = { + addr = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "https://bao.example.com:8200"; + description = '' + Where the swarm secret store listens, as this container reaches it. + + Set by the generated meta flake from the host's own `BAO_ADDR`, which + is the address this hive already uses, and setting it is what generates + `${unitName}.service`: at boot that unit logs in with the certificate + its hive delivered and reads this agent's own path back, failing if + either step does not succeed. `systemctl status ${unitName}` inside the + container is then the answer to "can this agent reach the store as + itself", which nothing else in the tree reports. + + `null` means the hive was given no store. No unit is generated then, + because an agent whose swarm never minted an identity has nothing to + log in with, and a failed unit at every boot would say that in the + loudest possible way about a deployment that never asked for it. + ''; + }; + }; + + config = lib.mkIf configured { + systemd.services.${unitName} = { + description = "prove this agent can authenticate to the swarm secret store as itself"; + after = [ "network.target" ]; + wantedBy = [ "multi-user.target" ]; + path = [ + pkgs.openbao + pkgs.coreutils + ]; + # Sized for a store that comes up around the same time this container + # does, not for one that is sealed: a few short attempts cover the race, + # and a longer window would only delay the report of a real failure. + # + # `StartLimit*` are `[Unit]` settings, so they go here and not in + # `serviceConfig` — systemd ignores them under `[Service]`. The window + # has to exceed `RestartSec` times the burst. + startLimitBurst = 4; + startLimitIntervalSec = 300; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + TimeoutStartSec = 30; + Restart = "on-failure"; + RestartSec = 15; + User = agentName; + Group = agentName; + # Bare ids, no paths: the terse `LoadCredential=` form that inherits a + # credential the service *manager* received, which is what the + # container manager passed in. ./queue.nix states the same shape. + LoadCredential = [ + certCredential + keyCredential + serverCaCredential + ]; + }; + environment = { + BAO_ADDR = cfg.addr; + # `%d` is `$CREDENTIALS_DIRECTORY`, per-unit and owned by `User=`. + BAO_CLIENT_CERT = "%d/${certCredential}"; + BAO_CLIENT_KEY = "%d/${keyCredential}"; + }; + script = '' + set -euo pipefail + + # Absent credentials are the first thing checked, because every later + # message would blame the store for a delivery that never happened. + # 🩸 The files are named and never read here: what went wrong is a + # property of the path, and the bytes at it are a private key. + missing= + for id in ${lib.escapeShellArg certCredential} ${lib.escapeShellArg keyCredential}; do + if [ ! -s "$CREDENTIALS_DIRECTORY/$id" ]; then + missing="$missing $id" + fi + done + if [ -n "$missing" ]; then + echo "this agent has no store identity: the hive delivered no$missing." >&2 + echo "swarm-controller publishes it at ${identityPath} when the agent is created, and the hive collects it from there." >&2 + exit 1 + fi + + # Only when one was delivered. Absent means the container verifies the + # store's listener against its own trust store, which is what a + # deployment with a real CA wants; pointing BAO_CACERT at a file that + # is not there would fail the handshake and name the wrong cause. + if [ -s "$CREDENTIALS_DIRECTORY/${serverCaCredential}" ]; then + export BAO_CACERT="$CREDENTIALS_DIRECTORY/${serverCaCredential}" + fi + + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` + # variables above only decide which certificate the TLS handshake + # presents; without a token `bao` asks its token helper instead, and + # that is a `sh` this unit's `path` does not carry. `-token-only` + # answers on stdout and skips the helper on both sides. + # + # No `name=`: a cert role pins both its authority and the common name + # it accepts, so this agent's leaf matches its own role and no other. + # Naming the role here would be a third copy of a string + # `swarm_secret_client::policy::agent_object_name` already owns. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "this agent's certificate was refused by the swarm secret store at $BAO_ADDR." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 + fi + export BAO_TOKEN + + # A token is not yet an answer: the login proves the certificate, this + # proves the policy attached to it. Reading this agent's own identity + # back is the smallest read its document grants, and it is the same + # check `swarm-controller` runs against the leaf before it reports the + # creation done — so a policy that drifted apart from the path fails + # here rather than in whichever pull needed it first. + # + # ⚠️ Output discarded, not printed: the field is a certificate and the + # object beside it is a private key. Nothing about this check needs a + # value, only whether the read succeeded. + if ! bao kv get -field=cert ${lib.escapeShellArg identityPath} >/dev/null 2>"$err"; then + echo "this agent logged in to the swarm secret store but cannot read ${identityPath}, so its policy does not cover its own path." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 + fi + + echo "authenticated to the swarm secret store at $BAO_ADDR as ${agentName} and read ${identityPath}." + ''; + }; + }; +} diff --git a/nix/agent-modules/default.nix b/nix/agent-modules/default.nix index 2ea0a867..bab54e47 100644 --- a/nix/agent-modules/default.nix +++ b/nix/agent-modules/default.nix @@ -40,6 +40,7 @@ in { imports = [ ./agent-service.nix + ./bao.nix ./bash-env.nix ./claude-settings.nix ./dashboard-links.nix diff --git a/nix/host-modules/hive-c0re/environment.nix b/nix/host-modules/hive-c0re/environment.nix index 268d014b..f660f3aa 100644 --- a/nix/host-modules/hive-c0re/environment.nix +++ b/nix/host-modules/hive-c0re/environment.nix @@ -334,6 +334,12 @@ in # handshake, naming neither. lib.optionalAttrs haveBaoClientIdentity { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + # The same address again under an agent-facing name, forwarded into every + # container's module block by `meta::render_flake`. Deliberately not the + # `BAO_ADDR` line itself: a container that inherited this hive's store + # environment wholesale would inherit the hive's certificate paths with it, + # and those name a credential no agent may present. + HIVE_AGENT_BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; # `%d`, not the paths themselves: the key is `0600` root-owned and this # daemon runs as hive-core, so it never gets read access to the original. # See the LoadCredential in ./default.nix. diff --git a/nix/module-eval.nix b/nix/module-eval.nix index cb28baec..572003c7 100644 --- a/nix/module-eval.nix +++ b/nix/module-eval.nix @@ -64,10 +64,12 @@ let # the meta flake hands a container, so what this evaluates is what an # agent gets. # - # Note `hyperhive`, not `services.hyperhive`: an agent container's options - # live at the top level. - agent = - extra: + # Takes a whole module rather than a settings attrset, so a fixture can + # reach either spelling of the agent tier. `user.name` is the one option + # with no usable default, set here at its real path and at `mkDefault` so a + # fixture naming its own agent still wins. + agentWith = + module: (nixosSystem { system = pkgs.stdenv.hostPlatform.system; modules = [ @@ -79,11 +81,20 @@ let }; boot.loader.grub.enable = false; system.stateVersion = "25.11"; - hyperhive = lib.recursiveUpdate { user.name = "a1"; } extra; + services.hyperhive.agent.user.name = lib.mkDefault "a1"; } + module ]; }).config; + # Note `hyperhive`, not `services.hyperhive`: the agent tier's options moved + # under `services.hyperhive.agent`, and ../agent-modules/renamed-options.nix + # keeps the top-level spelling reaching them. ⚠️ That shim covers the + # options that existed when the tier moved and nothing since, so a fixture + # for an option added afterwards has to go through [`agentWith`] and name + # the real path. + agent = extra: agentWith { hyperhive = extra; }; + allLocal = hive { deploy.singleHostSwarm = true; }; bare = hive { }; withCi = hive { deploy.forgejo.ci.enable = true; }; @@ -605,6 +616,12 @@ let }; agentNoQueue = agent { }; + # The agent side of the swarm secret store. The address is the whole switch — + # it is both what generates the login check and what that check points at — + # so it and the empty fixture beside it are the two arms worth having. + agentBao = agentWith { services.hyperhive.agent.bao.addr = "https://bao.t.local:8200"; }; + agentNoBao = agentWith { }; + # The memory-pressure pair. `claudeMemoryMaxBytes` is the container's own # cap, rendered per agent by meta.rs — the capped arm is the one every # real deploy gets, the uncapped arm is a hive that set `infinity` or a @@ -636,6 +653,7 @@ let }; agentHarness = machine: machine.systemd.services.hive-agent; agentSubagentDaemon = machine: machine.systemd.services.hive-subagent-daemon; + agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity; agentSettings = machine: machine.services.opentelemetry-collector.settings; # This hive's own collector, which is a HOST service — unlike the swarm @@ -1766,6 +1784,112 @@ let && !(u.environment ? HIVE_AGENT_OIDC_CLIENT_SECRET_FILE) && !(u.environment ? HIVE_AGENT_OIDC_CLIENT_ID_FILE); } + { + # The three ids `hive_c0re::lifecycle::agent_identity` forwards under. + # Neither end can discover the other's spelling, and a mismatch is a + # credential that is simply not there — which this unit then reports as + # a hive that delivered nothing. + name = "an agent with the store enabled imports every half of its identity"; + ok = + let + c = (agentBaoIdentity agentBao).serviceConfig.LoadCredential; + in + builtins.elem "hive-agent-bao-cert" c + && builtins.elem "hive-agent-bao-key" c + && builtins.elem "hive-agent-bao-server-ca" c; + } + { + # `%d` and not a path under the agent's state dir, for the reason the + # queue arm above gives: the host file is `0600` to the hive daemon, so + # the only copy this unprivileged unit can open is the one systemd puts + # in its own credentials directory. The address is the option's value + # rather than a literal that agrees with it today. + name = "the identity check presents its certificate out of the credentials directory"; + ok = + let + e = (agentBaoIdentity agentBao).environment; + in + e.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert" + && e.BAO_CLIENT_KEY == "%d/hive-agent-bao-key" + && e.BAO_ADDR == agentBao.services.hyperhive.agent.bao.addr; + } + { + # Same 403-not-a-miss reason as the hive-side readers: the path + # `swarm_secret_client::mtls::identity_path` builds is the one this + # agent's own policy stanza covers, and a path outside it is refused + # however correct it looks. Built from the agent's own name rather than + # from a literal, because the name is what makes it this agent's path + # and not some other agent's. + name = "the identity check reads the agent's own path"; + ok = + let + m = agentBao; + name = m.services.hyperhive.agent.user.name; + in + lib.hasInfix "secret/swarm/agents/${name}/bao-mtls" (agentBaoIdentity m).script; + } + { + # The whole point of the unit, and the thing a quieter default would + # undo: every arm of the check ends the unit non-zero, so an agent that + # cannot authenticate as itself says so at boot instead of at whichever + # pull needed the store first. + name = "the identity check fails the unit rather than degrading"; + ok = + let + u = agentBaoIdentity agentBao; + in + lib.hasInfix "exit 1" u.script + && !(lib.hasInfix "exit 0" u.script) + && u.serviceConfig.Restart == "on-failure"; + } + { + # Nothing about the identity may be printed, and the read-back is where + # that could slip: `bao kv get` on this path answers with certificate + # material, and the object beside it is a private key. The check needs + # only whether the read succeeded. + # + # The path goes through `lib.escapeShellArg` here for the same reason the + # module passes it through one — that helper decides whether an argument + # needs quotes at all, and this one (only `[a-z0-9/-]`) comes back bare. + # Spelling the quotes in by hand asserts a rendering nixpkgs chooses + # rather than the redirect this property is about. + name = "the identity check discards what it reads back"; + ok = + let + m = agentBao; + name = m.services.hyperhive.agent.user.name; + arg = lib.escapeShellArg "secret/swarm/agents/${name}/bao-mtls"; + in + lib.hasInfix "bao kv get -field=cert ${arg} >/dev/null" (agentBaoIdentity m).script; + } + { + # The absence arm, and what makes the four above able to fail. An agent + # whose swarm never minted an identity has nothing to log in with, and a + # failed unit at every boot would be the loudest possible statement + # about a deployment that never asked for one. + name = "an agent told no store address runs no identity check"; + ok = !(agentNoBao.systemd.services ? hive-agent-bao-identity); + } + { + # Where the store is and where the agent is told it is, one agreement + # spanning two modules. Asserted against the hive's own `BAO_ADDR` + # rather than a literal, because an agent pointed at a different + # spelling of the same store presents a certificate to a listener whose + # name it cannot verify. + name = "an agent is told the same store address its hive uses"; + ok = + let + e = allLocal.systemd.services.hive-c0re.environment; + in + e.HIVE_AGENT_BAO_ADDR == e.BAO_ADDR; + } + { + # A hive with no certificate of its own can collect no agent's identity, + # so forwarding an address would name a store nothing in the container + # can reach. The same gate the `BAO_*` pair beside it sits behind. + name = "a hive with no store identity forwards no store address to its agents"; + ok = !(bare.systemd.services.hive-c0re.environment ? HIVE_AGENT_BAO_ADDR); + } { # Two thirds of the container's cap, and a SOFT ceiling: the daemon's # cgroup holds every nested claude, so `MemoryHigh=` throttles the