mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

drop the shared hive agent login

Browse source 
Every agent now connects to the queue and reads logs with its own secret
from the store (#5181), so the per-hive `hive-<h>-agent` OIDC client has
no reader left. Remove it end to end:

- hive-c0re no longer forwards `hive-queue-agent-{secret,client-id}` into
  containers, and no longer forwards HIVE_AGENT_OIDC_TOKEN_ENDPOINT; the
  agent module that loaded them (nix/agent-modules/queue.nix) is gone, its
  options refused with a removal message.
- The host reader (glue-queue-agent-credential.nix, swarm-bao-queue-agent),
  its store role and policy unit (swarm-bao-queue-agent-policy), its PKI
  leaf and the deploy.bao.queueAgent* / queue.agentCredentialDir options
  are gone; the options are refused with a removal message.
- authelia mints no `hive-<h>-agent` client, and swarm-secret-publish no
  longer writes swarm/hives/<h>/queue/agent.
- swarm-secret-client drops queue::agent_client_path / queue::Credential.
- HIVE_AGENT_NATS_URL is now gated on the agents' queue address alone.

The log store's authelia check stays for operators. swarm-nats-auth still
refuses a `-agent` client id; that refusal and `agentClientSuffix` are left
for a follow-up.

Refs #4459
Refs #4765
This commit is contained in:
atlas 2026-10-13 09:49:32 +02:00 • committed by mara
commit 868c6cf96c
41 changed files with 286 additions and 1410 deletions
Download patch file Download diff file Expand all files Collapse all files

3
docs/swarm/README.md
View file

@ -303,8 +303,7 @@ subjects it publishes to, the credential may answer the one request it was just
sent, which is how a command gets its reply. The harness and the subagent daemon
connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to the
icon write and the schedule read arrive. An agent can't read another agent's
terminal, state, input, commands or replies. The queue refuses the hive's shared
agent client outright.
terminal, state, input, commands or replies.
Hives and the swarm controller get a subscribe allow-list too, each with its
own reply inbox `_INBOX.client.<client-id>.>`, which `swarm_queue_client::connect`

11
docs/swarm/bao.md
View file

@ -121,13 +121,12 @@ them and names them in config. The service principals' leaves are at the top of
`/var/lib/swarm-bao-pki/`, named after the principal.
For every hive in `swarm.hives` other than its own, the store's host signs
that hive's three leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`:
that hive's two leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`:
| file | subject | name it in |
| ---------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------- |
| `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` |
| `queue-agent.pem`, `queue-agent-key.pem` | `<deploy.bao.queueAgentCommonNamePrefix>-<hive>` | `deploy.bao.queueAgentClientCertFile`, `.queueAgentClientKeyFile` |
| `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
| file | subject | name it in |
| ------------------------------------ | ---------------------------------------------- | ------------------------------------------------------------------------------- |
| `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` |
| `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
Copy `/var/lib/swarm-bao-pki/ca.pem` with them and name it in
`deploy.bao.serverCaFile`. A hive added to `swarm.hives` gets its directory on

1
docs/swarm/credentials.md
View file

@ -43,7 +43,6 @@ the rest of the cell says how.
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host | must be stated | must be stated |
| `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |

66
docs/swarm/secrets.md
View file

@ -107,9 +107,8 @@ with nothing to authenticate with still receives every hive's telemetry —
its pushes to the stores go out unauthenticated and get refused there, and its
scrape jobs for the published targets read a credential file nothing
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice
`glue-queue-agent-credential.nix` makes for its own reader, rather than the hard assertion `swarm-grafana.nix`
uses for its non-optional one. The operator can still deliver the secret by
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
`swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.
@ -259,7 +258,6 @@ such as the root token.
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the
@ -268,34 +266,8 @@ publisher in authelia's container has written the secret, rather than skipping.
A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause.
The store's **first reader** is the agent containers' queue credential:
`glue-queue-agent-credential.nix` lands it as two files, the client secret and
the client id it authenticates, because that's how a queue client reads them —
the secret by path, the id by value. No local fallback exists, and none is
possible, so absent files mean an agent built before `swarm-logs` read its own
queue secret has no working `swarm-logs`, which is what a swarm looks like
before the publisher in authelia's container has run.
The reader runs before `hive-c0re.service`, which wants it (doesn't require
it), so an agent container never renders ahead of the credential; an
unreachable store delays the daemon's start rather than failing it.
That credential still has one hop left, because the reader of it lives inside
an agent container. **It crosses as a systemd credential, not as a bind
mount**, and the mode decides that: the secret is `root:0600` and a harness
runs as its own unprivileged agent user, so a bind would deliver a file that
user can't open. `hive-c0re` stats the two files — the directory is `0755`, so
it needs no read access to either — and hands them to
`systemd-nspawn --load-credential` through `hive-priv`, which runs as root.
Inside, `hive-agent.service` names the same two ids in `LoadCredential=`, which
lands them in its own `$CREDENTIALS_DIRECTORY`, owned by the agent user and by
nobody else, where a `swarm-logs` built before it read the agent's own queue
secret reads them. The harness itself doesn't: it connects to the queue with the agent's own credential. `hive-c0re`
never reads the bytes at any point: it runs as `hive-core`, which is the reason
a copy wasn't an option either. When the files aren't there the daemon forwards
nothing and says so in its journal, and the container starts without them.
The **second reader** is Grafana's OIDC client secret, and it's the first one
that's a _swarm service's_ own credential rather than a hive's. It has exactly
The store's **first reader** is Grafana's OIDC client secret, a _swarm
service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the publisher in
@ -338,7 +310,7 @@ the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
the Grafana host, which retries, while Grafana stays down because its config
reads that file.
The **third reader** is the swarm collector's own OIDC client secret, the
The **second reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
@ -347,13 +319,12 @@ differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
doesn't render the reading unit without them, the shape
`glue-queue-agent-credential.nix` uses for its own read.
doesn't render the reading unit without them.
The **fourth reader** is the secret store forwarder's OIDC client secret:
The **third reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
third in both directions. It renders unconditionally, because the export it
second in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
@ -412,14 +383,13 @@ next controller restart.
### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. Each of these eleven leaves has
identity and sharing one merges the identities. Each of these ten leaves has
a role and a policy of its own, and reaches one path, except the review
classifier's, the CI runner's, the builder's, the CI run poller's, the claude
login minter's, the forge host's and Grafana's, which reach more than one:
| unit | option pair under `deploy.bao.` | reads, or writes |
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
| `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin`, and Grafana's journal forwarder in its container | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client` and `swarm/services/<grafana forwarder client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
@ -431,12 +401,8 @@ login minter's, the forge host's and Grafana's, which reach more than one:
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
The first exists **per hive**, because the path it reads carries a hive name
and every hive runs its own reader. Its subject is
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
composed spelling as a hive name, so nobody can name a hive into another hive's
role. The other nine reach a path that names a swarm service rather than a hive,
so one role each is enough and their subjects are the flat
Each reaches a path that names a swarm service rather than a hive, so one role
each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
@ -449,7 +415,7 @@ OIDC client secret it reads.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all nine and defaults
all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from
the store's host and name it here: a service principal's from the top of
`/var/lib/swarm-bao-pki/`, another hive's queue-agent pair from
`/var/lib/swarm-bao-pki/`, another hive's leaves from
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
directory). That's one file per principal rather than one file
shared by nine, which is the whole of what this buys. `swarm-ci-runner.nix`
@ -461,7 +427,7 @@ naming both options, and the controller runs without forge access and the forge
without a login source until an operator places the leaf.
On a host without the store, where the forwarder never runs, forgetting one of
the other nine isn't a quiet degrade. A hand-configured remote-store hive that
the others isn't a quiet degrade. A hand-configured remote-store hive that
names the hive's own `clientCertFile` and misses a principal's pair has the
build refused, by the module of the reader it missed, naming that pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
@ -475,7 +441,7 @@ their readers hold a handle for longer than five minutes.
### What a missing leaf costs
Each of the three modules asserts its own pair, on the same condition: **this host
Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
@ -492,9 +458,7 @@ that matters on its own:
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no collector, no Grafana — has no
unit to skip, so the second clause is false. Only the queue-credential reader
has no toggle of its own to check: every hive runs one for its own agents, so
reading the store at all is what asks for it.
unit to skip, so the second clause is false.
The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives

2
docs/swarm/sso.md
View file

@ -225,7 +225,7 @@ Four consequences worth stating plainly:
Everything above is a person in a browser. A swarm also has callers that
hold no session and follow no redirect: the telemetry collector, the
queue's auth-callout responder, and each hive's own agents.
queue's auth-callout responder, and each hive.
**One client per hive, not one per service.** A hive's identity belongs to
the directory rather than to whichever service happens to consume it, so a

12
hive-c0re/src/lifecycle/agent_identity.rs
View file

@ -8,10 +8,9 @@
//! asked to authenticate in order to obtain the thing it authenticates with:
//! by the time it runs, its identity is already inside it.
//!
//! **A credential and not a bind mount**, the same answer and the same reason
//! as the queue secret beside it in [`super::host_config`]: the files are
//! `0600` to `hive-core` and the units that use them run as the unprivileged
//! agent user, so a bind would deliver a private key that user cannot open.
//! **A credential and not a bind mount**: the files are `0600` to `hive-core`
//! and the units that use them run as the unprivileged agent user, so a bind
//! would deliver a private key that user cannot open.
//! The container manager reads a `--load-credential` source as root and
//! re-exposes it under the consuming unit's own `User=`.
//!
@ -60,8 +59,7 @@ const SERVER_CA_CREDENTIAL: &str = "hive-agent-bao-server-ca";
///
/// Already-staged files survive a failed collection. A store that is briefly
/// unreachable then delivers the identity from the last successful read rather
/// than none at all — the same call as the queue reader's, which leaves its
/// files alone when the store says nothing.
/// than none at all.
pub async fn stage(agent: &str) -> Vec<CredentialMount> {
let dir = crate::paths::agent_identity_dir(agent);
match collect(agent, &dir).await {
@ -246,7 +244,7 @@ mod tests {
/// Half a delivery is worse than none: the container's check would fail
/// on the missing key and report a login problem instead of a delivery
/// one. Same both-or-neither rule the queue credential states.
/// one.
#[test]
fn a_certificate_without_its_key_forwards_nothing() {
let dir = tempfile::tempdir().expect("tempdir");

130
hive-c0re/src/lifecycle/host_config.rs
View file

@ -6,7 +6,7 @@
use std::path::{Path, PathBuf};
use anyhow::{Context, Result};
use hive_priv_sock::{BindMount, CredentialMount};
use hive_priv_sock::BindMount;
use hive_types::Permission;
use crate::coordinator::{AgentPaths, HiveEnv};
@ -417,70 +417,6 @@ pub fn hand_permissions(agent: &str, runtime_dir: &Path) {
}
}
/// Env var naming the host directory `swarm-bao-queue-agent.service` lands
/// this hive's agent queue credential in. Set by the hive-c0re NixOS module
/// from `deploy.hive-controller.queue.agentCredentialDir`; absent means this
/// daemon runs outside its unit, which is the same "no queue credential"
/// answer as an empty directory.
const QUEUE_CREDENTIAL_DIR_ENV: &str = "HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR";
/// systemd credential ids the two files arrive under inside the container.
/// `nix/agent-modules/queue.nix` spells the same two names in the harness
/// unit's `LoadCredential=`; neither side can discover the other's, so a
/// rename here is a rename there.
const QUEUE_SECRET_CREDENTIAL: &str = "hive-queue-agent-secret";
const QUEUE_CLIENT_ID_CREDENTIAL: &str = "hive-queue-agent-client-id";
/// Forward this hive's agent queue credential into `agent_name`'s container
/// as two systemd credentials, or nothing when the publisher has not landed
/// it yet.
///
/// **A credential and not a bind mount, because of the mode.** The secret is
/// `root:0600` on the host and the harness runs as the unprivileged agent
/// user, so a bind would deliver a file that user cannot open. nspawn's
/// `--load-credential` is read by the container manager as root and
/// re-exposed under the consuming unit's own `User=`, which is the whole
/// difference. hive-c0re never reads the bytes either way — it runs as
/// `hive-core` and only ever needs to know the file is *there*, which a
/// `0755` directory permits.
///
/// **Absent files are legal.** Authelia mints the secret on its first boot
/// and a publisher on that host puts it in the store, so "nothing at that
/// path" is the ordinary early state of a swarm rather than a fault. Saying
/// so at `info!` is what keeps it from being invisible: without a line here,
/// an agent that never connects looks identical to one that was never
/// configured.
fn queue_agent_credentials(agent_name: &str, dir: Option<&Path>) -> Vec<CredentialMount> {
let Some(dir) = dir else {
tracing::info!(
%agent_name,
"no {QUEUE_CREDENTIAL_DIR_ENV} in this daemon's environment — agent gets no swarm queue credential"
);
return Vec::new();
};
let secret = dir.join("secret");
let client_id = dir.join("client_id");
// Both or neither: a client holding one of the two comes up fine and
// never connects.
if !secret.is_file() || !client_id.is_file() {
tracing::info!(
%agent_name, dir = %dir.display(),
"swarm queue credential not published yet — agent gets no swarm queue credential"
);
return Vec::new();
}
vec![
CredentialMount {
name: QUEUE_SECRET_CREDENTIAL.to_owned(),
host_path: secret.to_string_lossy().into_owned(),
},
CredentialMount {
name: QUEUE_CLIENT_ID_CREDENTIAL.to_owned(),
host_path: client_id.to_string_lossy().into_owned(),
},
]
}
/// Idempotently rewrite the lines in `/etc/nixos-containers/<container>.conf`
/// that hive-c0re owns: `PRIVATE_NETWORK` (always 1), `HOST_ADDRESS` (the
/// bridge gateway IP) and `EXTRA_NSPAWN_FLAGS` (the runtime-dir bind). What
@ -537,18 +473,11 @@ async fn set_nspawn_flags(
// is needed here — the bind alone is enough.
let claude_mount = container_claude_mount(agent_name);
// The agent's own swarm-queue credential — the one thing forwarded in
// here, and forwarded *as a credential* rather than a bind for the
// reason `queue_agent_credentials` records.
let queue_credential_dir = std::env::var_os(QUEUE_CREDENTIAL_DIR_ENV).map(PathBuf::from);
let mut load_creds = queue_agent_credentials(agent_name, queue_credential_dir.as_deref());
// The agent's own identity *at* the swarm secret store, collected under
// this hive's certificate and forwarded the same way and for the same
// reason. Staged here rather than at spawn because this is the one place
// that writes the container's credential list — see
// this hive's certificate. Staged here rather than at spawn because this
// is the one place that writes the container's credential list — see
// `super::agent_identity` for the whole hop.
load_creds.extend(super::agent_identity::stage(agent_name).await);
let load_creds = super::agent_identity::stage(agent_name).await;
let mut binds: Vec<BindMount> = vec![
BindMount {
@ -678,10 +607,8 @@ async fn set_nspawn_flags(
#[cfg(test)]
mod tests {
use super::{
BindMount, HostMemory, MEMORY_CAP_FILE, QUEUE_CLIENT_ID_CREDENTIAL,
QUEUE_SECRET_CREDENTIAL, bind_child_agent_dirs, bind_own_agent_dirs,
holds_manage_root_agent, limits_to_apply, memory_cap_value, queue_agent_credentials,
stage_limits,
BindMount, HostMemory, MEMORY_CAP_FILE, bind_child_agent_dirs, bind_own_agent_dirs,
holds_manage_root_agent, limits_to_apply, memory_cap_value, stage_limits,
};
use crate::coordinator::HiveEnv;
@ -791,51 +718,6 @@ mod tests {
assert!(binds.is_empty());
}
/// The ordinary state of a swarm before the publisher on the authelia
/// host has run: the directory is named and empty. Forwarding a
/// credential whose source does not exist would make every agent
/// container refuse to start, so this has to be silence-with-a-log
/// rather than a partial list.
#[test]
fn an_unpublished_queue_credential_forwards_nothing() {
let dir = tempfile::tempdir().expect("tempdir");
assert!(queue_agent_credentials("iris", None).is_empty());
assert!(queue_agent_credentials("iris", Some(dir.path())).is_empty());
std::fs::write(dir.path().join("secret"), "s").expect("write secret");
assert!(
queue_agent_credentials("iris", Some(dir.path())).is_empty(),
"a secret with no client id is not a usable credential"
);
}
/// Both files present: two credentials, named the same two ids the
/// harness unit imports, pointing at the two files the reader unit
/// wrote.
#[test]
fn a_published_queue_credential_forwards_both_files() {
let dir = tempfile::tempdir().expect("tempdir");
std::fs::write(dir.path().join("secret"), "s").expect("write secret");
std::fs::write(dir.path().join("client_id"), "hive-h1-agent").expect("write client_id");
let creds = queue_agent_credentials("iris", Some(dir.path()));
let named: Vec<(String, String)> = creds
.into_iter()
.map(|c| (c.name, c.host_path))
.collect::<Vec<_>>();
assert_eq!(
named,
[
(
QUEUE_SECRET_CREDENTIAL.to_owned(),
dir.path().join("secret").to_string_lossy().into_owned(),
),
(
QUEUE_CLIENT_ID_CREDENTIAL.to_owned(),
dir.path().join("client_id").to_string_lossy().into_owned(),
),
]
);
}
#[test]
fn corrupt_swarm_permissions_deny_without_failing() {
let dir = tempfile::tempdir().expect("tempdir");

59
hive-c0re/src/meta.rs
View file

@ -455,12 +455,10 @@ const CANONICAL_INPUTS: &[&str] = &["nixpkgs"];
/// trivial to stub from tests (which build their own slice instead of
/// touching process-wide env).
const FORWARDED_VARS: &[&str] = &[
// Where the swarm queue is and where its tokens are minted, as an agent
// container reaches them. Both are addresses the *host* computes — the
// queue's from the bridge IP an agent routes over, never a loopback
// address, which inside a container is the agent itself.
// Where the swarm queue is, as an agent container reaches it. An address
// the *host* computes, never a loopback address, which inside a container
// is the agent itself.
"HIVE_AGENT_NATS_URL",
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
// Where the swarm secret store listens. The hive's own `BAO_ADDR` value,
// forwarded under a separate name so the container never inherits the
// hive's store environment wholesale — the address is the only part of it
@ -470,7 +468,7 @@ const FORWARDED_VARS: &[&str] = &[
// fetches to trust the swarm. Forwarded with the address, under the same gate.
"HIVE_AGENT_BAO_SERVICES_PKI_MOUNT",
// Where an agent's `swarm-logs` queries, computed by the host for the
// same reason as the two above: it is a gateway address, and a container
// same reason as the queue's: it is a gateway address, and a container
// cannot derive it from anything it holds.
"HIVE_AGENT_LOGS_QUERY_URL",
// The metrics store's origin, for `swarm-metrics`, for the same reason.
@ -494,18 +492,6 @@ const FORWARDED_VARS: &[&str] = &[
/// name. Half-wiring this map is a value that evaluates fine and is silently
/// wrong.
const FORWARDED_VAR_OPTIONS: &[(&str, &str)] = &[
// Read at build time to decide whether the harness unit imports the hive's
// queue credential at all — see `nix/agent-modules/queue.nix`. Both halves
// wired for this map's own reason: an agent whose option says no queue and
// whose env says otherwise imports no credential for `swarm-logs`.
(
"HIVE_AGENT_NATS_URL",
"services.hyperhive.agent.queue.natsUrl",
),
(
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
"services.hyperhive.agent.queue.tokenEndpoint",
),
// Read at build time to decide whether the in-container identity check is
// generated at all — see `nix/agent-modules/bao.nix`. Both halves wired
// for this map's own reason: the unit bakes the address into its own
@ -1561,35 +1547,22 @@ mod tests {
);
}
/// The queue coordinates travel as a pair and are wired through both
/// halves of the forwarded-var machinery: the env var the harness reads at
/// runtime and the option the harness *unit* is built from. An agent that
/// got only the env var would declare no credential and then report a
/// partial config — visible, but a rebuild away from working.
/// The queue address reaches the harness as env only. The agent module
/// declares no `services.hyperhive.agent.queue.*` option, so rendering one
/// into the meta flake fails every agent's evaluation.
#[test]
fn queue_coordinates_are_forwarded_as_both_env_and_option() {
for var in ["HIVE_AGENT_NATS_URL", "HIVE_AGENT_OIDC_TOKEN_ENDPOINT"] {
fn queue_address_is_forwarded_as_env_and_no_option() {
assert!(FORWARDED_VARS.contains(&"HIVE_AGENT_NATS_URL"));
assert!(
!FORWARDED_VARS.contains(&"HIVE_AGENT_OIDC_TOKEN_ENDPOINT"),
"no agent reads a token endpoint"
);
for (var, option) in FORWARDED_VAR_OPTIONS {
assert!(
FORWARDED_VARS.contains(&var),
"{var} must be forwarded into every agent's env"
!option.starts_with("services.hyperhive.agent.queue."),
"{var} renders {option}, which no agent module declares"
);
}
let mut out = String::new();
push_forwarded_var_options(
&mut out,
&[
("HIVE_AGENT_NATS_URL", "nats://10.42.0.1:4222".to_string()),
(
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
"https://auth.t.local/api/oidc/token".to_string(),
),
],
);
assert_eq!(
out,
" services.hyperhive.agent.queue.natsUrl = \"nats://10.42.0.1:4222\";\n\
\x20 services.hyperhive.agent.queue.tokenEndpoint = \"https://auth.t.local/api/oidc/token\";\n"
);
}
/// `swarm-metrics` is installed only when the option is set, and reads

10
nix/agent-modules/bao.nix
View file

@ -14,8 +14,7 @@
# is why: the host file is `0600` to the hive daemon and this unit runs as the
# unprivileged agent user. nspawn's `--load-credential` is read by the
# container manager as root and re-exposed under this unit's own `User=`; a
# bind would deliver a private key this user cannot open. Same answer, same
# reason, as ./queue.nix's credential pair.
# bind would deliver a private key this user cannot open.
#
# This is the courier's other end: the hive-side collector has no purpose
# without it, so the two ship together and neither is reachable alone.
@ -63,12 +62,11 @@ let
# Where the identity lives in the store, spelled from the same pieces the
# publisher uses. `swarm_secret_client::mtls::identity_path` builds
# `swarm/agents/<agent>/bao-mtls` and `path::MOUNT` is `secret`; this
# literal is the nix half of that one agreement, exactly as
# ../host-modules/glue-queue-agent-credential.nix spells its own.
# literal is the nix half of that one agreement.
identityPath = "secret/swarm/agents/${agentName}/bao-mtls";
# The address is the whole switch — no separate `enable`, the same shape
# ./queue.nix and ./logs.nix gate themselves with. A hive that has a store
# ./logs.nix gates itself with. A hive that has a store
# forwards its address and every agent on it gets the check; a hive that has
# none forwards nothing and no agent does. An `enable` beside it would be a
# knob whose only correct setting is whatever the address already says, and
@ -153,7 +151,7 @@ in
UMask = "0377";
# Bare ids, no paths: the terse `LoadCredential=` form that inherits a
# credential the service *manager* received, which is what the
# container manager passed in. ./queue.nix states the same shape.
# container manager passed in.
LoadCredential = [
certCredential
keyCredential

9
nix/agent-modules/default.nix
View file

@ -34,6 +34,12 @@ let
sets it. Pin it host-side with
services.hyperhive.deploy.hive-controller.claudeCodePackage.
'';
queueRemoved = ''
The harness reads the queue's address at runtime from HIVE_AGENT_NATS_URL,
which the hive forwards into every container, and authenticates with the
agent's own secret from the swarm secret store, so no agent build sets a
queue option. Remove the setting.
'';
# `claude` on PATH. A hive that pins a claude-code
# (`services.hyperhive.deploy.hive-controller.claudeCodePackage`) hands its
@ -81,7 +87,6 @@ in
./otel.nix
./otel-removed.nix
./packages.nix
./queue.nix
./queue-identity.nix
./user.nix
./screen.nix
@ -101,6 +106,8 @@ in
"agent"
"claudeCodePath"
] claudeCodePathRemoved)
(lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "queue" "natsUrl" ] queueRemoved)
(lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "queue" "tokenEndpoint" ] queueRemoved)
];
options.services.hyperhive.agent.icon = lib.mkOption {

20
nix/agent-modules/queue-identity.nix
View file

@ -1,13 +1,11 @@
# This agent's own credential for the swarm queue, read from the store by the
# harness itself.
#
# The credential beside it in ./queue.nix is keyed per **hive**: one OIDC
# client minted at deploy time and handed to every agent container on the
# hive, so at the queue's auth callout one agent is indistinguishable from its
# co-hived neighbours. This one is keyed per **agent** — `swarm-controller`
# mints it at agent creation into `swarm/agents/<agent>/queue`
# (`swarm_secret_client::queue::agent_queue_path`), at swarm level, with no
# hive anywhere in the chain.
# Keyed per **agent**, so at the queue's auth callout one agent is
# distinguishable from its co-hived neighbours: `swarm-controller` mints it at
# agent creation into `swarm/agents/<agent>/queue`
# (`swarm_secret_client::queue::agent_queue_path`), at swarm level, with no hive
# anywhere in the chain.
#
# The agent authenticates to the store as itself, with the certificate
# ./bao.nix already proves it can log in with, and reads its own path. A hive
@ -42,11 +40,7 @@ let
# too.
serverCaCredential = "hive-agent-bao-server-ca";
# The store's address is the whole switch, exactly as in ./bao.nix — and
# deliberately *not* the queue coordinates in ./queue.nix. Those are the
# hive's, and gating a swarm-minted per-agent credential on a hive-level
# option would put the hive back in a path whose entire purpose is not
# having one.
# The store's address is the whole switch, exactly as in ./bao.nix.
configured = cfg.addr != null;
in
{
@ -54,7 +48,7 @@ in
systemd.services.hive-agent = {
# A bare id, no path: the terse `LoadCredential=` form that inherits a
# credential the service *manager* received, which is what the container
# manager passed in. ./bao.nix and ./queue.nix state the same shape.
# manager passed in. ./bao.nix states the same shape.
serviceConfig.LoadCredential = [ serverCaCredential ];
# The live certificate pair exists once ./bao.nix's unit has run.
after = [ "hive-agent-bao-identity.service" ];

117
nix/agent-modules/queue.nix
View file

@ -1,117 +0,0 @@
# The hive's shared swarm-queue client credential, imported into the harness
# unit's credentials directory for `swarm-logs` (./logs.nix) to authenticate
# with. The harness itself connects to the queue with the agent's own
# credential (./queue-identity.nix) and reads neither file.
#
# The credential is two files, the secret and the client id beside it, so a
# reader never spells `hive-<name>-agent` a second time.
#
# ⚠️ The credential arrives as a systemd credential and NOT as a bind mount,
# and the mode is why: the host file is `root:0600` and the reader runs as the
# unprivileged agent user. nspawn's `--load-credential` (written by
# `hive_c0re::lifecycle::host_config`) is read by the container manager as
# root and re-exposed under this unit's own `User=`; a bind would deliver a
# file the agent user cannot open.
{
lib,
config,
...
}:
let
cfg = config.services.hyperhive.agent.queue;
configured = cfg.natsUrl != null && cfg.tokenEndpoint != null;
# The two ids `hive_c0re::lifecycle::host_config` forwards under. Neither
# side can discover the other's spelling, so a rename is a rename there too.
secretCredential = "hive-queue-agent-secret";
clientIdCredential = "hive-queue-agent-client-id";
# Where systemd materialises this unit's credentials. `%d` only expands
# *inside* a unit, so `swarm-logs` (./logs.nix), which is not one, needs the
# literal; deriving it from the unit name here is what keeps the two from
# disagreeing about which unit's credentials they mean.
credentialsDir = "/run/credentials/hive-agent.service";
in
{
options.services.hyperhive.agent.queue = {
natsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "tls://nats.example.com:4222";
description = ''
Where the swarm queue listens, as this container reaches it.
Set by the generated meta flake from the host's
{option}`services.hyperhive.deploy.hive-controller.queue.agentNatsUrl`,
which names the queue rather than a loopback address — inside this
container `127.0.0.1` is the agent itself.
`null` means this hive has not been given the queue's address for its
agents: the harness unit imports no credential, and the harness fails
at boot, since it cannot run without a queue connection. It is
deliberately not defaulted to anything: a guessed address builds fine
and talks to the wrong machine.
'';
};
tokenEndpoint = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "https://auth.example.com/api/oidc/token";
description = ''
The swarm IdP's OAuth2 token endpoint, where `swarm-logs` mints a
`client_credentials` token for the client named in the delivered
credential.
The harness unit imports that credential only when this and
{option}`services.hyperhive.agent.queue.natsUrl` are both set.
'';
};
# One declaration, two readers. The credential ids are this module's —
# `hive_c0re::lifecycle::host_config` and the `LoadCredential=` below are
# the pair that has to agree on them — and a second consumer that spelled
# them again would be a second source of truth for a string whose
# mismatch is a file that is simply not there.
clientIdFile = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "${credentialsDir}/${clientIdCredential}";
description = ''
Path the agent's OIDC client id is delivered at, for a consumer
outside the harness unit. Read-only: it is a fact about where the
credential lands, not a knob — see {option}`services.hyperhive.agent.logs.queryUrl`
for the consumer this exists for.
'';
};
clientSecretFile = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "${credentialsDir}/${secretCredential}";
description = ''
Path the agent's OIDC client secret is delivered at. Read-only for
the same reason as {option}`services.hyperhive.agent.queue.clientIdFile`.
🩸 A PATH and never a value. The file is `0400` to the agent user and
is read at the moment of a token request; nothing in this tree puts
its contents in an environment variable, where `/proc/<pid>/environ`
would publish them to every process in the container.
'';
};
};
config = lib.mkIf configured {
systemd.services.hive-agent = {
# Bare ids, no paths: this is the terse `LoadCredential=` form that
# inherits a credential the service *manager* received, which is what
# the container manager passed in. `man systemd.exec` also makes that
# form non-fatal when the credential is absent, which is exactly the
# behaviour a hive whose publisher has not run yet needs.
serviceConfig.LoadCredential = [
secretCredential
clientIdCredential
];
};
};
}

14
nix/container-modules/swarm-secret-publisher.nix
View file

@ -1,6 +1,6 @@
# The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its own and its agents'
# credentials, a service or the controller its own. It mints nothing: every
# whoever needs one can read it there: a hive, a service or the controller its
# own. It mints nothing: every
# value is an OIDC client secret authelia minted, and this unit copies it.
#
# Imported inside authelia's container, beside the mint: it reads each secret
@ -61,14 +61,6 @@ in
example = "secret/swarm/services/swarm-grafana/oidc/client";
description = "KV path its reader fetches the value from.";
};
withClientId = lib.mkOption {
type = lib.types.bool;
default = false;
description = ''
Also store the client id beside the value, as `client_id`, for
a reader that has to reconstruct it from the store alone.
'';
};
};
}
);
@ -170,7 +162,7 @@ in
# `value="$(cat "$src")"` would publish it to /proc for anyone in
# the container to read.
bao kv put ${lib.escapeShellArg s.path} \
value=@"$src"${lib.optionalString s.withClientId " client_id=${lib.escapeShellArg s.clientId}"}
value=@"$src"
published=$((published + 1))
else
# Not an error: authelia mints on its FIRST BOOT, so an absent file

1
nix/host-modules/default.nix
View file

@ -47,7 +47,6 @@
./glue-nats-bao-identity.nix
./glue-nats-oidc-client.nix
./glue-nats-wireguard.nix
./glue-queue-agent-credential.nix
./glue-builder-oidc-client.nix
./glue-ci-metrics-otel-oidc-client.nix
./glue-ci-runner-otel-oidc-client.nix

37
nix/host-modules/deploy.nix
View file

@ -53,6 +53,14 @@ let
puts it there. Remove this definition; the file it named is read by
nothing now and can be deleted.
'';
# Shared by the removed options of the hive's agent queue credential reader.
queueAgentCredentialRemoved = ''
A hive no longer fetches a shared queue credential for its agents: each
agent reads its own secret from the swarm's secret store under its own
certificate. Remove this definition; the leaf or directory it named is read
by nothing now and can be deleted.
'';
in
{
imports = [
@ -95,6 +103,35 @@ in
"statusPublish"
"clientSecretFile"
] statusSecretRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"bao"
"queueAgentCommonNamePrefix"
] queueAgentCredentialRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"bao"
"queueAgentClientCertFile"
] queueAgentCredentialRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"bao"
"queueAgentClientKeyFile"
] queueAgentCredentialRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"hive-controller"
"queue"
"agentCredentialDir"
] queueAgentCredentialRemoved)
# Same type, same meaning, new path — so a rename carries it exactly
# and existing configs keep evaluating with one warning naming both

4
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -31,10 +31,6 @@ let
# Reader unit → the gate its own module defines it under.
readersHere = {
# ./glue-queue-agent-credential.nix
swarm-bao-queue-agent =
deployCfg.hive-controller.enable
&& havePair baoDeploy.queueAgentClientCertFile baoDeploy.queueAgentClientKeyFile;
# ./swarm-grafana.nix
swarm-bao-grafana-oidc = deployCfg.grafana.enable;
# ./swarm-otel.nix

26
nix/host-modules/glue-bao-tls.nix
View file

@ -95,15 +95,13 @@ in
# 🩸 The readers below each present a leaf of their own rather than
# `client.pem` above. The leaf is what bao sees, so this pairing is the
# whole of what turns "one principal with the union of their grants" into
# one principal per grant — see ./swarm-bao.nix's `perHiveReaders` block
# for the grants themselves.
# one principal per grant — see ./swarm-bao.nix's `readStanza` block for
# the grants themselves.
#
# Defaulted here rather than in `glue-<consumer>-bao-identity.nix` files:
# those exist where the consumer is a CONTAINER with a
# `deploy.<service>.*` namespace of its own to point at. These are host
# units reading the store, which is the pairing this file already is.
queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem";
queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem";
grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem";
grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem";
otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem";
@ -223,20 +221,10 @@ in
# principal holding the union of their grants, and the union would
# include every agent credential in the swarm.
#
# ⚠️ The first carries THIS hive's name in the subject, unlike the three
# service leaves above. Its grant names one hive's path, because a queue
# credential lives under `swarm/hives/<name>/` and every hive runs a
# reader for its own; ./swarm-bao.nix writes one role per hive in the
# swarm directory to match. The OIDC readers need no such segment — a
# client is registered once per swarm.
#
# Minted whether or not the consumer runs here, for the reason the three
# leaves above give: on a hive that does not run the store this is the
# file an operator copies, and a leaf that only appears where its
# consumer does is one nobody can copy from anywhere.
[ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth
[ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \
${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth
@ -344,14 +332,14 @@ in
${lib.optionalString deployCfg.hive-controller.enable ''
# This hive's gateway-leaf identity, which opens
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its
# subject carries this hive's name, like the queue-agent leaf
# above, because ./swarm-bao.nix writes one role
# per hive. Only where a hive runs; the other hives' are below.
# subject carries this hive's name, because ./swarm-bao.nix writes
# one role per hive. Only where a hive runs; the other hives' are
# below.
[ -s ${pkiDir}/hive-leaf.pem ] || ${signLeaf} ${pkiDir} hive-leaf \
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${clientCn}"} "" clientAuth
''}
# The reader, queue-agent and gateway-leaf leaves of every other hive in
# The reader and gateway-leaf leaves of every other hive in
# the swarm directory: the files an operator copies to that hive's
# host. One directory per hive, with the basenames this hive's leaves
# use, so a hive's set is one directory to copy.
@ -364,8 +352,6 @@ in
install -d -m 0700 ${d}
[ -s ${d}/client.pem ] || ${signLeaf} ${d} client \
${lib.escapeShellArg h} "" clientAuth
[ -s ${d}/queue-agent.pem ] || ${signLeaf} ${d} queue-agent \
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${h}"} "" clientAuth
[ -s ${d}/hive-leaf.pem ] || ${signLeaf} ${d} hive-leaf \
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${h}"} "" clientAuth
''

284
nix/host-modules/glue-queue-agent-credential.nix
View file

@ -1,284 +0,0 @@
# Glue: this hive's agent queue credential comes out of the secret store.
#
# A cert login that fails LOUDLY because every state it fails on is one a
# retry fixes, then a read that degrades QUIETLY because no retry turns "no
# value there" into a value.
#
# ⚠️ Two files, not one, and that is the consumer's shape rather than a
# preference. `swarm-logs` inside the agent reads the secret and the client id
# each from its own file (`<prefix>_OIDC_CLIENT_SECRET_FILE`,
# `<prefix>_OIDC_CLIENT_ID_FILE`), so splitting them here is what lets
# hive-c0re hand both to an agent without parsing anything.
#
# ⚠️ An ABSENT file means this hive's agents do not connect to the queue, and
# that is correct rather than degraded. The publisher runs on the authelia
# host and authelia mints on its first boot, so "nothing at that path yet" is
# the ordinary early state of a swarm. Nothing here writes a local stand-in:
# there is no such thing as a locally valid OIDC client secret, so a placeholder would turn a hive that cannot connect
# into one that is refused, which reaches the agent as a timeout.
#
# 📌 This runs wherever a client identity is configured, NOT only where the
# store is. `deploy.bao.enable` would have been the co-location assumption
# itself; the reader needs a certificate, not a neighbour. On the store's own
# host ./glue-bao-tls.nix supplies one as a `mkDefault`; elsewhere an operator
# places the leaf and names it, and the same unit works. It also needs this
# host to run a hive (`deploy.hive-controller.enable`): the credential is for
# the agents hive-c0re creates here at runtime, and the unit writes under
# hive-c0re's own state directory.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
# What decides whether this unit exists at all. A reader is defined by holding
# a certificate the store accepts, and that is true on the store's own host
# and on a hive three networks away for exactly the same reason.
#
# 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's. Bao
# matches a cert-auth role on the CN, so one leaf shared by four readers
# would be ONE principal holding the union of four grants: read on
# `swarm/agents/*` AND `swarm/hives/<hive>/*`, when this unit reads one
# queue credential and nothing else. Its own leaf carries
# `<queueAgentCommonNamePrefix>-<hive>` and its role grants the single path
# below — still this hive's own, so the narrowing costs no reach.
haveClientIdentity =
baoDeploy.queueAgentClientCertFile != null && baoDeploy.queueAgentClientKeyFile != null;
# Does this host read the store at all — the hive's own leaf, which is the
# one thing a remote-store deployment has to place by hand. Only used to
# decide whether a missing per-principal leaf is a mistake or a deployment
# that has no store: a host holding neither is the supported no-store
# shape, and one holding this pair but not the pair above reads the store
# without a reader identity of its own.
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir;
secretFile = "${credentialDir}/secret";
clientIdFile = "${credentialDir}/client_id";
# Where the credential lives in the store, spelled from the same pieces the
# writer uses. Whoever writes it and whoever reads it must agree, and the
# agreement belongs in one visible place.
#
# ⚠️ The `hives/<name>` segment is not decoration — it is what the reader's
# own grant covers, so a path outside it is a 403 rather than a miss, however
# correct it looks. `swarm-secret-client`'s `queue::agent_client_path` builds
# the same string from the same pieces; this literal is the nix half of that
# one agreement.
#
# `hiveName` has no fallback here for the reason ./glue-bao-tls.nix gives at
# its own use of it: it is asserted set wherever
# `deploy.hive-controller.enable` is.
credentialPath = "secret/swarm/hives/${hyperhiveCfg.hiveName}/queue/agent";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
storeRetry = import ./lib/store-retry.nix { };
in
{
options.services.hyperhive.deploy.hive-controller.queue = {
agentCredentialDir = lib.mkOption {
type = lib.types.path;
default = "/var/lib/hyperhive/queue-agent";
description = ''
Host directory holding this hive's agent queue credential: `secret`
(the OIDC client secret, `0600`) and `client_id` (the client that
secret authenticates, `0644` — it is sent to the token endpoint on
every connection and is not itself a secret).
Both files are written by `swarm-bao-queue-agent.service`, which also
creates this directory. Named here because nothing else creates it:
the parent `/var/lib/hyperhive` belongs to `hive-c0re.service`'s
`StateDirectory=`, which re-applies its own mode on every start, and a
second declaration of *that* path is the two-mechanisms-one-path trap
./hive-gateway/default.nix records — so this directory is the unit's
to make and the parent stays c0re's.
Absent files mean this hive has no queue credential yet, which is what
a swarm looks like before the publisher in authelia's container has run.
'';
};
};
config = lib.mkMerge [
# ⚠️ A SEPARATE arm from the unit below, and that separation is the whole
# mechanism: the unit's arm is gated on `haveClientIdentity`, so an
# assertion written inside it could never be reached in the state it
# exists to report.
#
# Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the
# same refusal, named to this principal's own pair. Where Grafana's gate
# is `deploy.grafana.enable`, this reader's is the hive's own: every hive
# runs one for its own agents, so on a hive, reading the store at all is
# what asks for it. Hence `hiveReaderIdentity` — a host holding no hive
# leaf has no store to read and nothing is missing.
(lib.mkIf (deployCfg.hive-controller.enable && hiveReaderIdentity) {
assertions = [
{
assertion = haveClientIdentity;
message = ''
This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile
is set), so it needs the agent queue credential reader's own client
identity: set both
services.hyperhive.deploy.bao.queueAgentClientCertFile
services.hyperhive.deploy.bao.queueAgentClientKeyFile
swarm-bao-queue-agent.service fetches this hive's agent queue
credential out of the store, and without these it is not rendered
at all — leaving hive-c0re with no queue credential and no unit
that would ever have written one.
Every hive runs this reader for its own agents, so unlike the other
three principals there is no per-service toggle that turns it off:
a hive that reads the store at all is a hive that needs this pair.
⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one
is the hive's, and its grant reads every agent credential in the
swarm and this hive's own secrets; this role reads the one
queue-credential path. Pointing this option at the hive's leaf
would evaluate, deploy and log in — and undo the split.
On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere copy the pair from
/var/lib/swarm-bao-pki/hives/<hive>/ on the store's host and name it
here — see docs/swarm/bao.md.
'';
}
];
})
(lib.mkIf (deployCfg.hive-controller.enable && haveClientIdentity) {
systemd.services.swarm-bao-queue-agent = {
description = "fetch this hive's agent queue credential from the swarm secret store";
# Every one of these names a unit that exists only where the store runs,
# so the ordering is conditional even though the read is not: off-host
# there is nothing local to wait for, and the timeout below is what
# bounds the attempt instead. `Wants=` pki, for
# ./glue-bao-readers-store-order.nix's reason.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
# Ordered before hive-c0re, so no agent container renders ahead of an
# attempt at its credential. `Wants=`, not `Requires=`: a store this
# unit can't reach delays hive-c0re's start by one attempt
# (`TimeoutStartSec` below) rather than failing it — hive-c0re starts
# after that attempt with whatever credential is on disk, while
# ./lib/store-retry.nix keeps retrying in the background.
before = [ "hive-c0re.service" ];
wantedBy = [
"multi-user.target"
"hive-c0re.service"
];
path = [
baoDeploy.package
pkgs.coreutils
];
# ./lib/store-retry.nix.
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
# What actually bounds each attempt at the reads below. Stated here
# rather than left to systemd's default, so the number a boot waits on
# is in the file that waits.
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile;
BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
${atomicWriteSecret}
# `bao`'s own message is the only thing separating a missing value from
# a refused identity from an unreachable host. This unit's degraded
# mode is correct for all three, so it reports which one rather than
# asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Fails LOUDLY, unlike the reads below: the three states a login
# failure covers — store not up, sealed, role not written yet — are all
# things a retry fixes, and `Restart=on-failure` above is what retries.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
# Two reads of one object rather than one `-format=json` parsed with
# `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot
# actually disagree — the client id is derived from this hive's name, so
# a rotation landing between these two calls changes the secret and
# rewrites the same id.
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
# Both or neither: a client that finds one of the two comes up "fine"
# and never connects.
if [ -z "$secret" ] || [ -z "$client_id" ]; then
echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2
exit 0
fi
install -d -m 0755 ${lib.escapeShellArg credentialDir}
atomic_write_secret 0600 "" ${lib.escapeShellArg secretFile} "$secret"
# `0644` on purpose: an OIDC client id is presented to the token
# endpoint on every connection and is public by construction.
atomic_write_secret 0644 "" ${lib.escapeShellArg clientIdFile} "$client_id"
'';
};
})
];
}

3
nix/host-modules/hive-c0re/default.nix
View file

@ -333,8 +333,7 @@ in
# from an unprivileged daemon without widening either.
# The gate is the identity, never `deploy.bao.enable`: a hive that
# reads a store on another machine holds one of these and runs no
# store. ./glue-queue-agent-credential.nix gates its own reader the
# same way.
# store. ../swarm-otel.nix gates its own reader the same way.
lib.optionals baoClientIdentity [
"bao-client.pem:${baoDeploy.clientCertFile}"
"bao-client-key.pem:${baoDeploy.clientKeyFile}"

29
nix/host-modules/hive-c0re/environment.nix
View file

@ -11,9 +11,8 @@ let
cfg = config.services.hyperhive.c0re;
baoDeploy = config.services.hyperhive.deploy.bao;
baoCfg = config.services.hyperhive.swarm.bao;
# The same test ../glue-queue-agent-credential.nix applies, and for the same
# reason: a reader is defined by holding a certificate the store accepts,
# not by sharing a host with the store. Deriving this from
# A reader is defined by holding a certificate the store accepts, not by
# sharing a host with the store. Deriving this from
# `deploy.bao.enable` would be the co-location assumption itself.
haveBaoClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
in
@ -222,31 +221,15 @@ in
# the one the roster gave it.
HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}";
}
// {
# Where ../glue-queue-agent-credential.nix lands the AGENTS' queue
# credential. Read by `hive_c0re::lifecycle::host_config`, which stats the
# two files and forwards them into each container as systemd credentials.
# Never read for its contents here: the secret is `0600` root-owned and
# this daemon is `hive-core`, which is exactly why the transport is a
# credential rather than a bind mount.
HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR = toString config.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
}
//
# The agents' half of the same queue, forwarded by `hive_c0re::meta` into
# every container. A pair, gated together, because an agent that got one of
# them would report a half-configured queue instead of none.
# every container.
#
# ⚠️ By name, like `HIVE_C0RE_NATS_URL` above, and never a loopback
# address: inside a container `127.0.0.1` is the agent itself.
lib.optionalAttrs
(
config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null
&& config.services.hyperhive.swarm.statusPublish.tokenEndpoint != null
)
{
HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl;
HIVE_AGENT_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
}
lib.optionalAttrs (config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null) {
HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl;
}
//
# Where an agent reads the swarm's logs. Gated on the store identity below,
# because `swarm-logs` authenticates with the agent's own queue secret, read

Some files were not shown because too many files have changed in this diff Show more