vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-secret-client: hives no longer read swarm/services/*

Browse source 
The hive policy rendered by policy::render granted every hive cert read
on secret/data/swarm/services/*, which holds each swarm service's OIDC
client secret. No reader of that tree authenticates with a hive cert:
grafana, the otel collector and the store forwarder each read with
their own leaf, and the bao UI's secret is read by the store's granter.

render() now emits two stanzas: swarm/agents/* and the hive's own
swarm/hives/<hive>/*. Kind::Service had no other caller and is removed.
Docs and comments that described the grant as a decision are rewritten
to the current state.

Closes #4960
This commit is contained in:
atlas 2026-10-04 22:32:28 +02:00 • committed by mara
commit 2505008b83
13 changed files with 82 additions and 136 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/swarm/credentials.md
View file

@ -69,7 +69,7 @@ of the cell says how.
| `swarm/hives/<hive>/matrix/appservice-token` | one minter, on the authelia host | the hive process that presents the token to its homeserver, under the hive's own certificate | must be stated | must be stated |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/services/<clientId>/oidc/client` | authelia | the service process that presents the client secret, under the certificate of the host it runs on | must be stated | must be stated |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate; the bao UI's by the store's granter | must be stated | must be stated |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES -->

39
docs/trust-boundary/security.md
View file

@ -72,37 +72,24 @@ advertises a boundary it doesn't actually hold. A wide grant that says what it
is beats a narrow one that only looks narrow.
What still holds: the grant is **read-only** (a hive can't write an agent's
credential, so it can't hand itself an agent's identity), and it reaches three
prefixes and nothing else in the store — every agent's credentials, the reader's
_own_ entry under the hive namespace, which names the hive asking and so widens
nothing between them, and every **swarm service's** OIDC client secret.
credential, so it can't hand itself an agent's identity), and it reaches two
prefixes and nothing else in the store — every agent's credentials, and the
reader's _own_ entry under the hive namespace, which names the hive asking and
so widens nothing between them.
That third prefix has the same shape of reason as the first, and the same honest
cost. A swarm service (Grafana and the swarm collector are the two there today)
registers **one** client for the whole swarm, so its credential's path names
the service and never the host — and which hive runs a given service is a
`deploy.*` fact, per-host by definition, so nothing swarm-wide exists to scope
the grant to. The host running such a service has no store identity of its own
either; it reads with the certificate of the hive it is. **Any hive can
therefore read any swarm service's client secret**, which is worth what it
buys: every swarm service gets its credential the same way from anywhere,
instead of only where the identity provider happens to sit. Giving such a
service its own store identity is what would remove this rather than re-scope
it.
A hive reads no **swarm service's** OIDC client secret. Every reader of
`swarm/services/` presents a certificate that isn't a hive's: Grafana, the swarm
collector and the store forwarder each fetch theirs with a leaf of their own,
and the store's granter fetches the bao UI's.
**A tracked follow-up** narrows this, with the two candidate directions: scope
the grant per hive (and pay for the re-emission), or give each agent container
its own store identity so credentials never pass through a hive at all.
### Per-hive credentials live under the hive kind, not beside a service
### Per-hive credentials live under the hive kind
`swarm_secret_client::policy` scopes the **hive** stanza narrowly (unlike the
agent and service stanzas above) because a hive's own path names it, so
scoping to the reader's own name costs nothing and drifts nowhere. That only
holds if a credential that must be one-per-hive is actually stored under the
hive kind.
agent stanza above) because a hive's own path names it, so scoping to the
reader's own name costs nothing and drifts nowhere. That only holds if a
credential that must be one-per-hive is actually stored under the hive kind.
**A credential that must be one-per-hive goes under `Kind::Hive`**; putting it
under `Kind::Service` and expecting the grant to scope it — that's the mistake
under the agent prefix and expecting the grant to scope it — that's the mistake
this note exists to stop.
### Threat model: prompt injection → confused deputy

3
hive-c0re/src/matrix.rs
View file

@ -218,8 +218,7 @@ fn persist_sender_token(path: &std::path::Path, access_token: &str) -> Result<()
/// this process's environment, so the store read costs no plumbing through
/// [`ensure_all`]. That name is also the path's own hive segment, which is
/// what makes this read reach **this** hive's token and 403 on any other's:
/// the grant behind it is the hive's own `swarm/hives/<name>/*` stanza, not
/// the swarm-wide `swarm/services/*` one the shared token used to sit under.
/// the grant behind it is the hive's own `swarm/hives/<name>/*` stanza.
///
/// `None`, never an error, for every way this can come up empty — no hive
/// name, no `BAO_*` identity, an unreachable store, nothing at the path. All

9
nix/host-modules/swarm-bao.nix
View file

@ -759,7 +759,7 @@ let
#
# 🩸 Each of the four reads exactly ONE path in the store, and until this
# split each did it holding `deploy.bao.clientCertFile` — the hive's own leaf,
# whose policy (`swarm-secret-client`'s `policy::render`) grants read on
# whose policy (`swarm-secret-client`'s `policy::render`) granted read on
# `swarm/agents/*`, `swarm/hives/<hive>/*` AND `swarm/services/*`. Four
# principals presenting one certificate are ONE principal to bao, so the union
# of what the four need was the only grant expressible: the unit that fetches
@ -1142,8 +1142,7 @@ let
# own and lands it in a collector's container — because that route is
# proven and there is no second one worth inventing.
#
# Where the publisher on authelia's host leaves it. The `services` segment
# is `swarm-secret-client`'s `path::Kind::Service`: a swarm service's client
# Where the publisher on authelia's host leaves it. A swarm service's client
# is registered once for the whole swarm, so its secret is one value. The
# `swarm-forwarder-oidc` role's grant is this one path, so any other answers
# 403 however correct it looks.
@ -2288,8 +2287,8 @@ in
environment = {
BAO_ADDR = "https://${cfg.domain}:${toString cfg.port}";
# 🩸 Its OWN leaf, not `clientCertFile`: the hive's grant reads every
# agent's credential and every service's OIDC secret, and this unit
# needs one path. The pair is asserted set above.
# agent's credential, and this unit needs one path. The pair is
# asserted set above.
BAO_CLIENT_CERT = baoDeploy.forwarderOidcClientCertFile;
BAO_CLIENT_KEY = baoDeploy.forwarderOidcClientKeyFile;
}

3
nix/host-modules/swarm-grafana.nix
View file

@ -238,8 +238,7 @@ let
hostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${secretPath}";
hostSecretDir = builtins.dirOf hostSecretPath;
# Where the publisher on authelia's host leaves this client's secret. The
# `services` segment is `swarm-secret-client`'s `path::Kind::Service` — a
# Where the publisher on authelia's host leaves this client's secret. A
# swarm service's client is registered once for the whole swarm, so its
# secret is one value and not one per hive. Both ends compose this from the
# same swarm-wide client id, so a rename cannot leave one of them behind.

6
nix/host-modules/swarm-otel.nix
View file

@ -272,10 +272,8 @@ let
# Where the publisher on authelia's host leaves this client's secret —
# composed from the same swarm-wide `clientId` the registration in
# ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of
# them behind. The `services` segment is `swarm-secret-client`'s
# `path::Kind::Service`, the same prefix ./swarm-grafana.nix reads its own
# client secret under — one write grant in ./swarm-bao.nix and one hive read
# grant in `policy::render` already cover it.
# them behind. The same prefix ./swarm-grafana.nix reads its own client
# secret under; the `swarm-otel-oidc` role in ./swarm-bao.nix grants the read.
storeSecretPath = "secret/swarm/services/${cfg.clientId}/oidc/client";
# The operator-configured upstream, named once: the same exporter carries

8
nix/host-modules/swarm-secret-publisher.nix
View file

@ -60,8 +60,7 @@ let
# The swarm's own services, as opposed to its hives. One client for the whole
# swarm rather than one per hive, so one value in the store rather than a copy
# each: `swarm/services/<id>/oidc/client`, under the `services` kind
# `swarm-secret-client`'s `path::Kind` declares.
# each: `swarm/services/<id>/oidc/client`.
#
# The ids come from `swarm.*`, which is identical on every host — that is what
# lets this host name a service's client while running none of them, and it is
@ -267,9 +266,8 @@ in
fi
'') serviceClientIds}
# The controller's own client, under `controller/` rather than
# `services/`: every hive reads `services/*`, and this client may read
# every hive's status. The nix half of
# The controller's own client, under `controller/`, which no hive
# reads: this client may read every hive's status. The nix half of
# `swarm_secret_client::queue::controller_client_path`.
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${controllerClientId}.secret"}
if [ -s "$src" ]; then

2
nix/module-eval/bao-grants.nix
View file

@ -482,7 +482,7 @@ let
# ── the four readers that used to share the hive's own leaf ──────────────
#
# 🩸 Until this split all four presented `deploy.bao.clientCertFile`, whose
# policy grants read on `swarm/agents/*`, `swarm/hives/<hive>/*` AND
# policy granted read on `swarm/agents/*`, `swarm/hives/<hive>/*` AND
# `swarm/services/*`. Four principals behind one certificate are one
# principal to bao, so the only expressible grant was the union: the unit
# fetching Grafana's OIDC secret could fetch every agent credential in the

2
nix/module-eval/secret-publisher.nix
View file

@ -88,7 +88,7 @@ let
{
# The controller reads this exact path
# (`swarm_secret_client::queue::controller_client_path`, pinned by that
# crate's own test). Not under `services/`, which every hive reads: this
# crate's own test). Under `controller/`, which no hive reads: this
# client may read every hive's status.
name = "the publisher writes the controller's client under controller/, never services/";
ok =

33
swarm-secret-client/src/matrix.rs
View file

@ -60,18 +60,10 @@ pub fn hive_localpart(hive: &str) -> String {
/// The path holding `hive`'s matrix sender account's homeserver access token.
///
/// Keyed per **hive**, like [`appservice_token_path`] below and unlike the
/// per-homeserver constant this replaced. The old path
/// (`swarm/services/matrix/sender-token`) held **one value for the whole
/// swarm** and sat under the [`Kind::Service`] tree that
/// [`crate::policy::render`] grants *every* hive read on — so every hive could
/// read, and act as, the one shared account.
///
/// Under [`Kind::Hive`] the grant that reaches it is the hive's **own**
/// stanza, `secret/data/swarm/hives/<this hive>/*`, which interpolates the
/// name: hive `alpha` reads its own token and gets a 403 on `beta`'s. Nothing
/// was added to the policy to do that — the path moved out from under the
/// broad grant, which is the half that makes it real.
/// Keyed per **hive**, like [`appservice_token_path`] below. Under
/// [`Kind::Hive`] the grant that reaches it is the hive's **own** stanza,
/// `secret/data/swarm/hives/<this hive>/*`, which interpolates the name: hive
/// `alpha` reads its own token and gets a 403 on `beta`'s.
///
/// # Errors
/// [`Error::PathSegment`] when `hive` contains anything but `[A-Za-z0-9_-]`,
@ -112,9 +104,8 @@ pub(crate) const CONTROLLER: &str = "swarm-controller";
///
/// The matrix container mints it and publishes it here; `swarm-controller`
/// reads it to create agents' accounts. Under [`Kind::Controller`] because
/// that is the one kind [`crate::policy::render`] grants no hive: every hive
/// reads `agents/*`, its own `hives/<hive>/*` and `services/*`, so under any
/// of those this admin credential would be readable by every hive.
/// [`crate::policy::render`] grants no hive that kind: every hive reads
/// `agents/*` and its own `hives/<hive>/*`, and nothing else.
///
/// # Errors
/// Never in practice: the name segment is a constant. The `Result` is
@ -174,18 +165,6 @@ mod tests {
);
}
#[test]
fn the_sender_token_left_the_tree_every_hive_can_read() {
// 🩸 The whole point of the move. `policy::render` grants every hive
// read on `secret/data/swarm/services/*` — a deliberate grant that
// stays, because a service's OIDC secret is read with the certificate
// of whatever hive hosts it. What must not stay is this credential
// sitting inside it: under `services/` one shared token was readable
// by every hive, which is one matrix identity for the whole swarm.
let p = sender_token_path("pr1ma").expect("legal");
assert!(!p.starts_with("swarm/services/"), "{p}");
}
#[test]
fn one_hives_sender_token_is_not_another_hives() {
// The property the per-hive path exists for: two hives never name the

8
swarm-secret-client/src/path.rs
View file

@ -40,9 +40,6 @@ pub enum Kind {
/// is what uses it.
#[strum(serialize = "hives")]
Hive,
/// One swarm service — the things a swarm runs beside the controller.
#[strum(serialize = "services")]
Service,
/// The controller itself. There is one per swarm, so the name segment
/// below it does not vary; the shape stays uniform anyway, because one
/// grant pattern over `<kind>/<name>` is cheaper than a special case.
@ -53,7 +50,7 @@ pub enum Kind {
impl Kind {
/// Every kind, so callers that must cover the whole set can iterate rather
/// than restate it — a second list is a list that drifts.
pub const ALL: [Kind; 4] = [Kind::Agent, Kind::Hive, Kind::Service, Kind::Controller];
pub const ALL: [Kind; 3] = [Kind::Agent, Kind::Hive, Kind::Controller];
/// What to call the name in an error — singular, because the message reads
/// "hive name ... is not a single path segment".
@ -62,7 +59,6 @@ impl Kind {
match self {
Kind::Agent => "agent",
Kind::Hive => "hive",
Kind::Service => "service",
Kind::Controller => "controller",
}
}
@ -175,7 +171,7 @@ mod tests {
// decision. Spelling each one out is what makes adding a kind a
// deliberate edit.
let segments: Vec<&str> = Kind::ALL.iter().map(|k| (*k).into()).collect();
assert_eq!(segments, ["agents", "hives", "services", "controller"]);
assert_eq!(segments, ["agents", "hives", "controller"]);
}
#[test]

94
swarm-secret-client/src/policy.rs
View file

@ -16,16 +16,15 @@
//! narrow one that only looks narrow. The narrower shapes, and what they would
//! cost, are in `docs/trust-boundary/security.md`.
//!
//! ⚠️ The **service** stanza is wide for the same shape of reason: a swarm
//! service's client is registered once per swarm, so its path names the service
//! and never the host, and which hive runs a service is a `deploy.*` fact with
//! no swarm-wide spelling to scope against. Cost: the same doc.
//!
//! The **hive** stanza has no such problem and is therefore narrow: that path
//! names its principal, so scoping it to the reader's own name costs nothing
//! and drifts nowhere. Do not widen it to match its neighbours — the asymmetry
//! is the point, and only holds if a one-per-hive credential is stored under
//! [`Kind::Hive`], not [`Kind::Service`] — see docs/trust-boundary/security.md.
//! and drifts nowhere. Do not widen it to match the agent stanza — the
//! asymmetry is the point, and only holds if a one-per-hive credential is
//! stored under [`Kind::Hive`] — see docs/trust-boundary/security.md.
//!
//! A hive reads nothing under `swarm/services/` or `swarm/controller/`. A swarm
//! service's OIDC client secret is read under a certificate that is not a
//! hive's: `swarm-bao.nix`'s per-service readers, and its granter for the UI's.
//!
//! Rendering stays separate from writing so the text can be asserted with no store to talk to.
@ -127,8 +126,8 @@ fn list_stanza(path: &str) -> String {
format!("path \"{path}\" {{\n capabilities = [\"list\"]\n}}\n")
}
/// Render `hive`'s policy document: read on every agent's credentials, on this
/// hive's own, and on the swarm services'.
/// Render `hive`'s policy document: read on every agent's credentials and on
/// this hive's own.
///
/// The name is the only input, and it is deploy-time — so the document is
/// still a deploy-time object rather than derived state with a re-emission to
@ -136,15 +135,11 @@ fn list_stanza(path: &str) -> String {
///
/// Read-only: the controller mints these and never reads one back.
///
/// ⚠️ The controller kind is deliberately absent: a hive has no business
/// reading the credentials of the thing that provisions it. Adding it is a
/// boundary decision, not a consequence of the namespace growing.
///
/// The **service** kind is granted, and that was such a decision rather than a
/// consequence: a service whose identity provider is on another host reads its
/// own OIDC client secret with the certificate of the hive it runs on, that
/// being the only identity such a host has — so every hive can read every
/// service's. Bought and paid for in `docs/trust-boundary/security.md`.
/// ⚠️ The controller kind and the `swarm/services/` tree are deliberately
/// absent: a hive has no business reading the credentials of the thing that
/// provisions it, and a swarm service's OIDC client secret is read under a
/// certificate that is not a hive's. Adding either is a boundary decision,
/// not a consequence of the namespace growing.
///
/// The hive's *own* kind is granted, and that is the decision the agent-only
/// version of this grant said had to be made rather than assumed: a hive holds
@ -166,11 +161,7 @@ pub fn render(hive: &str) -> Result<String, Error> {
"{MOUNT}/data/{ROOT}/{}/{hive}/*",
<&str>::from(Kind::Hive)
));
let services = read_stanza(&format!(
"{MOUNT}/data/{ROOT}/{}/*",
<&str>::from(Kind::Service)
));
Ok(format!("{agents}{own}{services}"))
Ok(format!("{agents}{own}"))
}
/// Render `agent`'s policy document: read on that one agent's credentials, list
@ -186,13 +177,13 @@ pub fn render(hive: &str) -> Result<String, Error> {
///
/// ⚠️ **None of [`render`]'s breadth is inherited.** A hive's document grants
/// read on `swarm/agents/*` — *every* agent's credentials, not the ones that
/// hive hosts — and on `swarm/services/*` the same way; the module header says
/// why each is a decision rather than an oversight. Neither reason transfers:
/// an agent's path names the agent, so scoping costs nothing and drifts nowhere
/// (the argument that keeps [`render`]'s hive stanza narrow), and an agent has
/// no business with a service's OIDC secret, another agent's credentials, a
/// hive's, or the controller's. Narrow here does **not** narrow the hive's: a
/// hive still reads this agent's secrets, and closing that is its own decision.
/// hive hosts; the module header says why that is a decision rather than an
/// oversight. The reason does not transfer: an agent's path names the agent,
/// so scoping costs nothing and drifts nowhere (the argument that keeps
/// [`render`]'s hive stanza narrow), and an agent has no business with another
/// agent's credentials, a hive's, a service's, or the controller's. Narrow
/// here does **not** narrow the hive's: a hive still reads this agent's
/// secrets, and closing that is its own decision.
/// Nothing here writes, for the reason the hive's does not: an agent that could
/// write its own credentials could hand itself an identity it was never issued.
///
@ -216,26 +207,31 @@ mod tests {
use super::*;
#[test]
fn the_document_grants_two_whole_prefixes_and_this_hive_alone() {
fn the_document_grants_every_agent_and_this_hive_alone() {
assert_eq!(
render("pr1ma").expect("a plain name is legal"),
"path \"secret/data/swarm/agents/*\" {\n capabilities = [\"read\"]\n}\n\
path \"secret/data/swarm/hives/pr1ma/*\" {\n capabilities = [\"read\"]\n}\n\
path \"secret/data/swarm/services/*\" {\n capabilities = [\"read\"]\n}\n"
path \"secret/data/swarm/hives/pr1ma/*\" {\n capabilities = [\"read\"]\n}\n"
);
}
#[test]
fn the_service_stanza_covers_a_service_this_hive_was_never_named_beside() {
// The property the swarm-grafana delivery depends on: the host running a
// swarm service reads that service's client secret with its own hive
// certificate, and the path names the service rather than the host. A
// stanza narrowed to the reader's name would 403 every such read.
fn a_hive_reaches_no_swarm_services_secret() {
// A swarm service's OIDC client secret is read under a certificate that
// is not a hive's. Checked by walking the stanzas, so a wider grant
// that happened to cover the tree (`swarm/*`, say) fails too.
let p = render("pr1ma").expect("legal");
assert!(p.contains("path \"secret/data/swarm/services/*\""));
assert!(!p.contains("swarm/services"), "{p}");
let service = "swarm/services/swarm-grafana/oidc/client";
assert!(
!p.contains("services/pr1ma"),
"the service stanza is not scoped to the reader"
!stanza_paths(&p).iter().any(|g| covers(g, service)),
"a hive's document reaches {service}:\n{p}"
);
// The control: the walk does find a path the document grants.
assert!(
stanza_paths(&p)
.iter()
.any(|g| covers(g, "swarm/hives/pr1ma/queue/agent"))
);
}
@ -243,10 +239,6 @@ mod tests {
/// document rather than on the path function alone: it is the policy text
/// that decides what a hive may fetch, so "the path is per-hive" only
/// means something if the stanza that reaches it is too.
///
/// The `services/*` stanza above is deliberate and stays — it is how a
/// host reads the OIDC secret of a service it runs. The property here is
/// that the matrix sender token is no longer *inside* it.
#[test]
fn a_hive_reaches_its_own_matrix_sender_token_and_no_other_hives() {
let alpha = render("alpha").expect("legal");
@ -516,16 +508,16 @@ mod tests {
fn an_agents_document_is_not_the_hives() {
// The pin the two renderers need against each other: a later edit that
// made `render_agent` delegate to `render`, or vice versa, would hand
// an agent the hive's two wide stanzas.
// an agent the hive's `agents/*` stanza.
let agent = render_agent("atlas").expect("legal");
let hive = render("atlas").expect("legal");
assert_ne!(agent, hive);
assert!(!agent.contains("swarm/services/"));
assert!(!agent.contains("swarm/hives/"));
// And the hive's is untouched by this module's growth: still the three
// stanzas `the_document_grants_two_whole_prefixes_and_this_hive_alone`
// pins byte for byte, none of them narrowed to make the agent's look
// consistent with it.
// And the hive's is untouched by this module's growth: still the two
// stanzas `the_document_grants_every_agent_and_this_hive_alone` pins
// byte for byte, neither narrowed to make the agent's look consistent
// with it.
assert!(hive.contains("path \"secret/data/swarm/agents/*\""));
}

5
swarm-secret-client/src/queue.rs
View file

@ -63,9 +63,8 @@ pub fn agent_queue_path(agent: &str) -> Result<String, Error> {
/// The path holding `swarm-controller`'s own OIDC client secret: the identity
/// it connects to the queue with, and mints its other bearer tokens from.
///
/// Under [`Kind::Controller`] because no hive's policy reads that kind. The
/// client may read every hive's status, so a copy under `services/`, which
/// every hive reads, would hand that reach to every hive.
/// Under [`Kind::Controller`] because no hive's policy reads that kind, and
/// this client may read every hive's status.
///
/// The nix half is `swarm-secret-publisher.nix`, which writes it, and
/// `swarm-bao.nix`'s `controllerPolicyText`, which grants the read.