Watch
0
0
Fork
You've already forked hyperhive
0

drop the shared hive agent login

Every agent now connects to the queue and reads logs with its own secret
from the store (#5181), so the per-hive `hive-<h>-agent` OIDC client has
no reader left. Remove it end to end:

- hive-c0re no longer forwards `hive-queue-agent-{secret,client-id}` into
  containers, and no longer forwards HIVE_AGENT_OIDC_TOKEN_ENDPOINT; the
  agent module that loaded them (nix/agent-modules/queue.nix) is gone, its
  options refused with a removal message.
- The host reader (glue-queue-agent-credential.nix, swarm-bao-queue-agent),
  its store role and policy unit (swarm-bao-queue-agent-policy), its PKI
  leaf and the deploy.bao.queueAgent* / queue.agentCredentialDir options
  are gone; the options are refused with a removal message.
- authelia mints no `hive-<h>-agent` client, and swarm-secret-publish no
  longer writes swarm/hives/<h>/queue/agent.
- swarm-secret-client drops queue::agent_client_path / queue::Credential.
- HIVE_AGENT_NATS_URL is now gated on the agents' queue address alone.

The log store's authelia check stays for operators. swarm-nats-auth still
refuses a `-agent` client id; that refusal and `agentClientSuffix` are left
for a follow-up.

Refs #4459
Refs #4765
This commit is contained in:
atlas 2026-10-13 09:49:32 +02:00 • committed by mara
commit 868c6cf96c
41 changed files with 286 additions and 1410 deletions

View file

@ -303,8 +303,7 @@ subjects it publishes to, the credential may answer the one request it was just
sent, which is how a command gets its reply. The harness and the subagent daemon sent, which is how a command gets its reply. The harness and the subagent daemon
connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to the connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to the
icon write and the schedule read arrive. An agent can't read another agent's icon write and the schedule read arrive. An agent can't read another agent's
terminal, state, input, commands or replies. The queue refuses the hive's shared terminal, state, input, commands or replies.
agent client outright.
Hives and the swarm controller get a subscribe allow-list too, each with its Hives and the swarm controller get a subscribe allow-list too, each with its
own reply inbox `_INBOX.client.<client-id>.>`, which `swarm_queue_client::connect` own reply inbox `_INBOX.client.<client-id>.>`, which `swarm_queue_client::connect`

View file

@ -121,13 +121,12 @@ them and names them in config. The service principals' leaves are at the top of
`/var/lib/swarm-bao-pki/`, named after the principal. `/var/lib/swarm-bao-pki/`, named after the principal.
For every hive in `swarm.hives` other than its own, the store's host signs For every hive in `swarm.hives` other than its own, the store's host signs
that hive's three leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`: that hive's two leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`:
| file | subject | name it in | | file | subject | name it in |
| ---------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------- | | ------------------------------------ | ---------------------------------------------- | ------------------------------------------------------------------------------- |
| `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` | | `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` |
| `queue-agent.pem`, `queue-agent-key.pem` | `<deploy.bao.queueAgentCommonNamePrefix>-<hive>` | `deploy.bao.queueAgentClientCertFile`, `.queueAgentClientKeyFile` | | `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
| `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
Copy `/var/lib/swarm-bao-pki/ca.pem` with them and name it in Copy `/var/lib/swarm-bao-pki/ca.pem` with them and name it in
`deploy.bao.serverCaFile`. A hive added to `swarm.hives` gets its directory on `deploy.bao.serverCaFile`. A hive added to `swarm.hives` gets its directory on

View file

@ -43,7 +43,6 @@ the rest of the cell says how.
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting | | `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call | | `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host | must be stated | must be stated |
| `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it | | `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |

View file

@ -107,9 +107,8 @@ with nothing to authenticate with still receives every hive's telemetry —
its pushes to the stores go out unauthenticated and get refused there, and its its pushes to the stores go out unauthenticated and get refused there, and its
scrape jobs for the published targets read a credential file nothing scrape jobs for the published targets read a credential file nothing
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice `deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
`glue-queue-agent-credential.nix` makes for its own reader, rather than the hard assertion `swarm-grafana.nix` `swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches. collector this unit never reaches.
@ -259,7 +258,6 @@ such as the root token.
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` | | the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` | | the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` | | the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
The forge and the homeserver read their copies out of the store, the way The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the Grafana does below: the forge under the forge publisher's leaf, the
@ -268,34 +266,8 @@ publisher in authelia's container has written the secret, rather than skipping.
A silent skip produces a service whose login button always fails, which is a A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause. symptom many layers from its cause.
The store's **first reader** is the agent containers' queue credential: The store's **first reader** is Grafana's OIDC client secret, a _swarm
`glue-queue-agent-credential.nix` lands it as two files, the client secret and service's_ own credential rather than a hive's. It has exactly
the client id it authenticates, because that's how a queue client reads them —
the secret by path, the id by value. No local fallback exists, and none is
possible, so absent files mean an agent built before `swarm-logs` read its own
queue secret has no working `swarm-logs`, which is what a swarm looks like
before the publisher in authelia's container has run.
The reader runs before `hive-c0re.service`, which wants it (doesn't require
it), so an agent container never renders ahead of the credential; an
unreachable store delays the daemon's start rather than failing it.
That credential still has one hop left, because the reader of it lives inside
an agent container. **It crosses as a systemd credential, not as a bind
mount**, and the mode decides that: the secret is `root:0600` and a harness
runs as its own unprivileged agent user, so a bind would deliver a file that
user can't open. `hive-c0re` stats the two files — the directory is `0755`, so
it needs no read access to either — and hands them to
`systemd-nspawn --load-credential` through `hive-priv`, which runs as root.
Inside, `hive-agent.service` names the same two ids in `LoadCredential=`, which
lands them in its own `$CREDENTIALS_DIRECTORY`, owned by the agent user and by
nobody else, where a `swarm-logs` built before it read the agent's own queue
secret reads them. The harness itself doesn't: it connects to the queue with the agent's own credential. `hive-c0re`
never reads the bytes at any point: it runs as `hive-core`, which is the reason
a copy wasn't an option either. When the files aren't there the daemon forwards
nothing and says so in its journal, and the container starts without them.
The **second reader** is Grafana's OIDC client secret, and it's the first one
that's a _swarm service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host `swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the publisher in that runs Grafana. Where authelia is doesn't enter into it — the publisher in
@ -338,7 +310,7 @@ the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
the Grafana host, which retries, while Grafana stays down because its config the Grafana host, which retries, while Grafana stays down because its config
reads that file. reads that file.
The **third reader** is the swarm collector's own OIDC client secret, the The **second reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that `swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in runs the collector, and registration lives separately in
@ -347,13 +319,12 @@ differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana `deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it does — it
doesn't render the reading unit without them, the shape doesn't render the reading unit without them.
`glue-queue-agent-credential.nix` uses for its own read.
The **fourth reader** is the secret store forwarder's OIDC client secret: The **third reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by `swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the `glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
third in both directions. It renders unconditionally, because the export it second in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it container it delivers into. Nothing circular sits behind that: the identity it
@ -412,14 +383,13 @@ next controller restart.
### Per-principal identities ### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. Each of these eleven leaves has identity and sharing one merges the identities. Each of these ten leaves has
a role and a policy of its own, and reaches one path, except the review a role and a policy of its own, and reaches one path, except the review
classifier's, the CI runner's, the builder's, the CI run poller's, the claude classifier's, the CI runner's, the builder's, the CI run poller's, the claude
login minter's, the forge host's and Grafana's, which reach more than one: login minter's, the forge host's and Grafana's, which reach more than one:
| unit | option pair under `deploy.bao.` | reads, or writes | | unit | option pair under `deploy.bao.` | reads, or writes |
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
| `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin`, and Grafana's journal forwarder in its container | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client` and `swarm/services/<grafana forwarder client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` | | `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin`, and Grafana's journal forwarder in its container | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client` and `swarm/services/<grafana forwarder client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` | | `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` | | `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
@ -431,12 +401,8 @@ login minter's, the forge host's and Grafana's, which reach more than one:
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` | | `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` | | `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
The first exists **per hive**, because the path it reads carries a hive name Each reaches a path that names a swarm service rather than a hive, so one role
and every hive runs its own reader. Its subject is each is enough and their subjects are the flat
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
composed spelling as a hive name, so nobody can name a hive into another hive's
role. The other nine reach a path that names a swarm service rather than a hive,
so one role each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`, `deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`, `deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`, `deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
@ -449,7 +415,7 @@ OIDC client secret it reads.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all nine and defaults On a host that mints its own PKI, `glue-bao-tls.nix` signs all nine and defaults
all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from
the store's host and name it here: a service principal's from the top of the store's host and name it here: a service principal's from the top of
`/var/lib/swarm-bao-pki/`, another hive's queue-agent pair from `/var/lib/swarm-bao-pki/`, another hive's leaves from
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that `/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
directory). That's one file per principal rather than one file directory). That's one file per principal rather than one file
shared by nine, which is the whole of what this buys. `swarm-ci-runner.nix` shared by nine, which is the whole of what this buys. `swarm-ci-runner.nix`
@ -461,7 +427,7 @@ naming both options, and the controller runs without forge access and the forge
without a login source until an operator places the leaf. without a login source until an operator places the leaf.
On a host without the store, where the forwarder never runs, forgetting one of On a host without the store, where the forwarder never runs, forgetting one of
the other nine isn't a quiet degrade. A hand-configured remote-store hive that the others isn't a quiet degrade. A hand-configured remote-store hive that
names the hive's own `clientCertFile` and misses a principal's pair has the names the hive's own `clientCertFile` and misses a principal's pair has the
build refused, by the module of the reader it missed, naming that pair — see build refused, by the module of the reader it missed, naming that pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below. [what a missing leaf costs](#what-a-missing-leaf-costs) below.
@ -475,7 +441,7 @@ their readers hold a handle for longer than five minutes.
### What a missing leaf costs ### What a missing leaf costs
Each of the three modules asserts its own pair, on the same condition: **this host Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**. `clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without The refusal names the two options, so an operator acts on the message without
@ -492,9 +458,7 @@ that matters on its own:
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as - **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse. defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no collector, no Grafana — has no - **A consumer that's off** — no collector, no Grafana — has no
unit to skip, so the second clause is false. Only the queue-credential reader unit to skip, so the second clause is false.
has no toggle of its own to check: every hive runs one for its own agents, so
reading the store at all is what asks for it.
The collector keeps a separate degrade underneath this, unchanged: a host with no The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives store identity runs a collector with no OIDC client secret and still receives

View file

@ -225,7 +225,7 @@ Four consequences worth stating plainly:
Everything above is a person in a browser. A swarm also has callers that Everything above is a person in a browser. A swarm also has callers that
hold no session and follow no redirect: the telemetry collector, the hold no session and follow no redirect: the telemetry collector, the
queue's auth-callout responder, and each hive's own agents. queue's auth-callout responder, and each hive.
**One client per hive, not one per service.** A hive's identity belongs to **One client per hive, not one per service.** A hive's identity belongs to
the directory rather than to whichever service happens to consume it, so a the directory rather than to whichever service happens to consume it, so a

View file

@ -8,10 +8,9 @@
//! asked to authenticate in order to obtain the thing it authenticates with: //! asked to authenticate in order to obtain the thing it authenticates with:
//! by the time it runs, its identity is already inside it. //! by the time it runs, its identity is already inside it.
//! //!
//! **A credential and not a bind mount**, the same answer and the same reason //! **A credential and not a bind mount**: the files are `0600` to `hive-core`
//! as the queue secret beside it in [`super::host_config`]: the files are //! and the units that use them run as the unprivileged agent user, so a bind
//! `0600` to `hive-core` and the units that use them run as the unprivileged //! would deliver a private key that user cannot open.
//! agent user, so a bind would deliver a private key that user cannot open.
//! The container manager reads a `--load-credential` source as root and //! The container manager reads a `--load-credential` source as root and
//! re-exposes it under the consuming unit's own `User=`. //! re-exposes it under the consuming unit's own `User=`.
//! //!
@ -60,8 +59,7 @@ const SERVER_CA_CREDENTIAL: &str = "hive-agent-bao-server-ca";
/// ///
/// Already-staged files survive a failed collection. A store that is briefly /// Already-staged files survive a failed collection. A store that is briefly
/// unreachable then delivers the identity from the last successful read rather /// unreachable then delivers the identity from the last successful read rather
/// than none at all — the same call as the queue reader's, which leaves its /// than none at all.
/// files alone when the store says nothing.
pub async fn stage(agent: &str) -> Vec<CredentialMount> { pub async fn stage(agent: &str) -> Vec<CredentialMount> {
let dir = crate::paths::agent_identity_dir(agent); let dir = crate::paths::agent_identity_dir(agent);
match collect(agent, &dir).await { match collect(agent, &dir).await {
@ -246,7 +244,7 @@ mod tests {
/// Half a delivery is worse than none: the container's check would fail /// Half a delivery is worse than none: the container's check would fail
/// on the missing key and report a login problem instead of a delivery /// on the missing key and report a login problem instead of a delivery
/// one. Same both-or-neither rule the queue credential states. /// one.
#[test] #[test]
fn a_certificate_without_its_key_forwards_nothing() { fn a_certificate_without_its_key_forwards_nothing() {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");

View file

@ -6,7 +6,7 @@
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use hive_priv_sock::{BindMount, CredentialMount}; use hive_priv_sock::BindMount;
use hive_types::Permission; use hive_types::Permission;
use crate::coordinator::{AgentPaths, HiveEnv}; use crate::coordinator::{AgentPaths, HiveEnv};
@ -417,70 +417,6 @@ pub fn hand_permissions(agent: &str, runtime_dir: &Path) {
} }
} }
/// Env var naming the host directory `swarm-bao-queue-agent.service` lands
/// this hive's agent queue credential in. Set by the hive-c0re NixOS module
/// from `deploy.hive-controller.queue.agentCredentialDir`; absent means this
/// daemon runs outside its unit, which is the same "no queue credential"
/// answer as an empty directory.
const QUEUE_CREDENTIAL_DIR_ENV: &str = "HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR";
/// systemd credential ids the two files arrive under inside the container.
/// `nix/agent-modules/queue.nix` spells the same two names in the harness
/// unit's `LoadCredential=`; neither side can discover the other's, so a
/// rename here is a rename there.
const QUEUE_SECRET_CREDENTIAL: &str = "hive-queue-agent-secret";
const QUEUE_CLIENT_ID_CREDENTIAL: &str = "hive-queue-agent-client-id";
/// Forward this hive's agent queue credential into `agent_name`'s container
/// as two systemd credentials, or nothing when the publisher has not landed
/// it yet.
///
/// **A credential and not a bind mount, because of the mode.** The secret is
/// `root:0600` on the host and the harness runs as the unprivileged agent
/// user, so a bind would deliver a file that user cannot open. nspawn's
/// `--load-credential` is read by the container manager as root and
/// re-exposed under the consuming unit's own `User=`, which is the whole
/// difference. hive-c0re never reads the bytes either way — it runs as
/// `hive-core` and only ever needs to know the file is *there*, which a
/// `0755` directory permits.
///
/// **Absent files are legal.** Authelia mints the secret on its first boot
/// and a publisher on that host puts it in the store, so "nothing at that
/// path" is the ordinary early state of a swarm rather than a fault. Saying
/// so at `info!` is what keeps it from being invisible: without a line here,
/// an agent that never connects looks identical to one that was never
/// configured.
fn queue_agent_credentials(agent_name: &str, dir: Option<&Path>) -> Vec<CredentialMount> {
let Some(dir) = dir else {
tracing::info!(
%agent_name,
"no {QUEUE_CREDENTIAL_DIR_ENV} in this daemon's environment — agent gets no swarm queue credential"
);
return Vec::new();
};
let secret = dir.join("secret");
let client_id = dir.join("client_id");
// Both or neither: a client holding one of the two comes up fine and
// never connects.
if !secret.is_file() || !client_id.is_file() {
tracing::info!(
%agent_name, dir = %dir.display(),
"swarm queue credential not published yet — agent gets no swarm queue credential"
);
return Vec::new();
}
vec![
CredentialMount {
name: QUEUE_SECRET_CREDENTIAL.to_owned(),
host_path: secret.to_string_lossy().into_owned(),
},
CredentialMount {
name: QUEUE_CLIENT_ID_CREDENTIAL.to_owned(),
host_path: client_id.to_string_lossy().into_owned(),
},
]
}
/// Idempotently rewrite the lines in `/etc/nixos-containers/<container>.conf` /// Idempotently rewrite the lines in `/etc/nixos-containers/<container>.conf`
/// that hive-c0re owns: `PRIVATE_NETWORK` (always 1), `HOST_ADDRESS` (the /// that hive-c0re owns: `PRIVATE_NETWORK` (always 1), `HOST_ADDRESS` (the
/// bridge gateway IP) and `EXTRA_NSPAWN_FLAGS` (the runtime-dir bind). What /// bridge gateway IP) and `EXTRA_NSPAWN_FLAGS` (the runtime-dir bind). What
@ -537,18 +473,11 @@ async fn set_nspawn_flags(
// is needed here — the bind alone is enough. // is needed here — the bind alone is enough.
let claude_mount = container_claude_mount(agent_name); let claude_mount = container_claude_mount(agent_name);
// The agent's own swarm-queue credential — the one thing forwarded in
// here, and forwarded *as a credential* rather than a bind for the
// reason `queue_agent_credentials` records.
let queue_credential_dir = std::env::var_os(QUEUE_CREDENTIAL_DIR_ENV).map(PathBuf::from);
let mut load_creds = queue_agent_credentials(agent_name, queue_credential_dir.as_deref());
// The agent's own identity *at* the swarm secret store, collected under // The agent's own identity *at* the swarm secret store, collected under
// this hive's certificate and forwarded the same way and for the same // this hive's certificate. Staged here rather than at spawn because this
// reason. Staged here rather than at spawn because this is the one place // is the one place that writes the container's credential list — see
// that writes the container's credential list — see
// `super::agent_identity` for the whole hop. // `super::agent_identity` for the whole hop.
load_creds.extend(super::agent_identity::stage(agent_name).await); let load_creds = super::agent_identity::stage(agent_name).await;
let mut binds: Vec<BindMount> = vec![ let mut binds: Vec<BindMount> = vec![
BindMount { BindMount {
@ -678,10 +607,8 @@ async fn set_nspawn_flags(
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::{ use super::{
BindMount, HostMemory, MEMORY_CAP_FILE, QUEUE_CLIENT_ID_CREDENTIAL, BindMount, HostMemory, MEMORY_CAP_FILE, bind_child_agent_dirs, bind_own_agent_dirs,
QUEUE_SECRET_CREDENTIAL, bind_child_agent_dirs, bind_own_agent_dirs, holds_manage_root_agent, limits_to_apply, memory_cap_value, stage_limits,
holds_manage_root_agent, limits_to_apply, memory_cap_value, queue_agent_credentials,
stage_limits,
}; };
use crate::coordinator::HiveEnv; use crate::coordinator::HiveEnv;
@ -791,51 +718,6 @@ mod tests {
assert!(binds.is_empty()); assert!(binds.is_empty());
} }
/// The ordinary state of a swarm before the publisher on the authelia
/// host has run: the directory is named and empty. Forwarding a
/// credential whose source does not exist would make every agent
/// container refuse to start, so this has to be silence-with-a-log
/// rather than a partial list.
#[test]
fn an_unpublished_queue_credential_forwards_nothing() {
let dir = tempfile::tempdir().expect("tempdir");
assert!(queue_agent_credentials("iris", None).is_empty());
assert!(queue_agent_credentials("iris", Some(dir.path())).is_empty());
std::fs::write(dir.path().join("secret"), "s").expect("write secret");
assert!(
queue_agent_credentials("iris", Some(dir.path())).is_empty(),
"a secret with no client id is not a usable credential"
);
}
/// Both files present: two credentials, named the same two ids the
/// harness unit imports, pointing at the two files the reader unit
/// wrote.
#[test]
fn a_published_queue_credential_forwards_both_files() {
let dir = tempfile::tempdir().expect("tempdir");
std::fs::write(dir.path().join("secret"), "s").expect("write secret");
std::fs::write(dir.path().join("client_id"), "hive-h1-agent").expect("write client_id");
let creds = queue_agent_credentials("iris", Some(dir.path()));
let named: Vec<(String, String)> = creds
.into_iter()
.map(|c| (c.name, c.host_path))
.collect::<Vec<_>>();
assert_eq!(
named,
[
(
QUEUE_SECRET_CREDENTIAL.to_owned(),
dir.path().join("secret").to_string_lossy().into_owned(),
),
(
QUEUE_CLIENT_ID_CREDENTIAL.to_owned(),
dir.path().join("client_id").to_string_lossy().into_owned(),
),
]
);
}
#[test] #[test]
fn corrupt_swarm_permissions_deny_without_failing() { fn corrupt_swarm_permissions_deny_without_failing() {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");

View file

@ -455,12 +455,10 @@ const CANONICAL_INPUTS: &[&str] = &["nixpkgs"];
/// trivial to stub from tests (which build their own slice instead of /// trivial to stub from tests (which build their own slice instead of
/// touching process-wide env). /// touching process-wide env).
const FORWARDED_VARS: &[&str] = &[ const FORWARDED_VARS: &[&str] = &[
// Where the swarm queue is and where its tokens are minted, as an agent // Where the swarm queue is, as an agent container reaches it. An address
// container reaches them. Both are addresses the *host* computes — the // the *host* computes, never a loopback address, which inside a container
// queue's from the bridge IP an agent routes over, never a loopback // is the agent itself.
// address, which inside a container is the agent itself.
"HIVE_AGENT_NATS_URL", "HIVE_AGENT_NATS_URL",
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
// Where the swarm secret store listens. The hive's own `BAO_ADDR` value, // Where the swarm secret store listens. The hive's own `BAO_ADDR` value,
// forwarded under a separate name so the container never inherits the // forwarded under a separate name so the container never inherits the
// hive's store environment wholesale — the address is the only part of it // hive's store environment wholesale — the address is the only part of it
@ -470,7 +468,7 @@ const FORWARDED_VARS: &[&str] = &[
// fetches to trust the swarm. Forwarded with the address, under the same gate. // fetches to trust the swarm. Forwarded with the address, under the same gate.
"HIVE_AGENT_BAO_SERVICES_PKI_MOUNT", "HIVE_AGENT_BAO_SERVICES_PKI_MOUNT",
// Where an agent's `swarm-logs` queries, computed by the host for the // Where an agent's `swarm-logs` queries, computed by the host for the
// same reason as the two above: it is a gateway address, and a container // same reason as the queue's: it is a gateway address, and a container
// cannot derive it from anything it holds. // cannot derive it from anything it holds.
"HIVE_AGENT_LOGS_QUERY_URL", "HIVE_AGENT_LOGS_QUERY_URL",
// The metrics store's origin, for `swarm-metrics`, for the same reason. // The metrics store's origin, for `swarm-metrics`, for the same reason.
@ -494,18 +492,6 @@ const FORWARDED_VARS: &[&str] = &[
/// name. Half-wiring this map is a value that evaluates fine and is silently /// name. Half-wiring this map is a value that evaluates fine and is silently
/// wrong. /// wrong.
const FORWARDED_VAR_OPTIONS: &[(&str, &str)] = &[ const FORWARDED_VAR_OPTIONS: &[(&str, &str)] = &[
// Read at build time to decide whether the harness unit imports the hive's
// queue credential at all — see `nix/agent-modules/queue.nix`. Both halves
// wired for this map's own reason: an agent whose option says no queue and
// whose env says otherwise imports no credential for `swarm-logs`.
(
"HIVE_AGENT_NATS_URL",
"services.hyperhive.agent.queue.natsUrl",
),
(
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
"services.hyperhive.agent.queue.tokenEndpoint",
),
// Read at build time to decide whether the in-container identity check is // Read at build time to decide whether the in-container identity check is
// generated at all — see `nix/agent-modules/bao.nix`. Both halves wired // generated at all — see `nix/agent-modules/bao.nix`. Both halves wired
// for this map's own reason: the unit bakes the address into its own // for this map's own reason: the unit bakes the address into its own
@ -1561,35 +1547,22 @@ mod tests {
); );
} }
/// The queue coordinates travel as a pair and are wired through both /// The queue address reaches the harness as env only. The agent module
/// halves of the forwarded-var machinery: the env var the harness reads at /// declares no `services.hyperhive.agent.queue.*` option, so rendering one
/// runtime and the option the harness *unit* is built from. An agent that /// into the meta flake fails every agent's evaluation.
/// got only the env var would declare no credential and then report a
/// partial config — visible, but a rebuild away from working.
#[test] #[test]
fn queue_coordinates_are_forwarded_as_both_env_and_option() { fn queue_address_is_forwarded_as_env_and_no_option() {
for var in ["HIVE_AGENT_NATS_URL", "HIVE_AGENT_OIDC_TOKEN_ENDPOINT"] { assert!(FORWARDED_VARS.contains(&"HIVE_AGENT_NATS_URL"));
assert!(
!FORWARDED_VARS.contains(&"HIVE_AGENT_OIDC_TOKEN_ENDPOINT"),
"no agent reads a token endpoint"
);
for (var, option) in FORWARDED_VAR_OPTIONS {
assert!( assert!(
FORWARDED_VARS.contains(&var), !option.starts_with("services.hyperhive.agent.queue."),
"{var} must be forwarded into every agent's env" "{var} renders {option}, which no agent module declares"
); );
} }
let mut out = String::new();
push_forwarded_var_options(
&mut out,
&[
("HIVE_AGENT_NATS_URL", "nats://10.42.0.1:4222".to_string()),
(
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
"https://auth.t.local/api/oidc/token".to_string(),
),
],
);
assert_eq!(
out,
" services.hyperhive.agent.queue.natsUrl = \"nats://10.42.0.1:4222\";\n\
\x20 services.hyperhive.agent.queue.tokenEndpoint = \"https://auth.t.local/api/oidc/token\";\n"
);
} }
/// `swarm-metrics` is installed only when the option is set, and reads /// `swarm-metrics` is installed only when the option is set, and reads

View file

@ -14,8 +14,7 @@
# is why: the host file is `0600` to the hive daemon and this unit runs as the # is why: the host file is `0600` to the hive daemon and this unit runs as the
# unprivileged agent user. nspawn's `--load-credential` is read by the # unprivileged agent user. nspawn's `--load-credential` is read by the
# container manager as root and re-exposed under this unit's own `User=`; a # container manager as root and re-exposed under this unit's own `User=`; a
# bind would deliver a private key this user cannot open. Same answer, same # bind would deliver a private key this user cannot open.
# reason, as ./queue.nix's credential pair.
# #
# This is the courier's other end: the hive-side collector has no purpose # This is the courier's other end: the hive-side collector has no purpose
# without it, so the two ship together and neither is reachable alone. # without it, so the two ship together and neither is reachable alone.
@ -63,12 +62,11 @@ let
# Where the identity lives in the store, spelled from the same pieces the # Where the identity lives in the store, spelled from the same pieces the
# publisher uses. `swarm_secret_client::mtls::identity_path` builds # publisher uses. `swarm_secret_client::mtls::identity_path` builds
# `swarm/agents/<agent>/bao-mtls` and `path::MOUNT` is `secret`; this # `swarm/agents/<agent>/bao-mtls` and `path::MOUNT` is `secret`; this
# literal is the nix half of that one agreement, exactly as # literal is the nix half of that one agreement.
# ../host-modules/glue-queue-agent-credential.nix spells its own.
identityPath = "secret/swarm/agents/${agentName}/bao-mtls"; identityPath = "secret/swarm/agents/${agentName}/bao-mtls";
# The address is the whole switch — no separate `enable`, the same shape # The address is the whole switch — no separate `enable`, the same shape
# ./queue.nix and ./logs.nix gate themselves with. A hive that has a store # ./logs.nix gates itself with. A hive that has a store
# forwards its address and every agent on it gets the check; a hive that has # forwards its address and every agent on it gets the check; a hive that has
# none forwards nothing and no agent does. An `enable` beside it would be a # none forwards nothing and no agent does. An `enable` beside it would be a
# knob whose only correct setting is whatever the address already says, and # knob whose only correct setting is whatever the address already says, and
@ -153,7 +151,7 @@ in
UMask = "0377"; UMask = "0377";
# Bare ids, no paths: the terse `LoadCredential=` form that inherits a # Bare ids, no paths: the terse `LoadCredential=` form that inherits a
# credential the service *manager* received, which is what the # credential the service *manager* received, which is what the
# container manager passed in. ./queue.nix states the same shape. # container manager passed in.
LoadCredential = [ LoadCredential = [
certCredential certCredential
keyCredential keyCredential

View file

@ -34,6 +34,12 @@ let
sets it. Pin it host-side with sets it. Pin it host-side with
services.hyperhive.deploy.hive-controller.claudeCodePackage. services.hyperhive.deploy.hive-controller.claudeCodePackage.
''; '';
queueRemoved = ''
The harness reads the queue's address at runtime from HIVE_AGENT_NATS_URL,
which the hive forwards into every container, and authenticates with the
agent's own secret from the swarm secret store, so no agent build sets a
queue option. Remove the setting.
'';
# `claude` on PATH. A hive that pins a claude-code # `claude` on PATH. A hive that pins a claude-code
# (`services.hyperhive.deploy.hive-controller.claudeCodePackage`) hands its # (`services.hyperhive.deploy.hive-controller.claudeCodePackage`) hands its
@ -81,7 +87,6 @@ in
./otel.nix ./otel.nix
./otel-removed.nix ./otel-removed.nix
./packages.nix ./packages.nix
./queue.nix
./queue-identity.nix ./queue-identity.nix
./user.nix ./user.nix
./screen.nix ./screen.nix
@ -101,6 +106,8 @@ in
"agent" "agent"
"claudeCodePath" "claudeCodePath"
] claudeCodePathRemoved) ] claudeCodePathRemoved)
(lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "queue" "natsUrl" ] queueRemoved)
(lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "queue" "tokenEndpoint" ] queueRemoved)
]; ];
options.services.hyperhive.agent.icon = lib.mkOption { options.services.hyperhive.agent.icon = lib.mkOption {

View file

@ -1,13 +1,11 @@
# This agent's own credential for the swarm queue, read from the store by the # This agent's own credential for the swarm queue, read from the store by the
# harness itself. # harness itself.
# #
# The credential beside it in ./queue.nix is keyed per **hive**: one OIDC # Keyed per **agent**, so at the queue's auth callout one agent is
# client minted at deploy time and handed to every agent container on the # distinguishable from its co-hived neighbours: `swarm-controller` mints it at
# hive, so at the queue's auth callout one agent is indistinguishable from its # agent creation into `swarm/agents/<agent>/queue`
# co-hived neighbours. This one is keyed per **agent** — `swarm-controller` # (`swarm_secret_client::queue::agent_queue_path`), at swarm level, with no hive
# mints it at agent creation into `swarm/agents/<agent>/queue` # anywhere in the chain.
# (`swarm_secret_client::queue::agent_queue_path`), at swarm level, with no
# hive anywhere in the chain.
# #
# The agent authenticates to the store as itself, with the certificate # The agent authenticates to the store as itself, with the certificate
# ./bao.nix already proves it can log in with, and reads its own path. A hive # ./bao.nix already proves it can log in with, and reads its own path. A hive
@ -42,11 +40,7 @@ let
# too. # too.
serverCaCredential = "hive-agent-bao-server-ca"; serverCaCredential = "hive-agent-bao-server-ca";
# The store's address is the whole switch, exactly as in ./bao.nix — and # The store's address is the whole switch, exactly as in ./bao.nix.
# deliberately *not* the queue coordinates in ./queue.nix. Those are the
# hive's, and gating a swarm-minted per-agent credential on a hive-level
# option would put the hive back in a path whose entire purpose is not
# having one.
configured = cfg.addr != null; configured = cfg.addr != null;
in in
{ {
@ -54,7 +48,7 @@ in
systemd.services.hive-agent = { systemd.services.hive-agent = {
# A bare id, no path: the terse `LoadCredential=` form that inherits a # A bare id, no path: the terse `LoadCredential=` form that inherits a
# credential the service *manager* received, which is what the container # credential the service *manager* received, which is what the container
# manager passed in. ./bao.nix and ./queue.nix state the same shape. # manager passed in. ./bao.nix states the same shape.
serviceConfig.LoadCredential = [ serverCaCredential ]; serviceConfig.LoadCredential = [ serverCaCredential ];
# The live certificate pair exists once ./bao.nix's unit has run. # The live certificate pair exists once ./bao.nix's unit has run.
after = [ "hive-agent-bao-identity.service" ]; after = [ "hive-agent-bao-identity.service" ];

View file

@ -1,117 +0,0 @@
# The hive's shared swarm-queue client credential, imported into the harness
# unit's credentials directory for `swarm-logs` (./logs.nix) to authenticate
# with. The harness itself connects to the queue with the agent's own
# credential (./queue-identity.nix) and reads neither file.
#
# The credential is two files, the secret and the client id beside it, so a
# reader never spells `hive-<name>-agent` a second time.
#
# ⚠️ The credential arrives as a systemd credential and NOT as a bind mount,
# and the mode is why: the host file is `root:0600` and the reader runs as the
# unprivileged agent user. nspawn's `--load-credential` (written by
# `hive_c0re::lifecycle::host_config`) is read by the container manager as
# root and re-exposed under this unit's own `User=`; a bind would deliver a
# file the agent user cannot open.
{
lib,
config,
...
}:
let
cfg = config.services.hyperhive.agent.queue;
configured = cfg.natsUrl != null && cfg.tokenEndpoint != null;
# The two ids `hive_c0re::lifecycle::host_config` forwards under. Neither
# side can discover the other's spelling, so a rename is a rename there too.
secretCredential = "hive-queue-agent-secret";
clientIdCredential = "hive-queue-agent-client-id";
# Where systemd materialises this unit's credentials. `%d` only expands
# *inside* a unit, so `swarm-logs` (./logs.nix), which is not one, needs the
# literal; deriving it from the unit name here is what keeps the two from
# disagreeing about which unit's credentials they mean.
credentialsDir = "/run/credentials/hive-agent.service";
in
{
options.services.hyperhive.agent.queue = {
natsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "tls://nats.example.com:4222";
description = ''
Where the swarm queue listens, as this container reaches it.
Set by the generated meta flake from the host's
{option}`services.hyperhive.deploy.hive-controller.queue.agentNatsUrl`,
which names the queue rather than a loopback address — inside this
container `127.0.0.1` is the agent itself.
`null` means this hive has not been given the queue's address for its
agents: the harness unit imports no credential, and the harness fails
at boot, since it cannot run without a queue connection. It is
deliberately not defaulted to anything: a guessed address builds fine
and talks to the wrong machine.
'';
};
tokenEndpoint = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "https://auth.example.com/api/oidc/token";
description = ''
The swarm IdP's OAuth2 token endpoint, where `swarm-logs` mints a
`client_credentials` token for the client named in the delivered
credential.
The harness unit imports that credential only when this and
{option}`services.hyperhive.agent.queue.natsUrl` are both set.
'';
};
# One declaration, two readers. The credential ids are this module's —
# `hive_c0re::lifecycle::host_config` and the `LoadCredential=` below are
# the pair that has to agree on them — and a second consumer that spelled
# them again would be a second source of truth for a string whose
# mismatch is a file that is simply not there.
clientIdFile = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "${credentialsDir}/${clientIdCredential}";
description = ''
Path the agent's OIDC client id is delivered at, for a consumer
outside the harness unit. Read-only: it is a fact about where the
credential lands, not a knob — see {option}`services.hyperhive.agent.logs.queryUrl`
for the consumer this exists for.
'';
};
clientSecretFile = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "${credentialsDir}/${secretCredential}";
description = ''
Path the agent's OIDC client secret is delivered at. Read-only for
the same reason as {option}`services.hyperhive.agent.queue.clientIdFile`.
🩸 A PATH and never a value. The file is `0400` to the agent user and
is read at the moment of a token request; nothing in this tree puts
its contents in an environment variable, where `/proc/<pid>/environ`
would publish them to every process in the container.
'';
};
};
config = lib.mkIf configured {
systemd.services.hive-agent = {
# Bare ids, no paths: this is the terse `LoadCredential=` form that
# inherits a credential the service *manager* received, which is what
# the container manager passed in. `man systemd.exec` also makes that
# form non-fatal when the credential is absent, which is exactly the
# behaviour a hive whose publisher has not run yet needs.
serviceConfig.LoadCredential = [
secretCredential
clientIdCredential
];
};
};
}

View file

@ -1,6 +1,6 @@
# The unit that puts swarm-level secrets into the swarm's secret store, so # The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its own and its agents' # whoever needs one can read it there: a hive, a service or the controller its
# credentials, a service or the controller its own. It mints nothing: every # own. It mints nothing: every
# value is an OIDC client secret authelia minted, and this unit copies it. # value is an OIDC client secret authelia minted, and this unit copies it.
# #
# Imported inside authelia's container, beside the mint: it reads each secret # Imported inside authelia's container, beside the mint: it reads each secret
@ -61,14 +61,6 @@ in
example = "secret/swarm/services/swarm-grafana/oidc/client"; example = "secret/swarm/services/swarm-grafana/oidc/client";
description = "KV path its reader fetches the value from."; description = "KV path its reader fetches the value from.";
}; };
withClientId = lib.mkOption {
type = lib.types.bool;
default = false;
description = ''
Also store the client id beside the value, as `client_id`, for
a reader that has to reconstruct it from the store alone.
'';
};
}; };
} }
); );
@ -170,7 +162,7 @@ in
# `value="$(cat "$src")"` would publish it to /proc for anyone in # `value="$(cat "$src")"` would publish it to /proc for anyone in
# the container to read. # the container to read.
bao kv put ${lib.escapeShellArg s.path} \ bao kv put ${lib.escapeShellArg s.path} \
value=@"$src"${lib.optionalString s.withClientId " client_id=${lib.escapeShellArg s.clientId}"} value=@"$src"
published=$((published + 1)) published=$((published + 1))
else else
# Not an error: authelia mints on its FIRST BOOT, so an absent file # Not an error: authelia mints on its FIRST BOOT, so an absent file

View file

@ -47,7 +47,6 @@
./glue-nats-bao-identity.nix ./glue-nats-bao-identity.nix
./glue-nats-oidc-client.nix ./glue-nats-oidc-client.nix
./glue-nats-wireguard.nix ./glue-nats-wireguard.nix
./glue-queue-agent-credential.nix
./glue-builder-oidc-client.nix ./glue-builder-oidc-client.nix
./glue-ci-metrics-otel-oidc-client.nix ./glue-ci-metrics-otel-oidc-client.nix
./glue-ci-runner-otel-oidc-client.nix ./glue-ci-runner-otel-oidc-client.nix

View file

@ -53,6 +53,14 @@ let
puts it there. Remove this definition; the file it named is read by puts it there. Remove this definition; the file it named is read by
nothing now and can be deleted. nothing now and can be deleted.
''; '';
# Shared by the removed options of the hive's agent queue credential reader.
queueAgentCredentialRemoved = ''
A hive no longer fetches a shared queue credential for its agents: each
agent reads its own secret from the swarm's secret store under its own
certificate. Remove this definition; the leaf or directory it named is read
by nothing now and can be deleted.
'';
in in
{ {
imports = [ imports = [
@ -95,6 +103,35 @@ in
"statusPublish" "statusPublish"
"clientSecretFile" "clientSecretFile"
] statusSecretRemoved) ] statusSecretRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"bao"
"queueAgentCommonNamePrefix"
] queueAgentCredentialRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"bao"
"queueAgentClientCertFile"
] queueAgentCredentialRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"bao"
"queueAgentClientKeyFile"
] queueAgentCredentialRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"hive-controller"
"queue"
"agentCredentialDir"
] queueAgentCredentialRemoved)
# Same type, same meaning, new path — so a rename carries it exactly # Same type, same meaning, new path — so a rename carries it exactly
# and existing configs keep evaluating with one warning naming both # and existing configs keep evaluating with one warning naming both

View file

@ -31,10 +31,6 @@ let
# Reader unit → the gate its own module defines it under. # Reader unit → the gate its own module defines it under.
readersHere = { readersHere = {
# ./glue-queue-agent-credential.nix
swarm-bao-queue-agent =
deployCfg.hive-controller.enable
&& havePair baoDeploy.queueAgentClientCertFile baoDeploy.queueAgentClientKeyFile;
# ./swarm-grafana.nix # ./swarm-grafana.nix
swarm-bao-grafana-oidc = deployCfg.grafana.enable; swarm-bao-grafana-oidc = deployCfg.grafana.enable;
# ./swarm-otel.nix # ./swarm-otel.nix

View file

@ -95,15 +95,13 @@ in
# 🩸 The readers below each present a leaf of their own rather than # 🩸 The readers below each present a leaf of their own rather than
# `client.pem` above. The leaf is what bao sees, so this pairing is the # `client.pem` above. The leaf is what bao sees, so this pairing is the
# whole of what turns "one principal with the union of their grants" into # whole of what turns "one principal with the union of their grants" into
# one principal per grant — see ./swarm-bao.nix's `perHiveReaders` block # one principal per grant — see ./swarm-bao.nix's `readStanza` block for
# for the grants themselves. # the grants themselves.
# #
# Defaulted here rather than in `glue-<consumer>-bao-identity.nix` files: # Defaulted here rather than in `glue-<consumer>-bao-identity.nix` files:
# those exist where the consumer is a CONTAINER with a # those exist where the consumer is a CONTAINER with a
# `deploy.<service>.*` namespace of its own to point at. These are host # `deploy.<service>.*` namespace of its own to point at. These are host
# units reading the store, which is the pairing this file already is. # units reading the store, which is the pairing this file already is.
queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem";
queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem";
grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem"; grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem";
grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem"; grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem";
otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem"; otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem";
@ -223,20 +221,10 @@ in
# principal holding the union of their grants, and the union would # principal holding the union of their grants, and the union would
# include every agent credential in the swarm. # include every agent credential in the swarm.
# #
# ⚠️ The first carries THIS hive's name in the subject, unlike the three
# service leaves above. Its grant names one hive's path, because a queue
# credential lives under `swarm/hives/<name>/` and every hive runs a
# reader for its own; ./swarm-bao.nix writes one role per hive in the
# swarm directory to match. The OIDC readers need no such segment — a
# client is registered once per swarm.
#
# Minted whether or not the consumer runs here, for the reason the three # Minted whether or not the consumer runs here, for the reason the three
# leaves above give: on a hive that does not run the store this is the # leaves above give: on a hive that does not run the store this is the
# file an operator copies, and a leaf that only appears where its # file an operator copies, and a leaf that only appears where its
# consumer does is one nobody can copy from anywhere. # consumer does is one nobody can copy from anywhere.
[ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth
[ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \ [ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \
${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth ${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth
@ -344,14 +332,14 @@ in
${lib.optionalString deployCfg.hive-controller.enable '' ${lib.optionalString deployCfg.hive-controller.enable ''
# This hive's gateway-leaf identity, which opens # This hive's gateway-leaf identity, which opens
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its # `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its
# subject carries this hive's name, like the queue-agent leaf # subject carries this hive's name, because ./swarm-bao.nix writes
# above, because ./swarm-bao.nix writes one role # one role per hive. Only where a hive runs; the other hives' are
# per hive. Only where a hive runs; the other hives' are below. # below.
[ -s ${pkiDir}/hive-leaf.pem ] || ${signLeaf} ${pkiDir} hive-leaf \ [ -s ${pkiDir}/hive-leaf.pem ] || ${signLeaf} ${pkiDir} hive-leaf \
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${clientCn}"} "" clientAuth ${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${clientCn}"} "" clientAuth
''} ''}
# The reader, queue-agent and gateway-leaf leaves of every other hive in # The reader and gateway-leaf leaves of every other hive in
# the swarm directory: the files an operator copies to that hive's # the swarm directory: the files an operator copies to that hive's
# host. One directory per hive, with the basenames this hive's leaves # host. One directory per hive, with the basenames this hive's leaves
# use, so a hive's set is one directory to copy. # use, so a hive's set is one directory to copy.
@ -364,8 +352,6 @@ in
install -d -m 0700 ${d} install -d -m 0700 ${d}
[ -s ${d}/client.pem ] || ${signLeaf} ${d} client \ [ -s ${d}/client.pem ] || ${signLeaf} ${d} client \
${lib.escapeShellArg h} "" clientAuth ${lib.escapeShellArg h} "" clientAuth
[ -s ${d}/queue-agent.pem ] || ${signLeaf} ${d} queue-agent \
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${h}"} "" clientAuth
[ -s ${d}/hive-leaf.pem ] || ${signLeaf} ${d} hive-leaf \ [ -s ${d}/hive-leaf.pem ] || ${signLeaf} ${d} hive-leaf \
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${h}"} "" clientAuth ${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${h}"} "" clientAuth
'' ''

View file

@ -1,284 +0,0 @@
# Glue: this hive's agent queue credential comes out of the secret store.
#
# A cert login that fails LOUDLY because every state it fails on is one a
# retry fixes, then a read that degrades QUIETLY because no retry turns "no
# value there" into a value.
#
# ⚠️ Two files, not one, and that is the consumer's shape rather than a
# preference. `swarm-logs` inside the agent reads the secret and the client id
# each from its own file (`<prefix>_OIDC_CLIENT_SECRET_FILE`,
# `<prefix>_OIDC_CLIENT_ID_FILE`), so splitting them here is what lets
# hive-c0re hand both to an agent without parsing anything.
#
# ⚠️ An ABSENT file means this hive's agents do not connect to the queue, and
# that is correct rather than degraded. The publisher runs on the authelia
# host and authelia mints on its first boot, so "nothing at that path yet" is
# the ordinary early state of a swarm. Nothing here writes a local stand-in:
# there is no such thing as a locally valid OIDC client secret, so a placeholder would turn a hive that cannot connect
# into one that is refused, which reaches the agent as a timeout.
#
# 📌 This runs wherever a client identity is configured, NOT only where the
# store is. `deploy.bao.enable` would have been the co-location assumption
# itself; the reader needs a certificate, not a neighbour. On the store's own
# host ./glue-bao-tls.nix supplies one as a `mkDefault`; elsewhere an operator
# places the leaf and names it, and the same unit works. It also needs this
# host to run a hive (`deploy.hive-controller.enable`): the credential is for
# the agents hive-c0re creates here at runtime, and the unit writes under
# hive-c0re's own state directory.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
# What decides whether this unit exists at all. A reader is defined by holding
# a certificate the store accepts, and that is true on the store's own host
# and on a hive three networks away for exactly the same reason.
#
# 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's. Bao
# matches a cert-auth role on the CN, so one leaf shared by four readers
# would be ONE principal holding the union of four grants: read on
# `swarm/agents/*` AND `swarm/hives/<hive>/*`, when this unit reads one
# queue credential and nothing else. Its own leaf carries
# `<queueAgentCommonNamePrefix>-<hive>` and its role grants the single path
# below — still this hive's own, so the narrowing costs no reach.
haveClientIdentity =
baoDeploy.queueAgentClientCertFile != null && baoDeploy.queueAgentClientKeyFile != null;
# Does this host read the store at all — the hive's own leaf, which is the
# one thing a remote-store deployment has to place by hand. Only used to
# decide whether a missing per-principal leaf is a mistake or a deployment
# that has no store: a host holding neither is the supported no-store
# shape, and one holding this pair but not the pair above reads the store
# without a reader identity of its own.
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir;
secretFile = "${credentialDir}/secret";
clientIdFile = "${credentialDir}/client_id";
# Where the credential lives in the store, spelled from the same pieces the
# writer uses. Whoever writes it and whoever reads it must agree, and the
# agreement belongs in one visible place.
#
# ⚠️ The `hives/<name>` segment is not decoration — it is what the reader's
# own grant covers, so a path outside it is a 403 rather than a miss, however
# correct it looks. `swarm-secret-client`'s `queue::agent_client_path` builds
# the same string from the same pieces; this literal is the nix half of that
# one agreement.
#
# `hiveName` has no fallback here for the reason ./glue-bao-tls.nix gives at
# its own use of it: it is asserted set wherever
# `deploy.hive-controller.enable` is.
credentialPath = "secret/swarm/hives/${hyperhiveCfg.hiveName}/queue/agent";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
storeRetry = import ./lib/store-retry.nix { };
in
{
options.services.hyperhive.deploy.hive-controller.queue = {
agentCredentialDir = lib.mkOption {
type = lib.types.path;
default = "/var/lib/hyperhive/queue-agent";
description = ''
Host directory holding this hive's agent queue credential: `secret`
(the OIDC client secret, `0600`) and `client_id` (the client that
secret authenticates, `0644` — it is sent to the token endpoint on
every connection and is not itself a secret).
Both files are written by `swarm-bao-queue-agent.service`, which also
creates this directory. Named here because nothing else creates it:
the parent `/var/lib/hyperhive` belongs to `hive-c0re.service`'s
`StateDirectory=`, which re-applies its own mode on every start, and a
second declaration of *that* path is the two-mechanisms-one-path trap
./hive-gateway/default.nix records — so this directory is the unit's
to make and the parent stays c0re's.
Absent files mean this hive has no queue credential yet, which is what
a swarm looks like before the publisher in authelia's container has run.
'';
};
};
config = lib.mkMerge [
# ⚠️ A SEPARATE arm from the unit below, and that separation is the whole
# mechanism: the unit's arm is gated on `haveClientIdentity`, so an
# assertion written inside it could never be reached in the state it
# exists to report.
#
# Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the
# same refusal, named to this principal's own pair. Where Grafana's gate
# is `deploy.grafana.enable`, this reader's is the hive's own: every hive
# runs one for its own agents, so on a hive, reading the store at all is
# what asks for it. Hence `hiveReaderIdentity` — a host holding no hive
# leaf has no store to read and nothing is missing.
(lib.mkIf (deployCfg.hive-controller.enable && hiveReaderIdentity) {
assertions = [
{
assertion = haveClientIdentity;
message = ''
This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile
is set), so it needs the agent queue credential reader's own client
identity: set both
services.hyperhive.deploy.bao.queueAgentClientCertFile
services.hyperhive.deploy.bao.queueAgentClientKeyFile
swarm-bao-queue-agent.service fetches this hive's agent queue
credential out of the store, and without these it is not rendered
at all — leaving hive-c0re with no queue credential and no unit
that would ever have written one.
Every hive runs this reader for its own agents, so unlike the other
three principals there is no per-service toggle that turns it off:
a hive that reads the store at all is a hive that needs this pair.
⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one
is the hive's, and its grant reads every agent credential in the
swarm and this hive's own secrets; this role reads the one
queue-credential path. Pointing this option at the hive's leaf
would evaluate, deploy and log in — and undo the split.
On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere copy the pair from
/var/lib/swarm-bao-pki/hives/<hive>/ on the store's host and name it
here — see docs/swarm/bao.md.
'';
}
];
})
(lib.mkIf (deployCfg.hive-controller.enable && haveClientIdentity) {
systemd.services.swarm-bao-queue-agent = {
description = "fetch this hive's agent queue credential from the swarm secret store";
# Every one of these names a unit that exists only where the store runs,
# so the ordering is conditional even though the read is not: off-host
# there is nothing local to wait for, and the timeout below is what
# bounds the attempt instead. `Wants=` pki, for
# ./glue-bao-readers-store-order.nix's reason.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
# Ordered before hive-c0re, so no agent container renders ahead of an
# attempt at its credential. `Wants=`, not `Requires=`: a store this
# unit can't reach delays hive-c0re's start by one attempt
# (`TimeoutStartSec` below) rather than failing it — hive-c0re starts
# after that attempt with whatever credential is on disk, while
# ./lib/store-retry.nix keeps retrying in the background.
before = [ "hive-c0re.service" ];
wantedBy = [
"multi-user.target"
"hive-c0re.service"
];
path = [
baoDeploy.package
pkgs.coreutils
];
# ./lib/store-retry.nix.
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
# What actually bounds each attempt at the reads below. Stated here
# rather than left to systemd's default, so the number a boot waits on
# is in the file that waits.
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile;
BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
${atomicWriteSecret}
# `bao`'s own message is the only thing separating a missing value from
# a refused identity from an unreachable host. This unit's degraded
# mode is correct for all three, so it reports which one rather than
# asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Fails LOUDLY, unlike the reads below: the three states a login
# failure covers — store not up, sealed, role not written yet — are all
# things a retry fixes, and `Restart=on-failure` above is what retries.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
# Two reads of one object rather than one `-format=json` parsed with
# `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot
# actually disagree — the client id is derived from this hive's name, so
# a rotation landing between these two calls changes the secret and
# rewrites the same id.
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then
echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
# Both or neither: a client that finds one of the two comes up "fine"
# and never connects.
if [ -z "$secret" ] || [ -z "$client_id" ]; then
echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2
exit 0
fi
install -d -m 0755 ${lib.escapeShellArg credentialDir}
atomic_write_secret 0600 "" ${lib.escapeShellArg secretFile} "$secret"
# `0644` on purpose: an OIDC client id is presented to the token
# endpoint on every connection and is public by construction.
atomic_write_secret 0644 "" ${lib.escapeShellArg clientIdFile} "$client_id"
'';
};
})
];
}

View file

@ -333,8 +333,7 @@ in
# from an unprivileged daemon without widening either. # from an unprivileged daemon without widening either.
# The gate is the identity, never `deploy.bao.enable`: a hive that # The gate is the identity, never `deploy.bao.enable`: a hive that
# reads a store on another machine holds one of these and runs no # reads a store on another machine holds one of these and runs no
# store. ./glue-queue-agent-credential.nix gates its own reader the # store. ../swarm-otel.nix gates its own reader the same way.
# same way.
lib.optionals baoClientIdentity [ lib.optionals baoClientIdentity [
"bao-client.pem:${baoDeploy.clientCertFile}" "bao-client.pem:${baoDeploy.clientCertFile}"
"bao-client-key.pem:${baoDeploy.clientKeyFile}" "bao-client-key.pem:${baoDeploy.clientKeyFile}"

View file

@ -11,9 +11,8 @@ let
cfg = config.services.hyperhive.c0re; cfg = config.services.hyperhive.c0re;
baoDeploy = config.services.hyperhive.deploy.bao; baoDeploy = config.services.hyperhive.deploy.bao;
baoCfg = config.services.hyperhive.swarm.bao; baoCfg = config.services.hyperhive.swarm.bao;
# The same test ../glue-queue-agent-credential.nix applies, and for the same # A reader is defined by holding a certificate the store accepts, not by
# reason: a reader is defined by holding a certificate the store accepts, # sharing a host with the store. Deriving this from
# not by sharing a host with the store. Deriving this from
# `deploy.bao.enable` would be the co-location assumption itself. # `deploy.bao.enable` would be the co-location assumption itself.
haveBaoClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; haveBaoClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
in in
@ -222,31 +221,15 @@ in
# the one the roster gave it. # the one the roster gave it.
HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}"; HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}";
} }
// {
# Where ../glue-queue-agent-credential.nix lands the AGENTS' queue
# credential. Read by `hive_c0re::lifecycle::host_config`, which stats the
# two files and forwards them into each container as systemd credentials.
# Never read for its contents here: the secret is `0600` root-owned and
# this daemon is `hive-core`, which is exactly why the transport is a
# credential rather than a bind mount.
HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR = toString config.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
}
// //
# The agents' half of the same queue, forwarded by `hive_c0re::meta` into # The agents' half of the same queue, forwarded by `hive_c0re::meta` into
# every container. A pair, gated together, because an agent that got one of # every container.
# them would report a half-configured queue instead of none.
# #
# ⚠️ By name, like `HIVE_C0RE_NATS_URL` above, and never a loopback # ⚠️ By name, like `HIVE_C0RE_NATS_URL` above, and never a loopback
# address: inside a container `127.0.0.1` is the agent itself. # address: inside a container `127.0.0.1` is the agent itself.
lib.optionalAttrs lib.optionalAttrs (config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null) {
( HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl;
config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null }
&& config.services.hyperhive.swarm.statusPublish.tokenEndpoint != null
)
{
HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl;
HIVE_AGENT_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
}
// //
# Where an agent reads the swarm's logs. Gated on the store identity below, # Where an agent reads the swarm's logs. Gated on the store identity below,
# because `swarm-logs` authenticates with the agent's own queue secret, read # because `swarm-logs` authenticates with the agent's own queue secret, read

View file

@ -352,22 +352,14 @@ in
readOnly = true; readOnly = true;
default = "-agent"; default = "-agent";
description = '' description = ''
Suffix appended to a hive's own client id to name the client its Suffix the queue's auth-callout responder refuses on a hive's client
*agent containers* present — agents on hive `alpha` all present id: `${config.services.hyperhive.swarm.authelia.hiveClientPrefix}alpha${config.services.hyperhive.swarm.authelia.agentClientSuffix}`
`${config.services.hyperhive.swarm.authelia.hiveClientPrefix}alpha${config.services.hyperhive.swarm.authelia.agentClientSuffix}`. is denied, never read as hive `alpha-agent`. No client is minted with
Read-only for the same reason as `hiveClientPrefix`, and read by the it; an agent authenticates to the queue with its own secret.
same consumer with the same failure mode: a split spelling denies
every agent as a timeout.
A suffix on the hive's id rather than a prefix of its own, because Read-only for the same reason as `hiveClientPrefix`. A hive named with
`agent-alpha` reads as *the agent named alpha* — which is precisely this suffix is refused at evaluation, since the queue would refuse its
what this identity does not say. own client id.
One id per hive rather than per agent, because agents are created at
runtime and a per-agent client would make creating one a config
change plus a reload. The consequence is that this identity says
*which hive* an agent belongs to and never *which agent* — the
broker cannot tell two agents on one hive apart.
''; '';
}; };

View file

@ -230,8 +230,8 @@ let
# The secret publisher runs in this container, beside the mint whose secrets # The secret publisher runs in this container, beside the mint whose secrets
# it copies. It needs a store identity, and a reader is defined by holding a # it copies. It needs a store identity, and a reader is defined by holding a
# certificate the store accepts, never by standing next to it — the rule # certificate the store accepts, never by standing next to it — the rule
# ./glue-queue-agent-credential.nix states in full. `deploy.bao.enable` here # ./swarm-bao.nix's `deploy.bao` options block states in full.
# would be the co-location assumption itself. # `deploy.bao.enable` here would be the co-location assumption itself.
publisherCfg = deployCfg.swarm-secret-publisher; publisherCfg = deployCfg.swarm-secret-publisher;
publishes = publishes =
publisherCfg.enable publisherCfg.enable
@ -300,71 +300,6 @@ let
accessTokenSignedResponseAlg = "RS256"; accessTokenSignedResponseAlg = "RS256";
}) hyperhiveCfg.swarm.hives; }) hyperhiveCfg.swarm.hives;
# The one URL an agent's token has to be authorised at beyond the queue:
# the log store's machine query route. Read off `swarm.victorialogs.domain`
# — an unconditional option, declared whether or not this host runs the
# store — rather than restated, because it is also the string `swarm-logs`
# requests as its audience and the string `hive-c0re/environment.nix` hands
# an agent as its endpoint. Three readers, one formula; two spellings
# present as a valid token refused at the store.
logsQueryUrl = "https://${hyperhiveCfg.swarm.victorialogs.domain}/select/logsql/query";
# The identity an agent container presents to the swarm's queue. One per
# HIVE, not per agent, and that is the load-bearing choice rather than a
# shortcut: agents are created at **runtime**, so anything minted per agent
# would make creating one a config change plus an authelia reload. Keyed on
# the hive, this list's length tracks the roster above it — deploy-time, like
# every other entry here.
#
# ⚠️ The cost, stated rather than discovered later: every agent on a hive
# presents the SAME client id, so the broker can tell *hives* apart and not
# *agents*. Deliberate and deferred — closing it requires the client list to
# stop being static, which is the same problem the users-database writer
# already solves for identities.
#
# 🏷️ The id EXTENDS the hive's own (`hive-<name>-agent`) rather than taking a
# prefix of its own (`agent-<name>`), because that reads as *the agent called
# `<name>`* — which is the one thing this identity does not carry. The cost is
# that the two ids are not distinguishable by prefix, so the responder's
# agent rule must be tried BEFORE its hive rule; `policy.rs` says so at the
# match site, and the assertion below covers the case that ordering cannot.
#
# Mirrors `hiveClients` field for field so the two stay comparable. The
# signing algorithm is not load-bearing here — this client's consumers are
# the queue's auth-callout responder, which *introspects* rather than
# verifying offline, and authelia's own authz endpoint — but it matches its
# sibling rather than inventing a second answer to a question nobody asked.
agentClients = lib.mapAttrsToList (name: _: {
id = "${cfg.hiveClientPrefix}${name}${cfg.agentClientSuffix}";
description = "HyperHive agents on hive ${name}";
kind = "machine";
redirectUris = [ ];
# ⚠️ **Two** audiences, where `hiveClients` has one, because an agent
# presents this token at two different kinds of consumer. The queue
# introspects it and reads the client id; the log store's gateway
# location runs `auth_request` against authelia, which authorises a
# bearer token **by the URL being requested** and refuses one whose
# audience does not name it. Measured, not assumed: without the URL
# registered here the token endpoint answers `invalid_target` for it,
# and a token minted with no audience at all is answered by the gateway
# with a bare 401 that names no cause.
audience = [
"${cfg.hiveClientPrefix}${name}${cfg.agentClientSuffix}"
logsQueryUrl
];
# Without it the authz endpoint refuses an otherwise valid token and
# blames the token rather than the missing grant — the same note
# `glue-swarm-otel-oidc-client.nix` carries over the collector's client.
bearerAuthz = true;
# Stated rather than left on authelia's default, for
# `glue-swarm-otel-oidc-client.nix`'s reason: authelia permits only
# basic / JWT methods for a confidential client holding this scope, and
# enforces it in the startup validator. `swarm-queue-client` sends the
# credential as basic auth, so this is also what the minter already does.
tokenEndpointAuthMethod = "client_secret_basic";
accessTokenSignedResponseAlg = "RS256";
}) hyperhiveCfg.swarm.hives;
# `swarm-authelia-bridge`'s own identity — distinct from # `swarm-authelia-bridge`'s own identity — distinct from
# `swarm-controller`'s (`swarm-controller.nix`'s `queueClientId`). A # `swarm-controller`'s (`swarm-controller.nix`'s `queueClientId`). A
# resource server introspecting a token proves its OWN identity to the # resource server introspecting a token proves its OWN identity to the
@ -706,7 +641,7 @@ in
# wherever this module is, so `oidc.clients` is never actually empty # wherever this module is, so `oidc.clients` is never actually empty
# — see `oidcEnabled`'s comment above. # — see `oidcEnabled`'s comment above.
services.hyperhive.swarm.authelia.oidc.clients = services.hyperhive.swarm.authelia.oidc.clients =
lib.optionals cfg.oidc.hiveIdentities (hiveClients ++ agentClients) lib.optionals cfg.oidc.hiveIdentities hiveClients
++ [ ++ [
bridgeClient bridgeClient
forwarderClient forwarderClient
@ -749,18 +684,11 @@ in
} }
{ {
# A hive whose name ENDS with the agent suffix mints an id the # A hive whose name ENDS with the agent suffix mints an id the
# queue's responder reads as somebody else's agents: # queue's responder reads as a hive's agent client, which it refuses:
# `hive-foo-agent` parses as *the agents of hive foo* before it # `hive-foo-agent` parses as *the agents of hive foo* before it
# parses as *the hive foo-agent*, because the agent rule is the # parses as *the hive foo-agent*, because the agent rule is the
# more specific one and therefore runs first. # more specific one and therefore runs first. A NATS denial arrives
# # as a timeout, so the symptom names nothing.
# ⚠️ NOT covered by the duplicate-id assertion above, and the gap is
# the interesting half. That one fires only when BOTH `foo` and
# `foo-agent` are on the roster, because only then are two clients
# actually named the same string. With `foo-agent` alone there is no
# duplicate and nothing to see — the hive simply receives its
# agents' grant instead of its own, and a NATS denial arrives as a
# timeout, so the symptom names nothing.
# #
# Checkable here and nowhere downstream: the responder runs in a # Checkable here and nowhere downstream: the responder runs in a
# container with no view of the roster (`policy.rs`'s `hive_name` # container with no view of the roster (`policy.rs`'s `hive_name`
@ -774,10 +702,9 @@ in
+ lib.concatMapStringsSep ", " (h: "'${h}'") ( + lib.concatMapStringsSep ", " (h: "'${h}'") (
lib.filter (h: lib.hasSuffix cfg.agentClientSuffix h) (lib.attrNames hyperhiveCfg.swarm.hives) lib.filter (h: lib.hasSuffix cfg.agentClientSuffix h) (lib.attrNames hyperhiveCfg.swarm.hives)
) )
+ " — names ending with '${cfg.agentClientSuffix}', the suffix that " + " — names ending with '${cfg.agentClientSuffix}', the suffix the "
+ "marks a hive's agent containers. Such a hive's own client id is " + "queue refuses as a hive's agent client. Such a hive's own client "
+ "indistinguishable from another hive's agent client, and the " + "id reads as one, and the queue refuses it. Rename the hive.";
+ "queue resolves it as the agents. Rename the hive.";
} }
# The two obligations `authelia.bearer.authz` carries. Authelia # The two obligations `authelia.bearer.authz` carries. Authelia
# enforces both itself — but in its `preStart` validator, so a # enforces both itself — but in its `preStart` validator, so a

View file

@ -808,9 +808,9 @@ let
} }
''; '';
# One gateway-leaf issuer per hive in the swarm directory, for the reason # One gateway-leaf issuer per hive in the swarm directory: one subject for
# `perHiveReaders` below is per hive. One name for the pki role, the policy # every hive could ask for any hive's certificate. One name for the pki role,
# and the cert-auth role, as for the readers. # the policy and the cert-auth role, as for the readers below.
hiveLeafIssuers = lib.mapAttrsToList (hiveName: hive: { hiveLeafIssuers = lib.mapAttrsToList (hiveName: hive: {
name = "${baoDeploy.hiveLeafPkiRolePrefix}-${hiveName}"; name = "${baoDeploy.hiveLeafPkiRolePrefix}-${hiveName}";
cn = "${baoDeploy.hiveLeafCommonNamePrefix}-${hiveName}"; cn = "${baoDeploy.hiveLeafCommonNamePrefix}-${hiveName}";
@ -897,48 +897,21 @@ let
# `swarm/agents/*` AND `swarm/hives/<hive>/*`, and nothing under # `swarm/agents/*` AND `swarm/hives/<hive>/*`, and nothing under
# `swarm/services/`. # `swarm/services/`.
# Principals presenting one certificate are ONE principal to bao, so sharing # Principals presenting one certificate are ONE principal to bao, so sharing
# the hive's leaf would give each of them everything that leaf reads: the # the hive's leaf would give each of them everything that leaf reads: a unit
# unit that fetches this hive's queue credential could fetch every agent # that fetches a service's OIDC client secret could fetch every agent
# credential in the swarm, and a unit that fetches a service's OIDC client # credential in the swarm, and could not fetch its own secret at all, as
# secret could not fetch it at all, as that path is not the hive's to read. # that path is not the hive's to read.
# #
# Each gets its own leaf, its own cert-auth role and a policy holding the # Each gets its own leaf, its own cert-auth role and a policy holding the
# one `secret/data/…` path that principal's script actually names — spelled # one `secret/data/…` path that principal's script actually names — spelled
# to the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText` # to the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText`
# above gives for doing the same. # above gives for doing the same.
#
# ⚠️ The queue-credential reader is PER HIVE and the OIDC readers are not,
# and that asymmetry is the shape of the paths rather than a preference. A
# queue credential lives under `swarm/hives/<name>/`, and every hive runs a
# reader for its own; an OIDC client secret lives under
# `swarm/services/<client-id>/` and a swarm registers each of those exactly
# once. A single role for a per-hive reader would have to name `hives/*` to
# serve every hive — which would let one hive read another's queue
# credential, a reach no hive needs.
readStanza = path: '' readStanza = path: ''
path "${path}" { path "${path}" {
capabilities = ["read"] capabilities = ["read"]
} }
''; '';
# One reader per hive in the swarm directory. Written from the directory
# rather than from this host's own name because the policy is written where
# the STORE is and the reader runs where its hive is, so this host's name is
# not the reader's.
perHiveReaders =
{ rolePrefix, cnPrefix, ... }@spec:
lib.mapAttrsToList (hiveName: _: {
name = "${rolePrefix}-${hiveName}";
cn = "${cnPrefix}-${hiveName}";
policyText = readStanza (spec.path hiveName);
}) hyperhiveCfg.swarm.hives;
queueAgentReaders = perHiveReaders {
rolePrefix = "swarm-queue-agent";
cnPrefix = baoDeploy.queueAgentCommonNamePrefix;
path = h: "${credentialMountPath}/data/swarm/hives/${h}/queue/agent";
};
# Singletons, so a list of one rather than a second shape: the granting unit # Singletons, so a list of one rather than a second shape: the granting unit
# below takes a list either way and the units stay identical apart from # below takes a list either way and the units stay identical apart from
# the objects they write. # the objects they write.
@ -1257,9 +1230,9 @@ let
# One unit per PRINCIPAL, not one unit for all of them: these names are # One unit per PRINCIPAL, not one unit for all of them: these names are
# operator-facing strings, the same reason `swarm-bao-secret-publisher-policy` # operator-facing strings, the same reason `swarm-bao-secret-publisher-policy`
# below gives for not being more script inside the controller's unit. A hive # below gives for not being more script inside the controller's unit. A
# whose queue credential never arrives is diagnosed by the state of the unit named # reader whose secret never arrives is diagnosed by the state of the unit
# after it. # named after it.
readerPolicyUnit = readerPolicyUnit =
description: objects: description: objects:
granterPolicyUnit description ( granterPolicyUnit description (
@ -1589,7 +1562,7 @@ in
# `serverCaFile` live here too, and they are set by whoever authenticates TO # `serverCaFile` live here too, and they are set by whoever authenticates TO
# the store — which includes a host that runs none of it. A reader is defined # the store — which includes a host that runs none of it. A reader is defined
# by holding a certificate the store accepts, never by sharing a host with # by holding a certificate the store accepts, never by sharing a host with
# one; ./glue-queue-agent-credential.nix gates on exactly that and not on # one; ./swarm-otel.nix's store reader gates on exactly that and not on
# `enable`. # `enable`.
# Reading this block as store-runner-only is what makes an off-host reader # Reading this block as store-runner-only is what makes an off-host reader
# look inexpressible when it is already supported. # look inexpressible when it is already supported.
@ -2102,28 +2075,6 @@ in
''; '';
}; };
queueAgentCommonNamePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-queue-agent";
example = "swarm-bao-queue-agent.svc";
description = ''
Prefix of the subject the store's per-hive queue-credential roles
accept. The role for hive `h` accepts `<prefix>-h` and grants read on
exactly `swarm/hives/h/queue/agent` — the client credential that hive's
agent containers authenticate to the swarm queue with.
A **prefix** rather than one subject because every hive runs its own
reader and the path it reads carries that hive's name. One subject for
all of them would need a `hives/*` grant, which would let one hive read
another hive's queue credential.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its
fixed-subject siblings are: cert auth trusts the CA and
`allowed_common_names` is the whole narrowing, so a hive whose own leaf
carried one of these spellings would receive that grant.
'';
};
hiveLeafCommonNamePrefix = lib.mkOption { hiveLeafCommonNamePrefix = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "swarm-bao-hive-leaf"; default = "swarm-bao-hive-leaf";
@ -2134,9 +2085,8 @@ in
`<servicesPkiMountPath>/issue/<hiveLeafPkiRolePrefix>-h`, which issues `<servicesPkiMountPath>/issue/<hiveLeafPkiRolePrefix>-h`, which issues
for `h`'s domain and `*.` that domain and nothing else. for `h`'s domain and `*.` that domain and nothing else.
Per hive for the reason Per hive because one subject for every hive could ask for any hive's
{option}`services.hyperhive.deploy.bao.queueAgentCommonNamePrefix` is: certificate.
one subject for every hive could ask for any hive's certificate.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, like its siblings. ⚠️ Reserved as a hive name by ./swarm.nix, per hive, like its siblings.
''; '';
@ -2163,7 +2113,7 @@ in
Its grant is one path, Its grant is one path,
`swarm/services/<grafana client id>/oidc/client`, and read only. Not `swarm/services/<grafana client id>/oidc/client`, and read only. Not
per hive, unlike the two prefixes above: an OIDC client is registered per hive, unlike the prefix above: an OIDC client is registered
once per swarm, so the path names the service and never a hive. once per swarm, so the path names the service and never a hive.
⚠️ Same collision as its siblings, and the same answer: ./swarm.nix ⚠️ Same collision as its siblings, and the same answer: ./swarm.nix
@ -2206,37 +2156,6 @@ in
''; '';
}; };
queueAgentClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/queue-agent.pem";
description = ''
Certificate the unit that fetches this hive's queue client credential
presents to the store. Its subject must be
`<queueAgentCommonNamePrefix>-<this hive's name>`; cert auth matches on
the CN and the role accepts nothing else.
⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
Pointing this at the hive's own leaf would evaluate, deploy and log in —
and hand a unit that reads one credential a certificate that reads every
agent credential in the swarm and this hive's own secrets.
No default here: ./glue-bao-tls.nix points it at the leaf it mints,
wherever this host mints its own PKI. A deployment that reads a store on
another host names it by hand, alongside the leaf itself.
'';
};
queueAgentClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/queue-agent-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile`.
'';
};
grafanaOidcClientCertFile = lib.mkOption { grafanaOidcClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str; type = lib.types.nullOr lib.types.str;
default = null; default = null;
@ -2246,8 +2165,10 @@ in
to the store. Its subject must be to the store. Its subject must be
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`. {option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason ⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives. Pointing this at the hive's own leaf would evaluate, deploy and log in —
and hand a unit that reads one secret a certificate that reads every
agent credential in the swarm and this hive's own secrets.
''; '';
}; };
@ -2271,7 +2192,7 @@ in
{option}`services.hyperhive.deploy.bao.otelOidcCommonName`. {option}`services.hyperhive.deploy.bao.otelOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason ⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives. {option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile` gives.
''; '';
}; };
@ -2295,7 +2216,7 @@ in
{option}`services.hyperhive.deploy.bao.forwarderOidcCommonName`. {option}`services.hyperhive.deploy.bao.forwarderOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason ⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives. {option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile` gives.
''; '';
}; };
@ -3543,12 +3464,10 @@ in
''; '';
# 🩸 The principals that each log in with a leaf of their own, not as the # 🩸 The principals that each log in with a leaf of their own, not as the
# hive. See the block around `perHiveReaders` for what each grant is and # hive. See the block around `readStanza` for what each grant is; the
# why the first is per hive; the units are identical apart from the # units are identical apart from the objects they write, so the shape
# objects they write, so the shape lives in `readerPolicyUnit` and the # lives in `readerPolicyUnit` and the difference is visible here in one
# difference is visible here in one line each. # line each.
systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders;
systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders; systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders;
systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders; systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders;
@ -3693,7 +3612,6 @@ in
"swarm-bao-controller-policy.service" "swarm-bao-controller-policy.service"
"swarm-bao-secret-publisher-policy.service" "swarm-bao-secret-publisher-policy.service"
"swarm-bao-matrix-ctl-policy.service" "swarm-bao-matrix-ctl-policy.service"
"swarm-bao-queue-agent-policy.service"
"swarm-bao-grafana-oidc-policy.service" "swarm-bao-grafana-oidc-policy.service"
"swarm-bao-otel-oidc-policy.service" "swarm-bao-otel-oidc-policy.service"
"swarm-bao-forwarder-oidc-policy.service" "swarm-bao-forwarder-oidc-policy.service"

View file

@ -282,8 +282,8 @@ let
); );
# A reader of the store is defined by holding a certificate the store # A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule # accepts, never by standing next to it — the rule ./swarm-bao.nix's
# ./glue-queue-agent-credential.nix states in full. # `deploy.bao` options block states in full.
# #
# Also asserted rather than gating anything, and that is the ruling this # Also asserted rather than gating anything, and that is the ruling this
# module's second delivery route was deleted under: the store exists so that # module's second delivery route was deleted under: the store exists so that
@ -722,10 +722,10 @@ in
# to be wrong, and a gate whose false arm is a Grafana with no login. # to be wrong, and a gate whose false arm is a Grafana with no login.
# Recorded in docs/swarm/secrets.md. # Recorded in docs/swarm/secrets.md.
# #
# Shaped after ./glue-queue-agent-credential.nix, because it is a store # It is a store reader, and this is what the store's readers do: a cert
# reader and this is what the store's readers do: a cert login that fails # login that fails LOUDLY, since every state it fails on is one a retry
# LOUDLY, since every state it fails on is one a retry fixes, then a read # fixes, then a read that degrades QUIETLY, since no retry turns "no value
# that degrades QUIETLY, since no retry turns "no value there" into a value. # there" into a value.
# #
# ⚠️ An ABSENT secret is "not published yet", not a fault of this host's. # ⚠️ An ABSENT secret is "not published yet", not a fault of this host's.
# The publisher runs in authelia's container and authelia mints on its # The publisher runs in authelia's container and authelia mints on its

View file

@ -873,7 +873,7 @@ in
# terminal rows and its turn-state header # terminal rows and its turn-state header
# (`hive-agent::swarm_agent_state`), two subject families so a # (`hive-agent::swarm_agent_state`), two subject families so a
# subscriber following one agent's terminal is not also handed # subscriber following one agent's terminal is not also handed
# every header. The hive's shared agent client gets nothing. # every header.
# #
# `$$`, not `$`: systemd substitutes `$NAME` in `ExecStart` # `$$`, not `$`: systemd substitutes `$NAME` in `ExecStart`
# whether or not it is quoted, so a single dollar reaches the # whether or not it is quoted, so a single dollar reaches the

View file

@ -232,12 +232,11 @@ let
haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null; haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null;
# A reader of the store is defined by holding a certificate the store # A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule # accepts, never by standing next to it — the rule ./swarm-bao.nix's
# ./glue-queue-agent-credential.nix states in full. Unlike Grafana's identical- # `deploy.bao` options block states in full. Unlike Grafana's identical-
# looking flag, this one does not gate an assertion on its own: the # looking flag, this one does not gate an assertion on its own: the
# store-reading unit below simply does not render without it, the same choice # store-reading unit below simply does not render without it, because a
# ./glue-queue-agent-credential.nix makes for its own reader, because a collector with no client identity is # collector with no client identity is `haveCollectorSecret = false` above, and that is already a supported,
# `haveCollectorSecret = false` above, and that is already a supported,
# merely degraded shape rather than a service with no way in at all. What # merely degraded shape rather than a service with no way in at all. What
# IS asserted is narrower and lives in `assertions` below — see # IS asserted is narrower and lives in `assertions` below — see
# `hiveReaderIdentity`. # `hiveReaderIdentity`.
@ -539,7 +538,7 @@ in
# exists so a host holds ONE out-of-band secret — its client certificate — # exists so a host holds ONE out-of-band secret — its client certificate —
# and reads everything else with it. Recorded in docs/swarm/secrets.md. # and reads everything else with it. Recorded in docs/swarm/secrets.md.
# #
# Shaped after ./glue-queue-agent-credential.nix: a cert login that fails # Shaped after ./swarm-grafana.nix's store reader: a cert login that fails
# LOUDLY, since every state it fails on is one a retry fixes, then a read # LOUDLY, since every state it fails on is one a retry fixes, then a read
# that degrades QUIETLY, since no retry turns "no value there" into a # that degrades QUIETLY, since no retry turns "no value there" into a
# value. # value.

View file

@ -12,15 +12,9 @@ let
hyperhiveCfg = config.services.hyperhive; hyperhiveCfg = config.services.hyperhive;
hiveNames = lib.attrNames hyperhiveCfg.swarm.hives; hiveNames = lib.attrNames hyperhiveCfg.swarm.hives;
# The client id agent containers present, per hive — composed exactly as
# ./swarm-authelia.nix composes it, from the same two read-only options, so a
# rename there cannot leave this spelling behind.
agentClientId =
hive:
"${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}${hyperhiveCfg.swarm.authelia.agentClientSuffix}";
# The hive's own client id, which hive-c0re publishes the hive's status with. # The hive's own client id, which hive-c0re publishes the hive's status with.
# Composed as ./swarm-authelia.nix composes it, for the reason above. # Composed exactly as ./swarm-authelia.nix composes it, from the same
# read-only option, so a rename there cannot leave this spelling behind.
hiveClientId = hive: "${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}"; hiveClientId = hive: "${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}";
# The swarm's own services, as opposed to its hives. One client for the whole # The swarm's own services, as opposed to its hives. One client for the whole
@ -83,22 +77,14 @@ let
controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client"; controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client";
secrets = secrets =
# `client_id` beside the value: an agent's id is derived per hive and has
# to be reconstructable from the store alone.
map (hive: {
clientId = agentClientId hive;
path = "secret/swarm/hives/${hive}/queue/agent";
withClientId = true;
}) hiveNames
# Each hive's own client, which that hive's hive-c0re reads under its own # Each hive's own client, which that hive's hive-c0re reads under its own
# certificate. The nix half of # certificate. The nix half of
# `swarm_secret_client::queue::hive_client_path`. # `swarm_secret_client::queue::hive_client_path`.
++ map (hive: { map (hive: {
clientId = hiveClientId hive; clientId = hiveClientId hive;
path = "secret/swarm/hives/${hive}/oidc/client"; path = "secret/swarm/hives/${hive}/oidc/client";
}) hiveNames }) hiveNames
# No `client_id`: a service's client id is the swarm-wide option both # A service's client id is the swarm-wide option both ends already read.
# ends already read.
++ map (id: { ++ map (id: {
clientId = id; clientId = id;
path = "secret/swarm/services/${id}/oidc/client"; path = "secret/swarm/services/${id}/oidc/client";
@ -120,9 +106,8 @@ in
default = false; default = false;
description = '' description = ''
Publish the OIDC client secrets this host mints into the swarm's Publish the OIDC client secrets this host mints into the swarm's
secret store, so a hive that does not run authelia can read its secret store, so a hive that does not run authelia, a swarm service
own and its agents' credentials and a swarm service or the or the controller can read its own — from wherever it runs, this host
controller can read its own — from wherever it runs, this host
included. The unit, `swarm-secret-publish`, runs inside authelia's included. The unit, `swarm-secret-publish`, runs inside authelia's
container, so it exists only where authelia runs. container, so it exists only where authelia runs.
@ -137,10 +122,9 @@ in
publishing nothing. Whether the wiring is complete is the client publishing nothing. Whether the wiring is complete is the client
identity's job (see `baoClientCertFile`), not this option's. identity's job (see `baoClientCertFile`), not this option's.
Turning it off leaves every hive but this one without its agents' Turning it off leaves every hive unable to publish its status, the
credential, every hive unable to publish its status, the swarm swarm controller unable to start, the swarm's Grafana without any login
controller unable to start, the swarm's Grafana without any login at at all, its collector pushing unauthenticated, the queue's responder
all, its collector pushing unauthenticated, the queue's responder
unable to start, the homeserver unable to start, and the forge's login unable to start, the homeserver unable to start, and the forge's login
source without its secret. source without its secret.
Each secret has exactly one route and this is the producer's Each secret has exactly one route and this is the producer's
@ -185,9 +169,8 @@ in
internal = true; internal = true;
default = secrets; default = secrets;
description = '' description = ''
Every secret the publisher copies: the minted client's id, the store Every secret the publisher copies: the minted client's id and the store
path its reader fetches, and whether `client_id` is stored beside the path its reader fetches. Derived from `swarm.*` alone, so it is the same list on every
value. Derived from `swarm.*` alone, so it is the same list on every
host; ./swarm-authelia.nix hands it to the unit in its container. host; ./swarm-authelia.nix hands it to the unit in its container.
''; '';
}; };

View file

@ -64,17 +64,14 @@ let
deployCfg.bao.notifyRouterCommonName deployCfg.bao.notifyRouterCommonName
deployCfg.bao.forgePublisherCommonName deployCfg.bao.forgePublisherCommonName
] ]
# The per-hive subjects, spelled out per hive rather than as the prefix. # The per-hive subject, spelled out per hive rather than as the prefix.
# The prefix alone would reserve the wrong string: the role for hive # The prefix alone would reserve the wrong string: the role for hive
# `h` accepts `<prefix>-h`, so the name a hive must not BE is that composed # `h` accepts `<prefix>-h`, so the name a hive must not BE is that composed
# spelling, and a hive named it would present a leaf that role accepts. # spelling, and a hive named it would present a leaf that role accepts.
# #
# Every spelling is reserved for every declared hive, including the one the # Every spelling is reserved for every declared hive, including the one the
# CN belongs to — a hive cannot be named after its own reader either. # CN belongs to — a hive cannot be named after its own leaf issuer either.
++ lib.concatMap (hive: [ ++ map (hive: "${deployCfg.bao.hiveLeafCommonNamePrefix}-${hive}") hiveNames;
"${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}"
"${deployCfg.bao.hiveLeafCommonNamePrefix}-${hive}"
]) hiveNames;
# Public hostnames of the swarm's own services, in declaration order. # Public hostnames of the swarm's own services, in declaration order.
# `serviceDomains` below is this set sorted + deduplicated. # `serviceDomains` below is this set sorted + deduplicated.
@ -477,9 +474,8 @@ in
# its own network namespace and resolver, so this is its own option, even # its own network namespace and resolver, so this is its own option, even
# though by name it defaults to the same URL as `statusPublish.natsUrl`. # though by name it defaults to the same URL as `statusPublish.natsUrl`.
# #
# Only the address lives here. The credential does not: it is published per # Only the address lives here. Each agent reads its own credential from the
# hive and read out of the store by ./glue-queue-agent-credential.nix, which # store (`nix/agent-modules/queue-identity.nix`).
# owns `queue.agentCredentialDir` in the same namespace.
options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption { options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str; type = lib.types.nullOr lib.types.str;
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}"; default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
@ -496,10 +492,8 @@ in
agent's network namespace `127.0.0.1` is the agent, not this host, agent's network namespace `127.0.0.1` is the agent, not this host,
and the queue's certificate names no address anyway. and the queue's certificate names no address anyway.
Null means this hive's agents have not been given the queue's address. Null means this hive's agents have not been given the queue's address,
Together with and the harness is handed none.
{option}`services.hyperhive.swarm.statusPublish.tokenEndpoint` it is
what decides whether the harness is handed queue coordinates at all.
''; '';
}; };

View file

@ -24,16 +24,13 @@ let
agentHarness agentHarness
; ;
# The agent side of the swarm queue. Both coordinates set is the only state # An `agent.nix` still setting the queue coordinates the agent module no
# in which the harness unit declares a credential at all, so the pair and # longer declares.
# the empty fixture beside it are the two arms worth having.
agentQueue = agent { agentQueue = agent {
queue.natsUrl = "nats://10.42.0.1:4222"; queue.natsUrl = "nats://10.42.0.1:4222";
queue.tokenEndpoint = "https://auth.t.local/api/oidc/token"; queue.tokenEndpoint = "https://auth.t.local/api/oidc/token";
}; };
agentNoQueue = shared.agentBare;
# The agent side of the swarm secret store. The address is the whole switch — # The agent side of the swarm secret store. The address is the whole switch —
# it is both what generates the login check and what that check points at — # it is both what generates the login check and what that check points at —
# so it and the empty fixture beside it are the two arms worth having. # so it and the empty fixture beside it are the two arms worth having.
@ -106,21 +103,24 @@ let
ok = swarmMetricsIn agentLogsNoBao == null && swarmMetricsIn agentLogs != null; ok = swarmMetricsIn agentLogsNoBao == null && swarmMetricsIn agentLogs != null;
} }
{ {
# Both ids or neither: the secret authenticates nobody without the id it # The hive forwards no shared queue credential, so the harness unit
# belongs to, and `swarm-logs` reads both. # names none, whatever the agent's config says.
name = "an agent with queue coordinates imports both halves of its credential"; name = "an agent imports no hive-shared queue credential";
ok = ok =
let let
c = (agentHarness agentQueue).serviceConfig.LoadCredential; credentials = m: (agentHarness m).serviceConfig.LoadCredential or [ ];
in in
builtins.elem "hive-queue-agent-secret" c && builtins.elem "hive-queue-agent-client-id" c; !lib.any (lib.hasPrefix "hive-queue-agent-") (credentials agentQueue ++ credentials agentBao)
# Control: the lookup finds the store identity's credential.
&& lib.elem "hive-agent-bao-server-ca" (credentials agentBao);
} }
{ {
# An agent built before its hive was handed the queue's address. It # Both options are refused, naming where the address
# must declare nothing rather than name a credential that never # comes from now, rather than dropped silently.
# arrives. name = "an agent setting the removed queue options is refused";
name = "an agent with no queue coordinates declares no credential"; ok =
ok = !((agentHarness agentNoQueue).serviceConfig ? LoadCredential); lib.count (a: !a.assertion && lib.hasInfix "HIVE_AGENT_NATS_URL" a.message) agentQueue.assertions
== 2;
} }
{ {
# The harness exits when it cannot reach the queue. It has to come back # The harness exits when it cannot reach the queue. It has to come back

View file

@ -106,14 +106,11 @@ let
deploy.matrix.enable = true; deploy.matrix.enable = true;
}; };
# The store with none of the readers beside it. Grafana and the collector are # The store with none of the readers beside it: Grafana and the collector are
# simply off; the queue reader renders on any host holding # off.
# its leaf, which ./glue-bao-tls.nix mints here, so that leaf is taken away.
baoGrantNoReaders = hiveWithController { baoGrantNoReaders = hiveWithController {
deploy.bao.enable = true; deploy.bao.enable = true;
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token"; deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
deploy.bao.queueAgentClientCertFile = lib.mkForce null;
deploy.bao.queueAgentClientKeyFile = lib.mkForce null;
}; };
# The store with the forwarder's own pair taken away. The forwarder renders # The store with the forwarder's own pair taken away. The forwarder renders
@ -131,8 +128,6 @@ let
deploy.swarm-otel.enable = true; deploy.swarm-otel.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.queueAgentClientCertFile = "/etc/pki/bao-queue-agent.pem";
deploy.bao.queueAgentClientKeyFile = "/etc/pki/bao-queue-agent-key.pem";
deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem"; deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem";
deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem"; deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem";
deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem";
@ -142,7 +137,6 @@ let
# The readers ./glue-bao-readers-policy-order.nix orders after their policy # The readers ./glue-bao-readers-policy-order.nix orders after their policy
# units. # units.
policyReaders = [ policyReaders = [
"swarm-bao-queue-agent"
"swarm-bao-grafana-oidc" "swarm-bao-grafana-oidc"
"swarm-bao-otel-oidc" "swarm-bao-otel-oidc"
]; ];
@ -175,7 +169,6 @@ let
"swarm-bao-controller-policy" "swarm-bao-controller-policy"
"swarm-bao-secret-publisher-policy" "swarm-bao-secret-publisher-policy"
"swarm-bao-matrix-ctl-policy" "swarm-bao-matrix-ctl-policy"
"swarm-bao-queue-agent-policy"
"swarm-bao-grafana-oidc-policy" "swarm-bao-grafana-oidc-policy"
"swarm-bao-otel-oidc-policy" "swarm-bao-otel-oidc-policy"
"swarm-bao-forwarder-oidc-policy" "swarm-bao-forwarder-oidc-policy"
@ -599,19 +592,18 @@ let
# arms pin what each principal must NOT reach, so a later widening fails # arms pin what each principal must NOT reach, so a later widening fails
# here instead of being noticed in a store. # here instead of being noticed in a store.
{ {
name = "the queue-credential reader's grant is one hive's queue credential and nothing else"; # A hive holds no shared queue credential for its agents: no unit writes
# a role or policy for one, none reads one, and no granting unit names
# its path. The control: the same scan finds a reader's grant.
name = "no unit grants, reads or writes a hive-shared agent queue credential";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-queue-agent-policy.script; scripts = lib.mapAttrsToList (_: u: u.script or "") baoGrantWithConsumers.systemd.services;
in in
lib.hasInfix "path \"secret/data/swarm/hives/h1/queue/agent\" {" s !(baoGrantWithConsumers.systemd.services ? swarm-bao-queue-agent-policy)
&& lib.hasInfix "capabilities = [\"read\"]" s && !(baoGrantWithConsumers.systemd.services ? swarm-bao-queue-agent)
&& !(lib.hasInfix "secret/data/swarm/agents" s) && !(lib.any (lib.hasInfix "/queue/agent") scripts)
&& !(lib.hasInfix "secret/data/swarm/services" s) && lib.any (lib.hasInfix "/oidc/client") scripts;
&& !(lib.hasInfix "secret/data/swarm/hives/h1/*" s)
&& !(lib.hasInfix "secret/data/swarm/hives/h1/matrix" s)
&& !(lib.hasInfix "secret/data/swarm/hives/*" s)
&& !(lib.hasInfix "sys/policies/acl" s);
} }
{ {
# ⚠️ The client id is the path segment, so the negative arm that matters # ⚠️ The client id is the path segment, so the negative arm that matters
@ -856,11 +848,7 @@ let
# through a token that carries it, and a token is minted by a cert-auth # through a token that carries it, and a token is minted by a cert-auth
# role matching a CN. Distinct subjects is the whole mechanism — one # role matching a CN. Distinct subjects is the whole mechanism — one
# subject for every reader is one principal however the policies read. # subject for every reader is one principal however the policies read.
# name = "each of the three readers logs in under a subject of its own";
# The per-hive subject carries the hive name because its path does; the
# service subjects do not, because an OIDC client is registered once
# per swarm. Pinned so neither shape is tidied into the other.
name = "each of the four readers logs in under a subject of its own";
ok = ok =
let let
subjectOf = subjectOf =
@ -875,8 +863,7 @@ let
# the reason the three service principals above state. # the reason the three service principals above state.
&& !(lib.hasInfix "auth/cert/certs/hive-" s); && !(lib.hasInfix "auth/cert/certs/hive-" s);
in in
subjectOf "swarm-bao-queue-agent-policy" "swarm-queue-agent-h1" "swarm-bao-queue-agent-h1" subjectOf "swarm-bao-grafana-oidc-policy" "swarm-grafana-oidc" "swarm-bao-grafana-oidc"
&& subjectOf "swarm-bao-grafana-oidc-policy" "swarm-grafana-oidc" "swarm-bao-grafana-oidc"
&& subjectOf "swarm-bao-otel-oidc-policy" "swarm-otel-oidc" "swarm-bao-otel-oidc" && subjectOf "swarm-bao-otel-oidc-policy" "swarm-otel-oidc" "swarm-bao-otel-oidc"
&& subjectOf "swarm-bao-forwarder-oidc-policy" "swarm-forwarder-oidc" "swarm-bao-forwarder-oidc"; && subjectOf "swarm-bao-forwarder-oidc-policy" "swarm-forwarder-oidc" "swarm-bao-forwarder-oidc";
} }
@ -889,7 +876,7 @@ let
# #
# Each pair is asserted whole: a certificate with no key authenticates # Each pair is asserted whole: a certificate with no key authenticates
# nothing, so a half-set pair is a reader that does not render. # nothing, so a half-set pair is a reader that does not render.
name = "each of the four readers presents its own leaf, never the hive's"; name = "each of the three readers presents its own leaf, never the hive's";
ok = ok =
let let
b = baoGrantWithConsumers.services.hyperhive.deploy.bao; b = baoGrantWithConsumers.services.hyperhive.deploy.bao;
@ -898,8 +885,6 @@ let
b.clientKeyFile b.clientKeyFile
]; ];
own = [ own = [
b.queueAgentClientCertFile
b.queueAgentClientKeyFile
b.grafanaOidcClientCertFile b.grafanaOidcClientCertFile
b.grafanaOidcClientKeyFile b.grafanaOidcClientKeyFile
b.otelOidcClientCertFile b.otelOidcClientCertFile
@ -915,7 +900,6 @@ let
lib.all (p: p != null) own lib.all (p: p != null) own
&& !(lib.any (p: lib.elem p hiveLeaf) own) && !(lib.any (p: lib.elem p hiveLeaf) own)
&& lib.length (lib.unique own) == lib.length own && lib.length (lib.unique own) == lib.length own
&& presents "swarm-bao-queue-agent" b.queueAgentClientCertFile b.queueAgentClientKeyFile
&& presents "swarm-bao-grafana-oidc" b.grafanaOidcClientCertFile b.grafanaOidcClientKeyFile && presents "swarm-bao-grafana-oidc" b.grafanaOidcClientCertFile b.grafanaOidcClientKeyFile
&& presents "swarm-bao-otel-oidc" b.otelOidcClientCertFile b.otelOidcClientKeyFile && presents "swarm-bao-otel-oidc" b.otelOidcClientCertFile b.otelOidcClientKeyFile
&& presents "swarm-bao-forwarder-oidc" b.forwarderOidcClientCertFile b.forwarderOidcClientKeyFile; && presents "swarm-bao-forwarder-oidc" b.forwarderOidcClientCertFile b.forwarderOidcClientKeyFile;
@ -923,9 +907,7 @@ let
{ {
# The minting side of the same claim. A role matching a subject nothing # The minting side of the same claim. A role matching a subject nothing
# signs is a reader that cannot log in, so the leaves and the roles have # signs is a reader that cannot log in, so the leaves and the roles have
# to be asserted against each other — and the per-hive leaf carries # to be asserted against each other.
# THIS host's hive name, which is what makes one hive's leaf useless
# against another hive's role.
name = "the PKI unit signs a leaf per reader, each under that reader's own subject"; name = "the PKI unit signs a leaf per reader, each under that reader's own subject";
ok = ok =
let let
@ -936,8 +918,6 @@ let
# literal spanning both would pin this file's line wrapping rather than # literal spanning both would pin this file's line wrapping rather than
# the pairing it means to. # the pairing it means to.
lib.all (lib.flip lib.hasInfix s) [ lib.all (lib.flip lib.hasInfix s) [
"/queue-agent.pem ]"
"swarm-bao-queue-agent-h1 \"\" clientAuth"
"/grafana-oidc.pem ]" "/grafana-oidc.pem ]"
"swarm-bao-grafana-oidc \"\" clientAuth" "swarm-bao-grafana-oidc \"\" clientAuth"
"/otel-oidc.pem ]" "/otel-oidc.pem ]"
@ -965,11 +945,10 @@ let
# client certificate and the host is the side that has one, so a unit # client certificate and the host is the side that has one, so a unit
# rendered inside the store's container would have neither an identity # rendered inside the store's container would have neither an identity
# nor a route. Plus the ordering that makes the mounts exist first. # nor a route. Plus the ordering that makes the mounts exist first.
name = "the four readers' granting units are ordered after the mounts and rendered on the host"; name = "the three readers' granting units are ordered after the mounts and rendered on the host";
ok = ok =
let let
units = [ units = [
"swarm-bao-queue-agent-policy"
"swarm-bao-grafana-oidc-policy" "swarm-bao-grafana-oidc-policy"
"swarm-bao-otel-oidc-policy" "swarm-bao-otel-oidc-policy"
"swarm-bao-forwarder-oidc-policy" "swarm-bao-forwarder-oidc-policy"
@ -990,7 +969,7 @@ let
# The forwarder is listed apart from `policyReaders`: it renders wherever # The forwarder is listed apart from `policyReaders`: it renders wherever
# the store does, so it is never absent on a store host and never present # the store does, so it is never absent on a store host and never present
# on a remote one, and the two cases below would fail on it for that. # on a remote one, and the two cases below would fail on it for that.
name = "each of the four readers is ordered after the unit writing its role"; name = "each of the three readers is ordered after the unit writing its role";
ok = ok =
let let
s = baoGrantWithConsumers.systemd.services; s = baoGrantWithConsumers.systemd.services;
@ -1483,9 +1462,9 @@ let
} }
{ {
# The other hives' side: every hive in the directory but this one gets # The other hives' side: every hive in the directory but this one gets
# the same three leaves, under its own name, in a directory of its own # the same two leaves, under its own name, in a directory of its own
# that the operator copies to its host. # that the operator copies to its host.
name = "the PKI unit signs every other hive's reader, queue-agent and gateway-leaf identities"; name = "the PKI unit signs every other hive's reader and gateway-leaf identities";
ok = ok =
let let
s = baoGrantTwoHives.systemd.services.swarm-bao-pki.script; s = baoGrantTwoHives.systemd.services.swarm-bao-pki.script;
@ -1494,7 +1473,6 @@ let
file: cn: lib.hasInfix "/hives/h2/${file}.pem ]" s && lib.hasInfix " ${cn} \"\" clientAuth" s; file: cn: lib.hasInfix "/hives/h2/${file}.pem ]" s && lib.hasInfix " ${cn} \"\" clientAuth" s;
in in
signs "client" "h2" signs "client" "h2"
&& signs "queue-agent" "${b.queueAgentCommonNamePrefix}-h2"
&& signs "hive-leaf" "${b.hiveLeafCommonNamePrefix}-h2" && signs "hive-leaf" "${b.hiveLeafCommonNamePrefix}-h2"
# This hive's own leaves stay at the top of the directory. # This hive's own leaves stay at the top of the directory.
&& !(lib.hasInfix "/hives/h1/" s); && !(lib.hasInfix "/hives/h1/" s);

View file

@ -31,18 +31,10 @@ let
# nothing here mints a leaf and the operator names one placed by hand. The # nothing here mints a leaf and the operator names one placed by hand. The
# deployment this pairing exists to serve, and the one that was previously # deployment this pairing exists to serve, and the one that was previously
# inexpressible — the gate asked whether the store was a neighbour. # inexpressible — the gate asked whether the store was a neighbour.
#
# 🩸 One leaf PER READER, and this fixture is where that cost is visible: the
# queue-credential unit identifies itself to the store separately from the
# hive, so an operator placing leaves by hand places one for it as well as
# the hive's own. Naming only `clientCertFile` here would leave the unit
# unrendered — which is what the arms below would then be asserting about.
baoRemoteReader = hiveWithController { baoRemoteReader = hiveWithController {
deploy.matrix.enable = true; deploy.matrix.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.queueAgentClientCertFile = "/etc/pki/bao-queue-agent.pem";
deploy.bao.queueAgentClientKeyFile = "/etc/pki/bao-queue-agent-key.pem";
}; };
# A homeserver on a hive with NO store identity at all — neither a local # A homeserver on a hive with NO store identity at all — neither a local
@ -51,24 +43,15 @@ let
# fixture set its own, as ./lib.nix asks. # fixture set its own, as ./lib.nix asks.
inherit (shared) matrixNoBaoIdentity; inherit (shared) matrixNoBaoIdentity;
# 🩸 The hand-configured remote reader that named the hive's leaf and stopped. # 🩸 The hand-configured remote reader that named the hive's leaf and stopped:
# `baoRemoteReader` above is the same deployment done right; this one holds # it holds the hive's own leaf, so it demonstrably reads the store, and is
# the hive's own leaf, so it demonstrably reads the store, and is missing the # missing matrix-ctl's pair.
# queue reader's pair. Without the refusal it would render no queue reader
# and say nothing — which is what the refusal arms below are about.
baoRemoteReaderMissingLeaves = hiveWithController { baoRemoteReaderMissingLeaves = hiveWithController {
deploy.matrix.enable = true; deploy.matrix.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
}; };
# The same omission on a hive that does NOT run a homeserver: the queue
# refusal checks only the hive's own toggle.
remoteReaderNoMatrixMissingLeaves = hiveWithController {
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# The store's host with the hive's own reader pointed somewhere else. # The store's host with the hive's own reader pointed somewhere else.
matrixOnStoreHostReaderMoved = hiveWithController { matrixOnStoreHostReaderMoved = hiveWithController {
deploy.bao.enable = true; deploy.bao.enable = true;
@ -77,122 +60,15 @@ let
deploy.bao.clientKeyFile = "/etc/pki/h1-key.pem"; deploy.bao.clientKeyFile = "/etc/pki/h1-key.pem";
}; };
# Did a module refuse this host, and over which option. An assertion is a
# config VALUE until something forces it — `.config` never throws — so a
# fixture in a state the module refuses stays evaluable and the refusal reads
# back as data. Matched on the option name the message names rather than on
# its prose, because the option name is the part an operator has to act on
# and a message that stopped naming it would be the actual defect. Same shape
# as ./grafana.nix's `grafanaRefusedFor`, which reads `swarm-grafana.nix`'s
# own refusal.
refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions;
cases = [ cases = [
{ {
# A login failure is the store being unreachable, sealed, or not yet # A hive holds no shared queue credential for its agents, wherever the
# holding this host's role — all of which a retry fixes. A read that # store runs. The control: hive-c0re renders on both hosts.
# answers "nothing there" is not, so only the first is allowed to fail name = "no hive renders a reader for a hive-shared agent queue credential";
# the unit. ok = lib.all (m: !(m.systemd.services ? swarm-bao-queue-agent) && m.systemd.services ? hive-c0re) [
name = "the queue credential reader retries a failed login and still degrades on an empty read"; baoWithMatrix
ok = baoRemoteReader
let ];
u = baoWithMatrix.systemd.services.swarm-bao-queue-agent;
# Everything between the login's failure branch and the read's, which
# is where the exit that decides "retry or give up" lives.
afterLogin = lib.last (lib.splitString "bao login" u.script);
loginBranch = lib.head (lib.splitString "bao kv get" afterLogin);
in
u.serviceConfig.Restart or null == "on-failure"
&& u.startLimitBurst or 0 > 0
# The window has to outlast every attempt, or the burst is unreachable.
&& u.startLimitIntervalSec or 0 > (u.serviceConfig.RestartSec or 0) * (u.startLimitBurst or 0)
&& lib.hasInfix "exit 1" loginBranch
&& lib.hasInfix "exit 0" (lib.last (lib.splitString "bao kv get" u.script));
}
{
# The reader's own grant covers `swarm/hives/<this hive>/*`; a path
# outside it answers 403, not "no such key". This is the path
# `swarm_secret_client::queue::agent_client_path` builds from the same
# pieces. The negative arm is the rename this one is exposed to: a
# credential named for the queue rather than for the hive that presents
# it reads as correct and is refused on every boot.
name = "the agent queue credential path sits inside the prefix the reader is granted";
ok =
let
s = baoRemoteReader.systemd.services.swarm-bao-queue-agent.script;
in
lib.hasInfix "secret/swarm/hives/h1/queue/agent" s && !(lib.hasInfix "secret/swarm/queue/" s);
}
{
# The unit's output is the option's value, not a literal that agrees with
# it today: an operator moving the directory has to move both files. The
# prefix is asserted too because `hasInfix ""` is true — an option
# renamed out from under this arm would otherwise read empty and pass.
name = "the queue credential reader writes both files under the directory its option names";
ok =
let
m = baoRemoteReader;
dir = toString m.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
s = m.systemd.services.swarm-bao-queue-agent.script;
in
lib.hasPrefix "/var/lib/" dir
&& lib.hasInfix "${dir}/secret" s
&& lib.hasInfix "${dir}/client_id" s;
}
{
# No agent container may render before this unit has had its attempts,
# and the edge that guarantees it must delay hive-c0re rather than sink
# it: an unreachable store is this unit's `Restart=on-failure` window,
# not a reason for the daemon that renders every agent to fail its own
# start.
name = "the queue credential reader orders before hive-c0re and is wanted, not required, by it";
ok =
let
u = baoRemoteReader.systemd.services.swarm-bao-queue-agent;
in
builtins.elem "hive-c0re.service" (u.before or [ ])
&& builtins.elem "hive-c0re.service" (u.wantedBy or [ ])
&& !(builtins.elem "hive-c0re.service" (u.requiredBy or [ ]))
&& !(builtins.elem "hive-c0re.service" (u.requires or [ ]));
}
{
# Nothing local mints certificates off-host, so this orders against
# nothing there.
name = "an off-host reader names no unit the store's host would have provided";
# Membership first, then the value: indexing a missing unit throws, and a
# table that reports which property broke must not be the thing that dies.
ok =
let
s = baoRemoteReader.systemd.services;
u = s.swarm-bao-queue-agent;
in
s ? swarm-bao-queue-agent && !(lib.elem "swarm-bao-pki.service" (u.after ++ u.wants ++ u.requires));
}
{
# Presence control for the case above: the list is conditional, not gone.
# Wanted, never required: a refused pki start fails a requirer's job with
# `dependency`, which its `Restart=on-failure` never retries.
name = "a co-located reader still orders after and wants the local pki unit, and does not require it";
ok =
let
s = baoWithMatrix.systemd.services;
u = s.swarm-bao-queue-agent;
in
s ? swarm-bao-queue-agent
&& lib.elem "swarm-bao-pki.service" u.after
&& lib.elem "swarm-bao-pki.service" u.wants
&& u.requires == [ ];
}
{
# Nothing asserted this script before, which is how it kept a branch that
# named three states and threw away the only thing telling them apart. A
# missing value, a refused identity and an unreachable host all end in the
# same degraded mode here, correctly — what must survive is which one.
name = "the queue credential reader carries the store's own diagnostic into the journal";
ok =
let
s = baoWithMatrix.systemd.services.swarm-bao-queue-agent.script;
in
!(lib.hasInfix "2>/dev/null" s) && lib.hasInfix ''cat "''$err"'' s;
} }
{ {
# hive-c0re runs as hive-core and the client key is `0600` root-owned # hive-c0re runs as hive-core and the client key is `0600` root-owned
@ -404,46 +280,6 @@ let
&& !(c.systemd.services ? swarm-matrix-appservice-publish) && !(c.systemd.services ? swarm-matrix-appservice-publish)
&& !(lib.any (lib.hasPrefix "swarm-appservice.yaml") c.systemd.services.tuwunel.serviceConfig.LoadCredential); && !(lib.any (lib.hasPrefix "swarm-appservice.yaml") c.systemd.services.tuwunel.serviceConfig.LoadCredential);
} }
{
# 🩸 The arm the whole refusal exists for. The reader is gated on its own
# leaf, so the way this regresses is the build going green and the unit
# not rendering — which no presence check on a rendered unit can see,
# because the unit that is missing is the evidence. Read as a refusal
# naming each option, so an operator acts on the message without opening
# the nix.
name = "a remote reader missing the queue reader's leaf is refused, naming both options";
ok =
refusedOver baoRemoteReaderMissingLeaves "queueAgentClientCertFile"
&& refusedOver baoRemoteReaderMissingLeaves "queueAgentClientKeyFile";
}
{
name = "the queue refusal needs no homeserver";
ok = refusedOver remoteReaderNoMatrixMissingLeaves "queueAgentClientCertFile";
}
{
# ⚠️ The arm that keeps the refusal from being worse than the silence it
# replaced. A hive with NO store identity is the supported no-store
# deployment and also the state an operator passes through bringing a hive
# up — neither may fail to evaluate. Asserted as "no refusal names either
# option", not as "this one fixture is fine", because the way this breaks
# is a gate widened to the principal's leaf alone.
name = "a hive with no store identity at all is refused over none of the per-principal leaves";
ok = lib.all (option: !(refusedOver matrixNoBaoIdentity option)) [
"queueAgentClientCertFile"
"queueAgentClientKeyFile"
];
}
{
# The other half of the same guard, and the one an operator meets far more
# often: on the store's own host ./host-modules/glue-bao-tls.nix mkDefaults
# every pair, so there is nothing to name and nothing to refuse. Paired with
# the fully-named remote reader, which is the same deployment done by hand.
name = "neither a store host nor a correctly-named remote reader is refused";
ok = lib.all (m: !(refusedOver m "queueAgentClientCertFile")) [
baoWithMatrix
baoRemoteReader
];
}
]; ];
in in
runGroup "bao-matrix-reader" cases runGroup "bao-matrix-reader" cases

View file

@ -216,13 +216,17 @@ let
&& env shared.controllerAlone == { }; && env shared.controllerAlone == { };
} }
{ {
# Where the reader puts the files and where the daemon looks for them is # An agent connects to the queue with its own secret, so the hive hands
# one agreement spanning two modules. Asserted against the option rather # its agents no shared credential and no token endpoint to mint one at.
# than the literal so moving the directory moves both ends. # The control: the same environment carries the agents' queue address.
name = "hive-c0re is told where the agents' queue credential lands"; name = "hive-c0re forwards no shared agent queue credential";
ok = ok =
allLocal.systemd.services.hive-c0re.environment.HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR let
== toString allLocal.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir; e = allLocal.systemd.services.hive-c0re.environment;
in
!(e ? HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR)
&& !(e ? HIVE_AGENT_OIDC_TOKEN_ENDPOINT)
&& e ? HIVE_AGENT_NATS_URL;
} }
{ {
# Inside an agent container loopback is the agent itself, so the agents' # Inside an agent container loopback is the agent itself, so the agents'
@ -256,28 +260,10 @@ let
&& lib.elem "/hive-collector.hive/${bridge}" allLocal.services.dnsmasq.settings.address; && lib.elem "/hive-collector.hive/${bridge}" allLocal.services.dnsmasq.settings.address;
} }
{ {
# The agents mint against the swarm's IdP, the same endpoint the hive's # The absence arm, and what makes the agents' queue address cases above able
# own client uses — a hive-local guess would produce a token the queue # to fail.
# would not accept. name = "a hive with no agent queue address forwards none";
name = "the agents' token endpoint is the swarm IdP's"; ok = !(noAgentQueue.systemd.services.hive-c0re.environment ? HIVE_AGENT_NATS_URL);
ok =
let
e = allLocal.systemd.services.hive-c0re.environment;
in
lib.hasSuffix "/api/oidc/token" e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT
&& e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT == e.HIVE_C0RE_OIDC_TOKEN_ENDPOINT;
}
{
# The absence arm, and what makes the two above able to fail: a hive
# with no queue address for its agents must forward neither coordinate,
# because half a pair reaches the harness as a partial configuration
# rather than as none.
name = "a hive with no agent queue address forwards no agent queue coordinates";
ok =
let
e = noAgentQueue.systemd.services.hive-c0re.environment;
in
!(e ? HIVE_AGENT_NATS_URL) && !(e ? HIVE_AGENT_OIDC_TOKEN_ENDPOINT);
} }
{ {
# Control: `bare` sets none of them, and its serve.json carries the # Control: `bare` sets none of them, and its serve.json carries the

View file

@ -37,13 +37,6 @@ let
# homeserver across the network. # homeserver across the network.
matrixElsewhere = shared.bareWithController; matrixElsewhere = shared.bareWithController;
cases = [ cases = [
{
# Absence arm, and what makes the one above able to fail: with no leaf
# this unit would fail a TLS handshake on every boot, so it must not
# exist at all rather than retry its way through the start limit.
name = "a hive with no store identity renders no queue credential reader";
ok = !(matrixNoBaoIdentity.systemd.services ? swarm-bao-queue-agent);
}
{ {
# Two limits governed a matrix upload and nothing kept them in agreement, # Two limits governed a matrix upload and nothing kept them in agreement,
# so raising the documented one past the gateway's hardcoded cap changed # so raising the documented one past the gateway's hardcoded cap changed

View file

@ -43,14 +43,12 @@ let
# leaves, not a whole prefix, so a hive that inherited a dead entry there # leaves, not a whole prefix, so a hive that inherited a dead entry there
# would silently gain the ability to overwrite the swarm's matrix credential. # would silently gain the ability to overwrite the swarm's matrix credential.
# #
# 🩸 The two per-hive subjects have a different shape: those roles are # 🩸 The per-hive subject has a different shape: that role is written PER
# written PER HIVE, so the subject a hive must not be is # HIVE, so the subject a hive must not be is `<prefix>-<some hive's name>`
# `<prefix>-<some hive's name>` rather than the prefix itself. Reserving only # rather than the prefix itself. Reserving only the prefix would leave the
# the prefix would leave the composed spelling free, and a hive taking it # composed spelling free, and a hive taking it would present a leaf the other
# would present a leaf the other hive's role accepts — a hive reading another # hive's role accepts. The collision is with the OTHER hive's role, hence
# hive's queue credential, the exact widening the split exists to avoid. The # `other` beside `hl-other`.
# collision is with the OTHER hive's role, hence `other` beside `qr-other`
# and `hl-other`.
hiveNamedAfterEverySubject = hive { hiveNamedAfterEverySubject = hive {
deploy.swarm-otel.enable = false; deploy.swarm-otel.enable = false;
deploy.bao = { deploy.bao = {
@ -62,7 +60,6 @@ let
forwarderOidcCommonName = "fwctl"; forwarderOidcCommonName = "fwctl";
forgePublisherCommonName = "fpctl"; forgePublisherCommonName = "fpctl";
granterCommonName = "grctl"; granterCommonName = "grctl";
queueAgentCommonNamePrefix = "qr";
hiveLeafCommonNamePrefix = "hl"; hiveLeafCommonNamePrefix = "hl";
}; };
swarm.hives = { swarm.hives = {
@ -75,7 +72,6 @@ let
fpctl.domain = "fp.t.local"; fpctl.domain = "fp.t.local";
grctl.domain = "gr.t.local"; grctl.domain = "gr.t.local";
other.domain = "other.t.local"; other.domain = "other.t.local";
qr-other.domain = "q.t.local";
hl-other.domain = "hl.t.local"; hl-other.domain = "hl.t.local";
}; };
}; };
@ -216,16 +212,9 @@ let
{ {
# 🩸 The per-hive half, and the one a prefix-only reservation would miss: # 🩸 The per-hive half, and the one a prefix-only reservation would miss:
# the role is `<prefix>-<hive>`, so the reserved string has to be composed # the role is `<prefix>-<hive>`, so the reserved string has to be composed
# against every declared hive. Here hive `qr-other` collides with the role # against every declared hive. A hive taking this spelling would present
# written for hive `other` — a leaf that reads a credential belonging to a # a leaf hive `other`'s gateway-leaf role accepts, and could ask the store
# hive that is not it. # for that hive's certificate. The control is the legal roster below.
name = "a hive named after another hive's per-hive reader subject is refused";
ok = equalityGuardRefuses hiveNamedAfterEverySubject "qr-other";
}
{
# A hive taking this spelling would present a leaf hive `other`'s
# gateway-leaf role accepts, and could ask the store for that hive's
# certificate. The control is the legal roster below.
name = "a hive named after another hive's gateway-leaf subject is refused"; name = "a hive named after another hive's gateway-leaf subject is refused";
ok = equalityGuardRefuses hiveNamedAfterEverySubject "hl-other"; ok = equalityGuardRefuses hiveNamedAfterEverySubject "hl-other";
} }

View file

@ -77,13 +77,10 @@ let
# The same hive given its own store leaf by hand, which is what turns # The same hive given its own store leaf by hand, which is what turns
# publishing on away from the IdP's host: hive-c0re reads the status secret # publishing on away from the IdP's host: hive-c0re reads the status secret
# from the store under it. The queue-credential reader's pair comes with it, # from the store under it.
# because a hive that reads the store is refused at eval without one.
hiveLeaf = { hiveLeaf = {
deploy.bao.clientCertFile = "/etc/pki/h1.pem"; deploy.bao.clientCertFile = "/etc/pki/h1.pem";
deploy.bao.clientKeyFile = "/etc/pki/h1-key.pem"; deploy.bao.clientKeyFile = "/etc/pki/h1-key.pem";
deploy.bao.queueAgentClientCertFile = "/etc/pki/h1-queue-agent.pem";
deploy.bao.queueAgentClientKeyFile = "/etc/pki/h1-queue-agent-key.pem";
}; };
remotePublishing = hiveWithController (lib.recursiveUpdate remoteSecrets hiveLeaf); remotePublishing = hiveWithController (lib.recursiveUpdate remoteSecrets hiveLeaf);

View file

@ -80,7 +80,6 @@ let
# Each hive's hive-c0re reads this exact path # Each hive's hive-c0re reads this exact path
# (`swarm_secret_client::queue::hive_client_path`, pinned by that crate's # (`swarm_secret_client::queue::hive_client_path`, pinned by that crate's
# own test), for every hive in the roster, not only the one this host is. # own test), for every hive in the roster, not only the one this host is.
# The agents' client beside it stays a separate path.
name = "the publisher writes every hive's own client under that hive's prefix"; name = "the publisher writes every hive's own client under that hive's prefix";
ok = ok =
let let
@ -91,8 +90,7 @@ let
[ [
"h1" "h1"
"h2" "h2"
] ];
&& lib.hasInfix "secret/swarm/hives/h1/queue/agent" s;
} }
{ {
# The controller reads this exact path # The controller reads this exact path
@ -333,17 +331,17 @@ let
lib.hasInfix "value=@" code && !(lib.hasInfix "$(cat" code); lib.hasInfix "value=@" code && !(lib.hasInfix "$(cat" code);
} }
{ {
# Two ends of a wire nothing at eval time carries end to end: this is the # No hive holds a shared queue credential for its agents, so authelia
# path `swarm_secret_client::queue` resolves for the reader. Both hives # mints no agent client and the publisher writes no such path. The
# are asserted, so a publisher that knew one name rather than the roster # control: the same script writes each hive's own client.
# fails here rather than on the second hive ever added to a swarm. name = "the publisher writes no hive-shared agent queue credential";
name = "the publisher writes every hive in the roster to that hive's own queue path";
ok = ok =
let let
s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script; s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script;
in in
lib.hasInfix "secret/swarm/hives/h1/queue/agent" s !(lib.hasInfix "/queue/agent" s)
&& lib.hasInfix "secret/swarm/hives/h2/queue/agent" s; && !(lib.hasInfix "-agent.secret" s)
&& lib.hasInfix "secret/swarm/hives/h1/oidc/client" s;
} }
{ {
# A property of the SET, not of one unit: both of these authenticate by # A property of the SET, not of one unit: both of these authenticate by
@ -373,7 +371,7 @@ let
&& !(lib.hasInfix "bao kv" (lib.head (lib.splitString "bao login" c))); && !(lib.hasInfix "bao kv" (lib.head (lib.splitString "bao login" c)));
in in
holdsTokenFirst (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script holdsTokenFirst (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script
&& holdsTokenFirst baoWithMatrix.systemd.services.swarm-bao-queue-agent.script && holdsTokenFirst baoWithMatrix.systemd.services.swarm-bao-matrix-oidc.script
# Controls, so a clean verdict above means something. In order: a bare # Controls, so a clean verdict above means something. In order: a bare
# read is refused, a read placed before the login is refused, and a # read is refused, a read placed before the login is refused, and a
# login that exists only in a comment is refused — that last one is the # login that exists only in a comment is refused — that last one is the

View file

@ -95,7 +95,7 @@ pub fn hive_localpart(hive: &str) -> String {
/// The path holding `hive`'s matrix sender account's homeserver access token. /// The path holding `hive`'s matrix sender account's homeserver access token.
/// ///
/// Keyed per **hive**, like [`crate::queue::agent_client_path`]. Under /// Keyed per **hive**, like [`crate::queue::hive_client_path`]. Under
/// [`Kind::Hive`] the grant that reaches it is the hive's **own** stanza, /// [`Kind::Hive`] the grant that reaches it is the hive's **own** stanza,
/// `secret/data/swarm/hives/<this hive>/*`, which interpolates the name: hive /// `secret/data/swarm/hives/<this hive>/*`, which interpolates the name: hive
/// `alpha` reads its own token and gets a 403 on `beta`'s. /// `alpha` reads its own token and gets a 403 on `beta`'s.
@ -137,7 +137,7 @@ pub fn swarm_appservice_token_path() -> Result<String, Error> {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct Credential { pub struct Credential {
/// Named `value` like [`crate::forge::Credential::value`] and /// Named `value` like [`crate::forge::Credential::value`] and
/// [`crate::queue::Credential::value`], so a nix-side reader spells /// [`crate::queue::AgentCredential::value`], so a nix-side reader spells
/// `bao kv get -field=value` for every kind. /// `bao kv get -field=value` for every kind.
pub value: String, pub value: String,

View file

@ -40,10 +40,9 @@ pub fn identity_path(agent: &str) -> Result<String, Error> {
/// was issued from. /// was issued from.
/// ///
/// All three, because a reader has to reconstruct a usable identity from the /// All three, because a reader has to reconstruct a usable identity from the
/// store alone — the same requirement [`crate::queue::Credential`] states for /// store alone. The authority rides along so the cert-auth role and the
/// carrying its client id. The authority rides along so the cert-auth role /// certificate cannot be delivered from two different sources and silently
/// and the certificate cannot be delivered from two different sources and /// disagree; it is public material, unlike the other two fields.
/// silently disagree; it is public material, unlike the other two fields.
/// ///
/// ⚠️ **No `Debug` derive.** See the hand-written impl below: one of these /// ⚠️ **No `Debug` derive.** See the hand-written impl below: one of these
/// fields is a private key, and a derived `Debug` would put it in any log line /// fields is a private key, and a derived `Debug` would put it in any log line

View file

@ -11,7 +11,7 @@ use crate::Error;
/// The KV v2 mount every swarm secret lives under. /// The KV v2 mount every swarm secret lives under.
/// ///
/// A literal because the nix-side readers spell the same one, as in /// A literal because the nix-side readers spell the same one, as in
/// `glue-queue-agent-credential.nix`; an option nobody sets would be two ways /// `nix/agent-modules/bao.nix`; an option nobody sets would be two ways
/// to say one thing. /// to say one thing.
pub const MOUNT: &str = "secret"; pub const MOUNT: &str = "secret";

View file

@ -231,7 +231,7 @@ mod tests {
assert!( assert!(
stanza_paths(&p) stanza_paths(&p)
.iter() .iter()
.any(|g| covers(g, "swarm/hives/pr1ma/queue/agent")) .any(|g| covers(g, "swarm/hives/pr1ma/oidc/client"))
); );
} }

View file

@ -1,21 +1,10 @@
//! The queue agreements: where a queue credential lives in the store, and what //! The queue agreements: where a queue credential lives in the store, and what
//! the object at that path holds. //! the object at that path holds.
//! //!
//! **Two kinds live here, and which is which is the thing to get right.** //! An agent's credential, [`agent_queue_path`] / [`AgentCredential`], is keyed
//! //! per **agent**: a secret the swarm mints for one agent, at swarm level, with
//! - [`agent_client_path`] / [`Credential`] are keyed per **hive**: one OIDC //! no hive in the chain. It is what makes one agent distinguishable from its
//! client minted per hive at deploy time, shared by every agent container on //! co-hived neighbours.
//! it. It says which hive a caller belongs to, never which agent.
//! - [`agent_queue_path`] / [`AgentCredential`] are keyed per **agent**: a
//! secret the swarm mints for one agent, at swarm level, with no hive in the
//! chain. It is what makes one agent distinguishable from its co-hived
//! neighbours.
//!
//! They are separate objects rather than one with a nullable field because they
//! are minted by different principals on different events — the hive-scoped one
//! by authelia at deploy time, the per-agent one by `swarm-controller` when an
//! agent is created — and an object whose shape depends on who wrote it is a
//! reader that has to guess.
//! //!
//! The per-agent secret is deliberately **not** derived from the agent's mTLS //! The per-agent secret is deliberately **not** derived from the agent's mTLS
//! identity (the leaf at [`crate::mtls::identity_path`]). That leaf is for //! identity (the leaf at [`crate::mtls::identity_path`]). That leaf is for
@ -33,17 +22,6 @@ use crate::{
path::{Kind, principal_prefix}, path::{Kind, principal_prefix},
}; };
/// The path holding the client secret that agent containers on `hive` present
/// to the swarm queue.
///
/// # Errors
/// [`Error::PathSegment`] when `hive` contains anything but `[A-Za-z0-9_-]`,
/// which is what keeps one hive's name from addressing another hive's secret.
pub fn agent_client_path(hive: &str) -> Result<String, Error> {
let prefix = principal_prefix(Kind::Hive, hive)?;
Ok(format!("{prefix}/queue/agent"))
}
/// The path holding the secret **one agent** presents to the swarm queue. /// The path holding the secret **one agent** presents to the swarm queue.
/// ///
/// Under the agent's own principal prefix, and that is the whole reason for /// Under the agent's own principal prefix, and that is the whole reason for
@ -89,8 +67,7 @@ pub struct ControllerCredential {
/// The path holding `hive`'s own OIDC client secret: the `hive-<hive>` /// The path holding `hive`'s own OIDC client secret: the `hive-<hive>`
/// identity hive-c0re publishes the hive's status to the swarm queue with. /// identity hive-c0re publishes the hive's status to the swarm queue with.
/// ///
/// Not [`agent_client_path`], which is the agents' client on the same hive. /// Under the hive's own prefix, which the hive's policy reads
/// Both sit under the hive's own prefix, which the hive's policy reads
/// ([`crate::policy::render`]). /// ([`crate::policy::render`]).
/// ///
/// The nix half is `swarm-secret-publisher.nix`, which writes it. /// The nix half is `swarm-secret-publisher.nix`, which writes it.
@ -117,8 +94,8 @@ pub struct HiveClientCredential {
/// field still decode, because unknown fields are ignored. /// field still decode, because unknown fields are ignored.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct AgentCredential { pub struct AgentCredential {
/// The secret itself. Named to match [`Credential::value`] and /// The secret itself. Named to match [`crate::matrix::Credential::value`]
/// [`crate::matrix::Credential::value`] so a nix-side reader spells /// so a nix-side reader spells
/// `bao kv get -field=value` for every kind. /// `bao kv get -field=value` for every kind.
pub value: String, pub value: String,
@ -136,79 +113,10 @@ pub struct AgentCredential {
pub minted_at: Option<i64>, pub minted_at: Option<i64>,
} }
/// What the path holds: the client secret, plus the client id it belongs to.
///
/// The id rides with the secret for the same reason the homeserver rides with
/// a matrix token — a credential has to be reconstructable from the store
/// alone. Deriving it on the reading side instead would mean spelling
/// `hive-<name>-agent` in a second place, and the authelia module's own option
/// says what a split spelling costs: every agent is denied as a timeout.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct Credential {
/// The secret itself. Named to match [`crate::matrix::Credential::value`]
/// so a nix-side reader spells `bao kv get -field=value` for either kind.
pub value: String,
/// The OIDC client id the secret authenticates.
///
/// Required, unlike [`crate::matrix::Credential::homeserver`]: that field
/// is optional because it was added to objects already in the store, and
/// this path is new. An object without an id is not a usable credential,
/// so failing to decode is the wanted behaviour — the alternative is a
/// reader holding `None` whose only recourse is to spell the id itself.
pub client_id: String,
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
#[test]
fn a_hive_name_lands_under_its_own_principal_prefix() {
assert_eq!(
agent_client_path("alpha").expect("a plain name is legal"),
"swarm/hives/alpha/queue/agent"
);
}
#[test]
fn a_traversal_in_the_hive_name_is_refused() {
let e = agent_client_path("../beta").expect_err("a traversal is not");
assert!(matches!(e, Error::PathSegment { kind: "hive", .. }), "{e}");
}
#[test]
fn two_hives_never_share_a_path() {
assert_ne!(
agent_client_path("alpha").expect("legal"),
agent_client_path("beta").expect("legal")
);
}
#[test]
fn the_object_round_trips_through_the_store_representation() {
let c = Credential {
value: "s3cr3t".to_owned(),
client_id: "hive-alpha-agent".to_owned(),
};
let json = serde_json::to_string(&c).expect("serialises");
assert_eq!(
serde_json::from_str::<Credential>(&json).expect("deserialises"),
c
);
}
#[test]
fn the_field_names_the_nix_reader_asks_for_are_the_ones_written() {
let json = serde_json::to_value(Credential {
value: "s3cr3t".to_owned(),
client_id: "hive-alpha-agent".to_owned(),
})
.expect("serialises");
assert_eq!(json["value"], "s3cr3t");
assert_eq!(json["client_id"], "hive-alpha-agent");
}
/// Spelled out in full because `swarm-secret-publisher.nix` and /// Spelled out in full because `swarm-secret-publisher.nix` and
/// `swarm-bao.nix` write the same string by hand. /// `swarm-bao.nix` write the same string by hand.
#[test] #[test]
@ -231,9 +139,9 @@ mod tests {
let reads = |path: &str| granted.iter().any(|g| path.starts_with(g)); let reads = |path: &str| granted.iter().any(|g| path.starts_with(g));
let path = controller_client_path().expect("a constant segment"); let path = controller_client_path().expect("a constant segment");
assert!(!reads(&path), "{path} is readable under {granted:?}"); assert!(!reads(&path), "{path} is readable under {granted:?}");
// The control: the hive's own agent client IS under a hive stanza, so // The control: the hive's own client IS under a hive stanza, so the
// the assertion above can fail at all. // assertion above can fail at all.
let own = agent_client_path("alpha").expect("legal"); let own = hive_client_path("alpha").expect("legal");
assert!(reads(&own), "{own} should be readable under {granted:?}"); assert!(reads(&own), "{own} should be readable under {granted:?}");
} }
@ -381,29 +289,4 @@ mod tests {
serde_json::from_str::<AgentCredential>(r#"{"value":"s","hive":"alpha"}"#).is_err() serde_json::from_str::<AgentCredential>(r#"{"value":"s","hive":"alpha"}"#).is_err()
); );
} }
/// decodes as the other — the property that keeps a reader from picking up
/// a hive-shared credential where a per-agent one was meant.
#[test]
fn the_hive_credential_and_the_agent_credential_are_not_interchangeable() {
let hive_json = serde_json::to_string(&Credential {
value: "s".to_owned(),
client_id: "hive-alpha-agent".to_owned(),
})
.expect("serialises");
assert!(serde_json::from_str::<AgentCredential>(&hive_json).is_err());
let agent_json = serde_json::to_string(&AgentCredential {
value: "s".to_owned(),
agent: "atlas".to_owned(),
minted_at: None,
})
.expect("serialises");
assert!(serde_json::from_str::<Credential>(&agent_json).is_err());
assert_ne!(
agent_queue_path("atlas").expect("legal"),
agent_client_path("atlas").expect("legal"),
);
}
} }