drop the shared hive agent login
Every agent now connects to the queue and reads logs with its own secret from the store (#5181), so the per-hive `hive-<h>-agent` OIDC client has no reader left. Remove it end to end: - hive-c0re no longer forwards `hive-queue-agent-{secret,client-id}` into containers, and no longer forwards HIVE_AGENT_OIDC_TOKEN_ENDPOINT; the agent module that loaded them (nix/agent-modules/queue.nix) is gone, its options refused with a removal message. - The host reader (glue-queue-agent-credential.nix, swarm-bao-queue-agent), its store role and policy unit (swarm-bao-queue-agent-policy), its PKI leaf and the deploy.bao.queueAgent* / queue.agentCredentialDir options are gone; the options are refused with a removal message. - authelia mints no `hive-<h>-agent` client, and swarm-secret-publish no longer writes swarm/hives/<h>/queue/agent. - swarm-secret-client drops queue::agent_client_path / queue::Credential. - HIVE_AGENT_NATS_URL is now gated on the agents' queue address alone. The log store's authelia check stays for operators. swarm-nats-auth still refuses a `-agent` client id; that refusal and `agentClientSuffix` are left for a follow-up. Refs #4459 Refs #4765
This commit is contained in:
parent
d19bd83aa5
commit
868c6cf96c
41 changed files with 286 additions and 1410 deletions
|
|
@ -303,8 +303,7 @@ subjects it publishes to, the credential may answer the one request it was just
|
||||||
sent, which is how a command gets its reply. The harness and the subagent daemon
|
sent, which is how a command gets its reply. The harness and the subagent daemon
|
||||||
connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to the
|
connect with `_INBOX.agent.<agent>` as their inbox prefix, so the replies to the
|
||||||
icon write and the schedule read arrive. An agent can't read another agent's
|
icon write and the schedule read arrive. An agent can't read another agent's
|
||||||
terminal, state, input, commands or replies. The queue refuses the hive's shared
|
terminal, state, input, commands or replies.
|
||||||
agent client outright.
|
|
||||||
|
|
||||||
Hives and the swarm controller get a subscribe allow-list too, each with its
|
Hives and the swarm controller get a subscribe allow-list too, each with its
|
||||||
own reply inbox `_INBOX.client.<client-id>.>`, which `swarm_queue_client::connect`
|
own reply inbox `_INBOX.client.<client-id>.>`, which `swarm_queue_client::connect`
|
||||||
|
|
|
||||||
|
|
@ -121,12 +121,11 @@ them and names them in config. The service principals' leaves are at the top of
|
||||||
`/var/lib/swarm-bao-pki/`, named after the principal.
|
`/var/lib/swarm-bao-pki/`, named after the principal.
|
||||||
|
|
||||||
For every hive in `swarm.hives` other than its own, the store's host signs
|
For every hive in `swarm.hives` other than its own, the store's host signs
|
||||||
that hive's three leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`:
|
that hive's two leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`:
|
||||||
|
|
||||||
| file | subject | name it in |
|
| file | subject | name it in |
|
||||||
| ---------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------- |
|
| ------------------------------------ | ---------------------------------------------- | ------------------------------------------------------------------------------- |
|
||||||
| `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` |
|
| `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` |
|
||||||
| `queue-agent.pem`, `queue-agent-key.pem` | `<deploy.bao.queueAgentCommonNamePrefix>-<hive>` | `deploy.bao.queueAgentClientCertFile`, `.queueAgentClientKeyFile` |
|
|
||||||
| `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
|
| `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
|
||||||
|
|
||||||
Copy `/var/lib/swarm-bao-pki/ca.pem` with them and name it in
|
Copy `/var/lib/swarm-bao-pki/ca.pem` with them and name it in
|
||||||
|
|
|
||||||
|
|
@ -43,7 +43,6 @@ the rest of the cell says how.
|
||||||
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
|
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
|
||||||
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
|
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
|
||||||
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
|
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
|
||||||
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host | must be stated | must be stated |
|
|
||||||
| `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
|
| `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
|
||||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
|
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
|
||||||
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
|
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
|
||||||
|
|
|
||||||
|
|
@ -107,9 +107,8 @@ with nothing to authenticate with still receives every hive's telemetry —
|
||||||
its pushes to the stores go out unauthenticated and get refused there, and its
|
its pushes to the stores go out unauthenticated and get refused there, and its
|
||||||
scrape jobs for the published targets read a credential file nothing
|
scrape jobs for the published targets read a credential file nothing
|
||||||
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
|
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
|
||||||
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice
|
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
|
||||||
`glue-queue-agent-credential.nix` makes for its own reader, rather than the hard assertion `swarm-grafana.nix`
|
`swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
|
||||||
uses for its non-optional one. The operator can still deliver the secret by
|
|
||||||
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
|
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
|
||||||
collector this unit never reaches.
|
collector this unit never reaches.
|
||||||
|
|
||||||
|
|
@ -259,7 +258,6 @@ such as the root token.
|
||||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||||
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
|
|
||||||
|
|
||||||
The forge and the homeserver read their copies out of the store, the way
|
The forge and the homeserver read their copies out of the store, the way
|
||||||
Grafana does below: the forge under the forge publisher's leaf, the
|
Grafana does below: the forge under the forge publisher's leaf, the
|
||||||
|
|
@ -268,34 +266,8 @@ publisher in authelia's container has written the secret, rather than skipping.
|
||||||
A silent skip produces a service whose login button always fails, which is a
|
A silent skip produces a service whose login button always fails, which is a
|
||||||
symptom many layers from its cause.
|
symptom many layers from its cause.
|
||||||
|
|
||||||
The store's **first reader** is the agent containers' queue credential:
|
The store's **first reader** is Grafana's OIDC client secret, a _swarm
|
||||||
`glue-queue-agent-credential.nix` lands it as two files, the client secret and
|
service's_ own credential rather than a hive's. It has exactly
|
||||||
the client id it authenticates, because that's how a queue client reads them —
|
|
||||||
the secret by path, the id by value. No local fallback exists, and none is
|
|
||||||
possible, so absent files mean an agent built before `swarm-logs` read its own
|
|
||||||
queue secret has no working `swarm-logs`, which is what a swarm looks like
|
|
||||||
before the publisher in authelia's container has run.
|
|
||||||
The reader runs before `hive-c0re.service`, which wants it (doesn't require
|
|
||||||
it), so an agent container never renders ahead of the credential; an
|
|
||||||
unreachable store delays the daemon's start rather than failing it.
|
|
||||||
|
|
||||||
That credential still has one hop left, because the reader of it lives inside
|
|
||||||
an agent container. **It crosses as a systemd credential, not as a bind
|
|
||||||
mount**, and the mode decides that: the secret is `root:0600` and a harness
|
|
||||||
runs as its own unprivileged agent user, so a bind would deliver a file that
|
|
||||||
user can't open. `hive-c0re` stats the two files — the directory is `0755`, so
|
|
||||||
it needs no read access to either — and hands them to
|
|
||||||
`systemd-nspawn --load-credential` through `hive-priv`, which runs as root.
|
|
||||||
Inside, `hive-agent.service` names the same two ids in `LoadCredential=`, which
|
|
||||||
lands them in its own `$CREDENTIALS_DIRECTORY`, owned by the agent user and by
|
|
||||||
nobody else, where a `swarm-logs` built before it read the agent's own queue
|
|
||||||
secret reads them. The harness itself doesn't: it connects to the queue with the agent's own credential. `hive-c0re`
|
|
||||||
never reads the bytes at any point: it runs as `hive-core`, which is the reason
|
|
||||||
a copy wasn't an option either. When the files aren't there the daemon forwards
|
|
||||||
nothing and says so in its journal, and the container starts without them.
|
|
||||||
|
|
||||||
The **second reader** is Grafana's OIDC client secret, and it's the first one
|
|
||||||
that's a _swarm service's_ own credential rather than a hive's. It has exactly
|
|
||||||
one delivery route: `swarm-bao-grafana-oidc.service` reads
|
one delivery route: `swarm-bao-grafana-oidc.service` reads
|
||||||
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
|
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
|
||||||
that runs Grafana. Where authelia is doesn't enter into it — the publisher in
|
that runs Grafana. Where authelia is doesn't enter into it — the publisher in
|
||||||
|
|
@ -338,7 +310,7 @@ the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
|
||||||
the Grafana host, which retries, while Grafana stays down because its config
|
the Grafana host, which retries, while Grafana stays down because its config
|
||||||
reads that file.
|
reads that file.
|
||||||
|
|
||||||
The **third reader** is the swarm collector's own OIDC client secret, the
|
The **second reader** is the swarm collector's own OIDC client secret, the
|
||||||
same shape one level down: `swarm-bao-otel-oidc.service` reads
|
same shape one level down: `swarm-bao-otel-oidc.service` reads
|
||||||
`swarm/services/<client-id>/oidc/client` out of the store on every host that
|
`swarm/services/<client-id>/oidc/client` out of the store on every host that
|
||||||
runs the collector, and registration lives separately in
|
runs the collector, and registration lives separately in
|
||||||
|
|
@ -347,13 +319,12 @@ differs is what an absent credential means: the collector still receives
|
||||||
telemetry with none, so `swarm-otel.nix` doesn't assert
|
telemetry with none, so `swarm-otel.nix` doesn't assert
|
||||||
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
|
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
|
||||||
does — it
|
does — it
|
||||||
doesn't render the reading unit without them, the shape
|
doesn't render the reading unit without them.
|
||||||
`glue-queue-agent-credential.nix` uses for its own read.
|
|
||||||
|
|
||||||
The **fourth reader** is the secret store forwarder's OIDC client secret:
|
The **third reader** is the secret store forwarder's OIDC client secret:
|
||||||
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
|
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
|
||||||
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
|
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
|
||||||
third in both directions. It renders unconditionally, because the export it
|
second in both directions. It renders unconditionally, because the export it
|
||||||
authenticates has no unauthenticated mode to degrade into — and it orders
|
authenticates has no unauthenticated mode to degrade into — and it orders
|
||||||
itself **after** `container@swarm-bao`, because the store it reads runs in the
|
itself **after** `container@swarm-bao`, because the store it reads runs in the
|
||||||
container it delivers into. Nothing circular sits behind that: the identity it
|
container it delivers into. Nothing circular sits behind that: the identity it
|
||||||
|
|
@ -412,14 +383,13 @@ next controller restart.
|
||||||
### Per-principal identities
|
### Per-principal identities
|
||||||
|
|
||||||
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
||||||
identity and sharing one merges the identities. Each of these eleven leaves has
|
identity and sharing one merges the identities. Each of these ten leaves has
|
||||||
a role and a policy of its own, and reaches one path, except the review
|
a role and a policy of its own, and reaches one path, except the review
|
||||||
classifier's, the CI runner's, the builder's, the CI run poller's, the claude
|
classifier's, the CI runner's, the builder's, the CI run poller's, the claude
|
||||||
login minter's, the forge host's and Grafana's, which reach more than one:
|
login minter's, the forge host's and Grafana's, which reach more than one:
|
||||||
|
|
||||||
| unit | option pair under `deploy.bao.` | reads, or writes |
|
| unit | option pair under `deploy.bao.` | reads, or writes |
|
||||||
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
|
|
||||||
| `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin`, and Grafana's journal forwarder in its container | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client` and `swarm/services/<grafana forwarder client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` |
|
| `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin`, and Grafana's journal forwarder in its container | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client` and `swarm/services/<grafana forwarder client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` |
|
||||||
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
|
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
|
||||||
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
|
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
|
||||||
|
|
@ -431,12 +401,8 @@ login minter's, the forge host's and Grafana's, which reach more than one:
|
||||||
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
|
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
|
||||||
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
|
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
|
||||||
|
|
||||||
The first exists **per hive**, because the path it reads carries a hive name
|
Each reaches a path that names a swarm service rather than a hive, so one role
|
||||||
and every hive runs its own reader. Its subject is
|
each is enough and their subjects are the flat
|
||||||
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
|
|
||||||
composed spelling as a hive name, so nobody can name a hive into another hive's
|
|
||||||
role. The other nine reach a path that names a swarm service rather than a hive,
|
|
||||||
so one role each is enough and their subjects are the flat
|
|
||||||
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
||||||
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
|
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
|
||||||
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
|
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
|
||||||
|
|
@ -449,7 +415,7 @@ OIDC client secret it reads.
|
||||||
On a host that mints its own PKI, `glue-bao-tls.nix` signs all nine and defaults
|
On a host that mints its own PKI, `glue-bao-tls.nix` signs all nine and defaults
|
||||||
all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from
|
all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from
|
||||||
the store's host and name it here: a service principal's from the top of
|
the store's host and name it here: a service principal's from the top of
|
||||||
`/var/lib/swarm-bao-pki/`, another hive's queue-agent pair from
|
`/var/lib/swarm-bao-pki/`, another hive's leaves from
|
||||||
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
|
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
|
||||||
directory). That's one file per principal rather than one file
|
directory). That's one file per principal rather than one file
|
||||||
shared by nine, which is the whole of what this buys. `swarm-ci-runner.nix`
|
shared by nine, which is the whole of what this buys. `swarm-ci-runner.nix`
|
||||||
|
|
@ -461,7 +427,7 @@ naming both options, and the controller runs without forge access and the forge
|
||||||
without a login source until an operator places the leaf.
|
without a login source until an operator places the leaf.
|
||||||
|
|
||||||
On a host without the store, where the forwarder never runs, forgetting one of
|
On a host without the store, where the forwarder never runs, forgetting one of
|
||||||
the other nine isn't a quiet degrade. A hand-configured remote-store hive that
|
the others isn't a quiet degrade. A hand-configured remote-store hive that
|
||||||
names the hive's own `clientCertFile` and misses a principal's pair has the
|
names the hive's own `clientCertFile` and misses a principal's pair has the
|
||||||
build refused, by the module of the reader it missed, naming that pair — see
|
build refused, by the module of the reader it missed, naming that pair — see
|
||||||
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
|
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
|
||||||
|
|
@ -475,7 +441,7 @@ their readers hold a handle for longer than five minutes.
|
||||||
|
|
||||||
### What a missing leaf costs
|
### What a missing leaf costs
|
||||||
|
|
||||||
Each of the three modules asserts its own pair, on the same condition: **this host
|
Each of the four modules asserts its own pair, on the same condition: **this host
|
||||||
already reads the store** — it holds `deploy.bao.clientCertFile` and
|
already reads the store** — it holds `deploy.bao.clientCertFile` and
|
||||||
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
|
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
|
||||||
The refusal names the two options, so an operator acts on the message without
|
The refusal names the two options, so an operator acts on the message without
|
||||||
|
|
@ -492,9 +458,7 @@ that matters on its own:
|
||||||
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
|
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
|
||||||
defaults, so the third clause is false. Nothing to set, nothing to refuse.
|
defaults, so the third clause is false. Nothing to set, nothing to refuse.
|
||||||
- **A consumer that's off** — no collector, no Grafana — has no
|
- **A consumer that's off** — no collector, no Grafana — has no
|
||||||
unit to skip, so the second clause is false. Only the queue-credential reader
|
unit to skip, so the second clause is false.
|
||||||
has no toggle of its own to check: every hive runs one for its own agents, so
|
|
||||||
reading the store at all is what asks for it.
|
|
||||||
|
|
||||||
The collector keeps a separate degrade underneath this, unchanged: a host with no
|
The collector keeps a separate degrade underneath this, unchanged: a host with no
|
||||||
store identity runs a collector with no OIDC client secret and still receives
|
store identity runs a collector with no OIDC client secret and still receives
|
||||||
|
|
|
||||||
|
|
@ -225,7 +225,7 @@ Four consequences worth stating plainly:
|
||||||
|
|
||||||
Everything above is a person in a browser. A swarm also has callers that
|
Everything above is a person in a browser. A swarm also has callers that
|
||||||
hold no session and follow no redirect: the telemetry collector, the
|
hold no session and follow no redirect: the telemetry collector, the
|
||||||
queue's auth-callout responder, and each hive's own agents.
|
queue's auth-callout responder, and each hive.
|
||||||
|
|
||||||
**One client per hive, not one per service.** A hive's identity belongs to
|
**One client per hive, not one per service.** A hive's identity belongs to
|
||||||
the directory rather than to whichever service happens to consume it, so a
|
the directory rather than to whichever service happens to consume it, so a
|
||||||
|
|
|
||||||
|
|
@ -8,10 +8,9 @@
|
||||||
//! asked to authenticate in order to obtain the thing it authenticates with:
|
//! asked to authenticate in order to obtain the thing it authenticates with:
|
||||||
//! by the time it runs, its identity is already inside it.
|
//! by the time it runs, its identity is already inside it.
|
||||||
//!
|
//!
|
||||||
//! **A credential and not a bind mount**, the same answer and the same reason
|
//! **A credential and not a bind mount**: the files are `0600` to `hive-core`
|
||||||
//! as the queue secret beside it in [`super::host_config`]: the files are
|
//! and the units that use them run as the unprivileged agent user, so a bind
|
||||||
//! `0600` to `hive-core` and the units that use them run as the unprivileged
|
//! would deliver a private key that user cannot open.
|
||||||
//! agent user, so a bind would deliver a private key that user cannot open.
|
|
||||||
//! The container manager reads a `--load-credential` source as root and
|
//! The container manager reads a `--load-credential` source as root and
|
||||||
//! re-exposes it under the consuming unit's own `User=`.
|
//! re-exposes it under the consuming unit's own `User=`.
|
||||||
//!
|
//!
|
||||||
|
|
@ -60,8 +59,7 @@ const SERVER_CA_CREDENTIAL: &str = "hive-agent-bao-server-ca";
|
||||||
///
|
///
|
||||||
/// Already-staged files survive a failed collection. A store that is briefly
|
/// Already-staged files survive a failed collection. A store that is briefly
|
||||||
/// unreachable then delivers the identity from the last successful read rather
|
/// unreachable then delivers the identity from the last successful read rather
|
||||||
/// than none at all — the same call as the queue reader's, which leaves its
|
/// than none at all.
|
||||||
/// files alone when the store says nothing.
|
|
||||||
pub async fn stage(agent: &str) -> Vec<CredentialMount> {
|
pub async fn stage(agent: &str) -> Vec<CredentialMount> {
|
||||||
let dir = crate::paths::agent_identity_dir(agent);
|
let dir = crate::paths::agent_identity_dir(agent);
|
||||||
match collect(agent, &dir).await {
|
match collect(agent, &dir).await {
|
||||||
|
|
@ -246,7 +244,7 @@ mod tests {
|
||||||
|
|
||||||
/// Half a delivery is worse than none: the container's check would fail
|
/// Half a delivery is worse than none: the container's check would fail
|
||||||
/// on the missing key and report a login problem instead of a delivery
|
/// on the missing key and report a login problem instead of a delivery
|
||||||
/// one. Same both-or-neither rule the queue credential states.
|
/// one.
|
||||||
#[test]
|
#[test]
|
||||||
fn a_certificate_without_its_key_forwards_nothing() {
|
fn a_certificate_without_its_key_forwards_nothing() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
use anyhow::{Context, Result};
|
use anyhow::{Context, Result};
|
||||||
use hive_priv_sock::{BindMount, CredentialMount};
|
use hive_priv_sock::BindMount;
|
||||||
use hive_types::Permission;
|
use hive_types::Permission;
|
||||||
|
|
||||||
use crate::coordinator::{AgentPaths, HiveEnv};
|
use crate::coordinator::{AgentPaths, HiveEnv};
|
||||||
|
|
@ -417,70 +417,6 @@ pub fn hand_permissions(agent: &str, runtime_dir: &Path) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Env var naming the host directory `swarm-bao-queue-agent.service` lands
|
|
||||||
/// this hive's agent queue credential in. Set by the hive-c0re NixOS module
|
|
||||||
/// from `deploy.hive-controller.queue.agentCredentialDir`; absent means this
|
|
||||||
/// daemon runs outside its unit, which is the same "no queue credential"
|
|
||||||
/// answer as an empty directory.
|
|
||||||
const QUEUE_CREDENTIAL_DIR_ENV: &str = "HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR";
|
|
||||||
|
|
||||||
/// systemd credential ids the two files arrive under inside the container.
|
|
||||||
/// `nix/agent-modules/queue.nix` spells the same two names in the harness
|
|
||||||
/// unit's `LoadCredential=`; neither side can discover the other's, so a
|
|
||||||
/// rename here is a rename there.
|
|
||||||
const QUEUE_SECRET_CREDENTIAL: &str = "hive-queue-agent-secret";
|
|
||||||
const QUEUE_CLIENT_ID_CREDENTIAL: &str = "hive-queue-agent-client-id";
|
|
||||||
|
|
||||||
/// Forward this hive's agent queue credential into `agent_name`'s container
|
|
||||||
/// as two systemd credentials, or nothing when the publisher has not landed
|
|
||||||
/// it yet.
|
|
||||||
///
|
|
||||||
/// **A credential and not a bind mount, because of the mode.** The secret is
|
|
||||||
/// `root:0600` on the host and the harness runs as the unprivileged agent
|
|
||||||
/// user, so a bind would deliver a file that user cannot open. nspawn's
|
|
||||||
/// `--load-credential` is read by the container manager as root and
|
|
||||||
/// re-exposed under the consuming unit's own `User=`, which is the whole
|
|
||||||
/// difference. hive-c0re never reads the bytes either way — it runs as
|
|
||||||
/// `hive-core` and only ever needs to know the file is *there*, which a
|
|
||||||
/// `0755` directory permits.
|
|
||||||
///
|
|
||||||
/// **Absent files are legal.** Authelia mints the secret on its first boot
|
|
||||||
/// and a publisher on that host puts it in the store, so "nothing at that
|
|
||||||
/// path" is the ordinary early state of a swarm rather than a fault. Saying
|
|
||||||
/// so at `info!` is what keeps it from being invisible: without a line here,
|
|
||||||
/// an agent that never connects looks identical to one that was never
|
|
||||||
/// configured.
|
|
||||||
fn queue_agent_credentials(agent_name: &str, dir: Option<&Path>) -> Vec<CredentialMount> {
|
|
||||||
let Some(dir) = dir else {
|
|
||||||
tracing::info!(
|
|
||||||
%agent_name,
|
|
||||||
"no {QUEUE_CREDENTIAL_DIR_ENV} in this daemon's environment — agent gets no swarm queue credential"
|
|
||||||
);
|
|
||||||
return Vec::new();
|
|
||||||
};
|
|
||||||
let secret = dir.join("secret");
|
|
||||||
let client_id = dir.join("client_id");
|
|
||||||
// Both or neither: a client holding one of the two comes up fine and
|
|
||||||
// never connects.
|
|
||||||
if !secret.is_file() || !client_id.is_file() {
|
|
||||||
tracing::info!(
|
|
||||||
%agent_name, dir = %dir.display(),
|
|
||||||
"swarm queue credential not published yet — agent gets no swarm queue credential"
|
|
||||||
);
|
|
||||||
return Vec::new();
|
|
||||||
}
|
|
||||||
vec![
|
|
||||||
CredentialMount {
|
|
||||||
name: QUEUE_SECRET_CREDENTIAL.to_owned(),
|
|
||||||
host_path: secret.to_string_lossy().into_owned(),
|
|
||||||
},
|
|
||||||
CredentialMount {
|
|
||||||
name: QUEUE_CLIENT_ID_CREDENTIAL.to_owned(),
|
|
||||||
host_path: client_id.to_string_lossy().into_owned(),
|
|
||||||
},
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Idempotently rewrite the lines in `/etc/nixos-containers/<container>.conf`
|
/// Idempotently rewrite the lines in `/etc/nixos-containers/<container>.conf`
|
||||||
/// that hive-c0re owns: `PRIVATE_NETWORK` (always 1), `HOST_ADDRESS` (the
|
/// that hive-c0re owns: `PRIVATE_NETWORK` (always 1), `HOST_ADDRESS` (the
|
||||||
/// bridge gateway IP) and `EXTRA_NSPAWN_FLAGS` (the runtime-dir bind). What
|
/// bridge gateway IP) and `EXTRA_NSPAWN_FLAGS` (the runtime-dir bind). What
|
||||||
|
|
@ -537,18 +473,11 @@ async fn set_nspawn_flags(
|
||||||
// is needed here — the bind alone is enough.
|
// is needed here — the bind alone is enough.
|
||||||
let claude_mount = container_claude_mount(agent_name);
|
let claude_mount = container_claude_mount(agent_name);
|
||||||
|
|
||||||
// The agent's own swarm-queue credential — the one thing forwarded in
|
|
||||||
// here, and forwarded *as a credential* rather than a bind for the
|
|
||||||
// reason `queue_agent_credentials` records.
|
|
||||||
let queue_credential_dir = std::env::var_os(QUEUE_CREDENTIAL_DIR_ENV).map(PathBuf::from);
|
|
||||||
let mut load_creds = queue_agent_credentials(agent_name, queue_credential_dir.as_deref());
|
|
||||||
|
|
||||||
// The agent's own identity *at* the swarm secret store, collected under
|
// The agent's own identity *at* the swarm secret store, collected under
|
||||||
// this hive's certificate and forwarded the same way and for the same
|
// this hive's certificate. Staged here rather than at spawn because this
|
||||||
// reason. Staged here rather than at spawn because this is the one place
|
// is the one place that writes the container's credential list — see
|
||||||
// that writes the container's credential list — see
|
|
||||||
// `super::agent_identity` for the whole hop.
|
// `super::agent_identity` for the whole hop.
|
||||||
load_creds.extend(super::agent_identity::stage(agent_name).await);
|
let load_creds = super::agent_identity::stage(agent_name).await;
|
||||||
|
|
||||||
let mut binds: Vec<BindMount> = vec![
|
let mut binds: Vec<BindMount> = vec![
|
||||||
BindMount {
|
BindMount {
|
||||||
|
|
@ -678,10 +607,8 @@ async fn set_nspawn_flags(
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::{
|
use super::{
|
||||||
BindMount, HostMemory, MEMORY_CAP_FILE, QUEUE_CLIENT_ID_CREDENTIAL,
|
BindMount, HostMemory, MEMORY_CAP_FILE, bind_child_agent_dirs, bind_own_agent_dirs,
|
||||||
QUEUE_SECRET_CREDENTIAL, bind_child_agent_dirs, bind_own_agent_dirs,
|
holds_manage_root_agent, limits_to_apply, memory_cap_value, stage_limits,
|
||||||
holds_manage_root_agent, limits_to_apply, memory_cap_value, queue_agent_credentials,
|
|
||||||
stage_limits,
|
|
||||||
};
|
};
|
||||||
use crate::coordinator::HiveEnv;
|
use crate::coordinator::HiveEnv;
|
||||||
|
|
||||||
|
|
@ -791,51 +718,6 @@ mod tests {
|
||||||
assert!(binds.is_empty());
|
assert!(binds.is_empty());
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The ordinary state of a swarm before the publisher on the authelia
|
|
||||||
/// host has run: the directory is named and empty. Forwarding a
|
|
||||||
/// credential whose source does not exist would make every agent
|
|
||||||
/// container refuse to start, so this has to be silence-with-a-log
|
|
||||||
/// rather than a partial list.
|
|
||||||
#[test]
|
|
||||||
fn an_unpublished_queue_credential_forwards_nothing() {
|
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
|
||||||
assert!(queue_agent_credentials("iris", None).is_empty());
|
|
||||||
assert!(queue_agent_credentials("iris", Some(dir.path())).is_empty());
|
|
||||||
std::fs::write(dir.path().join("secret"), "s").expect("write secret");
|
|
||||||
assert!(
|
|
||||||
queue_agent_credentials("iris", Some(dir.path())).is_empty(),
|
|
||||||
"a secret with no client id is not a usable credential"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Both files present: two credentials, named the same two ids the
|
|
||||||
/// harness unit imports, pointing at the two files the reader unit
|
|
||||||
/// wrote.
|
|
||||||
#[test]
|
|
||||||
fn a_published_queue_credential_forwards_both_files() {
|
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
|
||||||
std::fs::write(dir.path().join("secret"), "s").expect("write secret");
|
|
||||||
std::fs::write(dir.path().join("client_id"), "hive-h1-agent").expect("write client_id");
|
|
||||||
let creds = queue_agent_credentials("iris", Some(dir.path()));
|
|
||||||
let named: Vec<(String, String)> = creds
|
|
||||||
.into_iter()
|
|
||||||
.map(|c| (c.name, c.host_path))
|
|
||||||
.collect::<Vec<_>>();
|
|
||||||
assert_eq!(
|
|
||||||
named,
|
|
||||||
[
|
|
||||||
(
|
|
||||||
QUEUE_SECRET_CREDENTIAL.to_owned(),
|
|
||||||
dir.path().join("secret").to_string_lossy().into_owned(),
|
|
||||||
),
|
|
||||||
(
|
|
||||||
QUEUE_CLIENT_ID_CREDENTIAL.to_owned(),
|
|
||||||
dir.path().join("client_id").to_string_lossy().into_owned(),
|
|
||||||
),
|
|
||||||
]
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn corrupt_swarm_permissions_deny_without_failing() {
|
fn corrupt_swarm_permissions_deny_without_failing() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
|
|
|
||||||
|
|
@ -455,12 +455,10 @@ const CANONICAL_INPUTS: &[&str] = &["nixpkgs"];
|
||||||
/// trivial to stub from tests (which build their own slice instead of
|
/// trivial to stub from tests (which build their own slice instead of
|
||||||
/// touching process-wide env).
|
/// touching process-wide env).
|
||||||
const FORWARDED_VARS: &[&str] = &[
|
const FORWARDED_VARS: &[&str] = &[
|
||||||
// Where the swarm queue is and where its tokens are minted, as an agent
|
// Where the swarm queue is, as an agent container reaches it. An address
|
||||||
// container reaches them. Both are addresses the *host* computes — the
|
// the *host* computes, never a loopback address, which inside a container
|
||||||
// queue's from the bridge IP an agent routes over, never a loopback
|
// is the agent itself.
|
||||||
// address, which inside a container is the agent itself.
|
|
||||||
"HIVE_AGENT_NATS_URL",
|
"HIVE_AGENT_NATS_URL",
|
||||||
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
|
|
||||||
// Where the swarm secret store listens. The hive's own `BAO_ADDR` value,
|
// Where the swarm secret store listens. The hive's own `BAO_ADDR` value,
|
||||||
// forwarded under a separate name so the container never inherits the
|
// forwarded under a separate name so the container never inherits the
|
||||||
// hive's store environment wholesale — the address is the only part of it
|
// hive's store environment wholesale — the address is the only part of it
|
||||||
|
|
@ -470,7 +468,7 @@ const FORWARDED_VARS: &[&str] = &[
|
||||||
// fetches to trust the swarm. Forwarded with the address, under the same gate.
|
// fetches to trust the swarm. Forwarded with the address, under the same gate.
|
||||||
"HIVE_AGENT_BAO_SERVICES_PKI_MOUNT",
|
"HIVE_AGENT_BAO_SERVICES_PKI_MOUNT",
|
||||||
// Where an agent's `swarm-logs` queries, computed by the host for the
|
// Where an agent's `swarm-logs` queries, computed by the host for the
|
||||||
// same reason as the two above: it is a gateway address, and a container
|
// same reason as the queue's: it is a gateway address, and a container
|
||||||
// cannot derive it from anything it holds.
|
// cannot derive it from anything it holds.
|
||||||
"HIVE_AGENT_LOGS_QUERY_URL",
|
"HIVE_AGENT_LOGS_QUERY_URL",
|
||||||
// The metrics store's origin, for `swarm-metrics`, for the same reason.
|
// The metrics store's origin, for `swarm-metrics`, for the same reason.
|
||||||
|
|
@ -494,18 +492,6 @@ const FORWARDED_VARS: &[&str] = &[
|
||||||
/// name. Half-wiring this map is a value that evaluates fine and is silently
|
/// name. Half-wiring this map is a value that evaluates fine and is silently
|
||||||
/// wrong.
|
/// wrong.
|
||||||
const FORWARDED_VAR_OPTIONS: &[(&str, &str)] = &[
|
const FORWARDED_VAR_OPTIONS: &[(&str, &str)] = &[
|
||||||
// Read at build time to decide whether the harness unit imports the hive's
|
|
||||||
// queue credential at all — see `nix/agent-modules/queue.nix`. Both halves
|
|
||||||
// wired for this map's own reason: an agent whose option says no queue and
|
|
||||||
// whose env says otherwise imports no credential for `swarm-logs`.
|
|
||||||
(
|
|
||||||
"HIVE_AGENT_NATS_URL",
|
|
||||||
"services.hyperhive.agent.queue.natsUrl",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
|
|
||||||
"services.hyperhive.agent.queue.tokenEndpoint",
|
|
||||||
),
|
|
||||||
// Read at build time to decide whether the in-container identity check is
|
// Read at build time to decide whether the in-container identity check is
|
||||||
// generated at all — see `nix/agent-modules/bao.nix`. Both halves wired
|
// generated at all — see `nix/agent-modules/bao.nix`. Both halves wired
|
||||||
// for this map's own reason: the unit bakes the address into its own
|
// for this map's own reason: the unit bakes the address into its own
|
||||||
|
|
@ -1561,35 +1547,22 @@ mod tests {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The queue coordinates travel as a pair and are wired through both
|
/// The queue address reaches the harness as env only. The agent module
|
||||||
/// halves of the forwarded-var machinery: the env var the harness reads at
|
/// declares no `services.hyperhive.agent.queue.*` option, so rendering one
|
||||||
/// runtime and the option the harness *unit* is built from. An agent that
|
/// into the meta flake fails every agent's evaluation.
|
||||||
/// got only the env var would declare no credential and then report a
|
|
||||||
/// partial config — visible, but a rebuild away from working.
|
|
||||||
#[test]
|
#[test]
|
||||||
fn queue_coordinates_are_forwarded_as_both_env_and_option() {
|
fn queue_address_is_forwarded_as_env_and_no_option() {
|
||||||
for var in ["HIVE_AGENT_NATS_URL", "HIVE_AGENT_OIDC_TOKEN_ENDPOINT"] {
|
assert!(FORWARDED_VARS.contains(&"HIVE_AGENT_NATS_URL"));
|
||||||
assert!(
|
assert!(
|
||||||
FORWARDED_VARS.contains(&var),
|
!FORWARDED_VARS.contains(&"HIVE_AGENT_OIDC_TOKEN_ENDPOINT"),
|
||||||
"{var} must be forwarded into every agent's env"
|
"no agent reads a token endpoint"
|
||||||
|
);
|
||||||
|
for (var, option) in FORWARDED_VAR_OPTIONS {
|
||||||
|
assert!(
|
||||||
|
!option.starts_with("services.hyperhive.agent.queue."),
|
||||||
|
"{var} renders {option}, which no agent module declares"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
let mut out = String::new();
|
|
||||||
push_forwarded_var_options(
|
|
||||||
&mut out,
|
|
||||||
&[
|
|
||||||
("HIVE_AGENT_NATS_URL", "nats://10.42.0.1:4222".to_string()),
|
|
||||||
(
|
|
||||||
"HIVE_AGENT_OIDC_TOKEN_ENDPOINT",
|
|
||||||
"https://auth.t.local/api/oidc/token".to_string(),
|
|
||||||
),
|
|
||||||
],
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
out,
|
|
||||||
" services.hyperhive.agent.queue.natsUrl = \"nats://10.42.0.1:4222\";\n\
|
|
||||||
\x20 services.hyperhive.agent.queue.tokenEndpoint = \"https://auth.t.local/api/oidc/token\";\n"
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `swarm-metrics` is installed only when the option is set, and reads
|
/// `swarm-metrics` is installed only when the option is set, and reads
|
||||||
|
|
|
||||||
|
|
@ -14,8 +14,7 @@
|
||||||
# is why: the host file is `0600` to the hive daemon and this unit runs as the
|
# is why: the host file is `0600` to the hive daemon and this unit runs as the
|
||||||
# unprivileged agent user. nspawn's `--load-credential` is read by the
|
# unprivileged agent user. nspawn's `--load-credential` is read by the
|
||||||
# container manager as root and re-exposed under this unit's own `User=`; a
|
# container manager as root and re-exposed under this unit's own `User=`; a
|
||||||
# bind would deliver a private key this user cannot open. Same answer, same
|
# bind would deliver a private key this user cannot open.
|
||||||
# reason, as ./queue.nix's credential pair.
|
|
||||||
#
|
#
|
||||||
# This is the courier's other end: the hive-side collector has no purpose
|
# This is the courier's other end: the hive-side collector has no purpose
|
||||||
# without it, so the two ship together and neither is reachable alone.
|
# without it, so the two ship together and neither is reachable alone.
|
||||||
|
|
@ -63,12 +62,11 @@ let
|
||||||
# Where the identity lives in the store, spelled from the same pieces the
|
# Where the identity lives in the store, spelled from the same pieces the
|
||||||
# publisher uses. `swarm_secret_client::mtls::identity_path` builds
|
# publisher uses. `swarm_secret_client::mtls::identity_path` builds
|
||||||
# `swarm/agents/<agent>/bao-mtls` and `path::MOUNT` is `secret`; this
|
# `swarm/agents/<agent>/bao-mtls` and `path::MOUNT` is `secret`; this
|
||||||
# literal is the nix half of that one agreement, exactly as
|
# literal is the nix half of that one agreement.
|
||||||
# ../host-modules/glue-queue-agent-credential.nix spells its own.
|
|
||||||
identityPath = "secret/swarm/agents/${agentName}/bao-mtls";
|
identityPath = "secret/swarm/agents/${agentName}/bao-mtls";
|
||||||
|
|
||||||
# The address is the whole switch — no separate `enable`, the same shape
|
# The address is the whole switch — no separate `enable`, the same shape
|
||||||
# ./queue.nix and ./logs.nix gate themselves with. A hive that has a store
|
# ./logs.nix gates itself with. A hive that has a store
|
||||||
# forwards its address and every agent on it gets the check; a hive that has
|
# forwards its address and every agent on it gets the check; a hive that has
|
||||||
# none forwards nothing and no agent does. An `enable` beside it would be a
|
# none forwards nothing and no agent does. An `enable` beside it would be a
|
||||||
# knob whose only correct setting is whatever the address already says, and
|
# knob whose only correct setting is whatever the address already says, and
|
||||||
|
|
@ -153,7 +151,7 @@ in
|
||||||
UMask = "0377";
|
UMask = "0377";
|
||||||
# Bare ids, no paths: the terse `LoadCredential=` form that inherits a
|
# Bare ids, no paths: the terse `LoadCredential=` form that inherits a
|
||||||
# credential the service *manager* received, which is what the
|
# credential the service *manager* received, which is what the
|
||||||
# container manager passed in. ./queue.nix states the same shape.
|
# container manager passed in.
|
||||||
LoadCredential = [
|
LoadCredential = [
|
||||||
certCredential
|
certCredential
|
||||||
keyCredential
|
keyCredential
|
||||||
|
|
|
||||||
|
|
@ -34,6 +34,12 @@ let
|
||||||
sets it. Pin it host-side with
|
sets it. Pin it host-side with
|
||||||
services.hyperhive.deploy.hive-controller.claudeCodePackage.
|
services.hyperhive.deploy.hive-controller.claudeCodePackage.
|
||||||
'';
|
'';
|
||||||
|
queueRemoved = ''
|
||||||
|
The harness reads the queue's address at runtime from HIVE_AGENT_NATS_URL,
|
||||||
|
which the hive forwards into every container, and authenticates with the
|
||||||
|
agent's own secret from the swarm secret store, so no agent build sets a
|
||||||
|
queue option. Remove the setting.
|
||||||
|
'';
|
||||||
|
|
||||||
# `claude` on PATH. A hive that pins a claude-code
|
# `claude` on PATH. A hive that pins a claude-code
|
||||||
# (`services.hyperhive.deploy.hive-controller.claudeCodePackage`) hands its
|
# (`services.hyperhive.deploy.hive-controller.claudeCodePackage`) hands its
|
||||||
|
|
@ -81,7 +87,6 @@ in
|
||||||
./otel.nix
|
./otel.nix
|
||||||
./otel-removed.nix
|
./otel-removed.nix
|
||||||
./packages.nix
|
./packages.nix
|
||||||
./queue.nix
|
|
||||||
./queue-identity.nix
|
./queue-identity.nix
|
||||||
./user.nix
|
./user.nix
|
||||||
./screen.nix
|
./screen.nix
|
||||||
|
|
@ -101,6 +106,8 @@ in
|
||||||
"agent"
|
"agent"
|
||||||
"claudeCodePath"
|
"claudeCodePath"
|
||||||
] claudeCodePathRemoved)
|
] claudeCodePathRemoved)
|
||||||
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "queue" "natsUrl" ] queueRemoved)
|
||||||
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "agent" "queue" "tokenEndpoint" ] queueRemoved)
|
||||||
];
|
];
|
||||||
|
|
||||||
options.services.hyperhive.agent.icon = lib.mkOption {
|
options.services.hyperhive.agent.icon = lib.mkOption {
|
||||||
|
|
|
||||||
|
|
@ -1,13 +1,11 @@
|
||||||
# This agent's own credential for the swarm queue, read from the store by the
|
# This agent's own credential for the swarm queue, read from the store by the
|
||||||
# harness itself.
|
# harness itself.
|
||||||
#
|
#
|
||||||
# The credential beside it in ./queue.nix is keyed per **hive**: one OIDC
|
# Keyed per **agent**, so at the queue's auth callout one agent is
|
||||||
# client minted at deploy time and handed to every agent container on the
|
# distinguishable from its co-hived neighbours: `swarm-controller` mints it at
|
||||||
# hive, so at the queue's auth callout one agent is indistinguishable from its
|
# agent creation into `swarm/agents/<agent>/queue`
|
||||||
# co-hived neighbours. This one is keyed per **agent** — `swarm-controller`
|
# (`swarm_secret_client::queue::agent_queue_path`), at swarm level, with no hive
|
||||||
# mints it at agent creation into `swarm/agents/<agent>/queue`
|
# anywhere in the chain.
|
||||||
# (`swarm_secret_client::queue::agent_queue_path`), at swarm level, with no
|
|
||||||
# hive anywhere in the chain.
|
|
||||||
#
|
#
|
||||||
# The agent authenticates to the store as itself, with the certificate
|
# The agent authenticates to the store as itself, with the certificate
|
||||||
# ./bao.nix already proves it can log in with, and reads its own path. A hive
|
# ./bao.nix already proves it can log in with, and reads its own path. A hive
|
||||||
|
|
@ -42,11 +40,7 @@ let
|
||||||
# too.
|
# too.
|
||||||
serverCaCredential = "hive-agent-bao-server-ca";
|
serverCaCredential = "hive-agent-bao-server-ca";
|
||||||
|
|
||||||
# The store's address is the whole switch, exactly as in ./bao.nix — and
|
# The store's address is the whole switch, exactly as in ./bao.nix.
|
||||||
# deliberately *not* the queue coordinates in ./queue.nix. Those are the
|
|
||||||
# hive's, and gating a swarm-minted per-agent credential on a hive-level
|
|
||||||
# option would put the hive back in a path whose entire purpose is not
|
|
||||||
# having one.
|
|
||||||
configured = cfg.addr != null;
|
configured = cfg.addr != null;
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
|
|
@ -54,7 +48,7 @@ in
|
||||||
systemd.services.hive-agent = {
|
systemd.services.hive-agent = {
|
||||||
# A bare id, no path: the terse `LoadCredential=` form that inherits a
|
# A bare id, no path: the terse `LoadCredential=` form that inherits a
|
||||||
# credential the service *manager* received, which is what the container
|
# credential the service *manager* received, which is what the container
|
||||||
# manager passed in. ./bao.nix and ./queue.nix state the same shape.
|
# manager passed in. ./bao.nix states the same shape.
|
||||||
serviceConfig.LoadCredential = [ serverCaCredential ];
|
serviceConfig.LoadCredential = [ serverCaCredential ];
|
||||||
# The live certificate pair exists once ./bao.nix's unit has run.
|
# The live certificate pair exists once ./bao.nix's unit has run.
|
||||||
after = [ "hive-agent-bao-identity.service" ];
|
after = [ "hive-agent-bao-identity.service" ];
|
||||||
|
|
|
||||||
|
|
@ -1,117 +0,0 @@
|
||||||
# The hive's shared swarm-queue client credential, imported into the harness
|
|
||||||
# unit's credentials directory for `swarm-logs` (./logs.nix) to authenticate
|
|
||||||
# with. The harness itself connects to the queue with the agent's own
|
|
||||||
# credential (./queue-identity.nix) and reads neither file.
|
|
||||||
#
|
|
||||||
# The credential is two files, the secret and the client id beside it, so a
|
|
||||||
# reader never spells `hive-<name>-agent` a second time.
|
|
||||||
#
|
|
||||||
# ⚠️ The credential arrives as a systemd credential and NOT as a bind mount,
|
|
||||||
# and the mode is why: the host file is `root:0600` and the reader runs as the
|
|
||||||
# unprivileged agent user. nspawn's `--load-credential` (written by
|
|
||||||
# `hive_c0re::lifecycle::host_config`) is read by the container manager as
|
|
||||||
# root and re-exposed under this unit's own `User=`; a bind would deliver a
|
|
||||||
# file the agent user cannot open.
|
|
||||||
{
|
|
||||||
lib,
|
|
||||||
config,
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
cfg = config.services.hyperhive.agent.queue;
|
|
||||||
configured = cfg.natsUrl != null && cfg.tokenEndpoint != null;
|
|
||||||
|
|
||||||
# The two ids `hive_c0re::lifecycle::host_config` forwards under. Neither
|
|
||||||
# side can discover the other's spelling, so a rename is a rename there too.
|
|
||||||
secretCredential = "hive-queue-agent-secret";
|
|
||||||
clientIdCredential = "hive-queue-agent-client-id";
|
|
||||||
|
|
||||||
# Where systemd materialises this unit's credentials. `%d` only expands
|
|
||||||
# *inside* a unit, so `swarm-logs` (./logs.nix), which is not one, needs the
|
|
||||||
# literal; deriving it from the unit name here is what keeps the two from
|
|
||||||
# disagreeing about which unit's credentials they mean.
|
|
||||||
credentialsDir = "/run/credentials/hive-agent.service";
|
|
||||||
in
|
|
||||||
{
|
|
||||||
options.services.hyperhive.agent.queue = {
|
|
||||||
natsUrl = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.str;
|
|
||||||
default = null;
|
|
||||||
example = "tls://nats.example.com:4222";
|
|
||||||
description = ''
|
|
||||||
Where the swarm queue listens, as this container reaches it.
|
|
||||||
|
|
||||||
Set by the generated meta flake from the host's
|
|
||||||
{option}`services.hyperhive.deploy.hive-controller.queue.agentNatsUrl`,
|
|
||||||
which names the queue rather than a loopback address — inside this
|
|
||||||
container `127.0.0.1` is the agent itself.
|
|
||||||
|
|
||||||
`null` means this hive has not been given the queue's address for its
|
|
||||||
agents: the harness unit imports no credential, and the harness fails
|
|
||||||
at boot, since it cannot run without a queue connection. It is
|
|
||||||
deliberately not defaulted to anything: a guessed address builds fine
|
|
||||||
and talks to the wrong machine.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
tokenEndpoint = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.str;
|
|
||||||
default = null;
|
|
||||||
example = "https://auth.example.com/api/oidc/token";
|
|
||||||
description = ''
|
|
||||||
The swarm IdP's OAuth2 token endpoint, where `swarm-logs` mints a
|
|
||||||
`client_credentials` token for the client named in the delivered
|
|
||||||
credential.
|
|
||||||
|
|
||||||
The harness unit imports that credential only when this and
|
|
||||||
{option}`services.hyperhive.agent.queue.natsUrl` are both set.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
# One declaration, two readers. The credential ids are this module's —
|
|
||||||
# `hive_c0re::lifecycle::host_config` and the `LoadCredential=` below are
|
|
||||||
# the pair that has to agree on them — and a second consumer that spelled
|
|
||||||
# them again would be a second source of truth for a string whose
|
|
||||||
# mismatch is a file that is simply not there.
|
|
||||||
clientIdFile = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
readOnly = true;
|
|
||||||
default = "${credentialsDir}/${clientIdCredential}";
|
|
||||||
description = ''
|
|
||||||
Path the agent's OIDC client id is delivered at, for a consumer
|
|
||||||
outside the harness unit. Read-only: it is a fact about where the
|
|
||||||
credential lands, not a knob — see {option}`services.hyperhive.agent.logs.queryUrl`
|
|
||||||
for the consumer this exists for.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
clientSecretFile = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
readOnly = true;
|
|
||||||
default = "${credentialsDir}/${secretCredential}";
|
|
||||||
description = ''
|
|
||||||
Path the agent's OIDC client secret is delivered at. Read-only for
|
|
||||||
the same reason as {option}`services.hyperhive.agent.queue.clientIdFile`.
|
|
||||||
|
|
||||||
🩸 A PATH and never a value. The file is `0400` to the agent user and
|
|
||||||
is read at the moment of a token request; nothing in this tree puts
|
|
||||||
its contents in an environment variable, where `/proc/<pid>/environ`
|
|
||||||
would publish them to every process in the container.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
config = lib.mkIf configured {
|
|
||||||
systemd.services.hive-agent = {
|
|
||||||
# Bare ids, no paths: this is the terse `LoadCredential=` form that
|
|
||||||
# inherits a credential the service *manager* received, which is what
|
|
||||||
# the container manager passed in. `man systemd.exec` also makes that
|
|
||||||
# form non-fatal when the credential is absent, which is exactly the
|
|
||||||
# behaviour a hive whose publisher has not run yet needs.
|
|
||||||
serviceConfig.LoadCredential = [
|
|
||||||
secretCredential
|
|
||||||
clientIdCredential
|
|
||||||
];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# The unit that puts swarm-level secrets into the swarm's secret store, so
|
# The unit that puts swarm-level secrets into the swarm's secret store, so
|
||||||
# whoever needs one can read it there: a hive its own and its agents'
|
# whoever needs one can read it there: a hive, a service or the controller its
|
||||||
# credentials, a service or the controller its own. It mints nothing: every
|
# own. It mints nothing: every
|
||||||
# value is an OIDC client secret authelia minted, and this unit copies it.
|
# value is an OIDC client secret authelia minted, and this unit copies it.
|
||||||
#
|
#
|
||||||
# Imported inside authelia's container, beside the mint: it reads each secret
|
# Imported inside authelia's container, beside the mint: it reads each secret
|
||||||
|
|
@ -61,14 +61,6 @@ in
|
||||||
example = "secret/swarm/services/swarm-grafana/oidc/client";
|
example = "secret/swarm/services/swarm-grafana/oidc/client";
|
||||||
description = "KV path its reader fetches the value from.";
|
description = "KV path its reader fetches the value from.";
|
||||||
};
|
};
|
||||||
withClientId = lib.mkOption {
|
|
||||||
type = lib.types.bool;
|
|
||||||
default = false;
|
|
||||||
description = ''
|
|
||||||
Also store the client id beside the value, as `client_id`, for
|
|
||||||
a reader that has to reconstruct it from the store alone.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
@ -170,7 +162,7 @@ in
|
||||||
# `value="$(cat "$src")"` would publish it to /proc for anyone in
|
# `value="$(cat "$src")"` would publish it to /proc for anyone in
|
||||||
# the container to read.
|
# the container to read.
|
||||||
bao kv put ${lib.escapeShellArg s.path} \
|
bao kv put ${lib.escapeShellArg s.path} \
|
||||||
value=@"$src"${lib.optionalString s.withClientId " client_id=${lib.escapeShellArg s.clientId}"}
|
value=@"$src"
|
||||||
published=$((published + 1))
|
published=$((published + 1))
|
||||||
else
|
else
|
||||||
# Not an error: authelia mints on its FIRST BOOT, so an absent file
|
# Not an error: authelia mints on its FIRST BOOT, so an absent file
|
||||||
|
|
|
||||||
|
|
@ -47,7 +47,6 @@
|
||||||
./glue-nats-bao-identity.nix
|
./glue-nats-bao-identity.nix
|
||||||
./glue-nats-oidc-client.nix
|
./glue-nats-oidc-client.nix
|
||||||
./glue-nats-wireguard.nix
|
./glue-nats-wireguard.nix
|
||||||
./glue-queue-agent-credential.nix
|
|
||||||
./glue-builder-oidc-client.nix
|
./glue-builder-oidc-client.nix
|
||||||
./glue-ci-metrics-otel-oidc-client.nix
|
./glue-ci-metrics-otel-oidc-client.nix
|
||||||
./glue-ci-runner-otel-oidc-client.nix
|
./glue-ci-runner-otel-oidc-client.nix
|
||||||
|
|
|
||||||
|
|
@ -53,6 +53,14 @@ let
|
||||||
puts it there. Remove this definition; the file it named is read by
|
puts it there. Remove this definition; the file it named is read by
|
||||||
nothing now and can be deleted.
|
nothing now and can be deleted.
|
||||||
'';
|
'';
|
||||||
|
|
||||||
|
# Shared by the removed options of the hive's agent queue credential reader.
|
||||||
|
queueAgentCredentialRemoved = ''
|
||||||
|
A hive no longer fetches a shared queue credential for its agents: each
|
||||||
|
agent reads its own secret from the swarm's secret store under its own
|
||||||
|
certificate. Remove this definition; the leaf or directory it named is read
|
||||||
|
by nothing now and can be deleted.
|
||||||
|
'';
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
imports = [
|
imports = [
|
||||||
|
|
@ -95,6 +103,35 @@ in
|
||||||
"statusPublish"
|
"statusPublish"
|
||||||
"clientSecretFile"
|
"clientSecretFile"
|
||||||
] statusSecretRemoved)
|
] statusSecretRemoved)
|
||||||
|
(lib.mkRemovedOptionModule [
|
||||||
|
"services"
|
||||||
|
"hyperhive"
|
||||||
|
"deploy"
|
||||||
|
"bao"
|
||||||
|
"queueAgentCommonNamePrefix"
|
||||||
|
] queueAgentCredentialRemoved)
|
||||||
|
(lib.mkRemovedOptionModule [
|
||||||
|
"services"
|
||||||
|
"hyperhive"
|
||||||
|
"deploy"
|
||||||
|
"bao"
|
||||||
|
"queueAgentClientCertFile"
|
||||||
|
] queueAgentCredentialRemoved)
|
||||||
|
(lib.mkRemovedOptionModule [
|
||||||
|
"services"
|
||||||
|
"hyperhive"
|
||||||
|
"deploy"
|
||||||
|
"bao"
|
||||||
|
"queueAgentClientKeyFile"
|
||||||
|
] queueAgentCredentialRemoved)
|
||||||
|
(lib.mkRemovedOptionModule [
|
||||||
|
"services"
|
||||||
|
"hyperhive"
|
||||||
|
"deploy"
|
||||||
|
"hive-controller"
|
||||||
|
"queue"
|
||||||
|
"agentCredentialDir"
|
||||||
|
] queueAgentCredentialRemoved)
|
||||||
|
|
||||||
# Same type, same meaning, new path — so a rename carries it exactly
|
# Same type, same meaning, new path — so a rename carries it exactly
|
||||||
# and existing configs keep evaluating with one warning naming both
|
# and existing configs keep evaluating with one warning naming both
|
||||||
|
|
|
||||||
|
|
@ -31,10 +31,6 @@ let
|
||||||
|
|
||||||
# Reader unit → the gate its own module defines it under.
|
# Reader unit → the gate its own module defines it under.
|
||||||
readersHere = {
|
readersHere = {
|
||||||
# ./glue-queue-agent-credential.nix
|
|
||||||
swarm-bao-queue-agent =
|
|
||||||
deployCfg.hive-controller.enable
|
|
||||||
&& havePair baoDeploy.queueAgentClientCertFile baoDeploy.queueAgentClientKeyFile;
|
|
||||||
# ./swarm-grafana.nix
|
# ./swarm-grafana.nix
|
||||||
swarm-bao-grafana-oidc = deployCfg.grafana.enable;
|
swarm-bao-grafana-oidc = deployCfg.grafana.enable;
|
||||||
# ./swarm-otel.nix
|
# ./swarm-otel.nix
|
||||||
|
|
|
||||||
|
|
@ -95,15 +95,13 @@ in
|
||||||
# 🩸 The readers below each present a leaf of their own rather than
|
# 🩸 The readers below each present a leaf of their own rather than
|
||||||
# `client.pem` above. The leaf is what bao sees, so this pairing is the
|
# `client.pem` above. The leaf is what bao sees, so this pairing is the
|
||||||
# whole of what turns "one principal with the union of their grants" into
|
# whole of what turns "one principal with the union of their grants" into
|
||||||
# one principal per grant — see ./swarm-bao.nix's `perHiveReaders` block
|
# one principal per grant — see ./swarm-bao.nix's `readStanza` block for
|
||||||
# for the grants themselves.
|
# the grants themselves.
|
||||||
#
|
#
|
||||||
# Defaulted here rather than in `glue-<consumer>-bao-identity.nix` files:
|
# Defaulted here rather than in `glue-<consumer>-bao-identity.nix` files:
|
||||||
# those exist where the consumer is a CONTAINER with a
|
# those exist where the consumer is a CONTAINER with a
|
||||||
# `deploy.<service>.*` namespace of its own to point at. These are host
|
# `deploy.<service>.*` namespace of its own to point at. These are host
|
||||||
# units reading the store, which is the pairing this file already is.
|
# units reading the store, which is the pairing this file already is.
|
||||||
queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem";
|
|
||||||
queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem";
|
|
||||||
grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem";
|
grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem";
|
||||||
grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem";
|
grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem";
|
||||||
otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem";
|
otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem";
|
||||||
|
|
@ -223,20 +221,10 @@ in
|
||||||
# principal holding the union of their grants, and the union would
|
# principal holding the union of their grants, and the union would
|
||||||
# include every agent credential in the swarm.
|
# include every agent credential in the swarm.
|
||||||
#
|
#
|
||||||
# ⚠️ The first carries THIS hive's name in the subject, unlike the three
|
|
||||||
# service leaves above. Its grant names one hive's path, because a queue
|
|
||||||
# credential lives under `swarm/hives/<name>/` and every hive runs a
|
|
||||||
# reader for its own; ./swarm-bao.nix writes one role per hive in the
|
|
||||||
# swarm directory to match. The OIDC readers need no such segment — a
|
|
||||||
# client is registered once per swarm.
|
|
||||||
#
|
|
||||||
# Minted whether or not the consumer runs here, for the reason the three
|
# Minted whether or not the consumer runs here, for the reason the three
|
||||||
# leaves above give: on a hive that does not run the store this is the
|
# leaves above give: on a hive that does not run the store this is the
|
||||||
# file an operator copies, and a leaf that only appears where its
|
# file an operator copies, and a leaf that only appears where its
|
||||||
# consumer does is one nobody can copy from anywhere.
|
# consumer does is one nobody can copy from anywhere.
|
||||||
[ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \
|
|
||||||
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth
|
|
||||||
|
|
||||||
[ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \
|
[ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \
|
||||||
${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth
|
${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth
|
||||||
|
|
||||||
|
|
@ -344,14 +332,14 @@ in
|
||||||
${lib.optionalString deployCfg.hive-controller.enable ''
|
${lib.optionalString deployCfg.hive-controller.enable ''
|
||||||
# This hive's gateway-leaf identity, which opens
|
# This hive's gateway-leaf identity, which opens
|
||||||
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its
|
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its
|
||||||
# subject carries this hive's name, like the queue-agent leaf
|
# subject carries this hive's name, because ./swarm-bao.nix writes
|
||||||
# above, because ./swarm-bao.nix writes one role
|
# one role per hive. Only where a hive runs; the other hives' are
|
||||||
# per hive. Only where a hive runs; the other hives' are below.
|
# below.
|
||||||
[ -s ${pkiDir}/hive-leaf.pem ] || ${signLeaf} ${pkiDir} hive-leaf \
|
[ -s ${pkiDir}/hive-leaf.pem ] || ${signLeaf} ${pkiDir} hive-leaf \
|
||||||
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${clientCn}"} "" clientAuth
|
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${clientCn}"} "" clientAuth
|
||||||
''}
|
''}
|
||||||
|
|
||||||
# The reader, queue-agent and gateway-leaf leaves of every other hive in
|
# The reader and gateway-leaf leaves of every other hive in
|
||||||
# the swarm directory: the files an operator copies to that hive's
|
# the swarm directory: the files an operator copies to that hive's
|
||||||
# host. One directory per hive, with the basenames this hive's leaves
|
# host. One directory per hive, with the basenames this hive's leaves
|
||||||
# use, so a hive's set is one directory to copy.
|
# use, so a hive's set is one directory to copy.
|
||||||
|
|
@ -364,8 +352,6 @@ in
|
||||||
install -d -m 0700 ${d}
|
install -d -m 0700 ${d}
|
||||||
[ -s ${d}/client.pem ] || ${signLeaf} ${d} client \
|
[ -s ${d}/client.pem ] || ${signLeaf} ${d} client \
|
||||||
${lib.escapeShellArg h} "" clientAuth
|
${lib.escapeShellArg h} "" clientAuth
|
||||||
[ -s ${d}/queue-agent.pem ] || ${signLeaf} ${d} queue-agent \
|
|
||||||
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${h}"} "" clientAuth
|
|
||||||
[ -s ${d}/hive-leaf.pem ] || ${signLeaf} ${d} hive-leaf \
|
[ -s ${d}/hive-leaf.pem ] || ${signLeaf} ${d} hive-leaf \
|
||||||
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${h}"} "" clientAuth
|
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${h}"} "" clientAuth
|
||||||
''
|
''
|
||||||
|
|
|
||||||
|
|
@ -1,284 +0,0 @@
|
||||||
# Glue: this hive's agent queue credential comes out of the secret store.
|
|
||||||
#
|
|
||||||
# A cert login that fails LOUDLY because every state it fails on is one a
|
|
||||||
# retry fixes, then a read that degrades QUIETLY because no retry turns "no
|
|
||||||
# value there" into a value.
|
|
||||||
#
|
|
||||||
# ⚠️ Two files, not one, and that is the consumer's shape rather than a
|
|
||||||
# preference. `swarm-logs` inside the agent reads the secret and the client id
|
|
||||||
# each from its own file (`<prefix>_OIDC_CLIENT_SECRET_FILE`,
|
|
||||||
# `<prefix>_OIDC_CLIENT_ID_FILE`), so splitting them here is what lets
|
|
||||||
# hive-c0re hand both to an agent without parsing anything.
|
|
||||||
#
|
|
||||||
# ⚠️ An ABSENT file means this hive's agents do not connect to the queue, and
|
|
||||||
# that is correct rather than degraded. The publisher runs on the authelia
|
|
||||||
# host and authelia mints on its first boot, so "nothing at that path yet" is
|
|
||||||
# the ordinary early state of a swarm. Nothing here writes a local stand-in:
|
|
||||||
# there is no such thing as a locally valid OIDC client secret, so a placeholder would turn a hive that cannot connect
|
|
||||||
# into one that is refused, which reaches the agent as a timeout.
|
|
||||||
#
|
|
||||||
# 📌 This runs wherever a client identity is configured, NOT only where the
|
|
||||||
# store is. `deploy.bao.enable` would have been the co-location assumption
|
|
||||||
# itself; the reader needs a certificate, not a neighbour. On the store's own
|
|
||||||
# host ./glue-bao-tls.nix supplies one as a `mkDefault`; elsewhere an operator
|
|
||||||
# places the leaf and names it, and the same unit works. It also needs this
|
|
||||||
# host to run a hive (`deploy.hive-controller.enable`): the credential is for
|
|
||||||
# the agents hive-c0re creates here at runtime, and the unit writes under
|
|
||||||
# hive-c0re's own state directory.
|
|
||||||
{
|
|
||||||
pkgs,
|
|
||||||
lib,
|
|
||||||
config,
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
hyperhiveCfg = config.services.hyperhive;
|
|
||||||
deployCfg = hyperhiveCfg.deploy;
|
|
||||||
baoCfg = hyperhiveCfg.swarm.bao;
|
|
||||||
|
|
||||||
baoDeploy = deployCfg.bao;
|
|
||||||
|
|
||||||
# What decides whether this unit exists at all. A reader is defined by holding
|
|
||||||
# a certificate the store accepts, and that is true on the store's own host
|
|
||||||
# and on a hive three networks away for exactly the same reason.
|
|
||||||
#
|
|
||||||
# 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's. Bao
|
|
||||||
# matches a cert-auth role on the CN, so one leaf shared by four readers
|
|
||||||
# would be ONE principal holding the union of four grants: read on
|
|
||||||
# `swarm/agents/*` AND `swarm/hives/<hive>/*`, when this unit reads one
|
|
||||||
# queue credential and nothing else. Its own leaf carries
|
|
||||||
# `<queueAgentCommonNamePrefix>-<hive>` and its role grants the single path
|
|
||||||
# below — still this hive's own, so the narrowing costs no reach.
|
|
||||||
haveClientIdentity =
|
|
||||||
baoDeploy.queueAgentClientCertFile != null && baoDeploy.queueAgentClientKeyFile != null;
|
|
||||||
|
|
||||||
# Does this host read the store at all — the hive's own leaf, which is the
|
|
||||||
# one thing a remote-store deployment has to place by hand. Only used to
|
|
||||||
# decide whether a missing per-principal leaf is a mistake or a deployment
|
|
||||||
# that has no store: a host holding neither is the supported no-store
|
|
||||||
# shape, and one holding this pair but not the pair above reads the store
|
|
||||||
# without a reader identity of its own.
|
|
||||||
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
|
|
||||||
|
|
||||||
credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir;
|
|
||||||
secretFile = "${credentialDir}/secret";
|
|
||||||
clientIdFile = "${credentialDir}/client_id";
|
|
||||||
|
|
||||||
# Where the credential lives in the store, spelled from the same pieces the
|
|
||||||
# writer uses. Whoever writes it and whoever reads it must agree, and the
|
|
||||||
# agreement belongs in one visible place.
|
|
||||||
#
|
|
||||||
# ⚠️ The `hives/<name>` segment is not decoration — it is what the reader's
|
|
||||||
# own grant covers, so a path outside it is a 403 rather than a miss, however
|
|
||||||
# correct it looks. `swarm-secret-client`'s `queue::agent_client_path` builds
|
|
||||||
# the same string from the same pieces; this literal is the nix half of that
|
|
||||||
# one agreement.
|
|
||||||
#
|
|
||||||
# `hiveName` has no fallback here for the reason ./glue-bao-tls.nix gives at
|
|
||||||
# its own use of it: it is asserted set wherever
|
|
||||||
# `deploy.hive-controller.enable` is.
|
|
||||||
credentialPath = "secret/swarm/hives/${hyperhiveCfg.hiveName}/queue/agent";
|
|
||||||
|
|
||||||
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
|
|
||||||
storeRetry = import ./lib/store-retry.nix { };
|
|
||||||
in
|
|
||||||
{
|
|
||||||
options.services.hyperhive.deploy.hive-controller.queue = {
|
|
||||||
agentCredentialDir = lib.mkOption {
|
|
||||||
type = lib.types.path;
|
|
||||||
default = "/var/lib/hyperhive/queue-agent";
|
|
||||||
description = ''
|
|
||||||
Host directory holding this hive's agent queue credential: `secret`
|
|
||||||
(the OIDC client secret, `0600`) and `client_id` (the client that
|
|
||||||
secret authenticates, `0644` — it is sent to the token endpoint on
|
|
||||||
every connection and is not itself a secret).
|
|
||||||
|
|
||||||
Both files are written by `swarm-bao-queue-agent.service`, which also
|
|
||||||
creates this directory. Named here because nothing else creates it:
|
|
||||||
the parent `/var/lib/hyperhive` belongs to `hive-c0re.service`'s
|
|
||||||
`StateDirectory=`, which re-applies its own mode on every start, and a
|
|
||||||
second declaration of *that* path is the two-mechanisms-one-path trap
|
|
||||||
./hive-gateway/default.nix records — so this directory is the unit's
|
|
||||||
to make and the parent stays c0re's.
|
|
||||||
|
|
||||||
Absent files mean this hive has no queue credential yet, which is what
|
|
||||||
a swarm looks like before the publisher in authelia's container has run.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
config = lib.mkMerge [
|
|
||||||
# ⚠️ A SEPARATE arm from the unit below, and that separation is the whole
|
|
||||||
# mechanism: the unit's arm is gated on `haveClientIdentity`, so an
|
|
||||||
# assertion written inside it could never be reached in the state it
|
|
||||||
# exists to report.
|
|
||||||
#
|
|
||||||
# Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the
|
|
||||||
# same refusal, named to this principal's own pair. Where Grafana's gate
|
|
||||||
# is `deploy.grafana.enable`, this reader's is the hive's own: every hive
|
|
||||||
# runs one for its own agents, so on a hive, reading the store at all is
|
|
||||||
# what asks for it. Hence `hiveReaderIdentity` — a host holding no hive
|
|
||||||
# leaf has no store to read and nothing is missing.
|
|
||||||
(lib.mkIf (deployCfg.hive-controller.enable && hiveReaderIdentity) {
|
|
||||||
assertions = [
|
|
||||||
{
|
|
||||||
assertion = haveClientIdentity;
|
|
||||||
message = ''
|
|
||||||
This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile
|
|
||||||
is set), so it needs the agent queue credential reader's own client
|
|
||||||
identity: set both
|
|
||||||
|
|
||||||
services.hyperhive.deploy.bao.queueAgentClientCertFile
|
|
||||||
services.hyperhive.deploy.bao.queueAgentClientKeyFile
|
|
||||||
|
|
||||||
swarm-bao-queue-agent.service fetches this hive's agent queue
|
|
||||||
credential out of the store, and without these it is not rendered
|
|
||||||
at all — leaving hive-c0re with no queue credential and no unit
|
|
||||||
that would ever have written one.
|
|
||||||
|
|
||||||
Every hive runs this reader for its own agents, so unlike the other
|
|
||||||
three principals there is no per-service toggle that turns it off:
|
|
||||||
a hive that reads the store at all is a hive that needs this pair.
|
|
||||||
|
|
||||||
⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one
|
|
||||||
is the hive's, and its grant reads every agent credential in the
|
|
||||||
swarm and this hive's own secrets; this role reads the one
|
|
||||||
queue-credential path. Pointing this option at the hive's leaf
|
|
||||||
would evaluate, deploy and log in — and undo the split.
|
|
||||||
|
|
||||||
On a hive that runs the store, glue-bao-tls.nix supplies both as
|
|
||||||
defaults and there is nothing to do. Elsewhere copy the pair from
|
|
||||||
/var/lib/swarm-bao-pki/hives/<hive>/ on the store's host and name it
|
|
||||||
here — see docs/swarm/bao.md.
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
];
|
|
||||||
})
|
|
||||||
|
|
||||||
(lib.mkIf (deployCfg.hive-controller.enable && haveClientIdentity) {
|
|
||||||
systemd.services.swarm-bao-queue-agent = {
|
|
||||||
description = "fetch this hive's agent queue credential from the swarm secret store";
|
|
||||||
# Every one of these names a unit that exists only where the store runs,
|
|
||||||
# so the ordering is conditional even though the read is not: off-host
|
|
||||||
# there is nothing local to wait for, and the timeout below is what
|
|
||||||
# bounds the attempt instead. `Wants=` pki, for
|
|
||||||
# ./glue-bao-readers-store-order.nix's reason.
|
|
||||||
after = lib.optionals baoDeploy.enable [
|
|
||||||
"swarm-bao-pki.service"
|
|
||||||
"container@${baoCfg.machine}.service"
|
|
||||||
];
|
|
||||||
wants = lib.optionals baoDeploy.enable [
|
|
||||||
"swarm-bao-pki.service"
|
|
||||||
"container@${baoCfg.machine}.service"
|
|
||||||
];
|
|
||||||
# Ordered before hive-c0re, so no agent container renders ahead of an
|
|
||||||
# attempt at its credential. `Wants=`, not `Requires=`: a store this
|
|
||||||
# unit can't reach delays hive-c0re's start by one attempt
|
|
||||||
# (`TimeoutStartSec` below) rather than failing it — hive-c0re starts
|
|
||||||
# after that attempt with whatever credential is on disk, while
|
|
||||||
# ./lib/store-retry.nix keeps retrying in the background.
|
|
||||||
before = [ "hive-c0re.service" ];
|
|
||||||
wantedBy = [
|
|
||||||
"multi-user.target"
|
|
||||||
"hive-c0re.service"
|
|
||||||
];
|
|
||||||
path = [
|
|
||||||
baoDeploy.package
|
|
||||||
pkgs.coreutils
|
|
||||||
];
|
|
||||||
# ./lib/store-retry.nix.
|
|
||||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
|
||||||
serviceConfig = storeRetry.serviceConfig // {
|
|
||||||
Type = "oneshot";
|
|
||||||
RemainAfterExit = true;
|
|
||||||
# What actually bounds each attempt at the reads below. Stated here
|
|
||||||
# rather than left to systemd's default, so the number a boot waits on
|
|
||||||
# is in the file that waits.
|
|
||||||
TimeoutStartSec = 30;
|
|
||||||
};
|
|
||||||
environment = {
|
|
||||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
|
||||||
BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile;
|
|
||||||
BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile;
|
|
||||||
}
|
|
||||||
# Absent means the system trust store, which is what a deployment with a
|
|
||||||
# real CA wants and what a self-signed one must not be left with.
|
|
||||||
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
|
|
||||||
BAO_CACERT = baoDeploy.serverCaFile;
|
|
||||||
};
|
|
||||||
script = ''
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
${atomicWriteSecret}
|
|
||||||
|
|
||||||
# `bao`'s own message is the only thing separating a missing value from
|
|
||||||
# a refused identity from an unreachable host. This unit's degraded
|
|
||||||
# mode is correct for all three, so it reports which one rather than
|
|
||||||
# asserting all three in a sentence of ours.
|
|
||||||
err="$(mktemp)"
|
|
||||||
trap 'rm -f "$err"' EXIT
|
|
||||||
|
|
||||||
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
|
|
||||||
# variables above only decide which certificate the TLS handshake
|
|
||||||
# presents; without a token `bao` asks its token helper instead, and
|
|
||||||
# that is a `sh` this unit's `path` does not carry. `-token-only`
|
|
||||||
# answers on stdout and skips the helper on both sides.
|
|
||||||
#
|
|
||||||
# Fails LOUDLY, unlike the reads below: the three states a login
|
|
||||||
# failure covers — store not up, sealed, role not written yet — are all
|
|
||||||
# things a retry fixes, and `Restart=on-failure` above is what retries.
|
|
||||||
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
|
|
||||||
echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2
|
|
||||||
if [ -s "$err" ]; then
|
|
||||||
cat "$err" >&2
|
|
||||||
else
|
|
||||||
echo "bao failed without writing a diagnostic." >&2
|
|
||||||
fi
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
export BAO_TOKEN
|
|
||||||
|
|
||||||
# Two reads of one object rather than one `-format=json` parsed with
|
|
||||||
# `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot
|
|
||||||
# actually disagree — the client id is derived from this hive's name, so
|
|
||||||
# a rotation landing between these two calls changes the secret and
|
|
||||||
# rewrites the same id.
|
|
||||||
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then
|
|
||||||
echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2
|
|
||||||
if [ -s "$err" ]; then
|
|
||||||
cat "$err" >&2
|
|
||||||
else
|
|
||||||
echo "bao failed without writing a diagnostic." >&2
|
|
||||||
fi
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then
|
|
||||||
echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2
|
|
||||||
if [ -s "$err" ]; then
|
|
||||||
cat "$err" >&2
|
|
||||||
else
|
|
||||||
echo "bao failed without writing a diagnostic." >&2
|
|
||||||
fi
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Both or neither: a client that finds one of the two comes up "fine"
|
|
||||||
# and never connects.
|
|
||||||
if [ -z "$secret" ] || [ -z "$client_id" ]; then
|
|
||||||
echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
install -d -m 0755 ${lib.escapeShellArg credentialDir}
|
|
||||||
|
|
||||||
atomic_write_secret 0600 "" ${lib.escapeShellArg secretFile} "$secret"
|
|
||||||
|
|
||||||
# `0644` on purpose: an OIDC client id is presented to the token
|
|
||||||
# endpoint on every connection and is public by construction.
|
|
||||||
atomic_write_secret 0644 "" ${lib.escapeShellArg clientIdFile} "$client_id"
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
})
|
|
||||||
];
|
|
||||||
}
|
|
||||||
|
|
@ -333,8 +333,7 @@ in
|
||||||
# from an unprivileged daemon without widening either.
|
# from an unprivileged daemon without widening either.
|
||||||
# The gate is the identity, never `deploy.bao.enable`: a hive that
|
# The gate is the identity, never `deploy.bao.enable`: a hive that
|
||||||
# reads a store on another machine holds one of these and runs no
|
# reads a store on another machine holds one of these and runs no
|
||||||
# store. ./glue-queue-agent-credential.nix gates its own reader the
|
# store. ../swarm-otel.nix gates its own reader the same way.
|
||||||
# same way.
|
|
||||||
lib.optionals baoClientIdentity [
|
lib.optionals baoClientIdentity [
|
||||||
"bao-client.pem:${baoDeploy.clientCertFile}"
|
"bao-client.pem:${baoDeploy.clientCertFile}"
|
||||||
"bao-client-key.pem:${baoDeploy.clientKeyFile}"
|
"bao-client-key.pem:${baoDeploy.clientKeyFile}"
|
||||||
|
|
|
||||||
|
|
@ -11,9 +11,8 @@ let
|
||||||
cfg = config.services.hyperhive.c0re;
|
cfg = config.services.hyperhive.c0re;
|
||||||
baoDeploy = config.services.hyperhive.deploy.bao;
|
baoDeploy = config.services.hyperhive.deploy.bao;
|
||||||
baoCfg = config.services.hyperhive.swarm.bao;
|
baoCfg = config.services.hyperhive.swarm.bao;
|
||||||
# The same test ../glue-queue-agent-credential.nix applies, and for the same
|
# A reader is defined by holding a certificate the store accepts, not by
|
||||||
# reason: a reader is defined by holding a certificate the store accepts,
|
# sharing a host with the store. Deriving this from
|
||||||
# not by sharing a host with the store. Deriving this from
|
|
||||||
# `deploy.bao.enable` would be the co-location assumption itself.
|
# `deploy.bao.enable` would be the co-location assumption itself.
|
||||||
haveBaoClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
|
haveBaoClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
|
||||||
in
|
in
|
||||||
|
|
@ -222,30 +221,14 @@ in
|
||||||
# the one the roster gave it.
|
# the one the roster gave it.
|
||||||
HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}";
|
HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}";
|
||||||
}
|
}
|
||||||
// {
|
|
||||||
# Where ../glue-queue-agent-credential.nix lands the AGENTS' queue
|
|
||||||
# credential. Read by `hive_c0re::lifecycle::host_config`, which stats the
|
|
||||||
# two files and forwards them into each container as systemd credentials.
|
|
||||||
# Never read for its contents here: the secret is `0600` root-owned and
|
|
||||||
# this daemon is `hive-core`, which is exactly why the transport is a
|
|
||||||
# credential rather than a bind mount.
|
|
||||||
HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR = toString config.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
|
|
||||||
}
|
|
||||||
//
|
//
|
||||||
# The agents' half of the same queue, forwarded by `hive_c0re::meta` into
|
# The agents' half of the same queue, forwarded by `hive_c0re::meta` into
|
||||||
# every container. A pair, gated together, because an agent that got one of
|
# every container.
|
||||||
# them would report a half-configured queue instead of none.
|
|
||||||
#
|
#
|
||||||
# ⚠️ By name, like `HIVE_C0RE_NATS_URL` above, and never a loopback
|
# ⚠️ By name, like `HIVE_C0RE_NATS_URL` above, and never a loopback
|
||||||
# address: inside a container `127.0.0.1` is the agent itself.
|
# address: inside a container `127.0.0.1` is the agent itself.
|
||||||
lib.optionalAttrs
|
lib.optionalAttrs (config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null) {
|
||||||
(
|
|
||||||
config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl != null
|
|
||||||
&& config.services.hyperhive.swarm.statusPublish.tokenEndpoint != null
|
|
||||||
)
|
|
||||||
{
|
|
||||||
HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl;
|
HIVE_AGENT_NATS_URL = config.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl;
|
||||||
HIVE_AGENT_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
|
|
||||||
}
|
}
|
||||||
//
|
//
|
||||||
# Where an agent reads the swarm's logs. Gated on the store identity below,
|
# Where an agent reads the swarm's logs. Gated on the store identity below,
|
||||||
|
|
|
||||||
|
|
@ -352,22 +352,14 @@ in
|
||||||
readOnly = true;
|
readOnly = true;
|
||||||
default = "-agent";
|
default = "-agent";
|
||||||
description = ''
|
description = ''
|
||||||
Suffix appended to a hive's own client id to name the client its
|
Suffix the queue's auth-callout responder refuses on a hive's client
|
||||||
*agent containers* present — agents on hive `alpha` all present
|
id: `${config.services.hyperhive.swarm.authelia.hiveClientPrefix}alpha${config.services.hyperhive.swarm.authelia.agentClientSuffix}`
|
||||||
`${config.services.hyperhive.swarm.authelia.hiveClientPrefix}alpha${config.services.hyperhive.swarm.authelia.agentClientSuffix}`.
|
is denied, never read as hive `alpha-agent`. No client is minted with
|
||||||
Read-only for the same reason as `hiveClientPrefix`, and read by the
|
it; an agent authenticates to the queue with its own secret.
|
||||||
same consumer with the same failure mode: a split spelling denies
|
|
||||||
every agent as a timeout.
|
|
||||||
|
|
||||||
A suffix on the hive's id rather than a prefix of its own, because
|
Read-only for the same reason as `hiveClientPrefix`. A hive named with
|
||||||
`agent-alpha` reads as *the agent named alpha* — which is precisely
|
this suffix is refused at evaluation, since the queue would refuse its
|
||||||
what this identity does not say.
|
own client id.
|
||||||
|
|
||||||
One id per hive rather than per agent, because agents are created at
|
|
||||||
runtime and a per-agent client would make creating one a config
|
|
||||||
change plus a reload. The consequence is that this identity says
|
|
||||||
*which hive* an agent belongs to and never *which agent* — the
|
|
||||||
broker cannot tell two agents on one hive apart.
|
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -230,8 +230,8 @@ let
|
||||||
# The secret publisher runs in this container, beside the mint whose secrets
|
# The secret publisher runs in this container, beside the mint whose secrets
|
||||||
# it copies. It needs a store identity, and a reader is defined by holding a
|
# it copies. It needs a store identity, and a reader is defined by holding a
|
||||||
# certificate the store accepts, never by standing next to it — the rule
|
# certificate the store accepts, never by standing next to it — the rule
|
||||||
# ./glue-queue-agent-credential.nix states in full. `deploy.bao.enable` here
|
# ./swarm-bao.nix's `deploy.bao` options block states in full.
|
||||||
# would be the co-location assumption itself.
|
# `deploy.bao.enable` here would be the co-location assumption itself.
|
||||||
publisherCfg = deployCfg.swarm-secret-publisher;
|
publisherCfg = deployCfg.swarm-secret-publisher;
|
||||||
publishes =
|
publishes =
|
||||||
publisherCfg.enable
|
publisherCfg.enable
|
||||||
|
|
@ -300,71 +300,6 @@ let
|
||||||
accessTokenSignedResponseAlg = "RS256";
|
accessTokenSignedResponseAlg = "RS256";
|
||||||
}) hyperhiveCfg.swarm.hives;
|
}) hyperhiveCfg.swarm.hives;
|
||||||
|
|
||||||
# The one URL an agent's token has to be authorised at beyond the queue:
|
|
||||||
# the log store's machine query route. Read off `swarm.victorialogs.domain`
|
|
||||||
# — an unconditional option, declared whether or not this host runs the
|
|
||||||
# store — rather than restated, because it is also the string `swarm-logs`
|
|
||||||
# requests as its audience and the string `hive-c0re/environment.nix` hands
|
|
||||||
# an agent as its endpoint. Three readers, one formula; two spellings
|
|
||||||
# present as a valid token refused at the store.
|
|
||||||
logsQueryUrl = "https://${hyperhiveCfg.swarm.victorialogs.domain}/select/logsql/query";
|
|
||||||
|
|
||||||
# The identity an agent container presents to the swarm's queue. One per
|
|
||||||
# HIVE, not per agent, and that is the load-bearing choice rather than a
|
|
||||||
# shortcut: agents are created at **runtime**, so anything minted per agent
|
|
||||||
# would make creating one a config change plus an authelia reload. Keyed on
|
|
||||||
# the hive, this list's length tracks the roster above it — deploy-time, like
|
|
||||||
# every other entry here.
|
|
||||||
#
|
|
||||||
# ⚠️ The cost, stated rather than discovered later: every agent on a hive
|
|
||||||
# presents the SAME client id, so the broker can tell *hives* apart and not
|
|
||||||
# *agents*. Deliberate and deferred — closing it requires the client list to
|
|
||||||
# stop being static, which is the same problem the users-database writer
|
|
||||||
# already solves for identities.
|
|
||||||
#
|
|
||||||
# 🏷️ The id EXTENDS the hive's own (`hive-<name>-agent`) rather than taking a
|
|
||||||
# prefix of its own (`agent-<name>`), because that reads as *the agent called
|
|
||||||
# `<name>`* — which is the one thing this identity does not carry. The cost is
|
|
||||||
# that the two ids are not distinguishable by prefix, so the responder's
|
|
||||||
# agent rule must be tried BEFORE its hive rule; `policy.rs` says so at the
|
|
||||||
# match site, and the assertion below covers the case that ordering cannot.
|
|
||||||
#
|
|
||||||
# Mirrors `hiveClients` field for field so the two stay comparable. The
|
|
||||||
# signing algorithm is not load-bearing here — this client's consumers are
|
|
||||||
# the queue's auth-callout responder, which *introspects* rather than
|
|
||||||
# verifying offline, and authelia's own authz endpoint — but it matches its
|
|
||||||
# sibling rather than inventing a second answer to a question nobody asked.
|
|
||||||
agentClients = lib.mapAttrsToList (name: _: {
|
|
||||||
id = "${cfg.hiveClientPrefix}${name}${cfg.agentClientSuffix}";
|
|
||||||
description = "HyperHive agents on hive ${name}";
|
|
||||||
kind = "machine";
|
|
||||||
redirectUris = [ ];
|
|
||||||
# ⚠️ **Two** audiences, where `hiveClients` has one, because an agent
|
|
||||||
# presents this token at two different kinds of consumer. The queue
|
|
||||||
# introspects it and reads the client id; the log store's gateway
|
|
||||||
# location runs `auth_request` against authelia, which authorises a
|
|
||||||
# bearer token **by the URL being requested** and refuses one whose
|
|
||||||
# audience does not name it. Measured, not assumed: without the URL
|
|
||||||
# registered here the token endpoint answers `invalid_target` for it,
|
|
||||||
# and a token minted with no audience at all is answered by the gateway
|
|
||||||
# with a bare 401 that names no cause.
|
|
||||||
audience = [
|
|
||||||
"${cfg.hiveClientPrefix}${name}${cfg.agentClientSuffix}"
|
|
||||||
logsQueryUrl
|
|
||||||
];
|
|
||||||
# Without it the authz endpoint refuses an otherwise valid token and
|
|
||||||
# blames the token rather than the missing grant — the same note
|
|
||||||
# `glue-swarm-otel-oidc-client.nix` carries over the collector's client.
|
|
||||||
bearerAuthz = true;
|
|
||||||
# Stated rather than left on authelia's default, for
|
|
||||||
# `glue-swarm-otel-oidc-client.nix`'s reason: authelia permits only
|
|
||||||
# basic / JWT methods for a confidential client holding this scope, and
|
|
||||||
# enforces it in the startup validator. `swarm-queue-client` sends the
|
|
||||||
# credential as basic auth, so this is also what the minter already does.
|
|
||||||
tokenEndpointAuthMethod = "client_secret_basic";
|
|
||||||
accessTokenSignedResponseAlg = "RS256";
|
|
||||||
}) hyperhiveCfg.swarm.hives;
|
|
||||||
|
|
||||||
# `swarm-authelia-bridge`'s own identity — distinct from
|
# `swarm-authelia-bridge`'s own identity — distinct from
|
||||||
# `swarm-controller`'s (`swarm-controller.nix`'s `queueClientId`). A
|
# `swarm-controller`'s (`swarm-controller.nix`'s `queueClientId`). A
|
||||||
# resource server introspecting a token proves its OWN identity to the
|
# resource server introspecting a token proves its OWN identity to the
|
||||||
|
|
@ -706,7 +641,7 @@ in
|
||||||
# wherever this module is, so `oidc.clients` is never actually empty
|
# wherever this module is, so `oidc.clients` is never actually empty
|
||||||
# — see `oidcEnabled`'s comment above.
|
# — see `oidcEnabled`'s comment above.
|
||||||
services.hyperhive.swarm.authelia.oidc.clients =
|
services.hyperhive.swarm.authelia.oidc.clients =
|
||||||
lib.optionals cfg.oidc.hiveIdentities (hiveClients ++ agentClients)
|
lib.optionals cfg.oidc.hiveIdentities hiveClients
|
||||||
++ [
|
++ [
|
||||||
bridgeClient
|
bridgeClient
|
||||||
forwarderClient
|
forwarderClient
|
||||||
|
|
@ -749,18 +684,11 @@ in
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# A hive whose name ENDS with the agent suffix mints an id the
|
# A hive whose name ENDS with the agent suffix mints an id the
|
||||||
# queue's responder reads as somebody else's agents:
|
# queue's responder reads as a hive's agent client, which it refuses:
|
||||||
# `hive-foo-agent` parses as *the agents of hive foo* before it
|
# `hive-foo-agent` parses as *the agents of hive foo* before it
|
||||||
# parses as *the hive foo-agent*, because the agent rule is the
|
# parses as *the hive foo-agent*, because the agent rule is the
|
||||||
# more specific one and therefore runs first.
|
# more specific one and therefore runs first. A NATS denial arrives
|
||||||
#
|
# as a timeout, so the symptom names nothing.
|
||||||
# ⚠️ NOT covered by the duplicate-id assertion above, and the gap is
|
|
||||||
# the interesting half. That one fires only when BOTH `foo` and
|
|
||||||
# `foo-agent` are on the roster, because only then are two clients
|
|
||||||
# actually named the same string. With `foo-agent` alone there is no
|
|
||||||
# duplicate and nothing to see — the hive simply receives its
|
|
||||||
# agents' grant instead of its own, and a NATS denial arrives as a
|
|
||||||
# timeout, so the symptom names nothing.
|
|
||||||
#
|
#
|
||||||
# Checkable here and nowhere downstream: the responder runs in a
|
# Checkable here and nowhere downstream: the responder runs in a
|
||||||
# container with no view of the roster (`policy.rs`'s `hive_name`
|
# container with no view of the roster (`policy.rs`'s `hive_name`
|
||||||
|
|
@ -774,10 +702,9 @@ in
|
||||||
+ lib.concatMapStringsSep ", " (h: "'${h}'") (
|
+ lib.concatMapStringsSep ", " (h: "'${h}'") (
|
||||||
lib.filter (h: lib.hasSuffix cfg.agentClientSuffix h) (lib.attrNames hyperhiveCfg.swarm.hives)
|
lib.filter (h: lib.hasSuffix cfg.agentClientSuffix h) (lib.attrNames hyperhiveCfg.swarm.hives)
|
||||||
)
|
)
|
||||||
+ " — names ending with '${cfg.agentClientSuffix}', the suffix that "
|
+ " — names ending with '${cfg.agentClientSuffix}', the suffix the "
|
||||||
+ "marks a hive's agent containers. Such a hive's own client id is "
|
+ "queue refuses as a hive's agent client. Such a hive's own client "
|
||||||
+ "indistinguishable from another hive's agent client, and the "
|
+ "id reads as one, and the queue refuses it. Rename the hive.";
|
||||||
+ "queue resolves it as the agents. Rename the hive.";
|
|
||||||
}
|
}
|
||||||
# The two obligations `authelia.bearer.authz` carries. Authelia
|
# The two obligations `authelia.bearer.authz` carries. Authelia
|
||||||
# enforces both itself — but in its `preStart` validator, so a
|
# enforces both itself — but in its `preStart` validator, so a
|
||||||
|
|
|
||||||
|
|
@ -808,9 +808,9 @@ let
|
||||||
}
|
}
|
||||||
'';
|
'';
|
||||||
|
|
||||||
# One gateway-leaf issuer per hive in the swarm directory, for the reason
|
# One gateway-leaf issuer per hive in the swarm directory: one subject for
|
||||||
# `perHiveReaders` below is per hive. One name for the pki role, the policy
|
# every hive could ask for any hive's certificate. One name for the pki role,
|
||||||
# and the cert-auth role, as for the readers.
|
# the policy and the cert-auth role, as for the readers below.
|
||||||
hiveLeafIssuers = lib.mapAttrsToList (hiveName: hive: {
|
hiveLeafIssuers = lib.mapAttrsToList (hiveName: hive: {
|
||||||
name = "${baoDeploy.hiveLeafPkiRolePrefix}-${hiveName}";
|
name = "${baoDeploy.hiveLeafPkiRolePrefix}-${hiveName}";
|
||||||
cn = "${baoDeploy.hiveLeafCommonNamePrefix}-${hiveName}";
|
cn = "${baoDeploy.hiveLeafCommonNamePrefix}-${hiveName}";
|
||||||
|
|
@ -897,48 +897,21 @@ let
|
||||||
# `swarm/agents/*` AND `swarm/hives/<hive>/*`, and nothing under
|
# `swarm/agents/*` AND `swarm/hives/<hive>/*`, and nothing under
|
||||||
# `swarm/services/`.
|
# `swarm/services/`.
|
||||||
# Principals presenting one certificate are ONE principal to bao, so sharing
|
# Principals presenting one certificate are ONE principal to bao, so sharing
|
||||||
# the hive's leaf would give each of them everything that leaf reads: the
|
# the hive's leaf would give each of them everything that leaf reads: a unit
|
||||||
# unit that fetches this hive's queue credential could fetch every agent
|
# that fetches a service's OIDC client secret could fetch every agent
|
||||||
# credential in the swarm, and a unit that fetches a service's OIDC client
|
# credential in the swarm, and could not fetch its own secret at all, as
|
||||||
# secret could not fetch it at all, as that path is not the hive's to read.
|
# that path is not the hive's to read.
|
||||||
#
|
#
|
||||||
# Each gets its own leaf, its own cert-auth role and a policy holding the
|
# Each gets its own leaf, its own cert-auth role and a policy holding the
|
||||||
# one `secret/data/…` path that principal's script actually names — spelled
|
# one `secret/data/…` path that principal's script actually names — spelled
|
||||||
# to the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText`
|
# to the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText`
|
||||||
# above gives for doing the same.
|
# above gives for doing the same.
|
||||||
#
|
|
||||||
# ⚠️ The queue-credential reader is PER HIVE and the OIDC readers are not,
|
|
||||||
# and that asymmetry is the shape of the paths rather than a preference. A
|
|
||||||
# queue credential lives under `swarm/hives/<name>/`, and every hive runs a
|
|
||||||
# reader for its own; an OIDC client secret lives under
|
|
||||||
# `swarm/services/<client-id>/` and a swarm registers each of those exactly
|
|
||||||
# once. A single role for a per-hive reader would have to name `hives/*` to
|
|
||||||
# serve every hive — which would let one hive read another's queue
|
|
||||||
# credential, a reach no hive needs.
|
|
||||||
readStanza = path: ''
|
readStanza = path: ''
|
||||||
path "${path}" {
|
path "${path}" {
|
||||||
capabilities = ["read"]
|
capabilities = ["read"]
|
||||||
}
|
}
|
||||||
'';
|
'';
|
||||||
|
|
||||||
# One reader per hive in the swarm directory. Written from the directory
|
|
||||||
# rather than from this host's own name because the policy is written where
|
|
||||||
# the STORE is and the reader runs where its hive is, so this host's name is
|
|
||||||
# not the reader's.
|
|
||||||
perHiveReaders =
|
|
||||||
{ rolePrefix, cnPrefix, ... }@spec:
|
|
||||||
lib.mapAttrsToList (hiveName: _: {
|
|
||||||
name = "${rolePrefix}-${hiveName}";
|
|
||||||
cn = "${cnPrefix}-${hiveName}";
|
|
||||||
policyText = readStanza (spec.path hiveName);
|
|
||||||
}) hyperhiveCfg.swarm.hives;
|
|
||||||
|
|
||||||
queueAgentReaders = perHiveReaders {
|
|
||||||
rolePrefix = "swarm-queue-agent";
|
|
||||||
cnPrefix = baoDeploy.queueAgentCommonNamePrefix;
|
|
||||||
path = h: "${credentialMountPath}/data/swarm/hives/${h}/queue/agent";
|
|
||||||
};
|
|
||||||
|
|
||||||
# Singletons, so a list of one rather than a second shape: the granting unit
|
# Singletons, so a list of one rather than a second shape: the granting unit
|
||||||
# below takes a list either way and the units stay identical apart from
|
# below takes a list either way and the units stay identical apart from
|
||||||
# the objects they write.
|
# the objects they write.
|
||||||
|
|
@ -1257,9 +1230,9 @@ let
|
||||||
|
|
||||||
# One unit per PRINCIPAL, not one unit for all of them: these names are
|
# One unit per PRINCIPAL, not one unit for all of them: these names are
|
||||||
# operator-facing strings, the same reason `swarm-bao-secret-publisher-policy`
|
# operator-facing strings, the same reason `swarm-bao-secret-publisher-policy`
|
||||||
# below gives for not being more script inside the controller's unit. A hive
|
# below gives for not being more script inside the controller's unit. A
|
||||||
# whose queue credential never arrives is diagnosed by the state of the unit named
|
# reader whose secret never arrives is diagnosed by the state of the unit
|
||||||
# after it.
|
# named after it.
|
||||||
readerPolicyUnit =
|
readerPolicyUnit =
|
||||||
description: objects:
|
description: objects:
|
||||||
granterPolicyUnit description (
|
granterPolicyUnit description (
|
||||||
|
|
@ -1589,7 +1562,7 @@ in
|
||||||
# `serverCaFile` live here too, and they are set by whoever authenticates TO
|
# `serverCaFile` live here too, and they are set by whoever authenticates TO
|
||||||
# the store — which includes a host that runs none of it. A reader is defined
|
# the store — which includes a host that runs none of it. A reader is defined
|
||||||
# by holding a certificate the store accepts, never by sharing a host with
|
# by holding a certificate the store accepts, never by sharing a host with
|
||||||
# one; ./glue-queue-agent-credential.nix gates on exactly that and not on
|
# one; ./swarm-otel.nix's store reader gates on exactly that and not on
|
||||||
# `enable`.
|
# `enable`.
|
||||||
# Reading this block as store-runner-only is what makes an off-host reader
|
# Reading this block as store-runner-only is what makes an off-host reader
|
||||||
# look inexpressible when it is already supported.
|
# look inexpressible when it is already supported.
|
||||||
|
|
@ -2102,28 +2075,6 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
queueAgentCommonNamePrefix = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "swarm-bao-queue-agent";
|
|
||||||
example = "swarm-bao-queue-agent.svc";
|
|
||||||
description = ''
|
|
||||||
Prefix of the subject the store's per-hive queue-credential roles
|
|
||||||
accept. The role for hive `h` accepts `<prefix>-h` and grants read on
|
|
||||||
exactly `swarm/hives/h/queue/agent` — the client credential that hive's
|
|
||||||
agent containers authenticate to the swarm queue with.
|
|
||||||
|
|
||||||
A **prefix** rather than one subject because every hive runs its own
|
|
||||||
reader and the path it reads carries that hive's name. One subject for
|
|
||||||
all of them would need a `hives/*` grant, which would let one hive read
|
|
||||||
another hive's queue credential.
|
|
||||||
|
|
||||||
⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its
|
|
||||||
fixed-subject siblings are: cert auth trusts the CA and
|
|
||||||
`allowed_common_names` is the whole narrowing, so a hive whose own leaf
|
|
||||||
carried one of these spellings would receive that grant.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
hiveLeafCommonNamePrefix = lib.mkOption {
|
hiveLeafCommonNamePrefix = lib.mkOption {
|
||||||
type = lib.types.str;
|
type = lib.types.str;
|
||||||
default = "swarm-bao-hive-leaf";
|
default = "swarm-bao-hive-leaf";
|
||||||
|
|
@ -2134,9 +2085,8 @@ in
|
||||||
`<servicesPkiMountPath>/issue/<hiveLeafPkiRolePrefix>-h`, which issues
|
`<servicesPkiMountPath>/issue/<hiveLeafPkiRolePrefix>-h`, which issues
|
||||||
for `h`'s domain and `*.` that domain and nothing else.
|
for `h`'s domain and `*.` that domain and nothing else.
|
||||||
|
|
||||||
Per hive for the reason
|
Per hive because one subject for every hive could ask for any hive's
|
||||||
{option}`services.hyperhive.deploy.bao.queueAgentCommonNamePrefix` is:
|
certificate.
|
||||||
one subject for every hive could ask for any hive's certificate.
|
|
||||||
|
|
||||||
⚠️ Reserved as a hive name by ./swarm.nix, per hive, like its siblings.
|
⚠️ Reserved as a hive name by ./swarm.nix, per hive, like its siblings.
|
||||||
'';
|
'';
|
||||||
|
|
@ -2163,7 +2113,7 @@ in
|
||||||
|
|
||||||
Its grant is one path,
|
Its grant is one path,
|
||||||
`swarm/services/<grafana client id>/oidc/client`, and read only. Not
|
`swarm/services/<grafana client id>/oidc/client`, and read only. Not
|
||||||
per hive, unlike the two prefixes above: an OIDC client is registered
|
per hive, unlike the prefix above: an OIDC client is registered
|
||||||
once per swarm, so the path names the service and never a hive.
|
once per swarm, so the path names the service and never a hive.
|
||||||
|
|
||||||
⚠️ Same collision as its siblings, and the same answer: ./swarm.nix
|
⚠️ Same collision as its siblings, and the same answer: ./swarm.nix
|
||||||
|
|
@ -2206,37 +2156,6 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
queueAgentClientCertFile = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.str;
|
|
||||||
default = null;
|
|
||||||
example = "/var/lib/swarm-bao-pki/queue-agent.pem";
|
|
||||||
description = ''
|
|
||||||
Certificate the unit that fetches this hive's queue client credential
|
|
||||||
presents to the store. Its subject must be
|
|
||||||
`<queueAgentCommonNamePrefix>-<this hive's name>`; cert auth matches on
|
|
||||||
the CN and the role accepts nothing else.
|
|
||||||
|
|
||||||
⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
|
|
||||||
Pointing this at the hive's own leaf would evaluate, deploy and log in —
|
|
||||||
and hand a unit that reads one credential a certificate that reads every
|
|
||||||
agent credential in the swarm and this hive's own secrets.
|
|
||||||
|
|
||||||
No default here: ./glue-bao-tls.nix points it at the leaf it mints,
|
|
||||||
wherever this host mints its own PKI. A deployment that reads a store on
|
|
||||||
another host names it by hand, alongside the leaf itself.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
queueAgentClientKeyFile = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.str;
|
|
||||||
default = null;
|
|
||||||
example = "/var/lib/swarm-bao-pki/queue-agent-key.pem";
|
|
||||||
description = ''
|
|
||||||
Private key for
|
|
||||||
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile`.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
grafanaOidcClientCertFile = lib.mkOption {
|
grafanaOidcClientCertFile = lib.mkOption {
|
||||||
type = lib.types.nullOr lib.types.str;
|
type = lib.types.nullOr lib.types.str;
|
||||||
default = null;
|
default = null;
|
||||||
|
|
@ -2246,8 +2165,10 @@ in
|
||||||
to the store. Its subject must be
|
to the store. Its subject must be
|
||||||
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`.
|
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`.
|
||||||
|
|
||||||
⚠️ Not the hive's own leaf, for the reason
|
⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
|
||||||
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
|
Pointing this at the hive's own leaf would evaluate, deploy and log in —
|
||||||
|
and hand a unit that reads one secret a certificate that reads every
|
||||||
|
agent credential in the swarm and this hive's own secrets.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -2271,7 +2192,7 @@ in
|
||||||
{option}`services.hyperhive.deploy.bao.otelOidcCommonName`.
|
{option}`services.hyperhive.deploy.bao.otelOidcCommonName`.
|
||||||
|
|
||||||
⚠️ Not the hive's own leaf, for the reason
|
⚠️ Not the hive's own leaf, for the reason
|
||||||
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
|
{option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile` gives.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -2295,7 +2216,7 @@ in
|
||||||
{option}`services.hyperhive.deploy.bao.forwarderOidcCommonName`.
|
{option}`services.hyperhive.deploy.bao.forwarderOidcCommonName`.
|
||||||
|
|
||||||
⚠️ Not the hive's own leaf, for the reason
|
⚠️ Not the hive's own leaf, for the reason
|
||||||
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
|
{option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile` gives.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -3543,12 +3464,10 @@ in
|
||||||
'';
|
'';
|
||||||
|
|
||||||
# 🩸 The principals that each log in with a leaf of their own, not as the
|
# 🩸 The principals that each log in with a leaf of their own, not as the
|
||||||
# hive. See the block around `perHiveReaders` for what each grant is and
|
# hive. See the block around `readStanza` for what each grant is; the
|
||||||
# why the first is per hive; the units are identical apart from the
|
# units are identical apart from the objects they write, so the shape
|
||||||
# objects they write, so the shape lives in `readerPolicyUnit` and the
|
# lives in `readerPolicyUnit` and the difference is visible here in one
|
||||||
# difference is visible here in one line each.
|
# line each.
|
||||||
systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders;
|
|
||||||
|
|
||||||
systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders;
|
systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders;
|
||||||
|
|
||||||
systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders;
|
systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders;
|
||||||
|
|
@ -3693,7 +3612,6 @@ in
|
||||||
"swarm-bao-controller-policy.service"
|
"swarm-bao-controller-policy.service"
|
||||||
"swarm-bao-secret-publisher-policy.service"
|
"swarm-bao-secret-publisher-policy.service"
|
||||||
"swarm-bao-matrix-ctl-policy.service"
|
"swarm-bao-matrix-ctl-policy.service"
|
||||||
"swarm-bao-queue-agent-policy.service"
|
|
||||||
"swarm-bao-grafana-oidc-policy.service"
|
"swarm-bao-grafana-oidc-policy.service"
|
||||||
"swarm-bao-otel-oidc-policy.service"
|
"swarm-bao-otel-oidc-policy.service"
|
||||||
"swarm-bao-forwarder-oidc-policy.service"
|
"swarm-bao-forwarder-oidc-policy.service"
|
||||||
|
|
|
||||||
|
|
@ -282,8 +282,8 @@ let
|
||||||
);
|
);
|
||||||
|
|
||||||
# A reader of the store is defined by holding a certificate the store
|
# A reader of the store is defined by holding a certificate the store
|
||||||
# accepts, never by standing next to it — the rule
|
# accepts, never by standing next to it — the rule ./swarm-bao.nix's
|
||||||
# ./glue-queue-agent-credential.nix states in full.
|
# `deploy.bao` options block states in full.
|
||||||
#
|
#
|
||||||
# Also asserted rather than gating anything, and that is the ruling this
|
# Also asserted rather than gating anything, and that is the ruling this
|
||||||
# module's second delivery route was deleted under: the store exists so that
|
# module's second delivery route was deleted under: the store exists so that
|
||||||
|
|
@ -722,10 +722,10 @@ in
|
||||||
# to be wrong, and a gate whose false arm is a Grafana with no login.
|
# to be wrong, and a gate whose false arm is a Grafana with no login.
|
||||||
# Recorded in docs/swarm/secrets.md.
|
# Recorded in docs/swarm/secrets.md.
|
||||||
#
|
#
|
||||||
# Shaped after ./glue-queue-agent-credential.nix, because it is a store
|
# It is a store reader, and this is what the store's readers do: a cert
|
||||||
# reader and this is what the store's readers do: a cert login that fails
|
# login that fails LOUDLY, since every state it fails on is one a retry
|
||||||
# LOUDLY, since every state it fails on is one a retry fixes, then a read
|
# fixes, then a read that degrades QUIETLY, since no retry turns "no value
|
||||||
# that degrades QUIETLY, since no retry turns "no value there" into a value.
|
# there" into a value.
|
||||||
#
|
#
|
||||||
# ⚠️ An ABSENT secret is "not published yet", not a fault of this host's.
|
# ⚠️ An ABSENT secret is "not published yet", not a fault of this host's.
|
||||||
# The publisher runs in authelia's container and authelia mints on its
|
# The publisher runs in authelia's container and authelia mints on its
|
||||||
|
|
|
||||||
|
|
@ -873,7 +873,7 @@ in
|
||||||
# terminal rows and its turn-state header
|
# terminal rows and its turn-state header
|
||||||
# (`hive-agent::swarm_agent_state`), two subject families so a
|
# (`hive-agent::swarm_agent_state`), two subject families so a
|
||||||
# subscriber following one agent's terminal is not also handed
|
# subscriber following one agent's terminal is not also handed
|
||||||
# every header. The hive's shared agent client gets nothing.
|
# every header.
|
||||||
#
|
#
|
||||||
# `$$`, not `$`: systemd substitutes `$NAME` in `ExecStart`
|
# `$$`, not `$`: systemd substitutes `$NAME` in `ExecStart`
|
||||||
# whether or not it is quoted, so a single dollar reaches the
|
# whether or not it is quoted, so a single dollar reaches the
|
||||||
|
|
|
||||||
|
|
@ -232,12 +232,11 @@ let
|
||||||
haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null;
|
haveCollectorSecret = deployCfg.swarm-otel.clientSecretFile != null;
|
||||||
|
|
||||||
# A reader of the store is defined by holding a certificate the store
|
# A reader of the store is defined by holding a certificate the store
|
||||||
# accepts, never by standing next to it — the rule
|
# accepts, never by standing next to it — the rule ./swarm-bao.nix's
|
||||||
# ./glue-queue-agent-credential.nix states in full. Unlike Grafana's identical-
|
# `deploy.bao` options block states in full. Unlike Grafana's identical-
|
||||||
# looking flag, this one does not gate an assertion on its own: the
|
# looking flag, this one does not gate an assertion on its own: the
|
||||||
# store-reading unit below simply does not render without it, the same choice
|
# store-reading unit below simply does not render without it, because a
|
||||||
# ./glue-queue-agent-credential.nix makes for its own reader, because a collector with no client identity is
|
# collector with no client identity is `haveCollectorSecret = false` above, and that is already a supported,
|
||||||
# `haveCollectorSecret = false` above, and that is already a supported,
|
|
||||||
# merely degraded shape rather than a service with no way in at all. What
|
# merely degraded shape rather than a service with no way in at all. What
|
||||||
# IS asserted is narrower and lives in `assertions` below — see
|
# IS asserted is narrower and lives in `assertions` below — see
|
||||||
# `hiveReaderIdentity`.
|
# `hiveReaderIdentity`.
|
||||||
|
|
@ -539,7 +538,7 @@ in
|
||||||
# exists so a host holds ONE out-of-band secret — its client certificate —
|
# exists so a host holds ONE out-of-band secret — its client certificate —
|
||||||
# and reads everything else with it. Recorded in docs/swarm/secrets.md.
|
# and reads everything else with it. Recorded in docs/swarm/secrets.md.
|
||||||
#
|
#
|
||||||
# Shaped after ./glue-queue-agent-credential.nix: a cert login that fails
|
# Shaped after ./swarm-grafana.nix's store reader: a cert login that fails
|
||||||
# LOUDLY, since every state it fails on is one a retry fixes, then a read
|
# LOUDLY, since every state it fails on is one a retry fixes, then a read
|
||||||
# that degrades QUIETLY, since no retry turns "no value there" into a
|
# that degrades QUIETLY, since no retry turns "no value there" into a
|
||||||
# value.
|
# value.
|
||||||
|
|
|
||||||
|
|
@ -12,15 +12,9 @@ let
|
||||||
hyperhiveCfg = config.services.hyperhive;
|
hyperhiveCfg = config.services.hyperhive;
|
||||||
hiveNames = lib.attrNames hyperhiveCfg.swarm.hives;
|
hiveNames = lib.attrNames hyperhiveCfg.swarm.hives;
|
||||||
|
|
||||||
# The client id agent containers present, per hive — composed exactly as
|
|
||||||
# ./swarm-authelia.nix composes it, from the same two read-only options, so a
|
|
||||||
# rename there cannot leave this spelling behind.
|
|
||||||
agentClientId =
|
|
||||||
hive:
|
|
||||||
"${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}${hyperhiveCfg.swarm.authelia.agentClientSuffix}";
|
|
||||||
|
|
||||||
# The hive's own client id, which hive-c0re publishes the hive's status with.
|
# The hive's own client id, which hive-c0re publishes the hive's status with.
|
||||||
# Composed as ./swarm-authelia.nix composes it, for the reason above.
|
# Composed exactly as ./swarm-authelia.nix composes it, from the same
|
||||||
|
# read-only option, so a rename there cannot leave this spelling behind.
|
||||||
hiveClientId = hive: "${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}";
|
hiveClientId = hive: "${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}";
|
||||||
|
|
||||||
# The swarm's own services, as opposed to its hives. One client for the whole
|
# The swarm's own services, as opposed to its hives. One client for the whole
|
||||||
|
|
@ -83,22 +77,14 @@ let
|
||||||
controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client";
|
controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client";
|
||||||
|
|
||||||
secrets =
|
secrets =
|
||||||
# `client_id` beside the value: an agent's id is derived per hive and has
|
|
||||||
# to be reconstructable from the store alone.
|
|
||||||
map (hive: {
|
|
||||||
clientId = agentClientId hive;
|
|
||||||
path = "secret/swarm/hives/${hive}/queue/agent";
|
|
||||||
withClientId = true;
|
|
||||||
}) hiveNames
|
|
||||||
# Each hive's own client, which that hive's hive-c0re reads under its own
|
# Each hive's own client, which that hive's hive-c0re reads under its own
|
||||||
# certificate. The nix half of
|
# certificate. The nix half of
|
||||||
# `swarm_secret_client::queue::hive_client_path`.
|
# `swarm_secret_client::queue::hive_client_path`.
|
||||||
++ map (hive: {
|
map (hive: {
|
||||||
clientId = hiveClientId hive;
|
clientId = hiveClientId hive;
|
||||||
path = "secret/swarm/hives/${hive}/oidc/client";
|
path = "secret/swarm/hives/${hive}/oidc/client";
|
||||||
}) hiveNames
|
}) hiveNames
|
||||||
# No `client_id`: a service's client id is the swarm-wide option both
|
# A service's client id is the swarm-wide option both ends already read.
|
||||||
# ends already read.
|
|
||||||
++ map (id: {
|
++ map (id: {
|
||||||
clientId = id;
|
clientId = id;
|
||||||
path = "secret/swarm/services/${id}/oidc/client";
|
path = "secret/swarm/services/${id}/oidc/client";
|
||||||
|
|
@ -120,9 +106,8 @@ in
|
||||||
default = false;
|
default = false;
|
||||||
description = ''
|
description = ''
|
||||||
Publish the OIDC client secrets this host mints into the swarm's
|
Publish the OIDC client secrets this host mints into the swarm's
|
||||||
secret store, so a hive that does not run authelia can read its
|
secret store, so a hive that does not run authelia, a swarm service
|
||||||
own and its agents' credentials and a swarm service or the
|
or the controller can read its own — from wherever it runs, this host
|
||||||
controller can read its own — from wherever it runs, this host
|
|
||||||
included. The unit, `swarm-secret-publish`, runs inside authelia's
|
included. The unit, `swarm-secret-publish`, runs inside authelia's
|
||||||
container, so it exists only where authelia runs.
|
container, so it exists only where authelia runs.
|
||||||
|
|
||||||
|
|
@ -137,10 +122,9 @@ in
|
||||||
publishing nothing. Whether the wiring is complete is the client
|
publishing nothing. Whether the wiring is complete is the client
|
||||||
identity's job (see `baoClientCertFile`), not this option's.
|
identity's job (see `baoClientCertFile`), not this option's.
|
||||||
|
|
||||||
Turning it off leaves every hive but this one without its agents'
|
Turning it off leaves every hive unable to publish its status, the
|
||||||
credential, every hive unable to publish its status, the swarm
|
swarm controller unable to start, the swarm's Grafana without any login
|
||||||
controller unable to start, the swarm's Grafana without any login at
|
at all, its collector pushing unauthenticated, the queue's responder
|
||||||
all, its collector pushing unauthenticated, the queue's responder
|
|
||||||
unable to start, the homeserver unable to start, and the forge's login
|
unable to start, the homeserver unable to start, and the forge's login
|
||||||
source without its secret.
|
source without its secret.
|
||||||
Each secret has exactly one route and this is the producer's
|
Each secret has exactly one route and this is the producer's
|
||||||
|
|
@ -185,9 +169,8 @@ in
|
||||||
internal = true;
|
internal = true;
|
||||||
default = secrets;
|
default = secrets;
|
||||||
description = ''
|
description = ''
|
||||||
Every secret the publisher copies: the minted client's id, the store
|
Every secret the publisher copies: the minted client's id and the store
|
||||||
path its reader fetches, and whether `client_id` is stored beside the
|
path its reader fetches. Derived from `swarm.*` alone, so it is the same list on every
|
||||||
value. Derived from `swarm.*` alone, so it is the same list on every
|
|
||||||
host; ./swarm-authelia.nix hands it to the unit in its container.
|
host; ./swarm-authelia.nix hands it to the unit in its container.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
|
||||||
|
|
@ -64,17 +64,14 @@ let
|
||||||
deployCfg.bao.notifyRouterCommonName
|
deployCfg.bao.notifyRouterCommonName
|
||||||
deployCfg.bao.forgePublisherCommonName
|
deployCfg.bao.forgePublisherCommonName
|
||||||
]
|
]
|
||||||
# The per-hive subjects, spelled out per hive rather than as the prefix.
|
# The per-hive subject, spelled out per hive rather than as the prefix.
|
||||||
# The prefix alone would reserve the wrong string: the role for hive
|
# The prefix alone would reserve the wrong string: the role for hive
|
||||||
# `h` accepts `<prefix>-h`, so the name a hive must not BE is that composed
|
# `h` accepts `<prefix>-h`, so the name a hive must not BE is that composed
|
||||||
# spelling, and a hive named it would present a leaf that role accepts.
|
# spelling, and a hive named it would present a leaf that role accepts.
|
||||||
#
|
#
|
||||||
# Every spelling is reserved for every declared hive, including the one the
|
# Every spelling is reserved for every declared hive, including the one the
|
||||||
# CN belongs to — a hive cannot be named after its own reader either.
|
# CN belongs to — a hive cannot be named after its own leaf issuer either.
|
||||||
++ lib.concatMap (hive: [
|
++ map (hive: "${deployCfg.bao.hiveLeafCommonNamePrefix}-${hive}") hiveNames;
|
||||||
"${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}"
|
|
||||||
"${deployCfg.bao.hiveLeafCommonNamePrefix}-${hive}"
|
|
||||||
]) hiveNames;
|
|
||||||
|
|
||||||
# Public hostnames of the swarm's own services, in declaration order.
|
# Public hostnames of the swarm's own services, in declaration order.
|
||||||
# `serviceDomains` below is this set sorted + deduplicated.
|
# `serviceDomains` below is this set sorted + deduplicated.
|
||||||
|
|
@ -477,9 +474,8 @@ in
|
||||||
# its own network namespace and resolver, so this is its own option, even
|
# its own network namespace and resolver, so this is its own option, even
|
||||||
# though by name it defaults to the same URL as `statusPublish.natsUrl`.
|
# though by name it defaults to the same URL as `statusPublish.natsUrl`.
|
||||||
#
|
#
|
||||||
# Only the address lives here. The credential does not: it is published per
|
# Only the address lives here. Each agent reads its own credential from the
|
||||||
# hive and read out of the store by ./glue-queue-agent-credential.nix, which
|
# store (`nix/agent-modules/queue-identity.nix`).
|
||||||
# owns `queue.agentCredentialDir` in the same namespace.
|
|
||||||
options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption {
|
options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption {
|
||||||
type = lib.types.nullOr lib.types.str;
|
type = lib.types.nullOr lib.types.str;
|
||||||
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
|
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
|
||||||
|
|
@ -496,10 +492,8 @@ in
|
||||||
agent's network namespace `127.0.0.1` is the agent, not this host,
|
agent's network namespace `127.0.0.1` is the agent, not this host,
|
||||||
and the queue's certificate names no address anyway.
|
and the queue's certificate names no address anyway.
|
||||||
|
|
||||||
Null means this hive's agents have not been given the queue's address.
|
Null means this hive's agents have not been given the queue's address,
|
||||||
Together with
|
and the harness is handed none.
|
||||||
{option}`services.hyperhive.swarm.statusPublish.tokenEndpoint` it is
|
|
||||||
what decides whether the harness is handed queue coordinates at all.
|
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -24,16 +24,13 @@ let
|
||||||
agentHarness
|
agentHarness
|
||||||
;
|
;
|
||||||
|
|
||||||
# The agent side of the swarm queue. Both coordinates set is the only state
|
# An `agent.nix` still setting the queue coordinates the agent module no
|
||||||
# in which the harness unit declares a credential at all, so the pair and
|
# longer declares.
|
||||||
# the empty fixture beside it are the two arms worth having.
|
|
||||||
agentQueue = agent {
|
agentQueue = agent {
|
||||||
queue.natsUrl = "nats://10.42.0.1:4222";
|
queue.natsUrl = "nats://10.42.0.1:4222";
|
||||||
queue.tokenEndpoint = "https://auth.t.local/api/oidc/token";
|
queue.tokenEndpoint = "https://auth.t.local/api/oidc/token";
|
||||||
};
|
};
|
||||||
|
|
||||||
agentNoQueue = shared.agentBare;
|
|
||||||
|
|
||||||
# The agent side of the swarm secret store. The address is the whole switch —
|
# The agent side of the swarm secret store. The address is the whole switch —
|
||||||
# it is both what generates the login check and what that check points at —
|
# it is both what generates the login check and what that check points at —
|
||||||
# so it and the empty fixture beside it are the two arms worth having.
|
# so it and the empty fixture beside it are the two arms worth having.
|
||||||
|
|
@ -106,21 +103,24 @@ let
|
||||||
ok = swarmMetricsIn agentLogsNoBao == null && swarmMetricsIn agentLogs != null;
|
ok = swarmMetricsIn agentLogsNoBao == null && swarmMetricsIn agentLogs != null;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# Both ids or neither: the secret authenticates nobody without the id it
|
# The hive forwards no shared queue credential, so the harness unit
|
||||||
# belongs to, and `swarm-logs` reads both.
|
# names none, whatever the agent's config says.
|
||||||
name = "an agent with queue coordinates imports both halves of its credential";
|
name = "an agent imports no hive-shared queue credential";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
c = (agentHarness agentQueue).serviceConfig.LoadCredential;
|
credentials = m: (agentHarness m).serviceConfig.LoadCredential or [ ];
|
||||||
in
|
in
|
||||||
builtins.elem "hive-queue-agent-secret" c && builtins.elem "hive-queue-agent-client-id" c;
|
!lib.any (lib.hasPrefix "hive-queue-agent-") (credentials agentQueue ++ credentials agentBao)
|
||||||
|
# Control: the lookup finds the store identity's credential.
|
||||||
|
&& lib.elem "hive-agent-bao-server-ca" (credentials agentBao);
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# An agent built before its hive was handed the queue's address. It
|
# Both options are refused, naming where the address
|
||||||
# must declare nothing rather than name a credential that never
|
# comes from now, rather than dropped silently.
|
||||||
# arrives.
|
name = "an agent setting the removed queue options is refused";
|
||||||
name = "an agent with no queue coordinates declares no credential";
|
ok =
|
||||||
ok = !((agentHarness agentNoQueue).serviceConfig ? LoadCredential);
|
lib.count (a: !a.assertion && lib.hasInfix "HIVE_AGENT_NATS_URL" a.message) agentQueue.assertions
|
||||||
|
== 2;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The harness exits when it cannot reach the queue. It has to come back
|
# The harness exits when it cannot reach the queue. It has to come back
|
||||||
|
|
|
||||||
|
|
@ -106,14 +106,11 @@ let
|
||||||
deploy.matrix.enable = true;
|
deploy.matrix.enable = true;
|
||||||
};
|
};
|
||||||
|
|
||||||
# The store with none of the readers beside it. Grafana and the collector are
|
# The store with none of the readers beside it: Grafana and the collector are
|
||||||
# simply off; the queue reader renders on any host holding
|
# off.
|
||||||
# its leaf, which ./glue-bao-tls.nix mints here, so that leaf is taken away.
|
|
||||||
baoGrantNoReaders = hiveWithController {
|
baoGrantNoReaders = hiveWithController {
|
||||||
deploy.bao.enable = true;
|
deploy.bao.enable = true;
|
||||||
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
|
||||||
deploy.bao.queueAgentClientCertFile = lib.mkForce null;
|
|
||||||
deploy.bao.queueAgentClientKeyFile = lib.mkForce null;
|
|
||||||
};
|
};
|
||||||
|
|
||||||
# The store with the forwarder's own pair taken away. The forwarder renders
|
# The store with the forwarder's own pair taken away. The forwarder renders
|
||||||
|
|
@ -131,8 +128,6 @@ let
|
||||||
deploy.swarm-otel.enable = true;
|
deploy.swarm-otel.enable = true;
|
||||||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||||
deploy.bao.queueAgentClientCertFile = "/etc/pki/bao-queue-agent.pem";
|
|
||||||
deploy.bao.queueAgentClientKeyFile = "/etc/pki/bao-queue-agent-key.pem";
|
|
||||||
deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem";
|
deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem";
|
||||||
deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem";
|
deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem";
|
||||||
deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem";
|
deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem";
|
||||||
|
|
@ -142,7 +137,6 @@ let
|
||||||
# The readers ./glue-bao-readers-policy-order.nix orders after their policy
|
# The readers ./glue-bao-readers-policy-order.nix orders after their policy
|
||||||
# units.
|
# units.
|
||||||
policyReaders = [
|
policyReaders = [
|
||||||
"swarm-bao-queue-agent"
|
|
||||||
"swarm-bao-grafana-oidc"
|
"swarm-bao-grafana-oidc"
|
||||||
"swarm-bao-otel-oidc"
|
"swarm-bao-otel-oidc"
|
||||||
];
|
];
|
||||||
|
|
@ -175,7 +169,6 @@ let
|
||||||
"swarm-bao-controller-policy"
|
"swarm-bao-controller-policy"
|
||||||
"swarm-bao-secret-publisher-policy"
|
"swarm-bao-secret-publisher-policy"
|
||||||
"swarm-bao-matrix-ctl-policy"
|
"swarm-bao-matrix-ctl-policy"
|
||||||
"swarm-bao-queue-agent-policy"
|
|
||||||
"swarm-bao-grafana-oidc-policy"
|
"swarm-bao-grafana-oidc-policy"
|
||||||
"swarm-bao-otel-oidc-policy"
|
"swarm-bao-otel-oidc-policy"
|
||||||
"swarm-bao-forwarder-oidc-policy"
|
"swarm-bao-forwarder-oidc-policy"
|
||||||
|
|
@ -599,19 +592,18 @@ let
|
||||||
# arms pin what each principal must NOT reach, so a later widening fails
|
# arms pin what each principal must NOT reach, so a later widening fails
|
||||||
# here instead of being noticed in a store.
|
# here instead of being noticed in a store.
|
||||||
{
|
{
|
||||||
name = "the queue-credential reader's grant is one hive's queue credential and nothing else";
|
# A hive holds no shared queue credential for its agents: no unit writes
|
||||||
|
# a role or policy for one, none reads one, and no granting unit names
|
||||||
|
# its path. The control: the same scan finds a reader's grant.
|
||||||
|
name = "no unit grants, reads or writes a hive-shared agent queue credential";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
s = baoGrantHere.systemd.services.swarm-bao-queue-agent-policy.script;
|
scripts = lib.mapAttrsToList (_: u: u.script or "") baoGrantWithConsumers.systemd.services;
|
||||||
in
|
in
|
||||||
lib.hasInfix "path \"secret/data/swarm/hives/h1/queue/agent\" {" s
|
!(baoGrantWithConsumers.systemd.services ? swarm-bao-queue-agent-policy)
|
||||||
&& lib.hasInfix "capabilities = [\"read\"]" s
|
&& !(baoGrantWithConsumers.systemd.services ? swarm-bao-queue-agent)
|
||||||
&& !(lib.hasInfix "secret/data/swarm/agents" s)
|
&& !(lib.any (lib.hasInfix "/queue/agent") scripts)
|
||||||
&& !(lib.hasInfix "secret/data/swarm/services" s)
|
&& lib.any (lib.hasInfix "/oidc/client") scripts;
|
||||||
&& !(lib.hasInfix "secret/data/swarm/hives/h1/*" s)
|
|
||||||
&& !(lib.hasInfix "secret/data/swarm/hives/h1/matrix" s)
|
|
||||||
&& !(lib.hasInfix "secret/data/swarm/hives/*" s)
|
|
||||||
&& !(lib.hasInfix "sys/policies/acl" s);
|
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# ⚠️ The client id is the path segment, so the negative arm that matters
|
# ⚠️ The client id is the path segment, so the negative arm that matters
|
||||||
|
|
@ -856,11 +848,7 @@ let
|
||||||
# through a token that carries it, and a token is minted by a cert-auth
|
# through a token that carries it, and a token is minted by a cert-auth
|
||||||
# role matching a CN. Distinct subjects is the whole mechanism — one
|
# role matching a CN. Distinct subjects is the whole mechanism — one
|
||||||
# subject for every reader is one principal however the policies read.
|
# subject for every reader is one principal however the policies read.
|
||||||
#
|
name = "each of the three readers logs in under a subject of its own";
|
||||||
# The per-hive subject carries the hive name because its path does; the
|
|
||||||
# service subjects do not, because an OIDC client is registered once
|
|
||||||
# per swarm. Pinned so neither shape is tidied into the other.
|
|
||||||
name = "each of the four readers logs in under a subject of its own";
|
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
subjectOf =
|
subjectOf =
|
||||||
|
|
@ -875,8 +863,7 @@ let
|
||||||
# the reason the three service principals above state.
|
# the reason the three service principals above state.
|
||||||
&& !(lib.hasInfix "auth/cert/certs/hive-" s);
|
&& !(lib.hasInfix "auth/cert/certs/hive-" s);
|
||||||
in
|
in
|
||||||
subjectOf "swarm-bao-queue-agent-policy" "swarm-queue-agent-h1" "swarm-bao-queue-agent-h1"
|
subjectOf "swarm-bao-grafana-oidc-policy" "swarm-grafana-oidc" "swarm-bao-grafana-oidc"
|
||||||
&& subjectOf "swarm-bao-grafana-oidc-policy" "swarm-grafana-oidc" "swarm-bao-grafana-oidc"
|
|
||||||
&& subjectOf "swarm-bao-otel-oidc-policy" "swarm-otel-oidc" "swarm-bao-otel-oidc"
|
&& subjectOf "swarm-bao-otel-oidc-policy" "swarm-otel-oidc" "swarm-bao-otel-oidc"
|
||||||
&& subjectOf "swarm-bao-forwarder-oidc-policy" "swarm-forwarder-oidc" "swarm-bao-forwarder-oidc";
|
&& subjectOf "swarm-bao-forwarder-oidc-policy" "swarm-forwarder-oidc" "swarm-bao-forwarder-oidc";
|
||||||
}
|
}
|
||||||
|
|
@ -889,7 +876,7 @@ let
|
||||||
#
|
#
|
||||||
# Each pair is asserted whole: a certificate with no key authenticates
|
# Each pair is asserted whole: a certificate with no key authenticates
|
||||||
# nothing, so a half-set pair is a reader that does not render.
|
# nothing, so a half-set pair is a reader that does not render.
|
||||||
name = "each of the four readers presents its own leaf, never the hive's";
|
name = "each of the three readers presents its own leaf, never the hive's";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
b = baoGrantWithConsumers.services.hyperhive.deploy.bao;
|
b = baoGrantWithConsumers.services.hyperhive.deploy.bao;
|
||||||
|
|
@ -898,8 +885,6 @@ let
|
||||||
b.clientKeyFile
|
b.clientKeyFile
|
||||||
];
|
];
|
||||||
own = [
|
own = [
|
||||||
b.queueAgentClientCertFile
|
|
||||||
b.queueAgentClientKeyFile
|
|
||||||
b.grafanaOidcClientCertFile
|
b.grafanaOidcClientCertFile
|
||||||
b.grafanaOidcClientKeyFile
|
b.grafanaOidcClientKeyFile
|
||||||
b.otelOidcClientCertFile
|
b.otelOidcClientCertFile
|
||||||
|
|
@ -915,7 +900,6 @@ let
|
||||||
lib.all (p: p != null) own
|
lib.all (p: p != null) own
|
||||||
&& !(lib.any (p: lib.elem p hiveLeaf) own)
|
&& !(lib.any (p: lib.elem p hiveLeaf) own)
|
||||||
&& lib.length (lib.unique own) == lib.length own
|
&& lib.length (lib.unique own) == lib.length own
|
||||||
&& presents "swarm-bao-queue-agent" b.queueAgentClientCertFile b.queueAgentClientKeyFile
|
|
||||||
&& presents "swarm-bao-grafana-oidc" b.grafanaOidcClientCertFile b.grafanaOidcClientKeyFile
|
&& presents "swarm-bao-grafana-oidc" b.grafanaOidcClientCertFile b.grafanaOidcClientKeyFile
|
||||||
&& presents "swarm-bao-otel-oidc" b.otelOidcClientCertFile b.otelOidcClientKeyFile
|
&& presents "swarm-bao-otel-oidc" b.otelOidcClientCertFile b.otelOidcClientKeyFile
|
||||||
&& presents "swarm-bao-forwarder-oidc" b.forwarderOidcClientCertFile b.forwarderOidcClientKeyFile;
|
&& presents "swarm-bao-forwarder-oidc" b.forwarderOidcClientCertFile b.forwarderOidcClientKeyFile;
|
||||||
|
|
@ -923,9 +907,7 @@ let
|
||||||
{
|
{
|
||||||
# The minting side of the same claim. A role matching a subject nothing
|
# The minting side of the same claim. A role matching a subject nothing
|
||||||
# signs is a reader that cannot log in, so the leaves and the roles have
|
# signs is a reader that cannot log in, so the leaves and the roles have
|
||||||
# to be asserted against each other — and the per-hive leaf carries
|
# to be asserted against each other.
|
||||||
# THIS host's hive name, which is what makes one hive's leaf useless
|
|
||||||
# against another hive's role.
|
|
||||||
name = "the PKI unit signs a leaf per reader, each under that reader's own subject";
|
name = "the PKI unit signs a leaf per reader, each under that reader's own subject";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
|
|
@ -936,8 +918,6 @@ let
|
||||||
# literal spanning both would pin this file's line wrapping rather than
|
# literal spanning both would pin this file's line wrapping rather than
|
||||||
# the pairing it means to.
|
# the pairing it means to.
|
||||||
lib.all (lib.flip lib.hasInfix s) [
|
lib.all (lib.flip lib.hasInfix s) [
|
||||||
"/queue-agent.pem ]"
|
|
||||||
"swarm-bao-queue-agent-h1 \"\" clientAuth"
|
|
||||||
"/grafana-oidc.pem ]"
|
"/grafana-oidc.pem ]"
|
||||||
"swarm-bao-grafana-oidc \"\" clientAuth"
|
"swarm-bao-grafana-oidc \"\" clientAuth"
|
||||||
"/otel-oidc.pem ]"
|
"/otel-oidc.pem ]"
|
||||||
|
|
@ -965,11 +945,10 @@ let
|
||||||
# client certificate and the host is the side that has one, so a unit
|
# client certificate and the host is the side that has one, so a unit
|
||||||
# rendered inside the store's container would have neither an identity
|
# rendered inside the store's container would have neither an identity
|
||||||
# nor a route. Plus the ordering that makes the mounts exist first.
|
# nor a route. Plus the ordering that makes the mounts exist first.
|
||||||
name = "the four readers' granting units are ordered after the mounts and rendered on the host";
|
name = "the three readers' granting units are ordered after the mounts and rendered on the host";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
units = [
|
units = [
|
||||||
"swarm-bao-queue-agent-policy"
|
|
||||||
"swarm-bao-grafana-oidc-policy"
|
"swarm-bao-grafana-oidc-policy"
|
||||||
"swarm-bao-otel-oidc-policy"
|
"swarm-bao-otel-oidc-policy"
|
||||||
"swarm-bao-forwarder-oidc-policy"
|
"swarm-bao-forwarder-oidc-policy"
|
||||||
|
|
@ -990,7 +969,7 @@ let
|
||||||
# The forwarder is listed apart from `policyReaders`: it renders wherever
|
# The forwarder is listed apart from `policyReaders`: it renders wherever
|
||||||
# the store does, so it is never absent on a store host and never present
|
# the store does, so it is never absent on a store host and never present
|
||||||
# on a remote one, and the two cases below would fail on it for that.
|
# on a remote one, and the two cases below would fail on it for that.
|
||||||
name = "each of the four readers is ordered after the unit writing its role";
|
name = "each of the three readers is ordered after the unit writing its role";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
s = baoGrantWithConsumers.systemd.services;
|
s = baoGrantWithConsumers.systemd.services;
|
||||||
|
|
@ -1483,9 +1462,9 @@ let
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The other hives' side: every hive in the directory but this one gets
|
# The other hives' side: every hive in the directory but this one gets
|
||||||
# the same three leaves, under its own name, in a directory of its own
|
# the same two leaves, under its own name, in a directory of its own
|
||||||
# that the operator copies to its host.
|
# that the operator copies to its host.
|
||||||
name = "the PKI unit signs every other hive's reader, queue-agent and gateway-leaf identities";
|
name = "the PKI unit signs every other hive's reader and gateway-leaf identities";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
s = baoGrantTwoHives.systemd.services.swarm-bao-pki.script;
|
s = baoGrantTwoHives.systemd.services.swarm-bao-pki.script;
|
||||||
|
|
@ -1494,7 +1473,6 @@ let
|
||||||
file: cn: lib.hasInfix "/hives/h2/${file}.pem ]" s && lib.hasInfix " ${cn} \"\" clientAuth" s;
|
file: cn: lib.hasInfix "/hives/h2/${file}.pem ]" s && lib.hasInfix " ${cn} \"\" clientAuth" s;
|
||||||
in
|
in
|
||||||
signs "client" "h2"
|
signs "client" "h2"
|
||||||
&& signs "queue-agent" "${b.queueAgentCommonNamePrefix}-h2"
|
|
||||||
&& signs "hive-leaf" "${b.hiveLeafCommonNamePrefix}-h2"
|
&& signs "hive-leaf" "${b.hiveLeafCommonNamePrefix}-h2"
|
||||||
# This hive's own leaves stay at the top of the directory.
|
# This hive's own leaves stay at the top of the directory.
|
||||||
&& !(lib.hasInfix "/hives/h1/" s);
|
&& !(lib.hasInfix "/hives/h1/" s);
|
||||||
|
|
|
||||||
|
|
@ -31,18 +31,10 @@ let
|
||||||
# nothing here mints a leaf and the operator names one placed by hand. The
|
# nothing here mints a leaf and the operator names one placed by hand. The
|
||||||
# deployment this pairing exists to serve, and the one that was previously
|
# deployment this pairing exists to serve, and the one that was previously
|
||||||
# inexpressible — the gate asked whether the store was a neighbour.
|
# inexpressible — the gate asked whether the store was a neighbour.
|
||||||
#
|
|
||||||
# 🩸 One leaf PER READER, and this fixture is where that cost is visible: the
|
|
||||||
# queue-credential unit identifies itself to the store separately from the
|
|
||||||
# hive, so an operator placing leaves by hand places one for it as well as
|
|
||||||
# the hive's own. Naming only `clientCertFile` here would leave the unit
|
|
||||||
# unrendered — which is what the arms below would then be asserting about.
|
|
||||||
baoRemoteReader = hiveWithController {
|
baoRemoteReader = hiveWithController {
|
||||||
deploy.matrix.enable = true;
|
deploy.matrix.enable = true;
|
||||||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||||
deploy.bao.queueAgentClientCertFile = "/etc/pki/bao-queue-agent.pem";
|
|
||||||
deploy.bao.queueAgentClientKeyFile = "/etc/pki/bao-queue-agent-key.pem";
|
|
||||||
};
|
};
|
||||||
|
|
||||||
# A homeserver on a hive with NO store identity at all — neither a local
|
# A homeserver on a hive with NO store identity at all — neither a local
|
||||||
|
|
@ -51,24 +43,15 @@ let
|
||||||
# fixture set its own, as ./lib.nix asks.
|
# fixture set its own, as ./lib.nix asks.
|
||||||
inherit (shared) matrixNoBaoIdentity;
|
inherit (shared) matrixNoBaoIdentity;
|
||||||
|
|
||||||
# 🩸 The hand-configured remote reader that named the hive's leaf and stopped.
|
# 🩸 The hand-configured remote reader that named the hive's leaf and stopped:
|
||||||
# `baoRemoteReader` above is the same deployment done right; this one holds
|
# it holds the hive's own leaf, so it demonstrably reads the store, and is
|
||||||
# the hive's own leaf, so it demonstrably reads the store, and is missing the
|
# missing matrix-ctl's pair.
|
||||||
# queue reader's pair. Without the refusal it would render no queue reader
|
|
||||||
# and say nothing — which is what the refusal arms below are about.
|
|
||||||
baoRemoteReaderMissingLeaves = hiveWithController {
|
baoRemoteReaderMissingLeaves = hiveWithController {
|
||||||
deploy.matrix.enable = true;
|
deploy.matrix.enable = true;
|
||||||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||||
};
|
};
|
||||||
|
|
||||||
# The same omission on a hive that does NOT run a homeserver: the queue
|
|
||||||
# refusal checks only the hive's own toggle.
|
|
||||||
remoteReaderNoMatrixMissingLeaves = hiveWithController {
|
|
||||||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
|
||||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
|
||||||
};
|
|
||||||
|
|
||||||
# The store's host with the hive's own reader pointed somewhere else.
|
# The store's host with the hive's own reader pointed somewhere else.
|
||||||
matrixOnStoreHostReaderMoved = hiveWithController {
|
matrixOnStoreHostReaderMoved = hiveWithController {
|
||||||
deploy.bao.enable = true;
|
deploy.bao.enable = true;
|
||||||
|
|
@ -77,122 +60,15 @@ let
|
||||||
deploy.bao.clientKeyFile = "/etc/pki/h1-key.pem";
|
deploy.bao.clientKeyFile = "/etc/pki/h1-key.pem";
|
||||||
};
|
};
|
||||||
|
|
||||||
# Did a module refuse this host, and over which option. An assertion is a
|
|
||||||
# config VALUE until something forces it — `.config` never throws — so a
|
|
||||||
# fixture in a state the module refuses stays evaluable and the refusal reads
|
|
||||||
# back as data. Matched on the option name the message names rather than on
|
|
||||||
# its prose, because the option name is the part an operator has to act on
|
|
||||||
# and a message that stopped naming it would be the actual defect. Same shape
|
|
||||||
# as ./grafana.nix's `grafanaRefusedFor`, which reads `swarm-grafana.nix`'s
|
|
||||||
# own refusal.
|
|
||||||
refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions;
|
|
||||||
cases = [
|
cases = [
|
||||||
{
|
{
|
||||||
# A login failure is the store being unreachable, sealed, or not yet
|
# A hive holds no shared queue credential for its agents, wherever the
|
||||||
# holding this host's role — all of which a retry fixes. A read that
|
# store runs. The control: hive-c0re renders on both hosts.
|
||||||
# answers "nothing there" is not, so only the first is allowed to fail
|
name = "no hive renders a reader for a hive-shared agent queue credential";
|
||||||
# the unit.
|
ok = lib.all (m: !(m.systemd.services ? swarm-bao-queue-agent) && m.systemd.services ? hive-c0re) [
|
||||||
name = "the queue credential reader retries a failed login and still degrades on an empty read";
|
baoWithMatrix
|
||||||
ok =
|
baoRemoteReader
|
||||||
let
|
];
|
||||||
u = baoWithMatrix.systemd.services.swarm-bao-queue-agent;
|
|
||||||
# Everything between the login's failure branch and the read's, which
|
|
||||||
# is where the exit that decides "retry or give up" lives.
|
|
||||||
afterLogin = lib.last (lib.splitString "bao login" u.script);
|
|
||||||
loginBranch = lib.head (lib.splitString "bao kv get" afterLogin);
|
|
||||||
in
|
|
||||||
u.serviceConfig.Restart or null == "on-failure"
|
|
||||||
&& u.startLimitBurst or 0 > 0
|
|
||||||
# The window has to outlast every attempt, or the burst is unreachable.
|
|
||||||
&& u.startLimitIntervalSec or 0 > (u.serviceConfig.RestartSec or 0) * (u.startLimitBurst or 0)
|
|
||||||
&& lib.hasInfix "exit 1" loginBranch
|
|
||||||
&& lib.hasInfix "exit 0" (lib.last (lib.splitString "bao kv get" u.script));
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# The reader's own grant covers `swarm/hives/<this hive>/*`; a path
|
|
||||||
# outside it answers 403, not "no such key". This is the path
|
|
||||||
# `swarm_secret_client::queue::agent_client_path` builds from the same
|
|
||||||
# pieces. The negative arm is the rename this one is exposed to: a
|
|
||||||
# credential named for the queue rather than for the hive that presents
|
|
||||||
# it reads as correct and is refused on every boot.
|
|
||||||
name = "the agent queue credential path sits inside the prefix the reader is granted";
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
s = baoRemoteReader.systemd.services.swarm-bao-queue-agent.script;
|
|
||||||
in
|
|
||||||
lib.hasInfix "secret/swarm/hives/h1/queue/agent" s && !(lib.hasInfix "secret/swarm/queue/" s);
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# The unit's output is the option's value, not a literal that agrees with
|
|
||||||
# it today: an operator moving the directory has to move both files. The
|
|
||||||
# prefix is asserted too because `hasInfix ""` is true — an option
|
|
||||||
# renamed out from under this arm would otherwise read empty and pass.
|
|
||||||
name = "the queue credential reader writes both files under the directory its option names";
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
m = baoRemoteReader;
|
|
||||||
dir = toString m.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
|
|
||||||
s = m.systemd.services.swarm-bao-queue-agent.script;
|
|
||||||
in
|
|
||||||
lib.hasPrefix "/var/lib/" dir
|
|
||||||
&& lib.hasInfix "${dir}/secret" s
|
|
||||||
&& lib.hasInfix "${dir}/client_id" s;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# No agent container may render before this unit has had its attempts,
|
|
||||||
# and the edge that guarantees it must delay hive-c0re rather than sink
|
|
||||||
# it: an unreachable store is this unit's `Restart=on-failure` window,
|
|
||||||
# not a reason for the daemon that renders every agent to fail its own
|
|
||||||
# start.
|
|
||||||
name = "the queue credential reader orders before hive-c0re and is wanted, not required, by it";
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
u = baoRemoteReader.systemd.services.swarm-bao-queue-agent;
|
|
||||||
in
|
|
||||||
builtins.elem "hive-c0re.service" (u.before or [ ])
|
|
||||||
&& builtins.elem "hive-c0re.service" (u.wantedBy or [ ])
|
|
||||||
&& !(builtins.elem "hive-c0re.service" (u.requiredBy or [ ]))
|
|
||||||
&& !(builtins.elem "hive-c0re.service" (u.requires or [ ]));
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Nothing local mints certificates off-host, so this orders against
|
|
||||||
# nothing there.
|
|
||||||
name = "an off-host reader names no unit the store's host would have provided";
|
|
||||||
# Membership first, then the value: indexing a missing unit throws, and a
|
|
||||||
# table that reports which property broke must not be the thing that dies.
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
s = baoRemoteReader.systemd.services;
|
|
||||||
u = s.swarm-bao-queue-agent;
|
|
||||||
in
|
|
||||||
s ? swarm-bao-queue-agent && !(lib.elem "swarm-bao-pki.service" (u.after ++ u.wants ++ u.requires));
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Presence control for the case above: the list is conditional, not gone.
|
|
||||||
# Wanted, never required: a refused pki start fails a requirer's job with
|
|
||||||
# `dependency`, which its `Restart=on-failure` never retries.
|
|
||||||
name = "a co-located reader still orders after and wants the local pki unit, and does not require it";
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
s = baoWithMatrix.systemd.services;
|
|
||||||
u = s.swarm-bao-queue-agent;
|
|
||||||
in
|
|
||||||
s ? swarm-bao-queue-agent
|
|
||||||
&& lib.elem "swarm-bao-pki.service" u.after
|
|
||||||
&& lib.elem "swarm-bao-pki.service" u.wants
|
|
||||||
&& u.requires == [ ];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Nothing asserted this script before, which is how it kept a branch that
|
|
||||||
# named three states and threw away the only thing telling them apart. A
|
|
||||||
# missing value, a refused identity and an unreachable host all end in the
|
|
||||||
# same degraded mode here, correctly — what must survive is which one.
|
|
||||||
name = "the queue credential reader carries the store's own diagnostic into the journal";
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
s = baoWithMatrix.systemd.services.swarm-bao-queue-agent.script;
|
|
||||||
in
|
|
||||||
!(lib.hasInfix "2>/dev/null" s) && lib.hasInfix ''cat "''$err"'' s;
|
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# hive-c0re runs as hive-core and the client key is `0600` root-owned
|
# hive-c0re runs as hive-core and the client key is `0600` root-owned
|
||||||
|
|
@ -404,46 +280,6 @@ let
|
||||||
&& !(c.systemd.services ? swarm-matrix-appservice-publish)
|
&& !(c.systemd.services ? swarm-matrix-appservice-publish)
|
||||||
&& !(lib.any (lib.hasPrefix "swarm-appservice.yaml") c.systemd.services.tuwunel.serviceConfig.LoadCredential);
|
&& !(lib.any (lib.hasPrefix "swarm-appservice.yaml") c.systemd.services.tuwunel.serviceConfig.LoadCredential);
|
||||||
}
|
}
|
||||||
{
|
|
||||||
# 🩸 The arm the whole refusal exists for. The reader is gated on its own
|
|
||||||
# leaf, so the way this regresses is the build going green and the unit
|
|
||||||
# not rendering — which no presence check on a rendered unit can see,
|
|
||||||
# because the unit that is missing is the evidence. Read as a refusal
|
|
||||||
# naming each option, so an operator acts on the message without opening
|
|
||||||
# the nix.
|
|
||||||
name = "a remote reader missing the queue reader's leaf is refused, naming both options";
|
|
||||||
ok =
|
|
||||||
refusedOver baoRemoteReaderMissingLeaves "queueAgentClientCertFile"
|
|
||||||
&& refusedOver baoRemoteReaderMissingLeaves "queueAgentClientKeyFile";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
name = "the queue refusal needs no homeserver";
|
|
||||||
ok = refusedOver remoteReaderNoMatrixMissingLeaves "queueAgentClientCertFile";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# ⚠️ The arm that keeps the refusal from being worse than the silence it
|
|
||||||
# replaced. A hive with NO store identity is the supported no-store
|
|
||||||
# deployment and also the state an operator passes through bringing a hive
|
|
||||||
# up — neither may fail to evaluate. Asserted as "no refusal names either
|
|
||||||
# option", not as "this one fixture is fine", because the way this breaks
|
|
||||||
# is a gate widened to the principal's leaf alone.
|
|
||||||
name = "a hive with no store identity at all is refused over none of the per-principal leaves";
|
|
||||||
ok = lib.all (option: !(refusedOver matrixNoBaoIdentity option)) [
|
|
||||||
"queueAgentClientCertFile"
|
|
||||||
"queueAgentClientKeyFile"
|
|
||||||
];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# The other half of the same guard, and the one an operator meets far more
|
|
||||||
# often: on the store's own host ./host-modules/glue-bao-tls.nix mkDefaults
|
|
||||||
# every pair, so there is nothing to name and nothing to refuse. Paired with
|
|
||||||
# the fully-named remote reader, which is the same deployment done by hand.
|
|
||||||
name = "neither a store host nor a correctly-named remote reader is refused";
|
|
||||||
ok = lib.all (m: !(refusedOver m "queueAgentClientCertFile")) [
|
|
||||||
baoWithMatrix
|
|
||||||
baoRemoteReader
|
|
||||||
];
|
|
||||||
}
|
|
||||||
];
|
];
|
||||||
in
|
in
|
||||||
runGroup "bao-matrix-reader" cases
|
runGroup "bao-matrix-reader" cases
|
||||||
|
|
|
||||||
|
|
@ -216,13 +216,17 @@ let
|
||||||
&& env shared.controllerAlone == { };
|
&& env shared.controllerAlone == { };
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# Where the reader puts the files and where the daemon looks for them is
|
# An agent connects to the queue with its own secret, so the hive hands
|
||||||
# one agreement spanning two modules. Asserted against the option rather
|
# its agents no shared credential and no token endpoint to mint one at.
|
||||||
# than the literal so moving the directory moves both ends.
|
# The control: the same environment carries the agents' queue address.
|
||||||
name = "hive-c0re is told where the agents' queue credential lands";
|
name = "hive-c0re forwards no shared agent queue credential";
|
||||||
ok =
|
ok =
|
||||||
allLocal.systemd.services.hive-c0re.environment.HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR
|
let
|
||||||
== toString allLocal.services.hyperhive.deploy.hive-controller.queue.agentCredentialDir;
|
e = allLocal.systemd.services.hive-c0re.environment;
|
||||||
|
in
|
||||||
|
!(e ? HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR)
|
||||||
|
&& !(e ? HIVE_AGENT_OIDC_TOKEN_ENDPOINT)
|
||||||
|
&& e ? HIVE_AGENT_NATS_URL;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# Inside an agent container loopback is the agent itself, so the agents'
|
# Inside an agent container loopback is the agent itself, so the agents'
|
||||||
|
|
@ -256,28 +260,10 @@ let
|
||||||
&& lib.elem "/hive-collector.hive/${bridge}" allLocal.services.dnsmasq.settings.address;
|
&& lib.elem "/hive-collector.hive/${bridge}" allLocal.services.dnsmasq.settings.address;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The agents mint against the swarm's IdP, the same endpoint the hive's
|
# The absence arm, and what makes the agents' queue address cases above able
|
||||||
# own client uses — a hive-local guess would produce a token the queue
|
# to fail.
|
||||||
# would not accept.
|
name = "a hive with no agent queue address forwards none";
|
||||||
name = "the agents' token endpoint is the swarm IdP's";
|
ok = !(noAgentQueue.systemd.services.hive-c0re.environment ? HIVE_AGENT_NATS_URL);
|
||||||
ok =
|
|
||||||
let
|
|
||||||
e = allLocal.systemd.services.hive-c0re.environment;
|
|
||||||
in
|
|
||||||
lib.hasSuffix "/api/oidc/token" e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT
|
|
||||||
&& e.HIVE_AGENT_OIDC_TOKEN_ENDPOINT == e.HIVE_C0RE_OIDC_TOKEN_ENDPOINT;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# The absence arm, and what makes the two above able to fail: a hive
|
|
||||||
# with no queue address for its agents must forward neither coordinate,
|
|
||||||
# because half a pair reaches the harness as a partial configuration
|
|
||||||
# rather than as none.
|
|
||||||
name = "a hive with no agent queue address forwards no agent queue coordinates";
|
|
||||||
ok =
|
|
||||||
let
|
|
||||||
e = noAgentQueue.systemd.services.hive-c0re.environment;
|
|
||||||
in
|
|
||||||
!(e ? HIVE_AGENT_NATS_URL) && !(e ? HIVE_AGENT_OIDC_TOKEN_ENDPOINT);
|
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# Control: `bare` sets none of them, and its serve.json carries the
|
# Control: `bare` sets none of them, and its serve.json carries the
|
||||||
|
|
|
||||||
|
|
@ -37,13 +37,6 @@ let
|
||||||
# homeserver across the network.
|
# homeserver across the network.
|
||||||
matrixElsewhere = shared.bareWithController;
|
matrixElsewhere = shared.bareWithController;
|
||||||
cases = [
|
cases = [
|
||||||
{
|
|
||||||
# Absence arm, and what makes the one above able to fail: with no leaf
|
|
||||||
# this unit would fail a TLS handshake on every boot, so it must not
|
|
||||||
# exist at all rather than retry its way through the start limit.
|
|
||||||
name = "a hive with no store identity renders no queue credential reader";
|
|
||||||
ok = !(matrixNoBaoIdentity.systemd.services ? swarm-bao-queue-agent);
|
|
||||||
}
|
|
||||||
{
|
{
|
||||||
# Two limits governed a matrix upload and nothing kept them in agreement,
|
# Two limits governed a matrix upload and nothing kept them in agreement,
|
||||||
# so raising the documented one past the gateway's hardcoded cap changed
|
# so raising the documented one past the gateway's hardcoded cap changed
|
||||||
|
|
|
||||||
|
|
@ -43,14 +43,12 @@ let
|
||||||
# leaves, not a whole prefix, so a hive that inherited a dead entry there
|
# leaves, not a whole prefix, so a hive that inherited a dead entry there
|
||||||
# would silently gain the ability to overwrite the swarm's matrix credential.
|
# would silently gain the ability to overwrite the swarm's matrix credential.
|
||||||
#
|
#
|
||||||
# 🩸 The two per-hive subjects have a different shape: those roles are
|
# 🩸 The per-hive subject has a different shape: that role is written PER
|
||||||
# written PER HIVE, so the subject a hive must not be is
|
# HIVE, so the subject a hive must not be is `<prefix>-<some hive's name>`
|
||||||
# `<prefix>-<some hive's name>` rather than the prefix itself. Reserving only
|
# rather than the prefix itself. Reserving only the prefix would leave the
|
||||||
# the prefix would leave the composed spelling free, and a hive taking it
|
# composed spelling free, and a hive taking it would present a leaf the other
|
||||||
# would present a leaf the other hive's role accepts — a hive reading another
|
# hive's role accepts. The collision is with the OTHER hive's role, hence
|
||||||
# hive's queue credential, the exact widening the split exists to avoid. The
|
# `other` beside `hl-other`.
|
||||||
# collision is with the OTHER hive's role, hence `other` beside `qr-other`
|
|
||||||
# and `hl-other`.
|
|
||||||
hiveNamedAfterEverySubject = hive {
|
hiveNamedAfterEverySubject = hive {
|
||||||
deploy.swarm-otel.enable = false;
|
deploy.swarm-otel.enable = false;
|
||||||
deploy.bao = {
|
deploy.bao = {
|
||||||
|
|
@ -62,7 +60,6 @@ let
|
||||||
forwarderOidcCommonName = "fwctl";
|
forwarderOidcCommonName = "fwctl";
|
||||||
forgePublisherCommonName = "fpctl";
|
forgePublisherCommonName = "fpctl";
|
||||||
granterCommonName = "grctl";
|
granterCommonName = "grctl";
|
||||||
queueAgentCommonNamePrefix = "qr";
|
|
||||||
hiveLeafCommonNamePrefix = "hl";
|
hiveLeafCommonNamePrefix = "hl";
|
||||||
};
|
};
|
||||||
swarm.hives = {
|
swarm.hives = {
|
||||||
|
|
@ -75,7 +72,6 @@ let
|
||||||
fpctl.domain = "fp.t.local";
|
fpctl.domain = "fp.t.local";
|
||||||
grctl.domain = "gr.t.local";
|
grctl.domain = "gr.t.local";
|
||||||
other.domain = "other.t.local";
|
other.domain = "other.t.local";
|
||||||
qr-other.domain = "q.t.local";
|
|
||||||
hl-other.domain = "hl.t.local";
|
hl-other.domain = "hl.t.local";
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
@ -216,16 +212,9 @@ let
|
||||||
{
|
{
|
||||||
# 🩸 The per-hive half, and the one a prefix-only reservation would miss:
|
# 🩸 The per-hive half, and the one a prefix-only reservation would miss:
|
||||||
# the role is `<prefix>-<hive>`, so the reserved string has to be composed
|
# the role is `<prefix>-<hive>`, so the reserved string has to be composed
|
||||||
# against every declared hive. Here hive `qr-other` collides with the role
|
# against every declared hive. A hive taking this spelling would present
|
||||||
# written for hive `other` — a leaf that reads a credential belonging to a
|
# a leaf hive `other`'s gateway-leaf role accepts, and could ask the store
|
||||||
# hive that is not it.
|
# for that hive's certificate. The control is the legal roster below.
|
||||||
name = "a hive named after another hive's per-hive reader subject is refused";
|
|
||||||
ok = equalityGuardRefuses hiveNamedAfterEverySubject "qr-other";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# A hive taking this spelling would present a leaf hive `other`'s
|
|
||||||
# gateway-leaf role accepts, and could ask the store for that hive's
|
|
||||||
# certificate. The control is the legal roster below.
|
|
||||||
name = "a hive named after another hive's gateway-leaf subject is refused";
|
name = "a hive named after another hive's gateway-leaf subject is refused";
|
||||||
ok = equalityGuardRefuses hiveNamedAfterEverySubject "hl-other";
|
ok = equalityGuardRefuses hiveNamedAfterEverySubject "hl-other";
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -77,13 +77,10 @@ let
|
||||||
|
|
||||||
# The same hive given its own store leaf by hand, which is what turns
|
# The same hive given its own store leaf by hand, which is what turns
|
||||||
# publishing on away from the IdP's host: hive-c0re reads the status secret
|
# publishing on away from the IdP's host: hive-c0re reads the status secret
|
||||||
# from the store under it. The queue-credential reader's pair comes with it,
|
# from the store under it.
|
||||||
# because a hive that reads the store is refused at eval without one.
|
|
||||||
hiveLeaf = {
|
hiveLeaf = {
|
||||||
deploy.bao.clientCertFile = "/etc/pki/h1.pem";
|
deploy.bao.clientCertFile = "/etc/pki/h1.pem";
|
||||||
deploy.bao.clientKeyFile = "/etc/pki/h1-key.pem";
|
deploy.bao.clientKeyFile = "/etc/pki/h1-key.pem";
|
||||||
deploy.bao.queueAgentClientCertFile = "/etc/pki/h1-queue-agent.pem";
|
|
||||||
deploy.bao.queueAgentClientKeyFile = "/etc/pki/h1-queue-agent-key.pem";
|
|
||||||
};
|
};
|
||||||
remotePublishing = hiveWithController (lib.recursiveUpdate remoteSecrets hiveLeaf);
|
remotePublishing = hiveWithController (lib.recursiveUpdate remoteSecrets hiveLeaf);
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -80,7 +80,6 @@ let
|
||||||
# Each hive's hive-c0re reads this exact path
|
# Each hive's hive-c0re reads this exact path
|
||||||
# (`swarm_secret_client::queue::hive_client_path`, pinned by that crate's
|
# (`swarm_secret_client::queue::hive_client_path`, pinned by that crate's
|
||||||
# own test), for every hive in the roster, not only the one this host is.
|
# own test), for every hive in the roster, not only the one this host is.
|
||||||
# The agents' client beside it stays a separate path.
|
|
||||||
name = "the publisher writes every hive's own client under that hive's prefix";
|
name = "the publisher writes every hive's own client under that hive's prefix";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
|
|
@ -91,8 +90,7 @@ let
|
||||||
[
|
[
|
||||||
"h1"
|
"h1"
|
||||||
"h2"
|
"h2"
|
||||||
]
|
];
|
||||||
&& lib.hasInfix "secret/swarm/hives/h1/queue/agent" s;
|
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The controller reads this exact path
|
# The controller reads this exact path
|
||||||
|
|
@ -333,17 +331,17 @@ let
|
||||||
lib.hasInfix "value=@" code && !(lib.hasInfix "$(cat" code);
|
lib.hasInfix "value=@" code && !(lib.hasInfix "$(cat" code);
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# Two ends of a wire nothing at eval time carries end to end: this is the
|
# No hive holds a shared queue credential for its agents, so authelia
|
||||||
# path `swarm_secret_client::queue` resolves for the reader. Both hives
|
# mints no agent client and the publisher writes no such path. The
|
||||||
# are asserted, so a publisher that knew one name rather than the roster
|
# control: the same script writes each hive's own client.
|
||||||
# fails here rather than on the second hive ever added to a swarm.
|
name = "the publisher writes no hive-shared agent queue credential";
|
||||||
name = "the publisher writes every hive in the roster to that hive's own queue path";
|
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script;
|
s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script;
|
||||||
in
|
in
|
||||||
lib.hasInfix "secret/swarm/hives/h1/queue/agent" s
|
!(lib.hasInfix "/queue/agent" s)
|
||||||
&& lib.hasInfix "secret/swarm/hives/h2/queue/agent" s;
|
&& !(lib.hasInfix "-agent.secret" s)
|
||||||
|
&& lib.hasInfix "secret/swarm/hives/h1/oidc/client" s;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# A property of the SET, not of one unit: both of these authenticate by
|
# A property of the SET, not of one unit: both of these authenticate by
|
||||||
|
|
@ -373,7 +371,7 @@ let
|
||||||
&& !(lib.hasInfix "bao kv" (lib.head (lib.splitString "bao login" c)));
|
&& !(lib.hasInfix "bao kv" (lib.head (lib.splitString "bao login" c)));
|
||||||
in
|
in
|
||||||
holdsTokenFirst (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script
|
holdsTokenFirst (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script
|
||||||
&& holdsTokenFirst baoWithMatrix.systemd.services.swarm-bao-queue-agent.script
|
&& holdsTokenFirst baoWithMatrix.systemd.services.swarm-bao-matrix-oidc.script
|
||||||
# Controls, so a clean verdict above means something. In order: a bare
|
# Controls, so a clean verdict above means something. In order: a bare
|
||||||
# read is refused, a read placed before the login is refused, and a
|
# read is refused, a read placed before the login is refused, and a
|
||||||
# login that exists only in a comment is refused — that last one is the
|
# login that exists only in a comment is refused — that last one is the
|
||||||
|
|
|
||||||
|
|
@ -95,7 +95,7 @@ pub fn hive_localpart(hive: &str) -> String {
|
||||||
|
|
||||||
/// The path holding `hive`'s matrix sender account's homeserver access token.
|
/// The path holding `hive`'s matrix sender account's homeserver access token.
|
||||||
///
|
///
|
||||||
/// Keyed per **hive**, like [`crate::queue::agent_client_path`]. Under
|
/// Keyed per **hive**, like [`crate::queue::hive_client_path`]. Under
|
||||||
/// [`Kind::Hive`] the grant that reaches it is the hive's **own** stanza,
|
/// [`Kind::Hive`] the grant that reaches it is the hive's **own** stanza,
|
||||||
/// `secret/data/swarm/hives/<this hive>/*`, which interpolates the name: hive
|
/// `secret/data/swarm/hives/<this hive>/*`, which interpolates the name: hive
|
||||||
/// `alpha` reads its own token and gets a 403 on `beta`'s.
|
/// `alpha` reads its own token and gets a 403 on `beta`'s.
|
||||||
|
|
@ -137,7 +137,7 @@ pub fn swarm_appservice_token_path() -> Result<String, Error> {
|
||||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
pub struct Credential {
|
pub struct Credential {
|
||||||
/// Named `value` like [`crate::forge::Credential::value`] and
|
/// Named `value` like [`crate::forge::Credential::value`] and
|
||||||
/// [`crate::queue::Credential::value`], so a nix-side reader spells
|
/// [`crate::queue::AgentCredential::value`], so a nix-side reader spells
|
||||||
/// `bao kv get -field=value` for every kind.
|
/// `bao kv get -field=value` for every kind.
|
||||||
pub value: String,
|
pub value: String,
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -40,10 +40,9 @@ pub fn identity_path(agent: &str) -> Result<String, Error> {
|
||||||
/// was issued from.
|
/// was issued from.
|
||||||
///
|
///
|
||||||
/// All three, because a reader has to reconstruct a usable identity from the
|
/// All three, because a reader has to reconstruct a usable identity from the
|
||||||
/// store alone — the same requirement [`crate::queue::Credential`] states for
|
/// store alone. The authority rides along so the cert-auth role and the
|
||||||
/// carrying its client id. The authority rides along so the cert-auth role
|
/// certificate cannot be delivered from two different sources and silently
|
||||||
/// and the certificate cannot be delivered from two different sources and
|
/// disagree; it is public material, unlike the other two fields.
|
||||||
/// silently disagree; it is public material, unlike the other two fields.
|
|
||||||
///
|
///
|
||||||
/// ⚠️ **No `Debug` derive.** See the hand-written impl below: one of these
|
/// ⚠️ **No `Debug` derive.** See the hand-written impl below: one of these
|
||||||
/// fields is a private key, and a derived `Debug` would put it in any log line
|
/// fields is a private key, and a derived `Debug` would put it in any log line
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ use crate::Error;
|
||||||
/// The KV v2 mount every swarm secret lives under.
|
/// The KV v2 mount every swarm secret lives under.
|
||||||
///
|
///
|
||||||
/// A literal because the nix-side readers spell the same one, as in
|
/// A literal because the nix-side readers spell the same one, as in
|
||||||
/// `glue-queue-agent-credential.nix`; an option nobody sets would be two ways
|
/// `nix/agent-modules/bao.nix`; an option nobody sets would be two ways
|
||||||
/// to say one thing.
|
/// to say one thing.
|
||||||
pub const MOUNT: &str = "secret";
|
pub const MOUNT: &str = "secret";
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -231,7 +231,7 @@ mod tests {
|
||||||
assert!(
|
assert!(
|
||||||
stanza_paths(&p)
|
stanza_paths(&p)
|
||||||
.iter()
|
.iter()
|
||||||
.any(|g| covers(g, "swarm/hives/pr1ma/queue/agent"))
|
.any(|g| covers(g, "swarm/hives/pr1ma/oidc/client"))
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,21 +1,10 @@
|
||||||
//! The queue agreements: where a queue credential lives in the store, and what
|
//! The queue agreements: where a queue credential lives in the store, and what
|
||||||
//! the object at that path holds.
|
//! the object at that path holds.
|
||||||
//!
|
//!
|
||||||
//! **Two kinds live here, and which is which is the thing to get right.**
|
//! An agent's credential, [`agent_queue_path`] / [`AgentCredential`], is keyed
|
||||||
//!
|
//! per **agent**: a secret the swarm mints for one agent, at swarm level, with
|
||||||
//! - [`agent_client_path`] / [`Credential`] are keyed per **hive**: one OIDC
|
//! no hive in the chain. It is what makes one agent distinguishable from its
|
||||||
//! client minted per hive at deploy time, shared by every agent container on
|
//! co-hived neighbours.
|
||||||
//! it. It says which hive a caller belongs to, never which agent.
|
|
||||||
//! - [`agent_queue_path`] / [`AgentCredential`] are keyed per **agent**: a
|
|
||||||
//! secret the swarm mints for one agent, at swarm level, with no hive in the
|
|
||||||
//! chain. It is what makes one agent distinguishable from its co-hived
|
|
||||||
//! neighbours.
|
|
||||||
//!
|
|
||||||
//! They are separate objects rather than one with a nullable field because they
|
|
||||||
//! are minted by different principals on different events — the hive-scoped one
|
|
||||||
//! by authelia at deploy time, the per-agent one by `swarm-controller` when an
|
|
||||||
//! agent is created — and an object whose shape depends on who wrote it is a
|
|
||||||
//! reader that has to guess.
|
|
||||||
//!
|
//!
|
||||||
//! The per-agent secret is deliberately **not** derived from the agent's mTLS
|
//! The per-agent secret is deliberately **not** derived from the agent's mTLS
|
||||||
//! identity (the leaf at [`crate::mtls::identity_path`]). That leaf is for
|
//! identity (the leaf at [`crate::mtls::identity_path`]). That leaf is for
|
||||||
|
|
@ -33,17 +22,6 @@ use crate::{
|
||||||
path::{Kind, principal_prefix},
|
path::{Kind, principal_prefix},
|
||||||
};
|
};
|
||||||
|
|
||||||
/// The path holding the client secret that agent containers on `hive` present
|
|
||||||
/// to the swarm queue.
|
|
||||||
///
|
|
||||||
/// # Errors
|
|
||||||
/// [`Error::PathSegment`] when `hive` contains anything but `[A-Za-z0-9_-]`,
|
|
||||||
/// which is what keeps one hive's name from addressing another hive's secret.
|
|
||||||
pub fn agent_client_path(hive: &str) -> Result<String, Error> {
|
|
||||||
let prefix = principal_prefix(Kind::Hive, hive)?;
|
|
||||||
Ok(format!("{prefix}/queue/agent"))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The path holding the secret **one agent** presents to the swarm queue.
|
/// The path holding the secret **one agent** presents to the swarm queue.
|
||||||
///
|
///
|
||||||
/// Under the agent's own principal prefix, and that is the whole reason for
|
/// Under the agent's own principal prefix, and that is the whole reason for
|
||||||
|
|
@ -89,8 +67,7 @@ pub struct ControllerCredential {
|
||||||
/// The path holding `hive`'s own OIDC client secret: the `hive-<hive>`
|
/// The path holding `hive`'s own OIDC client secret: the `hive-<hive>`
|
||||||
/// identity hive-c0re publishes the hive's status to the swarm queue with.
|
/// identity hive-c0re publishes the hive's status to the swarm queue with.
|
||||||
///
|
///
|
||||||
/// Not [`agent_client_path`], which is the agents' client on the same hive.
|
/// Under the hive's own prefix, which the hive's policy reads
|
||||||
/// Both sit under the hive's own prefix, which the hive's policy reads
|
|
||||||
/// ([`crate::policy::render`]).
|
/// ([`crate::policy::render`]).
|
||||||
///
|
///
|
||||||
/// The nix half is `swarm-secret-publisher.nix`, which writes it.
|
/// The nix half is `swarm-secret-publisher.nix`, which writes it.
|
||||||
|
|
@ -117,8 +94,8 @@ pub struct HiveClientCredential {
|
||||||
/// field still decode, because unknown fields are ignored.
|
/// field still decode, because unknown fields are ignored.
|
||||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
pub struct AgentCredential {
|
pub struct AgentCredential {
|
||||||
/// The secret itself. Named to match [`Credential::value`] and
|
/// The secret itself. Named to match [`crate::matrix::Credential::value`]
|
||||||
/// [`crate::matrix::Credential::value`] so a nix-side reader spells
|
/// so a nix-side reader spells
|
||||||
/// `bao kv get -field=value` for every kind.
|
/// `bao kv get -field=value` for every kind.
|
||||||
pub value: String,
|
pub value: String,
|
||||||
|
|
||||||
|
|
@ -136,79 +113,10 @@ pub struct AgentCredential {
|
||||||
pub minted_at: Option<i64>,
|
pub minted_at: Option<i64>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// What the path holds: the client secret, plus the client id it belongs to.
|
|
||||||
///
|
|
||||||
/// The id rides with the secret for the same reason the homeserver rides with
|
|
||||||
/// a matrix token — a credential has to be reconstructable from the store
|
|
||||||
/// alone. Deriving it on the reading side instead would mean spelling
|
|
||||||
/// `hive-<name>-agent` in a second place, and the authelia module's own option
|
|
||||||
/// says what a split spelling costs: every agent is denied as a timeout.
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
|
||||||
pub struct Credential {
|
|
||||||
/// The secret itself. Named to match [`crate::matrix::Credential::value`]
|
|
||||||
/// so a nix-side reader spells `bao kv get -field=value` for either kind.
|
|
||||||
pub value: String,
|
|
||||||
|
|
||||||
/// The OIDC client id the secret authenticates.
|
|
||||||
///
|
|
||||||
/// Required, unlike [`crate::matrix::Credential::homeserver`]: that field
|
|
||||||
/// is optional because it was added to objects already in the store, and
|
|
||||||
/// this path is new. An object without an id is not a usable credential,
|
|
||||||
/// so failing to decode is the wanted behaviour — the alternative is a
|
|
||||||
/// reader holding `None` whose only recourse is to spell the id itself.
|
|
||||||
pub client_id: String,
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_hive_name_lands_under_its_own_principal_prefix() {
|
|
||||||
assert_eq!(
|
|
||||||
agent_client_path("alpha").expect("a plain name is legal"),
|
|
||||||
"swarm/hives/alpha/queue/agent"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_traversal_in_the_hive_name_is_refused() {
|
|
||||||
let e = agent_client_path("../beta").expect_err("a traversal is not");
|
|
||||||
assert!(matches!(e, Error::PathSegment { kind: "hive", .. }), "{e}");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn two_hives_never_share_a_path() {
|
|
||||||
assert_ne!(
|
|
||||||
agent_client_path("alpha").expect("legal"),
|
|
||||||
agent_client_path("beta").expect("legal")
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn the_object_round_trips_through_the_store_representation() {
|
|
||||||
let c = Credential {
|
|
||||||
value: "s3cr3t".to_owned(),
|
|
||||||
client_id: "hive-alpha-agent".to_owned(),
|
|
||||||
};
|
|
||||||
let json = serde_json::to_string(&c).expect("serialises");
|
|
||||||
assert_eq!(
|
|
||||||
serde_json::from_str::<Credential>(&json).expect("deserialises"),
|
|
||||||
c
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn the_field_names_the_nix_reader_asks_for_are_the_ones_written() {
|
|
||||||
let json = serde_json::to_value(Credential {
|
|
||||||
value: "s3cr3t".to_owned(),
|
|
||||||
client_id: "hive-alpha-agent".to_owned(),
|
|
||||||
})
|
|
||||||
.expect("serialises");
|
|
||||||
assert_eq!(json["value"], "s3cr3t");
|
|
||||||
assert_eq!(json["client_id"], "hive-alpha-agent");
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Spelled out in full because `swarm-secret-publisher.nix` and
|
/// Spelled out in full because `swarm-secret-publisher.nix` and
|
||||||
/// `swarm-bao.nix` write the same string by hand.
|
/// `swarm-bao.nix` write the same string by hand.
|
||||||
#[test]
|
#[test]
|
||||||
|
|
@ -231,9 +139,9 @@ mod tests {
|
||||||
let reads = |path: &str| granted.iter().any(|g| path.starts_with(g));
|
let reads = |path: &str| granted.iter().any(|g| path.starts_with(g));
|
||||||
let path = controller_client_path().expect("a constant segment");
|
let path = controller_client_path().expect("a constant segment");
|
||||||
assert!(!reads(&path), "{path} is readable under {granted:?}");
|
assert!(!reads(&path), "{path} is readable under {granted:?}");
|
||||||
// The control: the hive's own agent client IS under a hive stanza, so
|
// The control: the hive's own client IS under a hive stanza, so the
|
||||||
// the assertion above can fail at all.
|
// assertion above can fail at all.
|
||||||
let own = agent_client_path("alpha").expect("legal");
|
let own = hive_client_path("alpha").expect("legal");
|
||||||
assert!(reads(&own), "{own} should be readable under {granted:?}");
|
assert!(reads(&own), "{own} should be readable under {granted:?}");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -381,29 +289,4 @@ mod tests {
|
||||||
serde_json::from_str::<AgentCredential>(r#"{"value":"s","hive":"alpha"}"#).is_err()
|
serde_json::from_str::<AgentCredential>(r#"{"value":"s","hive":"alpha"}"#).is_err()
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// decodes as the other — the property that keeps a reader from picking up
|
|
||||||
/// a hive-shared credential where a per-agent one was meant.
|
|
||||||
#[test]
|
|
||||||
fn the_hive_credential_and_the_agent_credential_are_not_interchangeable() {
|
|
||||||
let hive_json = serde_json::to_string(&Credential {
|
|
||||||
value: "s".to_owned(),
|
|
||||||
client_id: "hive-alpha-agent".to_owned(),
|
|
||||||
})
|
|
||||||
.expect("serialises");
|
|
||||||
assert!(serde_json::from_str::<AgentCredential>(&hive_json).is_err());
|
|
||||||
|
|
||||||
let agent_json = serde_json::to_string(&AgentCredential {
|
|
||||||
value: "s".to_owned(),
|
|
||||||
agent: "atlas".to_owned(),
|
|
||||||
minted_at: None,
|
|
||||||
})
|
|
||||||
.expect("serialises");
|
|
||||||
assert!(serde_json::from_str::<Credential>(&agent_json).is_err());
|
|
||||||
|
|
||||||
assert_ne!(
|
|
||||||
agent_queue_path("atlas").expect("legal"),
|
|
||||||
agent_client_path("atlas").expect("legal"),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue