mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: the swarm README and credentials page keep the overview, the rest moves to its own pages

Browse source 
The hive directory and the WireGuard mesh become handbook pages of their own,
the controller's mint-failure handling moves to controller.md, and the
credential-by-credential mechanics and the secret store UI's proxy move to a new
implementation page.
This commit is contained in:
iris 2026-11-14 12:43:46 +01:00 • committed by mara
commit 0e99662710
9 changed files with 559 additions and 514 deletions
Download patch file Download diff file Expand all files Collapse all files

3
docs/1-getting-started/nixos.md
View file

@ -105,7 +105,8 @@ Without the repo, the first agent's `update_meta` job step fails with
Leave the other options at their defaults until you need them:
- Spreading a swarm over more than one host, and the credentials a split swarm
needs: [`2-handbook/swarm/README.md`](../2-handbook/swarm/README.md) and
needs: [`2-handbook/swarm/README.md`](../2-handbook/swarm/README.md),
[`2-handbook/swarm/hive-directory.md`](../2-handbook/swarm/hive-directory.md) and
[`2-handbook/swarm/credentials.md`](../2-handbook/swarm/credentials.md).
A worked example: [`2-handbook/swarm/semi-spread-deploy.md`](../2-handbook/swarm/semi-spread-deploy.md).
- Every option, with its type and default:

231
docs/2-handbook/swarm/README.md
View file

@ -60,7 +60,6 @@ grafana, the controller, authelia, …) →
[`bao.md`](../../4-implementation/swarm/bao.md)
- **swarm CA** — the store root every hive's gateway leaf chains to, and what to
hand a peer. → [`ca.md`](../../4-implementation/swarm/ca.md)
- **agent-state backups** — the old snapshot store no longer exists; swarm-archive will replace it.
## Swarm controller
@ -69,13 +68,8 @@ the agent list and their wanted state, and the job graph. Creating an agent —
from the swarm UI or `swarmctl agent create --hive <h>` — queues its SSO
identity, forge user, config repo, store identity and matrix account, then
sends the hive a deploy message. A new agent starts `paused`. A store identity
that fails to mint logs `swarm jobq: mint_agent_identity failed` with the agent,
the step it stopped at and the error, which the Grafana alert rule
`hh-agent-identity-mint-failed` fires on. The agent is still created, without a
store identity, and the controller's five-minute credential pass mints one for
it (also when only the certificate exists and its cert-auth role is missing) as
long as the agent isn't `Destroyed`. A mint that runs after a hive declares the
agent `Destroyed` logs `identity mint skipped` and writes nothing.
that fails to mint doesn't stop the creation; the controller mints it later
([how](../../4-implementation/swarm/controller.md#agent-identity-minting)).
`services.hyperhive.deploy.swarm-controller.enable` runs it on this host;
`singleHostSwarm` turns it on. **Otherwise off: set it on the one host that
@ -130,222 +124,11 @@ if you want it. What that grant lets an agent reach on a hive is on the
_derived_, not declared: the directory lists every hive including
yourself, and `hiveName` says which one you are.
### Hive identity config
```nix
services.hyperhive = {
swarm = {
domain = "myswarm.example"; # required — the swarm's DNS domain
name = "myswarm"; # shared swarm display name (optional)
# required — the directory, identical on every host in the swarm.
# Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
hives = {
firsthive = { }; # this host, per hiveName
secondhive = { }; # a second hive
thirdhive.domain = "thirdhive.elsewhere.example"; # addressed off-convention
};
};
# required on a hive host — this hive's label in the directory
deploy.hive-controller.hiveName = "firsthive";
};
```
<!-- vale write-good.Passive = NO -->
Swarm options are identical on every host in the swarm, so `swarm.domain` is
**required** on every host that runs any hyperhive service; a host that
imports the module and enables nothing evaluates without it. `hiveName` is
required on a host that runs a hive (`deploy.hive-controller.hiveName`), and on no other. Eval fails with a hint naming
each. Neither defaults, because a guessed value here is a wrong hostname
that evaluates cleanly and deploys.
<!-- vale write-good.Passive = YES -->
**The directory is one attrset, identical on every host.** It describes
every hive in the swarm, **including this one**, keyed by `hiveName`; what
differs between hosts is only `hiveName`. It **must** contain an entry for
this host's `hiveName`, and an empty directory fails that check too — eval
names the missing hive. Peers are _every entry but this host's_, so a
directory without this host would make every hive a peer, itself included.
```
# hive A # hive B
deploy.hive-controller.hiveName = "firsthive"; deploy.hive-controller.hiveName = "secondhive";
swarm.hives = { … }; swarm.hives = { … }; # byte-identical
```
One entry per hive means two hosts can't hold _different_ facts about the
same third hive, such as a stale endpoint.
`domain` defaults to `<name>.<swarm.domain>`, so a conventional directory
is names only. Set it only for a hive addressed by something else. This
hive's own `services.hyperhive.domain` comes from its entry; it drives
`HYPERHIVE_HIVE_DOMAIN` in every container so agents can form qualified
labels (`agent1@firsthive.myswarm.example`). Setting `services.hyperhive.domain`
directly overrides the directory entry, with a deprecation warning: the
option is local to this host while every host shares the directory, so a
value written only here is invisible to the rest of the swarm.
`swarm.name` is display only — the dashboard chrome header and per-agent
system prompts — and federated hives at different domains can share one.
`hiveName` surfaces in the same places but is also the leftmost label of the
hive's domain. `swarm.name` names the group; `hiveName` names this hive.
The env-var chain and `qualify()` semantics:
[`conventions.md`](../../5-project/process/conventions.md) § Hive identity.
Trust inside a swarm comes from the secret store's `pki` root
([`ca.md`](../../4-implementation/swarm/ca.md)): every hive's gateway leaf chains to it. Trusting a
hive whose root this swarm doesn't own — another swarm's — has no
mechanism.
### What the directory feeds
1. **Swarm-wide hive roster** — swarm-controller reads this same
directory and serves it at `GET /api/hives`; the swarm UI's front page
renders it ([`ui.md`](ui.md)).
2. **Matrix federation** — when this host runs the homeserver (`deploy.matrix.enable`), tuwunel
federates with the peer's matrix server (discovered via the peer's
`.well-known/matrix/server` delegation, which the controller's host or the pinned hive's gateway serves).
Federation validates the peer's TLS certificate against the matrix
**container's** trust bundle, independent of this directory.
⚠️ On a hive with self-signed gateway certificates, the container
also trusts this hive's `trust-bundle.pem`, bound in at runtime and
added to the public CAs; it holds the store's `pki` root, so a peer whose
certificate chains to the same root validates. A peer outside this
swarm's root needs a CA-issued certificate (ACME). Federation firewall
and TLS requirements: [`2-handbook/swarm/matrix.md`](matrix.md).
3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
to configure `wg-hive`. See [below](#wireguard-inter-hive-mesh-optional).
### Reaching the queue and the store on a routable address
The mesh isn't the only way to reach the queue and the secret store from
another host. Set `deploy.nats.openFirewall` on the queue's host to open the
queue's port on every interface, and `deploy.bao.openFirewall` together with
`deploy.bao.extraListenAddresses` on the store's host to open the store's port
on the addresses it listens on. Both default to off. Each exposes a service
beyond the tunnel, which is the operator's decision. The queue requires TLS and
its `auth_callout` refuses every client it can't identify. The store requires a
client certificate on every listener, and evaluation fails when you
set `deploy.bao.openFirewall` without `deploy.bao.clientCaFile`. Point
`nats.<swarm.domain>` and `bao.<swarm.domain>` in your upstream DNS at the
address the other hives reach.
### Keeping a vhost off the public address
The gateway is one nginx per host, and a vhost answers on every address of the
host by default, IPv4 and IPv6. To serve some vhosts only inside a private
network (a VPN, say), list the host's addresses on that network in
`services.hyperhive.gateway.privateListenAddresses` and set `public = false`
on the vhosts that should stay inside:
```nix
services.hyperhive.gateway = {
privateListenAddresses = [ "10.100.0.1" "fd00:100::1" ];
vhosts."grafana.myswarm.example".public = false;
};
```
A `public = false` vhost listens on those addresses only. The public address
never serves it, and a request for its name there closes without a response.
Vhosts that keep the default `public = true` answer on the private addresses
too. The addresses can be IPv4 or IPv6 (`fd00::1` or `[fd00::1]`), and each
must exist on the host when nginx starts. The firewall stays your call: this
decides which vhosts nginx serves on which address, not which ports are open.
The gateway also listens on `[::]` by default, so the vhosts are reachable
over IPv6. `gateway.listenIPv6` follows `networking.enableIPv6`; turn it off
on a host whose kernel can't open an IPv6 socket. Details are in
[`gateway.md`](../../4-implementation/networking/gateway.md#listen-addresses).
### WireGuard inter-hive mesh (optional)
The peer config above uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hyperhive can configure a host-to-host
WireGuard mesh.
#### Generating keys
On each hive host:
```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
```
#### Config example (two hives)
```nix
# hive A (firsthive.myswarm.example, mesh IP 10.100.0.1)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.1/24";
listenPort = 51820; # optional, default 51820
};
# The same `hives` attrset both hosts hold — mesh fields included,
# since "where this hive can be dialled" is a fact about that hive.
swarm.hives = {
firsthive = {
domain = "firsthive.myswarm.example";
wireguardPublicKey = "base64keyA=";
wireguardEndpoint = "198.51.100.1:51820";
wireguardAddress = "10.100.0.1/32";
};
secondhive = {
domain = "secondhive.elsewhere.example";
wireguardPublicKey = "base64keyB=";
wireguardEndpoint = "203.0.113.42:51820";
wireguardAddress = "10.100.0.2/32";
};
};
};
# hive B (secondhive.elsewhere.example, mesh IP 10.100.0.2)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.2/24";
};
swarm.hives = {
# … identical to hive A's …
};
};
```
#### What the mesh does
- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the
host (not inside agent containers; containers reach peers via the
host's routing table).
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
- `swarm-wireguard.nix` reads each entry's `wireguardAddress` directly
from `services.hyperhive.swarm.hives`, minus this host's own hive, to
build `wg-hive`'s `allowedIPs`, so intra-swarm traffic can route over
the mesh address rather than the public domain.
- It sets `persistentKeepalive = 25` by default; override or null to
disable (not needed when both sides have public IPs and no NAT).
#### NAT / one-sided endpoints
If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.
If both hosts are behind NAT, you need a STUN relay or a third host
(exit node); hyperhive sets up neither.
Each hive carries an identity (`hiveName`, the `swarm.hives` directory) that
says which hive it's and how to reach the others; the directory feeds the
swarm UI's hive list, matrix federation and the optional WireGuard mesh. →
[the hive directory](hive-directory.md). A private link between hives instead
of public HTTPS: [WireGuard mesh](wireguard.md).
## Cross-references

301
docs/2-handbook/swarm/credentials.md
View file

@ -63,299 +63,26 @@ neither is a renaming of the other.
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
| **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config |
## Swarm-level secrets
## Swarm-level and hive-level secrets
The rows are in the [reference](../../4-implementation/swarm/credentials-reference.md#swarm-level--one-of-each-per-swarm).
Nothing outside authelia's container reads its keys (session, JWT, storage encryption,
OIDC HMAC and the OIDC issuer), which is why no delivery step exists for them. **That's the
test worth applying to any secret added here** — and the client secret's plaintext half is
the one row that fails it, which is the entire reason a delivery step exists.
The store still holds those five keys, at `swarm/services/swarm-authelia/keys/<name>`,
so that losing the host doesn't lose them. `authelia-swarm-secrets` in the container reads
each with the secret publisher's identity and writes it to the file authelia loads. A store
with no value takes the file's, so the key keeps its value; the unit mints a new key only when
there is neither. Nothing rotates a key: the storage-encryption key encrypts authelia's
database, and a new issuer key invalidates every token issued. A container without the
publisher's identity generates the files as before.
The same unit mints the client secrets, straight into the store. It reads
`swarm/hives/<hive>/oidc/client` or `swarm/services/<client id>/oidc/client` first, and
only when the store answers and holds none does it run `authelia crypto hash generate`,
create the leaf and write the digest. Authelia keeps only the digest on disk, in
`oidc-clients/<id>.digest`, and starts from it while the store is down. A store that
can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries after 15 seconds, then at growing intervals up to two minutes, so authelia starts meanwhile and `swarm-authelia-bridge` finds its secret once the store answers. A client with a
secret in the store but no digest gets the digest of that secret and keeps the secret; `authelia crypto hash generate` computes the digest,
with the password typed into its prompt through a pty, never in a command line.
`swarm-authelia-bridge` fetches its own secret from
`swarm/services/swarm-authelia-bridge/oidc/client` into
`/run/swarm-authelia-bridge/client-secret` before each start, and a store that doesn't
answer keeps the bridge down while it retries. A container without the publisher's
identity keeps the plaintext in `oidc-clients/<id>.secret`, as before.
The mint reads an `oidc-clients/<id>.secret` file an earlier release left once: when the store holds
no value for that client, it stores the file's value with `-cas=0`, so the secret keeps its value.
A store that holds a value wins, whatever the file says, and the file stays on disk, unused.
Three telemetry collectors exist, and each one fails that test: each reads its
own secret out of the store, under its own identity.
The **hive's** collector authenticates to the swarm's collector as its own hive,
so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client` under the hive's own leaf
(`deploy.bao.clientCertFile`), on every topology, into `/run/hive-otel-oidc` on
the host's tmpfs. The file never reaches persistent disk and a reboot fetches
it again; until the store answers, the collector runs on an empty file and the
swarm refuses its exports. A hive with no leaf has no
default, and the build stops at the collector's identity assertion unless
`services.hyperhive.otel.clientSecretFile` names a file delivered some other way.
The **swarm's** collector reads its own under a leaf of its own, the fourth
store reader — the same shape as Grafana's, described in full below. In short:
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
out of the store on every host that runs the collector and holds a store
identity, whether authelia is a network away or in the container next door.
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
the client it reads for unconditionally: authelia refuses a
bearer-authz client with no audience, and every host publishes at least
forgejo's `/metrics` as a scrape target, so the audience is never empty.
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
where an absent OIDC block is a container with no way in at all, a collector
with nothing to authenticate with still receives every hive's telemetry —
its pushes to the stores go out unauthenticated and get refused there, and its
scrape jobs for the published targets read a credential file nothing
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
`swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.
The **secret store's own** collector — the forwarder inside the `swarm-bao`
container — needs a delivery step too, and it takes the same route with one
principal of its own: `swarm-bao-forwarder-oidc.service` reads
`swarm/services/<client-id>/oidc/client` under a leaf of its own and
writes it to `/run/swarm-bao-forwarder-oidc` on the host's tmpfs, which the container bind-mounts read-only and where `LoadCredential` hands it to the
collector. The client id is its own
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
collector admits it on a receiver of its own — one identity per principal, so
this forwarder borrows neither the swarm collector's client nor
`swarm-controller`'s.
⚠️ **This one has no degraded mode, and the difference is structural.** The
route it exports over accepts an authenticated request and nothing else, so a
forwarder without the secret has nothing to fall back to: the unit fails and
retries, and the collector refuses to start until the file arrives, after every boot, rather than
exporting into a 401 while reporting healthy. Its ordering differs from the
unit above it for the same structural reason — it reads the store inside the
container it delivers into, so it runs **after** that container rather than
before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.
## Hive-level secrets
The rows are in the [reference](../../4-implementation/swarm/credentials-reference.md#hive-level--one-of-each-per-hive).
The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
mint in authelia's container has stored the secret, rather than skipping.
A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause.
The store's **first reader** is Grafana's OIDC client secret, a _swarm
service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the mint in
authelia's container writes that path whether the reader is a network away or in the
container next door.
That's a deliberate refusal of the shortcut. Copying the plaintext
straight out of authelia's tree when the two happen to share a host is one
round trip cheaper and costs a second delivery unit, a second way for the file
to be wrong, and a gate deciding between them — and the gate is the expensive
part, because whatever it's wrong about is an outage nobody can read. The store
exists so the only secrets a host holds out of band are **certificates**, and
a host reads everything else with them.
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (this module disables its local
login form unconditionally, so SSO isn't a feature of some topologies — it's the
only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. The swarm controller, the queue, the homeserver and the forge
register the same way (`glue-controller-oidc-client.nix`,
`glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
`glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
forge read their secrets from the store too: the responder itself under its own
leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
`swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
the option pair, when that leaf is unset — neither the responder nor the
homeserver starts without its secret, and the forge registers no login source
without its.
Grafana's admin password rides the same leaf and the same store. Where Grafana
and the store run on different hosts, deploy the store's host first:
`swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create
the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
the Grafana host, which retries, while Grafana stays down because its config
reads that file.
The **second reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
doesn't render the reading unit without them.
The **third reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
second in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not
anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix`
refuses the build and names `deploy.bao.forwarderOidcClientCertFile` /
`forwarderOidcClientKeyFile`.
A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs
a given service — placement is a `deploy.*` fact, per-host by definition — so
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
every swarm service's client secret, and that's stated in
`swarm-secret-client`'s `policy` module beside the grant itself.
⚠️ **The readers no longer present it.** Five units used to log in with
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
principal by the subject of the certificate it presents — so five readers behind
one leaf were one principal holding the union of their needs. Each now holds a
leaf of its own, and a policy naming only the path that unit reads. See
[per-principal identities](../../4-implementation/swarm/credentials.md#per-principal-identities).
Absence behaves
as it does for the other readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in.
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
credential you must already hold to authenticate from the thing it
authenticates you to, so the store's identity can't come from an authority the store
distributes — which excludes the store's `pki` root, and has nothing to do
with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
mints a CA that signs a fixed, short list of leaves — the store's server
certificate, the hive's own reader certificate, and one per service principal —
and distributes nothing. A deployment with a real
internal CA deletes that file and names its own paths in
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
A hive that reads from a store on **another** machine names the reader's half —
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
plus one leaf per principal it runs (the [table in the reference](../../4-implementation/swarm/credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
credentials that can't come out of the store, being what opens it; everything
else a hive needs does.
Agents are the one principal whose leaf the store issues. Its `pki-agents`
mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the
store, and `swarm-controller`, already logged in under its own host leaf, asks
that mount's one role for a `hive-agent-<agent>` client certificate at agent
creation. Host roles never pin that CA and agent roles pin only it, so an
agent's certificate opens that agent's own `swarm/agents/<agent>/*` and
nothing else: `read` on the values there and `list` on their names, and `create` (never `update`) on one leaf, `backend-env`, which the agent seeds from its environment file.
All agents share one policy, `hive-agent`, which the agent's cert-auth role
attaches and no other. It states the rule with `{{identity.entity.name}}`
where an agent's name would go, so the rule is one document and a change to it
is one write. `swarm-controller` writes it at agent creation and at its own
start.
For that the controller names each agent's identity entity after the agent: at
each certificate mint or renewal, before it writes the agent's cert-auth role,
and for every agent at its own start. It reads the cert mount's accessor and
creates the entity `<agent>` with the alias `hive-agent-<agent>`, or renames the
entity a first login created under a generated name. At a mint or renewal this is
a precondition: when the store refuses a call, or the alias sits on any other
entity, the mint fails and the role stays as it was, so a live agent keeps
working until an operator resolves it. The start pass only logs the same
failures, because it writes no role. The controller needs the `identity/` and
`sys/auth/cert` grants in `swarm-bao.nix`'s `controllerPolicyText` for that.
A cert-auth role that predates the shared policy may still name a policy
`hive-agent-<agent>`. The controller no longer writes or attaches these. The
objects stay in the store and keep granting what they did, to a token that
already carries them and to a role not yet rewritten. A live agent's role is
rewritten at its next certificate renewal.
A hive's cert-auth role `hive-<hive>` attaches one policy, `hive-hives`, shared
by all hives. A hive's certificate carries its bare name as
the common name, which is the login's alias name, so hives need no entity:
the policy states the hive's own path as
`swarm/hives/{{identity.entity.aliases.<accessor>.name}}/*`, where `<accessor>`
is the cert mount's accessor. The controller reads the accessor at its own start,
writes the policy, and rewrites the roles to attach it. No hive name can
spell `hive-hives`: it would be a hive called `hives`, and
`reserved-hive-fragments.nix` refuses the substring `hive`. When the controller
can't read the accessor or write the policy it logs a warning and leaves the
roles untouched, so a role that names a `hive-<hive>` policy keeps it.
### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
opening the nix.
That condition fires on exactly one shape, the hand-configured remote-store hive
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
that matters on its own:
- **A host with no store identity at all** holds no `clientCertFile` either, so
the first clause is false. That's the supported no-store deployment, and also
the one an operator passes through while bringing a hive up — neither should
fail to evaluate.
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no collector, no Grafana — has no
unit to skip, so the second clause is false.
The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives
telemetry. The assertion doesn't touch that shape, because that host has no
`clientCertFile` either.
Swarm-level secrets belong to a swarm service (authelia's keys and client
secrets, the collectors' and Grafana's OIDC secrets, the forge and queue
credentials); hive-level ones belong to a hive and its agents. Each has a unit
that mints or copies it and a reader that holds it only in memory. Which unit,
where the value lands and what a missing identity costs, secret by secret:
[how each credential reaches its reader](../../4-implementation/swarm/credential-delivery.md).
## The browser UI
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
(`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in
with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost
checks the session with authelia and proxies to an nginx inside the store's
container on
`127.0.0.1:<deploy.bao.uiProxyPort>`. That nginx forwards `/ui/` and `/v1/` to
a second openbao listener on `127.0.0.1:<deploy.bao.uiPort>`, answers 403 on
the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on
everything else. ⚠️ That listener asks for **no client certificate**, because
a browser has none to present, so on this one door a bao token is the whole
credential. Anything on the store's host that can dial loopback, and any
`admins` session through the vhost, needs only a token to use the API. Unseal
from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed
gateway where the UI is the only swarm name its host fronts, that host's nginx
doesn't wait for the store: it serves the hive leaf, or on a host that runs no
hive a throwaway self-signed certificate, on the UI's name (a browser warning)
until the unsealed store issues the services leaf, so the
stream passthrough readers use on that host comes up with the store sealed.
with the **OIDC** tab, or with a bao token under **Other**. ⚠️ The UI's
listener asks for **no client certificate**, because a browser has none to
present, so on this one door a bao token is the whole credential: anything on
the store's host that can dial loopback, and any `admins` session through the
vhost, needs only a token to use the API. Unseal from the host's `bao` CLI;
the UI's unseal form gets a 403. The proxy:
[the secret store browser UI](../../4-implementation/swarm/credential-delivery.md#the-secret-store-browser-ui).
## Adding a secret

141
docs/2-handbook/swarm/hive-directory.md Normal file
View file

@ -0,0 +1,141 @@
# The hive directory
How a hive knows which swarm it belongs to and how to reach the others: the
identity each hive carries, what the directory feeds, and how a hive reaches
the queue and the secret store on an address other hives can route to. The
[swarm page](README.md#hives-the-substrate) says what a hive is.
## Hive identity config
```nix
services.hyperhive = {
swarm = {
domain = "myswarm.example"; # required — the swarm's DNS domain
name = "myswarm"; # shared swarm display name (optional)
# required — the directory, identical on every host in the swarm.
# Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
hives = {
firsthive = { }; # this host, per hiveName
secondhive = { }; # a second hive
thirdhive.domain = "thirdhive.elsewhere.example"; # addressed off-convention
};
};
# required on a hive host — this hive's label in the directory
deploy.hive-controller.hiveName = "firsthive";
};
```
<!-- vale write-good.Passive = NO -->
Swarm options are identical on every host in the swarm, so `swarm.domain` is
**required** on every host that runs any hyperhive service; a host that
imports the module and enables nothing evaluates without it. `hiveName` is
required on a host that runs a hive (`deploy.hive-controller.hiveName`), and on no other. Eval fails with a hint naming
each. Neither defaults, because a guessed value here is a wrong hostname
that evaluates cleanly and deploys.
<!-- vale write-good.Passive = YES -->
**The directory is one attrset, identical on every host.** It describes
every hive in the swarm, **including this one**, keyed by `hiveName`; what
differs between hosts is only `hiveName`. It **must** contain an entry for
this host's `hiveName`, and an empty directory fails that check too — eval
names the missing hive. Peers are _every entry but this host's_, so a
directory without this host would make every hive a peer, itself included.
```
# hive A # hive B
deploy.hive-controller.hiveName = "firsthive"; deploy.hive-controller.hiveName = "secondhive";
swarm.hives = { … }; swarm.hives = { … }; # byte-identical
```
One entry per hive means two hosts can't hold _different_ facts about the
same third hive, such as a stale endpoint.
`domain` defaults to `<name>.<swarm.domain>`, so a conventional directory
is names only. Set it only for a hive addressed by something else. This
hive's own `services.hyperhive.domain` comes from its entry; it drives
`HYPERHIVE_HIVE_DOMAIN` in every container so agents can form qualified
labels (`agent1@firsthive.myswarm.example`). Setting `services.hyperhive.domain`
directly overrides the directory entry, with a deprecation warning: the
option is local to this host while every host shares the directory, so a
value written only here is invisible to the rest of the swarm.
`swarm.name` is display only — the dashboard chrome header and per-agent
system prompts — and federated hives at different domains can share one.
`hiveName` surfaces in the same places but is also the leftmost label of the
hive's domain. `swarm.name` names the group; `hiveName` names this hive.
How a hive name qualifies an agent, and the env vars that carry it:
[`conventions.md`](../../5-project/process/conventions.md) § Hive identity.
Trust inside a swarm comes from the secret store's `pki` root
([`ca.md`](../../4-implementation/swarm/ca.md)): every hive's gateway leaf chains to it. Trusting a
hive whose root this swarm doesn't own — another swarm's — has no
mechanism.
## What the directory feeds
1. **Swarm-wide hive roster** — swarm-controller reads this same
directory and serves it at `GET /api/hives`; the swarm UI's front page
renders it ([`ui.md`](ui.md)).
2. **Matrix federation** — when this host runs the homeserver (`deploy.matrix.enable`), tuwunel
federates with the peer's matrix server (discovered via the peer's
`.well-known/matrix/server` delegation, which the controller's host or the pinned hive's gateway serves).
Federation validates the peer's TLS certificate against the matrix
**container's** trust bundle, independent of this directory.
⚠️ On a hive with self-signed gateway certificates, the container
also trusts this hive's `trust-bundle.pem`, bound in at runtime and
added to the public CAs; it holds the store's `pki` root, so a peer whose
certificate chains to the same root validates. A peer outside this
swarm's root needs a CA-issued certificate (ACME). Federation firewall
and TLS requirements: [`2-handbook/swarm/matrix.md`](matrix.md).
3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
to configure `wg-hive`. See [the WireGuard page](wireguard.md).
## Reaching the queue and the store on a routable address
The mesh isn't the only way to reach the queue and the secret store from
another host. Set `deploy.nats.openFirewall` on the queue's host to open the
queue's port on every interface, and `deploy.bao.openFirewall` together with
`deploy.bao.extraListenAddresses` on the store's host to open the store's port
on the addresses it listens on. Both default to off. Each exposes a service
beyond the tunnel, which is the operator's decision. The queue requires TLS and
its `auth_callout` refuses every client it can't identify. The store requires a
client certificate on every listener, and evaluation fails when you
set `deploy.bao.openFirewall` without `deploy.bao.clientCaFile`. Point
`nats.<swarm.domain>` and `bao.<swarm.domain>` in your upstream DNS at the
address the other hives reach.
## Keeping a vhost off the public address
The gateway is one nginx per host, and a vhost answers on every address of the
host by default, IPv4 and IPv6. To serve some vhosts only inside a private
network (a VPN, say), list the host's addresses on that network in
`services.hyperhive.gateway.privateListenAddresses` and set `public = false`
on the vhosts that should stay inside:
```nix
services.hyperhive.gateway = {
privateListenAddresses = [ "10.100.0.1" "fd00:100::1" ];
vhosts."grafana.myswarm.example".public = false;
};
```
A `public = false` vhost listens on those addresses only. The public address
never serves it, and a request for its name there closes without a response.
Vhosts that keep the default `public = true` answer on the private addresses
too. The addresses can be IPv4 or IPv6 (`fd00::1` or `[fd00::1]`), and each
must exist on the host when nginx starts. The firewall stays your call: this
decides which vhosts nginx serves on which address, not which ports are open.
The gateway also listens on `[::]` by default, so the vhosts are reachable
over IPv6. `gateway.listenIPv6` follows `networking.enableIPv6`; turn it off
on a host whose kernel can't open an IPv6 socket. Details are in
[`gateway.md`](../../4-implementation/networking/gateway.md#listen-addresses).

80
docs/2-handbook/swarm/wireguard.md Normal file
View file

@ -0,0 +1,80 @@
# WireGuard mesh between hives (optional)
The peer config in [the hive directory](hive-directory.md#hive-identity-config) uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hyperhive can configure a host-to-host
WireGuard mesh.
## Generating keys
On each hive host:
```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
```
## Config example (two hives)
```nix
# hive A (firsthive.myswarm.example, mesh IP 10.100.0.1)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.1/24";
listenPort = 51820; # optional, default 51820
};
# The same `hives` attrset both hosts hold — mesh fields included,
# since "where this hive can be dialled" is a fact about that hive.
swarm.hives = {
firsthive = {
domain = "firsthive.myswarm.example";
wireguardPublicKey = "base64keyA=";
wireguardEndpoint = "198.51.100.1:51820";
wireguardAddress = "10.100.0.1/32";
};
secondhive = {
domain = "secondhive.elsewhere.example";
wireguardPublicKey = "base64keyB=";
wireguardEndpoint = "203.0.113.42:51820";
wireguardAddress = "10.100.0.2/32";
};
};
};
# hive B (secondhive.elsewhere.example, mesh IP 10.100.0.2)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.2/24";
};
swarm.hives = {
# … identical to hive A's …
};
};
```
## What the mesh does
- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the
host (not inside agent containers; containers reach peers via the
host's routing table).
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
- It reads each entry's `wireguardAddress` from `services.hyperhive.swarm.hives`,
minus this host's own hive, to build `wg-hive`'s `allowedIPs`, so intra-swarm
traffic can route over the mesh address rather than the public domain.
- It sets `persistentKeepalive = 25` by default; override or null to
disable (not needed when both sides have public IPs and no NAT).
## NAT / one-sided endpoints
If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.
If both hosts are behind NAT, you need a STUN relay or a third host
(exit node); hyperhive sets up neither.

2
docs/4-implementation/networking/network.md
View file

@ -109,7 +109,7 @@ The hive domain (`services.hyperhive.domain`, which the resolver answers
for) comes from this hive's entry in the swarm directory:
`swarm.hives.<hiveName>.domain`, default `<hiveName>.<swarm.domain>`.
Eval fails until `swarm.domain` and that entry exist
([`2-handbook/swarm/README.md`](../../2-handbook/swarm/README.md) § Hive identity config).
([`2-handbook/swarm/hive-directory.md`](../../2-handbook/swarm/hive-directory.md#hive-identity-config)).
## Bridge addressing

11
docs/4-implementation/swarm/controller.md
View file

@ -435,3 +435,14 @@ refused delivery logs `webhook: refused delivery` with the reason.
The controller registers the hooks within five minutes of having both the forge
token and the secret. It keeps trying, at most five minutes apart, for as long
as the forge or the secret store is down. A restart isn't needed to recover.
## Agent identity minting
A store identity that fails to mint logs `swarm jobq: mint_agent_identity
failed` with the agent, the step it stopped at and the error, which the Grafana
alert rule `hh-agent-identity-mint-failed` fires on. The agent is still
created, without a store identity, and the controller's five-minute credential
pass mints one for it (also when only the certificate exists and its cert-auth
role is missing) as long as the agent isn't `Destroyed`. A mint that runs after
a hive declares the agent `Destroyed` logs `identity mint skipped` and writes
nothing.

302
docs/4-implementation/swarm/credential-delivery.md Normal file
View file

@ -0,0 +1,302 @@
# How each credential reaches its reader
The swarm-level and hive-level secrets, one by one: which unit mints each, where
it lands, and what a missing identity costs. The rules a credential follows are
on [the handbook's credentials page](../../2-handbook/swarm/credentials.md); the
table of every credential is in
[`credentials-reference.md`](credentials-reference.md), and the store's own
mechanics are in [`credentials.md`](credentials.md).
## Swarm-level secrets
The rows are in the [reference](credentials-reference.md#swarm-level--one-of-each-per-swarm).
Nothing outside authelia's container reads its keys (session, JWT, storage encryption,
OIDC HMAC and the OIDC issuer), which is why no delivery step exists for them. **That's the
test worth applying to any secret added here** — and the client secret's plaintext half is
the one row that fails it, which is the entire reason a delivery step exists.
The store still holds those five keys, at `swarm/services/swarm-authelia/keys/<name>`,
so that losing the host doesn't lose them. `authelia-swarm-secrets` in the container reads
each with the secret publisher's identity and writes it to the file authelia loads. A store
with no value takes the file's, so the key keeps its value; the unit mints a new key only when
there is neither. Nothing rotates a key: the storage-encryption key encrypts authelia's
database, and a new issuer key invalidates every token issued. A container without the
publisher's identity generates the files as before.
The same unit mints the client secrets, straight into the store. It reads
`swarm/hives/<hive>/oidc/client` or `swarm/services/<client id>/oidc/client` first, and
only when the store answers and holds none does it run `authelia crypto hash generate`,
create the leaf and write the digest. Authelia keeps only the digest on disk, in
`oidc-clients/<id>.digest`, and starts from it while the store is down. A store that
can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries after 15 seconds, then at growing intervals up to two minutes, so authelia starts meanwhile and `swarm-authelia-bridge` finds its secret once the store answers. A client with a
secret in the store but no digest gets the digest of that secret and keeps the secret; `authelia crypto hash generate` computes the digest,
with the password typed into its prompt through a pty, never in a command line.
`swarm-authelia-bridge` fetches its own secret from
`swarm/services/swarm-authelia-bridge/oidc/client` into
`/run/swarm-authelia-bridge/client-secret` before each start, and a store that doesn't
answer keeps the bridge down while it retries. A container without the publisher's
identity keeps the plaintext in `oidc-clients/<id>.secret`, as before.
The mint reads an `oidc-clients/<id>.secret` file an earlier release left once: when the store holds
no value for that client, it stores the file's value with `-cas=0`, so the secret keeps its value.
A store that holds a value wins, whatever the file says, and the file stays on disk, unused.
Three telemetry collectors exist, and each one fails that test: each reads its
own secret out of the store, under its own identity.
The **hive's** collector authenticates to the swarm's collector as its own hive,
so it reads its secret as the hive: `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client` under the hive's own leaf
(`deploy.bao.clientCertFile`), on every topology, into `/run/hive-otel-oidc` on
the host's tmpfs. The file never reaches persistent disk and a reboot fetches
it again; until the store answers, the collector runs on an empty file and the
swarm refuses its exports. A hive with no leaf has no
default, and the build stops at the collector's identity assertion unless
`services.hyperhive.otel.clientSecretFile` names a file delivered some other way.
The **swarm's** collector reads its own under a leaf of its own, the fourth
store reader — the same shape as Grafana's, described in full below. In short:
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
out of the store on every host that runs the collector and holds a store
identity, whether authelia is a network away or in the container next door.
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
the client it reads for unconditionally: authelia refuses a
bearer-authz client with no audience, and every host publishes at least
forgejo's `/metrics` as a scrape target, so the audience is never empty.
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
where an absent OIDC block is a container with no way in at all, a collector
with nothing to authenticate with still receives every hive's telemetry —
its pushes to the stores go out unauthenticated and get refused there, and its
scrape jobs for the published targets read a credential file nothing
loads, so they can't get a token. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, rather than the hard assertion
`swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.
The **secret store's own** collector — the forwarder inside the `swarm-bao`
container — needs a delivery step too, and it takes the same route with one
principal of its own: `swarm-bao-forwarder-oidc.service` reads
`swarm/services/<client-id>/oidc/client` under a leaf of its own and
writes it to `/run/swarm-bao-forwarder-oidc` on the host's tmpfs, which the container bind-mounts read-only and where `LoadCredential` hands it to the
collector. The client id is its own
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
collector admits it on a receiver of its own — one identity per principal, so
this forwarder borrows neither the swarm collector's client nor
`swarm-controller`'s.
⚠️ **This one has no degraded mode, and the difference is structural.** The
route it exports over accepts an authenticated request and nothing else, so a
forwarder without the secret has nothing to fall back to: the unit fails and
retries, and the collector refuses to start until the file arrives, after every boot, rather than
exporting into a 401 while reporting healthy. Its ordering differs from the
unit above it for the same structural reason — it reads the store inside the
container it delivers into, so it runs **after** that container rather than
before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.
## Hive-level secrets
The rows are in the [reference](credentials-reference.md#hive-level--one-of-each-per-hive).
The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
mint in authelia's container has stored the secret, rather than skipping.
A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause.
The store's **first reader** is Grafana's OIDC client secret, a _swarm
service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the mint in
authelia's container writes that path whether the reader is a network away or in the
container next door.
That's a deliberate refusal of the shortcut. Copying the plaintext
straight out of authelia's tree when the two happen to share a host is one
round trip cheaper and costs a second delivery unit, a second way for the file
to be wrong, and a gate deciding between them — and the gate is the expensive
part, because whatever it's wrong about is an outage nobody can read. The store
exists so the only secrets a host holds out of band are **certificates**, and
a host reads everything else with them.
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (this module disables its local
login form unconditionally, so SSO isn't a feature of some topologies — it's the
only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. The swarm controller, the queue, the homeserver and the forge
register the same way (`glue-controller-oidc-client.nix`,
`glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
`glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
forge read their secrets from the store too: the responder itself under its own
leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
`swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
the option pair, when that leaf is unset — neither the responder nor the
homeserver starts without its secret, and the forge registers no login source
without its.
Grafana's admin password rides the same leaf and the same store. Where Grafana
and the store run on different hosts, deploy the store's host first:
`swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create
the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
the Grafana host, which retries, while Grafana stays down because its config
reads that file.
The **second reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
doesn't render the reading unit without them.
The **third reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
second in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not
anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix`
refuses the build and names `deploy.bao.forwarderOidcClientCertFile` /
`forwarderOidcClientKeyFile`.
A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs
a given service — placement is a `deploy.*` fact, per-host by definition — so
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
every swarm service's client secret, and that's stated in
`swarm-secret-client`'s `policy` module beside the grant itself.
⚠️ **The readers no longer present it.** Five units used to log in with
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
principal by the subject of the certificate it presents — so five readers behind
one leaf were one principal holding the union of their needs. Each now holds a
leaf of its own, and a policy naming only the path that unit reads. See
[per-principal identities](credentials.md#per-principal-identities).
Absence behaves
as it does for the other readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in.
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
credential you must already hold to authenticate from the thing it
authenticates you to, so the store's identity can't come from an authority the store
distributes — which excludes the store's `pki` root, and has nothing to do
with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
mints a CA that signs a fixed, short list of leaves — the store's server
certificate, the hive's own reader certificate, and one per service principal —
and distributes nothing. A deployment with a real
internal CA deletes that file and names its own paths in
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
A hive that reads from a store on **another** machine names the reader's half —
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
plus one leaf per principal it runs (the [table in the reference](credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
credentials that can't come out of the store, being what opens it; everything
else a hive needs does.
Agents are the one principal whose leaf the store issues. Its `pki-agents`
mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the
store, and `swarm-controller`, already logged in under its own host leaf, asks
that mount's one role for a `hive-agent-<agent>` client certificate at agent
creation. Host roles never pin that CA and agent roles pin only it, so an
agent's certificate opens that agent's own `swarm/agents/<agent>/*` and
nothing else: `read` on the values there and `list` on their names, and `create` (never `update`) on one leaf, `backend-env`, which the agent seeds from its environment file.
All agents share one policy, `hive-agent`, which the agent's cert-auth role
attaches and no other. It states the rule with `{{identity.entity.name}}`
where an agent's name would go, so the rule is one document and a change to it
is one write. `swarm-controller` writes it at agent creation and at its own
start.
For that the controller names each agent's identity entity after the agent: at
each certificate mint or renewal, before it writes the agent's cert-auth role,
and for every agent at its own start. It reads the cert mount's accessor and
creates the entity `<agent>` with the alias `hive-agent-<agent>`, or renames the
entity a first login created under a generated name. At a mint or renewal this is
a precondition: when the store refuses a call, or the alias sits on any other
entity, the mint fails and the role stays as it was, so a live agent keeps
working until an operator resolves it. The start pass only logs the same
failures, because it writes no role. The controller needs the `identity/` and
`sys/auth/cert` grants in `swarm-bao.nix`'s `controllerPolicyText` for that.
A cert-auth role that predates the shared policy may still name a policy
`hive-agent-<agent>`. The controller no longer writes or attaches these. The
objects stay in the store and keep granting what they did, to a token that
already carries them and to a role not yet rewritten. A live agent's role is
rewritten at its next certificate renewal.
A hive's cert-auth role `hive-<hive>` attaches one policy, `hive-hives`, shared
by all hives. A hive's certificate carries its bare name as
the common name, which is the login's alias name, so hives need no entity:
the policy states the hive's own path as
`swarm/hives/{{identity.entity.aliases.<accessor>.name}}/*`, where `<accessor>`
is the cert mount's accessor. The controller reads the accessor at its own start,
writes the policy, and rewrites the roles to attach it. No hive name can
spell `hive-hives`: it would be a hive called `hives`, and
`reserved-hive-fragments.nix` refuses the substring `hive`. When the controller
can't read the accessor or write the policy it logs a warning and leaves the
roles untouched, so a role that names a `hive-<hive>` policy keeps it.
### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
opening the nix.
That condition fires on exactly one shape, the hand-configured remote-store hive
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
that matters on its own:
- **A host with no store identity at all** holds no `clientCertFile` either, so
the first clause is false. That's the supported no-store deployment, and also
the one an operator passes through while bringing a hive up — neither should
fail to evaluate.
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no collector, no Grafana — has no
unit to skip, so the second clause is false.
The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives
telemetry. The assertion doesn't touch that shape, because that host has no
`clientCertFile` either.
## The secret store browser UI
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
(`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in
with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost
checks the session with authelia and proxies to an nginx inside the store's
container on
`127.0.0.1:<deploy.bao.uiProxyPort>`. That nginx forwards `/ui/` and `/v1/` to
a second openbao listener on `127.0.0.1:<deploy.bao.uiPort>`, answers 403 on
the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on
everything else. ⚠️ That listener asks for **no client certificate**, because
a browser has none to present, so on this one door a bao token is the whole
credential. Anything on the store's host that can dial loopback, and any
`admins` session through the vhost, needs only a token to use the API. Unseal
from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed
gateway where the UI is the only swarm name its host fronts, that host's nginx
doesn't wait for the store: it serves the hive leaf, or on a host that runs no
hive a throwaway self-signed certificate, on the UI's name (a browser warning)
until the unsealed store issues the services leaf, so the
stream passthrough readers use on that host comes up with the store sealed.

2
docs/4-implementation/swarm/credentials.md
View file

@ -168,7 +168,7 @@ On a host without the store, where the forwarder never runs, forgetting one of
the others isn't a quiet degrade. A hand-configured remote-store hive that
names the hive's own `clientCertFile` and misses a principal's pair has the
build refused, by the module of the reader it missed, naming that pair — see
[what a missing leaf costs](../../2-handbook/swarm/credentials.md#what-a-missing-leaf-costs) below.
[what a missing leaf costs](credential-delivery.md#what-a-missing-leaf-costs).
The roles of `swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth` issue
batch tokens with a five-minute TTL (`token_type=batch`, `token_ttl=5m`). The