docs: the credentials page keeps the approach and moves the mechanics out
credentials.md drops the target-shape framing and keeps the rules, where a secret lives, the swarm- and hive-level secrets and how to add one. The queue responder's keys, per-principal identities, how a reader reaches the store, the store UI's login and where the root lives move to 4-implementation/swarm/credentials.md. The per-secret reference moves to 4-implementation/swarm/credentials-reference.md. Links and comments follow.
This commit is contained in:
parent
c3791a0c8a
commit
ead667da42
15 changed files with 350 additions and 346 deletions
|
|
@ -50,7 +50,7 @@ when the token changed. It needs a store identity
|
|||
(`services.hyperhive.agent.bao.addr`); an agent without one gets no token.
|
||||
The unit never deletes the file: a `github-token` already in place stays
|
||||
when the store holds none or doesn't answer. Where the token lives and who
|
||||
reads it: [credentials-reference.md](../swarm/credentials-reference.md).
|
||||
reads it: [credentials-reference.md](../../4-implementation/swarm/credentials-reference.md).
|
||||
|
||||
## Security
|
||||
|
||||
|
|
|
|||
|
|
@ -52,7 +52,7 @@ grafana, the controller, authelia, …) →
|
|||
→ [`sso.md`](sso.md)
|
||||
- **secrets** — the rules a credential follows → [`credentials.md`](credentials.md);
|
||||
every credential the swarm holds, who mints it, who reads it, how it renews
|
||||
and where it lives → [`credentials-reference.md`](credentials-reference.md) ·
|
||||
and where it lives → [`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md) ·
|
||||
how the store comes up, who writes its grants and how it unseals →
|
||||
[`bao.md`](../../4-implementation/swarm/bao.md)
|
||||
- **swarm CA** — the store root every hive's gateway leaf chains to, and what to
|
||||
|
|
|
|||
|
|
@ -187,4 +187,4 @@ on disk. The archive host renders its users file (password hashes only) into
|
|||
`/run/swarm-archive-auth/`, which is tmpfs. With an external S3, it reads the
|
||||
S3 key from the store when a process starts and passes it to rclone in the
|
||||
environment. Hives read their restic password when restic asks for it. The paths and who writes and reads them are in the
|
||||
[credentials reference](credentials-reference.md).
|
||||
[credentials reference](../../4-implementation/swarm/credentials-reference.md).
|
||||
|
|
|
|||
|
|
@ -1,14 +1,10 @@
|
|||
# Credentials: the target shape
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
# Credentials
|
||||
|
||||
The swarm's credential store is bao. This page is the overview of how the swarm
|
||||
treats credentials; [`credentials-reference.md`](credentials-reference.md) lists
|
||||
each one: who mints it, who reads it, and how it renews, with the notes for
|
||||
operating them. It describes the target, not what's on disk today; the second half of the
|
||||
reference maps [the files that exist right now](credentials-reference.md#where-each-secret-lives-on-disk).
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
treats credentials; [`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md)
|
||||
lists each one: who mints it, who reads it, and how it renews, with the notes for
|
||||
operating them, and [`credentials.md`](../../4-implementation/swarm/credentials.md)
|
||||
holds the store's mechanics.
|
||||
|
||||
**Store secrets stay off disk.** A process pulls a value from the store at
|
||||
runtime and holds it in memory only.
|
||||
|
|
@ -23,7 +19,7 @@ itself.
|
|||
|
||||
**Each credential has a minter, a reader and a renewal story.** Who mints it, who
|
||||
reads it, and whether it renews without an operator are what the
|
||||
[reference table](credentials-reference.md) gives for every credential, with the
|
||||
[reference table](../../4-implementation/swarm/credentials-reference.md) gives for every credential, with the
|
||||
operating notes that go with them.
|
||||
|
||||
## Where a secret lives
|
||||
|
|
@ -55,7 +51,7 @@ container that starts after it.
|
|||
|
||||
### Topologies, by who places secrets
|
||||
|
||||
Read every row of the [reference tables](credentials-reference.md#where-each-secret-lives-on-disk) against one of these. This is a different cut
|
||||
Read every row of the [reference tables](../../4-implementation/swarm/credentials-reference.md#where-each-secret-lives-on-disk) against one of these. This is a different cut
|
||||
from the [deployment shapes](services.md#deployment-shapes) --- those
|
||||
say _where services run_, these say _who is responsible for a secret
|
||||
file being there_ --- the two lists don't line up one-to-one, and
|
||||
|
|
@ -69,7 +65,7 @@ neither is a renaming of the other.
|
|||
|
||||
## Swarm-level secrets
|
||||
|
||||
The rows are in the [reference](credentials-reference.md#swarm-level--one-of-each-per-swarm).
|
||||
The rows are in the [reference](../../4-implementation/swarm/credentials-reference.md#swarm-level--one-of-each-per-swarm).
|
||||
|
||||
Nothing outside authelia's container reads its keys (session, JWT, storage encryption,
|
||||
OIDC HMAC and the OIDC issuer), which is why no delivery step exists for them. **That's the
|
||||
|
|
@ -157,126 +153,9 @@ unit above it for the same structural reason — it reads the store inside the
|
|||
container it delivers into, so it runs **after** that container rather than
|
||||
before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.
|
||||
|
||||
### The queue's callout nkeys
|
||||
|
||||
The responder holds two nkeys besides the callout xkey below. It signs every
|
||||
user JWT it issues with the **issuer** key, an account key, so possession of
|
||||
the issuer seed is the authority to admit anyone to the queue. It connects to
|
||||
the queue as the **user** key, the one identity the server exempts from
|
||||
callout. You configure nothing about either: both live in the swarm secret
|
||||
store beside the xkey, and come into being on the queue's first boot.
|
||||
|
||||
- **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`,
|
||||
**public half** at `secret/swarm/public/swarm-nats/callout-issuer`.
|
||||
- **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`,
|
||||
**public half** at `secret/swarm/public/swarm-nats/callout-user`.
|
||||
|
||||
The unit that creates the xkey creates these too, under the same rules: only
|
||||
the `swarm-nats-auth` role reads or creates them, nothing replaces a stored
|
||||
seed, and the policy of `swarm-secret-publisher` denies it both seed paths.
|
||||
The server renders both public halves into its `auth_callout` block as it
|
||||
starts, and doesn't start without them.
|
||||
|
||||
A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or
|
||||
`/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what
|
||||
`swarm-nats-auth ensure-callout-keys` stores for that key when the store holds
|
||||
none, so the key keeps its public half. Once a read of the store returns a
|
||||
seed, the verb removes the file. When the store already holds a different
|
||||
seed, the verb keeps the store's seed, logs a warning naming the file, then
|
||||
removes it.
|
||||
|
||||
A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`,
|
||||
`calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`,
|
||||
under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the
|
||||
option and where the keys live now.
|
||||
|
||||
To replace either key, delete both of its paths the way the xkey's section
|
||||
below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`.
|
||||
|
||||
One consequence worth knowing before you debug it: the queue assembles its
|
||||
config each time the server starts rather than at build time, in every mode,
|
||||
because the callout keys exist only at runtime. A malformed config
|
||||
surfaces when the server starts instead of when the system builds. The server
|
||||
names the offending file and refuses to run.
|
||||
|
||||
### The queue's callout xkey
|
||||
|
||||
The server seals every auth-callout request to a curve key, the callout
|
||||
xkey, and the responder seals its reply back, so nothing else in the callout
|
||||
account or on the wire can read the request or the user JWT minted for it.
|
||||
You configure nothing about the key: it lives in the swarm secret store and
|
||||
comes into being on the queue's first boot.
|
||||
|
||||
- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`.
|
||||
- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`.
|
||||
|
||||
Only the `swarm-nats-auth` role reads either path, and it may create each but
|
||||
never replace it. The policy of `swarm-secret-publisher`, which writes the rest
|
||||
of `swarm/services/`, denies it the seed's path.
|
||||
|
||||
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
|
||||
`swarm-nats` container, under the responder's store leaf:
|
||||
|
||||
1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no
|
||||
seed, and otherwise only reads it. It needs no queue, so on a first boot
|
||||
the key exists before the server that needs it.
|
||||
2. It reads the public half back from the store into
|
||||
`/var/lib/nats-callout/callout-xkey.pub` inside the container.
|
||||
3. Each time `nats-server` starts, it renders that key into
|
||||
`auth_callout.xkey`. With no file, the server doesn't start.
|
||||
4. The responder starts after the server and reads the seed from the store.
|
||||
A responder that can't read it doesn't start, rather than answering
|
||||
requests it can't open.
|
||||
|
||||
The store has to be reachable each time the responder starts. When it's down,
|
||||
the fetch unit fails and retries for a day. The server starts with the key
|
||||
fetched last time, or not at all on a first boot. A responder that starts
|
||||
while the store is down exits and retries every 5 seconds, and until one
|
||||
start reads the seed the queue refuses every client, OIDC ones included. The
|
||||
retry that lands the file starts or restarts the server, and the responder
|
||||
comes up with it.
|
||||
|
||||
The responder's store leaf is `deploy.nats.authBaoClientCertFile` /
|
||||
`authBaoClientKeyFile`. On the store's host both default to the leaf
|
||||
`glue-bao-tls.nix` mints there. A queue host without the store gets no
|
||||
default, even when it holds the hive's own reader `deploy.bao.clientCertFile`:
|
||||
issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and
|
||||
name it in both options. Until you do, evaluation fails naming both.
|
||||
|
||||
Where the queue and the store run on different hosts, deploy the store's host
|
||||
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the
|
||||
keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of
|
||||
the seeds' paths; where the queue runs on that host too, the fetch waits for
|
||||
both. A queue host deployed before it can't read or create the keys, so its
|
||||
fetch fails, and the queue refuses every client until you deploy the store's
|
||||
host: the server starts only on public keys fetched before, and the responder
|
||||
can't load its seeds. Seed files the import is waiting for stay where they
|
||||
are. If that takes longer than the fetch's day of retries, run
|
||||
`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
|
||||
|
||||
Two starts that create the key at once can't both win: each write is a
|
||||
create that fails once the path holds anything, and the start that lost reads
|
||||
the winner's key back.
|
||||
|
||||
To replace the key, remove both paths with their metadata and every version,
|
||||
then restart the fetch unit on the queue's host. Its restart of `nats.service`
|
||||
restarts the responder too, which then loads the new seed:
|
||||
|
||||
```sh
|
||||
bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
|
||||
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
|
||||
systemctl restart swarm-bao-nats-callout-xkey
|
||||
```
|
||||
|
||||
`bao kv delete` isn't enough. It deletes the latest version but keeps the
|
||||
path's metadata, so the store still counts the path as existing and refuses
|
||||
the create that makes a new key. No principal hyperhive configures may
|
||||
delete either path's metadata, so run the two deletes with a token that may,
|
||||
such as the root token.
|
||||
|
||||
## Hive-level secrets
|
||||
|
||||
The rows are in the [reference](credentials-reference.md#hive-level--one-of-each-per-hive).
|
||||
The rows are in the [reference](../../4-implementation/swarm/credentials-reference.md#hive-level--one-of-each-per-hive).
|
||||
|
||||
The forge and the homeserver read their copies out of the store, the way
|
||||
Grafana does below: the forge under the forge publisher's leaf, the
|
||||
|
|
@ -365,7 +244,7 @@ every swarm service's client secret, and that's stated in
|
|||
principal by the subject of the certificate it presents — so five readers behind
|
||||
one leaf were one principal holding the union of their needs. Each now holds a
|
||||
leaf of its own, and a policy naming only the path that unit reads. See
|
||||
[per-principal identities](#per-principal-identities) below.
|
||||
[per-principal identities](../../4-implementation/swarm/credentials.md#per-principal-identities) below.
|
||||
|
||||
Absence behaves
|
||||
as it does for the other readers: a store that says "not here" leaves the
|
||||
|
|
@ -384,7 +263,7 @@ internal CA deletes that file and names its own paths in
|
|||
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
|
||||
A hive that reads from a store on **another** machine names the reader's half —
|
||||
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
|
||||
plus one leaf per principal it runs (the [table in the reference](credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
|
||||
plus one leaf per principal it runs (the [table in the reference](../../4-implementation/swarm/credentials-reference.md#store-identities-one-per-principal) names the options). Those are the
|
||||
credentials that can't come out of the store, being what opens it; everything
|
||||
else a hive needs does.
|
||||
|
||||
|
|
@ -431,75 +310,6 @@ spell `hive-hives`: it would be a hive called `hives`, and
|
|||
can't read the accessor or write the policy it logs a warning and leaves the
|
||||
roles untouched, so a role that names a `hive-<hive>` policy keeps it.
|
||||
|
||||
### Per-principal identities
|
||||
|
||||
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
||||
identity and sharing one merges the identities. The leaves, their option names
|
||||
and what each reaches are in the [reference](credentials-reference.md#store-identities-one-per-principal).
|
||||
|
||||
Each reaches a path that names a swarm service rather than a hive, so one role
|
||||
each is enough and their subjects are the flat
|
||||
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
||||
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
|
||||
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
|
||||
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`, `deploy.bao.archiveCommonName`,
|
||||
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
|
||||
The last three are the writers, and the last is the one leaf two units share:
|
||||
the forge's host holds one store identity for the token it publishes and the
|
||||
OIDC client secret it reads.
|
||||
|
||||
On a host that mints its own PKI, `glue-bao-tls.nix` signs all ten and defaults
|
||||
all twenty options, and there is nothing to do. Elsewhere you copy each leaf from
|
||||
the store's host and name it here: a service principal's from the top of
|
||||
`/var/lib/swarm-bao-pki/`, another hive's leaves from
|
||||
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](../../4-implementation/swarm/bao.md#tls) lists that
|
||||
directory). That's one file per principal rather than one file
|
||||
shared by ten, which is the whole of what this buys. `swarm-ci-runner.nix`
|
||||
refuses the build wherever the runner runs without its pair, as
|
||||
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
|
||||
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without
|
||||
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
|
||||
naming both options, and the controller runs without forge access and the forge
|
||||
without a login source until an operator places the leaf.
|
||||
|
||||
On a host without the store, where the forwarder never runs, forgetting one of
|
||||
the others isn't a quiet degrade. A hand-configured remote-store hive that
|
||||
names the hive's own `clientCertFile` and misses a principal's pair has the
|
||||
build refused, by the module of the reader it missed, naming that pair — see
|
||||
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
|
||||
|
||||
The roles of `swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth` issue
|
||||
batch tokens with a five-minute TTL (`token_type=batch`, `token_ttl=5m`). The
|
||||
responders log in on every check, and the store keeps no record of a batch token,
|
||||
so unauthenticated requests don't grow its token storage. Nothing can renew
|
||||
or revoke a batch token.
|
||||
|
||||
The other roles issue service tokens that live an hour (`token_ttl=1h` and
|
||||
`token_max_ttl=1h`). That covers the roles `swarm-bao.nix` writes, and the
|
||||
agent and hive roles the controller writes through
|
||||
`SecretStore::write_cert_role`, which takes the same value from
|
||||
`CERT_ROLE_TOKEN_TTL`. The store keeps a lease for every token until it
|
||||
expires, and the mount default of 32 days leaves one lease per login for a
|
||||
month; at the 17 thousand logins a day this swarm made, that came to half a
|
||||
million leases. A reader logs in per operation and holds the token for seconds.
|
||||
The longest hold is the controller's forge work, which stops after two minutes.
|
||||
A client that keeps a `SecretStore` handle longer than an hour must log in
|
||||
again, because nothing renews the token. A role the controller writes takes the
|
||||
TTL at its next write: a hive's at the controller's start, an agent's at its
|
||||
next certificate mint or renewal. At its start the controller also reads every
|
||||
existing `hive-agent-*` role and, for one whose `token_ttl` or `token_max_ttl`
|
||||
differs from the TTL, writes those two fields and no others. The store keeps the
|
||||
fields a write leaves out, so the role's certificate, allowed names and
|
||||
policies stay. A role already at the TTL isn't written. The start logs one line
|
||||
with `rewritten`, `already_current` and `failed` counts, and a role that fails
|
||||
gets a warning that names it, then the pass moves on. Tokens issued before
|
||||
expire on their own; the controller revokes nothing. At that rate an hour keeps
|
||||
about 700 tokens live.
|
||||
|
||||
The granter's role keeps 15 minutes. Only the one-time bootstrap token writes
|
||||
it, and a deployed store never rewrites it, so the longer TTL would not reach
|
||||
it; the granter logs in and finishes within seconds.
|
||||
|
||||
### What a missing leaf costs
|
||||
|
||||
Each of the four modules asserts its own pair, on the same condition: **this host
|
||||
|
|
@ -526,35 +336,6 @@ store identity runs a collector with no OIDC client secret and still receives
|
|||
telemetry. The assertion doesn't touch that shape, because that host has no
|
||||
`clientCertFile` either.
|
||||
|
||||
## How a reader reaches the store
|
||||
|
||||
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
|
||||
the host that runs the store that name resolves twice. `/etc/hosts` answers a
|
||||
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
|
||||
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
|
||||
and the network namespace that asked decides which half of it answers.
|
||||
|
||||
The container's half goes through nginx and **still doesn't terminate TLS**. It
|
||||
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
|
||||
ClientHello and splices the rest of the connection through byte for byte, so
|
||||
openbao completes the handshake itself. A vhost would decrypt here instead, and
|
||||
the client certificate — the store's whole authentication — would stop at nginx,
|
||||
leaving openbao seeing one client for every hive in the swarm. That's why the
|
||||
store has no vhost and gets a passthrough: not an exception to the no-vhost
|
||||
rule, the one shape that keeps it.
|
||||
|
||||
The stream listener binds the bridge IP rather than every address, because
|
||||
openbao already holds loopback on that port in the same network namespace and a
|
||||
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
|
||||
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
|
||||
on the bridge firewall and nowhere else.
|
||||
|
||||
Reaching the port grants nothing by itself: openbao answers nothing without a
|
||||
client certificate signed by `deploy.bao.clientCaFile` or by the store's own
|
||||
agent CA. The listener reads both from `listener-client-ca.pem`, which no
|
||||
cert-auth role pins. The passthrough carries
|
||||
whichever certificate the reader presents, unchanged.
|
||||
|
||||
## The browser UI
|
||||
|
||||
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
|
||||
|
|
@ -576,64 +357,8 @@ hive a throwaway self-signed certificate, on the UI's name (a browser warning)
|
|||
until the unsealed store issues the services leaf, so the
|
||||
stream passthrough readers use on that host comes up with the store sealed.
|
||||
|
||||
### OIDC login
|
||||
|
||||
The OIDC tab logs in through authelia as the client `swarm-bao-ui`
|
||||
(`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under
|
||||
`swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`, and
|
||||
`create` and `update` on
|
||||
`secret/data/swarm/services/review-classifier/anthropic-api-key`,
|
||||
`secret/data/swarm/services/review-classifier/acp-provider` and the cache's five
|
||||
leaves under `secret/data/swarm/services/attic/` (`push-token`, `pull-token`,
|
||||
`url`, `cache` and `public-key`). That shows the key tree and each key's
|
||||
versions and timestamps, and no value: the store refuses every `secret/data/`
|
||||
read. The writes are the leaves the operator fills in by hand: the review
|
||||
classifier's keys, and the cache's tokens and facts, which configure an external
|
||||
cache or replace the in-swarm cache's own. The cache's host units create those
|
||||
leaves only while the store holds none, so a saved value survives their next run
|
||||
and every deploy. The cache's signing key, `attic/jwt-secret`, isn't writable here.
|
||||
Anything more needs a token.
|
||||
|
||||
To replace one of the cache's leaves in the UI, open `secret` > `swarm` >
|
||||
`services` > `attic`, pick the leaf and choose **Create new version**. The
|
||||
form starts empty because the login can't read the current value: type the
|
||||
key `value` and the new value, and save. A new version replaces the whole
|
||||
secret, so `value` is the only key to enter. `bao kv put` with the same login
|
||||
works too.
|
||||
|
||||
| piece | written by |
|
||||
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
||||
| authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs |
|
||||
| its secret at `swarm/services/swarm-bao-ui/oidc/client` | authelia's mint, like every other service's |
|
||||
| the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token |
|
||||
| `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config |
|
||||
|
||||
The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token
|
||||
step. A store set up before the mount existed needs the
|
||||
[one-time granter step](../../1-getting-started/setup.md) again: it re-writes
|
||||
`bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own
|
||||
policy, which covers `auth/oidc/`. Until then
|
||||
`swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and
|
||||
exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs
|
||||
it each time the step succeeds, so OIDC login needs no further restart.
|
||||
|
||||
## The constraint that decides where the root lives
|
||||
|
||||
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
|
||||
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
|
||||
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
|
||||
That role can't issue a certificate for a swarm service; the store refuses it.
|
||||
|
||||
The store's `swarm-services` role issues them instead, from the root inside its
|
||||
`pki` mount. That root's private key never leaves the mount, and its
|
||||
certificate is the only anchor the swarm has. Each hive fetches the certificate
|
||||
from the store at boot, so nothing build-time can name it — the system reads
|
||||
`security.pki.certificateFiles`' value at build time, which makes it the wrong
|
||||
tool here. Host containers get a bind-mounted bundle assembled at boot; an
|
||||
agent fetches the root itself with its own store certificate.
|
||||
|
||||
## Adding a secret
|
||||
|
||||
State three things, in the row you add to the [reference](credentials-reference.md#where-each-secret-lives-on-disk): **who mints it**, **which
|
||||
State three things, in the row you add to the [reference](../../4-implementation/swarm/credentials-reference.md#where-each-secret-lives-on-disk): **who mints it**, **which
|
||||
container reads it**, and **what happens when they differ**. If they differ, it
|
||||
needs a delivery unit, and the unit copies — it doesn't bind.
|
||||
|
|
|
|||
|
|
@ -968,7 +968,7 @@ GitHub units never delete a file, so a token already fetched into `<state>`
|
|||
stays there.
|
||||
|
||||
Where each credential lives and who reads it:
|
||||
[`credentials-reference.md`](credentials-reference.md).
|
||||
[`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md).
|
||||
|
||||
## Enabling
|
||||
|
||||
|
|
|
|||
|
|
@ -246,7 +246,7 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
|
|||
minute. Nothing rotates a credential: to replace one, delete its leaf in the
|
||||
store and the next pass mints a new one. HTTP Basic crosses the bridge in
|
||||
clear between an agent and the proxy. See
|
||||
[the credentials reference](../../2-handbook/swarm/credentials-reference.md).
|
||||
[the credentials reference](../../4-implementation/swarm/credentials-reference.md).
|
||||
A request without a valid credential gets a 407, once the destination
|
||||
checks below have passed it.
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
The swarm runs one OpenBao store, `swarm-bao`. Every swarm-level
|
||||
credential an agent or service needs comes from it under that
|
||||
principal's own certificate identity. What lives in it and who reads what:
|
||||
[`credentials-reference.md`](../../2-handbook/swarm/credentials-reference.md). This page covers the store itself — how it
|
||||
[`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md). This page covers the store itself — how it
|
||||
comes up, who may write its grants, and how it's sealed and reached.
|
||||
|
||||
The operator steps (init, the one-time granter bootstrap) are in
|
||||
|
|
@ -119,7 +119,7 @@ Those are `mkDefault`s, so naming your own paths wins. Do that when your
|
|||
certificates come from a real internal CA; the store has no opinion about
|
||||
which. A host that does **not** deploy the store names the reader's three
|
||||
itself, plus a pair for every principal it runs — see
|
||||
[per-principal identities](../../2-handbook/swarm/credentials-reference.md#store-identities-one-per-principal) for the list.
|
||||
[per-principal identities](../../4-implementation/swarm/credentials-reference.md#store-identities-one-per-principal) for the list.
|
||||
They're the credentials the store can't hand you, being what opens it, so the
|
||||
store's host signs them and the operator copies them to the host that needs
|
||||
them and names them in config. The service principals' leaves are at the top of
|
||||
|
|
|
|||
|
|
@ -668,7 +668,7 @@ status. The same `git` fetches a config's own inputs, so the builder registers
|
|||
the helper for the forge's URL only, and the helper answers only an `https`
|
||||
request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the
|
||||
status with its own forge token. Details per path:
|
||||
[`credentials-reference.md`](../../2-handbook/swarm/credentials-reference.md).
|
||||
[`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md).
|
||||
|
||||
The builder exports build counts, durations and queue wait; see
|
||||
[Swarm builder metrics](../scheduler/observability.md#swarm-builder-metrics).
|
||||
|
|
|
|||
|
|
@ -24,7 +24,7 @@ IP SANs and no `localhost`. A request for any other name is a refusal from
|
|||
the store naming that name. The hive's policy grants `update` on that one
|
||||
issue path and `read` on its own `attic/public-key`, `attic/pull-token` and
|
||||
`squid-selfcheck`
|
||||
(see [the cache facts for hive-priv](../../2-handbook/swarm/credentials-reference.md) and the
|
||||
(see [the cache facts for hive-priv](../../4-implementation/swarm/credentials-reference.md) and the
|
||||
squid self-check password), and nothing
|
||||
else. `swarm-bao-hive-leaf-policy.service`, on the
|
||||
store's host, writes the role, the policy and the cert-auth role for every
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
|
||||
Every credential in the swarm's secret store, in one table: who mints it, who
|
||||
reads it, and whether it renews on its own. The overview and the rules a
|
||||
credential follows are in [`credentials.md`](credentials.md); this page is
|
||||
credential follows are in [`credentials.md`](../../2-handbook/swarm/credentials.md); this page is
|
||||
the detail, so it changes whenever someone adds a credential or moves its minting.
|
||||
The second half, [where each secret lives on disk](#where-each-secret-lives-on-disk), maps the
|
||||
files that exist today.
|
||||
|
|
@ -112,20 +112,20 @@ the next timer tick.
|
|||
|
||||
One row per secret, by the place that owns it: who generates it, where the file
|
||||
or store path is, and what an operator of a hive elsewhere places. How the
|
||||
pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.md).
|
||||
pieces fit, and the reasoning behind them, is in [`credentials.md`](../../2-handbook/swarm/credentials.md).
|
||||
|
||||
### Swarm-level — one of each per swarm
|
||||
|
||||
| secret | generated by | lives at | hive elsewhere |
|
||||
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
|
||||
| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/<name>` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store is down, refuses the login, or fails the read of a leaf (a refusal, a timeout, an error), the unit runs on the files, logs the store's error, and writes nothing to the store; the unit seeds or mints only a leaf the store reports as absent. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting |
|
||||
| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
||||
| swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice render` in `hive-matrix`. The store's pair wins; a store holding none takes the tokens an earlier version kept in the container, so the value doesn't change by moving. Only a store that answers, holds none, and a container with no such files gets a new pair | `/run/swarm-matrix-appservice/swarm.yaml` in `hive-matrix` (tmpfs), which tuwunel loads through `LoadCredential`. The earlier `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` stay, unused, and bao wins if they differ; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down keeps the homeserver down and mints nothing, and `render` retries every 30 seconds for 24 hours |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives/<hive>/oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services/<client id>/oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/<id>.secret`; a file an earlier release left seeds the store once when it holds no value, and stays unused | nothing to place: readers fetch it from the store ([how](credentials.md#hive-level-secrets)) |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives/<hive>/oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services/<client id>/oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/<id>.secret`; a file an earlier release left seeds the store once when it holds no value, and stays unused | nothing to place: readers fetch it from the store ([how](../../2-handbook/swarm/credentials.md#hive-level-secrets)) |
|
||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](credentials.md#hive-level-secrets) |
|
||||
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](../../2-handbook/swarm/credentials.md#swarm-level-secrets) |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](../../2-handbook/swarm/credentials.md#hive-level-secrets) |
|
||||
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
|
||||
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
|
||||
| Grafana's datasource encryption key | `swarm-bao-grafana-secret-key.service` stores it in the swarm secret store at `swarm/services/<grafana client id>/secret-key` when the store holds none, then copies it out, **on every host that runs Grafana**. It stores the key an earlier release generated at `/var/lib/grafana-secret/secret_key` in the container, so the key doesn't change, and mints a new one only on a host with no such file. Same grant as the admin password, create without update. The earlier file stays on disk, unused once the store holds the key. A store that's down or doesn't grant the leaf fails the unit, and Grafana doesn't start | `/run/swarm-grafana-secrets/secret-key` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container: `security.secret_key` | same unit, same path. Grafana doesn't start until the file exists. Changing the key leaves Grafana unable to read the datasource secrets it stored, so never delete the store path to rotate it |
|
||||
|
|
@ -139,7 +139,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
|
|||
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. It waits up to 60 seconds for Grafana's socket to answer before its first call. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
|
||||
| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents/<agent>/backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file |
|
||||
| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents/<agent>/proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it |
|
||||
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
|
||||
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](../../2-handbook/swarm/claude-login.md) |
|
||||
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the fetch fails fast while the store is down and retries with a delay that grows to 5 minutes; the attempt that succeeds starts the collector, which isn't part of the container's boot, and it resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) |
|
||||
|
|
@ -160,7 +160,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
|
|||
| secret | generated by | lives at |
|
||||
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](../../4-implementation/swarm/ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the cache's public key and pull token, for the host's nix daemon and hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` (tmpfs, root, `0600`, directory `0700`); it also writes `/run/hyperhive-nix/cache.conf` (tmpfs, root, `0644`, no secret in it): the cache's address as `extra-substituters`, the key as `extra-trusted-public-keys`, `netrc-file` and `ssl-cert-file`, which the host's nix.conf includes; and `/run/hyperhive-nix/netrc` (`root:root 0600`), the token's entry followed by the entries of the host's own netrc (`nix.settings.netrc-file`, else `/etc/nix/netrc`), merged again on every run. `ssl-cert-file` names `/run/hyperhive-nix/ca-bundle.crt` (`0644`, public), the host's own CA bundle followed by the swarm's root from `<deploy.hive-controller.tls.stateDir>/swarm-services-root.pem`, so that only nix trusts the root. The host's nix daemon reads it, so every client of the daemon can have a swarm closure substituted, and nothing unprivileged reads the token for the pull. The cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand; a hive that never got them has `hive-c0re` ask for a refresh and then fail the pull with `this hive has no cache facts yet`. |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
278
docs/4-implementation/swarm/credentials.md
Normal file
278
docs/4-implementation/swarm/credentials.md
Normal file
|
|
@ -0,0 +1,278 @@
|
|||
# Credentials: mechanics
|
||||
|
||||
The store's identities, the queue responder's keys, how a reader reaches the
|
||||
store, the store UI's login and where the root lives. The overview and the rules
|
||||
a credential follows are in [credentials](../../2-handbook/swarm/credentials.md);
|
||||
each credential's minter, reader and renewal are in
|
||||
[the reference](credentials-reference.md).
|
||||
|
||||
## The queue's callout nkeys
|
||||
|
||||
The responder holds two nkeys besides the callout xkey below. It signs every
|
||||
user JWT it issues with the **issuer** key, an account key, so possession of
|
||||
the issuer seed is the authority to admit anyone to the queue. It connects to
|
||||
the queue as the **user** key, the one identity the server exempts from
|
||||
callout. You configure nothing about either: both live in the swarm secret
|
||||
store beside the xkey, and come into being on the queue's first boot.
|
||||
|
||||
- **Issuer seed** at `secret/swarm/services/swarm-nats/callout/issuer-seed`,
|
||||
**public half** at `secret/swarm/public/swarm-nats/callout-issuer`.
|
||||
- **User seed** at `secret/swarm/services/swarm-nats/callout/user-seed`,
|
||||
**public half** at `secret/swarm/public/swarm-nats/callout-user`.
|
||||
|
||||
The unit that creates the xkey creates these too, under the same rules: only
|
||||
the `swarm-nats-auth` role reads or creates them, nothing replaces a stored
|
||||
seed, and the policy of `swarm-secret-publisher` denies it both seed paths.
|
||||
The server renders both public halves into its `auth_callout` block as it
|
||||
starts, and doesn't start without them.
|
||||
|
||||
A seed file at `/var/lib/swarm-nats-callout/issuer.seed` or
|
||||
`/var/lib/swarm-nats-callout/callout-user.seed` on the queue's host is what
|
||||
`swarm-nats-auth ensure-callout-keys` stores for that key when the store holds
|
||||
none, so the key keeps its public half. Once a read of the store returns a
|
||||
seed, the verb removes the file. When the store already holds a different
|
||||
seed, the verb keeps the store's seed, logs a warning naming the file, then
|
||||
removes it.
|
||||
|
||||
A hive that sets `autoGenerateCallout`, `calloutUserPublicKey`,
|
||||
`calloutIssuerPublicKey`, `calloutUserSeedFile` or `calloutIssuerSeedFile`,
|
||||
under `deploy.nats` or `swarm.nats`, fails at eval with a message naming the
|
||||
option and where the keys live now.
|
||||
|
||||
To replace either key, delete both of its paths the way the xkey's section
|
||||
below shows for the xkey, then restart `swarm-bao-nats-callout-xkey`.
|
||||
|
||||
One consequence worth knowing before you debug it: the queue assembles its
|
||||
config each time the server starts rather than at build time, in every mode,
|
||||
because the callout keys exist only at runtime. A malformed config
|
||||
surfaces when the server starts instead of when the system builds. The server
|
||||
names the offending file and refuses to run.
|
||||
|
||||
## The queue's callout xkey
|
||||
|
||||
The server seals every auth-callout request to a curve key, the callout
|
||||
xkey, and the responder seals its reply back, so nothing else in the callout
|
||||
account or on the wire can read the request or the user JWT minted for it.
|
||||
You configure nothing about the key: it lives in the swarm secret store and
|
||||
comes into being on the queue's first boot.
|
||||
|
||||
- **Seed** at `secret/swarm/services/swarm-nats/callout/xkey-seed`.
|
||||
- **Public half** at `secret/swarm/public/swarm-nats/callout-xkey`.
|
||||
|
||||
Only the `swarm-nats-auth` role reads either path, and it may create each but
|
||||
never replace it. The policy of `swarm-secret-publisher`, which writes the rest
|
||||
of `swarm/services/`, denies it the seed's path.
|
||||
|
||||
On the queue's host, `swarm-bao-nats-callout-xkey` runs before the
|
||||
`swarm-nats` container, under the responder's store leaf:
|
||||
|
||||
1. `swarm-nats-auth ensure-callout-keys` creates the pair when the store holds no
|
||||
seed, and otherwise only reads it. It needs no queue, so on a first boot
|
||||
the key exists before the server that needs it.
|
||||
2. It reads the public half back from the store into
|
||||
`/var/lib/nats-callout/callout-xkey.pub` inside the container.
|
||||
3. Each time `nats-server` starts, it renders that key into
|
||||
`auth_callout.xkey`. With no file, the server doesn't start.
|
||||
4. The responder starts after the server and reads the seed from the store.
|
||||
A responder that can't read it doesn't start, rather than answering
|
||||
requests it can't open.
|
||||
|
||||
The store has to be reachable each time the responder starts. When it's down,
|
||||
the fetch unit fails and retries for a day. The server starts with the key
|
||||
fetched last time, or not at all on a first boot. A responder that starts
|
||||
while the store is down exits and retries every 5 seconds, and until one
|
||||
start reads the seed the queue refuses every client, OIDC ones included. The
|
||||
retry that lands the file starts or restarts the server, and the responder
|
||||
comes up with it.
|
||||
|
||||
The responder's store leaf is `deploy.nats.authBaoClientCertFile` /
|
||||
`authBaoClientKeyFile`. On the store's host both default to the leaf
|
||||
`glue-bao-tls.nix` mints there. A queue host without the store gets no
|
||||
default, even when it holds the hive's own reader `deploy.bao.clientCertFile`:
|
||||
issue a leaf whose subject is `deploy.bao.natsAuthCommonName`, place it, and
|
||||
name it in both options. Until you do, evaluation fails naming both.
|
||||
|
||||
Where the queue and the store run on different hosts, deploy the store's host
|
||||
first. `swarm-bao-nats-auth-policy` there writes the role's grants on the
|
||||
keys' paths, and `swarm-bao-secret-publisher-policy` the publisher's denial of
|
||||
the seeds' paths; where the queue runs on that host too, the fetch waits for
|
||||
both. A queue host deployed before it can't read or create the keys, so its
|
||||
fetch fails, and the queue refuses every client until you deploy the store's
|
||||
host: the server starts only on public keys fetched before, and the responder
|
||||
can't load its seeds. Seed files the import is waiting for stay where they
|
||||
are. If that takes longer than the fetch's day of retries, run
|
||||
`systemctl restart swarm-bao-nats-callout-xkey` on the queue's host.
|
||||
|
||||
Two starts that create the key at once can't both win: each write is a
|
||||
create that fails once the path holds anything, and the start that lost reads
|
||||
the winner's key back.
|
||||
|
||||
To replace the key, remove both paths with their metadata and every version,
|
||||
then restart the fetch unit on the queue's host. Its restart of `nats.service`
|
||||
restarts the responder too, which then loads the new seed:
|
||||
|
||||
```sh
|
||||
bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
|
||||
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
|
||||
systemctl restart swarm-bao-nats-callout-xkey
|
||||
```
|
||||
|
||||
`bao kv delete` isn't enough. It deletes the latest version but keeps the
|
||||
path's metadata, so the store still counts the path as existing and refuses
|
||||
the create that makes a new key. No principal hyperhive configures may
|
||||
delete either path's metadata, so run the two deletes with a token that may,
|
||||
such as the root token.
|
||||
|
||||
## Per-principal identities
|
||||
|
||||
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
||||
identity and sharing one merges the identities. The leaves, their option names
|
||||
and what each reaches are in the [reference](credentials-reference.md#store-identities-one-per-principal).
|
||||
|
||||
Each reaches a path that names a swarm service rather than a hive, so one role
|
||||
each is enough and their subjects are the flat
|
||||
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
||||
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
|
||||
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
|
||||
`deploy.bao.ciMetricsCommonName`, `deploy.bao.atticCommonName`, `deploy.bao.archiveCommonName`,
|
||||
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
|
||||
The last three are the writers, and the last is the one leaf two units share:
|
||||
the forge's host holds one store identity for the token it publishes and the
|
||||
OIDC client secret it reads.
|
||||
|
||||
On a host that mints its own PKI, `glue-bao-tls.nix` signs all ten and defaults
|
||||
all twenty options, and there is nothing to do. Elsewhere you copy each leaf from
|
||||
the store's host and name it here: a service principal's from the top of
|
||||
`/var/lib/swarm-bao-pki/`, another hive's leaves from
|
||||
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](../../4-implementation/swarm/bao.md#tls) lists that
|
||||
directory). That's one file per principal rather than one file
|
||||
shared by ten, which is the whole of what this buys. `swarm-ci-runner.nix`
|
||||
refuses the build wherever the runner runs without its pair, as
|
||||
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
|
||||
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without
|
||||
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
|
||||
naming both options, and the controller runs without forge access and the forge
|
||||
without a login source until an operator places the leaf.
|
||||
|
||||
On a host without the store, where the forwarder never runs, forgetting one of
|
||||
the others isn't a quiet degrade. A hand-configured remote-store hive that
|
||||
names the hive's own `clientCertFile` and misses a principal's pair has the
|
||||
build refused, by the module of the reader it missed, naming that pair — see
|
||||
[what a missing leaf costs](../../2-handbook/swarm/credentials.md#what-a-missing-leaf-costs) below.
|
||||
|
||||
The roles of `swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth` issue
|
||||
batch tokens with a five-minute TTL (`token_type=batch`, `token_ttl=5m`). The
|
||||
responders log in on every check, and the store keeps no record of a batch token,
|
||||
so unauthenticated requests don't grow its token storage. Nothing can renew
|
||||
or revoke a batch token.
|
||||
|
||||
The other roles issue service tokens that live an hour (`token_ttl=1h` and
|
||||
`token_max_ttl=1h`). That covers the roles `swarm-bao.nix` writes, and the
|
||||
agent and hive roles the controller writes through
|
||||
`SecretStore::write_cert_role`, which takes the same value from
|
||||
`CERT_ROLE_TOKEN_TTL`. The store keeps a lease for every token until it
|
||||
expires, and the mount default of 32 days leaves one lease per login for a
|
||||
month; at the 17 thousand logins a day this swarm made, that came to half a
|
||||
million leases. A reader logs in per operation and holds the token for seconds.
|
||||
The longest hold is the controller's forge work, which stops after two minutes.
|
||||
A client that keeps a `SecretStore` handle longer than an hour must log in
|
||||
again, because nothing renews the token. A role the controller writes takes the
|
||||
TTL at its next write: a hive's at the controller's start, an agent's at its
|
||||
next certificate mint or renewal. At its start the controller also reads every
|
||||
existing `hive-agent-*` role and, for one whose `token_ttl` or `token_max_ttl`
|
||||
differs from the TTL, writes those two fields and no others. The store keeps the
|
||||
fields a write leaves out, so the role's certificate, allowed names and
|
||||
policies stay. A role already at the TTL isn't written. The start logs one line
|
||||
with `rewritten`, `already_current` and `failed` counts, and a role that fails
|
||||
gets a warning that names it, then the pass moves on. Tokens issued before
|
||||
expire on their own; the controller revokes nothing. At that rate an hour keeps
|
||||
about 700 tokens live.
|
||||
|
||||
The granter's role keeps 15 minutes. Only the one-time bootstrap token writes
|
||||
it, and a deployed store never rewrites it, so the longer TTL would not reach
|
||||
it; the granter logs in and finishes within seconds.
|
||||
|
||||
## How a reader reaches the store
|
||||
|
||||
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
|
||||
the host that runs the store that name resolves twice. `/etc/hosts` answers a
|
||||
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
|
||||
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
|
||||
and the network namespace that asked decides which half of it answers.
|
||||
|
||||
The container's half goes through nginx and **still doesn't terminate TLS**. It
|
||||
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
|
||||
ClientHello and splices the rest of the connection through byte for byte, so
|
||||
openbao completes the handshake itself. A vhost would decrypt here instead, and
|
||||
the client certificate — the store's whole authentication — would stop at nginx,
|
||||
leaving openbao seeing one client for every hive in the swarm. That's why the
|
||||
store has no vhost and gets a passthrough: not an exception to the no-vhost
|
||||
rule, the one shape that keeps it.
|
||||
|
||||
The stream listener binds the bridge IP rather than every address, because
|
||||
openbao already holds loopback on that port in the same network namespace and a
|
||||
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
|
||||
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
|
||||
on the bridge firewall and nowhere else.
|
||||
|
||||
Reaching the port grants nothing by itself: openbao answers nothing without a
|
||||
client certificate signed by `deploy.bao.clientCaFile` or by the store's own
|
||||
agent CA. The listener reads both from `listener-client-ca.pem`, which no
|
||||
cert-auth role pins. The passthrough carries
|
||||
whichever certificate the reader presents, unchanged.
|
||||
|
||||
## OIDC login to the store UI
|
||||
|
||||
The OIDC tab logs in through authelia as the client `swarm-bao-ui`
|
||||
(`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under
|
||||
`swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`, and
|
||||
`create` and `update` on
|
||||
`secret/data/swarm/services/review-classifier/anthropic-api-key`,
|
||||
`secret/data/swarm/services/review-classifier/acp-provider` and the cache's five
|
||||
leaves under `secret/data/swarm/services/attic/` (`push-token`, `pull-token`,
|
||||
`url`, `cache` and `public-key`). That shows the key tree and each key's
|
||||
versions and timestamps, and no value: the store refuses every `secret/data/`
|
||||
read. The writes are the leaves the operator fills in by hand: the review
|
||||
classifier's keys, and the cache's tokens and facts, which configure an external
|
||||
cache or replace the in-swarm cache's own. The cache's host units create those
|
||||
leaves only while the store holds none, so a saved value survives their next run
|
||||
and every deploy. The cache's signing key, `attic/jwt-secret`, isn't writable here.
|
||||
Anything more needs a token.
|
||||
|
||||
To replace one of the cache's leaves in the UI, open `secret` > `swarm` >
|
||||
`services` > `attic`, pick the leaf and choose **Create new version**. The
|
||||
form starts empty because the login can't read the current value: type the
|
||||
key `value` and the new value, and save. A new version replaces the whole
|
||||
secret, so `value` is the only key to enter. `bao kv put` with the same login
|
||||
works too.
|
||||
|
||||
| piece | written by |
|
||||
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
||||
| authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs |
|
||||
| its secret at `swarm/services/swarm-bao-ui/oidc/client` | authelia's mint, like every other service's |
|
||||
| the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token |
|
||||
| `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config |
|
||||
|
||||
The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token
|
||||
step. A store set up before the mount existed needs the
|
||||
[one-time granter step](../../1-getting-started/setup.md) again: it re-writes
|
||||
`bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own
|
||||
policy, which covers `auth/oidc/`. Until then
|
||||
`swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and
|
||||
exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs
|
||||
it each time the step succeeds, so OIDC login needs no further restart.
|
||||
|
||||
## The constraint that decides where the root lives
|
||||
|
||||
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
|
||||
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
|
||||
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
|
||||
That role can't issue a certificate for a swarm service; the store refuses it.
|
||||
|
||||
The store's `swarm-services` role issues them instead, from the root inside its
|
||||
`pki` mount. That root's private key never leaves the mount, and its
|
||||
certificate is the only anchor the swarm has. Each hive fetches the certificate
|
||||
from the store at boot, so nothing build-time can name it — the system reads
|
||||
`security.pki.certificateFiles`' value at build time, which makes it the wrong
|
||||
tool here. Host containers get a bind-mounted bundle assembled at boot; an
|
||||
agent fetches the root itself with its own store certificate.
|
||||
|
|
@ -6,7 +6,7 @@ differ, and why machine callers must fail closed. The operator's view is
|
|||
|
||||
## What secrets exist, and where each one lives
|
||||
|
||||
[`credentials-reference.md`](../../2-handbook/swarm/credentials-reference.md) tabulates every secret in the swarm, with its
|
||||
[`credentials-reference.md`](../../4-implementation/swarm/credentials-reference.md) tabulates every secret in the swarm, with its
|
||||
generator and its path, in one place — including authelia's own keys
|
||||
(session, JWT, storage-encryption, OIDC HMAC, OIDC issuer) and the two
|
||||
halves of each client secret. That page's two rules — a secret is always
|
||||
|
|
|
|||
|
|
@ -49,12 +49,13 @@ The services that exist once per swarm. Start at
|
|||
`services.hyperhive.agent.matrixAccounts`.
|
||||
- **What's `/knowledge`? How does the swarm-wide knowledge repo sync,
|
||||
and how to contribute a document?** → [`2-handbook/swarm/knowledge.md`](2-handbook/swarm/knowledge.md).
|
||||
- **Who mints each credential, who reads it, and how does it rotate — and
|
||||
where's that shape headed?** → [`2-handbook/swarm/credentials.md`](2-handbook/swarm/credentials.md)
|
||||
(the rules, where each secret goes and why) and
|
||||
[`2-handbook/swarm/credentials-reference.md`](2-handbook/swarm/credentials-reference.md) (the
|
||||
tables: each credential, current state and target state, and where each
|
||||
file lives today).
|
||||
- **Who mints each credential, who reads it, and how does it rotate?** →
|
||||
[`2-handbook/swarm/credentials.md`](2-handbook/swarm/credentials.md) (the
|
||||
rules, where each secret goes and why, adding a secret) and
|
||||
[`4-implementation/swarm/credentials-reference.md`](4-implementation/swarm/credentials-reference.md) (the
|
||||
tables: each credential and where each file lives). The store's identities,
|
||||
keys and login:
|
||||
[`4-implementation/swarm/credentials.md`](4-implementation/swarm/credentials.md).
|
||||
- **How does an agent get a prompt at set times, and why doesn't it fire
|
||||
twice after downtime?** → [`2-handbook/swarm/schedules.md`](2-handbook/swarm/schedules.md).
|
||||
- **How to create an agent from a ready-made set of permissions and
|
||||
|
|
|
|||
|
|
@ -123,7 +123,7 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients.
|
|||
icons, the cross-repo issue report, and the UI's quick links.
|
||||
- **Alerts** — the swarm Grafana's current alerts (`GET /api/alerts`), read
|
||||
at request time with a Viewer service-account token from the secret store
|
||||
→ [`docs/2-handbook/swarm/credentials-reference.md`](../docs/2-handbook/swarm/credentials-reference.md).
|
||||
→ [`docs/4-implementation/swarm/credentials-reference.md`](../docs/4-implementation/swarm/credentials-reference.md).
|
||||
|
||||
It reads its configuration once at startup, from the environment the nix module
|
||||
sets. It keeps the webhook HMAC secret in the secret store and
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
//!
|
||||
//! **The internal one** — [`agent_token`] — is the agent's own `main` account on
|
||||
//! the swarm's homeserver, minted with the swarm's appservice token and stored
|
||||
//! where the agent's daemon reads it. See docs/2-handbook/swarm/credentials-reference.md for why
|
||||
//! where the agent's daemon reads it. See docs/4-implementation/swarm/credentials-reference.md for why
|
||||
//! `main` is the one name [`put_matrix_account`] refuses to write.
|
||||
//! [`hive_sender`] mints each **hive's** sender account the same way, sharing this module's helpers with [`agent_token`].
|
||||
//! [`operator_admin`] makes authelia's operators homeserver admins with the same
|
||||
|
|
|
|||
Loading…
Reference in a new issue