docs: the forge, knowledge, archetypes, matrix and SSO pages move their implementation detail out
Account minting, knowledge sync, the archetype refresh, the matrix appservice flow and the SSO secret delivery move to pages under 4-implementation/swarm. Each handbook page keeps what an operator does or sees, with a pointer.
This commit is contained in:
iris
mara
parent
c6b24359cd
commit
0f39bdaaa4
5 changed files with 708 additions and 563 deletions
|
|
@ -54,48 +54,14 @@ an archetype from any other source name.
|
|||
|
||||
## How the list stays current
|
||||
|
||||
1. swarm-controller asks the builder for the list on the queue subject
|
||||
`$SWARM.builder.archetypes` once the queue connects, and every 10 minutes
|
||||
after that. The controller asks again after a pass that gets no answer or
|
||||
a failed evaluation: after 10 seconds, then 20, doubling up to 10 minutes,
|
||||
until one succeeds.
|
||||
2. The builder evaluates the `meta` of each `hyperhiveArchetypes` entry on
|
||||
its own hyperhive and nixpkgs sources, and answers with the JSON or the
|
||||
last lines `nix` printed:
|
||||
The swarm controller asks the builder for the list when the queue connects and
|
||||
every 10 minutes after. A builder host redeploy can change it, and the next
|
||||
pass picks the change up. The controller leaves an archetype that fails its check
|
||||
off the list and logs it. When no builder answers, the controller keeps the
|
||||
stored list, and a swarm with no builder shows no archetype select.
|
||||
|
||||
```sh
|
||||
nix eval --json <hyperhive>#hyperhiveArchetypes \
|
||||
--apply 'builtins.mapAttrs (_: a: a.meta)'
|
||||
```
|
||||
|
||||
3. The controller checks each archetype the way `POST /api/agents` checks
|
||||
its settings and stores the result in the `agent-archetypes` KV bucket,
|
||||
under the key `hyperhive`, when it differs from the stored one. The list
|
||||
leaves out an archetype that fails the check, and its name and reason go
|
||||
into the record's `refused` map and the controller's journal.
|
||||
4. `GET /api/archetypes` answers from that record, and the create form lists
|
||||
from it.
|
||||
|
||||
A builder host redeploy can change the list. The next pass, within 10
|
||||
minutes, picks it up. When no builder answers, the controller keeps the
|
||||
stored list and logs a warning on each pass. A swarm whose
|
||||
builder connects after the controller has no list until a retry succeeds. A swarm with no builder enabled shows no
|
||||
archetype select.
|
||||
|
||||
The stored value:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "/nix/store/…-hyperhive-flake-source",
|
||||
"archetypes": {
|
||||
"reviewer": {
|
||||
"description": "…",
|
||||
"permissions": ["messaging", "inbox", "execution"]
|
||||
}
|
||||
},
|
||||
"refused": { "broken": "limits: …" }
|
||||
}
|
||||
```
|
||||
The query, the stored record and the retry timing:
|
||||
[`archetypes.md`](../../4-implementation/swarm/archetypes.md).
|
||||
|
||||
## Creating an agent from an archetype
|
||||
|
||||
|
|
|
|||
|
|
@ -33,95 +33,34 @@ by a test). swarm-controller mints every agent token with it:
|
|||
| `write:notification` | Mark notifications read via `PATCH /notifications/threads/{id}`. |
|
||||
|
||||
**`HIVE_TOKEN_SCOPES`** (swarm-controller's `forge::hive_token`, pinned by a
|
||||
test): `read:repository` alone, for each hive's read-only account (next
|
||||
section).
|
||||
test): `read:repository` alone, for each hive's read-only account (see
|
||||
[Hive and agent accounts](#hive-and-agent-accounts)).
|
||||
|
||||
---
|
||||
|
||||
## How a hive reads the forge
|
||||
## Hive and agent accounts
|
||||
|
||||
Each hive in swarm-controller's hive directory has its own forge account,
|
||||
`hive-<hive>`. swarm-controller creates it, makes it a member of the `hives`
|
||||
team in `agent-configs`, and mints one token named `swarm-hive` into the
|
||||
swarm secret store at `swarm/hives/<hive>/forge/token`. The team has read
|
||||
permission on `repo.code` and `repo.pulls` of every repo in the org,
|
||||
including repos created later. A pass at start and every five minutes
|
||||
creates what's missing and re-mints a token that's missing or no longer
|
||||
matches the forge. A pass whose store or forge reads fail queues nothing, so
|
||||
an outage never rotates every hive's token.
|
||||
|
||||
swarm-controller refuses to create an agent whose name starts with `hive-`:
|
||||
these accounts own that prefix. An agent that already had such a
|
||||
name keeps it, so the accounts are also told apart by email. swarm-controller
|
||||
creates the hive's account with the email `hive-<hive>@hives.hyperhive.local`
|
||||
and refuses an account by that name with any other email: it never adds an
|
||||
agent's account to the team or mints a token for it. An agent's own forge
|
||||
account setup refuses the hive's account the same way.
|
||||
|
||||
hive-c0re reads the token with `forge::reader_token()`, under the hive's own
|
||||
store certificate, every time it reads the forge. A hive whose store holds
|
||||
none reads nothing. These reads go through it, and none of them needs a local
|
||||
forge container:
|
||||
|
||||
- cloning and pulling `/knowledge`
|
||||
|
||||
The hive builds nothing, so no nix process reads the forge as the hive's
|
||||
account. hive-c0re hands the token to each `git` command it runs
|
||||
(`http.extraHeader`) and keeps no copy on disk. A hive whose store holds no token
|
||||
reads nothing private from the forge. This also holds on a hive with no forge
|
||||
container.
|
||||
|
||||
Roll a change to any of this out store host first (its granter writes the
|
||||
controller's grant on `swarm/hives/+/forge/token`), then the controller host,
|
||||
then the hives. A hive that starts before its token exists reads nothing, and
|
||||
builds nothing, until the controller's next pass.
|
||||
|
||||
---
|
||||
|
||||
## Per-agent forge accounts
|
||||
|
||||
Each agent gets its own Forgejo user and access token. swarm-controller
|
||||
creates the user when it creates the agent, and mints one token named
|
||||
`swarm-agent` with the admin API into the swarm secret store at
|
||||
`swarm/agents/<agent>/forge-token`. A pass at start and every five minutes
|
||||
covers every agent with a store identity: it creates a missing forge user and
|
||||
re-mints any token that's missing or no longer matches the forge;
|
||||
a rotation deletes the old `swarm-agent` token first, so each agent holds at
|
||||
most one. The agent fetches the token under its own store certificate into
|
||||
`/run/hive-agent-forge-token/token` (`nix/agent-modules/forge-token.nix`),
|
||||
and `hive-forge`, the git credential helper, the `forge_notify` poller and
|
||||
the avatar sync read it from there, falling back to `<state>/forge-token`,
|
||||
the file hive-c0re wrote before. hive-c0re no longer creates agent users or
|
||||
mints agent tokens. swarm-controller deletes the `hyperhive-<unix-seconds>`
|
||||
tokens it left behind, for each agent whose current `swarm-agent` token is
|
||||
live; tokens of other names are never touched.
|
||||
Each hive has a read-only forge account, `hive-<hive>`, which it uses to clone
|
||||
and pull `/knowledge`. Each agent has its own Forgejo user and access token. The
|
||||
swarm controller creates both and keeps the tokens in the swarm secret store,
|
||||
so you set up nothing by hand. A token that goes missing or stops matching the
|
||||
forge is re-minted within five minutes.
|
||||
|
||||
Two things live in the `agent-configs` Forgejo organization:
|
||||
|
||||
- A config repo per agent (`agent-configs/<name>`). The
|
||||
agent is a **write collaborator on its own** repo — it can push
|
||||
config-change branches and open config PRs — but `main` is
|
||||
branch-protected by swarm-controller: the merge and approval
|
||||
allowlists are the `operators` team, and the agent can neither push
|
||||
`main` directly nor self-merge. An operator's merge in the Forgejo UI
|
||||
- A config repo per agent, `agent-configs/<name>`. The agent can push
|
||||
config-change branches and open config PRs there, but `main` is
|
||||
branch-protected: only the `operators` team can merge, and the agent can
|
||||
neither push `main` nor self-merge. An operator's merge in the Forgejo UI
|
||||
deploys the merged commit (see
|
||||
[config-changes.md § Config changes](../agents/config-changes.md#config-changes)).
|
||||
hive-c0re pushes nothing to this org and reads nothing from it.
|
||||
Repos stay private, so an agent can't read another
|
||||
agent's config. The agent's **persistent meta flake input** is this repo's `main` head,
|
||||
pinned by sha in `internal/meta` (`agent-<n>.url =
|
||||
git+<forge>/agent-configs/<n>.git?rev=<sha>`; see
|
||||
[config-changes.md § Meta flake](../agents/config-changes.md#meta-flake)).
|
||||
Repos stay private, so an agent can't read another agent's config.
|
||||
|
||||
The `hive-forge` CLI (separate workspace crate, see
|
||||
[`README.md`](../../README.md) file map) wraps the Forgejo REST API
|
||||
with the per-agent token; agents call it for issue / PR / comment
|
||||
ops as if it were a peer. All REST calls across the workspace
|
||||
(`hive-forge` verbs, hive-c0re provisioning, this poller) go through
|
||||
the typed `forgejo-api` crate; only non-`/api/v1` web-router routes
|
||||
(attachment / artifact downloads, log streaming) and the poller's
|
||||
enrichment fetches of server-provided subject URLs stay on raw
|
||||
reqwest.
|
||||
The `hive-forge` CLI wraps the Forgejo REST API with the agent's own token;
|
||||
agents call it for issue, PR and comment operations.
|
||||
|
||||
How the controller mints, rotates and reads each token, and the team that gives a hive read
|
||||
access: [`forge-accounts.md`](../../4-implementation/swarm/forge-accounts.md).
|
||||
|
||||
## Notifications
|
||||
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ read-only — agents never write through it. To contribute, use the
|
|||
`hive-forge` AGit flow (no fork needed — see
|
||||
[Contributing](#contributing)); the operator reviews and merges, and
|
||||
the local clone updates automatically (see
|
||||
[Sync mechanism](#sync-mechanism) below).
|
||||
[Staying current](#staying-current) below).
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
|
|
@ -38,91 +38,25 @@ and every few minutes after, and seeds an empty one with a `README.md`
|
|||
containing a contribution guide and a blank table of contents. Add an entry to that ToC each time you
|
||||
create a new document.
|
||||
|
||||
## Sync mechanism
|
||||
## Staying current
|
||||
|
||||
hive-c0re maintains the local clone at
|
||||
`/var/lib/hyperhive/knowledge` via two paths:
|
||||
Each hive keeps a local clone of the repo and mounts it read-only into every
|
||||
agent container. The swarm controller holds the single push hook on
|
||||
`internal/knowledge`, and on any push to main it sends an event that makes
|
||||
every hive pull. A periodic pull closes the gap for a hive that was offline, so
|
||||
one hive may briefly show older content than another. When a pull changes
|
||||
anything, every agent gets a `/knowledge updated` notice.
|
||||
|
||||
1. **Swarm event** — the swarm controller holds the single push hook on
|
||||
`internal/knowledge` (see
|
||||
[swarm-wide forge webhooks](../../4-implementation/swarm/controller.md#swarm-wide-forge-webhooks)).
|
||||
On any push to main, including merge commits, it sends an
|
||||
event to every hive over the swarm queue and each hive runs `git
|
||||
pull`, so agents see the new content on their next turn.
|
||||
**Don't add a per-hive hook.** A webhook has one target URL, so a second
|
||||
registration takes delivery away from whoever registered first.
|
||||
|
||||
A hive that's offline when the swarm controller sends the event
|
||||
doesn't get it on reconnect — the periodic pull below is what closes that gap. One
|
||||
hive briefly showing older `/knowledge` content than another is
|
||||
expected, and resolves by itself within the fallback interval.
|
||||
The clone lives on the host at `/var/lib/hyperhive/knowledge` and survives
|
||||
hive-c0re restarts and agent destroy or recreate; only a manual delete removes
|
||||
it. The mount leaves out `.git`, so agents see the documents but not the
|
||||
history.
|
||||
|
||||
**don't add a per-hive hook.** A webhook has exactly one target
|
||||
URL, so a second registration against the same repo doesn't add a
|
||||
recipient — it takes delivery away from whoever registered first.
|
||||
Earlier versions had each hive register its own, and later ones
|
||||
removed that leftover at startup. A hive upgraded straight past those
|
||||
versions still holds its old hook: delete any hook on the repo that
|
||||
points at a hive's own `/webhook/knowledge`.
|
||||
|
||||
2. **Periodic pull** — a background task in `hive-c0re::main`
|
||||
queues a pull on a fixed cadence as a fallback (webhook missed, c0re
|
||||
restarted between pushes). The pull is best-effort — a failure
|
||||
shows as a failed `KnowledgePull` node on the job queue, counts toward
|
||||
the `knowledge_pull` hive warning, and doesn't affect the rest of the daemon.
|
||||
|
||||
A failed pull of either kind, or the one at boot, also queues a retry after 5 s,
|
||||
doubling per failure in a row up to 5 min, until a pull succeeds. The
|
||||
`knowledge_pull` hive warning appears after seven failures in a row, about five
|
||||
minutes in, so a forge still starting at boot raises nothing.
|
||||
|
||||
Both paths, and the one pull at boot, queue the same `KnowledgePull` job
|
||||
node. It holds the working tree for its duration, so two pulls never run
|
||||
over each other — each trigger queues its own pass, and a pass that lands
|
||||
while another is already queued or running waits its turn, since the
|
||||
one ahead of it may have fetched before the push. The node runs
|
||||
`knowledge::pull()`, which also handles the change notice below.
|
||||
|
||||
### Change notice
|
||||
|
||||
When a pull actually moves the local clone's `HEAD` (a real change,
|
||||
not a no-op — for example the periodic pull finding nothing new), hive-c0re
|
||||
broadcasts a short notice to every currently registered agent's inbox:
|
||||
sender `system`, body `[system] /knowledge updated:` followed by a
|
||||
`git diff --stat <old>..<new>` summary of what changed (or a generic
|
||||
"see the repo" fallback if computing the diff itself fails). This is
|
||||
the same broadcast mechanism used for other hive-wide notices — inbox
|
||||
message only, no forced wake, and it carries the standard "this was a
|
||||
broadcast" hint. hive-c0re logs a diff or per-agent send failure
|
||||
without blocking the pull itself.
|
||||
|
||||
On a hive with no local clone yet, `knowledge::pull()` takes the clone
|
||||
instead of pulling, so every hive gets `/knowledge` from its first pull,
|
||||
whether or not it runs the forge. It authenticates with the hive's forge
|
||||
read token when it has one (see
|
||||
[forge.md](forge.md#how-a-hive-reads-the-forge)) and clones the public
|
||||
repo anonymously otherwise.
|
||||
|
||||
## State
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
- **Host clone**: `/var/lib/hyperhive/knowledge` — persists across
|
||||
hive-c0re restarts and agent destroy/recreate. Deleted only by
|
||||
manual operator action.
|
||||
- **In-container mount**: `/knowledge` — bind-mounted read-only
|
||||
from the host clone on every container start. Gone when container
|
||||
is stopped; reappears on next start with the current clone state.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
The mount deliberately **excludes `.git`**: hive-priv overlays an empty
|
||||
tmpfs at `/knowledge/.git`, so agents see the documents, not the branches or
|
||||
history. An older clone also kept a token in `.git/config`; `knowledge::pull()`
|
||||
rewrites that remote to the plain URL.
|
||||
|
||||
hive-priv adds the overlay only when the host clone has a `.git`. Before the
|
||||
first clone succeeds, the agent starts with an empty `/knowledge` and no
|
||||
overlay, because nspawn can't create the mount point inside the read-only
|
||||
bind. A container that started in that window sees the `.git` once the clone
|
||||
lands, until its next start.
|
||||
The sync jobs, retry timing, the change notice and the mount rules:
|
||||
[`knowledge.md`](../../4-implementation/swarm/knowledge.md).
|
||||
|
||||
## Contributing
|
||||
|
||||
|
|
@ -161,7 +95,7 @@ required.
|
|||
|
||||
4. The operator reviews and merges. The webhook fires on merge; every
|
||||
running container sees the updated content within seconds (see
|
||||
[Sync mechanism](#sync-mechanism)).
|
||||
[Staying current](#staying-current)).
|
||||
|
||||
Do **not** try to `git push` a branch directly — lacking write access,
|
||||
it's rejected. The `--agit` flow above is the no-fork path that works
|
||||
|
|
|
|||
|
|
@ -115,94 +115,16 @@ leases an address from the gateway's DHCP, resolves through its dnsmasq,
|
|||
and leaves through the bridge's NAT. It reaches the secret store,
|
||||
authelia and federation peers by name.
|
||||
|
||||
## Provisioning flow (appservice)
|
||||
## Accounts
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
Registration is closed. **The swarm's appservice** creates accounts: agents
|
||||
never see an appservice token, and an agent only ever receives its own
|
||||
`access_token`.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
The homeserver refuses registration. **The swarm's appservice** creates every account:
|
||||
agents never see an appservice token, and an agent only ever receives its own
|
||||
`access_token`. `swarm-controller` creates each agent's accounts and one sender
|
||||
account per hive, `@hive-<hive>:<server_name>`, an ordinary account that
|
||||
nothing promotes.
|
||||
|
||||
The appservice has no URL (`url: null` in its registration), so the
|
||||
homeserver never calls out to it and there is no service to run. What the
|
||||
registration buys is an identity the homeserver recognises — which is why
|
||||
no secret has to be equal on both sides of the wire, and why account
|
||||
creation doesn't depend on registration being open to anyone who learns
|
||||
a token.
|
||||
|
||||
1. **`swarm-matrix-ctl appservice render`**, inside the `hive-matrix`
|
||||
container and before tuwunel starts, fetches the `as_token` and
|
||||
spec-required `hs_token` from the store and renders the registration to
|
||||
`/run/swarm-matrix-appservice/swarm.yaml` (mode `0600 root:root`) on the
|
||||
container's tmpfs, so no token sits on disk. It mints a pair only when
|
||||
the store answers and holds none. While the store doesn't answer, `render`
|
||||
fails and retries, and tuwunel stays down.
|
||||
2. **systemd `LoadCredential=`** copies the registration into
|
||||
`/run/credentials/tuwunel.service/swarm-appservice.yaml`, owned by
|
||||
tuwunel's dynamic user with mode `0400`, at service start. The source
|
||||
file stays `root:root 0600` — no `chown :tuwunel` / `chmod 0640` /
|
||||
GID-pin gymnastics required. Keeps `DynamicUser = true` +
|
||||
`PrivateUsers = true` intact.
|
||||
3. tuwunel's `appservice_dir` points at the credentials directory. It
|
||||
reads only `.yaml`/`.yml` entries from there, so the sibling
|
||||
credentials are invisible to it. The `.yaml` suffix on the credential
|
||||
id is what makes this work.
|
||||
4. **hive-c0re** holds no appservice token, creates no account and does
|
||||
not talk to matrix.
|
||||
5. **`swarm-controller` creates every account**: each agent's, with its
|
||||
token stored at `swarm/agents/<agent>/matrix/main` for the agent's
|
||||
`hive-matrix-daemon` to read as the agent itself, and each hive's
|
||||
sender account (below).
|
||||
|
||||
A homeserver whose `swarm-matrix-ctl` has no store identity
|
||||
(`services.hyperhive.deploy.matrix.ctlBaoClientCertFile`) loads no
|
||||
registration, and nothing creates accounts on it.
|
||||
|
||||
On the store's host `ctlBaoClientCertFile` / `ctlBaoClientKeyFile` default
|
||||
to the leaf `glue-bao-tls.nix` mints there. A homeserver host without the
|
||||
store gets no default, even when it holds the hive's own reader
|
||||
`deploy.bao.clientCertFile`: issue a leaf whose subject is
|
||||
`deploy.bao.matrixCtlCommonName`, place it, and name it in both options.
|
||||
Until you do, `swarm-bao-matrix-oidc` fails naming both.
|
||||
|
||||
### The swarm's appservice, and its admin sender
|
||||
|
||||
The registration's id is `swarm`, its sender `@swarm`, and its user
|
||||
namespace non-exclusive. It's the swarm's identity on the homeserver:
|
||||
`swarm-controller` creates every agent's account and every hive's sender
|
||||
account with it.
|
||||
|
||||
The swarm secret store holds the registration's tokens, at
|
||||
`swarm/controller/swarm-controller/matrix/appservice-registration`.
|
||||
`swarm-matrix-ctl appservice render` fetches them and renders the
|
||||
registration tuwunel loads as a credential onto the `hive-matrix`
|
||||
container's tmpfs, before tuwunel starts.
|
||||
A render that can't reach the store fails and retries, and tuwunel, which fails to start without the registration, retries until one has landed.
|
||||
`appservice sync` runs beside tuwunel on a timer. The store's tokens win:
|
||||
when they differ from the rendered copy, it rewrites it. It
|
||||
restarts tuwunel whenever the registration is newer than tuwunel's last
|
||||
start, so the next run retries one that failed after writing, and starts
|
||||
one that failed because `render` couldn't reach the store. A store that
|
||||
holds none takes the tokens an earlier version kept in
|
||||
`/var/lib/swarm-matrix-appservice` with a create-only write, so a running
|
||||
swarm keeps its value; the files stay, unused. A host with neither files nor stored tokens mints a pair. `sync` then writes the `as_token`
|
||||
to `swarm/controller/swarm-controller/matrix/appservice-token`. No hive's
|
||||
policy reaches either path: only matrix-ctl (which holds `create` and
|
||||
`read` on the registration, and writes the token) and `swarm-controller`
|
||||
(which reads only the token) hold a grant.
|
||||
|
||||
`@swarm` is the **one** account the homeserver promotes to admin at boot
|
||||
(`admin_execute`). Only those two principals read its token, which is
|
||||
the difference from the hive sender below: every hive reads its own
|
||||
sender's token. `swarm` is a reserved name, so nothing can create an agent
|
||||
as this account.
|
||||
|
||||
`swarm-controller` mints each agent's account with the device id
|
||||
`hyperhive-<agent>`, and a login on that device replaces its token. The controller therefore reads
|
||||
the stored token back with `whoami` every five minutes and re-mints only
|
||||
when it's missing, unknown to the homeserver, or someone else's. An
|
||||
agent's daemon picks a re-minted token up on its next start, and a timer
|
||||
restarts it while it's down.
|
||||
How the registration gets rendered, where its tokens live and how the
|
||||
controller keeps an agent's token: [`matrix-provisioning.md`](../../4-implementation/swarm/matrix-provisioning.md).
|
||||
|
||||
### Operators are homeserver admins
|
||||
|
||||
|
|
@ -241,101 +163,25 @@ lands on.
|
|||
`services.hyperhive.deploy.swarm-controller.authBridgeUrl` and
|
||||
`services.hyperhive.deploy.swarm-controller.matrixHomeserverUrl` set.
|
||||
|
||||
### A hive's sender account, and why it isn't an admin
|
||||
|
||||
`@hive-<hive>:<server_name>` is the account a hive acts as.
|
||||
`swarm-controller` creates it with the swarm's appservice token, for every
|
||||
hive in its directory. It's an **ordinary account**: nothing promotes it;
|
||||
the homeserver's `admin_execute` promotes only `@swarm`.
|
||||
### Settings
|
||||
|
||||
**One account per hive.** The localpart carries the hive's name, so a swarm
|
||||
whose hives share a homeserver gives each of them its own identity: the
|
||||
homeserver attributes an action to the hive that took it, and revoking one
|
||||
hive's standing leaves the others alone.
|
||||
- **Federation** is on at the protocol level, but `trusted_servers = []` keeps
|
||||
it closed until you list peers in `trustedServers` (see
|
||||
[Configuration tuning](#configuration-tuning)).
|
||||
- **Registration** is off. The swarm's appservice still creates accounts, and
|
||||
the homeserver refuses everyone else.
|
||||
- **Encryption**: `services.hyperhive.deploy.matrix.allowEncryption` (default
|
||||
`false`) lets rooms on this homeserver use encryption. Turn it on for
|
||||
encrypted rooms with external homeservers, or to keep contents opaque to the
|
||||
homeserver admin. The agents' matrix client reads encrypted rooms either way.
|
||||
- **Key backup.** Every matrix account an agent has gets a recovery passphrase
|
||||
in the swarm secret store, so a crypto store lost to `--purge` or a token
|
||||
reset comes back with its history.
|
||||
|
||||
Its access token is the **sender token**. `swarm-controller`, its only
|
||||
minter, mints it for every hive with the swarm's appservice token and
|
||||
publishes it to `swarm/hives/<hive>/matrix/sender-token`, inside the hive's own
|
||||
read grant (`swarm/hives/<hive>/*`). Nothing on the hive uses it any more:
|
||||
hive-c0re used to present it for its Space and invite calls and no longer
|
||||
makes any. The name says what it authenticates as — an account the
|
||||
swarm's appservice creates — rather than any privilege level, because it
|
||||
carries none.
|
||||
|
||||
Promoting a user to homeserver admin and resetting a password both need an
|
||||
admin **sender**: `!admin …` messages into `#admins:<server_name>`, and
|
||||
tuwunel only treats a message as a command when its sender is already an
|
||||
admin. Of the swarm's own accounts only `@swarm` is a homeserver admin
|
||||
(operators are too, see
|
||||
[Operators are homeserver admins](#operators-are-homeserver-admins)), so
|
||||
both are swarm-level operations.
|
||||
|
||||
<details><summary>Appservice identity</summary>
|
||||
|
||||
`swarm-matrix-ctl appservice render` renders the registration from the
|
||||
store's copy of the appservice tokens before tuwunel starts, so every
|
||||
boot has both halves. With the store down or sealed, tuwunel waits for it.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
- **Access tokens live on the device that minted them.**
|
||||
`login_with_password` stays on, so the password fallback works
|
||||
too.
|
||||
- **The homeserver's `admin_execute` promotes only `@swarm` at boot**
|
||||
(above); a hive's sender account is never promoted.
|
||||
- **The value that matters lives at
|
||||
`swarm/controller/swarm-controller/matrix/appservice-token`.**
|
||||
`appservice sync` writes it there whenever the store's differs from the
|
||||
registration's `as_token`, and `swarm-controller` reads it from there.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
</details>
|
||||
|
||||
Initial rollout settings:
|
||||
|
||||
- `allow_federation = true` at the protocol level so swarms can be
|
||||
wired up later by extending `trustedServers` without a homeserver
|
||||
restart. `trusted_servers = []` keeps it effectively closed
|
||||
until you list peers.
|
||||
- `allow_registration = false`. tuwunel checks this flag only for
|
||||
requests that arrive **without** an appservice token, so the swarm's
|
||||
appservice still creates accounts and tuwunel refuses everyone else. With no
|
||||
registration token configured, `allow_registration = true` makes
|
||||
tuwunel refuse to start unless
|
||||
`yes_i_am_very_very_sure_…_open_registration_…` is also set.
|
||||
- `allow_encryption` — server-side E2EE switch, sourced from
|
||||
`services.hyperhive.deploy.matrix.allowEncryption` (**default `false`**, opt-in).
|
||||
Off by default because on the swarm's own homeserver the operator
|
||||
already controls the transport; turn it on for encrypted rooms on
|
||||
external / federated homeservers or to keep contents opaque to the
|
||||
homeserver admin. **The agent matrix client always supports decryption
|
||||
regardless of this flag** — it uses the `e2e-encryption` feature of
|
||||
`matrix-sdk` so it can read encrypted rooms it's invited to even when
|
||||
this homeserver doesn't permit room encryption. matrix-sdk stores
|
||||
crypto keys in the per-agent sqlite store under the state dir; they persist across
|
||||
restarts. `read_room` decrypts via
|
||||
`room.messages()` — UTD events surface as `event_type =
|
||||
"m.room.encrypted"` with `body = "[unable to decrypt]"`.
|
||||
The daemon bootstraps cross-signing on the bearer token alone, which
|
||||
works where the homeserver skips re-auth for a first key upload
|
||||
(MSC3967, as tuwunel does).
|
||||
- **Key backup.** `swarm-controller` mints a random recovery passphrase
|
||||
for every matrix account an agent has, at
|
||||
`swarm/agents/<agent>/matrix-recovery/<account>` in the swarm secret
|
||||
store, and never replaces it. When `hive-matrix-daemon` brings an
|
||||
account up it reads that passphrase: with no backup on the account's
|
||||
homeserver it turns on server-side key backup and secret storage
|
||||
(log line `matrix recovery enabled`), unless the account already has
|
||||
secret storage, which it never replaces (a `WARN` instead). With a
|
||||
backup there it recovers from it (`recovered from the server-side key backup`), which downloads the
|
||||
room keys and signs the new device with the account's existing
|
||||
cross-signing identity. A crypto store lost to `--purge` or to a
|
||||
primary-token reset therefore comes back with its history. With no passphrase
|
||||
stored yet the account comes up without backup and picks it up on the
|
||||
next daemon start. A failure is a `WARN` and never stops the daemon.
|
||||
|
||||
The module's retired hive-side code, its evaluation assertions and the
|
||||
fluffychat-web build fixes are in the
|
||||
[implementation notes](../../4-implementation/networking/matrix-homeserver.md).
|
||||
The details, and the retired hive-side code, are in
|
||||
[`matrix-provisioning.md`](../../4-implementation/swarm/matrix-provisioning.md)
|
||||
and the [implementation notes](../../4-implementation/networking/matrix-homeserver.md).
|
||||
|
||||
## Configuration tuning
|
||||
|
||||
|
|
|
|||
|
|
@ -104,186 +104,39 @@ this password is stored nowhere — record it now
|
|||
Authelia's file store has no SMTP notifier, so this is the only way a
|
||||
human account gets a new password once an operator forgets the old one.
|
||||
|
||||
## What secrets exist, and where each one lives
|
||||
## Logging in to the forge and matrix
|
||||
|
||||
[`credentials-reference.md`](credentials-reference.md) tabulates every secret in the swarm, with its
|
||||
generator and its path, in one place — including authelia's own keys
|
||||
(session, JWT, storage-encryption, OIDC HMAC, OIDC issuer) and the two
|
||||
halves of each client secret. That page's two rules — a secret is always
|
||||
a path, never a value, and the generator and the reader typically live in
|
||||
different containers — are why the client secret's plaintext half needs
|
||||
the delivery step below and the rest of authelia's keys don't.
|
||||
- **A person's first forge login creates their forge account**, named after
|
||||
their authelia username, provided the subject has an email. It starts as an
|
||||
ordinary user; `swarmctl forge make-admin <name>` makes it a site admin. A
|
||||
name that already has a local forge account isn't handed over: forgejo asks
|
||||
for that account's own password first. Agents, `core` and `swarm-controller`
|
||||
all have local accounts.
|
||||
- **A member of `admins` becomes a matrix homeserver admin** once their first
|
||||
matrix login has created the account. `swarm-controller` promotes them within
|
||||
five minutes, and never demotes. See
|
||||
[`matrix.md`](matrix.md#operators-are-homeserver-admins).
|
||||
|
||||
## Getting the plaintext to the relying party
|
||||
You place no client secret by hand. Authelia mints each one at its first boot
|
||||
and the host-side units deliver it to the forge and the homeserver, retrying
|
||||
until it lands, so there is no start order to get right. A forge on another host
|
||||
than the store needs the forge publisher's store leaf, in
|
||||
`deploy.bao.forgePublisherClientCertFile` and `forgePublisherClientKeyFile`.
|
||||
Without it the forge has no login source, and the unit fails naming both options.
|
||||
|
||||
Three cases, and they're genuinely different mechanisms rather than one
|
||||
mechanism with flags.
|
||||
|
||||
### 1. All-local — one host runs both
|
||||
|
||||
Nothing to configure at all. authelia mints each client's secret at its
|
||||
first boot, and the mint in its container creates it in the swarm
|
||||
secret store at `swarm/services/<client id>/oidc/client`, keeping only the digest on disk. A host-side unit
|
||||
then reads it from the store and writes it into the service's container:
|
||||
`swarm-bao-forge-oidc` for the forge, under the forge publisher's store
|
||||
leaf, into `/run/hive-forge-oidc/<client id>.secret` on the host, which
|
||||
`forgejo-sso-source` loads as a credential; `swarm-bao-matrix-oidc` for matrix, under
|
||||
matrix-ctl's. A unit that can't read the secret yet fails and retries
|
||||
every 30 seconds for a day, so there is no start order to get right. The
|
||||
service's own module contributes its client entry — callback URL
|
||||
included — to authelia's client list.
|
||||
|
||||
Each relying service's own glue module builds its callback once, and
|
||||
both authelia's client list and the service's own OIDC client config
|
||||
read that same value, so the redirect URI authelia accepts and the one
|
||||
the service actually sends can't drift apart. A mismatch there is a
|
||||
rejected login with no error text worth reading.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
⚠️ nixos-container refuses to start a container whose bind source is
|
||||
missing, and this secret doesn't exist until authelia's first boot has run
|
||||
and the mint has stored it. The forge therefore binds the **directory**
|
||||
`/run/hive-forge-oidc`, which a tmpfiles rule creates at boot, never the
|
||||
file. The directory is tmpfs, so the secret is gone after a reboot until
|
||||
the fetch runs again; forgejo keeps it in its login-source row, so logins
|
||||
still work meanwhile. Matrix's secret is written into the container's state
|
||||
directory instead.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
### 2. Swarm-managed services
|
||||
|
||||
The controller side owns provisioning: `swarmctl` writes both halves, the
|
||||
same way it already owns authelia's user store (`users.yml`, read and
|
||||
written in place).
|
||||
|
||||
### 3. A hive elsewhere
|
||||
|
||||
The forge's host needn't be the store's or authelia's host, and the
|
||||
operator places no secret file: `swarm-bao-forge-oidc` runs on the
|
||||
forge's host as in the all-local case and fetches the secret over the
|
||||
store's address. What that host needs is the forge publisher's store
|
||||
leaf, issued from the store's CA and copied over from the store's host:
|
||||
|
||||
```nix
|
||||
services.hyperhive = {
|
||||
swarm.authelia.url = "https://auth.example.com";
|
||||
deploy.bao = {
|
||||
forgePublisherClientCertFile = "/var/lib/swarm-bao-pki/forge-publisher.pem";
|
||||
forgePublisherClientKeyFile = "/var/lib/swarm-bao-pki/forge-publisher-key.pem";
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
**A forge host without that leaf still evaluates.** `swarm-bao-forge-oidc`
|
||||
renders and fails at start, naming both options, and the forge has no
|
||||
login source until an operator places the leaf. Where the store runs on
|
||||
the same host, `glue-bao-tls.nix` defaults both options to the leaf it
|
||||
mints.
|
||||
|
||||
## Where each relying party differs
|
||||
|
||||
The registration half is identical; what each service does with the
|
||||
result isn't.
|
||||
|
||||
| | forge | matrix |
|
||||
| ------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
||||
| how it learns the config | a oneshot calls `forgejo admin auth`, writing a login-source row into its database | tuwunel reads a `[[global.identity_provider]]` entry from its config file |
|
||||
| how it reads the secret | a path inside its container | the same path, handed on by `LoadCredential` |
|
||||
| callback URL | `<root>/user/oauth2/<source>/callback` | `<homeserver>/_matrix/client/unstable/login/sso/callback/<client_id>`, a shape tuwunel fixes rather than accepts |
|
||||
| cost of a malformed entry | the login source is missing | the homeserver can refuse to start |
|
||||
|
||||
Four consequences worth stating plainly:
|
||||
|
||||
- **A person's first forge login creates their forge account**, named
|
||||
after authelia's `preferred_username` (`[oauth2_client]` in the forge
|
||||
module), provided the subject has an email. It starts as an ordinary user; `swarmctl forge make-admin
|
||||
<name>` makes it a site admin. A name that already has a local forge
|
||||
account doesn't get it handed over: forgejo asks for that account's
|
||||
own password first (`ACCOUNT_LINKING = login`). Agents, `core` and
|
||||
`swarm-controller` all have local accounts, and `swarmctl user add`
|
||||
refuses none of those names.
|
||||
|
||||
- **A member of `admins` becomes a matrix homeserver admin** once their
|
||||
first matrix login has created the account: `swarm-controller` promotes
|
||||
them within five minutes, and never demotes. See
|
||||
[`2-handbook/swarm/matrix.md`](matrix.md#operators-are-homeserver-admins).
|
||||
|
||||
- **tuwunel re-reads its secret file on every OAuth exchange**, not only
|
||||
at startup, and its own sandboxing hides most paths from it. It gets the
|
||||
file through `LoadCredential` for the same reason the appservice
|
||||
registration does — that keeps `DynamicUser` and `PrivateUsers` intact,
|
||||
with no host-side ownership arrangement to maintain.
|
||||
- **Matrix SSO lives inside the homeserver.** The client-server API is
|
||||
spoken by non-browser clients holding matrix access tokens — every
|
||||
agent's own daemon — as well as by federation, so the homeserver
|
||||
serves `/_matrix/` directly and authenticates it itself. The
|
||||
forward-auth vhosts protect browser surfaces; this isn't one of them.
|
||||
Where each secret lives, the delivery mechanics and how forge and matrix
|
||||
differ: [`sso.md`](../../4-implementation/swarm/sso.md).
|
||||
|
||||
## Machine clients
|
||||
|
||||
Everything above is a person in a browser. A swarm also has callers that
|
||||
hold no session and follow no redirect: the telemetry collector, the
|
||||
queue's auth-callout responder, and each hive.
|
||||
|
||||
**One client per hive, not one per service.** A hive's identity belongs to
|
||||
the directory rather than to whichever service happens to consume it, so a
|
||||
hive holds a single OIDC client — `<hiveClientPrefix><hive>` — and mints a
|
||||
different token per service from it. The alternative, letting each
|
||||
consuming subsystem declare its own list, collides on the same client id
|
||||
the moment a second consumer appears.
|
||||
|
||||
**The audience is that client id.** A swarm service that has to tell hives
|
||||
apart needs one name both sides already agree on, and the client id is
|
||||
already that name. A parallel per-hive naming scheme would be a second
|
||||
thing to keep in step, and it drifts silently — a mismatch presents as a
|
||||
valid token refused at the target, which reads like a broken credential
|
||||
rather than a broken name.
|
||||
|
||||
**Authelia signs tokens (`RS256`), not opaque ones**, because a resource server
|
||||
that can't call the provider back is a real case here: the telemetry
|
||||
collector verifies offline against `/jwks.json`, and an opaque token gives
|
||||
it nothing to verify. The queue's responder introspects instead — a
|
||||
different question asked of the same token, and the reason both
|
||||
`/api/oidc/introspection` and `/jwks.json` have to stay reachable.
|
||||
|
||||
### Machine callers must fail closed
|
||||
|
||||
⚠️ **An error page that answers `200` is a security bug, not a cosmetic
|
||||
one.** The browser surface intercepts upstream errors and serves a
|
||||
friendly "SSO is unavailable" page; that page is a file, so it returns
|
||||
`200`. Any machine caller routed through it receives a success carrying
|
||||
HTML instead of the failure that actually happened:
|
||||
|
||||
- `/api/authz/auth-request` — nginx `auth_request` treats **any 2xx as
|
||||
success**, so a down provider means _access granted_
|
||||
- `/api/oidc/introspection` — a token check that answers `200`
|
||||
- `/api/oidc/token`, `/.well-known/openid-configuration` — a client
|
||||
parsing an error page as its JSON document
|
||||
|
||||
The gateway routes authelia's `/api/` and `/.well-known/` prefixes, and
|
||||
the bridge's `/bridge/`, **without** error interception. The split is by
|
||||
_audience_, not by an enumerated path list: a human gets the page, every
|
||||
machine caller gets the status.
|
||||
Enumerating endpoints individually would leave the next one added
|
||||
silently intercepted.
|
||||
|
||||
The same shape bites any machine route behind a browser-shaped gate: the
|
||||
caller follows a `302` to a login page, the login page answers `200`,
|
||||
and the caller reports success while nothing happened. Log ingest hit exactly this
|
||||
and lost eleven hours of delivery in silence.
|
||||
|
||||
**Checking it, if you change this routing.** Point the vhost at a dead
|
||||
upstream and compare three requests, not one:
|
||||
|
||||
1. through `/` — must still serve the friendly page
|
||||
2. through `/api/` — must deny
|
||||
3. a direct dial to authelia — must match what (2) did
|
||||
|
||||
All three matter. A change that silently deleted the browser page would
|
||||
pass a deny-only check, and one that quietly stopped denying would pass a
|
||||
page-only check. The person who introduced the split verified it this way.
|
||||
Everything above is a person in a browser. A swarm also has callers that hold
|
||||
no session and follow no redirect: the telemetry collector, the queue's
|
||||
auth-callout responder, and each hive. A hive holds a single OIDC client,
|
||||
`<hiveClientPrefix><hive>`, and mints a different token per service from it.
|
||||
The gateway answers machine callers with the real status rather than the
|
||||
friendly "SSO is unavailable" page, so a down provider denies access instead of
|
||||
granting it. The reasoning and how to check it after a routing change:
|
||||
[`sso.md`](../../4-implementation/swarm/sso.md#machine-callers-must-fail-closed).
|
||||
|
||||
## Operator CLI login
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue