docs: swarm/services.md is one blurb per service, the host detail moves to the service's implementation page
Each service section lost its container, credential and metrics detail to a Host setup section on its own implementation page, unchanged apart from links. The four agentAuth option rows that sat above the page title moved into the telemetry page's metrics option table.
This commit is contained in:
iris
mara
parent
60aede950b
commit
814edd1d67
8 changed files with 392 additions and 308 deletions
|
|
@ -681,3 +681,59 @@ is pure and runs in the container. Builds go to the host's nix daemon as an
|
|||
untrusted client: a config can set no daemon option, and its derivations build
|
||||
in the host's sandbox. A fixed-output derivation still has the network, as it
|
||||
does for any build on that host.
|
||||
|
||||
## Host setup
|
||||
|
||||
The swarm builder, in a `swarm-builder` container on the host that sets
|
||||
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host
|
||||
has the builder's store identity, as for the other swarm services: the host
|
||||
that runs the store has it. Set `deploy.builder.enable = false` to opt out. It
|
||||
evaluates unreviewed config PRs and builds them through its host's nix daemon,
|
||||
from a container with its own network namespace.
|
||||
|
||||
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
|
||||
swarm-controller queues a `build_agent` job. Jobs run one at a time, and a
|
||||
newer push replaces its PR's job while that job is still waiting. The builder
|
||||
builds that config at the PR's head on the agent base its own host runs,
|
||||
through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
|
||||
commit status on the head. The job shows on the swarm UI's Jobs page. A red
|
||||
status blocks nothing: the merge still deploys.
|
||||
|
||||
Separately, the builder builds each agent from `internal/meta` before the swarm
|
||||
sends its deploy, and a failed or unpushed build holds that deploy back (`build_meta` job;
|
||||
[deploy gate](#deploy-gate)). Those jobs share the one-build slot with
|
||||
the PR checks.
|
||||
|
||||
It needs neither the forge nor a hive on its host. It reaches the queue, the
|
||||
secret store and the forge by name, under its own `builder` leaf, and fetches
|
||||
the private config repo with the read-only token of its own `builder` forge
|
||||
account. On a host without the store, that leaf is the one file to copy, as
|
||||
for the other swarm services.
|
||||
|
||||
## Binary cache host setup
|
||||
|
||||
The swarm's attic server, in a `swarm-attic` container on the host that sets
|
||||
`deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has
|
||||
the cache's store identity, as for the other swarm services: the host that runs the store has
|
||||
it. For an external cache, override `swarm.attic.{domain,cache}` and don't set
|
||||
`deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under
|
||||
`allSwarmServices`.
|
||||
|
||||
The container sits on the hive bridge in a network namespace of its own, and atticd
|
||||
listens on that namespace's loopback only; the gateway reaches it over a unix socket.
|
||||
The gateway serves it at `swarm.attic.domain` (default
|
||||
`attic.<swarm domain>`, on every host) over TLS, with no login in front, because attic checks
|
||||
its own tokens. Storage is local disk in the container's state.
|
||||
|
||||
It needs no manual secret. Two host units, under the cache's own `attic` leaf,
|
||||
create its signing key, the cache and the push and pull tokens in the secret
|
||||
store, each only while absent. On a host without the store, that leaf is the one
|
||||
file to copy. The builder pushes green builds to it. How it works, and the
|
||||
external alternative, are in [Binary cache](#binary-cache).
|
||||
|
||||
The container forwards atticd's journal to the swarm collector like the other
|
||||
service containers (see [Telemetry collector](telemetry.md#telemetry-collector-otel)). The two host
|
||||
units' lines reach the log store too: through the hive collector's journal
|
||||
allow-list on a hive host, or through the swarm collector's whole-journal
|
||||
receiver where the host runs that. A host with neither keeps them in its own
|
||||
journal.
|
||||
|
|
|
|||
|
|
@ -101,3 +101,18 @@ variable:
|
|||
- finished runs per day, by outcome;
|
||||
- a table per PR over the time range: mean and longest successful run time,
|
||||
runs, failed runs.
|
||||
|
||||
## Host setup
|
||||
|
||||
The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host
|
||||
that sets `deploy.ci-metrics.enable`. It runs on the host, not in a container.
|
||||
`allSwarmServices` defaults it on wherever the host has the poller's store
|
||||
identity, as for the other swarm services. Set `deploy.ci-metrics.enable = false` to
|
||||
opt out.
|
||||
|
||||
Each pass pushes the run time and queue wait of every newly finished forge
|
||||
Actions run to the metrics store, through its own route on the swarm collector;
|
||||
the first pass pushes every run since the first. It reads the forge with the
|
||||
read-only token of its own `ci-metrics` forge account, under its own
|
||||
`ci-metrics` leaf. On a host without the store, that leaf is the one file to
|
||||
copy.
|
||||
|
|
|
|||
|
|
@ -118,3 +118,23 @@ gauges (`bao.client_cert.*`) come with them.
|
|||
`invalid` is a request that didn't decode or named no agent. A `start` spawns
|
||||
`claude auth login` and a `code` stores the credentials, so those two are the
|
||||
slow steps. `expired` counts logins stopped because their code never came.
|
||||
|
||||
## Host setup
|
||||
|
||||
The claude login minter, in a `swarm-claude-login` container on the host that
|
||||
sets `deploy.claude-login.enable`. `allSwarmServices` defaults it on wherever
|
||||
the host has the minter's store identity, like the other swarm services.
|
||||
Run one: a login in progress lives in the service's memory between its start
|
||||
and its code.
|
||||
|
||||
An operator logs an agent in from the **log in** button on its card in the
|
||||
swarm UI. swarm-controller relays the start and the pasted code over the queue,
|
||||
the minter runs `claude auth login` and writes the credentials to the agent's
|
||||
`swarm/agents/<agent>/claude-credentials`, and the agent's harness installs a
|
||||
version newer than the one it installed last.
|
||||
|
||||
It needs neither the forge nor a hive on its host. It reaches the queue and the
|
||||
secret store by name, under its own `claude-login` leaf, and reaches claude's
|
||||
login endpoints over the network. On a host without the store, that leaf is
|
||||
the one file to copy, as for the other swarm services. The flow and the version rule are
|
||||
in [`claude-login.md`](../../2-handbook/swarm/claude-login.md).
|
||||
|
|
|
|||
|
|
@ -433,3 +433,26 @@ no flow matched. `retried` is one the queue or the store failed on: it comes bac
|
|||
and counts again. A growing `consumer.pending` means the router is falling
|
||||
behind. The router reads the consumer counts at the start of each pull, at most
|
||||
30 seconds apart.
|
||||
|
||||
## Host setup
|
||||
|
||||
The notification router, in a `swarm-notify-router` container on the host
|
||||
that sets `deploy.notify-router.enable`. `allSwarmServices` defaults it on
|
||||
wherever the host has the router's store identity, on the review classifier's
|
||||
terms. Run one: it handles each issue's events in order.
|
||||
|
||||
swarm-controller keeps every `issues` and `pull_request` delivery, and every
|
||||
failed or cancelled CI run, from its instance-wide hooks in the `SWARM_EVENTS`
|
||||
stream. The router reads that
|
||||
stream and queues todos for the agents an event concerns: an agent assigned to
|
||||
an issue or pull request gets a todo, one taken off it gets that todo rewritten
|
||||
to say so, and closing it clears them. The author of a pull request gets a todo
|
||||
when a CI run on it fails or gets cancelled, and a push to the branch clears it. A push to an open pull request makes
|
||||
it ask swarm-controller to request the review classifier's review. Each agent's harness takes its todos
|
||||
from the `ROUTER_TODOS` stream, so an agent that's down gets them when it's
|
||||
back.
|
||||
|
||||
It needs neither the forge nor a hive on its host, and holds no forge token.
|
||||
It reaches the queue and the secret store by name, under its own
|
||||
`notify-router` leaf. On a host without the store, that leaf is the one file to
|
||||
copy, as for the other swarm services.
|
||||
|
|
|
|||
|
|
@ -194,7 +194,7 @@ own: opencode merges an agent's permission over `OPENCODE_PERMISSION`, so one
|
|||
there would offer the model tools again.
|
||||
|
||||
The service is `swarm-review-classifier`, and its container is the one
|
||||
[`services.md`](../../2-handbook/swarm/services.md#review-classifier) describes. It holds no forge
|
||||
[Host setup](#host-setup) describes. It holds no forge
|
||||
credential: `swarm-controller` reads the PR and sends it along.
|
||||
|
||||
### Swarm-level settings
|
||||
|
|
@ -315,7 +315,7 @@ deleted, still shows as set, because the controller can't read a value to check.
|
|||
|
||||
The classifier reads the leaf on every run, so a new key applies from the next
|
||||
run. The operator can still write the leaves in the secret store's UI
|
||||
([`services.md`](../../2-handbook/swarm/services.md#review-classifier)).
|
||||
([Host setup](#host-setup)).
|
||||
|
||||
The per-agent equivalent is [`inference.md`](inference.md), where the
|
||||
controller writes model and effort to a queue KV bucket
|
||||
|
|
@ -765,3 +765,34 @@ These get settled, and filled in here, as the work reaches them:
|
|||
`secret/swarm/services/review-classifier/forge/token`.
|
||||
- **Reviewer handoff.** How a later reviewer, human or agent, learns the
|
||||
classifier passed a PR before it starts its own review.
|
||||
|
||||
## Host setup
|
||||
|
||||
The review classifier's try-out service, in a `swarm-review-classifier`
|
||||
container on the host that sets `deploy.review-classifier.enable`.
|
||||
`allSwarmServices` defaults it on wherever the host has the classifier's store
|
||||
identity: the host that runs the store has it, and a services host with the
|
||||
store elsewhere gets the classifier only once it holds a copy of the pair (see
|
||||
below). Set `deploy.review-classifier.enable` directly to override either way.
|
||||
|
||||
The operator sets the key of the runtime the swarm settings choose in the
|
||||
swarm UI: the **classifier settings** card on `/jobs` takes a key for the
|
||||
chosen runtime and shows whether one exists and when it changed, never the
|
||||
key. "Set" means a key exists, not that it works: a run with a bad key
|
||||
still fails. You can also write the same two keys in the secret store's UI under the
|
||||
OIDC login, field `api_key`:
|
||||
|
||||
- the API key runtime's Anthropic key at
|
||||
`secret/swarm/services/review-classifier/anthropic-api-key`
|
||||
- the ACP runtime's provider key at
|
||||
`secret/swarm/services/review-classifier/acp-provider`
|
||||
|
||||
Nothing writes them for you: until the chosen one exists the service runs and
|
||||
every run fails with `no API key…` or `no ACP provider key…`. The service reads
|
||||
the key on every run, so writing or removing it takes effect on the next run,
|
||||
with no restart. Every run spends money on that key.
|
||||
|
||||
It needs neither the forge nor a hive on its host. It reaches the queue and the
|
||||
secret store by name, under its own `review-classifier` leaf. On a host
|
||||
without the store, that leaf is the one file to copy, as for the other swarm services.
|
||||
The swarm UI's try-out drives it through swarm-controller.
|
||||
|
|
|
|||
|
|
@ -184,3 +184,60 @@ upstream and compare three requests, not one:
|
|||
All three matter. A change that silently deleted the browser page would
|
||||
pass a deny-only check, and one that quietly stopped denying would pass a
|
||||
page-only check. The person who introduced the split verified it this way.
|
||||
|
||||
## Host setup
|
||||
|
||||
One authelia per swarm, in a `swarm-authelia` container, at
|
||||
`auth.<swarm-domain>`. Operator and agents are both subjects of the same
|
||||
provider, differentiated by roles and claims rather than by mechanism —
|
||||
there is one IdP and one auth path.
|
||||
|
||||
The container sits on the hive bridge in a network namespace of its own and
|
||||
dials the store, the collector and its own public name through it. It binds
|
||||
no TCP port: the login server, the metrics listener and the bridge each
|
||||
listen on a unix socket in `/run/swarm-authelia`, a directory only the
|
||||
gateway's nginx and the container's processes can open.
|
||||
|
||||
- **`deploy.authelia`** — run the container here.
|
||||
- **`swarm.authelia.url`** — where clients go to authenticate. Present
|
||||
on **every** hive and the same value on all of them:
|
||||
`https://<swarm.authelia.domain>`, whether or not this host runs the
|
||||
container. Set it explicitly when joining a swarm whose IdP is under
|
||||
another name.
|
||||
|
||||
Agent subjects come from swarm-controller's agent-creation job, written
|
||||
into the users database by `swarm-authelia-bridge`; human ones come from
|
||||
`swarmctl user add` → [setup.md § 2](../../1-getting-started/setup.md#2--your-sso-account).
|
||||
|
||||
The bridge listens on a unix socket on the authelia host, and the gateway
|
||||
publishes it as
|
||||
`https://<swarm.authelia.domain>/bridge/`, which is where
|
||||
`deploy.swarm-controller.authBridgeUrl` points by default on every host.
|
||||
It answers only swarm-controller's OIDC client
|
||||
(`swarm.controller.queueClientId`) and refuses every other token `403`.
|
||||
|
||||
On first boot this module seeds the users database with a placeholder subject marked `disabled`, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia's session, storage and other keys live in the swarm secret store; the container copies them into place at start, seeded from the files an earlier release generated there, and generates them itself only where it has no store identity. Nothing rotates them automatically; replacing one
|
||||
invalidates data already written (sessions, the encrypted store), so
|
||||
that's an operator action.
|
||||
|
||||
Storage is local sqlite and the notifier writes to a file. Both are
|
||||
small-deployment choices, and the scope is the justification: redis
|
||||
buys shared session state across replicas and there is one instance;
|
||||
SMTP exists to mail humans, and provisioning here is programmatic.
|
||||
|
||||
The swarm collector scrapes authelia's own metrics at
|
||||
`https://<swarm.authelia.domain>/metrics`, whichever host runs the
|
||||
container. The gateway admits only the collector's OAuth2 client there;
|
||||
any other session gets `deny`.
|
||||
|
||||
The container's `authelia-swarm-secrets` unit creates each client secret directly in
|
||||
the swarm secret store, under the secret publisher's own store leaf handed in as
|
||||
a system credential, and writes only the digest to disk; see
|
||||
[`credentials.md`](../../2-handbook/swarm/credentials.md). Authelia starts from its digests, so the IdP
|
||||
comes up while the store is down, and a unit that needs the store retries every
|
||||
30 seconds. The container's journal forwarder reads its client secret back from
|
||||
the store under the same leaf, and is off where the container has none.
|
||||
|
||||
See [`sso.md`](../../2-handbook/swarm/sso.md) for bootstrapping the first user and the OIDC
|
||||
relying-party flow, and [`credentials.md`](../../2-handbook/swarm/credentials.md) for where authelia
|
||||
generates and reads each of its keys.
|
||||
|
|
|
|||
|
|
@ -81,3 +81,28 @@ set them.
|
|||
Two things the daemon doesn't do yet: sweep scratch directories a crash left
|
||||
behind, and set whisper's thread count. The issue 6383 discussion has the
|
||||
choices behind the defaults.
|
||||
|
||||
## Host setup
|
||||
|
||||
The speech-to-text service, in a `swarm-stt` container on the host that sets
|
||||
`deploy.stt.enable`. `allSwarmServices` defaults it on wherever the host has the
|
||||
service's store identity, like the other swarm services. Run it where the inference
|
||||
hardware is: it transcribes with whisper.cpp.
|
||||
|
||||
An agent that receives a voice memo sends the audio to the service over the
|
||||
queue, on its own subject, and gets the text back. The service keeps each
|
||||
transcript for a day, keyed by a hash of the audio, so the same memo sent to several agents costs
|
||||
one whisper run. `deploy.stt.settings.modelSize` picks the
|
||||
model (`small` by default, or `base`), and `modelFile` takes any other ggml
|
||||
model. The cache time, the limits on file size, audio length and run time, and
|
||||
how many requests it holds at once are settings next to it.
|
||||
|
||||
The audio comes from rooms with outside people, so the service runs as its own
|
||||
unprivileged user under a tight systemd sandbox, with a memory ceiling and
|
||||
a CPU quota. Its network isn't restricted: the decoder runs in the same
|
||||
network namespace as the service and can read its credentials, so a decoder
|
||||
exploit could read them and send them anywhere the container can reach. Closing
|
||||
that's for the strict-egress work on service containers, or for running the
|
||||
decode with no network and no credentials. The file's first bytes must look like an audio container
|
||||
before anything decodes it. Its leaf and its queue client work as the notification router's
|
||||
do.
|
||||
|
|
|
|||
|
|
@ -31,12 +31,16 @@ services.hyperhive.deploy = {
|
|||
⚠️ **This starts a database that grows for as long as the swarm runs.**
|
||||
See `retentionPeriod` below before leaving it at its default.
|
||||
|
||||
| Option | When you'd touch it |
|
||||
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. |
|
||||
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. |
|
||||
| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. |
|
||||
| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://<swarm.victorialogs.domain>`), so Grafana and the store can run on different hosts. |
|
||||
| Option | When you'd touch it |
|
||||
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `deploy.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you can't get back. |
|
||||
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who shouldn't be able to reconfigure Grafana. |
|
||||
| `deploy.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store's swarm name (`https://<swarm.victoriametrics.domain>`), so Grafana and the store can run on different hosts. |
|
||||
| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://<swarm.victorialogs.domain>`), so Grafana and the store can run on different hosts. |
|
||||
| `deploy.victoriametrics.agentAuthRate` | Default `50r/s` per client address, for agent reads checked by `swarm-metrics-agent-auth`; over it and `agentAuthBurst`, the gateway answers 403. |
|
||||
| `deploy.victoriametrics.agentAuthBurst` | Default `500` requests above `agentAuthRate` before the 403s start. |
|
||||
| `deploy.victoriametrics.agentAuthVerdictTtl` | Default `60` seconds that `swarm-metrics-agent-auth` keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. A revoked `metrics` permission does too. |
|
||||
| `deploy.victoriametrics.agentAuthPackage` | The `swarm-logs-auth` build that runs as `swarm-metrics-agent-auth`. Set on its own: `deploy.victorialogs.authPackage` doesn't reach it. |
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue