8879225fa4
VictoriaLogs shipped without an operator-facing section: services.md covered SSO, metrics and the collector, and said nothing about where the swarm's logs land, how to read them, or how long they are kept. Adds one, answering the questions an operator arrives with — the name it serves on, that Grafana's Explore is the browser for it, retention, and that it binds loopback with the gateway vhost as the only way in.
157 lines
7.7 KiB
Markdown
157 lines
7.7 KiB
Markdown
# Swarm-wide services
|
|
|
|
Some things exist once per **swarm**, not once per hive: the forge, the
|
|
matrix homeserver, the SSO provider, the CA. Two options say where the
|
|
optional ones live, and everything else derives:
|
|
|
|
```nix
|
|
services.hyperhive.enableAllLocalDefaults = true; # everything on this box
|
|
# or, for a dedicated services host with hives elsewhere:
|
|
services.hyperhive.swarm.enableRequiredServices = true;
|
|
```
|
|
|
|
`enableAllLocalDefaults` is the all-on-one-box switch: it defaults both
|
|
`swarm.enableRequiredServices` (the shared services run here) and
|
|
`swarm.ca.autoConfigure` (the swarm CA is generated here). Each derived
|
|
toggle can still be set on its own, so "all local except X" needs no
|
|
further option.
|
|
|
|
**Both default to off**, and that is deliberate: a host cannot tell
|
|
whether it is meant to be the swarm's service host, so this is an
|
|
operator saying so rather than something inferred. With them off, a hive
|
|
is a *client* of those services — it configures how to reach them and
|
|
runs none of them.
|
|
|
|
The forge is the exception, and not because it is per-hive: it is
|
|
swarm-wide but **not optional**, being the canonical store for the meta
|
|
flake and every agent's config repo, so it deploys with hyperhive itself
|
|
and has no `enable` to derive from anything.
|
|
|
|
### SSO (authelia)
|
|
|
|
One authelia per swarm, in a `swarm-authelia` container, at
|
|
`auth.<swarm-domain>`. Operator and agents are both subjects of the same
|
|
provider, differentiated by roles and claims rather than by mechanism —
|
|
there is one IdP and one auth path.
|
|
|
|
- **`swarm.authelia.enable`** — run the container here. Defaults from
|
|
`swarm.enableRequiredServices`.
|
|
- **`swarm.authelia.url`** — where clients are sent to authenticate.
|
|
Present on **every** hive, defaulting to this host's own instance only
|
|
when this module is the thing running it; otherwise `null`, and a hive
|
|
joining someone else's swarm sets it explicitly. Null means "no SSO
|
|
configured", and consumers say so rather than guessing an address.
|
|
|
|
The users database is written by swarm-controller, not by hand: agents
|
|
are created and destroyed continuously, so the subject set is dynamic.
|
|
This module only guarantees the file exists and parses, so authelia
|
|
starts with nobody in it rather than failing to start — a provider with
|
|
no subjects yet is the correct state before anything has provisioned
|
|
them. Session and storage keys are generated in the container on first
|
|
boot and never rotated automatically; replacing one invalidates data
|
|
already written (sessions, the encrypted store), so that is an operator
|
|
action.
|
|
|
|
Storage is local sqlite and the notifier writes to a file. Both are
|
|
small-deployment choices, and the scope is the justification: redis
|
|
buys shared session state across replicas and there is one instance;
|
|
SMTP exists to mail humans, and provisioning here is programmatic.
|
|
|
|
See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC
|
|
relying-party flow, and [`secrets.md`](secrets.md) for where each of
|
|
authelia's keys is generated and read.
|
|
|
|
### Metrics (VictoriaMetrics + Grafana)
|
|
|
|
The swarm's telemetry lands in one VictoriaMetrics and is read through
|
|
one Grafana, in two containers at `metrics.<swarm-domain>` and
|
|
`grafana.<swarm-domain>`. Two containers rather than one so Grafana can
|
|
be restarted or broken without taking the time-series database with it.
|
|
|
|
Both follow `swarm.enableRequiredServices` like authelia and matrix, so
|
|
the swarm's service host gets them with everything else. They derive
|
|
together: a store with no UI is unreadable and a UI with no store is
|
|
empty. To run one without the other, set it directly:
|
|
|
|
```nix
|
|
services.hyperhive.swarm.victoriametrics.enable = true;
|
|
services.hyperhive.swarm.grafana.enable = false;
|
|
```
|
|
|
|
⚠️ **This starts a database that grows for as long as the swarm runs.**
|
|
See `retentionPeriod` below before leaving it at its default.
|
|
|
|
| Option | When you'd touch it |
|
|
|---|---|
|
|
| `swarm.victoriametrics.retentionPeriod` | Default `5y`. Lower it once you have measured how fast this swarm actually fills a disk — the default is deliberately generous because too-short silently discards history you cannot get back. |
|
|
| `swarm.grafana.oidc.role` | Default `Admin` for everyone who logs in. Lower to `Viewer`/`Editor` if the swarm grows operators who should not be able to reconfigure Grafana. |
|
|
| `swarm.grafana.datasourceUrl` | Only if you front VictoriaMetrics with something else. It defaults to the store on this host, which is the only thing it can reach. |
|
|
|
|
**Logging in.** Grafana is behind swarm SSO, so the accounts are the
|
|
authelia ones — there is no separate Grafana password, and the local
|
|
login form is switched off whenever SSO is configured. If you enable
|
|
Grafana on a host with no authelia, the form stays on and Grafana's
|
|
default `admin`/`admin` applies; change it before exposing that host.
|
|
|
|
**Where the data comes from.** The swarm's OTEL collector, below.
|
|
|
|
Neither container is reachable except through the gateway: both bind
|
|
loopback, and VictoriaMetrics' write endpoint takes no credential, so
|
|
the collector is the only intended writer.
|
|
|
|
### Logs (VictoriaLogs)
|
|
|
|
The swarm's service containers ship their journals to one VictoriaLogs at
|
|
`logs.<swarm-domain>`, behind the same SSO as everything else. It follows
|
|
`swarm.enableRequiredServices` too, and the collector below is what writes
|
|
to it.
|
|
|
|
**Reading them.** Open Grafana, pick **Explore**, and choose the
|
|
`VictoriaLogs` datasource — it is provisioned for you. Grafana's *Logs
|
|
Drilldown* app is deliberately not installed: it only supports Loki, and
|
|
no setting here changes that, so Explore is the log browser for this
|
|
swarm.
|
|
|
|
| Option | When you'd touch it |
|
|
|---|---|
|
|
| `swarm.victorialogs.retentionPeriod` | Default `30d`, far shorter than the metrics store's — logs are bulkier per unit of value and are usually read within days of being written. Raise it if you need to answer questions about last quarter. |
|
|
| `swarm.victorialogs.domain` | Only to rename it. |
|
|
| `swarm.victorialogs.port` | Only if something else on the services host already claims `9428`. |
|
|
|
|
Like the metrics store, it binds loopback and takes no credential of its
|
|
own: the gateway vhost is the only way in, and the collector is the only
|
|
intended writer.
|
|
|
|
### Telemetry collector (OTEL)
|
|
|
|
The swarm's collector receives from every hive's own collector and is the
|
|
only process that decides where telemetry goes: it writes the store above
|
|
and exports to `otel.endpoint`, doing both when both are configured. It
|
|
also holds the upstream credential, which is why no hive and no agent
|
|
needs one.
|
|
|
|
It follows `swarm.enableRequiredServices` like the services above, in a
|
|
`swarm-otel` container. Its `swarm.otel.port` defaults to `4319` rather
|
|
than OTLP's usual `4318`, which the hive tier already uses — swarm
|
|
containers share the host's network namespace, so two collectors on one
|
|
port is a coin toss at runtime rather than an error at build time.
|
|
|
|
Every hive's own collector reaches this one by its gateway name,
|
|
`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
|
|
by-domain-through-the-gateway shape every other swarm service uses, not a
|
|
loopback URL an operator has to redirect. There is nothing to set on a hive
|
|
that does not run the swarm's services; the name resolves through the
|
|
gateway either way.
|
|
|
|
| Option | When you'd touch it |
|
|
|---|---|
|
|
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
|
|
| `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
|
|
|
|
With neither `otel.endpoint` nor the store enabled, this collector is
|
|
refused at eval — a tier that receives samples and drops them looks
|
|
healthy while losing data.
|
|
|
|
Agent-side configuration, and what a hive's own collector does, are in
|
|
[`../observability.md`](../observability.md).
|
|
|