Two more per-service boards, cut from the withdrawn combined services
board: the log store (4 panels) and the metrics store (5).
The combined board was one page carrying panels for four different
services; the seams were already there, so this is a re-cut rather than
new authoring. Each board's subject is one service, and the services
page that remains will carry only panels whose subject is the *set*.
Both were imported by the operator before being committed here, so the
queries are known to draw against the live stores rather than only
parsing.
Split out of the combined swarm-services board, which mara declined as
multiple dashboards in one trench coat. Its 17 panels partition exactly
by subject -- 3 services / 5 metrics store / 5 queue / 4 log store, with
no panel belonging to two -- so this is a re-cut of verified panels, not
new authoring.
Seven panels: connections, subscriptions, slow consumers and JetStream
API errors as stats, messages over the range, plus pending bytes (the
leading indicator for a slow consumer) and JetStream store size against
its configured limit (the only capacity question the queue has).
Connections is red at zero, unlike every other count here: three things
hold a connection at all times, so an empty count means the queue is
refusing everyone -- and a callout denial reaches a client as a timeout
rather than an error, which makes this often the first visible sign.
The journald receiver was configured with a directory and no filter, so
the swarm's log store received every unit on the host that runs the
collector. On a hive whose services live on a workstation that includes
the operator's desktop session, in a store every swarm operator can read.
The receiver has no system-only switch and its `matches` field is an
allowlist too, so what the swarm collects has to be stated rather than
excluded. Each service module names its own units: a service that is not
running contributes nothing, and one added later arrives declared.
An empty list is fail-open — the receiver renders no filter at all and
reads everything — so it is asserted against.
grafana-lokiexplore-app cannot query VictoriaLogs. Its volume views call
/loki/api/v1/index/volume, which VictoriaLogs does not implement and
answers "unsupported path requested", and upstream's position is that
the datasource plugin must provide drilldown support. No configuration
here changes that, so the app only ever offered a menu entry that looks
like a broken feature rather than an absent one.
The option's own description already ruled its siblings out for fronting
backends this swarm does not run. That premise expired when the swarm
gained a log store, which is what made adding this one look reasonable —
so the reason is rewritten rather than the list alone: Logs Drilldown is
excluded on compatibility, not on absence.
Explore with the VictoriaLogs datasource is the log browser, and
docs/swarm/services.md now says so where an operator looks for it.
Grafana was the one swarm service that exports Prometheus metrics and had no
scrape target, so the swarm's metrics UI was invisible to the metrics it
displays.
It serves on a unix socket and claims no TCP port, which is deliberate and
stays that way; a prometheus target has to be a host:port, so nginx re-serves
the one endpoint a scraper needs on loopback. An exact-match location, not a
prefix: widening it would re-serve the whole UI without authorization.
The metrics section is now stated explicitly rather than inherited, because a
scrape target depends on it and a changed upstream default would take the
endpoint away while nginx kept answering.
Grafana could reach the swarm's metrics and not its logs, so the store that
landed with the collector pipeline had no reader.
Adds the VictoriaLogs datasource plugin and the Logs Drilldown app, and
provisions the datasource beside the metrics one. The drilldown matters as much
as the connection: an unfamiliar log stream is explorable without writing a
LogsQL query first, which is the difference between a store you can query and
one you can use.
The swarm-services board was mostly authelia, so it becomes its own page
and is trimmed out of that one (mara, #3494). Eleven panels: uptime,
authentications and failures, authorization decisions, requests and
verdicts by status code, request and OIDC latency, and the three Go
process signals.
OIDC latency gets its own panel rather than being folded into the
general one because authelia keeps a separate duration family for it,
and every machine-to-machine credential in this swarm is minted through
those endpoints -- averaging the two produces a number describing
neither.
Counters are counted over the dashboard range rather than rated. At this
volume a rate window contains no requests, so rate() returns zero and
draws a flat line, which is indistinguishable from a broken query; the
quantile panels are worse, because a quantile over all-zero buckets is
NaN and renders empty rather than zero.
Dashboard 2 of the set the operator asked for: how Claude is being used
rather than which agent is using it, so the axes are model, effort, token
type and query source. The only deliberate overlap with the agents page
is the cost/token headline.
Every panel was run against the live store before this landed. Three
panels were dropped rather than shipped, because their label has exactly
one live value today and a page of single-bar charts is the same silent
failure as an empty one.
Turn count and turn length are PROXIES and say so in their descriptions:
nothing exports turn stats, so a session record stands in for a turn,
which holds because each turn runs a new claude process.
Grafana served no dashboards: only datasources were provisioned, while
the module header already claimed dashboards were. This ships the agents
dashboard as a file provider and makes that sentence true.
The datasource uid is bound once and substituted into the dashboard at
build time. Committing the literal would make the dashboard a second
speller of a name the datasource already owns, and the drift failure is
silent -- panels render empty rather than erroring.
The shipped copy drops the `DS` datasource variable: it exists so an
operator can pick a store on manual import, and a provisioned dashboard
must not ask.
Setting declarativePlugins to any list -- the empty one included -- also
sets upstream's plugins.preinstall_disabled, which nixpkgs defaults to
declarativePlugins != null. Preinstall is what fetches the Drilldown apps
on first start, so claiming the plugin directory removed them without
touching anything a person had installed.
The comment above that line reasoned about the plugin path and the update
check and never about preinstall, and its closing claim -- nothing is
taken over on a hive with no plugins -- was true only of plugins a PERSON
installs. Upstream installs some itself, and those are the ones that went
missing.
Listing it as a package is better than the preinstall it replaces: store
pinned, no runtime fetch from grafana.com, survives a state reset. Its
Logs/Traces/Profiles siblings stay off -- they front Loki, Tempo and
Pyroscope, none of which this swarm runs.
Plugin management is server-admin scoped, and on an SSO hive nobody holds that
role: auto_assign_org_role grants an org role, and the built-in local admin
that does hold server admin cannot log in because the login form is disabled
whenever SSO is configured. Two individually-correct decisions leaving no path
to the plugin UI at all.
Declarative is the way through rather than a workaround for it -- plugins land
in the store and in git, survive a rebuild and a state reset, and the container
needs no runtime egress to grafana.com.
mkIf rather than passing the list through: upstream's default is null, while an
empty list is a real value pointing the plugin path at an empty store dir, so a
hive that sets nothing must keep seeing null.
Both hand-rolled the same concat with wantedBy + before and no requires,
so a failed assembly let the consumer start against a missing file and
trust nothing at all -- which fails every outbound TLS call while the unit
looks healthy. The helper puts requires on the consumer and verifies the
assembled bundle before moving it into place.
Removes each module's now-dead useSelfSigned binding: its only use was the
mkIf on the deleted block, and nix does not warn about an unused let
binding. Also corrects two comments the deletion orphaned -- one described
the removed path binding, the other pointed at a bundle service that no
longer exists.
A swarm service container shares the host netns and force-disables
resolvconf, so it inherits the /etc/resolv.conf nixos-containers copies in
at start (cp --remove-destination, host-side preStart, once per start) and
nothing ever refreshes it. That makes the container's resolver a snapshot of
the host's file at its boot instant.
When that snapshot is wrong the container can never recover, and the symptom
appears arbitrarily far from the cause: swarm-nats-auth cannot resolve
authelia's name, so it denies every auth-callout request and the queue
refuses every client with an authorization violation.
Give each of the four swarm containers a oneshot that writes the resolver
file itself, from the bridge IP, ordered before that container's first DNS
consumer. The shape is the one every agent container already uses.
networking.nameservers cannot do this: resolvconf is its only consumer and
these containers disable it, so setting it renders no file while still
evaluating cleanly. A static environment.etc entry cannot either -- it would
have to survive etc activation landing on the regular file the host already
copied there, which no eval can show.
Per the operator's call on #3372: rather than moving Grafana off the
forge's 3000 to another number, take it off TCP entirely.
The collision was possible because every swarm service container shares
the host's network namespace, which makes a port a swarm-wide resource
two modules can each claim believing it free — Grafana took upstream's
3000, so does the forge, and `grafana.<swarm-domain>` served the forge
with no bind error and nothing in any log. A socket has a path, and a
path collision is a build-time conflict rather than a runtime coin toss.
Three parts, none of which works alone:
- `protocol = "socket"` with `socket_gid` = nginx's static gid 60.
- Grafana joins that gid inside the container. A non-root process may
only chgrp to a group it belongs to, and a container has its own user
database — without the membership Grafana starts, the chown fails, and
the socket is simply unreachable.
- The socket dir is created host-side by tmpfiles and bind-mounted in. A
container's /run is an nspawn tmpfs, so it is not visible from the host
at /var/lib/nixos-containers/<name>/run; the same shape the per-agent
web.sock already uses.
The `port` option is gone rather than deprecated — nothing can set it to
a colliding value if it does not exist.
nixpkgs dropped the default for `services.grafana.settings.security.secret_key`
and asserts on null, so the module I merged an hour ago fails the build on any
host that enables it. That is a broken deploy, not a warning.
Generated in-container like authelia's own keys and for the same reason:
nothing outside the container ever reads it. Generated ONCE and kept — the
key signs Grafana's stored datasource secrets, and rotating it does not
re-encrypt what it already encrypted, so a fresh key per boot would leave
Grafana unable to read its own database. Delivered as `$__file{}`, so it
never enters the store.
The gate missed this because a container is a separate NixOS evaluation with
its own assertions: forcing the host's `config.assertions` never touched the
container's. `state/eval-3265.sh` now forces
`containers.swarm-grafana.config.assertions` and reports any that fail.
Second half of the metrics pair: a `swarm-grafana` container beside the
VictoriaMetrics store, provisioned with it as the default datasource and
fronted by the gateway on its own swarm-sibling name.
Behind swarm SSO, per the operator's call on #3265. The authelia client
and Grafana's callback URL both derive from `domain`, so the exact-match
string authelia checks cannot drift from the one Grafana sends. The
minted secret is delivered host-side (both container trees are only
addressable there) and reaches Grafana as a `$__file{}` reference rather
than a value, so it never enters the store.
The login form is disabled whenever SSO is configured: Grafana ships an
`admin`/`admin` account and this vhost is on the public gateway.
