Dashboard 2 of the set the operator asked for: how Claude is being used
rather than which agent is using it, so the axes are model, effort, token
type and query source. The only deliberate overlap with the agents page
is the cost/token headline.
Every panel was run against the live store before this landed. Three
panels were dropped rather than shipped, because their label has exactly
one live value today and a page of single-bar charts is the same silent
failure as an empty one.
Turn count and turn length are PROXIES and say so in their descriptions:
nothing exports turn stats, so a session record stands in for a turn,
which holds because each turn runs a new claude process.
Grafana served no dashboards: only datasources were provisioned, while
the module header already claimed dashboards were. This ships the agents
dashboard as a file provider and makes that sentence true.
The datasource uid is bound once and substituted into the dashboard at
build time. Committing the literal would make the dashboard a second
speller of a name the datasource already owns, and the drift failure is
silent -- panels render empty rather than erroring.
The shipped copy drops the `DS` datasource variable: it exists so an
operator can pick a store on manual import, and a provisioned dashboard
must not ask.
Setting declarativePlugins to any list -- the empty one included -- also
sets upstream's plugins.preinstall_disabled, which nixpkgs defaults to
declarativePlugins != null. Preinstall is what fetches the Drilldown apps
on first start, so claiming the plugin directory removed them without
touching anything a person had installed.
The comment above that line reasoned about the plugin path and the update
check and never about preinstall, and its closing claim -- nothing is
taken over on a hive with no plugins -- was true only of plugins a PERSON
installs. Upstream installs some itself, and those are the ones that went
missing.
Listing it as a package is better than the preinstall it replaces: store
pinned, no runtime fetch from grafana.com, survives a state reset. Its
Logs/Traces/Profiles siblings stay off -- they front Loki, Tempo and
Pyroscope, none of which this swarm runs.
Plugin management is server-admin scoped, and on an SSO hive nobody holds that
role: auto_assign_org_role grants an org role, and the built-in local admin
that does hold server admin cannot log in because the login form is disabled
whenever SSO is configured. Two individually-correct decisions leaving no path
to the plugin UI at all.
Declarative is the way through rather than a workaround for it -- plugins land
in the store and in git, survive a rebuild and a state reset, and the container
needs no runtime egress to grafana.com.
mkIf rather than passing the list through: upstream's default is null, while an
empty list is a real value pointing the plugin path at an empty store dir, so a
hive that sets nothing must keep seeing null.
Both hand-rolled the same concat with wantedBy + before and no requires,
so a failed assembly let the consumer start against a missing file and
trust nothing at all -- which fails every outbound TLS call while the unit
looks healthy. The helper puts requires on the consumer and verifies the
assembled bundle before moving it into place.
Removes each module's now-dead useSelfSigned binding: its only use was the
mkIf on the deleted block, and nix does not warn about an unused let
binding. Also corrects two comments the deletion orphaned -- one described
the removed path binding, the other pointed at a bundle service that no
longer exists.
A swarm service container shares the host netns and force-disables
resolvconf, so it inherits the /etc/resolv.conf nixos-containers copies in
at start (cp --remove-destination, host-side preStart, once per start) and
nothing ever refreshes it. That makes the container's resolver a snapshot of
the host's file at its boot instant.
When that snapshot is wrong the container can never recover, and the symptom
appears arbitrarily far from the cause: swarm-nats-auth cannot resolve
authelia's name, so it denies every auth-callout request and the queue
refuses every client with an authorization violation.
Give each of the four swarm containers a oneshot that writes the resolver
file itself, from the bridge IP, ordered before that container's first DNS
consumer. The shape is the one every agent container already uses.
networking.nameservers cannot do this: resolvconf is its only consumer and
these containers disable it, so setting it renders no file while still
evaluating cleanly. A static environment.etc entry cannot either -- it would
have to survive etc activation landing on the regular file the host already
copied there, which no eval can show.
Per the operator's call on #3372: rather than moving Grafana off the
forge's 3000 to another number, take it off TCP entirely.
The collision was possible because every swarm service container shares
the host's network namespace, which makes a port a swarm-wide resource
two modules can each claim believing it free — Grafana took upstream's
3000, so does the forge, and `grafana.<swarm-domain>` served the forge
with no bind error and nothing in any log. A socket has a path, and a
path collision is a build-time conflict rather than a runtime coin toss.
Three parts, none of which works alone:
- `protocol = "socket"` with `socket_gid` = nginx's static gid 60.
- Grafana joins that gid inside the container. A non-root process may
only chgrp to a group it belongs to, and a container has its own user
database — without the membership Grafana starts, the chown fails, and
the socket is simply unreachable.
- The socket dir is created host-side by tmpfiles and bind-mounted in. A
container's /run is an nspawn tmpfs, so it is not visible from the host
at /var/lib/nixos-containers/<name>/run; the same shape the per-agent
web.sock already uses.
The `port` option is gone rather than deprecated — nothing can set it to
a colliding value if it does not exist.
nixpkgs dropped the default for `services.grafana.settings.security.secret_key`
and asserts on null, so the module I merged an hour ago fails the build on any
host that enables it. That is a broken deploy, not a warning.
Generated in-container like authelia's own keys and for the same reason:
nothing outside the container ever reads it. Generated ONCE and kept — the
key signs Grafana's stored datasource secrets, and rotating it does not
re-encrypt what it already encrypted, so a fresh key per boot would leave
Grafana unable to read its own database. Delivered as `$__file{}`, so it
never enters the store.
The gate missed this because a container is a separate NixOS evaluation with
its own assertions: forcing the host's `config.assertions` never touched the
container's. `state/eval-3265.sh` now forces
`containers.swarm-grafana.config.assertions` and reports any that fail.
Second half of the metrics pair: a `swarm-grafana` container beside the
VictoriaMetrics store, provisioned with it as the default datasource and
fronted by the gateway on its own swarm-sibling name.
Behind swarm SSO, per the operator's call on #3265. The authelia client
and Grafana's callback URL both derive from `domain`, so the exact-match
string authelia checks cannot drift from the one Grafana sends. The
minted secret is delivered host-side (both container trees are only
addressable there) and reaches Grafana as a `$__file{}` reference rather
than a value, so it never enters the store.
The login form is disabled whenever SSO is configured: Grafana ships an
`admin`/`admin` account and this vhost is on the public gateway.
