Turns on gitea_issues_by_label and gitea_issues_by_repository, gated on
behindGateway exactly like the metrics endpoint itself -- they are only
reachable through the authenticated location that condition creates.
These are off by default upstream because they are the only metrics in
this section whose series count grows with the CONTENT of the forge
rather than with its feature set, so the question is cardinality.
Measured rather than assumed: ~19 distinct labels and ~29 repos on this
instance, well under 100 series, against a store that took +2234 from
nine host scrapers.
Still no TOKEN. Forgejo can guard the endpoint with its own static
bearer, but the swarm authenticates the scraper at the gateway, and a
second per-service credential would be the one that stops getting
rotated -- the reasoning already recorded above ENABLED applies
unchanged to the breakdowns.
The token file is 0600 root-owned and the precondition ran as the unit's
dynamic gitea-runner user, so it could not read it -- and it did not fail
closed. `[ -s ]` succeeds on a stat alone, grep then died with EACCES,
and `! grep` turned that error into true, so the gate exited 0 on a file
it had never read. It has reported 'a real token is present' without
looking for its entire existence, which is why the runner kept retrying
instead of holding still and saying why.
Prefix the step with + so it runs with full privileges. This grants the
runner nothing new: tokenFile becomes EnvironmentFile=, which systemd
already reads as root before dropping privileges, so the payload never
passes through an unprivileged reader either way -- the + only lets the
gate observe what systemd observes.
Also assert readability separately and loudly, rather than folding it
into the placeholder test: an unreadable file and a real token produced
the same verdict, so 'I could not look' was indistinguishable from 'I
looked and it is fine'.
Two more per-service boards, cut from the withdrawn combined services
board: the log store (4 panels) and the metrics store (5).
The combined board was one page carrying panels for four different
services; the seams were already there, so this is a re-cut rather than
new authoring. Each board's subject is one service, and the services
page that remains will carry only panels whose subject is the *set*.
Both were imported by the operator before being committed here, so the
queries are known to draw against the live stores rather than only
parsing.
Nothing in the tree rendered a `[credential "<forge>"]` entry for an
agent, so every agent's `~/.gitconfig` accumulated one by hand, per
generation of forge address. Append-only, none ever removed, and after
the domain move the live one absent entirely:
[credential "http://forge.<old-hive>"]
[credential "http://localhost:3000"]
The absence of a writer is the defect. A value interpolated at eval time
follows a rename; a value captured into a mutable home file does not.
hive-c0re's own gitconfig already derives its scope from
`swarm.forge.domain` and moved correctly for exactly that reason.
What made it expensive to diagnose is that it does not present as a
credential problem. `git fetch` against a stale remote still succeeds --
the old name redirects and a public read needs no auth -- so the break
surfaces only at the first authenticated push, long after the move, as
`could not read Username for '<new host>'`. That names a host the agent
was never configured for, which reads like DNS or TLS.
Same class as the CI runner keeping its registered address, one tier
down.
The shape is `github.nix`'s, unchanged: a small credential helper that
reads the token from the agent's state file at invocation, with the
token PATH baked in rather than the value, because claude's Bash tool
runs in a minimal env that never sources /etc/set-environment.
`environment.etc."gitconfig"` is already bound by github.nix; the two
merge rather than collide because the option is `lines`. Verified by
eval with both modules defining it before this was written -- a silent
last-wins there would drop one integration's credentials and look
exactly like this bug again.
The pending-bytes panel described itself as the leading indicator for
the slow-consumer stat 'beside it', and then sat alone in its own row
with eighteen grid units of empty space. A description that tells the
reader to look beside it, where nothing is, is worse than no
description.
Slow consumers and pending bytes are now adjacent in the top row, and
the two JetStream panels share the second. No gaps.
Split out of the combined swarm-services board, which mara declined as
multiple dashboards in one trench coat. Its 17 panels partition exactly
by subject -- 3 services / 5 metrics store / 5 queue / 4 log store, with
no panel belonging to two -- so this is a re-cut of verified panels, not
new authoring.
Seven panels: connections, subscriptions, slow consumers and JetStream
API errors as stats, messages over the range, plus pending bytes (the
leading indicator for a slow consumer) and JetStream store size against
its configured limit (the only capacity question the queue has).
Connections is red at zero, unlike every other count here: three things
hold a connection at all times, so an empty count means the queue is
refusing everyone -- and a callout denial reaches a client as a timeout
rather than an error, which makes this often the first visible sign.
They differ by one letter and a grep for the shorter name matches both.
The summary is what ships; the per-process scraper is deliberately absent,
and a reader (human or grep) cannot tell from the diff alone.
Nothing measured the host itself. hive-c0re emits per-container metrics
and each service exports its own, so the store could answer 'is this
container using more memory than last week' and could not answer 'is the
machine under them out of memory' -- the question every one of those
readings is implicitly relative to.
Hive tier rather than swarm: a host is owned by the hive running on it,
and the swarm tier runs one collector for the whole swarm, so it would
have to reach other machines to see theirs.
Every scraper the pinned receiver offers except process, which adds a
series set per running process -- unbounded on a hive host, where the
nine shipped are a fixed handful. Coverage and cardinality are different
axes and only the second can hurt the store.
Refs #3649
The collector's client is registered for authelia.bearer.authz, but the
prometheus scrape asked for no scopes, so every token came back carrying
none and authelia refused it at introspection with 'the requested scope
is invalid, unknown, or malformed'. Forgejo metrics were never scraped.
The rule was already stated one field below, for the audience: registered
is not requested. The two travel together, and a config read cannot see
the one that is missing -- every inspection of the rendered config came
back correct, because what was present was correct.
NODE_EXTRA_CA_CERTS beside it is Node's and additive. gitea-runner is Go,
whose trust store is replacing: it reads SSL_CERT_FILE and trusts only
what that names. So the container trusted the hive CA for its Node
actions and never for the daemon itself.
The daemon then fails startup with x509 unknown authority the moment it
reaches a TLS endpoint, crash-loops on Restart=on-failure, and no job is
picked up -- every pull request just sits at Waiting to run.
Not claimed as the whole cause of that outage: which address the daemon
holds is a separate question. The trust gap is real under every
explanation, and adding a CA to one unit cannot break a working path.
journaldUnits named "authelia"; upstream derives the unit from the instance
name, so the running unit is authelia-swarm and that entry matched nothing.
The receiver's units list is an allowlist, so a name matching no journal
entry is silently absent -- it reads as a service with nothing to say rather
than as a misconfiguration, and it excluded the busiest source on the box.
Use the unitName binding the rest of this module already uses, so the
allowlist cannot drift from the unit again.
The two callout assertions guarded the halves the server verifies with.
The responder needs the other halves, and nothing related them: a config
supplying only the public keys satisfies both, renders a syntactically
valid auth_callout block, and defines no responder unit.
Callout with no responder is the fail-closed state, so that queue refuses
every client — and a NATS denial arrives as a timeout, so the symptom is
every consumer hanging with nothing logged anywhere.
The build-time config check does run in this case and passes, because the
config is valid; what is missing is a unit, and the absence of a unit is
not an event.
Both services carried an `sso.enable` defaulting to false, so a swarm's
own forge and homeserver shipped with their identity provider switched
off unless an operator remembered two lines. Grafana never had the
toggle and is the shape the other two now match.
Behaves as if the setting were true: `ssoLocal` loses one conjunct, the
three assertions become unconditional, and the login source and
identity_provider render always.
The option is removed rather than defaulted, so a config that turned SSO
OFF fails where that line is instead of silently gaining a login
provider on the next rebuild.
The journald receiver was configured with a directory and no filter, so
the swarm's log store received every unit on the host that runs the
collector. On a hive whose services live on a workstation that includes
the operator's desktop session, in a store every swarm operator can read.
The receiver has no system-only switch and its `matches` field is an
allowlist too, so what the swarm collects has to be stated rather than
excluded. Each service module names its own units: a service that is not
running contributes nothing, and one added later arrives declared.
An empty list is fail-open — the receiver renders no filter at all and
reads everything — so it is asserted against.
grafana-lokiexplore-app cannot query VictoriaLogs. Its volume views call
/loki/api/v1/index/volume, which VictoriaLogs does not implement and
answers "unsupported path requested", and upstream's position is that
the datasource plugin must provide drilldown support. No configuration
here changes that, so the app only ever offered a menu entry that looks
like a broken feature rather than an absent one.
The option's own description already ruled its siblings out for fronting
backends this swarm does not run. That premise expired when the swarm
gained a log store, which is what made adding this one look reasonable —
so the reason is rewritten rather than the list alone: Logs Drilldown is
excluded on compatibility, not on absence.
Explore with the VictoriaLogs datasource is the log browser, and
docs/swarm/services.md now says so where an operator looks for it.
Grafana was the one swarm service that exports Prometheus metrics and had no
scrape target, so the swarm's metrics UI was invisible to the metrics it
displays.
It serves on a unix socket and claims no TCP port, which is deliberate and
stays that way; a prometheus target has to be a host:port, so nginx re-serves
the one endpoint a scraper needs on loopback. An exact-match location, not a
prefix: widening it would re-serve the whole UI without authorization.
The metrics section is now stated explicitly rather than inherited, because a
scrape target depends on it and a changed upstream default would take the
endpoint away while nginx kept answering.
Adds services.hyperhive.swarm.victorialogs.domain and a gateway vhost
gated by the same auth_request check against authelia that swarm-uis
own vhost uses (same swarmAuthRequest shape, copied not shared - see
the file top comment for why). The stores own listener stays
loopback-only and unauthenticated exactly as before; the collector
still writes to it directly, never through this vhost, so this only
adds a new authenticated read path.
Registers the new domain in swarm.nixs serviceDomains so the
swarm-services sub-CA issues for it (a missed entry silently falls
back to the hive leaf, which cannot cover a name under a different
apex - swarm.nixs own comment on that list documents the incident
this caused before).
Verified with a throwaway module-eval (same technique as the flake
module-eval check): vhost only exists when victorialogs.enable is
set, forceSSL/no addSSL, both locations present, domain correctly
registered/absent from serviceDomains, links entry present.
Grafana could reach the swarm's metrics and not its logs, so the store that
landed with the collector pipeline had no reader.
Adds the VictoriaLogs datasource plugin and the Logs Drilldown app, and
provisions the datasource beside the metrics one. The drilldown matters as much
as the connection: an unfamiliar log stream is explorable without writing a
LogsQL query first, which is the difference between a store you can query and
one you can use.
nixpkgs' collector unit already sets SupplementaryGroups to systemd-journal
unconditionally, with a comment saying why. Systemd list options concatenate,
so this module's copy rendered ["systemd-journal" "systemd-journal"] and made
this a second owner of a fact upstream may later change.
The bind mount stays, since that half is genuinely ours.
The logs half was gated on the swarm's log store being enabled, so turning
that store off stopped collection entirely rather than leaving the upstream
export. Logs now fan out exactly as metrics do: the store when it runs, the
operator's upstream when one is configured, both when both.
The receiver, the journal mount and the group grant follow whether there is
anywhere to send logs, not whether the local store exists. Tying the mount to
the store instead would render a receiver that can read nothing.
Deferred until the collector had a logs pipeline writing to it: a store
nothing writes to starts, answers queries and returns nothing, so the first
person to look concludes there were no logs rather than that nothing was
collecting them.
The journald receiver leaves the OTLP body empty and carries the entry as a
map of journal fields, so VictoriaLogs had no message to index and wrote a
placeholder into _msg on every record. Ingest returned 200, every field was
present, and a plain search for a line sitting in the store found nothing.
_msg_field names the field that holds the text. _stream_fields is the
difference between one stream for the whole host and one per unit per
machine; both are set by journald itself and both are low-cardinality.
A journald receiver reading the host's journal directory, a logs pipeline
stamped with the swarm tier's own resource processor, and an otlphttp
exporter pointed at VictoriaLogs. All four parts are conditional on the log
store being enabled, so a swarm without one renders exactly as before.
The host's directory is enough to see every container: nspawn links a
non-ephemeral container's journal guest-side, so the files live on the host
under the container's machine-id, and journalctl descends into those
subdirectories. Measured, along with the fixed systemd-journal gid that makes
the group grant meaningful across the bind mount.
An agent can verify that a unit was launched and never that it is
working: container journals are not reachable, so a diagnosis stops at
the first broken component -- which is precisely the component whose own
instrument is least likely to be legible. This is the store half of
collecting logs centrally so the question becomes answerable.
Mirrors swarm-victoriametrics deliberately: same container shape, same
loopback pin, same self-scrape arrangement. Two differences, both
intentional.
No gateway vhost. VictoriaLogs' ingest and query endpoints carry no
authentication of their own, exactly like the metrics store's -- and the
metrics store IS published under a resolvable name, which is an open
question rather than a settled design. Publishing this one the same way
would repeat that before the first instance is decided.
Retention defaults to 30d against the metrics store's 5y. Logs are
orders of magnitude larger per unit of time and their value decays much
faster: a log line answers what happened during an incident, a metric
answers whether this is worse than last quarter.
Not wired into enableRequiredServices yet -- that lands with the
collector pipeline, so we do not start a store nothing writes to.
prometheus-nats-exporter has never served a metric. Upstream's module
renders `-addr … -port … ${extraFlags} ${url}` and defaults extraFlags
to the empty list, but the binary refuses to start without at least one
collector: it exits 1 with "no Collectors specified". So the unit logged
Started, the process was gone milliseconds later, and every scrape was
refused -- up=0 continuously, scrape_duration 0.6ms, zero samples.
Verified by running the exact argv both ways: without a collector it
exits 1, with -varz it stays up and logs the listener. A bogus flag is
rejected with exit 2, so the check measures acceptance rather than
tolerance.
varz is the server itself, connz makes a client that will not stay
connected visible, and jsz covers JetStream, which this swarm uses for
the status KV. The rest describe a clustered deployment we do not have.
An empty list is not a neutral default when the program requires a flag,
and nothing about it is visible to evaluation -- which is why the gate
now reads the rendered ExecStart rather than only asserting enable.
The swarm-services board was mostly authelia, so it becomes its own page
and is trimmed out of that one (mara, #3494). Eleven panels: uptime,
authentications and failures, authorization decisions, requests and
verdicts by status code, request and OIDC latency, and the three Go
process signals.
OIDC latency gets its own panel rather than being folded into the
general one because authelia keeps a separate duration family for it,
and every machine-to-machine credential in this swarm is minted through
those endpoints -- averaging the two produces a number describing
neither.
Counters are counted over the dashboard range rather than rated. At this
volume a rate window contains no requests, so rate() returns zero and
draws a flat line, which is indistinguishable from a broken query; the
quantile panels are worse, because a quantile over all-zero buckets is
NaN and renders empty rather than zero.
The option's description claimed declaring an entry from the service's
own module put 'the scraper and the target on the same host by
construction rather than by luck'. It does not. It constrains where the
target is; nothing in it places the collector, and the two enable flags
are co-located by a shared lib.mkDefault rather than by construction.
Split across hosts, a target is silently never scraped — the service's
host declares an entry no local collector reads, the collector's host
never enabled the service. No error surfaces, and no assertion can
catch it: separate hosts are separate evaluations with no shared
context, so the doc telling the truth is the only mechanism there is.
The same paragraph already warned co-location was not a guarantee, four
lines below the sentence claiming it was; a reader arriving for
permission stopped at the permission. This one did.
swarm-nats carries the concrete caveat for its own contribution.
NATS has no Prometheus format of its own. It serves a JSON monitoring
endpoint, and prometheus-nats-exporter translates that — so this is two
changes in order, not one: without the monitoring endpoint the exporter
starts cleanly and scrapes nothing, which is the inert-config shape the
scrape work exists to avoid.
Both listeners are loopback and the exporter is the monitoring
endpoint's only intended reader: it is unauthenticated and /connz names
every connected client, so the address it binds is the whole access
control.
The scrape target is declared here rather than in the collector's
module, gated on a collector existing to read it — an entry exists only
where the service that named it runs.
mara, on review: "i thought we just swap a css file via nginx
config?" -- right instinct. The package-copy overlay (cp -r + install)
only made sense for hive-c0re's servedFrontend, which backs multiple
serve points (dashboard root + every per-agent gateway route) from one
swapped tree. swarm-ui has exactly one location serving cfg.package, so
an = /static/colors.css exact-match override -- the same idiom every
other single-path override on this vhost already uses (/api/whoami,
/api/docs) -- replaces the whole derivation with one location block.
Verified with the same throwaway nixosSystem eval as the previous
commit: unthemed case has no colors.css location and the / root is
cfg.package unchanged.
mara: "swarm dash is in my own colors, but tab colors on swarm ui are
the default catpucchin one." Root cause: hive-c0re/theme.nix's stylix
overlay only ever wrote a themed colors.css onto the dashboard/agent
frontend subtrees -- swarm-ui, served from its own separate package,
was never in scope.
Extracts the stylix-detection + colors.css-generation logic (previously
inline in theme.nix) into a shared nix/host-modules/stylix-theme.nix,
imported by both theme.nix and swarm-ui.nix -- one source instead of a
second copy that has to agree by inspection. swarm-ui.nix gains its own
themedPackage overlay (same shape as theme.nix's themedFrontend: copy
the package, overwrite static/colors.css) and serves that instead of
cfg.package directly when stylix is active; a clean passthrough
otherwise.
Verified with a throwaway nixosSystem eval (this repo's own flake
checks do not exercise gateway-module wiring) confirming the unthemed
path resolves cfg.package unchanged.
The scrape config names a client_secret_file; this is what puts a file
there. A host oneshot copies authelia's minted secret between the two
container trees — a copy and not a bind mount, because the secret does
not exist until authelia's first boot and nixos-container refuses to
start on a missing bind source, which on a fresh swarm is a permanent
stall presenting as broken metrics.
The collector runs under DynamicUser and the prometheus receiver opens
client_secret_file itself, as that user, so there is no uid to hand the
file to. LoadCredential reads it as root before the sandbox exists and
re-exposes it under a path that does not depend on which uid the unit
got; the scrape config points there. Both spellings derive from one
binding, since a mismatch is a file the collector cannot open and
nothing but a runtime 401 would say so.
The published targets now render as prometheus scrape jobs, so the
option's name is true: the collector reaches them by name over https,
using prometheus-native oauth2 with the audience set to the target's own
url. Registered is not the same as requested — a client that does not
ask for an audience gets a token with an empty one however complete its
registration looks.
The url is split into scheme, target and metrics_path rather than asked
for three times: two spellings of one address is a mismatch waiting to
happen, and the failure it produces is a valid token refused at the
target. A malformed or non-https url is an assertion rather than a null
dereference from inside the renderer.
`collectorAudiences` was a list of URLs services contributed so the
collector's client could be registered for them. Slice C needs the same
URLs as scrape jobs, and a job needs a name the audience list has no
room for — so services would have contributed to two options that must
agree.
They now contribute `publishedScrapeTargets` once, as `<job> = "<url>"`,
and the client's audiences derive from it. The drift that would have
needed maintaining is gone, and its failure mode was the quiet one: a
target whose audience was forgotten authenticates against nothing and
reads as a broken scrape rather than a missing registration.
Adds an assertion for the one collision the module system cannot catch.
Two definitions of the same key within one option are already refused
(measured); across the two scrape options nothing arbitrates, and both
entries would render into a single scrape_configs list under one
job_name.
Review found that nothing stopped `bearerAuthz = true` on an interactive
client. `renderClient` reads the flag only in the machine branch, so the
scope is never emitted: the client authenticates, is authorised for
nothing, and the build is green. `kind` defaults to `interactive`, so it
is reached by forgetting a field rather than by writing a wrong one —
this module's own failure mode one level up.
The other half of the review asked to relax the method assertion to
accept `null`, on the strength of the option's doc calling `null`
authelia's default. Measured instead: under `authelia.bearer.authz`
authelia refuses the omission outright, so the assertion was right and
the DOC was wrong. The doc now carries the exception, and the assertion
message says null is refused rather than leaving a reader to infer it.
The forge's `/metrics` is published behind the gateway and denied to
everyone, waiting on a client to allow. This is that client.
An audience is a URL: authelia validates a bearer token against the
address being requested, and a client may only request an audience it is
registered for, so registration is the authorisation. The URLs are owned
by the services that publish them while audiences attach to one client,
so services contribute to a list and this module builds the single entry
— the `gateway.localNames` split, forced here by client definitions
concatenating rather than merging into a shared entry.
The access-control rule asks the client list whether the collector is
registered rather than re-deriving the conditions that register it. The
two drifting is not a build failure: authelia refuses a rule naming an
unknown client in its startup validator, so SSO fails to restart.
A scraper reaches a service published behind the gateway by presenting an
access token to authelia's authz endpoint, which requires the client to
carry authelia.bearer.authz. Nothing could express that: scopes are
derived from kind, and a machine client rendered an empty list.
A named capability rather than a free-form scopes list, for the reason
the derivation exists — authelia refuses some scope/grant combinations
outright, openid with client_credentials among them, and a list makes
those expressible again.
The two assertions carry their weight: authelia checks the same
obligations, but in its preStart validator, so a violation builds and
deploys cleanly and then fails to restart with swarm SSO attached to it.
The collector names components `<kind>/<owner>` — a hive name for the
per-hive pipelines, the literal `swarm` for the swarm tier's own. Both
land in one attrset via `//`, so a hive named `swarm` replaced the swarm
tier's parts and lost its own: its receiver kept accepting pushes into a
pipeline that routed nowhere, and its samples lost the `hive` stamp that
makes attribution unforgeable. Zero failed assertions.
The reserved name is now bound once and interpolated at each swarm-tier
use, so the guard checks the same string the config emits rather than a
copy of it. A second swarm-tier pipeline joins the list and inherits the
check without touching the assertion.
There are two otel options one word apart: swarm.otel is the swarm's
collector, hyperhive.otel is the per-hive tier that ships upstream and
never reads scrapeTargets. The gate went through a let binding declared
800 lines from its use, so which one it referred to was not visible where
it mattered — mara had to ask.
Gating the wrong one is not a build error. It is a target that is either
always declared or never declared, and both look like working config.
The comment claimed the option's rule keeps scraper and target on one host
by construction. It does not. Both services default from
enableRequiredServices via mkDefault, which is an invitation to override
rather than a guarantee, so co-location is a property of the auto-deployed
topology and not of the module.
Gating the target on the collector's own enable makes the loopback address
honest: a host running authelia without a collector no longer declares a
target nothing can read. That absence was the part worth fixing, because it
is silent — no error, no metrics, nothing in a log to notice.
This does not make authelia scrapeable from another host. That needs the
endpoint published under a name with a certificate and an audience, which
is separate work; the option's docs now say so where someone splitting the
two would read it.
The endpoint was off, so nothing reported on the swarm's own SSO. Enabling
it alone would have added no data — the scraper that reads it only landed
with the swarm-tier prometheus receiver.
Loopback only, like the main listener and for a stronger reason: this
endpoint authenticates nothing and reports request volumes and outcomes for
every login on the swarm.
metricsPort is an option rather than a literal because every swarm container
shares the host netns, so two services picking the same port do not conflict
at build time — one loses at runtime with nothing in any log. 9959 is
upstream's default and is unclaimed across nix/.
Two review findings, folded together.
mara: a swarm integrated auto deployed forge always has metrics, so the
toggle is gone. The endpoint follows behindGateway instead, which is the
swarm-integrated shape and the condition the protected location lives
under. Serving it without that location would put it on a listener
openFirewall can expose with nothing in front.
argus: /metrics matched no access_control rule, so default_policy
one_factor governed it. That is any authenticated subject, which today
means any operator and tomorrow any agent. My audience argument covered
the Bearer path only; the same endpoint also accepts CookieSession, and a
cookie carries no audience at all, so the audience was never what stood
between a browser session and this data.
The rule is deny rather than a client-scoped allow because the collector's
client does not exist yet. Authelia refuses a subject naming an
unregistered client, and does so in a preStart validator rather than at
build time, so naming one early yields a green nixos-rebuild and dead
swarm SSO on the next restart. Denying until the client is registered
makes publishing the endpoint safe on its own; registering it is a
one-line change from deny to that allow.
Rule order is load-bearing: authelia takes the first match.
The gateway authenticates scrapers so services do not each grow a static
bearer of their own, but authelia's auth_request endpoint ran its default
strategies, which are cookie-only. A scraper's OAuth2 access token was
refused no matter how it was minted.
CookieSession is listed explicitly because authn_strategies replaces the
defaults rather than extending them. Omitting it evaluates, renders and
starts, and silently ends every operator session on the swarm UI, which
uses this same endpoint.
Unconditional rather than keyed to whichever service is scraped today:
this makes a scheme available, not an authorisation. Authelia refuses a
token carrying no audience for the requested URL and only issues a client
audiences it is registered for, so nothing passes until a client is
registered against a specific URL.
Forgejo can serve prometheus metrics but nothing turned them on, and
turning them on alone would have published them: forgejo serves
`/metrics` on its normal listener and the gateway vhost proxies `/` to
that listener, so the existing catch-all would have carried the endpoint
to anyone. The option is therefore one switch for both halves, and the
`= /metrics` location is an exact match so it outranks that prefix.
Authentication is the gateway's rather than forgejo's own `[metrics]
TOKEN`: a scraper presents an audience-scoped authelia token which nginx
checks via auth_request, so the swarm keeps one identity system instead
of gaining a static bearer per service.
The subrequest deliberately omits the `error_page 401 =302` that
swarm-ui uses. That redirect sends a browser to a login page; a scraper
would follow it and parse HTML as metrics.
Adds an "/api/whoami" same-origin nginx proxy to authelia's own
GET /api/user/info (session-cookie authenticated, no swarm-controller
code needed) and a new UserMenu header component: a generated initials
avatar (first letter of display name, coloured from the same seven
base16 chromatic slots the nav accent already cycles through) opening a
popover with the signed-in name, a link to authelia settings, and log
out — both reusing the existing "Authelia" entry from GET /api/links
rather than a second source of the domain.
Per mara's call on the open avatar-mechanism question: initials now,
a real uploaded photo (authelia's settings UI implies pics are
settable) is an explicit future item, not blocking this.
Dashboard 2 of the set the operator asked for: how Claude is being used
rather than which agent is using it, so the axes are model, effort, token
type and query source. The only deliberate overlap with the agents page
is the cost/token headline.
Every panel was run against the live store before this landed. Three
panels were dropped rather than shipped, because their label has exactly
one live value today and a page of single-bar charts is the same silent
failure as an empty one.
Turn count and turn length are PROXIES and say so in their descriptions:
nothing exports turn stats, so a session record stands in for a turn,
which holds because each turn runs a new claude process.
The operator asked for this when the dashboard was first reviewed and it was
deferred, not declined: the container metric family carried no hive label, so
selecting a hive emptied every container panel while the "All" default hid the
problem completely -- a `.*` matcher matches series where the label is absent.
That family now carries hive and swarm, verified against the live store rather
than inferred from the fix having merged: a hive selection returns the same 7
agents as the All default, at every window out to 168h, with a nonexistent
hive returning zero.
The agent list chains off the hive selection, so picking a hive narrows the
agent dropdown rather than leaving entries in it that resolve to nothing.
Grafana served no dashboards: only datasources were provisioned, while
the module header already claimed dashboards were. This ships the agents
dashboard as a file provider and makes that sentence true.
The datasource uid is bound once and substituted into the dashboard at
build time. Committing the literal would make the dashboard a second
speller of a name the datasource already owns, and the drift failure is
silent -- panels render empty rather than erroring.
The shipped copy drops the `DS` datasource variable: it exists so an
operator can pick a store on manual import, and a provisioned dashboard
must not ask.
