- nix/checks.nix: 'hyperhive#3685' in a source comment trips the
tracker-tag lint (prose-only, per hive-rules.md) — argus caught it on
review. Reworded without the ref.
- frontend/npm-deps-hash: the earlier scaffold commit
(be18f460) added preact/typescript to agent/package.json, which
regenerated package-lock.json but never re-derived this hash in
lockstep — nix flake check's frontend derivation was failing on a
hash mismatch. Regenerated via
'nix run nixpkgs#prefetch-npm-deps -- frontend/package-lock.json'.
- StatusChips.tsx: drop the stale 'useAgentState lands in a later
commit' comment now that Root.tsx already wires it.
- nix/checks.nix: add agent-typecheck, same shape as swarm-ui-typecheck,
so a real TS error in the agent package fails CI instead of just
esbuild-stripping silently.
Two findings on one panel, both from mara.
Form: 24 label series drawn as overlapping timeseries lines is
unreadable by construction. The panel is a current-count ranking, not a
time evolution, so it becomes a horizontal bar gauge on an instant
sort_desc query -- identity on the axis, magnitude in the bar, and a
fixed colour because colour was carrying no information that the axis
was not already carrying.
Title: the metric counts open AND closed issues. Measured, not assumed
-- the largest single label reads 1619 while gitea_issues_open reads
117, and no label can outnumber the set it is a subset of. The panel had
been asserting the opposite since it was written.
mara on the PR: "many hint texts are incorrect, most hives dont work on
hyperhive itself". Four panel descriptions asserted local practice as if
it were general:
Releases "expected to sit at zero on a hive that ships from
main" -- a claim about how THIS project releases
Automation surface "every hive registering the same webhook has bitten
this forge before" -- a war story from this
deployment, meaningless on any other
Organizations "near-constant by design" -- assumes our org layout
Issues open/closed "usually a bulk triage, not a productive afternoon"
-- a claim about our working rhythm
Each now describes what the metric IS and what a CHANGE means, and says
outright where the panel cannot know: whether zero releases is normal
depends on how a hive's projects ship, and a vertical step in closed is
some batch operation that this panel cannot distinguish from a burst of
real work.
The board title is NOT an instance of this and is unchanged -- all seven
dashboards use the 'hyperhive - <thing>' product namespace, checked
rather than assumed.
The forge restarted (new forgejo PID at 12:36), so it re-read [metrics]
and gitea_issues_by_label now exists -- 24 series carrying real label
names, read back out of the store rather than inferred from the metric
name appearing in a label-values list.
The panel shipped deliberately empty with EMPTY UNTIL THE FORGE RESTARTS
in its title and a matching noValue, because an empty graph and an
undeployed metric look identical. That warning is now false, and a stale
warning is worse than none: it tells a reader the panel cannot have data
at the moment it does.
Keeps the operational half rather than deleting it -- the restart
requirement is a fact about the NEXT metric someone enables, so it moves
into the description as a note and into noValue, where it fires only when
the series really is missing.
Adds the fourth per-service board. Unlike its three siblings it answers
"what does the forge CONTAIN", not "is the forge healthy" -- forgejo's
exporter publishes a census of stored objects and no request, error or
latency series at all.
That distinction is the board's own description rather than something a
reader has to infer, because a fourth per-service board that silently
answers a different question than the other three is worse than no board:
someone checks it during an incident and reads a flat line as calm.
Panels: repositories, users, orgs, issues open/closed, releases, the
open-vs-closed pair over time, repos-and-users together (they move in
lockstep when an agent is created, so divergence is the signal),
discussion volume, and the webhook/hooktask pair.
One panel ships deliberately EMPTY: issues-by-label. The metric is
enabled in the forge's config but forgejo reads its [metrics] section at
startup, so the series does not exist until that process restarts -- and
a config merge alone does not restart it. The panel title says so, the
description says how to tell "not deployed" from "no labelled issues",
and `noValue` says it again in the graph itself. An empty panel with no
explanation is the failure this board is trying not to be.
Gated: every panel's metric confirmed to exist in the store, with a
bogus name checked absent in the same run so the check can fail, plus a
known-positive control against the label-values query itself -- a
download that silently returned nothing would otherwise report every
metric missing. Gate kept as verify-3494-forge-board.sh.
The comment-block lint (added in 79dc8ca6) now trips on this block —
genuinely pre-existing, unrelated to that change, just newly caught.
Per the lint's own suggested remedy: relocated the full per-directive
reasoning plus both footguns (session-cache keying, the Host-header
clobber that can recurse a subrequest into itself) to a new
"Dialing another vhost by name" section in docs/gateway.md, and left
a short why + pointer comment in the source. No behavior change.
verifiedProxyTo (43ae164d) verified TLS but left Host to nixpkgs'
recommendedProxySettings, which sets Host to the CALLING vhost, not
the target. Since every consumer resolves back to this same gateway,
nginx picks the vhost to answer by Host header (not by the SNI
proxy_ssl_name already sends) — so every auth subrequest looped back
into its own vhost's auth_request, recursing until nginx's subrequest
depth limit turned it into a 500, on every domain gated by SSO.
Pin Host (and the rest of the header set nixpkgs' recommended include
would otherwise still be the one to set) inside verifiedProxyTo, and
set recommendedProxySettings = false on each of the four call sites so
nixpkgs' own copy — appended after a location's extraConfig — can't
clobber it back.
One list, in nix/reserved-names.nix, handed to everything that needs it
as HIVE_RESERVED_NAMES. Keeping it current becomes a config change
rather than a rebuild, and hive names and agent names -- one namespace
going forward -- are checked against the same file: swarm-otel.nix's
hand-written reservedOwners is gone.
Whitespace-separated rather than JSON, deliberately, unlike the
structured env vars beside it. Every entry is an Ident ([a-z0-9-]), so
whitespace cannot occur inside a name and the encoding is provably
lossless; JSON would mean either a parser dependency in a crate whose
purpose is to have none, or a copy of the parse in every consumer.
An UNSET variable is not "nothing is reserved". Both creation sites log
an error and return a warning saying the check did not run, so a
misconfigured deployment says so instead of silently accepting every
name. A blank value folds into unset: nix always renders a non-empty
list, so present-but-empty is a rendering fault, not a declaration.
Two guards whose subject moved out of their own file now assert their
own case is still in it, because a guard that can be retired by an edit
elsewhere is not a guard:
- swarm-otel.nix asserts reserved-names.nix still contains its
swarmTierName.
- hive-sh4re's sentinel drift test PANICS when the variable is missing
rather than skipping -- a drift test that quietly does nothing still
reports green. checks.nix and devshell.nix both export it so CI and a
local cargo test agree. Verified as a pair: with the variable set, 8
tests pass; with it unset, exactly the 4 drift tests fail and the
unrelated ones still pass.
Consumers reached authelia at `127.0.0.1:<port>`, which encoded a
co-location nobody agreed to: the gateway and authelia are not required
to share a host, so the literal is a requirement stated only by being
unwriteable any other way. Moving them to the name is the point of the
issue.
But a name over https is only half of "https and auth". nginx's
`proxy_ssl_verify` is OFF by default and there was no `proxy_ssl_*`
anywhere in the tree, so the obvious repoint would have produced an
encrypted, unauthenticated hop -- which works, and keeps working,
against any certificate at all.
Adds `gateway.lib.verifiedProxyTo <name>` next to the rest of the vhost
kit, so the convention has one definition rather than a copy in each
consuming module, and repoints the four call sites through it.
Each directive was checked against a real nginx with the opposite arm
run as a control:
- the CA *bundle* (root + intermediate) is accepted -- worth checking,
since `hive-ca-trust.nix` warns off consumers that read only one
certificate, and nginx is not one of those
- verification checks the chain: an unrelated CA fails
- and the HOSTNAME: a wrong `proxy_ssl_name` fails even with a good
chain. Chain-only would accept any cert this CA ever signed, which
for an internal CA is every service on the hive
- with verify off, the wrong CA passes -- so the failures above come
from verification, not from the connection
Bind addresses are untouched. This changes what consumers dial, not what
anything listens on.
The upstream gitea-actions-runner unit uses DynamicUser, and systemd
mounts a dynamic unit's state directory noexec. Measured from a job's
own /proc/self/mountinfo on the live runner, with /tmp as the control:
the state dir carries noexec, /tmp does not, and a script written and
chmod +x'd in the workspace fails execve with EACCES while the same
script in /tmp runs.
This was invisible for as long as every workflow compiled inside the
nix sandbox and executed nothing from the workspace. The first job that
built a binary into the runner's own target dir -- an instrumented
coverage run -- died on its first build script.
ExecPaths= re-mounts the state dir executable. Both spellings are
listed with the ignore-if-absent prefix because ExecPaths resolves
against the host root, where the dynamic-user layout makes
/var/lib/gitea-runner a symlink into private/.
/etc/gitconfig shipped `helper = git-credential-hive-forge`. git prepends
`git-credential-` to any helper value that is not an absolute path, so
that resolves to `git-credential-git-credential-hive-forge`, which does
not exist -- no helper runs at all. The sibling github.nix has always used
the short form.
Measured rather than read off the docs, with the arms isolated from the
personal ~/.gitconfig:
helper = git-credential-hive-forge -> 0 credentials, and git prints
"'credential-git-credential-hive-forge' is not a git command"
helper = hive-forge -> 1 credential, clean stderr
The reason this survived: every long-lived agent has a personal
~/.gitconfig naming the helper by ABSOLUTE path, which git accepts, so
pushes keep working and the stderr line reads as noise. The system config
is masked exactly where someone would notice it and bites a fresh agent
that has no such file.
The authelia vhost intercepts upstream errors and serves a friendly
"SSO unavailable" page. The `=` form of `error_page` takes its status
from the redirected location, and that location serves a file -- so the
page is returned as **200**.
That is right for a human typing the URL and wrong for every machine
caller, all of which reach authelia through this same vhost by name:
- `/api/authz/auth-request` -- nginx `auth_request` treats any 2xx as
success, so a down authelia means access GRANTED
- `/api/oidc/introspection` -- a token check answering 200
- `/api/oidc/token`, `/.well-known/openid-configuration` -- clients
parsing an HTML error page as their JSON document
Routes `/api/` and `/.well-known/` without the interception. A longer
prefix wins over `/`, and the intercept directives live inside the `/`
location rather than at server level, so they do not reach the new ones.
Split by AUDIENCE rather than by an enumerated path list: a human still
gets the page, and every machine caller -- including the login page's own
XHR, and any endpoint added later -- gets the real status.
Measured against a real nginx with a dead upstream, both arms: machine
paths return 502 where they returned 200+HTML, a subrequest through the
new prefix denies (matching a direct port dial) where through `/` it
served the protected content, and the browser control confirms the
friendly page survives. URI preservation checked separately against a
live echo upstream -- `proxy_pass` with no URI part passes the full
original path.
Adds `cargo llvm-cov` to the devshell and a coverage workflow that runs
only when someone asks for it.
Manual dispatch rather than nightly or per-PR, per the discussion on the
issue: a coverage run needs its own instrumented build and cannot reuse
the normal test artifacts, so it roughly doubles a test job. "Do we have
glaring holes" is a question someone asks occasionally, not a gate every
PR pays for, and not a number worth spending farm time on every night
whether or not anyone reads it. Manual dispatch costs nothing until the
answer is wanted.
Its own workflow file rather than a job in ci.yml: that file already has
a `workflow_dispatch` trigger so `hive-forge ci-rerun` can retrigger
without an empty commit, and a trigger there fires EVERY job in the
file -- a coverage job added there would run on every pull request.
No threshold and no --fail-under-lines. A coverage gate mostly teaches
people to write assertion-free tests that execute lines; the report is
the deliverable and the number is for a human to read.
The devshell needs LLVM_COV / LLVM_PROFDATA set explicitly: cargo
llvm-cov expects rustup's `llvm-tools-preview` beside the toolchain and
nixpkgs has no such component, so without them it aborts with "failed to
find llvm-tools-preview" -- which reads like a missing install rather
than a path the shell has to name.
Verified by running it: `cargo llvm-cov --package hive-types` produces a
real report (3 tests, 85.44% regions). Both the variable names and the
package were wrong on the first attempt and only running it said so --
the names take no `_PATH` suffix, and the binaries are in
`llvmPackages.llvm`, not `llvmPackages.bintools`, which is the linker
wrapper and ships neither.
Host metrics arrived carrying `hive` and no host attribute, so "which
host is out of memory" was answerable only as "which hive" -- true only
while a hive is one machine, which is the framing the swarm work exists
to end.
Adds the collector's own `resourcedetection` processor to the metrics
pipeline, emitting both `host.name` and `host.id`.
`resourcedetection` rather than a value picked in nix: this is the
canonical OTel mechanism for host identity, and choosing our own would
answer a question the tool already answers, differently from every other
deployment. A resource attribute is captured into the stored series, so
a private naming scheme is expensive in the way that lasts.
Both attributes, deliberately: `host.name` is readable and what a panel
groups by, but splits the series silently on a rename with nothing
linking old to new; `host.id` survives that and is unreadable alone.
Hosts get renamed and repurposed exactly when services move off one box.
The processor sits in the pipeline, so it applies to every receiver --
agent-pushed OTLP and scraped targets included, not just `hostmetrics`.
Keys verified against otelcol-contrib 0.151.0 with two deliberate
mutations rejected in the same run (a bogus resource attribute and a
bogus detector key), so "accepted" is distinguishable from a validator
that ignores what it does not recognise.
Turns on gitea_issues_by_label and gitea_issues_by_repository, gated on
behindGateway exactly like the metrics endpoint itself -- they are only
reachable through the authenticated location that condition creates.
These are off by default upstream because they are the only metrics in
this section whose series count grows with the CONTENT of the forge
rather than with its feature set, so the question is cardinality.
Measured rather than assumed: ~19 distinct labels and ~29 repos on this
instance, well under 100 series, against a store that took +2234 from
nine host scrapers.
Still no TOKEN. Forgejo can guard the endpoint with its own static
bearer, but the swarm authenticates the scraper at the gateway, and a
second per-service credential would be the one that stops getting
rotated -- the reasoning already recorded above ENABLED applies
unchanged to the breakdowns.
The token file is 0600 root-owned and the precondition ran as the unit's
dynamic gitea-runner user, so it could not read it -- and it did not fail
closed. `[ -s ]` succeeds on a stat alone, grep then died with EACCES,
and `! grep` turned that error into true, so the gate exited 0 on a file
it had never read. It has reported 'a real token is present' without
looking for its entire existence, which is why the runner kept retrying
instead of holding still and saying why.
Prefix the step with + so it runs with full privileges. This grants the
runner nothing new: tokenFile becomes EnvironmentFile=, which systemd
already reads as root before dropping privileges, so the payload never
passes through an unprivileged reader either way -- the + only lets the
gate observe what systemd observes.
Also assert readability separately and loudly, rather than folding it
into the placeholder test: an unreadable file and a real token produced
the same verdict, so 'I could not look' was indistinguishable from 'I
looked and it is fine'.
Two more per-service boards, cut from the withdrawn combined services
board: the log store (4 panels) and the metrics store (5).
The combined board was one page carrying panels for four different
services; the seams were already there, so this is a re-cut rather than
new authoring. Each board's subject is one service, and the services
page that remains will carry only panels whose subject is the *set*.
Both were imported by the operator before being committed here, so the
queries are known to draw against the live stores rather than only
parsing.
Nothing in the tree rendered a `[credential "<forge>"]` entry for an
agent, so every agent's `~/.gitconfig` accumulated one by hand, per
generation of forge address. Append-only, none ever removed, and after
the domain move the live one absent entirely:
[credential "http://forge.<old-hive>"]
[credential "http://localhost:3000"]
The absence of a writer is the defect. A value interpolated at eval time
follows a rename; a value captured into a mutable home file does not.
hive-c0re's own gitconfig already derives its scope from
`swarm.forge.domain` and moved correctly for exactly that reason.
What made it expensive to diagnose is that it does not present as a
credential problem. `git fetch` against a stale remote still succeeds --
the old name redirects and a public read needs no auth -- so the break
surfaces only at the first authenticated push, long after the move, as
`could not read Username for '<new host>'`. That names a host the agent
was never configured for, which reads like DNS or TLS.
Same class as the CI runner keeping its registered address, one tier
down.
The shape is `github.nix`'s, unchanged: a small credential helper that
reads the token from the agent's state file at invocation, with the
token PATH baked in rather than the value, because claude's Bash tool
runs in a minimal env that never sources /etc/set-environment.
`environment.etc."gitconfig"` is already bound by github.nix; the two
merge rather than collide because the option is `lines`. Verified by
eval with both modules defining it before this was written -- a silent
last-wins there would drop one integration's credentials and look
exactly like this bug again.
The pending-bytes panel described itself as the leading indicator for
the slow-consumer stat 'beside it', and then sat alone in its own row
with eighteen grid units of empty space. A description that tells the
reader to look beside it, where nothing is, is worse than no
description.
Slow consumers and pending bytes are now adjacent in the top row, and
the two JetStream panels share the second. No gaps.
Split out of the combined swarm-services board, which mara declined as
multiple dashboards in one trench coat. Its 17 panels partition exactly
by subject -- 3 services / 5 metrics store / 5 queue / 4 log store, with
no panel belonging to two -- so this is a re-cut of verified panels, not
new authoring.
Seven panels: connections, subscriptions, slow consumers and JetStream
API errors as stats, messages over the range, plus pending bytes (the
leading indicator for a slow consumer) and JetStream store size against
its configured limit (the only capacity question the queue has).
Connections is red at zero, unlike every other count here: three things
hold a connection at all times, so an empty count means the queue is
refusing everyone -- and a callout denial reaches a client as a timeout
rather than an error, which makes this often the first visible sign.
They differ by one letter and a grep for the shorter name matches both.
The summary is what ships; the per-process scraper is deliberately absent,
and a reader (human or grep) cannot tell from the diff alone.
Nothing measured the host itself. hive-c0re emits per-container metrics
and each service exports its own, so the store could answer 'is this
container using more memory than last week' and could not answer 'is the
machine under them out of memory' -- the question every one of those
readings is implicitly relative to.
Hive tier rather than swarm: a host is owned by the hive running on it,
and the swarm tier runs one collector for the whole swarm, so it would
have to reach other machines to see theirs.
Every scraper the pinned receiver offers except process, which adds a
series set per running process -- unbounded on a hive host, where the
nine shipped are a fixed handful. Coverage and cardinality are different
axes and only the second can hurt the store.
Refs #3649
The collector's client is registered for authelia.bearer.authz, but the
prometheus scrape asked for no scopes, so every token came back carrying
none and authelia refused it at introspection with 'the requested scope
is invalid, unknown, or malformed'. Forgejo metrics were never scraped.
The rule was already stated one field below, for the audience: registered
is not requested. The two travel together, and a config read cannot see
the one that is missing -- every inspection of the rendered config came
back correct, because what was present was correct.
NODE_EXTRA_CA_CERTS beside it is Node's and additive. gitea-runner is Go,
whose trust store is replacing: it reads SSL_CERT_FILE and trusts only
what that names. So the container trusted the hive CA for its Node
actions and never for the daemon itself.
The daemon then fails startup with x509 unknown authority the moment it
reaches a TLS endpoint, crash-loops on Restart=on-failure, and no job is
picked up -- every pull request just sits at Waiting to run.
Not claimed as the whole cause of that outage: which address the daemon
holds is a separate question. The trust gap is real under every
explanation, and adding a CA to one unit cannot break a working path.
journaldUnits named "authelia"; upstream derives the unit from the instance
name, so the running unit is authelia-swarm and that entry matched nothing.
The receiver's units list is an allowlist, so a name matching no journal
entry is silently absent -- it reads as a service with nothing to say rather
than as a misconfiguration, and it excluded the busiest source on the box.
Use the unitName binding the rest of this module already uses, so the
allowlist cannot drift from the unit again.
The two callout assertions guarded the halves the server verifies with.
The responder needs the other halves, and nothing related them: a config
supplying only the public keys satisfies both, renders a syntactically
valid auth_callout block, and defines no responder unit.
Callout with no responder is the fail-closed state, so that queue refuses
every client — and a NATS denial arrives as a timeout, so the symptom is
every consumer hanging with nothing logged anywhere.
The build-time config check does run in this case and passes, because the
config is valid; what is missing is a unit, and the absence of a unit is
not an event.
Both services carried an `sso.enable` defaulting to false, so a swarm's
own forge and homeserver shipped with their identity provider switched
off unless an operator remembered two lines. Grafana never had the
toggle and is the shape the other two now match.
Behaves as if the setting were true: `ssoLocal` loses one conjunct, the
three assertions become unconditional, and the login source and
identity_provider render always.
The option is removed rather than defaulted, so a config that turned SSO
OFF fails where that line is instead of silently gaining a login
provider on the next rebuild.
The journald receiver was configured with a directory and no filter, so
the swarm's log store received every unit on the host that runs the
collector. On a hive whose services live on a workstation that includes
the operator's desktop session, in a store every swarm operator can read.
The receiver has no system-only switch and its `matches` field is an
allowlist too, so what the swarm collects has to be stated rather than
excluded. Each service module names its own units: a service that is not
running contributes nothing, and one added later arrives declared.
An empty list is fail-open — the receiver renders no filter at all and
reads everything — so it is asserted against.
grafana-lokiexplore-app cannot query VictoriaLogs. Its volume views call
/loki/api/v1/index/volume, which VictoriaLogs does not implement and
answers "unsupported path requested", and upstream's position is that
the datasource plugin must provide drilldown support. No configuration
here changes that, so the app only ever offered a menu entry that looks
like a broken feature rather than an absent one.
The option's own description already ruled its siblings out for fronting
backends this swarm does not run. That premise expired when the swarm
gained a log store, which is what made adding this one look reasonable —
so the reason is rewritten rather than the list alone: Logs Drilldown is
excluded on compatibility, not on absence.
Explore with the VictoriaLogs datasource is the log browser, and
docs/swarm/services.md now says so where an operator looks for it.
Grafana was the one swarm service that exports Prometheus metrics and had no
scrape target, so the swarm's metrics UI was invisible to the metrics it
displays.
It serves on a unix socket and claims no TCP port, which is deliberate and
stays that way; a prometheus target has to be a host:port, so nginx re-serves
the one endpoint a scraper needs on loopback. An exact-match location, not a
prefix: widening it would re-serve the whole UI without authorization.
The metrics section is now stated explicitly rather than inherited, because a
scrape target depends on it and a changed upstream default would take the
endpoint away while nginx kept answering.
Adds services.hyperhive.swarm.victorialogs.domain and a gateway vhost
gated by the same auth_request check against authelia that swarm-uis
own vhost uses (same swarmAuthRequest shape, copied not shared - see
the file top comment for why). The stores own listener stays
loopback-only and unauthenticated exactly as before; the collector
still writes to it directly, never through this vhost, so this only
adds a new authenticated read path.
Registers the new domain in swarm.nixs serviceDomains so the
swarm-services sub-CA issues for it (a missed entry silently falls
back to the hive leaf, which cannot cover a name under a different
apex - swarm.nixs own comment on that list documents the incident
this caused before).
Verified with a throwaway module-eval (same technique as the flake
module-eval check): vhost only exists when victorialogs.enable is
set, forceSSL/no addSSL, both locations present, domain correctly
registered/absent from serviceDomains, links entry present.
Grafana could reach the swarm's metrics and not its logs, so the store that
landed with the collector pipeline had no reader.
Adds the VictoriaLogs datasource plugin and the Logs Drilldown app, and
provisions the datasource beside the metrics one. The drilldown matters as much
as the connection: an unfamiliar log stream is explorable without writing a
LogsQL query first, which is the difference between a store you can query and
one you can use.
nixpkgs' collector unit already sets SupplementaryGroups to systemd-journal
unconditionally, with a comment saying why. Systemd list options concatenate,
so this module's copy rendered ["systemd-journal" "systemd-journal"] and made
this a second owner of a fact upstream may later change.
The bind mount stays, since that half is genuinely ours.
The logs half was gated on the swarm's log store being enabled, so turning
that store off stopped collection entirely rather than leaving the upstream
export. Logs now fan out exactly as metrics do: the store when it runs, the
operator's upstream when one is configured, both when both.
The receiver, the journal mount and the group grant follow whether there is
anywhere to send logs, not whether the local store exists. Tying the mount to
the store instead would render a receiver that can read nothing.
Deferred until the collector had a logs pipeline writing to it: a store
nothing writes to starts, answers queries and returns nothing, so the first
person to look concludes there were no logs rather than that nothing was
collecting them.
The journald receiver leaves the OTLP body empty and carries the entry as a
map of journal fields, so VictoriaLogs had no message to index and wrote a
placeholder into _msg on every record. Ingest returned 200, every field was
present, and a plain search for a line sitting in the store found nothing.
_msg_field names the field that holds the text. _stream_fields is the
difference between one stream for the whole host and one per unit per
machine; both are set by journald itself and both are low-cardinality.
A journald receiver reading the host's journal directory, a logs pipeline
stamped with the swarm tier's own resource processor, and an otlphttp
exporter pointed at VictoriaLogs. All four parts are conditional on the log
store being enabled, so a swarm without one renders exactly as before.
The host's directory is enough to see every container: nspawn links a
non-ephemeral container's journal guest-side, so the files live on the host
under the container's machine-id, and journalctl descends into those
subdirectories. Measured, along with the fixed systemd-journal gid that makes
the group grant meaningful across the bind mount.
An agent can verify that a unit was launched and never that it is
working: container journals are not reachable, so a diagnosis stops at
the first broken component -- which is precisely the component whose own
instrument is least likely to be legible. This is the store half of
collecting logs centrally so the question becomes answerable.
Mirrors swarm-victoriametrics deliberately: same container shape, same
loopback pin, same self-scrape arrangement. Two differences, both
intentional.
No gateway vhost. VictoriaLogs' ingest and query endpoints carry no
authentication of their own, exactly like the metrics store's -- and the
metrics store IS published under a resolvable name, which is an open
question rather than a settled design. Publishing this one the same way
would repeat that before the first instance is decided.
Retention defaults to 30d against the metrics store's 5y. Logs are
orders of magnitude larger per unit of time and their value decays much
faster: a log line answers what happened during an incident, a metric
answers whether this is worse than last quarter.
Not wired into enableRequiredServices yet -- that lands with the
collector pipeline, so we do not start a store nothing writes to.
prometheus-nats-exporter has never served a metric. Upstream's module
renders `-addr … -port … ${extraFlags} ${url}` and defaults extraFlags
to the empty list, but the binary refuses to start without at least one
collector: it exits 1 with "no Collectors specified". So the unit logged
Started, the process was gone milliseconds later, and every scrape was
refused -- up=0 continuously, scrape_duration 0.6ms, zero samples.
Verified by running the exact argv both ways: without a collector it
exits 1, with -varz it stays up and logs the listener. A bogus flag is
rejected with exit 2, so the check measures acceptance rather than
tolerance.
varz is the server itself, connz makes a client that will not stay
connected visible, and jsz covers JetStream, which this swarm uses for
the status KV. The rest describe a clustered deployment we do not have.
An empty list is not a neutral default when the program requires a flag,
and nothing about it is visible to evaluation -- which is why the gate
now reads the rendered ExecStart rather than only asserting enable.
The swarm-services board was mostly authelia, so it becomes its own page
and is trimmed out of that one (mara, #3494). Eleven panels: uptime,
authentications and failures, authorization decisions, requests and
verdicts by status code, request and OIDC latency, and the three Go
process signals.
OIDC latency gets its own panel rather than being folded into the
general one because authelia keeps a separate duration family for it,
and every machine-to-machine credential in this swarm is minted through
those endpoints -- averaging the two produces a number describing
neither.
Counters are counted over the dashboard range rather than rated. At this
volume a rate window contains no requests, so rate() returns zero and
draws a flat line, which is indistinguishable from a broken query; the
quantile panels are worse, because a quantile over all-zero buckets is
NaN and renders empty rather than zero.
The option's description claimed declaring an entry from the service's
own module put 'the scraper and the target on the same host by
construction rather than by luck'. It does not. It constrains where the
target is; nothing in it places the collector, and the two enable flags
are co-located by a shared lib.mkDefault rather than by construction.
Split across hosts, a target is silently never scraped — the service's
host declares an entry no local collector reads, the collector's host
never enabled the service. No error surfaces, and no assertion can
catch it: separate hosts are separate evaluations with no shared
context, so the doc telling the truth is the only mechanism there is.
The same paragraph already warned co-location was not a guarantee, four
lines below the sentence claiming it was; a reader arriving for
permission stopped at the permission. This one did.
swarm-nats carries the concrete caveat for its own contribution.
NATS has no Prometheus format of its own. It serves a JSON monitoring
endpoint, and prometheus-nats-exporter translates that — so this is two
changes in order, not one: without the monitoring endpoint the exporter
starts cleanly and scrapes nothing, which is the inert-config shape the
scrape work exists to avoid.
Both listeners are loopback and the exporter is the monitoring
endpoint's only intended reader: it is unauthenticated and /connz names
every connected client, so the address it binds is the whole access
control.
The scrape target is declared here rather than in the collector's
module, gated on a collector existing to read it — an entry exists only
where the service that named it runs.
mara, on review: "i thought we just swap a css file via nginx
config?" -- right instinct. The package-copy overlay (cp -r + install)
only made sense for hive-c0re's servedFrontend, which backs multiple
serve points (dashboard root + every per-agent gateway route) from one
swapped tree. swarm-ui has exactly one location serving cfg.package, so
an = /static/colors.css exact-match override -- the same idiom every
other single-path override on this vhost already uses (/api/whoami,
/api/docs) -- replaces the whole derivation with one location block.
Verified with the same throwaway nixosSystem eval as the previous
commit: unthemed case has no colors.css location and the / root is
cfg.package unchanged.
mara: "swarm dash is in my own colors, but tab colors on swarm ui are
the default catpucchin one." Root cause: hive-c0re/theme.nix's stylix
overlay only ever wrote a themed colors.css onto the dashboard/agent
frontend subtrees -- swarm-ui, served from its own separate package,
was never in scope.
Extracts the stylix-detection + colors.css-generation logic (previously
inline in theme.nix) into a shared nix/host-modules/stylix-theme.nix,
imported by both theme.nix and swarm-ui.nix -- one source instead of a
second copy that has to agree by inspection. swarm-ui.nix gains its own
themedPackage overlay (same shape as theme.nix's themedFrontend: copy
the package, overwrite static/colors.css) and serves that instead of
cfg.package directly when stylix is active; a clean passthrough
otherwise.
Verified with a throwaway nixosSystem eval (this repo's own flake
checks do not exercise gateway-module wiring) confirming the unthemed
path resolves cfg.package unchanged.
The scrape config names a client_secret_file; this is what puts a file
there. A host oneshot copies authelia's minted secret between the two
container trees — a copy and not a bind mount, because the secret does
not exist until authelia's first boot and nixos-container refuses to
start on a missing bind source, which on a fresh swarm is a permanent
stall presenting as broken metrics.
The collector runs under DynamicUser and the prometheus receiver opens
client_secret_file itself, as that user, so there is no uid to hand the
file to. LoadCredential reads it as root before the sandbox exists and
re-exposes it under a path that does not depend on which uid the unit
got; the scrape config points there. Both spellings derive from one
binding, since a mismatch is a file the collector cannot open and
nothing but a runtime 401 would say so.
