The largest of these moves: sixteen references spelled through `let`
aliases across eight modules, plus eight more spelled as a path, plus
five documentation pages.
authelia is also the clearest case for why the two namespaces exist.
`swarm.authelia.url` is needed by *every* hive in the swarm — it says
where to send a browser to authenticate — while running the container is
the business of exactly one host. The client half and the server half
were sharing a namespace whose whole contract is "identical everywhere",
and only one of them could honour it.
`swarm.authelia.oidc.clients` stays where it is for the same reason:
several modules register a client there, gated on authelia running here,
and the registry itself is what the service *is* rather than a decision
about this machine.
One sweep note worth recording: a grep for `swarm.authelia.enable` misses
`swarmCfg.authelia.enable`, because the prefix is whatever the reading
file bound. Grepping the suffix `.authelia.enable` finds both, and found
a reference in swarm.nix that the path-shaped pattern did not.
One commit rather than two because they are not independent: the UI's
`enable` had the controller's as its literal default, so moving the
controller alone would leave the UI's default naming an option that no
longer exists.
The UI keeps that derivation in its new home — it is a view onto the
controller's state and reaches it over that daemon's unix socket, so the
host running the controller is the host that can serve it.
Three spellings had to move together for the UI, not one: the `default`,
the `defaultText` shown in the options doc, and the description prose
that names the old path in words. A grep for the option path finds the
first two.
The sweep also reached outside nix: `swarm-controller`'s crate README and
its `//!` module doc both named the option, as did this repo's own
CLAUDE.md and four pages under docs/. An option's name is API, and its
documentation lives wherever someone thought to write it down.
Same move as grafana, and the three belong together: they derive from one
switch and a store with no UI is as useless as a UI with no store.
`victorialogs` is the case that shows why the option-path sweep is not
enough on its own. It has **zero** references spelled
`swarm.victorialogs.enable` anywhere in the tree, and four spelled
through `let` aliases (`vlCfg.enable` in the collector, `cfg.enable` in
its own module). A sweep for the path would have reported nothing to do
and left every reader broken.
Prose moved with the code rather than being left behind: the comments in
swarm-required-services.nix that explained why the pair derives together
now sit above the assignments that do it, instead of above the gap where
they used to be.
`ssoLocal` in the grafana module read `cfg.enable`, where `cfg` is the
local binding for `services.hyperhive.swarm.grafana` — the option set the
previous commit removed `enable` from. Evaluation fails on the missing
attribute.
The sweep that missed it looked for the option *path*
(`grafana.enable`). This reference is spelled through a `let` alias, so
no grep for the path can find it, and a parse check cannot either: it is
syntactically valid and only wrong once the module system resolves the
attribute.
The check that does find it: for each file, extract the local bindings
that point at the option set being changed, then grep for
`<alias>.<removed-attr>`. Run against this tree it reports nothing for
grafana and sixteen references for authelia, which is the next module to
move.
`services.hyperhive.swarm.*` is meant to be identical on every host in a
swarm — it describes the swarm, and every hive needs all of it to be a
client. But it also carried the `enable` toggles, which are precisely the
values that must differ per machine. The namespace that should be the
same everywhere held the one thing that cannot be.
Adds `services.hyperhive.deploy.*` for a host's deployment decisions, and
moves the first of them (`swarm.grafana.enable` -> `deploy.grafana`) as
the pattern for the rest. Flat and named for the thing deployed rather
than grouped under a "swarm services" attribute: from the deploy side it
does not matter what kind of thing each one is, and a grouping by service
kind would re-encode the service-side taxonomy into a layer that does not
care about it.
Behaviour is unchanged. The move is a rename in the strict sense — same
type, same meaning, new path — so `mkRenamedOptionModule` carries it and
existing configs keep evaluating with one warning naming both paths. The
renames live in the new module rather than the service modules, so the
whole migration has a single home and a single file to delete when the
deprecation window closes.
harness-base.nix has never existed in this tree. Four comments named it,
or a `harness-base` module, as the place to look:
- weston-vnc.nix: the agent user is declared and home-chowned by
nix/agent-modules/user.nix
- hive-ci.nix: the sandbox-fallback reasoning lives in
nix/agent-modules/default.nix -- which the very next comment block in
the same file already cites correctly
- packages/default.nix: the per-bin consumer is
nix/agent-modules/packages.nix
- hive-sh4re/src/assets.rs: HIVE_ASSETS_DIR is set by
hive-c0re/environment.nix and agent-modules/default.nix +
agent-service.nix, and the package is built by nix/packages/assets.nix
-- not the equally nonexistent nix/assets.nix
assets.rs was twice declared out of scope on the sibling PR because it
names a module rather than a file. That distinction was real and
irrelevant: neither the module nor the nix/assets.nix path it points at
exists. Reading the wording is not checking the reference.
Every replacement path was verified to exist, with a deliberately bogus
path as a control.
Per mara on #3725: the on/off toggle is removed, and required env vars
unset lead to a crash. HIVE_NETWORK_ISOLATION is gone from
hive-network.nix -- it was the toggle.
Validation happens once at daemon startup rather than per container.
The variables are process-global, so a bad value breaks every container
rather than one: failing at boot gives a single diagnostic naming the
bad value, and cannot reach a state where some containers were
configured before it was noticed.
Option<NetworkIsolation> collapses to NetworkIsolation through the wire
type, client and helper, which deletes the branch instead of leaving it
unreachable. serde(default) is dropped on that field deliberately: a
request omitting isolation is now rejected rather than defaulting to a
container sharing the host's network namespace.
What this replaces was a silent security downgrade. Of the four ways
into the old fallback, two logged nothing at all -- a container came up
without isolation and the journal agreed it was fine.
Doc comments that still described the removed branch are updated
(argus's note on #3723 scoped that to this issue). The hive-priv one is
a minimal edit inside the block #3723 rewrites; de-splicing is that
PR's job.
- nix/checks.nix: 'hyperhive#3685' in a source comment trips the
tracker-tag lint (prose-only, per hive-rules.md) — argus caught it on
review. Reworded without the ref.
- frontend/npm-deps-hash: the earlier scaffold commit
(be18f460) added preact/typescript to agent/package.json, which
regenerated package-lock.json but never re-derived this hash in
lockstep — nix flake check's frontend derivation was failing on a
hash mismatch. Regenerated via
'nix run nixpkgs#prefetch-npm-deps -- frontend/package-lock.json'.
- StatusChips.tsx: drop the stale 'useAgentState lands in a later
commit' comment now that Root.tsx already wires it.
- nix/checks.nix: add agent-typecheck, same shape as swarm-ui-typecheck,
so a real TS error in the agent package fails CI instead of just
esbuild-stripping silently.
Two findings on one panel, both from mara.
Form: 24 label series drawn as overlapping timeseries lines is
unreadable by construction. The panel is a current-count ranking, not a
time evolution, so it becomes a horizontal bar gauge on an instant
sort_desc query -- identity on the axis, magnitude in the bar, and a
fixed colour because colour was carrying no information that the axis
was not already carrying.
Title: the metric counts open AND closed issues. Measured, not assumed
-- the largest single label reads 1619 while gitea_issues_open reads
117, and no label can outnumber the set it is a subset of. The panel had
been asserting the opposite since it was written.
mara on the PR: "many hint texts are incorrect, most hives dont work on
hyperhive itself". Four panel descriptions asserted local practice as if
it were general:
Releases "expected to sit at zero on a hive that ships from
main" -- a claim about how THIS project releases
Automation surface "every hive registering the same webhook has bitten
this forge before" -- a war story from this
deployment, meaningless on any other
Organizations "near-constant by design" -- assumes our org layout
Issues open/closed "usually a bulk triage, not a productive afternoon"
-- a claim about our working rhythm
Each now describes what the metric IS and what a CHANGE means, and says
outright where the panel cannot know: whether zero releases is normal
depends on how a hive's projects ship, and a vertical step in closed is
some batch operation that this panel cannot distinguish from a burst of
real work.
The board title is NOT an instance of this and is unchanged -- all seven
dashboards use the 'hyperhive - <thing>' product namespace, checked
rather than assumed.
The forge restarted (new forgejo PID at 12:36), so it re-read [metrics]
and gitea_issues_by_label now exists -- 24 series carrying real label
names, read back out of the store rather than inferred from the metric
name appearing in a label-values list.
The panel shipped deliberately empty with EMPTY UNTIL THE FORGE RESTARTS
in its title and a matching noValue, because an empty graph and an
undeployed metric look identical. That warning is now false, and a stale
warning is worse than none: it tells a reader the panel cannot have data
at the moment it does.
Keeps the operational half rather than deleting it -- the restart
requirement is a fact about the NEXT metric someone enables, so it moves
into the description as a note and into noValue, where it fires only when
the series really is missing.
Adds the fourth per-service board. Unlike its three siblings it answers
"what does the forge CONTAIN", not "is the forge healthy" -- forgejo's
exporter publishes a census of stored objects and no request, error or
latency series at all.
That distinction is the board's own description rather than something a
reader has to infer, because a fourth per-service board that silently
answers a different question than the other three is worse than no board:
someone checks it during an incident and reads a flat line as calm.
Panels: repositories, users, orgs, issues open/closed, releases, the
open-vs-closed pair over time, repos-and-users together (they move in
lockstep when an agent is created, so divergence is the signal),
discussion volume, and the webhook/hooktask pair.
One panel ships deliberately EMPTY: issues-by-label. The metric is
enabled in the forge's config but forgejo reads its [metrics] section at
startup, so the series does not exist until that process restarts -- and
a config merge alone does not restart it. The panel title says so, the
description says how to tell "not deployed" from "no labelled issues",
and `noValue` says it again in the graph itself. An empty panel with no
explanation is the failure this board is trying not to be.
Gated: every panel's metric confirmed to exist in the store, with a
bogus name checked absent in the same run so the check can fail, plus a
known-positive control against the label-values query itself -- a
download that silently returned nothing would otherwise report every
metric missing. Gate kept as verify-3494-forge-board.sh.
The comment-block lint (added in 79dc8ca6) now trips on this block —
genuinely pre-existing, unrelated to that change, just newly caught.
Per the lint's own suggested remedy: relocated the full per-directive
reasoning plus both footguns (session-cache keying, the Host-header
clobber that can recurse a subrequest into itself) to a new
"Dialing another vhost by name" section in docs/gateway.md, and left
a short why + pointer comment in the source. No behavior change.
verifiedProxyTo (43ae164d) verified TLS but left Host to nixpkgs'
recommendedProxySettings, which sets Host to the CALLING vhost, not
the target. Since every consumer resolves back to this same gateway,
nginx picks the vhost to answer by Host header (not by the SNI
proxy_ssl_name already sends) — so every auth subrequest looped back
into its own vhost's auth_request, recursing until nginx's subrequest
depth limit turned it into a 500, on every domain gated by SSO.
Pin Host (and the rest of the header set nixpkgs' recommended include
would otherwise still be the one to set) inside verifiedProxyTo, and
set recommendedProxySettings = false on each of the four call sites so
nixpkgs' own copy — appended after a location's extraConfig — can't
clobber it back.
One list, in nix/reserved-names.nix, handed to everything that needs it
as HIVE_RESERVED_NAMES. Keeping it current becomes a config change
rather than a rebuild, and hive names and agent names -- one namespace
going forward -- are checked against the same file: swarm-otel.nix's
hand-written reservedOwners is gone.
Whitespace-separated rather than JSON, deliberately, unlike the
structured env vars beside it. Every entry is an Ident ([a-z0-9-]), so
whitespace cannot occur inside a name and the encoding is provably
lossless; JSON would mean either a parser dependency in a crate whose
purpose is to have none, or a copy of the parse in every consumer.
An UNSET variable is not "nothing is reserved". Both creation sites log
an error and return a warning saying the check did not run, so a
misconfigured deployment says so instead of silently accepting every
name. A blank value folds into unset: nix always renders a non-empty
list, so present-but-empty is a rendering fault, not a declaration.
Two guards whose subject moved out of their own file now assert their
own case is still in it, because a guard that can be retired by an edit
elsewhere is not a guard:
- swarm-otel.nix asserts reserved-names.nix still contains its
swarmTierName.
- hive-sh4re's sentinel drift test PANICS when the variable is missing
rather than skipping -- a drift test that quietly does nothing still
reports green. checks.nix and devshell.nix both export it so CI and a
local cargo test agree. Verified as a pair: with the variable set, 8
tests pass; with it unset, exactly the 4 drift tests fail and the
unrelated ones still pass.
Consumers reached authelia at `127.0.0.1:<port>`, which encoded a
co-location nobody agreed to: the gateway and authelia are not required
to share a host, so the literal is a requirement stated only by being
unwriteable any other way. Moving them to the name is the point of the
issue.
But a name over https is only half of "https and auth". nginx's
`proxy_ssl_verify` is OFF by default and there was no `proxy_ssl_*`
anywhere in the tree, so the obvious repoint would have produced an
encrypted, unauthenticated hop -- which works, and keeps working,
against any certificate at all.
Adds `gateway.lib.verifiedProxyTo <name>` next to the rest of the vhost
kit, so the convention has one definition rather than a copy in each
consuming module, and repoints the four call sites through it.
Each directive was checked against a real nginx with the opposite arm
run as a control:
- the CA *bundle* (root + intermediate) is accepted -- worth checking,
since `hive-ca-trust.nix` warns off consumers that read only one
certificate, and nginx is not one of those
- verification checks the chain: an unrelated CA fails
- and the HOSTNAME: a wrong `proxy_ssl_name` fails even with a good
chain. Chain-only would accept any cert this CA ever signed, which
for an internal CA is every service on the hive
- with verify off, the wrong CA passes -- so the failures above come
from verification, not from the connection
Bind addresses are untouched. This changes what consumers dial, not what
anything listens on.
The upstream gitea-actions-runner unit uses DynamicUser, and systemd
mounts a dynamic unit's state directory noexec. Measured from a job's
own /proc/self/mountinfo on the live runner, with /tmp as the control:
the state dir carries noexec, /tmp does not, and a script written and
chmod +x'd in the workspace fails execve with EACCES while the same
script in /tmp runs.
This was invisible for as long as every workflow compiled inside the
nix sandbox and executed nothing from the workspace. The first job that
built a binary into the runner's own target dir -- an instrumented
coverage run -- died on its first build script.
ExecPaths= re-mounts the state dir executable. Both spellings are
listed with the ignore-if-absent prefix because ExecPaths resolves
against the host root, where the dynamic-user layout makes
/var/lib/gitea-runner a symlink into private/.
/etc/gitconfig shipped `helper = git-credential-hive-forge`. git prepends
`git-credential-` to any helper value that is not an absolute path, so
that resolves to `git-credential-git-credential-hive-forge`, which does
not exist -- no helper runs at all. The sibling github.nix has always used
the short form.
Measured rather than read off the docs, with the arms isolated from the
personal ~/.gitconfig:
helper = git-credential-hive-forge -> 0 credentials, and git prints
"'credential-git-credential-hive-forge' is not a git command"
helper = hive-forge -> 1 credential, clean stderr
The reason this survived: every long-lived agent has a personal
~/.gitconfig naming the helper by ABSOLUTE path, which git accepts, so
pushes keep working and the stderr line reads as noise. The system config
is masked exactly where someone would notice it and bites a fresh agent
that has no such file.
The authelia vhost intercepts upstream errors and serves a friendly
"SSO unavailable" page. The `=` form of `error_page` takes its status
from the redirected location, and that location serves a file -- so the
page is returned as **200**.
That is right for a human typing the URL and wrong for every machine
caller, all of which reach authelia through this same vhost by name:
- `/api/authz/auth-request` -- nginx `auth_request` treats any 2xx as
success, so a down authelia means access GRANTED
- `/api/oidc/introspection` -- a token check answering 200
- `/api/oidc/token`, `/.well-known/openid-configuration` -- clients
parsing an HTML error page as their JSON document
Routes `/api/` and `/.well-known/` without the interception. A longer
prefix wins over `/`, and the intercept directives live inside the `/`
location rather than at server level, so they do not reach the new ones.
Split by AUDIENCE rather than by an enumerated path list: a human still
gets the page, and every machine caller -- including the login page's own
XHR, and any endpoint added later -- gets the real status.
Measured against a real nginx with a dead upstream, both arms: machine
paths return 502 where they returned 200+HTML, a subrequest through the
new prefix denies (matching a direct port dial) where through `/` it
served the protected content, and the browser control confirms the
friendly page survives. URI preservation checked separately against a
live echo upstream -- `proxy_pass` with no URI part passes the full
original path.
Adds `cargo llvm-cov` to the devshell and a coverage workflow that runs
only when someone asks for it.
Manual dispatch rather than nightly or per-PR, per the discussion on the
issue: a coverage run needs its own instrumented build and cannot reuse
the normal test artifacts, so it roughly doubles a test job. "Do we have
glaring holes" is a question someone asks occasionally, not a gate every
PR pays for, and not a number worth spending farm time on every night
whether or not anyone reads it. Manual dispatch costs nothing until the
answer is wanted.
Its own workflow file rather than a job in ci.yml: that file already has
a `workflow_dispatch` trigger so `hive-forge ci-rerun` can retrigger
without an empty commit, and a trigger there fires EVERY job in the
file -- a coverage job added there would run on every pull request.
No threshold and no --fail-under-lines. A coverage gate mostly teaches
people to write assertion-free tests that execute lines; the report is
the deliverable and the number is for a human to read.
The devshell needs LLVM_COV / LLVM_PROFDATA set explicitly: cargo
llvm-cov expects rustup's `llvm-tools-preview` beside the toolchain and
nixpkgs has no such component, so without them it aborts with "failed to
find llvm-tools-preview" -- which reads like a missing install rather
than a path the shell has to name.
Verified by running it: `cargo llvm-cov --package hive-types` produces a
real report (3 tests, 85.44% regions). Both the variable names and the
package were wrong on the first attempt and only running it said so --
the names take no `_PATH` suffix, and the binaries are in
`llvmPackages.llvm`, not `llvmPackages.bintools`, which is the linker
wrapper and ships neither.
Host metrics arrived carrying `hive` and no host attribute, so "which
host is out of memory" was answerable only as "which hive" -- true only
while a hive is one machine, which is the framing the swarm work exists
to end.
Adds the collector's own `resourcedetection` processor to the metrics
pipeline, emitting both `host.name` and `host.id`.
`resourcedetection` rather than a value picked in nix: this is the
canonical OTel mechanism for host identity, and choosing our own would
answer a question the tool already answers, differently from every other
deployment. A resource attribute is captured into the stored series, so
a private naming scheme is expensive in the way that lasts.
Both attributes, deliberately: `host.name` is readable and what a panel
groups by, but splits the series silently on a rename with nothing
linking old to new; `host.id` survives that and is unreadable alone.
Hosts get renamed and repurposed exactly when services move off one box.
The processor sits in the pipeline, so it applies to every receiver --
agent-pushed OTLP and scraped targets included, not just `hostmetrics`.
Keys verified against otelcol-contrib 0.151.0 with two deliberate
mutations rejected in the same run (a bogus resource attribute and a
bogus detector key), so "accepted" is distinguishable from a validator
that ignores what it does not recognise.
Turns on gitea_issues_by_label and gitea_issues_by_repository, gated on
behindGateway exactly like the metrics endpoint itself -- they are only
reachable through the authenticated location that condition creates.
These are off by default upstream because they are the only metrics in
this section whose series count grows with the CONTENT of the forge
rather than with its feature set, so the question is cardinality.
Measured rather than assumed: ~19 distinct labels and ~29 repos on this
instance, well under 100 series, against a store that took +2234 from
nine host scrapers.
Still no TOKEN. Forgejo can guard the endpoint with its own static
bearer, but the swarm authenticates the scraper at the gateway, and a
second per-service credential would be the one that stops getting
rotated -- the reasoning already recorded above ENABLED applies
unchanged to the breakdowns.
The token file is 0600 root-owned and the precondition ran as the unit's
dynamic gitea-runner user, so it could not read it -- and it did not fail
closed. `[ -s ]` succeeds on a stat alone, grep then died with EACCES,
and `! grep` turned that error into true, so the gate exited 0 on a file
it had never read. It has reported 'a real token is present' without
looking for its entire existence, which is why the runner kept retrying
instead of holding still and saying why.
Prefix the step with + so it runs with full privileges. This grants the
runner nothing new: tokenFile becomes EnvironmentFile=, which systemd
already reads as root before dropping privileges, so the payload never
passes through an unprivileged reader either way -- the + only lets the
gate observe what systemd observes.
Also assert readability separately and loudly, rather than folding it
into the placeholder test: an unreadable file and a real token produced
the same verdict, so 'I could not look' was indistinguishable from 'I
looked and it is fine'.
Two more per-service boards, cut from the withdrawn combined services
board: the log store (4 panels) and the metrics store (5).
The combined board was one page carrying panels for four different
services; the seams were already there, so this is a re-cut rather than
new authoring. Each board's subject is one service, and the services
page that remains will carry only panels whose subject is the *set*.
Both were imported by the operator before being committed here, so the
queries are known to draw against the live stores rather than only
parsing.
Nothing in the tree rendered a `[credential "<forge>"]` entry for an
agent, so every agent's `~/.gitconfig` accumulated one by hand, per
generation of forge address. Append-only, none ever removed, and after
the domain move the live one absent entirely:
[credential "http://forge.<old-hive>"]
[credential "http://localhost:3000"]
The absence of a writer is the defect. A value interpolated at eval time
follows a rename; a value captured into a mutable home file does not.
hive-c0re's own gitconfig already derives its scope from
`swarm.forge.domain` and moved correctly for exactly that reason.
What made it expensive to diagnose is that it does not present as a
credential problem. `git fetch` against a stale remote still succeeds --
the old name redirects and a public read needs no auth -- so the break
surfaces only at the first authenticated push, long after the move, as
`could not read Username for '<new host>'`. That names a host the agent
was never configured for, which reads like DNS or TLS.
Same class as the CI runner keeping its registered address, one tier
down.
The shape is `github.nix`'s, unchanged: a small credential helper that
reads the token from the agent's state file at invocation, with the
token PATH baked in rather than the value, because claude's Bash tool
runs in a minimal env that never sources /etc/set-environment.
`environment.etc."gitconfig"` is already bound by github.nix; the two
merge rather than collide because the option is `lines`. Verified by
eval with both modules defining it before this was written -- a silent
last-wins there would drop one integration's credentials and look
exactly like this bug again.
The pending-bytes panel described itself as the leading indicator for
the slow-consumer stat 'beside it', and then sat alone in its own row
with eighteen grid units of empty space. A description that tells the
reader to look beside it, where nothing is, is worse than no
description.
Slow consumers and pending bytes are now adjacent in the top row, and
the two JetStream panels share the second. No gaps.
Split out of the combined swarm-services board, which mara declined as
multiple dashboards in one trench coat. Its 17 panels partition exactly
by subject -- 3 services / 5 metrics store / 5 queue / 4 log store, with
no panel belonging to two -- so this is a re-cut of verified panels, not
new authoring.
Seven panels: connections, subscriptions, slow consumers and JetStream
API errors as stats, messages over the range, plus pending bytes (the
leading indicator for a slow consumer) and JetStream store size against
its configured limit (the only capacity question the queue has).
Connections is red at zero, unlike every other count here: three things
hold a connection at all times, so an empty count means the queue is
refusing everyone -- and a callout denial reaches a client as a timeout
rather than an error, which makes this often the first visible sign.
They differ by one letter and a grep for the shorter name matches both.
The summary is what ships; the per-process scraper is deliberately absent,
and a reader (human or grep) cannot tell from the diff alone.
Nothing measured the host itself. hive-c0re emits per-container metrics
and each service exports its own, so the store could answer 'is this
container using more memory than last week' and could not answer 'is the
machine under them out of memory' -- the question every one of those
readings is implicitly relative to.
Hive tier rather than swarm: a host is owned by the hive running on it,
and the swarm tier runs one collector for the whole swarm, so it would
have to reach other machines to see theirs.
Every scraper the pinned receiver offers except process, which adds a
series set per running process -- unbounded on a hive host, where the
nine shipped are a fixed handful. Coverage and cardinality are different
axes and only the second can hurt the store.
Refs #3649
The collector's client is registered for authelia.bearer.authz, but the
prometheus scrape asked for no scopes, so every token came back carrying
none and authelia refused it at introspection with 'the requested scope
is invalid, unknown, or malformed'. Forgejo metrics were never scraped.
The rule was already stated one field below, for the audience: registered
is not requested. The two travel together, and a config read cannot see
the one that is missing -- every inspection of the rendered config came
back correct, because what was present was correct.
NODE_EXTRA_CA_CERTS beside it is Node's and additive. gitea-runner is Go,
whose trust store is replacing: it reads SSL_CERT_FILE and trusts only
what that names. So the container trusted the hive CA for its Node
actions and never for the daemon itself.
The daemon then fails startup with x509 unknown authority the moment it
reaches a TLS endpoint, crash-loops on Restart=on-failure, and no job is
picked up -- every pull request just sits at Waiting to run.
Not claimed as the whole cause of that outage: which address the daemon
holds is a separate question. The trust gap is real under every
explanation, and adding a CA to one unit cannot break a working path.
journaldUnits named "authelia"; upstream derives the unit from the instance
name, so the running unit is authelia-swarm and that entry matched nothing.
The receiver's units list is an allowlist, so a name matching no journal
entry is silently absent -- it reads as a service with nothing to say rather
than as a misconfiguration, and it excluded the busiest source on the box.
Use the unitName binding the rest of this module already uses, so the
allowlist cannot drift from the unit again.
The two callout assertions guarded the halves the server verifies with.
The responder needs the other halves, and nothing related them: a config
supplying only the public keys satisfies both, renders a syntactically
valid auth_callout block, and defines no responder unit.
Callout with no responder is the fail-closed state, so that queue refuses
every client — and a NATS denial arrives as a timeout, so the symptom is
every consumer hanging with nothing logged anywhere.
The build-time config check does run in this case and passes, because the
config is valid; what is missing is a unit, and the absence of a unit is
not an event.
Both services carried an `sso.enable` defaulting to false, so a swarm's
own forge and homeserver shipped with their identity provider switched
off unless an operator remembered two lines. Grafana never had the
toggle and is the shape the other two now match.
Behaves as if the setting were true: `ssoLocal` loses one conjunct, the
three assertions become unconditional, and the login source and
identity_provider render always.
The option is removed rather than defaulted, so a config that turned SSO
OFF fails where that line is instead of silently gaining a login
provider on the next rebuild.
The journald receiver was configured with a directory and no filter, so
the swarm's log store received every unit on the host that runs the
collector. On a hive whose services live on a workstation that includes
the operator's desktop session, in a store every swarm operator can read.
The receiver has no system-only switch and its `matches` field is an
allowlist too, so what the swarm collects has to be stated rather than
excluded. Each service module names its own units: a service that is not
running contributes nothing, and one added later arrives declared.
An empty list is fail-open — the receiver renders no filter at all and
reads everything — so it is asserted against.
grafana-lokiexplore-app cannot query VictoriaLogs. Its volume views call
/loki/api/v1/index/volume, which VictoriaLogs does not implement and
answers "unsupported path requested", and upstream's position is that
the datasource plugin must provide drilldown support. No configuration
here changes that, so the app only ever offered a menu entry that looks
like a broken feature rather than an absent one.
The option's own description already ruled its siblings out for fronting
backends this swarm does not run. That premise expired when the swarm
gained a log store, which is what made adding this one look reasonable —
so the reason is rewritten rather than the list alone: Logs Drilldown is
excluded on compatibility, not on absence.
Explore with the VictoriaLogs datasource is the log browser, and
docs/swarm/services.md now says so where an operator looks for it.
Grafana was the one swarm service that exports Prometheus metrics and had no
scrape target, so the swarm's metrics UI was invisible to the metrics it
displays.
It serves on a unix socket and claims no TCP port, which is deliberate and
stays that way; a prometheus target has to be a host:port, so nginx re-serves
the one endpoint a scraper needs on loopback. An exact-match location, not a
prefix: widening it would re-serve the whole UI without authorization.
The metrics section is now stated explicitly rather than inherited, because a
scrape target depends on it and a changed upstream default would take the
endpoint away while nginx kept answering.
Adds services.hyperhive.swarm.victorialogs.domain and a gateway vhost
gated by the same auth_request check against authelia that swarm-uis
own vhost uses (same swarmAuthRequest shape, copied not shared - see
the file top comment for why). The stores own listener stays
loopback-only and unauthenticated exactly as before; the collector
still writes to it directly, never through this vhost, so this only
adds a new authenticated read path.
Registers the new domain in swarm.nixs serviceDomains so the
swarm-services sub-CA issues for it (a missed entry silently falls
back to the hive leaf, which cannot cover a name under a different
apex - swarm.nixs own comment on that list documents the incident
this caused before).
Verified with a throwaway module-eval (same technique as the flake
module-eval check): vhost only exists when victorialogs.enable is
set, forceSSL/no addSSL, both locations present, domain correctly
registered/absent from serviceDomains, links entry present.
Grafana could reach the swarm's metrics and not its logs, so the store that
landed with the collector pipeline had no reader.
Adds the VictoriaLogs datasource plugin and the Logs Drilldown app, and
provisions the datasource beside the metrics one. The drilldown matters as much
as the connection: an unfamiliar log stream is explorable without writing a
LogsQL query first, which is the difference between a store you can query and
one you can use.
nixpkgs' collector unit already sets SupplementaryGroups to systemd-journal
unconditionally, with a comment saying why. Systemd list options concatenate,
so this module's copy rendered ["systemd-journal" "systemd-journal"] and made
this a second owner of a fact upstream may later change.
The bind mount stays, since that half is genuinely ours.
The logs half was gated on the swarm's log store being enabled, so turning
that store off stopped collection entirely rather than leaving the upstream
export. Logs now fan out exactly as metrics do: the store when it runs, the
operator's upstream when one is configured, both when both.
The receiver, the journal mount and the group grant follow whether there is
anywhere to send logs, not whether the local store exists. Tying the mount to
the store instead would render a receiver that can read nothing.
Deferred until the collector had a logs pipeline writing to it: a store
nothing writes to starts, answers queries and returns nothing, so the first
person to look concludes there were no logs rather than that nothing was
collecting them.
The journald receiver leaves the OTLP body empty and carries the entry as a
map of journal fields, so VictoriaLogs had no message to index and wrote a
placeholder into _msg on every record. Ingest returned 200, every field was
present, and a plain search for a line sitting in the store found nothing.
_msg_field names the field that holds the text. _stream_fields is the
difference between one stream for the whole host and one per unit per
machine; both are set by journald itself and both are low-cardinality.
A journald receiver reading the host's journal directory, a logs pipeline
stamped with the swarm tier's own resource processor, and an otlphttp
exporter pointed at VictoriaLogs. All four parts are conditional on the log
store being enabled, so a swarm without one renders exactly as before.
The host's directory is enough to see every container: nspawn links a
non-ephemeral container's journal guest-side, so the files live on the host
under the container's machine-id, and journalctl descends into those
subdirectories. Measured, along with the fixed systemd-journal gid that makes
the group grant meaningful across the bind mount.
An agent can verify that a unit was launched and never that it is
working: container journals are not reachable, so a diagnosis stops at
the first broken component -- which is precisely the component whose own
instrument is least likely to be legible. This is the store half of
collecting logs centrally so the question becomes answerable.
Mirrors swarm-victoriametrics deliberately: same container shape, same
loopback pin, same self-scrape arrangement. Two differences, both
intentional.
No gateway vhost. VictoriaLogs' ingest and query endpoints carry no
authentication of their own, exactly like the metrics store's -- and the
metrics store IS published under a resolvable name, which is an open
question rather than a settled design. Publishing this one the same way
would repeat that before the first instance is decided.
Retention defaults to 30d against the metrics store's 5y. Logs are
orders of magnitude larger per unit of time and their value decays much
faster: a log line answers what happened during an incident, a metric
answers whether this is worse than last quarter.
Not wired into enableRequiredServices yet -- that lands with the
collector pipeline, so we do not start a store nothing writes to.
