Turns on gitea_issues_by_label and gitea_issues_by_repository, gated on
behindGateway exactly like the metrics endpoint itself -- they are only
reachable through the authenticated location that condition creates.
These are off by default upstream because they are the only metrics in
this section whose series count grows with the CONTENT of the forge
rather than with its feature set, so the question is cardinality.
Measured rather than assumed: ~19 distinct labels and ~29 repos on this
instance, well under 100 series, against a store that took +2234 from
nine host scrapers.
Still no TOKEN. Forgejo can guard the endpoint with its own static
bearer, but the swarm authenticates the scraper at the gateway, and a
second per-service credential would be the one that stops getting
rotated -- the reasoning already recorded above ENABLED applies
unchanged to the breakdowns.
Forgejo's issue-list endpoint answers 'was this merged?' only inside the
nested pull_request object, while state says closed for a merged PR and
for one closed without merging alike. So the obvious top-level query is
null or ambiguous for every row, and with a // default it renders as a
confident 'nothing merged' that cannot ever be right -- a wrong answer
shaped exactly like a clean one.
Copy merged and merged_at up to the top level of each PR row so the
obvious query is the correct one. Additive: the nested object is left
untouched so an existing consumer keeps working, and issue rows have no
pull_request and pass through unchanged.
The head branch is deliberately not hoisted: this endpoint does not carry
it at all -- the row's ref is an empty string, not the branch -- so there
is nothing to lift. pr show has head_branch.
mara: "only the abstract jobq part was asked for in the first place" —
the previous revision still carried the full hive-c0re step-kind table
under a "core-specific nodes" heading; that catalogue belongs in
coordinator.md (where it already lived) alongside the rest of
hive-c0re's job-queue internals, not duplicated here.
jobq.md is now just the domain-agnostic model (graph of steps + shared
resource slots) plus the generic row/step/glyph framing for watching it
in the dashboard — no hive-c0re-specific step names anywhere.
Coordinator.md's job-queue section, docs/README.md, and the root
CLAUDE.md reading-paths index are updated to match.
mara: "the issue wanted jobq the abstract thing docs, but you
documented the hive nodes" — the original PR only had the
core-specific node table (the issue's second ask); it was missing the
first ask, an operator-facing explanation of the jobq engine itself
(graph of steps + shared resource slots) independent of what hive-c0re
specifically builds on top of it.
Adds that as its own section up front, and reframes the existing node
table under an explicit "core-specific nodes" heading so the two asks
read as clearly separate pieces.
hyperhive/hyperhive#2999 asked for an operator-facing page explaining
the job queue's node kinds in the abstract, without the coordinator's
implementation detail (leases, resource windows, module layout).
Add docs/jobq.md: what a queue row/step means, a plain-language table
of every node label an operator actually sees in the dashboard
(BU1LDS/R3BU1LD QU3U3 and swarm-ui's /jobs page both render the same
graph), and a rough shape for a typical rebuild.
Wire it into docs/README.md and the root CLAUDE.md reading-paths index
alongside coordinator.md. Add a pointer from coordinator.md's job-queue
section to the new page so the two don't try to explain the same thing
twice at different levels.
The token file is 0600 root-owned and the precondition ran as the unit's
dynamic gitea-runner user, so it could not read it -- and it did not fail
closed. `[ -s ]` succeeds on a stat alone, grep then died with EACCES,
and `! grep` turned that error into true, so the gate exited 0 on a file
it had never read. It has reported 'a real token is present' without
looking for its entire existence, which is why the runner kept retrying
instead of holding still and saying why.
Prefix the step with + so it runs with full privileges. This grants the
runner nothing new: tokenFile becomes EnvironmentFile=, which systemd
already reads as root before dropping privileges, so the payload never
passes through an unprivileged reader either way -- the + only lets the
gate observe what systemd observes.
Also assert readability separately and loudly, rather than folding it
into the placeholder test: an unreadable file and a real token produced
the same verdict, so 'I could not look' was indistinguishable from 'I
looked and it is fine'.
Upstream's ExecStartPre re-registers only when .runner is absent, the
labels changed, or the registration token hash changed -- never when the
instance URL changed. So writing a fresh token and restarting the runner
registers only if the hash happens to differ, which is Forgejo's choice
to make: whether the admin registration-token endpoint mints a new token
per request or hands back a stable one is not ours to depend on.
hive-c0re already decides correctly -- ensure_ci_runner_registered only
reaches this helper once it has concluded the credentials are absent or
stale -- but the remediation was a no-op, so re-registration was
requested every boot and never happened.
Remove .runner before the restart so upstream takes its absence branch,
the one it evaluates unconditionally. NotFound is success; any other
error propagates rather than reporting Ok for a registration that never
ran.
Two more per-service boards, cut from the withdrawn combined services
board: the log store (4 panels) and the metrics store (5).
The combined board was one page carrying panels for four different
services; the seams were already there, so this is a re-cut rather than
new authoring. Each board's subject is one service, and the services
page that remains will carry only panels whose subject is the *set*.
Both were imported by the operator before being committed here, so the
queries are known to draw against the live stores rather than only
parsing.
Nothing in the tree rendered a `[credential "<forge>"]` entry for an
agent, so every agent's `~/.gitconfig` accumulated one by hand, per
generation of forge address. Append-only, none ever removed, and after
the domain move the live one absent entirely:
[credential "http://forge.<old-hive>"]
[credential "http://localhost:3000"]
The absence of a writer is the defect. A value interpolated at eval time
follows a rename; a value captured into a mutable home file does not.
hive-c0re's own gitconfig already derives its scope from
`swarm.forge.domain` and moved correctly for exactly that reason.
What made it expensive to diagnose is that it does not present as a
credential problem. `git fetch` against a stale remote still succeeds --
the old name redirects and a public read needs no auth -- so the break
surfaces only at the first authenticated push, long after the move, as
`could not read Username for '<new host>'`. That names a host the agent
was never configured for, which reads like DNS or TLS.
Same class as the CI runner keeping its registered address, one tier
down.
The shape is `github.nix`'s, unchanged: a small credential helper that
reads the token from the agent's state file at invocation, with the
token PATH baked in rather than the value, because claude's Bash tool
runs in a minimal env that never sources /etc/set-environment.
`environment.etc."gitconfig"` is already bound by github.nix; the two
merge rather than collide because the option is `lines`. Verified by
eval with both modules defining it before this was written -- a silent
last-wins there would drop one integration's credentials and look
exactly like this bug again.
The pending-bytes panel described itself as the leading indicator for
the slow-consumer stat 'beside it', and then sat alone in its own row
with eighteen grid units of empty space. A description that tells the
reader to look beside it, where nothing is, is worse than no
description.
Slow consumers and pending bytes are now adjacent in the top row, and
the two JetStream panels share the second. No gaps.
Split out of the combined swarm-services board, which mara declined as
multiple dashboards in one trench coat. Its 17 panels partition exactly
by subject -- 3 services / 5 metrics store / 5 queue / 4 log store, with
no panel belonging to two -- so this is a re-cut of verified panels, not
new authoring.
Seven panels: connections, subscriptions, slow consumers and JetStream
API errors as stats, messages over the range, plus pending bytes (the
leading indicator for a slow consumer) and JetStream store size against
its configured limit (the only capacity question the queue has).
Connections is red at zero, unlike every other count here: three things
hold a connection at all times, so an empty count means the queue is
refusing everyone -- and a callout denial reaches a client as a timeout
rather than an error, which makes this often the first visible sign.
They differ by one letter and a grep for the shorter name matches both.
The summary is what ships; the per-process scraper is deliberately absent,
and a reader (human or grep) cannot tell from the diff alone.
Nothing measured the host itself. hive-c0re emits per-container metrics
and each service exports its own, so the store could answer 'is this
container using more memory than last week' and could not answer 'is the
machine under them out of memory' -- the question every one of those
readings is implicitly relative to.
Hive tier rather than swarm: a host is owned by the hive running on it,
and the swarm tier runs one collector for the whole swarm, so it would
have to reach other machines to see theirs.
Every scraper the pinned receiver offers except process, which adds a
series set per running process -- unbounded on a hive host, where the
nine shipped are a fixed handful. Coverage and cardinality are different
axes and only the second can hurt the store.
Refs #3649
The collector's client is registered for authelia.bearer.authz, but the
prometheus scrape asked for no scopes, so every token came back carrying
none and authelia refused it at introspection with 'the requested scope
is invalid, unknown, or malformed'. Forgejo metrics were never scraped.
The rule was already stated one field below, for the audience: registered
is not requested. The two travel together, and a config read cannot see
the one that is missing -- every inspection of the rendered config came
back correct, because what was present was correct.
NODE_EXTRA_CA_CERTS beside it is Node's and additive. gitea-runner is Go,
whose trust store is replacing: it reads SSL_CERT_FILE and trusts only
what that names. So the container trusted the hive CA for its Node
actions and never for the daemon itself.
The daemon then fails startup with x509 unknown authority the moment it
reaches a TLS endpoint, crash-loops on Restart=on-failure, and no job is
picked up -- every pull request just sits at Waiting to run.
Not claimed as the whole cause of that outage: which address the daemon
holds is a separate question. The trust gap is real under every
explanation, and adding a CA to one unit cannot break a working path.
act_runner records the --instance URL it was registered with and reads the
forge address from .runner and nowhere else: the nix option reaches only
`register`, and upstream re-registers on a changed token or labels, never on
a changed URL. Our own precond short-circuits on .runner existing, and
runner_valid asks whether the runner id still exists -- which after a rename
it does. So changing swarm.forge.domain left the runner dialling a name its
container's derived extraHosts no longer resolves, with every check green.
The symptom is CI going quiet rather than anything failing.
Compare the persisted address against the configured base as part of the
same early return. A disagreement mints a fresh registration token, which
changes the token hash upstream already keys on, so upstream removes .runner
and re-registers against the current --instance. This module never writes or
deletes that file; the deletion stays with the script that owns registration.
The comparison is host-only and lenient on purpose. Both sides render from
one nix expression -- http://${swarm.forge.domain}, reaching the runner as
instances.hive.url and this daemon as HIVE_FORGE_URL -- so they cannot drift,
while scheme, port and trailing slash are exactly the cosmetic differences
that would otherwise re-register on every boot. Unreadable, missing or
unparseable input keeps the existing credentials, matching runner_valid's
treatment of a transport error: only a positive disagreement counts.
journaldUnits named "authelia"; upstream derives the unit from the instance
name, so the running unit is authelia-swarm and that entry matched nothing.
The receiver's units list is an allowlist, so a name matching no journal
entry is silently absent -- it reads as a service with nothing to say rather
than as a misconfiguration, and it excluded the busiest source on the box.
Use the unitName binding the rest of this module already uses, so the
allowlist cannot drift from the unit again.
The two callout assertions guarded the halves the server verifies with.
The responder needs the other halves, and nothing related them: a config
supplying only the public keys satisfies both, renders a syntactically
valid auth_callout block, and defines no responder unit.
Callout with no responder is the fail-closed state, so that queue refuses
every client — and a NATS denial arrives as a timeout, so the symptom is
every consumer hanging with nothing logged anywhere.
The build-time config check does run in this case and passes, because the
config is valid; what is missing is a unit, and the absence of a unit is
not an event.
Both services carried an `sso.enable` defaulting to false, so a swarm's
own forge and homeserver shipped with their identity provider switched
off unless an operator remembered two lines. Grafana never had the
toggle and is the shape the other two now match.
Behaves as if the setting were true: `ssoLocal` loses one conjunct, the
three assertions become unconditional, and the login source and
identity_provider render always.
The option is removed rather than defaulted, so a config that turned SSO
OFF fails where that line is instead of silently gaining a login
provider on the next rebuild.
The journald receiver was configured with a directory and no filter, so
the swarm's log store received every unit on the host that runs the
collector. On a hive whose services live on a workstation that includes
the operator's desktop session, in a store every swarm operator can read.
The receiver has no system-only switch and its `matches` field is an
allowlist too, so what the swarm collects has to be stated rather than
excluded. Each service module names its own units: a service that is not
running contributes nothing, and one added later arrives declared.
An empty list is fail-open — the receiver renders no filter at all and
reads everything — so it is asserted against.
VictoriaLogs shipped without an operator-facing section: services.md
covered SSO, metrics and the collector, and said nothing about where the
swarm's logs land, how to read them, or how long they are kept.
Adds one, answering the questions an operator arrives with — the name it
serves on, that Grafana's Explore is the browser for it, retention, and
that it binds loopback with the gateway vhost as the only way in.
grafana-lokiexplore-app cannot query VictoriaLogs. Its volume views call
/loki/api/v1/index/volume, which VictoriaLogs does not implement and
answers "unsupported path requested", and upstream's position is that
the datasource plugin must provide drilldown support. No configuration
here changes that, so the app only ever offered a menu entry that looks
like a broken feature rather than an absent one.
The option's own description already ruled its siblings out for fronting
backends this swarm does not run. That premise expired when the swarm
gained a log store, which is what made adding this one look reasonable —
so the reason is rewritten rather than the list alone: Logs Drilldown is
excluded on compatibility, not on absence.
Explore with the VictoriaLogs datasource is the log browser, and
docs/swarm/services.md now says so where an operator looks for it.
Grafana was the one swarm service that exports Prometheus metrics and had no
scrape target, so the swarm's metrics UI was invisible to the metrics it
displays.
It serves on a unix socket and claims no TCP port, which is deliberate and
stays that way; a prometheus target has to be a host:port, so nginx re-serves
the one endpoint a scraper needs on loopback. An exact-match location, not a
prefix: widening it would re-serve the whole UI without authorization.
The metrics section is now stated explicitly rather than inherited, because a
scrape target depends on it and a changed upstream default would take the
endpoint away while nginx kept answering.
Every hive-labelled series in the store also carries an agent label, so a
hive is only ever visible as the sum of its agents — and a hive whose c0re
has stopped is indistinguishable from one that simply hosts none.
Adds three instruments to the exporter hive-c0re already runs, each a
projection of a value the process computes anyway: process.uptime (the
semconv name — the spec defines it as a double gauge in seconds, which is
exactly this instrument), hyperhive.hive.degraded, and
hyperhive.hive.warnings split by level. None carries an agent attribute;
that absence is what makes them selectable as hive-scoped.
The health pair reads warnings::readiness() rather than deriving its own
verdict, and degraded ships as a series instead of being left for a
dashboard query to compute from warnings{level="crit"} — either would put
the "what counts as unhealthy" rule in a second place that disagrees
silently the first time a degrading condition is added.
mara, on review: forge and matrix are swarm-level too (both are
swarm-wide singletons per their own module comments - one Forgejo,
one Matrix homeserver, not one per hive), and the single combined tree
made the host separation unclear. Split into two trees: everything a
hive host always runs, and everything that runs once per swarm on
whichever host opts in (can be the same host or a different one).
The diagram was single-hive only - no mention of swarm-controller,
swarm-ui, authelia SSO, the metrics/log stores, Grafana, NATS or the
swarm-level otel collector, despite all of them being real, shipped
services (several on by default via enableAllLocalDefaults). Adds a
swarm section alongside agent containers, same terse one-line-per-
service style the existing optional-containers section already uses.
swarm-ui gets a note that it has no own container (served straight
from the gateway's nginx, unlike every sibling swarm service).
Adds services.hyperhive.swarm.victorialogs.domain and a gateway vhost
gated by the same auth_request check against authelia that swarm-uis
own vhost uses (same swarmAuthRequest shape, copied not shared - see
the file top comment for why). The stores own listener stays
loopback-only and unauthenticated exactly as before; the collector
still writes to it directly, never through this vhost, so this only
adds a new authenticated read path.
Registers the new domain in swarm.nixs serviceDomains so the
swarm-services sub-CA issues for it (a missed entry silently falls
back to the hive leaf, which cannot cover a name under a different
apex - swarm.nixs own comment on that list documents the incident
this caused before).
Verified with a throwaway module-eval (same technique as the flake
module-eval check): vhost only exists when victorialogs.enable is
set, forceSSL/no addSSL, both locations present, domain correctly
registered/absent from serviceDomains, links entry present.
LinksMenu and SettingsMenu triggers mixed a full-colour emoji (link)
with a plain text glyph (gear) - different rendering paths mean
different, unfixable sizes/styles. Replace both with matching inline
SVG icons (feather/lucide gear + link glyphs), same viewBox/stroke/
size, so the two buttons finally share one rendering path.
Also: shared base.css never zeroed the default UA body margin, which
showed as a bg-coloured strip around the whole viewport edge on any
full-bleed header (swarm-uis .shell-header among them). Zeroed it in
the shared file so every consumer (dashboard, agent UI, swarm-ui) gets
the fix, not just swarm-ui.
Screenshot-verified at both desktop and phone widths.
Grafana could reach the swarm's metrics and not its logs, so the store that
landed with the collector pipeline had no reader.
Adds the VictoriaLogs datasource plugin and the Logs Drilldown app, and
provisions the datasource beside the metrics one. The drilldown matters as much
as the connection: an unfamiliar log stream is explorable without writing a
LogsQL query first, which is the difference between a store you can query and
one you can use.
The secrets page discussed 'the telemetry collector' as a reader needing no
delivery, but there are two: the hive's is a host unit and reads authelia's
file in place, while the swarm's runs in a container and gets a copy placed by
a host oneshot.
States plainly that the container one has no operator-provided variant, which
is a consequence of it running beside authelia rather than a gap.
nixpkgs' collector unit already sets SupplementaryGroups to systemd-journal
unconditionally, with a comment saying why. Systemd list options concatenate,
so this module's copy rendered ["systemd-journal" "systemd-journal"] and made
this a second owner of a fact upstream may later change.
The bind mount stays, since that half is genuinely ours.
The logs half was gated on the swarm's log store being enabled, so turning
that store off stopped collection entirely rather than leaving the upstream
export. Logs now fan out exactly as metrics do: the store when it runs, the
operator's upstream when one is configured, both when both.
The receiver, the journal mount and the group grant follow whether there is
anywhere to send logs, not whether the local store exists. Tying the mount to
the store instead would render a receiver that can read nothing.
Deferred until the collector had a logs pipeline writing to it: a store
nothing writes to starts, answers queries and returns nothing, so the first
person to look concludes there were no logs rather than that nothing was
collecting them.
The journald receiver leaves the OTLP body empty and carries the entry as a
map of journal fields, so VictoriaLogs had no message to index and wrote a
placeholder into _msg on every record. Ingest returned 200, every field was
present, and a plain search for a line sitting in the store found nothing.
_msg_field names the field that holds the text. _stream_fields is the
difference between one stream for the whole host and one per unit per
machine; both are set by journald itself and both are low-cardinality.
A journald receiver reading the host's journal directory, a logs pipeline
stamped with the swarm tier's own resource processor, and an otlphttp
exporter pointed at VictoriaLogs. All four parts are conditional on the log
store being enabled, so a swarm without one renders exactly as before.
The host's directory is enough to see every container: nspawn links a
non-ephemeral container's journal guest-side, so the files live on the host
under the container's machine-id, and journalctl descends into those
subdirectories. Measured, along with the fixed systemd-journal gid that makes
the group grant meaningful across the bind mount.
An agent can verify that a unit was launched and never that it is
working: container journals are not reachable, so a diagnosis stops at
the first broken component -- which is precisely the component whose own
instrument is least likely to be legible. This is the store half of
collecting logs centrally so the question becomes answerable.
Mirrors swarm-victoriametrics deliberately: same container shape, same
loopback pin, same self-scrape arrangement. Two differences, both
intentional.
No gateway vhost. VictoriaLogs' ingest and query endpoints carry no
authentication of their own, exactly like the metrics store's -- and the
metrics store IS published under a resolvable name, which is an open
question rather than a settled design. Publishing this one the same way
would repeat that before the first instance is decided.
Retention defaults to 30d against the metrics store's 5y. Logs are
orders of magnitude larger per unit of time and their value decays much
faster: a log line answers what happened during an incident, a metric
answers whether this is worse than last quarter.
Not wired into enableRequiredServices yet -- that lands with the
collector pipeline, so we do not start a store nothing writes to.
prometheus-nats-exporter has never served a metric. Upstream's module
renders `-addr … -port … ${extraFlags} ${url}` and defaults extraFlags
to the empty list, but the binary refuses to start without at least one
collector: it exits 1 with "no Collectors specified". So the unit logged
Started, the process was gone milliseconds later, and every scrape was
refused -- up=0 continuously, scrape_duration 0.6ms, zero samples.
Verified by running the exact argv both ways: without a collector it
exits 1, with -varz it stays up and logs the listener. A bogus flag is
rejected with exit 2, so the check measures acceptance rather than
tolerance.
varz is the server itself, connz makes a client that will not stay
connected visible, and jsz covers JetStream, which this swarm uses for
the status KV. The rest describe a clustered deployment we do not have.
An empty list is not a neutral default when the program requires a flag,
and nothing about it is visible to evaluation -- which is why the gate
now reads the rendered ExecStart rather than only asserting enable.
The swarm-services board was mostly authelia, so it becomes its own page
and is trimmed out of that one (mara, #3494). Eleven panels: uptime,
authentications and failures, authorization decisions, requests and
verdicts by status code, request and OIDC latency, and the three Go
process signals.
OIDC latency gets its own panel rather than being folded into the
general one because authelia keeps a separate duration family for it,
and every machine-to-machine credential in this swarm is minted through
those endpoints -- averaging the two produces a number describing
neither.
Counters are counted over the dashboard range rather than rated. At this
volume a rate window contains no requests, so rate() returns zero and
draws a flat line, which is indistinguishable from a broken query; the
quantile panels are worse, because a quantile over all-zero buckets is
NaN and renders empty rather than zero.
Part of #3591 (mara: "pop ups and menus appearing should animate").
Each popover mounts fresh on open ({open ? <div> : null}, not a state
transition), so a keyframe animation on the popover element itself is
the right tool -- same shape as Shell.css's own shell-page-enter
(fade + a slight translate/scale settle), including the identical
three-rule motion-guard (base rule, prefers-reduced-motion media
query, data-motion=reduce/allow explicit overrides).
Scope: LinksMenu, SettingsMenu, UserMenu -- the three header popovers.
Not included here (posted findings on the issue instead of guessing):
the refresh-interval picker's dropdown (native <select>, whose open
popup is OS/browser chrome outside CSS reach in current browsers --
"not themed" is a platform limitation, not a bug in this component's
own styling) and the jobs graph's node animations (JobqGraph is a
@hive/shared component consumed by both swarm-ui and the per-hive
dashboard, real design/implementation work on shared infra, not a
same-shape mechanical extension of an existing pattern).
Screenshot-verified the settled (post-animation) state renders
correctly; a static screenshot cannot show an in-flight CSS animation,
so this leans on exact structural parity with the already-shipped
Shell.css pattern for the animation's own correctness.
