Review catch: this PR relaxed the `otel.endpoint` assertion and staled the
canonical OTEL reference in the same stroke — `docs/observability.md` is
what CLAUDE.md points readers at for "what OTEL options are available",
and it still said required-full-stop while the new swarm/services.md
section said a local store satisfies it.
Also corrects the option's own description in otel.nix, which said the
same thing and renders into the generated options doc. Grepping the
reviewer's phrasing did not find that one; grepping the claim did.
Records the second destination where the "endpoint is where telemetry
ultimately goes" paragraph makes its claim, rather than only in the new
section a reader may not reach.
Review feedback: the metrics pair had nothing writing into it, and it sat
outside the switch that turns on every other swarm-wide service.
The collector now exports to VictoriaMetrics as well as upstream — a
fan-out, not a choice: a local store is for looking at this swarm, an
upstream is for whoever aggregates across swarms.
That makes a local store a complete destination on its own, so
`otel.endpoint` is no longer required when it runs here; a hive with
neither is still refused. The assertion only ever relaxes, so every
config that evaluated before still does.
`enableRequiredServices` now derives both halves, alongside matrix,
authelia and nats. They derive together because a store with no UI is
unreadable and a UI with no store is empty.
Second half of the metrics pair: a `swarm-grafana` container beside the
VictoriaMetrics store, provisioned with it as the default datasource and
fronted by the gateway on its own swarm-sibling name.
Behind swarm SSO, per the operator's call on #3265. The authelia client
and Grafana's callback URL both derive from `domain`, so the exact-match
string authelia checks cannot drift from the one Grafana sends. The
minted secret is delivered host-side (both container trees are only
addressable there) and reaches Grafana as a `$__file{}` reference rather
than a value, so it never enters the store.
The login form is disabled whenever SSO is configured: Grafana ships an
`admin`/`admin` account and this vhost is on the public gateway.
A local time-series database rather than only an external sink, so the
swarm dashboard stays readable when the outside world is not: a view of
the system must not depend on the system it views being healthy.
listenAddress is pinned to loopback. Upstream defaults it to every
interface, and the OTLP ingest path this exists to receive on is
unauthenticated — the gateway is the only intended client and it is on
this host, so a wider bind would publish a write endpoint to whatever
the host is reachable on.
retentionPeriod defaults high rather than being required, because the
two failure directions are not symmetric: too long fills a disk, which
is visible and recoverable by lowering it, while too short destroys
history silently and permanently. The operator lowers it once they have
measured how fast this swarm accumulates.
OTLP needs no flag. Measured against the pinned 1.146.0 rather than
inferred from the module's option list, which has no OTLP switch and so
reads as though the feature were missing: the running server answers
POST /opentelemetry/api/v1/push with 200, where a nonexistent path
answers 400.
mara, follow-up on PR#3365: "follow up with the same for jobq". Drops
mountJobqGraph the same way mountJobqRollup was dropped - builds.js
now calls render(h(JobqGraph, {...}), root) directly, split into
mountRebuildQueue() (the one-time replaceChildren() + first render,
called once at page init) and renderRebuildQueue() (the render-only
path the rebuild_queue_changed handler and the cancel flow reuse,
bumping a module-level jobqGraphToken instead of holding a mount
handle). The split matters here specifically: repeating
replaceChildren() on every refresh would wipe Preact's own tracked
children out from under its diffing instead of letting it update them
minimally - JobqRollup's simpler version didn't need this since it
only ever renders into its own dedicated section once per app
lifetime's worth of state, but the rebuild queue refreshes on every
`rebuild_queue_changed` tick.
Re-verified: npm run build (whole workspace) + swarm-ui typecheck
clean (JobqGraph is also used via JSX on swarm-ui's /jobs page,
untouched by this), comment-block + issue-ref lints clean, headless-
chromium screenshot of builds.html's R3BU1LD QU3U3 tab against a
mocked /api/jobq/graph payload - tree, filter checkboxes, and cancel
buttons all render identically to before.
mara, on review: expected the plain Preact pattern (render(h(Widget,
props), container), call again to update) rather than a custom
mountX() returning {refresh(), update()}. Preact's own render is
already the re-render/diff entry point, so the wrapper was indirection
this component didn't need - swarm.js (plain .js, no JSX pragma
required for h()/render() either) now calls render(h(JobqRollup,
{...refreshToken}), root) directly, bumping a module-level token to
force a refetch instead of holding a mount handle.
JobqGraph/mountJobqGraph (a separate, already-merged component) is
untouched - out of scope for this PR, flagged as a possible follow-up
if she wants the same simplification there.
Re-verified: npm run build (whole workspace) + swarm-ui typecheck
clean, comment-block + issue-ref lints clean, re-screenshotted the
dashboard SW4RM tab against the same mocked payload - identical
render, spinner now visibly mid-rotation in the frame (confirms the
animation is live, not just present in markup).
The extraction dropped the old banner's `spinner` class on the glyph
span (`el('span', { class: 'glyph spinner' }, ...)`) - JobqRollup.tsx
rendered a static `.jqr-glyph` with no animation rule anywhere.
`.spinner` (shared/base.css) is already imported by both consumers, so
just reuse it on the glyph rather than duplicating the keyframes.
argus caught this on review - a single-frame screenshot can't tell a
frozen spinner from a missing one, which is exactly why it slipped
past the PR's visual verification.
New @hive/shared/jobq-rollup.js (JobqRollup.tsx + jobq-rollup.css),
mirroring JobqGraph's shape exactly: JSX use plus an imperative
mountJobqRollup(container, props) for a plain-.js call site. Fetches
Vec<hive_jobq_wire::StateCount> off `endpoint`, sums Running+Finishing
roots as "running" and Pending roots as "queued", renders nothing when
both are zero. Optional `queueHref` adds a "view queue -> " link.
Swapped dashboard's hand-rolled queue-summary banner (swarm.js) over to
this component instead of keeping two parallel implementations - same
"one shared component" pattern JobqGraph already set for the rebuild
queue tree view. Mounted once into a new #jobq-rollup-section, kept as
a sibling of (not inside) #containers-section since that section gets
replaceChildren()-wiped on every container-state render, which would
tear down and remount a Preact tree on every tick. Refreshed via the
mount handle's .refresh() on rebuild_queue_changed, same as builds.js's
JobqGraph handle.
Also mounted in swarm-ui's /jobs page, above JobqGraph, with no
queueHref (a link back to the page you're already on is noise) - the
literal ask on hyperhive#3364.
Verified: npm run build (whole workspace) and swarm-ui typecheck both
clean, comment-block + issue-ref lints run manually, headless-chromium
screenshots of both the dashboard SW4RM tab and swarm-ui's /jobs page
against mocked /api/jobq/rollup payloads - banner renders identically
in both, with and without the queue link as expected.
Three host units poll up to 120s for a secret authelia mints on its first
boot, and all three are `Type=oneshot` with no `TimeoutStartSec`. systemd's
`DefaultTimeoutStartSec` is 90s, so it kills them at 90 — before the script
reaches its own `exit 1` and names the file that never appeared.
The wait itself is fine; what's lost is the diagnosis. On a fresh hive the
operator gets a bare start-timeout instead of "authelia has not minted
<path>", several layers from the container that was actually slow.
Found while writing the same unit for Grafana, where the timeout is set —
so this is the existing three catching up with it, not a new pattern.
The auth callback minted on every invocation, and async-nats runs it per
connection ATTEMPT. Its default reconnect delay backs off exponentially
and then clamps at four seconds, permanently. So a queue that could not
connect asked authelia for a token every four seconds indefinitely, and
authelia answered 429 Too Many Requests.
That rate limit then keeps itself alive: it outlives whatever first
broke the connection, and its log volume buries the original cause.
Observed in production tonight, once a redeploy fixed the TLS trust
problem that had been hiding it.
The callback now caches the token and re-mints only within two minutes
of expiry, which preserves the property that put the mint there — a
reconnect must never present a token that expired since it was minted —
while decoupling mint rate from retry rate. The reconnect cap moves to
one minute, exponential from half a second, so a brief outage still
recovers promptly and a persistent one stops hammering the identity
provider.
The module doc and the retry comment both argued for the old shape, so
both are corrected rather than left explaining why the bug was right.
Defining an environment key on a unit that does not exist creates a
fragment for it: inert, never activated, but present on every
non-controller hive that has a CA. hive-c0re runs everywhere so its
line needs no guard; the controller runs on one host in a swarm.
Caught in review. It evaluates and builds clean either way, which is
why it took a reviewer rather than a check.
The queue client built a bare reqwest::Client, so it trusted only the
platform roots. Against a swarm whose authelia is signed by the swarm
CA that is fatal: minting a token dies with 'invalid peer certificate:
UnknownIssuer', inside the auth callback, on a four-second retry loop,
with the queue never connecting.
The anchor was never missing. hive-tls.nix assembles trust-bundle.pem
and already hands it to hive-c0re as HIVE_TLS_CA_PATH; nothing pointed
the queue client at it.
QueueConfig gains an optional ca_file from <prefix>_OIDC_CA_FILE, read
outside the all-or-none tuple on purpose: a CA path with no queue is
meaningless rather than half-configured, and requiring it would break a
swarm fronted by a public certificate in order to fix one that is not.
add_root_certificate extends the default roots rather than replacing
them, so both deployments work.
A bad path fails loudly instead of falling back to the platform roots.
An operator who names a CA file wants that anchor; a silent fallback
turns their typo into UnknownIssuer five layers away.
hive-tls.nix names the bundle for both clients, beside the line that
already does it for hive-c0re, rather than having each consumer
re-derive the path.
The responder's third credential is not minted by the generator beside
it — authelia produces it on first boot, inside its own container, and
nothing ordered this unit against that. Losing the race cost the entire
queue: install exits 1, the responder never starts, and auth_callout
with no responder refuses every client. Fail-closed by design, so the
symptom lands on every queue client and nowhere near the cause. Seen on
a real boot at 20:31:31.
Now it waits for the file, bounded at two minutes. Where authelia runs
on another host the secret is never going to appear, and blocking the
queue container indefinitely would replace a clear failure with a hang;
after the timeout this fails exactly as it did before, having first
given the co-located case the seconds it needs.
TimeoutStartSec is set rather than left to the default because the
default is ninety seconds — a two-minute wait would be killed at ninety
and the operator would get a generic unit timeout instead of the
message naming the file.
Ordering after the authelia container is necessary and not sufficient:
the container being up says nothing about whether its in-container
secrets unit has finished. It only stops this spinning for the full
timeout on every boot.
A systemd credential named by an absolute path is fatal when the file is
missing, and the co-located queue secret is minted by authelia's first
boot in another container — which a host unit cannot order against. On a
real boot the daemon spent three of systemd's five default starts losing
that race before the file appeared; two seconds more would have reached
start-limit-hit, which does not self-heal.
An empty SetCredential acts as the default that makes the load
non-fatal, so the controller starts and serves its HTTP surface with the
queue unconfigured — a shape it already reports.
That alone would be a regression, though: a credential is snapshotted at
unit start, so the placeholder would freeze and the daemon would sit
degraded forever instead of recovering the way the restart loop
accidentally did. A path unit on the secret file closes it, and closes a
second gap in the same stroke — mint_token reads the secret on every
call precisely so a rotation takes effect without a restart, and a
snapshot in %d silently defeats that.
PathChanged and not PathExists: the latter activates immediately when
the file is already present at unit start, which would restart a healthy
daemon on every boot.
The option being consumed already warned about this shape — its own
description says a consumer has to wait for the secret because a
missing source turns a fresh hive into a boot-order deadlock. That
warning was written on the producing side and did not fire while I was
writing the consumer.
The client list was `cfg.oidc.clients ++ hiveClients`, where the first
half comes through the submodule and the second was a raw attrset from
this module's `let` block. That list is only half-typed: a field added
to the submodule exists on the declared entries and not on the derived
ones, so reading it plainly is an eval error the moment hive identities
are on. The operator asked whether it should be uniformly typed instead
of guarding each read, and it should.
The hive identities are now declared the same way an operator declares
a client, so the module system applies the submodule to them and every
option's default is present. Downstream reads one uniformly-typed list
and the guard added for the field that broke is gone with it.
allClients concatenates two shapes: cfg.oidc.clients comes through the
submodule and carries every option default, hiveClients is a raw attrset
built in the let block with four fields. renderClient read
tokenEndpointAuthMethod plainly, which is fine for a declared client and
an eval error for a derived one -- so all-local, where hiveIdentities is
on, stopped evaluating.
Read it with `or null`. The comment records that the list is not
uniformly typed, because the next field added to that submodule has the
same trap waiting.
enableRequiredServices asserts the services that exist once per swarm and
are optional -- its own description says so -- and the queue meets that
rule. It was left out because this file predates the swarm-nats container
by nine days and was never revisited, not because anyone decided against
it.
The all-local mode already derives nats.autoGenerateCallout, so it was
minting the queue's callout nkeys and never starting the queue. With this
the mode's loopback derivation for the controller points at something
that is actually running.
Review point: the co-located defaults are an auto-deploy special case and
belong with the other ones, not inside each option's own default. An
option should describe itself; the mode describes what a deployment shape
implies. local-defaults.nix already says exactly this in its header.
Options now default to empty (= unset, which the assertions refuse), and
enableAllLocalDefaults fills in loopback + the minted secret path.
The controller's queue coordinates were gated on `autheliaCfg.enable &&
natsCfg.enable` -- i.e. on this host running both. Split the hosts and
the whole env block vanished, which the daemon's own rule reads as "no
queue configured": it started cleanly, served /api/hives, and silently
never connected or published.
Make the coordinates options instead, defaulted to the co-located values
so nobody types a path they didn't need, and assert each one so a split
deployment fails at eval naming the option rather than running blind.
The queue is required for a controller -- /api/hives/status reads the KV
-- but co-location with it is not.
Answers mara's question on PR#3342 (lightweight dep for fmtAgo?): no
dep needed, Intl.RelativeTimeFormat is built into the runtime and its
narrow style produces the same '5m ago' shape, verified with a real
call rather than assumed from the spec.
Per mara's review on PR#3342 — the hive-status view shouldn't live
inline in App.tsx. Moves it to pages/OverviewPage.tsx, mirroring
JobsPage's shape: App.tsx stays routing-only, each page owns its own
fetch + render.
App.tsx now fetches GET /api/hives/status (the swarm-controller
aggregate: one row per roster hive, freshness derived at read time
from the status bucket) instead of GET /api/hives + a static
'configured' chip. Renders fresh/stale/never_reported/unknown as
StatusChip tones with a relative age, per the placeholder comment that
was already waiting on this endpoint to exist.
Adds a small local fmtAgo helper (src/util.ts) mirroring the
dashboard package's near-identical formatter — not worth sharing
across a vanilla-JS and a Preact/TS call site.
An SSO login whose derived localpart already exists silently gets a
random one instead: the candidate must be *available*, an existing
unlinked account is not, and `unique_id_fallbacks` defaults to true.
The operator's own login minted a stray account this way.
Fill the three identity_provider fields that decide which account a
login lands on: `trusted` (upstream's precondition -- self-hosted and
fully controlled -- is structural here, the issuer is always the swarm's
own authelia), `userid_claims` pinned to preferred_username so the
default ladder cannot fall through to an email local part, and
`unique_id_fallbacks = false` so a collision errors instead of
succeeding as the wrong user.
tuwunel authenticates at the token endpoint with the secret in the POST
body. Authelia enforces the method a client is REGISTERED with rather
than accepting whichever one arrives, and its default is
client_secret_basic — so the matrix login completed, consent was
granted, and the very last hop failed:
Client authentication failed ... The request was determined to be
using token_endpoint_auth_method client_secret_post, however the
OAuth 2.0 client registration does not allow this method.
The failure names neither the secret nor the redirect, and it lands
three layers from its cause, which is why it read as a credential
problem.
Adds a per-client tokenEndpointAuthMethod, null by default so every
existing client keeps authelia default (forgejo authenticates with
basic and is unaffected), and sets client_secret_post on the matrix
client only.
Both introduced by this PR's own JSX->TSX conversion, caught by argus:
the top-of-file comment grew to 42 lines converting it, and JobsPage.css
still pointed at 'JobqGraph.jsx's own comment' after the rename.
New /jobs route in swarm-ui, reusing the shared JobqGraph component
against swarm-controller's own GET /api/jobq/graph (same wire shape
hive-c0re's dashboard already consumes, different endpoint, no fork).
Converted JobqGraph.jsx to JobqGraph.tsx with real prop/wire types
(mirrors hive_jobq_wire's GraphNode/GraphDep/State by hand) instead of
a hand-maintained ambient .d.ts at the swarm-ui consumer side — the
.d.ts would duplicate the prop list and drift from the source the
moment the component's signature changes without the declaration being
touched. Both dashboard (untyped consumer, esbuild strips types) and
swarm-ui (tsc --noEmit) build/typecheck clean off the one file.
Same pattern frontend.nix already uses for the dashboard: the SVG
lives outside the npm tree at branding/hyperhive.svg, copied into
$out during the nix install phase (not build.mjs, so plain npm build
consumers still work, same split frontend.nix uses). Added the
matching <link rel="icon"> to index.html.
Review: the public keys are not an exception, the paragraph was
incomplete. It is private key material and access tokens that are
file-based so they stay out of the nix store; public material in a
public store is fine.
Fixes the rule itself, which removes the need for the special case the
previous commit added.
Review: high comment-to-code ratio. Measured — 197 of 337 added lines
were comments, and net of the block that only moved it was still +135
comment against +117 code.
Kept the constraints, the rejected alternatives and the costs; dropped
everything that restated what the code plainly does, and the
slash-stripping explanation that was duplicated in the generator.
secrets.md is the page that answers "who mints this and where does it
go" per topology, and the callout nkeys had no row. Adds one, plus the
`nk` invocations an operator needs on every topology that is not
all-local.
The note exists because the change provokes a question the page would
otherwise answer wrongly: this page opens with "a secret is a path,
never a value", and the queue takes two public keys as values. A reader
who meets those after reading that line concludes the page is stale.
A hive whose whole swarm is one box had to be handed two nkeys by hand
before its queue could authenticate anyone, which is the one deployment
shape where nobody else can supply them.
`autoGenerateCallout` mints both keypairs on the host on first boot,
keeps the seeds at 0600 host-side, and writes only the public halves
into a fragment the server reads at start. The all-local mode turns it
on; everywhere else the options stay operator-supplied and the
fail-closed eval assertions keep their full force.
The server config is not rewritten to do this. A wrapper includes
upstream's rendered `settings` verbatim plus the runtime fragment, and
the fragment wins — measured, along with the property that makes the
whole shape safe: the empty strings the options render in auto mode are
values `nats-server` refuses to start on, so any field the merge fails
to reach fails closed loudly rather than leaving a walk-in-able server.
The wrapper, the settings symlink and the fragment are siblings in one
runtime directory, and that is forced rather than tidy: NATS resolves an
include with filepath.Join against the config file's own directory,
which strips a leading slash, so an absolute include silently becomes a
relative one and the server never finds it. The includes are therefore
bare filenames. That also means nothing in the closure would otherwise
name the rendered settings, so the generator's symlink to it is what
keeps it from being garbage-collected under a running server.
`accounts` and `authorization` are defined once and rendered twice, into
`settings` and into the fragment template. Written out separately they
would diverge silently and backwards: the fragment is the later
definition, so a future edit to `settings` alone would be ignored on
exactly the hives that use auto mode.
`validateConfig` goes off in auto mode because `nats-server -t` rejects
the empty keys at build time; the parse check moves to server start,
where the fragment exists. Upstream's own option description names this
case.
frontend.nix stays the one authoritative explanation; swarm-ui.nix and
checks.nix now just point at it instead of each restating the
prefetch-npm-deps command in their own words.
The hash was hardcoded independently in nix/packages/frontend.nix,
nix/packages/swarm-ui.nix, and nix/checks.nix's inline
swarm-ui-typecheck derivation, all three building from the one
frontend/package-lock.json. Nothing enforced the three copies staying
in sync, and on a recent PR only some of them got updated when the
lockfile changed.
Moved the hash into a new file, frontend/npm-deps-hash (plain text, no
trailing newline, co-located with package-lock.json so it reads as
the lockfile's other half), and all three derivations now
builtins.readFile it instead of hardcoding their own copy. A lockfile
change now only needs prefetch-npm-deps + one file overwrite; the
other two derivations pick it up automatically.
Verified: nix eval against all three derivations' npmDepsHash
attribute (eval-only, not a build) confirms all three resolve to the
same value read from the one file.
fmt_args_generic's generic array handling collapsed `ids: [4]` — the
count — since mark_todos_done had no dedicated match arm. Added one,
matching the file's existing per-tool pattern (extracted into its own
helper to stay under the 100-line clippy limit on fmt_hyperhive_tool).
frontend/package-lock.json changed (preact added to dashboard + shared
in the earlier commits on this branch) but the fixed-output npm-deps
hash pinned in nix/checks.nix, nix/packages/frontend.nix, and
nix/packages/swarm-ui.nix (all three hash the one shared lockfile,
per checks.nix's own comment) wasn't updated to match — argus caught
it on review. Recomputed with nix run nixpkgs#prefetch-npm-deps.
