Relative hrefs (static/main.js) only resolve correctly for a route
one path segment deep - the browser resolves them against the
current URL's directory, so a route two-plus segments deep would
404 the JS/CSS silently once this is wired to a real server. Made
absolute (/static/main.js), matching what build.mjs's own comment
already documents these paths as being served at.
mara: the route should reflect creating an agent, and stay separate
from a future agent list page. Renamed AgentsPage -> CreateAgentPage
(file, component, css classes) and moved the route from /agents to
/create-agent - flat, not /agents/new, since index.html's relative
asset links only resolve correctly one path segment deep (filed
separately as a real bug, not fixed here). Leaves the bare /agents
path free for a future roster page.
Adds /agents: a name field that POSTs to swarm-controller's
POST /api/agents (from the CreateIdentity work), shows the queued
job node id, and links to /jobs to watch it settle. Scope matches
the issue exactly - no forge/deploy options, those aren't wired
server-side yet.
The auth-callout responder learned whether a token was valid, never whose
it was, so every admitted client got the same unscoped grant. Scoping a
grant to one hive's subjects needs an identity to scope it to.
`is_active` becomes `identify_caller` and returns the identity rather
than a boolean: `Ok(Some(client_id))` admits as that client, `Ok(None)`
denies. `active: true` with no `client_id` is a denial, and returning an
identity is what makes that unrepresentable instead of remembered --
there is no admitted-but-unscoped value to construct, so there is no
branch a later edit can forget to handle.
The module deliberately ignores `sub`/`scope`/`exp` on the grounds that
modelling a field implies checking it. That still holds: `active` remains
the whole admission decision, made in one place. `client_id` answers a
different question -- as whom -- and is used downstream of an admission
that has already happened, not as a second gate. The module docs say so,
next to the paragraph that would otherwise argue for deleting the field.
An inactive token has no identity even when the body names one, and an
empty `client_id` counts as absent: it would become a blank component in
a subject the grant is scoped to, which is not a narrower permission but
a different one. The two credential parameters are renamed `own_*` --
this responder's introspection credential is not the caller's identity,
and the function now takes one and returns the other.
The grant itself is still unscoped; narrowing it is the next slice.
Per the operator's call on #3372: rather than moving Grafana off the
forge's 3000 to another number, take it off TCP entirely.
The collision was possible because every swarm service container shares
the host's network namespace, which makes a port a swarm-wide resource
two modules can each claim believing it free — Grafana took upstream's
3000, so does the forge, and `grafana.<swarm-domain>` served the forge
with no bind error and nothing in any log. A socket has a path, and a
path collision is a build-time conflict rather than a runtime coin toss.
Three parts, none of which works alone:
- `protocol = "socket"` with `socket_gid` = nginx's static gid 60.
- Grafana joins that gid inside the container. A non-root process may
only chgrp to a group it belongs to, and a container has its own user
database — without the membership Grafana starts, the chown fails, and
the socket is simply unreachable.
- The socket dir is created host-side by tmpfiles and bind-mounted in. A
container's /run is an nspawn tmpfs, so it is not visible from the host
at /var/lib/nixos-containers/<name>/run; the same shape the per-agent
web.sock already uses.
The `port` option is gone rather than deprecated — nothing can set it to
a colliding value if it does not exist.
`swarm-controller.service` carried `SetCredential=queue-client.secret:` — an
empty value, which systemd's parser refuses:
/etc/systemd/system/swarm-controller.service:38:
Invalid syntax, ignoring: queue-client.secret:
So the line was dropped on every daemon-reload, and `LoadCredential=` was
fatal again — precisely the failure that default was added to prevent. On a
hive where authelia has not yet minted the secret, the controller refuses to
start rather than coming up with the queue unconfigured.
It looked correct for days because the credential file happened to exist, so
the fail-soft was never exercised.
Measured with `systemd-analyze verify`: an empty value is rejected, any
non-empty one is accepted. The placeholder is a real word rather than filler —
it reaches the token request as the client secret, so authelia refuses it and
the journal names something an operator can act on.
Gate: state/eval-setcredential.sh, with the parent commit as its mutation.
nixpkgs dropped the default for `services.grafana.settings.security.secret_key`
and asserts on null, so the module I merged an hour ago fails the build on any
host that enables it. That is a broken deploy, not a warning.
Generated in-container like authelia's own keys and for the same reason:
nothing outside the container ever reads it. Generated ONCE and kept — the
key signs Grafana's stored datasource secrets, and rotating it does not
re-encrypt what it already encrypted, so a fresh key per boot would leave
Grafana unable to read its own database. Delivered as `$__file{}`, so it
never enters the store.
The gate missed this because a container is a separate NixOS evaluation with
its own assertions: forcing the host's `config.assertions` never touched the
container's. `state/eval-3265.sh` now forces
`containers.swarm-grafana.config.assertions` and reports any that fail.
Review catch: this PR relaxed the `otel.endpoint` assertion and staled the
canonical OTEL reference in the same stroke — `docs/observability.md` is
what CLAUDE.md points readers at for "what OTEL options are available",
and it still said required-full-stop while the new swarm/services.md
section said a local store satisfies it.
Also corrects the option's own description in otel.nix, which said the
same thing and renders into the generated options doc. Grepping the
reviewer's phrasing did not find that one; grepping the claim did.
Records the second destination where the "endpoint is where telemetry
ultimately goes" paragraph makes its claim, rather than only in the new
section a reader may not reach.
Review feedback: the metrics pair had nothing writing into it, and it sat
outside the switch that turns on every other swarm-wide service.
The collector now exports to VictoriaMetrics as well as upstream — a
fan-out, not a choice: a local store is for looking at this swarm, an
upstream is for whoever aggregates across swarms.
That makes a local store a complete destination on its own, so
`otel.endpoint` is no longer required when it runs here; a hive with
neither is still refused. The assertion only ever relaxes, so every
config that evaluated before still does.
`enableRequiredServices` now derives both halves, alongside matrix,
authelia and nats. They derive together because a store with no UI is
unreadable and a UI with no store is empty.
Second half of the metrics pair: a `swarm-grafana` container beside the
VictoriaMetrics store, provisioned with it as the default datasource and
fronted by the gateway on its own swarm-sibling name.
Behind swarm SSO, per the operator's call on #3265. The authelia client
and Grafana's callback URL both derive from `domain`, so the exact-match
string authelia checks cannot drift from the one Grafana sends. The
minted secret is delivered host-side (both container trees are only
addressable there) and reaches Grafana as a `$__file{}` reference rather
than a value, so it never enters the store.
The login form is disabled whenever SSO is configured: Grafana ships an
`admin`/`admin` account and this vhost is on the public gateway.
A local time-series database rather than only an external sink, so the
swarm dashboard stays readable when the outside world is not: a view of
the system must not depend on the system it views being healthy.
listenAddress is pinned to loopback. Upstream defaults it to every
interface, and the OTLP ingest path this exists to receive on is
unauthenticated — the gateway is the only intended client and it is on
this host, so a wider bind would publish a write endpoint to whatever
the host is reachable on.
retentionPeriod defaults high rather than being required, because the
two failure directions are not symmetric: too long fills a disk, which
is visible and recoverable by lowering it, while too short destroys
history silently and permanently. The operator lowers it once they have
measured how fast this swarm accumulates.
OTLP needs no flag. Measured against the pinned 1.146.0 rather than
inferred from the module's option list, which has no OTLP switch and so
reads as though the feature were missing: the running server answers
POST /opentelemetry/api/v1/push with 200, where a nonexistent path
answers 400.
mara, follow-up on PR#3365: "follow up with the same for jobq". Drops
mountJobqGraph the same way mountJobqRollup was dropped - builds.js
now calls render(h(JobqGraph, {...}), root) directly, split into
mountRebuildQueue() (the one-time replaceChildren() + first render,
called once at page init) and renderRebuildQueue() (the render-only
path the rebuild_queue_changed handler and the cancel flow reuse,
bumping a module-level jobqGraphToken instead of holding a mount
handle). The split matters here specifically: repeating
replaceChildren() on every refresh would wipe Preact's own tracked
children out from under its diffing instead of letting it update them
minimally - JobqRollup's simpler version didn't need this since it
only ever renders into its own dedicated section once per app
lifetime's worth of state, but the rebuild queue refreshes on every
`rebuild_queue_changed` tick.
Re-verified: npm run build (whole workspace) + swarm-ui typecheck
clean (JobqGraph is also used via JSX on swarm-ui's /jobs page,
untouched by this), comment-block + issue-ref lints clean, headless-
chromium screenshot of builds.html's R3BU1LD QU3U3 tab against a
mocked /api/jobq/graph payload - tree, filter checkboxes, and cancel
buttons all render identically to before.
mara, on review: expected the plain Preact pattern (render(h(Widget,
props), container), call again to update) rather than a custom
mountX() returning {refresh(), update()}. Preact's own render is
already the re-render/diff entry point, so the wrapper was indirection
this component didn't need - swarm.js (plain .js, no JSX pragma
required for h()/render() either) now calls render(h(JobqRollup,
{...refreshToken}), root) directly, bumping a module-level token to
force a refetch instead of holding a mount handle.
JobqGraph/mountJobqGraph (a separate, already-merged component) is
untouched - out of scope for this PR, flagged as a possible follow-up
if she wants the same simplification there.
Re-verified: npm run build (whole workspace) + swarm-ui typecheck
clean, comment-block + issue-ref lints clean, re-screenshotted the
dashboard SW4RM tab against the same mocked payload - identical
render, spinner now visibly mid-rotation in the frame (confirms the
animation is live, not just present in markup).
The extraction dropped the old banner's `spinner` class on the glyph
span (`el('span', { class: 'glyph spinner' }, ...)`) - JobqRollup.tsx
rendered a static `.jqr-glyph` with no animation rule anywhere.
`.spinner` (shared/base.css) is already imported by both consumers, so
just reuse it on the glyph rather than duplicating the keyframes.
argus caught this on review - a single-frame screenshot can't tell a
frozen spinner from a missing one, which is exactly why it slipped
past the PR's visual verification.
New @hive/shared/jobq-rollup.js (JobqRollup.tsx + jobq-rollup.css),
mirroring JobqGraph's shape exactly: JSX use plus an imperative
mountJobqRollup(container, props) for a plain-.js call site. Fetches
Vec<hive_jobq_wire::StateCount> off `endpoint`, sums Running+Finishing
roots as "running" and Pending roots as "queued", renders nothing when
both are zero. Optional `queueHref` adds a "view queue -> " link.
Swapped dashboard's hand-rolled queue-summary banner (swarm.js) over to
this component instead of keeping two parallel implementations - same
"one shared component" pattern JobqGraph already set for the rebuild
queue tree view. Mounted once into a new #jobq-rollup-section, kept as
a sibling of (not inside) #containers-section since that section gets
replaceChildren()-wiped on every container-state render, which would
tear down and remount a Preact tree on every tick. Refreshed via the
mount handle's .refresh() on rebuild_queue_changed, same as builds.js's
JobqGraph handle.
Also mounted in swarm-ui's /jobs page, above JobqGraph, with no
queueHref (a link back to the page you're already on is noise) - the
literal ask on hyperhive#3364.
Verified: npm run build (whole workspace) and swarm-ui typecheck both
clean, comment-block + issue-ref lints run manually, headless-chromium
screenshots of both the dashboard SW4RM tab and swarm-ui's /jobs page
against mocked /api/jobq/rollup payloads - banner renders identically
in both, with and without the queue link as expected.
Three host units poll up to 120s for a secret authelia mints on its first
boot, and all three are `Type=oneshot` with no `TimeoutStartSec`. systemd's
`DefaultTimeoutStartSec` is 90s, so it kills them at 90 — before the script
reaches its own `exit 1` and names the file that never appeared.
The wait itself is fine; what's lost is the diagnosis. On a fresh hive the
operator gets a bare start-timeout instead of "authelia has not minted
<path>", several layers from the container that was actually slow.
Found while writing the same unit for Grafana, where the timeout is set —
so this is the existing three catching up with it, not a new pattern.
The auth callback minted on every invocation, and async-nats runs it per
connection ATTEMPT. Its default reconnect delay backs off exponentially
and then clamps at four seconds, permanently. So a queue that could not
connect asked authelia for a token every four seconds indefinitely, and
authelia answered 429 Too Many Requests.
That rate limit then keeps itself alive: it outlives whatever first
broke the connection, and its log volume buries the original cause.
Observed in production tonight, once a redeploy fixed the TLS trust
problem that had been hiding it.
The callback now caches the token and re-mints only within two minutes
of expiry, which preserves the property that put the mint there — a
reconnect must never present a token that expired since it was minted —
while decoupling mint rate from retry rate. The reconnect cap moves to
one minute, exponential from half a second, so a brief outage still
recovers promptly and a persistent one stops hammering the identity
provider.
The module doc and the retry comment both argued for the old shape, so
both are corrected rather than left explaining why the bug was right.
Defining an environment key on a unit that does not exist creates a
fragment for it: inert, never activated, but present on every
non-controller hive that has a CA. hive-c0re runs everywhere so its
line needs no guard; the controller runs on one host in a swarm.
Caught in review. It evaluates and builds clean either way, which is
why it took a reviewer rather than a check.
The queue client built a bare reqwest::Client, so it trusted only the
platform roots. Against a swarm whose authelia is signed by the swarm
CA that is fatal: minting a token dies with 'invalid peer certificate:
UnknownIssuer', inside the auth callback, on a four-second retry loop,
with the queue never connecting.
The anchor was never missing. hive-tls.nix assembles trust-bundle.pem
and already hands it to hive-c0re as HIVE_TLS_CA_PATH; nothing pointed
the queue client at it.
QueueConfig gains an optional ca_file from <prefix>_OIDC_CA_FILE, read
outside the all-or-none tuple on purpose: a CA path with no queue is
meaningless rather than half-configured, and requiring it would break a
swarm fronted by a public certificate in order to fix one that is not.
add_root_certificate extends the default roots rather than replacing
them, so both deployments work.
A bad path fails loudly instead of falling back to the platform roots.
An operator who names a CA file wants that anchor; a silent fallback
turns their typo into UnknownIssuer five layers away.
hive-tls.nix names the bundle for both clients, beside the line that
already does it for hive-c0re, rather than having each consumer
re-derive the path.
The responder's third credential is not minted by the generator beside
it — authelia produces it on first boot, inside its own container, and
nothing ordered this unit against that. Losing the race cost the entire
queue: install exits 1, the responder never starts, and auth_callout
with no responder refuses every client. Fail-closed by design, so the
symptom lands on every queue client and nowhere near the cause. Seen on
a real boot at 20:31:31.
Now it waits for the file, bounded at two minutes. Where authelia runs
on another host the secret is never going to appear, and blocking the
queue container indefinitely would replace a clear failure with a hang;
after the timeout this fails exactly as it did before, having first
given the co-located case the seconds it needs.
TimeoutStartSec is set rather than left to the default because the
default is ninety seconds — a two-minute wait would be killed at ninety
and the operator would get a generic unit timeout instead of the
message naming the file.
Ordering after the authelia container is necessary and not sufficient:
the container being up says nothing about whether its in-container
secrets unit has finished. It only stops this spinning for the full
timeout on every boot.
A systemd credential named by an absolute path is fatal when the file is
missing, and the co-located queue secret is minted by authelia's first
boot in another container — which a host unit cannot order against. On a
real boot the daemon spent three of systemd's five default starts losing
that race before the file appeared; two seconds more would have reached
start-limit-hit, which does not self-heal.
An empty SetCredential acts as the default that makes the load
non-fatal, so the controller starts and serves its HTTP surface with the
queue unconfigured — a shape it already reports.
That alone would be a regression, though: a credential is snapshotted at
unit start, so the placeholder would freeze and the daemon would sit
degraded forever instead of recovering the way the restart loop
accidentally did. A path unit on the secret file closes it, and closes a
second gap in the same stroke — mint_token reads the secret on every
call precisely so a rotation takes effect without a restart, and a
snapshot in %d silently defeats that.
PathChanged and not PathExists: the latter activates immediately when
the file is already present at unit start, which would restart a healthy
daemon on every boot.
The option being consumed already warned about this shape — its own
description says a consumer has to wait for the secret because a
missing source turns a fresh hive into a boot-order deadlock. That
warning was written on the producing side and did not fire while I was
writing the consumer.
The client list was `cfg.oidc.clients ++ hiveClients`, where the first
half comes through the submodule and the second was a raw attrset from
this module's `let` block. That list is only half-typed: a field added
to the submodule exists on the declared entries and not on the derived
ones, so reading it plainly is an eval error the moment hive identities
are on. The operator asked whether it should be uniformly typed instead
of guarding each read, and it should.
The hive identities are now declared the same way an operator declares
a client, so the module system applies the submodule to them and every
option's default is present. Downstream reads one uniformly-typed list
and the guard added for the field that broke is gone with it.
allClients concatenates two shapes: cfg.oidc.clients comes through the
submodule and carries every option default, hiveClients is a raw attrset
built in the let block with four fields. renderClient read
tokenEndpointAuthMethod plainly, which is fine for a declared client and
an eval error for a derived one -- so all-local, where hiveIdentities is
on, stopped evaluating.
Read it with `or null`. The comment records that the list is not
uniformly typed, because the next field added to that submodule has the
same trap waiting.
enableRequiredServices asserts the services that exist once per swarm and
are optional -- its own description says so -- and the queue meets that
rule. It was left out because this file predates the swarm-nats container
by nine days and was never revisited, not because anyone decided against
it.
The all-local mode already derives nats.autoGenerateCallout, so it was
minting the queue's callout nkeys and never starting the queue. With this
the mode's loopback derivation for the controller points at something
that is actually running.
Review point: the co-located defaults are an auto-deploy special case and
belong with the other ones, not inside each option's own default. An
option should describe itself; the mode describes what a deployment shape
implies. local-defaults.nix already says exactly this in its header.
Options now default to empty (= unset, which the assertions refuse), and
enableAllLocalDefaults fills in loopback + the minted secret path.
The controller's queue coordinates were gated on `autheliaCfg.enable &&
natsCfg.enable` -- i.e. on this host running both. Split the hosts and
the whole env block vanished, which the daemon's own rule reads as "no
queue configured": it started cleanly, served /api/hives, and silently
never connected or published.
Make the coordinates options instead, defaulted to the co-located values
so nobody types a path they didn't need, and assert each one so a split
deployment fails at eval naming the option rather than running blind.
The queue is required for a controller -- /api/hives/status reads the KV
-- but co-location with it is not.
Answers mara's question on PR#3342 (lightweight dep for fmtAgo?): no
dep needed, Intl.RelativeTimeFormat is built into the runtime and its
narrow style produces the same '5m ago' shape, verified with a real
call rather than assumed from the spec.
Per mara's review on PR#3342 — the hive-status view shouldn't live
inline in App.tsx. Moves it to pages/OverviewPage.tsx, mirroring
JobsPage's shape: App.tsx stays routing-only, each page owns its own
fetch + render.
App.tsx now fetches GET /api/hives/status (the swarm-controller
aggregate: one row per roster hive, freshness derived at read time
from the status bucket) instead of GET /api/hives + a static
'configured' chip. Renders fresh/stale/never_reported/unknown as
StatusChip tones with a relative age, per the placeholder comment that
was already waiting on this endpoint to exist.
Adds a small local fmtAgo helper (src/util.ts) mirroring the
dashboard package's near-identical formatter — not worth sharing
across a vanilla-JS and a Preact/TS call site.
An SSO login whose derived localpart already exists silently gets a
random one instead: the candidate must be *available*, an existing
unlinked account is not, and `unique_id_fallbacks` defaults to true.
The operator's own login minted a stray account this way.
Fill the three identity_provider fields that decide which account a
login lands on: `trusted` (upstream's precondition -- self-hosted and
fully controlled -- is structural here, the issuer is always the swarm's
own authelia), `userid_claims` pinned to preferred_username so the
default ladder cannot fall through to an email local part, and
`unique_id_fallbacks = false` so a collision errors instead of
succeeding as the wrong user.
tuwunel authenticates at the token endpoint with the secret in the POST
body. Authelia enforces the method a client is REGISTERED with rather
than accepting whichever one arrives, and its default is
client_secret_basic — so the matrix login completed, consent was
granted, and the very last hop failed:
Client authentication failed ... The request was determined to be
using token_endpoint_auth_method client_secret_post, however the
OAuth 2.0 client registration does not allow this method.
The failure names neither the secret nor the redirect, and it lands
three layers from its cause, which is why it read as a credential
problem.
Adds a per-client tokenEndpointAuthMethod, null by default so every
existing client keeps authelia default (forgejo authenticates with
basic and is unaffected), and sets client_secret_post on the matrix
client only.
Both introduced by this PR's own JSX->TSX conversion, caught by argus:
the top-of-file comment grew to 42 lines converting it, and JobsPage.css
still pointed at 'JobqGraph.jsx's own comment' after the rename.
