Commit graph

3,820 commits

Author SHA1 Message Date
damocles
9067ba52a9 swarm-controller: bulk-read endpoint for agent config-PR status 2026-08-19 21:51:55 +02:00
atlas
1f24193609 feat(swarm-victoriametrics): scrape the store's own prometheus endpoint
The scraper shipped with no targets, so nothing exercised it. This is its
first user, and the one with the least new surface: victoriametrics
publishes prometheus metrics on the listener it already serves queries on,
so there is no exporter, no extra port and no new reach — the collector's
otlphttp exporter already writes to that same loopback address.

Declared from this module rather than the collector's, per the option's own
rule: an entry exists only where the service that named it runs.
2026-08-19 21:40:40 +02:00
atlas
ff5c76c42f feat(#3520): scrape swarm-service prometheus endpoints at the swarm tier
Nothing in this deployment read a Prometheus endpoint, so enabling
/metrics on a managed service added an endpoint and no data. The swarm
collector receives OTLP pushes and does not scrape; VictoriaMetrics
stores what is pushed and has no scrape config. The pipeline was entirely
push-based and every such service is pull-based.

Adds a prometheus receiver, a resource/swarm processor and a
metrics/swarm pipeline alongside the per-hive ones.

The pipeline is separate because that is the ruling, not for tidiness:
every resource/<hive> processor UPSERTS a hive key, so a scraped swarm
sample routed through one would acquire the single label a swarm-level
service must not have. Keeping it out makes the absence structural rather
than something to remember to strip, the same way hive stays a property
of which authenticated receiver accepted a push.

Targets come from an option each service fills in from its own module,
under its own enable, rather than a list assembled here. That is what
puts the scraper and the target on the same host by construction: an
entry exists only where the service that named it runs. Co-location is
true of the all-local deployment and is not a guarantee, and that is
exactly the case where assuming it is invisible.

All three additions MERGE with the per-hive attrsets rather than
replacing them. A plain assignment would drop every hive's receiver,
processor and pipeline and still render a config the collector starts
cleanly on.

Empty target set emits no receiver, no processor and no pipeline — an
enabled scraper with nothing to scrape is the inert configuration this
issue is about, and the target set ships empty here because the targets
themselves are separate issues.

Formatting verified with nix fmt. The evaluation gate is not written yet;
nothing here has been evaluated against a fixture.
2026-08-19 21:40:40 +02:00
damocles
377dbb57e3 swarm-controller: fix broken intra-doc links (private items, rustdoc lint) 2026-08-19 21:12:06 +02:00
damocles
f3e42c93b4 swarm-controller: wire the config-PR webhook, not just the poll 2026-08-19 21:11:35 +02:00
damocles
66c494697f swarm-controller: track agent config-PR status independently of hives 2026-08-19 21:10:59 +02:00
atlas
3c390f8132 docs(#3255): the announce doc described the design the rework replaced
The function's own doc comment still said one subject per hive, four
lines above an inline comment correctly saying one publish for all of
them. Left behind by the single-writer rework.

Swept the rest of the touched files for the old design's vocabulary
rather than fixing only the line argus quoted: every other per-hive
mention is about webhook registration or the callout policy's
extra-subject template, both of which are still per-hive and still
accurate.
2026-08-19 21:05:52 +02:00
atlas
33958d78ae refactor(#3255): a constant, not a module with an essay
Review call: 46 lines of documentation around a single constant, part of
it already stale. The worst paragraph explained why the earlier per-hive
shape had been justified wrongly — history of a design that never
shipped, written into the file within an hour of that design being
dropped. A file is not a changelog; why it was wrong belongs in the PR.

The constant moves to lib.rs beside the status bucket name, keeping only
the rationale that stays true: three crates must agree on the string, and
the one that agrees hardest speaks neither jetstream nor kv, which is why
it cannot sit behind a feature gate.

status earns a module of its own because it holds a bucket name AND the
functions that open it. This held a constant.
2026-08-19 21:05:52 +02:00
atlas
9b939f4626 refactor(#3255): one knowledge subject, single writer and many readers
Review call: the event was addressed per hive — `$SWARM.events.<hive>.knowledge`,
published in a loop over the roster, granted through a wildcard. It does not
need to be. The payload is empty and the event means the same thing to every
hive, so one publish to one subject delivers exactly what N publishes to N
subjects did, and core NATS already fans out to whoever is subscribed. A hive
that was down misses it either way and reconciles on its next periodic pull.

That deletes rather than reshuffles: the roster loop, the wildcard, and the
shared subject-building function whose entire purpose was keeping the grant and
the publish from drifting apart. With one literal there is nothing to disagree
about.

The per-hive shape was justified by the callout policy's rule that an extra
subject must contain the hive name. That rule governs `extra_hive_subjects` —
what a HIVE may publish. This subject lives in the controller's reader grant,
which the rule does not constrain, so a real rule was carried across into a
decision it had no authority over.

Knowledge becomes its own category rather than a leaf under a general event
namespace, since a namespace shaped for events that do not exist yet is a
decision made before there is anything to decide from. The empty config-PR match
arm goes with it: an arm with no body claims this is where the deploy path is
handled, and it is not.

The deny test stays and matters more, not less: with one shared subject a forged
event would reach the whole swarm where a per-hive one reached a single hive.
2026-08-19 21:05:52 +02:00
atlas
84125b0806 docs(#3255): correct the reaper invariant the swarm webhook test guards
The test's doc named knowledge::ensure_webhook as one of two hive-side
reapers. That function is gone; its replacement matches the full URL and
cannot touch another hive's hook.

The assertion arm stays. The hazard is not what this repository's source
says, it is what is deployed — a hive still running the previous version
reaps by suffix until it is upgraded, so the arm guards the transition
rather than a current code path. Recorded when to drop it.
2026-08-19 21:05:52 +02:00
atlas
d2a550e685 feat(#3255): hives stop owning the knowledge webhook, and clean up their own
A webhook has exactly one target URL, so every hive registering one
against the shared internal/knowledge repository was last-writer-wins
rather than idempotent: all but the most recent silently stopped
receiving deliveries. The swarm controller holds the single registration
and now addresses an event to each hive over the queue instead.

This is a migration, not a deletion. Not registering any more fixes
nothing on a hive that has already run — the hook it created persists on
the forge, so the contention would survive on exactly the deployments
that have it while fresh installs looked fixed. The hive that created a
hook removes it.

It removes only its OWN, matched on the full URL rather than the
/webhook/knowledge suffix. A hook with that suffix and a different base
belongs to another hive, possibly one not yet upgraded, and deleting it
would break that hive's knowledge sync until it caught up. Reaping a
neighbour's registration is the behaviour being removed here; doing it
while fixing it would only invert the direction.

The predecessor did reap by suffix, to clear loopback hooks left by an
older single-hive layout. That was safe when a hive was alone on its
forge and is not safe now. The hive-side registrars also acted as reapers
of hooks under their own path, which is why the swarm hook lives under
/webhook/forge/; removing this registrar removes that reaper too.
Intended, and stated because no reviewer would infer it from the diff.

The receive endpoint goes with it. A live HMAC-verified
/webhook/knowledge that nothing can legitimately reach would tell the
next reader that this is how a hive learns about knowledge changes.

Docs move in the same commit: docs/swarm/README.md said two hooks exist
per swarm-wide repo and neither should be deleted, which is now true for
agent-configs and wrong for internal/knowledge — a half-correct
description being worse than an uncorrected one.
2026-08-19 21:05:52 +02:00
atlas
89050ef34b feat(#3255): hives subscribe to their own knowledge event
A hive learned the knowledge repository had changed only by registering
its own forge webhook. This subscribes to the per-hive subject the
controller now publishes on and calls the pull this daemon already runs
at boot.

Shares the hive's ONE queue connection rather than opening a second: a
second connect would double the auth-callout traffic against authelia and
give the two paths independent reconnect state, so one could be serving
while the other was still down. Same argument as the controller side.

No payload is read, because there is none to read — the webhook handler
this replaces took two fields from Forgejo and used neither, then ran
`git pull`, which re-derives everything from the repository.

At-most-once, and that is not a regression: a webhook delivery to a hive
that is down is lost identically today, and the boot pull covers it.
JetStream would require this end to publish to
`$JS.API.CONSUMER.CREATE.<stream>`, which the callout policy does not
grant, so durability would cost grants on both sides to remove a failure
the boot pull already handles.

⚠️ Documented at the call site rather than left implicit: a refused
subscription is indistinguishable from a quiet one, because NATS reports
authorization violations asynchronously on the connection. If hives stop
hearing events, the server log is the thing that knows.

futures-util comes from the workspace (same version swarm-controller
already uses), not a new dependency version.
2026-08-19 21:05:52 +02:00
atlas
bac4a8b6a1 feat(#3255): announce a knowledge change to every hive on the queue
The controller verified a knowledge delivery, logged it, and returned OK.
Nothing downstream ever heard about it, so a hive learned the repository
had changed only by registering its own webhook — which is the
last-writer-wins contention this issue is about.

The event carries no payload. The hive-side handler this replaces reads
two fields from Forgejo's push webhook and uses neither — both are
filters — then runs `git pull`, which re-derives everything from the
repository. What crosses the queue is an edge trigger, and fields would
invent a contract nobody reads.

One subject per hive, so the callout policy can express "this hive may
hear its own events" at all; a subject with no hive component is the same
subject for every hive.

`ConfigPr` deliveries are deliberately not forwarded. A hive does not
want to hear that a config PR opened — it wants to be told when to
rebuild from main, which the controller decides after a merge rather than
by relaying this delivery. That is deploy coordination's job, and the
empty arm is there so the omission reads as scoped rather than forgotten.

Fails soft: a missed announcement costs a hive stale knowledge until its
next boot pull, which is the same cost as a webhook delivery to a hive
that was down — what this replaces. A permission failure cannot be
observed at the call site (a NATS authorization violation is reported
asynchronously on the connection, reaching a client as a timeout or as
nothing), so the doc says the flush proves only that the bytes left this
process and points at the server log.
2026-08-19 21:05:52 +02:00
atlas
8baf1899d8 feat(#3255): name the swarm event subjects, and let the controller publish them
The controller could not emit an event at all: a reader's grant is
`reader_subjects()`, which is `$JS.API.*` only, so a publish to any event
subject would be refused — and a NATS refusal reaches the client as a
timeout, so the visible symptom would have been a hive that never hears
about a change, with nothing in any log naming a permission.

Adds `swarm_queue_client::events`, following `status::BUCKET`: three
crates must agree on these strings (the controller publishes, a hive
subscribes, the callout responder decides whether the publish is
permitted), and a literal repeated across crates is an agreement nothing
checks. The responder speaks neither jetstream nor kv, so the module is
unconditional and carries no NATS types, exactly as the bucket name is.

The grant takes the wildcard form from the same function the publisher
calls, so the two cannot drift; a separate wildcard constant would have
re-created the disagreement this module exists to prevent.

Tests pin that a reader gets the subject and that a hive does NOT — a
hive able to publish here could tell a neighbour the knowledge repo
changed when it had not, which is an unauthenticated write into someone
else's control path. That one asserts on the subject root rather than a
rendered subject, so a future event leaf fails it too instead of passing
because the test only knew about `knowledge`.

Both assertions mutation-tested: removing the grant fails the reader
test, granting a hive the subject fails the denial test, each on its own
assertion line, and the unmutated tree is green.
2026-08-19 21:05:52 +02:00
atlas
893b686c15 fix(#3527): a missing source must report as zero, not as empty
argus, reviewing, ran the guard against a source path that does not
exist rather than one that is empty. grep writes nothing to stdout in
that case, so `|| true` left the count variable empty and the -eq test
died with "integer expected" instead of reporting.

The unit still failed — cat hits the same missing file and set -e stops
it — but with a generic "no such file" rather than the message naming
which half is absent, which is the only thing this guard is for.

`|| echo 0` on all three counts. The gate gained the arm that was
missing: an absent source, asserted to fail THROUGH the guard rather
than merely to fail.
2026-08-19 20:27:00 +02:00
atlas
40557ffb74 fix(#3527): count the hive half, not the assembled bundle
The guard inspected the assembled file for any certificate. The system
store always holds certificates, so it passed unconditionally — including
in the one case it was written to catch, where the hive CA half
contributed nothing.

That half is the only one that matters here: every name these consumers
verify is issued by our own CA, so a bundle of nothing but public CAs is,
for this purpose, an empty bundle that measures as full. The failure is
silent and total — the unit reports success and every egress TLS call to
a swarm service then fails.

Counts the source on its own before assembling, and checks the result
carries what both halves brought, so a source truncated between the count
and the copy is caught too.

Scope is stated at the guard: it proves the anchor was contributed, not
that it is usable. A consumer reading only the first certificate ignores
it regardless, which is what took the swarm collector down, and no check
on this file can see that. Only a handshake can.
2026-08-19 20:16:16 +02:00
iris
149f745e76 docs(matrix): trim upgrade notes per review
- serverName upgrade note: state that it changed, drop the historical
  list of what it changed from/to
- openFirewall upgrade note: remove entirely, change is old enough
  that it's no longer worth flagging
2026-08-19 20:01:45 +02:00
iris
47f5e57c68 docs(matrix): demote breaking-change notes to clearly-scoped upgrade sections
serverName/gatewayHost and openFirewall both led with 'Breaking
change'/'has now moved twice' phrasing woven into the primary
explanation, reading as an active/ongoing concern rather than
settled current behaviour with a historical upgrade note attached.
Reframed to state the current default first, then a demarcated
'Upgrading...' section/note for deployments carrying an older
value forward — same pattern swarm/README.md's own upgrade notes
already use. No factual content changed, removed, or added.
2026-08-19 19:55:21 +02:00
iris
6d9a2fa9f6 swarm-ui: fix underline not showing on the initial page load
Root cause (found via a zoomed pixel-level screenshot check, not
guessed): the indicator span always exists in the DOM starting from
left:0/width:0/transparent, and the very first real position landed
via a CSS-transitioned change from that fallback rather than a snap —
so the underline visibly grew in from nothing over 140ms instead of
being there immediately, reading as entirely absent on a fast/slow
first paint alike depending on timing. Every subsequent navigation was
unaffected (always transitioning between two already-visible states).

Fixed with a one-render-only transition suppression
(indicatorSettledOnce, gated one tick behind the indicator's first
non-null commit) so the first placement snaps instead of animating in,
while every later hop still animates normally. Verified against the
exact repro: a fresh page load at the same short virtual-time-budget
that previously showed no underline now shows it immediately.
2026-08-19 19:54:52 +02:00
iris
daabb4fe6a swarm-ui: brand text follows the nav accent too
The brand/logo text now rides the same accent value the nav underline
uses, including mid-sweep, so the header reads as one accent changing
rather than the underline alone — per mara's review comment.
2026-08-19 19:54:52 +02:00
iris
a5584d0fcd swarm-ui: plain fade + multi-hop underline sweep, per review
Content entrance was a scale+overshoot pop; mara didn't like it on
review, swapped for a plain fade. The nav underline now hops through
every nav item it passes over on its way to the new active one
(position and colour together) instead of tweening directly between
the two endpoints, matching what she actually meant by "move through
color wheel" — a->c visibly touches b's colour along the way, not a
smooth continuous hue rotation.
2026-08-19 19:54:52 +02:00
iris
6be1bbd4f7 swarm-ui: page-switch animation (pop entrance + nav underline travel)
Shell's page content now remounts on every navigation and plays a
scale+fade pop-in entrance. The active nav link's underline is now a
single shared element that slides to its new position instead of
snapping, re-colouring to a per-tab accent — a discrete cycle through
the existing base16 chromatic slots, not a continuous hue rotation.
Both are gated on the data-motion/prefers-reduced-motion plumbing
lib/motion-apply.ts already had wired and waiting for a first real
consumer.
2026-08-19 19:54:52 +02:00
iris
213e0a1b89 swarm-ui: jobs page auto-refresh
Reuses HivesPage's exact RefreshIntervalPicker/useRefreshInterval
pattern (30s default, off/10s/30s/1m presets, pauses while the tab is
backgrounded). Ticks a refreshToken bump rather than doing its own
fetch — JobqGraph/JobqRollup already accept that prop as their
documented refetch lever, from the dashboard's rebuild-queue view.
2026-08-19 19:33:20 +02:00
iris
3b0180e7d6 docs: trim observability.md's duplicate option-reference sections
The hand-maintained per-option reference (services.hyperhive.otel.*)
restated what nixosOptionsDoc already generates from the same nix
module's mkOption descriptions into /options/ — two copies of the same
fact, one of which the build keeps honest and the other doesn't.
Trimmed to a pointer, keeping only the architecture/security prose a
flat per-option listing can't express. The port-conflict cross-
reference (hive tier 4318 vs swarm tier 4319) that isn't in either
option's own description is already covered in swarm/services.md,
which this doc already links to.
2026-08-19 19:30:39 +02:00
atlas
36810ae598 fix(#3529): create agent config repos in agent-configs, not agents
WIP — compiles per an earlier build, but the verifying build/test run was
cut short by a graceful stop. Re-run gate.sh before pushing.

The controller created every agent config repo, its collaborator entry,
its branch protection and its seeded agent.nix/flake.nix in the agents
org. Config repos live in agent-configs, which is where hive-c0re
reconciles, merges and mirrors them — so a repo created in agents is
invisible to all of those, and nothing errors, because both orgs exist
and both accept a repo.

Root cause was a doc comment asserting something false: AGENTS_ORG
claimed to be the same org hive-c0re uses for its config-repo path. It is
not — hive-c0re's agents org is the namespace repos an AGENT ASKS FOR
land in, and its config repos use agent-configs. The whole flow inherited
the wrong premise from that sentence.

The merge gate survives the move: hive-c0re provisions the operators team
in both orgs, with a comment recording that missing the agent-configs
copy once left every config repo unprotected.
2026-08-19 19:06:46 +02:00
damocles
c5bea0b8c4 forge-notify: widen earlier-activity marker to the full timeline 2026-08-19 19:06:16 +02:00
damocles
530eedd5f4 forge_notify: flag earlier comments the latest_comment_url blurb hides 2026-08-19 19:06:16 +02:00
atlas
b8b571fea0 fix(#3524): stop naming the trust bundle as the oidc issuer anchor
The swarm collector could never verify its OIDC issuer, so it exited at
startup on every boot and the hive tier dropped every metric.

`issuer_ca_path` loads only the FIRST certificate in the file it names.
The bundle assembled for this container is `system CAs ++ hive anchors`,
so the swarm CA sits ~123rd and was never in the pool: the extension got
whichever public CA sorts first, could verify nothing of ours, and failed
`x509: certificate signed by unknown authority` — with 125 valid
certificates in the file.

Leaving the option unset makes the extension use the process trust store,
which `trustBundle` already populates via `SSL_CERT_FILE`, and that
consumer reads every certificate regardless of order. One file, two
consumers, opposite parsing; the fix is to stop naming it twice rather
than to reorder the bundle.

Measured with the deployed binary against the deployed config, varying
only the CA source: root-only OK, root-first OK, root-last FAILS,
root-second FAILS, and unset-with-SSL_CERT_FILE OK against a control that
correctly fails when the anchor is absent.
2026-08-19 18:23:42 +02:00
damocles
f5ff8698f3 swarm-controller: extract jobq_metrics into its own hive-jobq-metrics crate 2026-08-19 18:10:22 +02:00
damocles
621245306f swarm-controller: move OTEL deps to workspace level, drop jobq_metrics singleton 2026-08-19 18:10:22 +02:00
damocles
1f5a7b71ed swarm-controller: export jobq state rollup as OTEL gauges 2026-08-19 18:10:22 +02:00
iris
b25aa157df swarm-ui: give Panel an optional header icon slot
Adds an `icon` prop to Panel (small emoji glyph left of the title,
aria-hidden, chosen per panel with no default) and wires it into every
current Panel caller: hives (bee), create-agent's form (robot) and info
panel (identity card, moved off the info panel's body copy where it
started as a one-off), jobs (puzzle piece), the components gallery
itself (toolbox), and the 404 page (compass). Adds a components-page
demo section and a whimsy-section pointer in the design guide.

Closes: #3508
2026-08-19 17:22:48 +02:00
damocles
582b5cf83d fix(#3500): make the swarm-controller forge account a site admin 2026-08-19 17:13:54 +02:00
damocles
aca39072d5 fix(#3072): stop waking every agent on a re-applied m.space.child state event 2026-08-19 17:12:01 +02:00
iris
1207a0e282 swarm-ui: create-agent layout fills the page width, centered
mara: the two cards were neither centered nor filling the space in
two-column mode. Dropped the page's own max-width entirely — Panel
already has no width opinion, and once both cards are equal-width
flex children there's no reason for an extra cap between them and
.shell-body's own 60em/centered column. Removed the now-pointless
wrapper div along with it.

Checked centered + filling behaviour at both a normal (1200px) and an
ultra-wide (1600px) viewport, and narrow-viewport stacking still
works.
2026-08-19 17:11:59 +02:00
iris
3b339b2232 swarm-ui: equal-width create-agent cards, full-width inputs
mara's follow-up: both cards should be the same size (flex: 1 1 0
instead of the form keeping its own narrower basis), and the form's
inputs should fill the card rather than stopping at their old 16em
cap. Dropped that cap from the shared form kit (.ui-form-control /
.ui-form-field) entirely rather than overriding it per-page — the kit
itself has no width opinion now, same as Panel already has none; a
page that wants a narrower field caps its layout, not the kit.

Checked ComponentsPage's standalone samples (no regression, just
wider) and narrow-viewport stacking (still moves the explanation
under the form).
2026-08-19 17:11:59 +02:00
iris
f30897b44e swarm-ui: create-agent info panel reads as the finished flow
mara's review: the info-panel copy should describe the intended end
state (a running agent on the chosen hive), not narrate today's
partial implementation (deploy isn't wired up server-side yet — noted
in this file's existing top comment for maintainers, kept separate
from the user-facing copy).
2026-08-19 17:11:59 +02:00
iris
fc65811d9f swarm-ui: fill create-agent's empty column with an explanatory panel
mara's follow-up on the field-alignment fix: narrowing the form panel
left a lot of bare space next to it, and asked for something that
fills it while helping a new user understand what the page does.

Adds a second panel beside the form (stacks under it on a narrow
viewport) explaining the job chain '/api/agents' actually queues:
an authelia identity, then a forge config repo — no container exists
yet after this page, and deploying one onto the chosen hive is a
separate step that isn't wired up server-side.
2026-08-19 17:11:59 +02:00
iris
f14fc154cc swarm-ui: fix create-agent form field alignment and panel width
TextField and SelectField's shared FormField wrapper had no width of
its own, so inside the form's shrink-to-fit flex column each field's
input/select resolved its 'width: 100%' against its own shrunk
wrapper rather than a shared column width — two fields with
differently-long labels ended up with differently-wide controls.
FormField now caps its own width the same way the control already
does, so every field in a form lines up regardless of label length.

Also wrapped the page in a max-width container: Panel has no width
opinion of its own, so it filled the full page column, leaving a lot
of bare panel to the right of the ~16em-wide form.
2026-08-19 17:11:59 +02:00
atlas
7880483b51 swarm-otel: stamp the swarm on every hive's pipeline
mara on the tracking issue: "we need the swarm label for upstream otel at
least (the out of swarm one)."

Stamped in the per-hive `resource` processor rather than on a separate
upstream-only pipeline, which would double the pipeline count to withhold one
constant label from the local store. It is redundant there — one metrics store
per swarm, so every series in it already belongs to this swarm — but a constant
label multiplies no series, and it means what leaves and what stays have the
same shape.

Upstream is where it stops being redundant: that is the one hop where several
swarms can land in one store, and samples that cannot name their swarm collide
there exactly as hives collided here before per-hive receivers existed.

`unknown` when unnamed rather than an absent label, copying the agent path so
a query never has to handle both "the label is missing" and "the label says
unknown".
2026-08-19 17:06:47 +02:00
atlas
738cc413e7 otel: wait for the telemetry client secret instead of racing it
`LoadCredential` naming a missing path is fatal at unit start, and this hive's
secret is minted by authelia's first-boot generator inside its own container —
nothing orders a host unit against that.

nixpkgs sets `Restart = "always"` on the collector with no `RestartSec`, so
that failure is instant: the unit burns systemd's 5-starts-in-10s allowance in
well under a second, lands in `start-limit-hit`, and stops retrying entirely.
`Restart = always` reads like it makes this self-healing and does the opposite
— a slow-failing unit retries until the secret appears, a fast-failing one
exhausts its limit before the thing it waits for can exist.

A oneshot converts the fast failure into a slow one, which is what that restart
policy is actually good at. Copied from `hive-forge-oidc-secret.service`, which
already solves this for the forge: bounded wait, then fail loudly naming the
file — never skip, because a skip yields a collector that starts and ships
nothing.

`TimeoutStartSec` exceeds the wait on purpose: `DefaultTimeoutStartSec` is 90s
and would kill the unit before it could emit that message.

The ordering against authelia's container is conditional — on a hive that does
not host the provider the secret is operator-provided, and naming a unit that
does not exist orders nothing, silently. The wait itself still applies there,
so a file that arrives late is tolerated rather than fatal.
2026-08-19 15:34:40 +02:00
atlas
9bd2b9e9e6 otel: a hive always authenticates — drop the unauthenticated mode
mara, reviewing this PR: "hives always require an identity, swarm controller
and auth is not optional."

So `requireHiveIdentity` is gone rather than defaulted, and with it every
branch that had to describe an unauthenticated collector. The swarm tier now
serves per-hive receivers only, and `/` answers 404 because there is no
swarm-wide inbox to route to. A hive with no credential is a build error, not
a quieter mode.

`hivePortBase` goes too: with per-hive receivers unconditional, `port` IS the
base of the range. That keeps one documented knob instead of adding a second,
and its advice ("move it if something else claims that range") still holds.

Two assertions replace the toggle — an empty hive roster, and a null
`authelia.url`. The second matters because a guessed issuer URL evaluates
cleanly, deploys cleanly, and then refuses every hive at runtime.

⚠️ `cfg.port` is deliberately no longer compared against the derived range in
the collision assertion: it is now the range's first element, so listing it
would make that assertion fire on every config.

This also retires the asymmetry guard added earlier in review — the state it
protected against (auth off on one side, credential still set on the other)
is no longer representable.
2026-08-19 15:27:09 +02:00
atlas
7da7915150 otel: refuse a half-configured escape hatch instead of 404ing silently
Turning ingest auth off without clearing a hive's credential leaves that
hive's collector authenticating and addressing its own path, while an
unauthenticated swarm tier serves one catch-all and forwards the URI
unchanged. The receiver is asked for a path it does not serve, so telemetry
stops with 404s and retries — no 401, no assertion, nothing in any log
naming auth.

Only reachable by overriding one side without the other, since both defaults
derive from the same flag. That is what makes it worth a build error rather
than a caveat: an operator who flips the documented escape hatch has no
reason to suspect the sending half.

Found in review by argus.
2026-08-19 15:27:09 +02:00
atlas
9dc60061e7 docs: authenticated telemetry ingest, and where the hive's secret comes from
Three readers, three different questions:

`observability.md` gets the operator's version — what to set (nothing, on an
all-local swarm), what to set elsewhere, what a misconfiguration looks like
(401s and a missing hive), and what turning it off actually costs.

`swarm/secrets.md` extends the existing client-secret row rather than adding
one, because this is a new *reader* of a secret already described. It notes the
one thing that makes this reader unusual: a host unit reads the file in place,
so unlike every other consumer there is no delivery step to get wrong.

`setup.md` gets one bullet under security notes, since a first-run hive gets
the property without doing anything and should know it has it.
2026-08-19 15:27:09 +02:00
atlas
cb787997bd otel: the hive tier presents its own identity to the swarm collector
The receiving half authenticates per hive, so this half has to prove which
hive it is. It mints a token against the swarm's authelia with this hive's
client and posts to that hive's path on the collector's gateway name.

Holding a credential is what decides whether this tier authenticates —
`clientSecretFile` non-null — rather than a second switch that could
disagree with it. The default is the secret this host's own authelia
minted, which is right exactly when the IdP runs here; a hive that is not
that host names wherever the file landed, the same manual-copy shape the
identities option already documents as unsolved.

Two things that a diff will not explain:

`endpoint_params.audience` is not redundant with the client's registered
audience. Registering only makes an audience permissible; a token minted
without asking for one carries `aud: []` and every receiver refuses it,
with a config that reads correctly at both ends.

`client_secret_file` keeps the secret out of nix altogether — the
collector opens the file itself. It is a real key of this extension,
checked against the shipped binary with a deliberate typo rejected in the
same run, so "accepted" is distinguishable from "ignores everything". The
path comes from systemd's `CREDENTIALS_DIRECTORY`, so nothing hardcodes a
`/run/credentials` layout.

An assertion covers the one deployment where this can go wrong silently:
a host running both tiers with ingest authenticated and no credential to
present would 401 against a collector on the same machine.
2026-08-19 15:27:09 +02:00
atlas
08faa0970e swarm-otel: authenticate ingest per hive, and stamp the hive from the receiver
The swarm collector accepted OTLP from anyone who could reach it, and took
the `hive` resource attribute from the payload. So any writer on the swarm
network could attribute metrics to any hive, and nothing downstream could
tell.

The label now comes from which receiver accepted the sample: one receiver
per hive, each behind an `oidc` extension verifying a token minted for that
hive's audience, each feeding a pipeline whose `resource` processor upserts
a constant. A sender cannot influence it, because the only input is which
authenticated port the bytes arrived on.

That multiplicity is forced rather than preferred. A processor cannot read
the token's claims — `from_context` reads request metadata, and asking it
for an auth claim yields nothing, silently, with a healthy startup — and
one receiver holding several credentials never reveals which one matched.

The per-hive ports are internal: a hive reaches its receiver as a path
under this collector's existing gateway name, so nginx (rendered from this
same evaluation) is the only thing that names a port. Fronting each hive
with its own vhost would need a certificate, a DNS name and a gateway entry
per hive to express routing the gateway already does.

Turning this on removes the unauthenticated receiver. While an open port
still accepts samples the per-hive receivers are decoration, so this is the
switch itself rather than a hardening layer beside it; a swarm that wants
the open receiver says so.

`hive-ca-trust.nix` grows `bundlePathFor`, because a consumer taking its own
CA argument has to name the bundle rather than just have `SSL_CERT_FILE`
exported at it.
2026-08-19 15:27:09 +02:00
atlas
0b98f0ac8f swarm-authelia: give each hive client an audience and JWT access tokens
A swarm service that has to tell hives apart needs the token itself to
say which hive presented it. Two client-registration fields were missing
for that, and both are properties of the client rather than of any one
consumer:

  - `audience` registers which `aud` values a client may request. It is a
    boundary, not a label: asking for one that is not registered is
    refused with `invalid_target`, so a hive cannot mint a token for
    another hive's slot.
  - `accessTokenSignedResponseAlg` switches the access token from
    authelia's default opaque handle to an RFC 9068 JWT, which is what
    lets a resource server verify a token against `/jwks.json` instead of
    asking authelia about every request.

Each hive's audience is its own client id rather than a new per-hive
string. The identity is the directory's, not the consumer's, and the id
is already published as `hiveClientPrefix` precisely so a second
consumer does not carry its own copy of the naming rule.

Both fields default to authelia's own behaviour, so no existing client
changes.
2026-08-19 15:27:09 +02:00
atlas
0f801c645f otel: build the hive tier's collector from contrib
The hive tier has to *present* a credential to the swarm tier, and the
upstream collector build ships no auth extensions at all — `oauth2client`
exists only in contrib. So this is what makes authenticated egress
expressible, not a preference between two equivalent packages.

It also matches the swarm tier, which has been on contrib since it was
written. Two tiers of the same pipeline built from different component
sets is a difference nobody would predict from reading either module.

Not a build-farm cost: contrib is fetched, not compiled.

No behaviour change on its own — nothing names an extension yet. The
config that does lands separately, because that is the commit whose
failure mode needs a running collector to detect: `otelcol validate`
accepts a receiver naming an absent extension and the process then dies
at startup, so a green build proves nothing about it.
2026-08-19 15:27:09 +02:00
damocles
bd4906ab17 hive-forge: unit-test blocking_open_count 2026-08-19 15:17:28 +02:00
damocles
a3f14f5126 hive-forge: surface issues an issue/pr blocks in list/issue/pr 2026-08-19 15:03:15 +02:00