A local time-series database rather than only an external sink, so the
swarm dashboard stays readable when the outside world is not: a view of
the system must not depend on the system it views being healthy.
listenAddress is pinned to loopback. Upstream defaults it to every
interface, and the OTLP ingest path this exists to receive on is
unauthenticated — the gateway is the only intended client and it is on
this host, so a wider bind would publish a write endpoint to whatever
the host is reachable on.
retentionPeriod defaults high rather than being required, because the
two failure directions are not symmetric: too long fills a disk, which
is visible and recoverable by lowering it, while too short destroys
history silently and permanently. The operator lowers it once they have
measured how fast this swarm accumulates.
OTLP needs no flag. Measured against the pinned 1.146.0 rather than
inferred from the module's option list, which has no OTLP switch and so
reads as though the feature were missing: the running server answers
POST /opentelemetry/api/v1/push with 200, where a nonexistent path
answers 400.
Three host units poll up to 120s for a secret authelia mints on its first
boot, and all three are `Type=oneshot` with no `TimeoutStartSec`. systemd's
`DefaultTimeoutStartSec` is 90s, so it kills them at 90 — before the script
reaches its own `exit 1` and names the file that never appeared.
The wait itself is fine; what's lost is the diagnosis. On a fresh hive the
operator gets a bare start-timeout instead of "authelia has not minted
<path>", several layers from the container that was actually slow.
Found while writing the same unit for Grafana, where the timeout is set —
so this is the existing three catching up with it, not a new pattern.
Defining an environment key on a unit that does not exist creates a
fragment for it: inert, never activated, but present on every
non-controller hive that has a CA. hive-c0re runs everywhere so its
line needs no guard; the controller runs on one host in a swarm.
Caught in review. It evaluates and builds clean either way, which is
why it took a reviewer rather than a check.
The queue client built a bare reqwest::Client, so it trusted only the
platform roots. Against a swarm whose authelia is signed by the swarm
CA that is fatal: minting a token dies with 'invalid peer certificate:
UnknownIssuer', inside the auth callback, on a four-second retry loop,
with the queue never connecting.
The anchor was never missing. hive-tls.nix assembles trust-bundle.pem
and already hands it to hive-c0re as HIVE_TLS_CA_PATH; nothing pointed
the queue client at it.
QueueConfig gains an optional ca_file from <prefix>_OIDC_CA_FILE, read
outside the all-or-none tuple on purpose: a CA path with no queue is
meaningless rather than half-configured, and requiring it would break a
swarm fronted by a public certificate in order to fix one that is not.
add_root_certificate extends the default roots rather than replacing
them, so both deployments work.
A bad path fails loudly instead of falling back to the platform roots.
An operator who names a CA file wants that anchor; a silent fallback
turns their typo into UnknownIssuer five layers away.
hive-tls.nix names the bundle for both clients, beside the line that
already does it for hive-c0re, rather than having each consumer
re-derive the path.
The responder's third credential is not minted by the generator beside
it — authelia produces it on first boot, inside its own container, and
nothing ordered this unit against that. Losing the race cost the entire
queue: install exits 1, the responder never starts, and auth_callout
with no responder refuses every client. Fail-closed by design, so the
symptom lands on every queue client and nowhere near the cause. Seen on
a real boot at 20:31:31.
Now it waits for the file, bounded at two minutes. Where authelia runs
on another host the secret is never going to appear, and blocking the
queue container indefinitely would replace a clear failure with a hang;
after the timeout this fails exactly as it did before, having first
given the co-located case the seconds it needs.
TimeoutStartSec is set rather than left to the default because the
default is ninety seconds — a two-minute wait would be killed at ninety
and the operator would get a generic unit timeout instead of the
message naming the file.
Ordering after the authelia container is necessary and not sufficient:
the container being up says nothing about whether its in-container
secrets unit has finished. It only stops this spinning for the full
timeout on every boot.
A systemd credential named by an absolute path is fatal when the file is
missing, and the co-located queue secret is minted by authelia's first
boot in another container — which a host unit cannot order against. On a
real boot the daemon spent three of systemd's five default starts losing
that race before the file appeared; two seconds more would have reached
start-limit-hit, which does not self-heal.
An empty SetCredential acts as the default that makes the load
non-fatal, so the controller starts and serves its HTTP surface with the
queue unconfigured — a shape it already reports.
That alone would be a regression, though: a credential is snapshotted at
unit start, so the placeholder would freeze and the daemon would sit
degraded forever instead of recovering the way the restart loop
accidentally did. A path unit on the secret file closes it, and closes a
second gap in the same stroke — mint_token reads the secret on every
call precisely so a rotation takes effect without a restart, and a
snapshot in %d silently defeats that.
PathChanged and not PathExists: the latter activates immediately when
the file is already present at unit start, which would restart a healthy
daemon on every boot.
The option being consumed already warned about this shape — its own
description says a consumer has to wait for the secret because a
missing source turns a fresh hive into a boot-order deadlock. That
warning was written on the producing side and did not fire while I was
writing the consumer.
The client list was `cfg.oidc.clients ++ hiveClients`, where the first
half comes through the submodule and the second was a raw attrset from
this module's `let` block. That list is only half-typed: a field added
to the submodule exists on the declared entries and not on the derived
ones, so reading it plainly is an eval error the moment hive identities
are on. The operator asked whether it should be uniformly typed instead
of guarding each read, and it should.
The hive identities are now declared the same way an operator declares
a client, so the module system applies the submodule to them and every
option's default is present. Downstream reads one uniformly-typed list
and the guard added for the field that broke is gone with it.
allClients concatenates two shapes: cfg.oidc.clients comes through the
submodule and carries every option default, hiveClients is a raw attrset
built in the let block with four fields. renderClient read
tokenEndpointAuthMethod plainly, which is fine for a declared client and
an eval error for a derived one -- so all-local, where hiveIdentities is
on, stopped evaluating.
Read it with `or null`. The comment records that the list is not
uniformly typed, because the next field added to that submodule has the
same trap waiting.
enableRequiredServices asserts the services that exist once per swarm and
are optional -- its own description says so -- and the queue meets that
rule. It was left out because this file predates the swarm-nats container
by nine days and was never revisited, not because anyone decided against
it.
The all-local mode already derives nats.autoGenerateCallout, so it was
minting the queue's callout nkeys and never starting the queue. With this
the mode's loopback derivation for the controller points at something
that is actually running.
Review point: the co-located defaults are an auto-deploy special case and
belong with the other ones, not inside each option's own default. An
option should describe itself; the mode describes what a deployment shape
implies. local-defaults.nix already says exactly this in its header.
Options now default to empty (= unset, which the assertions refuse), and
enableAllLocalDefaults fills in loopback + the minted secret path.
The controller's queue coordinates were gated on `autheliaCfg.enable &&
natsCfg.enable` -- i.e. on this host running both. Split the hosts and
the whole env block vanished, which the daemon's own rule reads as "no
queue configured": it started cleanly, served /api/hives, and silently
never connected or published.
Make the coordinates options instead, defaulted to the co-located values
so nobody types a path they didn't need, and assert each one so a split
deployment fails at eval naming the option rather than running blind.
The queue is required for a controller -- /api/hives/status reads the KV
-- but co-location with it is not.
An SSO login whose derived localpart already exists silently gets a
random one instead: the candidate must be *available*, an existing
unlinked account is not, and `unique_id_fallbacks` defaults to true.
The operator's own login minted a stray account this way.
Fill the three identity_provider fields that decide which account a
login lands on: `trusted` (upstream's precondition -- self-hosted and
fully controlled -- is structural here, the issuer is always the swarm's
own authelia), `userid_claims` pinned to preferred_username so the
default ladder cannot fall through to an email local part, and
`unique_id_fallbacks = false` so a collision errors instead of
succeeding as the wrong user.
tuwunel authenticates at the token endpoint with the secret in the POST
body. Authelia enforces the method a client is REGISTERED with rather
than accepting whichever one arrives, and its default is
client_secret_basic — so the matrix login completed, consent was
granted, and the very last hop failed:
Client authentication failed ... The request was determined to be
using token_endpoint_auth_method client_secret_post, however the
OAuth 2.0 client registration does not allow this method.
The failure names neither the secret nor the redirect, and it lands
three layers from its cause, which is why it read as a credential
problem.
Adds a per-client tokenEndpointAuthMethod, null by default so every
existing client keeps authelia default (forgejo authenticates with
basic and is unaffected), and sets client_secret_post on the matrix
client only.
Review: high comment-to-code ratio. Measured — 197 of 337 added lines
were comments, and net of the block that only moved it was still +135
comment against +117 code.
Kept the constraints, the rejected alternatives and the costs; dropped
everything that restated what the code plainly does, and the
slash-stripping explanation that was duplicated in the generator.
A hive whose whole swarm is one box had to be handed two nkeys by hand
before its queue could authenticate anyone, which is the one deployment
shape where nobody else can supply them.
`autoGenerateCallout` mints both keypairs on the host on first boot,
keeps the seeds at 0600 host-side, and writes only the public halves
into a fragment the server reads at start. The all-local mode turns it
on; everywhere else the options stay operator-supplied and the
fail-closed eval assertions keep their full force.
The server config is not rewritten to do this. A wrapper includes
upstream's rendered `settings` verbatim plus the runtime fragment, and
the fragment wins — measured, along with the property that makes the
whole shape safe: the empty strings the options render in auto mode are
values `nats-server` refuses to start on, so any field the merge fails
to reach fails closed loudly rather than leaving a walk-in-able server.
The wrapper, the settings symlink and the fragment are siblings in one
runtime directory, and that is forced rather than tidy: NATS resolves an
include with filepath.Join against the config file's own directory,
which strips a leading slash, so an absolute include silently becomes a
relative one and the server never finds it. The includes are therefore
bare filenames. That also means nothing in the closure would otherwise
name the rendered settings, so the generator's symlink to it is what
keeps it from being garbage-collected under a running server.
`accounts` and `authorization` are defined once and rendered twice, into
`settings` and into the fragment template. Written out separately they
would diverge silently and backwards: the fragment is the later
definition, so a future edit to `settings` alone would be ignored on
exactly the hives that use auto mode.
`validateConfig` goes off in auto mode because `nats-server -t` rejects
the empty keys at build time; the parse check moves to server start,
where the fragment exists. Upstream's own option description names this
case.
Three options, all three derived from ONE predicate — this host runs both
the queue and the IdP — so a defaulted set is all or nothing. Deriving
them per-service looks equivalent and is not: `enableRequiredServices`
turns on matrix and authelia but not nats, so an ordinary all-local hive
would resolve two of three and trip the assertion below. Making the
partial state unrepresentable is what keeps that assertion honest.
Deliberately not the shape swarm-controller uses. That module emits its
queue coordinates only when authelia and NATS are local, which is right
for a service that *is* a swarm-host service — but a hive is the one thing
in a swarm that routinely is not on the swarm host, so the same rule would
make status publishing work on exactly the deployment that needs it least.
There is no `enable`: three coordinates that are all set is the enable. An
extra flag would allow configured-but-off, which is one more state to
explain and one more way to be silently quiet.
A half-set trio is an eval error rather than a silent no-op, because its
runtime failure mode is the expensive kind — the daemon comes up fine,
never connects, and the hive reads never_reported on a dashboard nobody is
watching yet. With the defaults all-or-nothing, the assertion only ever
judges what an operator typed by hand.
The secret arrives by LoadCredential, not a copy: hive-c0re is a host
unit, so systemd hands it the file directly and the secret never gains a
second on-disk copy. The client id is not chosen here either — it is
`hive-<hiveName>`, the identity swarm-authelia.nix already declares for
every entry in the roster.
A deployed config still set `swarm.hives.<hive>.certFingerprint`, deleted
along with the dashboard feature it served. The module system's answer was
`The option ... does not exist`, which tells an operator nothing about why
it went, whether it moved, or what replaces it.
Re-declared invisible and internal, with a top-level assertion naming the
hives that still carry it and explaining that the swarm root CA replaces
per-hive leaf pinning.
`lib.mkRemovedOptionModule` cannot do this job, and it is worth writing
down why: neither of its halves survives the move into a submodule. Its
`apply = throw` fires only when the value is read, and nothing reads this
any more — that being the point of removing it. Its `config.assertions`
half would land on a submodule that declares no `assertions` option. It is
a top-level tool. This is the same shape swarm-peers-removed.nix already
uses for the analogous `peers.<hive>.caCert`.
An error rather than a warning, because re-declaring the option is what
stops the unhelpful message — and on its own that would turn a config that
used to fail into one that quietly evaluates with the setting ignored,
which is worse than the error it replaced.
Registering the OIDC login source makes an outbound HTTPS call - the CLI
fetches <issuer>/.well-known/openid-configuration to validate the
provider before writing the row. That URL is a swarm service name served
under the swarm CA, which the default system store has never heard of.
SSL_CERT_FILE was set on forgejo.service and not on
forgejo-sso-source.service, so the web service trusted the chain and the
registration one-shot did not. Same binary, same host, different unit.
The result was a 100% reproducible failure that no restart could fix:
nothing about restarting a unit gives it an environment variable it
never had. Every recorded run since the feature landed has failed with
x509: certificate signed by unknown authority.
The trust belongs to every process that makes the call, not to the
service that happens to be the obvious consumer.
Follow-up to the plumbing removal: docs/swarm/README.md gets the
biggest rewrite (drops the whole "Fingerprint format" section, fixes
the runtime-effects list, the WireGuard config example + "what the
mesh does" bullet), docs/gateway.md and hive-gateway/options.nix drop
4 "needs no certFingerprint" mentions, swarm-peers-removed.nix's
migration-warning text no longer tells an upgrading operator to carry
a field over that no longer exists, swarm.nix/swarm-wireguard.nix/
swarm-controller.nix/swarm-controller's main.rs get comment fixes
where they described the now-removed HYPERHIVE_PEERS shape. Also
caught one more stale "peer hives" mention in docs/web-ui/README.md's
SW4RM tab description that the first pass on this issue missed.
Mara wanted the underlying plumbing gone too, not just the dashboard
display. Traced every consumer before cutting:
- certFingerprint (services.hyperhive.swarm.hives.<name>.certFingerprint):
removed the nix option entirely. Its only consumer was the dashboard
code removed in the previous commits.
- HYPERHIVE_PEERS: removed entirely — the env var itself, the whole
block that built it in hive-c0re/environment.nix, and its entry in
meta.rs's FORWARDED_VARS (which forwarded it into every agent
container). Turned out to have zero real consumers, not just one:
the docs claimed hive-agent::identity::peers() read it for qualified
agent labels, but no such function exists — identity.rs only
qualifies THIS agent's own label with HYPERHIVE_HIVE_DOMAIN, nothing
peer-list-related. Grepped the whole hive-agent crate to confirm
before removing.
services.hyperhive.swarm.peerHives (the nix option HYPERHIVE_PEERS was
built from) is untouched — swarm-wireguard.nix reads it directly for
the wg-hive mesh, a real and unrelated consumer.
Verified: cargo build/clippy/test -p hive-c0re -p swarm-controller all
clean (needed nix develop -c per the usual -lsqlite3 gap), all touched
nix files pass nix-instantiate --parse, and a throwaway nixosSystem
eval confirms the wireguard mesh still configures a peer's
wireguardAddress into wg-hive correctly with certFingerprint gone.
Follow-up to the peer-hives removal: docs/web-ui/dashboard.md drops
the whole "P33R H1V3S" section + its stale #peers hash and tab-strip
mentions; docs/swarm/README.md's runtime-effects list and
cross-references point at swarm-ui's roster page instead. Nix comments
in hive-c0re/environment.nix, swarm.nix and swarm-wireguard.nix that
described HYPERHIVE_PEERS/certFingerprint as dashboard-consumed are
updated to say what's actually still true (agent identity only).
Deliberately NOT removed: the `certFingerprint` nix option itself.
It has no remaining Rust consumer, but deleting a documented,
potentially already-set option is a bigger, breaking change than
removing dead display code — flagged in docs/swarm/README.md with a
note instead of silently dropped, so an operator or reviewer can make
that call explicitly rather than have it made for them here.
The controller connects to the swarm queue as its own client and serves
what each hive last said about itself at GET /api/hives/status.
THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream
KV bucket (history 1) and the controller reads it per request, keeping no
copy. A cache here would be a second answer to the same question, free to
disagree with the first, and the disagreement surfaces as a hive reading
healthy on a dashboard while the bucket says otherwise. Whichever side
arrives first creates the bucket; both want the same shape.
Rows come from the roster rather than from the bucket, so an empty bucket
renders as a swarm nobody has heard from instead of a healthy one, and
`never_reported` stays distinct from `stale` - went quiet is a fault,
never spoke is usually a deployment that has not happened. Freshness is
derived at read time and never stored as a flag, because a stored
`healthy` boolean goes stale silently the moment nothing arrives, which
is the failure this endpoint is designed against. The timestamp is the
NATS server's, applied when the value landed, so a publisher cannot make
itself look fresher than it is.
Authentication is per connection attempt, not per process. Authelia
issues `client_credentials` tokens that expire in 3599s, and auth happens
at CONNECT, so a long-lived connection is fine but a reconnect an hour
later needs a token minted an hour later. `with_auth_callback` is re-run
by async-nats for each attempt, which handles expiry by construction
rather than by a timer - the alternative fails in the way this subsystem
exists to prevent, with the controller still serving while its data
quietly stops updating.
Three failure shapes are deliberate:
- A half-set environment is fatal; an absent one is not. Silently
behaving like an unconfigured host is how every hive ends up reading
`never_reported` with nothing to point at.
- The endpoint answers 503 rather than an empty list when the store
cannot be read. "I cannot reach the store" and "every hive is silent"
are different answers, and rendering the second turns a local fault
into an apparent swarm-wide outage.
- `retry_on_initial_connect` makes the daemon and the queue bootable in
either order, and the status handler refuses when the client is not
Connected rather than issuing a request into it - a request made in
that window does not fail, it waits, so every poll would hang and
learn nothing. `Pending` is the state a never-connected client is in,
which is why the test is `!= Connected` and not `== Disconnected`.
The rendering rules are a pure function over a map, so the semantics are
tested against a table rather than against a running server. The KV read,
the credential rotation and the 503 paths are covered behaviourally
instead: a real NATS server with a rotating token endpoint, asserting
that the controller recovers only when the credential rotates, and
mutation-tested by holding the credential wrong for the same window.
Second slice of #3274. Each entry in
`services.hyperhive.swarm.hives` gets an authelia client
`hive-<name>`, so a hive can authenticate to swarm services as
itself.
The clients are derived here rather than declared by a consuming
service, because a hive's identity belongs to the directory and not to
whichever service happens to use it first. mara's constraint is that a
hive's credentials "derive from the same identity" — one hive holds
one client and mints a different token per service from it — so the
queue declaring this list would mean the next consumer collides on the
same client id, and only at the moment it landed.
`oidc.hiveIdentities` defaults to whether the message queue is
enabled, that being the first service to need a hive to prove who it
is. It is an option rather than a hard-coded condition so the swarm
telemetry collector can turn it on without the queue.
No audience and no `access_token_signed_response_alg` here: the queue
learns which hive presented a token from the introspection response
(`{"active":true,"client_id":"hive-alpha",...}`, measured), so it needs
neither. The collector does need both, and they land with it.
Also asserts client ids are unique — newly reachable now that part of
the list is derived, since a hive named `x` and a service client named
`hive-x` never met before.
Refs #3274.
New `services.hyperhive.swarm.controller.links` option (listOf {label,
icon, url}, same shape as the per-agent hyperhive.dashboardLinks) plus
a new GET /api/links route serving it, same pattern as the existing
hives/GET /api/hives.
Rather than one central hardcoded list, each service's own module
contributes its own entry when actually enabled on the controller's
host: swarm-authelia.nix, hive-matrix.nix (gated on gui.enable too,
since / on that vhost only serves fluffychat then) and
hive-forge/default.nix (gated on behindGateway) each push one entry,
the same list-merge idiom services.hyperhive.gateway.localNames
already uses. swarm-ui.nix contributes a static entry for its own
same-origin swagger docs. Adding a future service's link is a nix-only
change to that service's own module.
Verified: cargo build/clippy/test -p swarm-controller clean, a
throwaway nixosSystem eval confirms all 4 entries merge correctly into
SWARM_CONTROLLER_LINKS, nix build .#swarm-controller succeeds.
`renderClient` could only emit the authorization-code shape, so a
daemon client was expressed as an interactive one with an empty
redirect list. Authelia permits only the grants a client names, and an
omitted `grant_types` means authorization-code alone — so that shape
cannot obtain a token at all.
Measured against authelia 4.39.20, rendering exactly what this module
produced for `swarm-nats`:
client_secret_basic → unauthorized_client: The OAuth 2.0 Client is
not allowed to use authorization grant
'client_credentials'
introspection → {"active":false} (works)
Introspection is all the queue's responder needs today, which is why
nothing was visibly broken while the comment in `swarm-nats.nix`
described a grant that was never configured.
Adds `kind = "interactive" | "machine"` rather than inferring from an
empty `redirectUris`, because the two differ in what authelia permits
and not merely in what is populated. `openid` is dropped from a machine
client's scopes because authelia refuses that combination outright — a
daemon receives an access token and never an id-token.
An assertion rejects redirect URIs on a machine client: they are not
harmlessly unused, they mean the author believed a browser was
involved.
Refs #3274.
The comment said "It parses; it does not prove a sample arrives",
which understates the gap in the direction that matters: it reads as
if a green build proves the collector *starts* and only runtime
delivery is unverified.
Measured while probing ingest-auth options for #3283: `otelcol
validate` ACCEPTS a receiver naming an auth extension that is absent
from the build, and the collector then dies at startup with `Failed
to start component`. So the check does not prove this config starts
at all.
Comment-only; no evaluated config changes.
Refs #3283.
Retention so a reader can ask what a hive last said without anyone
keeping a second copy. The container is ephemeral = false, so the store
survives a restart with no bind mount, and losing it degrades to the
true answer (nothing known) rather than a stale one.
The per-account grant is the part that is easy to miss: the server
option gives the SERVER JetStream, and an account gets it only from its
own. Measured against a running 2.14.1 with this exact two-account
shape - global-only makes kv operations fail 10039 while the server
starts cleanly and logs Starting JetStream, so no rendering check can
see it. Deliberately not granted to the callout account: the responder
mints credentials and has no business holding stream state.
The headersCredential section still said hive-c0re forwards the file
into each agent container - that is exactly the behaviour this change
removes, so the doc contradicted the code rather than merely lagging
it. Same for the collector section, which presented the two paths the
change collapses.
Also drops the dangling collector.enable reference in the network
section.
mara: 'there should only be the one via otel-collector'. Collapses
collector.enable away - enabling otel runs the collector, and agents
always export to it.
The argument for it is the same one the collector exists for: as long
as the direct path stays selectable, the credential-in-settings.json
hole stays selectable. An option that can reintroduce a hole is a hole
with extra steps. HYPERHIVE_OTEL_HEADERS_CREDENTIAL is now never
forwarded to containers at all rather than conditionally.
Two consequences, both deliberate:
- the enable option's own promise changes. It said each harness exports
directly so telemetry survives anything host-side being down; that is
now false for every hive rather than for opted-in ones. Stated in the
option text rather than left for a reader to discover.
- the collector-requires-headersCredential assertion is gone. It was
correct only under the shape it was written for: when the collector
was opt-in FOR the credential, opting in without one was pure
indirection. With one path it isn't - an upstream needing no auth
header is legitimate, and the assertion would have rejected it. The
upstream header block is now omitted entirely in that case, rather
than rendering an ${env:...} reference nothing sets.
The module hardcoded otlphttp, so a hive with protocol = "grpc" would
have had its agents' protocol respected end to end before the collector
and silently rewritten to HTTP after it. Splitting the path in two makes
the upstream half the one that has to keep honouring the option; the
agent half is pinned to the receiver's protocol and derived, not
configured.
The agent-facing endpoint becomes derived rather than `otel.endpoint`
re-defined: with the collector on, agents and hive-c0re's own exporter
send to the bridge address; with it off the expression is the old value
character for character. `otel.endpoint` keeps meaning "where telemetry
ultimately goes" on every hive that already sets it.
Suppressing HYPERHIVE_OTEL_HEADERS_CREDENTIAL is the half that actually
closes the hole this issue is about. The credential path host -> container
is sound (/run/credentials is dr-x------ root), but the hive-otel-header
oneshot then writes the value into the agent's own settings.json, mode
0600 and owned by the agent — so today every agent can read the hive's
upstream telemetry credential. With a collector there is nothing to
forward.
The protocol is derived for the same reason the endpoint is: the first
hop is now the collector's OTLP/HTTP receiver, while otel.protocol
describes the upstream link.
Renders services.opentelemetry-collector on the host: OTLP/HTTP receiver
on the bridge address, otlphttp exporter to otel.endpoint, and the
upstream credential delivered as EnvironmentFile so the collector
interpolates it at runtime and nix never sees the value.
Three things worth knowing, each measured rather than assumed:
- network.exposeHostPorts already exists and is wired (it opens the port
on the bridge interface only), so bridge reachability costs nothing.
- validateConfigFile defaults to isStorePath configFile, which is null on
the settings path - so upstream's default is OFF for exactly the way
this module configures it. Set true: it runs otelcol validate at build
time. It parses, it does not prove delivery.
- headersCredential's file is already NAME=value, i.e. EnvironmentFile
format, verified against a real settings.json rather than the doc.
An assertion refuses collector.enable with no headersCredential: the
collector exists to be the only holder of that token, and without one it
is indirection that reads as security.
Adds services.hyperhive.otel.collector.{enable,port}. No implementation
yet - this commit only declares the surface, so the option docs and the
default-off invariant can be reviewed on their own.
The reason the collector exists is the credential: without it every agent
needs headersCredential to reach the upstream, and the harness delivers
that token into the agent's own settings.json where the agent can read
it. The option text states the cost too - the collector becomes a
dependency in an export path that is currently direct.
The responder answered granted=true and the server still refused every
client:
Error non operator mode account "AUTH": attempted to use issuer_account
nats_jwt::Token::new_user always sets issuer_account, which is an
operator-mode field naming the account when a signing key rather than the
account identity key signed the token. In server-config mode - what this
module renders - its mere presence is fatal, and the account is named by
the claims' aud instead.
nats-jwt can express neither aud nor the omission, so the user JWT is now
hand-built by the same signer as the response wrapper, and nats-jwt moves
to dev-dependencies as the encoder's test oracle.
Every unit test passed throughout: they assert fields that must be
present, and the defect was a field that must be absent.
Measured against a running server built from this module's own rendered
config, not a hand-written one -- which is the fix to the method as much
as to the file. Two shapes died in a row: a bare name authenticates
anyone who sends it, and a name plus an nkey refuses to start at all
(nkey users do not take usernames or passwords).
A malformed key is fail-closed too, so the only way to a live server
here is a real key whose seed nobody but the responder holds.
argus caught it and reproduced it: a users entry carrying only a name
has no credential, so CONNECT {"user":"auth"} is accepted with no
password. The name is a literal in this module, so on the shared netns
that identity was walk-in-able from every agent container -- the same
hole the module exists to close, moved rather than closed.
An nkey rather than a password: only the public half appears in config,
the seed reaches the responder alone, so a hive with no responder has
nobody who can authenticate as it. Asserted at eval, because the
rendered config is valid and the server starts either way.
A swarm has one message queue; this is the container that runs it, in
the same shape as every other swarm service. Off by default, and off
means absent.
The auth_callout block is the interim state and the final one at once.
Measured on the pinned nats-server rather than reasoned about: an empty
authorization block, and one with an empty users list, both accept an
anonymous client and answer PONG. auth_callout sets auth_required and
refuses everyone until a responder approves them, so a config whose
responder does not exist yet denies by default -- and the responder
lands beside it without any of this being swapped out.
That matters more here than it would elsewhere because the container
shares the host netns, so an unauthenticated interim state would be
reachable from every agent container on the hive.
No package option, deliberately: nixpkgs' nats module resolves
pkgs.nats-server itself and exposes none, so one here would name
something it cannot control.
No gateway vhost either: NATS speaks its own TCP protocol, nginx cannot
front it, and cross-hive reach is the wireguard mesh.
Runs on the host: the two containers share a network namespace but not a
filesystem root, so this is the only place both trees are addressable.
A copy rather than a bindMounts entry. nixos-container refuses to start
when a bind source is missing, and the secret does not exist until
authelia's first boot has minted it. The registration token dodges that
with an activation script that pre-creates the file; that is unavailable
here, because tuwunel requires the secret to exist and be non-empty, so a
placeholder would satisfy the mount and then stop the homeserver.
Bounded wait then fail, never a silent skip: authelia's container can be
up while its generator is still minting.
Stage 1's actual login path. The provider entry is rendered only when
sso.enable is set, via optionalAttrs rather than an empty list, so a hive
that has not opted in renders byte-identical tuwunel settings.
The secret reaches tuwunel through LoadCredential rather than as a direct
path: upstream requires the file to survive systemd sandboxing and
suggests /etc/tuwunel/, which this container has no writable etc for. The
registration token two units below already solves it the same way, and
for the same reason -- DynamicUser and PrivateUsers stay intact.
Three assertions fail at eval instead of at boot. tuwunel reads identity
providers from its config file, so a half-configured one does not hide a
login button, it can stop the homeserver from starting.
Stage 1 of the staged rollout mara laid out on the issue: the options
and nothing that changes a running hive. `enable` defaults false, so
this commit is inert until someone opts in.
Three rules taken from hive-forge/default.nix rather than re-derived,
since it is the same pattern already merged and reviewed:
- `enable` ADDS a login path and leaves password login alone. An
identity provider that can take the homeserver offline when it
hiccups is worse than two ways in. Making authelia the only path is
tuwunel's `login_with_password`, a separate and reversible switch,
deliberately not folded in here.
- the client secret is a PATH, never a value: it has two holders in two
containers (authelia keeps a hash, the homeserver needs plaintext),
and a literal would be rendered into the world-readable store.
- required when enabled, no fallback. Worse here than for the forge:
tuwunel reads OIDC from its config file rather than a database row,
so a malformed block can stop the server outright instead of merely
hiding a button.
Also records the constraint that shapes the whole issue — matrix SSO
lives inside the homeserver, never behind a forward-auth proxy,
because the client-server API is spoken by non-browser clients holding
matrix access tokens plus federation.
Per mara's review: "describe what is, not what is not."
The header said "the swarm services' vhosts are NOT here", which
describes the file by its absences and dates immediately — the list of
what is missing grows every time a service is added, while the list of
what the file holds does not.
Reworded to say what it is: the hive's own gateway surface (the `_`
default server and the vhost named for the hive domain), with the scope
line stating where a swarm service's vhost lives rather than where it
doesn't.
The header still announced "the forge, matrix and authelia sub-domain
vhosts" and an "Accept-header SPA map". None of the three is true:
- the four swarm vhosts moved into their own service modules, so the
header named contents that are no longer in the file;
- `_` is described as serving the dashboard and per-agent routing, but
it is now a bare 444 catch-all — that surface moved to the vhost
named for the hive domain;
- the "Accept-header SPA map" is not merely gone, it is the mechanism
the file's own dashboard comment argues *against* ("routing is by
PATH, never content-type"), 140 lines below the header claiming it.
A file's opening comment is its orientation text — the first thing read
and the last thing checked — so a stale one costs more than a stale
line in the body. Rewritten to describe what the file holds now, and to
name the property the split established: a service owns its whole
gateway surface in its own module, and a service name appearing here is
the signal that the coupling is growing back.
The `_` vhost was serving the hive's own surface, so every dashboard and
agent-UI request matched the default server rather than a named vhost --
and so did a request for any name at all, including a raw IP.
Split it: `_` keeps only `return 444`, and the hive surface moves to a
vhost named for the hive domain. `_` is `mkDefault` so an operator can
claim default_server themselves, plus an assertion for the case where
they add one without turning ours off -- nginx refuses to start on a
duplicate default_server and nixpkgs asserts nothing, so that would
otherwise surface as a gateway outage at rebuild time.
Clause 2 of #3202, reading 1 (mara: "the all local stuff and swarm
services auto conf belong in those mods, not spread all over").
`gateway.localHostsEntry` is the gateway's only local-deployment knob —
`openFirewall` is about EXTERNAL exposure, `tls.acme` needs a public DNS
name, `hsts` is a hardening choice. It is now asserted by the mode in
local-defaults.nix, beside the three swarm toggles, instead of being the
one all-local implication an operator still had to know about.
`mkDefault`, so "all local except this" still needs no new option.
⚠️ The non-obvious half: this does NOT change what CONTAINERS resolve.
dnsmasq sets `no-hosts = true` unconditionally, so agents keep getting
the bridge IP from the authoritative `address=` rules rather than the
host's 127.0.0.1 — which would point every agent at its own netns. That
guard already existing is what makes this safe to default on; without it
this one line would break every agent's access to the forge.
Last of the four. The vhost, its `auth_request` block and the swarm
apex's dns record move into swarm-ui.nix; vhosts.nix drops `uiCfg`,
`controllerCfg` and `autheliaCfg` and is now 259 lines of hive surface
with no swarm service in it.
Also collapses a THIRD copy of the per-service list. `networking.hosts`
restated every service's name with its own copy of that service's guard,
after the vhosts and the dnsmasq records had each done the same. It asks
the same question — which names does this host answer for — so it now
reads the same answer: a service added later lands in /etc/hosts with no
edit, and cannot land there under a different condition than it used for
DNS.
The `forceSSL`-not-`addSSL` comment travels intact: it records that
authelia answers an http auth subrequest with 400 and nginx's
auth_request only understands 2xx/401/403, so the scheme is load-bearing
for this vhost and no other.
Moves the matrix sub-domain vhost out of the gateway's vhosts.nix, its
`address=` rule out of dnsmasq.nix, and the Accept-header
`$matrix_spa_target` map out of the gateway's appendHttpConfig — all
three into hive-matrix.nix.
The map is the one that had no business being where it was: it exists
solely for the SPA fallback in the vhost's `/` location, and
`appendHttpConfig` is a `lines` option, so a module can contribute to
it without the gateway assembling it.
The `.well-known/matrix/*` delegation deliberately stays on the hive's
own vhost. The spec requires it at the SERVER NAME, which is the hive
domain: that is the hive answering "where is my homeserver", not the
homeserver answering for itself. Moving it would have been the obvious
symmetric thing and it would have been wrong.
Moves the authelia vhost out of the gateway's vhosts.nix and its
`address=` rule out of dnsmasq.nix, into swarm-authelia.nix.
Both land inside that module's existing `cfg.enable` guard, which is
the load-bearing part: every hive in a swarm knows `authelia.url`, but
only the host that RUNS the container may claim the name. A client hive
declaring the vhost would answer for a service it does not run, and
publishing the DNS record would point every agent on its bridge at that
wrong answer.
The kit grows a fourth member, `errorPages`, because the vhost aims its
502/503/504 at the gateway's styled sso-unavailable page. Republished
rather than imported per module: a service rendering its own would drift
from the rest of the gateway the first time the theme changed.
