Each container now runs the journal forwarder the CI runner's, Grafana's and
the log store's containers run, pushing to a receiver of its own on the swarm
collector (serviceProducers hive-forge and hive-matrix). Each authenticates as
a new machine OIDC client of its own (swarm.forge.otel.clientId,
swarm.matrix.otel.clientId), registered by a glue file wherever authelia runs,
minted by authelia and published to the store by the secret publisher. The
forge's and the homeserver's browser login clients are unchanged. The client
ids are swarm-forge-collector and swarm-matrix-collector, not
hive-<x>-collector, which is the client id a hive named <x>-collector gets.
The fetch unit inside each container reads the client secret with the store
identity the host already holds for that service. The forge publisher's leaf
is handed into the forge container as an nspawn system credential, as the
grafana and log store containers do. matrix-ctl's leaf is already bound into
the homeserver's container, so its forwarder reads it at those paths, as
swarm-matrix-appservice-publish does; with no matrix-ctl identity the
homeserver's forwarder is off. Each of the two policies gains one read stanza
on its forwarder's client path. The host units that use those identities are
unchanged.
The two containers drop --link-journal=host, and the host collector stops
reading them.
Each container now runs the journal forwarder the queue's and the builder's
containers run, pushing to a receiver of its own on the swarm collector. Each
authenticates as a new machine OIDC client of its own
(swarm.<service>.otel.clientId), registered by a glue file wherever authelia
runs, minted by authelia and published to the store by the secret publisher.
Grafana's browser login client is unchanged.
The fetch unit inside each container reads the client secret with the store
identity the host already holds for that service (the runner's, Grafana's,
and the log store responder's), now also handed into the container as an
nspawn system credential. Each of those three policies gains one read stanza
on its forwarder's client path. The host units that use those identities are
unchanged.
The three containers drop --link-journal=host, and the host collector stops
reading them.
The publisher's `enable` default read `deploy.authelia.enable` from the
publisher's own module (swarm-secret-publisher.nix). It now defaults to
`false`, and glue-secret-publisher-authelia.nix sets it `mkDefault true`
wherever authelia runs, so the effective value is unchanged. The option
loses its `defaultText`, and its description names the glue.
The `deploy.authelia.enable` read in `active` stays: every file the unit
copies is under authelia's `hostClientSecretDir`, so it is a requirement
of the unit, not a co-location toggle.
Part of #3742.
Thirty-three module-eval fixtures were a second (or later) full
`nixosSystem` evaluation of a config another fixture already evaluated:
16 shapes defined identically in two to seven cluster files, plus
duplicates inside agent-queue-bao, core-toggle and
swarm-services-switch. Each cluster imports ./lib.nix itself, so nothing
was shared between them.
nix/module-eval/shared.nix now defines each of those 16 shapes once.
nix/checks.nix instantiates it a single time and passes it as `shared`
to the 23 clusters that read one, and each cluster rebinds the fixture
under its existing local name. Assertions are unchanged. Case text
changes only in core-toggle, where two cases built the forge-host
fixture inline and now read the file's `forgeHere`, and in two inline
`let` bindings (bao-otel-collector, grafana) that now take the shared
fixture.
Six folds are value-equal rather than textually equal: they drop an
explicit `deploy.hive-controller.enable = false` or
`deploy.swarm-otel.enable = false` that equals what the option already
resolves to for that config. Nothing under nix/host-modules sets either
option apart from the `mkDefault allSwarmServices` on swarm-otel, and
the only option-priority read there is `domain.highestPrio`.
Part of #5171.
deploy.review-classifier.enable (off by default, not derived from
allSwarmServices) puts swarm-review-classifier in its own network
namespace on the bridge, with claude-code and the daemon. A host unit
copies the classifier's store leaf into the container; the daemon reads
its queue client secret and the API key from the store with it.
- glue-bao-tls mints the review-classifier leaf; swarm-bao writes the
swarm-review-classifier role, reading exactly the API key path and the
classifier's queue client secret; swarm.nix reserves its CN.
- glue-review-classifier-oidc-client registers the machine client where
authelia runs, and the publisher carries its secret to the store.
- swarm-nats passes the client id to the responder as
--review-classifier-client.
- Where the host runs a hive-tier collector, claude exports its metrics
there; the daemon adds the repo and the PR to the resource attributes.
- module-eval: the new grant and leaf (bao-grants, now sixteen granting
units), the container's presence and its unit (core-toggle), and the
client registration and publishing (secret-publisher).
swarm-secret-publish.service was Type=oneshot with RemainAfterExit=true.
After its first successful run it stayed active (exited), so every later
start from swarm-secret-publish.path (PathChanged on authelia's client
secret dir) was a no-op: a secret minted or replaced after that first run
never reached the store, and nothing logged it.
Without RemainAfterExit the unit goes inactive after each run, so each
PathChanged event starts a fresh publish. Same shape as
swarm-bao-forge-publisher in hive-forge/default.nix. No unit requires,
wants, orders after, binds to or is part of swarm-secret-publish.service,
so nothing depended on it staying active.
This also covers the first-boot race where the publisher runs before
authelia's first mint: the mint writes the file, and the path unit now
starts the publisher again.
A module-eval case pins it: the path unit targets the service, the
service is a oneshot, and RemainAfterExit is unset or false.
Closes#5032
The forge's login source read its OIDC client secret from a file that
`hive-forge-oidc-secret` copied out of authelia's container tree. That
unit only existed where authelia ran on the forge's host (`ssoLocal`),
so a forge anywhere else needed the file placed by hand.
The publisher on authelia's host now carries the forge's client secret
to `secret/swarm/services/<forge client id>/oidc/client`, like every
other swarm service's. On the forge's host `swarm-bao-forge-oidc`, a copy
of `swarm-bao-matrix-oidc`, reads it under the forge publisher's leaf
(the forge host's one store identity, whose policy gains read on that
one path) and writes the file `forgejo-sso-source` reads, re-running the
login source when the value changes. It retries on the shared
store-retry shape while the store is down or the secret is not yet
published.
`hive-forge-oidc-secret` is removed, `clientSecretFile` defaults to the
in-container path wherever the forge runs, and the `ssoLocal` gate on
the login source's resolver ordering goes with it: the login source
fetches the issuer's discovery document on every forge host.
Refs #5015
hive-c0re publishes this hive's status to the swarm queue as the OIDC
client `hive-<name>`. Its secret came from
`deploy.hive-controller.statusPublish.clientSecretFile`, which defaulted
to authelia's minted file only on a host running both authelia and the
queue. Every other hive had no secret and never published, so the
controller showed it `never_reported`.
Now `swarm-secret-publish`, beside authelia, also copies each hive's
`hive-<name>` secret to `swarm/hives/<hive>/oidc/client`. Its policy
already allows `swarm/hives/*`. hive-c0re reads that path under the
hive's own store leaf, which the existing `swarm/hives/<hive>/*` grant
covers, and holds the value in memory.
The shared queue client no longer remembers a failed first connect for
the life of the process. A failed attempt is retried with a backoff
from 5s, doubling to a cap of 5 minutes; the dashboard banner stays up
until an attempt succeeds. The two status publishers wait for the
client instead of giving up on the first `None`, so a secret published
after boot is picked up without a restart.
Removed: the `statusPublish.clientSecretFile` option (both spellings,
now removed options with a message), its same-host default, its
LoadCredential, the assertion tying it to the other two coordinates,
and `swarm_queue_client::QueueConfig::from_env`, whose only caller was
this read. Publishing is now on wherever the queue URL and token
endpoint are set and the hive holds a store leaf.
Refs #5015
Rewrite comments and option descriptions under nix/ that narrate how the
code got here ("used to", "new name", "sat at the TOP", "no longer",
renamed/moved narration, attributions) to say what the code does now and
why. Comments and description strings only; no logic changes.
Files in open PR #4970's diff are left alone. Refs #3902
The shared `hive` fixture turned `deploy.hive-controller.enable` on for
every eval test. It is now off by default; `hiveWithController` turns it
on, and the fixtures whose claims read the hive tier (hive-c0re, the
queue-agent credential, the hive leaf, `deploy.hive-controller.*`, or an
absence that is only meaningful with a hive present) use it.
glue-nats-auth-bao-identity.nix still read the renamed
`services.hyperhive.enable`, which printed the "Obsolete option" trace
six times per CI run. The term is dropped: the pairing is gated on the
queue and the minted PKI, as its sibling glue-nats-bao-identity.nix is.
The three client registrations move into glue-{controller,nats,matrix}-oidc-client.nix,
gated only on deploy.authelia.enable and reading their ids from swarm.* options,
so the host running authelia registers every one of them whichever host runs
the service. The matrix callback is a new read-only swarm.matrix.sso.redirectUri,
read by both the homeserver's config and its client entry.
swarm-secret-publish now also publishes the queue's and the homeserver's minted
secrets to swarm/services/<id>/oidc/client. The queue host reads its responder's
secret there with swarm-bao-nats-auth-oidc under the responder's own leaf
(swarm-nats-auth role gains that one read path), and the homeserver's host with
swarm-bao-matrix-oidc under matrix-ctl's leaf (swarm-matrix-ctl role gains its
one read path). The host-file copies are gone: swarm-nats-auth-secrets now
delivers only the callout seeds and the store identity, and
hive-matrix-oidc-secret is removed. A fetch unit whose leaf is unset still
renders and fails naming the option pair.
Closes#4943
Every hive's homeserver loaded a per-hive `hyperhive` appservice
registration, and the swarm minted and delivered an as_token for it
through bao. No client presented that token: swarm-controller creates
every `@hive-<hive>` sender and every agent account with the swarm
appservice token (hive_sender.rs, agent_token.rs), and no Rust code read
the per-hive one.
Removed, with everything that existed only to serve it:
- hive-matrix.nix: the `hyperhive` registration, its activation-time
render, the host bind mount of /var/lib/hyperhive/matrix-appservice,
its LoadCredential entry, and the internal options
`deploy.matrix.appserviceTokenFile` / `appserviceRegistrationScript`.
tuwunel now loads only the swarm registration.
- glue-matrix-bao-token.nix and its `swarm-bao-matrix-token` unit.
- glue-bao-tls.nix: the matrix-token leaf and its defaults.
- swarm-bao.nix: the `swarm-matrix-token-<hive>` roles and policies
(`swarm-bao-matrix-token-policy`), and the options
`deploy.bao.matrixTokenClient{Cert,Key}File` and
`matrixTokenCommonNamePrefix`, now mkRemovedOptionModule.
- swarm-secret-publisher.nix: the per-hive token mint and its state dir.
- swarm.nix: the reserved `<matrixTokenCommonNamePrefix>-<hive>` CN.
- swarm-secret-client: `matrix::appservice_token_path`.
The swarm registration (swarm-matrix-ctl, swarm-controller) is untouched.
Comments and docs that pointed at the removed glue file now point at
glue-queue-agent-credential.nix, which has the same reader shape.
Refs #4428
The hive policy rendered by policy::render granted every hive cert read
on secret/data/swarm/services/*, which holds each swarm service's OIDC
client secret. No reader of that tree authenticates with a hive cert:
grafana, the otel collector and the store forwarder each read with
their own leaf, and the bao UI's secret is read by the store's granter.
render() now emits two stanzas: swarm/agents/* and the hive's own
swarm/hives/<hive>/*. Kind::Service had no other caller and is removed.
Docs and comments that described the grant as a decision are rewritten
to the current state.
Closes#4960
The bao UI at bao-ui.<swarm> took a raw store token and nothing else.
It now offers an OIDC tab: authelia's `admins` group logs in and lands
on `swarm-operator-viewer`, which is list+read on `secret/metadata/*`
and nothing under `secret/data/` or `sys/`.
- authelia registers an interactive client `swarm-bao-ui`
(glue-bao-ui-oidc-client.nix) with redirect
`https://bao-ui.<swarm>/ui/vault/auth/oidc/oidc/callback`; the secret
publisher carries its secret to
`secret/swarm/services/swarm-bao-ui/oidc/client`.
- `swarm-bao-granter-role` (bootstrap token) enables the `oidc` auth
mount with listing visibility `unauth`, asked before attempted like
cert/approle; `bao-bootstrap-policy.hcl` gains `sys/auth/oidc`.
- The granter's policy gains `auth/oidc/config`, `auth/oidc/role/swarm-*`
and read on that one secret leaf. It still holds no `sys/auth`.
- New granting unit `swarm-bao-operator-viewer-policy` writes the viewer
policy, and once the granter may configure `auth/oidc/config` (checked
through `sys/capabilities-self`), writes the mount's config from the
published secret and the role binding `groups=admins` to the viewer.
Before the bootstrap step re-runs it writes the policy, logs the step
and exits 0.
Route (a) per mara on #4775: enabling the auth method stays a
bootstrap-token step, re-run once on the live store.
module-eval pins the viewer policy's single metadata stanza, that the
granter's policy has no sys/auth path, the oidc enable in the bootstrap
unit, the exit-0 path, the config/role contents, and the client
registration + publish.
The controller's OIDC client secret (client `swarm-controller`, used for
the queue connection, the auth-bridge bearer and the OTLP push) came from
an operator-placed file, `deploy.swarm-controller.queue.clientSecretFile`,
handed in by `LoadCredential=`.
Now `swarm-secret-publish`, which already copies authelia's minted OIDC
secrets into the store, also publishes this one, to
`swarm/controller/swarm-controller/oidc/client`. That path sits under
`controller/`, which no hive's policy reads. The controller reads it once
at start with its existing store certificate and holds it in memory, as
`swarm_queue_client::ClientSecret::Value`. If the store is down, it
retries for about a minute and then fails the start, so `Restart=` tries
again.
Policy delta: the controller gets `read` on that leaf, and the publisher
gets `create`/`update` on that leaf.
Removed: the `queue.clientSecretFile` option (both spellings, now removed
options with a message), its singleHostSwarm default, the credential and
placeholder, and the path watcher plus its restart oneshot. A controller
without a store identity is now an eval error, because it has no other
way to get the secret.
bao's metrics were scraped by the SWARM collector over loopback, via a
`swarm.otel.scrapeTargets.bao` entry gated on `deploy.swarm-otel.enable`
— "does the swarm's collector run on THIS host". It had to be: loopback
only reaches a reader that landed on the same host.
What that rendered everywhere else was nothing at all. Off that host the
metrics listener was not emitted, so the store's metrics reached the
store nowhere, and a host with no entry is indistinguishable from a host
nobody asked to scrape.
Moves the scrape into the collector this container already runs, per
mara on #4537: "move the existing scraper to the local collector". The
container shares the host netns (privateNetwork = false), so the scrape
still dials 127.0.0.1 — the listener keeps its address, its
`metrics_only` narrowing and its loopback-only bind, and the API
listener's `tls_require_and_verify_client_cert` is untouched.
The listener and its `prometheus_retention_time` lose their gate: the
reader ships with the store now, so there is no host where the endpoint
has none. The metrics pipeline reuses the logs pipeline's `resource`
processor and `otlphttp` exporter, so both signals carry the same
`service.name` and leave by the one hop.
Logs are unaffected: `journaldUnits` and --link-journal=host stay until
every sibling swarm container has a collector of its own.
The module-eval absence arm "a store with no collector beside it serves
no metrics" is inverted rather than dropped — the condition it asserted
is the bug. Three cases join it: the job is in swarm-bao AND gone from
swarm-otel (a move, not a copy), the scrape target and listener are both
pinned to loopback, and the metrics pipeline shares its exporter with
the logs one.
The single module-eval derivation forced ~62 full nixosSystem
fixtures live at once to compute its cases list: 10.6GB peak RSS /
5m25s to evaluate, by far the dominant cost in nix flake check.
Splits it into 21 independent checks.module-eval-* derivations
(1-7 fixtures each) sharing builders/helpers via module-eval/lib.nix,
so no single derivation needs more than a handful of fixtures live
at once. A few cases spanning two clusters carry a small duplicated
fixture rather than threading shared state through lib.nix.
