The queue's metrics were scraped over loopback from the exporter, declared
only where the queue and the collector share a host, so a collector on any
other host had no job for it. Its target is now declared on every host as
`https://<swarm.nats.domain>/metrics`, and the queue's host gets a gateway
vhost on that name serving `= /metrics` from the exporter behind the
collector's auth subrequest (`/__nats_metrics_authz`, the forge's shape).
Every other path there is 403; the monitoring endpoint is never proxied.
The rules derived from `publishedScrapeTargets` admit the collector's
client and deny every other subject. The loopback entry, and with it
`glue-otel-scrape-targets.nix`, is gone.
In self-signed mode the vhost serves the queue's own `swarm-nats` leaf
(issued for exactly that name, `server_flag` set), copied 0640
root:nginx by `swarm-nats-gateway-cert`; until the leaf exists that unit
writes a throwaway pair, so nginx never starts on a missing file. The
name stays out of `swarm.serviceDomains`, so a host whose only swarm
service is the queue fronts no services name and its gateway neither
waits for nor needs the swarm-services leaf: no services-issuer pair has
to be placed on it. With ACME or an operator cert the vhost uses the
gateway's own TLS, as every other name does.
The queue's host now enables the gateway (`mkDefault`), as every other
swarm service's host does.
Fixes#5205
Grafana's own metrics were scraped over loopback through the
`grafana-metrics` nginx vhost, declared only where grafana and the
collector share a host, so a collector on any other host had no job for
it. Its target is now declared on every host as
`https://<swarm.grafana.domain>/metrics`, next to the other published
targets, and grafana's vhost serves `= /metrics` behind the collector's
auth subrequest (`/__grafana_metrics_authz`, the forge's shape). The
rules derived from `publishedScrapeTargets` admit the collector's client
there and deny every other subject.
The loopback scrape is gone with the vhost that served it and its
`swarm.grafana.metricsPort` option, so a single host still scrapes
grafana once, now by name. A config that sets that option now fails
evaluation with "option does not exist".
On the published vhost `/metrics` was answered with 403 for everyone.
It now answers the collector's token, and a plain session gets authelia's
`deny`. The design note that ruled the published vhost out for scraping
predates the collector's credential; it is replaced by the location's
own comment and a line on `swarm.grafana.domain`.
Part of #5205
The apex's `operatorsOnly` pair was guarded on
`deploy.swarm-controller.enable`, which is true only on the host that
runs the controller and with it the swarm UI's vhost. When authelia runs
on another host, that vhost still asks it, but the authelia host
rendered no rule for the apex at all, so every session reached the
swarm UI and its `/api/` through `default_policy = one_factor`.
The pair now renders whenever the swarm domain is set, like the bao UI's
pair. No vhost on a host that doesn't serve the apex asks about it, so
the extra rules there are inert. authelia 4.39.20's `validate-config`
accepts them on an authelia-only host's rendered access_control.
module-eval-swarm-otel-identity gains a case on the existing
`autheliaOnly` fixture (authelia without the controller): the apex has
exactly the allow+deny pair, the rule list is closed, and an account in
no group is refused `/api/agents` while `group:admins` passes.
Fixes#5120
authelia takes the first matching rule, and a rule with a `subject` does
not match a subject outside it. The swarm domain's and the bao UI's
`group:admins` rules had nothing after them, so an account in no group
skipped them and landed on `default_policy = one_factor`: any
authenticated account reached the swarm UI, its `/api/` (by cookie or by
bearer) and the bao UI.
Each of the two group rules is now an `operatorsOnly` pair: the group
allow, then a subject-less `deny` for the same domain, the shape
`collectorOnly` already gives the collector's metrics rules.
`default_policy` stays `one_factor`, which the forge and chat rely on.
module-eval-swarm-otel-identity gains two cases on `allLocal`: an
account in no group is refused the swarm domain (four paths) and the bao
UI while `group:admins` passes and the forge stays open; and a
predicate, `autheliaSubjectRulesClosed` in lib.nix, holds over the
rendered list: every subject-scoped allow is closed by a later
subject-less `deny` on its domain with no subject-less allow in between,
and nothing follows a domain-wide `deny` on its domain.
Fixes#5120
The metrics store's container gets privateNetwork = true with no
interface but loopback. VictoriaMetrics listens on
unix:/run/swarm-victoriametrics/victoriametrics.sock, a host directory
(root:nginx 0770) bind-mounted into the container, and every vhost
location that forwarded to 127.0.0.1:8428 now forwards to that socket.
The store runs as a static user whose primary group is nginx's gid,
with UMask 0007, so the socket it creates is 0770 group nginx. This
relies on VictoriaMetrics >= 1.153.0 honouring the umask (the locked
nixpkgs ships 1.153.0). nixpkgs' readiness loop curls a TCP URL, so
postStart is replaced with the same loop over --unix-socket.
swarm.victoriametrics.port is removed (nothing binds it any more), with
a mkRemovedOptionModule entry, and dropped from swarm-otel's host
port-collision assertion. The container's resolver unit is dropped:
there is no interface to reach the bridge resolver through, and the
store resolves no names.
The collector scrape, the collector push and Grafana's datasource
already reach the store by its published domain, so they are unchanged.
Refs #4815
Refs #3560
VictoriaLogs' own metrics were scraped over loopback, declared only on
the host running the store, so a collector on any other host had no job
for it. Its target is now declared on every host as
`https://<swarm.victorialogs.domain>/metrics`, next to forgejo's and
authelia's, and the store's vhost serves `= /metrics` behind a bare
`auth_request` to authelia. The rules derived from
`publishedScrapeTargets` admit the collector's client there and deny every
other subject. The loopback entry is gone, so a single host still scrapes
the store once, now by name.
On that vhost `/metrics` was reachable through the `/` location by any
authelia session. The exact-match location now takes it, so a session
gets a `deny` on `/metrics` and nothing else on the name changes.
Part of #5205
No access rule covered the push route on `metrics.<swarm>`, so a request
there fell to `default_policy = one_factor`: any logged-in session cookie
for the swarm could write samples into VictoriaMetrics.
The name now gets two rules with no `resources`: the collector's client
at `one_factor`, then `deny`. Only the push route and `/metrics` on that
vhost send an authelia subrequest (the read API, Grafana's datasource
included, is served with no check), and the collector is the one caller
of both, so a domain-wide pair covers them with no path or query regex
to slip past. It replaces the per-path `/metrics` pair the published
target would otherwise derive for that name. Both rules sit ahead of the
apex and store-UI rules, and authelia matches `domain` exactly, so they
shadow no other name.
Authelia 4.39.20, read from source: a rule with no `domain` wildcard
matches with `strings.EqualFold` (access_control_domain.go), an empty
`resources` or `subject` list matches everything (access_control_rule.go),
and the first matching rule wins (authorizer.go).
Fixes#5218
The store's own /metrics was scraped over loopback, declared under
deploy.victoriametrics.enable, so a collector on another host never had
a job for it. Declare it as a published target on every host instead,
keyed on swarm.victoriametrics.domain, and serve it on the metrics vhost
behind the existing authz subrequest. Authelia's per-target rules (one
collector rule, then deny) now cover it too.
The loopback entry goes, so a single-host swarm scrapes the store once.
#5215 moved authelia's scrape to `publishedScrapeTargets` and removed
its loopback entry from the authelia module. It was rebased at merge
onto c06aa78b6, which had already moved that loopback entry into
glue-otel-scrape-targets.nix, so the entry survived there. Every host
running authelia and the collector, `singleHostSwarm` included, then
failed swarm-otel.nix's job-name assertion.
The glue now declares only grafana and nats. forge-placement's case
that pinned authelia on loopback now pins its absence, and
swarm-otel-identity gains a case reading the assertion itself on the
single-host fixture, since the option-level case cannot fail a
rebuild the way the assertion does.
Fixes#5247
Authelia 4.39.20 matches `resources` against the cleaned path plus
`?<query>` (authorization/types.go:94 NewObject, utils/url.go:11-34
URLPathFullClean), while nginx's `= /metrics` serves the request whatever
its query. `^/metrics$` therefore missed `/metrics?x=1`: both the collector
rule and the trailing `deny` were skipped and a plain session reached
`default_policy = one_factor`. The pattern is now `^<path>([?].*)?$` for
every published target.
Forgejo strips trailing slashes before routing (routers/common/
middleware.go stripSlashesMiddleware) and serves metrics without a token
when none is configured, so `forge.<swarm>/metrics/` reached its metrics
through the ungated `/` location. The forge vhost now answers
`= /metrics/` with 404; nginx merges slashes and decodes `%2F` before
matching, so that also covers `/metrics//` and `/metrics%2F`.
Part of #5205
Authelia's metrics were scraped over loopback, declared only on a host
running both authelia and the collector, so a split deploy collected none.
Its target is now declared on every host as
`https://<swarm.authelia.domain>/metrics`, next to forgejo's, and authelia's
vhost serves `= /metrics` behind the same collector auth subrequest as the
forge's. The loopback entry is gone, so a single host still scrapes
authelia once, now by name.
The forge-only `metricsRule` becomes `metricsRules`, derived from every
`publishedScrapeTargets` entry. Each target gets the collector-scoped
`one_factor` rule followed by an unconditional `deny`. The `deny` is
needed because authelia 4.39 skips a rule whose `subject` does not
match and moves on. With nothing after the collector rule, any other
authenticated session reached `default_policy = one_factor` and was
admitted. That was already true of the forge's `/metrics`, and this
change closes it there too.
Part of #5205
The forgejo entry in `swarm.otel.publishedScrapeTargets` was declared
under `deploy.forgejo.enable`, so it existed only in the forge host's
evaluation. Its two readers evaluate elsewhere: the collector renders
its prometheus jobs from its own host's list, and authelia registers
the collector client's audience from its own host's list. With the
forge, the collector and authelia on different hosts, the collector
had no forgejo job and authelia would have refused its token.
Declare the entry unconditionally in swarm-otel-service.nix, keyed on
`swarm.forge.domain`, which every host evaluates identically. A swarm
with no forge scrapes a name that answers nothing and shows a failing
job, the same trade the collector and grafana OIDC clients already make.
Single-host render is unchanged: the entry already existed there.
Part of #5205
Each store reader (forge's OIDC fetch and token publisher, matrix's OIDC
fetch, the hive collector's fetch, Grafana's OIDC fetch and admin password,
the queue's callout-xkey unit, the CI runner's token fetch, the swarm
collector's OIDC fetch) used to read `deploy.bao.enable` in its own module
to add `after`/`wants` on `container@<store machine>` and
`after`/`requires` on `swarm-bao-pki.service`. Per the #3742 ruling, that
cross-service read moves into glue-bao-readers-store-order.nix, which sets
the same edges when the store and the reader are both on this host and
restates each reader's own gate so no stub unit is rendered.
Rendered `after`/`wants`/`requires` are identical as sets to main on 16
fixtures, store here and elsewhere. New module-eval cases in existing
fixtures pin both arms.
Part of #3742.
Thirty-three module-eval fixtures were a second (or later) full
`nixosSystem` evaluation of a config another fixture already evaluated:
16 shapes defined identically in two to seven cluster files, plus
duplicates inside agent-queue-bao, core-toggle and
swarm-services-switch. Each cluster imports ./lib.nix itself, so nothing
was shared between them.
nix/module-eval/shared.nix now defines each of those 16 shapes once.
nix/checks.nix instantiates it a single time and passes it as `shared`
to the 23 clusters that read one, and each cluster rebinds the fixture
under its existing local name. Assertions are unchanged. Case text
changes only in core-toggle, where two cases built the forge-host
fixture inline and now read the file's `forgeHere`, and in two inline
`let` bindings (bao-otel-collector, grafana) that now take the shared
fixture.
Six folds are value-equal rather than textually equal: they drop an
explicit `deploy.hive-controller.enable = false` or
`deploy.swarm-otel.enable = false` that equals what the option already
resolves to for that config. Nothing under nix/host-modules sets either
option apart from the `mkDefault allSwarmServices` on swarm-otel, and
the only option-priority read there is `domain.highestPrio`.
Part of #5171.
Drop the eval assertion b78190e1 added. It refused every self-signed
gateway host that does not run the store, which is the default hive
shape, and contradicts docs/swarm/secrets.md's supported no-store
deployment. The approved #4921 plan has such a host evaluate and fail
at runtime; `requireServicesIssuer` already exits 1 naming both
options, on the root-fetch branch and before issuing.
The module-eval case that pinned `hiveWithController { }` as refused
now pins it, and a hive-less gateway with no pair, as evaluating with
the unit's refusal in the script. The fixture pairs the assertion
forced into forge-placement, nats-tls and swarm-otel-identity are
reverted.
Refs #4921
swarm-services-cert issues the swarm-services leaf a self-signed gateway
serves, and on a host fronting no swarm name fetches the store's pki root.
It is gated on the gateway, not on a hive, yet its leaf pair was named
under deploy.hive-controller.tls and defaulted from the hive reader's cert
directory, and the root fetch presented the hive's own identity. A gateway
host with no hive had no such identity, so that branch was red by
construction.
- The pair moves to deploy.bao.servicesIssuerClient{Cert,Key}File, beside
the store's other host-unit readers, and glue-bao-tls.nix defaults it on
the store's host from the leaf it already mints. The old
deploy.hive-controller.tls.baoClient{Cert,Key}File paths are
mkRenamedOptionModule aliases. glue-services-issuer-bao-identity.nix,
which derived the default from deploy.bao.clientCertFile, is gone.
- The no-swarm-name root fetch presents the services-issuer pair. The hive
identity is now only hive-tls-resign's.
- A self-signed gateway with no pair fails evaluation, naming both options.
A pair named but unreadable still fails the unit, naming the paths.
Refs #4921
The four-way client-cert split gives each store reader its own leaf, and
three of the four readers render only where their own leaf exists. On a
host that mints its own PKI glue-bao-tls.nix defaults all eight, so there
is nothing to do; on a hand-configured remote-store hive, omitting one
pair used to mean that unit silently did not render — a privilege-
narrowing unit absent from a green build, with the missing unit as the
only evidence.
Each of the three now asserts its own pair, shaped after
swarm-grafana.nix's haveClientIdentity assertion and named to the pair it
needs. What differs from Grafana's is the gate: these fire only where the
host demonstrably reads the store (it holds deploy.bao.clientCertFile and
clientKeyFile) and the consumer is on. A host with no store identity is
the supported no-store deployment and still evaluates; the collector's
no-secret degrade is untouched, because that host holds no clientCertFile
either.
Also rewords three passive-voice sentences in docs/swarm/secrets.md that
vale flagged, and documents what the refusal costs and where it stays
silent.
The single module-eval derivation forced ~62 full nixosSystem
fixtures live at once to compute its cases list: 10.6GB peak RSS /
5m25s to evaluate, by far the dominant cost in nix flake check.
Splits it into 21 independent checks.module-eval-* derivations
(1-7 fixtures each) sharing builders/helpers via module-eval/lib.nix,
so no single derivation needs more than a handful of fixtures live
at once. A few cases spanning two clusters carry a small duplicated
fixture rather than threading shared state through lib.nix.
