217b3f1cea
The queue's metrics were scraped over loopback from the exporter, declared only where the queue and the collector share a host, so a collector on any other host had no job for it. Its target is now declared on every host as `https://<swarm.nats.domain>/metrics`, and the queue's host gets a gateway vhost on that name serving `= /metrics` from the exporter behind the collector's auth subrequest (`/__nats_metrics_authz`, the forge's shape). Every other path there is 403; the monitoring endpoint is never proxied. The rules derived from `publishedScrapeTargets` admit the collector's client and deny every other subject. The loopback entry, and with it `glue-otel-scrape-targets.nix`, is gone. In self-signed mode the vhost serves the queue's own `swarm-nats` leaf (issued for exactly that name, `server_flag` set), copied 0640 root:nginx by `swarm-nats-gateway-cert`; until the leaf exists that unit writes a throwaway pair, so nginx never starts on a missing file. The name stays out of `swarm.serviceDomains`, so a host whose only swarm service is the queue fronts no services name and its gateway neither waits for nor needs the swarm-services leaf: no services-issuer pair has to be placed on it. With ACME or an operator cert the vhost uses the gateway's own TLS, as every other name does. The queue's host now enables the gateway (`mkDefault`), as every other swarm service's host does. Fixes #5205
122 lines
5.1 KiB
Nix
122 lines
5.1 KiB
Nix
# The swarm's message queue as every hive sees it: where it is reached and
|
||
# what it is registered as, identical on every host. What the host running it
|
||
# decides, and the container itself, are in ./swarm-nats.nix.
|
||
{
|
||
lib,
|
||
config,
|
||
...
|
||
}:
|
||
let
|
||
swarmDomain = config.services.hyperhive.swarm.domain;
|
||
# Total on a null swarm domain, for the reason ./swarm-bao-service.nix gives.
|
||
domainBase = if swarmDomain == null then "invalid" else swarmDomain;
|
||
in
|
||
{
|
||
options.services.hyperhive.swarm.nats = {
|
||
# `enable` is `services.hyperhive.deploy.nats.enable` — see
|
||
# ./deploy.nix. What the queue IS to every hive: its domain, ports
|
||
# and client id.
|
||
|
||
# ⚠️ Deliberately NO `package` option for the NATS server, anywhere —
|
||
# not here and not under `deploy.nats` either, where every other
|
||
# service's build lives. `services.nats` upstream does not expose
|
||
# one; it resolves `pkgs.nats-server` itself, so an option would be
|
||
# ignored or need an overlay to mean anything, and an option that does
|
||
# not control what it names is worse than its absence. Pin the build
|
||
# with `nixpkgs.overlays` if you need to. (`deploy.nats.authPackage`
|
||
# is a different thing: the callout responder, which IS ours.)
|
||
|
||
domain = lib.mkOption {
|
||
type = lib.types.str;
|
||
default = "nats.${domainBase}";
|
||
defaultText = lib.literalExpression ''"nats.''${services.hyperhive.swarm.domain}"'';
|
||
description = ''
|
||
Name every client reaches the queue on, as
|
||
`tls://<domain>:<port>`, and the only name its certificate carries.
|
||
A **sibling** of the swarm's other service names, for the reason
|
||
{option}`services.hyperhive.swarm.bao.domain` gives.
|
||
|
||
The host running the queue answers it through `gateway.localNames`.
|
||
Every other hive resolves it through the operator's upstream DNS,
|
||
which is where a multi-host swarm needs a record for it.
|
||
|
||
Not in {option}`services.hyperhive.swarm.serviceDomains`. The gateway
|
||
on the queue's host serves `https://<domain>/metrics` to the swarm
|
||
collector, and in self-signed mode it serves the queue's own leaf
|
||
there rather than the swarm-services one, so that host never waits
|
||
for a services leaf (./swarm-nats.nix).
|
||
'';
|
||
};
|
||
|
||
port = lib.mkOption {
|
||
type = lib.types.port;
|
||
default = 4222;
|
||
description = ''
|
||
TCP port the queue listens on. 4222 is upstream's default and
|
||
sits outside hyperhive's claimed ranges (dashboard 7000, forge
|
||
3000, matrix 8008, every agent in 8100..8999 via FNV-1a hash).
|
||
|
||
Opened on the bridge interface, for agent containers, and on
|
||
`wg-hive` when this host is on the mesh, for other hives. Never
|
||
host-wide. TLS only: a client that does not speak it is refused.
|
||
|
||
Unlike {option}`monitorPort` and {option}`metricsPort`, the
|
||
address is not what bounds who may use this port: the queue's
|
||
`auth_callout` refuses every client it cannot identify.
|
||
'';
|
||
};
|
||
|
||
monitorPort = lib.mkOption {
|
||
type = lib.types.port;
|
||
default = 8222;
|
||
description = ''
|
||
Port NATS serves its **monitoring** endpoint on, bound to
|
||
loopback.
|
||
|
||
Not a metrics endpoint: the server has no Prometheus format of its
|
||
own. This serves `/varz`, `/connz`, `/routez` as JSON, and the
|
||
exporter below is what translates it — which is why enabling the
|
||
exporter without this produces a process that starts cleanly and
|
||
scrapes nothing.
|
||
|
||
⚠️ Loopback, and the exporter is the only intended reader. The
|
||
endpoint is unauthenticated and `/connz` names every connected
|
||
client, so the address it binds is the whole access control. Do
|
||
not widen it, and do not put it behind a gateway vhost expecting
|
||
that to add one.
|
||
'';
|
||
};
|
||
|
||
metricsPort = lib.mkOption {
|
||
type = lib.types.port;
|
||
default = 7777;
|
||
description = ''
|
||
Port the Prometheus exporter serves NATS's metrics on, bound to
|
||
loopback. The gateway's `= /metrics` location on
|
||
{option}`services.hyperhive.swarm.nats.domain` proxies it, behind
|
||
the swarm collector's authelia check.
|
||
|
||
⚠️ This and {option}`monitorPort` are two more claims on a port
|
||
space every swarm container shares — they run in this container but
|
||
`privateNetwork = false`, so a collision with any other hyperhive
|
||
service is a runtime coin toss over which process gets the port,
|
||
with nothing in any log saying so. Both defaults are upstream's
|
||
own (`nats-server` 8222, `prometheus-nats-exporter` 7777) and
|
||
neither is claimed elsewhere in this repo, checked when they were
|
||
added.
|
||
'';
|
||
};
|
||
|
||
clientId = lib.mkOption {
|
||
type = lib.types.str;
|
||
default = "swarm-nats";
|
||
description = ''
|
||
OAuth2 client id the queue's authentication path identifies
|
||
itself with. The host running authelia registers it, whichever
|
||
host runs the queue, and the queue's host reads its secret out of
|
||
the swarm secret store.
|
||
'';
|
||
};
|
||
|
||
};
|
||
}
|