Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7680b07fcb | ||
|
|
41bea3c6d9 | ||
|
|
1325f44707 | ||
|
|
ec562ca503 | ||
|
|
6b1fc4b040 | ||
|
|
e226e65954 |
12 changed files with 506 additions and 253 deletions
|
|
@ -330,10 +330,22 @@ go there is `deploy.swarm-otel.upstream.signals` (default `[ "metrics" "logs" ]`
|
|||
always get both. It holds
|
||||
the upstream credential, which is why no hive and no agent needs one.
|
||||
|
||||
It runs in a `swarm-otel` container. Its `swarm.otel.port` defaults to
|
||||
`4319` rather than OTLP's usual `4318`, which the hive tier uses — swarm
|
||||
containers share the host's network namespace, so two collectors on one
|
||||
port is a coin toss at runtime rather than an error at build time.
|
||||
It runs in a `swarm-otel` container on the hive bridge, in a network
|
||||
namespace of its own. It binds no OTLP port: every receiver (one per hive,
|
||||
the swarm tier's and one per service forwarder) listens on a unix socket in
|
||||
`/run/swarm-otel`, a directory only the gateway's nginx and the container can
|
||||
open, and the gateway forwards to them. The gateway's routes and the collector's
|
||||
receivers come from one list, so they name the same sockets. The
|
||||
`swarm-otel-sockets` unit connects to each socket as the gateway's user after
|
||||
the container starts, then requests every route through the gateway vhost. It
|
||||
fails if a socket refuses for 30 seconds, or if a route answers `000`, `502` or
|
||||
`504`, or lacks its own `X-Swarm-Otel-Route` response header, for 30 seconds.
|
||||
The gateway sets that header, named after the route, on every response from a
|
||||
route, so a collector that's up but not listening, or an nginx that doesn't
|
||||
have the route, shows as a failed unit. Any status other than those passes,
|
||||
the collector's 4xx included. A switch leaves a window of about 20 seconds in which
|
||||
nginx and the container are out of step: the container stops before nginx
|
||||
restarts and starts about 18 seconds after it. Senders retry through that window.
|
||||
|
||||
Every hive's own collector reaches this one by its gateway name,
|
||||
`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
|
||||
|
|
@ -345,7 +357,6 @@ gateway either way.
|
|||
| Option | When you'd touch it |
|
||||
| ------------------- | --------------------------------------------------------------------------------------- |
|
||||
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
|
||||
| `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
|
||||
|
||||
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
|
||||
`swarm-notify-router`, `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
|
||||
|
|
|
|||
|
|
@ -69,6 +69,19 @@ in
|
|||
own, and the gateway reaches it there, so it binds no host port. Remove
|
||||
this definition.
|
||||
'')
|
||||
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "otel" "port" ] ''
|
||||
The swarm collector's OTLP receivers listen on unix sockets in a network
|
||||
namespace of its own, and the gateway reaches them there, so there is no
|
||||
receiver port range. Remove this definition.
|
||||
'')
|
||||
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "otel" "producerPort" ] ''
|
||||
The swarm-tier receiver listens on a unix socket, so it binds no port.
|
||||
Remove this definition.
|
||||
'')
|
||||
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "otel" "storeProducerPort" ] ''
|
||||
The store forwarder's receiver listens on a unix socket, so it binds no
|
||||
port. Remove this definition.
|
||||
'')
|
||||
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "authelia" "port" ] ''
|
||||
authelia listens on a unix socket in a network namespace of its own, and
|
||||
the gateway reaches it there, so it binds no port. Remove this
|
||||
|
|
|
|||
|
|
@ -458,14 +458,13 @@ in
|
|||
assertion =
|
||||
!(lib.elem forwarderTelemetryPort [
|
||||
config.services.hyperhive.otel.telemetryPort
|
||||
config.services.hyperhive.swarm.otel.telemetryPort
|
||||
config.services.hyperhive.swarm.bao.otel.telemetryPort
|
||||
config.services.hyperhive.swarm.nats.otel.telemetryPort
|
||||
]);
|
||||
message = ''
|
||||
The forge's journal forwarder's telemetry port (${toString forwarderTelemetryPort})
|
||||
collides with another collector's in the host netns: the hive
|
||||
collector's, the swarm collector's, the store forwarder's or the
|
||||
collector's, the store forwarder's or the
|
||||
queue forwarder's.
|
||||
'';
|
||||
}
|
||||
|
|
|
|||
|
|
@ -144,9 +144,7 @@ in
|
|||
the value *configured* rather than implied, which is the point of
|
||||
the option. An undeclared port is invisible to anything comparing
|
||||
ports, so a second collector on the same host picking the same
|
||||
number fails at `bind()` rather than at eval —
|
||||
`services.hyperhive.swarm.otel.telemetryPort` sidesteps 8888 by
|
||||
hand for that reason.
|
||||
number fails at `bind()` rather than at eval.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -136,12 +136,10 @@ in
|
|||
failures. How you find out that telemetry is being lost, so it is
|
||||
worth keeping rather than switching off.
|
||||
|
||||
⚠️ **Deliberately neither 8888 nor 8889.** 8888 is the collector
|
||||
⚠️ **Deliberately not 8888.** 8888 is the collector
|
||||
binary's built-in default, which the hive tier
|
||||
({option}`services.hyperhive.otel.telemetryPort`) already binds, and
|
||||
8889 is the swarm tier's
|
||||
({option}`services.hyperhive.swarm.otel.telemetryPort`). This
|
||||
container runs with `privateNetwork = false`, so all three share the
|
||||
({option}`services.hyperhive.otel.telemetryPort`) already binds. This
|
||||
container runs with `privateNetwork = false`, so both share the
|
||||
host's network namespace whenever they are co-located — and unlike
|
||||
the OTLP ports this one appears nowhere in either config when it is
|
||||
left undeclared, so nothing that compares configured ports can see
|
||||
|
|
|
|||
|
|
@ -115,10 +115,8 @@ in
|
|||
serves its own metrics on (refused and dropped records, exporter
|
||||
failures).
|
||||
|
||||
⚠️ Not 8888, 8889 or 8890: the hive collector, the swarm collector
|
||||
and the store's forwarder bind those
|
||||
({option}`services.hyperhive.otel.telemetryPort`,
|
||||
{option}`services.hyperhive.swarm.otel.telemetryPort`,
|
||||
⚠️ Not 8888 or 8890: the hive collector and the store's forwarder
|
||||
bind those ({option}`services.hyperhive.otel.telemetryPort`,
|
||||
{option}`services.hyperhive.swarm.bao.otel.telemetryPort`), and this
|
||||
container shares the host netns with them when co-located. The second
|
||||
collector to bind a port dies with `bind: address already in use`.
|
||||
|
|
|
|||
|
|
@ -603,7 +603,6 @@ in
|
|||
cfg.metricsPort
|
||||
cfg.otel.telemetryPort
|
||||
config.services.hyperhive.otel.telemetryPort
|
||||
swarmOtelCfg.telemetryPort
|
||||
config.services.hyperhive.swarm.bao.otel.telemetryPort
|
||||
];
|
||||
in
|
||||
|
|
@ -612,7 +611,7 @@ in
|
|||
services.hyperhive.swarm.nats.otel.telemetryPort
|
||||
(${toString cfg.otel.telemetryPort}) collides with another port in
|
||||
the host netns: the queue's own ports, or the telemetry port of the
|
||||
hive collector, the swarm collector or the store's forwarder. Pick
|
||||
hive collector or the store's forwarder. Pick
|
||||
a free port.
|
||||
'';
|
||||
}
|
||||
|
|
|
|||
|
|
@ -50,34 +50,6 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
port = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 4319;
|
||||
description = ''
|
||||
First port of this collector's receiver range. Every hive in
|
||||
{option}`services.hyperhive.swarm.hives` gets its **own**
|
||||
authenticated receiver — that is what makes the `hive` label
|
||||
unforgeable — so the range is one port per hive, starting here, in
|
||||
sorted-name order.
|
||||
|
||||
⚠️ Internal. No client is ever told a port: a hive reaches its own
|
||||
receiver as `https://''${domain}/<hive>`, and the gateway routes on
|
||||
that path. So adding a hive, which renumbers the ones after it, is
|
||||
harmless — nginx is rendered from this same evaluation and moves
|
||||
with it.
|
||||
|
||||
⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the
|
||||
hive tier already uses it (`services.hyperhive.swarm.hiveOtel.collectorPort`)
|
||||
and every swarm container shares the host's network namespace. Two
|
||||
listeners claiming one port on one host is not a build failure —
|
||||
it is a runtime coin toss over which one gets it, with nothing in
|
||||
any log saying so. On the host running this collector, an
|
||||
assertion rejects a derived range that overlaps any other port
|
||||
this module or the hive tier declares, which is as far as a module
|
||||
can see.
|
||||
'';
|
||||
};
|
||||
|
||||
telemetryPort = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 8889;
|
||||
|
|
@ -87,73 +59,8 @@ in
|
|||
that telemetry is being lost, so it is worth keeping rather than
|
||||
switching off.
|
||||
|
||||
⚠️ **Deliberately not 8889's neighbour 8888**, which is the
|
||||
collector's built-in default and therefore what the hive tier
|
||||
already binds. Two collectors share a network namespace whenever
|
||||
they are co-located, and unlike the OTLP port this one appears
|
||||
nowhere in either config — it is a default inside the binary, so
|
||||
nothing that compares configured ports can see the clash. The
|
||||
second collector to start simply dies with
|
||||
`bind: address already in use`.
|
||||
'';
|
||||
};
|
||||
|
||||
producerPort = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 4390;
|
||||
description = ''
|
||||
Port the swarm tier's own OTLP/HTTP receiver for a
|
||||
**swarm-level** producer (`swarm-controller`'s vcs/jobq counters
|
||||
today) listens on, at `127.0.0.1`. Reached through the gateway at
|
||||
`https://''${domain}/''${producerName}/`, same shape as a hive's
|
||||
own receiver — a swarm-level producer may not share a host with
|
||||
this collector (the swarm services need not run on the same host
|
||||
as the swarm controller), so loopback-only reachability is not a
|
||||
supported shape here, as it is not for a hive.
|
||||
|
||||
Authenticated by `oidc/''${producerName}`, checking for the
|
||||
audience `services.hyperhive.swarm.controller.queueClientId`
|
||||
requests — not the per-hive `hiveClientPrefix` namespace, because
|
||||
a swarm-level producer is explicitly not a hive (see that
|
||||
option's own doc comment). The `swarm` label this receiver's
|
||||
samples get is still the one the receiver's own *existence*
|
||||
establishes, same as before — there is no hive identity being
|
||||
forged or attributed here, just a caller proving it is the one
|
||||
principal allowed to push into this pipeline.
|
||||
|
||||
⚠️ Deliberately NOT derived from `port + (number of hives)`: that
|
||||
range grows every time a hive is added, and a fixed offset from
|
||||
it would silently start colliding once the hive count caught up.
|
||||
Kept as its own reserved value instead, and the port-collision
|
||||
assertion still rejects a real collision (including one hive
|
||||
growth eventually causes) rather than starting a collector that
|
||||
quietly drops one receiver's samples.
|
||||
'';
|
||||
};
|
||||
|
||||
storeProducerPort = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 4391;
|
||||
description = ''
|
||||
Port the receiver for the **secret store's own forwarder** listens
|
||||
on, at `127.0.0.1`. Reached through the gateway at
|
||||
`https://''${domain}/''${storeProducerName}/`, the same
|
||||
path-per-producer shape a hive and the swarm-tier producer above
|
||||
both use.
|
||||
|
||||
Its own receiver, and not a second sender into
|
||||
{option}`services.hyperhive.swarm.otel.producerPort`, because an
|
||||
`oidc` authenticator checks exactly ONE audience: that receiver
|
||||
admits the audience `swarm-controller` registers for itself, and
|
||||
the store's forwarder is a different principal with a client of its
|
||||
own. Sharing the route would mean sharing that client — which is
|
||||
the thing the store exists to make unnecessary.
|
||||
|
||||
⚠️ A reserved value rather than an offset from
|
||||
{option}`services.hyperhive.swarm.otel.port`, for the reason stated
|
||||
on `producerPort`: that range grows with the hive count and would
|
||||
eventually walk into any fixed offset from it. The port-collision
|
||||
assertion covers this port too.
|
||||
Bound on the container's own loopback, so no other collector
|
||||
shares it; only this collector scrapes it.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -170,14 +77,6 @@ in
|
|||
the forwarder requests.
|
||||
'';
|
||||
};
|
||||
port = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
description = ''
|
||||
Loopback port of the producer's receiver on the collector's
|
||||
host. In the port-collision assertion with every other
|
||||
receiver there.
|
||||
'';
|
||||
};
|
||||
};
|
||||
}
|
||||
);
|
||||
|
|
@ -256,13 +155,15 @@ in
|
|||
scrapeTargets = lib.mkOption {
|
||||
type = lib.types.attrsOf lib.types.str;
|
||||
default = { };
|
||||
example = lib.literalExpression ''{ forgejo = "127.0.0.1:3000"; }'';
|
||||
example = lib.literalExpression ''{ collector = "127.0.0.1:8889"; }'';
|
||||
description = ''
|
||||
Prometheus exposition endpoints this collector scrapes, as
|
||||
`<job name> = "<host>:<port>"`.
|
||||
`<job name> = "<host>:<port>"`. The collector runs in a network namespace of
|
||||
its own, so `127.0.0.1` is the collector container's loopback: a target
|
||||
here is a listener inside that container, such as its own telemetry.
|
||||
|
||||
A path and query may follow the port —
|
||||
`"127.0.0.1:8202/v1/sys/metrics?format=prometheus"` — for an
|
||||
`"127.0.0.1:9100/stats?format=prometheus"` — for an
|
||||
exporter that does not serve `/metrics`. Both are optional and
|
||||
omitted when absent, so a bare `host:port` is scraped exactly as
|
||||
before.
|
||||
|
|
@ -270,11 +171,10 @@ in
|
|||
**A service declares its own entry, from its own module, under its
|
||||
own `enable`.** Do not assemble the list here: an entry then exists
|
||||
only where the service that named it runs, so a target is never
|
||||
declared on a host that does not serve it. An entry that also needs
|
||||
the collector on the same host (the loopback address is only honest
|
||||
then) is declared from a glue module instead, which reads both
|
||||
switches. Every swarm service is a `publishedScrapeTargets` entry
|
||||
instead, which a collector on any host scrapes.
|
||||
declared on a host that does not serve it. A swarm service on the
|
||||
host is not reachable here, since its loopback is not the collector's:
|
||||
it is a `publishedScrapeTargets` entry instead, which a collector on
|
||||
any host scrapes.
|
||||
|
||||
⚠️ **That constrains the TARGET, not the SCRAPER, and the difference
|
||||
is a silent gap.** Nothing here puts the collector on the same host.
|
||||
|
|
@ -383,73 +283,59 @@ in
|
|||
# The forwarder in ./swarm-bao.nix.
|
||||
${cfg.storeProducerName} = {
|
||||
clientId = baoCfg.otel.clientId;
|
||||
port = cfg.storeProducerPort;
|
||||
};
|
||||
# The forwarder in ./swarm-nats.nix, which addresses this route by the
|
||||
# queue container's name.
|
||||
swarm-nats = {
|
||||
clientId = hyperhiveCfg.swarm.nats.clientId;
|
||||
port = lib.mkDefault 4392;
|
||||
};
|
||||
# The forwarders in ./swarm-builder.nix, ./swarm-claude-login.nix,
|
||||
# ./swarm-notify-router.nix and ./swarm-review-classifier.nix, which
|
||||
# address their routes by container name.
|
||||
swarm-builder = {
|
||||
clientId = hyperhiveCfg.swarm.builder.clientId;
|
||||
port = lib.mkDefault 4393;
|
||||
};
|
||||
swarm-claude-login = {
|
||||
clientId = hyperhiveCfg.swarm.claude-login.clientId;
|
||||
port = lib.mkDefault 4394;
|
||||
};
|
||||
swarm-review-classifier = {
|
||||
clientId = hyperhiveCfg.swarm.review-classifier.clientId;
|
||||
port = lib.mkDefault 4395;
|
||||
};
|
||||
swarm-notify-router = {
|
||||
clientId = hyperhiveCfg.swarm.notify-router.clientId;
|
||||
port = lib.mkDefault 4404;
|
||||
};
|
||||
# The forwarders in ./swarm-ci-runner.nix, ./swarm-grafana.nix and
|
||||
# ./swarm-victorialogs.nix, each authenticated as a client of its own.
|
||||
swarm-ci-runner = {
|
||||
clientId = hyperhiveCfg.swarm.ci-runner.otel.clientId;
|
||||
port = lib.mkDefault 4396;
|
||||
};
|
||||
swarm-grafana = {
|
||||
clientId = hyperhiveCfg.swarm.grafana.otel.clientId;
|
||||
port = lib.mkDefault 4397;
|
||||
};
|
||||
swarm-victorialogs = {
|
||||
clientId = hyperhiveCfg.swarm.victorialogs.otel.clientId;
|
||||
port = lib.mkDefault 4398;
|
||||
};
|
||||
# The forwarders in ./hive-forge/default.nix and ./hive-matrix.nix, each
|
||||
# authenticated as a client of its own.
|
||||
hive-forge = {
|
||||
clientId = hyperhiveCfg.swarm.forge.otel.clientId;
|
||||
port = lib.mkDefault 4399;
|
||||
};
|
||||
hive-matrix = {
|
||||
clientId = hyperhiveCfg.swarm.matrix.otel.clientId;
|
||||
port = lib.mkDefault 4400;
|
||||
};
|
||||
# The forwarder in ./swarm-victoriametrics.nix, which reaches this route
|
||||
# through its host's nginx.
|
||||
swarm-victoriametrics = {
|
||||
clientId = hyperhiveCfg.swarm.victoriametrics.otel.clientId;
|
||||
port = lib.mkDefault 4401;
|
||||
};
|
||||
# The CI run poller in ./swarm-ci-metrics.nix, a host unit that pushes
|
||||
# its run samples here directly.
|
||||
swarm-ci-metrics = {
|
||||
clientId = hyperhiveCfg.swarm.ci-metrics.clientId;
|
||||
port = lib.mkDefault 4403;
|
||||
};
|
||||
# The forwarder in ./swarm-authelia.nix.
|
||||
swarm-authelia = {
|
||||
clientId = hyperhiveCfg.swarm.authelia.otel.clientId;
|
||||
port = lib.mkDefault 4402;
|
||||
};
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -168,16 +168,10 @@ let
|
|||
# `attrNames` is sorted, so this is a function of the hive SET and not of
|
||||
# the order anyone wrote it in.
|
||||
#
|
||||
# These ports are internal and appear in no URL: a hive addresses its own
|
||||
# receiver as a PATH on this collector's single gateway name, and nginx —
|
||||
# rendered from this same evaluation — is the only thing that ever names
|
||||
# the port. That is what makes deriving them safe here and unsafe in the
|
||||
# obvious other place: were a hive told a port, inserting a hive would
|
||||
# renumber the ones after it and silently move a port a running hive was
|
||||
# already sending to.
|
||||
hivePorts = lib.listToAttrs (
|
||||
lib.imap0 (i: h: lib.nameValuePair h (cfg.port + i)) (lib.attrNames hyperhiveCfg.swarm.hives)
|
||||
);
|
||||
# A hive addresses its own receiver as a PATH on this collector's single
|
||||
# gateway name; nginx, rendered from this same evaluation, is the only thing
|
||||
# that ever names the socket.
|
||||
hiveSockets = lib.genAttrs (lib.attrNames hyperhiveCfg.swarm.hives) socketOf;
|
||||
|
||||
# The swarm's authelia is reached by its gateway name, whose leaf is
|
||||
# issued by the swarm services sub-CA — so this container needs the same
|
||||
|
|
@ -301,11 +295,32 @@ let
|
|||
# Now always satisfied, as a consequence of the log store always existing.
|
||||
collectLogs = logExporterNames != [ ];
|
||||
|
||||
# Shared host netns, like every sibling swarm service: the hive tier
|
||||
# reaches this collector, and this collector reaches the metrics
|
||||
# store, without either crossing a network boundary that would need
|
||||
# its own trust material.
|
||||
privateNetwork = false;
|
||||
# A netns of its own on the hive bridge: the container dials authelia, the
|
||||
# stores and the upstream by name through the bridge, and the gateway reaches
|
||||
# every OTLP receiver over a socket, so no receiver binds a TCP port on any
|
||||
# interface.
|
||||
privateNetwork = true;
|
||||
|
||||
# Fixed, not an option: the directory's mode is what keeps the sockets
|
||||
# private, so it holds the receivers' sockets and nothing else.
|
||||
socketDir = "/run/swarm-otel";
|
||||
# One socket per receiver, named after its owner (hive, swarm tier or
|
||||
# service): `otlp/<owner>` is already unique, so the socket is too.
|
||||
socketOf = owner: "${socketDir}/${owner}.sock";
|
||||
# Every receiver's socket, by owner. The collector's receivers, the gateway's
|
||||
# routes and the readiness check all read this attrset, so they cannot name
|
||||
# different sockets.
|
||||
receiverSockets = hiveSockets // lib.genAttrs ([ swarmTierName ] ++ serviceProducerNames) socketOf;
|
||||
upstreamOf = sock: "http://unix:${sock}:/";
|
||||
# Set by nginx on every response from a route, whatever its status, so the
|
||||
# readiness check can tell a route the running nginx has from the catch-all.
|
||||
routeHeader = "X-Swarm-Otel-Route";
|
||||
allSockets = lib.attrValues receiverSockets;
|
||||
|
||||
# The gid the gateway's nginx runs as. Only the number crosses into the
|
||||
# container, which knows it as `gateway-nginx`.
|
||||
nginxGid = config.ids.gids.nginx;
|
||||
socketGroup = "gateway-nginx";
|
||||
storeRetry = import ./lib/store-retry.nix { };
|
||||
storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
|
||||
in
|
||||
|
|
@ -455,23 +470,23 @@ in
|
|||
# vhost would answer for a service it does not have.
|
||||
services.hyperhive.gateway.localNames = [ cfg.domain ];
|
||||
|
||||
# This host serves the vhost, and the container behind it resolves
|
||||
# through the hive's dnsmasq.
|
||||
# This host serves the vhost, and the hive's dnsmasq answers its name.
|
||||
services.hyperhive.gateway.enable = lib.mkDefault true;
|
||||
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
|
||||
|
||||
# The hive bridge carries the container's DHCP lease and its resolver.
|
||||
services.hyperhive.network.enable = lib.mkDefault true;
|
||||
|
||||
# The metrics counterpart to this collector's own journal, which its
|
||||
# receiver ships like any other unit's: the journal says the process is
|
||||
# alive, these counters say whether it is dropping what it receives.
|
||||
# Loopback works here because `privateNetwork = false` — this container
|
||||
# shares the host's netns, so `127.0.0.1` is where `telemetryPort` is
|
||||
# bound.
|
||||
# Loopback is the container's own: this collector scrapes `telemetryPort`
|
||||
# inside its netns, and nothing else reads it.
|
||||
services.hyperhive.swarm.otel.scrapeTargets.collector = "127.0.0.1:${toString cfg.telemetryPort}";
|
||||
|
||||
# OTLP/HTTP, not a browsable UI, but the same reverse-proxy shape as
|
||||
# every sibling swarm service: TLS terminates here, then plain http to
|
||||
# the co-located container over loopback (shared netns, like the store
|
||||
# this collector writes to).
|
||||
# every sibling swarm service: TLS terminates here, then plain http over
|
||||
# the receiver's unix socket.
|
||||
services.hyperhive.gateway.vhosts."${cfg.domain}" = {
|
||||
locations =
|
||||
# One name for the whole collector, and the hive is a path under
|
||||
|
|
@ -484,26 +499,26 @@ in
|
|||
# serves `/v1/metrics` and knows nothing about hives. Without it
|
||||
# the receiver sees `/<hive>/v1/metrics` and answers 404 to a
|
||||
# request that authenticated perfectly.
|
||||
#
|
||||
# ⚠️ A location with an `add_header` of its own inherits none of the
|
||||
# server-scope ones, so the security headers are repeated here.
|
||||
lib.mapAttrs' (
|
||||
h: p: lib.nameValuePair "/${h}/" { proxyPass = "http://127.0.0.1:${toString p}/"; }
|
||||
) hivePorts
|
||||
owner: sock:
|
||||
lib.nameValuePair "/${owner}/" {
|
||||
proxyPass = upstreamOf sock;
|
||||
extraConfig = ''
|
||||
${gatewayCfg.lib.securityHeaders}
|
||||
add_header ${routeHeader} ${owner} always;
|
||||
'';
|
||||
}
|
||||
) receiverSockets
|
||||
// {
|
||||
# The swarm-tier producer's own route — same trailing-slash
|
||||
# shape as the per-hive locations above, and load-bearing for
|
||||
# the same reason: it strips `/${producerName}` before the
|
||||
# request reaches the receiver, which knows nothing about the
|
||||
# path it was found at.
|
||||
"/${swarmTierName}/".proxyPass = "http://127.0.0.1:${toString cfg.producerPort}/";
|
||||
# There is no swarm-wide inbox, and a closed door is the honest
|
||||
# description of that. Every other route into this collector
|
||||
# belongs to exactly one hive or one swarm-tier producer.
|
||||
# belongs to exactly one hive, swarm-tier producer or service
|
||||
# forwarder, each on its own route for its own audience.
|
||||
"/".return = "404";
|
||||
}
|
||||
# Each service forwarder, on its own route for its own audience —
|
||||
# same trailing slashes, same reason.
|
||||
// lib.mapAttrs' (
|
||||
name: p: lib.nameValuePair "/${name}/" { proxyPass = "http://127.0.0.1:${toString p.port}/"; }
|
||||
) serviceProducers;
|
||||
};
|
||||
};
|
||||
|
||||
# The identities this tier authenticates against. Stated rather than
|
||||
|
|
@ -663,7 +678,18 @@ in
|
|||
lib.mkDefault collectorSecretInContainer
|
||||
);
|
||||
|
||||
systemd.tmpfiles.rules = lib.optionals haveClientIdentity (
|
||||
systemd.tmpfiles.rules = [
|
||||
# The socket directory, created host-side before the container starts:
|
||||
# nixos-container refuses to start when a bind source is missing.
|
||||
#
|
||||
# Root-owned, group nginx, setgid `2770`: the collector creates its
|
||||
# sockets here through the group (its uid is allocated inside the
|
||||
# container, so the host cannot name it), and a socket takes the
|
||||
# directory's group, so the gateway can connect. Everything else is
|
||||
# kept out.
|
||||
"d ${socketDir} 2770 root ${toString nginxGid} - -"
|
||||
]
|
||||
++ lib.optionals haveClientIdentity (
|
||||
[
|
||||
# The bind source must exist when the container starts.
|
||||
"d ${collectorSecretDir} 0700 root root -"
|
||||
|
|
@ -682,6 +708,62 @@ in
|
|||
# over a file that does not exist yet.
|
||||
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;
|
||||
|
||||
# A collector that is up but not listening on a socket the gateway names
|
||||
# answers every producer with a 502 and fails no unit. This one fails when
|
||||
# any receiver socket refuses a connection for 30s, or when the running
|
||||
# nginx does not route to one, so the switch reports it. It runs as the
|
||||
# gateway's user: a socket nginx cannot open fails too. On the socket any
|
||||
# HTTP status counts as accepted. Through nginx, 000, 502 and 504 fail, and so
|
||||
# does a response without its route's own header; any other status passes, a
|
||||
# collector's 4xx included.
|
||||
systemd.services.swarm-otel-sockets = {
|
||||
description = "wait for every swarm collector receiver socket to accept a connection";
|
||||
after = [
|
||||
"container@${cfg.machine}.service"
|
||||
"nginx.service"
|
||||
];
|
||||
wants = [ "container@${cfg.machine}.service" ];
|
||||
# Restarted with the container, so every start of the collector is
|
||||
# checked and not only the first one after a boot.
|
||||
partOf = [ "container@${cfg.machine}.service" ];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
path = [
|
||||
pkgs.curl
|
||||
pkgs.coreutils
|
||||
];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
User = config.services.nginx.user;
|
||||
TimeoutStartSec = 90;
|
||||
};
|
||||
script = ''
|
||||
for sock in ${lib.escapeShellArgs allSockets}; do
|
||||
for _ in $(seq 30); do
|
||||
curl -s -o /dev/null --max-time 5 --unix-socket "$sock" http://localhost/ && continue 2
|
||||
sleep 1
|
||||
done
|
||||
echo "no connection to $sock after 30s" >&2
|
||||
exit 1
|
||||
done
|
||||
|
||||
# The running nginx, not its config. `--resolve` pins the vhost to
|
||||
# this host's gateway. The route header is what separates a route nginx
|
||||
# has from its catch-all 404.
|
||||
for route in ${lib.escapeShellArgs (lib.attrNames receiverSockets)}; do
|
||||
for _ in $(seq 30); do
|
||||
out=$(curl -sk --max-time 5 --resolve ${cfg.domain}:443:127.0.0.1 -o /dev/null \
|
||||
-w '%{http_code} %header{${lib.toLower routeHeader}}' "https://${cfg.domain}/$route/") || out=000
|
||||
read -r code seen <<< "$out"
|
||||
case $code in 000 | 502 | 504) ;; *) [ "$seen" = "$route" ] && continue 2 ;; esac
|
||||
sleep 1
|
||||
done
|
||||
echo "https://${cfg.domain}/$route/ answered $code, route header '$seen', after 30s" >&2
|
||||
exit 1
|
||||
done
|
||||
'';
|
||||
};
|
||||
|
||||
assertions = [
|
||||
{
|
||||
# Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion —
|
||||
|
|
@ -872,44 +954,11 @@ in
|
|||
anything, and the two are scraped by different rules with
|
||||
different trust.
|
||||
|
||||
Rename one. A target is either reachable on loopback because it
|
||||
shares this host, or published and reached with a credential; it
|
||||
Rename one. A target is either reachable on the collector's own
|
||||
loopback, or published and reached with a credential; it
|
||||
should not be described as both.
|
||||
'';
|
||||
}
|
||||
{
|
||||
# A port collision between two listeners on one host is a runtime
|
||||
# coin toss with nothing in any log — the failure this whole
|
||||
# comment budget exists to prevent. Checked against every port
|
||||
# reachable from here; a port some other module picks is not.
|
||||
#
|
||||
# ⚠️ `cfg.port` is deliberately absent from `others`: it is the
|
||||
# FIRST element of the derived range, so listing it would make this
|
||||
# assertion fire on every config.
|
||||
assertion =
|
||||
let
|
||||
derived = lib.attrValues hivePorts;
|
||||
others = [
|
||||
cfg.telemetryPort
|
||||
cfg.producerPort
|
||||
config.services.hyperhive.swarm.hiveOtel.collectorPort
|
||||
]
|
||||
++ map (p: p.port) (lib.attrValues serviceProducers);
|
||||
all = derived ++ others;
|
||||
in
|
||||
lib.length (lib.unique all) == lib.length all;
|
||||
message = ''
|
||||
services.hyperhive.swarm.otel: the receiver range starting at
|
||||
port (${toString cfg.port}, one port per hive in
|
||||
services.hyperhive.swarm.hives) overlaps another port on this
|
||||
host.
|
||||
|
||||
Every swarm container shares the host's network namespace, so
|
||||
two listeners claiming one port is not a build failure — it is
|
||||
whichever process started first, silently. Move
|
||||
services.hyperhive.swarm.otel.port to a free range.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
containers.${cfg.machine} = {
|
||||
|
|
@ -918,19 +967,28 @@ in
|
|||
# No `--link-journal=host`: this container's journal stays in the guest,
|
||||
# where `journald/self` reads it.
|
||||
inherit privateNetwork;
|
||||
hostBridge = config.services.hyperhive.network.bridgeName;
|
||||
|
||||
# The upstream credential is operator-provided and lives on the host.
|
||||
# Read-only, and only when one is configured — binding a path that
|
||||
# does not exist makes nixos-container refuse to start the container,
|
||||
# which is a stall several layers from its cause.
|
||||
bindMounts =
|
||||
# A bind mount rather than a path both sides happen to know: a
|
||||
# container's `/run` is an nspawn tmpfs the host cannot see into.
|
||||
{
|
||||
${socketDir} = {
|
||||
hostPath = socketDir;
|
||||
isReadOnly = false;
|
||||
};
|
||||
}
|
||||
# The collector's OIDC secret, where the delivery unit writes it.
|
||||
lib.optionalAttrs haveClientIdentity {
|
||||
// lib.optionalAttrs haveClientIdentity {
|
||||
${collectorSecretDir} = {
|
||||
hostPath = collectorSecretDir;
|
||||
isReadOnly = true;
|
||||
};
|
||||
}
|
||||
# The upstream credential is operator-provided and lives on the host.
|
||||
# Read-only, and only when one is configured — binding a path that
|
||||
# does not exist makes nixos-container refuse to start the container,
|
||||
# which is a stall several layers from its cause.
|
||||
// lib.optionalAttrs (upstream.headersCredential != null) {
|
||||
${upstream.headersCredential} = {
|
||||
hostPath = upstream.headersCredential;
|
||||
|
|
@ -996,6 +1054,15 @@ in
|
|||
system.stateVersion = config.system.stateVersion;
|
||||
services.hyperhive.swarmContainer = { inherit privateNetwork; };
|
||||
|
||||
# The bridge address comes from the gateway's DHCP. `swarm-bridge-dns`
|
||||
# owns resolv.conf, so dhcpcd must not write it over that unit.
|
||||
networking.interfaces.eth0.useDHCP = true;
|
||||
networking.dhcpcd.extraConfig = "nohook resolv.conf";
|
||||
|
||||
# The sockets' group is the directory's, so that group is nginx's gid.
|
||||
# A static group, so the unit below can name it.
|
||||
users.groups.${socketGroup}.gid = nginxGid;
|
||||
|
||||
services.opentelemetry-collector = {
|
||||
enable = true;
|
||||
package = pkgs.opentelemetry-collector-contrib;
|
||||
|
|
@ -1018,22 +1085,24 @@ in
|
|||
# reveals which one matched.
|
||||
receivers =
|
||||
lib.mapAttrs' (
|
||||
h: p:
|
||||
h: sock:
|
||||
lib.nameValuePair "otlp/${h}" {
|
||||
protocols.http = {
|
||||
endpoint = "127.0.0.1:${toString p}";
|
||||
endpoint = sock;
|
||||
transport = "unix";
|
||||
auth.authenticator = "oidc/${h}";
|
||||
};
|
||||
}
|
||||
) hivePorts
|
||||
) hiveSockets
|
||||
# The swarm tier's OWN receiver — authenticated exactly
|
||||
# like a hive's, just against a different identity (the
|
||||
# swarm-level producer is not a hive; see
|
||||
# `producerPort`'s description for what its audience
|
||||
# checks and why).
|
||||
# swarm-level producer is not a hive). Its authenticator checks
|
||||
# the audience `swarm.controller.queueClientId` requests, not
|
||||
# the per-hive `hiveClientPrefix` namespace.
|
||||
// {
|
||||
"otlp/${swarmTierName}".protocols.http = {
|
||||
endpoint = "127.0.0.1:${toString cfg.producerPort}";
|
||||
endpoint = receiverSockets.${swarmTierName};
|
||||
transport = "unix";
|
||||
auth.authenticator = "oidc/${swarmTierName}";
|
||||
};
|
||||
}
|
||||
|
|
@ -1041,10 +1110,11 @@ in
|
|||
# principal of its own — a receiver per audience, because an
|
||||
# `oidc` authenticator checks one.
|
||||
// lib.mapAttrs' (
|
||||
name: p:
|
||||
name: _:
|
||||
lib.nameValuePair "otlp/${name}" {
|
||||
protocols.http = {
|
||||
endpoint = "127.0.0.1:${toString p.port}";
|
||||
endpoint = receiverSockets.${name};
|
||||
transport = "unix";
|
||||
auth.authenticator = "oidc/${name}";
|
||||
};
|
||||
}
|
||||
|
|
@ -1256,7 +1326,7 @@ in
|
|||
# Same rule, and the journald receiver's `storage:` above
|
||||
# would name a component the collector never starts.
|
||||
[ "file_storage" ]
|
||||
++ map (h: "oidc/${h}") (lib.attrNames hivePorts)
|
||||
++ map (h: "oidc/${h}") (lib.attrNames hiveSockets)
|
||||
# The swarm-tier producer's own authenticator — unconditional,
|
||||
# same reasoning as its receiver above: `otlp/${swarmTierName}`
|
||||
# always exists, so the authenticator it names must too, or an
|
||||
|
|
@ -1285,7 +1355,7 @@ in
|
|||
processors = [ "resource/${h}" ];
|
||||
exporters = exporterNames;
|
||||
}
|
||||
) hivePorts
|
||||
) hiveSockets
|
||||
# Its OWN pipeline, and that separation is the ruling, not a
|
||||
# tidiness choice: every `resource/<hive>` above UPSERTS a
|
||||
# `hive` key, so a scraped swarm sample routed through any of
|
||||
|
|
@ -1343,7 +1413,7 @@ in
|
|||
processors = [ "resource/${h}" ];
|
||||
exporters = logExporterNames;
|
||||
}
|
||||
) hivePorts
|
||||
) hiveSockets
|
||||
)
|
||||
# Logs fan out exactly as metrics do: the swarm's store when it
|
||||
# runs, the operator's upstream when one is configured, both
|
||||
|
|
@ -1446,7 +1516,7 @@ in
|
|||
# worth not re-deriving: `issuer_ca_file`, `ca_file` and
|
||||
# `tls.ca_file` are INVALID KEYS for this extension.)
|
||||
}
|
||||
) hivePorts
|
||||
) hiveSockets
|
||||
# The swarm-tier producer's own authenticator. Same extension
|
||||
# type and same `issuer_url`/no-`issuer_ca_path` shape as every
|
||||
# `oidc/${h}` above — only the audience differs, and
|
||||
|
|
@ -1534,7 +1604,7 @@ in
|
|||
}
|
||||
];
|
||||
}
|
||||
) hivePorts
|
||||
) hiveSockets
|
||||
# The swarm tier's own stamp: `swarm` and deliberately NO
|
||||
# `hive`. A scraped swarm service belongs to the swarm, not to
|
||||
# any one hive, so there is no honest value to put there — and
|
||||
|
|
@ -1623,6 +1693,22 @@ in
|
|||
serviceConfig.RestartSec = 5;
|
||||
}
|
||||
|
||||
{
|
||||
serviceConfig = {
|
||||
# The unit's user is allocated per start (`DynamicUser`), so
|
||||
# it reaches the socket directory through this group, and
|
||||
# `0007` makes each socket `0770`: group nginx can connect.
|
||||
SupplementaryGroups = [ socketGroup ];
|
||||
UMask = "0007";
|
||||
# `DynamicUser` forces `ProtectSystem=strict` over upstream's
|
||||
# `full`, so `/run` is read-only in the unit and the socket
|
||||
# directory's writable bind mount does not reach it.
|
||||
ReadWritePaths = [ socketDir ];
|
||||
# A socket left by a killed process would fail the next `bind`.
|
||||
ExecStartPre = [ "${pkgs.coreutils}/bin/rm -f ${lib.escapeShellArgs allSockets}" ];
|
||||
};
|
||||
}
|
||||
|
||||
# The credential file is already `NAME=value`, systemd's
|
||||
# EnvironmentFile format — so the secret reaches the process as
|
||||
# an environment variable without being read by nix, written to
|
||||
|
|
|
|||
|
|
@ -276,14 +276,13 @@ in
|
|||
assertion =
|
||||
!(lib.elem forwarderTelemetryPort [
|
||||
hyperhiveCfg.otel.telemetryPort
|
||||
hyperhiveCfg.swarm.otel.telemetryPort
|
||||
hyperhiveCfg.swarm.bao.otel.telemetryPort
|
||||
hyperhiveCfg.swarm.nats.otel.telemetryPort
|
||||
]);
|
||||
message = ''
|
||||
The log store's journal forwarder's telemetry port (${toString forwarderTelemetryPort})
|
||||
collides with another collector's in the host netns: the hive
|
||||
collector's, the swarm collector's, the store forwarder's or the
|
||||
collector's, the store forwarder's or the
|
||||
queue forwarder's.
|
||||
'';
|
||||
}
|
||||
|
|
|
|||
|
|
@ -153,7 +153,6 @@ let
|
|||
# Every other collector port on the host, and the queue's forwarder.
|
||||
taken = [
|
||||
8888
|
||||
8889
|
||||
8890
|
||||
h.services.hyperhive.swarm.nats.otel.telemetryPort
|
||||
]
|
||||
|
|
@ -266,7 +265,6 @@ let
|
|||
let
|
||||
taken = [
|
||||
8888
|
||||
8889
|
||||
8890
|
||||
h.services.hyperhive.swarm.nats.otel.telemetryPort
|
||||
]
|
||||
|
|
@ -795,7 +793,7 @@ let
|
|||
otel = h.services.hyperhive.swarm.otel;
|
||||
s = h.containers.swarm-otel.config.services.opentelemetry-collector.settings;
|
||||
clientOf = svc: h.services.hyperhive.swarm.${svc.name}.clientId;
|
||||
ports = map (svc: otel.serviceProducers.${svc.machine}.port) forwarderServices;
|
||||
socks = map (svc: s.receivers."otlp/${svc.machine}".protocols.http.endpoint) forwarderServices;
|
||||
in
|
||||
!h.services.hyperhive.deploy.builder.enable
|
||||
&& lib.all (
|
||||
|
|
@ -806,9 +804,9 @@ let
|
|||
&& lib.elem "otlp/${svc.machine}" s.service.pipelines."logs/${otel.producerName}".receivers
|
||||
&& h.services.nginx.virtualHosts.${otel.domain}.locations ? "/${svc.machine}/"
|
||||
) forwarderServices
|
||||
# Their receivers and the queue's on the collector's host, none sharing a port.
|
||||
# Their receivers and the queue's on the collector's host, none sharing a socket.
|
||||
&&
|
||||
lib.length (lib.unique (ports ++ [ otel.serviceProducers.swarm-nats.port ]))
|
||||
lib.length (lib.unique (socks ++ [ s.receivers."otlp/swarm-nats".protocols.http.endpoint ]))
|
||||
== lib.length forwarderServices + 1;
|
||||
}
|
||||
{
|
||||
|
|
@ -837,7 +835,9 @@ let
|
|||
h = shared.baoWithCollector;
|
||||
otel = h.services.hyperhive.swarm.otel;
|
||||
s = h.containers.swarm-otel.config.services.opentelemetry-collector.settings;
|
||||
ports = map (p: p.port) (lib.attrValues otel.serviceProducers);
|
||||
socks = map (n: s.receivers."otlp/${n}".protocols.http.endpoint) (
|
||||
lib.attrNames otel.serviceProducers
|
||||
);
|
||||
in
|
||||
lib.all (
|
||||
svc:
|
||||
|
|
@ -847,7 +847,7 @@ let
|
|||
&& lib.elem "otlp/${svc.machine}" s.service.pipelines."logs/${otel.producerName}".receivers
|
||||
&& h.services.nginx.virtualHosts.${otel.domain}.locations ? "/${svc.machine}/"
|
||||
) ownClientForwarders
|
||||
&& lib.length (lib.unique ports) == lib.length ports;
|
||||
&& lib.length (lib.unique socks) == lib.length socks;
|
||||
}
|
||||
{
|
||||
# Grafana's forwarder is a second client beside its browser login, which
|
||||
|
|
@ -1000,7 +1000,6 @@ let
|
|||
reader = lib.head s.service.telemetry.metrics.readers;
|
||||
taken = [
|
||||
8888
|
||||
8889
|
||||
8890
|
||||
h.services.hyperhive.swarm.nats.otel.telemetryPort
|
||||
]
|
||||
|
|
|
|||
|
|
@ -170,9 +170,10 @@ let
|
|||
lib.filter (l: swarmServiceLinkDomains ? ${l.label}) cfg.services.hyperhive.swarm.controller.links;
|
||||
|
||||
# Every service container at once: the services-only host plus the CI
|
||||
# runner. The CI runner, the review classifier, the builder, the claude login
|
||||
# minter, the metrics store, the homeserver, authelia and Grafana are the eight
|
||||
# containers on a netns of their own.
|
||||
# runner. The CI runner, the review classifier, the notification router, the
|
||||
# builder, the claude login minter, the metrics store, the homeserver,
|
||||
# authelia, Grafana and the collector are the ten containers on a netns of
|
||||
# their own.
|
||||
serviceContainersWithCi = hive {
|
||||
deploy.hive-controller.enable = false;
|
||||
deploy.allSwarmServices = true;
|
||||
|
|
@ -616,12 +617,14 @@ let
|
|||
}
|
||||
{
|
||||
# Both values come from ../container-modules/swarm-container.nix.
|
||||
# Four of the 14 service containers share the host netns (hive-forge, bao,
|
||||
# the queue, the log store) and ten have one of their own.
|
||||
name = "a service container on the host netns runs no firewall or resolvconf of its own";
|
||||
ok =
|
||||
let
|
||||
shared = lib.filter (c: !c.privateNetwork) (lib.attrValues serviceContainers);
|
||||
in
|
||||
lib.length shared == 5
|
||||
lib.length shared == 4
|
||||
&& lib.all (
|
||||
c: !c.config.networking.firewall.enable && !c.config.networking.resolvconf.enable
|
||||
) shared;
|
||||
|
|
@ -792,6 +795,270 @@ let
|
|||
&& lib.all (lib.hasPrefix "http://unix:${sock}:") forwarded
|
||||
&& c.bindMounts.${dir}.hostPath == dir;
|
||||
}
|
||||
{
|
||||
# On the bridge by DHCP, with every receiver on a socket: the hive's, the
|
||||
# swarm tier's and each service forwarder's. Every gateway location that
|
||||
# forwards goes through one of them, and each receiver's socket is the
|
||||
# one its location names. The three kinds are named so an empty filter
|
||||
# can't pass.
|
||||
name = "the swarm collector runs on a netns of its own and the gateway reaches every receiver over a socket";
|
||||
ok =
|
||||
let
|
||||
host = serviceContainersWithCi;
|
||||
net = host.services.hyperhive.network;
|
||||
c = host.containers.swarm-otel;
|
||||
otel = host.services.hyperhive.swarm.otel;
|
||||
dir = "/run/swarm-otel";
|
||||
sock = n: "${dir}/${n}.sock";
|
||||
locs = host.services.nginx.virtualHosts.${otel.domain}.locations;
|
||||
forwarded = lib.filter (p: p != null) (map (l: l.proxyPass) (lib.attrValues locs));
|
||||
inner = c.config;
|
||||
s = inner.services.opentelemetry-collector.settings;
|
||||
unit = inner.systemd.services.opentelemetry-collector.serviceConfig;
|
||||
nginxGid = host.ids.gids.nginx;
|
||||
receivers = lib.filterAttrs (n: _: lib.hasPrefix "otlp/" n) s.receivers;
|
||||
onSocket =
|
||||
owner:
|
||||
let
|
||||
r = s.receivers."otlp/${owner}".protocols.http;
|
||||
in
|
||||
r.transport == "unix"
|
||||
&& r.endpoint == sock owner
|
||||
&& locs."/${owner}/".proxyPass == "http://unix:${sock owner}:/";
|
||||
in
|
||||
c.privateNetwork
|
||||
&& c.hostBridge == net.bridgeName
|
||||
&& c.hostAddress == null
|
||||
&& c.localAddress == null
|
||||
&& inner.networking.interfaces.eth0.useDHCP
|
||||
&& inner.networking.firewall.enable
|
||||
&& onSocket "h1"
|
||||
&& onSocket otel.producerName
|
||||
&& onSocket "swarm-grafana"
|
||||
&& lib.all (n: (receivers.${n}.protocols.http.transport or null) == "unix") (
|
||||
lib.attrNames receivers
|
||||
)
|
||||
&&
|
||||
lib.length (lib.attrNames receivers)
|
||||
== lib.length (lib.attrNames host.services.hyperhive.swarm.hives)
|
||||
+ 1
|
||||
+ lib.length (lib.attrNames otel.serviceProducers)
|
||||
&& forwarded != [ ]
|
||||
&& lib.all (lib.hasPrefix "http://unix:${dir}/") forwarded
|
||||
&& c.bindMounts.${dir}.hostPath == dir
|
||||
&& builtins.elem "d ${dir} 2770 root ${toString nginxGid} - -" host.systemd.tmpfiles.rules
|
||||
&& inner.users.groups.gateway-nginx.gid == nginxGid
|
||||
&& lib.elem "gateway-nginx" unit.SupplementaryGroups
|
||||
&& unit.UMask == "0007";
|
||||
}
|
||||
{
|
||||
# The routes the gateway forwards to are exactly the sockets the
|
||||
# collector listens on, and none is a TCP address: a route left on
|
||||
# loopback after the collector moved into its own netns returns a 502 to
|
||||
# every producer while the collector runs. `verdict` is applied to
|
||||
# broken copies of the routes too, so a check that cannot fail is caught.
|
||||
name = "the gateway's collector routes are the collector's unix receivers and none is loopback";
|
||||
ok =
|
||||
let
|
||||
host = serviceContainersWithCi;
|
||||
otel = host.services.hyperhive.swarm.otel;
|
||||
s = host.containers.swarm-otel.config.services.opentelemetry-collector.settings;
|
||||
locs = host.services.nginx.virtualHosts.${otel.domain}.locations;
|
||||
sorted = xs: lib.sort lib.lessThan (lib.unique xs);
|
||||
unixReceivers = lib.filter (r: (r.protocols.http.transport or null) == "unix") (
|
||||
lib.attrValues s.receivers
|
||||
);
|
||||
receiverSocks = sorted (map (r: r.protocols.http.endpoint) unixReceivers);
|
||||
socketOfRoute =
|
||||
p:
|
||||
if lib.hasPrefix "http://unix:" p then
|
||||
lib.removeSuffix ":/" (lib.removePrefix "http://unix:" p)
|
||||
else
|
||||
p;
|
||||
verdict =
|
||||
routes:
|
||||
let
|
||||
passes = lib.filter (p: p != null) (map (l: l.proxyPass or null) (lib.attrValues routes));
|
||||
in
|
||||
sorted (map socketOfRoute passes) == receiverSocks
|
||||
&& !(lib.any (p: lib.hasInfix "127.0.0.1" p || lib.hasInfix "localhost" p) passes);
|
||||
in
|
||||
verdict locs
|
||||
&& lib.elem "/run/swarm-otel/h1.sock" receiverSocks
|
||||
&& lib.elem "/run/swarm-otel/${otel.producerName}.sock" receiverSocks
|
||||
&& lib.elem "/run/swarm-otel/swarm-grafana.sock" receiverSocks
|
||||
&&
|
||||
lib.length unixReceivers == lib.length (lib.attrNames host.services.hyperhive.swarm.hives)
|
||||
+ 1
|
||||
+ lib.length (lib.attrNames otel.serviceProducers)
|
||||
# Each broken copy must fail: a TCP route, a missing route, a route to
|
||||
# a socket no receiver has.
|
||||
&& !verdict (locs // { "/h1/".proxyPass = "http://127.0.0.1:4319/"; })
|
||||
&& !verdict (lib.removeAttrs locs [ "/h1/" ])
|
||||
&& !verdict (locs // { "/x/".proxyPass = "http://unix:/run/swarm-otel/x.sock:/"; });
|
||||
}
|
||||
{
|
||||
# After the container, restarted with it, and run as the gateway's user,
|
||||
# so a socket nginx cannot open fails the unit. Every receiver socket is
|
||||
# in the script.
|
||||
name = "the collector's receiver sockets are connect-checked after the container starts";
|
||||
ok =
|
||||
let
|
||||
host = serviceContainersWithCi;
|
||||
u = host.systemd.services.swarm-otel-sockets;
|
||||
s = host.containers.swarm-otel.config.services.opentelemetry-collector.settings;
|
||||
socks = map (r: r.protocols.http.endpoint) (
|
||||
lib.filter (r: (r.protocols.http.transport or null) == "unix") (lib.attrValues s.receivers)
|
||||
);
|
||||
in
|
||||
socks != [ ]
|
||||
&& lib.all (sock: lib.hasInfix sock u.script) socks
|
||||
&& !(lib.hasInfix "/run/swarm-otel/absent.sock" u.script)
|
||||
&& u.serviceConfig.Type == "oneshot"
|
||||
&& u.serviceConfig.User == host.services.nginx.user
|
||||
&& lib.elem "container@swarm-otel.service" u.after
|
||||
&& lib.elem "container@swarm-otel.service" u.partOf
|
||||
&& lib.elem "multi-user.target" u.wantedBy;
|
||||
}
|
||||
{
|
||||
# `DynamicUser` forces `ProtectSystem=strict`, which leaves `/run`
|
||||
# read-only unless the path is in `ReadWritePaths`. The controls: the
|
||||
# unit really is DynamicUser, and a copy without the path, or with some
|
||||
# other path, is read-only.
|
||||
name = "the collector unit can write the directory its receiver sockets are created in";
|
||||
ok =
|
||||
let
|
||||
host = serviceContainersWithCi;
|
||||
sc = host.containers.swarm-otel.config.systemd.services.opentelemetry-collector.serviceConfig;
|
||||
s = host.containers.swarm-otel.config.services.opentelemetry-collector.settings;
|
||||
socks = map (r: r.protocols.http.endpoint) (
|
||||
lib.filter (r: (r.protocols.http.transport or null) == "unix") (lib.attrValues s.receivers)
|
||||
);
|
||||
writes =
|
||||
c: sock:
|
||||
let
|
||||
strict = (c.DynamicUser or false) || (c.ProtectSystem or "no") == "strict";
|
||||
dir = dirOf sock;
|
||||
under = p: p == dir || lib.hasPrefix "${p}/" dir;
|
||||
in
|
||||
!strict || lib.any under (c.ReadWritePaths or [ ]);
|
||||
in
|
||||
socks != [ ]
|
||||
&& sc.DynamicUser
|
||||
&& lib.all (writes sc) socks
|
||||
&& !(lib.any (writes (sc // { ReadWritePaths = [ ]; })) socks)
|
||||
&& !(lib.any (writes (sc // { ReadWritePaths = [ "/var/lib/opentelemetry-collector" ]; })) socks);
|
||||
}
|
||||
{
|
||||
# The readiness unit tells a route nginx has from its catch-all 404 by
|
||||
# this header, so every route carries its own name in it, `always` so a
|
||||
# 4xx or 5xx from the collector carries it too. A location with an
|
||||
# `add_header` inherits none of the server-scope ones, so the security
|
||||
# headers have to be repeated. The catch-all carries no route header.
|
||||
name = "every collector route sets its own route header and keeps the security headers";
|
||||
ok =
|
||||
let
|
||||
host = serviceContainersWithCi;
|
||||
otel = host.services.hyperhive.swarm.otel;
|
||||
locs = host.services.nginx.virtualHosts.${otel.domain}.locations;
|
||||
sec = host.services.hyperhive.gateway.lib.securityHeaders;
|
||||
proxied = lib.filter (l: (locs.${l}.proxyPass or null) != null) (lib.attrNames locs);
|
||||
nameOf = l: lib.removeSuffix "/" (lib.removePrefix "/" l);
|
||||
verdict =
|
||||
ls:
|
||||
lib.all (
|
||||
l:
|
||||
let
|
||||
extra = ls.${l}.extraConfig or "";
|
||||
in
|
||||
lib.hasInfix "add_header X-Swarm-Otel-Route ${nameOf l} always;" extra
|
||||
&& lib.hasInfix "add_header X-Frame-Options" extra
|
||||
) proxied;
|
||||
in
|
||||
lib.length proxied > 2
|
||||
&& lib.hasInfix "add_header X-Frame-Options" sec
|
||||
&& !(lib.hasInfix "X-Swarm-Otel-Route" (locs."/".extraConfig or ""))
|
||||
&& verdict locs
|
||||
&& !verdict (
|
||||
locs
|
||||
// {
|
||||
"/h1/" = locs."/h1/" // {
|
||||
extraConfig = sec;
|
||||
};
|
||||
}
|
||||
)
|
||||
&& !verdict (
|
||||
locs
|
||||
// {
|
||||
"/h1/" = locs."/h1/" // {
|
||||
extraConfig = "add_header X-Swarm-Otel-Route h1 always;";
|
||||
};
|
||||
}
|
||||
)
|
||||
&& !verdict (
|
||||
locs
|
||||
// {
|
||||
"/h1/" = locs."/h1/" // {
|
||||
extraConfig = sec + "add_header X-Swarm-Otel-Route swarm always;";
|
||||
};
|
||||
}
|
||||
)
|
||||
&& !verdict (
|
||||
locs
|
||||
// {
|
||||
"/h1/" = locs."/h1/" // {
|
||||
extraConfig = sec + "add_header X-Swarm-Otel-Route h1;";
|
||||
};
|
||||
}
|
||||
);
|
||||
}
|
||||
{
|
||||
# The probe loop names each gateway route, and the routes are read from
|
||||
# the vhost, not from the receivers the probe is built from. A script
|
||||
# missing one route, treating 502 as a pass, or passing on status alone
|
||||
# without the route header, is rejected.
|
||||
name = "the readiness unit probes every collector route through the gateway";
|
||||
ok =
|
||||
let
|
||||
host = serviceContainersWithCi;
|
||||
otel = host.services.hyperhive.swarm.otel;
|
||||
locs = host.services.nginx.virtualHosts.${otel.domain}.locations;
|
||||
routes = lib.sort lib.lessThan (
|
||||
map (l: lib.removeSuffix "/" (lib.removePrefix "/" l)) (
|
||||
lib.filter (l: (locs.${l}.proxyPass or null) != null) (lib.attrNames locs)
|
||||
)
|
||||
);
|
||||
lines = script: map (lib.removePrefix " ") (lib.splitString "\n" script);
|
||||
probed =
|
||||
script:
|
||||
let
|
||||
loop = lib.findFirst (l: lib.hasPrefix "for route in " l) "" (lines script);
|
||||
in
|
||||
lib.sort lib.lessThan (
|
||||
lib.filter (w: w != "") (
|
||||
lib.splitString " " (lib.removeSuffix "; do" (lib.removePrefix "for route in " loop))
|
||||
)
|
||||
);
|
||||
verdict =
|
||||
script:
|
||||
probed script == routes
|
||||
&& lib.hasInfix "--resolve ${otel.domain}:443:127.0.0.1" script
|
||||
&& lib.hasInfix "https://${otel.domain}/$route/" script
|
||||
&& lib.hasInfix "case $code in 000 | 502 | 504) ;;" script
|
||||
&& lib.hasInfix "%header{x-swarm-otel-route}" script
|
||||
&& lib.hasInfix ''[ "$seen" = "$route" ] && continue 2'' script;
|
||||
script = host.systemd.services.swarm-otel-sockets.script;
|
||||
in
|
||||
lib.length routes > 2
|
||||
&& lib.elem "h1" routes
|
||||
&& lib.elem "swarm-grafana" routes
|
||||
&& verdict script
|
||||
&& !verdict (lib.replaceStrings [ " swarm-grafana" ] [ "" ] script)
|
||||
&& !verdict (lib.replaceStrings [ "000 | 502 | 504" ] [ "000 | 504" ] script)
|
||||
&& !verdict (lib.replaceStrings [ "--resolve" ] [ "--noresolve" ] script)
|
||||
&& !verdict (lib.replaceStrings [ "%header{x-swarm-otel-route}" ] [ "" ] script)
|
||||
&& !verdict (lib.replaceStrings [ ''[ "$seen" = "$route" ] && '' ] [ "" ] script);
|
||||
}
|
||||
{
|
||||
# The DynamicUser layout keeps the history under /var/lib/private; the
|
||||
# static user's StateDirectory is the plain path. The move runs as its
|
||||
|
|
|
|||
Loading…
Reference in a new issue