vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-otel: scrape grafana through its published name

Browse source 
Grafana's own metrics were scraped over loopback through the
`grafana-metrics` nginx vhost, declared only where grafana and the
collector share a host, so a collector on any other host had no job for
it. Its target is now declared on every host as
`https://<swarm.grafana.domain>/metrics`, next to the other published
targets, and grafana's vhost serves `= /metrics` behind the collector's
auth subrequest (`/__grafana_metrics_authz`, the forge's shape). The
rules derived from `publishedScrapeTargets` admit the collector's client
there and deny every other subject.

The loopback scrape is gone with the vhost that served it and its
`swarm.grafana.metricsPort` option, so a single host still scrapes
grafana once, now by name. A config that sets that option now fails
evaluation with "option does not exist".

On the published vhost `/metrics` was answered with 403 for everyone.
It now answers the collector's token, and a plain session gets authelia's
`deny`. The design note that ruled the published vhost out for scraping
predates the collector's credential; it is replaced by the location's
own comment and a line on `swarm.grafana.domain`.

Part of #5205
This commit is contained in:
atlas 2026-10-14 19:13:53 +02:00 • committed by mara
commit 54f6061d41
7 changed files with 145 additions and 99 deletions
Download patch file Download diff file Expand all files Collapse all files

24
nix/host-modules/glue-otel-scrape-targets.nix
View file

@ -1,21 +1,20 @@
# Glue: declare the loopback scrape targets of grafana and nats on the swarm's
# OTEL collector, wherever one of them shares a host with it. Authelia is a
# `publishedScrapeTargets` job (./swarm-otel-service.nix); a loopback job of
# Glue: declare the loopback scrape target of nats on the swarm's OTEL
# collector, wherever it shares a host with it. Authelia and grafana are
# `publishedScrapeTargets` jobs (./swarm-otel-service.nix); a loopback job of
# the same name fails ./swarm-otel.nix's job-name assertion.
#
# ONE FILE, TWO PAIRINGS — each service ← the collector, with the same shape.
# Deleting this leaves the collector scraping none of them: no error, no
# metrics.
# ONE PAIRING — nats ← the collector. Deleting this leaves the collector not
# scraping nats: no error, no metrics.
#
# Each target is gated on the service AND the collector being enabled here. The
# The target is gated on the service AND the collector being enabled here. The
# loopback address is only honest when both run on this host; a target nobody
# can read asserts a collection that is not happening. Both services default
# from `deploy.allSwarmServices`, but `mkDefault` is an invitation to override,
# so co-location is a property of the auto-deployed topology, not the modules.
# can read asserts a collection that is not happening. Both default from
# `deploy.allSwarmServices`, but `mkDefault` is an invitation to override, so
# co-location is a property of the auto-deployed topology, not the modules.
#
# ⚠️ Silent when split: separate hosts are separate evaluations with no shared
# context, so a host running a service and no collector declares nothing and
# the collector's host never learns of the service. No error, no warning — a
# context, so a host running nats and no collector declares nothing and
# the collector's host never learns of it. No error, no warning — a
# healthy exporter and an empty dashboard. Not guardable from here.
#
# ⚠️ `swarm.otel`, not `hyperhive.otel` — two collectors one word apart, and only
@ -33,7 +32,6 @@ let
in
{
config.services.hyperhive.swarm.otel.scrapeTargets = {
grafana = withCollector "grafana" "127.0.0.1:${toString swarmCfg.grafana.metricsPort}";
nats = withCollector "nats" "127.0.0.1:${toString swarmCfg.nats.metricsPort}";
};
}

53
nix/host-modules/swarm-grafana-service.nix
View file

@ -1,7 +1,7 @@
# The swarm's metrics UI as every hive sees it: the name it is served under,
# the port its `/metrics` is re-served on, and the OIDC client it is
# registered as, identical on every host. What the host running it decides,
# and the container itself, are in ./swarm-grafana.nix.
# The swarm's metrics UI as every hive sees it: the name it is served under
# and the OIDC client it is registered as, identical on every host. What the
# host running it decides, and the container itself, are in
# ./swarm-grafana.nix.
{
lib,
config,
@ -20,12 +20,11 @@ in
# `enable` is `services.hyperhive.deploy.grafana.enable` — see
# ./deploy.nix. Whether this host runs the swarm's Grafana is a
# deployment decision, and `swarm.*` has to be identical on every host.
# Here is what the service IS to every hive: the name it is served under,
# the port its `/metrics` is re-served on, and the OIDC client it is
# registered as. What the host running it decides — which build it runs,
# where its datasources point, which plugins are in its store path, and
# the socket directory it shares with nginx — is `deploy.grafana`, in
# ./swarm-grafana.nix.
# Here is what the service IS to every hive: the name it is served under
# and the OIDC client it is registered as. What the host running it
# decides — which build it runs, where its datasources point, which plugins
# are in its store path, and the socket directory it shares with nginx — is
# `deploy.grafana`, in ./swarm-grafana.nix.
options.services.hyperhive.swarm.grafana = {
machine = lib.mkOption {
type = lib.types.str;
@ -51,35 +50,13 @@ in
⚠️ Changing this changes the OAuth redirect URI, which authelia
matches exactly. Both sides move together because both derive from
this option; an operator who pins one by hand breaks the login.
'';
};
metricsPort = lib.mkOption {
type = lib.types.port;
default = 9095;
description = ''
Loopback port on which the gateway's nginx re-serves Grafana's
`/metrics`, and nothing else, so the swarm's collector can scrape it.
⚠️ **This is nginx's port, not Grafana's.** Grafana still claims none —
see {option}`services.hyperhive.deploy.grafana.socketDir` for why that
matters. A prometheus scrape target is a `host:port`, and it cannot
address a unix socket; rather than undo the socket decision, the one
endpoint a scraper needs gets a listener of its own.
Bound to loopback and unauthenticated, which is the same posture every
other entry in
{option}`services.hyperhive.swarm.otel.scrapeTargets` has: those
targets are trusted by *proximity* rather than by credential.
Deliberately **not** the published `grafana.<domain>` vhost, which
would put an authorization decision in front of a scrape: that vhost
answers `/metrics` with 403, because Grafana serves it without
authentication.
The number itself is arbitrary and free today;
`state/eval-port-collisions.sh` is what keeps it that way, since a
second claim on a port in this shared namespace produces no bind error
and nothing in any log.
The swarm collector scrapes Grafana's own metrics at
`https://<this name>/metrics`, from whichever host it runs on.
Grafana serves that endpoint to anyone, so the gateway puts it
behind authelia, whose rules admit only the collector's OAuth2
client there: see `publishedScrapeTargets` in
./swarm-otel-service.nix.
'';
};

67
nix/host-modules/swarm-grafana.nix
View file

@ -20,6 +20,7 @@ let
networkCfg = config.services.hyperhive.network;
hyperhiveCfg = config.services.hyperhive;
gatewayCfg = hyperhiveCfg.gateway;
autheliaDomain = hyperhiveCfg.swarm.authelia.domain;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
@ -459,12 +460,43 @@ in
listen = gatewayCfg.lib.listen;
extraConfig = gatewayCfg.lib.securityHeaders;
# Grafana serves its Prometheus exposition to anyone, signed in or not
# (`metrics.enabled` below, no `basic_auth_*`), so the published name
# refuses it; the scrape goes through the loopback `grafana-metrics`
# vhost instead. `=` is enough: nginx matches the decoded, normalised
# path, so `//metrics`, `/./metrics`, `/%6detrics` and `/metrics?x` all
# land here, and Grafana itself answers `/metrics/` with a login redirect.
locations."= /metrics".return = "403";
# (`metrics.enabled` below, no `basic_auth_*`), so this location's auth
# is the only gate on it. `=` is enough: nginx matches the decoded,
# normalised path, so `//metrics`, `/./metrics`, `/%6detrics` and
# `/metrics?x` all land here, and Grafana itself answers `/metrics/` with
# a login redirect.
#
# The collector requests exactly this URL, declared as
# `publishedScrapeTargets.grafana` in ./swarm-otel-service.nix; the two
# must agree, since that string is also the token's audience. Authelia's
# rules for it admit the collector's client and deny every other subject
# (`metricsRules` in ./swarm-authelia.nix).
locations."= /metrics" = {
proxyPass = "http://unix:${socketPath}:/metrics";
extraConfig = ''
auth_request /__grafana_metrics_authz;
'';
};
# The subrequest, same shape as the forge's `= /__forge_metrics_authz`
# (./hive-forge/default.nix): no `error_page 401 =302`, so a scraper
# gets the 401 rather than a login page parsed as metrics.
locations."= /__grafana_metrics_authz" = {
proxyPass = "https://${autheliaDomain}/api/authz/auth-request";
# nixpkgs appends its OWN `Host $host` after extraConfig, which would
# override verifiedProxyTo's — see hive-gateway/vhost-lib.nix.
recommendedProxySettings = false;
extraConfig = ''
internal;
${gatewayCfg.lib.verifiedProxyTo autheliaDomain}
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-Method $request_method;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
'';
};
locations."/" = {
proxyPass = "http://unix:${socketPath}:/";
proxyWebsockets = true;
@ -480,29 +512,6 @@ in
};
};
# The scrape listener: Grafana's `/metrics` and nothing else, on loopback.
#
# No `serverName` and no TLS on purpose — this listener exists for exactly
# one client on the same host, so there is no name to match and nothing to
# verify. It is the only vhost here that is not reachable from outside the
# machine, and that is what lets it skip the authorization the published
# one carries.
#
# ⚠️ `= /metrics` is an EXACT match, not a prefix: it is the difference
# between exposing one endpoint and re-serving the whole of Grafana on an
# unauthenticated port. Widening it would be a silent authorization bypass,
# since nothing else about this block would have to change. The published
# vhost above refuses `/metrics`, so this listener is the only way to it.
services.nginx.virtualHosts."grafana-metrics" = {
listen = [
{
addr = "127.0.0.1";
port = cfg.metricsPort;
}
];
locations."= /metrics".proxyPass = "http://unix:${socketPath}:/metrics";
};
# Order the container after the host CA service so the bind source below
# exists before nspawn sets the mount up.
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;

2
nix/host-modules/swarm-otel-service.nix
View file

@ -362,5 +362,7 @@ in
victoriametrics = "https://${vmCfg.domain}/metrics";
# The vhost location is in ./swarm-victorialogs.nix.
victorialogs = "https://${hyperhiveCfg.swarm.victorialogs.domain}/metrics";
# The vhost location is in ./swarm-grafana.nix.
grafana = "https://${hyperhiveCfg.swarm.grafana.domain}/metrics";
};
}