vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-otel: scrape grafana through its published name

Browse source 
Grafana's own metrics were scraped over loopback through the
`grafana-metrics` nginx vhost, declared only where grafana and the
collector share a host, so a collector on any other host had no job for
it. Its target is now declared on every host as
`https://<swarm.grafana.domain>/metrics`, next to the other published
targets, and grafana's vhost serves `= /metrics` behind the collector's
auth subrequest (`/__grafana_metrics_authz`, the forge's shape). The
rules derived from `publishedScrapeTargets` admit the collector's client
there and deny every other subject.

The loopback scrape is gone with the vhost that served it and its
`swarm.grafana.metricsPort` option, so a single host still scrapes
grafana once, now by name. A config that sets that option now fails
evaluation with "option does not exist".

On the published vhost `/metrics` was answered with 403 for everyone.
It now answers the collector's token, and a plain session gets authelia's
`deny`. The design note that ruled the published vhost out for scraping
predates the collector's credential; it is replaced by the location's
own comment and a line on `swarm.grafana.domain`.

Part of #5205
This commit is contained in:
atlas 2026-10-14 19:13:53 +02:00 • committed by mara
commit 54f6061d41
7 changed files with 145 additions and 99 deletions
Download patch file Download diff file Expand all files Collapse all files

67
nix/host-modules/swarm-grafana.nix
View file

@ -20,6 +20,7 @@ let
networkCfg = config.services.hyperhive.network;
hyperhiveCfg = config.services.hyperhive;
gatewayCfg = hyperhiveCfg.gateway;
autheliaDomain = hyperhiveCfg.swarm.authelia.domain;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
@ -459,12 +460,43 @@ in
listen = gatewayCfg.lib.listen;
extraConfig = gatewayCfg.lib.securityHeaders;
# Grafana serves its Prometheus exposition to anyone, signed in or not
# (`metrics.enabled` below, no `basic_auth_*`), so the published name
# refuses it; the scrape goes through the loopback `grafana-metrics`
# vhost instead. `=` is enough: nginx matches the decoded, normalised
# path, so `//metrics`, `/./metrics`, `/%6detrics` and `/metrics?x` all
# land here, and Grafana itself answers `/metrics/` with a login redirect.
locations."= /metrics".return = "403";
# (`metrics.enabled` below, no `basic_auth_*`), so this location's auth
# is the only gate on it. `=` is enough: nginx matches the decoded,
# normalised path, so `//metrics`, `/./metrics`, `/%6detrics` and
# `/metrics?x` all land here, and Grafana itself answers `/metrics/` with
# a login redirect.
#
# The collector requests exactly this URL, declared as
# `publishedScrapeTargets.grafana` in ./swarm-otel-service.nix; the two
# must agree, since that string is also the token's audience. Authelia's
# rules for it admit the collector's client and deny every other subject
# (`metricsRules` in ./swarm-authelia.nix).
locations."= /metrics" = {
proxyPass = "http://unix:${socketPath}:/metrics";
extraConfig = ''
auth_request /__grafana_metrics_authz;
'';
};
# The subrequest, same shape as the forge's `= /__forge_metrics_authz`
# (./hive-forge/default.nix): no `error_page 401 =302`, so a scraper
# gets the 401 rather than a login page parsed as metrics.
locations."= /__grafana_metrics_authz" = {
proxyPass = "https://${autheliaDomain}/api/authz/auth-request";
# nixpkgs appends its OWN `Host $host` after extraConfig, which would
# override verifiedProxyTo's — see hive-gateway/vhost-lib.nix.
recommendedProxySettings = false;
extraConfig = ''
internal;
${gatewayCfg.lib.verifiedProxyTo autheliaDomain}
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-Method $request_method;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
'';
};
locations."/" = {
proxyPass = "http://unix:${socketPath}:/";
proxyWebsockets = true;
@ -480,29 +512,6 @@ in
};
};
# The scrape listener: Grafana's `/metrics` and nothing else, on loopback.
#
# No `serverName` and no TLS on purpose — this listener exists for exactly
# one client on the same host, so there is no name to match and nothing to
# verify. It is the only vhost here that is not reachable from outside the
# machine, and that is what lets it skip the authorization the published
# one carries.
#
# ⚠️ `= /metrics` is an EXACT match, not a prefix: it is the difference
# between exposing one endpoint and re-serving the whole of Grafana on an
# unauthenticated port. Widening it would be a silent authorization bypass,
# since nothing else about this block would have to change. The published
# vhost above refuses `/metrics`, so this listener is the only way to it.
services.nginx.virtualHosts."grafana-metrics" = {
listen = [
{
addr = "127.0.0.1";
port = cfg.metricsPort;
}
];
locations."= /metrics".proxyPass = "http://unix:${socketPath}:/metrics";
};
# Order the container after the host CA service so the bind source below
# exists before nspawn sets the mount up.
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;