vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

grafana: query the log store with the metrics bearer, by its swarm name

Browse source 
Grafana's VictoriaLogs datasource now targets https://logs.<swarm> and sends
the same static bearer as the metrics datasource. The log store's vhost routes
a 64-hex Authorization bearer to a second swarm-metrics-auth instance
(swarm-logs-grafana-auth, cert role swarm-logs-auth, which gains a read on the
bearer's store path). swarm-agent. tokens and everything else keep their routes.

Part of #3742 (DoD 3, logs).
This commit is contained in:
atlas 2026-10-16 01:32:24 +02:00 • committed by mara
commit eff5c43bdc
9 changed files with 200 additions and 52 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/swarm/credentials.md
View file

@ -1,4 +1,4 @@
# Credentials: the target shape , and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API# Credentials: the target shape
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->

4
docs/swarm/secrets.md
View file

@ -44,7 +44,7 @@ neither is a renaming of the other.
## Swarm-level — one of each per swarm ## Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere | | secret | generated by | lives at | hive elsewhere |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | | swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
@ -54,7 +54,7 @@ neither is a renaming of the other.
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | | Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. `swarm-metrics-auth` on the metrics store's host reads the stored value itself, under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | | Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | | the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |

13
docs/swarm/services.md
View file

@ -1,3 +1,5 @@
| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://<swarm.victorialogs.domain>`), so Grafana and the store can run on different hosts. |
# Swarm-wide services # Swarm-wide services
Some things exist once per **swarm** rather than once per hive: the forge, Some things exist once per **swarm** rather than once per hive: the forge,
@ -247,6 +249,17 @@ swarm.
It binds loopback and takes no credential of its own: the gateway vhost is It binds loopback and takes no credential of its own: the gateway vhost is
the only way in, and the collector is the only intended writer. the only way in, and the collector is the only intended writer.
**Who may query.** `/select/logsql/` takes one of three credentials. An
agent presents its `swarm-agent.<agent>.<secret>` bearer, checked by
`swarm-logs-auth`. Grafana's datasource presents the same bearer as its
metrics datasource, which `swarm-bao-grafana-metrics-bearer` creates, and a
second `swarm-metrics-auth` instance (`swarm-logs-grafana-auth`) on the log
store's host checks it against the same store path; the gateway tells it from
the others by its shape, 64 hex digits. Anything else needs an operator's
authelia session. Because one bearer opens both read APIs, a rotation
(see [`secrets.md`](secrets.md)) restarts Grafana once, and the log store
accepts the new value within a minute.
The collector also scrapes this store's own metrics, at The collector also scrapes this store's own metrics, at
`https://<swarm.victorialogs.domain>/metrics`, so a collector on a host `https://<swarm.victorialogs.domain>/metrics`, so a collector on a host
without the store still gets them. The gateway admits only the collector's without the store still gets them. The gateway admits only the collector's

17
nix/host-modules/swarm-bao.nix
View file

@ -1003,10 +1003,10 @@ let
# of `adminPasswordStorePath` in ./swarm-grafana.nix. # of `adminPasswordStorePath` in ./swarm-grafana.nix.
grafanaAdminLeaf = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/admin/password"; grafanaAdminLeaf = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/admin/password";
# The bearer Grafana's metrics datasource presents at the metrics store. # The bearer Grafana's datasources present at the metrics and log stores.
# Grafana's leaf creates it, the store's responder reads it. The nix half of # Grafana's leaf creates it, the stores' responders read it. The nix half of
# `metricsBearerStorePath` in ./swarm-grafana.nix and of `--store-path` in # `metricsBearerStorePath` in ./swarm-grafana.nix and of `--store-path` in
# ./swarm-victoriametrics.nix. # ./swarm-victoriametrics.nix and ./swarm-victorialogs.nix.
grafanaMetricsBearerLeaf = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/metrics/bearer"; grafanaMetricsBearerLeaf = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/metrics/bearer";
createStanza = path: '' createStanza = path: ''
@ -1015,14 +1015,17 @@ let
} }
''; '';
# The log store's agent auth responder, which checks the secret an agent # The log store's auth responders: the agent one checks the secret an agent
# presents against the one stored for it: the same `+` leaf as the queue # presents against the one stored for it (the same `+` leaf as the queue
# responder's, and nothing else. # responder's), the other checks the bearer Grafana presents against
# Grafana's metrics bearer. Nothing else.
logsAuthReaders = [ logsAuthReaders = [
{ {
name = "swarm-logs-auth"; name = "swarm-logs-auth";
cn = baoDeploy.logsAuthCommonName; cn = baoDeploy.logsAuthCommonName;
policyText = readStanza "${credentialMountPath}/data/swarm/agents/+/queue"; policyText =
readStanza "${credentialMountPath}/data/swarm/agents/+/queue"
+ readStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}";
} }
]; ];

22
nix/host-modules/swarm-grafana.nix
View file

@ -274,8 +274,10 @@ let
# The bearer the metrics datasource presents at the metrics store's query # The bearer the metrics datasource presents at the metrics store's query
# API, which `swarm-metrics-auth` on the store's host checks against the # API, which `swarm-metrics-auth` on the store's host checks against the
# same store path. Minted once by `swarm-bao-grafana-metrics-bearer` below, # same store path, and the logs datasource at the log store's query API,
# under the same grant as the admin password. The file holds the whole # which a second `swarm-metrics-auth` instance in ./swarm-victorialogs.nix
# checks against it. Minted once by `swarm-bao-grafana-metrics-bearer`
# below, under the same grant as the admin password. The file holds the whole
# header value, `Bearer <value>`: nixpkgs accepts a `secureJsonData` entry # header value, `Bearer <value>`: nixpkgs accepts a `secureJsonData` entry
# only when it is exactly one `$__file{}`. # only when it is exactly one `$__file{}`.
metricsBearerStorePath = "secret/swarm/services/${cfg.oidc.clientId}/metrics/bearer"; metricsBearerStorePath = "secret/swarm/services/${cfg.oidc.clientId}/metrics/bearer";
@ -371,16 +373,20 @@ in
logsDatasourceUrl = lib.mkOption { logsDatasourceUrl = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "http://127.0.0.1:${toString hyperhiveCfg.swarm.victorialogs.port}"; default = "https://${hyperhiveCfg.swarm.victorialogs.domain}";
defaultText = lib.literalExpression ''"http://127.0.0.1:''${toString services.hyperhive.swarm.victorialogs.port}"''; defaultText = lib.literalExpression ''"https://''${services.hyperhive.swarm.victorialogs.domain}"'';
description = '' description = ''
Where the provisioned logs datasource points. Defaults to the log Where the provisioned logs datasource points. Defaults to the log
store on this host, which binds loopback, so this default reaches store's swarm name, served by the gateway of whichever host runs the
it only where Grafana and the store share a host. store, so Grafana and the store need not share a host.
Provisioned unconditionally, like the metrics datasource — the store Provisioned unconditionally, like the metrics datasource — the store
being off is a deployment choice rather than a reason to withhold the being off is a deployment choice rather than a reason to withhold the
connection, and an operator whose logs live elsewhere sets this. connection, and an operator whose logs live elsewhere sets this.
The datasource sends the same bearer as the metrics one, which the
store's `/select/logsql/` route checks. Whatever URL this names
receives that bearer.
''; '';
}; };
@ -1316,7 +1322,11 @@ in
# not in the pipeline. # not in the pipeline.
jsonData = { jsonData = {
inherit logLevelRules; inherit logLevelRules;
httpHeaderName1 = "Authorization";
}; };
# The metrics datasource's bearer; the log store's vhost
# accepts it on `/select/logsql/`.
secureJsonData.httpHeaderValue1 = "$__file{${metricsBearerPath}}";
} }
]; ];
}; };

79
nix/host-modules/swarm-victorialogs.nix
View file

@ -68,6 +68,15 @@ let
}; };
baoCfg = hyperhiveCfg.swarm.bao; baoCfg = hyperhiveCfg.swarm.bao;
# Grafana's bearer, checked by a second `swarm-metrics-auth` instance. It
# reads the store with the same leaf as `swarm-logs-auth`, which is granted
# the one extra path below. The same leaf as `grafanaBearerStorePath` in
# ./swarm-victoriametrics.nix and `grafanaMetricsBearerLeaf` in ./swarm-bao.nix.
grafanaAuthUnit = "swarm-logs-grafana-auth";
grafanaAuthSocket = "/run/${grafanaAuthUnit}/auth.sock";
grafanaAuthCredential = id: "/run/credentials/${grafanaAuthUnit}.service/${id}";
grafanaBearerStorePath = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/metrics/bearer";
# Shared host netns, like every sibling swarm container: Grafana reaches # Shared host netns, like every sibling swarm container: Grafana reaches
# this at 127.0.0.1:<port>. # this at 127.0.0.1:<port>.
privateNetwork = false; privateNetwork = false;
@ -210,6 +219,7 @@ in
services.nginx.appendHttpConfig = '' services.nginx.appendHttpConfig = ''
map $http_authorization $swarm_logs_auth_target { map $http_authorization $swarm_logs_auth_target {
"~^Bearer swarm-agent\." /__swarm_logs_agent_auth; "~^Bearer swarm-agent\." /__swarm_logs_agent_auth;
"~^Bearer [0-9a-f]{64}$" /__swarm_logs_grafana_auth;
default /__hive_authelia; default /__hive_authelia;
} }
limit_req_zone $binary_remote_addr zone=swarm_logs_agent_auth:1m rate=${deployCfg.victorialogs.authRate}; limit_req_zone $binary_remote_addr zone=swarm_logs_agent_auth:1m rate=${deployCfg.victorialogs.authRate};
@ -289,9 +299,9 @@ in
# permissions are a later thing, and this location is where one # permissions are a later thing, and this location is where one
# attaches when it exists. # attaches when it exists.
# #
# Two checks: an agent's own `swarm-agent.` bearer goes to # Three checks: an agent's own `swarm-agent.` bearer goes to
# `swarm-logs-auth`, anything else to authelia (`appendHttpConfig` # `swarm-logs-auth`, Grafana's 64-hex bearer to `swarm-metrics-auth`,
# above picks). # anything else to authelia (`appendHttpConfig` above picks).
"^~ /select/logsql/" = { "^~ /select/logsql/" = {
# No URI part, so the request path and its query string reach # No URI part, so the request path and its query string reach
# the store as the caller sent them. # the store as the caller sent them.
@ -345,6 +355,18 @@ in
''; '';
}; };
# Grafana's bearer, 64 hex digits (`openssl rand -hex 32`), checked by
# `swarm-metrics-auth`. The responder reads the store at most once per
# its cache TTL whatever the bearers, so no rate limit sits in front.
"= /__swarm_logs_grafana_auth" = {
proxyPass = "http://unix:${grafanaAuthSocket}:/";
extraConfig = ''
internal;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
'';
};
# The subrequest itself — same target, same header set, same # The subrequest itself — same target, same header set, same
# reasoning as `swarm-ui.nix`'s own copy (measured against the # reasoning as `swarm-ui.nix`'s own copy (measured against the
# pinned authelia binary, not copied from an example). # pinned authelia binary, not copied from an example).
@ -430,6 +452,57 @@ in
}; };
}; };
# Grafana's check on `/select/logsql/`: the metrics store's responder
# binary, pointed at the bearer Grafana's datasources share.
users.users.${grafanaAuthUnit} = {
isSystemUser = true;
group = grafanaAuthUnit;
description = "swarm log store auth_request responder for Grafana's bearer";
};
users.groups.${grafanaAuthUnit} = { };
systemd.services.${grafanaAuthUnit} = {
description = "swarm log store auth_request responder for Grafana's bearer";
wantedBy = [ "multi-user.target" ];
after = [
"network.target"
"swarm-bao-pki.service"
];
serviceConfig = {
ExecStart = lib.concatStringsSep " " [
"${deployCfg.victoriametrics.authPackage}/bin/swarm-metrics-auth"
"--socket ${grafanaAuthSocket}"
"--store-cert-role swarm-logs-auth"
"--store-path ${grafanaBearerStorePath}"
];
LoadCredential = lib.mapAttrsToList (id: path: "${id}:${path}") logsAuthStoreFiles;
# ⚠️ A static user, never `DynamicUser`: see the `swarm-logs-auth` unit above.
User = grafanaAuthUnit;
Group = grafanaAuthUnit;
RuntimeDirectory = grafanaAuthUnit;
RuntimeDirectoryMode = "0751";
RuntimeDirectoryPreserve = "yes";
Restart = "on-failure";
RestartSec = "5s";
SyslogIdentifier = grafanaAuthUnit;
ProtectSystem = "strict";
ProtectHome = true;
PrivateTmp = "disconnected";
PrivateDevices = true;
NoNewPrivileges = true;
RestrictSUIDSGID = true;
RemoveIPC = true;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = grafanaAuthCredential "bao-client.pem";
BAO_CLIENT_KEY = grafanaAuthCredential "bao-client-key.pem";
}
// lib.optionalAttrs (logsAuthStoreFiles ? "bao-ca.pem") {
BAO_CACERT = grafanaAuthCredential "bao-ca.pem";
};
};
containers.${cfg.machine} = { containers.${cfg.machine} = {
autoStart = true; autoStart = true;
ephemeral = false; ephemeral = false;

22
nix/module-eval/bao-grants.nix
View file

@ -726,18 +726,24 @@ let
lib.hasInfix "/nats-auth.pem ]" s && lib.hasInfix "swarm-nats-auth \"\" clientAuth" s; lib.hasInfix "/nats-auth.pem ]" s && lib.hasInfix "swarm-nats-auth \"\" clientAuth" s;
} }
{ {
# The same `+` leaf as the queue responder, read-only, and nothing else: # Every agent's queue secret and Grafana's metrics bearer, both
# this principal checks a presented secret and does nothing with it. # read-only, and nothing else: this principal checks a presented secret
name = "the log store responder's grant is every agent's queue secret, read-only, and nothing else"; # and does nothing with it.
name = "the log store responder's grant is every agent's queue secret and Grafana's bearer, read-only, and nothing else";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-logs-auth-policy.script; s = baoGrantHere.systemd.services.swarm-bao-logs-auth-policy.script;
in in
lib.hasInfix "path \"secret/data/swarm/agents/+/queue\" {\n capabilities = [\"read\"]\n}" s grantsIn s == [
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 1 {
&& !(lib.hasInfix "secret/data/swarm/agents/*" s) path = "secret/data/swarm/agents/+/queue";
&& !(lib.hasInfix "secret/data/swarm/hives" s) caps = [ "read" ];
&& !(lib.hasInfix "secret/data/swarm/services" s) }
{
path = "secret/data/swarm/services/swarm-grafana/metrics/bearer";
caps = [ "read" ];
}
]
&& lib.hasInfix "auth/cert/certs/swarm-logs-auth" s && lib.hasInfix "auth/cert/certs/swarm-logs-auth" s
&& lib.hasInfix "allowed_common_names=swarm-logs-auth" s && lib.hasInfix "allowed_common_names=swarm-logs-auth" s
&& lib.hasInfix "token_policies=swarm-logs-auth" s; && lib.hasInfix "token_policies=swarm-logs-auth" s;

25
nix/module-eval/core-toggle.nix
View file

@ -627,6 +627,22 @@ let
&& lib.hasInfix "path \"secret/data/${leaf}\" {\n capabilities = [\"read\"]\n}" s.swarm-bao-metrics-auth-policy.script && lib.hasInfix "path \"secret/data/${leaf}\" {\n capabilities = [\"read\"]\n}" s.swarm-bao-metrics-auth-policy.script
&& lib.hasInfix "path \"secret/data/${leaf}\" {\n capabilities = [\"create\", \"read\"]\n}" s.swarm-bao-grafana-oidc-policy.script; && lib.hasInfix "path \"secret/data/${leaf}\" {\n capabilities = [\"create\", \"read\"]\n}" s.swarm-bao-grafana-oidc-policy.script;
} }
{
# The log store's second responder checks the very bearer the metrics
# datasource sends, at the same store path: a different file would be a
# bearer the responder never reads.
name = "Grafana's logs datasource sends the bearer the log store's responder checks, at one store path";
ok =
let
datasources =
allLocal.containers.swarm-grafana.config.services.grafana.provision.datasources.settings.datasources;
metrics = lib.findFirst (d: d.uid == "swarm-victoriametrics") null datasources;
logs = lib.findFirst (d: d.uid == "swarm-victorialogs") null datasources;
leaf = "swarm/services/${allLocal.services.hyperhive.swarm.grafana.oidc.clientId}/metrics/bearer";
in
logs.secureJsonData.httpHeaderValue1 == metrics.secureJsonData.httpHeaderValue1
&& lib.hasInfix "--store-path ${leaf}" allLocal.systemd.services.swarm-logs-grafana-auth.serviceConfig.ExecStart;
}
{ {
# The responder reads the store as itself, with the leaf the store's # The responder reads the store as itself, with the leaf the store's
# host mints for it. # host mints for it.
@ -709,15 +725,23 @@ let
l = allLocal.services.nginx.virtualHosts."logs.t.local".locations; l = allLocal.services.nginx.virtualHosts."logs.t.local".locations;
http = allLocal.services.nginx.appendHttpConfig; http = allLocal.services.nginx.appendHttpConfig;
unit = allLocal.systemd.services.swarm-logs-auth; unit = allLocal.systemd.services.swarm-logs-auth;
grafanaUnit = allLocal.systemd.services.swarm-logs-grafana-auth;
in in
lib.hasInfix "map \$http_authorization \$swarm_logs_auth_target" http lib.hasInfix "map \$http_authorization \$swarm_logs_auth_target" http
&& lib.hasInfix "\"~^Bearer swarm-agent\\.\" /__swarm_logs_agent_auth;" http && lib.hasInfix "\"~^Bearer swarm-agent\\.\" /__swarm_logs_agent_auth;" http
&& lib.hasInfix "\"~^Bearer [0-9a-f]{64}$\" /__swarm_logs_grafana_auth;" http
&& lib.hasInfix "default /__hive_authelia;" http && lib.hasInfix "default /__hive_authelia;" http
&& lib.hasInfix "auth_request /__swarm_logs_auth;" l."^~ /select/logsql/".extraConfig && lib.hasInfix "auth_request /__swarm_logs_auth;" l."^~ /select/logsql/".extraConfig
&& lib.hasInfix "rewrite ^ \$swarm_logs_auth_target last;" l."= /__swarm_logs_auth".extraConfig && lib.hasInfix "rewrite ^ \$swarm_logs_auth_target last;" l."= /__swarm_logs_auth".extraConfig
&& lib.hasInfix "internal;" l."= /__swarm_logs_agent_auth".extraConfig && lib.hasInfix "internal;" l."= /__swarm_logs_agent_auth".extraConfig
&& l."= /__swarm_logs_agent_auth".proxyPass == "http://unix:/run/swarm-logs-auth/auth.sock:/" && l."= /__swarm_logs_agent_auth".proxyPass == "http://unix:/run/swarm-logs-auth/auth.sock:/"
&& lib.hasInfix "--socket /run/swarm-logs-auth/auth.sock" unit.serviceConfig.ExecStart && lib.hasInfix "--socket /run/swarm-logs-auth/auth.sock" unit.serviceConfig.ExecStart
&& lib.hasInfix "internal;" l."= /__swarm_logs_grafana_auth".extraConfig
&&
l."= /__swarm_logs_grafana_auth".proxyPass == "http://unix:/run/swarm-logs-grafana-auth/auth.sock:/"
&& lib.hasInfix "--socket /run/swarm-logs-grafana-auth/auth.sock" grafanaUnit.serviceConfig.ExecStart
&& lib.hasInfix "swarm-metrics-auth" grafanaUnit.serviceConfig.ExecStart
&& lib.hasInfix "--store-cert-role swarm-logs-auth" grafanaUnit.serviceConfig.ExecStart
&& l ? "= /__hive_authelia"; && l ? "= /__hive_authelia";
} }
{ {
@ -760,6 +784,7 @@ let
]); ]);
in in
owners ? swarm-logs-auth owners ? swarm-logs-auth
&& owners ? swarm-logs-grafana-auth
&& owners ? swarm-metrics-auth && owners ? swarm-metrics-auth
&& !(lib.any (u: hidden u.serviceConfig) (lib.attrValues owners)); && !(lib.any (u: hidden u.serviceConfig) (lib.attrValues owners));
} }

18
nix/module-eval/grafana.nix
View file

@ -378,6 +378,24 @@ let
lib.any (e: lib.hasInfix "PRIORITY:*" e) counts lib.any (e: lib.hasInfix "PRIORITY:*" e) counts
&& lib.any (e: lib.hasInfix "PRIORITY:\"\"" e) counts; && lib.any (e: lib.hasInfix "PRIORITY:\"\"" e) counts;
} }
{
# The logs datasource reaches the store through its swarm name — a
# loopback URL works only where Grafana and the store share a host — and
# sends the metrics datasource's bearer, which the store's `/select/logsql/`
# route checks.
name = "the logs datasource targets the log store's swarm name and sends the metrics bearer";
ok =
let
datasources =
grafanaWithAuthelia.containers.swarm-grafana.config.services.grafana.provision.datasources.settings.datasources;
metrics = lib.findFirst (d: d.uid == "swarm-victoriametrics") null datasources;
logs = lib.findFirst (d: d.uid == "swarm-victorialogs") null datasources;
in
logs.url == "https://${grafanaWithAuthelia.services.hyperhive.swarm.victorialogs.domain}"
&& !(lib.hasInfix "127.0.0.1" logs.url)
&& logs.jsonData.httpHeaderName1 == "Authorization"
&& logs.secureJsonData.httpHeaderValue1 == metrics.secureJsonData.httpHeaderValue1;
}
{ {
# The reader's half of the same mapping. The collector writes a severity # The reader's half of the same mapping. The collector writes a severity
# TEXT; nothing downstream reads it as a level unless the datasource is # TEXT; nothing downstream reads it as a level unless the datasource is