swarm-grafana: one delivery route for the OIDC client secret
The previous commit left two delivery paths and a three-way gate: `swarm-grafana-oidc-secret.service` copied authelia's minted plaintext out of its host tree wherever the two were co-located, `swarm-bao-grafana-oidc.service` read the same value from the swarm secret store wherever they were not, and `ssoConfigured && (ssoLocal || haveClientIdentity)` decided whether Grafana got an OIDC block at all. Delete the co-located path. The store reader is now THE delivery unit, in every deployment — the publisher on authelia's host writes `swarm/services/<id>/oidc/client` whether the reader is a network away or in the container next door. The ruling behind it: the store exists so a host holds ONE out-of-band secret, its client certificate, and reads everything else with it. Skipping the store when the producer happens to be local saves a round trip and costs a second delivery unit, a second way for the file to be wrong, and a gate to choose between them. The gate goes too, and both of its questions become assertions, scoped to hosts that run Grafana: - `swarm.authelia.url` must be set. `auth.disable_login_form` is unconditional — Grafana ships an admin/admin account on a public vhost — so dropping the OIDC block when the swarm names no IdP produced a container with no SSO and no password box, silently. An eval-time refusal naming the option is the only report that reaches anyone, the shape swarm-nats.nix already uses for the same option. - `deploy.bao.clientCertFile` / `clientKeyFile` must be set. This replaces a warning that nothing reads back, and its message names both options and where the leaf comes from. Fixtures follow. `grafanaWithAuthelia` gains the cert pair, because a co-located host is a store reader like any other. The old `grafanaRemoteAutheliaNoIdentity` is kept rather than deleted, renamed `grafanaNoIdentity`: the shape is still reachable, only its deliverable changed from silence to a refusal, and an arm now reads that refusal back. Its mirror `grafanaNoSso` covers the other assertion, each fixture wrong in exactly one way so an arm can name which refusal fired. Every positive keeps an explicit negative — the one-delivery-unit arm asserts the deleted unit is absent in both topologies rather than merely that the store reader is present. Refs #4234 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
atlas
mara
parent
4aa982cc2a
commit
815f977d7c
5 changed files with 295 additions and 233 deletions
|
|
@ -41,23 +41,23 @@ neither is a renaming of the other.
|
|||
|
||||
## Swarm-level — one of each per swarm
|
||||
|
||||
| secret | generated by | lives at | hive elsewhere |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
|
||||
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
|
||||
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
|
||||
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
||||
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
||||
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||
| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia |
|
||||
| Grafana's copy of its OIDC secret | `swarm-grafana-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store instead — nothing for an operator to place, see below |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
|
||||
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
||||
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
||||
| secret | generated by | lives at | hive elsewhere |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when `autoConfigure` is set | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
|
||||
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
|
||||
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
|
||||
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
||||
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
||||
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||
| the swarm collector's copy of its OIDC secret | `swarm-otel-oidc-secret.service` copies it from authelia's tree, **when authelia runs on this host** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | operator provides the file and names it in `deploy.swarm-otel.clientSecretFile` — the collector need not share a host with authelia |
|
||||
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when `deploy.nats.autoGenerateCallout` is set | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
|
||||
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
||||
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
||||
|
||||
Authelia mints the three keys for itself, in-container, precisely
|
||||
because nothing outside that container ever reads them. **That's the test worth
|
||||
|
|
@ -185,18 +185,32 @@ the harness logs that it has no queue coordinates — the same absent-and-legal
|
|||
rather than a container that refuses to start.
|
||||
|
||||
The **third reader** is Grafana's OIDC client secret, and it's the first one
|
||||
that's a _swarm service's_ own credential rather than a hive's.
|
||||
`swarm-grafana.nix` asks two questions where it used to ask one: whether this
|
||||
_swarm_ has an IdP (`swarm.authelia.url`) decides whether Grafana gets SSO
|
||||
settings at all, and whether authelia is on _this_ host decides only which of
|
||||
two units delivers the secret — the host copy out of authelia's tree, or
|
||||
`swarm-bao-grafana-oidc.service` reading `swarm/services/<client-id>/oidc/client`
|
||||
out of the store. Exactly one of the two exists in any deployment, and where
|
||||
neither can (no authelia here, no store identity here) the module warns and
|
||||
emits no OIDC settings rather than pointing Grafana at a file nothing writes.
|
||||
Registration moved to `glue-grafana-oidc-client.nix` for a related reason: a
|
||||
client is a row in authelia's own config, so it has to be declared on the host
|
||||
that runs authelia — which is not necessarily the host that runs Grafana.
|
||||
that's a _swarm service's_ own credential rather than a hive's. It has exactly
|
||||
one delivery route: `swarm-bao-grafana-oidc.service` reads
|
||||
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
|
||||
that runs Grafana. Where authelia is doesn't enter into it — the publisher on
|
||||
authelia's host writes that path whether the reader is a network away or in the
|
||||
container next door.
|
||||
|
||||
That's a deliberate refusal of the shortcut. Copying the plaintext
|
||||
straight out of authelia's tree when the two happen to share a host is one
|
||||
round trip cheaper and costs a second delivery unit, a second way for the file
|
||||
to be wrong, and a gate deciding between them — and the gate is the expensive
|
||||
part, because whatever it's wrong about is an outage nobody can read. The store
|
||||
exists so a host holds **one** out-of-band secret, its client certificate, and
|
||||
reads everything else with it.
|
||||
|
||||
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
|
||||
running Grafana requires `swarm.authelia.url` (its local login form is disabled
|
||||
unconditionally, so SSO isn't a feature of some topologies — it's the only way
|
||||
in), and it requires this host's `deploy.bao.clientCertFile` /
|
||||
`clientKeyFile`. Each refusal names the option to set. Both used to be silent:
|
||||
a null URL dropped the OIDC block, a missing leaf produced a warning, and
|
||||
either one left a Grafana with no SSO and no password box — nothing failed, so
|
||||
nothing said anything. Registration is separate and stays where authelia is
|
||||
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
|
||||
so it's declared on the host running authelia, which isn't necessarily the host
|
||||
running Grafana.
|
||||
|
||||
A service's secret is one value for the whole swarm rather than one per hive, so
|
||||
it lives under the `services` prefix, and a hive's read policy grants that prefix
|
||||
|
|
|
|||
|
|
@ -71,11 +71,11 @@ for the whole swarm, so its credential's path names the service and never the
|
|||
host — and which hive runs a given service is a `deploy.*` fact, per-host by
|
||||
definition, so nothing swarm-wide exists to scope the grant to. The host running
|
||||
such a service has no store identity of its own either; it reads with the
|
||||
certificate of the hive it is. So **any hive can read any swarm service's client
|
||||
secret**, which is worth what it buys: a swarm service whose identity provider
|
||||
lives on another machine can get its secret at all, instead of only working
|
||||
co-located. Giving such a service its own store identity is what would remove
|
||||
this rather than re-scope it.
|
||||
certificate of the hive it is. **Any hive can therefore read any swarm service's
|
||||
client secret**, which is worth what it buys: every swarm service gets its
|
||||
credential the same way from anywhere, instead of only where the identity
|
||||
provider happens to sit. Giving such a service its own store identity is what
|
||||
would remove this rather than re-scope it.
|
||||
|
||||
**A tracked follow-up** narrows this, with the two candidate directions: scope
|
||||
the grant per hive (and pay for the re-emission), or give each agent container
|
||||
|
|
|
|||
|
|
@ -83,38 +83,27 @@ let
|
|||
# Is SSO configured for this SWARM. Swarm-wide by construction — `swarm.*` is
|
||||
# identical on every host — and the option's own description is what makes
|
||||
# this the right question to ask: a null URL means "no SSO configured".
|
||||
ssoConfigured = autheliaCfg.url != null;
|
||||
|
||||
# Is authelia on THIS host, spelled exactly as the forge and matrix modules
|
||||
# spell it. All locality decides is where the minted plaintext is copied
|
||||
# FROM: authelia's own tree here, the swarm secret store otherwise.
|
||||
#
|
||||
# 🩸 This used to be `grafana.enable && authelia.enable` and was the ONLY
|
||||
# gate: registration, the mint's delivery and the OIDC block all hung off
|
||||
# it, so a swarm whose authelia ran elsewhere got Grafana with no SSO wiring
|
||||
# — and the login form is disabled whatever happens, so with nothing else.
|
||||
ssoLocal = deployCfg.authelia.enable;
|
||||
# 🩸 A subject of an assertion below, NOT a gate. Dropping the OIDC block when
|
||||
# this is false looks conservative and is the outage: `disable_login_form` is
|
||||
# unconditional a few hundred lines down, so a Grafana with no OIDC settings
|
||||
# is a Grafana with no login of any kind, arrived at silently. SSO is a
|
||||
# requirement of running this service, so an unconfigured swarm fails to
|
||||
# build and says which option to set.
|
||||
ssoConfigured = autheliaCfg.url != null;
|
||||
|
||||
# A reader of the store is defined by holding a certificate the store
|
||||
# accepts, never by standing next to it — the rule
|
||||
# ./glue-matrix-bao-token.nix states in full.
|
||||
#
|
||||
# Also asserted rather than gating anything, and that is the ruling this
|
||||
# module's second delivery route was deleted under: the store exists so that
|
||||
# ONE secret is placed out of band per host — this certificate — and every
|
||||
# other secret is read with it. A Grafana host without it has not been given
|
||||
# its identity yet, which is a thing to say out loud rather than to route
|
||||
# around by reaching into authelia's tree whenever it happens to be local.
|
||||
haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
|
||||
|
||||
# Whether Grafana gets an OIDC block: the swarm-wide question, AND that
|
||||
# something on this host delivers the secret that block names. The second
|
||||
# half is not redundant — a block naming a file no unit here writes cannot
|
||||
# produce a working login either way, and what Grafana does with an
|
||||
# unreadable `$__file{}` target (refuse to start, or start and fail every
|
||||
# login) is not worth betting the container on when not emitting the block is
|
||||
# available and leaves this host as it was.
|
||||
ssoWired = ssoConfigured && (ssoLocal || haveClientIdentity);
|
||||
|
||||
# The delivery that is not a copy: authelia is elsewhere, so the same minted
|
||||
# plaintext arrives out of the store that the publisher on authelia's host
|
||||
# wrote it into. Exclusive with the copy unit by construction — two units
|
||||
# writing one file is a race over which secret Grafana parses.
|
||||
storeDelivery = ssoWired && !ssoLocal;
|
||||
|
||||
autheliaUrl = toString autheliaCfg.url;
|
||||
|
||||
# Where the plaintext lands inside the container. Under /var/lib rather
|
||||
|
|
@ -123,9 +112,10 @@ let
|
|||
# intermittent one.
|
||||
secretPath = "/var/lib/grafana-oidc/${cfg.oidc.clientId}.secret";
|
||||
|
||||
# The same file seen from the host, which is where both delivery units write
|
||||
# it. Spelled once: two units landing a secret at two spellings of one path
|
||||
# is a Grafana that reads whichever of them it was configured with.
|
||||
# The same file seen from the host, which is where the delivery unit writes
|
||||
# it. Spelled once: the unit that writes it and the config that names it are
|
||||
# a few hundred lines apart, and a Grafana reading a path nothing writes is
|
||||
# a login that fails with nothing in any log about the file.
|
||||
hostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${secretPath}";
|
||||
hostSecretDir = builtins.dirOf hostSecretPath;
|
||||
|
||||
|
|
@ -401,15 +391,16 @@ in
|
|||
# The secret oneshots as well as grafana itself: each runs before it and
|
||||
# fails in ways grafana then reports only as a login that does not work.
|
||||
#
|
||||
# Each delivery unit is listed only where it exists, the way
|
||||
# ./hive-matrix.nix lists its own: a unit name that never renders is a
|
||||
# journald scrape target matching nothing, which reads as a quiet unit.
|
||||
# Unconditional, because every unit named here now renders in every
|
||||
# deployment. This list used to be assembled with `lib.optional` per
|
||||
# delivery route, which was the right shape while there were two — a unit
|
||||
# name that never renders is a journald scrape target matching nothing,
|
||||
# which reads as a quiet unit rather than an absent one.
|
||||
services.hyperhive.swarm.otel.journaldUnits = [
|
||||
"grafana"
|
||||
"swarm-grafana-secret-key"
|
||||
]
|
||||
++ lib.optional ssoLocal "swarm-grafana-oidc-secret"
|
||||
++ lib.optional storeDelivery "swarm-bao-grafana-oidc";
|
||||
"swarm-bao-grafana-oidc"
|
||||
];
|
||||
|
||||
services.hyperhive.swarm.controller.links = [
|
||||
{
|
||||
|
|
@ -423,36 +414,57 @@ in
|
|||
# host that runs authelia, and this whole block is gated on the host that
|
||||
# runs Grafana. ./glue-grafana-oidc-client.nix is where it moved to.
|
||||
|
||||
warnings = lib.optional (ssoConfigured && !ssoLocal && !haveClientIdentity) ''
|
||||
services.hyperhive.swarm.grafana: this swarm has an IdP
|
||||
(services.hyperhive.swarm.authelia.url) but authelia is not on this
|
||||
host and this host holds no swarm-secret-store identity, so nothing
|
||||
can deliver Grafana's OIDC client secret. Grafana will come up with
|
||||
no OIDC login and no local login form, which is no way in at all.
|
||||
|
||||
Name this host's store client leaf in
|
||||
services.hyperhive.deploy.bao.clientCertFile and
|
||||
services.hyperhive.deploy.bao.clientKeyFile — the same identity every
|
||||
other reader of the store uses. See docs/swarm/secrets.md.
|
||||
'';
|
||||
|
||||
# Both arms are what used to be a silent gate, and both fire only where
|
||||
# Grafana runs — this whole block is under `deploy.grafana.enable`. The
|
||||
# binding itself is asserted rather than a copy of its formula, the way
|
||||
# ./swarm-nats.nix's own `autheliaUrl` arm does it: two spellings of one
|
||||
# boolean is two places for a future edit to land in only one.
|
||||
assertions = [
|
||||
{
|
||||
# The URL is what `ssoConfigured` above reads, so with it null this
|
||||
# host mints and delivers a secret for an OIDC block it then does not
|
||||
# emit — SSO silently absent on the one deployment that has every
|
||||
# other piece of it. Only reachable by enabling authelia and clearing
|
||||
# its `url`, which is why it is an assertion and not a fallback.
|
||||
assertion = !ssoLocal || ssoConfigured;
|
||||
# SSO is not optional for this service, and the reason is a hundred
|
||||
# lines below in `auth.disable_login_form = true`: Grafana ships an
|
||||
# `admin`/`admin` account on a public vhost, so the password box is
|
||||
# off whatever the topology. Emitting no OIDC block when the swarm
|
||||
# names no IdP therefore produces a container with no way in at all —
|
||||
# a state no log names, since nothing failed. Failing the build and
|
||||
# naming the option is the only report that reaches anyone.
|
||||
assertion = ssoConfigured;
|
||||
message = ''
|
||||
services.hyperhive.swarm.grafana requires
|
||||
services.hyperhive.swarm.authelia.url when authelia is enabled.
|
||||
services.hyperhive.deploy.grafana.enable requires
|
||||
services.hyperhive.swarm.authelia.url — Grafana's only login is SSO,
|
||||
because its local login form is disabled unconditionally (it ships
|
||||
an admin/admin account and its vhost is on the public gateway).
|
||||
|
||||
Grafana exchanges its authorization code at
|
||||
`''${url}/api/oidc/token` from inside its container, so with the URL
|
||||
null there is no endpoint to name — and a null URL is also how this
|
||||
module reads "no SSO configured for this swarm", which would leave
|
||||
Grafana with no login of any kind on the host that mints its secret.
|
||||
It defaults to this host's own instance only when this host runs
|
||||
authelia. A hive that federates with a swarm sets it explicitly to
|
||||
wherever that provider lives. Grafana exchanges its authorization
|
||||
code at `''${url}/api/oidc/token` from inside its container, so a
|
||||
null URL leaves no endpoint to name.
|
||||
'';
|
||||
}
|
||||
{
|
||||
# The other half of one login: the OIDC block names a `$__file{}` that
|
||||
# `swarm-bao-grafana-oidc.service` below writes, and that unit reads
|
||||
# the store with this host's client certificate. No certificate, no
|
||||
# secret, and the same no-way-in Grafana as the arm above.
|
||||
assertion = haveClientIdentity;
|
||||
message = ''
|
||||
services.hyperhive.deploy.grafana.enable requires this host to hold a
|
||||
swarm-secret-store client identity: set both
|
||||
|
||||
services.hyperhive.deploy.bao.clientCertFile
|
||||
services.hyperhive.deploy.bao.clientKeyFile
|
||||
|
||||
Grafana's OIDC client secret is minted by authelia and read out of
|
||||
the store, on every host that runs Grafana — including the host that
|
||||
runs authelia. That is one delivery route rather than two, and it is
|
||||
what the store is for: this certificate is the single credential
|
||||
placed out of band, and every other secret comes from the store with
|
||||
it.
|
||||
|
||||
On a hive that runs the store, glue-bao-tls.nix supplies both as
|
||||
defaults and there is nothing to do. Elsewhere the leaf is issued
|
||||
from that CA out of band and named here — see docs/swarm/secrets.md.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
|
@ -524,79 +536,24 @@ in
|
|||
"d ${deployCfg.grafana.socketDir} 0750 ${toString grafanaUid} ${toString nginxGid} - -"
|
||||
];
|
||||
|
||||
# The local delivery, and the ONLY thing locality still decides. It runs on
|
||||
# the HOST because that is the only place both container trees are
|
||||
# addressable: they share this host's network namespace, which makes them
|
||||
# feel co-located, but their filesystem roots are separate — Grafana cannot
|
||||
# open a path inside authelia's tree however local the port looks.
|
||||
# THE delivery unit — one route, in every deployment. The secret authelia
|
||||
# minted arrives out of the swarm secret store, which the publisher on
|
||||
# authelia's host wrote it into, and that is true whether authelia is a
|
||||
# network away or in the container next door.
|
||||
#
|
||||
# ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container
|
||||
# refuses to start when a bind source is missing, and this secret does
|
||||
# not exist until authelia's first boot has minted it — so binding it
|
||||
# would make Grafana wait on a file that waits on a container that starts
|
||||
# after it.
|
||||
systemd.services.swarm-grafana-oidc-secret = lib.mkIf ssoLocal {
|
||||
description = "deliver Grafana's OIDC client secret from authelia";
|
||||
after = [ "container@${autheliaCfg.machine}.service" ];
|
||||
requires = [ "container@${autheliaCfg.machine}.service" ];
|
||||
before = [ "container@${cfg.machine}.service" ];
|
||||
wantedBy = [ "container@${cfg.machine}.service" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
SyslogIdentifier = "swarm-grafana-oidc-secret";
|
||||
# ⚠️ Longer than the wait below, and that is the whole point:
|
||||
# `DefaultTimeoutStartSec` is 90s, so a 120s bounded wait is killed
|
||||
# by systemd at 90 — before it can emit the error naming the file it
|
||||
# waited for. The timeout has to outlive the thing it is timing.
|
||||
TimeoutStartSec = "180s";
|
||||
};
|
||||
path = [ pkgs.coreutils ];
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
|
||||
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.oidc.clientId}.secret"}
|
||||
dst=${lib.escapeShellArg hostSecretPath}
|
||||
|
||||
# authelia's container is up, but its first-boot generator may still
|
||||
# be minting. Bounded wait, then fail: a silent skip here produces a
|
||||
# Grafana whose only login path dead-ends.
|
||||
deadline=$(( SECONDS + 120 ))
|
||||
while [ ! -s "$src" ]; do
|
||||
if [ "$SECONDS" -ge "$deadline" ]; then
|
||||
echo "authelia has not minted $src after 120s" >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
# Owned by Grafana's own uid, unlike the matrix sibling which lands
|
||||
# root-owned: tuwunel's secret is read by `LoadCredential` as root
|
||||
# before the sandbox exists, whereas Grafana expands `$__file{}`
|
||||
# itself, as itself, while parsing its config. These containers set
|
||||
# no `privateUsers`, so the host uid is the container uid, and both
|
||||
# sides take it from the same static NixOS id.
|
||||
#
|
||||
# Group is root, not grafana, and that is forced rather than chosen:
|
||||
# `ids.uids.grafana` is a static id but there is no `ids.gids.grafana`
|
||||
# — the group's gid is allocated at activation inside the container,
|
||||
# so the host cannot know it at eval time. Harmless here because 0400
|
||||
# grants the group nothing; if this mode ever widens, the gid has to
|
||||
# be discovered at runtime rather than assumed.
|
||||
install -D -m 0400 -o ${toString config.ids.uids.grafana} -g 0 "$src" "$dst"
|
||||
'';
|
||||
};
|
||||
|
||||
# The same secret, the same destination, for the deployment where authelia
|
||||
# is NOT here: it arrives out of the swarm secret store, which the
|
||||
# publisher on authelia's host wrote it into. Nothing above changes — this
|
||||
# is the second source for one file, never a second file.
|
||||
# 🩸 A second unit here used to copy the plaintext out of authelia's host
|
||||
# tree wherever the two were co-located, with the OIDC block gated on
|
||||
# either route being available. The ruling that deleted it: the store
|
||||
# exists so a host holds ONE out-of-band secret — its client certificate —
|
||||
# and reads everything else with it, so skipping the store when the
|
||||
# producer is local buys a round trip and costs a second delivery path, a
|
||||
# second way for the file to be wrong, and a gate whose false arm was a
|
||||
# Grafana with no login. Recorded in docs/swarm/secrets.md.
|
||||
#
|
||||
# Shaped after ./glue-queue-agent-credential.nix rather than after the copy
|
||||
# above, because it is a store reader and those are the store's readers: a
|
||||
# cert login that fails LOUDLY, since every state it fails on is one a
|
||||
# retry fixes, then a read that degrades QUIETLY, since no retry turns "no
|
||||
# value there" into a value.
|
||||
# Shaped after ./glue-queue-agent-credential.nix, because it is a store
|
||||
# reader and this is what the store's readers do: a cert login that fails
|
||||
# LOUDLY, since every state it fails on is one a retry fixes, then a read
|
||||
# that degrades QUIETLY, since no retry turns "no value there" into a value.
|
||||
#
|
||||
# ⚠️ An ABSENT secret is "not published yet", not a fault of this host's.
|
||||
# The publisher runs on the authelia host and authelia mints on its first
|
||||
|
|
@ -606,11 +563,10 @@ in
|
|||
# log anyone in into one whose token exchange is refused.
|
||||
#
|
||||
# ⇒ So the absence surfaces at Grafana rather than here, and this unit's job
|
||||
# is to leave the journal line that says which of the three it was. That is
|
||||
# the same division the copy unit above makes by failing rather than
|
||||
# skipping: the secret not arriving is a Grafana nobody can log into, and a
|
||||
# named cause is the only thing separating that from a mystery.
|
||||
systemd.services.swarm-bao-grafana-oidc = lib.mkIf storeDelivery {
|
||||
# is to leave the journal line that says which of the three it was: the
|
||||
# secret not arriving is a Grafana nobody can log into, and a named cause is
|
||||
# the only thing separating that from a mystery.
|
||||
systemd.services.swarm-bao-grafana-oidc = {
|
||||
description = "fetch Grafana's OIDC client secret from the swarm secret store";
|
||||
# Every one of these names a unit that exists only where the store runs.
|
||||
# `Requires=` on an absent unit fails the job outright, so the ordering
|
||||
|
|
@ -708,8 +664,19 @@ in
|
|||
# bash's own, so the plaintext never becomes an argument in /proc the
|
||||
# way `install <<<"$secret"` or an `echo` from `path` would.
|
||||
#
|
||||
# Same uid, group and mode as the copy above, for the reasons stated
|
||||
# there — this is the same file arriving by a different route.
|
||||
# Owned by Grafana's own uid, unlike the matrix sibling which lands
|
||||
# root-owned: tuwunel's secret is read by `LoadCredential` as root
|
||||
# before the sandbox exists, whereas Grafana expands `$__file{}`
|
||||
# itself, as itself, while parsing its config. This container sets no
|
||||
# `privateUsers`, so the host uid is the container uid, and both sides
|
||||
# take it from the same static NixOS id.
|
||||
#
|
||||
# Group is root, not grafana, and that is forced rather than chosen:
|
||||
# `ids.uids.grafana` is a static id but there is no `ids.gids.grafana`
|
||||
# — the group's gid is allocated at activation inside the container, so
|
||||
# the host cannot know it at eval time. Harmless here because 0400
|
||||
# grants the group nothing; if this mode ever widens, the gid has to be
|
||||
# discovered at runtime rather than assumed.
|
||||
install -d -m 0755 ${lib.escapeShellArg hostSecretDir}
|
||||
umask 077
|
||||
printf '%s\n' "$secret" > ${lib.escapeShellArg hostSecretPath}
|
||||
|
|
@ -913,13 +880,21 @@ in
|
|||
# public gateway, so SSO is the only way in — not merely the
|
||||
# preferred one.
|
||||
#
|
||||
# Deliberately NOT `ssoLocal`: whether a password box sits on a
|
||||
# public login page cannot depend on which host happens to run
|
||||
# authelia. The OIDC block below now answers the same swarm-wide
|
||||
# question, so the two no longer disagree.
|
||||
# Not conditional on anything, and the block below is not
|
||||
# either, which is what keeps the two from ever disagreeing.
|
||||
# Whether a password box sits on a public login page cannot
|
||||
# depend on which host happens to run authelia — and it used to,
|
||||
# in a module where the OIDC block was gated and this line was
|
||||
# not.
|
||||
auth.disable_login_form = true;
|
||||
}
|
||||
// lib.optionalAttrs ssoWired {
|
||||
|
||||
# Emitted in every deployment, because SSO is a requirement of
|
||||
# running this service rather than a feature of some topologies:
|
||||
# the assertions above refuse to build a Grafana that lacks an
|
||||
# IdP to point at or the identity to fetch its secret with. A
|
||||
# `lib.optionalAttrs` here would be the silent half of that pair,
|
||||
# dropping the only login this container has whenever a value
|
||||
# went null.
|
||||
"auth.generic_oauth" = {
|
||||
enabled = true;
|
||||
name = "HyperHive";
|
||||
|
|
@ -930,6 +905,9 @@ in
|
|||
# `database.password`, `security.admin_password` and
|
||||
# datasource `secureJsonData`), so nothing but this comment
|
||||
# stands between a literal and the store.
|
||||
#
|
||||
# `swarm-bao-grafana-oidc.service` above is what writes this
|
||||
# path, on every host that runs Grafana.
|
||||
client_secret = "$__file{${secretPath}}";
|
||||
scopes = "openid profile email groups";
|
||||
auth_url = "${autheliaUrl}/api/oidc/authorization";
|
||||
|
|
|
|||
|
|
@ -1,6 +1,13 @@
|
|||
# The unit that copies authelia's minted OIDC client secrets into the swarm's
|
||||
# secret store, so whoever needs one without hosting authelia can read it: a
|
||||
# hive its agents' credential, a swarm service its own.
|
||||
# secret store, so whoever needs one can read it there: a hive its agents'
|
||||
# credential, a swarm service its own.
|
||||
#
|
||||
# ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
|
||||
# into the store even when the service runs beside authelia, because its
|
||||
# reader fetches it from the store in every deployment — ./swarm-grafana.nix
|
||||
# states the ruling that made that the only route. Publishing "only when the
|
||||
# reader is elsewhere" would be a second shape of this unit, gated on a fact
|
||||
# about another host, to save a round trip on the one host that can afford it.
|
||||
#
|
||||
# ⚠️ IT RUNS WHERE AUTHELIA DOES, and that is the whole reason it exists as a
|
||||
# separate thing. `deploy.authelia.hostClientSecretDir`'s own description says
|
||||
|
|
@ -70,7 +77,8 @@ in
|
|||
description = ''
|
||||
Publish the OIDC client secrets this host mints into the swarm's
|
||||
secret store, so a hive that does not run authelia can read its
|
||||
agents' credential and a swarm service elsewhere can read its own.
|
||||
agents' credential and a swarm service can read its own — from
|
||||
wherever it runs, this host included.
|
||||
|
||||
Defaults to whether this host mints them, which is the only half of
|
||||
the question that is a property of *this* host.
|
||||
|
|
@ -82,10 +90,11 @@ in
|
|||
publishing nothing. Whether the wiring is complete is the client
|
||||
identity's job (see `baoClientCertFile`), not this option's.
|
||||
|
||||
Turning it off leaves every non-co-located hive without a delivery
|
||||
path, which is the state this exists to end — so the honest reason
|
||||
to set it false is a deployment delivering those secrets by some
|
||||
other mechanism it owns.
|
||||
Turning it off leaves every hive but this one without its agents'
|
||||
credential, and the swarm's Grafana without any login at all — its
|
||||
secret has exactly one route and this is the producer's end of it. So
|
||||
the honest reason to set it false is a deployment delivering those
|
||||
secrets by some other mechanism it owns.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -223,18 +223,22 @@ let
|
|||
swarm.grafana.package = pkgs.emptyDirectory;
|
||||
};
|
||||
|
||||
# The metrics UI beside the IdP, which is the deployment whose secret was
|
||||
# already delivered: a host copy out of authelia's own tree.
|
||||
# The metrics UI beside the IdP. It reads its secret out of the store like
|
||||
# every other Grafana host, so it needs a store identity like every other
|
||||
# Grafana host — the cert pair here is not scenery, it is the arm that would
|
||||
# have caught the deleted co-located copy unit coming back.
|
||||
grafanaWithAuthelia = hive {
|
||||
deploy.grafana.enable = true;
|
||||
deploy.grafana.plugins = [ ];
|
||||
deploy.grafana.package = pkgs.emptyDirectory;
|
||||
deploy.authelia.enable = true;
|
||||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||
};
|
||||
# The same UI with the IdP on ANOTHER host and a store leaf placed by hand —
|
||||
# the deployment that had no delivery path at all. Knowing an IdP is not
|
||||
# running one: `swarm.authelia.url` is what says this swarm has SSO, and
|
||||
# nothing about this host does.
|
||||
# The same UI with the IdP on ANOTHER host and a store leaf placed by hand.
|
||||
# Knowing an IdP is not running one: `swarm.authelia.url` is what says this
|
||||
# swarm has SSO, and nothing about this host does. Identical to the fixture
|
||||
# above in everything the delivery path reads, which is the point.
|
||||
grafanaRemoteAuthelia = hive {
|
||||
deploy.grafana.enable = true;
|
||||
deploy.grafana.plugins = [ ];
|
||||
|
|
@ -243,16 +247,48 @@ let
|
|||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||
};
|
||||
# The same again with the store identity taken away, which is the one shape
|
||||
# that can deliver this secret by NO route. It separates "the swarm has an
|
||||
# IdP" from "something here can fetch what it minted", and it is what keeps
|
||||
# Grafana's config from naming a file nothing writes.
|
||||
grafanaRemoteAutheliaNoIdentity = hive {
|
||||
# A Grafana host holding no store identity. This used to be the shape the
|
||||
# module went QUIET on — no OIDC block, a warning, and a container whose
|
||||
# login form is off regardless, so no way in and nothing failed. It is kept
|
||||
# rather than deleted because the shape is still reachable by an operator;
|
||||
# what changed is the deliverable, from a warning nothing reads back to a
|
||||
# refusal naming the two options to set. Only the identity is missing, so an
|
||||
# arm below can name which refusal fired.
|
||||
grafanaNoIdentity = hive {
|
||||
deploy.grafana.enable = true;
|
||||
deploy.grafana.plugins = [ ];
|
||||
deploy.grafana.package = pkgs.emptyDirectory;
|
||||
swarm.authelia.url = "https://auth.example.invalid";
|
||||
};
|
||||
# The mirror image: the identity is placed, and the swarm names no IdP. The
|
||||
# other half of "SSO must always be configured", and isolated the same way —
|
||||
# exactly one thing wrong, so the arm reads one refusal.
|
||||
grafanaNoSso = hive {
|
||||
deploy.grafana.enable = true;
|
||||
deploy.grafana.plugins = [ ];
|
||||
deploy.grafana.package = pkgs.emptyDirectory;
|
||||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||
};
|
||||
|
||||
# Did ./host-modules/swarm-grafana.nix refuse this host, and for which of its
|
||||
# two reasons. An assertion is a config VALUE until something forces it —
|
||||
# `.config` never throws — so a fixture in a state the module refuses is
|
||||
# evaluable and the refusal is readable as data. That is what lets a case
|
||||
# check that a misconfiguration is REPORTED, rather than only that it is not
|
||||
# silently accepted.
|
||||
#
|
||||
# Matched on the option name the message names, not on its prose, so the
|
||||
# wording stays rewordable: the option name is the part an operator has to
|
||||
# act on, and a message that stopped naming it would be the actual defect.
|
||||
grafanaRefusedFor =
|
||||
m: option:
|
||||
lib.any (
|
||||
a:
|
||||
!a.assertion
|
||||
&& lib.hasInfix "services.hyperhive.deploy.grafana.enable requires" a.message
|
||||
&& lib.hasInfix option a.message
|
||||
) m.assertions;
|
||||
|
||||
# The first slice to leave options on BOTH sides of the split, so the
|
||||
# fixture sets all three of them through the paths an existing config uses:
|
||||
|
|
@ -871,55 +907,80 @@ let
|
|||
# The absence class this whole file is for, and the reported defect in one
|
||||
# arm: the OIDC block hung off "authelia is on this host", so the split
|
||||
# deployment got a Grafana with no SSO settings and no login form — no way
|
||||
# in at all. The question it asks now is whether the SWARM has an IdP.
|
||||
name = "grafana is wired for SSO against an IdP on another host";
|
||||
# in at all. The block is emitted in every deployment now, so the negative
|
||||
# arm is not "no block elsewhere" but "the two do not name the same IdP":
|
||||
# each host's block has to point at the URL the SWARM names, and a block
|
||||
# built from `deploy.authelia` rather than `swarm.authelia.url` would pass
|
||||
# a presence check on both fixtures while sending one of them nowhere.
|
||||
name = "grafana's OIDC block names the swarm's IdP, wherever that IdP runs";
|
||||
ok =
|
||||
let
|
||||
s =
|
||||
grafanaRemoteAuthelia.containers.swarm-grafana.config.services.grafana.settings."auth.generic_oauth";
|
||||
oauth = m: m.containers.swarm-grafana.config.services.grafana.settings."auth.generic_oauth";
|
||||
remote = oauth grafanaRemoteAuthelia;
|
||||
local = oauth grafanaWithAuthelia;
|
||||
in
|
||||
s.enabled && lib.hasInfix "https://auth.example.invalid/api/oidc/token" s.token_url;
|
||||
remote.enabled
|
||||
&& lib.hasInfix "https://auth.example.invalid/api/oidc/token" remote.token_url
|
||||
&& local.enabled
|
||||
&& lib.hasInfix "https://auth.t.local/api/oidc/token" local.token_url
|
||||
&& !(lib.hasInfix "auth.example.invalid" local.token_url);
|
||||
}
|
||||
{
|
||||
# The control for the arm above, and the reason the gate is a conjunction:
|
||||
# a block naming a file no unit on this host writes cannot produce a
|
||||
# working login, so it is emitted only where a delivery route exists —
|
||||
# leaving a host with neither exactly as it was rather than pointing
|
||||
# Grafana's config at a secret that never arrives.
|
||||
name = "grafana emits no OIDC block when nothing on this host can deliver the secret";
|
||||
ok =
|
||||
!(
|
||||
grafanaRemoteAutheliaNoIdentity.containers.swarm-grafana.config.services.grafana.settings
|
||||
? "auth.generic_oauth"
|
||||
);
|
||||
}
|
||||
{
|
||||
# The other half of the arm above, and the only deliverable that shape
|
||||
# has: silence there is a Grafana nobody can log into for a reason no log
|
||||
# names. A warning is read back by nothing, so an unevaluated one is a
|
||||
# warning nobody sees — the same reason the peers fixture exists.
|
||||
name = "the undeliverable-secret shape warns instead of going quiet";
|
||||
ok =
|
||||
lib.any (
|
||||
w: lib.hasInfix "no swarm-secret-store identity" w
|
||||
) grafanaRemoteAutheliaNoIdentity.warnings
|
||||
&& !(lib.any (w: lib.hasInfix "no swarm-secret-store identity" w) grafanaRemoteAuthelia.warnings);
|
||||
}
|
||||
{
|
||||
# Exactly one delivery unit per deployment, and which one is the only
|
||||
# thing locality decides. Two units writing one path is a race over which
|
||||
# secret Grafana parses; zero is the defect this pair replaced.
|
||||
name = "grafana's OIDC secret has one delivery unit, chosen by where authelia is";
|
||||
# 🩸 The arm that guards the ruling this slice landed under. There is ONE
|
||||
# delivery route: the store reader, on every host that runs Grafana. The
|
||||
# negative names the deleted unit rather than a generic absence, because
|
||||
# the way this regresses is someone re-adding the co-located copy as an
|
||||
# optimisation — a second writer of one path, and a second shape of "the
|
||||
# secret is wrong" to debug.
|
||||
name = "grafana's OIDC secret has exactly one delivery unit, the store reader, in both topologies";
|
||||
ok =
|
||||
let
|
||||
local = grafanaWithAuthelia.systemd.services;
|
||||
remote = grafanaRemoteAuthelia.systemd.services;
|
||||
in
|
||||
local ? swarm-grafana-oidc-secret
|
||||
&& !(local ? swarm-bao-grafana-oidc)
|
||||
local ? swarm-bao-grafana-oidc
|
||||
&& remote ? swarm-bao-grafana-oidc
|
||||
&& !(local ? swarm-grafana-oidc-secret)
|
||||
&& !(remote ? swarm-grafana-oidc-secret);
|
||||
}
|
||||
{
|
||||
# What the deleted warning became. The shape is unchanged — a Grafana host
|
||||
# holding no store leaf — but silence there is a container nobody can log
|
||||
# into for a reason no log names, and a warning is read back by nothing.
|
||||
# The second arm is what makes this a refusal about the IDENTITY: this
|
||||
# fixture names an IdP, so a message about `swarm.authelia.url` here would
|
||||
# mean the two assertions had been collapsed into one conjunction.
|
||||
name = "a grafana host with no store identity is refused, naming the options to set";
|
||||
ok =
|
||||
grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientCertFile"
|
||||
&& grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientKeyFile"
|
||||
&& !(grafanaRefusedFor grafanaNoIdentity "swarm.authelia.url");
|
||||
}
|
||||
{
|
||||
# "SSO must always be configured", as an eval-time refusal rather than a
|
||||
# gate. A null URL used to drop the OIDC block silently, and
|
||||
# `disable_login_form` is unconditional a hundred lines below it, so that
|
||||
# combination produced a Grafana with no SSO and no password box — an
|
||||
# outage whose cause is a boolean that evaluated to false at build time
|
||||
# and left no trace. Same isolation as the arm above, mirrored.
|
||||
name = "a grafana host in a swarm with no IdP is refused, naming swarm.authelia.url";
|
||||
ok =
|
||||
grafanaRefusedFor grafanaNoSso "services.hyperhive.swarm.authelia.url"
|
||||
&& !(grafanaRefusedFor grafanaNoSso "deploy.bao.clientCertFile");
|
||||
}
|
||||
{
|
||||
# Without this the two arms above prove nothing: a refusal that fires on
|
||||
# every host is not a check, and both of these are hosts a swarm is
|
||||
# expected to have. Read through the same helper, so a message that
|
||||
# stopped naming its option would fail the arms above rather than pass
|
||||
# this one by accident.
|
||||
name = "neither grafana refusal fires on a correctly configured host, co-located or not";
|
||||
ok =
|
||||
!(grafanaRefusedFor grafanaWithAuthelia "services.hyperhive.swarm.authelia.url")
|
||||
&& !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.clientCertFile")
|
||||
&& !(grafanaRefusedFor grafanaRemoteAuthelia "services.hyperhive.swarm.authelia.url")
|
||||
&& !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.clientCertFile");
|
||||
}
|
||||
{
|
||||
# Same 403-not-a-miss reason as the matrix and queue arms below: the
|
||||
# reader's grant covers the `services` prefix, so a path outside it is
|
||||
|
|
|
|||
Loading…
Reference in a new issue