Compare commits
7 changed files with 27 additions and 519 deletions
|
|
@ -186,31 +186,6 @@ redefines it — the agent-facing value is *derived*
|
||||||
keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
|
keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
|
||||||
automatically; there is nothing to open by hand.
|
automatically; there is nothing to open by hand.
|
||||||
|
|
||||||
### Authenticated ingest
|
|
||||||
|
|
||||||
The swarm tier gives **each hive its own receiver**, and stamps the `hive` label
|
|
||||||
from whichever receiver accepted a sample. A hive therefore cannot report
|
|
||||||
metrics as another hive, and cannot relabel its own by editing what it sends —
|
|
||||||
the label is not taken from the payload at all.
|
|
||||||
|
|
||||||
**On an all-local swarm there is nothing to set.** Each hive already has an
|
|
||||||
identity, and its collector reads the secret that host's own authelia minted.
|
|
||||||
|
|
||||||
**On a hive that does not host the swarm's services**, the secret has to arrive
|
|
||||||
somehow — copy it across and name it:
|
|
||||||
|
|
||||||
```nix
|
|
||||||
services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret";
|
|
||||||
```
|
|
||||||
|
|
||||||
**There is no unauthenticated mode.** A hive always presents an identity, so a
|
|
||||||
missing credential is a build error rather than a quieter fallback — the
|
|
||||||
collector has no anonymous route to accept samples on, and every path it serves
|
|
||||||
belongs to exactly one hive.
|
|
||||||
|
|
||||||
Getting the secret wrong shows up as the hive's collector logging 401s from the
|
|
||||||
swarm tier and no metrics appearing for that hive.
|
|
||||||
|
|
||||||
### `services.hyperhive.otel.collector.port` — port, default `4318`
|
### `services.hyperhive.otel.collector.port` — port, default `4318`
|
||||||
|
|
||||||
The OTLP/HTTP port the hive tier listens on, bound to the bridge IP only. The
|
The OTLP/HTTP port the hive tier listens on, bound to the bridge IP only. The
|
||||||
|
|
|
||||||
|
|
@ -156,11 +156,6 @@ See [`tools/hivectl.md`](tools/hivectl.md) for every `hivectl` verb.
|
||||||
- All config changes (forge PRs on `agent-configs/<name>`) go through
|
- All config changes (forge PRs on `agent-configs/<name>`) go through
|
||||||
operator approval — agents can't unilaterally rebuild containers, by design.
|
operator approval — agents can't unilaterally rebuild containers, by design.
|
||||||
See [`boundary.md`](boundary.md) and [`security.md`](security.md).
|
See [`boundary.md`](boundary.md) and [`security.md`](security.md).
|
||||||
- **Telemetry ingest is authenticated per hive**, and the `hive` label comes
|
|
||||||
from which hive authenticated rather than from the payload — so no hive can
|
|
||||||
report metrics as another. A first-run all-local hive gets this with nothing
|
|
||||||
to configure; joining a swarm you don't host needs one secret copied across.
|
|
||||||
See [`observability.md`](observability.md#authenticated-ingest).
|
|
||||||
|
|
||||||
Once the hive is running, ruth records anything it needs to remember
|
Once the hive is running, ruth records anything it needs to remember
|
||||||
across restarts in `/agents/ruth/state/notes.md`.
|
across restarts in `/agents/ruth/state/notes.md`.
|
||||||
|
|
|
||||||
|
|
@ -45,7 +45,7 @@ Every row below is read against one of these.
|
||||||
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
||||||
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
||||||
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
||||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in the service's `sso.clientSecretFile` |
|
||||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||||
| wireguard private key | **the operator** — `wg genkey` | whatever `swarm.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
| wireguard private key | **the operator** — `wg genkey` | whatever `swarm.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||||
|
|
@ -56,13 +56,6 @@ because nothing outside that container ever reads them. **That is the test worth
|
||||||
applying to any secret added here** — and the client secret's plaintext half is
|
applying to any secret added here** — and the client secret's plaintext half is
|
||||||
the one row that fails it, which is the entire reason a delivery step exists.
|
the one row that fails it, which is the entire reason a delivery step exists.
|
||||||
|
|
||||||
One reader needs no delivery step: the **hive's telemetry collector**, which
|
|
||||||
authenticates to the swarm's collector as its own hive. It is a host unit rather
|
|
||||||
than a container, so on an all-local swarm it reads authelia's file where it
|
|
||||||
lies (through `LoadCredential`) and no second copy is made. On any other
|
|
||||||
topology it is an ordinary "operator provides the file" case — see
|
|
||||||
`services.hyperhive.otel.clientSecretFile`.
|
|
||||||
|
|
||||||
### Minting the queue's callout nkeys
|
### Minting the queue's callout nkeys
|
||||||
|
|
||||||
`nats.autoGenerateCallout` mints both keypairs on the host before the queue
|
`nats.autoGenerateCallout` mints both keypairs on the host before the queue
|
||||||
|
|
|
||||||
|
|
@ -39,26 +39,10 @@ let
|
||||||
# the bundle next to the CA and explains the split.
|
# the bundle next to the CA and explains the split.
|
||||||
caHostPath = "${tlsCfg.stateDir}/trust-bundle.pem";
|
caHostPath = "${tlsCfg.stateDir}/trust-bundle.pem";
|
||||||
caContainerPath = "/run/hive-ca/trust-bundle.pem";
|
caContainerPath = "/run/hive-ca/trust-bundle.pem";
|
||||||
|
|
||||||
# One definition, used by `trustBundle` to WRITE the bundle and published
|
|
||||||
# below so a caller can NAME it. Two copies of this path would be two
|
|
||||||
# things to keep in step, and the one that drifts is the reader.
|
|
||||||
bundleDirFor = name: "/run/${name}-ca";
|
|
||||||
bundlePathFor = name: "${bundleDirFor name}/trust-bundle.pem";
|
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
inherit useSelfSigned caContainerPath;
|
inherit useSelfSigned caContainerPath;
|
||||||
|
|
||||||
# Where `trustBundle` below puts the assembled bundle, for the callers
|
|
||||||
# that must NAME it rather than just have it exported. `SSL_CERT_FILE` is
|
|
||||||
# set for you and needs no path here; a consumer that takes its own CA
|
|
||||||
# argument (an OIDC verifier's `issuer_ca_path`, a client's `--cacert`)
|
|
||||||
# does, and the alternative is copying `/run/<name>-ca/…` to the call
|
|
||||||
# site. That copy breaks silently: the bundle keeps being written, the
|
|
||||||
# consumer keeps reading a path that no longer exists, and the failure
|
|
||||||
# surfaces as a TLS error naming the peer rather than the file.
|
|
||||||
inherit bundlePathFor;
|
|
||||||
|
|
||||||
# Fold into the container's `bindMounts` via `//`. Binds ONLY the public
|
# Fold into the container's `bindMounts` via `//`. Binds ONLY the public
|
||||||
# CA cert (never the `hive-tls` state dir — it holds the CA + leaf private
|
# CA cert (never the `hive-tls` state dir — it holds the CA + leaf private
|
||||||
# keys), read-only. Empty when not self-signed, so the whole trust path
|
# keys), read-only. Empty when not self-signed, so the whole trust path
|
||||||
|
|
@ -122,8 +106,8 @@ in
|
||||||
enable ? true,
|
enable ? true,
|
||||||
}:
|
}:
|
||||||
let
|
let
|
||||||
dir = bundleDirFor name;
|
dir = "/run/${name}-ca";
|
||||||
bundlePath = bundlePathFor name;
|
bundlePath = "${dir}/trust-bundle.pem";
|
||||||
unit = "${name}-ca-bundle";
|
unit = "${name}-ca-bundle";
|
||||||
source = if hostUnit then caHostPath else caContainerPath;
|
source = if hostUnit then caHostPath else caContainerPath;
|
||||||
in
|
in
|
||||||
|
|
|
||||||
|
|
@ -193,40 +193,6 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
clientSecretFile = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.str;
|
|
||||||
default =
|
|
||||||
if
|
|
||||||
config.services.hyperhive.swarm.authelia.enable && config.services.hyperhive.hiveName != null
|
|
||||||
then
|
|
||||||
"${config.services.hyperhive.swarm.authelia.hostClientSecretDir}/"
|
|
||||||
+ "${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${config.services.hyperhive.hiveName}.secret"
|
|
||||||
else
|
|
||||||
null;
|
|
||||||
defaultText = lib.literalExpression ''"''${swarm.authelia.hostClientSecretDir}/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when this host runs the swarm's IdP, else null'';
|
|
||||||
example = "/var/lib/secrets/hive-telemetry.secret";
|
|
||||||
description = ''
|
|
||||||
Absolute path to this hive's OAuth2 client secret, used to
|
|
||||||
authenticate to the swarm's collector as this hive.
|
|
||||||
|
|
||||||
**Whether this is set is what decides whether this tier
|
|
||||||
authenticates at all.** A credential is the only thing that makes
|
|
||||||
authenticated egress possible, so its presence is the condition
|
|
||||||
rather than a second switch that could disagree with it.
|
|
||||||
|
|
||||||
Defaults to the secret this host's own authelia minted, which is
|
|
||||||
correct exactly when the swarm's IdP runs here. On a hive that is
|
|
||||||
not that host, the file has to arrive some other way and this
|
|
||||||
option names wherever it landed — the same manual-copy shape
|
|
||||||
`services.hyperhive.swarm.authelia.oidc.hiveIdentities` documents,
|
|
||||||
where delivering a secret to a hive that is not this host is
|
|
||||||
deliberately not solved.
|
|
||||||
|
|
||||||
Read by `LoadCredential`, so it is never evaluated by nix, never
|
|
||||||
copied into the store and never passed in argv.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
metricIntervalMs = lib.mkOption {
|
metricIntervalMs = lib.mkOption {
|
||||||
type = lib.types.nullOr lib.types.ints.positive;
|
type = lib.types.nullOr lib.types.ints.positive;
|
||||||
default = null;
|
default = null;
|
||||||
|
|
@ -248,32 +214,8 @@ in
|
||||||
config = lib.mkIf config.services.hyperhive.otel.enable (
|
config = lib.mkIf config.services.hyperhive.otel.enable (
|
||||||
let
|
let
|
||||||
otel = config.services.hyperhive.otel;
|
otel = config.services.hyperhive.otel;
|
||||||
autheliaCfg = config.services.hyperhive.swarm.authelia;
|
|
||||||
swarmOtelCfg = config.services.hyperhive.swarm.otel;
|
|
||||||
hiveName = config.services.hyperhive.hiveName;
|
|
||||||
listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
|
listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
|
||||||
swarmName = "otlphttp/swarm";
|
swarmName = "otlphttp/swarm";
|
||||||
authName = "oauth2client/swarm";
|
|
||||||
|
|
||||||
# A hive always authenticates to the swarm's collector as itself, so
|
|
||||||
# this is not a mode — it is a precondition, and the assertion below
|
|
||||||
# is what enforces it. Kept as a name because several places have to
|
|
||||||
# read "do we have what it takes", and an eval error from a null path
|
|
||||||
# names this file rather than the option an operator has to set.
|
|
||||||
senderAuth = otel.clientSecretFile != null && hiveName != null;
|
|
||||||
|
|
||||||
# This hive's client id, and also the audience it must ASK for. Both
|
|
||||||
# are `hiveClientPrefix` + the hive's name because that is the one
|
|
||||||
# name the swarm already agrees on; the receiver one tier up derives
|
|
||||||
# the same string.
|
|
||||||
hiveClient = "${autheliaCfg.hiveClientPrefix}${toString hiveName}";
|
|
||||||
|
|
||||||
# systemd exports `CREDENTIALS_DIRECTORY` to any unit with
|
|
||||||
# `LoadCredential`, and the collector expands `${env:…}` at load. So
|
|
||||||
# the secret reaches the process as a PATH resolved at runtime — nix
|
|
||||||
# renders neither the value nor the directory, and nothing has to
|
|
||||||
# hardcode `/run/credentials/<unit>`.
|
|
||||||
credName = "swarm-client.secret";
|
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
# Reachable from agent containers and nowhere else: this opens
|
# Reachable from agent containers and nowhere else: this opens
|
||||||
|
|
@ -282,22 +224,6 @@ in
|
||||||
|
|
||||||
services.opentelemetry-collector = {
|
services.opentelemetry-collector = {
|
||||||
enable = true;
|
enable = true;
|
||||||
# Contrib, matching the swarm tier (./swarm-otel.nix). The upstream
|
|
||||||
# default build has no auth extensions at all, and this tier has to
|
|
||||||
# *present* a credential to the swarm tier — `oauth2client` lives
|
|
||||||
# only in contrib, so the package choice is what makes authenticated
|
|
||||||
# egress expressible rather than a preference.
|
|
||||||
#
|
|
||||||
# Not a build-farm cost: contrib is fetched, not compiled.
|
|
||||||
#
|
|
||||||
# ⚠️ Read the note directly below before adding any extension here.
|
|
||||||
# It describes precisely the trap this package unlocks: naming an
|
|
||||||
# extension the build lacks passes `validate` and then kills the
|
|
||||||
# collector at startup. With contrib the extensions exist — but the
|
|
||||||
# gap it warns about (a green build proving nothing about whether
|
|
||||||
# the process starts) is exactly why this module's auth wiring is
|
|
||||||
# gated by a probe that runs both collectors, not by eval.
|
|
||||||
package = pkgs.opentelemetry-collector-contrib;
|
|
||||||
# `validateConfigFile` defaults to `isStorePath configFile`,
|
# `validateConfigFile` defaults to `isStorePath configFile`,
|
||||||
# and `configFile` is null on the `settings` path — so the
|
# and `configFile` is null on the `settings` path — so the
|
||||||
# upstream default is OFF for exactly the way this module
|
# upstream default is OFF for exactly the way this module
|
||||||
|
|
@ -337,80 +263,15 @@ in
|
||||||
# change. `https://` because that name resolves through the
|
# change. `https://` because that name resolves through the
|
||||||
# gateway even on a co-located host — see `caTrust` above for
|
# gateway even on a co-located host — see `caTrust` above for
|
||||||
# the trust half that makes this verify.
|
# the trust half that makes this verify.
|
||||||
# The hive's own path under the collector's single name. The
|
endpoint = "https://${config.services.hyperhive.swarm.otel.domain}";
|
||||||
# swarm tier gives each hive its own authenticated receiver and
|
};
|
||||||
# routes to it by this prefix, so the path is not decoration —
|
|
||||||
# it selects WHICH receiver, and therefore which hive the
|
|
||||||
# samples get labelled as.
|
|
||||||
endpoint = "https://${swarmOtelCfg.domain}" + lib.optionalString senderAuth "/${toString hiveName}";
|
|
||||||
}
|
|
||||||
// lib.optionalAttrs senderAuth { auth.authenticator = authName; };
|
|
||||||
|
|
||||||
# ⚠️ An extension not listed here is INERT: the collector starts
|
|
||||||
# clean and the exporter naming it sends nothing authenticated.
|
|
||||||
service.extensions = lib.optional senderAuth authName;
|
|
||||||
|
|
||||||
service.pipelines.metrics = {
|
service.pipelines.metrics = {
|
||||||
receivers = [ "otlp" ];
|
receivers = [ "otlp" ];
|
||||||
exporters = [ swarmName ];
|
exporters = [ swarmName ];
|
||||||
};
|
};
|
||||||
}
|
|
||||||
// lib.optionalAttrs senderAuth {
|
|
||||||
extensions.${authName} = {
|
|
||||||
client_id = hiveClient;
|
|
||||||
# A real key, measured against this collector version rather
|
|
||||||
# than assumed — with a deliberate typo rejected in the same
|
|
||||||
# run, so "accepted" is distinguishable from "ignores
|
|
||||||
# everything". Keeps the secret out of nix entirely: the
|
|
||||||
# collector opens the file itself.
|
|
||||||
client_secret_file = "\${env:CREDENTIALS_DIRECTORY}/${credName}";
|
|
||||||
token_url = "${toString autheliaCfg.url}/api/oidc/token";
|
|
||||||
# ⚠️ THE AUDIENCE HAS TO BE REQUESTED, not merely granted.
|
|
||||||
# Registering it on the client only makes it permissible; a
|
|
||||||
# token minted without asking carries `aud: []` and every
|
|
||||||
# receiver refuses it — with a config that reads perfectly at
|
|
||||||
# both ends. Measured against authelia 4.39.20.
|
|
||||||
endpoint_params.audience = hiveClient;
|
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
};
|
|
||||||
|
|
||||||
# `LoadCredential` and not a copy-oneshot: this collector is a HOST
|
|
||||||
# unit, so there is no container boundary to cross and therefore no
|
|
||||||
# reason for a second on-disk copy of the secret. systemd hands it to
|
|
||||||
# the process in a private tmpfs and exports the directory, which is
|
|
||||||
# what the config above names.
|
|
||||||
systemd.services.opentelemetry-collector.serviceConfig = lib.optionalAttrs senderAuth {
|
|
||||||
LoadCredential = [ "${credName}:${otel.clientSecretFile}" ];
|
|
||||||
};
|
|
||||||
|
|
||||||
assertions = [
|
|
||||||
{
|
|
||||||
# A hive authenticates to the swarm's collector as itself — there
|
|
||||||
# is no unauthenticated path to fall back to, so a missing
|
|
||||||
# credential is a broken deployment rather than a quieter mode.
|
|
||||||
# Caught here because the alternative is a collector that starts
|
|
||||||
# cleanly, retries forever, and reports nothing to anyone.
|
|
||||||
assertion = senderAuth;
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.otel.enable is true but this hive has no
|
|
||||||
identity to present to the swarm's collector:
|
|
||||||
|
|
||||||
services.hyperhive.otel.clientSecretFile = ${
|
|
||||||
if otel.clientSecretFile == null then "null" else otel.clientSecretFile
|
|
||||||
}
|
|
||||||
services.hyperhive.hiveName = ${if hiveName == null then "null" else hiveName}
|
|
||||||
|
|
||||||
Every hive authenticates as itself — that is what makes the
|
|
||||||
`hive` label on its metrics mean anything — so both are
|
|
||||||
required.
|
|
||||||
|
|
||||||
On a host that runs the swarm's identity provider, the default
|
|
||||||
already points at the secret authelia minted. On a hive that
|
|
||||||
does not, copy that hive's secret across and name it here.
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
];
|
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -102,26 +102,11 @@ let
|
||||||
# and a field later added to the submodule then existed on the
|
# and a field later added to the submodule then existed on the
|
||||||
# declared entries and not on these, which is an eval error reachable
|
# declared entries and not on these, which is an eval error reachable
|
||||||
# only once hive identities are on.
|
# only once hive identities are on.
|
||||||
#
|
|
||||||
# `audience` is the hive's own client id rather than a second per-hive
|
|
||||||
# string invented here. A swarm service that has to tell hives apart
|
|
||||||
# needs one name per hive that both sides already agree on, and the
|
|
||||||
# client id is that name — published as `hiveClientPrefix` for exactly
|
|
||||||
# this reason. Minting a parallel naming scheme would be a second thing
|
|
||||||
# to keep in step, and the one that drifts is the one nobody tests.
|
|
||||||
#
|
|
||||||
# `RS256` because a resource server that cannot call this provider back
|
|
||||||
# is a real case here: the swarm's telemetry collector verifies tokens
|
|
||||||
# offline against `/jwks.json`, and an opaque token gives it nothing to
|
|
||||||
# verify. The queue's auth-callout responder introspects instead, which
|
|
||||||
# is a different question asked of the same token.
|
|
||||||
hiveClients = lib.mapAttrsToList (name: _: {
|
hiveClients = lib.mapAttrsToList (name: _: {
|
||||||
id = "${cfg.hiveClientPrefix}${name}";
|
id = "${cfg.hiveClientPrefix}${name}";
|
||||||
description = "HyperHive hive ${name}";
|
description = "HyperHive hive ${name}";
|
||||||
kind = "machine";
|
kind = "machine";
|
||||||
redirectUris = [ ];
|
redirectUris = [ ];
|
||||||
audience = [ "${cfg.hiveClientPrefix}${name}" ];
|
|
||||||
accessTokenSignedResponseAlg = "RS256";
|
|
||||||
}) hyperhiveCfg.swarm.hives;
|
}) hyperhiveCfg.swarm.hives;
|
||||||
|
|
||||||
# `swarm-authelia-bridge`'s own identity — distinct from
|
# `swarm-authelia-bridge`'s own identity — distinct from
|
||||||
|
|
@ -203,15 +188,6 @@ let
|
||||||
+ lib.optionalString (c.tokenEndpointAuthMethod != null) ''
|
+ lib.optionalString (c.tokenEndpointAuthMethod != null) ''
|
||||||
printf -- ' token_endpoint_auth_method: %s\n' ${lib.escapeShellArg c.tokenEndpointAuthMethod}
|
printf -- ' token_endpoint_auth_method: %s\n' ${lib.escapeShellArg c.tokenEndpointAuthMethod}
|
||||||
''
|
''
|
||||||
# Flow-style YAML, matching `scopes` below. The values are client ids
|
|
||||||
# and hive names, which `Ident` already constrains to `[a-z0-9-]` — no
|
|
||||||
# character in that set needs quoting in a YAML flow sequence.
|
|
||||||
+ lib.optionalString (c.audience != [ ]) ''
|
|
||||||
printf -- ' audience: [%s]\n' ${lib.escapeShellArg (lib.concatStringsSep ", " c.audience)}
|
|
||||||
''
|
|
||||||
+ lib.optionalString (c.accessTokenSignedResponseAlg != null) ''
|
|
||||||
printf -- ' access_token_signed_response_alg: %s\n' ${lib.escapeShellArg c.accessTokenSignedResponseAlg}
|
|
||||||
''
|
|
||||||
+ (
|
+ (
|
||||||
if c.kind == "machine" then
|
if c.kind == "machine" then
|
||||||
''
|
''
|
||||||
|
|
@ -517,63 +493,6 @@ in
|
||||||
the secret is at fault.
|
the secret is at fault.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
audience = lib.mkOption {
|
|
||||||
type = lib.types.listOf lib.types.str;
|
|
||||||
default = [ ];
|
|
||||||
example = [ "hive-alpha" ];
|
|
||||||
description = ''
|
|
||||||
Audiences (`aud`) this client is permitted to request a
|
|
||||||
token for. Empty means it asks for none, which is the
|
|
||||||
right answer for a client whose resource server does not
|
|
||||||
distinguish callers.
|
|
||||||
|
|
||||||
⚠️ Registering an audience only *permits* it — the value
|
|
||||||
lands in a token when the client **asks** for it at the
|
|
||||||
token endpoint, and a client that does not send
|
|
||||||
`audience=` receives a token with `aud: []` however
|
|
||||||
complete this list looks. Measured against authelia
|
|
||||||
4.39.20: the config reads exactly right and the resource
|
|
||||||
server rejects every token, because a config that grants
|
|
||||||
and a request that claims are two separate acts.
|
|
||||||
|
|
||||||
Requesting an audience that is *not* listed here is
|
|
||||||
refused with `invalid_target`, which is what makes this
|
|
||||||
usable as a boundary rather than a label: a client cannot
|
|
||||||
mint a token for a resource slot that is not its own.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
accessTokenSignedResponseAlg = lib.mkOption {
|
|
||||||
type = lib.types.nullOr (
|
|
||||||
lib.types.enum [
|
|
||||||
"none"
|
|
||||||
"RS256"
|
|
||||||
]
|
|
||||||
);
|
|
||||||
default = null;
|
|
||||||
example = "RS256";
|
|
||||||
description = ''
|
|
||||||
Signing algorithm for this client's **access** tokens.
|
|
||||||
`null` leaves authelia on its default, which issues an
|
|
||||||
opaque token (`authelia_at_…`) — a database handle that
|
|
||||||
carries no claims and means nothing to anyone but this
|
|
||||||
provider.
|
|
||||||
|
|
||||||
Set `RS256` when the resource server verifies the token
|
|
||||||
*itself* rather than asking this provider about it: that
|
|
||||||
yields an RFC 9068 JWT (`at+jwt`) carrying `aud`, `iss`
|
|
||||||
and `client_id`, verifiable against `/jwks.json` with no
|
|
||||||
round trip.
|
|
||||||
|
|
||||||
⚠️ This is what makes a token readable by an
|
|
||||||
OIDC-verifying consumer at all. A resource server given
|
|
||||||
an opaque token is not *misconfigured* — it is
|
|
||||||
structurally unable to verify it, and says so in terms
|
|
||||||
that point at the verifier rather than at the token's
|
|
||||||
format.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
|
||||||
|
|
@ -40,43 +40,6 @@ let
|
||||||
# the required-domain assertion in hive-network.nix should be what an
|
# the required-domain assertion in hive-network.nix should be what an
|
||||||
# operator sees, not a coercion error from here.
|
# operator sees, not a coercion error from here.
|
||||||
domainBase = if swarmDomain == null then "invalid" else swarmDomain;
|
domainBase = if swarmDomain == null then "invalid" else swarmDomain;
|
||||||
|
|
||||||
autheliaCfg = hyperhiveCfg.swarm.authelia;
|
|
||||||
|
|
||||||
# `attrNames` is sorted, so this is a function of the hive SET and not of
|
|
||||||
# the order anyone wrote it in.
|
|
||||||
#
|
|
||||||
# These ports are internal and appear in no URL: a hive addresses its own
|
|
||||||
# receiver as a PATH on this collector's single gateway name, and nginx —
|
|
||||||
# rendered from this same evaluation — is the only thing that ever names
|
|
||||||
# the port. That is what makes deriving them safe here and unsafe in the
|
|
||||||
# obvious other place: were a hive told a port, inserting a hive would
|
|
||||||
# renumber the ones after it and silently move a port a running hive was
|
|
||||||
# already sending to.
|
|
||||||
hivePorts = lib.listToAttrs (
|
|
||||||
lib.imap0 (i: h: lib.nameValuePair h (cfg.port + i)) (lib.attrNames hyperhiveCfg.swarm.hives)
|
|
||||||
);
|
|
||||||
|
|
||||||
# The swarm's authelia is reached by its gateway name, whose leaf is
|
|
||||||
# issued by the swarm services sub-CA — so this container needs the same
|
|
||||||
# runtime CA trust every other consumer of a swarm-service name needs.
|
|
||||||
# The CA is generated at runtime and cannot be baked into a derivation,
|
|
||||||
# which is why it arrives as a bind mount rather than
|
|
||||||
# `security.pki.certificateFiles`.
|
|
||||||
caTrust = import ./lib/hive-ca-trust.nix {
|
|
||||||
inherit lib;
|
|
||||||
tlsCfg = hyperhiveCfg.tls;
|
|
||||||
inherit gatewayCfg;
|
|
||||||
};
|
|
||||||
caBundle = caTrust.bundlePathFor cfg.machine;
|
|
||||||
|
|
||||||
# One list, read by every pipeline: the per-hive pipelines fan out to
|
|
||||||
# exactly the same destinations as the single pipeline they replace.
|
|
||||||
# Written once because "which exporters" is a property of this tier, not
|
|
||||||
# of which hive a sample came from.
|
|
||||||
exporterNames =
|
|
||||||
lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp")
|
|
||||||
++ lib.optional vmCfg.enable "otlphttp/victoriametrics";
|
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
options.services.hyperhive.swarm.otel = {
|
options.services.hyperhive.swarm.otel = {
|
||||||
|
|
@ -112,17 +75,7 @@ in
|
||||||
type = lib.types.port;
|
type = lib.types.port;
|
||||||
default = 4319;
|
default = 4319;
|
||||||
description = ''
|
description = ''
|
||||||
First port of this collector's receiver range. Every hive in
|
Port this collector's OTLP/HTTP receiver listens on.
|
||||||
{option}`services.hyperhive.swarm.hives` gets its **own**
|
|
||||||
authenticated receiver — that is what makes the `hive` label
|
|
||||||
unforgeable — so the range is one port per hive, starting here, in
|
|
||||||
sorted-name order.
|
|
||||||
|
|
||||||
⚠️ Internal. No client is ever told a port: a hive reaches its own
|
|
||||||
receiver as `https://''${domain}/<hive>`, and the gateway routes on
|
|
||||||
that path. So adding a hive, which renumbers the ones after it, is
|
|
||||||
harmless — nginx is rendered from this same evaluation and moves
|
|
||||||
with it.
|
|
||||||
|
|
||||||
⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the
|
⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the
|
||||||
hive tier already uses it (`services.hyperhive.otel.collector.port`)
|
hive tier already uses it (`services.hyperhive.otel.collector.port`)
|
||||||
|
|
@ -130,9 +83,7 @@ in
|
||||||
listeners claiming one port on one host is not a build failure —
|
listeners claiming one port on one host is not a build failure —
|
||||||
it is a runtime coin toss over which one gets it, with nothing in
|
it is a runtime coin toss over which one gets it, with nothing in
|
||||||
any log saying so. The same collision cost a release when grafana
|
any log saying so. The same collision cost a release when grafana
|
||||||
and the forge both defaulted to 3000. The assertions below check
|
and the forge both defaulted to 3000.
|
||||||
the whole derived range against every port this module and the hive
|
|
||||||
tier declare, which is as far as a module can see.
|
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -192,39 +143,11 @@ in
|
||||||
services.nginx.virtualHosts."${cfg.domain}" = (gatewayCfg.lib.tlsFor cfg.domain) // {
|
services.nginx.virtualHosts."${cfg.domain}" = (gatewayCfg.lib.tlsFor cfg.domain) // {
|
||||||
listen = gatewayCfg.lib.listen;
|
listen = gatewayCfg.lib.listen;
|
||||||
extraConfig = gatewayCfg.lib.securityHeaders;
|
extraConfig = gatewayCfg.lib.securityHeaders;
|
||||||
locations =
|
locations."/" = {
|
||||||
# One name for the whole collector, and the hive is a path under
|
proxyPass = "http://127.0.0.1:${toString cfg.port}";
|
||||||
# it. The alternative — a vhost per hive — needs a certificate,
|
|
||||||
# a DNS name and a `localNames` entry per hive to express the
|
|
||||||
# same routing the gateway already does for free.
|
|
||||||
#
|
|
||||||
# ⚠️ The trailing slash on both sides is load-bearing: it is what
|
|
||||||
# strips `/<hive>` before the request reaches the receiver, which
|
|
||||||
# serves `/v1/metrics` and knows nothing about hives. Without it
|
|
||||||
# the receiver sees `/<hive>/v1/metrics` and answers 404 to a
|
|
||||||
# request that authenticated perfectly.
|
|
||||||
lib.mapAttrs' (
|
|
||||||
h: p: lib.nameValuePair "/${h}/" { proxyPass = "http://127.0.0.1:${toString p}/"; }
|
|
||||||
) hivePorts
|
|
||||||
// {
|
|
||||||
# There is no swarm-wide inbox, and a closed door is the honest
|
|
||||||
# description of that. Every route into this collector belongs to
|
|
||||||
# exactly one hive.
|
|
||||||
"/".return = "404";
|
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
# Turning on the identities this tier authenticates against, which the
|
|
||||||
# option exists to allow: its own description names this module as the
|
|
||||||
# second consumer, so the queue is not a prerequisite for authenticated
|
|
||||||
# telemetry.
|
|
||||||
services.hyperhive.swarm.authelia.oidc.hiveIdentities = true;
|
|
||||||
|
|
||||||
# The CA bind source is written at runtime by a host unit, so the
|
|
||||||
# container has to start after it — otherwise nspawn sets up a mount
|
|
||||||
# over a file that does not exist yet.
|
|
||||||
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;
|
|
||||||
|
|
||||||
assertions = [
|
assertions = [
|
||||||
{
|
{
|
||||||
# The tier exists to hold the upstream credential and to write the
|
# The tier exists to hold the upstream credential and to write the
|
||||||
|
|
@ -241,67 +164,6 @@ in
|
||||||
metrics store.
|
metrics store.
|
||||||
'';
|
'';
|
||||||
}
|
}
|
||||||
{
|
|
||||||
# Without a roster there are no receivers at all, so this
|
|
||||||
# collector would listen on nothing while looking configured.
|
|
||||||
assertion = hyperhiveCfg.swarm.hives != { };
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.swarm.otel.enable is true but
|
|
||||||
services.hyperhive.swarm.hives is empty: ingest is authenticated
|
|
||||||
per hive, so an empty roster means this collector accepts nothing
|
|
||||||
from anyone.
|
|
||||||
|
|
||||||
List the swarm's hives.
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# A hive proves who it is with a token this provider mints, so
|
|
||||||
# there is no version of this collector that runs without one.
|
|
||||||
# Stated as an assertion rather than a fallback because a guessed
|
|
||||||
# issuer URL evaluates cleanly and refuses every hive at runtime.
|
|
||||||
assertion = autheliaCfg.url != null;
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.swarm.otel.enable is true but
|
|
||||||
services.hyperhive.swarm.authelia.url is null: every hive
|
|
||||||
authenticates to this collector as itself, and the token comes
|
|
||||||
from the swarm's identity provider.
|
|
||||||
|
|
||||||
Point authelia.url at the swarm's provider, or enable
|
|
||||||
services.hyperhive.swarm.authelia on the host that runs it.
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# A port collision between two listeners on one host is a runtime
|
|
||||||
# coin toss with nothing in any log — the failure this whole
|
|
||||||
# comment budget exists to prevent. Checked against every port
|
|
||||||
# reachable from here; a port some other module picks is not.
|
|
||||||
#
|
|
||||||
# ⚠️ `cfg.port` is deliberately absent from `others`: it is the
|
|
||||||
# FIRST element of the derived range, so listing it would make this
|
|
||||||
# assertion fire on every config.
|
|
||||||
assertion =
|
|
||||||
let
|
|
||||||
derived = lib.attrValues hivePorts;
|
|
||||||
others = [
|
|
||||||
cfg.telemetryPort
|
|
||||||
otelCfg.collector.port
|
|
||||||
]
|
|
||||||
++ lib.optional vmCfg.enable vmCfg.port;
|
|
||||||
all = derived ++ others;
|
|
||||||
in
|
|
||||||
lib.length (lib.unique all) == lib.length all;
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.swarm.otel: the receiver range starting at
|
|
||||||
port (${toString cfg.port}, one port per hive in
|
|
||||||
services.hyperhive.swarm.hives) overlaps another port on this
|
|
||||||
host.
|
|
||||||
|
|
||||||
Every swarm container shares the host's network namespace, so
|
|
||||||
two listeners claiming one port is not a build failure — it is
|
|
||||||
whichever process started first, silently. Move
|
|
||||||
services.hyperhive.swarm.otel.port to a free range.
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
];
|
];
|
||||||
|
|
||||||
containers.${cfg.machine} = {
|
containers.${cfg.machine} = {
|
||||||
|
|
@ -317,16 +179,12 @@ in
|
||||||
# Read-only, and only when one is configured — binding a path that
|
# Read-only, and only when one is configured — binding a path that
|
||||||
# does not exist makes nixos-container refuse to start the container,
|
# does not exist makes nixos-container refuse to start the container,
|
||||||
# which is a stall several layers from its cause.
|
# which is a stall several layers from its cause.
|
||||||
bindMounts =
|
bindMounts = lib.optionalAttrs (otelCfg.headersCredential != null) {
|
||||||
lib.optionalAttrs (otelCfg.headersCredential != null) {
|
|
||||||
${otelCfg.headersCredential} = {
|
${otelCfg.headersCredential} = {
|
||||||
hostPath = otelCfg.headersCredential;
|
hostPath = otelCfg.headersCredential;
|
||||||
isReadOnly = true;
|
isReadOnly = true;
|
||||||
};
|
};
|
||||||
}
|
};
|
||||||
# The public hive CA, read-only — only when something in here
|
|
||||||
# actually verifies a swarm-service name.
|
|
||||||
// caTrust.bindMount;
|
|
||||||
|
|
||||||
config =
|
config =
|
||||||
{ ... }:
|
{ ... }:
|
||||||
|
|
@ -342,18 +200,6 @@ in
|
||||||
inherit (config.services.hyperhive.network) bridgeIp;
|
inherit (config.services.hyperhive.network) bridgeIp;
|
||||||
dnsConsumers = [ "opentelemetry-collector.service" ];
|
dnsConsumers = [ "opentelemetry-collector.service" ];
|
||||||
})
|
})
|
||||||
]
|
|
||||||
# `SSL_CERT_FILE` REPLACES the trust store rather than adding to
|
|
||||||
# it, so a failed assembly yields an empty pool and every TLS
|
|
||||||
# call fails while the unit looks healthy. That is why this is
|
|
||||||
# the shared helper — it carries the `Requires` and the
|
|
||||||
# non-empty check — and not a local `cat`.
|
|
||||||
++ [
|
|
||||||
(caTrust.trustBundle {
|
|
||||||
inherit pkgs;
|
|
||||||
name = cfg.machine;
|
|
||||||
consumers = [ "opentelemetry-collector" ];
|
|
||||||
})
|
|
||||||
];
|
];
|
||||||
|
|
||||||
system.stateVersion = config.system.stateVersion;
|
system.stateVersion = config.system.stateVersion;
|
||||||
|
|
@ -373,24 +219,7 @@ in
|
||||||
# real sample through both tiers into the store.
|
# real sample through both tiers into the store.
|
||||||
validateConfigFile = true;
|
validateConfigFile = true;
|
||||||
settings = {
|
settings = {
|
||||||
# One receiver per hive, and that multiplicity is forced
|
receivers.otlp.protocols.http.endpoint = "127.0.0.1:${toString cfg.port}";
|
||||||
# rather than chosen. The `hive`
|
|
||||||
# label has to come from something the sender cannot write,
|
|
||||||
# and the only such thing here is WHICH RECEIVER accepted
|
|
||||||
# the sample: a processor cannot read the token's claims
|
|
||||||
# (`from_context` reads request metadata, and asking it for
|
|
||||||
# an auth claim yields nothing — silently, with a healthy
|
|
||||||
# startup), and one receiver holding many credentials never
|
|
||||||
# reveals which one matched.
|
|
||||||
receivers = lib.mapAttrs' (
|
|
||||||
h: p:
|
|
||||||
lib.nameValuePair "otlp/${h}" {
|
|
||||||
protocols.http = {
|
|
||||||
endpoint = "127.0.0.1:${toString p}";
|
|
||||||
auth.authenticator = "oidc/${h}";
|
|
||||||
};
|
|
||||||
}
|
|
||||||
) hivePorts;
|
|
||||||
|
|
||||||
exporters =
|
exporters =
|
||||||
lib.optionalAttrs vmCfg.enable {
|
lib.optionalAttrs vmCfg.enable {
|
||||||
|
|
@ -434,64 +263,16 @@ in
|
||||||
}
|
}
|
||||||
];
|
];
|
||||||
|
|
||||||
# ⚠️ An extension that is configured but not listed here is
|
service.pipelines.metrics = {
|
||||||
# INERT — the collector starts clean and the receiver
|
receivers = [ "otlp" ];
|
||||||
# naming it authenticates nothing. Derived from the same
|
|
||||||
# attrset as the receivers so the two cannot disagree.
|
|
||||||
service.extensions = map (h: "oidc/${h}") (lib.attrNames hivePorts);
|
|
||||||
|
|
||||||
# Fan-out, not a choice: with both configured the same
|
# Fan-out, not a choice: with both configured the same
|
||||||
# samples go upstream AND into the swarm's store. The store
|
# samples go upstream AND into the swarm's store. The store
|
||||||
# is for looking at this swarm; the upstream is for whoever
|
# is for looking at this swarm; the upstream is for whoever
|
||||||
# aggregates across swarms, and neither replaces the other.
|
# aggregates across swarms, and neither replaces the other.
|
||||||
# `exporterNames` is shared by every pipeline — where a
|
exporters =
|
||||||
# sample goes is a property of this tier, not of the hive
|
lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp")
|
||||||
# that sent it.
|
++ lib.optional vmCfg.enable "otlphttp/victoriametrics";
|
||||||
service.pipelines = lib.mapAttrs' (
|
};
|
||||||
h: _:
|
|
||||||
lib.nameValuePair "metrics/${h}" {
|
|
||||||
receivers = [ "otlp/${h}" ];
|
|
||||||
processors = [ "resource/${h}" ];
|
|
||||||
exporters = exporterNames;
|
|
||||||
}
|
|
||||||
) hivePorts;
|
|
||||||
}
|
|
||||||
// {
|
|
||||||
extensions = lib.mapAttrs' (
|
|
||||||
h: _:
|
|
||||||
lib.nameValuePair "oidc/${h}" {
|
|
||||||
issuer_url = autheliaCfg.url;
|
|
||||||
# The audience this hive's client is registered to
|
|
||||||
# request, and the reason one hive's token is refused by
|
|
||||||
# another hive's receiver. Same expression authelia
|
|
||||||
# registers it under — a second spelling here would deny
|
|
||||||
# every hive, as a 401 that blames the token.
|
|
||||||
audience = "${autheliaCfg.hiveClientPrefix}${h}";
|
|
||||||
# ⚠️ `issuer_ca_path`. `issuer_ca_file`, `ca_file` and
|
|
||||||
# `tls.ca_file` are all INVALID KEYS for this extension
|
|
||||||
# — measured, and the failure is a startup error naming
|
|
||||||
# the key rather than anything about certificates.
|
|
||||||
issuer_ca_path = caBundle;
|
|
||||||
}
|
|
||||||
) hivePorts;
|
|
||||||
|
|
||||||
# `upsert`, not `insert`: a sender that stamps its own
|
|
||||||
# `hive` must be OVERWRITTEN, not deferred to. This
|
|
||||||
# processor is the whole attribution boundary — the value
|
|
||||||
# is a constant per receiver, so it says which hive
|
|
||||||
# authenticated, not which hive claimed to be sending.
|
|
||||||
processors = lib.mapAttrs' (
|
|
||||||
h: _:
|
|
||||||
lib.nameValuePair "resource/${h}" {
|
|
||||||
attributes = [
|
|
||||||
{
|
|
||||||
key = "hive";
|
|
||||||
value = h;
|
|
||||||
action = "upsert";
|
|
||||||
}
|
|
||||||
];
|
|
||||||
}
|
|
||||||
) hivePorts;
|
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue