Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9bd2b9e9e6 | ||
|
|
7da7915150 | ||
|
|
9dc60061e7 | ||
|
|
cb787997bd | ||
|
|
08faa0970e | ||
|
|
0b98f0ac8f | ||
|
|
0f801c645f |
7 changed files with 519 additions and 27 deletions
|
|
@ -186,6 +186,31 @@ redefines it — the agent-facing value is *derived*
|
|||
keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
|
||||
automatically; there is nothing to open by hand.
|
||||
|
||||
### Authenticated ingest
|
||||
|
||||
The swarm tier gives **each hive its own receiver**, and stamps the `hive` label
|
||||
from whichever receiver accepted a sample. A hive therefore cannot report
|
||||
metrics as another hive, and cannot relabel its own by editing what it sends —
|
||||
the label is not taken from the payload at all.
|
||||
|
||||
**On an all-local swarm there is nothing to set.** Each hive already has an
|
||||
identity, and its collector reads the secret that host's own authelia minted.
|
||||
|
||||
**On a hive that does not host the swarm's services**, the secret has to arrive
|
||||
somehow — copy it across and name it:
|
||||
|
||||
```nix
|
||||
services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret";
|
||||
```
|
||||
|
||||
**There is no unauthenticated mode.** A hive always presents an identity, so a
|
||||
missing credential is a build error rather than a quieter fallback — the
|
||||
collector has no anonymous route to accept samples on, and every path it serves
|
||||
belongs to exactly one hive.
|
||||
|
||||
Getting the secret wrong shows up as the hive's collector logging 401s from the
|
||||
swarm tier and no metrics appearing for that hive.
|
||||
|
||||
### `services.hyperhive.otel.collector.port` — port, default `4318`
|
||||
|
||||
The OTLP/HTTP port the hive tier listens on, bound to the bridge IP only. The
|
||||
|
|
|
|||
|
|
@ -156,6 +156,11 @@ See [`tools/hivectl.md`](tools/hivectl.md) for every `hivectl` verb.
|
|||
- All config changes (forge PRs on `agent-configs/<name>`) go through
|
||||
operator approval — agents can't unilaterally rebuild containers, by design.
|
||||
See [`boundary.md`](boundary.md) and [`security.md`](security.md).
|
||||
- **Telemetry ingest is authenticated per hive**, and the `hive` label comes
|
||||
from which hive authenticated rather than from the payload — so no hive can
|
||||
report metrics as another. A first-run all-local hive gets this with nothing
|
||||
to configure; joining a swarm you don't host needs one secret copied across.
|
||||
See [`observability.md`](observability.md#authenticated-ingest).
|
||||
|
||||
Once the hive is running, ruth records anything it needs to remember
|
||||
across restarts in `/agents/ruth/state/notes.md`.
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ Every row below is read against one of these.
|
|||
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
||||
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
||||
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in the service's `sso.clientSecretFile` |
|
||||
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
||||
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `swarm.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
|
|
@ -56,6 +56,13 @@ because nothing outside that container ever reads them. **That is the test worth
|
|||
applying to any secret added here** — and the client secret's plaintext half is
|
||||
the one row that fails it, which is the entire reason a delivery step exists.
|
||||
|
||||
One reader needs no delivery step: the **hive's telemetry collector**, which
|
||||
authenticates to the swarm's collector as its own hive. It is a host unit rather
|
||||
than a container, so on an all-local swarm it reads authelia's file where it
|
||||
lies (through `LoadCredential`) and no second copy is made. On any other
|
||||
topology it is an ordinary "operator provides the file" case — see
|
||||
`services.hyperhive.otel.clientSecretFile`.
|
||||
|
||||
### Minting the queue's callout nkeys
|
||||
|
||||
`nats.autoGenerateCallout` mints both keypairs on the host before the queue
|
||||
|
|
|
|||
|
|
@ -39,10 +39,26 @@ let
|
|||
# the bundle next to the CA and explains the split.
|
||||
caHostPath = "${tlsCfg.stateDir}/trust-bundle.pem";
|
||||
caContainerPath = "/run/hive-ca/trust-bundle.pem";
|
||||
|
||||
# One definition, used by `trustBundle` to WRITE the bundle and published
|
||||
# below so a caller can NAME it. Two copies of this path would be two
|
||||
# things to keep in step, and the one that drifts is the reader.
|
||||
bundleDirFor = name: "/run/${name}-ca";
|
||||
bundlePathFor = name: "${bundleDirFor name}/trust-bundle.pem";
|
||||
in
|
||||
{
|
||||
inherit useSelfSigned caContainerPath;
|
||||
|
||||
# Where `trustBundle` below puts the assembled bundle, for the callers
|
||||
# that must NAME it rather than just have it exported. `SSL_CERT_FILE` is
|
||||
# set for you and needs no path here; a consumer that takes its own CA
|
||||
# argument (an OIDC verifier's `issuer_ca_path`, a client's `--cacert`)
|
||||
# does, and the alternative is copying `/run/<name>-ca/…` to the call
|
||||
# site. That copy breaks silently: the bundle keeps being written, the
|
||||
# consumer keeps reading a path that no longer exists, and the failure
|
||||
# surfaces as a TLS error naming the peer rather than the file.
|
||||
inherit bundlePathFor;
|
||||
|
||||
# Fold into the container's `bindMounts` via `//`. Binds ONLY the public
|
||||
# CA cert (never the `hive-tls` state dir — it holds the CA + leaf private
|
||||
# keys), read-only. Empty when not self-signed, so the whole trust path
|
||||
|
|
@ -106,8 +122,8 @@ in
|
|||
enable ? true,
|
||||
}:
|
||||
let
|
||||
dir = "/run/${name}-ca";
|
||||
bundlePath = "${dir}/trust-bundle.pem";
|
||||
dir = bundleDirFor name;
|
||||
bundlePath = bundlePathFor name;
|
||||
unit = "${name}-ca-bundle";
|
||||
source = if hostUnit then caHostPath else caContainerPath;
|
||||
in
|
||||
|
|
|
|||
|
|
@ -193,6 +193,40 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
clientSecretFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default =
|
||||
if
|
||||
config.services.hyperhive.swarm.authelia.enable && config.services.hyperhive.hiveName != null
|
||||
then
|
||||
"${config.services.hyperhive.swarm.authelia.hostClientSecretDir}/"
|
||||
+ "${config.services.hyperhive.swarm.authelia.hiveClientPrefix}${config.services.hyperhive.hiveName}.secret"
|
||||
else
|
||||
null;
|
||||
defaultText = lib.literalExpression ''"''${swarm.authelia.hostClientSecretDir}/''${swarm.authelia.hiveClientPrefix}''${hiveName}.secret" when this host runs the swarm's IdP, else null'';
|
||||
example = "/var/lib/secrets/hive-telemetry.secret";
|
||||
description = ''
|
||||
Absolute path to this hive's OAuth2 client secret, used to
|
||||
authenticate to the swarm's collector as this hive.
|
||||
|
||||
**Whether this is set is what decides whether this tier
|
||||
authenticates at all.** A credential is the only thing that makes
|
||||
authenticated egress possible, so its presence is the condition
|
||||
rather than a second switch that could disagree with it.
|
||||
|
||||
Defaults to the secret this host's own authelia minted, which is
|
||||
correct exactly when the swarm's IdP runs here. On a hive that is
|
||||
not that host, the file has to arrive some other way and this
|
||||
option names wherever it landed — the same manual-copy shape
|
||||
`services.hyperhive.swarm.authelia.oidc.hiveIdentities` documents,
|
||||
where delivering a secret to a hive that is not this host is
|
||||
deliberately not solved.
|
||||
|
||||
Read by `LoadCredential`, so it is never evaluated by nix, never
|
||||
copied into the store and never passed in argv.
|
||||
'';
|
||||
};
|
||||
|
||||
metricIntervalMs = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.ints.positive;
|
||||
default = null;
|
||||
|
|
@ -214,8 +248,32 @@ in
|
|||
config = lib.mkIf config.services.hyperhive.otel.enable (
|
||||
let
|
||||
otel = config.services.hyperhive.otel;
|
||||
autheliaCfg = config.services.hyperhive.swarm.authelia;
|
||||
swarmOtelCfg = config.services.hyperhive.swarm.otel;
|
||||
hiveName = config.services.hyperhive.hiveName;
|
||||
listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
|
||||
swarmName = "otlphttp/swarm";
|
||||
authName = "oauth2client/swarm";
|
||||
|
||||
# A hive always authenticates to the swarm's collector as itself, so
|
||||
# this is not a mode — it is a precondition, and the assertion below
|
||||
# is what enforces it. Kept as a name because several places have to
|
||||
# read "do we have what it takes", and an eval error from a null path
|
||||
# names this file rather than the option an operator has to set.
|
||||
senderAuth = otel.clientSecretFile != null && hiveName != null;
|
||||
|
||||
# This hive's client id, and also the audience it must ASK for. Both
|
||||
# are `hiveClientPrefix` + the hive's name because that is the one
|
||||
# name the swarm already agrees on; the receiver one tier up derives
|
||||
# the same string.
|
||||
hiveClient = "${autheliaCfg.hiveClientPrefix}${toString hiveName}";
|
||||
|
||||
# systemd exports `CREDENTIALS_DIRECTORY` to any unit with
|
||||
# `LoadCredential`, and the collector expands `${env:…}` at load. So
|
||||
# the secret reaches the process as a PATH resolved at runtime — nix
|
||||
# renders neither the value nor the directory, and nothing has to
|
||||
# hardcode `/run/credentials/<unit>`.
|
||||
credName = "swarm-client.secret";
|
||||
in
|
||||
{
|
||||
# Reachable from agent containers and nowhere else: this opens
|
||||
|
|
@ -224,6 +282,22 @@ in
|
|||
|
||||
services.opentelemetry-collector = {
|
||||
enable = true;
|
||||
# Contrib, matching the swarm tier (./swarm-otel.nix). The upstream
|
||||
# default build has no auth extensions at all, and this tier has to
|
||||
# *present* a credential to the swarm tier — `oauth2client` lives
|
||||
# only in contrib, so the package choice is what makes authenticated
|
||||
# egress expressible rather than a preference.
|
||||
#
|
||||
# Not a build-farm cost: contrib is fetched, not compiled.
|
||||
#
|
||||
# ⚠️ Read the note directly below before adding any extension here.
|
||||
# It describes precisely the trap this package unlocks: naming an
|
||||
# extension the build lacks passes `validate` and then kills the
|
||||
# collector at startup. With contrib the extensions exist — but the
|
||||
# gap it warns about (a green build proving nothing about whether
|
||||
# the process starts) is exactly why this module's auth wiring is
|
||||
# gated by a probe that runs both collectors, not by eval.
|
||||
package = pkgs.opentelemetry-collector-contrib;
|
||||
# `validateConfigFile` defaults to `isStorePath configFile`,
|
||||
# and `configFile` is null on the `settings` path — so the
|
||||
# upstream default is OFF for exactly the way this module
|
||||
|
|
@ -263,15 +337,80 @@ in
|
|||
# change. `https://` because that name resolves through the
|
||||
# gateway even on a co-located host — see `caTrust` above for
|
||||
# the trust half that makes this verify.
|
||||
endpoint = "https://${config.services.hyperhive.swarm.otel.domain}";
|
||||
};
|
||||
# The hive's own path under the collector's single name. The
|
||||
# swarm tier gives each hive its own authenticated receiver and
|
||||
# routes to it by this prefix, so the path is not decoration —
|
||||
# it selects WHICH receiver, and therefore which hive the
|
||||
# samples get labelled as.
|
||||
endpoint = "https://${swarmOtelCfg.domain}" + lib.optionalString senderAuth "/${toString hiveName}";
|
||||
}
|
||||
// lib.optionalAttrs senderAuth { auth.authenticator = authName; };
|
||||
|
||||
# ⚠️ An extension not listed here is INERT: the collector starts
|
||||
# clean and the exporter naming it sends nothing authenticated.
|
||||
service.extensions = lib.optional senderAuth authName;
|
||||
|
||||
service.pipelines.metrics = {
|
||||
receivers = [ "otlp" ];
|
||||
exporters = [ swarmName ];
|
||||
};
|
||||
}
|
||||
// lib.optionalAttrs senderAuth {
|
||||
extensions.${authName} = {
|
||||
client_id = hiveClient;
|
||||
# A real key, measured against this collector version rather
|
||||
# than assumed — with a deliberate typo rejected in the same
|
||||
# run, so "accepted" is distinguishable from "ignores
|
||||
# everything". Keeps the secret out of nix entirely: the
|
||||
# collector opens the file itself.
|
||||
client_secret_file = "\${env:CREDENTIALS_DIRECTORY}/${credName}";
|
||||
token_url = "${toString autheliaCfg.url}/api/oidc/token";
|
||||
# ⚠️ THE AUDIENCE HAS TO BE REQUESTED, not merely granted.
|
||||
# Registering it on the client only makes it permissible; a
|
||||
# token minted without asking carries `aud: []` and every
|
||||
# receiver refuses it — with a config that reads perfectly at
|
||||
# both ends. Measured against authelia 4.39.20.
|
||||
endpoint_params.audience = hiveClient;
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
# `LoadCredential` and not a copy-oneshot: this collector is a HOST
|
||||
# unit, so there is no container boundary to cross and therefore no
|
||||
# reason for a second on-disk copy of the secret. systemd hands it to
|
||||
# the process in a private tmpfs and exports the directory, which is
|
||||
# what the config above names.
|
||||
systemd.services.opentelemetry-collector.serviceConfig = lib.optionalAttrs senderAuth {
|
||||
LoadCredential = [ "${credName}:${otel.clientSecretFile}" ];
|
||||
};
|
||||
|
||||
assertions = [
|
||||
{
|
||||
# A hive authenticates to the swarm's collector as itself — there
|
||||
# is no unauthenticated path to fall back to, so a missing
|
||||
# credential is a broken deployment rather than a quieter mode.
|
||||
# Caught here because the alternative is a collector that starts
|
||||
# cleanly, retries forever, and reports nothing to anyone.
|
||||
assertion = senderAuth;
|
||||
message = ''
|
||||
services.hyperhive.otel.enable is true but this hive has no
|
||||
identity to present to the swarm's collector:
|
||||
|
||||
services.hyperhive.otel.clientSecretFile = ${
|
||||
if otel.clientSecretFile == null then "null" else otel.clientSecretFile
|
||||
}
|
||||
services.hyperhive.hiveName = ${if hiveName == null then "null" else hiveName}
|
||||
|
||||
Every hive authenticates as itself — that is what makes the
|
||||
`hive` label on its metrics mean anything — so both are
|
||||
required.
|
||||
|
||||
On a host that runs the swarm's identity provider, the default
|
||||
already points at the secret authelia minted. On a hive that
|
||||
does not, copy that hive's secret across and name it here.
|
||||
'';
|
||||
}
|
||||
];
|
||||
}
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -102,11 +102,26 @@ let
|
|||
# and a field later added to the submodule then existed on the
|
||||
# declared entries and not on these, which is an eval error reachable
|
||||
# only once hive identities are on.
|
||||
#
|
||||
# `audience` is the hive's own client id rather than a second per-hive
|
||||
# string invented here. A swarm service that has to tell hives apart
|
||||
# needs one name per hive that both sides already agree on, and the
|
||||
# client id is that name — published as `hiveClientPrefix` for exactly
|
||||
# this reason. Minting a parallel naming scheme would be a second thing
|
||||
# to keep in step, and the one that drifts is the one nobody tests.
|
||||
#
|
||||
# `RS256` because a resource server that cannot call this provider back
|
||||
# is a real case here: the swarm's telemetry collector verifies tokens
|
||||
# offline against `/jwks.json`, and an opaque token gives it nothing to
|
||||
# verify. The queue's auth-callout responder introspects instead, which
|
||||
# is a different question asked of the same token.
|
||||
hiveClients = lib.mapAttrsToList (name: _: {
|
||||
id = "${cfg.hiveClientPrefix}${name}";
|
||||
description = "HyperHive hive ${name}";
|
||||
kind = "machine";
|
||||
redirectUris = [ ];
|
||||
audience = [ "${cfg.hiveClientPrefix}${name}" ];
|
||||
accessTokenSignedResponseAlg = "RS256";
|
||||
}) hyperhiveCfg.swarm.hives;
|
||||
|
||||
# `swarm-authelia-bridge`'s own identity — distinct from
|
||||
|
|
@ -188,6 +203,15 @@ let
|
|||
+ lib.optionalString (c.tokenEndpointAuthMethod != null) ''
|
||||
printf -- ' token_endpoint_auth_method: %s\n' ${lib.escapeShellArg c.tokenEndpointAuthMethod}
|
||||
''
|
||||
# Flow-style YAML, matching `scopes` below. The values are client ids
|
||||
# and hive names, which `Ident` already constrains to `[a-z0-9-]` — no
|
||||
# character in that set needs quoting in a YAML flow sequence.
|
||||
+ lib.optionalString (c.audience != [ ]) ''
|
||||
printf -- ' audience: [%s]\n' ${lib.escapeShellArg (lib.concatStringsSep ", " c.audience)}
|
||||
''
|
||||
+ lib.optionalString (c.accessTokenSignedResponseAlg != null) ''
|
||||
printf -- ' access_token_signed_response_alg: %s\n' ${lib.escapeShellArg c.accessTokenSignedResponseAlg}
|
||||
''
|
||||
+ (
|
||||
if c.kind == "machine" then
|
||||
''
|
||||
|
|
@ -493,6 +517,63 @@ in
|
|||
the secret is at fault.
|
||||
'';
|
||||
};
|
||||
|
||||
audience = lib.mkOption {
|
||||
type = lib.types.listOf lib.types.str;
|
||||
default = [ ];
|
||||
example = [ "hive-alpha" ];
|
||||
description = ''
|
||||
Audiences (`aud`) this client is permitted to request a
|
||||
token for. Empty means it asks for none, which is the
|
||||
right answer for a client whose resource server does not
|
||||
distinguish callers.
|
||||
|
||||
⚠️ Registering an audience only *permits* it — the value
|
||||
lands in a token when the client **asks** for it at the
|
||||
token endpoint, and a client that does not send
|
||||
`audience=` receives a token with `aud: []` however
|
||||
complete this list looks. Measured against authelia
|
||||
4.39.20: the config reads exactly right and the resource
|
||||
server rejects every token, because a config that grants
|
||||
and a request that claims are two separate acts.
|
||||
|
||||
Requesting an audience that is *not* listed here is
|
||||
refused with `invalid_target`, which is what makes this
|
||||
usable as a boundary rather than a label: a client cannot
|
||||
mint a token for a resource slot that is not its own.
|
||||
'';
|
||||
};
|
||||
|
||||
accessTokenSignedResponseAlg = lib.mkOption {
|
||||
type = lib.types.nullOr (
|
||||
lib.types.enum [
|
||||
"none"
|
||||
"RS256"
|
||||
]
|
||||
);
|
||||
default = null;
|
||||
example = "RS256";
|
||||
description = ''
|
||||
Signing algorithm for this client's **access** tokens.
|
||||
`null` leaves authelia on its default, which issues an
|
||||
opaque token (`authelia_at_…`) — a database handle that
|
||||
carries no claims and means nothing to anyone but this
|
||||
provider.
|
||||
|
||||
Set `RS256` when the resource server verifies the token
|
||||
*itself* rather than asking this provider about it: that
|
||||
yields an RFC 9068 JWT (`at+jwt`) carrying `aud`, `iss`
|
||||
and `client_id`, verifiable against `/jwks.json` with no
|
||||
round trip.
|
||||
|
||||
⚠️ This is what makes a token readable by an
|
||||
OIDC-verifying consumer at all. A resource server given
|
||||
an opaque token is not *misconfigured* — it is
|
||||
structurally unable to verify it, and says so in terms
|
||||
that point at the verifier rather than at the token's
|
||||
format.
|
||||
'';
|
||||
};
|
||||
};
|
||||
}
|
||||
);
|
||||
|
|
|
|||
|
|
@ -40,6 +40,43 @@ let
|
|||
# the required-domain assertion in hive-network.nix should be what an
|
||||
# operator sees, not a coercion error from here.
|
||||
domainBase = if swarmDomain == null then "invalid" else swarmDomain;
|
||||
|
||||
autheliaCfg = hyperhiveCfg.swarm.authelia;
|
||||
|
||||
# `attrNames` is sorted, so this is a function of the hive SET and not of
|
||||
# the order anyone wrote it in.
|
||||
#
|
||||
# These ports are internal and appear in no URL: a hive addresses its own
|
||||
# receiver as a PATH on this collector's single gateway name, and nginx —
|
||||
# rendered from this same evaluation — is the only thing that ever names
|
||||
# the port. That is what makes deriving them safe here and unsafe in the
|
||||
# obvious other place: were a hive told a port, inserting a hive would
|
||||
# renumber the ones after it and silently move a port a running hive was
|
||||
# already sending to.
|
||||
hivePorts = lib.listToAttrs (
|
||||
lib.imap0 (i: h: lib.nameValuePair h (cfg.port + i)) (lib.attrNames hyperhiveCfg.swarm.hives)
|
||||
);
|
||||
|
||||
# The swarm's authelia is reached by its gateway name, whose leaf is
|
||||
# issued by the swarm services sub-CA — so this container needs the same
|
||||
# runtime CA trust every other consumer of a swarm-service name needs.
|
||||
# The CA is generated at runtime and cannot be baked into a derivation,
|
||||
# which is why it arrives as a bind mount rather than
|
||||
# `security.pki.certificateFiles`.
|
||||
caTrust = import ./lib/hive-ca-trust.nix {
|
||||
inherit lib;
|
||||
tlsCfg = hyperhiveCfg.tls;
|
||||
inherit gatewayCfg;
|
||||
};
|
||||
caBundle = caTrust.bundlePathFor cfg.machine;
|
||||
|
||||
# One list, read by every pipeline: the per-hive pipelines fan out to
|
||||
# exactly the same destinations as the single pipeline they replace.
|
||||
# Written once because "which exporters" is a property of this tier, not
|
||||
# of which hive a sample came from.
|
||||
exporterNames =
|
||||
lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp")
|
||||
++ lib.optional vmCfg.enable "otlphttp/victoriametrics";
|
||||
in
|
||||
{
|
||||
options.services.hyperhive.swarm.otel = {
|
||||
|
|
@ -75,7 +112,17 @@ in
|
|||
type = lib.types.port;
|
||||
default = 4319;
|
||||
description = ''
|
||||
Port this collector's OTLP/HTTP receiver listens on.
|
||||
First port of this collector's receiver range. Every hive in
|
||||
{option}`services.hyperhive.swarm.hives` gets its **own**
|
||||
authenticated receiver — that is what makes the `hive` label
|
||||
unforgeable — so the range is one port per hive, starting here, in
|
||||
sorted-name order.
|
||||
|
||||
⚠️ Internal. No client is ever told a port: a hive reaches its own
|
||||
receiver as `https://''${domain}/<hive>`, and the gateway routes on
|
||||
that path. So adding a hive, which renumbers the ones after it, is
|
||||
harmless — nginx is rendered from this same evaluation and moves
|
||||
with it.
|
||||
|
||||
⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the
|
||||
hive tier already uses it (`services.hyperhive.otel.collector.port`)
|
||||
|
|
@ -83,7 +130,9 @@ in
|
|||
listeners claiming one port on one host is not a build failure —
|
||||
it is a runtime coin toss over which one gets it, with nothing in
|
||||
any log saying so. The same collision cost a release when grafana
|
||||
and the forge both defaulted to 3000.
|
||||
and the forge both defaulted to 3000. The assertions below check
|
||||
the whole derived range against every port this module and the hive
|
||||
tier declare, which is as far as a module can see.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -143,11 +192,39 @@ in
|
|||
services.nginx.virtualHosts."${cfg.domain}" = (gatewayCfg.lib.tlsFor cfg.domain) // {
|
||||
listen = gatewayCfg.lib.listen;
|
||||
extraConfig = gatewayCfg.lib.securityHeaders;
|
||||
locations."/" = {
|
||||
proxyPass = "http://127.0.0.1:${toString cfg.port}";
|
||||
};
|
||||
locations =
|
||||
# One name for the whole collector, and the hive is a path under
|
||||
# it. The alternative — a vhost per hive — needs a certificate,
|
||||
# a DNS name and a `localNames` entry per hive to express the
|
||||
# same routing the gateway already does for free.
|
||||
#
|
||||
# ⚠️ The trailing slash on both sides is load-bearing: it is what
|
||||
# strips `/<hive>` before the request reaches the receiver, which
|
||||
# serves `/v1/metrics` and knows nothing about hives. Without it
|
||||
# the receiver sees `/<hive>/v1/metrics` and answers 404 to a
|
||||
# request that authenticated perfectly.
|
||||
lib.mapAttrs' (
|
||||
h: p: lib.nameValuePair "/${h}/" { proxyPass = "http://127.0.0.1:${toString p}/"; }
|
||||
) hivePorts
|
||||
// {
|
||||
# There is no swarm-wide inbox, and a closed door is the honest
|
||||
# description of that. Every route into this collector belongs to
|
||||
# exactly one hive.
|
||||
"/".return = "404";
|
||||
};
|
||||
};
|
||||
|
||||
# Turning on the identities this tier authenticates against, which the
|
||||
# option exists to allow: its own description names this module as the
|
||||
# second consumer, so the queue is not a prerequisite for authenticated
|
||||
# telemetry.
|
||||
services.hyperhive.swarm.authelia.oidc.hiveIdentities = true;
|
||||
|
||||
# The CA bind source is written at runtime by a host unit, so the
|
||||
# container has to start after it — otherwise nspawn sets up a mount
|
||||
# over a file that does not exist yet.
|
||||
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;
|
||||
|
||||
assertions = [
|
||||
{
|
||||
# The tier exists to hold the upstream credential and to write the
|
||||
|
|
@ -164,6 +241,67 @@ in
|
|||
metrics store.
|
||||
'';
|
||||
}
|
||||
{
|
||||
# Without a roster there are no receivers at all, so this
|
||||
# collector would listen on nothing while looking configured.
|
||||
assertion = hyperhiveCfg.swarm.hives != { };
|
||||
message = ''
|
||||
services.hyperhive.swarm.otel.enable is true but
|
||||
services.hyperhive.swarm.hives is empty: ingest is authenticated
|
||||
per hive, so an empty roster means this collector accepts nothing
|
||||
from anyone.
|
||||
|
||||
List the swarm's hives.
|
||||
'';
|
||||
}
|
||||
{
|
||||
# A hive proves who it is with a token this provider mints, so
|
||||
# there is no version of this collector that runs without one.
|
||||
# Stated as an assertion rather than a fallback because a guessed
|
||||
# issuer URL evaluates cleanly and refuses every hive at runtime.
|
||||
assertion = autheliaCfg.url != null;
|
||||
message = ''
|
||||
services.hyperhive.swarm.otel.enable is true but
|
||||
services.hyperhive.swarm.authelia.url is null: every hive
|
||||
authenticates to this collector as itself, and the token comes
|
||||
from the swarm's identity provider.
|
||||
|
||||
Point authelia.url at the swarm's provider, or enable
|
||||
services.hyperhive.swarm.authelia on the host that runs it.
|
||||
'';
|
||||
}
|
||||
{
|
||||
# A port collision between two listeners on one host is a runtime
|
||||
# coin toss with nothing in any log — the failure this whole
|
||||
# comment budget exists to prevent. Checked against every port
|
||||
# reachable from here; a port some other module picks is not.
|
||||
#
|
||||
# ⚠️ `cfg.port` is deliberately absent from `others`: it is the
|
||||
# FIRST element of the derived range, so listing it would make this
|
||||
# assertion fire on every config.
|
||||
assertion =
|
||||
let
|
||||
derived = lib.attrValues hivePorts;
|
||||
others = [
|
||||
cfg.telemetryPort
|
||||
otelCfg.collector.port
|
||||
]
|
||||
++ lib.optional vmCfg.enable vmCfg.port;
|
||||
all = derived ++ others;
|
||||
in
|
||||
lib.length (lib.unique all) == lib.length all;
|
||||
message = ''
|
||||
services.hyperhive.swarm.otel: the receiver range starting at
|
||||
port (${toString cfg.port}, one port per hive in
|
||||
services.hyperhive.swarm.hives) overlaps another port on this
|
||||
host.
|
||||
|
||||
Every swarm container shares the host's network namespace, so
|
||||
two listeners claiming one port is not a build failure — it is
|
||||
whichever process started first, silently. Move
|
||||
services.hyperhive.swarm.otel.port to a free range.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
containers.${cfg.machine} = {
|
||||
|
|
@ -179,12 +317,16 @@ in
|
|||
# Read-only, and only when one is configured — binding a path that
|
||||
# does not exist makes nixos-container refuse to start the container,
|
||||
# which is a stall several layers from its cause.
|
||||
bindMounts = lib.optionalAttrs (otelCfg.headersCredential != null) {
|
||||
${otelCfg.headersCredential} = {
|
||||
hostPath = otelCfg.headersCredential;
|
||||
isReadOnly = true;
|
||||
};
|
||||
};
|
||||
bindMounts =
|
||||
lib.optionalAttrs (otelCfg.headersCredential != null) {
|
||||
${otelCfg.headersCredential} = {
|
||||
hostPath = otelCfg.headersCredential;
|
||||
isReadOnly = true;
|
||||
};
|
||||
}
|
||||
# The public hive CA, read-only — only when something in here
|
||||
# actually verifies a swarm-service name.
|
||||
// caTrust.bindMount;
|
||||
|
||||
config =
|
||||
{ ... }:
|
||||
|
|
@ -200,6 +342,18 @@ in
|
|||
inherit (config.services.hyperhive.network) bridgeIp;
|
||||
dnsConsumers = [ "opentelemetry-collector.service" ];
|
||||
})
|
||||
]
|
||||
# `SSL_CERT_FILE` REPLACES the trust store rather than adding to
|
||||
# it, so a failed assembly yields an empty pool and every TLS
|
||||
# call fails while the unit looks healthy. That is why this is
|
||||
# the shared helper — it carries the `Requires` and the
|
||||
# non-empty check — and not a local `cat`.
|
||||
++ [
|
||||
(caTrust.trustBundle {
|
||||
inherit pkgs;
|
||||
name = cfg.machine;
|
||||
consumers = [ "opentelemetry-collector" ];
|
||||
})
|
||||
];
|
||||
|
||||
system.stateVersion = config.system.stateVersion;
|
||||
|
|
@ -219,7 +373,24 @@ in
|
|||
# real sample through both tiers into the store.
|
||||
validateConfigFile = true;
|
||||
settings = {
|
||||
receivers.otlp.protocols.http.endpoint = "127.0.0.1:${toString cfg.port}";
|
||||
# One receiver per hive, and that multiplicity is forced
|
||||
# rather than chosen. The `hive`
|
||||
# label has to come from something the sender cannot write,
|
||||
# and the only such thing here is WHICH RECEIVER accepted
|
||||
# the sample: a processor cannot read the token's claims
|
||||
# (`from_context` reads request metadata, and asking it for
|
||||
# an auth claim yields nothing — silently, with a healthy
|
||||
# startup), and one receiver holding many credentials never
|
||||
# reveals which one matched.
|
||||
receivers = lib.mapAttrs' (
|
||||
h: p:
|
||||
lib.nameValuePair "otlp/${h}" {
|
||||
protocols.http = {
|
||||
endpoint = "127.0.0.1:${toString p}";
|
||||
auth.authenticator = "oidc/${h}";
|
||||
};
|
||||
}
|
||||
) hivePorts;
|
||||
|
||||
exporters =
|
||||
lib.optionalAttrs vmCfg.enable {
|
||||
|
|
@ -263,16 +434,64 @@ in
|
|||
}
|
||||
];
|
||||
|
||||
service.pipelines.metrics = {
|
||||
receivers = [ "otlp" ];
|
||||
# Fan-out, not a choice: with both configured the same
|
||||
# samples go upstream AND into the swarm's store. The store
|
||||
# is for looking at this swarm; the upstream is for whoever
|
||||
# aggregates across swarms, and neither replaces the other.
|
||||
exporters =
|
||||
lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp")
|
||||
++ lib.optional vmCfg.enable "otlphttp/victoriametrics";
|
||||
};
|
||||
# ⚠️ An extension that is configured but not listed here is
|
||||
# INERT — the collector starts clean and the receiver
|
||||
# naming it authenticates nothing. Derived from the same
|
||||
# attrset as the receivers so the two cannot disagree.
|
||||
service.extensions = map (h: "oidc/${h}") (lib.attrNames hivePorts);
|
||||
|
||||
# Fan-out, not a choice: with both configured the same
|
||||
# samples go upstream AND into the swarm's store. The store
|
||||
# is for looking at this swarm; the upstream is for whoever
|
||||
# aggregates across swarms, and neither replaces the other.
|
||||
# `exporterNames` is shared by every pipeline — where a
|
||||
# sample goes is a property of this tier, not of the hive
|
||||
# that sent it.
|
||||
service.pipelines = lib.mapAttrs' (
|
||||
h: _:
|
||||
lib.nameValuePair "metrics/${h}" {
|
||||
receivers = [ "otlp/${h}" ];
|
||||
processors = [ "resource/${h}" ];
|
||||
exporters = exporterNames;
|
||||
}
|
||||
) hivePorts;
|
||||
}
|
||||
// {
|
||||
extensions = lib.mapAttrs' (
|
||||
h: _:
|
||||
lib.nameValuePair "oidc/${h}" {
|
||||
issuer_url = autheliaCfg.url;
|
||||
# The audience this hive's client is registered to
|
||||
# request, and the reason one hive's token is refused by
|
||||
# another hive's receiver. Same expression authelia
|
||||
# registers it under — a second spelling here would deny
|
||||
# every hive, as a 401 that blames the token.
|
||||
audience = "${autheliaCfg.hiveClientPrefix}${h}";
|
||||
# ⚠️ `issuer_ca_path`. `issuer_ca_file`, `ca_file` and
|
||||
# `tls.ca_file` are all INVALID KEYS for this extension
|
||||
# — measured, and the failure is a startup error naming
|
||||
# the key rather than anything about certificates.
|
||||
issuer_ca_path = caBundle;
|
||||
}
|
||||
) hivePorts;
|
||||
|
||||
# `upsert`, not `insert`: a sender that stamps its own
|
||||
# `hive` must be OVERWRITTEN, not deferred to. This
|
||||
# processor is the whole attribution boundary — the value
|
||||
# is a constant per receiver, so it says which hive
|
||||
# authenticated, not which hive claimed to be sending.
|
||||
processors = lib.mapAttrs' (
|
||||
h: _:
|
||||
lib.nameValuePair "resource/${h}" {
|
||||
attributes = [
|
||||
{
|
||||
key = "hive";
|
||||
value = h;
|
||||
action = "upsert";
|
||||
}
|
||||
];
|
||||
}
|
||||
) hivePorts;
|
||||
};
|
||||
};
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue