vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
hyperhive/nix/module-eval.nix
atlas 01ce968fb6 nix: split statusPublish and the otel secret into deploy.* 
Slices 8 and 9 of the swarm/deploy split, and the last two.

statusPublish had three coordinates under one namespace. Two of them
are this machine's — where the queue listens *as seen from here*, and
where its client secret sits on this disk — so they move to
`deploy.hive-controller.statusPublish.*`, the namespace of the daemon
that is their only reader. `tokenEndpoint` is the swarm's one address,
so it stays. That leaves `swarm.statusPublish` holding a single option:
a legitimate split, not a botched move.

The all-or-nothing assertion now spans both namespaces. It is repointed
in both its condition and its message, and the message spells all three
paths in full so an operator is never told to set two options under a
path that only has one. `environment.nix`'s guard and the value beside
it likewise read different namespaces on purpose.

The collector's secret moves the same way, for the same reason, to
`deploy.swarm-otel.*` — `enable` already lives there. That also retires
one of the eight cross-namespace assignments tracked in #4048: the
delivery unit set a `swarm.*` value under a `deploy.*` gate, and now
sets a `deploy.*` value under one.

module-eval gets a fixture per slice. `otelRemoteAuthelia` already set
the collector secret through its pre-rename path, so it becomes slice
9's old-path case as it stands — left spelled that way deliberately,
with a comment, so it is not read later as a missed site.

That fixture also turned out to be describing an impossible hive: it
said authelia lives elsewhere without saying where, so the
authenticator interpolated a null `swarm.authelia.url` into its
`token_url`. Nothing to do with the rename, and invisible to the
existing case over the same fixture, which reads `? auth` and `elem` —
both stop at names and never force the extension's value. Given the
address a remote-IdP deployment has.

Verified: 49 -> 51 properties, all holding.
2026-09-07 16:54:23 +02:00

899 lines
43 KiB
Nix
Raw Blame History

# `checks.module-eval` — the flake check that covers **nix**.
#
# Every other check in ./checks.nix is a Rust derivation, so a `.nix`-only
# diff moves no hash and `nix flake check` goes green **without evaluating
# what changed**. This derivation's hash is a function of the results
# below, so a change that flips a property rebuilds it and fails naming it.
#
# Anything expressible as a module `assertion` **should be one instead** —
# an assertion fires at deploy time for a real operator, not only in CI.
# What cannot is the **absence class**: "a hive that hasn't opted in
# renders what it did before", "this unit does not exist unless X" — claims
# about the rendered config rather than about a config being invalid.
#
# ⚠️ **Name a case for the PROPERTY it defends, never the ticket that
# prompted it**; a case named after a ticket has the ticket's lifetime.
#
# ⚠️ **This evaluates, it does not execute** — a command line, a request or
# a certificate needs something that *runs* it. A case needing a rendered
# file must stub what it drags in (`swarm.ui.package = pkgs.emptyDirectory`).
{
pkgs,
lib,
self,
nixosSystem,
}:
let
# Stub host, same shape ./docs/default.nix already uses: enough for a
# `nixosSystem` to evaluate, nothing that pulls a real disk or
# bootloader in.
hive =
extra:
(nixosSystem {
system = pkgs.stdenv.hostPlatform.system;
modules = [
self.nixosModules.default
{
fileSystems."/" = {
device = "/dev/null";
fsType = "tmpfs";
};
boot.loader.grub.enable = false;
system.stateVersion = "25.11";
# `recursiveUpdate`, not `//`: a plain `//` only merges the
# *top-level* keys of `extra` in, so an `extra` that touches a
# nested attr under an existing top-level key (e.g. `swarm.*`)
# silently drops every sibling under that key instead of merging
# into it — the same class of bug as the `services.hyperhive`
# double-nesting mistake this stub already had to dodge once,
# just one level down. `recursiveUpdate` merges nested attrsets
# all the way down instead.
services.hyperhive = lib.recursiveUpdate {
enable = true;
hiveName = "h1";
swarm.domain = "t.local";
swarm.hives.h1.domain = "h1.t.local";
} extra;
}
];
}).config;
allLocal = hive { deploy.singleHostSwarm = true; };
bare = hive { };
withCi = hive { deploy.forgejo.ci.enable = true; };
# A host configured against the pre-rename option path. `mkRenamedOptionModule`
# is the only thing carrying it, and nothing else in this suite would notice
# if it were dropped: the new path evaluates fine on its own, so a missing
# shim reads as a clean tree and breaks every existing operator config.
wireguardOldPath = hive {
swarm.wireguard.enable = true;
swarm.wireguard.address = "10.100.0.1/24";
swarm.wireguard.privateKeyFile = "/etc/wireguard/hive.key";
};
# Same shape for the forge, which SPLIT rather than moving whole: the five
# host-side options are set here through their pre-rename paths while the
# rest of `swarm.forge` stays put. All five are defined so that dropping any
# single shim entry fails the eval, not just the two the assertion reads.
forgeOldPath = hive {
swarm.forge.behindGateway = true;
swarm.forge.openFirewall = true;
swarm.forge.hostSwarmControllerTokenFile = "/etc/forge/sc.token";
swarm.forge.sso.clientSecretFile = "/etc/forge/oidc.secret";
swarm.forge.mirrors = [
{
upstream = "https://example.invalid/tool";
dest = "mirrors/tool";
}
];
};
# The homeserver's turn to split. All six host-side options are set through
# their pre-rename paths — including `gui.enable`, whose value is deliberately
# the opposite of its default so the definition has to actually land. All six
# so that dropping any single shim entry fails the eval, not just the two the
# assertion reads.
matrixOldPath = hive {
deploy.matrix.enable = true;
swarm.matrix.openFirewall = true;
swarm.matrix.trustedServers = [ "matrix.example.invalid" ];
swarm.matrix.maxRequestSize = 31457280;
swarm.matrix.registrationTokenFile = "/etc/matrix/register.token";
swarm.matrix.gui.enable = false;
swarm.matrix.sso.clientSecretFile = "/etc/matrix/oidc.secret";
};
# The queue's callout identity, fourth split slice. `autoGenerateCallout` is
# left FALSE on purpose: that is what makes the seed paths the thing deciding
# `responderConfigured`, so the assertion below is about the seeds rather
# than about the auto-mint branch. All five old paths are defined, so
# dropping any single shim entry fails the eval, not just the arms read.
natsOldPath = hive {
deploy.nats.enable = true;
swarm.nats.autoGenerateCallout = false;
swarm.nats.calloutUserPublicKey = "UTESTUSERPUBKEYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
swarm.nats.calloutIssuerPublicKey = "ATESTISSUERPUBKEYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
swarm.nats.calloutUserSeedFile = "/run/secrets/nats-user.seed";
swarm.nats.calloutIssuerSeedFile = "/run/secrets/nats-issuer.seed";
};
# Seventh split slice. Only `usersFile` has a rename entry: the other two
# movers are `readOnly`, and a rename module contributes a definition, which
# a read-only option refuses — see ./host-modules/deploy.nix. So the two arms
# below have different jobs. `usersFile` tests the rename; the nats one tests
# that a reader repointed to the new namespace still renders the derived
# path, which is the failure this slice could actually have shipped — seven
# of those reads went through an alias a path-shaped grep cannot see.
autheliaOldPath = hive {
deploy.authelia.enable = true;
deploy.nats.enable = true;
swarm.authelia.usersFile = "/var/lib/test-authelia/users.yml";
swarm.nats.autoGenerateCallout = false;
swarm.nats.calloutUserPublicKey = "UTESTUSERPUBKEYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
swarm.nats.calloutIssuerPublicKey = "ATESTISSUERPUBKEYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
swarm.nats.calloutUserSeedFile = "/run/secrets/nats-user.seed";
swarm.nats.calloutIssuerSeedFile = "/run/secrets/nats-issuer.seed";
};
# A deliberately odd `maxRequestSize`, so the number the assertion looks for
# cannot have come from a default or from another module's literal.
matrixBodyCap = hive {
deploy.singleHostSwarm = true;
deploy.matrix.maxRequestSize = 99000000;
};
controllerOldPath = hive {
deploy.swarm-controller.enable = true;
swarm.controller.socketPath = "/run/test-ctrl/ctrl.sock";
swarm.controller.forgeTokenFile = "/run/secrets/ctrl-forge.token";
swarm.controller.authBridgeUrl = "http://127.0.0.1:19097";
swarm.controller.queue.clientSecretFile = "/run/secrets/ctrl-queue.secret";
};
grafanaOldPath = hive {
deploy.grafana.enable = true;
swarm.grafana.socketDir = "/run/test-grafana-sock";
swarm.grafana.datasourceUrl = "http://127.0.0.1:19999";
swarm.grafana.logsDatasourceUrl = "http://127.0.0.1:19998";
swarm.grafana.plugins = [ ];
};
# The first slice to leave options on BOTH sides of the split, so the
# fixture sets all three of them through the paths an existing config uses:
# the two movers via their rename entries, `tokenEndpoint` via the path it
# kept. That is also what the all-or-nothing assertion wants, so this hive
# is a valid one rather than one that only evaluates because nothing forced
# the assertion.
statusPublishOldPath = hive {
swarm.statusPublish.natsUrl = "nats://10.0.0.9:4222";
swarm.statusPublish.tokenEndpoint = "https://auth.example.invalid/api/oidc/token";
swarm.statusPublish.clientSecretFile = "/run/secrets/status-client.secret";
};
baoPkcs11 = hive {
deploy.bao.enable = true;
deploy.bao.seal = "pkcs11";
};
baoShamir = hive {
deploy.bao.enable = true;
deploy.bao.seal = "shamir";
};
baoExplicitCerts = hive {
deploy.bao.enable = true;
deploy.bao.serverCertFile = "/etc/pki/bao.pem";
deploy.bao.serverKeyFile = "/etc/pki/bao-key.pem";
};
# The store and a service that reads from it, versus the store alone. The
# pair is what makes the reader's absence arm mean anything.
baoWithMatrix = hive {
deploy.bao.enable = true;
deploy.matrix.enable = true;
};
# A hive that reads from a store it does not run: no `deploy.bao.enable`, so
# nothing here mints a leaf and the operator names one placed by hand. The
# deployment this pairing exists to serve, and the one that was previously
# inexpressible — the gate asked whether the store was a neighbour.
baoRemoteReader = hive {
deploy.matrix.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# The same hive with the identity taken away, which separates "a homeserver
# is deployed" from "this host can authenticate to the store".
matrixNoBaoIdentity = hive { deploy.matrix.enable = true; };
baoNames = machine: machine.services.hyperhive.gateway.localNames;
baoTwoAddresses = hive {
deploy.bao.enable = true;
deploy.bao.extraListenAddresses = [ "10.0.0.1" ];
# Pinned, not incidental: the case counting these listeners is about the
# declared addresses, and a collector on this host would add one of its own.
deploy.swarm-otel.enable = false;
};
# The store with and without a collector on the same host. `scrapeTargets`
# is only ever read by a local collector, so the metrics endpoint is a
# function of the pairing rather than of the store.
baoWithCollector = hive {
deploy.bao.enable = true;
deploy.swarm-otel.enable = true;
# A second job declared as bare `host:port`, so the pair of cases below
# reads one rendered scrape list: the store's entry carries a path, this
# one carries none.
swarm.otel.scrapeTargets.plain = "127.0.0.1:9999";
};
baoNoCollector = hive {
deploy.bao.enable = true;
deploy.swarm-otel.enable = false;
};
# The config file openbao parses, not the nix that produces it: a setting it
# requires is absent here without anything in the module system minding, so
# the daemon's own startup is otherwise the first reader.
baoSettings = machine: machine.containers.swarm-bao.config.services.openbao.settings;
# The store's units live inside its container, so the gates below have to
# look there rather than at the host's service set.
baoUnits = machine: machine.containers.swarm-bao.config.systemd.services;
# The scrape list prometheus is handed, not the option a service declared:
# the address, the path and the query are one string on the way in and three
# fields on the way out, and only the second shape is what gets requested.
scrapeJob =
machine: job:
lib.findFirst (c: c.job_name == job) null
machine.containers.swarm-otel.config.services.opentelemetry-collector.settings.receivers.prometheus.config.scrape_configs;
# A swarm collector on a host that runs NEITHER store — the fully-spread
# shape from docs/swarm/services.md, and the one the old per-host gates made
# inexpressible. It is the whole point of the cases below that this hive is
# not a degenerate configuration but a supported one.
otelNoStores = hive {
deploy.swarm-otel.enable = true;
deploy.authelia.enable = true;
deploy.victoriametrics.enable = false;
deploy.victorialogs.enable = false;
};
otelSettings =
machine: machine.containers.swarm-otel.config.services.opentelemetry-collector.settings;
# authelia somewhere else, the credential delivered by hand. Whether this
# collector authenticates must follow the credential, never another
# service's placement.
#
# The `swarm.otel.clientSecretFile` below is the PRE-RENAME path. It predates
# the split and is deliberately left spelled that way: it makes this fixture
# the old-path case for that option too, so dropping its rename entry fails
# the eval here rather than only in a real operator's config.
otelRemoteAuthelia = hive {
deploy.swarm-otel.enable = true;
deploy.authelia.enable = false;
# Where that elsewhere IS. Running no IdP does not mean knowing no IdP:
# the authenticator this fixture exists to render puts this address in its
# `token_url`, so a hive with a secret and no URL has a credential it can
# present nowhere.
swarm.authelia.url = "https://auth.example.invalid";
swarm.otel.clientSecretFile = "/var/lib/swarm-otel-oidc/by-hand.secret";
};
# A priority collision is a property of the *option*, not
# of the merged value's interior — nix throws the moment the value is
# demanded at all, so `seq`-ing each `serviceConfig` value to WHNF is
# both necessary and sufficient. `deepSeq` over-specifies this: it keeps
# walking *into* the resulting value after the merge already succeeded,
# and a package/derivation-shaped value's `override`/`overrideAttrs`
# self-reference sends it into nixpkgs' fixpoint machinery and blows the
# stack (measured — this is not a hypothetical).
forceCiServiceConfigs =
let
svcs = withCi.containers.hive-ci.config.systemd.services;
vals = lib.concatMap (s: builtins.attrValues (s.serviceConfig or { })) (builtins.attrValues svcs);
in
builtins.foldl' (acc: v: builtins.seq v acc) true vals;
# Each case: a name stating the property, and `ok`.
cases = [
{
name = "a hive that has not opted into all-local runs no swarm controller";
ok = !bare.services.hyperhive.deploy.swarm-controller.enable;
}
{
name = "the all-local mode turns the swarm controller on";
ok = allLocal.services.hyperhive.deploy.swarm-controller.enable;
}
{
# The mesh moved namespace wholesale, so an existing config sets paths
# that no longer exist. Reading the *rendered interface* rather than the
# option: a rename that resolved but stopped reaching the module would
# satisfy an option-level check and still bring up no tunnel.
name = "a config written against the pre-rename wireguard path still configures the interface";
ok =
let
wg = wireguardOldPath.networking.wireguard.interfaces.wg-hive;
in
wg.ips == [ "10.100.0.1/24" ] && wg.privateKeyFile == "/etc/wireguard/hive.key";
}
{
# Same reasoning one namespace over, plus the shape the mesh did not
# have: `mirrors` is a single option of a list-of-submodule type, so its
# one rename entry has to carry a whole compound value rather than a
# scalar. Both arms read a rendered effect — the host firewall and the
# env var c0re seeds mirrors from — not the option.
name = "a config written against the pre-rename forge paths still opens the firewall and seeds the mirror";
ok =
let
ports = forgeOldPath.networking.firewall.allowedTCPPorts;
seeded = builtins.fromJSON forgeOldPath.systemd.services.hive-c0re.environment.HYPERHIVE_FORGE_MIRRORS;
in
builtins.elem 3000 ports && builtins.any (m: m.dest == "mirrors/tool") seeded;
}
{
# Third split, and the one whose readers were hardest to see: the
# registration token is read only through a `let` alias in another
# module, so no full path names it anywhere. Both arms read a rendered
# effect — the host firewall and the container's bind-mount table — so a
# rename that resolves but stops reaching the module still fails.
name = "a config written against the pre-rename matrix paths still opens the port and mounts the token";
ok =
let
ports = matrixOldPath.networking.firewall.allowedTCPPorts;
httpPort = matrixOldPath.services.hyperhive.swarm.matrix.httpPort;
in
builtins.elem httpPort ports
&& matrixOldPath.containers.hive-matrix.bindMounts ? "/etc/matrix/register.token";
}
{
# registrationTokenFile lost its override capability entirely
# (bao-delivered secrets don't need one — glue-matrix-bao-token.nix
# already writes into the fixed path instead of moving it), unlike its
# five siblings in the same rename. `matrixOldPath` above proves the
# override still *resolves* (mkDefault, not a crash) — this proves it
# also gets *rejected*, by a named assertion rather than nixpkgs' generic
# conflicting-definition text. Reads `.assertions` directly (cheap: a
# list of `{assertion; message;}`, not `system.build.toplevel`) rather
# than forcing a real build just to observe a boolean.
name = "overriding registrationTokenFile (even via the pre-rename shim) trips a named assertion, not a silent desync";
ok = lib.any (
a: !a.assertion && lib.hasInfix "registrationTokenFile" a.message
) matrixOldPath.assertions;
}
{
# Reads the DELIVERY UNIT, not the options: `responderConfigured` gates
# whether it exists at all, and the seed path is interpolated into its
# script. A rename that resolved but stopped reaching the module would
# leave the responder with no credentials and this case would catch it.
name = "a config written against the pre-rename nats paths still delivers the responder's seeds";
ok =
let
units = natsOldPath.systemd.services;
in
units ? swarm-nats-auth-secrets
&& lib.hasInfix "/run/secrets/nats-user.seed" units.swarm-nats-auth-secrets.script;
}
{
# Reads the host's tmpfiles rules, not the options: the socket directory
# nginx and the container share is created there, so a rename that
# resolved but stopped reaching the module would leave the gateway
# proxying to a path nothing creates. All four old paths are defined in
# the fixture, so removing any single shim fails the eval rather than
# only the one this assertion reads.
# All FOUR movers are asserted as rendered effects rather than as option
# values, so a rename that resolved but stopped reaching the module is
# caught per-option instead of only where one assertion happens to look.
name = "a config written against the pre-rename swarm-controller paths still reaches the unit";
ok =
let
u = controllerOldPath.systemd.services.swarm-controller;
creds = u.serviceConfig.LoadCredential;
in
u.environment.SWARM_CONTROLLER_SOCKET == "/run/test-ctrl/ctrl.sock"
&& u.environment.SWARM_CONTROLLER_AUTH_BRIDGE_URL == "http://127.0.0.1:19097"
&& lib.any (c: lib.hasInfix "/run/secrets/ctrl-queue.secret" c) creds
&& lib.any (c: lib.hasInfix "/run/secrets/ctrl-forge.token" c) creds;
}
{
name = "a config written against the pre-rename grafana paths still creates the socket directory";
ok = lib.any (
rule: lib.hasInfix "/run/test-grafana-sock" rule
) grafanaOldPath.systemd.tmpfiles.rules;
}
{
# Reads the daemon's rendered unit, not the options: the queue address
# arrives as an env var whose whole attrset is guarded on `natsUrl`, and
# the secret as a systemd credential. A rename that resolved but stopped
# reaching the module leaves hive-c0re coming up perfectly and reporting
# to nobody, which is the one failure this option set exists to prevent.
# The third arm is the option that did NOT move, read out of the same
# attrset: the guard and the value beside it now come from different
# namespaces, so a hive that renders one and drops the other is exactly
# what a split can silently produce.
name = "a config written against the pre-rename statusPublish paths still reaches the daemon";
ok =
let
u = statusPublishOldPath.systemd.services.hive-c0re;
in
u.environment.HIVE_C0RE_NATS_URL == "nats://10.0.0.9:4222"
&& u.environment.HIVE_C0RE_OIDC_TOKEN_ENDPOINT == "https://auth.example.invalid/api/oidc/token"
&& lib.any (c: lib.hasInfix "/run/secrets/status-client.secret" c) u.serviceConfig.LoadCredential;
}
{
# The collector's half of the same split, and a different arm from the
# authenticator case below: this one reads the PATH the unit loads, so a
# reader left on a source that is non-null but wrong still fails. The
# fixture spells the option its pre-rename way, so it covers the rename
# entry at the same time.
name = "a config written against the pre-rename otel secret path still loads it as a credential";
ok =
lib.any (c: lib.hasInfix "/var/lib/swarm-otel-oidc/by-hand.secret" c)
otelRemoteAuthelia.containers.swarm-otel.config.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential;
}
{
name = "a config written against the pre-rename authelia usersFile still reaches the bridge";
ok =
autheliaOldPath.containers.swarm-authelia.config.systemd.services.swarm-authelia-bridge.environment.SWARM_AUTHELIA_BRIDGE_USERS_FILE
== "/var/lib/test-authelia/users.yml";
}
{
# Not a rename test. `hostClientSecretDir` is `readOnly`, so the fixture
# cannot define it; what can break is a reader left pointing at the
# namespace it moved out of. Five modules read this through an
# `autheliaCfg` alias, where a path-shaped grep does not see it.
name = "a consumer of authelia's host client-secret dir renders it from the deploy namespace";
ok = lib.hasInfix "/var/lib/nixos-containers/swarm-authelia/var/lib/authelia-swarm/oidc-clients/" autheliaOldPath.systemd.services.swarm-nats-auth-secrets.script;
}
{
# The gateway's per-name issuer choice. If this ever collapses to a
# constant, every swarm-service vhost serves a certificate its CA
# is name-constrained out of — which evaluates cleanly and fails in
# a browser.
name = "a swarm service name gets the swarm-services leaf and the default server does not";
ok =
let
l = allLocal.services.hyperhive.gateway.lib;
in
(l.tlsFor "t.local").sslCertificate != (l.tlsFor "_").sslCertificate;
}
{
# nixos asserts when a vhost declares both, so this is also a
# statement that the `removeAttrs` upstream of it still happens.
name = "the swarm UI vhost forces TLS instead of merely adding it";
ok =
let
v = allLocal.services.nginx.virtualHosts."t.local";
in
v.forceSSL && !(v.addSSL or false);
}
{
name = "a hive with matrix off serves no matrix discovery endpoint";
ok =
!(builtins.hasAttr "= /.well-known/matrix/client" bare.services.nginx.virtualHosts."_".locations);
}
{
# main got eval-borked twice by this exact class of bug (once on the
# unit's `Restart` key, once on `RestartSec`) — a nixpkgs bump to
# `gitea-actions-runner.nix` adds a plain `serviceConfig.*`
# definition that collides with one of ours, and nix refuses to
# merge two plain definitions at *host* eval. No other check
# instantiates a host with `containers.hive-ci` actually enabled, so
# the collision only surfaces on operator deploy, not in CI.
name = "the CI container's unit definitions merge without a priority collision";
ok = forceCiServiceConfigs;
}
{
# The defect itself. These exporters used to be gated on the stores'
# PER-HOST enables, so a collector that did not share a host with them
# rendered none at all and dropped everything it received, from every
# hive — silently, because an absent exporter is not an error.
name = "a collector that hosts neither store still exports to both";
ok =
let
e = (otelSettings otelNoStores).exporters;
in
(e ? "otlphttp/victoriametrics") && (e ? "otlphttp/victorialogs");
}
{
# A swarm has one of each store, so the address is a swarm-level name.
# A loopback literal here is the co-location assumption written back in,
# and it renders, deploys and reports healthy while reaching nothing.
name = "the store exporters address the stores by name, never by loopback";
ok =
let
e = (otelSettings otelNoStores).exporters;
m = e."otlphttp/victoriametrics".metrics_endpoint;
l = e."otlphttp/victorialogs".logs_endpoint;
in
!(lib.hasInfix "127.0.0.1" m)
&& !(lib.hasInfix "127.0.0.1" l)
&& lib.hasInfix "metrics.t.local" m
&& lib.hasInfix "logs.t.local" l;
}
{
# The collector reaches these routes through the gateway now, so each
# store needs an ingest location of its own. Without one the write rides
# the `/` catch-all: unauthenticated on the metrics store, and into a
# browser redirect on the log store.
name = "each store's vhost has an authenticated ingest location";
ok =
let
v = allLocal.services.nginx.virtualHosts;
m = v."metrics.t.local".locations."= /opentelemetry/api/v1/push" or null;
l = v."logs.t.local".locations."= /insert/opentelemetry/v1/logs" or null;
in
m != null
&& l != null
&& lib.hasInfix "auth_request" m.extraConfig
&& lib.hasInfix "auth_request" l.extraConfig;
}
{
# Two limits governed a matrix upload and nothing kept them in agreement,
# so raising the documented one past the gateway's hardcoded cap changed
# nothing. The second clause is the control: the old directive is gone, so
# a pass means the value is derived rather than that `hasInfix` matched
# something incidental. It targets the whole directive rather than the
# bare size, because the rendered config carries the comment above it and
# a prose mention of the old value would defeat a looser arm.
name = "the matrix gateway's body cap is derived from maxRequestSize, not a literal";
ok =
let
c = matrixBodyCap.services.nginx.virtualHosts."chat.t.local".locations."/_matrix/".extraConfig;
in
lib.hasInfix "client_max_body_size 100048576;" c && !(lib.hasInfix "client_max_body_size 50M" c);
}
{
# The arm that actually protects something. A pusher handed
# `error_page 401 =302` FOLLOWS it and POSTs its batch at a login page,
# which answers 200 — ingest reporting healthy while storing nothing.
# The third clause is the positive control: the log store's browser
# location really does redirect, so this says the machine routes differ
# rather than that the string is absent from the whole file.
name = "the ingest locations answer 401 instead of redirecting a pusher";
ok =
let
v = allLocal.services.nginx.virtualHosts;
m = v."metrics.t.local".locations."= /opentelemetry/api/v1/push".extraConfig;
l = v."logs.t.local".locations."= /insert/opentelemetry/v1/logs".extraConfig;
browser = v."logs.t.local".locations."/".extraConfig;
in
!(lib.hasInfix "error_page" m)
&& !(lib.hasInfix "error_page" l)
&& lib.hasInfix "error_page" browser;
}
{
# Defining an exporter and REFERENCING it are two separate lists, and
# the second is where the original gate also lived. An exporter no
# pipeline names is as silent as one that does not exist — this case
# exists because a mutation that restored only the reference-side gate
# left every other case here green.
name = "every pipeline that has a store exporter defined actually sends to it";
ok =
let
s = otelSettings otelNoStores;
used = lib.unique (lib.concatMap (p: p.exporters) (lib.attrValues s.service.pipelines));
in
builtins.elem "otlphttp/victoriametrics" used && builtins.elem "otlphttp/victorialogs" used;
}
{
# The collector authenticates because it HOLDS a credential, not because
# authelia happens to share its host. Gating on the other service's
# placement renders a collector that pushes unauthenticated wherever
# authelia lives elsewhere — one of the supported shapes.
name = "a collector with a hand-delivered secret authenticates without authelia beside it";
ok =
let
s = otelSettings otelRemoteAuthelia;
in
(s.exporters."otlphttp/victoriametrics" ? auth)
&& builtins.elem "oauth2client/victoriametrics" s.service.extensions;
}
{
# An authenticator an exporter names but `service.extensions` omits is
# INERT — the collector starts clean and pushes unauthenticated until
# something at the far end refuses it. Checked as a set relation rather
# than by naming the two, so it keeps holding for exporters not written
# yet.
name = "every exporter authenticator is listed in service.extensions";
ok =
let
s = otelSettings otelNoStores;
named = lib.filter (v: v != null) (
lib.mapAttrsToList (_: e: e.auth.authenticator or null) s.exporters
);
in
named != [ ] && lib.all (a: builtins.elem a s.service.extensions) named;
}
{
# The store's seal is spread over six gates — the stanza, the
# provisioning unit, two bind mounts, a device and an EnvironmentFile.
# Rendering only some of them is the dangerous state: a store that
# says hardware-backed and seals with a software key, which no
# assertion can catch because every value is individually valid.
name = "a shamir store renders no TPM provisioning unit";
ok = !(baoUnits baoShamir ? swarm-bao-token);
}
{
# Presence control for the case above. Without it, a typo in the
# option name would satisfy the absence arm forever. The second half is
# the fix itself: the unit has to run where openbao's `DynamicUser` is
# allocated, and a host unit writing the same bytes has no name to hand
# them to.
name = "a pkcs11 store provisions the token in the container, not on the host";
ok = (baoUnits baoPkcs11 ? swarm-bao-token) && !(baoPkcs11.systemd.services ? swarm-bao-token);
}
{
# `allowedDevices` renders `DeviceAllow=` and nothing else — nspawn
# mounts its own /dev and cannot create device nodes, so permission to
# use a device that was never bound in opens nothing. Neither half
# fails on its own, which is why they are asserted as a pair.
name = "a pkcs11 store gets the TPM device bound in, not merely allowed";
ok =
let
c = baoPkcs11.containers.swarm-bao;
in
(c.bindMounts ? "/dev/tpmrm0") && builtins.any (d: d.node == "/dev/tpmrm0") c.allowedDevices;
}
{
# The stanza's label and the label the unit creates are two literals that
# have to name one object, and the mechanism is asserted with them
# because it is valid only for an RSA key: openbao takes AES-GCM or
# RSA-OAEP and a TPM 2.0 has neither GCM nor an opinion about which the
# seal asked for. Every wrong combination renders and deploys, and
# surfaces as a pkcs11 error at `operator init`.
name = "the seal asks for the RSA key the provisioning unit creates";
ok =
let
p = (baoSettings baoPkcs11).seal.pkcs11;
in
(p.mechanism or "") == "CKM_RSA_PKCS_OAEP"
&& lib.hasInfix "--algorithm=rsa2048 --key-label=${p.key_label or ""}" (
(baoUnits baoPkcs11).swarm-bao-token.script or ""
);
}
{
# `DynamicUser` implies `ProtectSystem=strict`, so the token directory is
# read-only to the seal however it is owned, and the group is the only
# handle on a uid allocated at start. Dropping either surfaces as a
# pkcs11 error deep in a library, naming neither the mount nor the user.
name = "the store's seal may write the token directory, and is in both its groups";
ok =
let
sc = (baoUnits baoPkcs11).openbao.serviceConfig;
in
# `or [ ]` rather than a bare select: the interesting mutation is the
# key being gone, and a select would abort the whole run with a nix
# trace instead of failing this case by name.
builtins.elem "/var/lib/swarm-bao-token" (sc.ReadWritePaths or [ ])
&& builtins.elem "swarm-bao-token" (sc.SupplementaryGroups or [ ])
&& builtins.elem "swarm-bao-tpm" (sc.SupplementaryGroups or [ ]);
}
{
# The device node belongs to the HOST and is matched by NUMBER, while the
# unit that opens it lives in the container — so the two sides holding
# the same gid is the entire mechanism. Letting either side auto-allocate
# renders cleanly, deploys cleanly, and leaves a 0660 node the seal
# cannot open. Compared rather than each checked against a literal: the
# property is that they AGREE, not what they agree on.
name = "the TPM group has the same gid on the host and inside the container";
ok =
let
host = baoPkcs11.users.groups.swarm-bao-tpm.gid or null;
inner = baoPkcs11.containers.swarm-bao.config.users.groups.swarm-bao-tpm.gid or null;
in
host != null && host == inner;
}
{
# Absence arm for the case above — a shamir store never opens a TPM, so
# it must not claim a device node's group. Without this, pinning the gid
# unconditionally would look identical.
name = "a shamir store claims no TPM device group";
ok = !(baoShamir.users.groups ? swarm-bao-tpm);
}
{
# The store's mTLS identity is a separate trust domain from both CAs in
# this tree, because it must not come from an authority the store will
# itself distribute. What supplies it is the glue, which mints a CA of
# the store's own — so an enabled store has all three paths, and if this
# ever reads null again the store stops coming up on its own.
name = "a deployed store is given its own certificate, key and client CA";
ok =
let
b = baoPkcs11.services.hyperhive.deploy.bao;
in
b.serverCertFile != null && b.serverKeyFile != null && b.clientCaFile != null;
}
{
# Everything the glue sets is `mkDefault`, and this is the case that
# says so: a deployment whose certificates come from somewhere the glue
# has never heard of must win. Also the presence control for the case
# above — a renamed option would read `null` on both and satisfy
# neither, but only this one names a value.
name = "an operator's own certificate path beats the glue's default";
ok = baoExplicitCerts.services.hyperhive.deploy.bao.serverCertFile == "/etc/pki/bao.pem";
}
{
# The store's first reader. Its unit belongs to the pairing, not to
# either service: matrix must not learn the store exists, and the store
# must not know who reads it.
name = "a store deployed beside the homeserver fetches its registration token";
ok = baoWithMatrix.systemd.services ? swarm-bao-matrix-token;
}
{
# Absence arm. A store with nothing to serve renders no reader, so the
# unit is a function of the PAIRING rather than of the store — which is
# the property that makes it glue instead of a feature of either side.
name = "a store with no homeserver beside it renders no token reader";
ok = !(baoPkcs11.systemd.services ? swarm-bao-matrix-token);
}
{
name = "a hive that names a client identity reads from a store it does not run";
ok = baoRemoteReader.systemd.services ? swarm-bao-matrix-token;
}
{
# Absence arm for the one above, and the reason the gate is the identity
# rather than the homeserver: without it, deploying matrix anywhere would
# render a reader that cannot authenticate.
name = "a homeserver with no way to authenticate to the store renders no token reader";
ok = !(matrixNoBaoIdentity.systemd.services ? swarm-bao-matrix-token);
}
{
# `Requires=` on a unit that does not exist fails the job, and nothing
# local mints certificates off-host — so this orders against nothing.
# Eval cannot see that failure; only the empty list here stands in for it.
name = "an off-host reader requires no unit the store's host would have provided";
# Membership first, then the value: indexing a missing unit throws, and a
# table that reports which property broke must not be the thing that dies.
ok =
let
s = baoRemoteReader.systemd.services;
in
s ? swarm-bao-matrix-token && s.swarm-bao-matrix-token.requires == [ ];
}
{
# Presence control for the case above: the list is conditional, not gone.
name = "a co-located reader still orders after the local pki unit";
ok =
let
s = baoWithMatrix.systemd.services;
in
s ? swarm-bao-matrix-token && s.swarm-bao-matrix-token.requires == [ "swarm-bao-pki.service" ];
}
{
# hive-c0re runs as hive-core and the client key is `0600` root-owned
# inside a `0700` directory, so the identity reaches the daemon as a
# systemd credential and the environment names `%d` rather than the
# file. Both halves are asserted together because either alone is a
# daemon that fails at the TLS handshake, naming neither.
name = "a reader hands hive-c0re a store identity the daemon cannot open itself";
ok =
let
s = baoRemoteReader.systemd.services;
in
s ? hive-c0re
&& (s.hive-c0re.environment.BAO_CLIENT_CERT or null) == "%d/bao-client.pem"
&& (s.hive-c0re.environment.BAO_CLIENT_KEY or null) == "%d/bao-client-key.pem"
&& builtins.elem "bao-client.pem:/etc/pki/bao-client.pem" s.hive-c0re.serviceConfig.LoadCredential
&& builtins.elem "bao-client-key.pem:/etc/pki/bao-client-key.pem" s.hive-c0re.serviceConfig.LoadCredential;
}
{
# Absence arm for the case above. A hive with no client identity gets no
# store environment at all — the daemon reports a queue it cannot serve
# rather than a handshake it cannot explain.
name = "a hive with no client identity gives hive-c0re no store environment";
ok =
let
s = matrixNoBaoIdentity.systemd.services;
in
s ? hive-c0re
&& !(s.hive-c0re.environment ? BAO_ADDR)
&& !(lib.any (c: lib.hasPrefix "bao-" c) s.hive-c0re.serviceConfig.LoadCredential);
}
{
# The CA is its own arm: absent means the system trust store, which is
# right for a deployment with a real CA and wrong for a self-signed one.
name = "a reader that names no store CA falls through to the system trust store";
ok =
let
s = baoRemoteReader.systemd.services;
in
s ? hive-c0re && !(s.hive-c0re.environment ? BAO_CACERT);
}
{
# Presence control for the arm above: the CA is conditional, not gone.
# Co-located, ./host-modules/glue-bao-tls.nix mints one and names it.
name = "a reader beside a self-signed store is given that store's CA";
ok =
let
s = baoWithMatrix.systemd.services;
in
s ? hive-c0re
&& (s.hive-c0re.environment.BAO_CACERT or null) == "%d/bao-ca.pem"
&& lib.any (c: lib.hasPrefix "bao-ca.pem:" c) s.hive-c0re.serviceConfig.LoadCredential;
}
{
# The name a reader dials has to resolve where the store runs; a
# multi-host swarm resolves it upstream instead.
name = "the store's host answers for the store's name";
ok = builtins.elem "bao.t.local" (baoNames baoPkcs11);
}
{
# Absence arm, and the one that matters: claiming a name this host does
# not serve points every local reader at the wrong machine.
name = "a hive that does not run the store claims no name for it";
ok = !(builtins.elem "bao.t.local" (baoNames bare));
}
{
# Raft refuses to start without it, and says so in a message that names
# neither the setting nor the stanza.
name = "the store advertises a cluster address";
ok = lib.hasPrefix "https://" ((baoSettings baoPkcs11).cluster_addr or "");
}
{
# Control for the case above: these settings are rendered per deployment,
# not constants a passing case could be indifferent to.
name = "a declared extra address renders a second listener beside loopback";
ok = builtins.length (builtins.attrNames (baoSettings baoTwoAddresses).listener) == 2;
}
{
# Retention is what serves the endpoint at all, so the listener alone
# would be a port that answers 404.
name = "a store beside a collector serves metrics on its own listener";
ok =
let
s = baoSettings baoWithCollector;
in
s.listener ? metrics && (s.telemetry.prometheus_retention_time or "0s") != "0s";
}
{
# openbao serves no `/metrics` at all, so a scrape of the default path
# 404s: the store looks like a dead exporter, and every panel built on
# it renders empty rather than erroring.
name = "the store's scrape asks for the path openbao serves";
ok =
let
j = scrapeJob baoWithCollector "bao";
in
(j.metrics_path or "") == "/v1/sys/metrics" && (j.params.format or [ ]) == [ "prometheus" ];
}
{
# Presence control for the case above: both fields are omitted rather
# than defaulted, so a target declared as bare `host:port` renders what
# it rendered before the path grammar existed.
name = "a target with no path renders neither metrics_path nor params";
ok =
let
j = scrapeJob baoWithCollector "plain";
in
j != null && !(j ? metrics_path) && !(j ? params);
}
{
# Absence arm. Unauthenticated by design, so it must not exist where
# nothing reads it.
name = "a store with no collector beside it serves no metrics";
ok =
let
s = baoSettings baoNoCollector;
in
!(s.listener ? metrics) && !(s ? telemetry);
}
];
bad = builtins.filter (c: !c.ok) cases;
# Escaped, because a name is prose and prose contains apostrophes. Hand-quoting
# broke the builder mid-report on the first such name that failed — and only
# ever on failure, so every green run agreed the reporter was fine.
report = lib.concatMapStringsSep "\n" (
c: " echo ${lib.escapeShellArg "FAILED: ${c.name}"} >&2"
) bad;
in
# The results are embedded in the builder text on purpose: that is what
# makes this derivation's hash depend on them, so a nix-only change that
# flips a case cannot be answered from cache.
pkgs.runCommand "hyperhive-module-eval" { } ''
${report}
${
if bad == [ ] then
"echo '${toString (builtins.length cases)} module properties hold' && touch $out"
else
"echo 'module-eval: ${toString (builtins.length bad)} of ${toString (builtins.length cases)} properties broke' >&2 && exit 1"
}
''
Reference in a new issue View git blame Copy permalink