vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
hyperhive/nix/module-eval.nix
atlas 8bd41fd84f bao: enable cert auth and give the controller a role for its policy 
The policy the granting unit already writes grants paths under
`auth/cert/certs/*`, and nothing in the tree creates that mount. Every
certificate login therefore fails against a path that is not there --
the controller's own, and the per-hive ones it is meant to issue
against the same mount.

Same unit, same bootstrap token: check whether cert auth is mounted,
enable it if not, then write a role binding CN `swarm-controller` to
the `swarm-controller` policy.

Idempotency is a read rather than a tolerated error. `auth enable`
fails on an existing mount, and recognising that would tie a rebuild to
an error string no run of this store has ever produced, so the unit
asks `bao auth list` and mounts only on absence. That read is why the
token policy in setup.md gains `sys/auth`.

Each grant came from `bao <cmd> -output-policy`, which prints what a
command requires without running it -- the same way controllerPolicyText
was derived. Enabling an auth method needs `sudo` on `sys/auth/cert`,
which the documented token did not have.

Gated on `clientCaFile`, not on the token alone: `swarm-bao-certs`
installs `client-ca.pem` only under that condition, and a role's
`certificate=` has to name a real CA. The policy write, which needs no
CA, is unchanged in that case.

Nothing can present a certificate for this role yet -- the only client
leaf the tree mints carries CN = the hive's name -- and none of this has
been run against a live store. Both are stated in setup.md.
2026-09-07 21:52:06 +02:00

1061 lines
51 KiB
Nix
Raw Blame History

# `checks.module-eval` — the flake check that covers **nix**.
#
# Every other check in ./checks.nix is a Rust derivation, so a `.nix`-only
# diff moves no hash and `nix flake check` goes green **without evaluating
# what changed**. This derivation's hash is a function of the results
# below, so a change that flips a property rebuilds it and fails naming it.
#
# Anything expressible as a module `assertion` **should be one instead** —
# an assertion fires at deploy time for a real operator, not only in CI.
# What cannot is the **absence class**: "a hive that hasn't opted in
# renders what it did before", "this unit does not exist unless X" — claims
# about the rendered config rather than about a config being invalid.
#
# ⚠️ **Name a case for the PROPERTY it defends, never the ticket that
# prompted it**; a case named after a ticket has the ticket's lifetime.
#
# ⚠️ **This evaluates, it does not execute** — a command line, a request or
# a certificate needs something that *runs* it. A case needing a rendered
# file must stub what it drags in (`deploy.swarm-ui.package = pkgs.emptyDirectory`).
{
pkgs,
lib,
self,
nixosSystem,
}:
let
# Stub host, same shape ./docs/default.nix already uses: enough for a
# `nixosSystem` to evaluate, nothing that pulls a real disk or
# bootloader in.
hive =
extra:
(nixosSystem {
system = pkgs.stdenv.hostPlatform.system;
modules = [
self.nixosModules.default
{
fileSystems."/" = {
device = "/dev/null";
fsType = "tmpfs";
};
boot.loader.grub.enable = false;
system.stateVersion = "25.11";
# `recursiveUpdate`, not `//`: a plain `//` only merges the
# *top-level* keys of `extra` in, so an `extra` that touches a
# nested attr under an existing top-level key (e.g. `swarm.*`)
# silently drops every sibling under that key instead of merging
# into it — the same class of bug as the `services.hyperhive`
# double-nesting mistake this stub already had to dodge once,
# just one level down. `recursiveUpdate` merges nested attrsets
# all the way down instead.
services.hyperhive = lib.recursiveUpdate {
enable = true;
hiveName = "h1";
swarm.domain = "t.local";
swarm.hives.h1.domain = "h1.t.local";
} extra;
}
];
}).config;
allLocal = hive { deploy.singleHostSwarm = true; };
bare = hive { };
withCi = hive { deploy.forgejo.ci.enable = true; };
# A host configured against the pre-rename option path. `mkRenamedOptionModule`
# is the only thing carrying it, and nothing else in this suite would notice
# if it were dropped: the new path evaluates fine on its own, so a missing
# shim reads as a clean tree and breaks every existing operator config.
wireguardOldPath = hive {
swarm.wireguard.enable = true;
swarm.wireguard.address = "10.100.0.1/24";
swarm.wireguard.privateKeyFile = "/etc/wireguard/hive.key";
};
# Same shape for the forge, which SPLIT rather than moving whole: the six
# host-side options are set here through their pre-rename paths while the
# rest of `swarm.forge` stays put. All six are defined so that dropping any
# single shim entry fails the eval, not just the two the assertion reads.
forgeOldPath = hive {
swarm.forge.package = pkgs.emptyDirectory;
swarm.forge.behindGateway = true;
swarm.forge.openFirewall = true;
swarm.forge.hostSwarmControllerTokenFile = "/etc/forge/sc.token";
swarm.forge.sso.clientSecretFile = "/etc/forge/oidc.secret";
swarm.forge.mirrors = [
{
upstream = "https://example.invalid/tool";
dest = "mirrors/tool";
}
];
};
# The homeserver's turn to split. All eight host-side options are set through
# their pre-rename paths — including `gui.enable`, whose value is deliberately
# the opposite of its default so the definition has to actually land, and both
# packages, stubbed to a derivation neither option defaults to for the same
# reason. All eight so that dropping any single shim entry fails the eval, not
# just the ones the assertions read.
matrixOldPath = hive {
deploy.matrix.enable = true;
swarm.matrix.openFirewall = true;
swarm.matrix.trustedServers = [ "matrix.example.invalid" ];
swarm.matrix.maxRequestSize = 31457280;
swarm.matrix.registrationTokenFile = "/etc/matrix/register.token";
swarm.matrix.gui.enable = false;
swarm.matrix.package = pkgs.emptyDirectory;
swarm.matrix.gui.package = pkgs.emptyDirectory;
swarm.matrix.sso.clientSecretFile = "/etc/matrix/oidc.secret";
};
# The queue's callout identity, fourth split slice. `autoGenerateCallout` is
# left FALSE on purpose: that is what makes the seed paths the thing deciding
# `responderConfigured`, so the assertion below is about the seeds rather
# than about the auto-mint branch. Every one of the seven old paths is
# defined — `enable` included, which is why it is spelled the old way here
# while the fixture below uses the new one — so dropping any single nats
# shim fails the eval, not just the arms read.
natsOldPath = hive {
swarm.nats.enable = true;
swarm.nats.autoGenerateCallout = false;
swarm.nats.calloutUserPublicKey = "UTESTUSERPUBKEYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
swarm.nats.calloutIssuerPublicKey = "ATESTISSUERPUBKEYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
swarm.nats.calloutUserSeedFile = "/run/secrets/nats-user.seed";
swarm.nats.calloutIssuerSeedFile = "/run/secrets/nats-issuer.seed";
swarm.nats.authPackage = pkgs.emptyDirectory;
};
# Seventh split slice, plus slice 10's two authelia packages. Of slice 7's
# movers only `usersFile` has a rename entry — the other two are `readOnly`,
# and a rename module contributes a definition, which a read-only option
# refuses; see ./host-modules/deploy.nix. `package` and `bridgePackage` are
# ordinary options, so they do carry one. The two arms
# below have different jobs. `usersFile` tests the rename; the nats one tests
# that a reader repointed to the new namespace still renders the derived
# path, which is the failure this slice could actually have shipped — seven
# of those reads went through an alias a path-shaped grep cannot see.
autheliaOldPath = hive {
deploy.authelia.enable = true;
deploy.nats.enable = true;
swarm.authelia.usersFile = "/var/lib/test-authelia/users.yml";
swarm.authelia.package = pkgs.emptyDirectory;
swarm.authelia.bridgePackage = pkgs.emptyDirectory;
swarm.nats.autoGenerateCallout = false;
swarm.nats.calloutUserPublicKey = "UTESTUSERPUBKEYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
swarm.nats.calloutIssuerPublicKey = "ATESTISSUERPUBKEYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
swarm.nats.calloutUserSeedFile = "/run/secrets/nats-user.seed";
swarm.nats.calloutIssuerSeedFile = "/run/secrets/nats-issuer.seed";
};
# A deliberately odd `maxRequestSize`, so the number the assertion looks for
# cannot have come from a default or from another module's literal.
matrixBodyCap = hive {
deploy.singleHostSwarm = true;
deploy.matrix.maxRequestSize = 99000000;
};
# `enable` is spelled the OLD way like everything else here, so all seven
# of the controller's shims are exercised rather than six.
controllerOldPath = hive {
swarm.controller.enable = true;
swarm.controller.package = pkgs.emptyDirectory;
swarm.controller.swarmctlPackage = pkgs.emptyDirectory;
swarm.controller.socketPath = "/run/test-ctrl/ctrl.sock";
swarm.controller.forgeTokenFile = "/run/secrets/ctrl-forge.token";
swarm.controller.authBridgeUrl = "http://127.0.0.1:19097";
swarm.controller.queue.clientSecretFile = "/run/secrets/ctrl-queue.secret";
};
# The swarm UI, whose namespace this slice empties: `swarm.ui` had exactly
# two options and both have moved, so BOTH old paths are set here and the
# whole namespace now lives or dies by its two rename entries.
uiOldPath = hive {
swarm.ui.enable = true;
swarm.ui.package = pkgs.emptyDirectory;
};
# The two stores, which had NO old-path fixture at all until this slice —
# their `enable` and `retentionPeriod` shims have been uncovered since they
# landed, which is precisely the "a missing shim reads as a clean tree"
# failure ./module-eval.nix's wireguard fixture was written to catch. All six
# old paths are set, so dropping any single entry fails the eval.
storesOldPath = hive {
swarm.victoriametrics.enable = true;
swarm.victorialogs.enable = true;
swarm.victoriametrics.retentionPeriod = "3d";
swarm.victorialogs.retentionPeriod = "5d";
swarm.victoriametrics.package = pkgs.emptyDirectory;
swarm.victorialogs.package = pkgs.emptyDirectory;
};
grafanaOldPath = hive {
deploy.grafana.enable = true;
swarm.grafana.socketDir = "/run/test-grafana-sock";
swarm.grafana.datasourceUrl = "http://127.0.0.1:19999";
swarm.grafana.logsDatasourceUrl = "http://127.0.0.1:19998";
swarm.grafana.plugins = [ ];
swarm.grafana.package = pkgs.emptyDirectory;
};
# The first slice to leave options on BOTH sides of the split, so the
# fixture sets all three of them through the paths an existing config uses:
# the two movers via their rename entries, `tokenEndpoint` via the path it
# kept. That is also what the all-or-nothing assertion wants, so this hive
# is a valid one rather than one that only evaluates because nothing forced
# the assertion.
statusPublishOldPath = hive {
swarm.statusPublish.natsUrl = "nats://10.0.0.9:4222";
swarm.statusPublish.tokenEndpoint = "https://auth.example.invalid/api/oidc/token";
swarm.statusPublish.clientSecretFile = "/run/secrets/status-client.secret";
};
baoPkcs11 = hive {
deploy.bao.enable = true;
deploy.bao.seal = "pkcs11";
};
baoShamir = hive {
deploy.bao.enable = true;
deploy.bao.seal = "shamir";
};
baoExplicitCerts = hive {
deploy.bao.enable = true;
deploy.bao.serverCertFile = "/etc/pki/bao.pem";
deploy.bao.serverKeyFile = "/etc/pki/bao-key.pem";
};
# The store and a service that reads from it, versus the store alone. The
# pair is what makes the reader's absence arm mean anything.
baoWithMatrix = hive {
deploy.bao.enable = true;
deploy.matrix.enable = true;
};
# A hive that reads from a store it does not run: no `deploy.bao.enable`, so
# nothing here mints a leaf and the operator names one placed by hand. The
# deployment this pairing exists to serve, and the one that was previously
# inexpressible — the gate asked whether the store was a neighbour.
baoRemoteReader = hive {
deploy.matrix.enable = true;
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
};
# The same hive with the identity taken away, which separates "a homeserver
# is deployed" from "this host can authenticate to the store".
matrixNoBaoIdentity = hive { deploy.matrix.enable = true; };
# The store, plus a placed bootstrap token: the only shape in which the
# swarm's first grant can be written at all.
baoGrantHere = hive {
deploy.bao.enable = true;
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
};
# The credential without the store. Writing the first grant is a store-side
# operation, so a host holding only the token has nothing to do — and this
# is the arm that separates "an operator placed a token" from "this box can
# act on it".
baoGrantNoStore = hive {
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
};
# The store and the token, with no CA to trust. `mkForce` because the PKI
# glue supplies one by default here — this is the deployment that brings its
# own certificates and has not named the authority yet, and it separates
# "the grant unit runs" from "cert auth can be set up".
baoGrantNoClientCa = hive {
deploy.bao.enable = true;
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
deploy.bao.clientCaFile = lib.mkForce null;
};
baoNames = machine: machine.services.hyperhive.gateway.localNames;
baoTwoAddresses = hive {
deploy.bao.enable = true;
deploy.bao.extraListenAddresses = [ "10.0.0.1" ];
# Pinned, not incidental: the case counting these listeners is about the
# declared addresses, and a collector on this host would add one of its own.
deploy.swarm-otel.enable = false;
};
# The store with and without a collector on the same host. `scrapeTargets`
# is only ever read by a local collector, so the metrics endpoint is a
# function of the pairing rather than of the store.
baoWithCollector = hive {
deploy.bao.enable = true;
deploy.swarm-otel.enable = true;
# A second job declared as bare `host:port`, so the pair of cases below
# reads one rendered scrape list: the store's entry carries a path, this
# one carries none.
swarm.otel.scrapeTargets.plain = "127.0.0.1:9999";
};
baoNoCollector = hive {
deploy.bao.enable = true;
deploy.swarm-otel.enable = false;
};
# The config file openbao parses, not the nix that produces it: a setting it
# requires is absent here without anything in the module system minding, so
# the daemon's own startup is otherwise the first reader.
baoSettings = machine: machine.containers.swarm-bao.config.services.openbao.settings;
# The store's units live inside its container, so the gates below have to
# look there rather than at the host's service set.
baoUnits = machine: machine.containers.swarm-bao.config.systemd.services;
# The scrape list prometheus is handed, not the option a service declared:
# the address, the path and the query are one string on the way in and three
# fields on the way out, and only the second shape is what gets requested.
scrapeJob =
machine: job:
lib.findFirst (c: c.job_name == job) null
machine.containers.swarm-otel.config.services.opentelemetry-collector.settings.receivers.prometheus.config.scrape_configs;
# A swarm collector on a host that runs NEITHER store — the fully-spread
# shape from docs/swarm/services.md, and the one the old per-host gates made
# inexpressible. It is the whole point of the cases below that this hive is
# not a degenerate configuration but a supported one.
otelNoStores = hive {
deploy.swarm-otel.enable = true;
deploy.authelia.enable = true;
deploy.victoriametrics.enable = false;
deploy.victorialogs.enable = false;
};
otelSettings =
machine: machine.containers.swarm-otel.config.services.opentelemetry-collector.settings;
# authelia somewhere else, the credential delivered by hand. Whether this
# collector authenticates must follow the credential, never another
# service's placement.
#
# The `swarm.otel.clientSecretFile` below is the PRE-RENAME path. It predates
# the split and is deliberately left spelled that way: it makes this fixture
# the old-path case for that option too, so dropping its rename entry fails
# the eval here rather than only in a real operator's config.
otelRemoteAuthelia = hive {
deploy.swarm-otel.enable = true;
deploy.authelia.enable = false;
# Where that elsewhere IS. Running no IdP does not mean knowing no IdP:
# the authenticator this fixture exists to render puts this address in its
# `token_url`, so a hive with a secret and no URL has a credential it can
# present nowhere.
swarm.authelia.url = "https://auth.example.invalid";
swarm.otel.clientSecretFile = "/var/lib/swarm-otel-oidc/by-hand.secret";
};
# A priority collision is a property of the *option*, not
# of the merged value's interior — nix throws the moment the value is
# demanded at all, so `seq`-ing each `serviceConfig` value to WHNF is
# both necessary and sufficient. `deepSeq` over-specifies this: it keeps
# walking *into* the resulting value after the merge already succeeded,
# and a package/derivation-shaped value's `override`/`overrideAttrs`
# self-reference sends it into nixpkgs' fixpoint machinery and blows the
# stack (measured — this is not a hypothetical).
forceCiServiceConfigs =
let
svcs = withCi.containers.hive-ci.config.systemd.services;
vals = lib.concatMap (s: builtins.attrValues (s.serviceConfig or { })) (builtins.attrValues svcs);
in
builtins.foldl' (acc: v: builtins.seq v acc) true vals;
# Each case: a name stating the property, and `ok`.
cases = [
{
name = "a hive that has not opted into all-local runs no swarm controller";
ok = !bare.services.hyperhive.deploy.swarm-controller.enable;
}
{
name = "the all-local mode turns the swarm controller on";
ok = allLocal.services.hyperhive.deploy.swarm-controller.enable;
}
{
# The mesh moved namespace wholesale, so an existing config sets paths
# that no longer exist. Reading the *rendered interface* rather than the
# option: a rename that resolved but stopped reaching the module would
# satisfy an option-level check and still bring up no tunnel.
name = "a config written against the pre-rename wireguard path still configures the interface";
ok =
let
wg = wireguardOldPath.networking.wireguard.interfaces.wg-hive;
in
wg.ips == [ "10.100.0.1/24" ] && wg.privateKeyFile == "/etc/wireguard/hive.key";
}
{
# Same reasoning one namespace over, plus the shape the mesh did not
# have: `mirrors` is a single option of a list-of-submodule type, so its
# one rename entry has to carry a whole compound value rather than a
# scalar. Both arms read a rendered effect — the host firewall and the
# env var c0re seeds mirrors from — not the option.
name = "a config written against the pre-rename forge paths still opens the firewall and seeds the mirror";
ok =
let
ports = forgeOldPath.networking.firewall.allowedTCPPorts;
seeded = builtins.fromJSON forgeOldPath.systemd.services.hive-c0re.environment.HYPERHIVE_FORGE_MIRRORS;
in
builtins.elem 3000 ports && builtins.any (m: m.dest == "mirrors/tool") seeded;
}
{
# Third split, and the one whose readers were hardest to see: the
# registration token is read only through a `let` alias in another
# module, so no full path names it anywhere. Both arms read a rendered
# effect — the host firewall and the container's bind-mount table — so a
# rename that resolves but stops reaching the module still fails.
name = "a config written against the pre-rename matrix paths still opens the port and mounts the token";
ok =
let
ports = matrixOldPath.networking.firewall.allowedTCPPorts;
httpPort = matrixOldPath.services.hyperhive.swarm.matrix.httpPort;
in
builtins.elem httpPort ports
&& matrixOldPath.containers.hive-matrix.bindMounts ? "/etc/matrix/register.token";
}
{
# registrationTokenFile lost its override capability entirely
# (bao-delivered secrets don't need one — glue-matrix-bao-token.nix
# already writes into the fixed path instead of moving it), unlike its
# five siblings in the same rename. `matrixOldPath` above proves the
# override still *resolves* (mkDefault, not a crash) — this proves it
# also gets *rejected*, by a named assertion rather than nixpkgs' generic
# conflicting-definition text. Reads `.assertions` directly (cheap: a
# list of `{assertion; message;}`, not `system.build.toplevel`) rather
# than forcing a real build just to observe a boolean.
name = "overriding registrationTokenFile (even via the pre-rename shim) trips a named assertion, not a silent desync";
ok = lib.any (
a: !a.assertion && lib.hasInfix "registrationTokenFile" a.message
) matrixOldPath.assertions;
}
{
# Reads the DELIVERY UNIT, not the options: `responderConfigured` gates
# whether it exists at all, and the seed path is interpolated into its
# script. A rename that resolved but stopped reaching the module would
# leave the responder with no credentials and this case would catch it.
name = "a config written against the pre-rename nats paths still delivers the responder's seeds";
ok =
let
units = natsOldPath.systemd.services;
in
units ? swarm-nats-auth-secrets
&& lib.hasInfix "/run/secrets/nats-user.seed" units.swarm-nats-auth-secrets.script;
}
{
# Reads the host's tmpfiles rules, not the options: the socket directory
# nginx and the container share is created there, so a rename that
# resolved but stopped reaching the module would leave the gateway
# proxying to a path nothing creates. All four old paths are defined in
# the fixture, so removing any single shim fails the eval rather than
# only the one this assertion reads.
# All FOUR movers are asserted as rendered effects rather than as option
# values, so a rename that resolved but stopped reaching the module is
# caught per-option instead of only where one assertion happens to look.
name = "a config written against the pre-rename swarm-controller paths still reaches the unit";
ok =
let
u = controllerOldPath.systemd.services.swarm-controller;
creds = u.serviceConfig.LoadCredential;
in
u.environment.SWARM_CONTROLLER_SOCKET == "/run/test-ctrl/ctrl.sock"
&& u.environment.SWARM_CONTROLLER_AUTH_BRIDGE_URL == "http://127.0.0.1:19097"
&& lib.any (c: lib.hasInfix "/run/secrets/ctrl-queue.secret" c) creds
&& lib.any (c: lib.hasInfix "/run/secrets/ctrl-forge.token" c) creds;
}
{
name = "a config written against the pre-rename grafana paths still creates the socket directory";
ok = lib.any (
rule: lib.hasInfix "/run/test-grafana-sock" rule
) grafanaOldPath.systemd.tmpfiles.rules;
}
{
# Reads the daemon's rendered unit, not the options: the queue address
# arrives as an env var whose whole attrset is guarded on `natsUrl`, and
# the secret as a systemd credential. A rename that resolved but stopped
# reaching the module leaves hive-c0re coming up perfectly and reporting
# to nobody, which is the one failure this option set exists to prevent.
# The third arm is the option that did NOT move, read out of the same
# attrset: the guard and the value beside it now come from different
# namespaces, so a hive that renders one and drops the other is exactly
# what a split can silently produce.
name = "a config written against the pre-rename statusPublish paths still reaches the daemon";
ok =
let
u = statusPublishOldPath.systemd.services.hive-c0re;
in
u.environment.HIVE_C0RE_NATS_URL == "nats://10.0.0.9:4222"
&& u.environment.HIVE_C0RE_OIDC_TOKEN_ENDPOINT == "https://auth.example.invalid/api/oidc/token"
&& lib.any (c: lib.hasInfix "/run/secrets/status-client.secret" c) u.serviceConfig.LoadCredential;
}
{
# The collector's half of the same split, and a different arm from the
# authenticator case below: this one reads the PATH the unit loads, so a
# reader left on a source that is non-null but wrong still fails. The
# fixture spells the option its pre-rename way, so it covers the rename
# entry at the same time.
name = "a config written against the pre-rename otel secret path still loads it as a credential";
ok =
lib.any (c: lib.hasInfix "/var/lib/swarm-otel-oidc/by-hand.secret" c)
otelRemoteAuthelia.containers.swarm-otel.config.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential;
}
{
# Reads the rendered unit INSIDE the container, which is where the write
# happens: reaching the store locally is what lets the grant be written
# without a client certificate at all.
name = "a store host with a placed bootstrap token renders the granting unit inside the container";
ok =
let
u = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy;
in
lib.hasInfix "/run/secrets/bao-bootstrap.token" u.script
&& u.unitConfig.ConditionPathExists == "/run/secrets/bao-bootstrap.token";
}
{
# The grants themselves, and the `hive-` prefix is the whole point:
# without it the controller can rewrite the policy that constrains it,
# which is a privilege escalation that renders, deploys and looks fine.
# Readable here only because the HCL is piped as an argument rather than
# written to a store path.
name = "the controller's bao grants cannot reach the policy that constrains it";
ok =
let
s = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script;
in
lib.hasInfix "sys/policies/acl/hive-*" s && !(lib.hasInfix "sys/policies/acl/*" s);
}
{
# The policy above grants paths under a mount nothing else creates, so
# the unit that writes the policy has to create it too — otherwise every
# certificate login fails against a path that is not there.
name = "the granting unit creates the cert auth mount and the controller's role";
ok =
let
s = baoGrantHere.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script;
in
lib.hasInfix "bao auth enable cert" s
&& lib.hasInfix "auth/cert/certs/swarm-controller" s
&& lib.hasInfix "/var/lib/swarm-bao-tls/client-ca.pem" s;
}
{
# The arm that makes the one above mean something. A role's trust anchor
# is the CA, so with none named there is nothing to write — and the
# policy write, which needs no CA, must survive that.
#
# ⚠️ Matched on the COMMANDS, not on `auth/cert/certs`: the policy text is
# embedded in this same script and grants that very path, so the shorter
# infix is present either way and the arm could never fail.
name = "with no client CA the unit still writes the policy and skips the role";
ok =
let
s =
baoGrantNoClientCa.containers.swarm-bao.config.systemd.services.swarm-bao-controller-policy.script;
in
lib.hasInfix "bao policy write" s
&& !(lib.hasInfix "bao auth enable cert" s)
&& !(lib.hasInfix "client-ca.pem" s);
}
{
# The arm that makes the one above mean something, and the property the
# host-side half depends on: no store here, so no bind mount and no unit.
# Without it a hive that merely names a token would drag the store's
# container config into its evaluation.
# Reads the vhost's rendered `root`, not the option: the UI is served
# straight out of a store path, so a shim that resolves but stops
# reaching the module would leave nginx pointing at the default build.
name = "a config written against the pre-rename swarm-ui paths still serves the operator's build";
#
# ⚠️ `unsafeDiscardStringContext` on both sides, load-bearing rather than
# tidy: interpolating a derivation carries string *context*, and this
# suite renders its results into a `buildCommand` that may not reference
# store paths. Comparing the paths as plain text is the intent — the case
# asks "does nginx point HERE", not "depend on what it points at".
ok =
let
want = builtins.unsafeDiscardStringContext "${pkgs.emptyDirectory}";
# `or ""` is NOT enough: nginx's `locations.<l>.root` is `nullOr`, so
# a vhost that HAS the attribute set to null skips the default and
# reaches the coercion. Filter nulls, then discard context.
roots = map builtins.unsafeDiscardStringContext (
lib.filter (r: r != null) (
map (v: v.locations."/".root or null) (builtins.attrValues uiOldPath.services.nginx.virtualHosts)
)
);
in
builtins.elem want roots;
}
{
# Reads the package the CONTAINER renders, not the option: a shim that
# resolves but stops reaching the module would leave the store running
# nixpkgs' default while the operator's override read back fine.
name = "a config written against the pre-rename store paths still picks the operator's package";
ok =
storesOldPath.containers.swarm-victorialogs.config.services.victorialogs.package
== pkgs.emptyDirectory
&&
storesOldPath.containers.swarm-victoriametrics.config.services.victoriametrics.package
== pkgs.emptyDirectory;
}
{
name = "a bootstrap token on a host that runs no store grants nothing";
ok = !(baoGrantNoStore.systemd.services ? swarm-bao-bootstrap-dir);
}
{
name = "a config written against the pre-rename authelia usersFile still reaches the bridge";
ok =
autheliaOldPath.containers.swarm-authelia.config.systemd.services.swarm-authelia-bridge.environment.SWARM_AUTHELIA_BRIDGE_USERS_FILE
== "/var/lib/test-authelia/users.yml";
}
{
# Not a rename test. `hostClientSecretDir` is `readOnly`, so the fixture
# cannot define it; what can break is a reader left pointing at the
# namespace it moved out of. Five modules read this through an
# `autheliaCfg` alias, where a path-shaped grep does not see it.
name = "a consumer of authelia's host client-secret dir renders it from the deploy namespace";
ok = lib.hasInfix "/var/lib/nixos-containers/swarm-authelia/var/lib/authelia-swarm/oidc-clients/" autheliaOldPath.systemd.services.swarm-nats-auth-secrets.script;
}
{
# The gateway's per-name issuer choice. If this ever collapses to a
# constant, every swarm-service vhost serves a certificate its CA
# is name-constrained out of — which evaluates cleanly and fails in
# a browser.
name = "a swarm service name gets the swarm-services leaf and the default server does not";
ok =
let
l = allLocal.services.hyperhive.gateway.lib;
in
(l.tlsFor "t.local").sslCertificate != (l.tlsFor "_").sslCertificate;
}
{
# nixos asserts when a vhost declares both, so this is also a
# statement that the `removeAttrs` upstream of it still happens.
name = "the swarm UI vhost forces TLS instead of merely adding it";
ok =
let
v = allLocal.services.nginx.virtualHosts."t.local";
in
v.forceSSL && !(v.addSSL or false);
}
{
name = "a hive with matrix off serves no matrix discovery endpoint";
ok =
!(builtins.hasAttr "= /.well-known/matrix/client" bare.services.nginx.virtualHosts."_".locations);
}
{
# main got eval-borked twice by this exact class of bug (once on the
# unit's `Restart` key, once on `RestartSec`) — a nixpkgs bump to
# `gitea-actions-runner.nix` adds a plain `serviceConfig.*`
# definition that collides with one of ours, and nix refuses to
# merge two plain definitions at *host* eval. No other check
# instantiates a host with `containers.hive-ci` actually enabled, so
# the collision only surfaces on operator deploy, not in CI.
name = "the CI container's unit definitions merge without a priority collision";
ok = forceCiServiceConfigs;
}
{
# The defect itself. These exporters used to be gated on the stores'
# PER-HOST enables, so a collector that did not share a host with them
# rendered none at all and dropped everything it received, from every
# hive — silently, because an absent exporter is not an error.
name = "a collector that hosts neither store still exports to both";
ok =
let
e = (otelSettings otelNoStores).exporters;
in
(e ? "otlphttp/victoriametrics") && (e ? "otlphttp/victorialogs");
}
{
# A swarm has one of each store, so the address is a swarm-level name.
# A loopback literal here is the co-location assumption written back in,
# and it renders, deploys and reports healthy while reaching nothing.
name = "the store exporters address the stores by name, never by loopback";
ok =
let
e = (otelSettings otelNoStores).exporters;
m = e."otlphttp/victoriametrics".metrics_endpoint;
l = e."otlphttp/victorialogs".logs_endpoint;
in
!(lib.hasInfix "127.0.0.1" m)
&& !(lib.hasInfix "127.0.0.1" l)
&& lib.hasInfix "metrics.t.local" m
&& lib.hasInfix "logs.t.local" l;
}
{
# The collector reaches these routes through the gateway now, so each
# store needs an ingest location of its own. Without one the write rides
# the `/` catch-all: unauthenticated on the metrics store, and into a
# browser redirect on the log store.
name = "each store's vhost has an authenticated ingest location";
ok =
let
v = allLocal.services.nginx.virtualHosts;
m = v."metrics.t.local".locations."= /opentelemetry/api/v1/push" or null;
l = v."logs.t.local".locations."= /insert/opentelemetry/v1/logs" or null;
in
m != null
&& l != null
&& lib.hasInfix "auth_request" m.extraConfig
&& lib.hasInfix "auth_request" l.extraConfig;
}
{
# Two limits governed a matrix upload and nothing kept them in agreement,
# so raising the documented one past the gateway's hardcoded cap changed
# nothing. The second clause is the control: the old directive is gone, so
# a pass means the value is derived rather than that `hasInfix` matched
# something incidental. It targets the whole directive rather than the
# bare size, because the rendered config carries the comment above it and
# a prose mention of the old value would defeat a looser arm.
name = "the matrix gateway's body cap is derived from maxRequestSize, not a literal";
ok =
let
c = matrixBodyCap.services.nginx.virtualHosts."chat.t.local".locations."/_matrix/".extraConfig;
in
lib.hasInfix "client_max_body_size 100048576;" c && !(lib.hasInfix "client_max_body_size 50M" c);
}
{
# The arm that actually protects something. A pusher handed
# `error_page 401 =302` FOLLOWS it and POSTs its batch at a login page,
# which answers 200 — ingest reporting healthy while storing nothing.
# The third clause is the positive control: the log store's browser
# location really does redirect, so this says the machine routes differ
# rather than that the string is absent from the whole file.
name = "the ingest locations answer 401 instead of redirecting a pusher";
ok =
let
v = allLocal.services.nginx.virtualHosts;
m = v."metrics.t.local".locations."= /opentelemetry/api/v1/push".extraConfig;
l = v."logs.t.local".locations."= /insert/opentelemetry/v1/logs".extraConfig;
browser = v."logs.t.local".locations."/".extraConfig;
in
!(lib.hasInfix "error_page" m)
&& !(lib.hasInfix "error_page" l)
&& lib.hasInfix "error_page" browser;
}
{
# Defining an exporter and REFERENCING it are two separate lists, and
# the second is where the original gate also lived. An exporter no
# pipeline names is as silent as one that does not exist — this case
# exists because a mutation that restored only the reference-side gate
# left every other case here green.
name = "every pipeline that has a store exporter defined actually sends to it";
ok =
let
s = otelSettings otelNoStores;
used = lib.unique (lib.concatMap (p: p.exporters) (lib.attrValues s.service.pipelines));
in
builtins.elem "otlphttp/victoriametrics" used && builtins.elem "otlphttp/victorialogs" used;
}
{
# The collector authenticates because it HOLDS a credential, not because
# authelia happens to share its host. Gating on the other service's
# placement renders a collector that pushes unauthenticated wherever
# authelia lives elsewhere — one of the supported shapes.
name = "a collector with a hand-delivered secret authenticates without authelia beside it";
ok =
let
s = otelSettings otelRemoteAuthelia;
in
(s.exporters."otlphttp/victoriametrics" ? auth)
&& builtins.elem "oauth2client/victoriametrics" s.service.extensions;
}
{
# An authenticator an exporter names but `service.extensions` omits is
# INERT — the collector starts clean and pushes unauthenticated until
# something at the far end refuses it. Checked as a set relation rather
# than by naming the two, so it keeps holding for exporters not written
# yet.
name = "every exporter authenticator is listed in service.extensions";
ok =
let
s = otelSettings otelNoStores;
named = lib.filter (v: v != null) (
lib.mapAttrsToList (_: e: e.auth.authenticator or null) s.exporters
);
in
named != [ ] && lib.all (a: builtins.elem a s.service.extensions) named;
}
{
# The store's seal is spread over six gates — the stanza, the
# provisioning unit, two bind mounts, a device and an EnvironmentFile.
# Rendering only some of them is the dangerous state: a store that
# says hardware-backed and seals with a software key, which no
# assertion can catch because every value is individually valid.
name = "a shamir store renders no TPM provisioning unit";
ok = !(baoUnits baoShamir ? swarm-bao-token);
}
{
# Presence control for the case above. Without it, a typo in the
# option name would satisfy the absence arm forever. The second half is
# the fix itself: the unit has to run where openbao's `DynamicUser` is
# allocated, and a host unit writing the same bytes has no name to hand
# them to.
name = "a pkcs11 store provisions the token in the container, not on the host";
ok = (baoUnits baoPkcs11 ? swarm-bao-token) && !(baoPkcs11.systemd.services ? swarm-bao-token);
}
{
# `allowedDevices` renders `DeviceAllow=` and nothing else — nspawn
# mounts its own /dev and cannot create device nodes, so permission to
# use a device that was never bound in opens nothing. Neither half
# fails on its own, which is why they are asserted as a pair.
name = "a pkcs11 store gets the TPM device bound in, not merely allowed";
ok =
let
c = baoPkcs11.containers.swarm-bao;
in
(c.bindMounts ? "/dev/tpmrm0") && builtins.any (d: d.node == "/dev/tpmrm0") c.allowedDevices;
}
{
# The stanza's label and the label the unit creates are two literals that
# have to name one object, and the mechanism is asserted with them
# because it is valid only for an RSA key: openbao takes AES-GCM or
# RSA-OAEP and a TPM 2.0 has neither GCM nor an opinion about which the
# seal asked for. Every wrong combination renders and deploys, and
# surfaces as a pkcs11 error at `operator init`.
name = "the seal asks for the RSA key the provisioning unit creates";
ok =
let
p = (baoSettings baoPkcs11).seal.pkcs11;
in
(p.mechanism or "") == "CKM_RSA_PKCS_OAEP"
&& lib.hasInfix "--algorithm=rsa2048 --key-label=${p.key_label or ""}" (
(baoUnits baoPkcs11).swarm-bao-token.script or ""
);
}
{
# `DynamicUser` implies `ProtectSystem=strict`, so the token directory is
# read-only to the seal however it is owned, and the group is the only
# handle on a uid allocated at start. Dropping either surfaces as a
# pkcs11 error deep in a library, naming neither the mount nor the user.
name = "the store's seal may write the token directory, and is in both its groups";
ok =
let
sc = (baoUnits baoPkcs11).openbao.serviceConfig;
in
# `or [ ]` rather than a bare select: the interesting mutation is the
# key being gone, and a select would abort the whole run with a nix
# trace instead of failing this case by name.
builtins.elem "/var/lib/swarm-bao-token" (sc.ReadWritePaths or [ ])
&& builtins.elem "swarm-bao-token" (sc.SupplementaryGroups or [ ])
&& builtins.elem "swarm-bao-tpm" (sc.SupplementaryGroups or [ ]);
}
{
# The device node belongs to the HOST and is matched by NUMBER, while the
# unit that opens it lives in the container — so the two sides holding
# the same gid is the entire mechanism. Letting either side auto-allocate
# renders cleanly, deploys cleanly, and leaves a 0660 node the seal
# cannot open. Compared rather than each checked against a literal: the
# property is that they AGREE, not what they agree on.
name = "the TPM group has the same gid on the host and inside the container";
ok =
let
host = baoPkcs11.users.groups.swarm-bao-tpm.gid or null;
inner = baoPkcs11.containers.swarm-bao.config.users.groups.swarm-bao-tpm.gid or null;
in
host != null && host == inner;
}
{
# Absence arm for the case above — a shamir store never opens a TPM, so
# it must not claim a device node's group. Without this, pinning the gid
# unconditionally would look identical.
name = "a shamir store claims no TPM device group";
ok = !(baoShamir.users.groups ? swarm-bao-tpm);
}
{
# The store's mTLS identity is a separate trust domain from both CAs in
# this tree, because it must not come from an authority the store will
# itself distribute. What supplies it is the glue, which mints a CA of
# the store's own — so an enabled store has all three paths, and if this
# ever reads null again the store stops coming up on its own.
name = "a deployed store is given its own certificate, key and client CA";
ok =
let
b = baoPkcs11.services.hyperhive.deploy.bao;
in
b.serverCertFile != null && b.serverKeyFile != null && b.clientCaFile != null;
}
{
# Everything the glue sets is `mkDefault`, and this is the case that
# says so: a deployment whose certificates come from somewhere the glue
# has never heard of must win. Also the presence control for the case
# above — a renamed option would read `null` on both and satisfy
# neither, but only this one names a value.
name = "an operator's own certificate path beats the glue's default";
ok = baoExplicitCerts.services.hyperhive.deploy.bao.serverCertFile == "/etc/pki/bao.pem";
}
{
# The store's first reader. Its unit belongs to the pairing, not to
# either service: matrix must not learn the store exists, and the store
# must not know who reads it.
name = "a store deployed beside the homeserver fetches its registration token";
ok = baoWithMatrix.systemd.services ? swarm-bao-matrix-token;
}
{
# Absence arm. A store with nothing to serve renders no reader, so the
# unit is a function of the PAIRING rather than of the store — which is
# the property that makes it glue instead of a feature of either side.
name = "a store with no homeserver beside it renders no token reader";
ok = !(baoPkcs11.systemd.services ? swarm-bao-matrix-token);
}
{
name = "a hive that names a client identity reads from a store it does not run";
ok = baoRemoteReader.systemd.services ? swarm-bao-matrix-token;
}
{
# Absence arm for the one above, and the reason the gate is the identity
# rather than the homeserver: without it, deploying matrix anywhere would
# render a reader that cannot authenticate.
name = "a homeserver with no way to authenticate to the store renders no token reader";
ok = !(matrixNoBaoIdentity.systemd.services ? swarm-bao-matrix-token);
}
{
# `Requires=` on a unit that does not exist fails the job, and nothing
# local mints certificates off-host — so this orders against nothing.
# Eval cannot see that failure; only the empty list here stands in for it.
name = "an off-host reader requires no unit the store's host would have provided";
# Membership first, then the value: indexing a missing unit throws, and a
# table that reports which property broke must not be the thing that dies.
ok =
let
s = baoRemoteReader.systemd.services;
in
s ? swarm-bao-matrix-token && s.swarm-bao-matrix-token.requires == [ ];
}
{
# Presence control for the case above: the list is conditional, not gone.
name = "a co-located reader still orders after the local pki unit";
ok =
let
s = baoWithMatrix.systemd.services;
in
s ? swarm-bao-matrix-token && s.swarm-bao-matrix-token.requires == [ "swarm-bao-pki.service" ];
}
{
# hive-c0re runs as hive-core and the client key is `0600` root-owned
# inside a `0700` directory, so the identity reaches the daemon as a
# systemd credential and the environment names `%d` rather than the
# file. Both halves are asserted together because either alone is a
# daemon that fails at the TLS handshake, naming neither.
name = "a reader hands hive-c0re a store identity the daemon cannot open itself";
ok =
let
s = baoRemoteReader.systemd.services;
in
s ? hive-c0re
&& (s.hive-c0re.environment.BAO_CLIENT_CERT or null) == "%d/bao-client.pem"
&& (s.hive-c0re.environment.BAO_CLIENT_KEY or null) == "%d/bao-client-key.pem"
&& builtins.elem "bao-client.pem:/etc/pki/bao-client.pem" s.hive-c0re.serviceConfig.LoadCredential
&& builtins.elem "bao-client-key.pem:/etc/pki/bao-client-key.pem" s.hive-c0re.serviceConfig.LoadCredential;
}
{
# Absence arm for the case above. A hive with no client identity gets no
# store environment at all — the daemon reports a queue it cannot serve
# rather than a handshake it cannot explain.
name = "a hive with no client identity gives hive-c0re no store environment";
ok =
let
s = matrixNoBaoIdentity.systemd.services;
in
s ? hive-c0re
&& !(s.hive-c0re.environment ? BAO_ADDR)
&& !(lib.any (c: lib.hasPrefix "bao-" c) s.hive-c0re.serviceConfig.LoadCredential);
}
{
# The CA is its own arm: absent means the system trust store, which is
# right for a deployment with a real CA and wrong for a self-signed one.
name = "a reader that names no store CA falls through to the system trust store";
ok =
let
s = baoRemoteReader.systemd.services;
in
s ? hive-c0re && !(s.hive-c0re.environment ? BAO_CACERT);
}
{
# Presence control for the arm above: the CA is conditional, not gone.
# Co-located, ./host-modules/glue-bao-tls.nix mints one and names it.
name = "a reader beside a self-signed store is given that store's CA";
ok =
let
s = baoWithMatrix.systemd.services;
in
s ? hive-c0re
&& (s.hive-c0re.environment.BAO_CACERT or null) == "%d/bao-ca.pem"
&& lib.any (c: lib.hasPrefix "bao-ca.pem:" c) s.hive-c0re.serviceConfig.LoadCredential;
}
{
# The name a reader dials has to resolve where the store runs; a
# multi-host swarm resolves it upstream instead.
name = "the store's host answers for the store's name";
ok = builtins.elem "bao.t.local" (baoNames baoPkcs11);
}
{
# Absence arm, and the one that matters: claiming a name this host does
# not serve points every local reader at the wrong machine.
name = "a hive that does not run the store claims no name for it";
ok = !(builtins.elem "bao.t.local" (baoNames bare));
}
{
# Raft refuses to start without it, and says so in a message that names
# neither the setting nor the stanza.
name = "the store advertises a cluster address";
ok = lib.hasPrefix "https://" ((baoSettings baoPkcs11).cluster_addr or "");
}
{
# Control for the case above: these settings are rendered per deployment,
# not constants a passing case could be indifferent to.
name = "a declared extra address renders a second listener beside loopback";
ok = builtins.length (builtins.attrNames (baoSettings baoTwoAddresses).listener) == 2;
}
{
# Retention is what serves the endpoint at all, so the listener alone
# would be a port that answers 404.
name = "a store beside a collector serves metrics on its own listener";
ok =
let
s = baoSettings baoWithCollector;
in
s.listener ? metrics && (s.telemetry.prometheus_retention_time or "0s") != "0s";
}
{
# openbao serves no `/metrics` at all, so a scrape of the default path
# 404s: the store looks like a dead exporter, and every panel built on
# it renders empty rather than erroring.
name = "the store's scrape asks for the path openbao serves";
ok =
let
j = scrapeJob baoWithCollector "bao";
in
(j.metrics_path or "") == "/v1/sys/metrics" && (j.params.format or [ ]) == [ "prometheus" ];
}
{
# Presence control for the case above: both fields are omitted rather
# than defaulted, so a target declared as bare `host:port` renders what
# it rendered before the path grammar existed.
name = "a target with no path renders neither metrics_path nor params";
ok =
let
j = scrapeJob baoWithCollector "plain";
in
j != null && !(j ? metrics_path) && !(j ? params);
}
{
# Absence arm. Unauthenticated by design, so it must not exist where
# nothing reads it.
name = "a store with no collector beside it serves no metrics";
ok =
let
s = baoSettings baoNoCollector;
in
!(s.listener ? metrics) && !(s ? telemetry);
}
];
bad = builtins.filter (c: !c.ok) cases;
# Escaped, because a name is prose and prose contains apostrophes. Hand-quoting
# broke the builder mid-report on the first such name that failed — and only
# ever on failure, so every green run agreed the reporter was fine.
report = lib.concatMapStringsSep "\n" (
c: " echo ${lib.escapeShellArg "FAILED: ${c.name}"} >&2"
) bad;
in
# The results are embedded in the builder text on purpose: that is what
# makes this derivation's hash depend on them, so a nix-only change that
# flips a case cannot be answered from cache.
pkgs.runCommand "hyperhive-module-eval" { } ''
${report}
${
if bad == [ ] then
"echo '${toString (builtins.length cases)} module properties hold' && touch $out"
else
"echo 'module-eval: ${toString (builtins.length bad)} of ${toString (builtins.length cases)} properties broke' >&2 && exit 1"
}
''
Reference in a new issue View git blame Copy permalink