ed2ec52fe5
Every gateway asked the store's `pki/issue/swarm-services` for the whole swarm's service set, so a private key on any gateway host could serve a valid certificate for services that host does not front and never has. `swarm.localServiceDomains` derives the per-host subset by filtering `swarm.serviceDomains` against the vhosts this host actually renders — the deploy flags those vhosts are already guarded on, read once rather than copied into a second filter. The leaf request and the coverage guard that decides whether to re-issue both read it, so they cannot disagree about which names the leaf owes. The sub-CA's name constraint and the role's `allowed_domains` stay the swarm-wide set: every host's subset is inside it, and narrowing the constraint per host would turn one signing into N.
647 lines
30 KiB
Nix
647 lines
30 KiB
Nix
# The swarm's directory: one entry per hive, **including this one**,
|
||
# identical on every host in the swarm. `services.hyperhive.hiveName`
|
||
# says which entry is us, and `peerHives` below derives the rest.
|
||
#
|
||
# Why a directory rather than a per-host peer list: every field here is
|
||
# intrinsic to the hive it describes — none of them says anything about
|
||
# the *pair*. A list where every field is intrinsic is a directory each
|
||
# host was keeping its own copy of, which is O(n²) duplication that
|
||
# deduplicates without loss. It is also a correctness gain: two hosts
|
||
# could hold different endpoints for the same third hive and nothing
|
||
# detected it. One entry per hive makes that unrepresentable.
|
||
#
|
||
# Consumed by swarm-controller's own hive directory (its `/api/hives`,
|
||
# swarm-controller.nix) and the mesh in ./swarm-wireguard.nix. The mesh
|
||
# lives there rather than here because bringing up an interface is
|
||
# host networking rather than swarm bookkeeping, and a host that runs
|
||
# no hive still needs it.
|
||
{
|
||
lib,
|
||
config,
|
||
...
|
||
}:
|
||
let
|
||
cfg = config.services.hyperhive;
|
||
swarmCfg = cfg.swarm;
|
||
deployCfg = cfg.deploy;
|
||
|
||
hiveNames = lib.attrNames swarmCfg.hives;
|
||
|
||
nameGuards = import ./lib/name-guards.nix { inherit lib; };
|
||
|
||
# Names no hive may BE, and words no hive name may CONTAIN. Both files
|
||
# explain their own admission rules; the second says why they are not one
|
||
# list. They are applied HERE rather than in the module that happens to
|
||
# consume them, because a hive name composes identifiers whatever else is
|
||
# enabled — ./swarm-otel.nix used to hold these guards and gated them on its
|
||
# own `enable`, so a swarm without the collector had no check at all.
|
||
reservedNames = import ../reserved-names.nix;
|
||
reservedFragments = import ../reserved-hive-fragments.nix;
|
||
|
||
# Subjects the store's cert-auth roles accept. A LIST, not one string: cert
|
||
# auth trusts the CA and `allowed_common_names` is the whole narrowing, so
|
||
# every role added beside these widens what a hive name must not collide
|
||
# with. A hive's own leaf carries its name as the CN, so a hive named after
|
||
# one of these presents a certificate that role accepts.
|
||
certAuthCns = [
|
||
deployCfg.bao.controllerCommonName
|
||
deployCfg.bao.secretPublisherCommonName
|
||
deployCfg.bao.matrixCtlCommonName
|
||
deployCfg.bao.grafanaOidcCommonName
|
||
deployCfg.bao.otelOidcCommonName
|
||
deployCfg.bao.forwarderOidcCommonName
|
||
deployCfg.bao.servicesIssuerCommonName
|
||
deployCfg.bao.natsCommonName
|
||
]
|
||
# The two per-hive readers' subjects, spelled out per hive rather than as the
|
||
# prefix. The prefix alone would reserve the wrong string: the role for hive
|
||
# `h` accepts `<prefix>-h`, so the name a hive must not BE is that composed
|
||
# spelling, and a hive named it would present a leaf that role accepts.
|
||
#
|
||
# Both spellings are reserved for every declared hive, including the one the
|
||
# CN belongs to — a hive cannot be named after its own reader either.
|
||
++ lib.concatMap (hive: [
|
||
"${deployCfg.bao.matrixTokenCommonNamePrefix}-${hive}"
|
||
"${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}"
|
||
]) hiveNames;
|
||
|
||
# Public hostnames of the swarm's own services, in declaration order.
|
||
# `serviceDomains` below is this set sorted + deduplicated.
|
||
#
|
||
# ⚠️ These are NOT required to be under `swarm.domain`. An earlier
|
||
# revision asserted that, reasoning that the services sub-CA is
|
||
# constrained to the swarm's tree — but the sub-CA is constrained to
|
||
# the **configured names** (./swarm-ca.nix) and the swarm root carries
|
||
# no name constraints at all, so any configured name is issuable. The
|
||
# assertion encoded an intended shape, not a property of the code, and
|
||
# it rejected the supported migration path: a hive pinning its old
|
||
# `forge.<hive domain>` while joining a swarm at a different apex.
|
||
serviceDomains' = [
|
||
swarmCfg.forge.domain
|
||
swarmCfg.matrix.gatewayHost
|
||
swarmCfg.authelia.domain
|
||
]
|
||
# The swarm UI's name is a SIBLING of the other three, not a parent of
|
||
# them — the apex is as much a name needing a certificate as
|
||
# `forge.<apex>` is, and no CA in the hierarchy issues for it
|
||
# implicitly. Left out, its vhost falls back to the hive leaf and the
|
||
# swarm's front page opens with a name mismatch.
|
||
++ lib.optional (deployCfg.swarm-ui.enable && swarmCfg.domain != null) swarmCfg.domain
|
||
# Every swarm service that claims a gateway name belongs here, and
|
||
# these three were missing it. Membership is what `gateway.lib.tlsFor`
|
||
# consults to pick the services leaf over the hive one, so a name
|
||
# absent from this list is served the HIVE certificate — which cannot
|
||
# cover a name under a different apex however the sub-CA is set up.
|
||
#
|
||
# ⚠️ How that stayed invisible: a mismatch a browser shows as a
|
||
# click-through warning is one a machine client rejects outright. The
|
||
# metrics UI and store looked fine for as long as only people opened
|
||
# them; the collector's exporter — same defect, no human in the loop —
|
||
# failed every POST and dropped the samples.
|
||
++ lib.optional deployCfg.grafana.enable swarmCfg.grafana.domain
|
||
++ lib.optional deployCfg.victoriametrics.enable swarmCfg.victoriametrics.domain
|
||
++ lib.optional deployCfg.swarm-otel.enable swarmCfg.otel.domain
|
||
# VictoriaLogs' vhost is new (was previously unpublished entirely — see
|
||
# swarm-victorialogs.nix's file-top comment) and needs the same
|
||
# membership every gateway-published swarm service needs: absent from
|
||
# this list, `gateway.lib.tlsFor` falls back to the hive leaf, which
|
||
# cannot cover a name under a different apex — see the ⚠️ above this
|
||
# list for what that looked like the last time a name was missed here.
|
||
++ lib.optional deployCfg.victorialogs.enable swarmCfg.victorialogs.domain;
|
||
|
||
# Hives whose entry still carries the removed `certFingerprint`. Scanned
|
||
# here, at top level, because that is the only place an assertion about a
|
||
# submodule field can actually live — see the option's own comment.
|
||
pinnedHives = lib.attrNames (
|
||
lib.filterAttrs (_: hive: hive.certFingerprint != null) swarmCfg.hives
|
||
);
|
||
|
||
# Whether this hive's minted status secret is on THIS host, which is the
|
||
# one status-publishing coordinate that is a statement about this
|
||
# machine's disk.
|
||
#
|
||
# ⚠️ Neither the token endpoint nor the queue's URL is gated on it. Both
|
||
# are the swarm's one address: the endpoint derived from
|
||
# `swarm.authelia.url`, the queue from its name, which resolves on every
|
||
# hive. So they are correct wherever the queue and the IdP run, and do
|
||
# not ask where anything runs.
|
||
queueLocal = deployCfg.nats.enable && deployCfg.authelia.enable && cfg.hiveName != null;
|
||
in
|
||
{
|
||
options.services.hyperhive.swarm.hives = lib.mkOption {
|
||
type = lib.types.attrsOf (
|
||
lib.types.submodule (
|
||
{ name, ... }:
|
||
{
|
||
options = {
|
||
domain = lib.mkOption {
|
||
type = lib.types.str;
|
||
# `<name>.<swarm.domain>` is a derivation from two values an
|
||
# operator had to state explicitly (both are required), not a
|
||
# guess — and it is what makes this directory worth copying:
|
||
# a conventional swarm is `{ pr1ma = { }; umbra = { }; }`,
|
||
# names only, with a non-conventional hive saying so and only
|
||
# that. A shared file is read far more often than written.
|
||
#
|
||
# Total rather than a throw when `swarm.domain` is unset, for
|
||
# the reason ./hive-network.nix:155 gives in full: defaults
|
||
# that interpolate the domain are forced *while the assertion
|
||
# list evaluates*, so a throw here would replace the message
|
||
# naming the missing option with a coercion error naming this
|
||
# one. `.invalid` is reserved (RFC 2606) and fails loudly at
|
||
# resolution if it ever escaped — which the required-domain
|
||
# assertion is there to stop.
|
||
default = if swarmCfg.domain == null then "${name}.invalid" else "${name}.${swarmCfg.domain}";
|
||
defaultText = lib.literalExpression ''"''${name}.''${services.hyperhive.swarm.domain}"'';
|
||
example = "lab.example.com";
|
||
description = ''
|
||
Public DNS domain this hive occupies — used for
|
||
swarm-controller's hive roster, agent identity
|
||
(qualified `agent@domain` labels), and Matrix federation
|
||
discovery.
|
||
|
||
Defaults to `<name>.<swarm.domain>`, the convention every
|
||
hive in a swarm follows, so a conventional directory is
|
||
names only. Set it for a hive that is addressed by
|
||
something else.
|
||
'';
|
||
};
|
||
|
||
wireguardPublicKey = lib.mkOption {
|
||
type = lib.types.nullOr lib.types.str;
|
||
default = null;
|
||
example = "base64pubkey=";
|
||
description = ''
|
||
WireGuard public key for this hive's host. Required when
|
||
`services.hyperhive.deploy.wireguard.enable = true` and
|
||
you want this hive reachable over the mesh. Null = TLS-
|
||
only peering (public internet, no mesh tunnel).
|
||
'';
|
||
};
|
||
|
||
wireguardEndpoint = lib.mkOption {
|
||
type = lib.types.nullOr lib.types.str;
|
||
default = null;
|
||
example = "203.0.113.1:51820";
|
||
description = ''
|
||
WireGuard endpoint for this hive in `host:port` form.
|
||
Null = this hive has no reachable endpoint, so the tunnel
|
||
is initiated from the other side.
|
||
|
||
Reads like a fact about the relationship and is not: it
|
||
says whether *this* hive can be dialled, which every other
|
||
hive in the swarm needs the same answer to.
|
||
'';
|
||
};
|
||
|
||
wireguardAddress = lib.mkOption {
|
||
type = lib.types.nullOr lib.types.str;
|
||
default = null;
|
||
example = "10.100.0.2/32";
|
||
description = ''
|
||
IP address (with prefix) of this hive's host on the
|
||
WireGuard mesh. Used as the `allowedIPs` for its
|
||
WireGuard config entry (`./swarm-wireguard.nix`), so
|
||
intra-swarm traffic can route over the mesh address
|
||
rather than the public domain. Required to include
|
||
a hive in the mesh (entries missing this field are
|
||
silently excluded from `wg-hive`).
|
||
'';
|
||
};
|
||
|
||
# Removed, and re-declared only so an existing definition
|
||
# produces an error that says *what happened*. Without it the
|
||
# module system says `The option ... does not exist`, which
|
||
# tells an operator nothing about why it went or what replaced
|
||
# it — and this field was set on real deployments.
|
||
#
|
||
# ⚠️ `lib.mkRemovedOptionModule` cannot do this job here, and
|
||
# neither of its two halves survives the move into a submodule:
|
||
# its `apply = throw` fires only when the value is READ, and
|
||
# nothing reads this any more (that being the point of removing
|
||
# it); its `config.assertions` half would land on a submodule
|
||
# that declares no `assertions` option at all. It is a
|
||
# top-level tool. The assertion below is the working shape, and
|
||
# it is the same one ./swarm-peers-removed.nix uses for the
|
||
# analogous `peers.<hive>.caCert`.
|
||
certFingerprint = lib.mkOption {
|
||
type = lib.types.nullOr lib.types.str;
|
||
default = null;
|
||
visible = false;
|
||
internal = true;
|
||
description = ''
|
||
Removed. Trust inside a swarm comes from the swarm root CA
|
||
(`services.hyperhive.swarm.ca`, docs/swarm/ca.md), which
|
||
replaces per-hive leaf pinning entirely.
|
||
'';
|
||
};
|
||
};
|
||
}
|
||
)
|
||
);
|
||
default = { };
|
||
example = {
|
||
pr1ma = {
|
||
wireguardAddress = "10.100.0.1/32";
|
||
wireguardEndpoint = "203.0.113.1:51820";
|
||
};
|
||
edge = {
|
||
domain = "edge.elsewhere.example";
|
||
wireguardAddress = "10.100.0.2/32";
|
||
};
|
||
};
|
||
description = ''
|
||
Every hive in this swarm, keyed by `hiveName` — **including this
|
||
host's own hive**. The same attrset is meant to be identical on
|
||
every host in the swarm, so it can be written once and shared;
|
||
`services.hyperhive.hiveName` is what makes a given host read it
|
||
as "me and four others" rather than "five peers".
|
||
|
||
Each entry needs no fields at all in the conventional case: a
|
||
hive's `domain` defaults to `<name>.<swarm.domain>`, so the whole
|
||
directory is usually a list of names.
|
||
|
||
It must contain an entry for `hiveName`, which is asserted — this
|
||
host's own address is read out of it (it is where
|
||
`services.hyperhive.domain` derives from), and a hive that lists
|
||
everyone but itself would otherwise derive its own peer set as
|
||
*everything* and peer with itself.
|
||
'';
|
||
};
|
||
|
||
options.services.hyperhive.swarm.peerHives = lib.mkOption {
|
||
type = lib.types.attrsOf (lib.types.attrsOf lib.types.unspecified);
|
||
readOnly = true;
|
||
internal = true;
|
||
description = ''
|
||
Read-only: `hives` minus this host's own entry. Derived once here
|
||
rather than in each consumer, because "everything that isn't me"
|
||
is a filter four different modules were re-implementing and only
|
||
one of them has to be wrong for a hive to peer with itself.
|
||
'';
|
||
};
|
||
|
||
options.services.hyperhive.swarm.serviceDomains = lib.mkOption {
|
||
type = lib.types.listOf lib.types.str;
|
||
readOnly = true;
|
||
internal = true;
|
||
description = ''
|
||
Read-only: the public hostnames of the swarm's own services, in a
|
||
stable sorted order. Second derived set alongside `peerHives`, and
|
||
here for the same reason — the CA that name-constrains these and
|
||
the leaf that carries them as SANs must agree exactly, and two
|
||
modules each assembling the list is how they stop agreeing.
|
||
|
||
Sorted and deduplicated deliberately: consumers compare this list
|
||
against what they issued last time to decide whether to re-issue,
|
||
so an unstable order would churn a certificate that other things
|
||
are meant to pin.
|
||
'';
|
||
};
|
||
|
||
options.services.hyperhive.swarm.localServiceDomains = lib.mkOption {
|
||
type = lib.types.listOf lib.types.str;
|
||
readOnly = true;
|
||
internal = true;
|
||
description = ''
|
||
Read-only: the names in `serviceDomains` **this host actually
|
||
fronts**, which is what its services leaf may carry. The swarm-wide
|
||
set stays what the sub-CA is name-constrained to and what the
|
||
store's issuing role permits; narrowing the leaf is what stops a
|
||
private key on one gateway presenting a valid certificate for
|
||
services that host does not serve and never has.
|
||
|
||
Read off the rendered `services.nginx.virtualHosts` rather than off
|
||
the per-service deploy flags: those flags are what the vhosts are
|
||
already guarded on, so a second reading of them is a copy that
|
||
drifts from the thing it claims to describe. Filtering
|
||
`serviceDomains` rather than assembling a list beside it is that
|
||
option's own constraint, and it applies here for the same reason.
|
||
'';
|
||
};
|
||
|
||
config = {
|
||
services.hyperhive.swarm.peerHives = lib.filterAttrs (name: _: name != cfg.hiveName) swarmCfg.hives;
|
||
|
||
services.hyperhive.swarm.serviceDomains = lib.sort (a: b: a < b) (
|
||
lib.unique (lib.filter (d: d != null && d != "") serviceDomains')
|
||
);
|
||
|
||
services.hyperhive.swarm.localServiceDomains = lib.filter (
|
||
d: config.services.nginx.virtualHosts ? ${d}
|
||
) swarmCfg.serviceDomains;
|
||
|
||
assertions = [
|
||
{
|
||
# An EMPTY `hives` fires this too, deliberately: since
|
||
# `swarm.domain` became required, every hive is in a swarm — a
|
||
# swarm of one is still a swarm — so a directory with no entry
|
||
# for this host is missing one either way. It also has to fire
|
||
# here, because this host's own domain is now read out of the
|
||
# directory: without the entry `services.hyperhive.domain` is
|
||
# null and the generic required-domain assertion in
|
||
# ./hive-network.nix would fire instead, naming an option the
|
||
# operator should no longer be setting.
|
||
#
|
||
# Guarded on `hiveName != null` so the required-hiveName
|
||
# assertion in ./hyperhive.nix is what fires for that case —
|
||
# two assertions naming the same missing value is noise.
|
||
assertion = cfg.hiveName == null || swarmCfg.hives ? ${cfg.hiveName};
|
||
message = ''
|
||
services.hyperhive.swarm.hives has no entry for this hive
|
||
(services.hyperhive.hiveName = "${toString cfg.hiveName}").
|
||
|
||
`hives` describes every hive in the swarm including this one,
|
||
so that every host can share one identical attrset, and this
|
||
hive's own domain is derived from its entry. Add:
|
||
|
||
services.hyperhive.swarm.hives."${toString cfg.hiveName}" = { };
|
||
|
||
No fields are needed: `domain` defaults to
|
||
`<name>.<swarm.domain>`. Set it in the entry if this hive is
|
||
addressed by something else.
|
||
|
||
Declared hives: ${lib.concatStringsSep ", " (lib.attrNames swarmCfg.hives)}
|
||
'';
|
||
}
|
||
{
|
||
# An error rather than a warning, deliberately. Re-declaring the
|
||
# removed option above is what stops "option does not exist"; on
|
||
# its own it would also turn a config that used to FAIL into one
|
||
# that quietly evaluates with the setting ignored, which is a
|
||
# worse answer than the unhelpful error it replaces. The operator
|
||
# asked for a removal error, so this stays fatal until the field
|
||
# is gone from their config.
|
||
#
|
||
# Wording follows `lib.mkRemovedOptionModule`'s, so it reads like
|
||
# every other removal the module system reports.
|
||
assertion = pinnedHives == [ ];
|
||
message = ''
|
||
The option definition `services.hyperhive.swarm.hives.<hive>.certFingerprint'
|
||
no longer has any effect; please remove it.
|
||
|
||
Still set on: ${lib.concatStringsSep ", " pinnedHives}
|
||
|
||
It pinned a peer hive's TLS leaf for hive-c0re's own peer HTTPS
|
||
checks, and was removed along with the dashboard feature it
|
||
existed to serve — nothing else ever consumed it.
|
||
|
||
There is no replacement, and none is needed for a hive inside
|
||
this swarm: trust comes from the swarm root CA
|
||
(services.hyperhive.swarm.ca — see docs/swarm/ca.md), which
|
||
every hive chains to, so one anchor replaces per-hive pinning.
|
||
What that genuinely drops is trusting a hive whose root this
|
||
swarm does NOT own — another swarm's, or one keeping its own
|
||
CA. That is a cross-swarm problem and wants a mechanism
|
||
designed for it.
|
||
'';
|
||
}
|
||
{
|
||
# Deliberately an assertion and not a silent "then publish
|
||
# nothing": a secret without the coordinates it is presented at is
|
||
# a config an operator believes is working, and its runtime failure
|
||
# mode is the expensive one — the daemon comes up fine, never
|
||
# connects, and the hive reads `never_reported` on a dashboard
|
||
# nobody is watching yet.
|
||
#
|
||
# It encodes a property of the code rather than an intended shape —
|
||
# hive-c0re genuinely cannot publish with two of three coordinates —
|
||
# which is the distinction the `serviceDomains'` comment at the top
|
||
# of this file was written about.
|
||
#
|
||
# The three coordinates live in two namespaces now: the token
|
||
# endpoint is the swarm's one address, the other two are this
|
||
# machine's. The message spells all three paths out, because an
|
||
# operator told only the option names would look for them under
|
||
# one prefix and find one of them.
|
||
#
|
||
# ⚠️ Asymmetric on purpose. The token endpoint and the queue's URL
|
||
# both default to the swarm's one address on every hive: they say
|
||
# where the IdP and the queue are, not that this hive publishes, so
|
||
# either without the secret is publishing off rather than a half-set
|
||
# trio. Only the client secret is a claim to be publishing, and it
|
||
# needs the other two to mean anything.
|
||
assertion =
|
||
deployCfg.hive-controller.statusPublish.clientSecretFile == null
|
||
|| (
|
||
deployCfg.hive-controller.statusPublish.natsUrl != null
|
||
&& swarmCfg.statusPublish.tokenEndpoint != null
|
||
);
|
||
message = ''
|
||
This hive has a status-publishing client secret but not the
|
||
coordinates to publish with it.
|
||
|
||
Currently:
|
||
deploy.hive-controller.statusPublish.natsUrl
|
||
= ${toString deployCfg.hive-controller.statusPublish.natsUrl}
|
||
swarm.statusPublish.tokenEndpoint
|
||
= ${toString swarmCfg.statusPublish.tokenEndpoint}
|
||
deploy.hive-controller.statusPublish.clientSecretFile
|
||
= ${toString deployCfg.hive-controller.statusPublish.clientSecretFile}
|
||
|
||
Set the missing ones to publish this hive's status to the
|
||
swarm, or set clientSecretFile to null to turn publishing off.
|
||
'';
|
||
}
|
||
(nameGuards.mustNotEqual {
|
||
option = "services.hyperhive.swarm.hives";
|
||
names = hiveNames;
|
||
reserved = reservedNames ++ certAuthCns;
|
||
why = ''
|
||
A hive's name is what other components address it by, and two
|
||
of those uses resolve the clash silently rather than erroring.
|
||
|
||
The collector names components `<kind>/<owner>` with the hive
|
||
name as owner, and `//` merges them: the swarm tier's parts
|
||
win, that hive's pipeline and `hive=` stamp disappear, and it
|
||
keeps pushing into a route that goes nowhere.
|
||
|
||
The secret store's cert-auth roles match on a certificate's
|
||
common name, and a hive's own leaf carries its name. A hive
|
||
named after a role's subject presents a certificate that role
|
||
accepts — so it receives that principal's grants, which for
|
||
the controller means write access to every hive's credentials
|
||
and policies.
|
||
|
||
Rename the hive.
|
||
'';
|
||
})
|
||
(nameGuards.mustNotContain {
|
||
option = "services.hyperhive.swarm.hives";
|
||
names = hiveNames;
|
||
fragments = reservedFragments;
|
||
why = ''
|
||
Hive-scoped identifiers are composed from a hive name —
|
||
`hive-<name>`, `hive-<name>-agent` — so a name containing one of
|
||
these produces an identifier that is also somebody else's. The
|
||
failure is a wrong grant rather than an error: the client
|
||
authenticates and receives another principal's permissions, and a
|
||
NATS denial arrives as a timeout. Rename the hive.
|
||
'';
|
||
})
|
||
];
|
||
};
|
||
|
||
# `allSwarmServices` is declared in ./swarm-required-services.nix
|
||
# together with the per-service `enable`s it asserts — it is a
|
||
# deployment-shape switch rather than swarm bookkeeping, so it lives
|
||
# with its consequences instead of here.
|
||
|
||
options.services.hyperhive.swarm.snapshotStore = {
|
||
address = lib.mkOption {
|
||
type = lib.types.nullOr lib.types.str;
|
||
default = null;
|
||
example = "10.100.0.1";
|
||
description = ''
|
||
Mesh address of the swarm's snapshot store — the single
|
||
`btrfs receive` endpoint every hive in this swarm pushes agent
|
||
snapshots to. Bare IP, no prefix.
|
||
|
||
There is exactly **one** store per swarm, not one per peer: the
|
||
receiver keys destinations by *agent*, so an agent that migrates
|
||
between hives keeps a single unbroken incremental chain. Per-hive
|
||
stores would split that chain in two, which is the case the store
|
||
exists to serve.
|
||
|
||
Null means this swarm has no store configured, and pushing fails
|
||
saying so rather than guessing an address. Set it on every hive
|
||
that pushes; the receiving host separately sets
|
||
`services.hyperhive.snapshotStore.enable`.
|
||
'';
|
||
};
|
||
|
||
port = lib.mkOption {
|
||
type = lib.types.port;
|
||
default = 51821;
|
||
description = ''
|
||
TCP port the swarm's snapshot store listens on. Must match the
|
||
receiving host's `services.hyperhive.snapshotStore.port`.
|
||
|
||
Defaulted (unlike `address`) because it is a shared convention
|
||
both sides read from the same option docs — whereas an address
|
||
is deployment-specific and cannot be guessed.
|
||
'';
|
||
};
|
||
};
|
||
|
||
# How this hive reaches the swarm queue to offer its own status
|
||
# (hive-c0re's `swarm_status`). Three coordinates: the token endpoint
|
||
# below follows the swarm's IdP address wherever that is, and the two
|
||
# under `deploy.hive-controller` default from the local swarm services
|
||
# when this host runs them and are set by hand otherwise.
|
||
#
|
||
# There is no `enable`: three coordinates that are all set is the
|
||
# enable. An extra flag would let a hive be configured-but-off, which
|
||
# is one more state to explain and one more way to be silently quiet.
|
||
options.services.hyperhive.swarm.statusPublish = {
|
||
tokenEndpoint = lib.mkOption {
|
||
type = lib.types.nullOr lib.types.str;
|
||
default = "${swarmCfg.authelia.url}/api/oidc/token";
|
||
defaultText = lib.literalExpression ''"''${swarm.authelia.url}/api/oidc/token"'';
|
||
example = "https://auth.example.com/api/oidc/token";
|
||
description = ''
|
||
The swarm IdP's OAuth2 token endpoint. This hive mints a
|
||
`client_credentials` access token there and presents it when
|
||
connecting to the queue, which authenticates it as
|
||
`hive-<hiveName>` — the client
|
||
{file}`nix/host-modules/swarm-authelia.nix` already declares for
|
||
every entry in {option}`services.hyperhive.swarm.hives`.
|
||
'';
|
||
};
|
||
};
|
||
|
||
# What stays above is the one thing every hive shares: the address of the
|
||
# swarm's token endpoint. The other two coordinates are this machine's —
|
||
# where the queue is *as seen from here*, and where its secret sits on this
|
||
# disk — so they hang off the daemon that uses them. `hive-controller` is
|
||
# hive-c0re's new name, which is why the namespace is the daemon's rather
|
||
# than a `deploy.statusPublish` of its own.
|
||
#
|
||
# ⚠️ The all-or-nothing assertion above now spans both namespaces. That is
|
||
# one service's own options either side of the split, not a service reaching
|
||
# into a foreign `deploy.*`; its message names all three paths so an operator
|
||
# is not told to set two options under a path that has one.
|
||
options.services.hyperhive.deploy.hive-controller.statusPublish = {
|
||
natsUrl = lib.mkOption {
|
||
type = lib.types.nullOr lib.types.str;
|
||
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
|
||
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}"'';
|
||
example = "tls://nats.example.com:4222";
|
||
description = ''
|
||
Where the swarm queue listens, as seen from *this* hive.
|
||
|
||
Defaults to the queue's name on every hive. The queue's own host
|
||
resolves it locally, and any other hive through the operator's
|
||
DNS. TLS only, and by name, since the queue's certificate carries
|
||
the name and no address.
|
||
|
||
A URL alone does not publish.
|
||
{option}`services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile`
|
||
is what turns publishing on. Without it this hive computes its own
|
||
readiness as always and offers it to nobody, and the swarm
|
||
controller reports it `never_reported`, which is the honest
|
||
reading.
|
||
'';
|
||
};
|
||
|
||
clientSecretFile = lib.mkOption {
|
||
type = lib.types.nullOr lib.types.str;
|
||
default =
|
||
if queueLocal then
|
||
"${deployCfg.authelia.hostClientSecretDir}/hive-${cfg.hiveName}.secret"
|
||
else
|
||
null;
|
||
defaultText = lib.literalExpression ''"''${deploy.authelia.hostClientSecretDir}/hive-''${hiveName}.secret" when this host runs both the queue and the IdP, else null'';
|
||
example = "/var/lib/secrets/swarm-queue-client.secret";
|
||
description = ''
|
||
Path to a file holding the plaintext client secret for this
|
||
hive's `hive-<hiveName>` identity.
|
||
|
||
A path and not a value: a secret in the Nix store is world
|
||
readable, and one in the environment is readable by anything
|
||
that can open {file}`/proc/<pid>/environ`.
|
||
|
||
Setting it is what makes this hive publish its status; null
|
||
turns publishing off.
|
||
|
||
Defaults to authelia's own minted secret when the IdP runs on
|
||
this host. On any other hive the secret has to get here somehow,
|
||
and the swarm does not distribute it — copy it out of the swarm
|
||
host's
|
||
{option}`services.hyperhive.deploy.authelia.hostClientSecretDir`
|
||
with whatever secret management this deployment already uses.
|
||
'';
|
||
};
|
||
};
|
||
|
||
# The same queue, reached from one layer further in. An agent container has
|
||
# its own network namespace and resolver, so this is its own option, even
|
||
# though by name it defaults to the same URL as `statusPublish.natsUrl`.
|
||
#
|
||
# Only the address lives here. The credential does not: it is published per
|
||
# hive and read out of the store by ./glue-queue-agent-credential.nix, which
|
||
# owns `queue.agentCredentialDir` in the same namespace.
|
||
options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption {
|
||
type = lib.types.nullOr lib.types.str;
|
||
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
|
||
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}"'';
|
||
example = "tls://nats.example.com:4222";
|
||
description = ''
|
||
Where the swarm queue listens, as an agent *container* on this host
|
||
reaches it.
|
||
|
||
Defaults to the queue's name on every hive. The agent resolves it
|
||
through the bridge: on the queue's own host dnsmasq answers it with
|
||
the bridge address, and elsewhere it forwards to the operator's DNS,
|
||
as for the store's name. ⚠️ Never a loopback address: inside an
|
||
agent's network namespace `127.0.0.1` is the agent, not this host,
|
||
and the queue's certificate names no address anyway.
|
||
|
||
Null means this hive's agents have not been given the queue's address.
|
||
Together with
|
||
{option}`services.hyperhive.swarm.statusPublish.tokenEndpoint` it is
|
||
what decides whether the harness is handed queue coordinates at all.
|
||
'';
|
||
};
|
||
|
||
}
|