vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

Compare commits

base: vinzenz:22a87f72685839a9fafbc8e968169d0fa05ca92d
Branches Tags
vinzenz:main
..
compare: vinzenz:ebcc5bde89db739297e8d3c751a0a58f767bed2e
Branches Tags
vinzenz:main
10 changed files with 315 additions and 700 deletions
Download patch file Download diff file Expand all files Collapse all files

44
docs/swarm/ca.md
View file

@ -37,42 +37,26 @@ Moving the swarm CA onto its own host is then a matter of moving
`services.hyperhive.swarm.ca.stateDir` and leaving `autoConfigure` off —
there is no second code path to switch to.
### The swarm's service names come from the secret store
### The warning about uncovered service names
The swarm's service names (`swarm.forge.domain`,
`swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to
siblings of the hive domain — `forge.<swarm.domain>`, not
`forge.<hive domain>`. The hive CA's leaf is a **single-label** wildcard
over the hive's own domain, so it can't cover them.
over the hive's own domain, so it can't cover them; only the
swarm-services leaf can, and this host signs that one only under
`autoConfigure`. A hive with neither serves the hive leaf on those names
and every client sees a name mismatch.
The secret store's `pki` mount issues that leaf instead, from a
root generated inside it. `swarm-services-cert.service` logs in with
this host's services-issuer certificate, calls
`pki/issue/swarm-services`, and writes the result to
`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already
copies from. Every hive does this with its own identity, so holding the
swarm root's private key stopped being what decides whether a hive can
serve its swarm's names.
The same `services.hyperhive.swarm.serviceDomains` that builds the SANs
also populates the role's `allowed_domains`, so asking for a name nobody
configured is a refusal from the store naming that name — not a
certificate quietly issued for it.
**The root's public certificate is a file, on every hive:**
`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the
leaf it signed and folded into `trust-bundle.pem`. On the host running
the store it's also at
`/var/lib/swarm-bao-services-pki/services-root.pem`. That's the file to
hand a browser, and reading it needs no store login — which matters,
because every store listener demands a client certificate.
**The granting unit generates the root once, and never again.** It asks
the mount whether it already has an issuer (`bao list pki/issuers`)
before calling `pki/root/generate/internal`, so a rebuild or a reboot
re-asserts the role and the grant without touching the anchor. A root
that changed per boot would invalidate every certificate issued under it
and every browser taught to trust it.
`hive-tls` emits an eval-time **warning** naming the uncovered names.
It's deliberately not an assertion: this module can see what _it_ is
able to issue, but not an operator-installed sub-CA in
`swarm.ca.stateDir`, an external ACME setup, or a certificate delivered
by any other means. If you have arranged one, expect the warning and
ignore it. Otherwise either install the sub-CA, or pin the
names back under the hive domain — a supported migration, since the
sub-CA carries a name constraint on the _configured_ names and the
swarm root carries no name constraints at all.
## Constraints on the material

2
docs/swarm/secrets.md
View file

@ -45,7 +45,7 @@ neither is a renaming of the other.
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when the operator sets `autoConfigure` | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for |
| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives |
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |

12
docs/swarm/ui.md
View file

@ -91,12 +91,12 @@ own options, the way `swarm-ui.nix` and `swarm-authelia.nix` do.
<!-- vale write-good.Passive = YES -->
⚠️ The certificate one is the hardest to predict and the most visible when
missed. `serviceDomains` is _both_ the `allowed_domains` the secret
store's `pki/roles/swarm-services` narrows to and the leaf's SAN list,
and the apex is a **sibling** of `forge.<swarm>` / `chat.<swarm>` /
`auth.<swarm>`, not a parent — no CA in the hierarchy issues for it
implicitly. Left out, the vhost falls back to the hive leaf and the
swarm's front page opens with a name mismatch.
missed. `serviceDomains` is _both_ the services sub-CA's
`nameConstraints` set and the leaf's SAN list, and the apex is a
**sibling** of `forge.<swarm>` / `chat.<swarm>` / `auth.<swarm>`, not a
parent — no CA in the hierarchy issues for it implicitly. Left out, the
vhost falls back to the hive leaf and the swarm's front page opens with a
name mismatch.
## Quick links

1
nix/host-modules/default.nix
View file

@ -30,7 +30,6 @@
./glue-matrix-ctl-bao-identity.nix
./glue-queue-agent-credential.nix
./glue-secret-publisher-bao-identity.nix
./glue-services-issuer-bao-identity.nix
./glue-swarm-bao-otel-oidc-client.nix
./glue-swarm-otel-oidc-client.nix
./swarm-authelia.nix

15
nix/host-modules/glue-bao-tls.nix
View file

@ -208,21 +208,6 @@ in
[ -s ${pkiDir}/otel-oidc.pem ] || ${signLeaf} ${pkiDir} otel-oidc \
${lib.escapeShellArg deployCfg.bao.otelOidcCommonName} "" clientAuth
# The identity a hive presents to ask the store's `pki` mount for the
# swarm-services certificate its gateway serves. Minted here like the
# three above, and the reason is the sharpest of the four: this leaf
# is what OPENS the mint, so it cannot come out of it. A certificate
# authority that issues the credential admitting you to it is the
# cycle this file's header exists to keep out — which is exactly why
# the swarm-services root moved into the store and this one did not.
#
# Its own CN, not the reader's: the hive's own leaf reads every secret
# its policy names, while this principal may do one thing — `update`
# on `pki/issue/swarm-services` — and giving a renewal unit the wider
# credential would undo the separation the role was created for.
[ -s ${pkiDir}/services-issuer.pem ] || ${signLeaf} ${pkiDir} services-issuer \
${lib.escapeShellArg deployCfg.bao.servicesIssuerCommonName} "" clientAuth
'';
};
};

43
nix/host-modules/glue-services-issuer-bao-identity.nix
View file

@ -1,43 +0,0 @@
# Glue: point the swarm-services certificate unit at the bao leaf minted for
# it.
#
# ONE PAIRING PER FILE — the services-issuer principal ← bao, and nothing
# else. Deleting this leaves a hive that takes an operator-provided path to
# that credential, which is what any deployment not minting its own already
# does.
#
# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the
# leaf, because the thing that owns a private key owns issuing from it. What
# belongs here is the pairing: which paths `swarm-services-cert` reads.
#
# ⚠️ Gated on the leaf existing, not on the store running here — the same rule
# ./glue-controller-bao-identity.nix states, and it matters more for this
# principal than for any of its siblings: every hive in a swarm serves the
# swarm's service names and therefore needs this certificate, while exactly
# one of them hosts the store.
#
# Everything is `mkDefault`. An operator naming their own paths wins.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoDeploy = deployCfg.bao;
# Where ./glue-bao-tls.nix puts the leaves, derived from the reader's own
# path rather than repeating that file's directory literal: an operator who
# moves the PKI moves both, and the two cannot drift apart.
haveMintedPki = baoDeploy.clientCertFile != null;
pkiDir = if haveMintedPki then builtins.dirOf baoDeploy.clientCertFile else null;
in
{
config = lib.mkIf (hyperhiveCfg.enable && haveMintedPki) {
services.hyperhive.deploy.hive-controller.tls = {
baoClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
baoClientKeyFile = lib.mkDefault "${pkiDir}/services-issuer-key.pem";
};
};
}

512
nix/host-modules/hive-tls.nix
View file

@ -10,22 +10,26 @@ let
gatewayCfg = config.services.hyperhive.gateway;
swarmCaCfg = config.services.hyperhive.swarm.ca;
domain = hyperhiveCfg.domain;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = hyperhiveCfg.deploy.bao;
# The endpoint the services leaf comes out of. Read off the store's own
# options rather than spelled here: ./swarm-bao.nix enables the mount,
# generates its root and writes the grant that names this exact path, and a
# second module composing its own is how the grant and the caller stop
# agreeing.
baoServicesPkiMount = baoDeploy.servicesPkiMountPath;
baoServicesPkiRole = baoDeploy.servicesPkiRoleName;
# Derived once in ./swarm.nix and read here + in ./swarm-bao.nix, so
# the names this leaf carries as SANs and the names the store's
# `pki/roles/swarm-services` is narrowed to cannot disagree.
# Derived once in ./swarm.nix and read here + in ./swarm-ca.nix, so
# the names this leaf carries as SANs and the names the sub-CA is
# constrained to cannot disagree.
swarmServiceDomains = hyperhiveCfg.swarm.serviceDomains;
# True for the names `signHiveLeaf` below actually covers: the hive
# domain itself, or ONE label under it. `*.<domain>` is a single-label
# wildcard — `a.b.<domain>` does not match it — so the depth check is
# the whole point rather than a nicety.
coveredByHiveLeaf =
n:
n == domain
|| (lib.hasSuffix ".${domain}" n && !lib.hasInfix "." (lib.removeSuffix ".${domain}" n));
# Service names this host can serve a *matching* certificate for, and
# the ones it can't. A name is issuable here when the hive leaf covers
# it, or when this host issues the swarm-services leaf — which needs
# the swarm root's private key, i.e. `swarm.ca.autoConfigure`.
uncoveredServiceDomains = lib.filter (n: !coveredByHiveLeaf n) swarmServiceDomains;
# The host-managed hive CA is the trust anchor for self-signed mode.
# It is only stood up when the gateway actually serves a self-signed
# cert: the gateway must be in self-signed mode. `domain` is required
@ -178,38 +182,57 @@ let
"$d/ca.pem" "$d/ca-key.pem"
'';
# Shared by every unit that decides whether to re-issue a leaf. It
# lives here rather than in one of them because the guards have to
# agree: they answer the same question at different times
# (`hive-tls-ca` at service activation, i.e. on the deploy;
# `hive-tls-resign` from a weekly timer for a host that stays up long
# enough to drift; `swarm-services-cert` for the leaf the store
# issues). A rule implemented in one and not the others is worse than
# one implemented in none — it looks fixed and only fires on whichever
# path you did not take, which is exactly how a corrected
# `serviceDomains` still served a stale leaf.
# The swarm-services leaf: signed by the services sub-CA, covering the
# swarm's service names. Those are *siblings* of the hive domain, not
# children, so the hive CA is name-constrained out of them and cannot
# sign this however its SAN list is written.
#
# Skipped when the sub-CA isn't on disk: it only exists where the
# swarm CA is autoconfigured, and a hive that gets its certs from its
# operator has nothing for this to do.
signServicesLeaf = lib.optionalString (swarmServiceDomains != [ ]) ''
servicesCa=${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca.pem"}
servicesCaKey=${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca-key.pem"}
if [ -s "$servicesCa" ] && [ -s "$servicesCaKey" ]; then
${signLeafScript} "$d" swarm-services \
${lib.escapeShellArg (builtins.head swarmServiceDomains)} \
${lib.escapeShellArg (lib.concatMapStringsSep "," (n: "DNS:${n}") swarmServiceDomains)} \
"$servicesCa" "$servicesCaKey"
else
echo "no swarm-services sub-CA at $servicesCa — skipping the services leaf"
fi
'';
# Shared by BOTH units that decide whether to re-sign. It lives here
# rather than in one of them because the two guards have to agree: they
# answer the same question at different times (`hive-tls-ca` at service
# activation, i.e. on the deploy; `hive-tls-resign` from a weekly timer
# for a host that stays up long enough to drift). A rule implemented in
# one and not the other is worse than one implemented in neither — it
# looks fixed and only fires on whichever path you did not take, which
# is exactly how a corrected `serviceDomains` still served a stale leaf.
#
# Expects `$d` (state dir) to be set; defines `$leaf`-adjacent names and
# `covers`.
leafCoverage = ''
svcleaf="$d/swarm-services.pem"
svcroot="$d/swarm-services-root.pem"
hiveNames=${lib.escapeShellArg "${domain} *.${domain}"}
svcNames=${lib.escapeShellArg (lib.concatStringsSep " " swarmServiceDomains)}
# A hive with no configured service names has no services leaf to
# hold, and its absence there is the correct state rather than a
# stale one. Anywhere else it is expected: the leaf comes from the
# secret store's `pki` mount now, not from a sub-CA that exists on
# one host in the swarm, so "this host cannot issue it" is no longer
# one of the answers.
# The services leaf is only expected where the sub-CA exists;
# elsewhere its absence is the correct state, not a stale leaf.
want_svc=${if swarmServiceDomains == [ ] then "0" else "1"}
if [ ! -s ${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca.pem"} ]; then
want_svc=0
fi
# Expiry is not the only way a leaf goes wrong. One signed when the
# configured name set was smaller stays valid for its whole lifetime
# while omitting every name added since — so a config change can
# evaluate, build and deploy cleanly while the gateway keeps serving a
# certificate that does not cover the new service.
# certificate that does not cover the new service. The services sub-CA
# already reconciles this way (its `.names` comparison in
# ../swarm-ca.nix); the leaves did not.
#
# Reads the names out of the CERTIFICATE, not a sidecar file: the pem
# is what nginx serves, and a bookkeeping file drifts from it the
@ -231,20 +254,13 @@ let
[ "$_missing" = 0 ]
}
# True when the leaf this host SIGNS is present and carries its
# configured names. Expiry is the callers' own business — they use
# different windows.
#
# ⚠️ The services leaf is deliberately NOT part of this any more, and
# the omission is load-bearing rather than a simplification. Its two
# callers sign under the hive CA's key; the services leaf comes out
# of the secret store, which neither of them can reach. A guard that
# went on reporting it stale would make `hive-tls-ca` re-sign the
# hive leaf forever over a file it has no way to fix. The
# `swarm-services-cert` unit below holds the same rule for that leaf,
# using the `covers` helper above so the two cannot drift.
# True when every leaf this host is supposed to hold is present and
# carries its configured names. Expiry is the callers' own business —
# they use different windows.
leavesCoverNames() {
covers "$leaf" "$hiveNames"
covers "$leaf" "$hiveNames" || return 1
[ "$want_svc" = 0 ] && return 0
covers "$svcleaf" "$svcNames"
}
'';
in
@ -297,44 +313,6 @@ in
'';
};
baoClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/services-issuer.pem";
description = ''
Certificate `swarm-services-cert` presents to the secret store when it
asks for the swarm-services leaf the gateway serves.
Its subject has to be
{option}`services.hyperhive.deploy.bao.servicesIssuerCommonName` — cert
auth matches on the common name, and this is the one role whose policy
permits `pki/issue/swarm-services`.
A **separate** credential from
{option}`services.hyperhive.deploy.bao.clientCertFile`, which is this
hive's own reader identity and whose policy reads every secret the
hive is entitled to. A unit that renews one server certificate has no
business holding that.
On a hive that mints its own store PKI a glue module supplies the leaf
as a `mkDefault`; everywhere else it is the credential an operator
places, for the reason the store's own client certificate is.
A path, never a value.
'';
};
baoClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/services-issuer-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.hive-controller.tls.baoClientCertFile`.
Both or neither — a certificate with no key authenticates nothing.
'';
};
leafValidityDays = lib.mkOption {
type = lib.types.int;
default = 30;
@ -355,17 +333,40 @@ in
};
config = lib.mkIf active {
# 🚫 The "these service names are outside this hive's domain and this
# host cannot sign for them" warning is GONE, and its absence is the
# deliverable rather than a tidy-up. It fired on
# `!swarm.ca.autoConfigure` — i.e. "this host does not hold the swarm
# root key" — which was the whole reason a hive could end up serving
# its own leaf on a swarm-service name. The services leaf now comes
# from the secret store's `pki` mount, which every hive reaches over
# the network with its own identity, so holding the root key stopped
# being the thing that decides. Left in place it would warn, on every
# rebuild of every hive that is not the CA host, about a fallback
# that no longer happens.
# A swarm service name that is a sibling of the hive domain rather
# than a child needs the swarm-services leaf, and this host only
# signs that one when it holds the swarm root key. Without it nginx
# falls back to the hive leaf on those names and every client sees a
# name mismatch — on a config that evaluates and deploys cleanly.
#
# ⚠️ A WARNING, NOT AN ASSERTION, and the distinction is the point:
# this module can see what *it* can issue; it cannot see an
# operator-installed services sub-CA, an external ACME setup, or a
# cert delivered by any other means. A rebuild must not be blocked
# by a conclusion this host is not in a position to reach. Report
# the observation and let the operator judge it.
#
# (`certDir` / ACME modes don't reach here at all: `active` is
# self-signed-only, and there the operator's cert decides.)
warnings = lib.optional (uncoveredServiceDomains != [ ] && !swarmCaCfg.autoConfigure) ''
services.hyperhive: these swarm service names are outside this
hive's domain (${domain}), so the hive CA's leaf does not cover
them:
${lib.concatMapStringsSep "\n" (n: " ${n}") uncoveredServiceDomains}
services.hyperhive.swarm.ca.autoConfigure is false, so this host
does not sign the swarm-services leaf either, and the gateway will
serve the hive leaf on those names — a certificate-name mismatch
for browsers and for agents' git-over-https.
If you have already arranged certificates for them — an
operator-installed services sub-CA in ${swarmCaCfg.stateDir}, or
services.hyperhive.gateway.tls.{certDir,acme} — this is expected
and you can ignore it. Otherwise pin the names back under
${domain} (services.hyperhive.swarm.{forge.domain,
matrix.gatewayHost, authelia.domain}) or install the sub-CA.
'';
# Generate (and rotate) the hive CA + gateway leaf before anything
# serves it. Idempotent: the CA is created once and reused; the leaf
@ -380,22 +381,26 @@ in
# against the unit that reads the file.
before = [ "hive-gateway-self-signed-cert.service" ];
requiredBy = [ "hive-gateway-self-signed-cert.service" ];
# The issuance below needs the swarm root key on disk. When this
# host generates it (single-host swarm) that unit must have run
# first; when the operator provides the material there is no unit
# to wait for, so the dependency is conditional rather than a unit
# that exists and does nothing.
# The issuance below needs the swarm root key on disk, and (for the
# services leaf) the services sub-CA it signs under. When this host
# generates them (single-host swarm) both units must have run first;
# when the operator provides the material there is no unit to wait
# for, so the dependency is conditional rather than a unit that
# exists and does nothing.
#
# `swarm-services-cert` is ordered before this one rather than the
# other way round, and not because this unit needs the leaf — it
# never touches it. The trust bundle written at the end of this
# script anchors on every root a consumer must hold, and the
# services root is one of them, so the file has to be on disk
# before the `cat`. Get it wrong and the bundle is a boot behind:
# correct-looking, and missing the anchor for exactly the names the
# gateway serves with it.
after = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ];
requires = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ];
# Without waiting for swarm-services-ca specifically, this unit races
# it: if hive-tls-ca finishes first, it finds no services-ca.pem yet,
# silently skips signing the services leaf (the same as "operator
# hasn't set one up"), and the gateway comes up with a vhost pointed
# at a cert that was never written.
after = lib.optionals swarmCaCfg.autoConfigure [
"swarm-ca.service"
"swarm-services-ca.service"
];
requires = lib.optionals swarmCaCfg.autoConfigure [
"swarm-ca.service"
"swarm-services-ca.service"
];
path = [ pkgs.openssl ];
serviceConfig = {
Type = "oneshot";
@ -491,14 +496,23 @@ in
# up to a week late and does nothing for the rebuild that changed
# the names in the first place.
#
# The name check has to include the SERVICES leaf even though the
# condition is written around the hive one, because both are signed
# in this block: a fresh `gateway.pem` otherwise suppresses the
# re-sign of a `swarm-services.pem` that is missing or stale, which
# is what left a corrected `serviceDomains` still mis-served.
resign=0
{ [ -s "$leaf" ] && [ -s "$leafk" ]; } || resign=1
openssl x509 -in "$leaf" -noout -checkend 2592000 >/dev/null 2>&1 || resign=1
leavesCoverNames || resign=1
if [ "$want_svc" = 1 ] && [ ! -s "$svcleaf" ]; then
resign=1
fi
if [ "$resign" = 1 ]; then
echo "signing gateway leaf at $leaf (missing, near expiry, or missing a configured name)"
${signHiveLeaf}
${signServicesLeaf}
fi
# --- Trust bundle: what a consumer must TRUST, as opposed to
@ -520,19 +534,10 @@ in
# hive where the optional file is legitimately absent. The
# reflexive `|| true` is worse; it also swallows a real failure to
# read the hive CA.
#
# `swarm-services-root.pem` joins the set for the reason the swarm
# root is in it: it is an ANCHOR a consumer terminates at, not an
# intermediate. It is also what keeps this change from being a
# flag day — the swarm root stays in the bundle beside it, so a
# leaf still chaining to the old services sub-CA and one issued by
# the store both verify against the same file, and a hive can be
# rebuilt before or after its peers.
bundle="$d/trust-bundle.pem"
anchors=("$ca")
if [ -s "$prev" ]; then anchors+=("$prev"); fi
if [ -s "$root" ]; then anchors+=("$root"); fi
if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi
cat "''${anchors[@]}" > "$bundle"
chmod 0644 "$bundle"
'';
@ -577,17 +582,12 @@ in
d=${lib.escapeShellArg cfg.stateDir}
leaf="$d/gateway.pem"
# ⚠️ EVERY leaf this host SIGNS must be covered here. A leaf that
# ⚠️ EVERY leaf this host issues must be covered here. A leaf that
# first-boot issuance creates and this unit does not know about
# looks perfect for its entire validity and then expires with no
# warning — the failure is invisible until it is total. The name
# checks come from the shared snippet, so this unit and
# `hive-tls-ca` cannot disagree about what a good leaf is.
#
# The services leaf is not one of them any longer: it is issued
# by the secret store, whose key this unit does not hold and
# cannot re-sign under. Renewing it is `swarm-services-cert`'s
# job, at boot, which is a cadence its own issue owns.
# warning — the failure is invisible until it is total. `svcleaf`
# and the name checks come from the shared snippet, so this unit
# and `hive-tls-ca` cannot disagree about what a good leaf is.
${leafCoverage}
# Re-sign only when a leaf is within half its validity of expiry.
@ -598,17 +598,19 @@ in
[ -s "$1" ] && openssl x509 -in "$1" -noout -checkend "$halflife" >/dev/null 2>&1
}
if fresh "$leaf" && leavesCoverNames; then
if fresh "$leaf" && { [ "$want_svc" = 0 ] || fresh "$svcleaf"; } \
&& leavesCoverNames; then
echo "leaves valid, and covering the configured names — no resign needed"
exit 0
fi
echo "a leaf is missing, near expiry, or missing a configured name — re-signing"
before="$(sha256sum "$leaf" 2>/dev/null || true)"
before="$(sha256sum "$leaf" "$svcleaf" 2>/dev/null || true)"
${signHiveLeaf}
${signServicesLeaf}
after="$(sha256sum "$leaf" 2>/dev/null || true)"
after="$(sha256sum "$leaf" "$svcleaf" 2>/dev/null || true)"
if [ "$before" != "$after" ]; then
echo "gateway leaf rotated — re-importing and reloading nginx"
systemctl restart hive-gateway-self-signed-cert.service
@ -619,256 +621,6 @@ in
'';
};
# The swarm-services leaf, issued by the secret store's `pki` mount.
#
# ⚠️ This unit replaces a chain, not a command. What used to happen
# was two openssl hops on whichever host held the swarm root key: a
# name-constrained `swarm-services` sub-CA (../swarm-ca.nix), then a
# leaf signed under it here. Both are gone. The store's
# `pki/roles/swarm-services` carries the same narrowing as an
# `allowed_domains` list enforced at issue time, so the intermediate
# that used to encode it in x509 `nameConstraints` has nothing left
# to express — and with the root inside the mount there is nothing
# for it to be an intermediate OF.
#
# The names it may carry are not this unit's to assert: the role
# refuses anything outside `allowed_domains`, which is read from the
# same `swarm.serviceDomains` the SANs below are built from. A
# mismatch is a refusal from the store naming the offending name,
# not a certificate quietly issued for something nobody configured.
systemd.services.swarm-services-cert = {
description = "Issue the swarm-services TLS leaf from the secret store's PKI";
wantedBy = [ "multi-user.target" ];
# ⚠️ The ordering the old shape got by ACCIDENT, written down.
# Nothing used to declare "no TLS until the store is up" — it held
# because `swarm-services-ca.service` happened to block the gateway.
# Now it is a real edge: the gateway's certificate comes out of the
# store, so the store being up is a precondition rather than a
# coincidence, and `requiredBy` is what makes a store that never
# comes up an outage that says so instead of a gateway serving the
# wrong name.
before = [
"hive-tls-ca.service"
"hive-gateway-self-signed-cert.service"
];
requiredBy = [ "hive-gateway-self-signed-cert.service" ];
# The store's container, where it runs here. On a hive that reads a
# store hosted elsewhere no such unit exists and systemd ignores
# the name, which is the correct behaviour rather than a gap: what
# this unit actually needs is the store reachable, and on a remote
# store nothing local can order against that.
after = [
"container@${baoCfg.machine}.service"
"swarm-bao-services-issuer-policy.service"
];
wants = [ "container@${baoCfg.machine}.service" ];
path = [
baoDeploy.package
pkgs.jq
pkgs.openssl
pkgs.coreutils
pkgs.systemd
];
# Sized like the store's own granting units, and for the same
# reason: under `seal = "shamir"` an operator unseals BY HAND, and
# the login below fails for as long as that takes. 2880 × 30s is
# 24h inside a 25h window — `StartLimit*` are `[Unit]` settings, so
# the window must exceed `RestartSec × burst` or it closes between
# attempts and the burst is never reached.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
UMask = "0077";
SyslogIdentifier = "swarm-services-cert";
Restart = "on-failure";
RestartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs (cfg.baoClientCertFile != null) {
BAO_CLIENT_CERT = cfg.baoClientCertFile;
}
// lib.optionalAttrs (cfg.baoClientKeyFile != null) {
BAO_CLIENT_KEY = cfg.baoClientKeyFile;
}
# Absent means the system trust store, which is what a deployment
# with a real CA wants and what a self-signed one must not be left
# with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
d=${lib.escapeShellArg cfg.stateDir}
install -d -m 0755 "$d"
${leafCoverage}
if [ "$want_svc" = 0 ]; then
echo "no swarm service domains configured — nothing to issue"
exit 0
fi
# Same rule as the hive leaf's, at a different cadence: re-issue
# when the file is missing, within 30 days of expiry, or no
# longer carrying every configured name. The name check is what
# makes adding a swarm service take effect on the rebuild that
# added it rather than whenever the certificate happens to lapse.
reissue=0
{ [ -s "$svcleaf" ] && [ -s "$d/swarm-services-key.pem" ]; } || reissue=1
openssl x509 -in "$svcleaf" -noout -checkend 2592000 >/dev/null 2>&1 || reissue=1
covers "$svcleaf" "$svcNames" || reissue=1
[ -s "$svcroot" ] || reissue=1
if [ "$reissue" = 0 ]; then
echo "swarm-services leaf valid and covering the configured names — leaving it alone"
exit 0
fi
${
if cfg.baoClientCertFile == null || cfg.baoClientKeyFile == null then
''
echo "no bao client certificate configured for this host, so the gateway's" >&2
echo "swarm-services certificate cannot be requested from the store." >&2
echo "Set services.hyperhive.deploy.hive-controller.tls.baoClient{Cert,Key}File" >&2
echo "to a leaf the store's CA signed with CN=${baoDeploy.servicesIssuerCommonName}." >&2
exit 1''
else
""
}
# `bao`'s own message is the only thing separating an unreachable
# store from a refused identity from a role that would not issue
# these names, and this unit retries on all three — so it reports
# which one rather than asserting all three in a sentence of ours.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting: the
# `BAO_CLIENT_*` variables only decide which certificate the
# handshake presents. Without a token `bao` asks its token
# helper, and that is a `sh` this unit's `path` does not carry —
# `-token-only` answers on stdout and skips the helper.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to the swarm secret store with this host's services-issuer certificate." >&2
cat "$err" >&2
exit 1
fi
export BAO_TOKEN
echo "requesting the swarm-services leaf from the store"
resp="$(mktemp "$d/swarm-services.json.XXXXXX")"
trap 'rm -f "$err" "$resp"' EXIT
# ONE call, and the response carries all three artefacts —
# certificate, private key and issuing CA. Splitting it into a
# local CSR plus `pki/sign` would buy a key that never crosses
# the wire, and cost the thing that makes this unit worth having:
# the store, not this host, decides what the certificate says.
if ! bao write -format=json \
${lib.escapeShellArg "${baoServicesPkiMount}/issue/${baoServicesPkiRole}"} \
common_name=${lib.escapeShellArg (builtins.head swarmServiceDomains)} \
alt_names=${lib.escapeShellArg (lib.concatStringsSep "," swarmServiceDomains)} \
> "$resp" 2>"$err"; then
echo "the store refused to issue the swarm-services certificate." >&2
cat "$err" >&2
exit 1
fi
# Three fields out of ONE issuance, which is why the response is
# captured and then split rather than asked for three times with
# `bao write -field=…`: each of those calls would mint a separate
# certificate, and the key would belong to a different one than
# the leaf.
#
# `jq -r` into a file the shell opened, never into a variable and
# never into an argv — the private key does not become visible in
# /proc on its way to disk.
#
# `// empty` on every one of them: `jq -r` renders a missing
# field as the four characters `null`, which is a non-empty file
# and would sail past the check below into a certificate slot.
umask 077
jq -r '.data.private_key // empty' < "$resp" > "$d/swarm-services-key.pem.new"
jq -r '.data.certificate // empty' < "$resp" > "$d/swarm-services-leaf.new"
jq -r '.data.issuing_ca // empty' < "$resp" > "$svcroot.new"
for f in "$d/swarm-services-key.pem.new" "$d/swarm-services-leaf.new" "$svcroot.new"; do
if [ ! -s "$f" ]; then
echo "the store's response was missing a field: $f is empty" >&2
exit 1
fi
done
# nginx serves this file verbatim. It carries the leaf ALONE and
# not leaf-plus-issuer, which is the one visible consequence of
# flattening the chain: the issuer here is a self-signed root, so
# appending it would ship an anchor a client either already holds
# — in which case it is redundant — or does not, in which case
# shipping it proves nothing. The old file appended the services
# sub-CA because that WAS an intermediate, and a client anchored
# on the swarm root could not build the middle of the chain by
# itself.
mv -f "$d/swarm-services-leaf.new" "$svcleaf"
mv -f "$d/swarm-services-key.pem.new" "$d/swarm-services-key.pem"
chmod 0644 "$svcleaf"
chmod 0600 "$d/swarm-services-key.pem"
# The anchor, world-readable, beside the leaf it signed. This is
# the file an operator adds to a browser's trust store, and the
# one `hive-tls-ca` folds into `trust-bundle.pem`. Public
# certificate material only — the private half never leaves the
# store's `pki` mount, which is what `root/generate/internal`
# buys.
rootchanged=0
if ! cmp -s "$svcroot.new" "$svcroot"; then
rootchanged=1
fi
mv -f "$svcroot.new" "$svcroot"
chmod 0644 "$svcroot"
# Propagation, for the RETRY path only. Ordered before both of
# these, so on a normal boot they have not run yet, `is-active`
# is false, and ordering alone does the work. What this covers is
# the store coming up hours after the gateway did: nginx serves a
# *copy* of the leaf, so re-issuing the source changes nothing
# until the copy is remade.
#
# ⚠️ `--no-block`, and it is not a preference. This unit declares
# `Before=` both of these, so a blocking `systemctl restart`
# enqueues a job that systemd will not start until this unit is
# active — and this unit is not active until its ExecStart
# returns, which is waiting on that job. A deadlock, held until
# the 24h retry window's `TimeoutStartSec` fires. Queueing the
# job and letting it run once we exit is the only ordering that
# terminates.
#
# The cost is real and is the reason `hive-tls-resign` does NOT
# do this: a propagation that fails after we return does not fail
# us, so the leaf can rotate on disk while nginx keeps the old
# copy. That unit blocks because nothing orders it before its
# consumers; this one cannot.
if systemctl is-active --quiet nginx.service; then
echo "swarm-services leaf rotated — re-importing and reloading nginx"
systemctl restart --no-block hive-gateway-self-signed-cert.service
systemctl reload --no-block nginx.service
fi
# The bundle is assembled by `hive-tls-ca`, which is ordered
# after this unit and therefore already holds the current root on
# any normal boot. On the retry path it ran a long time ago, so a
# CHANGED root has to reach it — and only a changed one, or every
# boot would bounce a unit with nothing to do.
if [ "$rootchanged" = 1 ] && systemctl is-active --quiet hive-tls-ca.service; then
echo "the services root changed — rebuilding the trust bundle"
systemctl restart --no-block hive-tls-ca.service
fi
'';
};
systemd.timers.hive-tls-resign = {
description = "Weekly gateway-leaf re-sign and propagation";
wantedBy = [ "timers.target" ];

243
nix/host-modules/swarm-bao.nix
View file

@ -300,48 +300,15 @@ let
# self-signed CA under `/var/lib/swarm-bao-pki`, and it stays outside the
# store permanently: bao cannot issue the credential that opens bao.
#
# The mount's own root is generated into it by the bootstrap unit below and
# its private key never leaves — `root/generate/internal` keeps it inside
# the store, which is the whole reason the mount exists rather than another
# key beside ./swarm-ca.nix's.
#
# An option rather than a literal because ./hive-tls.nix issues through it
# and has to spell it the same way.
servicesPkiMountPath = baoDeploy.servicesPkiMountPath;
# Subject of the root generated into that mount. A label for a human reading
# a chain, not an identity anything authenticates against — same fall-through
# ./swarm-ca.nix:29-37 uses, and for the same reason: a hive that has set
# neither a swarm name nor a domain must still evaluate.
servicesPkiRootLabel =
if hyperhiveCfg.swarm.name != null then
hyperhiveCfg.swarm.name
else if hyperhiveCfg.domain != null then
hyperhiveCfg.domain
else
"hyperhive";
# Where the store's host publishes that root's PUBLIC certificate. Its own
# directory rather than ./glue-bao-tls.nix's `/var/lib/swarm-bao-pki`: that
# one holds the CA key that opens the store, is 0700 for exactly that
# reason, and this file is the opposite kind of thing — the one piece of
# certificate material meant to be copied out and handed around.
servicesPkiRootDir = "/var/lib/swarm-bao-services-pki";
servicesPkiRootFile = "${servicesPkiRootDir}/services-root.pem";
# ~30y, matching ./swarm-ca.nix's root rather than its services sub-CA. This
# is an ANCHOR: an operator adds it to a browser's trust store by hand, and
# every rotation is a manual re-trust on every machine that holds it. The
# sub-CA's short window bought cheap rotation because the root above it
# absorbed the churn; nothing sits above this one.
servicesPkiRootTtl = "262800h";
# Nothing issues through this mount yet — see the role below.
servicesPkiMountPath = "pki";
# The role every swarm-service certificate will be issued through, and the
# whole narrowing of this mount. A role rather than a name-constrained
# sub-CA: the engine enforces the same "these names and no others" at issue
# time that ./swarm-ca.nix builds into x509 `nameConstraints` today, so a
# second CA in the chain would re-implement a check that already happens.
servicesPkiRoleName = baoDeploy.servicesPkiRoleName;
servicesPkiRoleName = "swarm-services";
# The names that role may issue for. Read out of the swarm-tier option
# rather than assembled here, exactly as ./swarm-ca.nix:45 and
@ -350,22 +317,23 @@ let
# composing its own list is how they stop agreeing.
swarmServiceDomains = hyperhiveCfg.swarm.serviceDomains;
# The identity that asks this mount for a certificate. A FOURTH principal
# rather than any existing one, for the reason `secretPublisherCommonName`
# gives for being a second: the controller may rewrite every hive's policy
# and login role, and a unit whose whole job is renewing one server
# certificate has no business holding that.
# The identity that will ask this mount for a certificate, once something
# does. A THIRD principal rather than either existing one, for the reason
# `secretPublisherCommonName` gives for being a second: the controller may
# rewrite every hive's policy and login role, and a unit whose whole job is
# renewing one server certificate has no business holding that.
#
# `update` and not `create`: bao's issue endpoint is a POST to an existing
# path, which is `update` in ACL terms — `create` here would grant nothing
# and read as though it did.
#
# ⚠️ NOTHING IS ATTACHED TO THIS POLICY. A bao policy grants only through a
# token that carries it, so until a login role names it this is an inert
# declaration of intent — which is the point: the cert-auth role that
# attaches it needs a leaf carrying its CN, and minting that is
# ./glue-bao-tls.nix's job, one step further along than this one.
servicesIssuerPolicyName = "swarm-services-issuer";
# The subject the login role accepts, and the CN ./glue-bao-tls.nix signs
# the matching leaf with. An option rather than a literal for the reason
# `controllerCn` gives: two files have to spell it identically.
servicesIssuerCn = baoDeploy.servicesIssuerCommonName;
servicesIssuerPolicyText = ''
path "${servicesPkiMountPath}/issue/${servicesPkiRoleName}" {
capabilities = ["update"]
@ -1003,65 +971,6 @@ in
'';
};
servicesPkiMountPath = lib.mkOption {
type = lib.types.str;
default = "pki";
description = ''
Mount path of the PKI engine the swarm's **service** certificates are
issued from, and whose root is the anchor those certificates chain to.
An option rather than a literal because two files have to spell it
identically: this module enables the mount, generates its root and
writes the grant that points at `<path>/issue/`, while ./hive-tls.nix
calls that endpoint. A mount path only one of them knows is a grant
naming a location nobody issues from.
⚠️ NOT the store's own client-auth PKI. That one is
./glue-bao-tls.nix's self-signed CA on disk, and it stays outside the
store permanently — bao cannot issue the credential that opens bao.
'';
};
servicesPkiRoleName = lib.mkOption {
type = lib.types.str;
default = "swarm-services";
description = ''
Role on {option}`services.hyperhive.deploy.bao.servicesPkiMountPath`
that the swarm-services certificate is issued through, and the whole
narrowing of that mount.
A role rather than a name-constrained sub-CA: the engine enforces the
same "these names and no others" at issue time, from an
`allowed_domains` read off
{option}`services.hyperhive.swarm.serviceDomains`, that an
intermediate would encode in x509 `nameConstraints` — so a second
authority in the chain would re-implement a check that has already
happened.
'';
};
servicesIssuerCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-services-issuer";
example = "swarm-services-issuer.svc";
description = ''
Subject the store's services-issuer cert-auth role accepts — the
identity a hive presents when it asks the `pki` mount for the
swarm-services certificate its gateway serves.
A **fourth** identity rather than reuse of any sibling above, and the
reasoning is theirs: this principal's whole grant is one `update` on
`pki/issue/swarm-services`, while the controller's includes rewriting
every hive's policy and login role. Handing a certificate-renewal unit
the controller's leaf would give that separation away in one line.
⚠️ Same collision as its siblings, and the same answer: ./swarm.nix
feeds this value into the guard on
{option}`services.hyperhive.swarm.hives`, so a hive named after it
fails evaluation rather than silently receiving this grant.
'';
};
matrixCtlHiveName = lib.mkOption {
type = lib.types.str;
default = toString hyperhiveCfg.hiveName;
@ -1498,7 +1407,6 @@ in
"swarm-bao-certs"
"swarm-bao-token"
"swarm-bao-forwarder-oidc"
"swarm-bao-services-issuer-policy"
];
# 🚫 No `swarm.otel.scrapeTargets.bao` entry any more, and its absence is
@ -1840,69 +1748,21 @@ in
*) bao secrets enable -path=${credentialMountPath} kv-v2 ;;
esac
# The PKI engine the swarm's service certificates come from. Asked
# rather than attempted for the same reason as the mount above:
# The PKI engine the swarm's service certificates are to come from.
# Asked rather than attempted for the same reason as the mount above:
# `secrets enable` errors on a path already in use, so a second
# rebuild would fail a unit that has nothing left to do.
#
# ⚠️ Deliberately EMPTY: nothing here generates an issuer into it, so
# the role below can be written but cannot yet issue. That is the
# whole shape of this step — the mount, its narrowing and its grant
# exist, and the binary that uses them arrives separately. Undoing it
# is `bao secrets disable ${servicesPkiMountPath}`.
case "$mounts" in
*'"${servicesPkiMountPath}/"'*) ;;
*) bao secrets enable -path=${servicesPkiMountPath} pki ;;
esac
# ⚠️ THE ROOT, AND THE ONE THING THIS UNIT MUST NEVER DO TWICE.
#
# `root/generate/internal` mints a new self-signed CA every time it
# is called. Calling it unconditionally would hand the swarm a
# different anchor on every boot: every certificate issued under the
# previous one stops verifying, and every browser an operator taught
# to trust the old root has to be re-taught. So the generation is
# guarded, and the guard asks BAO rather than looking for a marker
# file — a file on this host's disk is a claim about a store that
# may live in a container, may have been restored from a snapshot,
# and may have had the mount disabled and re-enabled underneath it.
# The mount's own issuer list is the only answer that cannot be
# stale.
#
# `bao list ${servicesPkiMountPath}/issuers` is a 404 (non-zero) on a
# mount with no issuer and a key list once one exists, so its exit
# status IS the question — no output parsing, no error string to
# recognise. The default issuer is what `${servicesPkiMountPath}/issue/`
# signs with, and there is exactly one.
if ! bao list ${lib.escapeShellArg "${servicesPkiMountPath}/issuers"} >/dev/null 2>&1; then
echo "generating the swarm services root into the ${servicesPkiMountPath} mount"
# `-field=issuing_ca` discards the rest of the response. The
# private key is not in it under `internal` and cannot be — that
# is the point of the endpoint — and the certificate is read back
# below from a path that works on every subsequent boot too.
bao write -field=issuing_ca ${lib.escapeShellArg "${servicesPkiMountPath}/root/generate/internal"} \
common_name=${lib.escapeShellArg "swarm-services-ca ${servicesPkiRootLabel}"} \
issuer_name=${lib.escapeShellArg servicesPkiRoleName} \
ttl=${servicesPkiRootTtl} \
key_type=rsa \
key_bits=4096 >/dev/null
else
echo "the ${servicesPkiMountPath} mount already has an issuer — leaving it alone"
fi
# The anchor, as a world-readable file on the store's own host.
#
# Written on EVERY run and not only after a generation, so a store
# whose root predates this unit still publishes it. It is a public
# certificate — the material a browser has to be told to trust —
# and the private half stays in the mount, so 0644 is the correct
# mode rather than a concession.
#
# Why a file at all when `bao read -field=certificate
# ${servicesPkiMountPath}/cert/ca` returns the same bytes: every
# listener sets `tls_require_and_verify_client_cert`, so reading it
# from the store costs a client certificate. An operator adding the
# root to a browser is not necessarily holding one.
install -d -m 0755 ${lib.escapeShellArg servicesPkiRootDir}
bao read -field=certificate ${lib.escapeShellArg "${servicesPkiMountPath}/cert/ca"} \
> ${lib.escapeShellArg "${servicesPkiRootFile}.new"}
chmod 0644 ${lib.escapeShellArg "${servicesPkiRootFile}.new"}
mv -f ${lib.escapeShellArg "${servicesPkiRootFile}.new"} ${lib.escapeShellArg servicesPkiRootFile}
# `bao write` is an upsert, so this re-asserts the role on every
# rebuild rather than failing on one that exists — same shape as the
# cert-auth roles below, and the reason the mount needs the `case`
@ -1939,6 +1799,14 @@ in
client_flag=false \
key_type=rsa \
key_bits=4096
# Idempotent for the same reason the controller's policy above is: a
# rebuild re-asserts it. Written here rather than in a unit of its
# own because it attaches to nothing yet — the sibling-unit rule
# below is about a second *principal* that logs in, and this policy
# has no login role until the leaf carrying its CN exists.
printf '%s' ${lib.escapeShellArg servicesIssuerPolicyText} |
bao policy write ${lib.escapeShellArg servicesIssuerPolicyName} -
''
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
@ -2090,57 +1958,6 @@ in
systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders;
# A FOURTH sibling, same shape and same reasons as the two above. This
# one is what turns `swarm-services-issuer` from a declaration into a
# grant: a bao policy reaches nothing until a login role hands it to a
# token, so the `bao write auth/cert/certs/…` below is the whole
# difference between a mount nobody may issue from and one the gateway's
# certificate comes out of.
#
# The policy text moved here from the controller's unit, where it sat
# while it attached to nothing — a policy and the role that carries it
# belong in one place, and now there is a principal to put them with.
systemd.services.swarm-bao-services-issuer-policy = lib.mkIf haveBootstrapToken {
description = "write the swarm services issuer's bao policy and cert-auth role";
after = [
"container@${cfg.machine}.service"
"swarm-bao-controller-policy.service"
];
wantedBy = [ "multi-user.target" ];
path = [
baoCli
pkgs.coreutils
];
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
# Same unseal wait as its three siblings above, for the reason stated
# there: under `seal = "shamir"` a human unseals by hand.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
Restart = "on-failure";
RestartSec = 30;
};
script = ''
set -euo pipefail
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
export BAO_TOKEN
printf '%s' ${lib.escapeShellArg servicesIssuerPolicyText} |
bao policy write ${lib.escapeShellArg servicesIssuerPolicyName} -
''
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
bao write auth/cert/certs/${lib.escapeShellArg servicesIssuerPolicyName} \
certificate=@${tlsDir}/client-ca.pem \
allowed_common_names=${lib.escapeShellArg servicesIssuerCn} \
token_policies=${lib.escapeShellArg servicesIssuerPolicyName} \
display_name=${lib.escapeShellArg servicesIssuerCn}
'';
};
# The CA bind source is written at runtime by a host unit, so the
# container has to start after it — otherwise nspawn sets up a mount
# over a file that does not exist yet.

140
nix/host-modules/swarm-ca.nix
View file

@ -14,13 +14,6 @@
# for the obvious reason; the cert as a consequence, and that one costs
# something real — nothing whose trust store is built at build time can
# name it: `docs/swarm/ca.md::Distributing the root`.
#
# ⚠️ This root no longer issues the swarm's SERVICE certificates, and the
# `swarm-services-ca` sub-CA that used to sit under it is gone rather than
# moved. Those come out of the secret store's own `pki` mount now, anchored
# on a root generated inside it (./swarm-bao.nix), and ./hive-tls.nix asks
# for them over the network instead of signing them here. What is left here
# is one job: the anchor each hive's own CA is issued under.
{
lib,
config,
@ -43,6 +36,13 @@ let
hyperhiveCfg.domain
else
"hyperhive";
# Derived once in ./swarm.nix, read here and by ./hive-tls.nix: the
# CA that name-constrains these and the leaf that carries them as SANs
# must agree exactly, and two modules each assembling the list is how
# they stop agreeing. It is also this unit's *rotation trigger* below,
# which is why the ordering is stable there rather than here.
serviceDomains = hyperhiveCfg.swarm.serviceDomains;
in
{
options.services.hyperhive.swarm.ca = {
@ -90,6 +90,23 @@ in
'';
};
servicesValidityDays = lib.mkOption {
type = lib.types.int;
default = 1825;
description = ''
Validity window of the swarm-services sub-CA in days (~5y).
Deliberately far shorter than the root's: this CA is *meant* to
be re-issued — adding a swarm service changes its name
constraints and rotates it — so a long window buys nothing, and
a short one keeps the rotation path exercised rather than
theoretical.
Rotating it is cheap in the way rotating the root is not: it
touches only the swarm-service vhosts, and no peer hive holds it
as an anchor.
'';
};
validityDays = lib.mkOption {
type = lib.types.int;
default = 10950;
@ -108,8 +125,11 @@ in
config = lib.mkIf (hyperhiveCfg.enable && cfg.autoConfigure) {
# A CA that fails to issue is invisible until something makes a TLS call
# hours later, so this oneshot is worth more than most services.
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-ca" ];
# hours later, so these two oneshots are worth more than most services.
services.hyperhive.swarm.otel.journaldUnits = [
"swarm-ca"
"swarm-services-ca"
];
systemd.services.swarm-ca = {
description = "Generate the swarm root CA when absent";
@ -163,5 +183,107 @@ in
chmod 0644 "$root"
'';
};
# The swarm-services sub-CA: issues leaves for the swarm's own
# service names, which no hive CA can sign — each of those is
# name-constrained to its own hive's domain, and the service names
# are siblings of it, not children.
#
# Rotation is the point of it being separate (mara: "swarm services
# sub ca that can rotate independently of swarm root ca"): the
# constraint enumerates the exact service names, so adding a service
# re-issues *this* and never touches the root or any hive CA.
systemd.services.swarm-services-ca = {
description = "Issue the swarm-services sub-CA under the swarm root";
wantedBy = [ "multi-user.target" ];
after = [ "swarm-ca.service" ];
requires = [ "swarm-ca.service" ];
path = [ pkgs.openssl ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
UMask = "0077";
SyslogIdentifier = "swarm-services-ca";
};
script = ''
set -euo pipefail
d=${lib.escapeShellArg cfg.stateDir}
root="$d/root.pem"
rootk="$d/root-key.pem"
ca="$d/services-ca.pem"
cak="$d/services-ca-key.pem"
# The name set this CA was last issued for. Comparing against it
# is what makes re-issuance happen exactly when the service
# names change — not every boot, and not never.
names="$d/services-ca.names"
want=${lib.escapeShellArg (lib.concatStringsSep "\n" serviceDomains)}
if [ -z "$want" ]; then
echo "no swarm service domains configured — nothing to issue for"
exit 0
fi
# Same half-provisioned guard as the root: a key beside a cert
# that did not sign it looks like it works and issues nothing
# anyone will trust.
if { [ -e "$ca" ] && [ ! -e "$cak" ]; } || { [ -e "$cak" ] && [ ! -e "$ca" ]; }; then
echo "services sub-CA half-provisioned ($ca / $cak) — refusing to generate over it" >&2
exit 1
fi
if [ -s "$ca" ] && [ -s "$cak" ] && [ -f "$names" ] \
&& [ "$(cat "$names")" = "$want" ]; then
echo "services sub-CA present and covers the configured names — leaving it alone"
exit 0
fi
if [ ! -s "$root" ] || [ ! -s "$rootk" ]; then
echo "no swarm root CA at $root — cannot issue the services sub-CA under it" >&2
exit 1
fi
echo "issuing services sub-CA at $ca for: $(echo "$want" | tr '\n' ' ')"
csr="$(mktemp "$d/services-ca.csr.XXXXXX")"
ext="$(mktemp "$d/services-ca.ext.XXXXXX")"
trap 'rm -f "$csr" "$ext"' EXIT
openssl req -newkey rsa:4096 -nodes -sha256 \
-keyout "$cak" -out "$csr" \
-subj "/CN=swarm-services-ca ${swarmLabel}"
{
# pathlen:0 — this signs leaves and delegates no further.
printf 'basicConstraints=critical,CA:TRUE,pathlen:0\n'
printf 'keyUsage=critical,keyCertSign,cRLSign\n'
printf 'subjectKeyIdentifier=hash\n'
printf 'authorityKeyIdentifier=keyid:always\n'
# Constrained to the exact service names, not to the whole
# swarm domain: a leaked services CA should mint `forge.`,
# `chat.`, `auth.` and nothing else. The IP exclusions are not
# redundant — a DNS constraint says nothing about an
# iPAddress SAN, and an unconstrained name type is a name
# type this CA is unconstrained for.
#
# ⚠️ EVERY entry carries its own `permitted;` / `excluded;`
# prefix. openssl's parser takes the qualifier per subtree, not
# once for a run of them: `permitted;DNS:a,DNS:b` is rejected
# outright with `v2i_NAME_CONSTRAINTS: invalid syntax`, which
# fails the whole unit. The hive CA next door emits exactly one
# permitted name, so the missing-prefix form is accidentally
# valid there and does NOT generalise — this list is always
# longer than one.
printf 'nameConstraints=critical,%s,excluded;IP:0.0.0.0/0.0.0.0,excluded;IP:0:0:0:0:0:0:0:0/0:0:0:0:0:0:0:0\n' \
"$(echo "$want" | sed 's/^/permitted;DNS:/' | paste -sd, -)"
} > "$ext"
openssl x509 -req -in "$csr" -CA "$root" -CAkey "$rootk" \
-CAcreateserial -days ${toString cfg.servicesValidityDays} -sha256 \
-extfile "$ext" -out "$ca"
printf '%s' "$want" > "$names"
chmod 0600 "$cak"
chmod 0644 "$ca" "$names"
'';
};
};
}

1
nix/host-modules/swarm.nix
View file

@ -49,7 +49,6 @@ let
deployCfg.bao.matrixCtlCommonName
deployCfg.bao.grafanaOidcCommonName
deployCfg.bao.otelOidcCommonName
deployCfg.bao.servicesIssuerCommonName
]
# The two per-hive readers' subjects, spelled out per hive rather than as the
# prefix. The prefix alone would reserve the wrong string: the role for hive