vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-tls: the gateway leaf uses the services-issuer identity, not a hive's

Browse source 
swarm-services-cert issues the swarm-services leaf a self-signed gateway
serves, and on a host fronting no swarm name fetches the store's pki root.
It is gated on the gateway, not on a hive, yet its leaf pair was named
under deploy.hive-controller.tls and defaulted from the hive reader's cert
directory, and the root fetch presented the hive's own identity. A gateway
host with no hive had no such identity, so that branch was red by
construction.

- The pair moves to deploy.bao.servicesIssuerClient{Cert,Key}File, beside
  the store's other host-unit readers, and glue-bao-tls.nix defaults it on
  the store's host from the leaf it already mints. The old
  deploy.hive-controller.tls.baoClient{Cert,Key}File paths are
  mkRenamedOptionModule aliases. glue-services-issuer-bao-identity.nix,
  which derived the default from deploy.bao.clientCertFile, is gone.
- The no-swarm-name root fetch presents the services-issuer pair. The hive
  identity is now only hive-tls-resign's.
- A self-signed gateway with no pair fails evaluation, naming both options.
  A pair named but unreadable still fails the unit, naming the paths.

Refs #4921
This commit is contained in:
atlas 2026-10-06 09:48:16 +02:00 • committed by mara
commit 5761398d60
13 changed files with 319 additions and 173 deletions
Download patch file Download diff file Expand all files Collapse all files

1
nix/host-modules/default.nix
View file

@ -40,7 +40,6 @@
./glue-nats-oidc-client.nix
./glue-queue-agent-credential.nix
./glue-secret-publisher-bao-identity.nix
./glue-services-issuer-bao-identity.nix
./glue-swarm-bao-otel-oidc-client.nix
./glue-swarm-otel-oidc-client.nix
./swarm-authelia-service.nix

11
nix/host-modules/deploy.nix
View file

@ -174,6 +174,17 @@ in
[ "services" "hyperhive" "deploy" "ci-runner" "jobTimeout" ]
)
# The gateway's services-issuer pair is a swarm-service identity, not a
# hive's, so it sits with the store's other host-unit readers.
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "baoClientCertFile" ]
[ "services" "hyperhive" "deploy" "bao" "servicesIssuerClientCertFile" ]
)
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "baoClientKeyFile" ]
[ "services" "hyperhive" "deploy" "bao" "servicesIssuerClientKeyFile" ]
)
# The runner toggle sat under `swarm.forge.ci` before `deploy.forgejo.ci`.
# Both were per-host, so each forwards straight to its final path.
(lib.mkRenamedOptionModule

6
nix/host-modules/glue-bao-tls.nix
View file

@ -107,6 +107,8 @@ in
forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem";
ciRunnerClientCertFile = lib.mkDefault "${pkiDir}/ci-runner.pem";
ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem";
servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
servicesIssuerClientKeyFile = lib.mkDefault "${pkiDir}/services-issuer-key.pem";
# The granter: every `swarm-bao-*-policy` unit on this host logs in with
# it. ⚠️ Unlike every other leaf here, never the file an operator copies:
@ -224,8 +226,8 @@ in
[ -s ${pkiDir}/forwarder-oidc.pem ] || ${signLeaf} ${pkiDir} forwarder-oidc \
${lib.escapeShellArg deployCfg.bao.forwarderOidcCommonName} "" clientAuth
# The identity a hive presents to ask the store's `pki` mount for the
# swarm-services certificate its gateway serves. Minted here like the
# The identity a gateway host presents to ask the store's `pki` mount
# for the swarm-services certificate it serves. Minted here like the
# three above, and the reason is the sharpest of the four: this leaf
# is what OPENS the mint, so it cannot come out of it. A certificate
# authority that issues the credential admitting you to it is the

4
nix/host-modules/glue-hive-tls-bao-identity.nix
View file

@ -6,7 +6,7 @@
# that credential, which is what every hive on a host without the store does.
#
# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the
# leaf, for the reason ./glue-services-issuer-bao-identity.nix gives.
# leaf, because the thing that owns a private key owns issuing from it.
#
# Gated on the minted PKI existing, as that file is, and on a hive running
# here: a host without one has no hive leaf to ask for.
@ -21,7 +21,7 @@ let
baoDeploy = config.services.hyperhive.deploy.bao;
# Where ./glue-bao-tls.nix puts the leaves, derived from the reader's own
# path as ./glue-services-issuer-bao-identity.nix derives it.
# path rather than repeating that file's directory literal.
haveMintedPki = baoDeploy.clientCertFile != null;
pkiDir = if haveMintedPki then builtins.dirOf baoDeploy.clientCertFile else null;
in

43
nix/host-modules/glue-services-issuer-bao-identity.nix
View file

@ -1,43 +0,0 @@
# Glue: point the swarm-services certificate unit at the bao leaf minted for
# it.
#
# ONE PAIRING PER FILE — the services-issuer principal ← bao, and nothing
# else. Deleting this leaves a hive that takes an operator-provided path to
# that credential, which is what any deployment not minting its own already
# does.
#
# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the
# leaf, because the thing that owns a private key owns issuing from it. What
# belongs here is the pairing: which paths `swarm-services-cert` reads.
#
# ⚠️ Gated on the leaf existing, not on the store running here — the same rule
# ./glue-controller-bao-identity.nix states, and it matters more for this
# principal than for any of its siblings: every hive in a swarm serves the
# swarm's service names and therefore needs this certificate, while exactly
# one of them hosts the store.
#
# Everything is `mkDefault`. An operator naming their own paths wins.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoDeploy = deployCfg.bao;
# Where ./glue-bao-tls.nix puts the leaves, derived from the reader's own
# path rather than repeating that file's directory literal: an operator who
# moves the PKI moves both, and the two cannot drift apart.
haveMintedPki = baoDeploy.clientCertFile != null;
pkiDir = if haveMintedPki then builtins.dirOf baoDeploy.clientCertFile else null;
in
{
config = lib.mkIf haveMintedPki {
services.hyperhive.deploy.hive-controller.tls = {
baoClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
baoClientKeyFile = lib.mkDefault "${pkiDir}/services-issuer-key.pem";
};
};
}

181
nix/host-modules/hive-tls.nix
View file

@ -72,41 +72,77 @@ let
else
"${baoDeploy.hiveLeafPkiRolePrefix}-${hyperhiveCfg.hiveName}";
# The hive's own store identity: its subject is
# `<hiveLeafCommonNamePrefix>-<hiveName>`, and its policy may issue from
# `hiveLeafRole` alone. It also carries the root fetch's handshake.
haveHiveIdentity = cfg.baoHiveIdentityCertFile != null && cfg.baoHiveIdentityKeyFile != null;
showPath = p: if p == null then "unset" else p;
hiveIdentityOptions = "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile (${showPath cfg.baoHiveIdentityCertFile}) and services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile (${showPath cfg.baoHiveIdentityKeyFile})";
# Fails the unit, naming both options, when the identity is unset or a file
# is unreadable. Every hive must hold one.
requireHiveIdentity =
# Fails the unit, naming both options, when an identity is unset or a file
# is unreadable.
requireIdentity =
{
holder,
what,
certFile,
keyFile,
certOption,
keyOption,
unsetHint ? "",
}:
purpose:
if haveHiveIdentity then
let
options = "${certOption} (${showPath certFile}) and ${keyOption} (${showPath keyFile})";
in
if certFile != null && keyFile != null then
''
missing=""
for f in ${
lib.escapeShellArgs [
cfg.baoHiveIdentityCertFile
cfg.baoHiveIdentityKeyFile
certFile
keyFile
]
}; do
[ -r "$f" ] || missing="$missing $f"
done
if [ -n "$missing" ]; then
echo "this hive has no usable store identity, so it cannot ${purpose}: cannot read$missing." >&2
echo "place the files named by ${hiveIdentityOptions}." >&2
echo "${holder} has no usable ${what}, so it cannot ${purpose}: cannot read$missing." >&2
echo "place the files named by ${options}." >&2
exit 1
fi
''
else
''
echo "this hive has no store identity, so it cannot ${purpose}." >&2
echo "set ${hiveIdentityOptions} and place both files." >&2
echo "${holder} has no ${what}, so it cannot ${purpose}." >&2
echo "set ${options} and place both files.${unsetHint}" >&2
exit 1
'';
# The hive's own store identity: its subject is
# `<hiveLeafCommonNamePrefix>-<hiveName>`, and its policy may issue from
# `hiveLeafRole` alone. Every hive must hold one.
requireHiveIdentity = requireIdentity {
holder = "this hive";
what = "store identity";
certFile = cfg.baoHiveIdentityCertFile;
keyFile = cfg.baoHiveIdentityKeyFile;
certOption = "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile";
keyOption = "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile";
};
# The swarm-wide services-issuer identity `swarm-services-cert` presents,
# whether or not a hive runs here. Its subject is
# `servicesIssuerCommonName`, and its policy may issue from
# `pki/issue/swarm-services` alone.
servicesIssuerCert = baoDeploy.servicesIssuerClientCertFile;
servicesIssuerKey = baoDeploy.servicesIssuerClientKeyFile;
haveServicesIssuer = servicesIssuerCert != null && servicesIssuerKey != null;
requireServicesIssuer = requireIdentity {
holder = "this host";
what = "services-issuer identity";
certFile = servicesIssuerCert;
keyFile = servicesIssuerKey;
certOption = "services.hyperhive.deploy.bao.servicesIssuerClientCertFile";
keyOption = "services.hyperhive.deploy.bao.servicesIssuerClientKeyFile";
unsetHint = " Its subject must be CN=${baoDeploy.servicesIssuerCommonName}.";
};
# Shared by every unit that decides whether to re-issue a leaf. It
# lives here rather than in one of them because the guards have to
# agree: they answer the same question for different leaves
@ -195,11 +231,11 @@ let
servicesCertEnvironment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs (cfg.baoClientCertFile != null) {
BAO_CLIENT_CERT = cfg.baoClientCertFile;
// lib.optionalAttrs (servicesIssuerCert != null) {
BAO_CLIENT_CERT = servicesIssuerCert;
}
// lib.optionalAttrs (cfg.baoClientKeyFile != null) {
BAO_CLIENT_KEY = cfg.baoClientKeyFile;
// lib.optionalAttrs (servicesIssuerKey != null) {
BAO_CLIENT_KEY = servicesIssuerKey;
}
# Absent means the system trust store, which is what a deployment
# with a real CA wants and what a self-signed one must not be left
@ -208,9 +244,8 @@ let
BAO_CACERT = baoDeploy.serverCaFile;
};
# The root-only fetch. Same store and server CA as `servicesCertEnvironment`,
# but the hive's own identity: a host fronting no swarm name holds no
# services-issuer pair.
# The root-only fetch: the same store, server CA and services-issuer pair
# as `servicesCertEnvironment`, handed to curl.
storeRootCurlTls = lib.escapeShellArgs (
lib.optionals (baoDeploy.serverCaFile != null) [
"--cacert"
@ -218,9 +253,9 @@ let
]
++ [
"--cert"
cfg.baoHiveIdentityCertFile
servicesIssuerCert
"--key"
cfg.baoHiveIdentityKeyFile
servicesIssuerKey
]
);
@ -262,62 +297,21 @@ in
'';
};
baoClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/services-issuer.pem";
description = ''
Certificate `swarm-services-cert` presents to the secret store when it
asks for the swarm-services leaf the gateway serves.
Its subject has to be
{option}`services.hyperhive.deploy.bao.servicesIssuerCommonName` — cert
auth matches on the common name, and this is the one role whose policy
permits `pki/issue/swarm-services`.
A **separate** credential from
{option}`services.hyperhive.deploy.bao.clientCertFile`, which is this
hive's own reader identity and whose policy reads every secret the
hive is entitled to, and from
{option}`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile`,
which the root fetch on a hive fronting no swarm name presents.
On a hive that mints its own store PKI a glue module supplies the leaf
as a `mkDefault`; everywhere else it is the credential an operator
places, for the reason the store's own client certificate is.
A path, never a value.
'';
};
baoClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/services-issuer-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.hive-controller.tls.baoClientCertFile`.
Both or neither — a certificate with no key authenticates nothing.
'';
};
baoHiveIdentityCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/hive-leaf.pem";
description = ''
This hive's own store identity. `hive-tls-resign` presents it to ask
the secret store for the gateway leaf, and `swarm-services-cert` on a
hive fronting no swarm name presents it for the handshake of the root
fetch.
the secret store for the gateway leaf.
Its subject has to be
`<services.hyperhive.deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`;
that cert-auth role's policy may issue from this hive's gateway-leaf
role and nothing else.
Every hive with a self-signed gateway needs it: without it both units
fail, naming this option and
Every hive with a self-signed gateway needs it: without it that unit
fails, naming this option and
{option}`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile`.
On the host that runs the store a glue module supplies the leaf as a
`mkDefault`; on every other host an operator places it.
@ -339,6 +333,30 @@ in
};
config = lib.mkIf active {
assertions = [
{
assertion = haveServicesIssuer;
message = ''
services.hyperhive.gateway serves the self-signed leaf on this host,
which needs the swarm-services issuer's store identity: set both
services.hyperhive.deploy.bao.servicesIssuerClientCertFile
services.hyperhive.deploy.bao.servicesIssuerClientKeyFile
swarm-services-cert.service presents it to ask the swarm secret
store for the swarm-services leaf and to fetch the store's pki root.
Both default to the pair glue-bao-tls.nix mints when
services.hyperhive.deploy.bao.enable is true (the host that runs
the store). Otherwise copy services-issuer.pem and
services-issuer-key.pem from the store host's /var/lib/swarm-bao-pki/
and name them in those two options, or give the gateway a
certificate of its own (services.hyperhive.gateway.tls.certDir or
services.hyperhive.gateway.tls.acme.enable).
'';
}
];
# The state dir and the trust bundle. Never waits on the store: the
# store's container is ordered after this unit, which writes the
# bundle it bind-mounts, so the leaf is `hive-tls-resign`'s.
@ -603,9 +621,9 @@ in
# store nothing local can order against that.
#
# `swarm-bao-pki` mints the services-issuer leaf the login below
# presents, and is absent on a hive holding an operator-placed one —
# presents, and is absent on a host holding an operator-placed one —
# ignored there for the same reason, rather than being a dependency
# only some hives may declare.
# only some hosts may declare.
after = [
"container@${baoCfg.machine}.service"
"swarm-bao-pki.service"
@ -643,13 +661,12 @@ in
# bundle: every other hive's swarm-service names chain to it.
# `ca/pem` takes no token; the TLS client certificate is the
# store's listener's demand, not bao's, so this presents the
# hive's own identity and never logs in with it.
# A hive without a readable identity fails here, naming the
# options: every hive must hold one.
# services-issuer pair and never logs in with it. Without a
# readable pair the unit fails here, naming the options.
if [ "$want_svc" = 0 ]; then
${
if haveHiveIdentity then
requireHiveIdentity "fetch the store's ${baoServicesPkiMount} root"
if haveServicesIssuer then
requireServicesIssuer "fetch the store's ${baoServicesPkiMount} root"
+ ''
echo "no swarm service domains configured — fetching only the store's pki root"
err="$(mktemp)"
@ -683,7 +700,7 @@ in
fi
exit 0''
else
requireHiveIdentity "fetch the store's ${baoServicesPkiMount} root"
requireServicesIssuer "fetch the store's ${baoServicesPkiMount} root"
}
fi
@ -725,17 +742,7 @@ in
exit 0
fi
${
if cfg.baoClientCertFile == null || cfg.baoClientKeyFile == null then
''
echo "no bao client certificate configured for this host, so the gateway's" >&2
echo "swarm-services certificate cannot be requested from the store." >&2
echo "Set services.hyperhive.deploy.hive-controller.tls.baoClient{Cert,Key}File" >&2
echo "to a leaf the store's CA signed with CN=${baoDeploy.servicesIssuerCommonName}." >&2
exit 1''
else
""
}
${requireServicesIssuer "ask the store for the swarm-services leaf"}
# `bao`'s own message is the only thing separating an unreachable
# store from a refused identity from a role that would not issue

37
nix/host-modules/swarm-bao.nix
View file

@ -1950,6 +1950,43 @@ in
'';
};
servicesIssuerClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/services-issuer.pem";
description = ''
Certificate `swarm-services-cert` presents to the store: to log in and
ask for the swarm-services leaf the gateway serves, and for the
handshake of the root fetch on a host fronting no swarm name. Its
subject must be
{option}`services.hyperhive.deploy.bao.servicesIssuerCommonName`.
⚠️ Neither the reader's
{option}`services.hyperhive.deploy.bao.clientCertFile` nor a hive's own
identity: this one's policy permits `update` on
`pki/issue/swarm-services` and nothing else, and it is the same on
every gateway host, whether or not a hive runs there.
Required on every host whose gateway serves the self-signed leaf;
unset fails evaluation.
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
On a gateway host without the store, it is a file the operator copies
from the store's host. A path, never a value.
'';
};
servicesIssuerClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/services-issuer-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.servicesIssuerClientCertFile`.
A path, never a value.
'';
};
granterCommonName = lib.mkOption {
type = lib.types.str;
default = "bao-granter";