`bao operator init` fails at the seal with CKR_MECHANISM_INVALID. The
pkcs11 seal asks for AES-GCM by default and the TPM does not offer it: a
TPM 2.0's symmetric modes are CBC/CFB/CTR/OFB/ECB, and openbao accepts
only AEAD mechanisms — AES-GCM or RSA-OAEP — so RSA-OAEP is the single
mechanism both sides implement.
Measured on the deployed token (`pkcs11-tool --list-mechanisms`, #3860):
no AES-GCM, and `RSA-PKCS-OAEP, keySize={1024,2048}, hw, encrypt,
decrypt` present. CBC is not a fallback — openbao's
`MechanismFromString` rejects `CKM_AES_CBC_PAD` as deprecated and its
encrypt path implements exactly the two AEAD mechanisms.
The key gets a new label so a store provisioned by the earlier module
keeps its unusable AES key without the two resolving to one label, and
the addkey step is now keyed on the label rather than on the store not
existing — otherwise an existing deployment never gains the RSA key.
#3880 bound /dev/tpmrm0 into the store's container, which was necessary and
not sufficient: a bind mount preserves the host's ownership, openbao runs as a
DynamicUser with an empty CapabilityBoundingSet (so no CAP_DAC_OVERRIDE), and
the host applies no tpm udev rule at all — measured, 0 hits across all 41 host
closures against 62 rule files as a control. The node therefore keeps the
kernel default and the seal cannot open it.
A name cannot fix this. NixOS allocates system groups at activation, per
machine, so `tss` — or any group declared on both sides — gets two different
ids, and the device node carries the number. mara picked pinning a gid with an
overridable default (deploy.bao.tpmGid).
The default sits above the range NixOS auto-assigns system groups from
(400-999, measured in update-users-groups.pl) and above the normal-user range,
and below systemd's DynamicUser range (61184-65519), so it collides with
nothing any of those allocate.
The module-eval case compares the two sides rather than checking each against
a literal: the property is that they AGREE, not what they agree on. Its
absence arm is a shamir store, which never opens a TPM and must not claim a
device node's group — without it, pinning unconditionally would look identical.
openbao runs as a DynamicUser, so the uid that has to open the token
store is allocated by the container's PID 1 and cannot be named by a
host unit — the store was created root-owned 0700 and the seal could
never read it. The provisioning unit moves inside the container and
hands the sqlite store over by group; the host keeps only the mkdir the
bind mount needs, create-only so a reboot does not re-impose 0700.
Two further layers blocked the same start, both measured while fixing
this one:
- DynamicUser implies ProtectSystem=strict (systemd.exec(5)), so the
bind mount was read-only to openbao however it was owned, and the
pkcs11 library opens its store read-write. ReadWritePaths= is
required and was absent.
- allowedDevices renders DeviceAllow= and nothing else, and nspawn
builds its own /dev as a fixed tmpfs and cannot create device nodes
— verified against a live container, whose /dev holds no host
devices at all. /dev/tpmrm0 was therefore absent inside swarm-bao,
not merely unpermitted. It is now bound in.
Whether openbao's dynamic uid may *open* that node is a third question:
the tss gid is dynamically allocated, so no name or number means the
same thing on both sides of the boundary. Filed separately rather than
guessed at here.
The two module-eval cases that asserted the unit on the host now assert
it in the container and absent from the host, and two new cases pin the
device bind and the write access — each was individually valid and
collectively required, which is the state no assertion catches. Both
new cases select with `or [ ]`: mutation-testing them showed that a bare
select aborts the run with a nix trace instead of failing the case by
name, which also hid the second failure behind the first.
The store declared no journald units and served no metrics: nothing in
`swarm-bao.nix` mentioned either, while every sibling swarm service
declares both.
Metrics get their own loopback listener rather than a flag on the API
one, and that follows from what a scraper can express rather than from
taste: `swarm.otel.scrapeTargets` is `host:port`, plaintext and with no
credential, while the API listener is TLS and demands a client
certificate once a client CA is set. `metrics_only` narrows the new
listener to the metrics path; `prometheus_retention_time` is what serves
the endpoint at all.
Measured against openbao 2.6.2 before writing any of it: the metrics
path answers 200 on such a listener *while the node is sealed and
uninitialised*, 503 on the API listener, and 404 for a non-metrics path
on the metrics listener.
The listener exists only where a collector does — it is unauthenticated
by design for now, and an endpoint with no reader would be exposure
bought for nothing.
The port cannot be the API port + 1: openbao derives every listener's
cluster address as its own port plus one, so that number is already
taken. An assertion says so, since the failure is otherwise a race with
no log line.
Journald units are declared by the module that defines each unit, not
gathered here, matching the option's own rule — a name nothing defines
is silently ignored, so a central list would read as coverage on hives
that have neither glue module.
Refs #3849
openbao has never started on a hive that deploys it. With raft storage
and no `cluster_addr`, `bao server` exits 1 before serving anything:
cluster address must be set when using raft storage
Reproduced against openbao 2.6.2 with the module's own config shape:
the same JSON minus `cluster_addr` exits 1 with that line, and with it
the server comes up and configures every listener.
Both addresses are built from `swarm.bao.domain` rather than a bind
address — that is the URL a reader already dials (`BAO_ADDR` in
glue-matrix-bao-token) and the name the server certificate carries.
Cluster traffic sits one port up, upstream's convention.
The `unknown or unsupported field loopback` warning in the same startup
log is unrelated and cosmetic: openbao's unknown-field check does not
know about named listener blocks, but the parser honours `type` and
configures each one (measured, two named listeners, both served).
Noted in the module so the next reader does not chase it; the JSON
array form that avoids the warning does not typecheck against nixpkgs'
`settings.listener` (`attrsOf`).
module-eval gains the regression gate plus a control that the settings
it reads vary per deployment.
Refs #3860
Nothing resolved `bao.<swarm.domain>`. It is absent from
`swarm.serviceDomains` and, unlike every other swarm service, contributed
no `gateway.localNames` entry — so the name a reader dials answered only
if an operator had published an external record, which nothing asks them
to do.
Cross-hive traffic goes via the domain either way; only what it resolves
to varies, and for a multi-host swarm that is the operator's upstream DNS.
This covers the deployment with no upstream record to configure.
DNS only, and that asymmetry is deliberate: bao stays out of
`serviceDomains` and gets no vhost. Its leaf comes from its own CA, so a
services-CA entry would mint one nothing uses, and nginx terminating TLS
would strip the client certificate the store authenticates every hive
with. `swarm-nats.nix` documents the same exclusion for itself.
module-eval gains the presence and absence arms: claiming a name this
host does not serve would point every local reader at the wrong machine.
Refs #3862
`swarm-bao.nix` declared the store's half of the mTLS pair as options —
`serverCertFile`, `serverKeyFile`, `clientCaFile` — and left the reader's
half as a literal inside `glue-bao-tls.nix`, which only runs where
`deploy.bao.enable` is set. A hive that did not host the store therefore
could not read from it and could not be pointed at a certificate even
when one had been placed by hand.
Adds `clientCertFile`, `clientKeyFile` and `serverCaFile` beside their
three server siblings, `mkDefault`ed by the glue to the leaf it already
mints, and moves `glue-matrix-bao-token.nix` onto them. Its gate becomes
"this host holds an identity" rather than "the store is a neighbour",
and the unit ordering that names store-local units is now conditional --
`Requires=` on an absent unit fails the job.
`serverCaFile` is separate from `clientCaFile` on purpose: one is the
store choosing which readers to trust, the other a reader choosing which
store to trust. Self-signing collapses them to one file, which is a
property of that deployment and not of the pairing.
Closes#3855.
The state directory was bind-mounted from the host so a nixos-container
destroy could not take the swarm's secrets with it. No sibling service does
that -- swarm-grafana keeps its sqlite database inside the container on
ephemeral = false -- and the bind is what broke the store: upstream pairs
StateDirectory= with DynamicUser=, systemd relocates the state to
/var/lib/private/openbao, and that rename fails EBUSY on an active mount
point, so the unit died at STATE_DIRECTORY before bao ever ran.
The TLS material still has to cross the boundary, because a host unit writes
it and the container reads it, so it moves to its own small bind at
/var/lib/swarm-bao-tls rather than riding along in the state directory. That
directory is 0755 and read-only inside: the certificate and client CA are
public and are read straight off the mount.
The private key is not. install -m 0600 leaves it root-owned and the service
runs as a DynamicUser, so the bind-mounted file is unreadable to it -- which
the old layout hid, because StateDirectory chowned the whole tree on the way
past. LoadCredential is systemd's mechanism for precisely this: PID 1 opens
the source as root and re-exposes it inside the unit owned by the service's
own account.
`services.hyperhive.tls.{stateDir,caValidityDays,leafValidityDays}` sat at
the top of `services.hyperhive`, which is meant to be everything about
hyperhive rather than the settings of one hive. Where the hive CA lives,
how long it lasts and how long the leaves it signs last are decisions of
the host holding the key — `deploy.*`, by the same rule as the switches
that moved before them.
`hive-controller` is hive-c0re's new name (mara on the issue), so the
knobs hang off the daemon that owns the CA rather than off a bare `tls`
at the root. mkRenamedOptionModule entries carry existing configs.
⚠️ Unlike the two switch renames, these names are NOT unique, so this was
swept by ALIAS BINDING rather than by identifier: hive-tls.nix alone holds
two options spelled `stateDir` — its own `cfg.stateDir` and the swarm CA's
`swarmCaCfg.stateDir`, four sites that must not move. Nine files bind an
alias to this config; the rename followed those bindings.
Two sites were invisible to the obvious check, and an unanchored sweep for
`hyperhive\.tls\b` is what found them: the option declaration (`= {` after
the path, so no trailing `.` or `;`) and the alias convention documented in
a comment in lib/hive-ca-trust.nix.
Also renamed the `<tls.stateDir>` shorthand in four docs and two Rust doc
comments, anchored on its delimiters — the new path contains the old one
as a substring, so an unanchored replace would have doubled the prefix.
Three places said nothing supplies them — the module header, the
serverCertFile and clientCaFile descriptions, and the assertion message
an operator actually hits. All three were true when written and
falsified by glue-bao-tls.nix landing in the same PR, which mkDefaults
all three paths.
The load-bearing half is unchanged: the store never reaches for an
authority, because it must not take its identity from one it will itself
distribute. What was wrong is the claim that therefore nothing fills them
in. The service module declares no default and does not know where the
value comes from; the glue supplies one where the store is deployed; an
operator's own path beats it.
The assertion also says what reaching it now means — the glue is absent,
or something set the paths back to null.
OpenBao in a swarm-bao nixos-container, unsealed by the host TPM through
the built-in pkcs11 seal, with shamir as the documented opt-out for
hardware that has no TPM. The seal guard asks the package whether it was
built with an HSM rather than reading its version, and falls back to the
version cliff only when a package advertises no tags -- it fails closed.
Every listener serves TLS: loopback unconditionally, because the host
running the store is always one of its readers, plus whatever
extraListenAddresses names, because which network the other hives share
is a deployment fact. Client-certificate verification turns on only when
clientCaFile says what to trust.
The store's identity is an input, and nothing in THIS module fills it
in. Service-to-store mTLS is a separate trust domain from the gateway's
HTTPS certificates and from both CAs in this tree: a store must not take
its identity from an authority it will itself distribute, or it cannot
come up before the thing that issues it. The module asserts when the
paths are unset rather than reaching for whichever CA happens to be
wired; what supplies them on a self-contained deploy is the next commit.
The leaf is copied to the container's state dir by a host unit rather
than bind-mounted: nixos-container refuses to start when a bind source
is missing, and a directory holding a leaf usually holds the CA's key
beside it.
