The unit treats every failure as permanent: it prints why and `exit 0`s,
with no `Restart=`, so one bad moment costs the whole boot. Two of its
failure modes deserve that and one does not.
`bao login` fails when the store is unreachable, sealed, or has not been
given this host's cert-auth role yet. All three are transient. Measured
on this morning's rebuild:
11:24:05 Started Container 'swarm-bao'
11:24:06 could not log in to swarm-bao with this host's certificate
11:24:07 Success! Data written to: auth/cert/certs/swarm-secret-publisher
It lost by one second, and stayed degraded for the boot. The publisher
next to it hit the same race and recovered on its first retry, because it
has `Restart=on-failure`.
`bao kv get` returning nothing is the opposite: the store answered, and
holds no token at that path. A retry cannot improve it, so that branch
keeps `exit 0` and the local token.
The bound is sized for this race, not for an unseal.
`swarm-bao-controller-policy` waits 2880 x 30s because a shamir unseal is
a human action and that unit blocks nothing. This one is `Before=` the
homeserver's container, so every retry is time the homeserver may spend
waiting -- 4 x 15s covers a container-start race with margin, and a store
still sealed after it degrades exactly as it does today.
`StartLimit*` are `[Unit]` settings and are ignored under `[Service]`, so
they are top-level attrs here. The module-eval case asserts the window
outlasts `RestartSec x burst`, since a burst that cannot be reached is a
unit that looks like it retries and does not.
Refs #4303
`BAO_CLIENT_CERT` decides which certificate the TLS handshake presents. It is
not an identity: cert auth is a login, and a `bao kv` call without a token asks
its token helper for one instead — a `sh` neither unit carries on `path`.
Measured on this host, from `swarm-bao-matrix-token.service`:
swarm-bao did not return secret/swarm/matrix/registration-token
failed to get token helper: error expanding config path "":
exec: "sh": executable file not found in $PATH
So the first credential meant to travel through the store never has, and the
publisher added last week would not have either.
`-token-only` rather than a plain login: storing is the default, and it stores
through that same helper, so the obvious form reproduces the failure one line
further down. It is `-field=token -no-store`, which keeps the token on stdout
and out of the filesystem.
The two units degrade differently on purpose, and that is preserved. The matrix
fetch is `Wants=`-only and must not hold up the homeserver, so a refused login
reports why and keeps the token already in place. The publisher is
`Restart=on-failure`, where a store that cannot authenticate this host is worth
retrying and "published 0" would read as an ordinary quiet day.
`swarm-bao.nix` is untouched: it authenticates with the bootstrap token from a
file, which is a real identity and not a cert exchange. Its shape is where the
export idiom here comes from.
The reader had one failure branch, and it could not fail. It named three
states — the store holds nothing, the store is sealed, the store is
unreachable — treated all three as success, and sent bao's stderr to
/dev/null, which is the only thing that tells them apart.
The degraded mode is right and is unchanged: a missing registration token
means new agent accounts cannot be provisioned and nothing else breaks,
so this still exits 0 and leaves the local token alone. What changes is
that the journal now carries the store's own message instead of a
sentence of ours asserting all three at once.
Deliberately not branching on that message. Distinguishing "no value
found" from "permission denied" programmatically would mean matching
prose from an external tool that I cannot reach a live instance of to
confirm, which is inventing a discriminator rather than reporting one.
Nothing asserted this script before — eleven module-eval cases cover the
unit's existence and its ordering, none its contents, which is how the
branch stayed. The case added here covers the property that was missing.
Gated: `96 module properties hold`, exit 0 (95 on the base commit, +1 =
the case added). `nix fmt` reported 0 changed over 723 files emitted in a
fresh worktree. Tracker-tag, comment-block and doc-pointer lints all exit
0 after staging.
`swarm.*` is what a hive needs to be a *client* of the swarm. For the
homeserver that is what it IS from anywhere: its package, the name it
answers to, the ports and URLs it is reached on, and the client id it is
registered under. Whether it is exposed, which peers it trusts, how large
a request it accepts and where its host-local secrets sit are decisions
of the machine running it, so openFirewall, trustedServers,
maxRequestSize, registrationTokenFile, gui.enable and
sso.clientSecretFile move to `deploy.matrix.*`.
Two sub-blocks split rather than moving whole, on their own evidence.
`gui.enable` is whether THIS host serves the web client; `gui.package` is
which client, an artifact identity, and stays. `sso.clientSecretFile` is a
path on one host; `clientId` must match the id in authelia's register, so
it is swarm-wide. Each half now points at the other, because the rendered
docs put them on separate pages.
hive-gateway passed the whole `swarm.matrix` attrset into vhosts.nix, so
that file read a moving option through an argument with no option path
anywhere in it. It now takes `matrixDeployCfg` beside `matrixCfg` — the
only shape that carries a split namespace across that boundary.
While there: vhosts.nix read `matrixCfg.enable`, which has been a rename
alias for `deploy.matrix.enable` since the enable moved. Reading it made
the module system print `Obsolete option services.hyperhive.swarm.matrix.
enable is used` on EVERY evaluation of every host — a deprecation warning
no operator could silence, because the config tripping it was ours. That
shim lives in hive-matrix.nix rather than in this file's table, which is
why deploy.nix's header claim to be their single home is now qualified
in the new block's comment.
glue-matrix-bao-token.nix read the registration token through its own
`matrixCfg` alias; with that read repointed, the binding had no reader
left, so it goes, and the comment naming it is reworded.
module-eval gains a case configuring a hive through all six OLD paths and
asserting two rendered effects — the host firewall's port list and the
container's bind-mount table — because the new paths evaluate fine
without the shims. `gui.enable` is set to the opposite of its default so
the definition has to land rather than agreeing with it by accident.
The store declared no journald units and served no metrics: nothing in
`swarm-bao.nix` mentioned either, while every sibling swarm service
declares both.
Metrics get their own loopback listener rather than a flag on the API
one, and that follows from what a scraper can express rather than from
taste: `swarm.otel.scrapeTargets` is `host:port`, plaintext and with no
credential, while the API listener is TLS and demands a client
certificate once a client CA is set. `metrics_only` narrows the new
listener to the metrics path; `prometheus_retention_time` is what serves
the endpoint at all.
Measured against openbao 2.6.2 before writing any of it: the metrics
path answers 200 on such a listener *while the node is sealed and
uninitialised*, 503 on the API listener, and 404 for a non-metrics path
on the metrics listener.
The listener exists only where a collector does — it is unauthenticated
by design for now, and an endpoint with no reader would be exposure
bought for nothing.
The port cannot be the API port + 1: openbao derives every listener's
cluster address as its own port plus one, so that number is already
taken. An assertion says so, since the failure is otherwise a race with
no log line.
Journald units are declared by the module that defines each unit, not
gathered here, matching the option's own rule — a name nothing defines
is silently ignored, so a central list would read as coverage on hives
that have neither glue module.
Refs #3849
`swarm-bao.nix` declared the store's half of the mTLS pair as options —
`serverCertFile`, `serverKeyFile`, `clientCaFile` — and left the reader's
half as a literal inside `glue-bao-tls.nix`, which only runs where
`deploy.bao.enable` is set. A hive that did not host the store therefore
could not read from it and could not be pointed at a certificate even
when one had been placed by hand.
Adds `clientCertFile`, `clientKeyFile` and `serverCaFile` beside their
three server siblings, `mkDefault`ed by the glue to the leaf it already
mints, and moves `glue-matrix-bao-token.nix` onto them. Its gate becomes
"this host holds an identity" rather than "the store is a neighbour",
and the unit ordering that names store-local units is now conditional --
`Requires=` on an absent unit fails the job.
`serverCaFile` is separate from `clientCaFile` on purpose: one is the
store choosing which readers to trust, the other a reader choosing which
store to trust. Self-signing collapses them to one file, which is a
property of that deployment and not of the pairing.
Closes#3855.
The comment said it bounds the wait rather than hanging the boot, and
nothing in the unit bounded anything. What actually held was systemd's
default start timeout plus the homeserver only Wants= this unit — both
true, neither stated here, so a reader had to already know them.
TimeoutStartSec puts the number a boot waits on in the file that waits,
and the comment now names the degradation: hitting it keeps the locally
minted token, same as every other failure path in this unit.
Caught by argus reviewing the slice it shipped in.
Fetches an opaque 32-byte value and writes it where hive-matrix.nix
already looks, so the homeserver never learns the store exists.
Chosen over authelia's OIDC secret deliberately: that one needs a
.secret AND a matching .digest, so shipping it first would debug 'can a
reader authenticate and get bytes back' and 'did we write authelia's
file format right' at once, with an SSO outage as the failure mode. Here
the failure is narrow -- new agent accounts cannot be provisioned,
existing ones untouched.
Every failure path keeps the local token: no such key, sealed store,
unreachable store, empty value. The activation script's mint-if-absent
is untouched, so a hive with no store behaves exactly as it does today.
matrixMachine is a literal because hive-matrix.nix declares no `machine`
option -- `matrixCfg.machine` parses cleanly and fails at module-system
resolution, which is the kind of error only reading the target module
catches.
