The log-store board reports rows ingested, disk size, free disk and errors —
every one of them a fact about VictoriaLogs itself. None of them can say
whether a particular unit or host is contributing, so aggregate ingestion
reads healthy on host-tier units while a whole tier ships nothing, and there
was no way to tell those apart from a dashboard.
Add four panels to that board rather than a second one. Every other board here
is per service — agents, authelia, openbao, forge, queue, the two stores — so
a second board about the same service would have made an operator guess which
of the two answers their question.
Nothing in the new panels names a unit: both breakdowns discover their rows
from the data, so a source that starts shipping appears without an edit, and
one that never existed is simply not there.
The ungrouped total is a control, not a summary. An empty breakdown renders
the same whether the query is malformed or the source genuinely never shipped;
with the total beside it, nonzero-and-empty is a broken query and
zero-and-empty is an empty store. Being on one board buys a second reading it
could not have alone: that total and `Log rows ingested` are the same quantity
measured by querying and from the store's own metrics, so the two disagreeing
means rows arrived that no query can reach — which is the failure this
pipeline actually had once.
The two breakdowns are bargauges rather than tables. A `stats` query returns
one frame per group, and a table panel renders one frame at a time behind a
series picker; bargauge consumes multi-series natively, for the same reason
the timeseries beside it always did. The alternative was a table plus
labelsToFields plus merge plus organize — three assumptions where this needs
none, in a spot I cannot render to check.
Series are named by their label rather than `rows{_SYSTEMD_UNIT="x.service"}`,
which also cleans up the timeseries legend. The datasource supports
legendFormat; its query editor's own placeholder is `{{label}}`.
These are the first panels to query the logs datasource at all; the other
eight boards are prometheus, including this one until now, which reads
VictoriaLogs' self-metrics out of VictoriaMetrics. So `renderDashboard` grows
a `@logsDatasourceUid@` substitution alongside the metrics one. The binding it
points at already existed, and its comment claiming a dashboard panel named it
only becomes true with this commit.
The uid is unchanged, so existing links and bookmarks still resolve; only the
title widens to match what the board now covers.
The query model was read out of the plugin in the store rather than guessed:
`queryType` is one of hits/instant/logs/stats/statsRange, and Stats/StatsRange
are the two that consume `expr`. The stream field names come from the module
that builds them — swarm-otel.nix's `_stream_fields=_HOSTNAME,_SYSTEMD_UNIT`.
The queries themselves are confirmed against the live store: mara ran the
by-unit one over seven days and it returned rows.
Refs #4084
glue-bao-tls.nix signs a third leaf. It is minted whether or not a
controller runs here, because the case it serves is the one where it
does not: a controller elsewhere needs a leaf from this CA and cannot
sign one, so issuing it here turns "obtain a certificate out of band"
into "copy this file".
glue-controller-bao-identity.nix holds the pairing and nothing else --
which paths this host's controller reads. Gated on the leaf existing
rather than on deploy.bao.enable, so a controller on the store's host
and one three networks away with an out-of-band leaf get the same
wiring; gating on the store would have made the co-located case the
only supported shape. The directory comes from deploy.bao.clientCertFile
rather than repeating glue-bao-tls.nix's literal, so moving the PKI
moves both.
module-eval gains three cases and two fixtures, because nothing asserted
the PKI script before: an earlier commit added a leaf to that rendered
unit and left the derivation unchanged. The fixture's CN is deliberately
a value no default could supply, so "the role and the leaf both carry
it" says they read one option rather than that both happen to say
swarm-controller.
Gates: 62 module properties hold (59 before, plus these three), on a
derivation hash that actually moved -- this suite is a cache hit when
only fixtures change, so an unchanged hash would have meant the cases
never ran. nix fmt clean, all three scripts/check-*.sh exit 0.
Two halves of one interface that currently exists in one file only.
The CN was a `let` binding in swarm-bao.nix. Whatever mints the
controller's leaf has to spell it identically, and that lives in another
file, so a literal in one place is an interface with no name. It becomes
`deploy.bao.controllerCommonName` -- under bao because it is a property
of the role this module writes, not a credential belonging to the
controller.
The certificate gets `deploy.swarm-controller.baoClientCertFile` /
`baoClientKeyFile` rather than reusing `deploy.bao.clientCertFile`. That
one means "this host as a reader" and carries the hive's name, while the
controller's policy lets it create roles for every hive; one certificate
serving both would hand that power to whatever else reads the store
here.
Both default to null. The glue that mints the leaf sets them with
`mkDefault`, the same way glue-bao-tls.nix already supplies the reader's
paths -- so the controller module names no path of bao's, and deleting
the glue leaves a controller that takes operator-provided ones.
Nothing reads the new options yet; the minting half is the next commit.
`module-eval`'s derivation is unchanged, which is the honest result for
a change that adds options without altering any asserted value -- it
proves the tree still evaluates, not that anything behaves differently.
The policy the granting unit already writes grants paths under
`auth/cert/certs/*`, and nothing in the tree creates that mount. Every
certificate login therefore fails against a path that is not there --
the controller's own, and the per-hive ones it is meant to issue
against the same mount.
Same unit, same bootstrap token: check whether cert auth is mounted,
enable it if not, then write a role binding CN `swarm-controller` to
the `swarm-controller` policy.
Idempotency is a read rather than a tolerated error. `auth enable`
fails on an existing mount, and recognising that would tie a rebuild to
an error string no run of this store has ever produced, so the unit
asks `bao auth list` and mounts only on absence. That read is why the
token policy in setup.md gains `sys/auth`.
Each grant came from `bao <cmd> -output-policy`, which prints what a
command requires without running it -- the same way controllerPolicyText
was derived. Enabling an auth method needs `sudo` on `sys/auth/cert`,
which the documented token did not have.
Gated on `clientCaFile`, not on the token alone: `swarm-bao-certs`
installs `client-ca.pem` only under that condition, and a role's
`certificate=` has to name a real CA. The policy write, which needs no
CA, is unchanged in that case.
Nothing can present a certificate for this role yet -- the only client
leaf the tree mints carries CN = the hive's name -- and none of this has
been run against a live store. Both are stated in setup.md.
The last of the twelve. `swarm.forge.package` becomes
`deploy.forgejo.package`, beside the `behindGateway` / `mirrors` /
host-secret options that were already there.
⚠️ `deploy.forgejo`, not `deploy.forge`. The forge joins the controller
and the UI as a mover where the two sides simply do not share a name,
so the shim comment says which one.
Four readers, and one of them is why an anchored grep was the wrong
instrument: `:678` reads `cfg.package.data`, a sub-attribute, which
`cfg.package}` and `cfg.package/` both miss.
Three copies of the same sentence claimed the package stays under
`swarm.forge` — `deploy.nix`, this module's own `deploy.forgejo`
header, and `docs/integrations/forge.md`. The docs one is outside
`nix/` entirely. A namespace split gets described wherever the split is
explained, which is not only where the option is declared.
`forgeOldPath` gains the old path, and its comment's count of the
host-side options goes five → six with it.
Both are regressions from earlier commits in this same slice, not
pre-existing drift:
- the nats comment listed "the responder package" among what stays
under `swarm.nats`; `authPackage` moved in a17286c4
- the matrix comment listed "its package" among what `swarm.matrix`
keeps; both matrix packages moved in 1b2da54c
Each move edited the prose in the module that declares the option —
`swarm-nats.nix` carries the same "responder package" sentence and it
was corrected there — but deploy.nix keeps its own one-paragraph
description of every service, next to that service's shims. It is a
per-module prose site that does not live in the module, so a sweep
scoped to the module cannot reach it.
The counts in both comments were checked and are correct as written:
the matrix block has six shims and the nats block five. The `*.package`
entries append to the separate block further down, which already has a
comment saying what it is, so no count moved. Each fix is one clause.
`package` (the daemon build) and `swarmctlPackage` (the operator CLI
installed beside it) are host decisions, so they join the
`deploy.swarm-controller` block that already holds `enable`, the socket
path and the three local-disk credentials.
⚠️ The target namespace is spelled out in the shim comment because
getting it wrong here does not fail. `deploy.hive-controller` also
exists — it is hive-c0re's, carrying `tls.*` and `statusPublish.*`, and
is referenced from four modules. A rename pointing at it would land on
a live, unrelated option and evaluate cleanly. The module already warned
about this above its own deploy block; the warning now sits next to the
entries it governs too.
`swarm-ui.nix`'s package description cited `swarm.controller.package` as
the precedent for "wired from flake.nix rather than defaulted to pkgs".
That was true when written and stops being true here, so it moves with
it — it lives in a module this slice already finished, which is exactly
where a per-module sweep stops looking.
These were the last two `services.hyperhive.swarm.*` writers in
flake.nix; that file now has none.
Riding along, and not a defect fix: `controllerOldPath` spelled `enable`
the NEW way while every other entry used the old one, so six of the
seven controller shims were exercised and the seventh was not. It is
named for testing old paths, so it now uses the old path throughout.
Unlike the nats fixture there was no comment claiming otherwise — this
is coverage the fixture was already shaped to provide.
`package` (which authelia build runs in the container) and
`bridgePackage` (which `swarm-authelia-bridge` build writes the users
file) are both host decisions, so they join the `deploy.authelia` block
that already holds `enable`, `usersFile` and the two host-side paths.
Six readers, and one of them is why this commit touches a second module:
`swarm-controller.nix` reads `autheliaCfg.package` for
`SWARMCTL_AUTHELIA_BIN`. A per-module sweep never sees that — it is a
cross-reference between two movers, and whichever landed second would
have inherited a dangling alias. The alias itself stays: it still has
three live uses for `url` and `bridgeUrl`.
Two comments needed real edits rather than a rename:
- The `deploy.authelia` header said "all three of these are the same
kind of thing — a filesystem path". It is five options now and two
are packages, so the sentence distinguishes the builds from the
paths, and keeps "a hive that does not run it has nothing at any of
those paths" scoped to the paths it is true of.
- `module-eval.nix` said "Only `usersFile` has a rename entry",
explaining that authelia's other slice-7 movers are `readOnly` and a
rename module cannot contribute a definition to those. That reasoning
is worth keeping, but the claim stopped being true one line above the
two entries this commit adds.
The fixture gains both old paths; no new case. `autheliaOldPath`'s
existing arms already fail the eval if either shim goes missing.
`swarm.nats.authPackage` is the callout responder's build — a host
decision like every other package in this slice — so it joins the
`deploy.nats` block that already holds `enable`, the seed paths and the
auto-mint toggle. Its one reader is the `ExecStart` in this file.
Three comments moved with it, and none of them survived the move
unedited:
- The "deliberately NO `package` option" note now says *anywhere* —
it argued against a `package` under `swarm.nats`, and after the move
a reader's next question is why there isn't one under `deploy.nats`
either, where every other service's build now lives.
- The description's "see the note above" stopped resolving: the note
is ~80 lines up and in a different option block, so it names
`options.services.hyperhive.swarm.nats` explicitly.
- The split-rationale comment listed "the responder package" among
what the queue IS to every hive. It's what the host running it
decides, so it moved to that half of the sentence.
`flake.nix` writes this option (`lib.mkDefault`), so the writer is
repointed too — three of those left, all in the same block.
The eval fixture gains the old path rather than a new case: the shim
is what a dropped rename would break, and `natsOldPath` already fails
the eval if any single entry goes missing.
Same rule as the four movers before it. What makes this one different is
that `package` was the LAST option under `services.hyperhive.swarm.ui` —
`enable` moved in an earlier slice — so the namespace now holds nothing
and survives only as its two rename entries.
Two consequences worth naming:
flake.nix set `swarm.ui.package` directly. Left alone the shim would
have kept it working, which is exactly the trap: this repo's own flake
would then be setting the option through its own deprecation shim and
warning on every eval. It is repointed here. Four sibling assignments in
that file name movers this commit does not touch; they move with theirs.
The `cfg` alias is deleted. With the one code reader repointed it bound
nothing but an empty attrset — which still evaluates, so nothing would
have failed; it would just have sat there implying `swarm.ui` still held
something. Three comments naming `cfg.package` are repointed with it.
`uiOldPath` sets both old paths, so dropping either rename fails the
eval. Its case reads the vhost's rendered `root` rather than the option,
because that is the half a resolving-but-unwired shim would break.
Not touched: swarm-ui.nix's description says its default is wired "the
same way `swarm.controller.package` is", which is true until the
controller moves. It belongs to that commit, not this one.
Refs #3772.
Same rule as matrix and grafana: which build a service runs is a decision
of the host that runs it. Both stores already had a `deploy.<store>`
option for retention, so the package joins something rather than opening
a namespace.
The prose in both modules claimed the package as part of "what the store
IS from any hive's point of view" — a client hive needs the domain and
the port to reach a store, never the build it runs. deploy.nix's own
comment made the same claim about the pair and is corrected with them.
Separately, and the reason this commit adds a fixture rather than a line:
NEITHER STORE HAD AN OLD-PATH FIXTURE AT ALL. `swarm.victorialogs.` and
`swarm.victoriametrics.` had zero hits in module-eval.nix, so the
`enable` shims from the first slice and both `retentionPeriod` shims have
been uncovered since they landed — the suite would have gone green with
any of them deleted. That is precisely what the wireguard fixture's own
comment warns about: a missing shim reads as a clean tree and breaks
every existing operator config.
`storesOldPath` therefore sets all six old paths, not just the two this
commit moves. The case reads the package the CONTAINER renders rather
than the option, so a shim that resolves but stops reaching the module
fails too.
Refs #3772.
Same rule as the matrix pair: which build a service runs is a decision of
the host that runs it, so `package` follows `enable` out of the namespace
every hive must agree on byte for byte.
Grafana already had a `deploy.grafana` block holding the datasources, the
plugin list and the socket directory, so this is the package joining
options it belongs with rather than a new namespace.
Prose: the swarm-side header listed "its package" among what stays; the
deploy-side header listed what the host decides without it. Both edited,
which is the fix — a substitution on the first alone would have left the
second silently incomplete.
`swarm-grafana.nix:219` was on the sweep list and is NOT edited: it says
what stays above is what the service is to every hive, and that stays
true once the package leaves. The stale half was the sentence after it.
Refs #3772.
Which build a service runs is a decision of the host that runs it, so
`*.package` belongs with `enable` rather than in the namespace every hive
must agree on byte for byte.
The `gui` pair was the sharpest case and the reason this starts here:
`gui.enable` moved to `deploy.*` in an earlier slice while `gui.package`
stayed behind, so one web client's two halves sat in two namespaces —
and hive-matrix.nix said so, in a comment directly above the option.
The diff removes a stated inconsistency rather than introducing a rule.
Both old paths keep working through `mkRenamedOptionModule`; both are
settable (neither is `readOnly`), so both get a shim. The `matrixOldPath`
fixture in module-eval.nix now sets eight options through their
pre-rename paths instead of six, so dropping either new shim fails the
eval rather than silently dropping a definition.
Prose swept in the same commit: three comments claimed the package was
swarm-wide or named the pre-rename path, and docs/web-ui/dashboard.md
named `swarm.matrix.gui.package` two lines above
`deploy.matrix.gui.enable` — the same asymmetry this commit exists to
remove, displayed in the docs. A shim keeps the old spelling valid, so
no build, lint or broken-reference search would have found that line.
Refs #3772.
Cert auth answers a role, so nothing in the swarm can authenticate to
the store until some role exists. Creating the first one therefore
cannot itself use a certificate — the credential has to come from
outside that cycle, and an operator places it.
`deploy.bao.bootstrapTokenFile` names that token. A one-shot unit inside
the store's container reads it and writes the `swarm-controller` policy;
`local-defaults.nix` supplies the path (never the file) on an all-local
deploy, since co-location makes only the location derivable. `bao
operator init` stays an operator action in every shape.
The unit runs in the container rather than on the host because writing
the first grant is a store-side operation: it reaches the store locally
and needs no client certificate. Gating it on the store being here is
therefore not the co-location assumption glue-matrix-bao-token.nix
warns about — a reader has to work from anywhere, the first write never
does.
Policy only, deliberately: a cert-auth role binds a certificate and the
controller has no bao identity yet — it holds no leaf and contains no
bao code at all. The two certificates that do exist are both wrong to
bind. `clientCertFile` is the store host's own reader leaf rather than
the controller's, and the CA that signed it also signs every other
reader's, so binding that would let any reader authenticate as the
controller. Whoever gives the controller an identity writes the role.
The grants are scoped to `hive-*`. Without the prefix the controller
could rewrite the policy that constrains it.
Two things the module-eval arms pin: the unit renders inside the
container with the token path in both its script and its
ConditionPathExists, and a host that names a token while running no
store gets neither the unit nor the host-side directory.
The capabilities were derived with `bao write -output-policy` rather
than written from memory, and the setup.md commands were run against a
real binary for shape before being written down.
Refs #3726.
Slices 8 and 9 of the swarm/deploy split, and the last two.
statusPublish had three coordinates under one namespace. Two of them
are this machine's — where the queue listens *as seen from here*, and
where its client secret sits on this disk — so they move to
`deploy.hive-controller.statusPublish.*`, the namespace of the daemon
that is their only reader. `tokenEndpoint` is the swarm's one address,
so it stays. That leaves `swarm.statusPublish` holding a single option:
a legitimate split, not a botched move.
The all-or-nothing assertion now spans both namespaces. It is repointed
in both its condition and its message, and the message spells all three
paths in full so an operator is never told to set two options under a
path that only has one. `environment.nix`'s guard and the value beside
it likewise read different namespaces on purpose.
The collector's secret moves the same way, for the same reason, to
`deploy.swarm-otel.*` — `enable` already lives there. That also retires
one of the eight cross-namespace assignments tracked in #4048: the
delivery unit set a `swarm.*` value under a `deploy.*` gate, and now
sets a `deploy.*` value under one.
module-eval gets a fixture per slice. `otelRemoteAuthelia` already set
the collector secret through its pre-rename path, so it becomes slice
9's old-path case as it stands — left spelled that way deliberately,
with a comment, so it is not read later as a missed site.
That fixture also turned out to be describing an impossible hive: it
said authelia lives elsewhere without saying where, so the
authenticator interpolated a null `swarm.authelia.url` into its
`token_url`. Nothing to do with the rename, and invisible to the
existing case over the same fixture, which reads `? auth` and `elem` —
both stop at names and never force the extension's value. Given the
address a remote-IdP deployment has.
Verified: 49 -> 51 properties, all holding.
Two limits governed a matrix upload and nothing kept them in
agreement: the documented option reached the homeserver, while the
nginx location every client request traverses carried its own
`client_max_body_size 50M`. Raising the option past 50M changed
nothing — nginx returned 413 at the proxy, naming neither matrix nor
the option that had just been raised.
The cap is not set equal to the option. Equality would regress the
default: at 20 MB a 25 MB upload is currently rejected by the
homeserver, with a matrix error a client can act on, and equality
turns that into a gateway 413. The proxy must never be the party that
rejects, so it tracks the option with headroom. This is the shape the
forge vhost already uses for git, where a generous proxy cap lets the
application own the real limit.
The module-eval arm sets a deliberately odd size so the number it
looks for cannot have come from a default, and asserts the rendered
location. Its control names the whole directive rather than the bare
old value: nginx comments render into the config, so the comment above
the directive mentions 50M and a looser arm matched itself.
Two arms with different jobs. `usersFile` is configured through its old
path, so dropping the rename entry fails the eval. `hostClientSecretDir`
cannot be configured at all — it is read-only and derived — so its arm
asserts that a consumer renders the derived path, which is what breaks
if a reader is left pointing at the namespace the option moved out of.
Seven of those reads went through an `autheliaCfg` alias rather than a
full path, where a path-shaped grep does not see them. That, not the
rename, was the failure this change could have shipped.
The two read-only options get no rename entry. A rename module
contributes a definition to its target and a read-only option refuses a
second one, so a shim for either makes every evaluation fail with "set
multiple times", naming the option's own declaration as the rival. Of
the 38 distinct leaves the rename table already targets, these are the
only two declared read-only.
`usersFile`, `hostClientSecretDir` and `hostUsersFile` are filesystem
paths that only exist on the machine running the `swarm-authelia`
container. A hive that does not run it has nothing at any of them, so
they fail the swarm-wide test the namespace's header states.
`usersFile` is a path *inside* the container and still belongs on the
deploy side: a path's scope is the scope of the filesystem it names, and
that filesystem is this host's container root.
The two `host*` options are `readOnly` and derived, so nothing can set
them through the rename shims — those serve readers. The shims are still
required: `mkRenamedOptionModule` is what keeps an out-of-tree module
reading the old path resolving at all.
Seven reads went through an alias rather than a full path
(`autheliaCfg.hostClientSecretDir` in five modules, `.hostUsersFile` in a
sixth, `swarmCfg.authelia.hostClientSecretDir` in a seventh). Every one
of those files already binds `deployCfg`, so the repoint needed no new
alias. Four more sites were prose, including one in `docs/`, which no
grep restricted to `nix/` would have found.
`swarm.*` is what a hive needs to be a *client* of the swarm. For the
controller that is the quick-links it publishes, the client id it is
registered under, how stale a snapshot may read, and its packages. Where
its socket lives and where the three credentials it reads sit are
decisions of the machine running it, so socketPath, forgeTokenFile,
authBridgeUrl and queue.clientSecretFile move to
`deploy.swarm-controller.*`.
⚠️ `deploy.swarm-controller`, not `deploy.hive-controller`. Both exist on
main — the latter is hive-c0re's namespace, where `tls.*` lives — and a
definition on the wrong one lands on a live unrelated option instead of
failing. The shim table is anchored on both sides for that reason.
`queue.clientSecretFile` is a nested field, not a top-level option: it
moves out of `queue` and leaves natsUrl and tokenEndpoint behind, so the
far side needs `deploy.swarm-controller.queue.clientSecretFile`. That is
the `forge.sso` split shape, applied again rather than reinvented — the
endpoints a client dials are swarm-wide, the secret beside them is a path
on one host.
local-defaults.nix set that secret from INSIDE
`config.services.hyperhive.swarm = { ... }`, where a bare `controller.`
prefix means `swarm.controller`. Left there it would still resolve —
through the rename — and warn on every evaluation of a single-host swarm,
which is the same defect an earlier slice fixed for the matrix module. It
moves out into its own `deploy` statement beside the controller's
`enable`. `queue.natsUrl` stays bare: it is a stayer.
swarm-ui.nix read `socketPath` through its own `controllerCfg` alias.
With that repointed the binding had no reader left, so it goes. Unlike
the dead `natsCfg` an earlier slice removed, this one was live until this
commit — the move orphaned it. `deployCfg` was already bound there.
Prose that named a moved option by its full path is requalified: the
assertion message for the queue secret (operator-facing, and inside
`config` where an options-block sweep would miss it) and
`swarm-controller/README.md`'s socket-path line. Left bare on purpose:
`docs/swarm/README.md` and the README's own "never point `socketPath` at
a directory that carries anything else" both name the FEATURE, not a
path.
module-eval configures a hive through all four OLD paths and asserts a
rendered effect for EACH of them — the unit's socket env, its auth-bridge
env, and both credential paths in `LoadCredential`. Asserting all four
rather than one means a rename that resolves but stops reaching the
module is caught per-option, not only where a single assertion happens to
look.
`swarm.*` is what a hive needs to be a *client* of the swarm. For Grafana
that is the package, the name it is served under, the port its `/metrics`
is re-served on, and the OIDC client it is registered as. Where its
datasources point, which plugins sit in its store path and the directory
it shares a socket with nginx through are decisions of the machine
running it, so datasourceUrl, logsDatasourceUrl, plugins and socketDir
move to `deploy.grafana.*`.
The two URLs are the interesting half. `swarm-grafana.nix`'s own summary
sentence said what stays is "its package, domain, and wiring" — and both
datasource URLs ARE wiring, so that sentence is rewritten with the move
rather than left asserting the opposite of what the module does. They
move because a URL's scope is the scope of what it ADDRESSES, not the
fact that it is a URL: docs/swarm/services.md already said datasourceUrl
"defaults to the store on this host, which is the only thing it can
reach", because that store binds loopback. The doc argues the move.
`socketDir` was already ruled host-side: the directory is shared between
the host's nginx and the container, and it is the same shape as the
options that moved in earlier slices.
Declared in swarm-grafana.nix under the `deploy.*` path, following
swarm-nats.nix; deploy.nix carries only the renames. One reader outside
the options block is prose: `metricsPort` STAYS and cross-referenced
`{option}services.hyperhive.swarm.grafana.socketDir`, which the split
makes a pointer to another rendered page — nixosOptionsDoc emits
swarm.md and deploy.md separately. Requalified, along with the one line
in docs/swarm/services.md that named a mover.
module-eval configures a hive through all four OLD paths and asserts a
rendered effect: the host tmpfiles rule that creates the socket
directory carries the fixture's custom path. The new paths evaluate fine
without the shims, so dropping them has to read as a clean tree; all
four are defined in the fixture, so removing any single shim fails the
eval rather than only the one the assertion reads.
`swarm.*` is what a hive needs to be a *client* of the swarm. For the
queue that is the ports it answers on, the client id it is registered
under, and the responder package. Whether this host mints its own
auth-callout keypairs and where the seeds sit are decisions of the
machine running it, so autoGenerateCallout, both seed files and both
public keys move to `deploy.nats.*`.
The two PUBLIC keys are the non-obvious half, so the reasoning is here
rather than in a comment nobody re-reads. A public key looks swarm-wide,
and docs/swarm/secrets.md says the user key "is published to every
client that connects" — which is true and does not make it swarm
config: peers RECEIVE it over the wire at connect time, they never
configure it. What decides the placement is that its seed is
unambiguously host-side, and splitting a keypair across two namespaces
is worse than either placement.
local-defaults.nix set `nats.autoGenerateCallout` from INSIDE
`config.services.hyperhive.swarm = { ... }`, so the bare prefix meant
`swarm.nats` and no full-path grep could see it. It moves out of that
block into a `deploy` statement rather than being rewritten in place.
swarm-controller.nix bound `natsCfg` and never used it — one mention,
no bare pass, while its sibling bindings have 5 and 10. Pre-existing
dead code, found by this slice's own alias sweep, removed with it.
Four assertion messages and five doc lines named moved options by a
short form (`nats.calloutUserSeedFile`) that is ambiguous now that both
`swarm.nats` and `deploy.nats` exist; one opened with
"services.hyperhive.swarm.nats has callout public keys", which the split
makes false. Prose that names a VALUE rather than a path — the `nk`
pipeline's `# -> calloutUserPublicKey` — is left bare on purpose.
module-eval configures a hive through all five OLD paths and asserts the
responder's delivery unit exists and interpolates the seed path.
`autoGenerateCallout` is deliberately FALSE there: that is what makes
the seed options decide `responderConfigured`, so the assertion is about
the seeds rather than about the auto-mint branch.
`swarm.*` is what a hive needs to be a *client* of the swarm. For the
homeserver that is what it IS from anywhere: its package, the name it
answers to, the ports and URLs it is reached on, and the client id it is
registered under. Whether it is exposed, which peers it trusts, how large
a request it accepts and where its host-local secrets sit are decisions
of the machine running it, so openFirewall, trustedServers,
maxRequestSize, registrationTokenFile, gui.enable and
sso.clientSecretFile move to `deploy.matrix.*`.
Two sub-blocks split rather than moving whole, on their own evidence.
`gui.enable` is whether THIS host serves the web client; `gui.package` is
which client, an artifact identity, and stays. `sso.clientSecretFile` is a
path on one host; `clientId` must match the id in authelia's register, so
it is swarm-wide. Each half now points at the other, because the rendered
docs put them on separate pages.
hive-gateway passed the whole `swarm.matrix` attrset into vhosts.nix, so
that file read a moving option through an argument with no option path
anywhere in it. It now takes `matrixDeployCfg` beside `matrixCfg` — the
only shape that carries a split namespace across that boundary.
While there: vhosts.nix read `matrixCfg.enable`, which has been a rename
alias for `deploy.matrix.enable` since the enable moved. Reading it made
the module system print `Obsolete option services.hyperhive.swarm.matrix.
enable is used` on EVERY evaluation of every host — a deprecation warning
no operator could silence, because the config tripping it was ours. That
shim lives in hive-matrix.nix rather than in this file's table, which is
why deploy.nix's header claim to be their single home is now qualified
in the new block's comment.
glue-matrix-bao-token.nix read the registration token through its own
`matrixCfg` alias; with that read repointed, the binding had no reader
left, so it goes, and the comment naming it is reworded.
module-eval gains a case configuring a hive through all six OLD paths and
asserting two rendered effects — the host firewall's port list and the
container's bind-mount table — because the new paths evaluate fine
without the shims. `gui.enable` is set to the opposite of its default so
the definition has to land rather than agreeing with it by accident.
`swarm.*` is what a hive needs to be a *client* of the swarm. For the
forge that is what it IS from any hive's point of view: its package, the
names and ports it answers on, the URLs it advertises, and the client id
it is registered under. How it is served, what it mirrors and where its
host-local secrets sit are decisions of the machine running it, so
behindGateway, openFirewall, mirrors, sso.clientSecretFile and
hostSwarmControllerTokenFile move to `deploy.forgejo.*`.
Unlike the wireguard mesh this SPLITS a module rather than relocating a
whole namespace. `sso` splits with it: `clientId` stays because it must
match the id in authelia's register, while the secret beside it is a
path on one host. Moving the whole `sso` block for symmetry with `ci`
was considered and rejected on exactly that asymmetry.
Declared in hive-forge/default.nix under the `deploy.*` path, following
swarm-victorialogs.nix; deploy.nix carries only the renames. `mirrors`
renames in one entry rather than one per field — it is a single option
of a list-of-submodule type, so the rename carries its whole value,
where `ci` needed five because it is a plain attrset of options.
Readers outside the module: hive-ci.nix binds `deploy.forgejo` for its
behindGateway assertion; swarm-authelia.nix and swarm-controller.nix
read theirs off the `deployCfg` they already bind. hivectl's `open`
printed `services.hyperhive.forge.behindGateway` in an operator-facing
hint — a path that never existed, missing `swarm.` — and hive-c0re's
state_snapshot doc comment carried the same defect; both now name the
new path. The rendered docs put the two halves on separate pages, so
the five descriptions of staying options that explain themselves in
terms of `behindGateway` now qualify it in full.
module-eval gains a forge case configured entirely through the old
paths, asserting the rendered firewall ports and the mirror env var
c0re seeds from: the new paths evaluate fine without the shims, so
dropping them reads as a clean tree. All five old paths are defined in
the fixture, so removing any single shim entry fails the eval rather
than only the two the assertion reads.
`swarm.*` is what a hive needs to be a *client* of the swarm; the mesh is
none of it. A peer needs this host's `wireguardEndpoint` -- the roster entry
in swarm.nix, which stays -- and nothing about the interface this host
brings up. The module already said so: "plain host networking that a machine
which runs no hive at all still needs."
All five options move, so the namespace relocates rather than splitting.
`listenPort` is the one that reads the other way: it is what this host
*binds*, while the port a peer *dials* lives inside `wireguardEndpoint`.
Declared in swarm-wireguard.nix under the `deploy.*` path, following
swarm-victorialogs.nix; deploy.nix carries only the renames, per its own
"a single file to delete when the deprecation window closes". Deliberately
NOT added to deploy.nix's own options block: every entry there is a swarm
service this host deploys, and the mesh is host networking.
hivectl/src/wg.rs generates the config snippet an operator pastes, so it
moves too -- otherwise the tool's own output trips the deprecation warning.
module-eval gains a case that configures a host through the OLD path and
asserts the rendered wg-hive interface, because the new path evaluates
fine without the shim: dropping it reads as a clean tree.
The block header says `deploy.bao.*` is what the host RUNNING the store
decides and that none of it means anything on a host that only reads
secrets. Three options in that block are exactly what a read-only host must
set, and their own descriptions say so a screen further down — so reading
the options never corrects the header.
It reads as though an off-host reader were inexpressible. It is already
supported: glue-matrix-bao-token gates on the identity rather than on
`enable`, and module-eval's baoRemoteReader is that deployment.
Also drops a sentence describing what used to be undeclarable.
`credential.rs` calls `SecretStore::from_env`, and nothing set `BAO_*` for
this unit — only `swarm-bao-matrix-token` had them. Fixing that alone would
not have been enough: hive-c0re runs as hive-core, and glue-bao-tls mints the
client key `0600` inside a `0700` directory, so the daemon cannot read the
file even when it is named.
Both halves go through LoadCredential, which this unit already uses for the
swarm-status client secret: root reads the identity at unit start, hive-core
sees it under `%d`, and no second on-disk copy exists.
The gate is the identity, never `deploy.bao.enable` — a hive that reads a
store on another machine holds a certificate and runs no store. Four
module-eval cases: the co-located and off-host presence arms, the absence arm
for a hive with no identity, and a presence/absence pair for the optional CA.
The swarm collector reads /var/log/journal and has never seen a single
container unit. mara's count-by-unit against VictoriaLogs returns four
units, all host-tier; openbao -- which runs inside the swarm-bao
container -- is absent.
Cause: nixos-containers.nix hardcodes `--link-journal=try-guest` for
every non-ephemeral container. With `guest`, the host's
/var/log/journal/<machine-id> is a SYMLINK into the container's transient
root; a reader in the host namespace cannot follow it, and it dangles as
soon as the container stops. `ls -la /var/log/journal/` on the host shows
one real directory and a pile of `-> /tmp/nspawn-root-*` links dating
back to May.
swarm-otel.nix asserted the opposite, and that assertion is why the
receiver's path was considered sufficient: it said the files "live here"
and are "bind-mounted into the guest rather than the other way round".
That describes `--link-journal=host`. The same sentence names the flag we
actually use. The flag was right and the behaviour it described was not,
so grepping for the flag confirmed the comment and taught nothing.
`containers.<name>.extraFlags` feeds EXTRA_NSPAWN_FLAGS, which the
invocation expands after the hardcoded flag, so `--link-journal=host`
wins. The comment now describes what the code does instead of the other
way round.
Two payoffs, and the smaller one is the one the issue is about:
container logs become collectable, and -- independently -- they become
durable at all, rather than dying with the container.
Ten identical edits because ten host-modules hand-roll their own
container block; that duplication is #3773, not something to invent an
abstraction for here.
NOT VERIFIED: that systemd-nspawn honours the last `--link-journal` of
two. Everything else here is read out of nixpkgs; that step is a claim
about its argument parsing which cannot be exercised without starting a
container. It is settled by deploying one and re-running the `ls`: the
machine-id entry becomes a real directory instead of a symlink.
Refs #3849
Filtering the gateway's journal by severity returned nothing while nginx
was logging [error] continuously. Measured against nginx.service:
`-p warning` and `-p err` over 24h both said "No entries", while the
unfiltered query returned a continuous stream of [error] lines.
The failure is inverted rather than merely missing. A severity filter is
how you ask "is anything wrong", and here it answered "no" *because* the
errors were misfiled -- one step from recording "gateway clean, no
warnings in 24h".
Cause: the module default is `error_log stderr`, and systemd files that
whole stream at one priority regardless of what each line says. Routing
to syslog preserves per-line severity, so [error] lands as err and every
existing severity filter starts working without changing a consumer.
Uses the module's own `logError` option rather than appendConfig:
nixpkgs emits `error_log ${cfg.logError};` at exactly one site, in the
main context, so the option guarantees the placement. Transport only --
no level suffix, so nginx keeps its default `error` threshold and this
changes where lines are filed, not which lines exist.
Gated with `nginx -t` on the flake-locked nginx against the exact string
shipped here, with a deliberately invalid syslog target as the control
that the test can fail on this directive. gixy (what the module's own
validateConfigFile actually runs) reports no issues.
What that does NOT prove: that journald files the lines at the right
severity. `nginx -t` is a parse check; only a deploy shows the effect.
Pure `nix fmt` output from the commit before this one — no hand edits.
203 files: 52 md, 42 tsx, 32 js, 32 css, 21 ts, 13 html, 8 json, 3 mjs.
Reproduce with `nix develop -c nix fmt` on the parent commit; the result
should be byte-identical to this tree.
None of the 13 `.prettierignore` entries appears here — verified by
intersecting the changed-file list against the ignore file, with a
control proving the intersection finds a match when one exists.
`docs/` was reorganised into topic subdirectories and the references to it
were not moved with it. Thirteen distinct paths and three relative links no
longer resolved, spread across nix, css, html, js, markdown and
.prettierignore — a stale pointer is not a markdown problem, it is wherever
someone wrote a path down.
Each mapping resolved to exactly one target. `docs/matrix.md` was the sole
ambiguous basename: .prettierignore lists `docs/tools/matrix.md` separately
and that entry still resolves, so the stale one is the integrations doc.
The three relative links were each one `../` too deep — from `<crate>/src/`
two levels reach the repo root. `hive-agent/src/login.rs` already had the
correct form, in the same crate, at the same depth.
.prettierignore is repointed rather than dropped, though nothing in the tree
runs prettier: no treefmt entry, no CI job, no package script. Whether that
config should exist at all is a separate question from whether it names
files that do.
The collector bind-mounts /var/log/journal into its container and points
its journald receiver at that directory. journald only writes there when
it stores persistently: with storage = "volatile" the journal lives in
/run/log/journal, and with "none" there is none at all. Nothing in the
tree sets or asserts services.journald.storage, so a host with either
value deploys a collector that cannot start — nixos-container refuses to
start when a bind source is missing, which is the same trap the
swarm-bao-certs and hostClientSecretDir comments document.
A bindMounts entry never creates its hostPath, and unlike every other
runtime-written bind source in the tree there is no unit to order the
container after: each of those declares before + requiredBy on its
producing unit, and this directory has no producer. journald was told to
store persistently or it does not exist. That leaves an assertion as the
only place to say so.
"auto" is deliberately accepted: it uses the directory when it exists,
and evaluation cannot see whether it does. Certainty is what separates
an assertion from a warning, and only "volatile" and "none" are certain.
Verified by reading config.assertions for all three values rather than
system.build.toplevel, which raises the first failing assertion in the
whole config and so reports a minimal stub's unrelated ones instead:
persistent -> mine [] others 3
volatile -> mine [the message, quoting "volatile"] others 3
none -> mine [the message, quoting "none"] others 3
The others count is identical across the three, so the storage value is
the only thing that differs between them, and a populated list is what
makes the empty one mean "this assertion passed".
`docs/swarm/ca.md` said it under *Constraints on the material* and again
under *Distributing the root*; this module's header said it a third time,
directly above a line already pointing at that doc.
Measured: 6-gram intersection between the module's comments and the doc,
32 -> 6. The sentence now appears once in the tree.
Module header keeps what a reader editing this file cannot see from the
config: that one root makes cross-hive trust O(1), that the two
provisioning modes differ only in who writes the artifacts, and the ⚠️
that both root key and root cert are runtime files — the cert as a
consequence, and that one costs something real, since nothing whose trust
store is built at build time can name it. Everything behind those is a
`docs/x.md::Section` pointer with the heading checked to exist.
Dropped the O(n²)-per-peer-pinning story: the live reason is that one root
is O(1), and how it used to be configured is history.
In the doc, *Distributing the root* now links to *Constraints on the
material* rather than restating it. Intra-doc `#anchor` links are already
the convention there — 85 of them across `docs/`, 8 in `gateway.md` alone.
Comments + one doc paragraph; no behaviour change. Refs #3901.
The module header restated `docs/scheduler/ci.md`'s credential-isolation
paragraph nearly word for word, and re-told the nix-sandbox gotcha that
`docs/process/gotchas.md` owns — both immediately above lines that already
pointed at those docs.
Measured, not eyeballed: 6-gram intersection between this file's comments
and `docs/scheduler/ci.md`, 74 -> 24.
Kept, because neither is visible from the config that implements it and a
reader changing this file needs both:
* the container has its own network namespace, not the host's, so a PR's
nix build cannot reach host-loopback services — a prompt-injection
boundary rather than a preference;
* the forge ADMIN token never enters this container; only a registration
token is handed in.
Each is now one sentence naming the property and its consequence, with the
mechanism behind a `docs/x.md::Section` pointer whose heading was checked
to exist (`Container isolation`, `Container design`).
Also restated in the present: the container is non-ephemeral because the
runner writes its registered credentials on first registration and reuses
them, which is why an ephemeral one would re-register every restart. That
was previously a description of what gitea-actions-runner does, with the
reason left implicit.
Comments only; no behaviour change. Refs #3901.
Two comment blocks explained things `docs/networking/gateway.md` already
explains, one of them directly above a line pointing at that very doc. A
pointer under a duplicate is not a pointer; it is a duplicate with a
citation.
Measured, not eyeballed: 6-gram intersection between this file's comments
and that doc, 73 -> 31. The residual is mostly the one-line summaries kept
at the call sites, which necessarily share phrasing with the doc they
summarise — that is the floor of the metric, not remaining duplication.
* the self-signed-cert import unit: kept the 🚨 that stops someone
collapsing it into "point nginx at the CA dir", both jobs at one line
each, and the blast radius (whole gateway, not one vhost). Dropped the
`BIO_new_file()` mechanism and the production-outage narrative — the
doc carries both, and the second is history.
* the nginx-reload note: rewritten to say the one thing this file needs
to say, that nothing here reloads nginx and hive-c0re drives it from
the host through hive-priv. The per-unit-state dispatch table is the
doc's, and only the doc has it in full.
Both pointers use the `docs/x.md::Section` form and name headings verified
to exist; the first replaces a parenthetical `("Self-signed TLS")` spelling
that no check can resolve.
Comments only; no behaviour change. Refs #3901.
Comments cite nix modules, scripts and crate source files constantly,
and nothing evaluates a comment — so when a file moves, the reference
rots silently and `nix flake check` stays green. A reader following one
finds nothing and cannot tell whether the file was renamed, deleted, or
never existed.
Seven such references, each repointed at the file that actually holds
the thing the sentence is about rather than at the directory the old
name became:
hive-c0re/src/agent_config/limits.rs hive-agent/src/mcp.rs
-> hive-agent-mcp/src/mcp/mod.rs
hive-agent-mcp/src/mcp/mod.rs hive-c0re/src/limits.rs
-> hive-c0re/src/agent_config/limits.rs
(and the module path in the doc
comment above it, which was stale
in the same way)
hive-c0re/src/forge/mod.rs hive-c0re/src/knowledge.rs
-> hive-c0re/src/workers/knowledge.rs
nix/host-modules/hive-c0re/options.nix hive-c0re/src/hive_stats.rs
-> hive-c0re/src/stats/hive_stats.rs
nix/packages/default.nix nix/host-modules/hive-c0re.nix
-> .../hive-c0re/options.nix
nix/agent-modules/network.nix nix/host-modules/hive-gateway.nix
-> .../hive-gateway/dnsmasq.nix
frontend/README.md nix/modules/frontend.nix
-> nix/packages/frontend.nix
The two `limits.rs` comments are a matched pair: each names the other's
old path, so the "keep in sync" instruction they exist to carry pointed
both ways at nothing.
Where a flat module became a directory the target is the file that
declares the named thing, not `default.nix` by reflex — the
`preBuildAgentTemplates` option is declared in `options.nix`, and the
DHCP pool that sentence is about lives in `dnsmasq.nix`.
Comments only; no behaviour change. Refs #3923, which is about whether a
gate should cover this class at all — that question is unanswered and
this does not close it.
`docs/swarm/sso.md` described a person in a browser. The swarm's other
callers — the telemetry collector, the queue's auth-callout responder, each
hive's agents — hold no session and follow no redirect, and nothing operator-
facing said how they authenticate. Its relying-party table is forge and
matrix, both browser surfaces.
The new section carries what `swarm-authelia.nix` was holding in comments:
one client per hive because identity belongs to the directory, the audience
being that client id rather than a parallel naming scheme, and signed rather
than opaque tokens because the collector verifies offline against
`/jwks.json` while the queue introspects.
It also states the fail-closed rule once, in the place a reader looks before
touching a vhost: an error page answers 200, and `auth_request` reads any 2xx
as access granted. That shape has now appeared three times — this module's
`/api/` prefix and both of victorialogs' routes — which is what makes it
documentation rather than a comment.
The two comment blocks those replace shrink to the part that is genuinely
local: the submodule-typing reason these clients are a definition rather than
an append, and a loud warning against folding the machine prefix back into
`/`. The security warning stays at the site; only its consequence list moves.
Comments 495 -> 465 lines. Option `description` strings are untouched: they
are the source `pkgs.nixosOptionsDoc` renders into the operator's options
reference, so trimming one would delete published documentation rather than a
duplicate.
Per mara's go-ahead on hyperhive#3902 ("getting started is good, but
terminal rendering does not go in there i think"):
Moved 21 top-level docs/*.md files into 7 new topic subdirectories
(existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched):
getting-started/ setup.md
agent-lifecycle/ agent-hierarchy.md, approvals.md, persistence.md
trust-boundary/ boundary.md, security.md
integrations/ forge.md, matrix.md, github.md, knowledge.md
networking/ gateway.md, network.md, snapshot-store.md
scheduler/ jobq.md, coordinator.md, ci.md, observability.md
process/ conventions.md, gotchas.md, pr-review-gate.md
web-ui/ terminal-rendering.md (moved into the EXISTING dir,
per mara's correction to the original getting-started
guess -- it's UI implementation detail, not onboarding)
The physical layout now matches docs/README.md's own topical headers,
which already amounted to this taxonomy -- see the scoping comment on
the issue for the two findings that motivated this (a genuine
duplication between CLAUDE.md's old "Reading paths" list and
docs/README.md's grouped one, since drifted out of sync with each
other; and the flat layout not matching the grouping we already had).
Fixed every cross-reference this moved across the whole repo (~120
files: docs/ internal links at every depth, Rust doc comments, nix
module option docs, crate READMEs) -- verified two ways: a grep sweep
confirming zero remaining references to any old path, and a script
that resolves every markdown link in docs/**/*.md + CLAUDE.md +
README.md against the filesystem and reports anything that doesn't
exist (zero broken links).
Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to
a pointer at docs/README.md, now the single index. Rewrote
docs/README.md itself to use the new subdirectory paths and added the
one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md).
Classified all 22 docs/*.md files first via a haiku subagent (mara's
suggestion) on two axes -- proposed grouping and operator-vs-
implementation focus -- before finalizing the taxonomy; spot-checked
the report and found internal inconsistencies (its classification
table disagreed with its own summary section for a few files), so this
taxonomy is my original proposal + the one correction mara gave
directly, not a blind application of the subagent's table. The
operator-focus data it gathered is still useful for a follow-up
content pass (docs skewing 'mixed' rather than pure operator-facing),
not addressed in this PR -- structure only.
nix fmt clean, both pre-push lints clean.
`docs/swarm/sso.md` already argues that the OIDC provider is derived from
the client list rather than carrying its own `enable` — authelia refuses to
start without clients, so a separate flag would be a second fact free to
disagree. The module header said it again, and the definition site said it a
third time.
The doc keeps the argument. The definition site keeps a short version,
because someone changing `oidcEnabled` needs the reason there and not two
files away, along with the part no doc carries: that it is unconditionally
true today and stays derived so the OIDC-gated code below is self-documenting
about why it is conditional.
The header also restated the users-store rationale the doc's "does not
provision users" bullet owns. What survives there is the one clause the doc
does not have — why a file backend is right rather than a placeholder for
LDAP, the subject set being bounded by one swarm.
503 comment lines to 495. Comment-only: still 36 module-eval properties.
Two named anti-patterns from the comment-budget issue, applied to the file
that carries the most comment lines in nix/host-modules.
The header re-explained the two-tier model and the co-location argument that
`docs/observability.md` already owns — "a boundary that disappears locally is
one the local deployment stops testing" appears in both, nearly word for word.
The doc is the right home for it; what stays here is the one fact the doc does
not carry, that this is a container because `services.opentelemetry-collector`
is a singleton option already taken by the hive tier.
The `logs_endpoint` block keeps every instruction and loses the narration:
use `logs_endpoint` not `endpoint`, both spellings pass `otelcol validate`
and every wrong path answers 400, `_msg_field` is required or text search
finds nothing while ingest still answers 200, `_stream_fields` sets stream
cardinality.
Six smaller trims of "measured"/"verified"/"once". One pointed at a probe
script under a single agent's state directory — a path no other reader can
open, in source every reader loads.
What stays: `validateConfigFile`'s "a parser, not a wiring check", the
`issuer_ca_path` warning, the swarm-tier stamp. Non-obvious, in no doc, and
the target is a share of the whole tree rather than a per-file quota.
604 comment lines to 574. Comment-only: the module still evaluates to the
same 36 module-eval properties.
The collector's store exporters were gated on deploy.victoriametrics.enable /
deploy.victorialogs.enable — options that say "this host RUNS the store", not
"the swarm has one". A collector that did not share a host with the stores
rendered no exporter at all and dropped everything it received, from every
hive, silently: an absent exporter is not an error.
mara ruled the premise rather than the mechanism ("the swarm always has a
store"), so there is no gate and no new option for that. Both exporters are
unconditional and address the stores by domain, per the cross-host rule from
the OIDC client and secret-delivery unit #3517 already built. The logs exporter
had the identical bug and is fixed here too.
Both stores gained a machine ingest route, required in the same change: the
exporter now targets https://logs.<domain>/insert/..., and that vhost is
browser-shaped, so shipping the collector half alone would have regressed logs
ingestion that works today. Neither ingest location carries
`error_page 401 =302` — a pusher handed a redirect follows it and POSTs at a
login page, which answers 200.
Whether the collector authenticates follows the CREDENTIAL, never another
service's placement: `clientSecretFile` is a nullable option, and the delivery
unit — the one thing here that may know where authelia runs, since it copies
out of its container — sets it by mkDefault. An earlier revision gated this on
deploy.authelia.enable directly, which put a different service's co-location in
the collector's own config.
Also removed rather than relaxed: the assertion that this collector has
"somewhere to send". It read the store's per-host enable, so it rejected at
eval exactly the deployment reaching the stores by domain exists for.
Deliberately not replaced with an authentication assertion — a collector on a
host of its own is a supported shape, and refusing to build it would make this
fix illegal where the bug bites hardest.
Knock-on worth review: collectLogs is now always satisfied, so journald
collection is unconditional.
Config shape validated against otelcol-contrib 0.151.0 `validate`, with a
bogus-key control confirming the validator checks the extension schema.
module-eval: 31 properties.
Nine panels against the metric names openbao actually emits — the prefix is
`vault_`, the fork kept upstream's names, and a board built on `openbao_`
would render perfectly and show nothing.
The top row works on a SEALED store, which is what a hive has until an
operator runs `operator init`: seal state, uptime, goroutines and a request
count that reads 0 rather than going blank. Everything below is empty until
then, and each panel's description says so rather than leaving an operator to
wonder which of the two it is looking at.
The seal panel reduces with `max()`. Once unsealed there are two
`vault_core_unsealed` series and the `cluster=""` one stays 0 forever, so a
bare selector reads SEALED on a healthy store — invisible while sealed, which
is why the metric names were sampled in both states.
The store's scrape target rendered as `host:port` alone, so the collector
requested `/metrics` — a path openbao does not serve on any listener. Every
scrape since the target landed has 404'd, which reads as a dead exporter
rather than a wrong address, and no bao sample has ever reached the store.
`scrapeTargets` values may now carry a path and query. Both are optional and
omitted when absent, so every existing target renders the config it rendered
before. The query cannot ride in `metrics_path`: prometheus percent-encodes
the `?`, so it has to become `params`.
Measured against openbao 2.6.2 and otelcol-contrib 0.151.0, the pinned
versions: `/metrics` 404s ("this listener only serves the metrics endpoint"),
`/v1/sys/metrics` answers JSON, and only `?format=prometheus` — or an Accept
header the collector happens to send today — returns exposition text. The
query param is the half that does not depend on content negotiation.
`bao operator init` fails at the seal with CKR_MECHANISM_INVALID. The
pkcs11 seal asks for AES-GCM by default and the TPM does not offer it: a
TPM 2.0's symmetric modes are CBC/CFB/CTR/OFB/ECB, and openbao accepts
only AEAD mechanisms — AES-GCM or RSA-OAEP — so RSA-OAEP is the single
mechanism both sides implement.
Measured on the deployed token (`pkcs11-tool --list-mechanisms`, #3860):
no AES-GCM, and `RSA-PKCS-OAEP, keySize={1024,2048}, hw, encrypt,
decrypt` present. CBC is not a fallback — openbao's
`MechanismFromString` rejects `CKM_AES_CBC_PAD` as deprecated and its
encrypt path implements exactly the two AEAD mechanisms.
The key gets a new label so a store provisioned by the earlier module
keeps its unusable AES key without the two resolving to one label, and
the addkey step is now keyed on the label rather than on the store not
existing — otherwise an existing deployment never gains the RSA key.
#3880 bound /dev/tpmrm0 into the store's container, which was necessary and
not sufficient: a bind mount preserves the host's ownership, openbao runs as a
DynamicUser with an empty CapabilityBoundingSet (so no CAP_DAC_OVERRIDE), and
the host applies no tpm udev rule at all — measured, 0 hits across all 41 host
closures against 62 rule files as a control. The node therefore keeps the
kernel default and the seal cannot open it.
A name cannot fix this. NixOS allocates system groups at activation, per
machine, so `tss` — or any group declared on both sides — gets two different
ids, and the device node carries the number. mara picked pinning a gid with an
overridable default (deploy.bao.tpmGid).
The default sits above the range NixOS auto-assigns system groups from
(400-999, measured in update-users-groups.pl) and above the normal-user range,
and below systemd's DynamicUser range (61184-65519), so it collides with
nothing any of those allocate.
The module-eval case compares the two sides rather than checking each against
a literal: the property is that they AGREE, not what they agree on. Its
absence arm is a shamir store, which never opens a TPM and must not claim a
device node's group — without it, pinning unconditionally would look identical.
openbao runs as a DynamicUser, so the uid that has to open the token
store is allocated by the container's PID 1 and cannot be named by a
host unit — the store was created root-owned 0700 and the seal could
never read it. The provisioning unit moves inside the container and
hands the sqlite store over by group; the host keeps only the mkdir the
bind mount needs, create-only so a reboot does not re-impose 0700.
Two further layers blocked the same start, both measured while fixing
this one:
- DynamicUser implies ProtectSystem=strict (systemd.exec(5)), so the
bind mount was read-only to openbao however it was owned, and the
pkcs11 library opens its store read-write. ReadWritePaths= is
required and was absent.
- allowedDevices renders DeviceAllow= and nothing else, and nspawn
builds its own /dev as a fixed tmpfs and cannot create device nodes
— verified against a live container, whose /dev holds no host
devices at all. /dev/tpmrm0 was therefore absent inside swarm-bao,
not merely unpermitted. It is now bound in.
Whether openbao's dynamic uid may *open* that node is a third question:
the tss gid is dynamically allocated, so no name or number means the
same thing on both sides of the boundary. Filed separately rather than
guessed at here.
The two module-eval cases that asserted the unit on the host now assert
it in the container and absent from the host, and two new cases pin the
device bind and the write access — each was individually valid and
collectively required, which is the state no assertion catches. Both
new cases select with `or [ ]`: mutation-testing them showed that a bare
select aborts the run with a nix trace instead of failing the case by
name, which also hid the second failure behind the first.
The store declared no journald units and served no metrics: nothing in
`swarm-bao.nix` mentioned either, while every sibling swarm service
declares both.
Metrics get their own loopback listener rather than a flag on the API
one, and that follows from what a scraper can express rather than from
taste: `swarm.otel.scrapeTargets` is `host:port`, plaintext and with no
credential, while the API listener is TLS and demands a client
certificate once a client CA is set. `metrics_only` narrows the new
listener to the metrics path; `prometheus_retention_time` is what serves
the endpoint at all.
Measured against openbao 2.6.2 before writing any of it: the metrics
path answers 200 on such a listener *while the node is sealed and
uninitialised*, 503 on the API listener, and 404 for a non-metrics path
on the metrics listener.
The listener exists only where a collector does — it is unauthenticated
by design for now, and an endpoint with no reader would be exposure
bought for nothing.
The port cannot be the API port + 1: openbao derives every listener's
cluster address as its own port plus one, so that number is already
taken. An assertion says so, since the failure is otherwise a race with
no log line.
Journald units are declared by the module that defines each unit, not
gathered here, matching the option's own rule — a name nothing defines
is silently ignored, so a central list would read as coverage on hives
that have neither glue module.
Refs #3849
openbao has never started on a hive that deploys it. With raft storage
and no `cluster_addr`, `bao server` exits 1 before serving anything:
cluster address must be set when using raft storage
Reproduced against openbao 2.6.2 with the module's own config shape:
the same JSON minus `cluster_addr` exits 1 with that line, and with it
the server comes up and configures every listener.
Both addresses are built from `swarm.bao.domain` rather than a bind
address — that is the URL a reader already dials (`BAO_ADDR` in
glue-matrix-bao-token) and the name the server certificate carries.
Cluster traffic sits one port up, upstream's convention.
The `unknown or unsupported field loopback` warning in the same startup
log is unrelated and cosmetic: openbao's unknown-field check does not
know about named listener blocks, but the parser honours `type` and
configures each one (measured, two named listeners, both served).
Noted in the module so the next reader does not chase it; the JSON
array form that avoids the warning does not typecheck against nixpkgs'
`settings.listener` (`attrsOf`).
module-eval gains the regression gate plus a control that the settings
it reads vary per deployment.
Refs #3860
