Per mara's go-ahead on hyperhive#3902 ("getting started is good, but
terminal rendering does not go in there i think"):
Moved 21 top-level docs/*.md files into 7 new topic subdirectories
(existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched):
getting-started/ setup.md
agent-lifecycle/ agent-hierarchy.md, approvals.md, persistence.md
trust-boundary/ boundary.md, security.md
integrations/ forge.md, matrix.md, github.md, knowledge.md
networking/ gateway.md, network.md, snapshot-store.md
scheduler/ jobq.md, coordinator.md, ci.md, observability.md
process/ conventions.md, gotchas.md, pr-review-gate.md
web-ui/ terminal-rendering.md (moved into the EXISTING dir,
per mara's correction to the original getting-started
guess -- it's UI implementation detail, not onboarding)
The physical layout now matches docs/README.md's own topical headers,
which already amounted to this taxonomy -- see the scoping comment on
the issue for the two findings that motivated this (a genuine
duplication between CLAUDE.md's old "Reading paths" list and
docs/README.md's grouped one, since drifted out of sync with each
other; and the flat layout not matching the grouping we already had).
Fixed every cross-reference this moved across the whole repo (~120
files: docs/ internal links at every depth, Rust doc comments, nix
module option docs, crate READMEs) -- verified two ways: a grep sweep
confirming zero remaining references to any old path, and a script
that resolves every markdown link in docs/**/*.md + CLAUDE.md +
README.md against the filesystem and reports anything that doesn't
exist (zero broken links).
Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to
a pointer at docs/README.md, now the single index. Rewrote
docs/README.md itself to use the new subdirectory paths and added the
one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md).
Classified all 22 docs/*.md files first via a haiku subagent (mara's
suggestion) on two axes -- proposed grouping and operator-vs-
implementation focus -- before finalizing the taxonomy; spot-checked
the report and found internal inconsistencies (its classification
table disagreed with its own summary section for a few files), so this
taxonomy is my original proposal + the one correction mara gave
directly, not a blind application of the subagent's table. The
operator-focus data it gathered is still useful for a follow-up
content pass (docs skewing 'mixed' rather than pure operator-facing),
not addressed in this PR -- structure only.
nix fmt clean, both pre-push lints clean.
`docs/swarm/sso.md` already argues that the OIDC provider is derived from
the client list rather than carrying its own `enable` — authelia refuses to
start without clients, so a separate flag would be a second fact free to
disagree. The module header said it again, and the definition site said it a
third time.
The doc keeps the argument. The definition site keeps a short version,
because someone changing `oidcEnabled` needs the reason there and not two
files away, along with the part no doc carries: that it is unconditionally
true today and stays derived so the OIDC-gated code below is self-documenting
about why it is conditional.
The header also restated the users-store rationale the doc's "does not
provision users" bullet owns. What survives there is the one clause the doc
does not have — why a file backend is right rather than a placeholder for
LDAP, the subject set being bounded by one swarm.
503 comment lines to 495. Comment-only: still 36 module-eval properties.
Two named anti-patterns from the comment-budget issue, applied to the file
that carries the most comment lines in nix/host-modules.
The header re-explained the two-tier model and the co-location argument that
`docs/observability.md` already owns — "a boundary that disappears locally is
one the local deployment stops testing" appears in both, nearly word for word.
The doc is the right home for it; what stays here is the one fact the doc does
not carry, that this is a container because `services.opentelemetry-collector`
is a singleton option already taken by the hive tier.
The `logs_endpoint` block keeps every instruction and loses the narration:
use `logs_endpoint` not `endpoint`, both spellings pass `otelcol validate`
and every wrong path answers 400, `_msg_field` is required or text search
finds nothing while ingest still answers 200, `_stream_fields` sets stream
cardinality.
Six smaller trims of "measured"/"verified"/"once". One pointed at a probe
script under a single agent's state directory — a path no other reader can
open, in source every reader loads.
What stays: `validateConfigFile`'s "a parser, not a wiring check", the
`issuer_ca_path` warning, the swarm-tier stamp. Non-obvious, in no doc, and
the target is a share of the whole tree rather than a per-file quota.
604 comment lines to 574. Comment-only: the module still evaluates to the
same 36 module-eval properties.
The collector's store exporters were gated on deploy.victoriametrics.enable /
deploy.victorialogs.enable — options that say "this host RUNS the store", not
"the swarm has one". A collector that did not share a host with the stores
rendered no exporter at all and dropped everything it received, from every
hive, silently: an absent exporter is not an error.
mara ruled the premise rather than the mechanism ("the swarm always has a
store"), so there is no gate and no new option for that. Both exporters are
unconditional and address the stores by domain, per the cross-host rule from
the OIDC client and secret-delivery unit #3517 already built. The logs exporter
had the identical bug and is fixed here too.
Both stores gained a machine ingest route, required in the same change: the
exporter now targets https://logs.<domain>/insert/..., and that vhost is
browser-shaped, so shipping the collector half alone would have regressed logs
ingestion that works today. Neither ingest location carries
`error_page 401 =302` — a pusher handed a redirect follows it and POSTs at a
login page, which answers 200.
Whether the collector authenticates follows the CREDENTIAL, never another
service's placement: `clientSecretFile` is a nullable option, and the delivery
unit — the one thing here that may know where authelia runs, since it copies
out of its container — sets it by mkDefault. An earlier revision gated this on
deploy.authelia.enable directly, which put a different service's co-location in
the collector's own config.
Also removed rather than relaxed: the assertion that this collector has
"somewhere to send". It read the store's per-host enable, so it rejected at
eval exactly the deployment reaching the stores by domain exists for.
Deliberately not replaced with an authentication assertion — a collector on a
host of its own is a supported shape, and refusing to build it would make this
fix illegal where the bug bites hardest.
Knock-on worth review: collectLogs is now always satisfied, so journald
collection is unconditional.
Config shape validated against otelcol-contrib 0.151.0 `validate`, with a
bogus-key control confirming the validator checks the extension schema.
module-eval: 31 properties.
Nine panels against the metric names openbao actually emits — the prefix is
`vault_`, the fork kept upstream's names, and a board built on `openbao_`
would render perfectly and show nothing.
The top row works on a SEALED store, which is what a hive has until an
operator runs `operator init`: seal state, uptime, goroutines and a request
count that reads 0 rather than going blank. Everything below is empty until
then, and each panel's description says so rather than leaving an operator to
wonder which of the two it is looking at.
The seal panel reduces with `max()`. Once unsealed there are two
`vault_core_unsealed` series and the `cluster=""` one stays 0 forever, so a
bare selector reads SEALED on a healthy store — invisible while sealed, which
is why the metric names were sampled in both states.
The store's scrape target rendered as `host:port` alone, so the collector
requested `/metrics` — a path openbao does not serve on any listener. Every
scrape since the target landed has 404'd, which reads as a dead exporter
rather than a wrong address, and no bao sample has ever reached the store.
`scrapeTargets` values may now carry a path and query. Both are optional and
omitted when absent, so every existing target renders the config it rendered
before. The query cannot ride in `metrics_path`: prometheus percent-encodes
the `?`, so it has to become `params`.
Measured against openbao 2.6.2 and otelcol-contrib 0.151.0, the pinned
versions: `/metrics` 404s ("this listener only serves the metrics endpoint"),
`/v1/sys/metrics` answers JSON, and only `?format=prometheus` — or an Accept
header the collector happens to send today — returns exposition text. The
query param is the half that does not depend on content negotiation.
`bao operator init` fails at the seal with CKR_MECHANISM_INVALID. The
pkcs11 seal asks for AES-GCM by default and the TPM does not offer it: a
TPM 2.0's symmetric modes are CBC/CFB/CTR/OFB/ECB, and openbao accepts
only AEAD mechanisms — AES-GCM or RSA-OAEP — so RSA-OAEP is the single
mechanism both sides implement.
Measured on the deployed token (`pkcs11-tool --list-mechanisms`, #3860):
no AES-GCM, and `RSA-PKCS-OAEP, keySize={1024,2048}, hw, encrypt,
decrypt` present. CBC is not a fallback — openbao's
`MechanismFromString` rejects `CKM_AES_CBC_PAD` as deprecated and its
encrypt path implements exactly the two AEAD mechanisms.
The key gets a new label so a store provisioned by the earlier module
keeps its unusable AES key without the two resolving to one label, and
the addkey step is now keyed on the label rather than on the store not
existing — otherwise an existing deployment never gains the RSA key.
#3880 bound /dev/tpmrm0 into the store's container, which was necessary and
not sufficient: a bind mount preserves the host's ownership, openbao runs as a
DynamicUser with an empty CapabilityBoundingSet (so no CAP_DAC_OVERRIDE), and
the host applies no tpm udev rule at all — measured, 0 hits across all 41 host
closures against 62 rule files as a control. The node therefore keeps the
kernel default and the seal cannot open it.
A name cannot fix this. NixOS allocates system groups at activation, per
machine, so `tss` — or any group declared on both sides — gets two different
ids, and the device node carries the number. mara picked pinning a gid with an
overridable default (deploy.bao.tpmGid).
The default sits above the range NixOS auto-assigns system groups from
(400-999, measured in update-users-groups.pl) and above the normal-user range,
and below systemd's DynamicUser range (61184-65519), so it collides with
nothing any of those allocate.
The module-eval case compares the two sides rather than checking each against
a literal: the property is that they AGREE, not what they agree on. Its
absence arm is a shamir store, which never opens a TPM and must not claim a
device node's group — without it, pinning unconditionally would look identical.
openbao runs as a DynamicUser, so the uid that has to open the token
store is allocated by the container's PID 1 and cannot be named by a
host unit — the store was created root-owned 0700 and the seal could
never read it. The provisioning unit moves inside the container and
hands the sqlite store over by group; the host keeps only the mkdir the
bind mount needs, create-only so a reboot does not re-impose 0700.
Two further layers blocked the same start, both measured while fixing
this one:
- DynamicUser implies ProtectSystem=strict (systemd.exec(5)), so the
bind mount was read-only to openbao however it was owned, and the
pkcs11 library opens its store read-write. ReadWritePaths= is
required and was absent.
- allowedDevices renders DeviceAllow= and nothing else, and nspawn
builds its own /dev as a fixed tmpfs and cannot create device nodes
— verified against a live container, whose /dev holds no host
devices at all. /dev/tpmrm0 was therefore absent inside swarm-bao,
not merely unpermitted. It is now bound in.
Whether openbao's dynamic uid may *open* that node is a third question:
the tss gid is dynamically allocated, so no name or number means the
same thing on both sides of the boundary. Filed separately rather than
guessed at here.
The two module-eval cases that asserted the unit on the host now assert
it in the container and absent from the host, and two new cases pin the
device bind and the write access — each was individually valid and
collectively required, which is the state no assertion catches. Both
new cases select with `or [ ]`: mutation-testing them showed that a bare
select aborts the run with a nix trace instead of failing the case by
name, which also hid the second failure behind the first.
The store declared no journald units and served no metrics: nothing in
`swarm-bao.nix` mentioned either, while every sibling swarm service
declares both.
Metrics get their own loopback listener rather than a flag on the API
one, and that follows from what a scraper can express rather than from
taste: `swarm.otel.scrapeTargets` is `host:port`, plaintext and with no
credential, while the API listener is TLS and demands a client
certificate once a client CA is set. `metrics_only` narrows the new
listener to the metrics path; `prometheus_retention_time` is what serves
the endpoint at all.
Measured against openbao 2.6.2 before writing any of it: the metrics
path answers 200 on such a listener *while the node is sealed and
uninitialised*, 503 on the API listener, and 404 for a non-metrics path
on the metrics listener.
The listener exists only where a collector does — it is unauthenticated
by design for now, and an endpoint with no reader would be exposure
bought for nothing.
The port cannot be the API port + 1: openbao derives every listener's
cluster address as its own port plus one, so that number is already
taken. An assertion says so, since the failure is otherwise a race with
no log line.
Journald units are declared by the module that defines each unit, not
gathered here, matching the option's own rule — a name nothing defines
is silently ignored, so a central list would read as coverage on hives
that have neither glue module.
Refs #3849
openbao has never started on a hive that deploys it. With raft storage
and no `cluster_addr`, `bao server` exits 1 before serving anything:
cluster address must be set when using raft storage
Reproduced against openbao 2.6.2 with the module's own config shape:
the same JSON minus `cluster_addr` exits 1 with that line, and with it
the server comes up and configures every listener.
Both addresses are built from `swarm.bao.domain` rather than a bind
address — that is the URL a reader already dials (`BAO_ADDR` in
glue-matrix-bao-token) and the name the server certificate carries.
Cluster traffic sits one port up, upstream's convention.
The `unknown or unsupported field loopback` warning in the same startup
log is unrelated and cosmetic: openbao's unknown-field check does not
know about named listener blocks, but the parser honours `type` and
configures each one (measured, two named listeners, both served).
Noted in the module so the next reader does not chase it; the JSON
array form that avoids the warning does not typecheck against nixpkgs'
`settings.listener` (`attrsOf`).
module-eval gains the regression gate plus a control that the settings
it reads vary per deployment.
Refs #3860
Nothing resolved `bao.<swarm.domain>`. It is absent from
`swarm.serviceDomains` and, unlike every other swarm service, contributed
no `gateway.localNames` entry — so the name a reader dials answered only
if an operator had published an external record, which nothing asks them
to do.
Cross-hive traffic goes via the domain either way; only what it resolves
to varies, and for a multi-host swarm that is the operator's upstream DNS.
This covers the deployment with no upstream record to configure.
DNS only, and that asymmetry is deliberate: bao stays out of
`serviceDomains` and gets no vhost. Its leaf comes from its own CA, so a
services-CA entry would mint one nothing uses, and nginx terminating TLS
would strip the client certificate the store authenticates every hive
with. `swarm-nats.nix` documents the same exclusion for itself.
module-eval gains the presence and absence arms: claiming a name this
host does not serve would point every local reader at the wrong machine.
Refs #3862
`swarm-bao.nix` declared the store's half of the mTLS pair as options —
`serverCertFile`, `serverKeyFile`, `clientCaFile` — and left the reader's
half as a literal inside `glue-bao-tls.nix`, which only runs where
`deploy.bao.enable` is set. A hive that did not host the store therefore
could not read from it and could not be pointed at a certificate even
when one had been placed by hand.
Adds `clientCertFile`, `clientKeyFile` and `serverCaFile` beside their
three server siblings, `mkDefault`ed by the glue to the leaf it already
mints, and moves `glue-matrix-bao-token.nix` onto them. Its gate becomes
"this host holds an identity" rather than "the store is a neighbour",
and the unit ordering that names store-local units is now conditional --
`Requires=` on an absent unit fails the job.
`serverCaFile` is separate from `clientCaFile` on purpose: one is the
store choosing which readers to trust, the other a reader choosing which
store to trust. Self-signing collapses them to one file, which is a
property of that deployment and not of the pairing.
Closes#3855.
The UI and the swarm-controller it fronts are one service to a reader and to a
certificate, so a separate option only ever created two spellings of one
address — and the controller's own public URL was built from one of them while
an allow-list entry named the other.
Removed rather than aliased. A config still setting it was pinning a name the
controller never used; an alias would preserve exactly that mismatch, so
`mkRemovedOptionModule` says so and fails the build.
⚠️ The option was TOTAL and `swarm.domain` is NULLABLE. That is not incidental:
`ui.domain` fell back to `swarm.invalid` so hive-network.nix's required-domain
assertion is what an operator sees, rather than a coercion error from a
consumer. Each site keeps that property its own way — swarm-ui binds a local
total (the idiom swarm-otel.nix already uses as `domainBase`), and the three
sites already gated on `deploy.swarm-ui.enable` gain `&& domain != null`, which
is stricter than a placeholder and puts no invented name into rendered config.
One behaviour change worth naming: `swarm.nix`'s certificate SAN list used to
receive the placeholder `swarm.invalid` on a null domain and now receives
nothing. `lib.optional` would otherwise have put a literal `null` in it.
Verified with `checks.module-eval` rather than `nix-instantiate --parse`: parse
proves syntax, and every failure this change could plausibly cause — a dead
binding, a renamed reference — is an evaluation error.
Her review: too much text, and an assertion helper instead of that much code
where the file should just say "this option cannot contain that".
`lib/name-guards.nix` gets `mustNotEqual` / `mustNotContain`; both call sites in
swarm-otel become four fields each — the option, the names, the list, and the
sentence explaining the failure. The message plumbing (find every offender,
quote them, name the list) is one shape shared by both.
Offenders are printed after their label rather than before it, because no
English verb agrees with both one name and five.
Converted the pre-existing equality assertion too. Leaving one hand-rolled
beside one helper-built is the worse of the two outcomes, and it is the same
guard.
Verified by evaluating both guards directly rather than only parsing: clean
rosters pass, `foo-agent`/`beehive` fail containment, `swarm` fails equality,
and `forgeworks`/`operator-hq` still pass — the control that keeps the two
lists from being merged.
Hive-scoped identifiers are built by joining a hive name with fixed words —
`hive-<name>`, `hive-<name>-agent` — so a hive called `foo-agent` produces
exactly the id hive `foo`'s agent containers produce. The queue's responder
resolves it as the agents, and that hive silently receives an agent grant
instead of its own; a NATS denial reaches a client as a timeout, so nothing
names the cause. Equality cannot see this: the two NAMES differ, only the
composed strings collide.
Deliberately a second file rather than three more entries in
`reserved-names.nix`. That list is matched by equality against a whole name and
every entry is a value some component emits; applying containment to it would
refuse `forgeworks` and `operator-hq` for failures that do not exist. The two
lists differ in subject (hive-only vs both), matcher (substring vs equality)
and admission rule, and each file's header says so.
No Rust change: `HIVE_RESERVED_NAMES` answers "may an AGENT be called this",
which stays an equality question — an agent name is never composed into these
identifiers.
Measured against the predicate rather than assumed:
pr1ma=ok alpha=ok forgeworks=ok operator-hq=ok
foo-agent=REFUSED beehive=REFUSED swarm=REFUSED myswarmx=REFUSED
The four `ok` cases are the control: `forgeworks` and `operator-hq` are exactly
what a merged list would have refused.
27 lines to 11. What went: the measured deny line (it is in the PR and the
commit that made the change), the "one value, two readers" argument for reading
the option the target URL is built from, and a note about `unique`.
What stayed is the part nobody can re-derive from the code: `external` must not
be dropped, both hyperhive hosts are private addresses the builtin denies, and
the entry is deliberately not gated on the per-host `deploy.swarm-ui.enable`
because a spread deployment has the forge and the swarm host on different
machines.
Her review: don't declare a `let` for a single use. The comment that justified
the binding moves to the use site and drops the claim it was carrying — that
`swarm.ui.domain` and `swarm.domain` can diverge — keeping only the part that
holds either way: this reads the same option swarm-controller builds its public
URL from, so the allow-list entry and the target it must match are one value
rather than two spellings.
`swarm.domain` is not the value swarm-controller addresses this hive with —
`swarm-controller.nix` builds `SWARM_CONTROLLER_PUBLIC_URL` from
`swarm.ui.domain`, which merely defaults to it and is documented as
overridable. Allowing the one and being delivered the other would work until
someone takes that option up, then fail as a refused delivery with nothing
naming the cause.
Also states why the entry is NOT gated on `deploy.swarm-ui.enable`, which is
what makes the controller register the hooks at all: in a spread deployment the
forge and the swarm UI are different hosts, so that per-host flag is false
exactly where the allow-list entry is needed.
The comment-block lint caught the prose at 35 lines; trimmed to the parts a
reader cannot re-derive.
`webhook.ALLOWED_HOST_LIST` named `external` plus the HIVE domain. The
swarm-controller's webhooks target the SWARM domain, which is a different host
on the same private gateway IP — so `external` denied it and the hive entry did
not cover it. Every swarm-controller delivery has been failing there.
Found from the sender's log, which names the host and the rule it broke:
deny 'constellation.darkest.space(10.42.0.1:443)'
webhook can only call allowed HTTP servers (check your
webhook.ALLOWED_HOST_LIST setting)
This is the layer under the DNS fix: resolution now succeeds and reaches
10.42.0.1:443, and forgejo refuses to dial it. The two failures look identical
from the receiving end — no delivery, no log line — which is why reading the
sender was what separated them.
`optional` because a hive with no swarm has no such domain; `unique` because an
all-local deployment can legitimately set both to the same string.
mara (PR#3820): the operator doesn't type /matrix/ into the browser — it's a swarm-level link (hive-matrix.nix registers it into services.hyperhive.swarm.controller.links, rendered by swarm-ui's LinksMenu). Corrected the M4TR1X page doc, and fixed the hive-matrix.nix comment that still described the removed H0M3 tile in present tense while I was in there.
The forge container used the host's resolvers, where the swarm domain has
no records — so every outbound call to a swarm name failed with "no such
host". Forgejo's webhook deliveries died there: knowledge, config-pr and
vcs-activity alike, which is why the swarm-controller's receiver has never
logged a verified delivery and knowledge propagation has been running on
its hourly fallback.
An earlier fix taught the container authelia's name with a `networking.hosts`
entry. That covered one name and left the rest, so use the resolver every
other service container already uses and drop the override — hive-matrix
records why a second answer that can disagree with the first is worse than
none.
Closes#3824
The dashboard's log surface is an allow-list: the four harness units in
every agent container, the hive-ci / hive-forge / hive-matrix machines,
and three host units. `swarm.otel.journaldUnits` only named the host
three plus forgejo — so the units an operator can read in the UI were
absent from the store they are supposed to be queryable in, and nothing
reports that, because a name that matches nothing is not an error.
Adds the four agent units, the CI runner and the homeserver. The agent
units are declared by the hive-c0re module rather than by the
agent-modules that define them: those are a guest evaluation and cannot
reach a host option.
Closes#3818
The state directory was bind-mounted from the host so a nixos-container
destroy could not take the swarm's secrets with it. No sibling service does
that -- swarm-grafana keeps its sqlite database inside the container on
ephemeral = false -- and the bind is what broke the store: upstream pairs
StateDirectory= with DynamicUser=, systemd relocates the state to
/var/lib/private/openbao, and that rename fails EBUSY on an active mount
point, so the unit died at STATE_DIRECTORY before bao ever ran.
The TLS material still has to cross the boundary, because a host unit writes
it and the container reads it, so it moves to its own small bind at
/var/lib/swarm-bao-tls rather than riding along in the state directory. That
directory is 0755 and read-only inside: the certificate and client CA are
public and are read straight off the mount.
The private key is not. install -m 0600 leaves it root-owned and the service
runs as a DynamicUser, so the bind-mounted file is unreadable to it -- which
the old layout hid, because StateDirectory chowned the whole tree on the way
past. LoadCredential is systemd's mechanism for precisely this: PID 1 opens
the source as root and re-exposes it inside the unit owned by the service's
own account.
`services.hyperhive.tls.{stateDir,caValidityDays,leafValidityDays}` sat at
the top of `services.hyperhive`, which is meant to be everything about
hyperhive rather than the settings of one hive. Where the hive CA lives,
how long it lasts and how long the leaves it signs last are decisions of
the host holding the key — `deploy.*`, by the same rule as the switches
that moved before them.
`hive-controller` is hive-c0re's new name (mara on the issue), so the
knobs hang off the daemon that owns the CA rather than off a bare `tls`
at the root. mkRenamedOptionModule entries carry existing configs.
⚠️ Unlike the two switch renames, these names are NOT unique, so this was
swept by ALIAS BINDING rather than by identifier: hive-tls.nix alone holds
two options spelled `stateDir` — its own `cfg.stateDir` and the swarm CA's
`swarmCaCfg.stateDir`, four sites that must not move. Nine files bind an
alias to this config; the rename followed those bindings.
Two sites were invisible to the obvious check, and an unanchored sweep for
`hyperhive\.tls\b` is what found them: the option declaration (`= {` after
the path, so no trailing `.` or `;`) and the alias convention documented in
a comment in lib/hive-ca-trust.nix.
Also renamed the `<tls.stateDir>` shorthand in four docs and two Rust doc
comments, anchored on its delimiters — the new path contains the old one
as a substring, so an unanchored replace would have doubled the prefix.
mara: "deploy.nix still says it everywhere". Seven option descriptions
each told the reader that this service derives from the swarm-services
switch — the same duplication just removed from docs/swarm/services.md,
one layer over, and rendered into the generated options doc.
Each keeps only what is true of that service and nothing else: the
metrics store is paired with grafana, the log store has clients rather
than second instances, the homeserver can be placed on its own host, a
swarm has one SSO provider. The store's paragraph about where it runs
being a separate question from that it runs survives, because that is
about the store, not about the switch.
swarm-ui keeps its positive derivation (the controller) and loses only
the "rather than from allSwarmServices" half.
Deliberately not touched, as a different question rather than an
oversight: the *mode* option's description in local-defaults.nix, whose
whole job is to name what it asserts; the module comments in
swarm-authelia.nix / swarm.nix, which are read by someone editing that
module rather than by an operator picking an option; and ui.md's
contrast, which she has not ruled on.
Same defect as the switch below it, one tier up: it sat at the TOP of
`services.hyperhive`, a namespace that is meant to be everything about
hyperhive rather than the settings of a single hive. Whether this box is
the whole deployment is as per-host as a decision gets.
The name follows mara's sentence for what it means — "everything in the
swarm is running on this host" — rather than naming its mechanism.
"Defaults" was doing no work: it is not a defaults toggle, it is a claim
about where the swarm lives, and the pair now reads as the containment it
already was, singleHostSwarm implying allSwarmServices plus this hive.
One site was a setter rather than a reference: module-eval's `allLocal`
fixture passes an attrset merged into `services.hyperhive`, so its key
carries the path and had to become `deploy.singleHostSwarm`. A rename by
bare identifier is right for the twelve prose mentions and wrong for
exactly this one, which is worth knowing before the next rename.
Both halves of the old name were wrong about the subject. The services
are required of the SWARM, not of the host, and the option says whether
THIS host runs them — so it described the wrong thing and sat in the
namespace that has to be identical on every host. The new name is mara's
own phrasing of what it means: "deploy all swarm level services on this
host".
mkRenamedOptionModule carries existing configs, read-side references
included, so this warns rather than failing to evaluate.
Three sites were not just the identifier:
- local-defaults.nix set it inside `config.services.hyperhive.swarm =
{ … }`. It moves out as a path beside the other deploy.* setter rather
than into a second `deploy = { … }` attrset — the warning that file
already carries about `swarm` applies to any second definition of the
same parent.
- swarm-required-services.nix bound only `swarmCfg`, now unused; it binds
and reads `deployCfg`.
- Two comments in that file described a half-migrated state, where the
switch asserted some `swarm.*.enable` toggles and some `deploy.*` ones.
Every one of them has been `deploy.*` for several slices now.
hive-metric counters are delta by default; VictoriaMetrics has no
delta-to-cumulative processor of its own and official guidance is to
feed it cumulative. Converting at the hive tier, closest to the
origin, means every downstream consumer (swarm tier, VictoriaMetrics,
upstream) sees ordinary cumulative counters and existing PromQL idioms
(rate()/increase()) keep working unchanged. Default config is a no-op
passthrough for anything already cumulative (Claude Code's own export,
hostmetrics, Prometheus scrapes) -- verified against the actual
collector binary (otelcol-contrib validate), not assumed: an empty
config validates clean, an invalid key is rejected.
The comment said it bounds the wait rather than hanging the boot, and
nothing in the unit bounded anything. What actually held was systemd's
default start timeout plus the homeserver only Wants= this unit — both
true, neither stated here, so a reader had to already know them.
TimeoutStartSec puts the number a boot waits on in the file that waits,
and the comment now names the degradation: hitting it keeps the locally
minted token, same as every other failure path in this unit.
Caught by argus reviewing the slice it shipped in.
Three places said nothing supplies them — the module header, the
serverCertFile and clientCaFile descriptions, and the assertion message
an operator actually hits. All three were true when written and
falsified by glue-bao-tls.nix landing in the same PR, which mkDefaults
all three paths.
The load-bearing half is unchanged: the store never reaches for an
authority, because it must not take its identity from one it will itself
distribute. What was wrong is the claim that therefore nothing fills them
in. The service module declares no default and does not know where the
value comes from; the glue supplies one where the store is deployed; an
operator's own path beats it.
The assertion also says what reaching it now means — the glue is absent,
or something set the paths back to null.
Fetches an opaque 32-byte value and writes it where hive-matrix.nix
already looks, so the homeserver never learns the store exists.
Chosen over authelia's OIDC secret deliberately: that one needs a
.secret AND a matching .digest, so shipping it first would debug 'can a
reader authenticate and get bytes back' and 'did we write authelia's
file format right' at once, with an SSO outage as the failure mode. Here
the failure is narrow -- new agent accounts cannot be provisioned,
existing ones untouched.
Every failure path keeps the local token: no such key, sealed store,
unreachable store, empty value. The activation script's mint-if-absent
is untouched, so a hive with no store behaves exactly as it does today.
matrixMachine is a literal because hive-matrix.nix declares no `machine`
option -- `matrixCfg.machine` parses cleanly and fails at module-system
resolution, which is the kind of error only reading the target module
catches.
A CA that signs exactly two things -- the store's server certificate and
the client certificate of whoever reads from it -- and distributes
nothing. Not the hive CA, not the swarm CA: the store will distribute
both, and an authority you must already hold a certificate from cannot
be one the store hands out. Not the gateway's HTTPS material either,
self-signed or ACME; that is a different trust domain with a different
audience.
The minting lives here rather than in swarm-bao.nix because it is an
opinion about where the store's identity comes from -- the most
consequential one available. The service serves what it is handed. A
deployment with a real internal CA drops this file and names its own
paths in serverCertFile / clientCaFile, and nothing in the store
changes. Ordering simplifies too: with the unit and its consumer in one
module, before/requiredBy is internal rather than a cross-module fact.
Idempotent on ABSENCE only. Re-issuing the CA invalidates every client
certificate already trusting it, so a rebuild that refreshed it would
lock out every reader in the swarm at once.
OpenBao in a swarm-bao nixos-container, unsealed by the host TPM through
the built-in pkcs11 seal, with shamir as the documented opt-out for
hardware that has no TPM. The seal guard asks the package whether it was
built with an HSM rather than reading its version, and falls back to the
version cliff only when a package advertises no tags -- it fails closed.
Every listener serves TLS: loopback unconditionally, because the host
running the store is always one of its readers, plus whatever
extraListenAddresses names, because which network the other hives share
is a deployment fact. Client-certificate verification turns on only when
clientCaFile says what to trust.
The store's identity is an input, and nothing in THIS module fills it
in. Service-to-store mTLS is a separate trust domain from the gateway's
HTTPS certificates and from both CAs in this tree: a store must not take
its identity from an authority it will itself distribute, or it cannot
come up before the thing that issues it. The module asserts when the
paths are unset rather than reaching for whichever CA happens to be
wired; what supplies them on a self-contained deploy is the next commit.
The leaf is copied to the container's state dir by a host unit rather
than bind-mounted: nixos-container refuses to start when a bind source
is missing, and a directory holding a leaf usually holds the CA's key
beside it.
A swarm has one store, so running it is a fact about this machine and
belongs beside the other deploy.* toggles rather than in the namespace
every host agrees on. Attrset with an enable, not a bare bool, so a
second deployment decision has somewhere to go.
Asserted from swarm.enableRequiredServices alongside the other
once-per-swarm services, and asserted as a mkDefault -- which is what
keeps *where* the store runs a separate question from *that* it runs.
Set it directly to put the store on a host of its own; a hive that does
not run it is a client either way, reaching it by name.
Two descriptions in the file still narrated the old swarm.* layout as
something that used to be true, and the queue's claimed it belongs on
the shared-services host. Both now say what holds.
Both retentionPeriods move to deploy.<store>.retentionPeriod. Measured
before moving: each is read exactly once, in the container definition of
the module that deploys the store, and no client hive consults either --
so they are options on the auto-deployed service rather than something
every host has to agree on.
What stays in swarm.* is what a client needs to reach the store: the
package, the name it answers on, the port. Both modules said "what stays
here is its package, domain, retention and wiring" in their options-block
comments; retention no longer does, so those say so.
Renames go in deploy.nix with the rest, so the whole move keeps one home
and one file to delete when the deprecation window closes.
Last of the swarm.* -> deploy.* moves for the enable-shaped options.
Whether this host also runs the CI runner is a per-machine decision, so
it was sitting in the namespace that has to be identical swarm-wide.
Renames all five options, not just enable: name, concurrency, labels and
package all describe the runner THIS host would run, and leaving them
behind would keep the original defect for four more options. One rename
entry each, because ci is a plain attrset rather than a submodule type,
so there is no parent path to rename in one go.
deploy.forgejo is the name deploy.nix's own header already specifies,
including this exact case -- "ci (does this host run the runner too) is
exactly that shape". It is the only entry with no enable: the forge is
not optional, it deploys with hyperhive itself, so running the runner is
the only deployment decision it has.
Also fixes hive-c0re/src/forge/ci_runner.rs, whose doc comment named
services.hyperhive.forge.ci.enable -- missing the swarm. segment, so it
had never resolved to a real option.
HIVE_MATRIX_URL was conditioned on this host deploying the homeserver,
so a hive that is a client of the swarm's matrix forwarded nothing and
every agent's hive-matrix-daemon no-opped. hive-c0re already provisions
those agents' accounts against a remote homeserver through
HIVE_MATRIX_API_URL three lines below, whose gate is `apiUrl != null` —
it registered the users and then never told them where to log in.
gatewayHost is swarm-scoped and defaults to chat.<swarm-domain>, so it
names the swarm's one homeserver from any member hive. Its being set is
the real question, and a null swarm domain is already caught by the
assertion in hive-network.nix rather than by this expression.
swarm.matrix.enable was the last toggle still spelled swarm.*, which
is the namespace every host in the swarm agrees on - and "does this
host run the homeserver" is exactly what differs between them. It
moves to deploy.matrix.enable with the rest; the genuinely swarm-wide
matrix settings stay where they are.
mkRenamedOptionModule keeps existing configs evaluating with one
warning naming both paths.
The sweep for readers had to be unanchored and cross-language: three
of them were outside nix, including a hive-c0re error message telling
the operator to set the old name. A dotted grep also cannot see the
rename module's own list form, so that was checked separately.
Per review: `deploy.otel` does not imply swarm level, and there is a
hive-tier collector too -- `services.hyperhive.otel.enable`, which every
hive runs. The tier is the entire distinction between the two, so the
name has to carry it, matching `deploy.swarm-controller` and
`deploy.swarm-ui`.
15 swarm-tier references renamed across 8 files. The hive-tier
collector's 16 references are deliberately untouched, verified as a
control on the same command.
Three spellings needed three different patterns, all inside this one
rename: the dotted path; the `mkRenamedOptionModule` target written as a
nix LIST (`[ ... "deploy" "otel" "enable" ]`), which no dotted grep can
match; and prose in docs/observability.md spelled `deploy.otel` with
neither `.enable` nor a leading dot. Unanchored `deploy\.otel\b` is the
only pattern that finds all three.
Round-3 review catch: the all-local mode still set
`services.hyperhive.swarm.controller.enable` -- the exact path this
branch's own `mkRenamedOptionModule` deprecates. It forwards correctly,
so nothing broke, but every eval with `enableAllLocalDefaults = true`
printed a deprecation warning and the tree depended on the shim it is
retiring.
It survived the rewrite because the path is SPLIT ACROSS NESTING:
`swarm = { controller.enable = ...; }`. No grep for a dotted path can
match text that is not contiguous -- not the prefix-anchored pattern the
rewrite used, and not the suffix-anchored one added after that missed
hive-tls.nix. What finds it is the bare tail with no prefix at all
(`^\s*controller\.enable\s*=`), which reports exactly one hit outside
deploy.nix's declarations.
The assignment moves out of the `swarm` attrset rather than staying in
it: `deploy.*` is a different top-level path, so the collision the
neighbouring warning describes -- two definitions of `swarm`, the nested
one silently lost -- does not apply.
Two corrections from review, applied forward on this branch rather than
by rewriting it.
`deploy.<service>` was a bare bool, which makes
`deploy.forgejo = { enable; ci; }` unrepresentable -- the nested
CI-runner sub-option this namespace was designed around. Every entry is
now an attrset with an `enable`, so a second per-host deployment
decision becomes an ordinary addition rather than a migration.
`deploy.controller` is now `deploy.swarm-controller`, consistent with
`deploy.swarm-ui`, which was introduced in the same commit.
89 references rewritten across 24 files -- nix, Rust, docs, and the
repo's own CLAUDE.md.
The prefix-anchored sweep missed exactly one, and it was live code:
hive-tls.nix spells it `hyperhiveCfg.deploy.controller` -- the only
`hyperhiveCfg` prefix among 45 references. A suffix grep
(`\.deploy\.<name>`) finds it; a path-anchored one cannot, because the
head of a reference is whatever alias the reading file happens to bind.
The one move where the suffix grep is the wrong instrument. There are two
otel options a word apart — `swarm.otel.enable` (one per swarm) and
`otel.enable` (one per hive, every hive runs it) — so `\.otel\.enable`
matches twenty-five references of which most must not change. The module
already carries a comment warning about exactly this, on a line that
names `swarm.otel` in full rather than through a binding.
Triaged by hand and confirmed the other way round: after the move, a grep
for the per-hive option still finds it in the files that should keep it.
Also worth recording what the alias sweep cannot do. `swarm.nix` reads
this as `swarmCfg.otel.enable`, where `swarmCfg = cfg.swarm` and `cfg =
config.services.hyperhive` — an alias bound to an alias, two hops from
the option path. No syntactic sweep resolves that chain, which is why the
module system's own evaluation is the only complete check here and the
static sweeps are a way to narrow the work, not to finish it.
Same shape as the others. Three of the eight references were prose
rather than code: two assertion messages naming the option to an
operator, and a `defaultText` in the authelia module, whose
`oidc.hiveIdentities` derives from whether the queue runs here.
Also corrects a comment in swarm.nix that this commit's own hunk sits
directly beneath. It said `enableRequiredServices` turns on matrix and
authelia "but NOT nats (nats has no mode that enables it)", which stopped
being true when the queue was added to that switch — before this change,
not by it. The reasoning it introduces is still right and still worth
keeping; only the premise had expired.
The largest of these moves: sixteen references spelled through `let`
aliases across eight modules, plus eight more spelled as a path, plus
five documentation pages.
authelia is also the clearest case for why the two namespaces exist.
`swarm.authelia.url` is needed by *every* hive in the swarm — it says
where to send a browser to authenticate — while running the container is
the business of exactly one host. The client half and the server half
were sharing a namespace whose whole contract is "identical everywhere",
and only one of them could honour it.
`swarm.authelia.oidc.clients` stays where it is for the same reason:
several modules register a client there, gated on authelia running here,
and the registry itself is what the service *is* rather than a decision
about this machine.
One sweep note worth recording: a grep for `swarm.authelia.enable` misses
`swarmCfg.authelia.enable`, because the prefix is whatever the reading
file bound. Grepping the suffix `.authelia.enable` finds both, and found
a reference in swarm.nix that the path-shaped pattern did not.
One commit rather than two because they are not independent: the UI's
`enable` had the controller's as its literal default, so moving the
controller alone would leave the UI's default naming an option that no
longer exists.
The UI keeps that derivation in its new home — it is a view onto the
controller's state and reaches it over that daemon's unix socket, so the
host running the controller is the host that can serve it.
Three spellings had to move together for the UI, not one: the `default`,
the `defaultText` shown in the options doc, and the description prose
that names the old path in words. A grep for the option path finds the
first two.
The sweep also reached outside nix: `swarm-controller`'s crate README and
its `//!` module doc both named the option, as did this repo's own
CLAUDE.md and four pages under docs/. An option's name is API, and its
documentation lives wherever someone thought to write it down.
Same move as grafana, and the three belong together: they derive from one
switch and a store with no UI is as useless as a UI with no store.
`victorialogs` is the case that shows why the option-path sweep is not
enough on its own. It has **zero** references spelled
`swarm.victorialogs.enable` anywhere in the tree, and four spelled
through `let` aliases (`vlCfg.enable` in the collector, `cfg.enable` in
its own module). A sweep for the path would have reported nothing to do
and left every reader broken.
Prose moved with the code rather than being left behind: the comments in
swarm-required-services.nix that explained why the pair derives together
now sit above the assignments that do it, instead of above the gap where
they used to be.
`ssoLocal` in the grafana module read `cfg.enable`, where `cfg` is the
local binding for `services.hyperhive.swarm.grafana` — the option set the
previous commit removed `enable` from. Evaluation fails on the missing
attribute.
The sweep that missed it looked for the option *path*
(`grafana.enable`). This reference is spelled through a `let` alias, so
no grep for the path can find it, and a parse check cannot either: it is
syntactically valid and only wrong once the module system resolves the
attribute.
The check that does find it: for each file, extract the local bindings
that point at the option set being changed, then grep for
`<alias>.<removed-attr>`. Run against this tree it reports nothing for
grafana and sixteen references for authelia, which is the next module to
move.
`services.hyperhive.swarm.*` is meant to be identical on every host in a
swarm — it describes the swarm, and every hive needs all of it to be a
client. But it also carried the `enable` toggles, which are precisely the
values that must differ per machine. The namespace that should be the
same everywhere held the one thing that cannot be.
Adds `services.hyperhive.deploy.*` for a host's deployment decisions, and
moves the first of them (`swarm.grafana.enable` -> `deploy.grafana`) as
the pattern for the rest. Flat and named for the thing deployed rather
than grouped under a "swarm services" attribute: from the deploy side it
does not matter what kind of thing each one is, and a grouping by service
kind would re-encode the service-side taxonomy into a layer that does not
care about it.
Behaviour is unchanged. The move is a rename in the strict sense — same
type, same meaning, new path — so `mkRenamedOptionModule` carries it and
existing configs keep evaluating with one warning naming both paths. The
renames live in the new module rather than the service modules, so the
whole migration has a single home and a single file to delete when the
deprecation window closes.
harness-base.nix has never existed in this tree. Four comments named it,
or a `harness-base` module, as the place to look:
- weston-vnc.nix: the agent user is declared and home-chowned by
nix/agent-modules/user.nix
- hive-ci.nix: the sandbox-fallback reasoning lives in
nix/agent-modules/default.nix -- which the very next comment block in
the same file already cites correctly
- packages/default.nix: the per-bin consumer is
nix/agent-modules/packages.nix
- hive-sh4re/src/assets.rs: HIVE_ASSETS_DIR is set by
hive-c0re/environment.nix and agent-modules/default.nix +
agent-service.nix, and the package is built by nix/packages/assets.nix
-- not the equally nonexistent nix/assets.nix
assets.rs was twice declared out of scope on the sibling PR because it
names a module rather than a file. That distinction was real and
irrelevant: neither the module nor the nix/assets.nix path it points at
exists. Reading the wording is not checking the reference.
Every replacement path was verified to exist, with a deliberately bogus
path as a control.
