`credential.rs` calls `SecretStore::from_env`, and nothing set `BAO_*` for
this unit — only `swarm-bao-matrix-token` had them. Fixing that alone would
not have been enough: hive-c0re runs as hive-core, and glue-bao-tls mints the
client key `0600` inside a `0700` directory, so the daemon cannot read the
file even when it is named.
Both halves go through LoadCredential, which this unit already uses for the
swarm-status client secret: root reads the identity at unit start, hive-core
sees it under `%d`, and no second on-disk copy exists.
The gate is the identity, never `deploy.bao.enable` — a hive that reads a
store on another machine holds a certificate and runs no store. Four
module-eval cases: the co-located and off-host presence arms, the absence arm
for a hive with no identity, and a presence/absence pair for the optional CA.
The swarm collector reads /var/log/journal and has never seen a single
container unit. mara's count-by-unit against VictoriaLogs returns four
units, all host-tier; openbao -- which runs inside the swarm-bao
container -- is absent.
Cause: nixos-containers.nix hardcodes `--link-journal=try-guest` for
every non-ephemeral container. With `guest`, the host's
/var/log/journal/<machine-id> is a SYMLINK into the container's transient
root; a reader in the host namespace cannot follow it, and it dangles as
soon as the container stops. `ls -la /var/log/journal/` on the host shows
one real directory and a pile of `-> /tmp/nspawn-root-*` links dating
back to May.
swarm-otel.nix asserted the opposite, and that assertion is why the
receiver's path was considered sufficient: it said the files "live here"
and are "bind-mounted into the guest rather than the other way round".
That describes `--link-journal=host`. The same sentence names the flag we
actually use. The flag was right and the behaviour it described was not,
so grepping for the flag confirmed the comment and taught nothing.
`containers.<name>.extraFlags` feeds EXTRA_NSPAWN_FLAGS, which the
invocation expands after the hardcoded flag, so `--link-journal=host`
wins. The comment now describes what the code does instead of the other
way round.
Two payoffs, and the smaller one is the one the issue is about:
container logs become collectable, and -- independently -- they become
durable at all, rather than dying with the container.
Ten identical edits because ten host-modules hand-roll their own
container block; that duplication is #3773, not something to invent an
abstraction for here.
NOT VERIFIED: that systemd-nspawn honours the last `--link-journal` of
two. Everything else here is read out of nixpkgs; that step is a claim
about its argument parsing which cannot be exercised without starting a
container. It is settled by deploying one and re-running the `ls`: the
machine-id entry becomes a real directory instead of a symlink.
Refs #3849
Filtering the gateway's journal by severity returned nothing while nginx
was logging [error] continuously. Measured against nginx.service:
`-p warning` and `-p err` over 24h both said "No entries", while the
unfiltered query returned a continuous stream of [error] lines.
The failure is inverted rather than merely missing. A severity filter is
how you ask "is anything wrong", and here it answered "no" *because* the
errors were misfiled -- one step from recording "gateway clean, no
warnings in 24h".
Cause: the module default is `error_log stderr`, and systemd files that
whole stream at one priority regardless of what each line says. Routing
to syslog preserves per-line severity, so [error] lands as err and every
existing severity filter starts working without changing a consumer.
Uses the module's own `logError` option rather than appendConfig:
nixpkgs emits `error_log ${cfg.logError};` at exactly one site, in the
main context, so the option guarantees the placement. Transport only --
no level suffix, so nginx keeps its default `error` threshold and this
changes where lines are filed, not which lines exist.
Gated with `nginx -t` on the flake-locked nginx against the exact string
shipped here, with a deliberately invalid syslog target as the control
that the test can fail on this directive. gixy (what the module's own
validateConfigFile actually runs) reports no issues.
What that does NOT prove: that journald files the lines at the right
severity. `nginx -t` is a parse check; only a deploy shows the effect.
Pure `nix fmt` output from the commit before this one — no hand edits.
203 files: 52 md, 42 tsx, 32 js, 32 css, 21 ts, 13 html, 8 json, 3 mjs.
Reproduce with `nix develop -c nix fmt` on the parent commit; the result
should be byte-identical to this tree.
None of the 13 `.prettierignore` entries appears here — verified by
intersecting the changed-file list against the ignore file, with a
control proving the intersection finds a match when one exists.
`docs/` was reorganised into topic subdirectories and the references to it
were not moved with it. Thirteen distinct paths and three relative links no
longer resolved, spread across nix, css, html, js, markdown and
.prettierignore — a stale pointer is not a markdown problem, it is wherever
someone wrote a path down.
Each mapping resolved to exactly one target. `docs/matrix.md` was the sole
ambiguous basename: .prettierignore lists `docs/tools/matrix.md` separately
and that entry still resolves, so the stale one is the integrations doc.
The three relative links were each one `../` too deep — from `<crate>/src/`
two levels reach the repo root. `hive-agent/src/login.rs` already had the
correct form, in the same crate, at the same depth.
.prettierignore is repointed rather than dropped, though nothing in the tree
runs prettier: no treefmt entry, no CI job, no package script. Whether that
config should exist at all is a separate question from whether it names
files that do.
The collector bind-mounts /var/log/journal into its container and points
its journald receiver at that directory. journald only writes there when
it stores persistently: with storage = "volatile" the journal lives in
/run/log/journal, and with "none" there is none at all. Nothing in the
tree sets or asserts services.journald.storage, so a host with either
value deploys a collector that cannot start — nixos-container refuses to
start when a bind source is missing, which is the same trap the
swarm-bao-certs and hostClientSecretDir comments document.
A bindMounts entry never creates its hostPath, and unlike every other
runtime-written bind source in the tree there is no unit to order the
container after: each of those declares before + requiredBy on its
producing unit, and this directory has no producer. journald was told to
store persistently or it does not exist. That leaves an assertion as the
only place to say so.
"auto" is deliberately accepted: it uses the directory when it exists,
and evaluation cannot see whether it does. Certainty is what separates
an assertion from a warning, and only "volatile" and "none" are certain.
Verified by reading config.assertions for all three values rather than
system.build.toplevel, which raises the first failing assertion in the
whole config and so reports a minimal stub's unrelated ones instead:
persistent -> mine [] others 3
volatile -> mine [the message, quoting "volatile"] others 3
none -> mine [the message, quoting "none"] others 3
The others count is identical across the three, so the storage value is
the only thing that differs between them, and a populated list is what
makes the empty one mean "this assertion passed".
`docs/swarm/ca.md` said it under *Constraints on the material* and again
under *Distributing the root*; this module's header said it a third time,
directly above a line already pointing at that doc.
Measured: 6-gram intersection between the module's comments and the doc,
32 -> 6. The sentence now appears once in the tree.
Module header keeps what a reader editing this file cannot see from the
config: that one root makes cross-hive trust O(1), that the two
provisioning modes differ only in who writes the artifacts, and the ⚠️
that both root key and root cert are runtime files — the cert as a
consequence, and that one costs something real, since nothing whose trust
store is built at build time can name it. Everything behind those is a
`docs/x.md::Section` pointer with the heading checked to exist.
Dropped the O(n²)-per-peer-pinning story: the live reason is that one root
is O(1), and how it used to be configured is history.
In the doc, *Distributing the root* now links to *Constraints on the
material* rather than restating it. Intra-doc `#anchor` links are already
the convention there — 85 of them across `docs/`, 8 in `gateway.md` alone.
Comments + one doc paragraph; no behaviour change. Refs #3901.
The module header restated `docs/scheduler/ci.md`'s credential-isolation
paragraph nearly word for word, and re-told the nix-sandbox gotcha that
`docs/process/gotchas.md` owns — both immediately above lines that already
pointed at those docs.
Measured, not eyeballed: 6-gram intersection between this file's comments
and `docs/scheduler/ci.md`, 74 -> 24.
Kept, because neither is visible from the config that implements it and a
reader changing this file needs both:
* the container has its own network namespace, not the host's, so a PR's
nix build cannot reach host-loopback services — a prompt-injection
boundary rather than a preference;
* the forge ADMIN token never enters this container; only a registration
token is handed in.
Each is now one sentence naming the property and its consequence, with the
mechanism behind a `docs/x.md::Section` pointer whose heading was checked
to exist (`Container isolation`, `Container design`).
Also restated in the present: the container is non-ephemeral because the
runner writes its registered credentials on first registration and reuses
them, which is why an ephemeral one would re-register every restart. That
was previously a description of what gitea-actions-runner does, with the
reason left implicit.
Comments only; no behaviour change. Refs #3901.
Two comment blocks explained things `docs/networking/gateway.md` already
explains, one of them directly above a line pointing at that very doc. A
pointer under a duplicate is not a pointer; it is a duplicate with a
citation.
Measured, not eyeballed: 6-gram intersection between this file's comments
and that doc, 73 -> 31. The residual is mostly the one-line summaries kept
at the call sites, which necessarily share phrasing with the doc they
summarise — that is the floor of the metric, not remaining duplication.
* the self-signed-cert import unit: kept the 🚨 that stops someone
collapsing it into "point nginx at the CA dir", both jobs at one line
each, and the blast radius (whole gateway, not one vhost). Dropped the
`BIO_new_file()` mechanism and the production-outage narrative — the
doc carries both, and the second is history.
* the nginx-reload note: rewritten to say the one thing this file needs
to say, that nothing here reloads nginx and hive-c0re drives it from
the host through hive-priv. The per-unit-state dispatch table is the
doc's, and only the doc has it in full.
Both pointers use the `docs/x.md::Section` form and name headings verified
to exist; the first replaces a parenthetical `("Self-signed TLS")` spelling
that no check can resolve.
Comments only; no behaviour change. Refs #3901.
Comments cite nix modules, scripts and crate source files constantly,
and nothing evaluates a comment — so when a file moves, the reference
rots silently and `nix flake check` stays green. A reader following one
finds nothing and cannot tell whether the file was renamed, deleted, or
never existed.
Seven such references, each repointed at the file that actually holds
the thing the sentence is about rather than at the directory the old
name became:
hive-c0re/src/agent_config/limits.rs hive-agent/src/mcp.rs
-> hive-agent-mcp/src/mcp/mod.rs
hive-agent-mcp/src/mcp/mod.rs hive-c0re/src/limits.rs
-> hive-c0re/src/agent_config/limits.rs
(and the module path in the doc
comment above it, which was stale
in the same way)
hive-c0re/src/forge/mod.rs hive-c0re/src/knowledge.rs
-> hive-c0re/src/workers/knowledge.rs
nix/host-modules/hive-c0re/options.nix hive-c0re/src/hive_stats.rs
-> hive-c0re/src/stats/hive_stats.rs
nix/packages/default.nix nix/host-modules/hive-c0re.nix
-> .../hive-c0re/options.nix
nix/agent-modules/network.nix nix/host-modules/hive-gateway.nix
-> .../hive-gateway/dnsmasq.nix
frontend/README.md nix/modules/frontend.nix
-> nix/packages/frontend.nix
The two `limits.rs` comments are a matched pair: each names the other's
old path, so the "keep in sync" instruction they exist to carry pointed
both ways at nothing.
Where a flat module became a directory the target is the file that
declares the named thing, not `default.nix` by reflex — the
`preBuildAgentTemplates` option is declared in `options.nix`, and the
DHCP pool that sentence is about lives in `dnsmasq.nix`.
Comments only; no behaviour change. Refs #3923, which is about whether a
gate should cover this class at all — that question is unanswered and
this does not close it.
`docs/swarm/sso.md` described a person in a browser. The swarm's other
callers — the telemetry collector, the queue's auth-callout responder, each
hive's agents — hold no session and follow no redirect, and nothing operator-
facing said how they authenticate. Its relying-party table is forge and
matrix, both browser surfaces.
The new section carries what `swarm-authelia.nix` was holding in comments:
one client per hive because identity belongs to the directory, the audience
being that client id rather than a parallel naming scheme, and signed rather
than opaque tokens because the collector verifies offline against
`/jwks.json` while the queue introspects.
It also states the fail-closed rule once, in the place a reader looks before
touching a vhost: an error page answers 200, and `auth_request` reads any 2xx
as access granted. That shape has now appeared three times — this module's
`/api/` prefix and both of victorialogs' routes — which is what makes it
documentation rather than a comment.
The two comment blocks those replace shrink to the part that is genuinely
local: the submodule-typing reason these clients are a definition rather than
an append, and a loud warning against folding the machine prefix back into
`/`. The security warning stays at the site; only its consequence list moves.
Comments 495 -> 465 lines. Option `description` strings are untouched: they
are the source `pkgs.nixosOptionsDoc` renders into the operator's options
reference, so trimming one would delete published documentation rather than a
duplicate.
Per mara's go-ahead on hyperhive#3902 ("getting started is good, but
terminal rendering does not go in there i think"):
Moved 21 top-level docs/*.md files into 7 new topic subdirectories
(existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched):
getting-started/ setup.md
agent-lifecycle/ agent-hierarchy.md, approvals.md, persistence.md
trust-boundary/ boundary.md, security.md
integrations/ forge.md, matrix.md, github.md, knowledge.md
networking/ gateway.md, network.md, snapshot-store.md
scheduler/ jobq.md, coordinator.md, ci.md, observability.md
process/ conventions.md, gotchas.md, pr-review-gate.md
web-ui/ terminal-rendering.md (moved into the EXISTING dir,
per mara's correction to the original getting-started
guess -- it's UI implementation detail, not onboarding)
The physical layout now matches docs/README.md's own topical headers,
which already amounted to this taxonomy -- see the scoping comment on
the issue for the two findings that motivated this (a genuine
duplication between CLAUDE.md's old "Reading paths" list and
docs/README.md's grouped one, since drifted out of sync with each
other; and the flat layout not matching the grouping we already had).
Fixed every cross-reference this moved across the whole repo (~120
files: docs/ internal links at every depth, Rust doc comments, nix
module option docs, crate READMEs) -- verified two ways: a grep sweep
confirming zero remaining references to any old path, and a script
that resolves every markdown link in docs/**/*.md + CLAUDE.md +
README.md against the filesystem and reports anything that doesn't
exist (zero broken links).
Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to
a pointer at docs/README.md, now the single index. Rewrote
docs/README.md itself to use the new subdirectory paths and added the
one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md).
Classified all 22 docs/*.md files first via a haiku subagent (mara's
suggestion) on two axes -- proposed grouping and operator-vs-
implementation focus -- before finalizing the taxonomy; spot-checked
the report and found internal inconsistencies (its classification
table disagreed with its own summary section for a few files), so this
taxonomy is my original proposal + the one correction mara gave
directly, not a blind application of the subagent's table. The
operator-focus data it gathered is still useful for a follow-up
content pass (docs skewing 'mixed' rather than pure operator-facing),
not addressed in this PR -- structure only.
nix fmt clean, both pre-push lints clean.
`docs/swarm/sso.md` already argues that the OIDC provider is derived from
the client list rather than carrying its own `enable` — authelia refuses to
start without clients, so a separate flag would be a second fact free to
disagree. The module header said it again, and the definition site said it a
third time.
The doc keeps the argument. The definition site keeps a short version,
because someone changing `oidcEnabled` needs the reason there and not two
files away, along with the part no doc carries: that it is unconditionally
true today and stays derived so the OIDC-gated code below is self-documenting
about why it is conditional.
The header also restated the users-store rationale the doc's "does not
provision users" bullet owns. What survives there is the one clause the doc
does not have — why a file backend is right rather than a placeholder for
LDAP, the subject set being bounded by one swarm.
503 comment lines to 495. Comment-only: still 36 module-eval properties.
Two named anti-patterns from the comment-budget issue, applied to the file
that carries the most comment lines in nix/host-modules.
The header re-explained the two-tier model and the co-location argument that
`docs/observability.md` already owns — "a boundary that disappears locally is
one the local deployment stops testing" appears in both, nearly word for word.
The doc is the right home for it; what stays here is the one fact the doc does
not carry, that this is a container because `services.opentelemetry-collector`
is a singleton option already taken by the hive tier.
The `logs_endpoint` block keeps every instruction and loses the narration:
use `logs_endpoint` not `endpoint`, both spellings pass `otelcol validate`
and every wrong path answers 400, `_msg_field` is required or text search
finds nothing while ingest still answers 200, `_stream_fields` sets stream
cardinality.
Six smaller trims of "measured"/"verified"/"once". One pointed at a probe
script under a single agent's state directory — a path no other reader can
open, in source every reader loads.
What stays: `validateConfigFile`'s "a parser, not a wiring check", the
`issuer_ca_path` warning, the swarm-tier stamp. Non-obvious, in no doc, and
the target is a share of the whole tree rather than a per-file quota.
604 comment lines to 574. Comment-only: the module still evaluates to the
same 36 module-eval properties.
The collector's store exporters were gated on deploy.victoriametrics.enable /
deploy.victorialogs.enable — options that say "this host RUNS the store", not
"the swarm has one". A collector that did not share a host with the stores
rendered no exporter at all and dropped everything it received, from every
hive, silently: an absent exporter is not an error.
mara ruled the premise rather than the mechanism ("the swarm always has a
store"), so there is no gate and no new option for that. Both exporters are
unconditional and address the stores by domain, per the cross-host rule from
the OIDC client and secret-delivery unit #3517 already built. The logs exporter
had the identical bug and is fixed here too.
Both stores gained a machine ingest route, required in the same change: the
exporter now targets https://logs.<domain>/insert/..., and that vhost is
browser-shaped, so shipping the collector half alone would have regressed logs
ingestion that works today. Neither ingest location carries
`error_page 401 =302` — a pusher handed a redirect follows it and POSTs at a
login page, which answers 200.
Whether the collector authenticates follows the CREDENTIAL, never another
service's placement: `clientSecretFile` is a nullable option, and the delivery
unit — the one thing here that may know where authelia runs, since it copies
out of its container — sets it by mkDefault. An earlier revision gated this on
deploy.authelia.enable directly, which put a different service's co-location in
the collector's own config.
Also removed rather than relaxed: the assertion that this collector has
"somewhere to send". It read the store's per-host enable, so it rejected at
eval exactly the deployment reaching the stores by domain exists for.
Deliberately not replaced with an authentication assertion — a collector on a
host of its own is a supported shape, and refusing to build it would make this
fix illegal where the bug bites hardest.
Knock-on worth review: collectLogs is now always satisfied, so journald
collection is unconditional.
Config shape validated against otelcol-contrib 0.151.0 `validate`, with a
bogus-key control confirming the validator checks the extension schema.
module-eval: 31 properties.
Nine panels against the metric names openbao actually emits — the prefix is
`vault_`, the fork kept upstream's names, and a board built on `openbao_`
would render perfectly and show nothing.
The top row works on a SEALED store, which is what a hive has until an
operator runs `operator init`: seal state, uptime, goroutines and a request
count that reads 0 rather than going blank. Everything below is empty until
then, and each panel's description says so rather than leaving an operator to
wonder which of the two it is looking at.
The seal panel reduces with `max()`. Once unsealed there are two
`vault_core_unsealed` series and the `cluster=""` one stays 0 forever, so a
bare selector reads SEALED on a healthy store — invisible while sealed, which
is why the metric names were sampled in both states.
The store's scrape target rendered as `host:port` alone, so the collector
requested `/metrics` — a path openbao does not serve on any listener. Every
scrape since the target landed has 404'd, which reads as a dead exporter
rather than a wrong address, and no bao sample has ever reached the store.
`scrapeTargets` values may now carry a path and query. Both are optional and
omitted when absent, so every existing target renders the config it rendered
before. The query cannot ride in `metrics_path`: prometheus percent-encodes
the `?`, so it has to become `params`.
Measured against openbao 2.6.2 and otelcol-contrib 0.151.0, the pinned
versions: `/metrics` 404s ("this listener only serves the metrics endpoint"),
`/v1/sys/metrics` answers JSON, and only `?format=prometheus` — or an Accept
header the collector happens to send today — returns exposition text. The
query param is the half that does not depend on content negotiation.
`bao operator init` fails at the seal with CKR_MECHANISM_INVALID. The
pkcs11 seal asks for AES-GCM by default and the TPM does not offer it: a
TPM 2.0's symmetric modes are CBC/CFB/CTR/OFB/ECB, and openbao accepts
only AEAD mechanisms — AES-GCM or RSA-OAEP — so RSA-OAEP is the single
mechanism both sides implement.
Measured on the deployed token (`pkcs11-tool --list-mechanisms`, #3860):
no AES-GCM, and `RSA-PKCS-OAEP, keySize={1024,2048}, hw, encrypt,
decrypt` present. CBC is not a fallback — openbao's
`MechanismFromString` rejects `CKM_AES_CBC_PAD` as deprecated and its
encrypt path implements exactly the two AEAD mechanisms.
The key gets a new label so a store provisioned by the earlier module
keeps its unusable AES key without the two resolving to one label, and
the addkey step is now keyed on the label rather than on the store not
existing — otherwise an existing deployment never gains the RSA key.
#3880 bound /dev/tpmrm0 into the store's container, which was necessary and
not sufficient: a bind mount preserves the host's ownership, openbao runs as a
DynamicUser with an empty CapabilityBoundingSet (so no CAP_DAC_OVERRIDE), and
the host applies no tpm udev rule at all — measured, 0 hits across all 41 host
closures against 62 rule files as a control. The node therefore keeps the
kernel default and the seal cannot open it.
A name cannot fix this. NixOS allocates system groups at activation, per
machine, so `tss` — or any group declared on both sides — gets two different
ids, and the device node carries the number. mara picked pinning a gid with an
overridable default (deploy.bao.tpmGid).
The default sits above the range NixOS auto-assigns system groups from
(400-999, measured in update-users-groups.pl) and above the normal-user range,
and below systemd's DynamicUser range (61184-65519), so it collides with
nothing any of those allocate.
The module-eval case compares the two sides rather than checking each against
a literal: the property is that they AGREE, not what they agree on. Its
absence arm is a shamir store, which never opens a TPM and must not claim a
device node's group — without it, pinning unconditionally would look identical.
openbao runs as a DynamicUser, so the uid that has to open the token
store is allocated by the container's PID 1 and cannot be named by a
host unit — the store was created root-owned 0700 and the seal could
never read it. The provisioning unit moves inside the container and
hands the sqlite store over by group; the host keeps only the mkdir the
bind mount needs, create-only so a reboot does not re-impose 0700.
Two further layers blocked the same start, both measured while fixing
this one:
- DynamicUser implies ProtectSystem=strict (systemd.exec(5)), so the
bind mount was read-only to openbao however it was owned, and the
pkcs11 library opens its store read-write. ReadWritePaths= is
required and was absent.
- allowedDevices renders DeviceAllow= and nothing else, and nspawn
builds its own /dev as a fixed tmpfs and cannot create device nodes
— verified against a live container, whose /dev holds no host
devices at all. /dev/tpmrm0 was therefore absent inside swarm-bao,
not merely unpermitted. It is now bound in.
Whether openbao's dynamic uid may *open* that node is a third question:
the tss gid is dynamically allocated, so no name or number means the
same thing on both sides of the boundary. Filed separately rather than
guessed at here.
The two module-eval cases that asserted the unit on the host now assert
it in the container and absent from the host, and two new cases pin the
device bind and the write access — each was individually valid and
collectively required, which is the state no assertion catches. Both
new cases select with `or [ ]`: mutation-testing them showed that a bare
select aborts the run with a nix trace instead of failing the case by
name, which also hid the second failure behind the first.
The store declared no journald units and served no metrics: nothing in
`swarm-bao.nix` mentioned either, while every sibling swarm service
declares both.
Metrics get their own loopback listener rather than a flag on the API
one, and that follows from what a scraper can express rather than from
taste: `swarm.otel.scrapeTargets` is `host:port`, plaintext and with no
credential, while the API listener is TLS and demands a client
certificate once a client CA is set. `metrics_only` narrows the new
listener to the metrics path; `prometheus_retention_time` is what serves
the endpoint at all.
Measured against openbao 2.6.2 before writing any of it: the metrics
path answers 200 on such a listener *while the node is sealed and
uninitialised*, 503 on the API listener, and 404 for a non-metrics path
on the metrics listener.
The listener exists only where a collector does — it is unauthenticated
by design for now, and an endpoint with no reader would be exposure
bought for nothing.
The port cannot be the API port + 1: openbao derives every listener's
cluster address as its own port plus one, so that number is already
taken. An assertion says so, since the failure is otherwise a race with
no log line.
Journald units are declared by the module that defines each unit, not
gathered here, matching the option's own rule — a name nothing defines
is silently ignored, so a central list would read as coverage on hives
that have neither glue module.
Refs #3849
openbao has never started on a hive that deploys it. With raft storage
and no `cluster_addr`, `bao server` exits 1 before serving anything:
cluster address must be set when using raft storage
Reproduced against openbao 2.6.2 with the module's own config shape:
the same JSON minus `cluster_addr` exits 1 with that line, and with it
the server comes up and configures every listener.
Both addresses are built from `swarm.bao.domain` rather than a bind
address — that is the URL a reader already dials (`BAO_ADDR` in
glue-matrix-bao-token) and the name the server certificate carries.
Cluster traffic sits one port up, upstream's convention.
The `unknown or unsupported field loopback` warning in the same startup
log is unrelated and cosmetic: openbao's unknown-field check does not
know about named listener blocks, but the parser honours `type` and
configures each one (measured, two named listeners, both served).
Noted in the module so the next reader does not chase it; the JSON
array form that avoids the warning does not typecheck against nixpkgs'
`settings.listener` (`attrsOf`).
module-eval gains the regression gate plus a control that the settings
it reads vary per deployment.
Refs #3860
Nothing resolved `bao.<swarm.domain>`. It is absent from
`swarm.serviceDomains` and, unlike every other swarm service, contributed
no `gateway.localNames` entry — so the name a reader dials answered only
if an operator had published an external record, which nothing asks them
to do.
Cross-hive traffic goes via the domain either way; only what it resolves
to varies, and for a multi-host swarm that is the operator's upstream DNS.
This covers the deployment with no upstream record to configure.
DNS only, and that asymmetry is deliberate: bao stays out of
`serviceDomains` and gets no vhost. Its leaf comes from its own CA, so a
services-CA entry would mint one nothing uses, and nginx terminating TLS
would strip the client certificate the store authenticates every hive
with. `swarm-nats.nix` documents the same exclusion for itself.
module-eval gains the presence and absence arms: claiming a name this
host does not serve would point every local reader at the wrong machine.
Refs #3862
`swarm-bao.nix` declared the store's half of the mTLS pair as options —
`serverCertFile`, `serverKeyFile`, `clientCaFile` — and left the reader's
half as a literal inside `glue-bao-tls.nix`, which only runs where
`deploy.bao.enable` is set. A hive that did not host the store therefore
could not read from it and could not be pointed at a certificate even
when one had been placed by hand.
Adds `clientCertFile`, `clientKeyFile` and `serverCaFile` beside their
three server siblings, `mkDefault`ed by the glue to the leaf it already
mints, and moves `glue-matrix-bao-token.nix` onto them. Its gate becomes
"this host holds an identity" rather than "the store is a neighbour",
and the unit ordering that names store-local units is now conditional --
`Requires=` on an absent unit fails the job.
`serverCaFile` is separate from `clientCaFile` on purpose: one is the
store choosing which readers to trust, the other a reader choosing which
store to trust. Self-signing collapses them to one file, which is a
property of that deployment and not of the pairing.
Closes#3855.
The UI and the swarm-controller it fronts are one service to a reader and to a
certificate, so a separate option only ever created two spellings of one
address — and the controller's own public URL was built from one of them while
an allow-list entry named the other.
Removed rather than aliased. A config still setting it was pinning a name the
controller never used; an alias would preserve exactly that mismatch, so
`mkRemovedOptionModule` says so and fails the build.
⚠️ The option was TOTAL and `swarm.domain` is NULLABLE. That is not incidental:
`ui.domain` fell back to `swarm.invalid` so hive-network.nix's required-domain
assertion is what an operator sees, rather than a coercion error from a
consumer. Each site keeps that property its own way — swarm-ui binds a local
total (the idiom swarm-otel.nix already uses as `domainBase`), and the three
sites already gated on `deploy.swarm-ui.enable` gain `&& domain != null`, which
is stricter than a placeholder and puts no invented name into rendered config.
One behaviour change worth naming: `swarm.nix`'s certificate SAN list used to
receive the placeholder `swarm.invalid` on a null domain and now receives
nothing. `lib.optional` would otherwise have put a literal `null` in it.
Verified with `checks.module-eval` rather than `nix-instantiate --parse`: parse
proves syntax, and every failure this change could plausibly cause — a dead
binding, a renamed reference — is an evaluation error.
Her review: too much text, and an assertion helper instead of that much code
where the file should just say "this option cannot contain that".
`lib/name-guards.nix` gets `mustNotEqual` / `mustNotContain`; both call sites in
swarm-otel become four fields each — the option, the names, the list, and the
sentence explaining the failure. The message plumbing (find every offender,
quote them, name the list) is one shape shared by both.
Offenders are printed after their label rather than before it, because no
English verb agrees with both one name and five.
Converted the pre-existing equality assertion too. Leaving one hand-rolled
beside one helper-built is the worse of the two outcomes, and it is the same
guard.
Verified by evaluating both guards directly rather than only parsing: clean
rosters pass, `foo-agent`/`beehive` fail containment, `swarm` fails equality,
and `forgeworks`/`operator-hq` still pass — the control that keeps the two
lists from being merged.
Hive-scoped identifiers are built by joining a hive name with fixed words —
`hive-<name>`, `hive-<name>-agent` — so a hive called `foo-agent` produces
exactly the id hive `foo`'s agent containers produce. The queue's responder
resolves it as the agents, and that hive silently receives an agent grant
instead of its own; a NATS denial reaches a client as a timeout, so nothing
names the cause. Equality cannot see this: the two NAMES differ, only the
composed strings collide.
Deliberately a second file rather than three more entries in
`reserved-names.nix`. That list is matched by equality against a whole name and
every entry is a value some component emits; applying containment to it would
refuse `forgeworks` and `operator-hq` for failures that do not exist. The two
lists differ in subject (hive-only vs both), matcher (substring vs equality)
and admission rule, and each file's header says so.
No Rust change: `HIVE_RESERVED_NAMES` answers "may an AGENT be called this",
which stays an equality question — an agent name is never composed into these
identifiers.
Measured against the predicate rather than assumed:
pr1ma=ok alpha=ok forgeworks=ok operator-hq=ok
foo-agent=REFUSED beehive=REFUSED swarm=REFUSED myswarmx=REFUSED
The four `ok` cases are the control: `forgeworks` and `operator-hq` are exactly
what a merged list would have refused.
27 lines to 11. What went: the measured deny line (it is in the PR and the
commit that made the change), the "one value, two readers" argument for reading
the option the target URL is built from, and a note about `unique`.
What stayed is the part nobody can re-derive from the code: `external` must not
be dropped, both hyperhive hosts are private addresses the builtin denies, and
the entry is deliberately not gated on the per-host `deploy.swarm-ui.enable`
because a spread deployment has the forge and the swarm host on different
machines.
Her review: don't declare a `let` for a single use. The comment that justified
the binding moves to the use site and drops the claim it was carrying — that
`swarm.ui.domain` and `swarm.domain` can diverge — keeping only the part that
holds either way: this reads the same option swarm-controller builds its public
URL from, so the allow-list entry and the target it must match are one value
rather than two spellings.
`swarm.domain` is not the value swarm-controller addresses this hive with —
`swarm-controller.nix` builds `SWARM_CONTROLLER_PUBLIC_URL` from
`swarm.ui.domain`, which merely defaults to it and is documented as
overridable. Allowing the one and being delivered the other would work until
someone takes that option up, then fail as a refused delivery with nothing
naming the cause.
Also states why the entry is NOT gated on `deploy.swarm-ui.enable`, which is
what makes the controller register the hooks at all: in a spread deployment the
forge and the swarm UI are different hosts, so that per-host flag is false
exactly where the allow-list entry is needed.
The comment-block lint caught the prose at 35 lines; trimmed to the parts a
reader cannot re-derive.
`webhook.ALLOWED_HOST_LIST` named `external` plus the HIVE domain. The
swarm-controller's webhooks target the SWARM domain, which is a different host
on the same private gateway IP — so `external` denied it and the hive entry did
not cover it. Every swarm-controller delivery has been failing there.
Found from the sender's log, which names the host and the rule it broke:
deny 'constellation.darkest.space(10.42.0.1:443)'
webhook can only call allowed HTTP servers (check your
webhook.ALLOWED_HOST_LIST setting)
This is the layer under the DNS fix: resolution now succeeds and reaches
10.42.0.1:443, and forgejo refuses to dial it. The two failures look identical
from the receiving end — no delivery, no log line — which is why reading the
sender was what separated them.
`optional` because a hive with no swarm has no such domain; `unique` because an
all-local deployment can legitimately set both to the same string.
mara (PR#3820): the operator doesn't type /matrix/ into the browser — it's a swarm-level link (hive-matrix.nix registers it into services.hyperhive.swarm.controller.links, rendered by swarm-ui's LinksMenu). Corrected the M4TR1X page doc, and fixed the hive-matrix.nix comment that still described the removed H0M3 tile in present tense while I was in there.
The forge container used the host's resolvers, where the swarm domain has
no records — so every outbound call to a swarm name failed with "no such
host". Forgejo's webhook deliveries died there: knowledge, config-pr and
vcs-activity alike, which is why the swarm-controller's receiver has never
logged a verified delivery and knowledge propagation has been running on
its hourly fallback.
An earlier fix taught the container authelia's name with a `networking.hosts`
entry. That covered one name and left the rest, so use the resolver every
other service container already uses and drop the override — hive-matrix
records why a second answer that can disagree with the first is worse than
none.
Closes#3824
The dashboard's log surface is an allow-list: the four harness units in
every agent container, the hive-ci / hive-forge / hive-matrix machines,
and three host units. `swarm.otel.journaldUnits` only named the host
three plus forgejo — so the units an operator can read in the UI were
absent from the store they are supposed to be queryable in, and nothing
reports that, because a name that matches nothing is not an error.
Adds the four agent units, the CI runner and the homeserver. The agent
units are declared by the hive-c0re module rather than by the
agent-modules that define them: those are a guest evaluation and cannot
reach a host option.
Closes#3818
The state directory was bind-mounted from the host so a nixos-container
destroy could not take the swarm's secrets with it. No sibling service does
that -- swarm-grafana keeps its sqlite database inside the container on
ephemeral = false -- and the bind is what broke the store: upstream pairs
StateDirectory= with DynamicUser=, systemd relocates the state to
/var/lib/private/openbao, and that rename fails EBUSY on an active mount
point, so the unit died at STATE_DIRECTORY before bao ever ran.
The TLS material still has to cross the boundary, because a host unit writes
it and the container reads it, so it moves to its own small bind at
/var/lib/swarm-bao-tls rather than riding along in the state directory. That
directory is 0755 and read-only inside: the certificate and client CA are
public and are read straight off the mount.
The private key is not. install -m 0600 leaves it root-owned and the service
runs as a DynamicUser, so the bind-mounted file is unreadable to it -- which
the old layout hid, because StateDirectory chowned the whole tree on the way
past. LoadCredential is systemd's mechanism for precisely this: PID 1 opens
the source as root and re-exposes it inside the unit owned by the service's
own account.
`services.hyperhive.tls.{stateDir,caValidityDays,leafValidityDays}` sat at
the top of `services.hyperhive`, which is meant to be everything about
hyperhive rather than the settings of one hive. Where the hive CA lives,
how long it lasts and how long the leaves it signs last are decisions of
the host holding the key — `deploy.*`, by the same rule as the switches
that moved before them.
`hive-controller` is hive-c0re's new name (mara on the issue), so the
knobs hang off the daemon that owns the CA rather than off a bare `tls`
at the root. mkRenamedOptionModule entries carry existing configs.
⚠️ Unlike the two switch renames, these names are NOT unique, so this was
swept by ALIAS BINDING rather than by identifier: hive-tls.nix alone holds
two options spelled `stateDir` — its own `cfg.stateDir` and the swarm CA's
`swarmCaCfg.stateDir`, four sites that must not move. Nine files bind an
alias to this config; the rename followed those bindings.
Two sites were invisible to the obvious check, and an unanchored sweep for
`hyperhive\.tls\b` is what found them: the option declaration (`= {` after
the path, so no trailing `.` or `;`) and the alias convention documented in
a comment in lib/hive-ca-trust.nix.
Also renamed the `<tls.stateDir>` shorthand in four docs and two Rust doc
comments, anchored on its delimiters — the new path contains the old one
as a substring, so an unanchored replace would have doubled the prefix.
mara: "deploy.nix still says it everywhere". Seven option descriptions
each told the reader that this service derives from the swarm-services
switch — the same duplication just removed from docs/swarm/services.md,
one layer over, and rendered into the generated options doc.
Each keeps only what is true of that service and nothing else: the
metrics store is paired with grafana, the log store has clients rather
than second instances, the homeserver can be placed on its own host, a
swarm has one SSO provider. The store's paragraph about where it runs
being a separate question from that it runs survives, because that is
about the store, not about the switch.
swarm-ui keeps its positive derivation (the controller) and loses only
the "rather than from allSwarmServices" half.
Deliberately not touched, as a different question rather than an
oversight: the *mode* option's description in local-defaults.nix, whose
whole job is to name what it asserts; the module comments in
swarm-authelia.nix / swarm.nix, which are read by someone editing that
module rather than by an operator picking an option; and ui.md's
contrast, which she has not ruled on.
Same defect as the switch below it, one tier up: it sat at the TOP of
`services.hyperhive`, a namespace that is meant to be everything about
hyperhive rather than the settings of a single hive. Whether this box is
the whole deployment is as per-host as a decision gets.
The name follows mara's sentence for what it means — "everything in the
swarm is running on this host" — rather than naming its mechanism.
"Defaults" was doing no work: it is not a defaults toggle, it is a claim
about where the swarm lives, and the pair now reads as the containment it
already was, singleHostSwarm implying allSwarmServices plus this hive.
One site was a setter rather than a reference: module-eval's `allLocal`
fixture passes an attrset merged into `services.hyperhive`, so its key
carries the path and had to become `deploy.singleHostSwarm`. A rename by
bare identifier is right for the twelve prose mentions and wrong for
exactly this one, which is worth knowing before the next rename.
Both halves of the old name were wrong about the subject. The services
are required of the SWARM, not of the host, and the option says whether
THIS host runs them — so it described the wrong thing and sat in the
namespace that has to be identical on every host. The new name is mara's
own phrasing of what it means: "deploy all swarm level services on this
host".
mkRenamedOptionModule carries existing configs, read-side references
included, so this warns rather than failing to evaluate.
Three sites were not just the identifier:
- local-defaults.nix set it inside `config.services.hyperhive.swarm =
{ … }`. It moves out as a path beside the other deploy.* setter rather
than into a second `deploy = { … }` attrset — the warning that file
already carries about `swarm` applies to any second definition of the
same parent.
- swarm-required-services.nix bound only `swarmCfg`, now unused; it binds
and reads `deployCfg`.
- Two comments in that file described a half-migrated state, where the
switch asserted some `swarm.*.enable` toggles and some `deploy.*` ones.
Every one of them has been `deploy.*` for several slices now.
hive-metric counters are delta by default; VictoriaMetrics has no
delta-to-cumulative processor of its own and official guidance is to
feed it cumulative. Converting at the hive tier, closest to the
origin, means every downstream consumer (swarm tier, VictoriaMetrics,
upstream) sees ordinary cumulative counters and existing PromQL idioms
(rate()/increase()) keep working unchanged. Default config is a no-op
passthrough for anything already cumulative (Claude Code's own export,
hostmetrics, Prometheus scrapes) -- verified against the actual
collector binary (otelcol-contrib validate), not assumed: an empty
config validates clean, an invalid key is rejected.
The comment said it bounds the wait rather than hanging the boot, and
nothing in the unit bounded anything. What actually held was systemd's
default start timeout plus the homeserver only Wants= this unit — both
true, neither stated here, so a reader had to already know them.
TimeoutStartSec puts the number a boot waits on in the file that waits,
and the comment now names the degradation: hitting it keeps the locally
minted token, same as every other failure path in this unit.
Caught by argus reviewing the slice it shipped in.
Three places said nothing supplies them — the module header, the
serverCertFile and clientCaFile descriptions, and the assertion message
an operator actually hits. All three were true when written and
falsified by glue-bao-tls.nix landing in the same PR, which mkDefaults
all three paths.
The load-bearing half is unchanged: the store never reaches for an
authority, because it must not take its identity from one it will itself
distribute. What was wrong is the claim that therefore nothing fills them
in. The service module declares no default and does not know where the
value comes from; the glue supplies one where the store is deployed; an
operator's own path beats it.
The assertion also says what reaching it now means — the glue is absent,
or something set the paths back to null.
Fetches an opaque 32-byte value and writes it where hive-matrix.nix
already looks, so the homeserver never learns the store exists.
Chosen over authelia's OIDC secret deliberately: that one needs a
.secret AND a matching .digest, so shipping it first would debug 'can a
reader authenticate and get bytes back' and 'did we write authelia's
file format right' at once, with an SSO outage as the failure mode. Here
the failure is narrow -- new agent accounts cannot be provisioned,
existing ones untouched.
Every failure path keeps the local token: no such key, sealed store,
unreachable store, empty value. The activation script's mint-if-absent
is untouched, so a hive with no store behaves exactly as it does today.
matrixMachine is a literal because hive-matrix.nix declares no `machine`
option -- `matrixCfg.machine` parses cleanly and fails at module-system
resolution, which is the kind of error only reading the target module
catches.
A CA that signs exactly two things -- the store's server certificate and
the client certificate of whoever reads from it -- and distributes
nothing. Not the hive CA, not the swarm CA: the store will distribute
both, and an authority you must already hold a certificate from cannot
be one the store hands out. Not the gateway's HTTPS material either,
self-signed or ACME; that is a different trust domain with a different
audience.
The minting lives here rather than in swarm-bao.nix because it is an
opinion about where the store's identity comes from -- the most
consequential one available. The service serves what it is handed. A
deployment with a real internal CA drops this file and names its own
paths in serverCertFile / clientCaFile, and nothing in the store
changes. Ordering simplifies too: with the unit and its consumer in one
module, before/requiredBy is internal rather than a cross-module fact.
Idempotent on ABSENCE only. Re-issuing the CA invalidates every client
certificate already trusting it, so a rebuild that refreshed it would
lock out every reader in the swarm at once.
OpenBao in a swarm-bao nixos-container, unsealed by the host TPM through
the built-in pkcs11 seal, with shamir as the documented opt-out for
hardware that has no TPM. The seal guard asks the package whether it was
built with an HSM rather than reading its version, and falls back to the
version cliff only when a package advertises no tags -- it fails closed.
Every listener serves TLS: loopback unconditionally, because the host
running the store is always one of its readers, plus whatever
extraListenAddresses names, because which network the other hives share
is a deployment fact. Client-certificate verification turns on only when
clientCaFile says what to trust.
The store's identity is an input, and nothing in THIS module fills it
in. Service-to-store mTLS is a separate trust domain from the gateway's
HTTPS certificates and from both CAs in this tree: a store must not take
its identity from an authority it will itself distribute, or it cannot
come up before the thing that issues it. The module asserts when the
paths are unset rather than reaching for whichever CA happens to be
wired; what supplies them on a self-contained deploy is the next commit.
The leaf is copied to the container's state dir by a host unit rather
than bind-mounted: nixos-container refuses to start when a bind source
is missing, and a directory holding a leaf usually holds the CA's key
beside it.
A swarm has one store, so running it is a fact about this machine and
belongs beside the other deploy.* toggles rather than in the namespace
every host agrees on. Attrset with an enable, not a bare bool, so a
second deployment decision has somewhere to go.
Asserted from swarm.enableRequiredServices alongside the other
once-per-swarm services, and asserted as a mkDefault -- which is what
keeps *where* the store runs a separate question from *that* it runs.
Set it directly to put the store on a host of its own; a hive that does
not run it is a client either way, reaching it by name.
Two descriptions in the file still narrated the old swarm.* layout as
something that used to be true, and the queue's claimed it belongs on
the shared-services host. Both now say what holds.
Both retentionPeriods move to deploy.<store>.retentionPeriod. Measured
before moving: each is read exactly once, in the container definition of
the module that deploys the store, and no client hive consults either --
so they are options on the auto-deployed service rather than something
every host has to agree on.
What stays in swarm.* is what a client needs to reach the store: the
package, the name it answers on, the port. Both modules said "what stays
here is its package, domain, retention and wiring" in their options-block
comments; retention no longer does, so those say so.
Renames go in deploy.nix with the rest, so the whole move keeps one home
and one file to delete when the deprecation window closes.
Last of the swarm.* -> deploy.* moves for the enable-shaped options.
Whether this host also runs the CI runner is a per-machine decision, so
it was sitting in the namespace that has to be identical swarm-wide.
Renames all five options, not just enable: name, concurrency, labels and
package all describe the runner THIS host would run, and leaving them
behind would keep the original defect for four more options. One rename
entry each, because ci is a plain attrset rather than a submodule type,
so there is no parent path to rename in one go.
deploy.forgejo is the name deploy.nix's own header already specifies,
including this exact case -- "ci (does this host run the runner too) is
exactly that shape". It is the only entry with no enable: the forge is
not optional, it deploys with hyperhive itself, so running the runner is
the only deployment decision it has.
Also fixes hive-c0re/src/forge/ci_runner.rs, whose doc comment named
services.hyperhive.forge.ci.enable -- missing the swarm. segment, so it
had never resolved to a real option.
HIVE_MATRIX_URL was conditioned on this host deploying the homeserver,
so a hive that is a client of the swarm's matrix forwarded nothing and
every agent's hive-matrix-daemon no-opped. hive-c0re already provisions
those agents' accounts against a remote homeserver through
HIVE_MATRIX_API_URL three lines below, whose gate is `apiUrl != null` —
it registered the users and then never told them where to log in.
gatewayHost is swarm-scoped and defaults to chat.<swarm-domain>, so it
names the swarm's one homeserver from any member hive. Its being set is
the real question, and a null swarm domain is already caught by the
assertion in hive-network.nix rather than by this expression.
swarm.matrix.enable was the last toggle still spelled swarm.*, which
is the namespace every host in the swarm agrees on - and "does this
host run the homeserver" is exactly what differs between them. It
moves to deploy.matrix.enable with the rest; the genuinely swarm-wide
matrix settings stay where they are.
mkRenamedOptionModule keeps existing configs evaluating with one
warning naming both paths.
The sweep for readers had to be unanchored and cross-language: three
of them were outside nix, including a hive-c0re error message telling
the operator to set the old name. A dotted grep also cannot see the
rename module's own list form, so that was checked separately.
