`check-doc-refs.sh` resolves the PATH half of a `docs/x.md::Section`
pointer and stops there. The section half rots the same way, and more
quietly: the file still exists, so every path-shaped check stays green
while the pointer names nothing.
Eighteen sites, five distinct pointers, each retargeted at a heading
verified to exist rather than at the nearest plausible one:
docs/web-ui.md::Container row
-> docs/web-ui/dashboard.md::Container row
docs/web-ui.md::Shared terminal pane
-> docs/web-ui/shape.md::Shared terminal pane
Both sections moved out when docs/web-ui.md became a two-heading
index. The path still resolves, which is exactly why nothing
caught them.
approvals.md::Helper events to the manager
-> approvals.md::Helper events to the submitting agent
Renamed with the manager special-casing removal; the pointer kept
the old vocabulary.
approvals.md::Migration from the pre-tag
-> approvals.md::Startup migrations (older hosts)
Same content, including the HIVE_SKIP_META_MIGRATION kill switch
the citing comment names.
agent-hierarchy.md::Current state
-> ::Where the tree lives (topology.rs, container_view.rs)
-> ::Reparenting (topology.rs's set_parent, host-sock)
Split by what each site actually asks for rather than repointed
uniformly: two want the format and the source-of-truth rule, two
want the reparenting validation.
Three known-dead pointers are deliberately left alone:
* `docs/integrations/forge.md::Sources` sits on a line PR #3927
rewrites; fixing it here would conflict for no gain.
* `docs/web-ui/shape.md::One unified channel` names real text that is
bold inline rather than a heading — which of those counts as
resolvable is the open question on #3922.
* `persistence.md::Harness state files` should point at a heading
whose own text contains backticks, and the backticked-pointer form
cannot nest them. That is a limit of the convention, not a typo.
Comments only; no behaviour change. Refs #3922.
Eleven doc comments pointed at `docs/` files as markdown links. Ten of
them render as broken hyperlinks in the docs rustdoc CI builds, and
nothing in the tree can tell.
Rustdoc renders a page at `target/doc/<crate>/<module…>/`, so a relative
link resolves against that directory and not against the source file it
was typed in. Every one of these except the single crate-root `//!` was
written for a reader resolving from the source tree, which is one `../`
short at module level and two short one directory deeper.
Two measurements on a throwaway crate, same build and same
`RUSTDOCFLAGS="-D rustdoc::all"`:
* a bogus intra-doc link `[`no_such_item`]` is a hard error, so the
`docs-rustdoc` check in nix/checks.nix works for its class;
* a relative link to a nonexistent file in the same comment produces
no diagnostic at all and lands in the html verbatim as
href="../../../docs/does-not-exist.md".
So the class is invisible to the one gate whose stated purpose is to
stop a doc pointer dangling — and it is worse than the plain-text
failure that gate's comment describes, because a broken href still
looks clickable.
Fixing the depths was the other option and is rejected: the correct
depth is a function of how deeply the module is nested, so any module
move silently breaks it again, and no check we have would notice.
The link text was already the canonical pointer — `docs/x.md::Section`,
the same repo-root-relative form used everywhere else in the tree and
the form scripts/check-doc-refs.sh gates. Dropping the `[…](…)` wrapper
keeps every byte of information a reader uses and removes the only part
that was ever wrong.
Refs #3926.
Comments cite nix modules, scripts and crate source files constantly,
and nothing evaluates a comment — so when a file moves, the reference
rots silently and `nix flake check` stays green. A reader following one
finds nothing and cannot tell whether the file was renamed, deleted, or
never existed.
Seven such references, each repointed at the file that actually holds
the thing the sentence is about rather than at the directory the old
name became:
hive-c0re/src/agent_config/limits.rs hive-agent/src/mcp.rs
-> hive-agent-mcp/src/mcp/mod.rs
hive-agent-mcp/src/mcp/mod.rs hive-c0re/src/limits.rs
-> hive-c0re/src/agent_config/limits.rs
(and the module path in the doc
comment above it, which was stale
in the same way)
hive-c0re/src/forge/mod.rs hive-c0re/src/knowledge.rs
-> hive-c0re/src/workers/knowledge.rs
nix/host-modules/hive-c0re/options.nix hive-c0re/src/hive_stats.rs
-> hive-c0re/src/stats/hive_stats.rs
nix/packages/default.nix nix/host-modules/hive-c0re.nix
-> .../hive-c0re/options.nix
nix/agent-modules/network.nix nix/host-modules/hive-gateway.nix
-> .../hive-gateway/dnsmasq.nix
frontend/README.md nix/modules/frontend.nix
-> nix/packages/frontend.nix
The two `limits.rs` comments are a matched pair: each names the other's
old path, so the "keep in sync" instruction they exist to carry pointed
both ways at nothing.
Where a flat module became a directory the target is the file that
declares the named thing, not `default.nix` by reflex — the
`preBuildAgentTemplates` option is declared in `options.nix`, and the
DHCP pool that sentence is about lives in `dnsmasq.nix`.
Comments only; no behaviour change. Refs #3923, which is about whether a
gate should cover this class at all — that question is unanswered and
this does not close it.
`docs/swarm/sso.md` described a person in a browser. The swarm's other
callers — the telemetry collector, the queue's auth-callout responder, each
hive's agents — hold no session and follow no redirect, and nothing operator-
facing said how they authenticate. Its relying-party table is forge and
matrix, both browser surfaces.
The new section carries what `swarm-authelia.nix` was holding in comments:
one client per hive because identity belongs to the directory, the audience
being that client id rather than a parallel naming scheme, and signed rather
than opaque tokens because the collector verifies offline against
`/jwks.json` while the queue introspects.
It also states the fail-closed rule once, in the place a reader looks before
touching a vhost: an error page answers 200, and `auth_request` reads any 2xx
as access granted. That shape has now appeared three times — this module's
`/api/` prefix and both of victorialogs' routes — which is what makes it
documentation rather than a comment.
The two comment blocks those replace shrink to the part that is genuinely
local: the submodule-typing reason these clients are a definition rather than
an append, and a loud warning against folding the machine prefix back into
`/`. The security warning stays at the site; only its consequence list moves.
Comments 495 -> 465 lines. Option `description` strings are untouched: they
are the source `pkgs.nixosOptionsDoc` renders into the operator's options
reference, so trimming one would delete published documentation rather than a
duplicate.
Continues #3901 (dashboard/#3906, agent+swarm-ui/#3917, this closes out
the packages/shared slice — jobq-graph was already done separately).
Scanned the rest of shared/src for the same 'used to be X, now Y'
pattern: one real hit. The other two matches (hive-btn.js's is=
attribute history, hive-side-panel.js's one-word /* legacy */ comment)
are load-bearing or too trivial to touch.
Continues #3901. Same pattern as tabs.js/swarm.js: cut 'used to be
X, now Y'/'moved to'/'no longer' change-history narration down to
the current design fact, keep every load-bearing rationale intact
(the ResizeObserver feedback-loop note in Header.tsx, the Dropdown-
vs-real-<a> semantics in MetaNav.tsx, etc.).
The operator never received a coverage report. The job existed but was
`workflow_dispatch` only, so nothing had ever run it — "there is a CI job
for it" was true and produced nothing.
Its header argued against a schedule: an instrumented build roughly doubles
a test job, and a nightly number nobody reads is farm time for nothing. That
was a cost judgement, and it has been made differently. The comment changes
with the trigger rather than staying to contradict it.
The hour is deliberate: the runner's job capacity defaults to 1
(`services.hyperhive.deploy.forgejo.ci.concurrency`, applied as
`runner.capacity` in nix/host-modules/hive-ci.nix), so at that setting this
job holds the runner for its whole timeout and everything else queues.
`--summary-only` wrote into the run log, which is not a place anyone
receives anything. The summary is now teed to a file and uploaded, so a
scheduled run leaves a report to fetch. `if: always()` keeps the partial
output from a failed run.
`set -o pipefail` is load-bearing: without it the step's status is `tee`'s,
so a failed run would report success and upload an empty report. Verified
that shape rather than assuming it — without the guard a failing pipeline
exits 0, with it 1, and a succeeding one still exits 0.
mara on hyperhive/website#59 ("is this slice 1? i expected the options
to fold into the main docs sidebar as well so that you have one all
docs tree" / "the same thing already happens: crate readmes get
included as well. add the virtual options dir"): apply the exact
technique already used for docs/crates/<crate>.md to the nixosOptionsDoc
bundle (packages.docs) — project it into the reference-docs tree at
docs/options/*.md rather than leaving it a wholly separate flake output
options.nix has to fetch and render on its own pipeline.
Simpler than the crates/ case: the options bundle is already
self-contained CommonMark with no relative doc-links needing rewriting
(nixosOptionsDoc's transformOptions already points every option
declaration at an absolute forge URL), so this is a straight
recursive copy, not a per-file sed pass.
website's docs.nix needs no changes to pick this up — its sidebar walk
is already generic over subdirectories, same reason the earlier
crates/ virtualization needed none. The website-side follow-up (has
docs.nix render these instead of options.nix's separate pipeline, and
what that means for the current /options/ URL) is a separate change on
that repo, not touched here.
Two of the three motion gaps mara flagged on the swarm-ui jobs graph
(the third, node status changes as a tree of nesting divs rather than
a node-link diagram, has no edges to animate today — see the issue
thread for that scoping correction).
- New node mount: .jg-node gets a fade+slide-in keyframe. No JS change
needed — Preact only creates a new .jg-node DOM node when its key
(the node id) is genuinely new, so this only plays on first
appearance, not every fetch re-render.
- State change flash: NodeView tracks each node's previous state via
a ref; on a real change it adds .jg-state-flash to the glyph span
(removed on animationend), driving a scale pulse.
Both follow the same three-rule motion-guard shape as Shell.css's
shell-page-enter / LinksMenu.css's links-menu-popover-enter.
Making `AgentState` `Copy` in the preceding commit is a change to every
consumer of the type, not to the crate that declares it. `hive-c0re` grew its
own consumer while this branch was in review, and under `-D pedantic` a
one-byte enum taken by reference is `trivially_copy_pass_by_ref` and a
`.clone()` on it is `clone_on_copy`. Neither crate is wrong alone; the merge
is.
`decide` now takes the state by value and the call sites drop the `&`. No
behaviour change — the function only matches on the value.
The hive-side loop landed without anything to converge to: nothing wrote
`$KV.hive-wanted.<hive>`, so in production only the "no key" branch ran.
This is the writer.
`WantedWriter` mirrors `StatusReader` — that module reads what hives report,
this one writes what they are told, so it holds a client rather than a bucket
handle and resolves the store on first use. It shares the status reader's
connection: the controller has exactly one by design, and a second connect
would double the auth-callout traffic and give the two paths independent
reconnect state.
The value under a hive's key is the map of every agent on that hive, so a
plain `put` of a single-agent change would drop a concurrent change to a
different agent, with only one revision of history to not recover from.
Writes are read-modify-write against the entry revision, and only
`WrongLastRevision` / `AlreadyExists` count as a lost race — every other
error returns immediately rather than spinning the retry loop and then
blaming a concurrent writer that never existed.
`apply` is split out and tested because it holds the invariant: declaring
one agent preserves the rest, and a current value that will not decode is an
error rather than a fresh start. Overwriting a document nobody can read
discards every other agent's declaration.
Two routes, no swarmctl verb and no jobq node: `create_agent` needs a graph
because it is multi-step, and one CAS'd write is not.
`build_app` is extracted from `main` in the same change because `main` sat at
exactly the `too_many_lines` limit, so adding an endpoint tripped a lint
about the startup sequence. The route list is the part that grows.
argus caught this on review: ci.md's own content never changed (0 diff on the
move), but it went from one level under docs/ to two (docs/scheduler/ci.md),
so its `../.forgejo/workflows/ci.yml` link now needs `../../`.
Also fixes the verification script itself — it only checked link targets
ending in .md or / (a systematic blind spot for a link to any other file
type, which is exactly how this one slipped through both the grep sweep and
the link-resolution check). Re-ran the corrected version repo-wide (93
tracked .md files, matches argus's own count) — zero broken links.
Per mara's go-ahead on hyperhive#3902 ("getting started is good, but
terminal rendering does not go in there i think"):
Moved 21 top-level docs/*.md files into 7 new topic subdirectories
(existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched):
getting-started/ setup.md
agent-lifecycle/ agent-hierarchy.md, approvals.md, persistence.md
trust-boundary/ boundary.md, security.md
integrations/ forge.md, matrix.md, github.md, knowledge.md
networking/ gateway.md, network.md, snapshot-store.md
scheduler/ jobq.md, coordinator.md, ci.md, observability.md
process/ conventions.md, gotchas.md, pr-review-gate.md
web-ui/ terminal-rendering.md (moved into the EXISTING dir,
per mara's correction to the original getting-started
guess -- it's UI implementation detail, not onboarding)
The physical layout now matches docs/README.md's own topical headers,
which already amounted to this taxonomy -- see the scoping comment on
the issue for the two findings that motivated this (a genuine
duplication between CLAUDE.md's old "Reading paths" list and
docs/README.md's grouped one, since drifted out of sync with each
other; and the flat layout not matching the grouping we already had).
Fixed every cross-reference this moved across the whole repo (~120
files: docs/ internal links at every depth, Rust doc comments, nix
module option docs, crate READMEs) -- verified two ways: a grep sweep
confirming zero remaining references to any old path, and a script
that resolves every markdown link in docs/**/*.md + CLAUDE.md +
README.md against the filesystem and reports anything that doesn't
exist (zero broken links).
Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to
a pointer at docs/README.md, now the single index. Rewrote
docs/README.md itself to use the new subdirectory paths and added the
one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md).
Classified all 22 docs/*.md files first via a haiku subagent (mara's
suggestion) on two axes -- proposed grouping and operator-vs-
implementation focus -- before finalizing the taxonomy; spot-checked
the report and found internal inconsistencies (its classification
table disagreed with its own summary section for a few files), so this
taxonomy is my original proposal + the one correction mara gave
directly, not a blind application of the subagent's table. The
operator-focus data it gathered is still useful for a follow-up
content pass (docs skewing 'mixed' rather than pure operator-facing),
not addressed in this PR -- structure only.
nix fmt clean, both pre-push lints clean.
mara set a 20-line comment maximum and scoped this file for the first pass.
Measured rather than eyeballed: this file had exactly one block over 20 — the
28-line header. Every other block in it is 8 lines or fewer, so the pass is
that header plus one removal.
The header keeps the four things a reader can act on — why this check exists,
assertion versus the absence class, name a case for the property it defends,
and evaluates-rather-than-executes with its stubbing rule — and loses the
connective prose around them.
The verdict-reading paragraph goes back out. It landed by a misclick on a PR
she meant to close, and the judgement is the one she gave its sibling: the
technique works, it just does not earn source space.
The lint still enforces 30. 208 blocks repo-wide exceed 20, so flipping MAX
belongs to the overhaul rather than to this file's pass.
`docs/swarm/sso.md` already argues that the OIDC provider is derived from
the client list rather than carrying its own `enable` — authelia refuses to
start without clients, so a separate flag would be a second fact free to
disagree. The module header said it again, and the definition site said it a
third time.
The doc keeps the argument. The definition site keeps a short version,
because someone changing `oidcEnabled` needs the reason there and not two
files away, along with the part no doc carries: that it is unconditionally
true today and stays derived so the OIDC-gated code below is self-documenting
about why it is conditional.
The header also restated the users-store rationale the doc's "does not
provision users" bullet owns. What survives there is the one clause the doc
does not have — why a file backend is right rather than a placeholder for
LDAP, the subject set being bounded by one swarm.
503 comment lines to 495. Comment-only: still 36 module-eval properties.
Two named anti-patterns from the comment-budget issue, applied to the file
that carries the most comment lines in nix/host-modules.
The header re-explained the two-tier model and the co-location argument that
`docs/observability.md` already owns — "a boundary that disappears locally is
one the local deployment stops testing" appears in both, nearly word for word.
The doc is the right home for it; what stays here is the one fact the doc does
not carry, that this is a container because `services.opentelemetry-collector`
is a singleton option already taken by the hive tier.
The `logs_endpoint` block keeps every instruction and loses the narration:
use `logs_endpoint` not `endpoint`, both spellings pass `otelcol validate`
and every wrong path answers 400, `_msg_field` is required or text search
finds nothing while ingest still answers 200, `_stream_fields` sets stream
cardinality.
Six smaller trims of "measured"/"verified"/"once". One pointed at a probe
script under a single agent's state directory — a path no other reader can
open, in source every reader loads.
What stays: `validateConfigFile`'s "a parser, not a wiring check", the
`issuer_ca_path` warning, the swarm-tier stamp. Non-obvious, in no doc, and
the target is a share of the whole tree rather than a per-file quota.
604 comment lines to 574. Comment-only: the module still evaluates to the
same 36 module-eval properties.
Per mara's guidance on hyperhive#3901 (target ~15% comment density
overall, less where obvious, more where not; prefer docs for
abstract/general topics; don't restate facts in multiple places;
don't document history).
tabs.js: removed 4 pure "X now lives in Y" / "moved to Z" historical
asides (the underlying facts are already documented in
docs/web-ui/dashboard.md, not lost by removing the floating in-code
aside) and merged one comment block that had drifted into restating
the same fact twice (one ticker feeding two live displays, documented
as if it were two separate tickers).
swarm.js: cut a comment narrating the removal history of two features
that no longer exist in the code (a per-agent queued-badge and a
client-side jobq tally), keeping only the design constraint still in
force (why the jobq-derived state here is deliberately narrow); cut a
comment documenting a removed CSS class's history down to a statement
of the current class's purpose; cut a comment restating the
jobq-rollup rendering rationale already stated once, above, down to a
one-line pointer; trimmed a "legacy flat layout is bit-identical"
history clause down to what the depth-0 case actually renders.
The boot sweep's error arm substituted `Wanted::from_running(running)`,
which is the one value for which `reconcile_action` returns `Noop` — both
ways. An agent whose `agent_power` row could not be read therefore could
never enter `drifted`, so on a fresh rev marker a corrupt row produced one
`warn!` per boot and no other signal, indefinitely.
Classify the unreadable case as its own outcome instead: the agent gets a
boot `Reconcile`, whose `get_or_seed` fails as a per-agent node — a surface
the dashboard already renders — and the failure stops at that one agent.
The classification moved into `boot_action`, a pure function, because the
loop had no tests at all. The first of the five asserts the unreadable case
across every (fresh × running) combination, which is exactly the matrix the
fabricated value made unreachable.
The check's results are already embedded in its builder text — the comment
above `runCommand` says so — but nothing said that this makes the verdict
readable straight out of the derivation. So mutation-testing a case, which
is the documented way to prove a case can fail, has been costing a build on
the shared farm per arm when it costs none.
Command verified against both arms: on an unmutated tree it prints the
`N properties hold` line, and with a case deliberately broken it prints that
case's `FAILED:` line. The jq shape is envelope-agnostic, since `derivation
show` grew a `{"derivations": …}` wrapper.
`PowerStore::get` ended `Ok(row.and_then(|s| Wanted::parse(&s)))`, so a
row holding an unparseable value collapsed into the same `Ok(None)` as
no row at all. Its doc comment stated the narrower case as the whole one
("`None` when the agent has no row yet"), which is what kept the gap
invisible.
The damage is not that a caller reads the wrong value. `get_or_seed`
takes `None` to mean "never seeded" and writes
`Wanted::from_running(running)` — so a corrupted row silently
**overwrote itself** with whatever the container happened to be doing at
that moment, which is precisely the inference an authoritative intent
store exists to prevent. Its own comment says the DB is authoritative
once seeded.
The error arm was already in the signature; this makes it reachable.
Every caller already handles `Result`: the two `get_or_seed` paths
propagate, so a corrupt row now fails that agent's boot reconcile
instead of erasing itself, and `workers::wanted` already skips an agent
whose intent could not be read.
The second test is the one that matters — it asserts the row still holds
its original bytes afterwards, because the overwrite, not the read, was
the bug. Both tests carry a valid and an absent row alongside, so
neither can pass by breaking `get` for everything.
Found by argus reviewing #3891, which reads this store but does not
write it.
mara's call on the PR: "dont make the enum open, we will just add
entries later". The catch-all variant is gone, and with it the per-agent
inert path.
What changes is where version skew lands, not whether it is handled. An
unrecognised value used to be one agent this hive left alone; it is now a
decode failure for the whole declaration, so a hive running older code
converges *nothing* rather than obeying the agents it happened to
understand. That fails closed instead of dangerous, and it is the right
trade when both ends ship together — which is what "add entries later"
assumes.
The test moved with the property rather than being rewritten in place:
`an_unknown_state_fails_the_whole_declaration` lives in
swarm-queue-client, where the decode is, with a valid entry beside it as
the control. `hive-c0re` keeps a coverage check that every state this
build knows produces an action somewhere — asserting inertness there
would be asserting something the type system no longer lets me build.
The deploy event is a nudge with no second path: core NATS is
at-most-once, so a hive that was down when the controller published
simply never learns that an agent is meant to exist here. This adds the
repair path — one boot-time DAG node that reads this hive's own key in
the `hive-wanted` bucket and converges the agents it names.
Two semantics settled on the issue thread, and both are places where a
plausible implementation is the wrong one:
- **Absence is not a deletion order.** No bucket, no key, or an agent
the value does not name all mean the controller has said nothing.
Swarm-side lifecycle does not yet cover agents that predate it, so
"converge to exactly this set" would tear down every agent the swarm
has not adopted. `plan` only ever inspects the agents a declaration
names.
- **An unrecognised state is inert.** `AgentState` is an open enum: a
value this build cannot read deserialises into `Unrecognised` and is
left alone. A closed enum would force "not `Up`" onto a state like
`paused`, so a controller that learned a new value would take agents
down on every hive not yet updated.
Divergence is measured against the hive's **stored power intent**, not
the container's observed running state — an agent that is down while its
intent says `Up` is already the boot reconcile's work, and a loop reading
`is_running` would insert a start DAG behind that reconcile's back on
every boot. A hive that already agrees with its declaration queues
nothing at all.
`queue_first_deploy` is extracted from the deploy-event path rather than
open-coded here, for the power-intent seed: without it `first_deploy`'s
tail `Reconcile` seeds `Wanted` from a container that exists but has not
started yet, which locks the agent to `Offline` on its first reconcile.
The read is authorised as-is: `store.get` takes async-nats' direct-get
arm (the KV bucket is created with `allow_direct`), which is exactly the
`$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.<hive>` subject
`swarm-nats-auth` grants a hive. The fallback subject is not granted, and
a refused NATS request surfaces as a timeout rather than an error.
Nothing writes the bucket yet — the controller-side writer is the other
half of #3124, so this does not close it.
This PR makes a swarm collector on a host with neither store — and no
authelia — an expressible deployment; `module-eval`'s `otelRemoteAuthelia`
case is exactly that. `secrets.md` still asserted the opposite in two
places, and one of them was the "hive elsewhere" column reading `n/a`.
The delivery unit is the convenience for the co-located case, not the
definition of it: whether the collector authenticates follows the
credential, never another service's placement.
The collector's store exporters were gated on deploy.victoriametrics.enable /
deploy.victorialogs.enable — options that say "this host RUNS the store", not
"the swarm has one". A collector that did not share a host with the stores
rendered no exporter at all and dropped everything it received, from every
hive, silently: an absent exporter is not an error.
mara ruled the premise rather than the mechanism ("the swarm always has a
store"), so there is no gate and no new option for that. Both exporters are
unconditional and address the stores by domain, per the cross-host rule from
the OIDC client and secret-delivery unit #3517 already built. The logs exporter
had the identical bug and is fixed here too.
Both stores gained a machine ingest route, required in the same change: the
exporter now targets https://logs.<domain>/insert/..., and that vhost is
browser-shaped, so shipping the collector half alone would have regressed logs
ingestion that works today. Neither ingest location carries
`error_page 401 =302` — a pusher handed a redirect follows it and POSTs at a
login page, which answers 200.
Whether the collector authenticates follows the CREDENTIAL, never another
service's placement: `clientSecretFile` is a nullable option, and the delivery
unit — the one thing here that may know where authelia runs, since it copies
out of its container — sets it by mkDefault. An earlier revision gated this on
deploy.authelia.enable directly, which put a different service's co-location in
the collector's own config.
Also removed rather than relaxed: the assertion that this collector has
"somewhere to send". It read the store's per-host enable, so it rejected at
eval exactly the deployment reaching the stores by domain exists for.
Deliberately not replaced with an authentication assertion — a collector on a
host of its own is a supported shape, and refusing to build it would make this
fix illegal where the bug bites hardest.
Knock-on worth review: collectLogs is now always satisfied, so journald
collection is unconditional.
Config shape validated against otelcol-contrib 0.151.0 `validate`, with a
bogus-key control confirming the validator checks the extension schema.
module-eval: 31 properties.
Nine panels against the metric names openbao actually emits — the prefix is
`vault_`, the fork kept upstream's names, and a board built on `openbao_`
would render perfectly and show nothing.
The top row works on a SEALED store, which is what a hive has until an
operator runs `operator init`: seal state, uptime, goroutines and a request
count that reads 0 rather than going blank. Everything below is empty until
then, and each panel's description says so rather than leaving an operator to
wonder which of the two it is looking at.
The seal panel reduces with `max()`. Once unsealed there are two
`vault_core_unsealed` series and the `cluster=""` one stays 0 forever, so a
bare selector reads SEALED on a healthy store — invisible while sealed, which
is why the metric names were sampled in both states.
The store's scrape target rendered as `host:port` alone, so the collector
requested `/metrics` — a path openbao does not serve on any listener. Every
scrape since the target landed has 404'd, which reads as a dead exporter
rather than a wrong address, and no bao sample has ever reached the store.
`scrapeTargets` values may now carry a path and query. Both are optional and
omitted when absent, so every existing target renders the config it rendered
before. The query cannot ride in `metrics_path`: prometheus percent-encodes
the `?`, so it has to become `params`.
Measured against openbao 2.6.2 and otelcol-contrib 0.151.0, the pinned
versions: `/metrics` 404s ("this listener only serves the metrics endpoint"),
`/v1/sys/metrics` answers JSON, and only `?format=prometheus` — or an Accept
header the collector happens to send today — returns exposition text. The
query param is the half that does not depend on content negotiation.
`bao operator init` fails at the seal with CKR_MECHANISM_INVALID. The
pkcs11 seal asks for AES-GCM by default and the TPM does not offer it: a
TPM 2.0's symmetric modes are CBC/CFB/CTR/OFB/ECB, and openbao accepts
only AEAD mechanisms — AES-GCM or RSA-OAEP — so RSA-OAEP is the single
mechanism both sides implement.
Measured on the deployed token (`pkcs11-tool --list-mechanisms`, #3860):
no AES-GCM, and `RSA-PKCS-OAEP, keySize={1024,2048}, hw, encrypt,
decrypt` present. CBC is not a fallback — openbao's
`MechanismFromString` rejects `CKM_AES_CBC_PAD` as deprecated and its
encrypt path implements exactly the two AEAD mechanisms.
The key gets a new label so a store provisioned by the earlier module
keeps its unusable AES key without the two resolving to one label, and
the addkey step is now keyed on the label rather than on the store not
existing — otherwise an existing deployment never gains the RSA key.
#3880 bound /dev/tpmrm0 into the store's container, which was necessary and
not sufficient: a bind mount preserves the host's ownership, openbao runs as a
DynamicUser with an empty CapabilityBoundingSet (so no CAP_DAC_OVERRIDE), and
the host applies no tpm udev rule at all — measured, 0 hits across all 41 host
closures against 62 rule files as a control. The node therefore keeps the
kernel default and the seal cannot open it.
A name cannot fix this. NixOS allocates system groups at activation, per
machine, so `tss` — or any group declared on both sides — gets two different
ids, and the device node carries the number. mara picked pinning a gid with an
overridable default (deploy.bao.tpmGid).
The default sits above the range NixOS auto-assigns system groups from
(400-999, measured in update-users-groups.pl) and above the normal-user range,
and below systemd's DynamicUser range (61184-65519), so it collides with
nothing any of those allocate.
The module-eval case compares the two sides rather than checking each against
a literal: the property is that they AGREE, not what they agree on. Its
absence arm is a shamir store, which never opens a TPM and must not claim a
device node's group — without it, pinning unconditionally would look identical.
mara (#3877): issue report too narrow — the 8-column table was
clipped by .shell-body's 60em readable-line-length cap, which is right
for the rest of the UI's cards/forms but too tight for a wide table.
damocles diagnosed the root cause and scoped the shape (route
allowlist + an additive .shell-body-wide modifier, no prop plumbing
through App.tsx) before asking for a nod; mara then routed the actual
build to me. Implemented that shape as scoped: /issues opts into a
90em cap via WIDE_BODY_ROUTES, every other route keeps 60em untouched.
Verified with real headless-chromium screenshots at 1280px: the issue
report's full 8-column row (through 'transitively blocks') now fits
with no horizontal scroll, and a second screenshot of /agents confirms
every other route is unaffected.
openbao runs as a DynamicUser, so the uid that has to open the token
store is allocated by the container's PID 1 and cannot be named by a
host unit — the store was created root-owned 0700 and the seal could
never read it. The provisioning unit moves inside the container and
hands the sqlite store over by group; the host keeps only the mkdir the
bind mount needs, create-only so a reboot does not re-impose 0700.
Two further layers blocked the same start, both measured while fixing
this one:
- DynamicUser implies ProtectSystem=strict (systemd.exec(5)), so the
bind mount was read-only to openbao however it was owned, and the
pkcs11 library opens its store read-write. ReadWritePaths= is
required and was absent.
- allowedDevices renders DeviceAllow= and nothing else, and nspawn
builds its own /dev as a fixed tmpfs and cannot create device nodes
— verified against a live container, whose /dev holds no host
devices at all. /dev/tpmrm0 was therefore absent inside swarm-bao,
not merely unpermitted. It is now bound in.
Whether openbao's dynamic uid may *open* that node is a third question:
the tss gid is dynamically allocated, so no name or number means the
same thing on both sides of the boundary. Filed separately rather than
guessed at here.
The two module-eval cases that asserted the unit on the host now assert
it in the container and absent from the host, and two new cases pin the
device bind and the write access — each was individually valid and
collectively required, which is the state no assertion catches. Both
new cases select with `or [ ]`: mutation-testing them showed that a bare
select aborts the run with a nix trace instead of failing the case by
name, which also hid the second failure behind the first.
The case counting that fixture's listeners is about the addresses
declared on it. A collector on the same host adds a metrics listener,
so the expected count depended on a flag the case never mentions —
found by mutating the gate it sits next to, which failed both.
The store declared no journald units and served no metrics: nothing in
`swarm-bao.nix` mentioned either, while every sibling swarm service
declares both.
Metrics get their own loopback listener rather than a flag on the API
one, and that follows from what a scraper can express rather than from
taste: `swarm.otel.scrapeTargets` is `host:port`, plaintext and with no
credential, while the API listener is TLS and demands a client
certificate once a client CA is set. `metrics_only` narrows the new
listener to the metrics path; `prometheus_retention_time` is what serves
the endpoint at all.
Measured against openbao 2.6.2 before writing any of it: the metrics
path answers 200 on such a listener *while the node is sealed and
uninitialised*, 503 on the API listener, and 404 for a non-metrics path
on the metrics listener.
The listener exists only where a collector does — it is unauthenticated
by design for now, and an endpoint with no reader would be exposure
bought for nothing.
The port cannot be the API port + 1: openbao derives every listener's
cluster address as its own port plus one, so that number is already
taken. An assertion says so, since the failure is otherwise a race with
no log line.
Journald units are declared by the module that defines each unit, not
gathered here, matching the option's own rule — a name nothing defines
is silently ignored, so a central list would read as coverage on hives
that have neither glue module.
Refs #3849
openbao has never started on a hive that deploys it. With raft storage
and no `cluster_addr`, `bao server` exits 1 before serving anything:
cluster address must be set when using raft storage
Reproduced against openbao 2.6.2 with the module's own config shape:
the same JSON minus `cluster_addr` exits 1 with that line, and with it
the server comes up and configures every listener.
Both addresses are built from `swarm.bao.domain` rather than a bind
address — that is the URL a reader already dials (`BAO_ADDR` in
glue-matrix-bao-token) and the name the server certificate carries.
Cluster traffic sits one port up, upstream's convention.
The `unknown or unsupported field loopback` warning in the same startup
log is unrelated and cosmetic: openbao's unknown-field check does not
know about named listener blocks, but the parser honours `type` and
configures each one (measured, two named listeners, both served).
Noted in the module so the next reader does not chase it; the JSON
array form that avoids the warning does not typecheck against nixpkgs'
`settings.listener` (`attrsOf`).
module-eval gains the regression gate plus a control that the settings
it reads vary per deployment.
Refs #3860
Nothing resolved `bao.<swarm.domain>`. It is absent from
`swarm.serviceDomains` and, unlike every other swarm service, contributed
no `gateway.localNames` entry — so the name a reader dials answered only
if an operator had published an external record, which nothing asks them
to do.
Cross-hive traffic goes via the domain either way; only what it resolves
to varies, and for a multi-host swarm that is the operator's upstream DNS.
This covers the deployment with no upstream record to configure.
DNS only, and that asymmetry is deliberate: bao stays out of
`serviceDomains` and gets no vhost. Its leaf comes from its own CA, so a
services-CA entry would mint one nothing uses, and nginx terminating TLS
would strip the client certificate the store authenticates every hive
with. `swarm-nats.nix` documents the same exclusion for itself.
module-eval gains the presence and absence arms: claiming a name this
host does not serve would point every local reader at the wrong machine.
Refs #3862
The bucket the previous commit adds had no grants, so the controller
could not create or write it and no hive could read its own key.
Measured against nats-server 2.14.4 rather than extended by analogy,
because the shapes are not symmetric:
- controller: `STREAM.INFO` + `STREAM.CREATE` on `KV_hive-wanted` and
`$KV.hive-wanted.>`. With only today's status grants, `get_key_value`
timed out and the server named the two missing stream subjects.
- hive: `STREAM.INFO` plus **one** direct-get subject carrying its own
key. A KV read is a publish — `store.get` is a request — and the
direct-get subject embeds the key, so the read scopes per hive. By
analogy with `reader_subjects` this would have been `.>`, handing
every hive every hive's wanted set.
Both refusals were verified to fire, not assumed: as `alpha`,
`get beta` was refused naming
`$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.beta`, and
`put alpha` was refused naming `$KV.hive-wanted.alpha`.
`a_reader_may_list_and_fetch_but_not_write` asserted a reader holds no
`$KV.` subject at all, which held only while status was the sole
bucket. Narrowed to the status bucket — the invariant it defends is
that the controller cannot forge a hive's own report, and the wanted
bucket runs the other way.
Refs #3124
The controller declares each hive's wanted agent set; the hive reads its own
key. Deliberately not a mirror of `status`, which the module documents as a
table: authored by the controller rather than the hive, DECLARED rather than
observed, and — the row that decides how it must be built — unrecoverable if
the store is lost, where status regenerates because every hive republishes
what it is.
Two open functions rather than one shared `open_or_create`. `status` shares its
constructor because either end may legitimately arrive first on a fresh swarm;
here the writer is single and known, so the hive gets a read-only open
returning `Option` and holds no grant to create the bucket. Its absence is the
ordinary pre-publication state, not an error a hive could fix.
Absence is also not a deletion order, which the module says at the place an
implementer will meet it: swarm-side lifecycle does not yet cover agents that
predate it, so a hive finding no key has learned nothing about what it runs —
converging to an empty set would tear those down.
No consumer yet. This is the half that is invariant under the scope semantics
and cadence still being decided.
`hive_subjects` granted `$JS.API.STREAM.INFO`/`CREATE` on `hive-notices`
but never the per-hive subject a publish actually targets, so every
`notices::publish` was refused with a permissions violation naming
`hive-notices.<hive>`.
The gap survived because `notices_subjects`'s own doc comment asserted
`extra_hive_subjects` covered it "since that one *is* per-hive". Nothing
in `nix/` has ever passed `--hive-publish-subject` (`git grep` → 0 hits;
control: `--reader-client` is passed at `nix/host-modules/swarm-nats.nix:711`),
so the publish had been refused for as long as the feature existed.
Subject comes from `swarm_queue_client::notices::subject`, the same
constant the publisher formats from, rather than a literal here.
Refs #3859
`echo 'FAILED: ${c.name}'` hand-quoted a string that is prose, so the
first case name containing an apostrophe to actually fail terminated the
builder script mid-report. The remaining failures never printed, and the
error read as a shell syntax error rather than as a broken property.
Only failing cases are ever echoed, so every green run agreed the
reporter was fine -- the defect was reachable exclusively at the moment
the report was the thing being relied on. Measured: with two cases
failing, the old form printed one line and an EOF error, the escaped
form prints both plus the count.
`swarm-bao.nix` declared the store's half of the mTLS pair as options —
`serverCertFile`, `serverKeyFile`, `clientCaFile` — and left the reader's
half as a literal inside `glue-bao-tls.nix`, which only runs where
`deploy.bao.enable` is set. A hive that did not host the store therefore
could not read from it and could not be pointed at a certificate even
when one had been placed by hand.
Adds `clientCertFile`, `clientKeyFile` and `serverCaFile` beside their
three server siblings, `mkDefault`ed by the glue to the leaf it already
mints, and moves `glue-matrix-bao-token.nix` onto them. Its gate becomes
"this host holds an identity" rather than "the store is a neighbour",
and the unit ordering that names store-local units is now conditional --
`Requires=` on an absent unit fails the job.
`serverCaFile` is separate from `clientCaFile` on purpose: one is the
store choosing which readers to trust, the other a reader choosing which
store to trust. Self-signing collapses them to one file, which is a
property of that deployment and not of the pairing.
Closes#3855.
mara asked on PR #3854 to remove the revive action alongside the
blank-name spawn form. Both posted to /api/request-spawn as the last
two frontend callers; with this the dashboard has no UI path left that
hits that endpoint (backend removal/decision stays damocles's call).
Drops the K3PT ST4T3 tombstone row's ⊕ R3V1V3 form, leaving PURG3 as
the only per-tombstone action. Updated the two doc/comment spots that
described the now-gone button.
Frontend half of removing hive-level agent spawning (swarm-level
creation now covers it, and its forge-works confirmation just landed
in hive/hive-chat). Removes call.js's blank-name spawn-request form
(R3QU3ST SP4WN) and its now-orphaned .spawnform CSS.
Deliberately keeps core.js's tombstone-list revive action (R3V1V3) --
a different feature (respawn an *existing* agent, reusing its kept
state), not agent creation, and not what this issue asked to remove.
It happens to post to the same /api/request-spawn action with the
tombstone's own name pre-filled rather than a blank input.
Backend endpoint + wiring is a separate slice (damocles).
claude's real OAuth URL is one long unbroken query-string token, no
whitespace for the browser to break on. Without overflow-wrap the
<a> just kept going past the login card's right edge instead of
wrapping at the card's bounded width (mara reported this with a
screenshot). overflow-wrap: anywhere lets it break mid-token as a
last resort, scoped to just the URL line (.login-url) rather than
the whole card.
