Slipped past the #4574 lint — 1473 chars vs. the 100-char cap, caught
by pre-push while working on an unrelated branch. Same treatment as
every other panel: state what the number is, not the reasoning.
Cut every panel/board description down to a short statement of what the
number is. Drops the design-reasoning prose, edge-case notes, and
operator-attributed quotes that had accumulated in an earlier pass —
useful context while building the boards, wrong shape for a tooltip an
operator glances at mid-incident.
The mapping above needs something that says whether it is still working
after whoever wrote it has gone. A timeseries rather than a stat, so a
regression is a line lifting off zero rather than a number nobody reads.
Two series, and the split is the point: rows that carried a PRIORITY and
arrived with no severity anyway (a broken mapping — this one must reach
zero and stay there), against rows that never had a priority to map. The
latter is Claude Code's own OTLP telemetry, which emits log records with no
severity set at the source; mapping cannot reach it, so it is named rather
than folded into one number that never goes to zero.
Needs no provisioning change — logstore.json is already in the shipped
dashboard list.
Records reached VictoriaLogs carrying the journal's raw PRIORITY and
severity_text "Unspecified" — every line in the store, at every tier, with
no level a query or a dashboard could read. VictoriaLogs has no ingest
parameter naming a level field; it auto-detects one by field name, so the
mapping has to happen in the collector.
A stanza severity_parser on each journald receiver, from one shared file
rather than a copy per tier: the two receivers are unrelated config (a
fixed stanza in the agent container, a parameterised block inside the
swarm-otel container) and a drifted copy fails silently — every line still
arrives, labelled as the wrong thing.
Two details that are easy to get wrong and quiet when wrong. PRIORITY
counts down in urgency where the OTEL severity counts up, so the table is
written as a table. And overwrite_text is required: without it the parser
sets the severity number and leaves the text as the raw digit, so
severity_text arrives as the literal "6" — populated, and not a level
anything renders.
Two swarm-wide facts were being read off this machine's deploy set, so the
answer differed between two hosts of one swarm:
- `swarm.authelia.oidc.hiveIdentities` defaulted to `deploy.nats.enable`,
so whether a hive gets an identity at all depended on whether the IdP
host happened to also run the queue. It is on by default now: a swarm's
hives have identities, and the clients are inert until used.
- `swarm.statusPublish.tokenEndpoint` defaulted through `queueLocal`
(`deploy.nats.enable && deploy.authelia.enable`), so a hive that was not
the swarm host had no token endpoint even when the swarm's IdP was
reachable and named. It follows `swarm.authelia.url` now — the same
derivation `swarm-controller.nix`'s own `queue.tokenEndpoint` already
uses, which is correct for a remote provider.
`deploy.nix:1-30` is what makes this a rule rather than a preference:
`swarm.*` is "identical on every host, byte for byte" and `deploy.*` is
"necessarily different on every host". A swarm value derived from a deploy
value cannot satisfy both.
The all-or-nothing status-publish assertion follows: the token endpoint is
no longer one of the coordinates that says this hive publishes — every hive
in a swarm with an IdP has one — so the two per-host coordinates are what
must agree, and they now require the endpoint rather than being counted
beside it.
`queueLocal` itself stays for the three remaining host-local addresses
(`natsUrl`, `clientSecretFile`, `agentNatsUrl`): each of those is a
`deploy.*` value that genuinely differs per host.
Closes#4048
Fix wording flagged on #4521 review: "Single nginx in front of every
hyperhive web surface" and "Runs on the host" claimed a deployment-wide
topology. There's one nginx per host that has something on it, and
potentially more inside service containers. Reword to describe what
this module builds on this host, with no count claim swarm-wide.
`services.hyperhive.gateway.enable`, `gateway.dns.enable` and
`network.enable` replace the `hyperhive.enable` gate on all three. Each
defaults to false; the modules that need one assert it with `mkDefault
true` from inside the guard their own deployment already carries, and
`swarm-required-services.nix` — the module that owns what the
swarm-services toggle implies — asserts all three explicitly.
hive-c0re asserts all three unconditionally, so an ordinary hive keeps
getting them with no opt-in: it is the host's only knowledge that agent
containers exist.
The resolver moves to its own `hive-gateway/dns.nix` so it can be gated
without reindenting the nginx half of the module.
Reinstates `network.enable`, dropping its `mkRemovedOptionModule` shim.
A config still carrying `network.enable = false` from before the removal
now switches the bridge off instead of failing eval.
Also deletes a duplicate `centralToggleOff` fixture in nix/module-eval.nix.
Two sibling slices added it independently (c5f60fd5, ce3b3d94); the merge
was textually clean and left `main` failing to evaluate at all, so this
file could not be gated without removing one.
The receiver was defined for `scrapeTargets != {} || publishedScrapeTargets
!= {}`, but the swarm-tier pipeline named it only for `scrapeTargets != {}`.
A collector with published targets and no loopback ones therefore rendered
scrape configs that belonged to no pipeline: requested, parsed, delivered
nowhere, and valid enough to deploy.
Latent, not live: the module seeds `scrapeTargets.collector` under its own
`enable`, so the loopback set is never empty in a real deploy and the
disagreeing arm is unreachable today. Nothing would have noticed if that
seeding became conditional, which is why the module-eval case pins the
pipeline rather than trusting the seed to keep masking it.
Refs #4511
`services.hyperhive.enable` is going away (Refs #4500). These four blocks
ANDed it with a second condition that was already the load-bearing one:
the bridge-firewall hole asks whether an operator named any
`network.exposeHostPorts`, and the three swarm-level services ask their
own `deploy.*.enable`, each of which defaults false (swarm-ui derives
from the controller's toggle, a sibling deployment decision) and none of
which is derived from the hive toggle. So dropping the conjunct turns
nothing on by itself — the remaining condition still decides.
The way that claim fails is by something becoming unconditional, so
module-eval gets absence arms for all four, each asserted on a host with
the hive ON and on one with it OFF, plus the two controls that make the
absences mean something.
The swarm-controller arm is probed by the credential oneshot and by the
daemon's `ExecStart` rather than by the unit name: hive-tls defines an
environment key on that name, which leaves an inert fragment behind on
any hive with a CA whether or not the controller runs there.
hive-forge's publicUrl and behindGateway, and swarm-controller's
forgeTokenFile, no longer gate their default on
config.services.hyperhive.enable — none of the three has a reader that
depends on hyperhive being enabled to make sense of the value, so the
extra condition only added a hidden coupling. module-eval.nix gains a
centralToggleOff fixture plus four cases asserting each affected
default now resolves identically whether the toggle is on or off.
Refs #4500
`swarm-controller` mints an agent's mTLS leaf at creation and publishes it
at `swarm/agents/<agent>/bao-mtls`. Nothing read it back. This adds the
hop that carries it the rest of the way, and the in-container consumer
that proves the hop works.
Host side, `lifecycle::agent_identity` reads the row under *this hive's*
own certificate — the hive is a principal the store already knows — and
stages the leaf and its key `0600` under a new `agent-identity/<name>`
state dir, deliberately outside every bind-mounted tree. Both files go in
as systemd credentials rather than binds, the same answer and the same
mode reason as the queue secret beside it: the staged key is unreadable
to the unprivileged agent user, and the container manager reads a
`--load-credential` source as root before re-exposing it under the
consuming unit's own `User=`. The agent is never asked to authenticate in
order to obtain the thing it authenticates with.
Container side, `hive-agent-bao-identity.service` logs in with that
certificate and reads the agent's own path back, failing the unit when
either step does not succeed. It fails loudly where the hive-side readers
degrade quietly, because a refused certificate means an agent that
believes it reaches the store and never does — a cause only the login
itself can name.
The address is the whole switch, no separate `enable`, matching how
`queue.nix` and `logs.nix` already gate themselves. A hive with a store
forwards `HIVE_AGENT_BAO_ADDR` and every agent on it gets the check; a
hive without one forwards nothing and no agent does. That is what keeps
the delivery from landing in a container with nothing to read it.
The hive can now reach an agent's identity, so hive privilege covers
agent privilege. Accepted, not mitigated: the alternative is an agent
fetching its own credential with a credential it does not yet have.
Refs #4137
`swarm/agents/<agent>/bao-mtls` did not exist, and neither did any
per-agent identity at the secret store: `policy::agent_object_name`,
`render_agent` and `render_agent_with_queue` had been written and never
called outside their own tests. An agent's only "per-agent" secret today
is read under the HIVE's certificate, through a wide grant on
`swarm/agents/*` — so "per-agent" was presentational.
The swarm now mints the certificate, so no hive ever needs the capability
to mint one. `swarm-controller` is the service that does it: it already
logs in to the store, and its existing grant already covers exactly the
three objects written here (`create/update` on
`secret/data/swarm/agents/*`, `sys/policies/acl/hive-*` and
`auth/cert/certs/hive-*`). No new bao grant, and nothing co-located — a
cert-auth role pins its authority by value, per role, so the controller
issues from its own CA on its own host and pins that CA in the role it
writes. No existing role changes.
The mint node does not report success on a write. After publishing it
connects again, with the leaf it just issued and under the role it just
wrote, and reads the path back — so the policy, the role, the common name
and the leaf are exercised in production on every agent creation. A
certificate this code mints that the role this code writes will not accept
turns the job node red at creation time instead of surfacing later as an
agent container that cannot start.
`TriggerDeploy` gains an `after_any` edge on the mint, not `after_ok`: a
hive cannot pass down a certificate the swarm has not published, but a
host with no authority configured must still create agents exactly as it
does today.
The private key is generated in memory and never written to disk on the
controller — `SecretStore::connect_with_identity` takes the PEM the minter
is already holding, so nothing is written out purely to be logged in with.
Refs #4137
mara: "the subagent should already stamp the agent on the metric. so i want
to see count of distinct subagent names per agent name over time" — it
does. New timeseries panel-46 on the subagents tab: one line per parent
agent, counting distinct subagent label values with a turn in the trailing
$__rate_interval at each point. Rolling, not cumulative — can fall as well
as rise, same shape as any other rate()-derived panel here.
Placed at y:15, below where #4491's panel-45 (stat tile, y:11-15) sits once
merged, and numbered panel-46 rather than reusing panel-45 — argus caught
that this PR and #4491 both independently added a panel-45 in the same grid
slot from a shared unmerged base, which would have collided on whichever
merged second.
Narrower than a true concurrency gauge (subagents open at this exact
instant) — that needs hive-subagent-mcp to publish its own metric, which
nothing does today; noted as a separate, harder ask on #4488.
Bargauge panels display series in the order the datasource returns them —
Grafana has no native sort for that (long-standing upstream gap, grafana#17245).
The reliable fix is sorting at the PromQL level with sort_desc() on an
instant vector, which forge.json's 'Issues by label' panel already does.
Applies the same pattern to the other 14 multi-series bargauge panels across
agents.json, claude-usage.json, and authelia.json (the latter two also
needed instant: true added, since sort_desc only works on instant vectors).
logstore.json's two VictoriaLogs-backed bargauge panels use LogsQL, not
PromQL, and are left out of this pass — LogsQL's sort-after-stats has a
documented ordering bug in some versions; needs its own verification.
mara: "show subagent count in time range" — the existing 'Subagent turns'
tile answers volume (how many turns), not this (how many differently-named
subagents ran at all). New panel-45 counts subagent label values with at
least one turn in the selected range.
readOnly makes a bad socketPath inexpressible rather than diagnosed
after the fact. Since readOnly rejects any definition including one
arriving through a rename shim, drop the deploy.nix rename entry for
it and the fixture line exercising it, and update the controllerOldPath
comment's shim count from seven to six.
Refs #4208
The socket is 0666 (nginx needs to connect(2) as a different user), so its
directory is the only thing bounding what the gateway's nginx can reach —
the crate's README already says so. A unit test pins DEFAULT_SOCKET, but
that only catches a developer editing the const; an operator setting the
documented socketPath option to a shared dir (notably /run/hyperhive,
which holds the host admin socket, or /run/hive) went unenforced.
Denylist rather than the general "not any other module's socket dir"
form — narrower, but it's what the README names and lands without pulling
in other modules' constants.
Refs #4208
First slice of the move off the hand-rolled swarm services sub-CA: the
store gains the three objects the eventual minter needs, and nothing
else.
- a `pki` secrets mount, enabled ask-first off the same `bao secrets
list` snapshot the kv-v2 mount beside it uses, so a rebuild that finds
it mounted does nothing;
- a `swarm-services` role on it, `allowed_domains` read straight out of
`swarm.serviceDomains` — the same swarm-tier list swarm-ca.nix
name-constrains its sub-CA to and hive-tls.nix carries as the leaf's
SANs — with subdomains, localhost, globs and IP SANs all off;
- a `swarm-services-issuer` policy granting `update` on
`pki/issue/swarm-services`.
Inert: the mount has no issuer generated into it, the role therefore
cannot issue, and no login role attaches the policy, so no token in the
swarm carries it. No consumer changes; swarm-ca.nix still mints the
services sub-CA exactly as before. Reversible with
`bao secrets disable pki`.
The cert-auth role that attaches the policy waits for the leaf carrying
its CN, which glue-bao-tls.nix mints — a later step.
Refs #4256
An agent can reach VictoriaLogs only through the gateway, and since the
machine query route landed the way to read it has been to hand-roll a
client_credentials token request and a curl, per query. This is the CLI
that closes that: `swarm-logs query '<LogsQL>'`, matched log lines on
stdout, so the answer pipes into grep like any other command's.
Built to the plan posted on the tracker thread: own crate, own
docs/tools reference generated off the clap tree, `query` as the one
verb, and the JSON error body surfaced on a non-200 rather than
swallowed. No `tail`: streaming is a different endpoint with a different
response shape, and folding it in here would be a fatter scope than the
ask.
Minting the token is NOT implemented here — swarm-queue-client already
owns the client_credentials request, its error type and its CA handling,
and a token-endpoint fix has to be findable in one place. What this crate
adds is the agent-shaped half: the client id arrives as a *file* beside
the secret, so nothing outside nix/agent-modules/queue.nix spells
`hive-<name>-agent` twice. That is the same problem hive-agent's
swarm_queue module solves, and swarm-logs/src/auth.rs is its `decide`
restated over this binary's inputs.
⚠️ The plan named one thing to verify empirically before calling the auth
settled: whether authelia's bearer policy for the logs vhost accepts the
agent client's audience. Measured from inside a container: it does not.
The client minted a token fine but with `aud: []` and `scp: []`, asking
for the logs URL as an audience answered `invalid_target`, and presenting
the audience-less token to the gateway answered a bare 401. So
swarm-authelia.nix's agentClients gains `authelia.bearer.authz` and the
query URL as a second audience — authelia authorises a bearer token by
the URL being requested, and that URL is now one binding read by three
places rather than three spellings of one address.
The URL reaches an agent the same way its queue coordinates do: computed
on the host (a container cannot derive a gateway address), forwarded by
hive_c0re::meta into the container's option set, and consumed by a new
agent module that installs the binary *wrapped* with its coordinates —
the shape swarm-controller.nix installs swarmctl in. Gated on the queue
credential as well as on the URL: a binary that can only answer 401 is
worse than no binary, because an agent reads a 401 as "no logs", which is
the exact confusion the store's machine route was added to end.
Adds services.hyperhive.deploy.swarm-controller.matrixHomeserverUrl,
threaded to the daemon as SWARM_CONTROLLER_MATRIX_HOMESERVER_URL, and a
Rust helper (homeserver_or_configured_default) that lets a caller-supplied
homeserver keep overriding it. Config plumbing only: put_matrix_account
does not call the helper yet, so this is a no-op for every current caller.
Refs #4345
Adds approle to the auth methods enabled during swarm-bao bootstrap,
using the same idempotent check pattern as cert. No role or policy
is configured yet — those are handled in later slices.
Refs #4386
`glue-matrix-bao-token.nix` has read
`secret/swarm/hives/<hive>/matrix/appservice-token` since it landed, but
nothing ever wrote that path. The store was empty in every deployment, so
every read degraded to "keep what activation minted" and each hive stayed
the origin of a value the swarm has to agree on — two hives never
converged.
`swarm-secret-publish` is now the producer. It already holds a store
identity, already writes under the hive prefix, and already runs per
hive in the roster, so the mint is a third loop beside the two OIDC
copies rather than a second shape of this unit.
Idempotence comes from a record of its own, not from the store: this
principal is granted `create`/`update` with no `read`, so it cannot ask
whether a hive already has a token. It keeps what it minted under
`StateDirectory=` (0700 dir, 0600 file) and mints only when that file is
missing or empty; the `put` runs every time, because re-putting the same
bytes changes nothing for a reader while a mint whose publish failed must
not be left as a token this host holds and no hive can reach.
The token never becomes a nix literal and never reaches argv: the mint
redirects into a file, and the publish hands bao `value=@<path>` so bao
opens it itself — the same handling the OIDC loops use.
`hive-matrix.nix`'s activation mint stays as the genuine first-boot
fallback. It already fires only when the token file is absent, so it
cannot clobber a value the store delivered; `hs_token` has no swarm half
and is still minted there for real.
Refs #4402
Nothing reads it any more: hive-c0re creates accounts as the hive's
appservice, so the mint, the host file, the bind mount, the
`LoadCredential` entry and tuwunel's `registration_token_file` all go.
⚠️ `allow_registration` has to go to `false` in the same change, and not
as hardening. tuwunel refuses to START when registration is allowed with
no token configured — it demands
`yes_i_am_very_very_sure_…_open_registration_…` instead — so dropping the
token and leaving the flag true is not a lax homeserver, it is one that
does not boot. The flag is checked only for requests arriving without an
appservice token, so hive-c0re provisions exactly as before and everyone
else is refused outright.
The swarm secret store keeps its role, repointed at the credential that
replaced the token (`swarm/hives/<hive>/matrix/appservice-token`). Its
unit now also re-runs hive-matrix's own registration renderer after
writing the file: the token is half an agreement, and a registration
still naming the previous value authenticates nobody. The renderer is
shared through an internal option rather than copied, so the
registration's shape has one home.
Both spellings of `registrationTokenFile` become
`mkRemovedOptionModule` with a message naming what replaced them. A hive
that never set the option — the default — is unaffected; one that pinned
it fails to evaluate with instructions instead of a silent no-op.
An upgraded hive needs no intervention: the activation script has both
halves in place before the homeserver restarts, existing agents keep the
tokens their devices already hold, and the old token file is left on
disk read by nothing. docs/integrations/matrix.md spells the path out.
Refs #4402
The hive creates matrix accounts with a shared registration token today,
which means the secret that authorises account creation has to reach both
hive-c0re and tuwunel and stay identical in both. An appservice
registration replaces that with an identity: one token that says "this is
the hive's provisioner", carried in an ordinary credential file.
`url = null`, so nothing is served and no daemon is introduced — with no
URL the homeserver never calls out, and the registration exists purely to
give the `as_token` meaning.
Delivered through `appservice_dir` rather than a `[global.appservice.<id>]`
stanza, because a stanza's `as_token` would be a nix literal and a nix
literal is a world-readable store path. The file is minted and rendered by
a host activation script, bind-mounted into the container, and handed to
the homeserver by `LoadCredential` — the same two steps the registration
token and the OIDC client secret already take, and for the same reason
(0600 root on the host, `DynamicUser=true` in the container).
`sender_localpart` is the hive admin account on purpose: loading a
registration creates its sender user on a zero-user database inside
`Services::start()`, and the `admin_execute` promotion runs after that and
still before the HTTP listener accepts anything. So a fresh homeserver has
a joined, power-level-100 admin on its first boot without anyone having
won the first-registered-user grant. `admin_execute_errors_ignore` is set
because a failing startup command otherwise aborts startup outright.
Nothing reads the registration yet — hive-c0re still provisions through
the registration token, which is untouched here.
Refs #4402
`swarmctl agent create <name> --hive <hive>` POSTs `/api/agents` to
swarm-controller over the daemon's unix socket and prints the queued
job's node id.
It deliberately does not wait. The endpoint queues a DAG whose last node
*publishes* a deploy message; the hive's `hive-c0re` then converges on
its own clock, out of the controller's sight. So even a fully settled
graph would not mean the agent is up, and there is nothing this CLI
could wait for that would let it claim otherwise. Printing the id is
exactly what the response says and all of what it says.
Transport is a bare hyper HTTP/1.1 client handshaked onto a tokio
`UnixStream` via `hyper_util::rt::TokioIo` — the same crate family
`hivectl/src/watch.rs` and `hive-agent/src/web_ui/proxy.rs` already use,
all of it already workspace-pinned. The request/response shapes are a
local mirror rather than a shared crate: the controller's own types are
private to its binary and this crate does not link it, the same
separation `hivectl` keeps from `hive-c0re`.
Errors are reduced to one actionable line — the controller answers
RFC 9457 problem+json, so an unknown `--hive` reaches the operator as
the roster of hives that would have worked rather than a body dump.
Response `warnings` are printed when non-empty.
The nix module wraps the binary with `SWARM_CONTROLLER_SOCKET`, read
from the same `socketPath` the daemon binds.
Refs #4399
The swarm can already tell whether an agent is alive — the `agent-status`
KV bucket republishes once a minute — but not what it is doing right now.
A header bar wants the second thing, and a minute-old answer to "is this
agent thinking" is the wrong answer most of the time it is read.
`hive-agent` now publishes a turn-state header to
`$SWARM.agent-state.<hive>.<agent>`, a core subject beside the terminal
rows it already sends. It goes out **on transition, not on a timer**: the
publisher watches the event bus, rebuilds the header, and sends only when
the serialised result differs from the last one it sent — so a second
periodic writer, which is the problem this exists to fix, is not what
replaces the bucket.
The payload is the published contract a swarm-level renderer is written
against, so the test asserts on the serialised JSON keys rather than on
Rust field names. Two fields deliberately depart from the per-agent web
UI's `StateSnapshot`: `turn_state_since` is an ISO 8601 UTC string rather
than unix seconds, matching the sibling `$SWARM.term` subject's stamp, and
`agent_state` carries the swarm's own `AgentState` vocabulary rather than
a `paused` boolean, so a reader can compare actual against wanted without
translating. `turn_state` and `agent_state` stay two separate fields:
neither vocabulary contains the other's values.
Swarm-side, `GET /api/agents/{name}/state/stream` relays the subject as
SSE, resolving the agent's hive at request time exactly as the terminal
stream does and passing the bytes through without parsing them.
The broker grant is a second `--agent-publish-subject` rather than a
widening of the existing one, so the terminal family and the header family
stay independently revocable, and a `module-eval` arm pins the rendered
flag and its argument together — the doubled dollar included, since a
single one expands to nothing in `ExecStart` and yields a grant that
matches nothing.
Refs #3802
The swarm collector's OIDC client secret only existed where authelia
did: `swarm-otel-oidc-secret.service` copied the minted plaintext out
of authelia's container tree, reachable only because the two share a
host's network namespace. A swarm that placed authelia elsewhere
delivered nothing, and the option's own description said so —
"a deployment that places authelia elsewhere points this at a file it
delivers itself." Same gap as #3853 and #4234, and this is the
swarm-otel twin of #4234's fix for Grafana.
Mirrors PR #4361 (Grafana) almost exactly:
- `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, in every
deployment, replacing the co-located copy unit outright — one
delivery route, not two, per the ruling that landed under #4234.
- Client registration moved out of `swarm-otel.nix`'s own `config`
block (gated on this host running the collector) into
`glue-swarm-otel-oidc-client.nix` (gated on this host running
authelia), the same split `glue-grafana-oidc-client.nix` made. It
was broken the same way: a split deployment registered the client
nowhere at all, so authelia never minted a secret for the publisher
to send on.
- The publisher's `services` prefix (write grant in `swarm-bao.nix`,
hive read grant in `policy::render`) already covers any service's
path — nothing to add there. `swarm-secret-publisher.nix` only grew
`serviceClientIds` by one entry.
One judgement call, stated rather than buried: the store-reading unit
renders only where this host holds a client identity
(`deploy.bao.clientCertFile`/`clientKeyFile`), rather than asserting
it the way `swarm-grafana.nix` does. Grafana's local login form is
disabled unconditionally, so a Grafana with no OIDC secret has no way
in at all — that earns a hard refusal. This collector without a
credential still receives every hive's telemetry; only its own pushes
to the stores go out unauthenticated and get refused there, an
already-supported degrade the module's own `haveCollectorSecret` flag
named before this change. So the reading unit follows the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use
for their own optional readers: no unit when the identity is absent,
not a build refusal.
Fixtures mirror #4361's: `otelBaoWithAuthelia`/`otelBaoRemoteAuthelia`
are the positive pair (co-located and split, both reading through the
store), `otelNoIdentity` is the negative — no reading unit, no
assertion firing, `clientSecretFile` left null.
Refs #4258
The v2-conversion script carried over the local-test dashboard's own
title ('agents-full-test') instead of the real one, since it read
spec.title straight from the generated ground-truth file without
overriding it back. Caught on a diff review, not by CI — nothing
schema-level flags a wrong-but-valid string.
Confirmed empirically (real Grafana 13.0.7, a file-based provisioner
identical in shape to swarm-grafana.nix's) that the deployed version
provisions v2-schema dashboards natively, tabs included, no migration
path needed. Supersedes the row-based split from the previous commit
on this branch: same four groups (activity/cost/health/subagents), same
17 panels with identical content (targets, fieldConfig, options,
descriptions all carried over unchanged — only the document shape
around them moved from a flat panels array + row wrapper panels to the
v2 schema's spec.elements map + spec.layout TabsLayout referencing them
by id), now rendered as real browser tabs instead of accordion rows.
Generated the panel-content half of this file mechanically rather than
hand-authoring the new schema: posted the pre-conversion (v1, flat,
17-panel) dashboard to a local Grafana 13.0.7 and read it back through
the v2beta1 API, which is Grafana's own conversion of the exact same
panel content into the v2 element shape — then replaced only the
generated layout (a flat GridLayout) with a hand-authored TabsLayout
grouping the same element references into the four tabs. Verified the
result end-to-end against the same local instance: file-provisioned
(confirmed via the loaded object's own grafana.app/managedBy annotation),
and a real screenshot of each of the four tabs rendering with its
correct panels.
Cost, health (cpu/mem/disk), and subagents, plus an activity row for the
panels that fit none of the three (agent/turn counts, lines-of-code and
CLAUDE.md-size trends). Rows rather than tabs: Grafana's tab layout is a
v2/Scenes-schema feature, and this deployment's file-based provisioner
doesn't reliably load v2-schema dashboards, so classic-schema collapsible
rows are the closest available equivalent — one section open, the rest
collapsed. activity stays expanded so the dashboard opens on something
useful instead of a wall of accordion headers.
An agent container cannot dial the store's loopback listener: the bridge
to-loopback DROP rule is there precisely to stop that, and the store
authenticates every reader by client certificate, so the usual answer —
a gateway vhost — is the one shape that cannot work. A terminating proxy
strips the certificate and bao sees nginx as the client for every hive.
nginx's stream module does not terminate. `ssl_preread` reads the SNI off
the ClientHello and splices the rest of the connection through byte for
byte, so bao completes the handshake itself and authenticates the client
it actually has. That is the no-vhost rule kept, not bent.
The listener binds the bridge IP rather than every address, because bao
already holds `127.0.0.1:<port>` in the same netns and a wildcard bind
there is EADDRINUSE — nginx would fail to start, taking the gateway with
it. Nothing moves as a result: the name already resolves two ways, so a
host-side reader still goes straight to loopback and an agent goes
through the passthrough, both on one `BAO_ADDR`.
Renders only inside the store's own `deploy.bao.enable` region; a host
that runs no store grows no listener and opens no port.
Per-agent certificates and per-agent policy are separate work.
Refs #4386
autheliaCfg/forgeCfg/vmCfg/vlCfg were local re-exports of
hyperhiveCfg.swarm.<subpath> — a shape that hides the full option
path from grep, the same hazard #4356 fixed for the tls alias.
Removed the six let-bindings and spelled the full
hyperhiveCfg.swarm.<subpath>.<field> path at every use site instead.
None of the read fields (url, machine, hiveClientPrefix,
agentClientSuffix, domain, port) sit on the old side of a
mkRenamedOptionModule in deploy.nix, so inlining the alias's own
current path is correct as-is.
Refs #4363
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous commit left two delivery paths and a three-way gate:
`swarm-grafana-oidc-secret.service` copied authelia's minted plaintext
out of its host tree wherever the two were co-located,
`swarm-bao-grafana-oidc.service` read the same value from the swarm
secret store wherever they were not, and `ssoConfigured && (ssoLocal ||
haveClientIdentity)` decided whether Grafana got an OIDC block at all.
Delete the co-located path. The store reader is now THE delivery unit,
in every deployment — the publisher on authelia's host writes
`swarm/services/<id>/oidc/client` whether the reader is a network away
or in the container next door. The ruling behind it: the store exists so
a host holds ONE out-of-band secret, its client certificate, and reads
everything else with it. Skipping the store when the producer happens to
be local saves a round trip and costs a second delivery unit, a second
way for the file to be wrong, and a gate to choose between them.
The gate goes too, and both of its questions become assertions, scoped
to hosts that run Grafana:
- `swarm.authelia.url` must be set. `auth.disable_login_form` is
unconditional — Grafana ships an admin/admin account on a public
vhost — so dropping the OIDC block when the swarm names no IdP
produced a container with no SSO and no password box, silently. An
eval-time refusal naming the option is the only report that reaches
anyone, the shape swarm-nats.nix already uses for the same option.
- `deploy.bao.clientCertFile` / `clientKeyFile` must be set. This
replaces a warning that nothing reads back, and its message names both
options and where the leaf comes from.
Fixtures follow. `grafanaWithAuthelia` gains the cert pair, because a
co-located host is a store reader like any other. The old
`grafanaRemoteAutheliaNoIdentity` is kept rather than deleted, renamed
`grafanaNoIdentity`: the shape is still reachable, only its deliverable
changed from silence to a refusal, and an arm now reads that refusal
back. Its mirror `grafanaNoSso` covers the other assertion, each fixture
wrong in exactly one way so an arm can name which refusal fired. Every
positive keeps an explicit negative — the one-delivery-unit arm asserts
the deleted unit is absent in both topologies rather than merely that
the store reader is present.
Refs #4234
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Grafana's OIDC client secret only existed where authelia did. One
`ssoLocal` gate — `grafana.enable && authelia.enable` — decided the
client registration, the minted secret's delivery and the whole
`auth.generic_oauth` block, so a swarm whose authelia runs on another
host got Grafana with no SSO wiring at all. The local login form is
disabled unconditionally, so that is no way in.
Split the one gate into the two questions it was conflating:
- `ssoConfigured` — does this SWARM have an identity provider
(`swarm.authelia.url`, which is swarm-wide and whose own description
makes null mean "no SSO configured"). With a delivery route present
this is what emits Grafana's OIDC block.
- `ssoLocal` — is authelia on THIS host, now spelled as the forge and
matrix modules spell it. It decides only which unit delivers the
secret.
Where authelia is elsewhere, `swarm-bao-grafana-oidc.service` reads the
secret from the swarm secret store, shaped after
glue-queue-agent-credential.nix: cert login fails loudly because a retry
fixes every state it fails on, the read degrades quietly because no
retry turns "no value there" into a value, and nothing writes a
stand-in. The producer is the publisher that already runs on authelia's
host, which gains the swarm's service clients beside the per-hive ones
at `swarm/services/<id>/oidc/client` — with the write grant in
swarm-bao.nix and the hive read grant in `policy::render` to match.
Registration moved to glue-grafana-oidc-client.nix. It has to be
declared where authelia's config is rendered, and swarm-grafana.nix's
config block hangs off this host running Grafana.
Two judgement calls stated rather than buried: a hive's read policy now
grants the whole `services` prefix, because a service's path names the
service and nothing swarm-wide records which hive runs it (cost recorded
in docs/trust-boundary/security.md); and the client is registered on any
authelia host, because no swarm-wide "this swarm has a Grafana" fact
exists to gate it on.
Refs #4234
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hive-subagent-mcp already stamps a subagent=<name> resource attribute
(alongside the parent's own agent=<name>) on every claude_code.* sample
a spawned subagent turn emits -- the label already existed, it just
wasn't plotted anywhere. Adds five panels: subagent cost, subagent
turns, subagent cost share (of all cost), cost by subagent name, and
cost by parent agent. No backend changes needed.
The dashboard already templated an $agent variable and broke turns and
active-time out by agent, but cost and token totals only split by
model, effort and query_source -- there was no panel answering "how
much did each agent spend/use", the aggregate-by-agent view mara asked
for.
An agent can reach the log store only through the gateway, and the only
location that exists for a reader is `/`, which is the operator's browser
route. That route ends in `error_page 401 =302`, and an unauthenticated
caller which follows the redirect gets authelia's login page as HTTP 200
with an HTML body — so a client that checks the status code records a query
that succeeded and matched no logs. Measured on the live gateway; it is the
root cause behind four operator round-trips already.
So the query API gets its own location, `^~ /select/logsql/`, with bare
`auth_request` and no login fallback: an unauthenticated caller gets a 401
it cannot mistake for an empty result. `^~` keeps it ahead of the `/`
catch-all and of any regex location added later, and it stops short of
`/select/vmui/`, which is the browser UI and stays on the browser route.
The query is forwarded unmodified — no filter parameter is injected, so any
authenticated caller reads the whole swarm's logs. That is the rule mara
set: read permissions are a later thing, and this location is where one
attaches when it exists.
Refs #3870
The queue's payload ceiling was justified by what the queue was about to
carry; it carries it now, so the comment says so.
The other two sites say "a hive with no queue configured". The swarm has
exactly one queue and a hive cannot lack it — only its coordinates, its
credential, or its ability to reach it. That wording is already used
everywhere else the absence is named; these two predate it.
The docs section on the agents' queue coordinates stopped at delivering
them and never said what the connection is for. It now names the subject
and the degrade rule, which is the part an operator reading an agent's
terminal at the swarm needs.
Refs #3805
The responder has had an agent arm since the principal was minted, but
no deployment ever passed `--agent-publish-subject`, and an empty list
is a refusal by design: `Policy::permissions` returns `None` rather than
a grant that can do nothing, so every agent was turned away at CONNECT.
The subject itself is the one this thread settled on, `$SWARM.term`
namespaced per hive.
The value has to reach the responder with a literal dollar. systemd
substitutes `$NAME` in `ExecStart` whether or not the word is quoted,
so a single dollar expands `SWARM` — unset, therefore empty — and the
responder is handed `.term.{hive}.>`. That grant validates (it carries
the `{hive}` placeholder), is accepted, and matches nothing any agent
publishes to, so the failure surfaces as an authorization violation far
from its cause. `$$` in the unit text is the escape for one dollar.
The module-eval case reads the rendered unit rather than the module
source, because the single-dollar version renders perfectly well; the
doubled dollar is the only thing that distinguishes them before deploy.
The grant is per-hive, not per-agent: an agent's identity names its
hive, so any agent in a hive can publish as another. That is the
tradeoff ruled acceptable for now, tracked separately for tightening.
Refs #3805
The swarm always has exactly one queue; a hive can only lack its
address. Reworded every prose site this PR added that stated or
implied the opposite, to name what is actually absent (coordinates,
credential, or address) instead of the queue itself.
Refs #3805
The host end: `HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR` tells the daemon
where the reader unit put the files, and a new
`deploy.hive-controller.queue.agentNatsUrl` says where the queue is as an
agent *container* reaches it. That address defaults to the bridge one and
never to loopback — `statusPublish.natsUrl` beside it is loopback and
correct, because hive-c0re shares the host netns and an agent does not.
Paired with the swarm's token endpoint, gated together, and forwarded by
`hive_c0re::meta` as both an env var and an agent option: the harness
reads the variable at runtime, its unit is built from the option.
The agent end: `nix/agent-modules/queue.nix` declares that option pair
and, when set, has the harness unit inherit the two credentials by name.
Bare-id `LoadCredential=` is the terse form documented for inheriting
what the service manager received, and is non-fatal when the credential
is absent — which a hive whose publisher has not run yet needs.
No `HIVE_AGENT_OIDC_CA_FILE`: the meta flake already embeds the hive CA
and the swarm root into each container's trust store at build time, and
reqwest's rustls backend verifies against it.
Refs #3805
Ruled: swarm-bao-queue-agent.service must run before hive-c0re.service
and be wanted (not required) by it, so no agent container renders
ahead of the reader's attempt at its credential. An unreachable store
delays hive-c0re's start by the reader's own start-limit window rather
than failing it outright.
Refs #4314
The swarm queue's rendered settings never named max_payload, so it ran
on nats-server's upstream default of 1 MiB. That default is about to
be too small: the broker is going to carry agent terminal rows
(whole TermMsg bodies) published as complete messages rather than
split, and a publish over the limit does not truncate — the server
answers -ERR 'Maximum Payload Violation' and closes the connection,
dropping the row.
Set max_payload = 8388608 explicitly in the settings merge, with a
comment on what it bounds and what bounds it (max_pending, which
nats-server refuses to start past). Add a module-eval case that reads
the rendered container config so a future edit that drops or shadows
the key fails eval instead of surfacing as a dropped row in
production.
Refs #3805
The publisher on the authelia host has been writing
`secret/swarm/hives/<hive>/queue/agent` — the OIDC client secret agent
containers present to the swarm queue, plus the client id it belongs to —
and nothing read it. This is the reader: a oneshot `swarm-bao-queue-agent`
that logs in with the host's certificate and lands the two fields as two
files under `deploy.hive-controller.queue.agentCredentialDir`, the secret
`0600` and the client id `0644`.
Two files rather than one because that is the consumer's shape:
`swarm_queue_client::QueueConfig::from_env` takes the secret as a path and
the client id as a value, so the split here is what keeps the next slice
from parsing anything.
Same shape as the store's first reader, `glue-matrix-bao-token.nix` — a
cert login that fails loudly under `Restart=on-failure` because every state
it fails on is one a retry fixes, then reads that degrade quietly because no
retry turns "no value there" into a value. Unlike the matrix token there is
no local fallback and none is possible, so absent files mean this hive's
agents do not connect, which is the ordinary state of a swarm before the
publisher has run.
Nothing consumes the files yet and this unit is ordered `Before=` nothing.
The next slice bind-mounts them into agent containers through hive-c0re and
adds the ordering edge along with them.
Refs #3805
`swarm-bao-matrix-token` reads `secret/swarm/matrix/registration-token`
and is refused with `Code: 403 — permission denied`, measured on this
host at 16:17:49Z after a successful cert login.
path.rs makes every swarm path `swarm/<kind>/<name>/…` where Kind is a
closed set of four: agents, hives, services, controller. `matrix` sits
where a kind belongs, so policy.rs's read document — which emits exactly
`swarm/agents/*` and `swarm/hives/<hive>/*` — cannot cover it. The
module's own doc predicted this: "a misspelled kind is a 403 at provision
time rather than anything a compiler sees".
Moves the token to `swarm/hives/<hive>/matrix/registration-token`, built
through principal_prefix(Kind::Hive, …) like its per-hive sibling
queue::agent_client_path. The policy is untouched: render already grants
that prefix. mara chose this over widening the namespace.
The nix reader interpolates hyperhiveCfg.hiveName, with the no-fallback
reasoning glue-bao-tls.nix already gives at its own use of it.
path.rs's MOUNT doc justified itself by citing the old literal, which
this commit deletes; rewritten to cite the nix reader instead.
Scope: this makes the read reachable, not the value present. Nothing
writes that path yet, and a 403 says nothing about presence — the two
are separate findings and only the first is fixed here. No migration:
nothing ever wrote the old path and no read ever succeeded.
Gate: cargo fmt 0, cargo test -p swarm-secret-client 0 — 32 passed
against a 29-passed baseline with the change stashed, so the three new
tests are accounted for rather than assumed.
Refs #4308
Every agent container reports the hostname `nixos`, so every log line it
ships carries that as its `_HOSTNAME`. Measured: host `muede-lpt2`,
`hive-matrix` (declared as `containers.hive-matrix`) `hive-matrix`, and
`h-atlas` `nixos`.
nixpkgs sets the hostname in the merge function of the
`containers.<name>.config` option (nixos-containers.nix:524), so it reaches
a guest evaluated through that option and nothing else. Agent containers are
`nixos-container create --flake meta#<name>` — an independent `nixosSystem`
off the meta flake, which never evaluates that submodule.
nspawn also names a container's hostname after the machine by default; that
is ruled out as the source here, because `h-atlas`'s machine name is
`h-atlas` and it reports `nixos`.
The machine name rather than the logical one: `stats/otel_metrics.rs:345`
already labels metrics `container.name = "h-<name>"`, so the logical name
would make logs say `atlas` while metrics say `h-atlas` — a prefix transform
on every join between the two signals. Declarative containers and nspawn
both use the machine name too, so this is one rule with no exception for
agents.
The emission sits next to `hyperhive.user.name = name;`, which already
derives the container's unix user from the agent name; the hostname was the
one identity attr nobody wired.
Also drops the prose in swarm-otel.nix and module-eval.nix that explained
`_MACHINE_ID` by "every container is `nixos`" — that motivating example is
what this commit removes, and the argument for `_MACHINE_ID` never depended
on it.
Checked before editing: nothing in the tree assumes the hostname is
"nixos" (0 hits across *.rs and *.nix), and nothing reads the hostname at
runtime.
Gate: cargo fmt 0, clippy -D warnings 0, cargo test -p hive-c0re meta::
0 (23 run, 22 passed, 1 ignored), nix fmt 0 changed.
Refs #4304
The retry comment claimed "every retry is time the homeserver may spend
waiting". That is the conservative reading, not a measured one, and argus
is right that it is probably wrong: systemd ordering typically resolves
once a unit's first start job completes, success or failure, and an
auto-restart after that is not a new ordering-relevant job.
If that holds, the container proceeds after the first failed attempt --
same timing as today -- and what the retries buy is the token file being
correct within ~60s for whatever starts next, rather than this boot's
race getting a second chance.
Both readings justify the same tight bound, for different reasons, so the
comment now says the ordering behaviour is unverified instead of picking
one. An agent container cannot reach a systemd manager to settle it.
Refs #4303
