`swarm/agents/<agent>/bao-mtls` did not exist, and neither did any
per-agent identity at the secret store: `policy::agent_object_name`,
`render_agent` and `render_agent_with_queue` had been written and never
called outside their own tests. An agent's only "per-agent" secret today
is read under the HIVE's certificate, through a wide grant on
`swarm/agents/*` — so "per-agent" was presentational.
The swarm now mints the certificate, so no hive ever needs the capability
to mint one. `swarm-controller` is the service that does it: it already
logs in to the store, and its existing grant already covers exactly the
three objects written here (`create/update` on
`secret/data/swarm/agents/*`, `sys/policies/acl/hive-*` and
`auth/cert/certs/hive-*`). No new bao grant, and nothing co-located — a
cert-auth role pins its authority by value, per role, so the controller
issues from its own CA on its own host and pins that CA in the role it
writes. No existing role changes.
The mint node does not report success on a write. After publishing it
connects again, with the leaf it just issued and under the role it just
wrote, and reads the path back — so the policy, the role, the common name
and the leaf are exercised in production on every agent creation. A
certificate this code mints that the role this code writes will not accept
turns the job node red at creation time instead of surfacing later as an
agent container that cannot start.
`TriggerDeploy` gains an `after_any` edge on the mint, not `after_ok`: a
hive cannot pass down a certificate the swarm has not published, but a
host with no authority configured must still create agents exactly as it
does today.
The private key is generated in memory and never written to disk on the
controller — `SecretStore::connect_with_identity` takes the PEM the minter
is already holding, so nothing is written out purely to be logged in with.
Refs #4137
`services.hyperhive.agent.matrix.enable` was a second source of truth for
a fact the account set already carried: after ①-③ the hive-internal
`main` account is an ordinary `matrixAccounts` entry, so "does this agent
have matrix" and "does this agent have an account" were the same question
asked twice, with the boolean able to disagree.
The option is gone and a non-empty `matrixAccounts` now gates the daemon
unit, its token path-watcher and the injected `extraMcpServers.matrix`
entry.
That is only a real condition because `matrixAccounts.main` is itself
gated: it is declared when `matrix.url != null`, never unconditionally. A
`main` with no homeserver is an account the daemon can never log in as,
so declaring one always would have made the signal trivially true and
turned matrix on for every agent in every hive. With the URL gate, the
empty set is reachable exactly for an agent the hive gave no homeserver
and whose operator declared no account of its own — the state the old
`enable = false` expressed.
Assertions: "extras require enable" is deleted, having become the
definition of the thing it checked (an external-only account with its own
homeserver is now rendered rather than rejected). `main.tokenFile` stays
pinned, re-guarded on `? main` instead of on the flag, since `main` is
absent whenever the URL is null and an unguarded index would throw there.
Both spellings of the option get `mkRemovedOptionModule`, following
../host-modules/deploy.nix's registrationTokenFile pair rather than a
silent delete: the definition whose meaning changes is `false`, and left
undeclared it would be ignored and hand the agent the tools its operator
turned off. Failing the eval with the replacement spelling is the only
outcome that cannot.
module-eval gains the three arms — URL, nothing, external-only — with the
middle one carrying why it exists: it is the only thing in the suite that
would notice `main` becoming unconditional again.
Refs #4475
mara: "the subagent should already stamp the agent on the metric. so i want
to see count of distinct subagent names per agent name over time" — it
does. New timeseries panel-46 on the subagents tab: one line per parent
agent, counting distinct subagent label values with a turn in the trailing
$__rate_interval at each point. Rolling, not cumulative — can fall as well
as rise, same shape as any other rate()-derived panel here.
Placed at y:15, below where #4491's panel-45 (stat tile, y:11-15) sits once
merged, and numbered panel-46 rather than reusing panel-45 — argus caught
that this PR and #4491 both independently added a panel-45 in the same grid
slot from a shared unmerged base, which would have collided on whichever
merged second.
Narrower than a true concurrency gauge (subagents open at this exact
instant) — that needs hive-subagent-mcp to publish its own metric, which
nothing does today; noted as a separate, harder ask on #4488.
`matrixAccounts` is meant to be the agent's full account list, but the
hive-internal `main` account was outside it: the nix module emitted only
the extras and `hive-matrix-daemon` prepended a `main` it synthesized
from the per-agent paths, with the option schema forbidding the name
outright.
nix/agent-modules/matrix.nix now declares `main` itself, as an ordinary
entry under `matrix.enable`, from the state-dir paths the module already
used for its token path-watcher (now a shared `stateDir` binding) plus
`matrix.url`. The whole set, `main` included, is serialized to
HIVE_MATRIX_ACCOUNTS.
accounts::configured therefore synthesizes `main` only when the parsed
list carries none, and otherwise takes the declared one verbatim —
hoisting it to index 0, since the daemon reads index 0 as the primary
and nix serializes an attrset, so `main` sorts wherever its key falls.
Declared xor synthesized: an agent whose harness predates this entry
keeps working, a current one gets its own, and there is no arrangement
where `main` is duplicated or missing.
The reserved-name assertion is replaced rather than dropped: the name
must now be legal (the module uses it), but `main`'s tokenFile stays
pinned to `<state>/matrix-token`, since hive-c0re provisions the
hive-internal token there and nowhere else — a retarget would evaluate
fine and then never restore. The other two fields are mkDefault and free
to override.
Refs #4475
Bargauge panels display series in the order the datasource returns them —
Grafana has no native sort for that (long-standing upstream gap, grafana#17245).
The reliable fix is sorting at the PromQL level with sort_desc() on an
instant vector, which forge.json's 'Issues by label' panel already does.
Applies the same pattern to the other 14 multi-series bargauge panels across
agents.json, claude-usage.json, and authelia.json (the latter two also
needed instant: true added, since sort_desc only works on instant vectors).
logstore.json's two VictoriaLogs-backed bargauge panels use LogsQL, not
PromQL, and are left out of this pass — LogsQL's sort-after-stats has a
documented ordering bug in some versions; needs its own verification.
mara: "show subagent count in time range" — the existing 'Subagent turns'
tile answers volume (how many turns), not this (how many differently-named
subagents ran at all). New panel-45 counts subagent label values with at
least one turn in the selected range.
Every per-agent harness option lived at the top-level `hyperhive.*` while
the host tier has always been `services.hyperhive.*`. Move all 52 agent-tier
option leaves (33 top-level names across 16 modules) to
`services.hyperhive.agent.*`, repoint every read, and keep existing agent
configs evaluating through one `mkRenamedOptionModule` per old leaf path in
the new nix/agent-modules/renamed-options.nix.
The shims are per leaf rather than per namespace: `user`, `mcp`, `otel`,
`queue`, `docs`, `forge`, `frontend`, `github`, `gui`, `logs`, `matrix` and
`cargo` are plain attrsets of declarations, not submodule-typed options, so
a parent-path rename would not reach their children. Three read-only
options (`frontend.mergedDist`, `queue.clientIdFile`,
`queue.clientSecretFile`) deliberately get no shim — a rename contributes a
definition, which a read-only option refuses; the exclusions are commented
in place.
Refs #4473
Same slice, same fact: socketPath lost its pre-rename path, so only three
old paths are defined in the fixture now, not four. Missed this sentence
while fixing the count three lines below it.
readOnly means socketPath can only ever hold the default, so the fixture's
old custom value can no longer reach the unit and the check.module-eval
assertion for it was failing. Assert the default instead of deleting the
check -- it still catches the env var vanishing or being misspelled, which
deleting it would not.
Also corrects the block's stale 'FOUR movers' count: socketPath lost its
pre-rename shim in the same slice, so only three of the asserted values
are still rename-shim movers.
readOnly makes a bad socketPath inexpressible rather than diagnosed
after the fact. Since readOnly rejects any definition including one
arriving through a rename shim, drop the deploy.nix rename entry for
it and the fixture line exercising it, and update the controllerOldPath
comment's shim count from seven to six.
Refs #4208
The socket is 0666 (nginx needs to connect(2) as a different user), so its
directory is the only thing bounding what the gateway's nginx can reach —
the crate's README already says so. A unit test pins DEFAULT_SOCKET, but
that only catches a developer editing the const; an operator setting the
documented socketPath option to a shared dir (notably /run/hyperhive,
which holds the host admin socket, or /run/hive) went unenforced.
Denylist rather than the general "not any other module's socket dir"
form — narrower, but it's what the README names and lands without pulling
in other modules' constants.
Refs #4208
First slice of the move off the hand-rolled swarm services sub-CA: the
store gains the three objects the eventual minter needs, and nothing
else.
- a `pki` secrets mount, enabled ask-first off the same `bao secrets
list` snapshot the kv-v2 mount beside it uses, so a rebuild that finds
it mounted does nothing;
- a `swarm-services` role on it, `allowed_domains` read straight out of
`swarm.serviceDomains` — the same swarm-tier list swarm-ca.nix
name-constrains its sub-CA to and hive-tls.nix carries as the leaf's
SANs — with subdomains, localhost, globs and IP SANs all off;
- a `swarm-services-issuer` policy granting `update` on
`pki/issue/swarm-services`.
Inert: the mount has no issuer generated into it, the role therefore
cannot issue, and no login role attaches the policy, so no token in the
swarm carries it. No consumer changes; swarm-ca.nix still mints the
services sub-CA exactly as before. Reversible with
`bao secrets disable pki`.
The cert-auth role that attaches the policy waits for the leaf carrying
its CN, which glue-bao-tls.nix mints — a later step.
Refs #4256
An agent can reach VictoriaLogs only through the gateway, and since the
machine query route landed the way to read it has been to hand-roll a
client_credentials token request and a curl, per query. This is the CLI
that closes that: `swarm-logs query '<LogsQL>'`, matched log lines on
stdout, so the answer pipes into grep like any other command's.
Built to the plan posted on the tracker thread: own crate, own
docs/tools reference generated off the clap tree, `query` as the one
verb, and the JSON error body surfaced on a non-200 rather than
swallowed. No `tail`: streaming is a different endpoint with a different
response shape, and folding it in here would be a fatter scope than the
ask.
Minting the token is NOT implemented here — swarm-queue-client already
owns the client_credentials request, its error type and its CA handling,
and a token-endpoint fix has to be findable in one place. What this crate
adds is the agent-shaped half: the client id arrives as a *file* beside
the secret, so nothing outside nix/agent-modules/queue.nix spells
`hive-<name>-agent` twice. That is the same problem hive-agent's
swarm_queue module solves, and swarm-logs/src/auth.rs is its `decide`
restated over this binary's inputs.
⚠️ The plan named one thing to verify empirically before calling the auth
settled: whether authelia's bearer policy for the logs vhost accepts the
agent client's audience. Measured from inside a container: it does not.
The client minted a token fine but with `aud: []` and `scp: []`, asking
for the logs URL as an audience answered `invalid_target`, and presenting
the audience-less token to the gateway answered a bare 401. So
swarm-authelia.nix's agentClients gains `authelia.bearer.authz` and the
query URL as a second audience — authelia authorises a bearer token by
the URL being requested, and that URL is now one binding read by three
places rather than three spellings of one address.
The URL reaches an agent the same way its queue coordinates do: computed
on the host (a container cannot derive a gateway address), forwarded by
hive_c0re::meta into the container's option set, and consumed by a new
agent module that installs the binary *wrapped* with its coordinates —
the shape swarm-controller.nix installs swarmctl in. Gated on the queue
credential as well as on the URL: a binary that can only answer 401 is
worse than no binary, because an agent reads a 401 as "no logs", which is
the exact confusion the store's machine route was added to end.
A unicast DHCP renewal reply currently reaches dhcpcd only by matching
the firewall's ESTABLISHED,RELATED conntrack rule against the outbound
request. When that conntrack entry has already expired the reply is
dropped silently, with no log line anywhere. The client's broadcast
paths (DISCOVER, rebind) bypass netfilter entirely via a raw BPF
socket and never depend on this state — only the unicast renewal path
does.
This removes that dependency by accepting DHCP client traffic
unconditionally, gated on the firewall being enabled at all. It does
not identify or claim to fix the cause of any particular observed
renewal failure.
Refs #3389
Adds services.hyperhive.deploy.swarm-controller.matrixHomeserverUrl,
threaded to the daemon as SWARM_CONTROLLER_MATRIX_HOMESERVER_URL, and a
Rust helper (homeserver_or_configured_default) that lets a caller-supplied
homeserver keep overriding it. Config plumbing only: put_matrix_account
does not call the helper yet, so this is a no-op for every current caller.
Refs #4345
The subagent daemon put `model` straight onto claude's argv with no
validation, so an agent could spawn nested sessions on any model the
operator had deliberately kept off its harness. Forward the existing
`hyperhive.availableModels` onto the daemon unit as
HIVE_AVAILABLE_MODELS (same rail `HIVE_TOOL_GROUPS` uses) and check
`start`/`continue` against it before building the config.
Default open: an absent var restricts nothing, so an agent deployed
before this keeps working. An omitted `model` is always allowed — it
lets claude pick its own default rather than naming one.
Refs #4436
Adds approle to the auth methods enabled during swarm-bao bootstrap,
using the same idempotent check pattern as cert. No role or policy
is configured yet — those are handled in later slices.
Refs #4386
`glue-matrix-bao-token.nix` has read
`secret/swarm/hives/<hive>/matrix/appservice-token` since it landed, but
nothing ever wrote that path. The store was empty in every deployment, so
every read degraded to "keep what activation minted" and each hive stayed
the origin of a value the swarm has to agree on — two hives never
converged.
`swarm-secret-publish` is now the producer. It already holds a store
identity, already writes under the hive prefix, and already runs per
hive in the roster, so the mint is a third loop beside the two OIDC
copies rather than a second shape of this unit.
Idempotence comes from a record of its own, not from the store: this
principal is granted `create`/`update` with no `read`, so it cannot ask
whether a hive already has a token. It keeps what it minted under
`StateDirectory=` (0700 dir, 0600 file) and mints only when that file is
missing or empty; the `put` runs every time, because re-putting the same
bytes changes nothing for a reader while a mint whose publish failed must
not be left as a token this host holds and no hive can reach.
The token never becomes a nix literal and never reaches argv: the mint
redirects into a file, and the publish hands bao `value=@<path>` so bao
opens it itself — the same handling the OIDC loops use.
`hive-matrix.nix`'s activation mint stays as the genuine first-boot
fallback. It already fires only when the token file is absent, so it
cannot clobber a value the store delivered; `hs_token` has no swarm half
and is still minted there for real.
Refs #4402
Nothing reads it any more: hive-c0re creates accounts as the hive's
appservice, so the mint, the host file, the bind mount, the
`LoadCredential` entry and tuwunel's `registration_token_file` all go.
⚠️ `allow_registration` has to go to `false` in the same change, and not
as hardening. tuwunel refuses to START when registration is allowed with
no token configured — it demands
`yes_i_am_very_very_sure_…_open_registration_…` instead — so dropping the
token and leaving the flag true is not a lax homeserver, it is one that
does not boot. The flag is checked only for requests arriving without an
appservice token, so hive-c0re provisions exactly as before and everyone
else is refused outright.
The swarm secret store keeps its role, repointed at the credential that
replaced the token (`swarm/hives/<hive>/matrix/appservice-token`). Its
unit now also re-runs hive-matrix's own registration renderer after
writing the file: the token is half an agreement, and a registration
still naming the previous value authenticates nobody. The renderer is
shared through an internal option rather than copied, so the
registration's shape has one home.
Both spellings of `registrationTokenFile` become
`mkRemovedOptionModule` with a message naming what replaced them. A hive
that never set the option — the default — is unaffected; one that pinned
it fails to evaluate with instructions instead of a silent no-op.
An upgraded hive needs no intervention: the activation script has both
halves in place before the homeserver restarts, existing agents keep the
tokens their devices already hold, and the old token file is left on
disk read by nothing. docs/integrations/matrix.md spells the path out.
Refs #4402
The hive creates matrix accounts with a shared registration token today,
which means the secret that authorises account creation has to reach both
hive-c0re and tuwunel and stay identical in both. An appservice
registration replaces that with an identity: one token that says "this is
the hive's provisioner", carried in an ordinary credential file.
`url = null`, so nothing is served and no daemon is introduced — with no
URL the homeserver never calls out, and the registration exists purely to
give the `as_token` meaning.
Delivered through `appservice_dir` rather than a `[global.appservice.<id>]`
stanza, because a stanza's `as_token` would be a nix literal and a nix
literal is a world-readable store path. The file is minted and rendered by
a host activation script, bind-mounted into the container, and handed to
the homeserver by `LoadCredential` — the same two steps the registration
token and the OIDC client secret already take, and for the same reason
(0600 root on the host, `DynamicUser=true` in the container).
`sender_localpart` is the hive admin account on purpose: loading a
registration creates its sender user on a zero-user database inside
`Services::start()`, and the `admin_execute` promotion runs after that and
still before the HTTP listener accepts anything. So a fresh homeserver has
a joined, power-level-100 admin on its first boot without anyone having
won the first-registered-user grant. `admin_execute_errors_ignore` is set
because a failing startup command otherwise aborts startup outright.
Nothing reads the registration yet — hive-c0re still provisions through
the registration token, which is untouched here.
Refs #4402
`build_config` now resolves a subagent's `--tools` from
`HIVE_TOOL_GROUPS`, the same var the harness resolves its own session
from — but the meta renderer writes that var onto the `hive-agent` unit
alone (`systemd.services.${service}.environment`), and the subagent
daemon is a separate unit. It would therefore have resolved the default
groups no matter what the agent was actually granted.
That direction is safe — the default groups add no built-ins, so the
resolution is a subset of the parent's either way, never a superset — but
it isn't what the code says it does: an agent granted `web_tools` would
spawn subagents silently without `WebFetch`/`WebSearch`, and the "same
set as the parent" property would be true only for agents whose groups
happen not to matter.
Forward the var onto the daemon's unit, read off the harness unit rather
than re-derived, so there is one place it is decided. Absent stays
absent: `or null`, which systemd drops from the unit, leaving the daemon
the same fallback the harness would take.
Refs #4416
`goal_reached`/`need_help` took the session name as a tool argument, so
identity was an assertion by the caller and the only guard on it was
`occupancy()` — "does that name have a turn in flight", which two
concurrently running siblings both satisfy for each other. A subagent
could stop its sibling's run by naming it.
Identity moves into the URL. Each spawned run is minted an unguessable
token (`Uuid::new_v4`, the OS CSPRNG), the URL carrying it goes into that
one subagent's own `--mcp-config`, and the route resolves it back to a
session before dispatching to a handler bound to that session. Neither
tool takes a `name` any more: a subagent has no field in which to name a
sibling, and a sibling's name — which a brief may well mention — is not a
token.
One route with a path parameter, not a route per session: the `Router` is
built once at startup and subagents come and go for the daemon's whole
life. An unminted or revoked token gets a bare 404, the same answer either
way, so nothing enumerates. A run's token is revoked when the run ends
(`finish_turn`) or when a call never reached a spawn.
Two things fall out of that:
- the config file becomes one per session. A single shared path was
already a race between two `start`s; with a per-session URL in it, the
loser would read the winner's identity.
- `occupancy()` stops being the identity guard and is gone from the signal
path entirely rather than kept "just in case" — a revoked token can't
reach it, and it never answered the question it was standing in for.
It still backs `status`, which is what it was always actually for.
Refs #4403
Refs #4413
`start` takes an optional `goal`. With one set a session stops being a
single turn: when a turn ends and nothing has said to stop, the daemon
spawns another turn re-prompting the subagent toward that goal, up to
`max_turns` (default 5, per-session). Without a goal nothing changes —
one turn, one todo, same as before.
Four things end a run, each recorded distinctly and reported by `status`:
the turn ending with no goal, `goal_reached`, `need_help`, and the turn
cap. The last says so out loud rather than stopping quietly — the todo
states the harness limit was reached and the goal was never reported
reached. Every stop extends the done message rather than replacing it,
and lands in the session's report file when it has one. The path is
never inferred: it comes from `start`'s `report_file` or from the
subagent naming where it wrote.
`goal_reached` and `need_help` are the subagent's own, served on a second
route (`/signal/mcp`) that carries those two tools and nothing else, so
reporting on a run can't become starting one. `goal_reached` is built as
a label, never a gate: it is self-reported by a subagent that has just
been re-prompted with "you haven't reached the goal", which is exactly
the incentive to claim it — the same failure class as a build report
asserting the tests pass. Every surface that renders it says so.
`need_help` is the blocking signal, and shows in `status` as its own
state so a parent polling it sees the block without reading a file.
`status` also carries `turn N of M`: with 4330's last-event age, that
separates working from wedged from out of turns off one answer.
Two bugs the new tests caught: a `tokio::fs::File` was dropped without
flushing, so the report line was written to nothing, and the plain idle
answer dropped the turn counter.
Also documents `await_resume`'s third case — a closed channel with no
send, which fails open the same as `Underway` — per argus on #4411.
Refs #4403
`swarmctl agent create <name> --hive <hive>` POSTs `/api/agents` to
swarm-controller over the daemon's unix socket and prints the queued
job's node id.
It deliberately does not wait. The endpoint queues a DAG whose last node
*publishes* a deploy message; the hive's `hive-c0re` then converges on
its own clock, out of the controller's sight. So even a fully settled
graph would not mean the agent is up, and there is nothing this CLI
could wait for that would let it claim otherwise. Printing the id is
exactly what the response says and all of what it says.
Transport is a bare hyper HTTP/1.1 client handshaked onto a tokio
`UnixStream` via `hyper_util::rt::TokioIo` — the same crate family
`hivectl/src/watch.rs` and `hive-agent/src/web_ui/proxy.rs` already use,
all of it already workspace-pinned. The request/response shapes are a
local mirror rather than a shared crate: the controller's own types are
private to its binary and this crate does not link it, the same
separation `hivectl` keeps from `hive-c0re`.
Errors are reduced to one actionable line — the controller answers
RFC 9457 problem+json, so an unknown `--hive` reaches the operator as
the roster of hives that would have worked rather than a body dump.
Response `warnings` are printed when non-empty.
The nix module wraps the binary with `SWARM_CONTROLLER_SOCKET`, read
from the same `socketPath` the daemon binds.
Refs #4399
swarm-controller's `InitAgentConfigRepo` node already covers config-repo
creation, so this deletes a duplicate rather than a capability; old
`init_config` rows are skipped by `collect_lenient` with no migration, by
operator decision.
Refs #4398
The swarm can already tell whether an agent is alive — the `agent-status`
KV bucket republishes once a minute — but not what it is doing right now.
A header bar wants the second thing, and a minute-old answer to "is this
agent thinking" is the wrong answer most of the time it is read.
`hive-agent` now publishes a turn-state header to
`$SWARM.agent-state.<hive>.<agent>`, a core subject beside the terminal
rows it already sends. It goes out **on transition, not on a timer**: the
publisher watches the event bus, rebuilds the header, and sends only when
the serialised result differs from the last one it sent — so a second
periodic writer, which is the problem this exists to fix, is not what
replaces the bucket.
The payload is the published contract a swarm-level renderer is written
against, so the test asserts on the serialised JSON keys rather than on
Rust field names. Two fields deliberately depart from the per-agent web
UI's `StateSnapshot`: `turn_state_since` is an ISO 8601 UTC string rather
than unix seconds, matching the sibling `$SWARM.term` subject's stamp, and
`agent_state` carries the swarm's own `AgentState` vocabulary rather than
a `paused` boolean, so a reader can compare actual against wanted without
translating. `turn_state` and `agent_state` stay two separate fields:
neither vocabulary contains the other's values.
Swarm-side, `GET /api/agents/{name}/state/stream` relays the subject as
SSE, resolving the agent's hive at request time exactly as the terminal
stream does and passing the bytes through without parsing them.
The broker grant is a second `--agent-publish-subject` rather than a
widening of the existing one, so the terminal family and the header family
stay independently revocable, and a `module-eval` arm pins the rendered
flag and its argument together — the doubled dollar included, since a
single one expands to nothing in `ExecStart` and yields a grant that
matches nothing.
Refs #3802
otelNoIdentity points swarm.authelia.url at a remote authelia while
deploy.authelia.enable stays false, so ssoLocal is false and nothing
mkDefaults the forge's deploy.forgejo.sso.clientSecretFile. The forge
module's config is gated only on services.hyperhive.enable (there is
no deploy.forgejo.enable to opt out of), so its SSO assertion fired
against every fixture that lacks the secret — entirely orthogonal to
what this fixture tests. Supply the by-hand path, mirroring
otelRemoteAuthelia's own hand-delivered secret.
The swarm.otel.journaldUnits = [ "nginx" ] line added in the previous
commit targeted an assertion that was never firing: journaldUnits
already defaults non-empty, so the effective list just contained
"nginx" twice. Removed as redundant.
Refs #4374
Refs #4374
- otelNoIdentity: name journaldUnits so the fixture trips the store-
identity path it's testing instead of swarm-otel's unrelated
journald-safety assertion (an empty list with log collection on is
refused as "collect everything", not "collect nothing").
- otelNoStores: give it a bao client identity. The secret gate moved
from deployCfg.authelia.enable to a real client cert/key pair, so a
fixture meaning "no telemetry stores" now needs its own secret
identity to keep exercising the exporter/authenticator wiring it
was written for.
- docs/swarm/secrets.md: two vale fixes — a contraction, and drop a
condescending "simply".
The swarm collector's OIDC client secret only existed where authelia
did: `swarm-otel-oidc-secret.service` copied the minted plaintext out
of authelia's container tree, reachable only because the two share a
host's network namespace. A swarm that placed authelia elsewhere
delivered nothing, and the option's own description said so —
"a deployment that places authelia elsewhere points this at a file it
delivers itself." Same gap as #3853 and #4234, and this is the
swarm-otel twin of #4234's fix for Grafana.
Mirrors PR #4361 (Grafana) almost exactly:
- `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, in every
deployment, replacing the co-located copy unit outright — one
delivery route, not two, per the ruling that landed under #4234.
- Client registration moved out of `swarm-otel.nix`'s own `config`
block (gated on this host running the collector) into
`glue-swarm-otel-oidc-client.nix` (gated on this host running
authelia), the same split `glue-grafana-oidc-client.nix` made. It
was broken the same way: a split deployment registered the client
nowhere at all, so authelia never minted a secret for the publisher
to send on.
- The publisher's `services` prefix (write grant in `swarm-bao.nix`,
hive read grant in `policy::render`) already covers any service's
path — nothing to add there. `swarm-secret-publisher.nix` only grew
`serviceClientIds` by one entry.
One judgement call, stated rather than buried: the store-reading unit
renders only where this host holds a client identity
(`deploy.bao.clientCertFile`/`clientKeyFile`), rather than asserting
it the way `swarm-grafana.nix` does. Grafana's local login form is
disabled unconditionally, so a Grafana with no OIDC secret has no way
in at all — that earns a hard refusal. This collector without a
credential still receives every hive's telemetry; only its own pushes
to the stores go out unauthenticated and get refused there, an
already-supported degrade the module's own `haveCollectorSecret` flag
named before this change. So the reading unit follows the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use
for their own optional readers: no unit when the identity is absent,
not a build refusal.
Fixtures mirror #4361's: `otelBaoWithAuthelia`/`otelBaoRemoteAuthelia`
are the positive pair (co-located and split, both reading through the
store), `otelNoIdentity` is the negative — no reading unit, no
assertion firing, `clientSecretFile` left null.
Refs #4258
The v2-conversion script carried over the local-test dashboard's own
title ('agents-full-test') instead of the real one, since it read
spec.title straight from the generated ground-truth file without
overriding it back. Caught on a diff review, not by CI — nothing
schema-level flags a wrong-but-valid string.
Confirmed empirically (real Grafana 13.0.7, a file-based provisioner
identical in shape to swarm-grafana.nix's) that the deployed version
provisions v2-schema dashboards natively, tabs included, no migration
path needed. Supersedes the row-based split from the previous commit
on this branch: same four groups (activity/cost/health/subagents), same
17 panels with identical content (targets, fieldConfig, options,
descriptions all carried over unchanged — only the document shape
around them moved from a flat panels array + row wrapper panels to the
v2 schema's spec.elements map + spec.layout TabsLayout referencing them
by id), now rendered as real browser tabs instead of accordion rows.
Generated the panel-content half of this file mechanically rather than
hand-authoring the new schema: posted the pre-conversion (v1, flat,
17-panel) dashboard to a local Grafana 13.0.7 and read it back through
the v2beta1 API, which is Grafana's own conversion of the exact same
panel content into the v2 element shape — then replaced only the
generated layout (a flat GridLayout) with a hand-authored TabsLayout
grouping the same element references into the four tabs. Verified the
result end-to-end against the same local instance: file-provisioned
(confirmed via the loaded object's own grafana.app/managedBy annotation),
and a real screenshot of each of the four tabs rendering with its
correct panels.
Cost, health (cpu/mem/disk), and subagents, plus an activity row for the
panels that fit none of the three (agent/turn counts, lines-of-code and
CLAUDE.md-size trends). Rows rather than tabs: Grafana's tab layout is a
v2/Scenes-schema feature, and this deployment's file-based provisioner
doesn't reliably load v2-schema dashboards, so classic-schema collapsible
rows are the closest available equivalent — one section open, the rest
collapsed. activity stays expanded so the dashboard opens on something
useful instead of a wall of accordion headers.
An agent container cannot dial the store's loopback listener: the bridge
to-loopback DROP rule is there precisely to stop that, and the store
authenticates every reader by client certificate, so the usual answer —
a gateway vhost — is the one shape that cannot work. A terminating proxy
strips the certificate and bao sees nginx as the client for every hive.
nginx's stream module does not terminate. `ssl_preread` reads the SNI off
the ClientHello and splices the rest of the connection through byte for
byte, so bao completes the handshake itself and authenticates the client
it actually has. That is the no-vhost rule kept, not bent.
The listener binds the bridge IP rather than every address, because bao
already holds `127.0.0.1:<port>` in the same netns and a wildcard bind
there is EADDRINUSE — nginx would fail to start, taking the gateway with
it. Nothing moves as a result: the name already resolves two ways, so a
host-side reader still goes straight to loopback and an agent goes
through the passthrough, both on one `BAO_ADDR`.
Renders only inside the store's own `deploy.bao.enable` region; a host
that runs no store grows no listener and opens no port.
Per-agent certificates and per-agent policy are separate work.
Refs #4386
validateConfigFile runs `otelcol validate` at nix build time, in a pure
sandbox where systemd's StateDirectory= has not run yet, so the
directory named by extensions.file_storage.directory does not exist.
Setting create_directory = true lets the validator create it itself,
matching what happens at runtime once StateDirectory= has acted.
Refs #4375
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
autheliaCfg/forgeCfg/vmCfg/vlCfg were local re-exports of
hyperhiveCfg.swarm.<subpath> — a shape that hides the full option
path from grep, the same hazard #4356 fixed for the tls alias.
Removed the six let-bindings and spelled the full
hyperhiveCfg.swarm.<subpath>.<field> path at every use site instead.
None of the read fields (url, machine, hiveClientPrefix,
agentClientSuffix, domain, port) sit on the old side of a
mkRenamedOptionModule in deploy.nix, so inlining the alias's own
current path is correct as-is.
Refs #4363
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous commit left two delivery paths and a three-way gate:
`swarm-grafana-oidc-secret.service` copied authelia's minted plaintext
out of its host tree wherever the two were co-located,
`swarm-bao-grafana-oidc.service` read the same value from the swarm
secret store wherever they were not, and `ssoConfigured && (ssoLocal ||
haveClientIdentity)` decided whether Grafana got an OIDC block at all.
Delete the co-located path. The store reader is now THE delivery unit,
in every deployment — the publisher on authelia's host writes
`swarm/services/<id>/oidc/client` whether the reader is a network away
or in the container next door. The ruling behind it: the store exists so
a host holds ONE out-of-band secret, its client certificate, and reads
everything else with it. Skipping the store when the producer happens to
be local saves a round trip and costs a second delivery unit, a second
way for the file to be wrong, and a gate to choose between them.
The gate goes too, and both of its questions become assertions, scoped
to hosts that run Grafana:
- `swarm.authelia.url` must be set. `auth.disable_login_form` is
unconditional — Grafana ships an admin/admin account on a public
vhost — so dropping the OIDC block when the swarm names no IdP
produced a container with no SSO and no password box, silently. An
eval-time refusal naming the option is the only report that reaches
anyone, the shape swarm-nats.nix already uses for the same option.
- `deploy.bao.clientCertFile` / `clientKeyFile` must be set. This
replaces a warning that nothing reads back, and its message names both
options and where the leaf comes from.
Fixtures follow. `grafanaWithAuthelia` gains the cert pair, because a
co-located host is a store reader like any other. The old
`grafanaRemoteAutheliaNoIdentity` is kept rather than deleted, renamed
`grafanaNoIdentity`: the shape is still reachable, only its deliverable
changed from silence to a refusal, and an arm now reads that refusal
back. Its mirror `grafanaNoSso` covers the other assertion, each fixture
wrong in exactly one way so an arm can name which refusal fired. Every
positive keeps an explicit negative — the one-delivery-unit arm asserts
the deleted unit is absent in both topologies rather than merely that
the store reader is present.
Refs #4234
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Grafana's OIDC client secret only existed where authelia did. One
`ssoLocal` gate — `grafana.enable && authelia.enable` — decided the
client registration, the minted secret's delivery and the whole
`auth.generic_oauth` block, so a swarm whose authelia runs on another
host got Grafana with no SSO wiring at all. The local login form is
disabled unconditionally, so that is no way in.
Split the one gate into the two questions it was conflating:
- `ssoConfigured` — does this SWARM have an identity provider
(`swarm.authelia.url`, which is swarm-wide and whose own description
makes null mean "no SSO configured"). With a delivery route present
this is what emits Grafana's OIDC block.
- `ssoLocal` — is authelia on THIS host, now spelled as the forge and
matrix modules spell it. It decides only which unit delivers the
secret.
Where authelia is elsewhere, `swarm-bao-grafana-oidc.service` reads the
secret from the swarm secret store, shaped after
glue-queue-agent-credential.nix: cert login fails loudly because a retry
fixes every state it fails on, the read degrades quietly because no
retry turns "no value there" into a value, and nothing writes a
stand-in. The producer is the publisher that already runs on authelia's
host, which gains the swarm's service clients beside the per-hive ones
at `swarm/services/<id>/oidc/client` — with the write grant in
swarm-bao.nix and the hive read grant in `policy::render` to match.
Registration moved to glue-grafana-oidc-client.nix. It has to be
declared where authelia's config is rendered, and swarm-grafana.nix's
config block hangs off this host running Grafana.
Two judgement calls stated rather than buried: a hive's read policy now
grants the whole `services` prefix, because a service's path names the
service and nothing swarm-wide records which hive runs it (cost recorded
in docs/trust-boundary/security.md); and the client is registered on any
authelia host, because no swarm-wide "this swarm has a Grafana" fact
exists to gate it on.
Refs #4234
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The journald receiver runs with --lines=0, so every collector start
only ships what's written after it starts, and a restart silently
loses whatever landed while it was down. Point it at a file_storage
extension so the read cursor survives a restart; start_at stays at
its 'end' default since the cursor now covers everything after the
first run.
Refs #3818
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
hive-subagent-mcp already stamps a subagent=<name> resource attribute
(alongside the parent's own agent=<name>) on every claude_code.* sample
a spawned subagent turn emits -- the label already existed, it just
wasn't plotted anywhere. Adds five panels: subagent cost, subagent
turns, subagent cost share (of all cost), cost by subagent name, and
cost by parent agent. No backend changes needed.
The dashboard already templated an $agent variable and broke turns and
active-time out by agent, but cost and token totals only split by
model, effort and query_source -- there was no panel answering "how
much did each agent spend/use", the aggregate-by-agent view mara asked
for.
Three cases beside the existing ingest ones, since the read route has the
same trap and one property of its own.
- the route exists and carries `auth_request`;
- it has no login fallback — with the browser location as the positive
control, so a pass means the two routes differ rather than that
`error_page` and the login host are absent from the whole vhost;
- it injects no filter, on `extraConfig` and on `proxyPass` both, because
VictoriaLogs takes its filters as request parameters and those ride an
upstream URI as easily as a directive. A filter arriving later is then a
visible diff here rather than a quiet change of rule.
Each absence arm leads with a clause that proves the location resolved and
that `hasInfix` finds what is really in that string, so "not there" cannot
be read off an unreadable path.
Refs #3870
An agent can reach the log store only through the gateway, and the only
location that exists for a reader is `/`, which is the operator's browser
route. That route ends in `error_page 401 =302`, and an unauthenticated
caller which follows the redirect gets authelia's login page as HTTP 200
with an HTML body — so a client that checks the status code records a query
that succeeded and matched no logs. Measured on the live gateway; it is the
root cause behind four operator round-trips already.
So the query API gets its own location, `^~ /select/logsql/`, with bare
`auth_request` and no login fallback: an unauthenticated caller gets a 401
it cannot mistake for an empty result. `^~` keeps it ahead of the `/`
catch-all and of any regex location added later, and it stops short of
`/select/vmui/`, which is the browser UI and stays on the browser route.
The query is forwarded unmodified — no filter parameter is injected, so any
authenticated caller reads the whole swarm's logs. That is the rule mara
set: read permissions are a later thing, and this location is where one
attaches when it exists.
Refs #3870
systemd's default OOMPolicy=stop tears the whole unit down the moment the
kernel kills any process in its cgroup, so a single over-large subagent
takes the daemon and every sibling session with it — measured on a live
agent, both units show OOMPolicy=stop today, which is exactly the "every
live subagent session was cut mid-turn" symptom.
That blast radius is also what would make the preceding commit a bad
trade: deliberately putting this unit first in the OOM queue is only an
improvement if losing one subagent isn't losing all of them. OOMPolicy=
continue scopes the loss to the process the kernel actually chose, and
leaves the daemon alive to report the kill instead of vanishing and being
restarted with an empty session map.
Refs #4316
Both units ran at OOMScoreAdjust=0, so under container memory pressure the
kernel picked purely on footprint — and the agent's own claude is often
the fattest process in the container, which means the session supervising
the work died before the work did.
The sign is the load-bearing part and is easy to invert: a HIGHER
OOMScoreAdjust means MORE likely to be killed, because the kernel adds it
to the badness score it derives from the process's memory footprint and
then kills the highest scorer. So hive-subagent-daemon gets +500 (first in
line) and hive-agent gets -500 (last in line). Written backwards this
makes the reported bug worse rather than better, so module-eval pins the
order as an inequality.
Both values are inherited by the nested claude each unit spawns as a
child, so ordering the units orders the sessions underneath them. -500
rather than -1000 on the harness: fully exempting it would leave the
kernel nothing to kill in a container whose only large process is the
harness.
Refs #4316
