Watch
0
0
Fork
You've already forked hyperhive
0
Commit graph

5,493 commits

Author SHA1 Message Date
iris
93ae435287 swarm-ui: drop an unused export and test the classifier settings logic
The runtime list was exported but only its type was used, so the type is
now written out. The form's pure logic (draft, checks, request body,
change detection) gets tests under node's built-in runner, run by the
existing swarm-ui check.
2026-10-14 16:20:37 +02:00
iris
a04096ea19 swarm-ui: choose the review classifier's runtime on the settings page
The classifier settings page showed model and effort only, so the API-key
or ACP runtime could be switched only through a raw API call. It now has a
runtime select, and under ACP the provider's URL, model and prices. A PUT
carries the provider only on the ACP runtime, so the stored record stays
for a switch back. Closes #5249
2026-10-14 16:20:37 +02:00
atlas
739c35757a hive-matrix-mcp: test the retry path and the once-only todo clear
The startup todo wipe runs once in main, before any sync loop, so a
secondary that comes up later cannot clear the todos the running
accounts' sweeps already deduped. Nothing tested that.

main now reads HIVE_AGENT_SOCKET once at startup and hands the path to
wake::clear_all_todos, the only way left to wipe the whole set;
send_todo_clear clears one key only. bring_up_account is split into
restore_account (everything that talks to the network) and the
synchronous account_sync_loop, and the retry path restores with the
former and builds the loop itself. Neither the loop nor the retry path
is given the socket.

The new test runs a late secondary's retry and its sync loop against a
listener standing in for the harness socket, after startup's wipe, and
asserts the harness saw exactly one wipe. It pins that seam, not every
place in restore_account a wipe could be added.
2026-10-14 16:20:04 +02:00
atlas
bd6009ff33 hive-matrix-mcp: test the secondary retry loop and the registry lock
The retry loop's body was only reachable through bring_up_account, which
needs a live homeserver, so nothing exercised what happens when a down
secondary comes up or fails, or that the Registry's new RwLock is never
held across the network call. The unit tests covered the pure decision
(DownAccounts, retry_delay) only.

retry_down now delegates to retry_loop / retry_once, which take the
bring-up step and the snapshot path as parameters; retry_down passes
bring_up_account and the real path, so behaviour is unchanged. Tests use
offline matrix-sdk clients and fake bring-ups to show that:
- a secondary that comes up is inserted, leaves the down set, has its
  sync loop sent on and is published in the snapshot, while the primary
  and the other accounts stay the same clients;
- one that fails (error or no token) stays down, is not published, and
  waits twice as long next time; one that is not yet due is not tried;
- readers are not blocked while a bring-up is pending;
- inserts under four busy reader threads neither deadlock nor lose an
  account;
- the loop, under paused time, retries at 2, 6, 14 and 30 minutes.

tokio's test-util feature is added as a dev-dependency for the last.
2026-10-14 16:20:04 +02:00
atlas
53049962b8 hive-matrix-mcp: quieter retries, unique snapshot tmp file, accurate comments
Audit follow-ups on the secondary-account retry.

The accounts snapshot is written by the heartbeat task and, now, by the
retry loop. Both used one fixed tmp path, so concurrent writers could
clobber it or fail the rename. Each write now names its own tmp file
(pid + counter); a test hammers it from four threads.

A permanently bad secondary logged three warnings per attempt, every 16
minutes, for as long as the daemon ran. DownAccounts::failed now reports
whether a failure is news (first, or a different reason from the last);
only that is a warning, repeats are debug. The two per-attempt warnings
in client.rs and credential.rs are covered by that line, which names the
reason, so the store-token one drops to debug.

The comments in client.rs and a credential.rs test, and persistence.md,
still described the old skip-until-restart behaviour or implied that a
store-listed account without a credential is retried here; that case is
handled by the linked-name set changing, not by this loop.
2026-10-14 16:20:04 +02:00
atlas
7f26238c87 hive-matrix-mcp: retry a secondary account that failed to come up
The daemon's two-minute refresh compared the store's linked account
names with the names it booted with. A secondary that was named but did
not come up - no token at bring-up, a token the homeserver rejected
(M_UNKNOWN_TOKEN), or a homeserver still down after the ~52 s boot
retries - kept its name in the set, so nothing ever noticed it again:
it stayed down until the daemon restarted for some other reason, and
an operator re-linking a rejected account changed no name.

Each such secondary is now tried again on the same two-minute period,
one per tick, after one refresh and then doubling waits capped at
sixteen minutes. One that comes up joins the registry (now behind a
lock) and its sync loop joins the running set; the primary and the
other accounts are not restarted. The store is still only read.

The startup todo wipe moves out of bring_up_account to run once before
any sync loop: run per account, a late secondary would have cleared the
todos the running accounts' sweeps had already deduped.

Part of #5017 (slice M).
2026-10-14 16:20:04 +02:00
atlas
2fb6b39f9f review classifier: every ACP run gets one shared, empty cwd
opencode writes the session's cwd into the system message's environment
block ("Working directory: …", "Workspace root folder: …"), ahead of the
PR's user message. Each ACP classification ran with its per-call scratch
directory, /tmp/review-classifier-<pid>-<counter>, as cwd, so every call
sent a different system message and the provider's prompt cache could not
match past it.

The session's cwd is now one fixed directory, <tmp>/review-classifier-acp-cwd,
shared by every run. HOME (the config, system.md and opencode's own state)
stays a directory per run, removed after it, as before. With no tool and
no project config nothing writes into the cwd: a live opencode 1.15.10 run
through initialize, session/new and a prompt left it empty, with its state
all under HOME. So concurrent runs can share it, and no state carries from
one run to the next.

Refs #5245
2026-10-14 14:45:59 +02:00
atlas
345c6e3731 docs: swarm services publish VictoriaLogs' /metrics to the collector 2026-10-14 14:45:31 +02:00
atlas
5fe04da559 swarm-otel: scrape VictoriaLogs through its published name
VictoriaLogs' own metrics were scraped over loopback, declared only on
the host running the store, so a collector on any other host had no job
for it. Its target is now declared on every host as
`https://<swarm.victorialogs.domain>/metrics`, next to forgejo's and
authelia's, and the store's vhost serves `= /metrics` behind a bare
`auth_request` to authelia. The rules derived from
`publishedScrapeTargets` admit the collector's client there and deny every
other subject. The loopback entry is gone, so a single host still scrapes
the store once, now by name.

On that vhost `/metrics` was reachable through the `/` location by any
authelia session. The exact-match location now takes it, so a session
gets a `deny` on `/metrics` and nothing else on the name changes.

Part of #5205
2026-10-14 14:45:31 +02:00
atlas
f19a9f546e hive-agent: export a per-file line count for subagent role files
A role file is read into every subagent dispatch that names it, and
nothing measured its size. A new watch on the same 15-minute tick as
claude_md_watch records `hyperhive.agent.subagent_role.lines`, one
series per `<role>.md` under the state dir's `subagent_roles/`,
labelled `role` with the file stem. Metric only: no todo, no threshold.

Fixes #4558
2026-10-14 13:53:20 +02:00
atlas
74b63dd031 review classifier: commit shas and index lines stay out of the prompt
Every classification's shared context began with the PR's head sha,
headed each commit with its sha, and carried each file's
`index <hash>..<hash>` line in the diff. A push or rebase rewrites all
of those, so the provider's prompt cache missed from the first line of
the user message even when the commit messages and hunks were
unchanged (#5245: "cache hits show as 0.1x").

The header now names only repo#pr, each commit is headed by its
position, and the diff drops `index` lines before lockfiles collapse.
The `diff --git` paths and every hunk stay. The TryOut's head_sha is
untouched: the controller still posts the review against it, and the
journal line still logs it.

On the ACP runtime the cost sink also adds each classification's tokens
to `hyperhive.review_classifier.token.usage`, typed input / output /
cacheRead / cacheCreation like claude's own `claude_code.token.usage`,
so the hit rate is a number in the metrics store on both runtimes.
2026-10-14 13:52:36 +02:00
atlas
57d6d67556 docs: swarm services publish VictoriaMetrics' /metrics to the collector 2026-10-14 13:27:08 +02:00
atlas
223cca2b56 swarm-authelia: admit only the collector anywhere on the metrics store's name
No access rule covered the push route on `metrics.<swarm>`, so a request
there fell to `default_policy = one_factor`: any logged-in session cookie
for the swarm could write samples into VictoriaMetrics.

The name now gets two rules with no `resources`: the collector's client
at `one_factor`, then `deny`. Only the push route and `/metrics` on that
vhost send an authelia subrequest (the read API, Grafana's datasource
included, is served with no check), and the collector is the one caller
of both, so a domain-wide pair covers them with no path or query regex
to slip past. It replaces the per-path `/metrics` pair the published
target would otherwise derive for that name. Both rules sit ahead of the
apex and store-UI rules, and authelia matches `domain` exactly, so they
shadow no other name.

Authelia 4.39.20, read from source: a rule with no `domain` wildcard
matches with `strings.EqualFold` (access_control_domain.go), an empty
`resources` or `subject` list matches everything (access_control_rule.go),
and the first matching rule wins (authorizer.go).

Fixes #5218
2026-10-14 13:27:08 +02:00
atlas
338a767a61 swarm-otel: scrape VictoriaMetrics through its published name
The store's own /metrics was scraped over loopback, declared under
deploy.victoriametrics.enable, so a collector on another host never had
a job for it. Declare it as a published target on every host instead,
keyed on swarm.victoriametrics.domain, and serve it on the metrics vhost
behind the existing authz subrequest. Authelia's per-target rules (one
collector rule, then deny) now cover it too.

The loopback entry goes, so a single-host swarm scrapes the store once.
2026-10-14 13:27:08 +02:00
atlas
6d42abb285 review classifier: under ACP the prompt is a real system message
The ACP runtime handed the classifier's system prompt to hive-runtime's
system_prompt_file, which prepends it to the first user message, so the
instructions and the PR text sat in the same user turn.

The rendered opencode.json now defines a review-classifier agent whose
prompt is {file:./system.md} and selects it with default_agent. opencode
1.15.10 sends an agent's prompt as the system-role message (in place of
its per-model default prompt), followed only by its environment block;
the first user message is the PR material alone. The API key runtime
keeps --system-prompt.

The agent carries no permission or tools key, and a test pins its exact
shape: opencode merges an agent's own permission after
OPENCODE_PERMISSION, so a permission there would offer tools again
(checked on real opencode: {"*":"allow"} on the agent offered all ten
tools despite OPENCODE_PERMISSION={"*":"deny"}).

Follow-up to #5214.
2026-10-14 12:49:33 +02:00
atlas
7d24feaf25 swarmctl: on a controller host, trust the swarm's root without SWARMCTL_CA_FILE
`swarmctl login` (and any `--swarm` verb) on a controller host failed TLS
with `UnknownIssuer` against the swarm's own gateway: its leaf chains to
the store's `pki` root, which no system store holds, and the only way
through was `--ca-file`/`SWARMCTL_CA_FILE` by hand.

The host's swarmctl wrapper now sets `SSL_CERT_FILE` to the bundle the
swarm-controller daemon already trusts (`/run/swarm-controller-ca/
trust-bundle.pem`: system CAs plus the swarm's root), when the gateway is
self-signed. swarmctl's reqwest/rustls resolves roots the same way the
daemon's does, so this is the path the daemon uses to reach the same
gateway. `--set` rather than a default, since a nix shell exports its own
`SSL_CERT_FILE`.

Pointing `SWARMCTL_CA_FILE` at the bundle instead would not work (#5243):
`--ca-file` adds each certificate as a root and fails the whole client on
one it can't parse.

Hosts without the controller (an operator laptop) still need `--ca-file`.

Refs #5239
Fixes #5243
2026-10-14 12:49:16 +02:00
atlas
07f093c400 swarm-otel: authelia is no longer declared as both a loopback and a published scrape target
#5215 moved authelia's scrape to `publishedScrapeTargets` and removed
its loopback entry from the authelia module. It was rebased at merge
onto c06aa78b6, which had already moved that loopback entry into
glue-otel-scrape-targets.nix, so the entry survived there. Every host
running authelia and the collector, `singleHostSwarm` included, then
failed swarm-otel.nix's job-name assertion.

The glue now declares only grafana and nats. forge-placement's case
that pinned authelia on loopback now pins its absence, and
swarm-otel-identity gains a case reading the assertion itself on the
single-host fixture, since the option-level case cannot fail a
rebuild the way the assertion does.

Fixes #5247
2026-10-14 10:28:48 +02:00
atlas
531cc6bf91 swarm-controller: the builder's forge calls re-read the token on refusal too
#5177 added five forge requests under the controller's token that went
straight to the forgejo client: the commit status in
`Client::create_commit_status`, and in `forge/builder_account.rs` the
hives-team add, the user read, the repo-creation lockdown and the team
membership read. Each now goes through `Api::call`, so a 401 re-reads
the token from the store and sends the request once more, like the
other controller forge calls.

The read-back of the builder's newly minted token in
`ensure_builder_forge_token` stays a direct client: it sends the
builder's own token, not the controller's, the same as the read-backs
in `agent_token`, `classifier_account` and `hive_token`.
2026-10-14 10:16:18 +02:00
atlas
6ba29c4d33 swarm-controller: re-read the forge token from the store when the forge rejects it
The controller read its forge token from the swarm secret store once, when
its client slot filled, and kept sending it until the daemon restarted. A
token the forge re-minted or revoked left every forge-backed path failing
with 401 until someone restarted the controller, and a restart drops the
runtime-only job graph.

Every forge request now goes through `forge::api::Api::call`. On a 401 it
asks a `StoredSecret` (the queue clients' debounced, single-flight re-read
with backoff) for a token to try instead, rebuilds the forgejo client with
it, and sends the request once more. A second 401, or a store still holding
the refused token, surfaces the error as before. Only 401 counts: unlike an
IdP, the forge answers 400 to a malformed request, and a re-read started by
one would put a real 401 that follows inside the backoff.

forgejo-api bakes the token into its own reqwest client's default headers
and takes no client or middleware from outside, so there is no single
request function to hook. Each `self.api.<endpoint>(..)` site becomes
`self.api.call(|api| api.<endpoint>(..).into_future())`; option structs a
site moved into the call are cloned, since the closure can run twice. The
`OnceLock<Arc<Client>>` slot and its `get()` callers are unchanged.

Refs #5017
2026-10-14 10:16:18 +02:00
atlas
e7bf337624 docs: swarm services publish authelia's /metrics to the collector 2026-10-14 10:15:41 +02:00
atlas
8eb11a91da swarm-authelia: match the metrics rules with or without a query string
Authelia 4.39.20 matches `resources` against the cleaned path plus
`?<query>` (authorization/types.go:94 NewObject, utils/url.go:11-34
URLPathFullClean), while nginx's `= /metrics` serves the request whatever
its query. `^/metrics$` therefore missed `/metrics?x=1`: both the collector
rule and the trailing `deny` were skipped and a plain session reached
`default_policy = one_factor`. The pattern is now `^<path>([?].*)?$` for
every published target.

Forgejo strips trailing slashes before routing (routers/common/
middleware.go stripSlashesMiddleware) and serves metrics without a token
when none is configured, so `forge.<swarm>/metrics/` reached its metrics
through the ungated `/` location. The forge vhost now answers
`= /metrics/` with 404; nginx merges slashes and decodes `%2F` before
matching, so that also covers `/metrics//` and `/metrics%2F`.

Part of #5205
2026-10-14 10:15:41 +02:00
atlas
393e84afd0 swarm-otel: scrape authelia through its published name
Authelia's metrics were scraped over loopback, declared only on a host
running both authelia and the collector, so a split deploy collected none.
Its target is now declared on every host as
`https://<swarm.authelia.domain>/metrics`, next to forgejo's, and authelia's
vhost serves `= /metrics` behind the same collector auth subrequest as the
forge's. The loopback entry is gone, so a single host still scrapes
authelia once, now by name.

The forge-only `metricsRule` becomes `metricsRules`, derived from every
`publishedScrapeTargets` entry. Each target gets the collector-scoped
`one_factor` rule followed by an unconditional `deny`. The `deny` is
needed because authelia 4.39 skips a rule whose `subject` does not
match and moves on. With nothing after the collector rule, any other
authenticated session reached `default_policy = one_factor` and was
admitted. That was already true of the forge's `/metrics`, and this
change closes it there too.

Part of #5205
2026-10-14 10:15:41 +02:00
atlas
1219091788 glue: the controller's authelia and otel environment moves out of the controller module
The authelia and swarm-otel co-location reads (swarm-controller.nix
autheliaEnv, otelEnv) move into glue-controller-env.nix.
2026-10-14 10:15:29 +02:00
atlas
4a36e53982 glue: nats firewall on the wireguard interface moves out of the nats module
The wg-hive allowedTCPPorts rule gated on deploy.wireguard.enable was a
cross-service toggle read in swarm-nats.nix; it now lives in
glue-nats-wireguard.nix under the same condition and value.
2026-10-14 10:15:19 +02:00
atlas
7d95973d83 swarmctl: --swarm works after the subcommand, takes a bare domain, and login defaults to the host's swarm
`swarmctl login --swarm X` failed with "unexpected argument", and
`login --help` listed no `--swarm`: the flag was only accepted before the
verb. `--swarm` and `--ca-file` are now global, so they parse on either side
of the subcommand and show in every subcommand's help.

`--swarm` / `SWARMCTL_SWARM` accept a bare domain and prepend `https://`.
A value with a scheme is parsed as before, so `http://` is still refused.

On a swarm-controller host, the swarmctl wrapper sets
`SWARMCTL_LOGIN_DEFAULT_SWARM=https://<swarm.domain>` (`--set-default`).
Only `login` reads it, after `--swarm` and `SWARMCTL_SWARM`. The wrapper
does not set `SWARMCTL_SWARM`: that would send every socket verb (agent
create, schedule, inference, forge) through the gateway and make each need
a login. Other verbs resolve exactly as before.

No CA file is wired. The only CA material the module names is the
controller's assembled trust bundle (system CAs plus the hive anchor).
`--ca-file` hands every certificate in it to rustls-platform-verifier as
an extra root, and that fails the whole client on any one it cannot
parse, so pointing it at the system CAs is a risk nothing here can test.
The controller daemon trusts the same CA through `SSL_CERT_FILE`, which
skips unparsable certificates; using that for swarmctl is left open.

Machines that are not swarm-controller hosts (an operator laptop) get no
default; that case waits on an answer in the issue.

Fixes #5239
2026-10-14 10:15:09 +02:00
atlas
62e45df88c docs/swarm/credentials.md: restore the API-key row lost in the rebase 2026-10-14 10:14:36 +02:00
atlas
2c6ae15383 swarm-review-classifier: test that Otel::record exports the run's cost
Nothing exercised the real record path: the acp tests use a mock sink and
the cost tests only covered amount(). Split the counter construction and
the add out of Otel::record so a test can run them against an in-memory
exporter, and assert the counter name, the amount and the four labels.

Dev-dependency: opentelemetry_sdk's `testing` feature (same crate; adds
tokio to its lockfile entry).
2026-10-14 10:14:36 +02:00
atlas
1b4e764e5c review classifier: run on ACP when the swarm settings choose it
The swarm settings gain `runtime` (`api-key`, the default, or `acp`) and
an `acp` provider record: an https base URL, the provider's model, and
its prices per million tokens. A PUT now changes only the fields its
body names, so the settings form that sends model, effort and limit
leaves the runtime alone; `null` clears `effort` or `acp`. The body stays
strict, and the stored document keeps #5192's lenient read.

The controller sends the runtime in each run request. On `acp` the
classifier runs one prompt on a new `opencode acp` session per
classification through hive-runtime's AcpRuntime, in a scratch
directory, with a config rendered from the settings. opencode runs
`read` and other tools without asking, so OPENCODE_PERMISSION
{"*":"deny"} takes every tool away from the model, and a deny-all policy
refuses any request on top. The title call, which resends the prompt,
is turned off. The deadline cancels the turn, and a turn that does not
end within 15s of that is dropped, which also covers an agent stuck
before its prompt. The answer is the final assistant text, as ACP has
no result event.

The provider key is read per run from
swarm/services/review-classifier/acp-provider, which the bao UI's
admins login may create and update, and reaches opencode through
AcpRuntime::with_provider_key and the trampoline's allowlist.

Each ACP classification's cost is computed from the configured prices
and the reported tokens, and added to
hyperhive.review_classifier.cost.usage labelled repo, pr, model and
currency.

Part of #5096
2026-10-14 10:14:36 +02:00
atlas
38ee68c31d hive otel: keep file_storage defined when the collector authenticates upstream
The hive collector's `settings` was `{ extensions.file_storage = …; … }
// lib.optionalAttrs senderAuth { extensions.oauth2client/swarm = …; }`.
`//` is shallow, so whenever senderAuth held (every valid hive config)
the right-hand `extensions` replaced the left one and `file_storage`
vanished, while `service.extensions` still listed it. `otelcol validate`
then failed the config.yaml build:

  Error: service::extensions: references extension "file_storage" which is not configured

Broken by 02831fb5e (PR #5178), which added `extensions.file_storage`
to the left-hand attrset; the `//` block for the auth extension predates
it (cb787997bd). Both extensions now live in one `extensions` attrset.

module-eval-hive-otel gains a case that every name in
`service.extensions` is defined under `extensions`; it evaluates false
on the parent commit and true here.

Fixes #5240
2026-10-14 10:13:58 +02:00
flake-bot
c7fdedd08c nix flake update 2026-10-14 10:13:45 +02:00
atlas
0754b740b7 swarm-ui: name the subagent cap in the three descriptions that list inference fields
swarm-controller README, the swarmctl inference doc comment (and the
regenerated swarmctl-cli.md) and the hive-agent swarm_inference module doc
still listed only model, effort and subagent models.
2026-10-14 10:13:29 +02:00
atlas
461795b2cf swarm-ui: fix the stray doc text, name the agent in the cap-read warning
docs/swarm/inference.md had a table-row suffix pasted before the H1 by a
bad substitution, and the PUT row had lost it; the text is back in the
row.

A failed read of the agent-inference key already applied no cap (an
unreachable swarm must not block subagent work) and warned, but the
warning did not say which agent. It now names the agent and the error,
a test pins that, and subagent.md says a failed read starts anyway.

The max-subagents field stays a text field with a digit pattern: the
review-settings page's numeric field (concurrent reviews) is a TextField
with pattern="[0-9]+" and validation, and no field in swarm-ui uses
type="number". The new field gets the same pattern.
2026-10-14 10:13:29 +02:00
atlas
9a71d1e094 swarm-ui: per-agent cap on concurrently running subagents
Nothing capped how many subagents an agent runs at once; the operator
wants to set that per agent from swarm-ui (#5234, shape (b)).

AgentInference gains `max_subagents: Option<u32>`. An absent field
decodes as None, so every stored value still parses and means "no cap",
which is today's behaviour.

swarm-controller accepts 1..=32 on PUT. 0 would refuse every start; 32
is far more concurrent claude sessions than one agent container has
memory for, so anything above it is a typo, not a cap.

The subagent daemon already read the agent's agent-inference key for
subagent_models; it now reads it on every start and continue (not only
those naming a model) and refuses a new run once the cap's worth of
names is tracked, counted under the same lock as the name reservation.
Every tracked slot counts: running, starting, a goal run between turns,
and an interrupted one still exiting. A goal re-prompt never goes
through the reservation, so it is never refused. A failed read applies
no cap, as it falls back for models.

swarmctl's `inference set` does GET, merge, PUT; its mirror type now
carries the field so a set from the CLI keeps a cap set in swarm-ui
instead of dropping it, and `get` prints it.
2026-10-14 10:13:29 +02:00
atlas
6d7fc0dd99 docs: builder.md says the client secret is re-read on refusal too
The builder reads its queue client secret at start and again when
authelia refuses it. The credentials table in builder.md and the
swarm-builder README still said it was read only at start.
2026-10-14 03:30:46 +02:00
atlas
47d3ccfede module-eval: bao-grants counts the eighteen granting units 2026-10-14 03:30:46 +02:00
atlas
cafa120d77 swarm-builder: scope the forge token to the forge, one build at a time, opt-in
argus's review on #5177: the git credential helper answered every `get`,
so a config PR's own flake inputs could have git send the builder's forge
token to any host. The helper is now configured under
`credential.<forge-url>.helper` and itself answers only an https request
whose host is the forge's; a test runs it as git does and checks that
other hosts, http and non-get operations get nothing.

Builds are bounded: every `build_agent` node needs the one `builder` jobq
resource, so one runs at a time and the rest wait pending, and a newer
head cancels its PR's pending build (a running one finishes and writes on
its own commit). The builder also holds a one-permit semaphore around each
build.

`builder` and `review-classifier`, the forge logins swarm-controller owns,
join nix/reserved-names.nix, with a test tying the list to both constants.

The builder is now opt-in like the CI runner: `allSwarmServices` no longer
turns it on, since it runs unreviewed config PRs through the host's nix
daemon. mara had not answered atlas's question on the PR.
2026-10-14 03:30:46 +02:00
atlas
b68b74d687 swarm-builder: build each agent config PR head and post a commit status
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).

When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.

The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.

Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.

This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
2026-10-14 03:30:46 +02:00
atlas
aece1c26d0 hive-matrix-mcp: never enable recovery over secret storage it did not create
`recovery().enable()` refuses only when a key backup exists. It then makes
a new secret-storage key and sets it as the account's default regardless
of the default already there, so an account with secret storage but no
backup (set up in another client, backup since deleted) would have its
default key replaced and its other clients' recovery key locked out.

`recover_or_enable` now asks `secret_storage().is_enabled()` before
`enable` and returns an error instead, which `ensure_recovery` logs at
WARN with the account and the passphrase's store path. The decision is
behind a small trait over the matrix-sdk calls so a fake can pin it: with
no backup and secret storage present, neither `enable` nor `recover` runs.

Review: argus on #5188.
2026-10-14 02:09:57 +02:00
atlas
45568f6e60 matrix: back up agents' room keys with a controller-minted recovery passphrase
An agent's matrix encryption keys lived only in its local crypto store, so
`destroy --purge` or a primary-account token reset lost every encrypted
room's history with no way back (#5022).

swarm-controller now mints a random passphrase (32 bytes from the kernel's
CSPRNG, base64url) for every account listed under each agent's `matrix/`
directory, at `swarm/agents/<agent>/matrix-recovery/<account>`, in a
five-minute pass. The write is a cas=0 create after a read; a path that
holds anything is never replaced, and a create that loses a race counts as
done. The controller's existing `data/swarm/agents/*` grant covers the
write and the agent's own `agents/<agent>/*` read grant covers the read, so
no policy changes.

hive-matrix-daemon reads the passphrase when it brings an account up, after
cross-signing: with no backup on the homeserver it enables recovery with it
(backup + secret storage); with one there it recovers, which imports the
backup key and the self-signing key, and room keys then download through
`BackupDownloadStrategy::OneShot`. No passphrase yet logs at INFO and the
account comes up without backup; any failure logs WARN and never stops the
daemon.

Recovery runs after `ensure_cross_signing`, not before: on a new account
cross-signing must exist locally for `enable` to export it into secret
storage, and on an existing identity `bootstrap_cross_signing_if_needed` is
a no-op, leaving `recover` to import and sign.

Part of #5022 (S1+S2, with the S4 docs).
2026-10-14 02:09:57 +02:00
atlas
bff605035c swarm-victorialogs: size the agent-auth limit for a host booting its agents together
All agents on a host reach the gateway from one address and share one
limit_req budget; a host boot is one cache miss per agent at once.
Defaults go from 5r/s burst 20 to 50r/s burst 500. The module-eval case
reads the option values.
2026-10-14 01:54:54 +02:00
atlas
8be4030d03 swarm-logs-auth: rate-limit verification and cache positive verdicts
Any request on logs.<swarm> carrying a well-formed swarm-agent bearer
made swarm-logs-auth do a bao cert login and KV read before it could
refuse, with no cap on the rate.

nginx now applies limit_req (zone swarm_logs_agent_auth, keyed by
client address, 5r/s burst 20 nodelay by default) on the internal
location that forwards to swarm-logs-auth, answering 403 when over,
since auth_request passes through only 401/403 and swarm-logs retries
a 401. The authelia path and the query location carry no limit.

swarm-logs-auth remembers bearers it has granted, keyed by SHA-256 of
the bearer, for --verdict-ttl seconds (deploy.victorialogs.authVerdictTtl,
60 by default), holding at most 4096 and evicting the oldest. Denials
are never remembered. A replaced or deleted queue secret therefore
keeps working at the log store for up to the TTL after its last check.

Closes #5190
2026-10-14 01:54:54 +02:00
atlas
3be6096c26 glue: the secret publisher's authelia-dependent settings move out of the publisher module
The publisher's `enable` default read `deploy.authelia.enable` from the
publisher's own module (swarm-secret-publisher.nix). It now defaults to
`false`, and glue-secret-publisher-authelia.nix sets it `mkDefault true`
wherever authelia runs, so the effective value is unchanged. The option
loses its `defaultText`, and its description names the glue.

The `deploy.authelia.enable` read in `active` stays: every file the unit
copies is under authelia's `hostClientSecretDir`, so it is a requirement
of the unit, not a co-location toggle.

Part of #3742.
2026-10-14 01:52:53 +02:00
atlas
c06aa78b63 glue: otel scrape targets for authelia, grafana and nats move out of the services
Part of #3742.
2026-10-14 01:40:40 +02:00
atlas
6c1ab87f68 swarm-otel: declare forgejo's published scrape target on every host
The forgejo entry in `swarm.otel.publishedScrapeTargets` was declared
under `deploy.forgejo.enable`, so it existed only in the forge host's
evaluation. Its two readers evaluate elsewhere: the collector renders
its prometheus jobs from its own host's list, and authelia registers
the collector client's audience from its own host's list. With the
forge, the collector and authelia on different hosts, the collector
had no forgejo job and authelia would have refused its token.

Declare the entry unconditionally in swarm-otel-service.nix, keyed on
`swarm.forge.domain`, which every host evaluates identically. A swarm
with no forge scrapes a name that answers nothing and shows a failing
job, the same trade the collector and grafana OIDC clients already make.

Single-host render is unchanged: the entry already existed there.

Part of #5205
2026-10-13 22:15:13 +02:00
atlas
4a3f21fc2b glue: forge's controller-dependent settings move out of the forge module 2026-10-13 22:14:28 +02:00
atlas
9407db69b5 glue: consumers' start-after-bao ordering moves into one glue module
Each store reader (forge's OIDC fetch and token publisher, matrix's OIDC
fetch, the hive collector's fetch, Grafana's OIDC fetch and admin password,
the queue's callout-xkey unit, the CI runner's token fetch, the swarm
collector's OIDC fetch) used to read `deploy.bao.enable` in its own module
to add `after`/`wants` on `container@<store machine>` and
`after`/`requires` on `swarm-bao-pki.service`. Per the #3742 ruling, that
cross-service read moves into glue-bao-readers-store-order.nix, which sets
the same edges when the store and the reader are both on this host and
restates each reader's own gate so no stub unit is rendered.

Rendered `after`/`wants`/`requires` are identical as sets to main on 16
fixtures, store here and elsewhere. New module-eval cases in existing
fixtures pin both arms.

Part of #3742.
2026-10-13 22:05:16 +02:00
atlas
40ebd45c66 glue: hive-otel's ordering after authelia moves out of the otel module
The hive collector's secret-wait unit read deploy.authelia.enable to
order itself after authelia's container. That ordering is now set by
glue-hive-otel-authelia.nix, which restates the unit's own gate (otel's
`waits`). The rendered unit is unchanged.

The hive-otel module-eval fixture `hiveOtel` gains authelia, and a case
pins the edge with authelia here and its absence without.

Part of #3742.
2026-10-13 22:04:52 +02:00
damocles
8e58ad8717 fix passive-voice vale errors in docs + hive-otel module-eval test for the new journald receiver 2026-10-13 22:03:15 +02:00
damocles
02831fb5e9 hive-c0re: replace the agents_crashing banner with a crash-watcher log line + grafana alert 2026-10-13 22:03:15 +02:00
atlas
6c9c2cef45 grafana: the published vhost refuses /metrics
Grafana runs with metrics.enabled and no basic_auth_*, so its Prometheus
exposition was served to any anonymous client of grafana.<swarm> through
the published vhost's `location "/"`. Add an exact-match `= /metrics`
location returning 403 there. The loopback `grafana-metrics` vhost and
the collector's scrape of it are unchanged.

Comments on the metricsPort option and the loopback vhost now state that
the published vhost refuses /metrics; the gateway doc says the same.
A module-eval case pins the refusing location and that `/` still proxies.
2026-10-13 21:42:29 +02:00