Watch
0
0
Fork
You've already forked hyperhive
0
Commit graph hyperhive/docs/swarm
Author SHA1 Message Date
atlas
09612c025b docs: the nats gateway cert unit and the queue's /metrics vhost get their secrets.md and gateway.md rows 2026-10-15 08:05:01 +02:00
atlas
e6e9c894d6 docs: the swarm queue publishes its /metrics to the collector 2026-10-15 08:03:46 +02:00
atlas
d24dbaeadf swarm builder: the all-swarm-services switch turns it on, like every other swarm service
deploy.builder.enable defaults on under deploy.allSwarmServices (mkDefault),
where the host has the builder's store identity pair, as for the review
classifier. An explicit false still wins.
2026-10-15 05:44:37 +02:00
iris
5b49e0e539 swarm ui: a contraction in the review section of the UI doc 2026-10-15 03:38:48 +02:00
iris
a68d73fa74 swarm ui: one review page
The two review tabs, runs and try-out, are one: /review. It is a column of
three sections: try a classification and the classifier settings, each a
folded bar (the settings' bar says what they are now), then the recent runs,
always open. The settings had been pinned under the try-out form; they are now
a section of their own. A new Disclosure primitive is the folding bar, a
native details wearing the Panel's classes, so the folded and open sections
read as the same surface.

/review-runs and /review-tryout are gone, with no redirect.
2026-10-15 03:38:48 +02:00
atlas
8abea789fd docs: swarm services publish Grafana's /metrics to the collector
The gateway no longer has a loopback-only vhost, so the TLS section
drops the paragraph that named `grafana-metrics` as its one instance.
2026-10-14 23:45:04 +02:00
atlas
4a9acd1d0c authelia: a user outside a rule's group is denied instead of falling through to one_factor
authelia takes the first matching rule, and a rule with a `subject` does
not match a subject outside it. The swarm domain's and the bao UI's
`group:admins` rules had nothing after them, so an account in no group
skipped them and landed on `default_policy = one_factor`: any
authenticated account reached the swarm UI, its `/api/` (by cookie or by
bearer) and the bao UI.

Each of the two group rules is now an `operatorsOnly` pair: the group
allow, then a subject-less `deny` for the same domain, the shape
`collectorOnly` already gives the collector's metrics rules.
`default_policy` stays `one_factor`, which the forge and chat rely on.

module-eval-swarm-otel-identity gains two cases on `allLocal`: an
account in no group is refused the swarm domain (four paths) and the bao
UI while `group:admins` passes and the forge stays open; and a
predicate, `autheliaSubjectRulesClosed` in lib.nix, holds over the
rendered list: every subject-scoped allow is closed by a later
subject-less `deny` on its domain with no subject-less allow in between,
and nothing follows a domain-wide `deny` on its domain.

Fixes #5120
2026-10-14 23:41:13 +02:00
atlas
afeb0f9794 swarm UI: an agent that needs a Claude login shows it on its card
The agent's turn-state header ($SWARM.agent-state.<agent>) gains
`needs_login`, set by hive-agent from its own LoginState, so the flag
reaches the swarm without hive-c0re (mara on #5268: "id prefer the agent
container sending the info it needs login without hive").

- swarm-queue-client: `agent_login::Login`, the shared wire type, beside
  `agent_inference::Effective`. `needs_login: Option<bool>` with
  `#[serde(default)]`: a header from an older agent decodes as `None`
  (unknown), never as "needs login" or "logged in".
- hive-agent: the header publisher reads the shared LoginState cell.
  Every writer of that cell emits a status on the bus afterwards, so a
  login change goes out on the next rebuild, not the 60 s refresh.
- swarm-controller: `agent_login::Reported` follows every agent's header
  and keeps the latest flag; `GET /api/agents/status` merges it into each
  row as `needs_login` (null when unknown).
- swarm-ui: AgentCard shows a warning-tone "login needed" badge when the
  row's `needs_login` is true.

Refs #5268
2026-10-14 20:07:46 +02:00
iris
417125b33c swarm-ui: a start page at / and the hive directory at /hives
/ is now the swarm's name over the digital-rain backdrop the hive
dashboard's home has, and the swarm name in the header links to it. The
hive directory moves to /hives, where the nav's hives tab points. The
swarm name is read once per page load by a shared hook, the header and
the start page both use it. Closes #5252
2026-10-14 18:36:52 +02:00
atlas
c90bd705a7 review try-out: author the PR branch at the account's noreply address
The commit author's email was the forge account's own address, read with
the site-admin token, which put an email the account may keep private
into the target repo's history. It is now the address Forgejo itself
uses for such an account's web commits, `<lowercased login>@<NO_REPLY_ADDRESS>`
(models/user/user.go GetPlaceholderEmail). NO_REPLY_ADDRESS defaults to
`noreply.<DOMAIN>` (modules/setting/service.go) and the forge module
leaves it unset, with DOMAIN set from services.hyperhive.swarm.forge.domain,
the same option SWARM_CONTROLLER_FORGE_URL is built from; so the domain is
that URL's host. The account is still read, to refuse a user with no
forge account, but its email no longer is.

Refs #5259
2026-10-14 18:36:36 +02:00
atlas
49b15e7775 review try-out: open a PR with the classifications on the page
The try-out page gets an "open a PR" button. It posts the page's
classifications to the new POST /api/review-classifier/open-pr, which
merges them by name into the repo's rules file on the default branch
and has the review-classifier forge account commit the result on a new
branch, review-classifier/<user>-<8 hex>, through the contents API. The
commit's author is the forge account named like the session's
Remote-User, read with the controller's admin token. The answer links
to the forge's compare page, where the user opens the PR as themselves.

The push uses the account's existing token from the store
(swarm/services/review-classifier/forge/token, scope write:repository),
so the account needs write access to the repo; a refused push answers
403 naming that. In a YAML rules file, the entries the merge leaves
alone keep their text, as long as the result reads back as the merged
list; otherwise the file is written out whole.

DefaultBranchFiles now carries each file's blob sha, which the forge
needs to replace a file, and the repo's web URL for the link.

Fixes #5259
Refs #5096
2026-10-14 18:36:36 +02:00
atlas
6ed88ad70a review classifier: classifications of one PR share a cached prefix
On the API key runtime the classifier now hands claude its prompt as one
stream-json user message (`--input-format stream-json`) with two text
blocks: the PR's shared context, carrying `cache_control: ephemeral`, and
the classification's question.

As plain stdin text the whole prompt was one block, and claude puts its
only message breakpoint on the last block, after the question. The cache
writes an entry only where a breakpoint is, so no entry ended at the
shared-context/question boundary. Every classification wrote the whole
prompt and none read it (production: cacheRead ~0, cacheCreation ~input).
The system-prompt prefix (~270 tokens) is under the minimum cacheable
length. Captures against a local listener with a dummy key (claude
2.1.223) show the client breakpoint survives on the shared block, and that
claude adds none there on its own.

The PR stays in the user role: moving it into the system prompt would give
untrusted PR text the system prompt's authority.

`Model::ask` takes a `Prompt { shared, question }`. The ACP runtime joins
it back into the same text it sent before.

Refs #5245
2026-10-14 18:26:11 +02:00
atlas
5e653ab9b5 swarm-victoriametrics: own network namespace, gateway over a unix socket
The metrics store's container gets privateNetwork = true with no
interface but loopback. VictoriaMetrics listens on
unix:/run/swarm-victoriametrics/victoriametrics.sock, a host directory
(root:nginx 0770) bind-mounted into the container, and every vhost
location that forwarded to 127.0.0.1:8428 now forwards to that socket.

The store runs as a static user whose primary group is nginx's gid,
with UMask 0007, so the socket it creates is 0770 group nginx. This
relies on VictoriaMetrics >= 1.153.0 honouring the umask (the locked
nixpkgs ships 1.153.0). nixpkgs' readiness loop curls a TCP URL, so
postStart is replaced with the same loop over --unix-socket.

swarm.victoriametrics.port is removed (nothing binds it any more), with
a mkRemovedOptionModule entry, and dropped from swarm-otel's host
port-collision assertion. The container's resolver unit is dropped:
there is no interface to reach the bridge resolver through, and the
store resolves no names.

The collector scrape, the collector push and Grafana's datasource
already reach the store by its published domain, so they are unchanged.

Refs #4815
Refs #3560
2026-10-14 18:15:20 +02:00
iris
a04096ea19 swarm-ui: choose the review classifier's runtime on the settings page
The classifier settings page showed model and effort only, so the API-key
or ACP runtime could be switched only through a raw API call. It now has a
runtime select, and under ACP the provider's URL, model and prices. A PUT
carries the provider only on the ACP runtime, so the stored record stays
for a switch back. Closes #5249
2026-10-14 16:20:37 +02:00
atlas
7f26238c87 hive-matrix-mcp: retry a secondary account that failed to come up
The daemon's two-minute refresh compared the store's linked account
names with the names it booted with. A secondary that was named but did
not come up - no token at bring-up, a token the homeserver rejected
(M_UNKNOWN_TOKEN), or a homeserver still down after the ~52 s boot
retries - kept its name in the set, so nothing ever noticed it again:
it stayed down until the daemon restarted for some other reason, and
an operator re-linking a rejected account changed no name.

Each such secondary is now tried again on the same two-minute period,
one per tick, after one refresh and then doubling waits capped at
sixteen minutes. One that comes up joins the registry (now behind a
lock) and its sync loop joins the running set; the primary and the
other accounts are not restarted. The store is still only read.

The startup todo wipe moves out of bring_up_account to run once before
any sync loop: run per account, a late secondary would have cleared the
todos the running accounts' sweeps had already deduped.

Part of #5017 (slice M).
2026-10-14 16:20:04 +02:00
atlas
2fb6b39f9f review classifier: every ACP run gets one shared, empty cwd
opencode writes the session's cwd into the system message's environment
block ("Working directory: …", "Workspace root folder: …"), ahead of the
PR's user message. Each ACP classification ran with its per-call scratch
directory, /tmp/review-classifier-<pid>-<counter>, as cwd, so every call
sent a different system message and the provider's prompt cache could not
match past it.

The session's cwd is now one fixed directory, <tmp>/review-classifier-acp-cwd,
shared by every run. HOME (the config, system.md and opencode's own state)
stays a directory per run, removed after it, as before. With no tool and
no project config nothing writes into the cwd: a live opencode 1.15.10 run
through initialize, session/new and a prompt left it empty, with its state
all under HOME. So concurrent runs can share it, and no state carries from
one run to the next.

Refs #5245
2026-10-14 14:45:59 +02:00
atlas
345c6e3731 docs: swarm services publish VictoriaLogs' /metrics to the collector 2026-10-14 14:45:31 +02:00
atlas
74b63dd031 review classifier: commit shas and index lines stay out of the prompt
Every classification's shared context began with the PR's head sha,
headed each commit with its sha, and carried each file's
`index <hash>..<hash>` line in the diff. A push or rebase rewrites all
of those, so the provider's prompt cache missed from the first line of
the user message even when the commit messages and hunks were
unchanged (#5245: "cache hits show as 0.1x").

The header now names only repo#pr, each commit is headed by its
position, and the diff drops `index` lines before lockfiles collapse.
The `diff --git` paths and every hunk stay. The TryOut's head_sha is
untouched: the controller still posts the review against it, and the
journal line still logs it.

On the ACP runtime the cost sink also adds each classification's tokens
to `hyperhive.review_classifier.token.usage`, typed input / output /
cacheRead / cacheCreation like claude's own `claude_code.token.usage`,
so the hit rate is a number in the metrics store on both runtimes.
2026-10-14 13:52:36 +02:00
atlas
57d6d67556 docs: swarm services publish VictoriaMetrics' /metrics to the collector 2026-10-14 13:27:08 +02:00
atlas
223cca2b56 swarm-authelia: admit only the collector anywhere on the metrics store's name
No access rule covered the push route on `metrics.<swarm>`, so a request
there fell to `default_policy = one_factor`: any logged-in session cookie
for the swarm could write samples into VictoriaMetrics.

The name now gets two rules with no `resources`: the collector's client
at `one_factor`, then `deny`. Only the push route and `/metrics` on that
vhost send an authelia subrequest (the read API, Grafana's datasource
included, is served with no check), and the collector is the one caller
of both, so a domain-wide pair covers them with no path or query regex
to slip past. It replaces the per-path `/metrics` pair the published
target would otherwise derive for that name. Both rules sit ahead of the
apex and store-UI rules, and authelia matches `domain` exactly, so they
shadow no other name.

Authelia 4.39.20, read from source: a rule with no `domain` wildcard
matches with `strings.EqualFold` (access_control_domain.go), an empty
`resources` or `subject` list matches everything (access_control_rule.go),
and the first matching rule wins (authorizer.go).

Fixes #5218
2026-10-14 13:27:08 +02:00
atlas
338a767a61 swarm-otel: scrape VictoriaMetrics through its published name
The store's own /metrics was scraped over loopback, declared under
deploy.victoriametrics.enable, so a collector on another host never had
a job for it. Declare it as a published target on every host instead,
keyed on swarm.victoriametrics.domain, and serve it on the metrics vhost
behind the existing authz subrequest. Authelia's per-target rules (one
collector rule, then deny) now cover it too.

The loopback entry goes, so a single-host swarm scrapes the store once.
2026-10-14 13:27:08 +02:00
atlas
6d42abb285 review classifier: under ACP the prompt is a real system message
The ACP runtime handed the classifier's system prompt to hive-runtime's
system_prompt_file, which prepends it to the first user message, so the
instructions and the PR text sat in the same user turn.

The rendered opencode.json now defines a review-classifier agent whose
prompt is {file:./system.md} and selects it with default_agent. opencode
1.15.10 sends an agent's prompt as the system-role message (in place of
its per-model default prompt), followed only by its environment block;
the first user message is the PR material alone. The API key runtime
keeps --system-prompt.

The agent carries no permission or tools key, and a test pins its exact
shape: opencode merges an agent's own permission after
OPENCODE_PERMISSION, so a permission there would offer tools again
(checked on real opencode: {"*":"allow"} on the agent offered all ten
tools despite OPENCODE_PERMISSION={"*":"deny"}).

Follow-up to #5214.
2026-10-14 12:49:33 +02:00
atlas
6ba29c4d33 swarm-controller: re-read the forge token from the store when the forge rejects it
The controller read its forge token from the swarm secret store once, when
its client slot filled, and kept sending it until the daemon restarted. A
token the forge re-minted or revoked left every forge-backed path failing
with 401 until someone restarted the controller, and a restart drops the
runtime-only job graph.

Every forge request now goes through `forge::api::Api::call`. On a 401 it
asks a `StoredSecret` (the queue clients' debounced, single-flight re-read
with backoff) for a token to try instead, rebuilds the forgejo client with
it, and sends the request once more. A second 401, or a store still holding
the refused token, surfaces the error as before. Only 401 counts: unlike an
IdP, the forge answers 400 to a malformed request, and a re-read started by
one would put a real 401 that follows inside the backoff.

forgejo-api bakes the token into its own reqwest client's default headers
and takes no client or middleware from outside, so there is no single
request function to hook. Each `self.api.<endpoint>(..)` site becomes
`self.api.call(|api| api.<endpoint>(..).into_future())`; option structs a
site moved into the call are cloned, since the closure can run twice. The
`OnceLock<Arc<Client>>` slot and its `get()` callers are unchanged.

Refs #5017
2026-10-14 10:16:18 +02:00
atlas
e7bf337624 docs: swarm services publish authelia's /metrics to the collector 2026-10-14 10:15:41 +02:00
atlas
62e45df88c docs/swarm/credentials.md: restore the API-key row lost in the rebase 2026-10-14 10:14:36 +02:00
atlas
1b4e764e5c review classifier: run on ACP when the swarm settings choose it
The swarm settings gain `runtime` (`api-key`, the default, or `acp`) and
an `acp` provider record: an https base URL, the provider's model, and
its prices per million tokens. A PUT now changes only the fields its
body names, so the settings form that sends model, effort and limit
leaves the runtime alone; `null` clears `effort` or `acp`. The body stays
strict, and the stored document keeps #5192's lenient read.

The controller sends the runtime in each run request. On `acp` the
classifier runs one prompt on a new `opencode acp` session per
classification through hive-runtime's AcpRuntime, in a scratch
directory, with a config rendered from the settings. opencode runs
`read` and other tools without asking, so OPENCODE_PERMISSION
{"*":"deny"} takes every tool away from the model, and a deny-all policy
refuses any request on top. The title call, which resends the prompt,
is turned off. The deadline cancels the turn, and a turn that does not
end within 15s of that is dropped, which also covers an agent stuck
before its prompt. The answer is the final assistant text, as ACP has
no result event.

The provider key is read per run from
swarm/services/review-classifier/acp-provider, which the bao UI's
admins login may create and update, and reaches opencode through
AcpRuntime::with_provider_key and the trampoline's allowlist.

Each ACP classification's cost is computed from the configured prices
and the reported tokens, and added to
hyperhive.review_classifier.cost.usage labelled repo, pr, model and
currency.

Part of #5096
2026-10-14 10:14:36 +02:00
atlas
461795b2cf swarm-ui: fix the stray doc text, name the agent in the cap-read warning
docs/swarm/inference.md had a table-row suffix pasted before the H1 by a
bad substitution, and the PUT row had lost it; the text is back in the
row.

A failed read of the agent-inference key already applied no cap (an
unreachable swarm must not block subagent work) and warned, but the
warning did not say which agent. It now names the agent and the error,
a test pins that, and subagent.md says a failed read starts anyway.

The max-subagents field stays a text field with a digit pattern: the
review-settings page's numeric field (concurrent reviews) is a TextField
with pattern="[0-9]+" and validation, and no field in swarm-ui uses
type="number". The new field gets the same pattern.
2026-10-14 10:13:29 +02:00
atlas
9a71d1e094 swarm-ui: per-agent cap on concurrently running subagents
Nothing capped how many subagents an agent runs at once; the operator
wants to set that per agent from swarm-ui (#5234, shape (b)).

AgentInference gains `max_subagents: Option<u32>`. An absent field
decodes as None, so every stored value still parses and means "no cap",
which is today's behaviour.

swarm-controller accepts 1..=32 on PUT. 0 would refuse every start; 32
is far more concurrent claude sessions than one agent container has
memory for, so anything above it is a typo, not a cap.

The subagent daemon already read the agent's agent-inference key for
subagent_models; it now reads it on every start and continue (not only
those naming a model) and refuses a new run once the cap's worth of
names is tracked, counted under the same lock as the name reservation.
Every tracked slot counts: running, starting, a goal run between turns,
and an interrupted one still exiting. A goal re-prompt never goes
through the reservation, so it is never refused. A failed read applies
no cap, as it falls back for models.

swarmctl's `inference set` does GET, merge, PUT; its mirror type now
carries the field so a set from the CLI keeps a cap set in swarm-ui
instead of dropping it, and `get` prints it.
2026-10-14 10:13:29 +02:00
atlas
6d7fc0dd99 docs: builder.md says the client secret is re-read on refusal too
The builder reads its queue client secret at start and again when
authelia refuses it. The credentials table in builder.md and the
swarm-builder README still said it was read only at start.
2026-10-14 03:30:46 +02:00
atlas
cafa120d77 swarm-builder: scope the forge token to the forge, one build at a time, opt-in
argus's review on #5177: the git credential helper answered every `get`,
so a config PR's own flake inputs could have git send the builder's forge
token to any host. The helper is now configured under
`credential.<forge-url>.helper` and itself answers only an https request
whose host is the forge's; a test runs it as git does and checks that
other hosts, http and non-get operations get nothing.

Builds are bounded: every `build_agent` node needs the one `builder` jobq
resource, so one runs at a time and the rest wait pending, and a newer
head cancels its PR's pending build (a running one finishes and writes on
its own commit). The builder also holds a one-permit semaphore around each
build.

`builder` and `review-classifier`, the forge logins swarm-controller owns,
join nix/reserved-names.nix, with a test tying the list to both constants.

The builder is now opt-in like the CI runner: `allSwarmServices` no longer
turns it on, since it runs unreviewed config PRs through the host's nix
daemon. mara had not answered atlas's question on the PR.
2026-10-14 03:30:46 +02:00
atlas
b68b74d687 swarm-builder: build each agent config PR head and post a commit status
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).

When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.

The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.

Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.

This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
2026-10-14 03:30:46 +02:00
atlas
45568f6e60 matrix: back up agents' room keys with a controller-minted recovery passphrase
An agent's matrix encryption keys lived only in its local crypto store, so
`destroy --purge` or a primary-account token reset lost every encrypted
room's history with no way back (#5022).

swarm-controller now mints a random passphrase (32 bytes from the kernel's
CSPRNG, base64url) for every account listed under each agent's `matrix/`
directory, at `swarm/agents/<agent>/matrix-recovery/<account>`, in a
five-minute pass. The write is a cas=0 create after a read; a path that
holds anything is never replaced, and a create that loses a race counts as
done. The controller's existing `data/swarm/agents/*` grant covers the
write and the agent's own `agents/<agent>/*` read grant covers the read, so
no policy changes.

hive-matrix-daemon reads the passphrase when it brings an account up, after
cross-signing: with no backup on the homeserver it enables recovery with it
(backup + secret storage); with one there it recovers, which imports the
backup key and the self-signing key, and room keys then download through
`BackupDownloadStrategy::OneShot`. No passphrase yet logs at INFO and the
account comes up without backup; any failure logs WARN and never stops the
daemon.

Recovery runs after `ensure_cross_signing`, not before: on a new account
cross-signing must exist locally for `enable` to export it into secret
storage, and on an existing identity `bootstrap_cross_signing_if_needed` is
a no-op, leaving `recover` to import and sign.

Part of #5022 (S1+S2, with the S4 docs).
2026-10-14 02:09:57 +02:00
atlas
bff605035c swarm-victorialogs: size the agent-auth limit for a host booting its agents together
All agents on a host reach the gateway from one address and share one
limit_req budget; a host boot is one cache miss per agent at once.
Defaults go from 5r/s burst 20 to 50r/s burst 500. The module-eval case
reads the option values.
2026-10-14 01:54:54 +02:00
atlas
8be4030d03 swarm-logs-auth: rate-limit verification and cache positive verdicts
Any request on logs.<swarm> carrying a well-formed swarm-agent bearer
made swarm-logs-auth do a bao cert login and KV read before it could
refuse, with no cap on the rate.

nginx now applies limit_req (zone swarm_logs_agent_auth, keyed by
client address, 5r/s burst 20 nodelay by default) on the internal
location that forwards to swarm-logs-auth, answering 403 when over,
since auth_request passes through only 401/403 and swarm-logs retries
a 401. The authelia path and the query location carry no limit.

swarm-logs-auth remembers bearers it has granted, keyed by SHA-256 of
the bearer, for --verdict-ttl seconds (deploy.victorialogs.authVerdictTtl,
60 by default), holding at most 4096 and evicting the oldest. Denials
are never remembered. A replaced or deleted queue secret therefore
keeps working at the log store for up to the TTL after its last check.

Closes #5190
2026-10-14 01:54:54 +02:00
atlas
6c1ab87f68 swarm-otel: declare forgejo's published scrape target on every host
The forgejo entry in `swarm.otel.publishedScrapeTargets` was declared
under `deploy.forgejo.enable`, so it existed only in the forge host's
evaluation. Its two readers evaluate elsewhere: the collector renders
its prometheus jobs from its own host's list, and authelia registers
the collector client's audience from its own host's list. With the
forge, the collector and authelia on different hosts, the collector
had no forgejo job and authelia would have refused its token.

Declare the entry unconditionally in swarm-otel-service.nix, keyed on
`swarm.forge.domain`, which every host evaluates identically. A swarm
with no forge scrapes a name that answers nothing and shows a failing
job, the same trade the collector and grafana OIDC clients already make.

Single-host render is unchanged: the entry already existed there.

Part of #5205
2026-10-13 22:15:13 +02:00
atlas
eba6428b6b docs: review-classifier truncation sentences in active voice 2026-10-13 21:23:03 +02:00
atlas
3f5f348270 review classifier: truncate an over-long reason instead of dropping the verdict
An answer whose reason ran past 500 characters was rejected whole, so the
verdict was lost and the one error row turned the review into no_verdict.
The reason is cut to the cap, ending in an ellipsis; verdict and certainty
are kept.
2026-10-13 21:23:03 +02:00
atlas
44346d8c6c nix: warn on the ignored swarm-ui switches instead of failing eval
deploy.swarm-ui.enable and swarm.ui.enable are declared again as
nullOr bool, default null, and do nothing. Setting either adds an eval
warning that swarm-ui now runs wherever swarm-controller runs.
2026-10-13 20:05:19 +02:00
atlas
f9954b91b4 nix: run swarm-ui wherever swarm-controller runs
Remove deploy.swarm-ui.enable. Every reader (the swarm-ui vhost, the
authelia operator rule for the swarm apex, hive-network's
anyServiceEnabled) now reads deploy.swarm-controller.enable, and the
controller's webhook public URL is gated on the swarm domain alone,
since the vhost it names exists wherever the daemon does.

Both spellings of the old switch (deploy.swarm-ui.enable and the
already-renamed swarm.ui.enable) become mkRemovedOptionModule entries,
so a host that still sets either fails evaluation with a message
naming the controller toggle instead of silently changing.

Ruling (mara, #3742): "swarm-ui always runs wherever controller runs".

Part of #3742
2026-10-13 20:05:19 +02:00
atlas
a6935e57c3 swarm-queue-client: re-read a held OIDC client secret when the IdP refuses it
hive-c0re (`hive-<name>`), swarm-controller and swarm-review-classifier
read their queue client secret from the store once and held it as
`ClientSecret::Value` for the life of the process. After authelia
re-minted a secret and swarm-secret-publish stored the new one, every
later token mint presented the old value and was refused until the
daemon restarted.

`ClientSecret::Stored` replaces `Value`: a `StoredSecret` holds the
secret in memory with a caller-supplied read callback, so the crate
takes no store dependency. When the token endpoint answers 400 or 401,
`mint_token` (and its blocking sibling, used by the controller's OTLP
exporters) re-reads it and, if the store holds a different secret,
mints once more with it. One re-read runs at a time and is spawned so
it outlives a caller that gives up. A re-read that fails or finds the
refused value delays the next one, 5 s doubling to 5 min, so a secret
that stays wrong costs one store read per delay. Logs name the client
id and never the secret.

swarm-nats-auth's `oidc_secret` was the template; its re-read machinery
now sits on `StoredSecret`, so the tree has one copy. The responder
gains the same re-read backoff.

The controller keeps its `IDENTITY` OnceLock: it now holds a
`QueueConfig` whose secret is re-read, and the lazily built vcs-metrics
exporter reads the identity from it.

agent_renewal.rs's module doc said a re-minted agent queue secret only
reaches the agent at its next start; hive-agent reads the store on every
connection attempt after its first, so the doc now says that.

Part of #5017 (slice Q).
2026-10-13 19:16:04 +02:00
atlas
be9b2475a9 docs: say what refresh-consumer does when it cannot read a timestamp 2026-10-13 17:00:56 +02:00
iris
baf282131f swarm-ui: the review try-out page shows and changes the review classifier's settings
A classifier settings panel under the try-out form reads and saves the model,
effort and how many requested reviews run at once through the settings route.
A save replaces all three; a refused one shows the route's reason and leaves
the form as typed. There is no API key field.
2026-10-13 12:24:47 +02:00
iris
57407cb437 swarm-ui: the resource caps dialog says when a change takes effect
The hive reads the caps about every 30 seconds and a running agent takes a
changed cap at its next spawn or rebuild, so the dialog and the ui doc no
longer say the hive applies a change within seconds.
2026-10-13 09:01:31 +02:00
iris
a376714116 swarm-ui: an agent's detail shows its resource caps and edits them
A resource caps row in the agent details reads GET /api/agents/{name}/limits
and shows the cpu quota and memory max, each reading as the hive deciding
when the swarm sets none. An edit dialog PUTs the filled fields, which
replaces the stored value whole, and a clear action DELETEs it. A refusal,
such as a percentage with a leading zero, shows as the route's reason under
the fields and leaves them as typed.
2026-10-13 09:01:31 +02:00
atlas
0f53e9be6c swarm-logs: authenticate as the agent with its own queue secret
swarm-logs reads `swarm/agents/<agent>/queue` from the swarm secret
store with the agent's own store certificate (the read hive-agent makes
for its queue connection) and presents it as
`Bearer swarm-agent.<agent>.<secret>`, which the gateway's
swarm-logs-auth checks. On a 401 it reads the secret again and retries
once, so a secret re-minted between the read and the check does not
fail the query.

The OIDC mint against the hive's shared `hive-<h>-agent` client is gone
from swarm-logs, along with the unused `HIVE_AGENT_OIDC_CA_FILE` anchor.
logs.nix now gates on the store address instead of the queue's token
endpoint and wraps the binary with the paths queue-identity.nix gives
the harness; the host forwards the query URL wherever it forwards a
store identity.

The shared client is still delivered into containers; removing that is

Part of #4459.
2026-10-13 08:38:05 +02:00
atlas
53d0d59be0 swarm-victoriametrics: forward only the store's read API
The metrics vhost's `/` proxied every path to VictoriaMetrics with no
auth_request, so anything that could reach metrics.<swarm> could write
(`/api/v1/import`, `/api/v1/write`, ...) and delete
(`/api/v1/admin/tsdb/delete_series`) series. Only the OTLP push route
was gated.

`/` now answers 403. The read routes Grafana's Prometheus datasource
and vmui use are forwarded by exact or `^~` prefix location, each with
the matched path as its proxy_pass URI. The bare domain, which the
swarm UI's Metrics quick link opens, redirects to vmui instead of the
store's index page. The OTLP push location is unchanged.

Closes #5186
2026-10-13 08:13:37 +02:00
atlas
3464a912bf schedules: keep projecting a schedule whose definition stops decoding
A reconcile projected only the definitions this build decodes, so after
a rollback an agent's already-projected schedule whose definition a newer
build rewrote was dropped from its key within a minute and stopped
firing. The projection now keeps an agent's stored entries whose id
belongs to an unreadable definition, as they were; it never adds one to
a key that did not hold it. A deleted readable definition still leaves
every key.
2026-10-13 08:12:50 +02:00
atlas
29aa242201 controller stores: recover from a record they can't decode
After a rollback, a record a newer build wrote (a new enum variant, a
changed required field) stopped decoding, and the strict
`schedules::read` turned that into an error on every read and write of
the key until someone deleted it by hand.

- agent_limits / inference (one document per agent): an undecodable
  document reads as unset, with a WARN naming the bucket, key and error.
  The PUT's "equal to stored, skip the write" shortcut no longer applies
  to an undecodable document, so a PUT of the defaults replaces it too.
- schedule definitions (one document, every schedule): salvaged per
  definition, as the review-classifier run list is. An undecodable
  definition is kept as its raw JSON, hidden from the list, edits and
  projections, written back by every commit after the readable ones, and
  its id is never minted again. A document with no list of schedules
  stays an error: reading it as empty would empty every agent's key.
- per-agent agent-schedules: an undecodable key reads as no schedules
  and is rewritten on the next projection, even when the agent has none.

`schedules::read` is replaced by `read_or_default`; `wanted` keeps its
own strict read.

Closes #5193
2026-10-13 08:12:50 +02:00
atlas
daaf920e06 review classifier: keep stored runs this build can't decode
A run record this build couldn't decode was dropped from the list it
read, and the next recorded run wrote the list back without it, so a
rollback followed by a roll-forward lost the newer build's runs for
good. The stored list is now a list of entries, each a decoded run or
the raw JSON of one that didn't decode. The route still shows only the
decoded runs. An undecodable entry keeps its place, counts toward the
100, and is written back as it was read. A read that finds any logs one
WARN with their count and the first error.

The settings decode derived its known fields from what
`Settings::default()` serializes, so a later field skipped when unset
would have been stripped from a valid stored document and lost on the
next PUT. The known fields are now an explicit list. A test serializes
a fully populated `Settings`, checks that its keys match the list, and
checks that `decode` round-trips it.
2026-10-13 08:10:51 +02:00
atlas
6f0a72b246 review classifier: recover from a stored record it can't decode
The controller keeps the classifier's newest runs as one JSON array and
its settings as one JSON object, each under one KV key. A document this
build could not decode, as after a rollback past a build that added an
outcome or a settings field, failed every read and every write until
someone deleted the key by hand. A classifier newer than its controller
refused every run request for the missing `model`.

The run store now decodes the array one record at a time: a record that
does not decode is dropped with a WARN naming the bucket, key, index and
error, and a document that is not an array reads as no runs. Listing
keeps working, and the next recorded run writes back only the records
that decoded.

The stored settings ignore a field this build does not know; one that
still does not decode reads as the defaults with a WARN. The PUT body
keeps `deny_unknown_fields`. A PUT now always writes, so a PUT of the
defaults also replaces an undecodable document.

A run request without `model` runs on haiku, the swarm-wide setting's
default, now one constant in swarm-queue-client.

`schedules::read` gains a sibling, `read_with`, taking the decoder.

Closes #5162
2026-10-13 08:10:51 +02:00