The retries covered about a minute, and the swarm-bao restart that lost
#5286's review lasted 3m19s. Each try now gets its own forge deadline,
a try with no answer by then is retried like a refused connection, and
the waits are 5s, 15s, 45s, 90s and 150s: six tries over about five
minutes.
The docs also said to request the review again after a post-failed run.
That does nothing while the account's earlier request is still pending,
so they now say to remove the request and then request it again.
A review request's verdict was lost when the post failed: on 2026-10-14
the classifier approved #5286, but its post hit swarm-bao mid-restart
(cert login: connection refused), the run was recorded post-failed, and
nothing on the PR showed it.
post() now tries the post up to four times, waiting 5s, 15s and 45s,
when it fails because the connection to the secret store or the forge
failed or timed out. Any other error fails at once, as before. The
verdict is computed once and reposted as is. All tries share the one
forge deadline the single post had, and the warn log carries the number
of tries.
network.md listed matrix among the shared-netns containers and the
private-netns list named only agents and the CI runner; it now says
every privateNetwork container keeps its own firewall. The authelia
module no longer calls the matrix container shared, and the hive-c0re
comment on having no homeserver fallback no longer names port 8008.
Part of #4815.
tuwunel bound `0.0.0.0:<httpPort>` in the host netns, so the homeserver
port was open on every host address, and the gateway proxied to it on
host loopback.
The container now sets `privateNetwork = true` and hangs off the hive
bridge with a DHCP lease, the same attach the CI runner, builder and
review classifier use (resolvconf on, `networking.nameservers` the bridge
IP). tuwunel listens only on `/run/hive-matrix/tuwunel.sock`, a host
directory (root:nginx 0770) bind-mounted into the container; its primary
group is nginx's gid and the socket is 0660. The gateway's `/_matrix/`
location proxies to the socket.
The static resolv.conf goes: it existed because resolvconf's host
tracking came up empty in the shared netns.
`swarm.matrix.httpPort` and `{swarm,deploy}.matrix.openFirewall` are
removed with a message: there is no host port to choose or open. The
`.well-known/matrix/*` fallback to `<hive>:<httpPort>` is dropped with
them; it only rendered with `gatewayHost = null`, which the homeserver's
SSO assertion already rejects.
Part of #4815.
The swarm lives in the swarm UI; this tab lists the hive's agents. The tab
is 4G3NTS at #agents, its pane and count pill follow, and the module that
renders the containers is agents.js. Comments and docs that named SW4RM or
swarm.js follow.
The build lifecycle used to be its own page with three sub-tabs. They are
now dashboard tabs (rebuild, meta, buildlogs) beside SW4RM and
P3RM1SS10NS: builds.js is a module tabs.js imports, so the dashboard's one
tab strip, /api/state snapshot and event stream (plus the meta_inputs_changed
and meta_update_running kinds) serve them, and the page, its bundle entry,
stylesheet and home tile are gone. The SW4RM banner's "view queue" link
points at the #rebuild tab, and the rebuild tab carries a count pill of the
rebuilds in flight.
An empty tab-count pill drew a "0": its display beat the UA [hidden] rule.
`swarm-bao-pki` was a `RemainAfterExit` oneshot. Once active, nothing
re-ran it: a switch restarts it only when its definition changes, and a
`Requires=`/`Wants=` on an active unit is satisfied without running it.
On muede-lpt2 the 10:57Z switch that added the builder line did change
the unit, so switch-to-configuration stopped it in its stop phase. A
start of `container@swarm-bao` 4 s later (before the daemon-reload at
10:57:35Z) pulled it back in through `swarm-bao-certs` with the OLD
definition, which had no builder line: 48 ms, nothing missing. The start
phase then found it active and did nothing. The 20:14Z switch that
enabled the builder left the unit unchanged, so it was never re-run, and
`container@swarm-builder` crash-looped on the absent `builder.pem`.
The unit now goes inactive after each run. Every switch starts
`multi-user.target`, which runs it again post-reload with the current
definition, and every unit that wants or requires it runs it on its own
start. Each line is idempotent on presence, so an existing leaf is never
re-signed.
The three consumers started from `machines.target` rather than
`multi-user.target` also want it, since `After=` alone does not order
against a job in another transaction: the builder's and the review
classifier's containers (nspawn `--load-credential`) and
`swarm-nats-auth-secrets` (pulled in by the queue's container).
Fixes#5284
The hive no longer sets an agent's CPU or memory cap: caps come from the swarm, else the hive-wide defaults. The container load tab was the only hive UI that set them, and its figures are already exported as OTEL metrics, so with the tab gone nothing wrote meta/resource-limits.json.
Removed: the file's reader and writer, the `hivectl agent set-limits` verb with its host-socket request, the `GET /api/container-resources` and `POST /api/resource-limits/{name}` dashboard endpoints, and the `cpu_quota`/`memory_max` fields of the container view that only the tab read. A leftover resource-limits.json is ignored.
The cpu, memory and disk figures are exported as OTEL metrics and graphed in Grafana, and the caps are set in the swarm UI, so the C0NT41N3R L04D tab on the C0R3 page only duplicated both. C0R3 is now the kept-state page alone, without a tab strip.
The socket is wanted by sockets.target and nothing orders the wireguard
unit before it, so the mesh address can be missing at bind time
(EADDRNOTAVAIL) and the receiver never listens. FreeBind lets the bind
succeed regardless, and also covers the interface coming back after a flap.
Also assert the mesh address is not a wildcard, which the comment above the
socket already claimed, and cover the rendered socket in a module-eval case.
A separate $SWARM.control.<agent> subject carries commands, apart from
the text subject. swarm-controller exposes POST /api/agents/{name}/cancel
as a NATS request; the harness answers whether a turn was actually
stopped, through the same cancel path as the dashboard button. The agent
credential gets subscribe on its control subject and allow_responses to
answer; only the controller may publish there.
Refs #3803
The review body's table is now: result | check | certainty / threshold |
reason. The result cell is a symbol for what the row did to the review,
after polarity (✅ counted and passed, ❌ counted and failed, ❔ not
counted or no answer), then the model's verdict. The overall line carries
✅ / ❌. Polarity, threshold and counted are folded into those cells.
The prompt asks for a reason under 350 characters; the enforced cap
stays 500, so an overshoot is truncated less often.
deploy.builder.enable defaults on under deploy.allSwarmServices (mkDefault),
where the host has the builder's store identity pair, as for the review
classifier. An explicit false still wins.
The two review tabs, runs and try-out, are one: /review. It is a column of
three sections: try a classification and the classifier settings, each a
folded bar (the settings' bar says what they are now), then the recent runs,
always open. The settings had been pinned under the try-out form; they are now
a section of their own. A new Disclosure primitive is the folding bar, a
native details wearing the Panel's classes, so the folded and open sections
read as the same surface.
/review-runs and /review-tryout are gone, with no redirect.
Three follow-ups to argus's review of #5237:
- A down secondary's retry runs `restore` under RESTORE_TIMEOUT (2 min).
Tries run one per tick on one future, so a restore that never resolved
held back every other down account's retries for good; a try that
outlasts the timeout now counts as a failure like any other.
- `client::build_and_restore` warns about a rejected token only for the
primary, which exits right after. For a secondary it logs at debug: the
retry loop already warns once per outage (`DownAccounts::failed` is news
on the first failure and on a changed reason, and an account that came
up starts afresh), so the extra warn repeated on every try.
- `a_secondary_that_comes_up_late_does_not_wipe_the_todos` runs the late
bring-up in a child process of the test binary with HIVE_AGENT_SOCKET
pointed at the test's listener. In-process the variable was unset, so a
wipe placed back in the bring-up path was a no-op and the test passed.
New tests: a_second_rejection_in_a_row_does_not_warn_again (captured logs, alone
in a child process since tracing caches callsite interest process-wide;
a stub whoami answering M_UNKNOWN_TOKEN, the real build_and_restore),
a_failure_after_the_account_came_up_is_news_again, and
an_account_whose_restore_hangs_does_not_hold_back_the_others (paused time).
authelia takes the first matching rule, and a rule with a `subject` does
not match a subject outside it. The swarm domain's and the bao UI's
`group:admins` rules had nothing after them, so an account in no group
skipped them and landed on `default_policy = one_factor`: any
authenticated account reached the swarm UI, its `/api/` (by cookie or by
bearer) and the bao UI.
Each of the two group rules is now an `operatorsOnly` pair: the group
allow, then a subject-less `deny` for the same domain, the shape
`collectorOnly` already gives the collector's metrics rules.
`default_policy` stays `one_factor`, which the forge and chat rely on.
module-eval-swarm-otel-identity gains two cases on `allLocal`: an
account in no group is refused the swarm domain (four paths) and the bao
UI while `group:admins` passes and the forge stays open; and a
predicate, `autheliaSubjectRulesClosed` in lib.nix, holds over the
rendered list: every subject-scoped allow is closed by a later
subject-less `deny` on its domain with no subject-less allow in between,
and nothing follows a domain-wide `deny` on its domain.
Fixes#5120
The agent's turn-state header ($SWARM.agent-state.<agent>) gains
`needs_login`, set by hive-agent from its own LoginState, so the flag
reaches the swarm without hive-c0re (mara on #5268: "id prefer the agent
container sending the info it needs login without hive").
- swarm-queue-client: `agent_login::Login`, the shared wire type, beside
`agent_inference::Effective`. `needs_login: Option<bool>` with
`#[serde(default)]`: a header from an older agent decodes as `None`
(unknown), never as "needs login" or "logged in".
- hive-agent: the header publisher reads the shared LoginState cell.
Every writer of that cell emits a status on the bus afterwards, so a
login change goes out on the next rebuild, not the 60 s refresh.
- swarm-controller: `agent_login::Reported` follows every agent's header
and keeps the latest flag; `GET /api/agents/status` merges it into each
row as `needs_login` (null when unknown).
- swarm-ui: AgentCard shows a warning-tone "login needed" badge when the
row's `needs_login` is true.
Refs #5268
/ is now the swarm's name over the digital-rain backdrop the hive
dashboard's home has, and the swarm name in the header links to it. The
hive directory moves to /hives, where the nav's hives tab points. The
swarm name is read once per page load by a shared hook, the header and
the start page both use it. Closes#5252
The commit author's email was the forge account's own address, read with
the site-admin token, which put an email the account may keep private
into the target repo's history. It is now the address Forgejo itself
uses for such an account's web commits, `<lowercased login>@<NO_REPLY_ADDRESS>`
(models/user/user.go GetPlaceholderEmail). NO_REPLY_ADDRESS defaults to
`noreply.<DOMAIN>` (modules/setting/service.go) and the forge module
leaves it unset, with DOMAIN set from services.hyperhive.swarm.forge.domain,
the same option SWARM_CONTROLLER_FORGE_URL is built from; so the domain is
that URL's host. The account is still read, to refuse a user with no
forge account, but its email no longer is.
Refs #5259
The try-out page gets an "open a PR" button. It posts the page's
classifications to the new POST /api/review-classifier/open-pr, which
merges them by name into the repo's rules file on the default branch
and has the review-classifier forge account commit the result on a new
branch, review-classifier/<user>-<8 hex>, through the contents API. The
commit's author is the forge account named like the session's
Remote-User, read with the controller's admin token. The answer links
to the forge's compare page, where the user opens the PR as themselves.
The push uses the account's existing token from the store
(swarm/services/review-classifier/forge/token, scope write:repository),
so the account needs write access to the repo; a refused push answers
403 naming that. In a YAML rules file, the entries the merge leaves
alone keep their text, as long as the result reads back as the merged
list; otherwise the file is written out whole.
DefaultBranchFiles now carries each file's blob sha, which the forge
needs to replace a file, and the repo's web URL for the link.
Fixes#5259
Refs #5096
On the API key runtime the classifier now hands claude its prompt as one
stream-json user message (`--input-format stream-json`) with two text
blocks: the PR's shared context, carrying `cache_control: ephemeral`, and
the classification's question.
As plain stdin text the whole prompt was one block, and claude puts its
only message breakpoint on the last block, after the question. The cache
writes an entry only where a breakpoint is, so no entry ended at the
shared-context/question boundary. Every classification wrote the whole
prompt and none read it (production: cacheRead ~0, cacheCreation ~input).
The system-prompt prefix (~270 tokens) is under the minimum cacheable
length. Captures against a local listener with a dummy key (claude
2.1.223) show the client breakpoint survives on the shared block, and that
claude adds none there on its own.
The PR stays in the user role: moving it into the system prompt would give
untrusted PR text the system prompt's authority.
`Model::ask` takes a `Prompt { shared, question }`. The ACP runtime joins
it back into the same text it sent before.
Refs #5245
The metrics store's container gets privateNetwork = true with no
interface but loopback. VictoriaMetrics listens on
unix:/run/swarm-victoriametrics/victoriametrics.sock, a host directory
(root:nginx 0770) bind-mounted into the container, and every vhost
location that forwarded to 127.0.0.1:8428 now forwards to that socket.
The store runs as a static user whose primary group is nginx's gid,
with UMask 0007, so the socket it creates is 0770 group nginx. This
relies on VictoriaMetrics >= 1.153.0 honouring the umask (the locked
nixpkgs ships 1.153.0). nixpkgs' readiness loop curls a TCP URL, so
postStart is replaced with the same loop over --unix-socket.
swarm.victoriametrics.port is removed (nothing binds it any more), with
a mkRemovedOptionModule entry, and dropped from swarm-otel's host
port-collision assertion. The container's resolver unit is dropped:
there is no interface to reach the bridge resolver through, and the
store resolves no names.
The collector scrape, the collector push and Grafana's datasource
already reach the store by its published domain, so they are unchanged.
Refs #4815
Refs #3560
With DynamicUser=true and RuntimeDirectoryPreserve=yes, systemd creates
the unit's runtime directory under /run/private/ (0700 root:root) and
symlinks /run/swarm-logs-auth to it (systemd.exec(5),
RuntimeDirectoryPreserve=). nginx runs as the unprivileged nginx user, so
its connect() to /run/swarm-logs-auth/auth.sock failed with EACCES, the
auth subrequest ended in a 502, and auth_request answered every agent
bearer on /select/logsql/ with a 500.
The unit now runs as a static swarm-logs-auth system user and group, the
shape swarm-controller already uses for its 0751 + Preserve socket dir.
RestrictSUIDSGID and RemoveIPC, which DynamicUser implied and the unit
did not set itself, are set explicitly.
module-eval-core-toggle gains a case that fails when any unit creating
the runtime directory of a unix: upstream of nginx combines DynamicUser
with RuntimeDirectoryPreserve. docs/process/gotchas.md records the trap.
Fixes#5235
The classifier settings page showed model and effort only, so the API-key
or ACP runtime could be switched only through a raw API call. It now has a
runtime select, and under ACP the provider's URL, model and prices. A PUT
carries the provider only on the ACP runtime, so the stored record stays
for a switch back. Closes#5249
Audit follow-ups on the secondary-account retry.
The accounts snapshot is written by the heartbeat task and, now, by the
retry loop. Both used one fixed tmp path, so concurrent writers could
clobber it or fail the rename. Each write now names its own tmp file
(pid + counter); a test hammers it from four threads.
A permanently bad secondary logged three warnings per attempt, every 16
minutes, for as long as the daemon ran. DownAccounts::failed now reports
whether a failure is news (first, or a different reason from the last);
only that is a warning, repeats are debug. The two per-attempt warnings
in client.rs and credential.rs are covered by that line, which names the
reason, so the store-token one drops to debug.
The comments in client.rs and a credential.rs test, and persistence.md,
still described the old skip-until-restart behaviour or implied that a
store-listed account without a credential is retried here; that case is
handled by the linked-name set changing, not by this loop.
The daemon's two-minute refresh compared the store's linked account
names with the names it booted with. A secondary that was named but did
not come up - no token at bring-up, a token the homeserver rejected
(M_UNKNOWN_TOKEN), or a homeserver still down after the ~52 s boot
retries - kept its name in the set, so nothing ever noticed it again:
it stayed down until the daemon restarted for some other reason, and
an operator re-linking a rejected account changed no name.
Each such secondary is now tried again on the same two-minute period,
one per tick, after one refresh and then doubling waits capped at
sixteen minutes. One that comes up joins the registry (now behind a
lock) and its sync loop joins the running set; the primary and the
other accounts are not restarted. The store is still only read.
The startup todo wipe moves out of bring_up_account to run once before
any sync loop: run per account, a late secondary would have cleared the
todos the running accounts' sweeps had already deduped.
Part of #5017 (slice M).
opencode writes the session's cwd into the system message's environment
block ("Working directory: …", "Workspace root folder: …"), ahead of the
PR's user message. Each ACP classification ran with its per-call scratch
directory, /tmp/review-classifier-<pid>-<counter>, as cwd, so every call
sent a different system message and the provider's prompt cache could not
match past it.
The session's cwd is now one fixed directory, <tmp>/review-classifier-acp-cwd,
shared by every run. HOME (the config, system.md and opencode's own state)
stays a directory per run, removed after it, as before. With no tool and
no project config nothing writes into the cwd: a live opencode 1.15.10 run
through initialize, session/new and a prompt left it empty, with its state
all under HOME. So concurrent runs can share it, and no state carries from
one run to the next.
Refs #5245
A role file is read into every subagent dispatch that names it, and
nothing measured its size. A new watch on the same 15-minute tick as
claude_md_watch records `hyperhive.agent.subagent_role.lines`, one
series per `<role>.md` under the state dir's `subagent_roles/`,
labelled `role` with the file stem. Metric only: no todo, no threshold.
Fixes#4558
Every classification's shared context began with the PR's head sha,
headed each commit with its sha, and carried each file's
`index <hash>..<hash>` line in the diff. A push or rebase rewrites all
of those, so the provider's prompt cache missed from the first line of
the user message even when the commit messages and hunks were
unchanged (#5245: "cache hits show as 0.1x").
The header now names only repo#pr, each commit is headed by its
position, and the diff drops `index` lines before lockfiles collapse.
The `diff --git` paths and every hunk stay. The TryOut's head_sha is
untouched: the controller still posts the review against it, and the
journal line still logs it.
On the ACP runtime the cost sink also adds each classification's tokens
to `hyperhive.review_classifier.token.usage`, typed input / output /
cacheRead / cacheCreation like claude's own `claude_code.token.usage`,
so the hit rate is a number in the metrics store on both runtimes.
No access rule covered the push route on `metrics.<swarm>`, so a request
there fell to `default_policy = one_factor`: any logged-in session cookie
for the swarm could write samples into VictoriaMetrics.
The name now gets two rules with no `resources`: the collector's client
at `one_factor`, then `deny`. Only the push route and `/metrics` on that
vhost send an authelia subrequest (the read API, Grafana's datasource
included, is served with no check), and the collector is the one caller
of both, so a domain-wide pair covers them with no path or query regex
to slip past. It replaces the per-path `/metrics` pair the published
target would otherwise derive for that name. Both rules sit ahead of the
apex and store-UI rules, and authelia matches `domain` exactly, so they
shadow no other name.
Authelia 4.39.20, read from source: a rule with no `domain` wildcard
matches with `strings.EqualFold` (access_control_domain.go), an empty
`resources` or `subject` list matches everything (access_control_rule.go),
and the first matching rule wins (authorizer.go).
Fixes#5218
The store's own /metrics was scraped over loopback, declared under
deploy.victoriametrics.enable, so a collector on another host never had
a job for it. Declare it as a published target on every host instead,
keyed on swarm.victoriametrics.domain, and serve it on the metrics vhost
behind the existing authz subrequest. Authelia's per-target rules (one
collector rule, then deny) now cover it too.
The loopback entry goes, so a single-host swarm scrapes the store once.
The ACP runtime handed the classifier's system prompt to hive-runtime's
system_prompt_file, which prepends it to the first user message, so the
instructions and the PR text sat in the same user turn.
The rendered opencode.json now defines a review-classifier agent whose
prompt is {file:./system.md} and selects it with default_agent. opencode
1.15.10 sends an agent's prompt as the system-role message (in place of
its per-model default prompt), followed only by its environment block;
the first user message is the PR material alone. The API key runtime
keeps --system-prompt.
The agent carries no permission or tools key, and a test pins its exact
shape: opencode merges an agent's own permission after
OPENCODE_PERMISSION, so a permission there would offer tools again
(checked on real opencode: {"*":"allow"} on the agent offered all ten
tools despite OPENCODE_PERMISSION={"*":"deny"}).
Follow-up to #5214.
The controller read its forge token from the swarm secret store once, when
its client slot filled, and kept sending it until the daemon restarted. A
token the forge re-minted or revoked left every forge-backed path failing
with 401 until someone restarted the controller, and a restart drops the
runtime-only job graph.
Every forge request now goes through `forge::api::Api::call`. On a 401 it
asks a `StoredSecret` (the queue clients' debounced, single-flight re-read
with backoff) for a token to try instead, rebuilds the forgejo client with
it, and sends the request once more. A second 401, or a store still holding
the refused token, surfaces the error as before. Only 401 counts: unlike an
IdP, the forge answers 400 to a malformed request, and a re-read started by
one would put a real 401 that follows inside the backoff.
forgejo-api bakes the token into its own reqwest client's default headers
and takes no client or middleware from outside, so there is no single
request function to hook. Each `self.api.<endpoint>(..)` site becomes
`self.api.call(|api| api.<endpoint>(..).into_future())`; option structs a
site moved into the call are cloned, since the closure can run twice. The
`OnceLock<Arc<Client>>` slot and its `get()` callers are unchanged.
Refs #5017
`swarmctl login --swarm X` failed with "unexpected argument", and
`login --help` listed no `--swarm`: the flag was only accepted before the
verb. `--swarm` and `--ca-file` are now global, so they parse on either side
of the subcommand and show in every subcommand's help.
`--swarm` / `SWARMCTL_SWARM` accept a bare domain and prepend `https://`.
A value with a scheme is parsed as before, so `http://` is still refused.
On a swarm-controller host, the swarmctl wrapper sets
`SWARMCTL_LOGIN_DEFAULT_SWARM=https://<swarm.domain>` (`--set-default`).
Only `login` reads it, after `--swarm` and `SWARMCTL_SWARM`. The wrapper
does not set `SWARMCTL_SWARM`: that would send every socket verb (agent
create, schedule, inference, forge) through the gateway and make each need
a login. Other verbs resolve exactly as before.
No CA file is wired. The only CA material the module names is the
controller's assembled trust bundle (system CAs plus the hive anchor).
`--ca-file` hands every certificate in it to rustls-platform-verifier as
an extra root, and that fails the whole client on any one it cannot
parse, so pointing it at the system CAs is a risk nothing here can test.
The controller daemon trusts the same CA through `SSL_CERT_FILE`, which
skips unparsable certificates; using that for swarmctl is left open.
Machines that are not swarm-controller hosts (an operator laptop) get no
default; that case waits on an answer in the issue.
Fixes#5239
The swarm settings gain `runtime` (`api-key`, the default, or `acp`) and
an `acp` provider record: an https base URL, the provider's model, and
its prices per million tokens. A PUT now changes only the fields its
body names, so the settings form that sends model, effort and limit
leaves the runtime alone; `null` clears `effort` or `acp`. The body stays
strict, and the stored document keeps #5192's lenient read.
The controller sends the runtime in each run request. On `acp` the
classifier runs one prompt on a new `opencode acp` session per
classification through hive-runtime's AcpRuntime, in a scratch
directory, with a config rendered from the settings. opencode runs
`read` and other tools without asking, so OPENCODE_PERMISSION
{"*":"deny"} takes every tool away from the model, and a deny-all policy
refuses any request on top. The title call, which resends the prompt,
is turned off. The deadline cancels the turn, and a turn that does not
end within 15s of that is dropped, which also covers an agent stuck
before its prompt. The answer is the final assistant text, as ACP has
no result event.
The provider key is read per run from
swarm/services/review-classifier/acp-provider, which the bao UI's
admins login may create and update, and reaches opencode through
AcpRuntime::with_provider_key and the trampoline's allowlist.
Each ACP classification's cost is computed from the configured prices
and the reported tokens, and added to
hyperhive.review_classifier.cost.usage labelled repo, pr, model and
currency.
Part of #5096
