The queue's metrics were scraped over loopback from the exporter, declared
only where the queue and the collector share a host, so a collector on any
other host had no job for it. Its target is now declared on every host as
`https://<swarm.nats.domain>/metrics`, and the queue's host gets a gateway
vhost on that name serving `= /metrics` from the exporter behind the
collector's auth subrequest (`/__nats_metrics_authz`, the forge's shape).
Every other path there is 403; the monitoring endpoint is never proxied.
The rules derived from `publishedScrapeTargets` admit the collector's
client and deny every other subject. The loopback entry, and with it
`glue-otel-scrape-targets.nix`, is gone.
In self-signed mode the vhost serves the queue's own `swarm-nats` leaf
(issued for exactly that name, `server_flag` set), copied 0640
root:nginx by `swarm-nats-gateway-cert`; until the leaf exists that unit
writes a throwaway pair, so nginx never starts on a missing file. The
name stays out of `swarm.serviceDomains`, so a host whose only swarm
service is the queue fronts no services name and its gateway neither
waits for nor needs the swarm-services leaf: no services-issuer pair has
to be placed on it. With ACME or an operator cert the vhost uses the
gateway's own TLS, as every other name does.
The queue's host now enables the gateway (`mkDefault`), as every other
swarm service's host does.
Fixes#5205
deploy.builder.enable defaults on under deploy.allSwarmServices (mkDefault),
where the host has the builder's store identity pair, as for the review
classifier. An explicit false still wins.
The two review tabs, runs and try-out, are one: /review. It is a column of
three sections: try a classification and the classifier settings, each a
folded bar (the settings' bar says what they are now), then the recent runs,
always open. The settings had been pinned under the try-out form; they are now
a section of their own. A new Disclosure primitive is the folding bar, a
native details wearing the Panel's classes, so the folded and open sections
read as the same surface.
/review-runs and /review-tryout are gone, with no redirect.
Three follow-ups to argus's review of #5237:
- A down secondary's retry runs `restore` under RESTORE_TIMEOUT (2 min).
Tries run one per tick on one future, so a restore that never resolved
held back every other down account's retries for good; a try that
outlasts the timeout now counts as a failure like any other.
- `client::build_and_restore` warns about a rejected token only for the
primary, which exits right after. For a secondary it logs at debug: the
retry loop already warns once per outage (`DownAccounts::failed` is news
on the first failure and on a changed reason, and an account that came
up starts afresh), so the extra warn repeated on every try.
- `a_secondary_that_comes_up_late_does_not_wipe_the_todos` runs the late
bring-up in a child process of the test binary with HIVE_AGENT_SOCKET
pointed at the test's listener. In-process the variable was unset, so a
wipe placed back in the bring-up path was a no-op and the test passed.
New tests: a_second_rejection_in_a_row_does_not_warn_again (captured logs, alone
in a child process since tracing caches callsite interest process-wide;
a stub whoami answering M_UNKNOWN_TOKEN, the real build_and_restore),
a_failure_after_the_account_came_up_is_news_again, and
an_account_whose_restore_hangs_does_not_hold_back_the_others (paused time).
Grafana's own metrics were scraped over loopback through the
`grafana-metrics` nginx vhost, declared only where grafana and the
collector share a host, so a collector on any other host had no job for
it. Its target is now declared on every host as
`https://<swarm.grafana.domain>/metrics`, next to the other published
targets, and grafana's vhost serves `= /metrics` behind the collector's
auth subrequest (`/__grafana_metrics_authz`, the forge's shape). The
rules derived from `publishedScrapeTargets` admit the collector's client
there and deny every other subject.
The loopback scrape is gone with the vhost that served it and its
`swarm.grafana.metricsPort` option, so a single host still scrapes
grafana once, now by name. A config that sets that option now fails
evaluation with "option does not exist".
On the published vhost `/metrics` was answered with 403 for everyone.
It now answers the collector's token, and a plain session gets authelia's
`deny`. The design note that ruled the published vhost out for scraping
predates the collector's credential; it is replaced by the location's
own comment and a line on `swarm.grafana.domain`.
Part of #5205
Two comments said authelia's rule requires `group:operators`. The group
is `admins` (`operatorGroup` in swarm-authelia.nix), and the rule is now
an `operatorsOnly` pair.
The apex's `operatorsOnly` pair was guarded on
`deploy.swarm-controller.enable`, which is true only on the host that
runs the controller and with it the swarm UI's vhost. When authelia runs
on another host, that vhost still asks it, but the authelia host
rendered no rule for the apex at all, so every session reached the
swarm UI and its `/api/` through `default_policy = one_factor`.
The pair now renders whenever the swarm domain is set, like the bao UI's
pair. No vhost on a host that doesn't serve the apex asks about it, so
the extra rules there are inert. authelia 4.39.20's `validate-config`
accepts them on an authelia-only host's rendered access_control.
module-eval-swarm-otel-identity gains a case on the existing
`autheliaOnly` fixture (authelia without the controller): the apex has
exactly the allow+deny pair, the rule list is closed, and an account in
no group is refused `/api/agents` while `group:admins` passes.
Fixes#5120
authelia takes the first matching rule, and a rule with a `subject` does
not match a subject outside it. The swarm domain's and the bao UI's
`group:admins` rules had nothing after them, so an account in no group
skipped them and landed on `default_policy = one_factor`: any
authenticated account reached the swarm UI, its `/api/` (by cookie or by
bearer) and the bao UI.
Each of the two group rules is now an `operatorsOnly` pair: the group
allow, then a subject-less `deny` for the same domain, the shape
`collectorOnly` already gives the collector's metrics rules.
`default_policy` stays `one_factor`, which the forge and chat rely on.
module-eval-swarm-otel-identity gains two cases on `allLocal`: an
account in no group is refused the swarm domain (four paths) and the bao
UI while `group:admins` passes and the forge stays open; and a
predicate, `autheliaSubjectRulesClosed` in lib.nix, holds over the
rendered list: every subject-scoped allow is closed by a later
subject-less `deny` on its domain with no subject-less allow in between,
and nothing follows a domain-wide `deny` on its domain.
Fixes#5120
Adds a build & check section, maps every workspace member, links
gotchas.md, and trims entries down to purpose + warnings (the rest lives
in each crate's //! doc). Also drops the comment in turn.rs that repeated
the module doc.
The agent's turn-state header ($SWARM.agent-state.<agent>) gains
`needs_login`, set by hive-agent from its own LoginState, so the flag
reaches the swarm without hive-c0re (mara on #5268: "id prefer the agent
container sending the info it needs login without hive").
- swarm-queue-client: `agent_login::Login`, the shared wire type, beside
`agent_inference::Effective`. `needs_login: Option<bool>` with
`#[serde(default)]`: a header from an older agent decodes as `None`
(unknown), never as "needs login" or "logged in".
- hive-agent: the header publisher reads the shared LoginState cell.
Every writer of that cell emits a status on the bus afterwards, so a
login change goes out on the next rebuild, not the 60 s refresh.
- swarm-controller: `agent_login::Reported` follows every agent's header
and keeps the latest flag; `GET /api/agents/status` merges it into each
row as `needs_login` (null when unknown).
- swarm-ui: AgentCard shows a warning-tone "login needed" badge when the
row's `needs_login` is true.
Refs #5268
The rain read only the OS reduced-motion query, once, at mount. The settings
menu's motion override (data-motion on the document) wins over the OS in
either direction everywhere else, and the shell applies it from an effect
that runs after this one. Whether the rain may move is now rainAllowed(), and
the component re-evaluates it on mount, on every document attribute change
and on the media query's change, starting or stopping the interval and
clearing the canvas.
The rain's column maths and fade colour move from MatrixRain into
shell/rain.ts, and the request-once swarm name loader from useSwarmName
into shell/swarm-name.ts. Both are plain functions with their input
passed in, so tests cover them; the components keep the DOM and canvas
work.
isFullBleedRoute and isWideBodyRoute are plain functions of a path, so
they move from Shell.tsx to shell/routes.ts, where the new tests cover
the start page, the hive directory's move to /hives, the terminal
paths and the wide-body routes.
/ is now the swarm's name over the digital-rain backdrop the hive
dashboard's home has, and the swarm name in the header links to it. The
hive directory moves to /hives, where the nav's hives tab points. The
swarm name is read once per page load by a shared hook, the header and
the start page both use it. Closes#5252
The form's one hint put the PR number first, so with the PR field empty
(which opening a PR doesn't need) it hid the repo or row problem that
was actually disabling the open-PR button. runProblem now checks the
rows first and the PR number last, and is tested.
Refs #5259
The commit author's email was the forge account's own address, read with
the site-admin token, which put an email the account may keep private
into the target repo's history. It is now the address Forgejo itself
uses for such an account's web commits, `<lowercased login>@<NO_REPLY_ADDRESS>`
(models/user/user.go GetPlaceholderEmail). NO_REPLY_ADDRESS defaults to
`noreply.<DOMAIN>` (modules/setting/service.go) and the forge module
leaves it unset, with DOMAIN set from services.hyperhive.swarm.forge.domain,
the same option SWARM_CONTROLLER_FORGE_URL is built from; so the domain is
that URL's host. The account is still read, to refuse a user with no
forge account, but its email no longer is.
Refs #5259
The try-out page's checks and request body get tests under the
package's node runner: the repo check, a row's checks in their order,
a shared name, and the open-a-PR gate (repo, then each row in order,
then a shared name, with no PR number). Also the row a request sends,
and the row a loaded rule becomes.
Refs #5259
push_classifier_branch's request moves into push_branch, which takes the
forge client, so a test can run it against a stand-in forge. The tests
pin the POST to /repos/o/r/contents and its body: base and new branch,
no force overwrite, the message, the operator as author with the
service account as committer, the base64 content, and update with the
blob sha or create without one.
The page's open-a-PR gating becomes rowsProblem in TryOutTypes.ts,
beside the other form checks. swarm-ui has no test runner, so the
typecheck is the only check on it.
Refs #5259
The try-out page gets an "open a PR" button. It posts the page's
classifications to the new POST /api/review-classifier/open-pr, which
merges them by name into the repo's rules file on the default branch
and has the review-classifier forge account commit the result on a new
branch, review-classifier/<user>-<8 hex>, through the contents API. The
commit's author is the forge account named like the session's
Remote-User, read with the controller's admin token. The answer links
to the forge's compare page, where the user opens the PR as themselves.
The push uses the account's existing token from the store
(swarm/services/review-classifier/forge/token, scope write:repository),
so the account needs write access to the repo; a refused push answers
403 naming that. In a YAML rules file, the entries the merge leaves
alone keep their text, as long as the result reads back as the merged
list; otherwise the file is written out whole.
DefaultBranchFiles now carries each file's blob sha, which the forge
needs to replace a file, and the repo's web URL for the link.
Fixes#5259
Refs #5096
On the API key runtime the classifier now hands claude its prompt as one
stream-json user message (`--input-format stream-json`) with two text
blocks: the PR's shared context, carrying `cache_control: ephemeral`, and
the classification's question.
As plain stdin text the whole prompt was one block, and claude puts its
only message breakpoint on the last block, after the question. The cache
writes an entry only where a breakpoint is, so no entry ended at the
shared-context/question boundary. Every classification wrote the whole
prompt and none read it (production: cacheRead ~0, cacheCreation ~input).
The system-prompt prefix (~270 tokens) is under the minimum cacheable
length. Captures against a local listener with a dummy key (claude
2.1.223) show the client breakpoint survives on the shared block, and that
claude adds none there on its own.
The PR stays in the user role: moving it into the system prompt would give
untrusted PR text the system prompt's authority.
`Model::ask` takes a `Prompt { shared, question }`. The ACP runtime joins
it back into the same text it sent before.
Refs #5245
The socket scheme needs VictoriaMetrics 1.153.0 or later: older builds
create the socket 0600 whatever the umask, so nginx could not connect
and every consumer would get 502 with the module evaluating clean.
deploy.victoriametrics.package is an operator option, so an assertion
now refuses an older one, naming the option and the reason.
Turning DynamicUser off moves the state directory from
/var/lib/private/victoriametrics (behind a /var/lib/victoriametrics
symlink) to a plain /var/lib/victoriametrics. A oneshot ordered before
the store moves the old directory there and chowns it to the static
user when the plain path is the old symlink, empty, or missing, and does
nothing once the private directory is gone. Without it the persistent
container's history depends on systemd's own DynamicUser-off migration.
Refs #4815
The metrics store's container gets privateNetwork = true with no
interface but loopback. VictoriaMetrics listens on
unix:/run/swarm-victoriametrics/victoriametrics.sock, a host directory
(root:nginx 0770) bind-mounted into the container, and every vhost
location that forwarded to 127.0.0.1:8428 now forwards to that socket.
The store runs as a static user whose primary group is nginx's gid,
with UMask 0007, so the socket it creates is 0770 group nginx. This
relies on VictoriaMetrics >= 1.153.0 honouring the umask (the locked
nixpkgs ships 1.153.0). nixpkgs' readiness loop curls a TCP URL, so
postStart is replaced with the same loop over --unix-socket.
swarm.victoriametrics.port is removed (nothing binds it any more), with
a mkRemovedOptionModule entry, and dropped from swarm-otel's host
port-collision assertion. The container's resolver unit is dropped:
there is no interface to reach the bridge resolver through, and the
store resolves no names.
The collector scrape, the collector push and Grafana's datasource
already reach the store by its published domain, so they are unchanged.
Refs #4815
Refs #3560
DynamicUser implies PrivateTmp=disconnected (systemd.exec(5)), a tmpfs
of the unit's own rather than a private directory reachable from the
host namespace. The explicit setting now matches what the unit ran
with before it moved to a static user.
Refs #5235
With DynamicUser=true and RuntimeDirectoryPreserve=yes, systemd creates
the unit's runtime directory under /run/private/ (0700 root:root) and
symlinks /run/swarm-logs-auth to it (systemd.exec(5),
RuntimeDirectoryPreserve=). nginx runs as the unprivileged nginx user, so
its connect() to /run/swarm-logs-auth/auth.sock failed with EACCES, the
auth subrequest ended in a 502, and auth_request answered every agent
bearer on /select/logsql/ with a 500.
The unit now runs as a static swarm-logs-auth system user and group, the
shape swarm-controller already uses for its 0751 + Preserve socket dir.
RestrictSUIDSGID and RemoveIPC, which DynamicUser implied and the unit
did not set itself, are set explicitly.
module-eval-core-toggle gains a case that fails when any unit creating
the runtime directory of a unix: upstream of nginx combines DynamicUser
with RuntimeDirectoryPreserve. docs/process/gotchas.md records the trap.
Fixes#5235
TestNats picked a port by binding 127.0.0.1:0, dropping the listener
and handing the number to nats-server. Until that server bound it, the
port was free for a concurrent test's bind(0) to return again. Both
servers were then started on it (with the same store dir, since that was
keyed by pid and port); the loser exited silently and both tests'
clients reached the winner. term_stream's tests all use the stream
term-main-iris, and one of them creates it with max_message_size 4096,
so a test sharing its server got "stream name already in use with a
different configuration" (10058) from open_or_create's
get-then-create.
The server now binds a port of its choice (-p -1) and announces it in a
ports file, and each server has its own tempdir for the store and that
file, so a test can only reach the server it started.
Fixes#5261
The runtime list was exported but only its type was used, so the type is
now written out. The form's pure logic (draft, checks, request body,
change detection) gets tests under node's built-in runner, run by the
existing swarm-ui check.
The classifier settings page showed model and effort only, so the API-key
or ACP runtime could be switched only through a raw API call. It now has a
runtime select, and under ACP the provider's URL, model and prices. A PUT
carries the provider only on the ACP runtime, so the stored record stays
for a switch back. Closes#5249
The startup todo wipe runs once in main, before any sync loop, so a
secondary that comes up later cannot clear the todos the running
accounts' sweeps already deduped. Nothing tested that.
main now reads HIVE_AGENT_SOCKET once at startup and hands the path to
wake::clear_all_todos, the only way left to wipe the whole set;
send_todo_clear clears one key only. bring_up_account is split into
restore_account (everything that talks to the network) and the
synchronous account_sync_loop, and the retry path restores with the
former and builds the loop itself. Neither the loop nor the retry path
is given the socket.
The new test runs a late secondary's retry and its sync loop against a
listener standing in for the harness socket, after startup's wipe, and
asserts the harness saw exactly one wipe. It pins that seam, not every
place in restore_account a wipe could be added.
The retry loop's body was only reachable through bring_up_account, which
needs a live homeserver, so nothing exercised what happens when a down
secondary comes up or fails, or that the Registry's new RwLock is never
held across the network call. The unit tests covered the pure decision
(DownAccounts, retry_delay) only.
retry_down now delegates to retry_loop / retry_once, which take the
bring-up step and the snapshot path as parameters; retry_down passes
bring_up_account and the real path, so behaviour is unchanged. Tests use
offline matrix-sdk clients and fake bring-ups to show that:
- a secondary that comes up is inserted, leaves the down set, has its
sync loop sent on and is published in the snapshot, while the primary
and the other accounts stay the same clients;
- one that fails (error or no token) stays down, is not published, and
waits twice as long next time; one that is not yet due is not tried;
- readers are not blocked while a bring-up is pending;
- inserts under four busy reader threads neither deadlock nor lose an
account;
- the loop, under paused time, retries at 2, 6, 14 and 30 minutes.
tokio's test-util feature is added as a dev-dependency for the last.
Audit follow-ups on the secondary-account retry.
The accounts snapshot is written by the heartbeat task and, now, by the
retry loop. Both used one fixed tmp path, so concurrent writers could
clobber it or fail the rename. Each write now names its own tmp file
(pid + counter); a test hammers it from four threads.
A permanently bad secondary logged three warnings per attempt, every 16
minutes, for as long as the daemon ran. DownAccounts::failed now reports
whether a failure is news (first, or a different reason from the last);
only that is a warning, repeats are debug. The two per-attempt warnings
in client.rs and credential.rs are covered by that line, which names the
reason, so the store-token one drops to debug.
The comments in client.rs and a credential.rs test, and persistence.md,
still described the old skip-until-restart behaviour or implied that a
store-listed account without a credential is retried here; that case is
handled by the linked-name set changing, not by this loop.
The daemon's two-minute refresh compared the store's linked account
names with the names it booted with. A secondary that was named but did
not come up - no token at bring-up, a token the homeserver rejected
(M_UNKNOWN_TOKEN), or a homeserver still down after the ~52 s boot
retries - kept its name in the set, so nothing ever noticed it again:
it stayed down until the daemon restarted for some other reason, and
an operator re-linking a rejected account changed no name.
Each such secondary is now tried again on the same two-minute period,
one per tick, after one refresh and then doubling waits capped at
sixteen minutes. One that comes up joins the registry (now behind a
lock) and its sync loop joins the running set; the primary and the
other accounts are not restarted. The store is still only read.
The startup todo wipe moves out of bring_up_account to run once before
any sync loop: run per account, a late secondary would have cleared the
todos the running accounts' sweeps had already deduped.
Part of #5017 (slice M).
opencode writes the session's cwd into the system message's environment
block ("Working directory: …", "Workspace root folder: …"), ahead of the
PR's user message. Each ACP classification ran with its per-call scratch
directory, /tmp/review-classifier-<pid>-<counter>, as cwd, so every call
sent a different system message and the provider's prompt cache could not
match past it.
The session's cwd is now one fixed directory, <tmp>/review-classifier-acp-cwd,
shared by every run. HOME (the config, system.md and opencode's own state)
stays a directory per run, removed after it, as before. With no tool and
no project config nothing writes into the cwd: a live opencode 1.15.10 run
through initialize, session/new and a prompt left it empty, with its state
all under HOME. So concurrent runs can share it, and no state carries from
one run to the next.
Refs #5245
VictoriaLogs' own metrics were scraped over loopback, declared only on
the host running the store, so a collector on any other host had no job
for it. Its target is now declared on every host as
`https://<swarm.victorialogs.domain>/metrics`, next to forgejo's and
authelia's, and the store's vhost serves `= /metrics` behind a bare
`auth_request` to authelia. The rules derived from
`publishedScrapeTargets` admit the collector's client there and deny every
other subject. The loopback entry is gone, so a single host still scrapes
the store once, now by name.
On that vhost `/metrics` was reachable through the `/` location by any
authelia session. The exact-match location now takes it, so a session
gets a `deny` on `/metrics` and nothing else on the name changes.
Part of #5205
A role file is read into every subagent dispatch that names it, and
nothing measured its size. A new watch on the same 15-minute tick as
claude_md_watch records `hyperhive.agent.subagent_role.lines`, one
series per `<role>.md` under the state dir's `subagent_roles/`,
labelled `role` with the file stem. Metric only: no todo, no threshold.
Fixes#4558
Every classification's shared context began with the PR's head sha,
headed each commit with its sha, and carried each file's
`index <hash>..<hash>` line in the diff. A push or rebase rewrites all
of those, so the provider's prompt cache missed from the first line of
the user message even when the commit messages and hunks were
unchanged (#5245: "cache hits show as 0.1x").
The header now names only repo#pr, each commit is headed by its
position, and the diff drops `index` lines before lockfiles collapse.
The `diff --git` paths and every hunk stay. The TryOut's head_sha is
untouched: the controller still posts the review against it, and the
journal line still logs it.
On the ACP runtime the cost sink also adds each classification's tokens
to `hyperhive.review_classifier.token.usage`, typed input / output /
cacheRead / cacheCreation like claude's own `claude_code.token.usage`,
so the hit rate is a number in the metrics store on both runtimes.
No access rule covered the push route on `metrics.<swarm>`, so a request
there fell to `default_policy = one_factor`: any logged-in session cookie
for the swarm could write samples into VictoriaMetrics.
The name now gets two rules with no `resources`: the collector's client
at `one_factor`, then `deny`. Only the push route and `/metrics` on that
vhost send an authelia subrequest (the read API, Grafana's datasource
included, is served with no check), and the collector is the one caller
of both, so a domain-wide pair covers them with no path or query regex
to slip past. It replaces the per-path `/metrics` pair the published
target would otherwise derive for that name. Both rules sit ahead of the
apex and store-UI rules, and authelia matches `domain` exactly, so they
shadow no other name.
Authelia 4.39.20, read from source: a rule with no `domain` wildcard
matches with `strings.EqualFold` (access_control_domain.go), an empty
`resources` or `subject` list matches everything (access_control_rule.go),
and the first matching rule wins (authorizer.go).
Fixes#5218
The store's own /metrics was scraped over loopback, declared under
deploy.victoriametrics.enable, so a collector on another host never had
a job for it. Declare it as a published target on every host instead,
keyed on swarm.victoriametrics.domain, and serve it on the metrics vhost
behind the existing authz subrequest. Authelia's per-target rules (one
collector rule, then deny) now cover it too.
The loopback entry goes, so a single-host swarm scrapes the store once.
The ACP runtime handed the classifier's system prompt to hive-runtime's
system_prompt_file, which prepends it to the first user message, so the
instructions and the PR text sat in the same user turn.
The rendered opencode.json now defines a review-classifier agent whose
prompt is {file:./system.md} and selects it with default_agent. opencode
1.15.10 sends an agent's prompt as the system-role message (in place of
its per-model default prompt), followed only by its environment block;
the first user message is the PR material alone. The API key runtime
keeps --system-prompt.
The agent carries no permission or tools key, and a test pins its exact
shape: opencode merges an agent's own permission after
OPENCODE_PERMISSION, so a permission there would offer tools again
(checked on real opencode: {"*":"allow"} on the agent offered all ten
tools despite OPENCODE_PERMISSION={"*":"deny"}).
Follow-up to #5214.
`swarmctl login` (and any `--swarm` verb) on a controller host failed TLS
with `UnknownIssuer` against the swarm's own gateway: its leaf chains to
the store's `pki` root, which no system store holds, and the only way
through was `--ca-file`/`SWARMCTL_CA_FILE` by hand.
The host's swarmctl wrapper now sets `SSL_CERT_FILE` to the bundle the
swarm-controller daemon already trusts (`/run/swarm-controller-ca/
trust-bundle.pem`: system CAs plus the swarm's root), when the gateway is
self-signed. swarmctl's reqwest/rustls resolves roots the same way the
daemon's does, so this is the path the daemon uses to reach the same
gateway. `--set` rather than a default, since a nix shell exports its own
`SSL_CERT_FILE`.
Pointing `SWARMCTL_CA_FILE` at the bundle instead would not work (#5243):
`--ca-file` adds each certificate as a root and fails the whole client on
one it can't parse.
Hosts without the controller (an operator laptop) still need `--ca-file`.
Refs #5239Fixes#5243
