#5177 added five forge requests under the controller's token that went
straight to the forgejo client: the commit status in
`Client::create_commit_status`, and in `forge/builder_account.rs` the
hives-team add, the user read, the repo-creation lockdown and the team
membership read. Each now goes through `Api::call`, so a 401 re-reads
the token from the store and sends the request once more, like the
other controller forge calls.
The read-back of the builder's newly minted token in
`ensure_builder_forge_token` stays a direct client: it sends the
builder's own token, not the controller's, the same as the read-backs
in `agent_token`, `classifier_account` and `hive_token`.
The controller read its forge token from the swarm secret store once, when
its client slot filled, and kept sending it until the daemon restarted. A
token the forge re-minted or revoked left every forge-backed path failing
with 401 until someone restarted the controller, and a restart drops the
runtime-only job graph.
Every forge request now goes through `forge::api::Api::call`. On a 401 it
asks a `StoredSecret` (the queue clients' debounced, single-flight re-read
with backoff) for a token to try instead, rebuilds the forgejo client with
it, and sends the request once more. A second 401, or a store still holding
the refused token, surfaces the error as before. Only 401 counts: unlike an
IdP, the forge answers 400 to a malformed request, and a re-read started by
one would put a real 401 that follows inside the backoff.
forgejo-api bakes the token into its own reqwest client's default headers
and takes no client or middleware from outside, so there is no single
request function to hook. Each `self.api.<endpoint>(..)` site becomes
`self.api.call(|api| api.<endpoint>(..).into_future())`; option structs a
site moved into the call are cloned, since the closure can run twice. The
`OnceLock<Arc<Client>>` slot and its `get()` callers are unchanged.
Refs #5017
The swarm settings gain `runtime` (`api-key`, the default, or `acp`) and
an `acp` provider record: an https base URL, the provider's model, and
its prices per million tokens. A PUT now changes only the fields its
body names, so the settings form that sends model, effort and limit
leaves the runtime alone; `null` clears `effort` or `acp`. The body stays
strict, and the stored document keeps #5192's lenient read.
The controller sends the runtime in each run request. On `acp` the
classifier runs one prompt on a new `opencode acp` session per
classification through hive-runtime's AcpRuntime, in a scratch
directory, with a config rendered from the settings. opencode runs
`read` and other tools without asking, so OPENCODE_PERMISSION
{"*":"deny"} takes every tool away from the model, and a deny-all policy
refuses any request on top. The title call, which resends the prompt,
is turned off. The deadline cancels the turn, and a turn that does not
end within 15s of that is dropped, which also covers an agent stuck
before its prompt. The answer is the final assistant text, as ACP has
no result event.
The provider key is read per run from
swarm/services/review-classifier/acp-provider, which the bao UI's
admins login may create and update, and reaches opencode through
AcpRuntime::with_provider_key and the trampoline's allowlist.
Each ACP classification's cost is computed from the configured prices
and the reported tokens, and added to
hyperhive.review_classifier.cost.usage labelled repo, pr, model and
currency.
Part of #5096
swarm-controller README, the swarmctl inference doc comment (and the
regenerated swarmctl-cli.md) and the hive-agent swarm_inference module doc
still listed only model, effort and subagent models.
Nothing capped how many subagents an agent runs at once; the operator
wants to set that per agent from swarm-ui (#5234, shape (b)).
AgentInference gains `max_subagents: Option<u32>`. An absent field
decodes as None, so every stored value still parses and means "no cap",
which is today's behaviour.
swarm-controller accepts 1..=32 on PUT. 0 would refuse every start; 32
is far more concurrent claude sessions than one agent container has
memory for, so anything above it is a typo, not a cap.
The subagent daemon already read the agent's agent-inference key for
subagent_models; it now reads it on every start and continue (not only
those naming a model) and refuses a new run once the cap's worth of
names is tracked, counted under the same lock as the name reservation.
Every tracked slot counts: running, starting, a goal run between turns,
and an interrupted one still exiting. A goal re-prompt never goes
through the reservation, so it is never refused. A failed read applies
no cap, as it falls back for models.
swarmctl's `inference set` does GET, merge, PUT; its mirror type now
carries the field so a set from the CLI keeps a cap set in swarm-ui
instead of dropping it, and `get` prints it.
argus's review on #5177: the git credential helper answered every `get`,
so a config PR's own flake inputs could have git send the builder's forge
token to any host. The helper is now configured under
`credential.<forge-url>.helper` and itself answers only an https request
whose host is the forge's; a test runs it as git does and checks that
other hosts, http and non-get operations get nothing.
Builds are bounded: every `build_agent` node needs the one `builder` jobq
resource, so one runs at a time and the rest wait pending, and a newer
head cancels its PR's pending build (a running one finishes and writes on
its own commit). The builder also holds a one-permit semaphore around each
build.
`builder` and `review-classifier`, the forge logins swarm-controller owns,
join nix/reserved-names.nix, with a test tying the list to both constants.
The builder is now opt-in like the CI runner: `allSwarmServices` no longer
turns it on, since it runs unreviewed config PRs through the host's nix
daemon. mara had not answered atlas's question on the PR.
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).
When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.
The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.
Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.
This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
An agent's matrix encryption keys lived only in its local crypto store, so
`destroy --purge` or a primary-account token reset lost every encrypted
room's history with no way back (#5022).
swarm-controller now mints a random passphrase (32 bytes from the kernel's
CSPRNG, base64url) for every account listed under each agent's `matrix/`
directory, at `swarm/agents/<agent>/matrix-recovery/<account>`, in a
five-minute pass. The write is a cas=0 create after a read; a path that
holds anything is never replaced, and a create that loses a race counts as
done. The controller's existing `data/swarm/agents/*` grant covers the
write and the agent's own `agents/<agent>/*` read grant covers the read, so
no policy changes.
hive-matrix-daemon reads the passphrase when it brings an account up, after
cross-signing: with no backup on the homeserver it enables recovery with it
(backup + secret storage); with one there it recovers, which imports the
backup key and the self-signing key, and room keys then download through
`BackupDownloadStrategy::OneShot`. No passphrase yet logs at INFO and the
account comes up without backup; any failure logs WARN and never stops the
daemon.
Recovery runs after `ensure_cross_signing`, not before: on a new account
cross-signing must exist locally for `enable` to export it into secret
storage, and on an existing identity `bootstrap_cross_signing_if_needed` is
a no-op, leaving `recover` to import and sign.
Part of #5022 (S1+S2, with the S4 docs).
Forgejo's markdown renders $...$ and $$...$$ as KaTeX; a reason quoting
nix `${...}` put two $ in one cell and the span between them became a KaTeX
parse error. The cell escaper now backslash-escapes $ like its other markup.
Rule source: the math inline parser triggers on '$' and only starts a span
there (modules/markup/markdown/math/inline_parser.go, Trigger/Parse in
go-gitea/gitea, which Forgejo carries); goldmark's CommonMark backslash
escape consumes '\$' first, so the parser never sees it. Escaping each $
also covers '$$'. Backslashes are already escaped, so '\\$' stays literal.
hive-c0re (`hive-<name>`), swarm-controller and swarm-review-classifier
read their queue client secret from the store once and held it as
`ClientSecret::Value` for the life of the process. After authelia
re-minted a secret and swarm-secret-publish stored the new one, every
later token mint presented the old value and was refused until the
daemon restarted.
`ClientSecret::Stored` replaces `Value`: a `StoredSecret` holds the
secret in memory with a caller-supplied read callback, so the crate
takes no store dependency. When the token endpoint answers 400 or 401,
`mint_token` (and its blocking sibling, used by the controller's OTLP
exporters) re-reads it and, if the store holds a different secret,
mints once more with it. One re-read runs at a time and is spawned so
it outlives a caller that gives up. A re-read that fails or finds the
refused value delays the next one, 5 s doubling to 5 min, so a secret
that stays wrong costs one store read per delay. Logs name the client
id and never the secret.
swarm-nats-auth's `oidc_secret` was the template; its re-read machinery
now sits on `StoredSecret`, so the tree has one copy. The responder
gains the same re-read backoff.
The controller keeps its `IDENTITY` OnceLock: it now holds a
`QueueConfig` whose secret is re-read, and the lazily built vcs-metrics
exporter reads the identity from it.
agent_renewal.rs's module doc said a re-minted agent queue secret only
reaches the agent at its next start; hive-agent reads the store on every
connection attempt after its first, so the doc now says that.
Part of #5017 (slice Q).
A reconcile projected only the definitions this build decodes, so after
a rollback an agent's already-projected schedule whose definition a newer
build rewrote was dropped from its key within a minute and stopped
firing. The projection now keeps an agent's stored entries whose id
belongs to an unreadable definition, as they were; it never adds one to
a key that did not hold it. A deleted readable definition still leaves
every key.
After a rollback, a record a newer build wrote (a new enum variant, a
changed required field) stopped decoding, and the strict
`schedules::read` turned that into an error on every read and write of
the key until someone deleted it by hand.
- agent_limits / inference (one document per agent): an undecodable
document reads as unset, with a WARN naming the bucket, key and error.
The PUT's "equal to stored, skip the write" shortcut no longer applies
to an undecodable document, so a PUT of the defaults replaces it too.
- schedule definitions (one document, every schedule): salvaged per
definition, as the review-classifier run list is. An undecodable
definition is kept as its raw JSON, hidden from the list, edits and
projections, written back by every commit after the readable ones, and
its id is never minted again. A document with no list of schedules
stays an error: reading it as empty would empty every agent's key.
- per-agent agent-schedules: an undecodable key reads as no schedules
and is rewritten on the next projection, even when the agent has none.
`schedules::read` is replaced by `read_or_default`; `wanted` keeps its
own strict read.
Closes#5193
A run record this build couldn't decode was dropped from the list it
read, and the next recorded run wrote the list back without it, so a
rollback followed by a roll-forward lost the newer build's runs for
good. The stored list is now a list of entries, each a decoded run or
the raw JSON of one that didn't decode. The route still shows only the
decoded runs. An undecodable entry keeps its place, counts toward the
100, and is written back as it was read. A read that finds any logs one
WARN with their count and the first error.
The settings decode derived its known fields from what
`Settings::default()` serializes, so a later field skipped when unset
would have been stripped from a valid stored document and lost on the
next PUT. The known fields are now an explicit list. A test serializes
a fully populated `Settings`, checks that its keys match the list, and
checks that `decode` round-trips it.
The controller keeps the classifier's newest runs as one JSON array and
its settings as one JSON object, each under one KV key. A document this
build could not decode, as after a rollback past a build that added an
outcome or a settings field, failed every read and every write until
someone deleted the key by hand. A classifier newer than its controller
refused every run request for the missing `model`.
The run store now decodes the array one record at a time: a record that
does not decode is dropped with a WARN naming the bucket, key, index and
error, and a document that is not an array reads as no runs. Listing
keeps working, and the next recorded run writes back only the records
that decoded.
The stored settings ignore a field this build does not know; one that
still does not decode reads as the defaults with a WARN. The PUT body
keeps `deny_unknown_fields`. A PUT now always writes, so a PUT of the
defaults also replaces an undecodable document.
A run request without `model` runs on haiku, the swarm-wide setting's
default, now one constant in swarm-queue-client.
`schedules::read` gains a sibling, `read_with`, taking the decoder.
Closes#5162
Re-creating an agent mints its store identity the same way, so the
backfill verb and the `POST /api/agents/{name}/identity` route that only
it called go. The `MintAgentIdentity` job node stays: agent creation and
the renewal pass still queue it.
Part of #4765 (c).
The classifier ran every classification on a hard-coded `haiku` with no
`--effort`, and the controller ran at most two requested reviews at once,
also hard-coded, so changing any of them took a code change and a
redeploy.
swarm-controller now keeps one settings document,
`{model, effort, max_concurrent_reviews}`, in its own
`review-classifier-settings` KV bucket under `all`, the way it keeps the
schedule definitions. No document means today's values: haiku, no
`--effort`, two reviews. `GET /api/review-classifier/settings` reads it;
`PUT` replaces it whole, needs the gateway's `Remote-User` like the
try-out POST, and refuses a bad model (the inference route's check), an
effort outside the five levels or a limit outside 1..=8 with
problem+json 400.
Every run request (try-out and requested review) now carries the model
and effort, and the classifier passes them to claude instead of its
constant, which is gone. A try-out reads the settings when it is posted;
a review reads them for the limit before it waits for a place and again
for the model and effort once it has one. The review semaphore is
replaced by a running count under a resizable limit, so a PUT, or the
next review's read, changes how many run without a restart, and a lower
limit lets the runs already going finish. MAX_WAITING_REVIEWS is
unchanged.
Closes#5151, part of #5096.
Every run is recorded when it ends: a try-out when its reply settles it,
a requested review once its review is posted or the post fails, and a
request turned away by a full queue once its comment is posted. A
duplicate request that starts nothing is not recorded. The newest 100
are one JSON array under `recent` in a new controller-owned KV bucket,
`review-classifier-runs`, written with compare-and-set as
`schedule-definitions` is (five attempts). A failed write is logged; the
review is already posted by then.
Each recorded run is also one info event, "review classifier run
finished", with the record's fields as tracing fields. The controller
logs natively to the journal, so they reach VictoriaLogs as F_<NAME>.
GET /api/review-classifier/runs returns the list, newest first, gated
by the gateway like the try-out GET. post_classifier_review now returns
the posted review's html_url, which the record keeps as review_url.
Closes#5153, part of #5096.
Eight classifications in .hyperhive/review-classifier.yml: no AI
advertising in commit messages, no committed secrets, no clippy
allow attributes, no debug leftovers, no dead code, tests with
behaviour changes, docs with operator-facing changes, one purpose per PR.
A swarm-controller test parses the real file so it can't rot, and the
rust source fileset in nix/rust.nix includes it so the test finds it
in the build sandbox.
systemd ignores `08%` and `085480%` for CPUQuota= and `08%` for MemoryMax=
("Invalid ... ignoring"), which leaves the agent uncapped. `hundredths` now
refuses a whole part that starts with `0` unless it is exactly `0`, for both
fields. `0.5%` and `0.01%` stay accepted.
systemd ignores a CPUQuota= or MemoryMax= it can't parse ("Invalid ...,
ignoring") and runs the unit without that limit. A swarm cap replaces
the hive's answer, so such a value left the agent uncapped while the
route, GET and the dashboard all said it was capped. argus on #4268 (R3)
found values the route took that systemd ignores: 12.345%, 400.123%,
0.001%, 17000000T, 99999999999999999999T and 99999999999999999999%.
The validators now accept a subset of what systemd parses, measured
against `systemd-analyze verify` (systemd 260):
- a percentage takes at most two decimals and is above zero, up to 100%
for memory_max and 21474836.47% for cpu_quota (systemd keeps the quota
in hundredths of a percent in a C int, not a u64)
- a size is one parse_size term: digits, an optional fraction, then
nothing, B, K, M, G or T, case-sensitive. The whole part and the
fraction digits fit u64, the whole part (plus one with a fraction)
fits u64 once scaled, and the byte count is at least 1 and below
2^64 - 1, as systemd checks them.
P and E, which systemd reads, stay refused because hive-c0re's validator
refuses them, and one value the hive refuses stops every cap update on
that hive. Values are stored as given: lower-case suffixes and a B
after a multiplier are refused rather than normalized, as systemd
refuses them.
The newline-injection values from the hive's test are in both bad lists
and go through the router in a_cap_with_a_second_line_is_a_400_and_the_caps_stay.
GET reads through a new agent_limits::open that returns None for a
missing bucket, answered as {}, instead of creating the bucket.
An agent's CPU and memory caps, set from the swarm. Keyed by the agent
alone, in a new `agent-limits` KV bucket that swarm-controller alone
writes, the same shape as `agent-inference` (#5034): mara on #4268,
"agents are not tied to a hive. dont tie the limits to the hive."
PUT {"cpu_quota": "400%", "memory_max": "8G"} every field optional,
replaces the stored value
GET the stored caps, {} when there are none
DELETE 204
PUT refuses an agent no hive's wanted state declares (as the inference
route does), unknown fields (422), zero and anything systemd does not take,
before the queue is asked, and stores memory in the spelling systemd takes
(8gb -> 8G). The hive hands these values to systemd verbatim.
The routes sit under /api/, behind the swarm UI vhost's authelia
subrequest like every sibling; the core-toggle module-eval case pins that
gate.
The bucket's reader on the hive comes in the next commit.
Contributes to #3865.
Two read-only routes feed them: GET /api/pulls/repos lists the repos with an
open pull request, and GET /api/pulls?repo=owner/name lists a repo's open
pull requests, highest number first. The page offers the lists through the
browser's own suggestion list and keeps both fields free text, since a run
also works on a merged or closed PR.
Follow-ups from the review of #5142.
A `Done` answer with fewer or more rows than the classifications sent, or
none, now posts the no-verdict comment instead of the classifier's overall
verdict, so a short answer can't approve.
Requests are accepted per repo, PR and head into a set bounded at the two
running plus eight waiting. A request for a key already running or waiting
starts nothing and logs at info; a request with every place taken starts no
run and posts a no-verdict comment saying so. Before, every delivery spawned
a task waiting on the semaphore, so toggling the review request queued one
paid run per toggle.
Table cells get a zero-width space after `#`, `!`, `://` and `www.`, so a
model's reason can't render an issue or PR reference or a bare URL as a link.
The secret publisher's policy denies the classifier's forge token leaf, the
way #5139 denied Grafana's admin password, with a module-eval-bao-grants arm
pinning it. The controller posts reviews with whatever token is there, so a
publisher write picked the account reviews come from.
Closes#5143, part of #5096.
Requesting the forge account `review-classifier` as a reviewer now runs the
repo's rules against the PR and posts the result as that account's review,
on the head commit the run read. A push starts no run; only a fresh review
request does.
- An instance-wide `pull_request_review_request` system hook delivers to
`/webhook/forge/review-request`. A request for any other reviewer, or a
removed request, does nothing.
- The controller reads the PR, loads the rules with the same reader as
`GET /api/review-classifier/rules`, sends one request on the try-out
subject, and posts APPROVED or REQUEST_CHANGES with the table as the body.
- With no clean verdict (PR or rules don't load, the run fails, or any
classification has no answer) it posts a COMMENT saying why, never an
approval. The event is one function, `no_clean_verdict_event`.
- Table cells are flattened, markup-escaped, and every `@` gets a
zero-width space, so a model's reason can't break the table or mention
anyone.
- At most two requested reviews run at once; the rest wait in memory.
- The account is minted the way hive accounts are: a five-minute pass
queues a `MintClassifierForgeToken` node when the token at
`swarm/services/review-classifier/forge/token` is missing or stale. The
token is `write:repository`. The controller's bao policy gains that leaf
(create/read/update), pinned in module-eval bao-grants. The classifier
never sees the token.
The account gets no repo access automatically; a repo admin adds it by
hand. A re-request retires the account's earlier review: Forgejo clears
`official` on a reviewer's earlier reviews when it is requested again
(models/issues/review.go AddReviewRequest, v16.0.5), and approvals and
rejection blocks count official reviews only.
Refs #5137
The rules file is now read with the existing forgejo-api client: the
commit's root tree, then the `.hyperhive` tree, then the
`review-classifier.{yml,json}` blob, base64-decoded. Every request takes
single-segment path parameters, so nothing depends on how a nested path
is escaped, and the controller keeps one HTTP transport to the forge.
The second reqwest client, `Client::with_token` and `raw_file_url` are
gone.
A tree the forge truncated is an error, not a missing file. A 404 on a
tree or blob doesn't come back as a forge 404, so only the repo lookup's
404 reads as "no such repo".
The `is_repo_not_found` doc comment is back on its function.
Part of #5096.
forgejo-api escapes the whole file path of a raw-file request, so the
rules file went out as `raw/.hyperhive%2Freview-classifier.yml` and
reached the forge only if every proxy and router in between decoded the
`%2F`. The controller now sends that one request itself: `raw_file_url`
escapes each path segment on its own and joins them with `/`, and
`forge::Client` holds a second HTTP client with the same token for it.
A 404 from the default-branch lookup now means "no commit on the
default branch", like an empty repo, instead of falling through to the
route's "no repo" answer. Only the repo lookup's 404 still means the
repo doesn't exist.
Tests cover the URL shape, the reads against a loopback stub forge
that serves only the literal path, the branch 404, 503 with no forge
access, 503 at the forge deadline on a paused clock, and a bad
polarity value.
Part of #5096.
`GET /api/review-classifier/rules?repo=owner/name` reads
`.hyperhive/review-classifier.yml` or `.json` from the repo's default
branch with the controller's forge token, on every request, and answers
the classifications in the try-out POST's shape, plus the file, the
branch and the commit it was read at. The swarm UI's try-out page loads
them into its editor (#5134).
The file is one `classifications` list of the try-out's `Classification`.
A missing file, both files, a parse error, any unknown key, an empty list
or more than ten entries is a problem+json error, never a pass. The
checks the POST makes on its classifications move into one function both
paths call.
The doc names the file and its schema, replacing the placeholders, and
documents the route.
Part of #5096.
The routes' annotations now list every 400 reason (including the JSON
shape rejections), the 503 causes, the 31-minute running window, the
404 after a controller restart, and that an errored row reads
not-applicable with certainty 0 and an empty reason, so the page keys
off error, and that overall can be approve beside errored rows.
POST /api/review-classifier/try-out validates the run (owner/name repo,
a PR number, one to ten classifications with distinct names, thresholds
up to 100), reads the PR's head commit, commits and diff with the
controller's forge token, sends one request to the review classifier on
the queue and answers 202 with an id. GET .../try-out/{id} reports the
run as running, done (rows and overall verdict) or failed (the error).
Results live in memory for an hour after the reply; there is no run
store. POST then GET because a run can outlast the gateway's 60-second
proxy read timeout. Both routes are utoipa-annotated, so they show on
/api/docs/.
A replay consumer now has a 5s idle heartbeat, and the row stream ends at
the first consumer error instead of skipping it. When the server drops the
consumer, as on a restart, two missed heartbeats end the SSE response and
the browser reconnects with Last-Event-ID. Before, the response stayed
open and delivered nothing.
A Last-Event-ID above the stream's newest sequence now replays as a fresh
reader. The server accepts a start past the end and delivers nothing
until the stream reaches it, so a stale id stalled the replay.
swarm-controller's stream tests run against a real nats-server on
loopback (`test_nats`, binary from HIVE_NATS_SERVER, exported by
checks.nix and devshell.nix). They cover: ensuring the stream twice, an
existing stream with other limits being opened as it is, the replay
arriving before live rows, a reconnect getting every later row exactly
once, and a dropped consumer ending the row stream.
The limits test now compares against the subagent stream's own config.
Two comments that described the agent's terminal route as live-only are
corrected.
swarm-controller now creates `term-main-<agent>` for every live agent, in
the same one-minute pass and with the same config style as
`term-sub-<agent>` (#4897): subjects `$SWARM.term.<agent>` only, max_age
24h, max_bytes 64 MiB, discard Old, max_message_size from the queue's
max_payload. An existing stream is opened as it is. The agent keeps its
plain publish on `$SWARM.term.<agent>` and gets no JetStream subject.
`GET /api/agents/{name}/term/stream` creates an ephemeral push consumer on
that stream starting 100 rows back, so a swarm-ui terminal opens with
recent history and then follows new rows, in order, from one
subscription. Each SSE event carries the row's stream sequence as its id,
and a reconnect's Last-Event-ID resumes after it. Without the stream the
route falls back to the live core subscription it used before.
The stream is `term-main-<agent>`, not `term-<agent>`: an agent may be
named `sub-x`, whose `term-<agent>` would be agent `x`'s subagent stream.
Its one literal subject cannot match `$SWARM.term.<agent>.sub.>`, which
needs more tokens, so the two streams never overlap.
The reader gains `$JS.API.CONSUMER.CREATE.*.*.$SWARM.term.*`, the
extended create form whose last tokens are the consumer's filter. Checked
against nats-server 2.14.4, the trailing `*` also admits a literal `*` or
`>` filter, so a consumer under it reaches any terminal row, main or
subagent; those are rows the reader already subscribes to and reads with
`STREAM.MSG.GET.*`.
Closes#4826
swarm-controller re-issues an agent's store leaf at half its validity, but
the running container kept presenting the copy its hive delivered at
start, so a re-issued leaf reached the agent only on its next restart.
hive-agent-bao-identity now seeds a live copy of the pair in its own
runtime directory from the delivered credentials, and re-runs every 10
minutes: it logs in with the live leaf, reads its own
swarm/agents/<agent>/bao-mtls (a read render_agent already grants), and
when the leaf changed, logs in with the new pair and swaps it in only if
the store accepts it.
Every unit in the container that logs in to the store presents the live
pair: hive-agent (queue-identity.nix), hive-matrix-daemon,
hive-subagent-daemon, hive-agent-forge-token, hive-agent-forge-accounts
and hive-agent-github-token. They keep loading the server CA as a
credential. The long-running daemons are ordered after the refresher so
the live pair exists when they start.
It renews only while the held leaf is still valid: the login that reads
the new leaf is made with the current one.
Refs #5017
`GET /api/agents/{name}/inference` answered `"configured": {}` both for
an agent with no key in `agent-inference` and for a stored value with no
fields, so a reader could not tell "the swarm sets nothing" from "a
setting with every field unset". It now reads the key as
`Option<AgentInference>`: an absent or deleted key is `null`, a stored
`{}` stays `{}`. The shared `schedules::read` helper is unchanged.
`swarmctl agent inference get` prints "no swarm setting; the agent runs
on its own model and effort" for `null`; `set` merges into an empty
value in that case, as before.
The agent itself still treats no key and `{}` the same.
Refs #5034
`check_model` accepted a value like `--dangerously-skip-permissions`,
and the stored model ends up in the agent's `claude` argv. A PUT whose
`model`, or any `subagent_models` entry (same validator), starts with
`-` is now a 400 with the same problem+json shape as the other model
checks. A dash inside a name (`claude-opus-4-8`) is still accepted.
Raised by argus on #5084.
Refs #5034
The header on `$SWARM.agent-state.<agent>` went out only when a bus event
changed it, and the core subject keeps nothing for a late subscriber. An
idle agent therefore said nothing to a reader that attached after its
last change: swarm-controller after a restart showed `effective: null`
for that agent until its next turn, which can be hours.
The publish loop now also wakes every 60 s and re-sends the current
header through the same `publish` path (clearing the dedup state first).
Transitions still go out as they happen. Same subject, same grant.
An agent whose queue connection is down now logs `swarm agent state:
publish skipped` once a minute even while idle, so the
`hh-swarm-agent-state-publish-skipped` alert also fires for idle
disconnected agents, not only busy ones.
Refs #5034
Each agent now reports the model and effort it runs on, and where each
comes from (`swarm`, `agent_nix`, `local_file` or `default`), in its
turn-state header on `$SWARM.agent-state.<agent>`: `effort`,
`model_source` and `effort_source` join `model`. The shared
`swarm_queue_client::agent_inference::{Effective, Source}` types spell
those fields for both ends.
The Bus keeps a source next to the model and effort. `Bus::new` takes it
from the local fallback chain (`harness_state::local_model` /
`local_effort`, which now return the source with the value), a `/model`
or `/effort` pick sets `local_file`, and the swarm poller sets `swarm`
or, on a clear, the local chain's source. A swarm value equal to the one
the agent already runs is now applied once, so the source moves to
`swarm` and the header is republished; before, `next()` skipped it as
unchanged.
swarm-controller subscribes to every agent's header with the reader
grant it already holds for the state relay, and keeps the latest
inference fields per agent in memory (`inference::Reported`).
`GET /api/agents/{name}/inference` now answers
`{configured, effective}`, `effective` being `null` until the agent's
first header with these fields since the controller started.
`swarmctl agent inference get` prints both; `set` reads `configured`.
Refs #5034
swarm-controller keeps each agent's inference settings in a new
`agent-inference` KV bucket, one key per agent, and is its only writer
(`GET`/`PUT`/`DELETE /api/agents/{name}/inference`; `swarmctl agent
inference get|set|clear`). Every field is optional. Each agent reads its
own key every 10 s through a direct-get grant on that key alone, and a
set `model` or `effort` replaces the Bus value, so the next turn runs on
it and the badge follows.
The swarm value is applied without persisting it: `Bus::set_model` and
`set_effort` are split into a persisting setter and a non-persisting
`apply_*`, and the poller uses the latter. Otherwise the swarm value
would overwrite `hyperhive-model`/`hyperhive-effort`, and clearing it, or
a restart, would no longer find the agent's own value. An unset field
changes nothing; a field cleared after being applied goes back to the
local value (`harness_state::local_model`/`local_effort`, the same
resolution `Bus::new` uses), read again at the clear.
`subagent_models` is validated, stored and shown by `get`; no agent
applies it yet.
Refs #5034
forgejo-api's client has no request timeout, so a forge that accepts a
connection and never answers held a forge-calling job node forever:
the node never reached Done or Failed, and kept its executor task.
Every executor arm that awaits the forge now runs its work under the
same deadline the periodic passes use. On elapsing, the node fails
through its existing Outcome::Failed path with
"<node>: no answer from the forge within 120s".
Nodes: create_repo, create_forge_user, add_repo_member,
grant_docs_read, init_agent_config_repo, mint_agent_forge_token,
mint_hive_forge_token.
The bound is per node, not per request: the nodes make 1 to about 15
sequential forge and store requests each (mint_hive_forge_token the
most), and the requests live inside forge::Client methods the periodic
passes share. One 2-minute bound covers both, so #5065's
within_pass_deadline / PASS_DEADLINE become within_forge_deadline /
FORGE_DEADLINE, with a log line that no longer says "retrying next
tick" (true of a pass, not of a node).
The InitAgentConfigRepo arm is lifted out of run_swarm_node, like its
siblings, to keep that function under clippy::too_many_lines.
Refs #5066
forgejo-api 0.11.1 builds its reqwest client with no timeout and offers
no way to hand it a caller-built one (the `client` field is private,
`new`/`with_user_agent` are the only constructors, `Endpoint` is
sealed). A forge that accepts a connection and then stalls therefore
held each of the six periodic reconcile loops on one request forever,
with no log line.
Wrap each loop's per-tick work in `forge::within_pass_deadline`
(`PASS_DEADLINE`, 2 min, under the 5-minute interval): on elapsed the
pass is dropped, a warn is logged, and the loop waits for its next tick.
Every pass is an idempotent reconcile, so abandoning one mid-way leaves
nothing the next pass does not redo.
Loops: config-pr poll, runner registration token, hive forge tokens,
agent forge tokens (observe + legacy sweep), docs grants, swarm forge
objects.
Refs #5060
Each of these wrote its file in place (or set the mode after the
write), so a write cut short by a kill or a full disk left a partial
file behind, and some secrets existed for a moment at umask mode:
- swarm-authelia/swarm-grafana/hive-forge/swarm-bao generators: a
partial file passed the `-s`/`-e` guard and was kept forever. They now
capture the value first and land it with `atomic_write_secret`.
- swarm-controller webhook secret: a truncated file failed the 64-hex
check and the secret was regenerated, out of step with the forge
hooks. It is now written 0600 to a sibling temp, fsynced, renamed.
- hive-c0re htpasswd: the temp is now fsynced and set 0644 before the
rename, like the sibling agents.conf writer, instead of relying on
c0re's umask.
- hivectl wg init: the private key is created 0400 in a temp and
renamed, instead of written at umask mode and chmod'ed after.
Closes#5068
- swarm-controller webhook: regenerate the secret only when the file is
absent. Any other read error is returned with the path; the caller
already logs "webhook secret unavailable" and the endpoint answers 503.
Overwriting meant every registered hook kept the old secret and every
delivery failed HMAC, with nothing re-registering them.
- swarm-queue-client wanted: open_read_only/watch return Result; only
STREAM_NOT_FOUND on the bucket maps to None. hive-c0re's wanted pull
fails the DAG node with the cause instead of logging "nothing has been
declared"; the watch warns with the error chain.
- swarm-nats-auth: warn when the OIDC secret re-read task dies.
- swarm-controller forge: listing a repo's config PRs failing is warn,
not debug.
Closes#5056
With actionsFromSelf, Forgejo resolves bare `uses: actions/<name>` against
the forge itself, so every action a workflow names needs a mirror there.
Only actions/checkout was seeded, and coverage.yml's
`actions/upload-artifact@v3` has failed at job setup with "repository not
found" since the CI runner moved to swarm-ci-runner (e2da5509).
The single hardcoded checkout mirror becomes a list of action names both
share, mirrored from github.com/actions/* like checkout already was. v3
points at the same commit (ff15f03) on github.com and code.forgejo.org.
The controller's forge-objects pass creates any missing mirror on every
pass, and the changed SWARM_CONTROLLER_FORGE_MIRRORS env restarts the unit
on switch, so an existing forge picks the mirror up on deploy.
Closes#5040
No behaviour change. Casts replaced by try_from / Duration math; schedules.rs
reuses agent_renewal::unix_now; the five swarm-queue-client KV open-or-create
copies share one private kv::open_or_create.
The publisher was wanted only by container@hive-forge.service. A switch
starts a new unit only through the targets that want it, and the container
is already running, so the deploy that introduced the publisher left it dead
until the next container restart or boot; the path unit cannot help, because
the mint's only-once guard never rewrites the file.
- The publisher is also wanted by multi-user.target, like swarm-secret-publish.
- A token file that is not there yet exits 0 instead of waiting 120s and
failing: the path unit runs the publisher when the mint writes it, so a
switch with the forge container down does not fail. TimeoutStartSec is 60.
- swarm-controller loses its After= on the publisher: a switch issues the two
jobs separately, so it ordered nothing there, and the background re-read
covers the gap.
- The controller's background re-read logs at info once the backoff is
capped, one line per five minutes, so a forge host that never got its
publisher leaf stays visible; earlier retries stay at debug.
Refs #4998
A controller that started before the forge's host had published its token
ran without forge access until someone restarted it: on a split-host first
deploy that is a manual step. Now the forge client lives in a slot that is
filled once (`Late<T>`, an `Arc<OnceLock<T>>`):
- `forge::enable` tries the store once at start. On failure it logs once and
re-reads in the background with backoff (30s doubling to a 5-minute cap,
no log per retry), fills the slot and logs once when the token is there.
- The forge passes (#5014's per-hive token mint among them), the config-PR
poll and the webhook registration start from that moment, through
`start_forge`'s callback.
- Job nodes and HTTP handlers read the slot when they run, so they answer
"no forge" before and work after.
A token replaced in the store after the slot is filled still reaches the
controller with its next restart. The round-1 change that shared
`queue_identity::fetch_with_retry` is reverted; nothing else uses it.
Refs #4998
The controller's forge token never entered bao: the forge minted it in its
container, hive-forge-swarm-controller-token copied it to a host file, and the
controller LoadCredential'ed that file. forgeTokenFile defaulted to that path
only where the forge ran, so a controller on any other host had no forge
access unless an operator copied the file by hand.
Now the forge's host publishes it and the controller reads it:
- swarm-bao-forge-publisher (replaces hive-forge-swarm-controller-token)
waits for the in-container token, logs in with its own forge-publisher leaf
and writes it to swarm/controller/swarm-controller/forge/token. It runs on
every forge container start, the old unit's trigger.
- New principal swarm-forge-publisher: leaf minted by glue-bao-tls.nix,
policy create/update on that one path, written by
swarm-bao-forge-publisher-policy, CN reserved as a hive name.
- The controller policy gains read on that path. swarm-controller reads it
at start under its own certificate with the OIDC secret's retry, and runs
without forge access if it still cannot.
- forgeTokenFile, its LoadCredential and the forgeEnv gate are gone;
SWARM_CONTROLLER_FORGE_URL is always set. hostSwarmControllerTokenFile is
removed, and a tmpfiles rule deletes the old host copy on the forge host.
The in-container file stays: the mint unit's only-once guard reads it, and
it is the publisher's source.
Refs #4998
Agent creation gains a GrantDocsRead node (after CreateForgeUser) that adds
the agent as a read collaborator on internal/docs, and forge::docs_grant
runs a pass over the store's hive-agent-* roster at start and every five
minutes that queues the same job for any agent that is not one. The pass only
adds; an unreadable roster or forge read changes nothing.
hive-c0re drops shared_docs_access, its call in sync_agent, and the
SHARED_ORG / SHARED_DOCS_REPO constants only it used.
`hive-<hive>` is each hive's read-only forge account, and an agent's forge
account takes the agent's name, so an agent called `hive-<x>` would claim
hive `<x>`'s account. `hive_types::reserved_agent_prefix_violation` names
the rule, and `broken_name_rules` applies it, so both creation paths that
go through it (`POST /api/agents` and placing an undeclared agent on a
hive) refuse such a name for a new agent. An agent already in the roster
keeps its name with a warning, as for every other naming rule.
The email check between hive and agent accounts stays as the second
guard, for agents created before this rule.
Refs #4851
