The controller read its forge token from the swarm secret store once, when
its client slot filled, and kept sending it until the daemon restarted. A
token the forge re-minted or revoked left every forge-backed path failing
with 401 until someone restarted the controller, and a restart drops the
runtime-only job graph.
Every forge request now goes through `forge::api::Api::call`. On a 401 it
asks a `StoredSecret` (the queue clients' debounced, single-flight re-read
with backoff) for a token to try instead, rebuilds the forgejo client with
it, and sends the request once more. A second 401, or a store still holding
the refused token, surfaces the error as before. Only 401 counts: unlike an
IdP, the forge answers 400 to a malformed request, and a re-read started by
one would put a real 401 that follows inside the backoff.
forgejo-api bakes the token into its own reqwest client's default headers
and takes no client or middleware from outside, so there is no single
request function to hook. Each `self.api.<endpoint>(..)` site becomes
`self.api.call(|api| api.<endpoint>(..).into_future())`; option structs a
site moved into the call are cloned, since the closure can run twice. The
`OnceLock<Arc<Client>>` slot and its `get()` callers are unchanged.
Refs #5017
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).
When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.
The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.
Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.
This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
forgejo-api's client has no request timeout, so a forge that accepts a
connection and never answers held a forge-calling job node forever:
the node never reached Done or Failed, and kept its executor task.
Every executor arm that awaits the forge now runs its work under the
same deadline the periodic passes use. On elapsing, the node fails
through its existing Outcome::Failed path with
"<node>: no answer from the forge within 120s".
Nodes: create_repo, create_forge_user, add_repo_member,
grant_docs_read, init_agent_config_repo, mint_agent_forge_token,
mint_hive_forge_token.
The bound is per node, not per request: the nodes make 1 to about 15
sequential forge and store requests each (mint_hive_forge_token the
most), and the requests live inside forge::Client methods the periodic
passes share. One 2-minute bound covers both, so #5065's
within_pass_deadline / PASS_DEADLINE become within_forge_deadline /
FORGE_DEADLINE, with a log line that no longer says "retrying next
tick" (true of a pass, not of a node).
The InitAgentConfigRepo arm is lifted out of run_swarm_node, like its
siblings, to keep that function under clippy::too_many_lines.
Refs #5066
forgejo-api 0.11.1 builds its reqwest client with no timeout and offers
no way to hand it a caller-built one (the `client` field is private,
`new`/`with_user_agent` are the only constructors, `Endpoint` is
sealed). A forge that accepts a connection and then stalls therefore
held each of the six periodic reconcile loops on one request forever,
with no log line.
Wrap each loop's per-tick work in `forge::within_pass_deadline`
(`PASS_DEADLINE`, 2 min, under the 5-minute interval): on elapsed the
pass is dropped, a warn is logged, and the loop waits for its next tick.
Every pass is an idempotent reconcile, so abandoning one mid-way leaves
nothing the next pass does not redo.
Loops: config-pr poll, runner registration token, hive forge tokens,
agent forge tokens (observe + legacy sweep), docs grants, swarm forge
objects.
Refs #5060
`hive-<hive>` is each hive's read-only forge account, and an agent's forge
account takes the agent's name, so an agent called `hive-<x>` would claim
hive `<x>`'s account. `hive_types::reserved_agent_prefix_violation` names
the rule, and `broken_name_rules` applies it, so both creation paths that
go through it (`POST /api/agents` and placing an undeclared agent on a
hive) refuse such a name for a new agent. An agent already in the roster
keeps its name with a warning, as for every other naming rule.
The email check between hive and agent accounts stays as the second
guard, for agents created before this rule.
Refs #4851
swarm-controller now gives every hive in its directory a forge account,
`hive-<hive>`, in a new `hives` team of `agent-configs` (read on
`repo.code` + `repo.pulls`, every repo including later ones), and mints
one `read:repository` token named `swarm-hive` into the store at
`swarm/hives/<hive>/forge/token`. A five-minute pass queues a
`MintHiveForgeToken` node for a hive whose token is missing or no longer
matches the forge, whose account is missing, or whose team membership or
team settings drifted. A hive whose store or forge read failed is never
queued, so an outage cannot rotate every hive's token. The token decision
reuses `agent_token`'s classifier and replace step, now parameterised by
name and scopes.
An agent may be called `hive-<x>`. The hive account is created with
`hive-<hive>@hives.hyperhive.local` and the job refuses an account by that
name with any other email before it locks it down, adds it to the team or
mints for it; agent creation already refuses the hive's account by the
same email check.
The controller's bao policy gains the `swarm/hives/+/forge/token` leaf
(create/read/update), pinned by the bao-grants module-eval assertion. The
hive's own policy already reads `swarm/hives/<hive>/*`.
hive-c0re reads that token through `forge::reader_token()`, falling back
to the core token file when the store holds none, and uses it for:
fetching a merged config rev (swarm deploy, `hivectl forge
reconcile-config`), reseeding a purged `applied`, the meta-update
cascade's fast-forward, seeding `proposed` on first deploy, and finding
the PR that merged a refused rev. Those reads no longer require a local
`hive-forge` container. The `/knowledge` clone moves from the forge
host's boot sweep into the knowledge pull, so every hive clones it.
Posting the deploy-failure comment on that PR stays on the core token.
Refs #4851
