wait_until_ready only checked READY_TIMEOUT between polls, so a half-open forge
hung the first get_version forever. The probe now goes through forge::bounded; a
timeout counts as not-ready and falls into the existing retry/deadline path.
forgejo-api 0.11.1 takes no caller-built client, so a request to a forge that
accepts the connection and never answers waited forever. Every forge::api()
call site now goes through forge::bounded, a 10s timeout that errors naming the
operation. The PR-comment calls run inline in swarm_status::drain_swarm_events,
the one task serving both swarm subscriptions, so a stalled forge stopped both.
Refs #5066
Bounding is_running turned a hung systemctl into "not running" for
callers that record or act on the answer: the broad-stop restore set
dropped the agent, start_many could rebuild-and-restart a running
agent, reconcile could seed Offline. is_running is back to the
unbounded call it was.
The periodic workers read through the bounded running_state instead:
crash_watch (unchanged), and the swarm_agent_status publisher via the
new container_view::read_agent_status_bounded, which skips the agent
for that sweep when the unit state is unreadable. The GetAgentMeta
socket handler keeps the unbounded read_agent_status_live.
priv_client::list_containers now waits at most 30s (LIST_WAIT) for
hive-priv's `nixos-container list`, via the existing `within` helper. Its
periodic callers (crash_watch, agent_sockets, swarm_agent_status, wanted,
mcp_sockets, swarm_status via container_view) already treat an Err as a
skipped cycle, so a hung listing no longer stops their loops for good.
lifecycle::is_running / unit_state run `systemctl is-active` with
kill_on_drop and a 10s timeout (IS_ACTIVE_TIMEOUT); a timeout warns and
reads as not-running / UnitState::Other, the same as a systemctl that
could not be spawned. The new lifecycle::running_state returns None for
that case, and crash_watch keeps the agent's previous observation on None
so a hung systemctl is neither reported as a crash nor as a recovery.
`call` itself and every mutating op (start/stop/destroy/send-snapshot/
daemon-reload) stay unbounded.
Closes#5061
The future `Scheduler::claim_next` hands back completed its node only
after `run`'s future returned. A panic while running (the workspace
unwinds) skipped that completion, so the node stayed `Running` and its
resource units stayed held until the process restarted; later nodes
needing the same resource never started.
The run future is now polled under `catch_unwind`. A panic completes
the node as `Outcome::Failed("runner panicked: <msg>")` with nothing
grown, through the same `complete_growing` path as an ordinary failure:
`AFTER_OK` dependents are skipped, pending children cascade, and the
node's units are released.
The node's future is boxed: wrapping it inline pushed the futures
swarm-controller and hive-c0re spawn over clippy's `large_futures`
limit.
Closes#5058
vaultrs sets no request timeout by default, so a store that accepts a connection and never answers hung every SecretStore call forever. connect_with_identity now sets VAULT_REQUEST_TIMEOUT; a stalled store surfaces as Error::Vault.
- swarm-controller webhook: regenerate the secret only when the file is
absent. Any other read error is returned with the path; the caller
already logs "webhook secret unavailable" and the endpoint answers 503.
Overwriting meant every registered hook kept the old secret and every
delivery failed HMAC, with nothing re-registering them.
- swarm-queue-client wanted: open_read_only/watch return Result; only
STREAM_NOT_FOUND on the bucket maps to None. hive-c0re's wanted pull
fails the DAG node with the cause instead of logging "nothing has been
declared"; the watch warns with the error chain.
- swarm-nats-auth: warn when the OIDC secret re-read task dies.
- swarm-controller forge: listing a repo's config PRs failing is warn,
not debug.
Closes#5056
The forge clamps page_size to 50. branches, pr reviews, the review
bodies merged into `comments <pr>`, and view's comment list each sent
one request, so anything past the first page was dropped silently.
labels::repo_labels did loop, but asked for 100/page and stopped on
`len < 100`, so a clamped full page of 50 ended the walk.
Add verbs::all_pages: requests 50/page until a short page, capped at
50 pages with a stderr warning when the cap is hit. Use it in all of
the above; repo-labels now calls labels::repo_labels instead of its
own copy of the loop.
list --json now prints the same "N of M shown" stderr trailer as the
pretty output; its paging and stdout shape are unchanged.
Closes#5051
HIVE_FORGE_URL (set + parseable) and HIVE_ASSETS_DIR are checked once in
main before workers start. Short revs in container_view and pr_status
take 12 chars instead of byte-slicing.
With actionsFromSelf, Forgejo resolves bare `uses: actions/<name>` against
the forge itself, so every action a workflow names needs a mirror there.
Only actions/checkout was seeded, and coverage.yml's
`actions/upload-artifact@v3` has failed at job setup with "repository not
found" since the CI runner moved to swarm-ci-runner (e2da5509).
The single hardcoded checkout mirror becomes a list of action names both
share, mirrored from github.com/actions/* like checkout already was. v3
points at the same commit (ff15f03) on github.com and code.forgejo.org.
The controller's forge-objects pass creates any missing mirror on every
pass, and the changed SWARM_CONTROLLER_FORGE_MIRRORS env restarts the unit
on switch, so an existing forge picks the mirror up on deploy.
Closes#5040
No behaviour change. Casts replaced by try_from / Duration math; schedules.rs
reuses agent_renewal::unix_now; the five swarm-queue-client KV open-or-create
copies share one private kv::open_or_create.
Add secret_file::write_secret_0600: create the parent dir, write a
sibling temp file opened create_new + mode 0600, sync_all, rename over
the target. Use it for the matrix sender token, the space/chat room-id
files and the forge core token, which previously wrote at the process
umask, then chmod-ed with the error discarded, and dropped the
create_dir_all error. agent_identity::write_identity uses it too
instead of its own open-0600 loop; that loop opened with create +
truncate, so a pre-existing file kept its old mode.
Also log instead of discarding:
- the manager's applied-rev marker write in auto_update
- the knowledge `git remote set-url` scrub (spawn error and non-zero
exit; the exit status only, since stderr may echo the old URL)
- the parent create_dir_all in paths::move_if_legacy
Closes#5037
The six hand-written granter policy units (secret-publisher, matrix-ctl,
services-issuer, nats-tls, hive-leaf, operator-viewer) repeated the same
skeleton readerPolicyUnit already carried: ordering after the controller's
unit, pki requirement, path, granter environment, unseal retry window and
oneshot serviceConfig. They differed only in description and the script
after the granter login.
granterPolicyUnit now owns that skeleton; the six units and readerPolicyUnit
pass a description and a script body. Every rendered unit is unchanged
byte-for-byte.
The publisher was wanted only by container@hive-forge.service. A switch
starts a new unit only through the targets that want it, and the container
is already running, so the deploy that introduced the publisher left it dead
until the next container restart or boot; the path unit cannot help, because
the mint's only-once guard never rewrites the file.
- The publisher is also wanted by multi-user.target, like swarm-secret-publish.
- A token file that is not there yet exits 0 instead of waiting 120s and
failing: the path unit runs the publisher when the mint writes it, so a
switch with the forge container down does not fail. TimeoutStartSec is 60.
- swarm-controller loses its After= on the publisher: a switch issues the two
jobs separately, so it ordered nothing there, and the background re-read
covers the gap.
- The controller's background re-read logs at info once the backoff is
capped, one line per five minutes, so a forge host that never got its
publisher leaf stays visible; earlier retries stay at debug.
Refs #4998
The publisher was a oneshot with RemainAfterExit=true, started only through
`wantedBy = container@hive-forge.service`. Once it had run it stayed
`active (exited)`, so a later container start made its start a no-op: a token
re-minted without a host reboot or a unit change never reached the store.
- RemainAfterExit is dropped, so each start runs the script again.
- systemd.paths.swarm-bao-forge-publisher watches the token file in the
container (PathChanged) and starts the publisher when the mint writes it,
the same shape as swarm-secret-publish's path unit.
- The code comment, secrets.md and credentials.md now name both triggers;
forge-placement pins the missing RemainAfterExit and the path unit.
Refs #4998
A controller that started before the forge's host had published its token
ran without forge access until someone restarted it: on a split-host first
deploy that is a manual step. Now the forge client lives in a slot that is
filled once (`Late<T>`, an `Arc<OnceLock<T>>`):
- `forge::enable` tries the store once at start. On failure it logs once and
re-reads in the background with backoff (30s doubling to a 5-minute cap,
no log per retry), fills the slot and logs once when the token is there.
- The forge passes (#5014's per-hive token mint among them), the config-PR
poll and the webhook registration start from that moment, through
`start_forge`'s callback.
- Job nodes and HTTP handlers read the slot when they run, so they answer
"no forge" before and work after.
A token replaced in the store after the slot is filled still reaches the
controller with its next restart. The round-1 change that shared
`queue_identity::fetch_with_retry` is reverted; nothing else uses it.
Refs #4998
The controller's forge token never entered bao: the forge minted it in its
container, hive-forge-swarm-controller-token copied it to a host file, and the
controller LoadCredential'ed that file. forgeTokenFile defaulted to that path
only where the forge ran, so a controller on any other host had no forge
access unless an operator copied the file by hand.
Now the forge's host publishes it and the controller reads it:
- swarm-bao-forge-publisher (replaces hive-forge-swarm-controller-token)
waits for the in-container token, logs in with its own forge-publisher leaf
and writes it to swarm/controller/swarm-controller/forge/token. It runs on
every forge container start, the old unit's trigger.
- New principal swarm-forge-publisher: leaf minted by glue-bao-tls.nix,
policy create/update on that one path, written by
swarm-bao-forge-publisher-policy, CN reserved as a hive name.
- The controller policy gains read on that path. swarm-controller reads it
at start under its own certificate with the OIDC secret's retry, and runs
without forge access if it still cannot.
- forgeTokenFile, its LoadCredential and the forgeEnv gate are gone;
SWARM_CONTROLLER_FORGE_URL is always set. hostSwarmControllerTokenFile is
removed, and a tmpfiles rule deletes the old host copy on the forge host.
The in-container file stays: the mint unit's only-once guard reads it, and
it is the publisher's source.
Refs #4998
Agent creation gains a GrantDocsRead node (after CreateForgeUser) that adds
the agent as a read collaborator on internal/docs, and forge::docs_grant
runs a pass over the store's hive-agent-* roster at start and every five
minutes that queues the same job for any agent that is not one. The pass only
adds; an unreadable roster or forge read changes nothing.
hive-c0re drops shared_docs_access, its call in sync_agent, and the
SHARED_ORG / SHARED_DOCS_REPO constants only it used.
`hive-<hive>` is each hive's read-only forge account, and an agent's forge
account takes the agent's name, so an agent called `hive-<x>` would claim
hive `<x>`'s account. `hive_types::reserved_agent_prefix_violation` names
the rule, and `broken_name_rules` applies it, so both creation paths that
go through it (`POST /api/agents` and placing an undeclared agent on a
hive) refuse such a name for a new agent. An agent already in the roster
keeps its name with a warning, as for every other naming rule.
The email check between hive and agent accounts stays as the second
guard, for agents created before this rule.
Refs #4851
swarm-controller now gives every hive in its directory a forge account,
`hive-<hive>`, in a new `hives` team of `agent-configs` (read on
`repo.code` + `repo.pulls`, every repo including later ones), and mints
one `read:repository` token named `swarm-hive` into the store at
`swarm/hives/<hive>/forge/token`. A five-minute pass queues a
`MintHiveForgeToken` node for a hive whose token is missing or no longer
matches the forge, whose account is missing, or whose team membership or
team settings drifted. A hive whose store or forge read failed is never
queued, so an outage cannot rotate every hive's token. The token decision
reuses `agent_token`'s classifier and replace step, now parameterised by
name and scopes.
An agent may be called `hive-<x>`. The hive account is created with
`hive-<hive>@hives.hyperhive.local` and the job refuses an account by that
name with any other email before it locks it down, adds it to the team or
mints for it; agent creation already refuses the hive's account by the
same email check.
The controller's bao policy gains the `swarm/hives/+/forge/token` leaf
(create/read/update), pinned by the bao-grants module-eval assertion. The
hive's own policy already reads `swarm/hives/<hive>/*`.
hive-c0re reads that token through `forge::reader_token()`, falling back
to the core token file when the store holds none, and uses it for:
fetching a merged config rev (swarm deploy, `hivectl forge
reconcile-config`), reseeding a purged `applied`, the meta-update
cascade's fast-forward, seeding `proposed` on first deploy, and finding
the PR that merged a refused rev. Those reads no longer require a local
`hive-forge` container. The `/knowledge` clone moves from the forge
host's boot sweep into the knowledge pull, so every hive clones it.
Posting the deploy-failure comment on that PR stays on the core token.
Refs #4851
