hive-c0re waited on hive-priv's answer with no timeout, so a hung
hive-priv would stall the disk sampler indefinitely. The wait is now
hive-priv's own 10 min bound plus a minute (the bound is shared as
MEASURE_AGENT_DISK_TIMEOUT); a timeout is an error, so the agent shows
unknown.
An agent with none of state/, work/ or the rootfs has no measurable
size. hive-priv now answers with an error for it instead of a total of
0, which read as a real measurement.
The disk sampler ran `du -sxb` as the unprivileged hive-core user over
each agent's state/, work/ and container rootfs. hive-core cannot read
the rootfs or all of the agent-owned files, so `du` failed and each
failed path silently counted as 0: the dashboard showed a few MB for
agents using over 13 GB.
Add a hive-priv `MeasureAgentDisk { agent_name }` op. hive-priv
validates the name, derives the three paths itself (the request carries
no path), skips the ones that don't exist, and runs one
`du -s -x -b -c -P` as root with a 10 min timeout. A timeout, a non-zero
exit or output without a `total` line is an error.
hive-c0re calls it every 30 min instead of running `du` every 5 min:
each pass walks every agent's full tree as root on the disk the agents
work on, and the sizes move slowly. A failed measurement is cached as
None, so `disk_bytes` is null, the OTel storage.usage sample is omitted
and the dashboard shows "unknown", never 0.
`Coordinator::agent_work_dir` had no other caller and is removed.
Closes#4956
The CI runner was a hive deploy module: hive-ci.nix ran it beside the
forge, and hive-c0re registered it through the forge's admin API with
the `core` token, writing the registration token through hive-priv. It
could only run on the forge's host, and it kept an admin-scoped forge
token on a hive.
Now the runner is a swarm service:
- swarm-ci-runner.nix runs it in a `swarm-ci-runner` container on the
host that sets deploy.ci-runner.enable, reaching the forge by name.
It needs neither the forge nor a hive there.
- swarm-controller reads the forge's runner registration token with its
own admin token and writes it to swarm/services/ci-runner/
registration-token whenever it differs from what it last wrote. Its
grant on that path is create/update only.
- The runner's own leaf (CN swarm-ci-runner, role and policy of the same
name) reads that one path. swarm-bao-ci-runner.service fetches it into
the runner's EnvironmentFile and restarts the runner on a change.
- A pre-start step removes .runner when the daemon's journal since its
last start holds Forgejo's "unregistered runner" refusal (the daemon
exits 1 when its startup Declare gets it), or when .runner names
another forge address; upstream's pre-start step then registers again.
Removed from the hive side: hive-ci.nix and deploy.forgejo.ci.*,
forge/ci_runner.rs, PrivRequest::RegisterCiRunner and its handler and
ReadWritePaths entry, InfraContainer::Ci with hivectl's --ci and the
dashboard's hive-ci journal entry, HYPERHIVE_FORGE_CI_ENABLED, and
read:admin,write:admin from CORE_TOKEN_SCOPES: nothing else in hive-c0re
calls /api/v1/admin/*. The old ci options, under both swarm.forge.ci and
deploy.forgejo.ci, now fail evaluation naming deploy.ci-runner.
The forge's half of the old ci.enable, the actions/checkout mirror and
DEFAULT_ACTIONS_URL = self, is deploy.forgejo.actionsFromSelf.
Closes#4954
Agents keep git worktrees and cargo target dirs under /home/<agent>,
which lives in the container rootfs and is deleted on destroy. They
can't move them under state/, because a claude started there walks up
into state/CLAUDE.md and takes on the agent's identity.
Add a per-agent work/ dir next to state/ on the host
(/var/lib/hyperhive/agents/<n>/work), bind-mounted RW at
/agents/<n>/work. hive-c0re creates it in ensure_state_dir and in
set_nspawn_flags (the rebuild path reaches only the latter), the
agent's activation chowns it to the agent user like state/ and
harness/, and measure_agent_disk counts it. destroy never touches the
agent root, so work/ survives it; purge removes the whole root, so
work/ goes with it.
Closes#4977
Every hive's homeserver loaded a per-hive `hyperhive` appservice
registration, and the swarm minted and delivered an as_token for it
through bao. No client presented that token: swarm-controller creates
every `@hive-<hive>` sender and every agent account with the swarm
appservice token (hive_sender.rs, agent_token.rs), and no Rust code read
the per-hive one.
Removed, with everything that existed only to serve it:
- hive-matrix.nix: the `hyperhive` registration, its activation-time
render, the host bind mount of /var/lib/hyperhive/matrix-appservice,
its LoadCredential entry, and the internal options
`deploy.matrix.appserviceTokenFile` / `appserviceRegistrationScript`.
tuwunel now loads only the swarm registration.
- glue-matrix-bao-token.nix and its `swarm-bao-matrix-token` unit.
- glue-bao-tls.nix: the matrix-token leaf and its defaults.
- swarm-bao.nix: the `swarm-matrix-token-<hive>` roles and policies
(`swarm-bao-matrix-token-policy`), and the options
`deploy.bao.matrixTokenClient{Cert,Key}File` and
`matrixTokenCommonNamePrefix`, now mkRemovedOptionModule.
- swarm-secret-publisher.nix: the per-hive token mint and its state dir.
- swarm.nix: the reserved `<matrixTokenCommonNamePrefix>-<hive>` CN.
- swarm-secret-client: `matrix::appservice_token_path`.
The swarm registration (swarm-matrix-ctl, swarm-controller) is untouched.
Comments and docs that pointed at the removed glue file now point at
glue-queue-agent-credential.nix, which has the same reader shape.
Refs #4428
Nothing submits an approval since scheduled prompts left hive-c0re, so the
queue only acted on rows written before that.
- hive-c0re: the approvals store and its migrations, approve/deny/
resolve actions, the ResolveApproval tail node (and meta_update's
approval_id), POST /api/approve and /api/deny, the approval views on
/api/state, the approval_resolved dashboard event, the orphan GC,
destroy/purge failing pending rows, and the approval rows in
get_loose_ends. DestroyBookkeeping loses the purge flag, which only
worded the approval-failure note.
- The CancelLooseEnd request goes: its only hive-c0re arm was the
approval one. cancel_loose_end handles reminder and todo in the harness.
- Wire types: hive_sh4re::approvals (ReminderStats moves to inbox),
LooseEnd::Approval, CancelLooseEndKind, HelperEvent::ApprovalResolved,
ToolGroup::Approvals, HostRequest::{Pending,Approve,Deny} and
HostResponse.approvals. HOST_SOCK_VERSION goes to 2 for the removed
field.
- hivectl loses the approvals verbs; docs/tools/hivectl-cli.md is
regenerated.
- The system prompt drops approval_resolved.
- The approvals table stays in broker.sqlite, unread, the same way the
operator_questions table was left when its store was removed.
- An `approvals` entry still in tool-groups.json is skipped with a warning
by ToolGroup::parse_list, like any unknown group.
- docs/agent-lifecycle/approvals.md is renamed to config-changes.md so the
code's doc pointers resolve; its content follows in the docs commit.
The hive policy rendered by policy::render granted every hive cert read
on secret/data/swarm/services/*, which holds each swarm service's OIDC
client secret. No reader of that tree authenticates with a hive cert:
grafana, the otel collector and the store forwarder each read with
their own leaf, and the bao UI's secret is read by the store's granter.
render() now emits two stanzas: swarm/agents/* and the hive's own
swarm/hives/<hive>/*. Kind::Service had no other caller and is removed.
Docs and comments that described the grant as a decision are rewritten
to the current state.
Closes#4960
An agent's harness connects to the swarm queue with the agent's own
credential or not at all. The first connect retries after 5s, doubling
(7 attempts, about five minutes), with a warning per failed attempt;
when every attempt fails the harness logs an error and exits 1. Missing
coordinates (HIVE_AGENT_NATS_URL, or the secret store's BAO_ADDR /
HIVE_AGENT_NAME / certificate) fail the boot outright.
Previously the first connect's outcome was memoized for the life of the
process: a store 403 or queue refusal at boot left the agent on the
hive's shared client, or disconnected, until someone restarted it, and
an agent with no queue coordinates ran on without one. Both paths are
gone, with the hive-scoped subjects that only the shared client used.
The harness unit now restarts after 30s instead of 2s, so a swarm-wide
outage doesn't restart every agent every 2s, and has no start limit, so
it keeps coming back however long the outage lasts.
The hive's shared client credential is still imported into the harness
unit for swarm-logs; the two HIVE_AGENT_OIDC_*_FILE variables that
pointed the harness at it are dropped, as is the unset
HIVE_AGENT_OIDC_CA_FILE override.
With the fallback gone, the pieces #4944 added only for it go too:
swarm_input's skip-on-shared-client branch and swarm-queue-client's
connect_with_inbox_prefix. swarm-nats-auth's grant for the hive's shared
agent client is unchanged.
Closes#4909
Scheduled prompts are swarm-level now (swarmctl schedule, #4936). The hive
keeps the rows it has only so the operator can copy them over.
- hive-c0re no longer fires anything: the scheduled-prompts worker is gone,
and with it every create/edit/cancel/pause/resume/fire path (dashboard
endpoints, agent-socket verbs, the SchedulePrompt approval kind and its
approve handler). The store keeps open + list; GET /api/schedules stays.
- A v6 approvals migration drops schedule_prompt rows.
- Nothing submits approvals any more, so the submit path goes too:
Approvals::submit_kind (now a test fixture), the ApprovalAdded dashboard
event and its emitter.
- Coordinator::push_todo, whose only caller was the worker, goes, and with
it hive_host_sock::agent_todo_socket and hive-c0re's hive-agent-sock /
hive-sock-client deps.
- The schedules_changed dashboard event goes; nothing mutates the rows.
- Agents lose request_schedule_prompt, cancel_schedule, fire_schedule_now,
edit_schedule and list_schedules, the `scheduling` tool group, their
system-prompt line and their transcript formatters. `remind` is unchanged.
sync_agent no longer runs ensure_user_email, ensure_repo_creation_disabled
or ensure_config_repo. swarm-controller does all three before any deploy:
CreateForgeUser sets the account email and max_repo_creation = 0 (and the
agent_token backfill re-runs CreateForgeUser), CreateRepo creates and
protects agent-configs/<agent>, AddRepoMember grants the agent write on it,
and InitAgentConfigRepo seeds it.
What goes with them: the email-aligned and repo-creation-disabled marker
paths and the legacy email-marker relocate, sparse_edit_user_option,
is_forbidden and its tests, and ensure_org_repo. A config repo deleted by
hand is no longer recreated on the next spawn; the operator recreates the
agent at swarm level. Comments in swarm-controller that described the
removed hive passes now state only the current behaviour.
Closes#4848
hive-c0re logs in with the hive's matrix sender token and creates the
Space and rooms over swarm.matrix.apiUrl. On a hive that does not run
the homeserver that crosses the network, so the default is now
https://<gatewayHost>. hive-c0re already verifies against the hive trust
bundle (SSL_CERT_FILE: system CAs, swarm root and the store's services
root), which covers the gateway's leaf for that name.
Also drops the remaining loopback-default wording in hive-c0re's matrix
module and the openFirewall description, and pins the https default on a
hive host without the homeserver in module-eval-matrix-core.
- swarm.serviceDomains lists every swarm service name on every host. The
store's issuing role is built from it, so the store now admits a leaf
for grafana, metrics, otel, logs, the bao UI and the apex whichever host
runs the store. Each gateway's own leaf stays narrowed by
localServiceDomains.
- swarm.matrix.apiUrl defaults to http://<swarm.matrix.gatewayHost>
(null when gatewayHost is null) instead of loopback on the matrix host.
- The authelia vhost proxies /bridge/ to swarm-authelia-bridge with the
prefix stripped, and deploy.swarm-controller.authBridgeUrl defaults to
${swarm.authelia.url}/bridge on every host. swarm.authelia.bridgeUrl is
removed.
- swarm-authelia-bridge admits only swarm-controller's OIDC client on
every request (403 for any other active token), configured by
SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID (renamed from
..._OPERATOR_READER_CLIENT_ID, which gated ListOperators alone).
- swarm.peerHives is removed; swarm-wireguard.nix filters this host's own
hive out of swarm.hives itself.
Closes#4933
The hook a hive registered on the agent-configs org points at its
/webhook/config-pr route, which no hive serves any more, so every
config-repo event fails delivery to it. The forge sweep now deletes it.
Only a hook whose URL equals this hive's own exactly is removed. The same
path on another base belongs to another hive and is left alone. A forge
error is logged and boot continues; once the hook is gone the step is a
no-op.
Closes#4850
Folds in #4853: hive-c0re no longer writes branch protection on
`agent-configs` repos (apply_config_repo_branch_protection,
config_repo_protection_edit, record_branch_protection_result and the
now-unused main_branch_protection_option go). swarm-controller's
CreateRepo rule and its forge-objects convergence own `main`'s gate.
Nothing merges into config `main` as `core` any more, so the converged
rule's merge user list is empty. `push_config` still pushes `main` as
core, through push rights, not the merge whitelist.
Refs #4850
Refs #4853
An operator's merge on the forge deploys a config PR through
swarm-controller's DeployRequest{rev}. The hive-side path that queued a
MergeConfigPr approval and merged the PR as `core` goes:
- the `/webhook/config-pr` receiver, its HMAC secret, the WebhookRegister
boot node and the org-hook registration; the hive vhost's `/webhook/`
location
- the 5-minute config-PR poll
- ApprovalKind::MergeConfigPr, its dashboard card, and the deploy DAG it
drove (DeployWindow, MergeVerify, DeployApply, FinalizeDeploy,
DeployTail), with verify_commit, the two-phase meta deploy, rollback
refs, the PR-failure comment and forge/pr_merge.rs
- `fetched_sha`, `sha_short`/`pr_number` on approval events, and
`sha`/`tag` on HelperEvent::ApprovalResolved: only the merge path set
them
`config_repo`, `merged_pr_for_commit` and `post_pr_comment` move to
forge/pr_comment.rs for the merged-rev deploy's refusal comment.
Approvals v5 drops stored `merge_config_pr` rows; a test reopens a v4
database holding them.
Closes#4850
A config PR merged in the Forgejo UI changed nothing on the hive: the
hive's webhook ignores `closed`, its poll then cancels the dashboard
card, and `applied/main` stays where it was.
swarm-controller reads `merged`/`merge_commit_sha` off the
`pull_request` delivery it already receives for `agent-configs`, finds
the hive placing the agent by scanning every hive's wanted state (the
scan `declarations_elsewhere` already ran, factored out), and queues a
`TriggerDeploy` carrying the rev. Zero or several claimants deploy
nothing and log the claimants.
`DeployRequest` gains `rev: Option<String>` with `serde(default)`, so
rev-less payloads from either side keep decoding.
hive-c0re, given a rev for an agent it runs: a no-op when
`applied/main` already is the rev (a dashboard merge deploys its own
PR); otherwise it fetches the forge `main` with the core token,
requires the rev to descend from `applied/main` (the ancestry gate,
factored out of `run_deploy_merge_verify`), fast-forwards by CAS and
queues the usual relocking rebuild. No eval-verify on this path, per
mara (#4850 c90075). A refusal is commented on the PR that merged the
rev, found by commit.
swarm-controller's forge-objects pass converges every config repo's
`main` rule to merge whitelist `operators` + `core` and approval
whitelist `operators`. The hive's boot PATCH stops forcing
`enable_approvals_whitelist` off, so the two do not fight.
Refs #4850
The swarm-controller builds the Forge quick link from
services.hyperhive.swarm.forge.domain, replacing hive-forge/default.nix's
per-host entry, so all seven swarm-service links come from swarm-level
options.
Also drops the remaining references to the removed matrix GUI switch:
the HiveUrls / Urls / hive_urls docs, the hivectl.md `open` note and
the gateway.md vhost-map rows, which name `gatewayHost` instead. The
grafana, victoriametrics and victorialogs modules' comments no longer
mention a quick-link they do not define.
Refs #4885
Removes services.hyperhive.deploy.matrix.gui.enable and its
swarm.matrix.gui.enable alias; both are mkRemovedOptionModule stubs. A
host running the homeserver serves fluffychat at gatewayHost's vhost,
and the hive's /matrix/ redirect follows the same condition.
The swarm-controller builds the Matrix quick link from
swarm.matrix.gatewayHost, replacing hive-matrix.nix's per-host entry.
HIVE_MATRIX_PUBLIC_URL is set on every hive with a gatewayHost, so
`hivectl open matrix` resolves off the homeserver's host too.
Drops HIVE_MATRIX_GUI_ENABLED and the dashboard's matrix_gui_enabled
field; nothing in the frontend reads it.
Refs #4885
hive-c0re/server.rs and hive-host-sock/lib.rs still described the
deleted behindGateway option as if a forge URL could be None for that
reason. Reword both to match the HiveUrls doc's publicUrl wording.
Refs #4885
The forge always sits behind the gateway, so `deploy.forgejo.behindGateway`
(and its `swarm.forge.behindGateway` rename alias) is removed and its
true-branch behaviour is now unconditional within `deploy.forgejo.enable`:
https ROOT_URL on the gateway's httpsPort, the forge vhost and local DNS
name, the swarm-ui quick link, the published metrics scrape target, forgejo
metrics, the authelia `/metrics` rule, and `publicUrl` defaulting to
`https://<forge.domain>`.
Removed with it: the direct-port `http://<domain>:<httpPort>/` ROOT_URL
branch, the hive-ci assertion that the option is true, the core-toggle
cases that only exercised the false branch (the services-leaf case reads
`bare`, which never enabled the forge either). `hivectl open forge` now
points at `swarm.forge.publicUrl`, which can still be set to null.
Refs #4885
`verify_hmac` is the only authentication gate on the public
`/webhook/config-pr` endpoint, and neither of its guard clauses nor the
handler's "unavailable" → 503 / otherwise → 401 mapping had a test.
The tests build a real `AppState` over the tempdir `Coordinator` that
the socket_server schedule tests already use. That helper is widened
from `pub(in crate::socket_server)` to `pub(crate)` and re-exported from
`socket_server` under `#[cfg(test)]`, so non-test code is unchanged.
Removing the missing-secret guard (replacing it with
`unwrap_or_default()`) fails
verify_hmac_rejects_every_delivery_when_no_secret_is_loaded and
config_pr_answers_503_when_no_secret_is_loaded. Deleting the
missing-header guard fails
verify_hmac_rejects_a_delivery_with_no_usable_signature_header. Breaking
the "unavailable" match fails the 503 test.
Without the missing-header guard, an unsigned delivery is still refused
by `verify_signature` (no `sha256=` prefix). The test therefore pins the
guard's own message rather than the bare rejection.
Closes#4654
An operator links an agent's GitHub personal access token in the swarm UI
(LinkGithubAccountForm, "link github account" on /agents). swarm-controller's
PUT /api/hives/{hive}/agents/{agent}/github-account stores it at
swarm/agents/<agent>/github-token (swarm_secret_client::github), a flat leaf
under the agent's prefix that the agent's existing read grant already covers:
no policy change, and no list grant, since there is one token per agent.
In the agent, hive-agent-github-token (oneshot + 2-minute timer, as the agent
user, under its own store certificate, ordered before hive-github-notify)
reads that path and writes <state>/github-token, 0600 and agent-owned, the
file the gh wrapper, git credential helper and hive-github-notify already
read. It replaces the file by rename only when the bytes changed and never
deletes it: a hive-written github-token stays until a token is linked in the
swarm UI. It is installed only with a store address and
services.hyperhive.agent.github.enable.
Removed: the dashboard's CR3D3NTIALS page (credentials.html/js/css, its
build entries and H0M3 tile; GITHUB was its only tab), hive-c0re's
dashboard/matrix_accounts.rs with GET/POST /api/github-account,
priv_client::write_agent_github_token, the host socket's
SetAgentGithubToken and `hivectl github set-token`, and hive-priv's
WriteAgentGithubToken with write_agent_state_file, its only caller gone.
Docs: integrations/github.md and swarm/ui.md describe the swarm path,
swarm/credentials.md gains the store-path row, and the hive UI docs,
hivectl docs and security.md's hive-priv table drop the removed pieces.
Closes#4347
The previous commit removed dashboard.md's M4TR1X section. These
comments and the `deploy.matrix.gui.enable` option description still
described a hive-dashboard M4TR1X tab or cited that section. They now
state what the option does: it serves the client on the gateway vhost
and adds the swarm UI's Matrix quick link (docs/swarm/ui.md::Quick links).
Refs #3902
An operator now links an agent's external forge account (label, base URL,
token) in the swarm UI. swarm-controller stores it at
swarm/agents/<agent>/forge/<label>. There is no index: the store's
listing of the agent's forge/ directory is the set of accounts.
In the agent, hive-agent-forge-accounts (oneshot + 2-minute timer, as
the agent user, under its own store certificate) lists
swarm/agents/<agent>/forge/ with the `list` #4866 grants an agent on its
own metadata subtree, reads each account, and writes
<state>/forge-<label>-token and forge-<label>.json in the names and shape
hive-forge -f already reads. An empty listing (a 404, which `bao kv list
-format=json` answers with `{}` and an empty stderr) is zero accounts; a
denial or an unreachable store fails the unit. It never deletes: files
for labels not listed, including ones the hive wrote, stay as they are.
Removed: the dashboard FORGES tab (credentials.js/html section and its
CSS), hive-c0re's extra_forges.rs and its routes, priv_client's
extra-forge calls, and hive-priv's WriteAgentExtraForgeAccount /
DeleteAgentExtraForgeAccount with their helpers. The GITHUB tab and
WriteAgentGithubToken stay.
Also: persistence.md's matrix avatar note names the exit-75 restart on a
changed account listing, not the dashboard, as what brings a linked
account up.
Refs #4348
hive-matrix-daemon now learns which external matrix accounts it has from
the swarm secret store, under the agent's own certificate, and the hive
push chain for matrix is gone.
The daemon lists swarm/agents/<agent>/matrix/ (the `list` its policy
grants on its own metadata subtree), reads each account's homeserver
from its credential, and brings the accounts up with their tokens from
the store. Every two minutes it lists again and exits with 75 when the
set of linked accounts changed; the unit restarts on 75 without counting
a failure. A listed name whose credential reads as absent is skipped and
logged once. At start it removes the matrix-token-<a> /
matrix-account-<a>.json pairs a hive delivered (a sidecar marks a pair
as delivered; a declared tokenFile keeps its token).
Removed: CredentialNotice and the $SWARM.credential.* subject and NATS
grant, the controller's publish and its queue precondition on the PUT
route, hive-c0re's credential subscription arm and workers/credential.rs,
priv_client::write_agent_matrix_token, hive-priv's WriteAgentMatrixToken
and its helpers, and the daemon's state-dir account discovery.
Kept: WriteAgentGithubToken and the external-forge path
(WriteAgentExtraForgeAccount, extra_forges.rs) are untouched, and a
declared matrixAccounts tokenFile is still read when the store has no
token for that account.
Refs #4348
mara ruled on #4849 (c88934): "remove create_repo tool". The tool ran in
hive-c0re with the hive's core token, so it only ever worked for agents
on the hive that runs the forge.
Removed:
- the create_repo MCP tool and CreateRepoArgs (hive-agent-mcp)
- wire variants Request::CreateRepo and Response::RepoCreated
(hive-core-agent-sock)
- hive-c0re's handle_create_repo, its valid_repo_name check and the
dispatch arm
- forge::create_agent_repo and apply_operator_branch_protection, which
had no other caller, plus AGENTS_ORG and OPERATORS_TEAM, whose only
users they were
- the tool's docs (docs/tools/forge.md repo management, docs/turn-loop/
mcp.md, the conventions tool-group table) and the doc comments that
named it (hive-sock-client's response timeout, ensure_repo_creation_
disabled, the security doc's merge-gate bullet)
ToolGroup::Forge is kept with no tools, the same way b88a5b24 kept
Lifecycle, so existing meta/capabilities.json grants still parse.
Forge state is untouched: existing agents/* repos keep their collaborators
and operators-team branch protection. The swarm-controller's own
create_repo (config-org repos) is a different path and is unchanged.
Closes#4849
Every hive is in a swarm and every swarm runs matrix, so every swarm has a
swarm-controller, and since #4810 its hive_sender pass mints each hive's
@hive-<hive>: sender token into the store every five minutes. The two
other minters of that token go:
- swarm-matrix-ctl mint: the systemd.services.swarm-matrix-ctl unit in the
hive-matrix container, Command::Mint and src/mint.rs. The binary, its
appservice render/publish verbs, ctlPackage, ctlActive and the ctl cert
role stay. bao-matrix-reader's checks on the deleted unit are removed;
the leaf-identity and no-token-in-env checks now look at
swarm-matrix-appservice-publish, which runs under the same identity.
- the hive-side mint ladder in hive-c0re's ensure_hive_user
(register/appservice-login/password-login with the local as_token), with
read_appservice_token, paths::matrix_appservice_token and the helpers
only it used. ensure_hive_user now takes the store's token, keeps the
file when the store has none or can't be reached, and fails otherwise.
- hivectl matrix sync-admin: the verb, HostRequest::MatrixSyncAdmin and
handle_matrix_sync_admin. The periodic MatrixSweep (ensure_all) is
unchanged apart from no longer reading the local as_token.
This removes the double-mint race #4810's review flagged: two minters
logging in on one pinned device could leave a dead token in the store
until the next pass.
Closes#4813Closes#4814
A hive whose homeserver runs on another host has no local
matrix-appservice-token, so hive-c0re's matrix sweep returned before
reaching the store read in ensure_hive_user: no @hive-<name>: token, no
Space, no chat room, no invites, and a sweep-health banner.
swarm-controller now mints @hive-<name>: with the swarm appservice
token for every hive in its directory, as a MintHiveSenderToken job
node queued by a five-minute pass, and stores it at
swarm/hives/<name>/matrix/sender-token, the same matrix::Credential
swarm-matrix-ctl writes there. It is keep-if-live, reusing agent_token's
classify/plan: a stored token whoami confirms as @hive-<name>: is left
alone, so only an absent or dead one is minted. agent_token's probe and
mint steps are lifted into probe_at/mint_at so both passes share them.
swarm-matrix-ctl mint still writes the path for its own hive when it is
empty. If both mint an empty path at once, one token is invalidated
(same pinned device); the next pass classifies it Revoked and re-mints.
hive-c0re's ensure_all no longer returns when there is no local
as_token. ensure_hive_user reads the store first on every sweep and
overwrites its token file when the store's token differs, keeps the
file when the store has none, mints with the local as_token only when
neither holds one, and fails with one error when there is nothing at
all. The decision is sender_source, unit-tested.
The controller's bao policy gains create/read/update on
swarm/hives/+/matrix/sender-token (`+`, since `*` is a glob only at the
end of a path), pinned in module-eval.
Refs #4427
swarm-controller's POST /api/agents now refuses (409) a name the swarm
has already placed on a different hive: a non-Destroyed declaration in
that hive's wanted state, or a SetAgentWanted node still queued for it.
The same name on the same hive is that agent being re-created and goes
through. A wanted state that cannot be read refuses (503/500) instead of
reading as "placed nowhere". Creations are serialised from that read to
the graph insert so two concurrent creations of one name cannot both
pass.
Hive-level creation is removed: hivectl `agent create` / `request-create`,
HostRequest::Spawn / RequestSpawn, the dashboard POST /api/request-spawn
route, and ApprovalKind::Spawn with its approve/resolve arms and the
approval-carrying `templates::spawn`. The swarm path (deploy request or
wanted-state sweep -> queue_first_deploy -> templates::first_deploy) used
none of them. Old `spawn` approval rows are skipped by collect_lenient,
as `init_config` rows were in a3b672d1.
policy.rs's comment on agent_object_name stated swarm-wide name
uniqueness as a fact; it now says where it is enforced and what that
check cannot see.
Refs #4396
The CR3D3NTIALS page's MATRIX tab was the only caller of
`POST /api/matrix-account-login` (provision/log in an external matrix
account through the hive) and `GET /api/matrix-accounts` (its account
list). External matrix accounts are linked from the swarm UI now
(`LinkMatrixAccountForm` -> swarm-controller), so the hive-side UI and
both routes go. `priv_client::restart_matrix_daemon` had no other caller
and goes with them.
Already-provisioned credentials keep working: the `matrix-token-<name>`
files and `matrix-account-<name>.json` sidecars the old route wrote are
still discovered by hive-matrix-mcp (`accounts::configured` ->
`discover_token_accounts`), the `matrix-token*` path unit still re-fires
the daemon, and `WriteAgentMatrixToken` stays for the swarm credential
worker. Removing that usage waits on moving the existing creds to
swarm level.
The GITHUB tab is the credentials page's default tab now.
Refs #4348
Human matrix accounts come from SSO, not hivectl. Matrix homeserver
admin will come from authelia's admins group (sync tracked in #4585);
password reset moves to swarm level (#4798). promote-user and
reset-password were already broken from the hive: the hive's sender
account has no admin sender to call the admin room with, only the
swarm's does.
Removes the three hivectl matrix verbs, their HostRequest variants,
their hive-c0re handlers, and the admin-room helpers (discover room id,
send-and-poll, event-id extraction, password/success parsing) that
only they used. sync-admin and invite are unchanged.
Refs #4585
post_purge_tombstone discarded fail_pending_for_agent's error with
let _ =. Mirrors #4740's fix for the identical discard in
job_queue/exec.rs's run_destroy_bookkeeping: warn and continue, since
the purge itself has already succeeded by this point.
Refs #4747
du_bytes returned None uniformly for every du failure, and both
du_bytes and measure_agent_disk had no logging at all, so a container
rootfs du couldn't read (permission, I/O error, unparseable output)
silently reported as 0 disk with no signal in the journal.
Distinguish the expected case — the path plain doesn't exist, e.g. a
destroyed-but-kept agent's rootfs or a not-yet-created state dir — from
an actual read failure, and warn only on the latter, with the path plus
whichever of the failure (spawn error, exit status, stderr, unparseable
stdout) applies.
prebuild_toplevel awaited nix build with a bare child.wait().await, so
a wedged nix-daemon (unreachable remote builder, stuck build slot)
hung the rebuild job forever with no way for the job queue to recover
short of restarting hive-c0re.
Mirrors #4741's hive-priv fix: the child now leads its own process
group, and after PREBUILD_TIMEOUT (1h, four times CI's observed
cold-cache flake check) the whole group is SIGKILLed and the call
fails with a named timeout error instead of hanging.
Refs #4723, #4741
hive-screen-mcp ran `grim` / `wtype` through an unbounded
`Command::output()` and spoke RFB to neatvnc with no deadline, so a
wedged compositor or a VNC server that accepts and never speaks held the
agent's turn forever. Each subprocess now has a 30s limit and is killed
when it hits it; each RFB exchange has a 10s limit. Both come back to the
agent as the tool's text result, like every other failure in this crate.
hive-c0re operator-facing text that described behaviour the code lacks:
- `hivectl matrix reset-password` printed a "next: hivectl matrix
create-user" hint that fails for every target (agents are refused, a
non-agent hits M_USER_IN_USE). The line is gone.
- a failed `nixos-container update` appended the container's journal
tail, read with `journalctl -M`. Since the job DAG, `update` only runs
from the `Swap` node on a stopped container, so the read always came
back empty. The helper is removed; the error still points at the
build log.
- the matrix sweep comment in main.rs said it re-provisions agent token
files; `ensure_all` creates no agent accounts or tokens.
- the knowledge-pull comments named a webhook caller that no longer
exists and claimed a race was "fixed at its source".
- `handle_spawn`'s doc and the `mcp_sockets` module doc named callers of
`register_agent` / a rollback that do not exist.
Refs #4723
mara chose to accept extra queued sweep passes over adding a new
hive-jobq primitive (or a hive-c0re one-off) for "don't queue another
of this kind". Every sweep caller now plain-inserts its node; the
capacity-1 Dep::Resource per sweep kind (MatrixSweep/KnowledgeTree)
still keeps two passes of the same kind from running concurrently, it
just no longer collapses a tick that lands while one is live or queued
into the existing one.
The matrix sweep and the /knowledge pull each had concurrent callers
(#4723 item 4). Two overlapping knowledge pulls fail on .git/index.lock
and the remote-tracking ref lock: 30 of 30 concurrent replays of the
reset/clean/pull sequence in a scratch repo errored, 0 of 10 sequential
ones did. Two overlapping matrix sweeps on a hive with no persisted
Space / chat-room id both miss the by-name lookup and both createRoom
(from reading the code, not reproduced against a homeserver). On every
boot the MatrixSweep DAG node and the main.rs loop's immediate first
call ran at once.
Every sweep now runs as a job node, and each sweep's node holds its own
capacity-1 queue resource (Resource::MatrixSweep,
Resource::KnowledgeTree), the MetaWindow pattern: the scheduler never
starts a second pass of one sweep while the first holds the resource,
and different sweeps still run side by side.
- templates::matrix_sweep / templates::knowledge_pull build the node
with its resource; boot, the periodic loops and the swarm event all
use them.
- JobQueue::insert_unless_live folds a submission into a live node of
the same kind instead of queueing another. Periodic ticks fold into a
queued or running pass. The swarm knowledge event folds into a queued
pull only, and queues one behind a running pull, which may have
fetched before the push.
- The main.rs matrix loop no longer sweeps immediately at startup; the
boot MatrixSweep node is the startup pass, as KnowledgePull already
was for knowledge.
- The executors bound each pass (10 min matrix, 5 min knowledge), since
a hung pass would otherwise hold its resource against every later one,
and own the sweep-health banners, so every pass reports to them.
Replaces the SweepLock version of this branch, per review.
Refs #4723
/etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290)
that pre-created every agent's bind sources. The start preamble already
creates them for every c0re-driven start, and on this host only hive-c0re
starts agent containers. The file was also the reason the socket dir's
owner had to be declared there, which is how it spent its life at
`0777 root root` whenever the uid could not be resolved (#4742).
- hive-priv gains `EnsureAgentSocketDir { name }`, called from
`set_nspawn_flags` in every start path. It creates
`/run/hive-agent/<name>` `0751 root:root` with mkdirat relative to an
O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a
directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a
directory is left alone. hive-c0re's own create_dir_all went: its /run
is read-only under ProtectSystem=strict.
- The container's `hive-agent-user-migrate` activation chowns that dir to
the agent user and sets 0751, the same way it already handles state/ and
harness/. It refuses a symlink or non-directory there, since `test -d`
and chmod follow links. No host-side passwd parse, and no window where
the dir is world-writable.
- `/run/hyperhive/agents/<name>` stays created by hive-c0re itself
(`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re
binds as hive-core, so it must not become root- or agent-owned.
- The `/run/hive-agent` parent is declared in hive-priv.nix, `0755
root:root`, instead of hive-gateway's hive-core rule. hive-priv is its
only writer now, and hive-priv's ReadWritePaths needs it to exist.
- The manager start in `ensure_root_agent` now goes through
`converge_start_preamble` + `start_with_fallback`. It was a bare start,
so after a reboot the manager's bind sources existed only because of the
tmpfiles file, and its limits drop-in did not exist at all.
- Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`,
`priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles
body builder and their tests, plus the three call sites.
- Legacy: hive-priv unlinks the file at every start, ignoring ENOENT.
`SyncAgentTmpfiles` stays one release as a payload-ignoring variant that
does the same unlink and returns Ok, for an older hive-c0re.
Salvaged from #4752: the boundary.md correction that nginx only dials,
because ProtectSystem=strict makes its /run read-only.
Behaviour change: a manual `nixos-container start h-<name>` right after a
reboot, before hive-c0re has started that agent, now fails on a missing
bind source instead of starting.
Closes#4742
An agent name that is not a valid Ident made `Coordinator::agent_paths`
panic. Job payloads carry names as plain strings (the swarm's published
wanted state is one source), and a panic inside a job-queue node never
reaches `complete_growing`, so the node's resources (the deploy
window included) were held until hive-c0re restarted. `agent_paths` now
returns an error; the job-queue nodes, the admin-socket spawn and
set-limits paths, the root-agent spawn and the dashboard set-limits
handler propagate it.
`lifecycle::list().await.unwrap_or_default()` turned a failed container
list into "no agents":
- meta-update cascade: the lock bump committed and zero rebuilds fanned
out, reported as success. The cascade is now resolved before the lock
bump and a list failure fails the node.
- dashboard update-all: queued nothing and returned 200 "ok". Now 500
with the error.
- container rescan: every row was emitted as removed and the cache
emptied. Now the last snapshot stands; `hivectl status` gets an error.
- dashboard journal: answered 404 "no managed container". Now 500.
- spawn/rebuild port-collision check: silently skipped. Now fails.
- startup migration: the per-agent phases ran over nothing, and phase 3
handed an empty agent list to `meta::sync_agents`, which renders the
meta flake with exactly the agents it is given. Both now log the list
failure and skip.
The hive-jobq scheduler still leaks a node's resources on any executor
panic; that root is not addressed here.
Refs #4723
invite_user_id mapped every 403 M_FORBIDDEN to Ok(()). The membership
pre-check already skips invited/joined users, so the 403s that reach the
POST are mostly real refusals (banned target, sender without power),
including `hivectl matrix invite`. A 403 is now success only when a
membership re-read shows the user invited or joined; otherwise it is an
error carrying the status and body.
admin_room_send_and_poll read the send response's event_id with
unwrap_or_default() and, when it was missing, walked every recent event
unanchored, so an older bot reply (an earlier reset password) could be
returned as this command's result. A send response without an event_id
is now an error.
run_destroy_bookkeeping discarded fail_pending_for_agent's error; it now
warns like its neighbouring steps.
ensure_config_pr_webhook returned as soon as a hook with the target URL
existed, so a regenerated webhook-secret never reached Forgejo and every
config-PR delivery failed HMAC until the 5-minute poll caught up.
Forgejo's edit-hook API ignores `secret` and never returns it, so the
SHA-256 of the secret last registered is recorded at
forge/config-pr-webhook-secret-sha256; when it doesn't match, the
same-URL hook is deleted and recreated. The paths.rs doc claiming
re-registration on change now describes this.
Refs #4723
rustls is built with both `ring` (async-nats's `ring` feature) and
`aws-lc-rs` (reqwest's `rustls` feature), so it cannot pick a
process-level default by itself. Since the queue started requiring TLS
(1d261b3f), async-nats builds its config with `ClientConfig::builder()`,
which panics without an installed default. The panic kills the async-nats
connector task, and every queue client (swarm-controller, hive-c0re, all
hive-agents) has sat in `Pending` since the 2026-09-25 23:04Z deploy.
Add `swarm_queue_client::install_crypto_provider()`, which installs
aws-lc-rs and ignores the "already installed" error. It is called first in
`main` of every binary that links async-nats: hive-agent, hive-c0re,
swarm-controller, swarm-nats-auth. `connect()` also calls it, so a new
binary that dials through this crate is covered without remembering to.
aws-lc-rs because reqwest already falls back to it when no default is
installed, so HTTPS in these processes keeps its current provider. The
other rustls users in the tree reach it only through reqwest, which never
panics here.
Closes#4738
Both accept loops returned on the first accept() error. The AgentSocket
stayed in Coordinator.agents, so mcp_sockets::sync_on_start skipped the
agent, and only a Start node or a hive-c0re restart bound it again. The
unit sets no LimitNOFILE, so a single EMFILE at the 1024-fd soft limit
cut the agent off from send/recv/ack with one warn line as the only
trace.
Both loops now share accept_until_fatal. Connection errors
(ECONNABORTED/ECONNRESET/ECONNREFUSED) retry at once, any other error
retries after 1 s, following axum's serve loop that the dashboard
already runs. Errors that leave the listening fd unusable (EBADF,
EFAULT, EINVAL, ENOTSOCK, EOPNOTSUPP) log at error and exit the
process: nothing re-binds a listener while hive-c0re runs, the manager
listener has no re-bind path at all, and the unit's Restart=on-failure
restart runs start_manager and sync_on_start, which re-bind every
socket.
mcp_sockets.rs now states that invariant instead of asserting that a
listener can only disappear on restart.
LimitNOFILE is left unset: per-agent fd use is a listener plus a
Recv long-poll plus short-lived requests, and a higher limit would raise
the memory ceiling of the per-line bound (bound x open connections).
Closes#4721
serve() read each request with read_line into an unbounded String, so
size checks such as the 4 KiB Send body limit ran only after the whole
line was buffered. Any process in an agent container can write to
/run/hive/mcp.sock, and one stream with no newline grew hive-c0re's heap
until the host OOM-killed it.
Read at most 16 MiB per line. The largest request a client sends is an
OperatorMsg from the agent web UI's /send, whose body axum's default
limit caps at 2 MiB (the gateway's nginx admits 10 MiB); JSON escaping
can roughly double that, and the cap is 4x the result. A longer line
gets a Response::Err, the same shape as the parse-error path, and the
connection is closed because the rest of the line is still unread.
The line is read as bytes and parsed with serde_json::from_slice, so a
line with invalid UTF-8 now gets a parse-error response instead of the
connection being dropped.
Closes#4718
hive-sock-client: each attempt now bounds connect (5s), write (10s) and
the wait for the response (60s by default). The response bound is per
call through the new `request_within`, which hive-agent's serve-loop
`Recv` uses with its 180s long-poll plus 30s headroom. A response
timeout is terminal rather than retried: the server holds the request,
so a retry re-sends something it may still act on and multiplies the
wait by the backoff schedule.
Outbound HTTP: the matrix login/whoami clients in swarm-controller and
hive-c0re's dashboard (5s connect, 30s request), the authelia-bridge
client (5s/30s; ensuring an identity runs an argon2 hash first) and the
ci-runner forge calls (5s/15s, config_pr_poll's forge budget) get a
connect_timeout and a request timeout. Timeout errors name the bound
that fired.
hive-agent's unix-socket extra web proxy bounds the connect (5s) and
the wait for the response head (30s, the http sibling's budget); the
body read stays unbounded.
Refs #4723
