hive-screen-mcp ran `grim` / `wtype` through an unbounded
`Command::output()` and spoke RFB to neatvnc with no deadline, so a
wedged compositor or a VNC server that accepts and never speaks held the
agent's turn forever. Each subprocess now has a 30s limit and is killed
when it hits it; each RFB exchange has a 10s limit. Both come back to the
agent as the tool's text result, like every other failure in this crate.
hive-c0re operator-facing text that described behaviour the code lacks:
- `hivectl matrix reset-password` printed a "next: hivectl matrix
create-user" hint that fails for every target (agents are refused, a
non-agent hits M_USER_IN_USE). The line is gone.
- a failed `nixos-container update` appended the container's journal
tail, read with `journalctl -M`. Since the job DAG, `update` only runs
from the `Swap` node on a stopped container, so the read always came
back empty. The helper is removed; the error still points at the
build log.
- the matrix sweep comment in main.rs said it re-provisions agent token
files; `ensure_all` creates no agent accounts or tokens.
- the knowledge-pull comments named a webhook caller that no longer
exists and claimed a race was "fixed at its source".
- `handle_spawn`'s doc and the `mcp_sockets` module doc named callers of
`register_agent` / a rollback that do not exist.
Refs #4723
The matrix sweep and the /knowledge pull each had concurrent callers
(#4723 item 4). Two overlapping knowledge pulls fail on .git/index.lock
and the remote-tracking ref lock: 30 of 30 concurrent replays of the
reset/clean/pull sequence in a scratch repo errored, 0 of 10 sequential
ones did. Two overlapping matrix sweeps on a hive with no persisted
Space / chat-room id both miss the by-name lookup and both createRoom
(from reading the code, not reproduced against a homeserver). On every
boot the MatrixSweep DAG node and the main.rs loop's immediate first
call ran at once.
Every sweep now runs as a job node, and each sweep's node holds its own
capacity-1 queue resource (Resource::MatrixSweep,
Resource::KnowledgeTree), the MetaWindow pattern: the scheduler never
starts a second pass of one sweep while the first holds the resource,
and different sweeps still run side by side.
- templates::matrix_sweep / templates::knowledge_pull build the node
with its resource; boot, the periodic loops and the swarm event all
use them.
- JobQueue::insert_unless_live folds a submission into a live node of
the same kind instead of queueing another. Periodic ticks fold into a
queued or running pass. The swarm knowledge event folds into a queued
pull only, and queues one behind a running pull, which may have
fetched before the push.
- The main.rs matrix loop no longer sweeps immediately at startup; the
boot MatrixSweep node is the startup pass, as KnowledgePull already
was for knowledge.
- The executors bound each pass (10 min matrix, 5 min knowledge), since
a hung pass would otherwise hold its resource against every later one,
and own the sweep-health banners, so every pass reports to them.
Replaces the SweepLock version of this branch, per review.
Refs #4723
The orgs agent-configs/internal/agents (plus mirror owners), the
operators team in agents and agent-configs, the pull-mirrors,
internal/docs, internal/knowledge (public, README-seeded) and the
agent-configs org avatar are one set per forge. hive-c0re ensured them in
its boot sweep, as the core admin, and only on the hive co-located with
the forge container.
swarm-controller now reconciles them at start and every 5 minutes
(forge/objects.rs: observe -> pure plan -> apply). A failed object logs
a warn line plus a pass summary and is retried next tick. create_repo
ensures the agent-configs org and its operators team first, so a config
repo's merge gate never depends on the periodic pass having run.
hive-c0re drops ensure_org, SEEDED_ORGS, ensure_mirrors/ensure_mirror_repo,
ensure_operators_team, ensure_shared_docs_repo, ensure_knowledge_repo/
set_repo_public, seed_readme, ensure_config_org_avatar and the one-shot
knowledge::remove_webhook cleanup, with their now-unused helpers.
nix: the mirror list moves from the hive-c0re unit
(HYPERHIVE_FORGE_MIRRORS) to the swarm-controller unit
(SWARM_CONTROLLER_FORGE_MIRRORS), with an eval warning when mirrors are
declared on a host that runs no controller. c0re.orgAvatarPng is renamed
to deploy.swarm-controller.configOrgAvatarPng.
Refs #3782
A webhook has exactly one target URL, so every hive registering one
against the shared internal/knowledge repository was last-writer-wins
rather than idempotent: all but the most recent silently stopped
receiving deliveries. The swarm controller holds the single registration
and now addresses an event to each hive over the queue instead.
This is a migration, not a deletion. Not registering any more fixes
nothing on a hive that has already run — the hook it created persists on
the forge, so the contention would survive on exactly the deployments
that have it while fresh installs looked fixed. The hive that created a
hook removes it.
It removes only its OWN, matched on the full URL rather than the
/webhook/knowledge suffix. A hook with that suffix and a different base
belongs to another hive, possibly one not yet upgraded, and deleting it
would break that hive's knowledge sync until it caught up. Reaping a
neighbour's registration is the behaviour being removed here; doing it
while fixing it would only invert the direction.
The predecessor did reap by suffix, to clear loopback hooks left by an
older single-hive layout. That was safe when a hive was alone on its
forge and is not safe now. The hive-side registrars also acted as reapers
of hooks under their own path, which is why the swarm hook lives under
/webhook/forge/; removing this registrar removes that reaper too.
Intended, and stated because no reviewer would infer it from the diff.
The receive endpoint goes with it. A live HMAC-verified
/webhook/knowledge that nothing can legitimately reach would tell the
next reader that this is how a hive learns about knowledge changes.
Docs move in the same commit: docs/swarm/README.md said two hooks exist
per swarm-wide repo and neither should be deleted, which is now true for
agent-configs and wrong for internal/knowledge — a half-correct
description being worse than an uncorrected one.
A bare `git pull --ff-only` merges whatever `branch.<current>.merge`
lists. A clone carrying more than one such entry aborts with "Cannot
fast-forward to multiple branches", which takes /knowledge out for every
agent on the hive - and the daemon neither writes that config nor can
see it, so the call worked only for as long as it happened to stay
clean.
Reproduced against two throwaway bare repos: duplicate merge entries
give that exact fatal, exit 128. A detached HEAD gives a different
error, and a missing tracking config does not reproduce at all - the
clone of an empty repo does write the tracking entry, so my first
explanation was wrong.
Naming origin and main makes the pull independent of local branch
config: a stray entry degrades to "the pull did not pick it up" instead
of breaking the shared mount. Does not explain how a second entry
appeared; nothing in this tree writes branch config.
Remove or fix broken documentation links that accumulate silently:
- container_view.rs: HiveEnv reference
- forge/mod.rs: READY_TIMEOUT and webhook handler links
- workers/knowledge.rs: webhook handler link
- job_queue/model.rs: Claim::deps and WireNode::data references
- stats/hive_stats.rs: read_skill_breakdown reference
- stores/audit_log.rs: global() reference
- workers/agent_sockets.rs: ambiguous agent_sockets::write reference
- coordinator.rs: systemd.services.<harness> formatting
- resource_limits.rs: ambiguous write/read references
Some broken links were to deleted functions/types; these are replaced
with prose descriptions. Others referenced items outside this crate or
were private; these are replaced with plain text references or qualified
paths as appropriate.
Fixes: #3245
`forge_git_url` spliced `core:<token>@` between scheme and authority, and
that URL is a process argument. `/proc/<pid>/cmdline` is mode 0444 —
world-readable — so the core admin token, which provisions every agent's
forge account, was published to any local user for the lifetime of each
git child. Seven call sites built such a URL.
The credential now travels in the environment instead:
`git_command_authed` sets `http.extraHeader` via `GIT_CONFIG_*`, which
git reads exactly like a config file, and `/proc/<pid>/environ` is 0400 —
owner-only. Same credential, materially smaller audience. The remote is a
plain `http://forge/<org>/<repo>.git`, and `forge_git_url` no longer takes
a token, so the old shape cannot be rebuilt by accident.
`knowledge`'s clone was the one place a credentialed URL was stored as a
named remote — git persists the clone URL into `.git/config`, so the
token sat on disk and every later `pull` authenticated from there. That
is the case `forge::repos::push_config` documents as forbidden ("the
tokenised URL ... deliberately never stored as a named remote"). `pull`
now rewrites `origin` to the plain URL first, which also scrubs the
persisted token from existing deployments, and authenticates from the
environment when a token is available. The repo is public, so the pull
still works without one.
Three call sites also stopped spawning `Command::new("git")` directly,
so they honour the `HYPERHIVE_GIT` path the NixOS module bakes in and
the `kill_on_drop` every other git spawn gets.
The two URL-shape tests now assert the *absence* of a credential, and a
new one decodes the header back to `core:<token>` — without that, a
malformed header would leave every forge operation silently anonymous
with the other assertions still green.
Two issues flagged by argus in PR #2388 review:
1. Empty-key fallback: when load_or_generate() failed, webhook_secret was
String::new(). An attacker knowing this could forge deliveries with a
valid HMAC of the empty key. Fix: change to Option<String>; on None,
skip hook registration entirely and return 503 from /webhook/* handlers
(rather than 401 with a misleadingly-verifiable empty-key HMAC).
2. Stale hook cleanup: on upgrade from old code, old loopback hooks
(http://127.0.0.1:.../webhook/knowledge, .../webhook/config-pr) were
left alongside the new domain-URL hook. Fix: during ensure_webhook /
ensure_config_pr_webhook, after listing hooks, delete any that end with
our path suffix but point at a different base URL.
clippy + nix fmt clean.
Both webhook registrations (knowledge push + config-PR pull_request) now
use the public hive domain instead of loopback:
https://<HYPERHIVE_HIVE_DOMAIN>/webhook/{knowledge,config-pr}
This routes deliveries through the gateway, bypassing the Forgejo SSRF
guard that blocked loopback delivery and silently broke the config-PR
merge flow since launch.
Changes:
- webhook_secret: new module — auto-generate + persist a 32-byte HMAC
secret to STATE_ROOT/webhook-secret on first startup; verify
X-Hub-Signature-256 on every incoming webhook POST (HMAC-SHA256).
- forge/mod.rs: ensure_config_pr_webhook now takes hive_domain +
webhook_secret; sets secret in Forgejo hook config.
- workers/knowledge.rs: ensure_webhook same update.
- dashboard/webhook.rs: both handlers read raw Bytes first, verify HMAC,
then parse JSON. Returns 401 on signature mismatch.
- dashboard/mod.rs: AppState carries webhook_secret; serve() takes it.
- main.rs: load/generate secret at startup; pass to registration tasks
+ dashboard; add 5-minute config-PR polling fallback task.
- forge/config_pr_poll.rs: new — scan agent-configs/* for open PRs with
no pending MergeConfigPr approval; queue them. Idempotent.
- stores/approvals.rs: has_pending_merge_config_pr() for poll dedup.
- nix/modules/hive-gateway.nix: remove dashboardAuth from /webhook/
location (HMAC replaces basic auth for webhook endpoints; Forgejo
cannot send HTTP Basic credentials with webhook deliveries).
Replace the hardcoded FORGE_HTTP const with forge_http_base() which
reads HIVE_FORGE_URL from the environment (already set unconditionally
by hive-c0re.nix to http://<forge.domain>). Add forge_git_url() helper
that inserts core:<token> credentials between scheme and authority for
git push/clone URLs.
All call sites updated:
- forge/mod.rs: api() OnceLock + new forge_git_url/forge_http_base fns
- forge/repos.rs: push_meta, push_config, ensure_meta_remote
- forge/pr_merge.rs: tokenised_repo_url delegate + test loosened
- workers/knowledge.rs: clone + push URLs
- socket_server/mod.rs: clone_url in RepoCreated response
No new env var: HIVE_FORGE_URL was already the right knob (mara).
Closes#1868. Closes#2174 (this supersedes the operators-team fix from
the closed#2218, which is re-applied in the ensure_operators_team call
that was already merged separately).
