Watch
0
0
Fork
You've already forked hyperhive
0
hyperhive/docs/4-implementation/swarm/knowledge.md

108 lines
5.5 KiB
Markdown

# Knowledge repo: sync, change notice and state
How a hive keeps its clone of `internal/knowledge` current, tells agents when it
changes, and mounts it into each container. The operator's view is
[the knowledge page](../../2-handbook/swarm/knowledge.md).
## Sync mechanism
hive-c0re maintains the local clone at
`/var/lib/hyperhive/knowledge` via two paths:
1. **Swarm event** — the swarm controller holds the single push hook on
`internal/knowledge` (see
[swarm-wide forge webhooks](controller.md#swarm-wide-forge-webhooks)). On any push to main, including merge commits, it sends an
event to every hive over the swarm queue and each hive runs `git
pull`, so agents see the new content on their next turn.
A hive that's offline when the swarm controller sends the event
doesn't get it on reconnect — the periodic pull below is what closes that gap. One
hive briefly showing older `/knowledge` content than another is
expected, and resolves by itself within the fallback interval.
**don't add a per-hive hook.** A webhook has exactly one target
URL, so a second registration against the same repo doesn't add a
recipient — it takes delivery away from whoever registered first.
Earlier versions had each hive register its own, and later ones
removed that leftover at startup. A hive upgraded straight past those
versions still holds its old hook: delete any hook on the repo that
points at a hive's own `/webhook/knowledge`.
2. **Periodic pull** — a background task in `hive-c0re::main`
queues a pull on a fixed cadence as a fallback (webhook missed, c0re
restarted between pushes). The pull is best-effort — a failure
shows as a failed `KnowledgePull` node on the job queue, counts toward
the `knowledge_pull` hive warning, and doesn't affect the rest of the daemon.
A failed pull of either kind, or the one at boot, also queues a retry after 5 s,
doubling per failure in a row up to 5 min, until a pull succeeds. The
`knowledge_pull` hive warning appears after seven failures in a row, about five
minutes in, so a forge still starting at boot raises nothing.
Both paths, and the one pull at boot, queue the same `KnowledgePull` job
node. It holds the working tree for its duration, so two pulls never run
over each other — each trigger queues its own pass, and a pass that lands
while another is already queued or running waits its turn, since the
one ahead of it may have fetched before the push. The node runs
`knowledge::pull()`, which also handles the change notice below.
### Change notice
When a pull actually moves the local clone's `HEAD` (a real change,
not a no-op — for example the periodic pull finding nothing new), hive-c0re
broadcasts a short notice to every currently registered agent's inbox:
sender `system`, body `[system] /knowledge updated:` followed by a
`git diff --stat <old>..<new>` summary of what changed (or a generic
"see the repo" fallback if computing the diff itself fails). This is
the same broadcast mechanism used for other hive-wide notices — inbox
message only, no forced wake, and it carries the standard "this was a
broadcast" hint. hive-c0re logs a diff or per-agent send failure
without blocking the pull itself.
On a hive with no local clone yet, `knowledge::pull()` takes the clone
instead of pulling, so every hive gets `/knowledge` from its first pull,
whether or not it runs the forge. It authenticates with the hive's forge
read token when it has one (see
[forge-accounts.md](forge-accounts.md#how-a-hive-reads-the-forge)) and clones the public
repo anonymously otherwise.
### The agent's own clone
Each agent harness also keeps its own shallow clone of `internal/knowledge`
(`hive-agent/src/knowledge_sync.rs`), under the harness dir as `knowledge/`,
without going through the hive. The agent's queue token may subscribe to
`$SWARM.knowledge` (and not publish it), so the harness hears the same event
as the hive: on it, once at start and hourly as a fallback, it fetches and
resets the clone to the remote head (`GIT_TERMINAL_PROMPT=0`, a five-minute
limit per git command, anonymous because the repo is public). When the head
moved it raises one keyed `knowledge` todo carrying the `git diff --stat`, so
a second change before the agent reads the first updates the same row; the
first clone raises nothing. A failed or timed-out sync logs a warning, and the
next trigger retries it.
This runs next to the host mount for now: `/knowledge` is still the host's
bind mount and the `/knowledge updated` broadcast above still goes out, until
a later change removes the host side.
## State
<!-- vale write-good.Passive = NO -->
- **Host clone**: `/var/lib/hyperhive/knowledge` — persists across
hive-c0re restarts and agent destroy/recreate. Deleted only by
manual operator action.
- **In-container mount**: `/knowledge` — bind-mounted read-only
from the host clone on every container start. Gone when container
is stopped; reappears on next start with the current clone state.
<!-- vale write-good.Passive = YES -->
The mount deliberately **excludes `.git`**: hive-priv overlays an empty
tmpfs at `/knowledge/.git`, so agents see the documents, not the branches or
history. An older clone also kept a token in `.git/config`; `knowledge::pull()`
rewrites that remote to the plain URL.
hive-priv adds the overlay only when the host clone has a `.git`. Before the
first clone succeeds, the agent starts with an empty `/knowledge` and no
overlay, because nspawn can't create the mount point inside the read-only
bind. A container that started in that window sees the `.git` once the clone
lands, until its next start.