mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: move implementation sections out of the handbook pages

Browse source 
Three handbook pages carried sections an operator never reads, named after
source paths and build internals. They move to pages under 4-implementation,
and each handbook page keeps a short pointer.

- forge.md: the notification poller (what it polls, how it formats,
  de-duplicates and marks notifications read) is now
  agent-tools/forge-notify.md. The forge page keeps a short Notifications
  section.
- matrix.md: the retired hive-side matrix code, the evaluation assertions and
  the fluffychat-web build fixes are now networking/matrix-homeserver.md.
- github.md: the Implementation section (how `gh` and `git push`
  authenticate, and the GitHub poller) is now agent-tools/github.md.

docs/README.md and the agent-tools README list the new pages, and AGENTS.md's
note on mixed operator and implementation docs now says to move the detail
and leave a pointer.
This commit is contained in:
iris 2026-11-07 11:49:28 +01:00 • committed by mara
commit 97671e28d5
4 changed files with 504 additions and 466 deletions
Download patch file Download diff file Expand all files Collapse all files

5
docs/4-implementation/agent-tools/README.md
View file

@ -18,6 +18,11 @@ behavior. The operator's own tools are in
- **[forge](forge.md)** — the `hive-forge` Forgejo CLI every agent has
for issues, PRs, and comments. Not an MCP tool — a binary agents
shell out to instead of ad-hoc curl.
- **[forge-notify](forge-notify.md)** — the daemon that wakes an agent on
forge activity: what it polls, how it formats and de-duplicates a
notification, and when it marks one read.
- **[github](github.md)** — how an agent's `gh` and `git push` authenticate
to GitHub, and its GitHub notification poller.
- **[forge-cli](forge-cli.md)** — the exhaustive, autogenerated
flag-by-flag reference for `hive-forge`, kept in lockstep with the
binary by CI the same way `hivectl-cli.md` is.

280
docs/4-implementation/agent-tools/forge-notify.md Normal file
View file

@ -0,0 +1,280 @@
# The forge notification poller
How the daemon that wakes an agent on forge activity works: what it polls, how it formats and de-duplicates a notification, and when it marks one read. The operator-side picture, with the forge's per-agent accounts, is in [the swarm forge](../../2-handbook/swarm/forge.md); the verbs an agent calls are in [`forge.md`](forge.md).
## Notification poller (`hive-forge-notify/src/notify.rs`)
Its own long-running per-agent daemon (`hive-forge-notify`, a sibling of
`hive-bash-daemon` / `hive-matrix-daemon`) — it used
to be a background task inside the `hive-agent` serve loop. Polls
`GET /api/v1/notifications?all=false` every 30 seconds (Forgejo's
unread-only filter), formats each notification as a broker
`Wake { from: "forge" }` message, and delivers it to the agent's own
inbox so claude's normal turn loop picks it up.
The daemon waits about ten minutes for the agent's forge token, then exits 0,
which its `Restart=on-failure` doesn't restart. A path unit
(`hive-forge-notify.path`) starts it again when the fetched token file or the
state-dir `forge-token` changes, so a token minted later still starts the
polling.
The crate builds a second, independent binary for a different host — see
[github.md](github.md#notifications).
The host-specific calls — list unread, mark read, resolve own login —
live behind `Source` in `hive-forge-notify/src/source.rs`;
classification, formatting, dedupe and todo delivery live in `notify.rs`,
common to both binaries. Todo
keys here are the bare thread ids, and must stay that way: renaming them
would orphan every in-flight forge todo on the first restart after a
deploy.
### Mark-read on delivery
On a **successful** broker delivery, `forge_notify` marks the thread
read on forge straight away (`PATCH /notifications/threads/{id}`). The
broker inbox is the durable work queue now — each delivered wake is a
sqlite row with its own ack lifecycle — so the forge unread flag no
longer needs to track whether the agent has _processed_ a
notification. Clearing it on delivery keeps forge's unread set **small
by construction**: at rest it holds only threads that failed to
deliver plus whatever arrived since the last 30s poll.
That size property is the whole point. A container rebuild starts the
poller with no memory of what it delivered, re-scans `?all=false`, and
finds nothing stale — the delivered threads are already read on forge.
Forge's own read-state is thus the durable, cross-rebuild record of
what `forge_notify` has delivered; there is **no persisted cursor**.
**Gotcha:** don't reintroduce a persisted dedup cursor here. A design
that leaves threads unread and tracks delivery via a separately persisted
cursor is fragile — losing that cursor across a rebuild re-delivers the
agent's entire still-unread backlog as a flood of fresh wakes. Forge's
own read-state is the only durable record this design needs.
**Read-before-comment guard doesn't block a fresh wake.** hive-forge's
read-before-comment guard (which keys off forge unread-state) doesn't
force the agent to view a thread before commenting on it: the broker
wake already carries the notification body, so _delivery is the read_.
An agent that wants the full thread still runs `hive-forge comments` /
`view`.
**In-process dedupe (small, ephemeral).** A single-process map (thread
id → last-delivered `updated_at`) guards the narrow window where a
mark-read call _transiently fails_ and the thread reappears unread in
the next poll before its `updated_at` bumps — so a flaky PATCH doesn't
re-fire the wake. it's **not persisted** and resets on restart (forge
read-state covers the durable case). Each poll prunes it to the ids in
the single `limit=UNREAD_FETCH_LIMIT` (50) fetch page, so it can never
exceed that many entries (a debug assertion pins the invariant; the
fetch limit and the bound are the same constant). `forge_notify` leaves
a failed _delivery_ unread and out of the map, so it resurfaces next tick.
`forge_notify` marks self-echo notifications (the agent's own writes, see below)
read directly without a delivery — same `mark_read` call, no wake.
### Activation gates (graceful no-ops)
The poller starts switched off and stays that way for any of:
- `HIVE_FORGE_URL` not set or not parseable as a URL (a hand-run
harness; hive-c0re sets it from `swarm.forge.domain` on every hive).
- no token in `$HIVE_FORGE_TOKEN_FILE` or `<state>/forge-token` (agent has no forge
account — pre-provisioning or destroy-without-purge race).
- Initial client construction fails (the typed `forgejo-api` client
for the API calls, or the plain reqwest client kept for the
best-effort enrichment fetches of server-provided subject URLs;
both unlikely; treated as fatal-to-the-task only).
Switched off = the spawned task returns immediately. All other failure
modes (HTTP errors, parse errors, mark-read failures) are
best-effort: logged at debug/warn and retried next tick.
### Self-notification filtering
Forgejo fires notifications for the agent's own actions (it opened a
PR, posted a comment, submitted a review). Surfacing those would
loop claude on its own writes. `forge_notify` drops the comment/review case
silently (mark-read without delivery):
- **Self-authored comments / reviews** — comment payload's
`user.login` matches `own_login`.
- **Self-authored creations** (an agent opening its own PR/issue) — the
already-fetched subject payload's poster `user.login` matches
`own_login`. `forge_notify` drops only _creations_ this way; a later state
change on the agent's own subject comes from someone else, so it still surfaces.
`forge_notify` fetches `own_login` at startup via `GET /api/v1/user`. On fetch
failure the filter degrades open (no filtering) rather than crashing
the task — a noisy inbox beats a silently stuck poller — but the fetch
is **re-attempted on each poll tick** until it succeeds, so a boot-time
failure (the forge not yet reachable) self-heals instead of leaving
self-echo filtering off for the whole process lifetime.
### Body excerpt + truncation + heading escape
The wake message embeds the comment / review body so the agent sees
actual content without a follow-up fetch — **not** a new issue/PR's own
description, which the wrapper table below deliberately omits. Three
pipeline steps in order:
1. **Truncate** to `BODY_TRUNCATE = 500` chars at a char-boundary;
appends `…` when cut. Truncation happens BEFORE escape so the
mention-overflow diff (next step) compares like-for-like against
the raw body.
2. **Mention overflow extraction** — when truncation actually
trimmed content, walk the full body line-by-line and surface any
`@username` lines that fell outside the embed window. Rendered as
a trailing `mentions (truncated from body):\n > <line>` block.
Mention detection requires the `@` to be at line start or
following a non-username byte, so email-style `foo@bar.com` does
NOT count.
3. **ATX heading escape** — for each line that's a strict
`CommonMark` ATX heading (1-6 leading `#`s followed by a space,
tab, or end-of-line), prepend `\` so the embedded body doesn't
blow into a top-level h1/h2 inside the wrapper message when the
dashboard renders it. Lines like `#tag`, `#123`, `#!/bin/bash` <!-- lint:allow: syntax examples, not real tags -->
**aren't** headings — no escape, no cosmetic noise. Indented
"headings" inside lists / nested quotes keep their leading
whitespace.
The strict ATX rule is deliberate: `\#tag` and `#tag` render
identically, so an over-eager escape just adds visual clutter
without changing behavior. Setext-style headings (`title\n====`)
aren't handled — rarer in practice, would need multi-line
lookahead.
### Wrapper format
Five shapes, distinguished by the notification's classification:
| Trigger | Wrapper |
| ----------------------------------- | ------------------------------------------------------------------------------ |
| Comment on issue / PR | `[comment on PR #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
| Review submission | `[PR approved #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
| New issue / PR | `[new PR #N owner/repo] title\nurl: ...\nassignee: ...` |
| Later activity (open, not creation) | `[activity on PR #N owner/repo] title\nurl: ...\nassignee: ...` |
| State change | `[PR merged #N owner/repo] title\nurl: ...\nassignee: ...` |
The issue/PR's own description is deliberately never embedded here (only a
comment/review body is — that's the actual new content a notification
exists to surface): repeating a subject's own description on every "new"
or "activity on" wake for it's noise the agent already has via the URL,
not context (mara, triage: "just the issue title should be enough").
Review labels come from the Forgejo `state` field: `APPROVED` →
`approved`, `REQUEST_CHANGES` → `changes requested`, `COMMENT` →
`review comment`. `forge_notify` drops `PENDING` (review saved but not
submitted yet — no peer-visible event). Unknown states fall back to
the generic comment wrapper.
A review submitted with **no body** renders `reviewed by: <author>` in
place of the `<author>: <body>` line — deliberately worded to not collide
with the meta-suffix `reviewer:` line (requested reviewers, below).
### Inline review comments
A review's line comments belong to neither the review body nor the
`latest_comment_url`. A review with only inline comments has no such url
and would otherwise land as a bare `activity on PR`. Every PR wake therefore
fetches the PR's reviews (`/pulls/N/reviews`, up to four pages of 50) and,
for the reviews new to the agent, appends an
`inline review comments (read them with `hive-forge pr reviews N`):` block
saying where they sit: one `author: path:line, path, ...` line per review
author (bare `path` when Forgejo dropped the position because the line left
the diff; control characters in a path become spaces). The comment text stays
out of the wake, and the agent reads the full review with the CLI.
A review counts as new when it landed strictly after the thread's previously
delivered `updated_at`. Without a baseline every review counts, and so does
one with no submit time, so nothing disappears. It doesn't count when the
agent wrote it, when it's a `PENDING` draft, or when `comments_count` is 0.
The notifier fetches the newest 5 such reviews and lists at most 10 locations
(`(+N more)` for the rest). Best effort: a failed fetch yields less text and
never drops the wake. Cost: one reviews fetch per PR wake, plus one per new
commented review.
### Merge/close vs a later comment
A notification carrying a `latest_comment_url` normally takes the comment
path. But a merged/closed subject **keeps** its `latest_comment_url` set,
so a just-merged PR that had any prior discussion would route to the
comment path and render `[comment on PR]` (with a stale pre-merge comment
body) instead of `[PR merged]` — the agent never learns its PR merged.
When the notification IS the merge/close transition — its
event time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` of the
subject's `closed_at` (set for both `merged` and `closed`) — the
state-change path wins even with a comment url present
(`state_change_is_current`). A genuine **later** comment on an
already-closed subject bumps `updated_at` well past `closed_at`, so it
stays on the comment path and keeps its comment body. Missing/unparseable
timestamps default to the state-change path, so a merge is never silently
hidden behind a stale comment.
#### Merge racing a comment
The one gap the timestamp cut leaves: a genuine comment posted **within
`NEW_ITEM_TOLERANCE_SECS` of the merge** bumps `updated_at` close enough
to `closed_at` that `state_change_is_current` returns `true` — so it takes
the state-change path, dropping its body. Best of both worlds:
on the merge/close path `forge_notify` fetches the `latest_comment_url` comment and, when
its `created_at` is strictly **after** the subject's `closed_at`
(`comment_is_after_close`) — in other words, it raced the merge rather than being the
pre-merge last comment the subject keeps — append it as a
`comment by <author>: <excerpt>` block before the meta suffix
(`fresh_post_close_comment_tail`). The wake carries **both** `[PR merged]`
and the racing comment. `forge_notify` leaves the kept pre-merge comment (created before
`closed_at`) off, drops a self-authored racing comment (don't
echo the agent's own write), and appends nothing for a missing/unparseable `created_at`/
`closed_at` (conservative — only surface a comment it can
positively place after the close). Cost: one extra comment fetch on
merge/close notifications, acceptable given how rare they are.
### "new" vs "activity on"
A review submitted with **no body** carries no `latest_comment_url`,
so it misses the comment path and lands on the state-change path with
`state == "open"` — exactly like a freshly opened PR. Labeling that
`new PR` is misleading: agents dismiss it as a duplicate of the
original open notification and miss the review. The `open`
state only earns the `new <kind>` label when the notification's event
time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` (120s) of the
subject's `created_at`. `forge_notify` labels anything later `activity on <kind>`
— neutral and non-misleading, since it can't cheaply say _what_ the
activity was without an extra reviews fetch. Missing/unparseable
timestamps default to `new` (preserve prior behavior rather than mask a
genuine new item). A small dependency-free
RFC 3339 helper (`parse_rfc3339`) parses timestamps.
`forge_notify` extracts the number from `subject.html_url`'s last path segment
(strips `#anchor` first), and the repo slug from `repository.full_name`.
Both degrade gracefully when absent (number → blank, repo → blank)
so unexpected Forgejo shapes don't crash the formatter.
### Meta suffix
Every wrapper ends with one or more of:
- `assignee: <list>` — always present; `unassigned` when empty so
the line shape is stable.
- `reviewer: <list>` — PR notifications only, present only when
`requested_reviewers` is non-empty.
### Review-request override
For new PRs, the kind label flips to `[review requested #N
owner/repo]` when `own_login` appears in `requested_reviewers`,
regardless of the Forgejo `reason` field. Forgejo doesn't reliably
set `reason == "review_requested"` (often null instead), so the
fallback checks the subject payload directly. `is_new` gates detection
so the label only fires once on PR creation, not on every
subsequent comment.
### Subscription management
The poller does **not** autounsubscribe from repo watches — it
delivers every unread notification it's handed. A hive-forge CLI
subscription verb bounds the firehose explicitly (dropping broad repo
watches an agent doesn't need) — the poller never guesses which
watches to drop. See the `subscription` verb in
[`docs/4-implementation/agent-tools/forge.md`](forge.md).

86
docs/4-implementation/agent-tools/github.md Normal file
View file

@ -0,0 +1,86 @@
# GitHub accounts: how the agent authenticates
How the agent's `gh` and `git push` authenticate, and how the GitHub notification poller works. What an operator does to give an agent an account is in [GitHub accounts](../../2-handbook/external-accounts/github.md).
## How the agent uses it
When enabled, the container gets:
- **A `gh` wrapper** on `PATH` (shadowing the raw `gh`) that exports
`GH_TOKEN` from the token file at invocation, then execs real `gh`. So
`gh pr create`, `gh api …`, etc. just work — `gh` derives the identity
from the token.
- **A git credential helper** (`git-credential-hive-github`), wired via a
host-scoped `/etc/gitconfig` entry for `https://github.com`, so
`git push https://github.com/<owner>/<repo>` authenticates as
`x-access-token` + the PAT (GitHub ignores the username for PAT auth).
Host-scoped, so it never touches the forge (`localhost:3000`) or any
other remote.
Both scripts read the token from `<state>/github-token` **at invocation
time**, so a PAT written (or rotated) mid-session takes effect immediately
— no container rebuild or restart. Until the file exists, `gh` / `git push`
fail unauthenticated. The build bakes the token path into the scripts
(rather than reading it from an env var), because claude's Bash tool runs in a
minimal environment that wouldn't carry one.
## Notifications
`hive-github-notify` polls github.com for the agent, turning each unread
notification thread into a todo. it's a **separate binary and a
separate systemd unit** from the internal forge's poller
(`hive-forge-notify`, see [the forge notification poller](forge-notify.md)),
installed by `nix/agent-modules/github.nix` under
`services.hyperhive.agent.github.enable`. Both binaries ship from the one
`hive-forge-notify` derivation, so the unit is a second `ExecStart`
path, not a new package.
Two units rather than one daemon with two loops, because it puts the
decision in nix: a hive built without this module has **no github poller
in its closure at all**, which is what makes GitHub access separable
rather than merely switched off. it's also why this isn't a cargo
feature — a feature would unify across the workspace and cost every
crate its build cache.
At runtime the poller needs the PAT above. No PAT, no polling: the unit
logs why and exits 0, which is why it's `Restart = on-failure` and
never `always` — a clean exit on a PAT-less agent must not become a
restart loop. The poller waits about ten minutes for the PAT to appear
before it gives up. A path unit (`hive-github-notify.path`) starts the poller
again when `<state>/github-token` changes, so a PAT that lands later still
enables polling. Once polling, it re-reads the file on every
tick, so it picks up a rotated PAT on the next tick without a restart.
Forgejo models its notifications API on GitHub's, so one tolerant
parse serves both: `id`, `repository.full_name`,
`subject {title,url,latest_comment_url}` and `updated_at` line up field
for field. Lenient deserializers absorb the two real differences —
GitHub sends the thread id as a _string_ where Forgejo
sends a number, and says `PullRequest` where Forgejo says `Pull`.
`hive-github-notify` prefixes todo keys `gh:<id>` so a github thread id can't collide with a
forge one.
Two host differences worth knowing before touching this code:
- **Auth scheme, not just value.** Forgejo takes
`Authorization: token <t>`; GitHub wants `Bearer <t>` plus `Accept:
application/vnd.github+json`, `X-GitHub-Api-Version` and a
`User-Agent`. Sending Forgejo's form to GitHub doesn't error — it
authenticates as _nobody_ and silently drops to the unauthenticated
rate limit. The cheap way to tell the two apart is the rate-limit
header: `x-ratelimit-remaining` near 5000 is an authenticated user,
near 60 is anonymous.
- **GitHub sets the cadence.** It returns `X-Poll-Interval` (60s in
practice, slower than the poller's own tick) and rate-limits callers who
ignore it, so the loop re-arms to the server's interval whenever that's
_slower_ than the poller's. A hint faster than the poller's own tick
isn't a reason to poll harder.
⚠️ **This needs the `notifications` scope on the PAT.** A token minted
for `gh` + `git push` typically carries `repo` only, which is enough to
push and open PRs but **not** to read the notification stream (nor to
mark a thread read, which is the same scope). A PAT without it doesn't
break anything: the poller logs the refusal and stays quiet, and the
agent never gets GitHub wakes. If an agent's GitHub
notifications never arrive, check the token's scopes first — the
symptom is silence, not an error.

95
docs/4-implementation/networking/matrix-homeserver.md Normal file
View file

@ -0,0 +1,95 @@
# The matrix homeserver: implementation notes
What the homeserver's NixOS module does that an operator never sees: the hive-side code it replaced, the evaluation assertions it makes, and the fluffychat-web build fixes. The operator's view of the homeserver is in [the swarm matrix page](../../2-handbook/swarm/matrix.md).
## Retired hive-side matrix code
Earlier versions of hive-c0re created a private "hive" Matrix Space and a
`hive-chat` room, invited every agent to both, kept the hive's sender token
on disk and offered `hivectl matrix invite`. Matrix is swarm-level, so
hive-c0re dropped all of it and no longer talks to the homeserver. The Space, the
room and the files `/var/lib/hyperhive/matrix/{access-token,space-room-id,chat-room-id}`
are harmless leftovers, so delete them by hand if you like.
## Assertion rationale
`config.assertions` in this module fail eval early rather than ship
surprising behaviour:
- **`services.hyperhive.swarm.matrix.gatewayHost != ""`** — same footgun as `forge.domain`:
empty string renders `.<hive>`-shaped garbage in both nginx
`server_name` (treated as wildcard catch-all, surprising) and
`/etc/hosts` (malformed entry). `null` is the right opt-out shape;
a config assertion rejects empty string explicitly.
SSO is unconditional, so the three below are requirements of running a
homeserver at all rather than of a setting:
- **Keep `deploy.matrix.sso.clientSecretFile` set** — it defaults to
`/run/hive-matrix-oidc/<client id>.secret` wherever the homeserver runs,
and `swarm-bao-matrix-oidc` writes the secret there from the swarm secret
store under matrix-ctl's leaf. The path is on the host's tmpfs, so every
reboot fetches the secret again and it never reaches persistent disk. After
a reboot the homeserver doesn't start until the store answers; the
store-retry timer then starts it with no manual step. A path
outside `/run/hive-matrix-oidc/`, or a null, fails at eval, not at boot:
tuwunel reads its identity providers from the config file, so a half-configured
one can stop the homeserver from starting outright rather than merely
hiding a login button.
- **Set `swarm.authelia.url`** — without a provider URL there
is nothing to discover against.
- **Set `swarm.matrix.gatewayHost != null`** — the SSO callback URL is
format-locked to `<homeserver>/_matrix/client/unstable/login/sso/callback/<client_id>`,
and the identity provider needs a public name to redirect the
browser to.
`server_name`'s own bogus-value guard lives in `hive-network.nix`
(`services.hyperhive.domain != null`), not here — see
[`docs/4-implementation/networking/network.md`](network.md).
## fluffychat-web build fixes
`pkgs.fluffychat-web` ships from `flutter341.buildFlutterApplication`,
which has two upstream gaps for fluffychat's web target:
- The dart web-worker entry point (`web/native_executor.dart`) isn't
compiled — `buildFlutterApplication` only runs `flutter build web`
on the main entry.
- `native_imaging`'s C source isn't built — emscripten isn't a
flutter-builder native build input.
Both fixed in `nix/host-modules/hive-matrix.nix` via two derivations:
- **`fluffychat-web-imaging`** builds `Imaging.{js,wasm}` from the
`native_imaging` C source via `pkgs.emscripten`. Source comes
from `pkgs.fluffychat-web.passthru.pubspecLock.dependencySources.native_imaging`
— already in the build closure of the flutter app, so no parallel
hash pin and version autosyncs with nixpkgs bumps. Build closure
is ~3.6 GiB (emscripten LLVM); runtime closure is just the two
output files. `dontConfigure = true` because cmake runs inside
`js/Makefile` via `emcmake cmake`, not at the package root. The
build script needs `HOME` + `EM_CACHE` writable for emscripten's
on-demand sysroot build (libc, libc++ → wasm).
- **`fluffychat-web-fixed`** is `pkgs.fluffychat-web` plus a
`postInstall` patch that (a) compiles `web/native_executor.dart`
via `dart compile js` (dart from the flutter341 closure, no
incremental cost) and (b) installs `fluffychat-web-imaging`'s
outputs into `$out`.
Two subtle details worth knowing before touching either derivation:
- **`make -C js`** instead of `cd js; make` — keeps the build-phase
pwd at the source root so `installPhase` doesn't have to know
about the cd. Robust against future reorders / `dontBuild`.
- **`web/native_executor.dart`** as a build-CWD-relative path,
_not_ `$src/web/...` — `dart`'s `package_config.json` walk-up
needs to hit `buildFlutterApplication`'s pub-get output
(`.dart_tool/` in the build CWD). Walking up from a read-only
`$src/` store path finds no `.dart_tool/` and errors with
"Couldn't resolve the package 'matrix'."
Drop both derivations when nixpkgs's flutter builder grows worker
- emcc support upstream.
Mount point is `chat.<swarm-domain>/` (the `gatewayHost` vhost);
upstream `--base-href "/"` is correct at sub-domain root, no override.