docs: sort the top-level directories into five sections by what the reader wants
Every docs directory moves under 1-getting-started, 2-handbook, 4-implementation or 5-project; the options reference (section 3) comes from the website and has no directory here. trust-boundary stays whole under the handbook for now. web-ui, swarm and tools split between handbook and implementation, and the command-line references move to 2-handbook/reference. Links, path references in code and nix, and the docs index follow the moves; no page content changes.
This commit is contained in:
iris
mara
parent
53ccbcb4b3
commit
6739bb903d
1 changed files with 736 additions and 701 deletions
|
|
@ -1,395 +0,0 @@
|
|||
# Swarm forge (Forgejo)
|
||||
|
||||
Private Forgejo instance running in a nixos-container (`hive-forge`, the
|
||||
container and CLI name), used as the
|
||||
swarm's persistent code-collaboration surface (issues, PRs, reviews,
|
||||
attachments). Configured via `services.hyperhive.swarm.forge.*` (what the
|
||||
forge IS to every hive — names, ports, URLs) plus
|
||||
`services.hyperhive.deploy.forgejo.*` (what the host running it decides —
|
||||
which build it runs, how it's served, what it mirrors, where its
|
||||
host-local secrets sit).
|
||||
Container
|
||||
shape, ROOT_URL / sub-domain routing, and operator-vs-in-cluster URL
|
||||
handling live in [`docs/networking/gateway.md`](../networking/gateway.md); this file owns the
|
||||
per-agent integration story and the notification pump that wakes
|
||||
each agent on relevant activity.
|
||||
|
||||
## Token scopes
|
||||
|
||||
Two scope sets:
|
||||
|
||||
**`AGENT_TOKEN_SCOPES`** (swarm-controller's `forge::agent_token`, pinned
|
||||
by a test). swarm-controller mints every agent token with it:
|
||||
|
||||
| Scope | Why |
|
||||
| -------------------- | ------------------------------------------------------------------------------------------------------- |
|
||||
| `write:repository` | Create, clone, push, delete repos; merge PRs. |
|
||||
| `write:issue` | Open / comment / review issues **and** pull requests (Forgejo namespaces PR conversation under issues). |
|
||||
| `write:user` | Edit own profile, create repos under own user. |
|
||||
| `write:organization` | Create + manage orgs (lets agents share a forge namespace). |
|
||||
| `read:user` | Token-owner endpoint used for self-identification at harness startup. |
|
||||
| `write:misc` | Hooks, attachments, the rest of the long tail. |
|
||||
| `read:notification` | Poll `GET /notifications` for unread events. |
|
||||
| `write:notification` | Mark notifications read via `PATCH /notifications/threads/{id}`. |
|
||||
|
||||
**`HIVE_TOKEN_SCOPES`** (swarm-controller's `forge::hive_token`, pinned by a
|
||||
test): `read:repository` alone, for each hive's read-only account (next
|
||||
section).
|
||||
|
||||
---
|
||||
|
||||
## How a hive reads the forge
|
||||
|
||||
Each hive in swarm-controller's hive directory has its own forge account,
|
||||
`hive-<hive>`. swarm-controller creates it, makes it a member of the `hives`
|
||||
team in `agent-configs`, and mints one token named `swarm-hive` into the
|
||||
swarm secret store at `swarm/hives/<hive>/forge/token`. The team has read
|
||||
permission on `repo.code` and `repo.pulls` of every repo in the org,
|
||||
including repos created later. A pass at start and every five minutes
|
||||
creates what's missing and re-mints a token that's missing or no longer
|
||||
matches the forge. A pass whose store or forge reads fail queues nothing, so
|
||||
an outage never rotates every hive's token.
|
||||
|
||||
swarm-controller refuses to create an agent whose name starts with `hive-`:
|
||||
these accounts own that prefix. An agent that already had such a
|
||||
name keeps it, so the accounts are also told apart by email. swarm-controller
|
||||
creates the hive's account with the email `hive-<hive>@hives.hyperhive.local`
|
||||
and refuses an account by that name with any other email: it never adds an
|
||||
agent's account to the team or mints a token for it. An agent's own forge
|
||||
account setup refuses the hive's account the same way.
|
||||
|
||||
hive-c0re reads the token with `forge::reader_token()`, under the hive's own
|
||||
store certificate, every time it reads the forge. A hive whose store holds
|
||||
none reads nothing. These reads go through it, and none of them needs a local
|
||||
forge container:
|
||||
|
||||
- cloning and pulling `/knowledge`
|
||||
|
||||
The hive builds nothing, so no nix process reads the forge as the hive's
|
||||
account. hive-c0re hands the token to each `git` command it runs
|
||||
(`http.extraHeader`) and keeps no copy on disk. A hive whose store holds no token
|
||||
reads nothing private from the forge. This also holds on a hive with no forge
|
||||
container.
|
||||
|
||||
Roll a change to any of this out store host first (its granter writes the
|
||||
controller's grant on `swarm/hives/+/forge/token`), then the controller host,
|
||||
then the hives. A hive that starts before its token exists reads nothing, and
|
||||
builds nothing, until the controller's next pass.
|
||||
|
||||
---
|
||||
|
||||
## Per-agent forge accounts
|
||||
|
||||
Each agent gets its own Forgejo user and access token. swarm-controller
|
||||
creates the user when it creates the agent, and mints one token named
|
||||
`swarm-agent` with the admin API into the swarm secret store at
|
||||
`swarm/agents/<agent>/forge-token`. A pass at start and every five minutes
|
||||
covers every agent with a store identity: it creates a missing forge user and
|
||||
re-mints any token that's missing or no longer matches the forge;
|
||||
a rotation deletes the old `swarm-agent` token first, so each agent holds at
|
||||
most one. The agent fetches the token under its own store certificate into
|
||||
`/run/hive-agent-forge-token/token` (`nix/agent-modules/forge-token.nix`),
|
||||
and `hive-forge`, the git credential helper, the `forge_notify` poller and
|
||||
the avatar sync read it from there, falling back to `<state>/forge-token`,
|
||||
the file hive-c0re wrote before. hive-c0re no longer creates agent users or
|
||||
mints agent tokens. swarm-controller deletes the `hyperhive-<unix-seconds>`
|
||||
tokens it left behind, for each agent whose current `swarm-agent` token is
|
||||
live; tokens of other names are never touched.
|
||||
|
||||
Two things live in the `agent-configs` Forgejo organization:
|
||||
|
||||
- A config repo per agent (`agent-configs/<name>`). The
|
||||
agent is a **write collaborator on its own** repo — it can push
|
||||
config-change branches and open config PRs — but `main` is
|
||||
branch-protected by swarm-controller: the merge and approval
|
||||
allowlists are the `operators` team, and the agent can neither push
|
||||
`main` directly nor self-merge. An operator's merge in the Forgejo UI
|
||||
deploys the merged commit (see
|
||||
[config-changes.md § Config changes](../agent-lifecycle/config-changes.md#config-changes)).
|
||||
hive-c0re pushes nothing to this org and reads nothing from it.
|
||||
Repos stay private, so an agent can't read another
|
||||
agent's config. The agent's **persistent meta flake input** is this repo's `main` head,
|
||||
pinned by sha in `internal/meta` (`agent-<n>.url =
|
||||
git+<forge>/agent-configs/<n>.git?rev=<sha>`; see
|
||||
[config-changes.md § Meta flake](../agent-lifecycle/config-changes.md#meta-flake)).
|
||||
|
||||
The `hive-forge` CLI (separate workspace crate, see
|
||||
[`README.md`](../README.md) file map) wraps the Forgejo REST API
|
||||
with the per-agent token; agents call it for issue / PR / comment
|
||||
ops as if it were a peer. All REST calls across the workspace
|
||||
(`hive-forge` verbs, hive-c0re provisioning, this poller) go through
|
||||
the typed `forgejo-api` crate; only non-`/api/v1` web-router routes
|
||||
(attachment / artifact downloads, log streaming) and the poller's
|
||||
enrichment fetches of server-provided subject URLs stay on raw
|
||||
reqwest.
|
||||
|
||||
## Notification poller (`hive-forge-notify/src/notify.rs`)
|
||||
|
||||
Its own long-running per-agent daemon (`hive-forge-notify`, a sibling of
|
||||
`hive-bash-daemon` / `hive-matrix-daemon`) — it used
|
||||
to be a background task inside the `hive-agent` serve loop. Polls
|
||||
`GET /api/v1/notifications?all=false` every 30 seconds (Forgejo's
|
||||
unread-only filter), formats each notification as a broker
|
||||
`Wake { from: "forge" }` message, and delivers it to the agent's own
|
||||
inbox so claude's normal turn loop picks it up.
|
||||
|
||||
The crate builds a second, independent binary for a different host — see
|
||||
[github.md](github.md#notifications).
|
||||
|
||||
The host-specific calls — list unread, mark read, resolve own login —
|
||||
live behind `Source` in `hive-forge-notify/src/source.rs`;
|
||||
classification, formatting, dedupe and todo delivery live in `notify.rs`,
|
||||
common to both binaries. Todo
|
||||
keys here are the bare thread ids, and must stay that way: renaming them
|
||||
would orphan every in-flight forge todo on the first restart after a
|
||||
deploy.
|
||||
|
||||
### Mark-read on delivery
|
||||
|
||||
On a **successful** broker delivery, `forge_notify` marks the thread
|
||||
read on forge straight away (`PATCH /notifications/threads/{id}`). The
|
||||
broker inbox is the durable work queue now — each delivered wake is a
|
||||
sqlite row with its own ack lifecycle — so the forge unread flag no
|
||||
longer needs to track whether the agent has _processed_ a
|
||||
notification. Clearing it on delivery keeps forge's unread set **small
|
||||
by construction**: at rest it holds only threads that failed to
|
||||
deliver plus whatever arrived since the last 30s poll.
|
||||
|
||||
That size property is the whole point. A container rebuild starts the
|
||||
poller with no memory of what it delivered, re-scans `?all=false`, and
|
||||
finds nothing stale — the delivered threads are already read on forge.
|
||||
Forge's own read-state is thus the durable, cross-rebuild record of
|
||||
what `forge_notify` has delivered; there is **no persisted cursor**.
|
||||
|
||||
**Gotcha:** don't reintroduce a persisted dedup cursor here. A design
|
||||
that leaves threads unread and tracks delivery via a separately persisted
|
||||
cursor is fragile — losing that cursor across a rebuild re-delivers the
|
||||
agent's entire still-unread backlog as a flood of fresh wakes. Forge's
|
||||
own read-state is the only durable record this design needs.
|
||||
|
||||
**Read-before-comment guard doesn't block a fresh wake.** hive-forge's
|
||||
read-before-comment guard (which keys off forge unread-state) doesn't
|
||||
force the agent to view a thread before commenting on it: the broker
|
||||
wake already carries the notification body, so _delivery is the read_.
|
||||
An agent that wants the full thread still runs `hive-forge comments` /
|
||||
`view`.
|
||||
|
||||
**In-process dedupe (small, ephemeral).** A single-process map (thread
|
||||
id → last-delivered `updated_at`) guards the narrow window where a
|
||||
mark-read call _transiently fails_ and the thread reappears unread in
|
||||
the next poll before its `updated_at` bumps — so a flaky PATCH doesn't
|
||||
re-fire the wake. it's **not persisted** and resets on restart (forge
|
||||
read-state covers the durable case). Each poll prunes it to the ids in
|
||||
the single `limit=UNREAD_FETCH_LIMIT` (50) fetch page, so it can never
|
||||
exceed that many entries (a debug assertion pins the invariant; the
|
||||
fetch limit and the bound are the same constant). `forge_notify` leaves
|
||||
a failed _delivery_ unread and out of the map, so it resurfaces next tick.
|
||||
|
||||
`forge_notify` marks self-echo notifications (the agent's own writes, see below)
|
||||
read directly without a delivery — same `mark_read` call, no wake.
|
||||
|
||||
### Activation gates (graceful no-ops)
|
||||
|
||||
The poller starts switched off and stays that way for any of:
|
||||
|
||||
- `HIVE_FORGE_URL` not set or not parseable as a URL (a hand-run
|
||||
harness; hive-c0re sets it from `swarm.forge.domain` on every hive).
|
||||
- no token in `$HIVE_FORGE_TOKEN_FILE` or `<state>/forge-token` (agent has no forge
|
||||
account — pre-provisioning or destroy-without-purge race).
|
||||
- Initial client construction fails (the typed `forgejo-api` client
|
||||
for the API calls, or the plain reqwest client kept for the
|
||||
best-effort enrichment fetches of server-provided subject URLs;
|
||||
both unlikely; treated as fatal-to-the-task only).
|
||||
|
||||
Switched off = the spawned task returns immediately. All other failure
|
||||
modes (HTTP errors, parse errors, mark-read failures) are
|
||||
best-effort: logged at debug/warn and retried next tick.
|
||||
|
||||
### Self-notification filtering
|
||||
|
||||
Forgejo fires notifications for the agent's own actions (it opened a
|
||||
PR, posted a comment, submitted a review). Surfacing those would
|
||||
loop claude on its own writes. `forge_notify` drops the comment/review case
|
||||
silently (mark-read without delivery):
|
||||
|
||||
- **Self-authored comments / reviews** — comment payload's
|
||||
`user.login` matches `own_login`.
|
||||
- **Self-authored creations** (an agent opening its own PR/issue) — the
|
||||
already-fetched subject payload's poster `user.login` matches
|
||||
`own_login`. `forge_notify` drops only _creations_ this way; a later state
|
||||
change on the agent's own subject comes from someone else, so it still surfaces.
|
||||
|
||||
`forge_notify` fetches `own_login` at startup via `GET /api/v1/user`. On fetch
|
||||
failure the filter degrades open (no filtering) rather than crashing
|
||||
the task — a noisy inbox beats a silently stuck poller — but the fetch
|
||||
is **re-attempted on each poll tick** until it succeeds, so a boot-time
|
||||
failure (the forge not yet reachable) self-heals instead of leaving
|
||||
self-echo filtering off for the whole process lifetime.
|
||||
|
||||
### Body excerpt + truncation + heading escape
|
||||
|
||||
The wake message embeds the comment / review body so the agent sees
|
||||
actual content without a follow-up fetch — **not** a new issue/PR's own
|
||||
description, which the wrapper table below deliberately omits. Three
|
||||
pipeline steps in order:
|
||||
|
||||
1. **Truncate** to `BODY_TRUNCATE = 500` chars at a char-boundary;
|
||||
appends `…` when cut. Truncation happens BEFORE escape so the
|
||||
mention-overflow diff (next step) compares like-for-like against
|
||||
the raw body.
|
||||
2. **Mention overflow extraction** — when truncation actually
|
||||
trimmed content, walk the full body line-by-line and surface any
|
||||
`@username` lines that fell outside the embed window. Rendered as
|
||||
a trailing `mentions (truncated from body):\n > <line>` block.
|
||||
Mention detection requires the `@` to be at line start or
|
||||
following a non-username byte, so email-style `foo@bar.com` does
|
||||
NOT count.
|
||||
3. **ATX heading escape** — for each line that's a strict
|
||||
`CommonMark` ATX heading (1-6 leading `#`s followed by a space,
|
||||
tab, or end-of-line), prepend `\` so the embedded body doesn't
|
||||
blow into a top-level h1/h2 inside the wrapper message when the
|
||||
dashboard renders it. Lines like `#tag`, `#123`, `#!/bin/bash` <!-- lint:allow: syntax examples, not real tags -->
|
||||
**aren't** headings — no escape, no cosmetic noise. Indented
|
||||
"headings" inside lists / nested quotes keep their leading
|
||||
whitespace.
|
||||
|
||||
The strict ATX rule is deliberate: `\#tag` and `#tag` render
|
||||
identically, so an over-eager escape just adds visual clutter
|
||||
without changing behavior. Setext-style headings (`title\n====`)
|
||||
aren't handled — rarer in practice, would need multi-line
|
||||
lookahead.
|
||||
|
||||
### Wrapper format
|
||||
|
||||
Five shapes, distinguished by the notification's classification:
|
||||
|
||||
| Trigger | Wrapper |
|
||||
| ----------------------------------- | ------------------------------------------------------------------------------ |
|
||||
| Comment on issue / PR | `[comment on PR #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
|
||||
| Review submission | `[PR approved #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
|
||||
| New issue / PR | `[new PR #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
| Later activity (open, not creation) | `[activity on PR #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
| State change | `[PR merged #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
|
||||
The issue/PR's own description is deliberately never embedded here (only a
|
||||
comment/review body is — that's the actual new content a notification
|
||||
exists to surface): repeating a subject's own description on every "new"
|
||||
or "activity on" wake for it's noise the agent already has via the URL,
|
||||
not context (mara, triage: "just the issue title should be enough").
|
||||
|
||||
Review labels come from the Forgejo `state` field: `APPROVED` →
|
||||
`approved`, `REQUEST_CHANGES` → `changes requested`, `COMMENT` →
|
||||
`review comment`. `forge_notify` drops `PENDING` (review saved but not
|
||||
submitted yet — no peer-visible event). Unknown states fall back to
|
||||
the generic comment wrapper.
|
||||
|
||||
A review submitted with **no body** renders `reviewed by: <author>` in
|
||||
place of the `<author>: <body>` line — deliberately worded to not collide
|
||||
with the meta-suffix `reviewer:` line (requested reviewers, below).
|
||||
|
||||
### Inline review comments
|
||||
|
||||
A review's line comments belong to neither the review body nor the
|
||||
`latest_comment_url`. A review with only inline comments has no such url
|
||||
and would otherwise land as a bare `activity on PR`. Every PR wake therefore
|
||||
fetches the PR's reviews (`/pulls/N/reviews`, up to four pages of 50) and,
|
||||
for the reviews new to the agent, appends an
|
||||
`inline review comments (read them with `hive-forge pr reviews N`):` block
|
||||
saying where they sit: one `author: path:line, path, ...` line per review
|
||||
author (bare `path` when Forgejo dropped the position because the line left
|
||||
the diff; control characters in a path become spaces). The comment text stays
|
||||
out of the wake, and the agent reads the full review with the CLI.
|
||||
|
||||
A review counts as new when it landed strictly after the thread's previously
|
||||
delivered `updated_at`. Without a baseline every review counts, and so does
|
||||
one with no submit time, so nothing disappears. It doesn't count when the
|
||||
agent wrote it, when it's a `PENDING` draft, or when `comments_count` is 0.
|
||||
The notifier fetches the newest 5 such reviews and lists at most 10 locations
|
||||
(`(+N more)` for the rest). Best effort: a failed fetch yields less text and
|
||||
never drops the wake. Cost: one reviews fetch per PR wake, plus one per new
|
||||
commented review.
|
||||
|
||||
### Merge/close vs a later comment
|
||||
|
||||
A notification carrying a `latest_comment_url` normally takes the comment
|
||||
path. But a merged/closed subject **keeps** its `latest_comment_url` set,
|
||||
so a just-merged PR that had any prior discussion would route to the
|
||||
comment path and render `[comment on PR]` (with a stale pre-merge comment
|
||||
body) instead of `[PR merged]` — the agent never learns its PR merged.
|
||||
When the notification IS the merge/close transition — its
|
||||
event time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` of the
|
||||
subject's `closed_at` (set for both `merged` and `closed`) — the
|
||||
state-change path wins even with a comment url present
|
||||
(`state_change_is_current`). A genuine **later** comment on an
|
||||
already-closed subject bumps `updated_at` well past `closed_at`, so it
|
||||
stays on the comment path and keeps its comment body. Missing/unparseable
|
||||
timestamps default to the state-change path, so a merge is never silently
|
||||
hidden behind a stale comment.
|
||||
|
||||
#### Merge racing a comment
|
||||
|
||||
The one gap the timestamp cut leaves: a genuine comment posted **within
|
||||
`NEW_ITEM_TOLERANCE_SECS` of the merge** bumps `updated_at` close enough
|
||||
to `closed_at` that `state_change_is_current` returns `true` — so it takes
|
||||
the state-change path, dropping its body. Best of both worlds:
|
||||
on the merge/close path `forge_notify` fetches the `latest_comment_url` comment and, when
|
||||
its `created_at` is strictly **after** the subject's `closed_at`
|
||||
(`comment_is_after_close`) — in other words, it raced the merge rather than being the
|
||||
pre-merge last comment the subject keeps — append it as a
|
||||
`comment by <author>: <excerpt>` block before the meta suffix
|
||||
(`fresh_post_close_comment_tail`). The wake carries **both** `[PR merged]`
|
||||
and the racing comment. `forge_notify` leaves the kept pre-merge comment (created before
|
||||
`closed_at`) off, drops a self-authored racing comment (don't
|
||||
echo the agent's own write), and appends nothing for a missing/unparseable `created_at`/
|
||||
`closed_at` (conservative — only surface a comment it can
|
||||
positively place after the close). Cost: one extra comment fetch on
|
||||
merge/close notifications, acceptable given how rare they are.
|
||||
|
||||
### "new" vs "activity on"
|
||||
|
||||
A review submitted with **no body** carries no `latest_comment_url`,
|
||||
so it misses the comment path and lands on the state-change path with
|
||||
`state == "open"` — exactly like a freshly opened PR. Labeling that
|
||||
`new PR` is misleading: agents dismiss it as a duplicate of the
|
||||
original open notification and miss the review. The `open`
|
||||
state only earns the `new <kind>` label when the notification's event
|
||||
time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` (120s) of the
|
||||
subject's `created_at`. `forge_notify` labels anything later `activity on <kind>`
|
||||
— neutral and non-misleading, since it can't cheaply say _what_ the
|
||||
activity was without an extra reviews fetch. Missing/unparseable
|
||||
timestamps default to `new` (preserve prior behavior rather than mask a
|
||||
genuine new item). A small dependency-free
|
||||
RFC 3339 helper (`parse_rfc3339`) parses timestamps.
|
||||
|
||||
`forge_notify` extracts the number from `subject.html_url`'s last path segment
|
||||
(strips `#anchor` first), and the repo slug from `repository.full_name`.
|
||||
Both degrade gracefully when absent (number → blank, repo → blank)
|
||||
so unexpected Forgejo shapes don't crash the formatter.
|
||||
|
||||
### Meta suffix
|
||||
|
||||
Every wrapper ends with one or more of:
|
||||
|
||||
- `assignee: <list>` — always present; `unassigned` when empty so
|
||||
the line shape is stable.
|
||||
- `reviewer: <list>` — PR notifications only, present only when
|
||||
`requested_reviewers` is non-empty.
|
||||
|
||||
### Review-request override
|
||||
|
||||
For new PRs, the kind label flips to `[review requested #N
|
||||
owner/repo]` when `own_login` appears in `requested_reviewers`,
|
||||
regardless of the Forgejo `reason` field. Forgejo doesn't reliably
|
||||
set `reason == "review_requested"` (often null instead), so the
|
||||
fallback checks the subject payload directly. `is_new` gates detection
|
||||
so the label only fires once on PR creation, not on every
|
||||
subsequent comment.
|
||||
|
||||
### Subscription management
|
||||
|
||||
The poller does **not** autounsubscribe from repo watches — it
|
||||
delivers every unread notification it's handed. A hive-forge CLI
|
||||
subscription verb bounds the firehose explicitly (dropping broad repo
|
||||
watches an agent doesn't need) — the poller never guesses which
|
||||
watches to drop. See the `subscription` verb in
|
||||
[`docs/tools/forge.md`](../tools/forge.md).
|
||||
Loading…
Reference in a new issue