Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
cc2503d9c7 | ||
|
|
36b67fcda3 | ||
|
|
39b95c2ede | ||
|
|
5d24bedd60 |
205 changed files with 10075 additions and 6049 deletions
|
|
@ -1,7 +1,11 @@
|
|||
# Auto-generated by `hivectl markdown-docs` — do not format.
|
||||
# The hivectl-docs CI check diffs this against fresh binary output,
|
||||
# so reformatting it would break that check.
|
||||
# Generated by `<tool> markdown-docs` — do not format. The matching
|
||||
# *-docs-fresh check in nix/checks.nix diffs each against fresh binary
|
||||
# output, so any reformatting fails that check permanently: the
|
||||
# generator, not the formatter, owns these files' layout. Every CLI
|
||||
# reference doc belongs here — adding one and forgetting this line is
|
||||
# what made swarmctl's fail.
|
||||
docs/tools/hivectl-cli.md
|
||||
docs/tools/swarmctl-cli.md
|
||||
|
||||
# Files with multi-line list-item continuations that prettier strips to col 0.
|
||||
# prettier's `proseWrap: "preserve"` prevents prose reflow but not list-item
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ claude --name <memorable> --model <cheaper-than-you> --dangerously-skip-permissi
|
|||
by name for follow-ups.
|
||||
- **`--model <cheaper-than-you>`** - think about which model the task
|
||||
actually needs; don't spend a bigger model's tokens than your own on
|
||||
mechanical work a cheaper one handles fine. Never use a *bigger* model
|
||||
mechanical work a cheaper one handles fine. Never use a _bigger_ model
|
||||
than yourself for a sub-agent - if the task needs that much capability,
|
||||
it's not the "mechanical batch" case this skill is for.
|
||||
- **`--dangerously-skip-permissions`** - required in headless (`-p`)
|
||||
|
|
@ -68,7 +68,7 @@ work while the sub-agent grinds.
|
|||
cases: "leave it as-is where genuinely unclear; do not guess."
|
||||
- **Ask for a report file** - per-item results + anything skipped and
|
||||
why, so you can verify without re-deriving.
|
||||
- **Tune on ONE item first**, eyeball the result, fix the prompt, *then*
|
||||
- **Tune on ONE item first**, eyeball the result, fix the prompt, _then_
|
||||
turn it loose on the full batch. A prompt bug replicated across 100
|
||||
items is 100 cleanups.
|
||||
|
||||
|
|
@ -98,7 +98,7 @@ running at once instead of handing the whole thing to one.
|
|||
loop.
|
||||
- **Mind the container's memory cap before picking N.** Your whole
|
||||
container shares one `MemoryMax` (a few GB by default) with every
|
||||
subagent you spawn *and* your own process. A `claude` process plus its
|
||||
subagent you spawn _and_ your own process. A `claude` process plus its
|
||||
MCP servers can hold several hundred MB to ~1GB depending on the task;
|
||||
spawning a dozen at once on a small container doesn't just slow
|
||||
things down, it can OOM the whole container — taking your own
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ steps in the right order.
|
|||
## 1. Check for a duplicate before filing
|
||||
|
||||
`hive-forge list` has a server-side full-text `--search` flag that covers
|
||||
title *and* body - use it before writing anything:
|
||||
title _and_ body - use it before writing anything:
|
||||
|
||||
```sh
|
||||
hive-forge list --search "keyword or two" --state all
|
||||
|
|
@ -43,8 +43,8 @@ Prefer naming the broken behavior or missing capability over a specific
|
|||
error string or one-off symptom - "hive-forge list has no way to filter by
|
||||
milestone" beats "list command confusing" or "can't find issues for
|
||||
milestone 11 easily". If you're not sure which framing is more durable,
|
||||
lean toward whatever a colleague hitting the *same underlying gap* through
|
||||
a *different* symptom would still recognize.
|
||||
lean toward whatever a colleague hitting the _same underlying gap_ through
|
||||
a _different_ symptom would still recognize.
|
||||
|
||||
## 3. Body: what's wrong, not how to fix it
|
||||
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ dimension directly.
|
|||
hardcoded, so check `repo-labels` first if you don't already know the
|
||||
scopes in use.
|
||||
- **PRs with no formally requested reviewer** - `hive-forge lint
|
||||
no-reviewer` flags PRs with zero requested reviewers at all; add
|
||||
no-reviewer` flags PRs with zero requested reviewers at all; add
|
||||
`--reviewer <name>` to instead flag PRs where that specific person
|
||||
isn't among the requested reviewers. This checks the forge's actual
|
||||
reviewer-request state, not a text `@name` mention anywhere in the
|
||||
|
|
@ -42,12 +42,13 @@ dimension directly.
|
|||
|
||||
Run the checks relevant to what you're verifying rather than assuming
|
||||
one covers everything - "is triage caught up" usually means unassigned
|
||||
+ unlabeled at minimum, with no-reviewer added if reviews matter for
|
||||
your workflow. Each verb supports `--json` if you want to fold the
|
||||
results into something else instead of reading the human-readable output.
|
||||
|
||||
- unlabeled at minimum, with no-reviewer added if reviews matter for
|
||||
your workflow. Each verb supports `--json` if you want to fold the
|
||||
results into something else instead of reading the human-readable output.
|
||||
|
||||
This is reactive/on-demand coverage checking, not a scheduled rollup or
|
||||
historical trend - if you need to track compliance *over time* (was
|
||||
historical trend - if you need to track compliance _over time_ (was
|
||||
triage worse last week than this week), that's a different, bigger
|
||||
ask than running these checks - don't build that unprompted just
|
||||
because the checks exist.
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ description: Take a real screenshot of an HTML/CSS page from inside a container
|
|||
# Headless Screenshot
|
||||
|
||||
"No browser in this container" usually means no browser is
|
||||
*pre-installed*, not that one is unreachable. `nixpkgs#chromium` is a
|
||||
_pre-installed_, not that one is unreachable. `nixpkgs#chromium` is a
|
||||
`nix shell` away, and a single headless invocation renders a page and
|
||||
writes a PNG without a running server, a display, or any scripted
|
||||
interaction library.
|
||||
|
|
@ -65,7 +65,7 @@ peer agent's GUI-testing capability, try rendering it yourself first:
|
|||
This is a static render, not a browser automation tool. Verifying an
|
||||
interaction (a click, a hover state, a form submission) needs
|
||||
puppeteer/playwright or similar - reach for those only when the
|
||||
question is genuinely about *behavior*, not appearance. A missing
|
||||
question is genuinely about _behavior_, not appearance. A missing
|
||||
emoji/glyph rendering as a box in the screenshot is usually a font
|
||||
availability artifact of the headless environment, not a real bug -
|
||||
don't mistake one for the other when reporting results.
|
||||
|
|
@ -80,7 +80,7 @@ judge icon/glyph layout, not just to note "there'd be an icon here").
|
|||
- **Tofu boxes instead of emoji.** Plain `nixpkgs#chromium` has no
|
||||
color-emoji font, so any 🖥/📡/🔨-style glyph renders as an empty
|
||||
box. Add `nixpkgs#noto-fonts-color-emoji` to the `nix shell` - but a
|
||||
`FONTCONFIG_FILE` that lists *only* the emoji dir throws away the
|
||||
`FONTCONFIG_FILE` that lists _only_ the emoji dir throws away the
|
||||
base fonts and blanks all normal text instead, which is its own
|
||||
false alarm. Merge, don't override:
|
||||
```
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ work in your own style. That framing should hold throughout.
|
|||
|
||||
## Before you start
|
||||
|
||||
1. **Read the linked issue/task** - understand *why* the change exists,
|
||||
1. **Read the linked issue/task** - understand _why_ the change exists,
|
||||
not just what it does.
|
||||
2. **Check CI status first.** A failing build or test suite is an
|
||||
automatic block - don't write a detailed review of code that doesn't
|
||||
|
|
|
|||
|
|
@ -28,7 +28,7 @@ requester-as-parent edge written at its `init_config` approval (so
|
|||
agent, including the bootstrap container (`ruth`) — it's just another
|
||||
root. The manager is reparentable like any other agent; there's no
|
||||
"structurally root" carve-out. Its privileges live on its MCP socket,
|
||||
not its tree position (see *Manager special-casing today* below).
|
||||
not its tree position (see _Manager special-casing today_ below).
|
||||
|
||||
### Reparenting
|
||||
|
||||
|
|
@ -88,15 +88,15 @@ umount-old / mount-new / restart-cascade step.
|
|||
|
||||
## Planned topology semantics (once ancestor-based enforcement lands)
|
||||
|
||||
| operation | who can do it |
|
||||
| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
|
||||
| `kill` / `start` / `restart` / `update` (any descendant) | any ancestor |
|
||||
| `request_init_config` (spawn a new child) | any agent, child added under self |
|
||||
| config change via forge PR (any descendant's config) | any ancestor |
|
||||
| `get_logs` (any descendant) | any ancestor |
|
||||
| moderate reminders (cancel any open thread of a descendant) | any ancestor |
|
||||
| `send` / `recv` routing | parent ↔ same-parent siblings ↔ self ↔ descendants; explicit allow-list for anyone else |
|
||||
| `request_update_meta_inputs` (bump meta lock) | root agents only (today: just `manager`) |
|
||||
| operation | who can do it |
|
||||
| ----------------------------------------------------------- | --------------------------------------------------------------------------------------- |
|
||||
| `kill` / `start` / `restart` / `update` (any descendant) | any ancestor |
|
||||
| `request_init_config` (spawn a new child) | any agent, child added under self |
|
||||
| config change via forge PR (any descendant's config) | any ancestor |
|
||||
| `get_logs` (any descendant) | any ancestor |
|
||||
| moderate reminders (cancel any open thread of a descendant) | any ancestor |
|
||||
| `send` / `recv` routing | parent ↔ same-parent siblings ↔ self ↔ descendants; explicit allow-list for anyone else |
|
||||
| `request_update_meta_inputs` (bump meta lock) | root agents only (today: just `manager`) |
|
||||
|
||||
"Ancestor" walks `ContainerView.parent` chains; cycles are guarded by a
|
||||
visited-set at dispatch time (a malformed `topology.json` can't lock
|
||||
|
|
@ -138,11 +138,11 @@ other agents don't:
|
|||
capability.
|
||||
- **Prompt/tools** — the system prompt uses `<!-- role:agent -->` /
|
||||
`<!-- role:manager -->` marker blocks, and a `Flavor::{Agent,
|
||||
Manager}` switch picks the MCP tool allow-list claude sees. Both are
|
||||
Manager}` switch picks the MCP tool allow-list claude sees. Both are
|
||||
already parametrised on a single flavour value, so the planned
|
||||
per-capability-group version (`cap:<group>` prompt blocks + a
|
||||
matching tool allow-list) is additive rather than a rewrite.
|
||||
- **State dirs** — *not* special-cased: `HYPERHIVE_STATE_DIR` is
|
||||
- **State dirs** — _not_ special-cased: `HYPERHIVE_STATE_DIR` is
|
||||
injected uniformly via `systemd.globalEnvironment` for every
|
||||
container including the manager, so all token/state paths resolve
|
||||
through it the same way everywhere.
|
||||
|
|
|
|||
|
|
@ -114,7 +114,7 @@ Nothing to run — it is served on the swarm apex
|
|||
decide whether you can actually open it:
|
||||
|
||||
- **You are in `admins`** (step 3). The gateway asks authelia whether
|
||||
you have a session; the rule that makes it mean *operator* wants the
|
||||
you have a session; the rule that makes it mean _operator_ wants the
|
||||
group. Without it you log in and still get bounced.
|
||||
- **The name resolves to this host.** It is published to the hive's own
|
||||
resolver and to `/etc/hosts` when `gateway.localHostsEntry` is on; from
|
||||
|
|
|
|||
|
|
@ -112,7 +112,7 @@ On a **successful** broker delivery, `forge_notify` marks the thread
|
|||
read on forge straight away (`PATCH /notifications/threads/{id}`). The
|
||||
broker inbox is the durable work queue now — each delivered wake is a
|
||||
sqlite row with its own ack lifecycle — so the forge unread flag no
|
||||
longer needs to track whether the agent has *processed* a
|
||||
longer needs to track whether the agent has _processed_ a
|
||||
notification. Clearing it on delivery keeps forge's unread set **tiny
|
||||
by construction**: at rest it holds only threads that failed to
|
||||
deliver plus whatever arrived since the last 30s poll.
|
||||
|
|
@ -132,19 +132,19 @@ own read-state is the only durable record this design needs.
|
|||
**Read-before-comment guard doesn't block a fresh wake.** hive-forge's
|
||||
read-before-comment guard (which keys off forge unread-state) does not
|
||||
force the agent to view a thread before commenting on it: the broker
|
||||
wake already carries the notification body, so *delivery is the read*.
|
||||
wake already carries the notification body, so _delivery is the read_.
|
||||
An agent that wants the full thread still runs `hive-forge comments` /
|
||||
`view`.
|
||||
|
||||
**In-process dedupe (tiny, ephemeral).** A single-process map (thread
|
||||
id → last-delivered `updated_at`) guards the narrow window where a
|
||||
mark-read call *transiently fails* and the thread reappears unread in
|
||||
mark-read call _transiently fails_ and the thread reappears unread in
|
||||
the next poll before its `updated_at` bumps — so a flaky PATCH doesn't
|
||||
re-fire the wake. It is **not persisted** and resets on restart (forge
|
||||
read-state covers the durable case). Each poll prunes it to the ids in
|
||||
the single `limit=UNREAD_FETCH_LIMIT` (50) fetch page, so it can never
|
||||
exceed that many entries (a debug assertion pins the invariant; the
|
||||
fetch limit and the bound are the same constant). A failed *delivery*
|
||||
fetch limit and the bound are the same constant). A failed _delivery_
|
||||
is left unread and out of the map, so it resurfaces next tick.
|
||||
|
||||
Self-echo notifications (the agent's own writes, see below) are marked
|
||||
|
|
@ -225,13 +225,13 @@ lookahead.
|
|||
|
||||
Five shapes, distinguished by the notification's classification:
|
||||
|
||||
| Trigger | Wrapper |
|
||||
| ----------------------------------- | --------------------------------------------------------------------------------- |
|
||||
| Trigger | Wrapper |
|
||||
| ----------------------------------- | ------------------------------------------------------------------------------ |
|
||||
| Comment on issue / PR | `[comment on PR #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
|
||||
| Review submission | `[PR approved #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
|
||||
| New issue / PR | `[new PR #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
| Later activity (open, not creation) | `[activity on PR #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
| State change | `[PR merged #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
| New issue / PR | `[new PR #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
| Later activity (open, not creation) | `[activity on PR #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
| State change | `[PR merged #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
|
||||
The issue/PR's own description is deliberately never embedded here (only a
|
||||
comment/review body is — that's the actual new content a notification
|
||||
|
|
|
|||
|
|
@ -109,7 +109,7 @@ Forgejo's notifications API is modelled on GitHub's, so one tolerant
|
|||
parse serves both: `id`, `repository.full_name`,
|
||||
`subject {title,url,latest_comment_url}` and `updated_at` line up field
|
||||
for field. The two real differences are absorbed by lenient
|
||||
deserializers — GitHub sends the thread id as a *string* where Forgejo
|
||||
deserializers — GitHub sends the thread id as a _string_ where Forgejo
|
||||
sends a number, and says `PullRequest` where Forgejo says `Pull`. Todo
|
||||
keys are prefixed `gh:<id>` so a github thread id cannot collide with a
|
||||
forge one.
|
||||
|
|
@ -118,16 +118,16 @@ Two host differences worth knowing before touching this code:
|
|||
|
||||
- **Auth scheme, not just value.** Forgejo takes
|
||||
`Authorization: token <t>`; GitHub wants `Bearer <t>` plus `Accept:
|
||||
application/vnd.github+json`, `X-GitHub-Api-Version` and a
|
||||
application/vnd.github+json`, `X-GitHub-Api-Version` and a
|
||||
`User-Agent`. Sending Forgejo's form to GitHub does not error — it
|
||||
authenticates as *nobody* and silently drops to the unauthenticated
|
||||
authenticates as _nobody_ and silently drops to the unauthenticated
|
||||
rate limit. The cheap way to tell the two apart is the rate-limit
|
||||
header: `x-ratelimit-remaining` near 5000 is an authenticated user,
|
||||
near 60 is anonymous.
|
||||
- **GitHub sets the cadence.** It returns `X-Poll-Interval` (60s in
|
||||
practice, slower than our own tick) and rate-limits callers who ignore
|
||||
it, so the loop re-arms to the server's interval whenever that is
|
||||
*slower* than ours. A hint faster than our own tick is not a reason to
|
||||
_slower_ than ours. A hint faster than our own tick is not a reason to
|
||||
poll harder.
|
||||
|
||||
⚠️ **This needs the `notifications` scope on the PAT.** A token minted
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@ hive-c0re maintains the local clone at
|
|||
`internal/knowledge` (see `docs/swarm/README.md` § Swarm-wide forge
|
||||
webhooks). On any push to main, including merge commits, it sends an
|
||||
event to every hive over the swarm queue and each hive runs `git
|
||||
pull`, so agents see the new content on their next turn.
|
||||
pull`, so agents see the new content on their next turn.
|
||||
|
||||
A hive that is offline when the event is sent does not get it on
|
||||
reconnect — the periodic pull below is what closes that gap. So one
|
||||
|
|
|
|||
|
|
@ -43,13 +43,13 @@ untouched by any of it.
|
|||
└─────────┘ └─────────┘ └─────────┘
|
||||
```
|
||||
|
||||
| container | netns | IPv4 | listens / reached via |
|
||||
| -------------- | ----------------------- | -------------------- | -------------------------------------------------------------------------------------------- |
|
||||
| `hive-gateway` | host (shared) | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
|
||||
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
|
||||
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
|
||||
| `hive-ci` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
|
||||
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
|
||||
| container | netns | IPv4 | listens / reached via |
|
||||
| -------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| `hive-gateway` | host (shared) | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
|
||||
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
|
||||
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
|
||||
| `hive-ci` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
|
||||
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
|
||||
|
||||
The flows, end to end:
|
||||
|
||||
|
|
@ -57,7 +57,7 @@ The flows, end to end:
|
|||
host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
|
||||
- **DNS** — agents and the service containers query `bridgeIp:53`; hive
|
||||
zones are answered authoritatively with the bridge IP, everything else
|
||||
forwards to the host's resolvers (see *Resolver behaviour* below). Each
|
||||
forwards to the host's resolvers (see _Resolver behaviour_ below). Each
|
||||
container points its own `resolv.conf` there, and one that instead
|
||||
inherits the host's resolves no swarm name at all — those records exist
|
||||
only on the bridge.
|
||||
|
|
@ -169,7 +169,7 @@ agent containers.
|
|||
The **host** firewall is the only firewall. The shared-netns infra
|
||||
containers (gateway, forge, matrix) set
|
||||
`networking.firewall.enable = false`: a NixOS firewall inside a
|
||||
shared-netns container runs against the *host* ruleset — at container
|
||||
shared-netns container runs against the _host_ ruleset — at container
|
||||
boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
|
||||
from the container's (empty) port list, and deletes the host's
|
||||
`nixos-nat-*` chains without recreating them, silently wiping the
|
||||
|
|
@ -183,7 +183,7 @@ By default agents can only reach the host on 80/443 (+53 DNS), so a
|
|||
host-side service on another port — e.g. a dev OTLP collector you want
|
||||
agents to reach directly — is unreachable. (hyperhive's own telemetry
|
||||
needs none of this: `otel.enable` opens its collector's port itself, and
|
||||
`otel.endpoint` is the *upstream*, which no agent ever dials. See
|
||||
`otel.endpoint` is the _upstream_, which no agent ever dials. See
|
||||
`docs/scheduler/observability.md`.)
|
||||
|
||||
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
|
||||
|
|
@ -206,12 +206,12 @@ Each agent container runs in a private network namespace with a dedicated
|
|||
veth pair attached to the bridge. The following table summarises what
|
||||
the nix side sets up unconditionally:
|
||||
|
||||
| effect | mechanism |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
|
||||
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
|
||||
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
|
||||
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
|
||||
| effect | mechanism |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
|
||||
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
|
||||
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
|
||||
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host |
|
||||
| c0re signal | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them |
|
||||
|
||||
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
|
||||
|
|
@ -262,7 +262,7 @@ wiring is runtime:
|
|||
|
||||
**Why isolation is safe**: hive-c0re's control-plane sockets are unix
|
||||
domain sockets bind-mounted into containers, not network listeners — see
|
||||
the *Control plane (no network)* bullet under [Network
|
||||
the _Control plane (no network)_ bullet under [Network
|
||||
map](#network-map) above. `PRIVATE_NETWORK=1` has no effect on a path
|
||||
that never touches the network stack.
|
||||
|
||||
|
|
|
|||
|
|
@ -43,8 +43,8 @@ Note that the mesh is gated on `swarm.wireguard.enable`, **not** on
|
|||
|
||||
## Pointing a hive at it
|
||||
|
||||
The block above configures the host that *receives*. Every hive that
|
||||
*pushes* separately needs to be told where the store is:
|
||||
The block above configures the host that _receives_. Every hive that
|
||||
_pushes_ separately needs to be told where the store is:
|
||||
|
||||
```nix
|
||||
services.hyperhive.swarm.snapshotStore = {
|
||||
|
|
@ -65,9 +65,9 @@ inconsistency:
|
|||
|
||||
Note the option lives under `swarm.*` while the receiving host's lives
|
||||
under `services.hyperhive.snapshotStore`. That is the distinction the
|
||||
two namespaces carry throughout: `swarm.*` describes *the swarm* as seen
|
||||
from this host, and a bare `services.hyperhive.<service>` describes *a
|
||||
role this host performs*. A store host sets both --- one to run the
|
||||
two namespaces carry throughout: `swarm.*` describes _the swarm_ as seen
|
||||
from this host, and a bare `services.hyperhive.<service>` describes _a
|
||||
role this host performs_. A store host sets both --- one to run the
|
||||
receiver, one only if it also runs a hive that pushes.
|
||||
|
||||
With it set, `hivectl agent <name> subvol snapshot push <label>
|
||||
|
|
@ -88,7 +88,7 @@ that address provably came from the holder of that private key. A
|
|||
packet that reaches the receiver has therefore already been
|
||||
authenticated by the kernel.
|
||||
|
||||
Layering TLS client certs on top would authenticate *the same fact* a
|
||||
Layering TLS client certs on top would authenticate _the same fact_ a
|
||||
second time, and add a credential with an expiry --- a migration that
|
||||
fails because a renewal quietly didn't happen, discovered on the day
|
||||
you need to move an agent.
|
||||
|
|
@ -98,14 +98,14 @@ you need to move an agent.
|
|||
The destination is keyed by **agent**.
|
||||
|
||||
This is not cosmetic. After a migration, an agent's next incremental
|
||||
send arrives from a *different* hive than the previous one. Keying by
|
||||
send arrives from a _different_ hive than the previous one. Keying by
|
||||
hive would split that agent's snapshot chain across two directories,
|
||||
and `btrfs send -p` would fail to find its parent --- breaking exactly
|
||||
the case the store exists to serve.
|
||||
|
||||
## What the sender can and cannot choose
|
||||
|
||||
A `btrfs send` stream carries no notion of *which agent* it belongs to,
|
||||
A `btrfs send` stream carries no notion of _which agent_ it belongs to,
|
||||
and the subvolume name inside it is chosen by the sender. So the
|
||||
protocol is one `agent <name>` header line, then the raw stream.
|
||||
|
||||
|
|
@ -129,7 +129,7 @@ address being set are assertions, not documentation --- bound to
|
|||
state.
|
||||
|
||||
Binding is not sufficient on its own. NixOS's firewall is default-deny
|
||||
and filters in netfilter, *before* a packet reaches a bound socket, so
|
||||
and filters in netfilter, _before_ a packet reaches a bound socket, so
|
||||
the port is opened explicitly --- and scoped to the mesh interface:
|
||||
|
||||
```nix
|
||||
|
|
@ -176,8 +176,8 @@ and means it should get the treatment a backup host gets --- restricted
|
|||
access, and a decision (rather than an omission) on encryption at rest.
|
||||
|
||||
The trap is the label: this box holds backup-grade data while not being
|
||||
called a backup, so it can end up with backup-grade *exposure* and
|
||||
non-backup-grade *controls*. Nobody puts a migration staging area on
|
||||
called a backup, so it can end up with backup-grade _exposure_ and
|
||||
non-backup-grade _controls_. Nobody puts a migration staging area on
|
||||
the access-review list.
|
||||
|
||||
### What a snapshot contains
|
||||
|
|
@ -202,7 +202,7 @@ gap is tracked separately.
|
|||
|
||||
### Retention
|
||||
|
||||
Retention lives on the *sending* side (last-N by count, swept
|
||||
Retention lives on the _sending_ side (last-N by count, swept
|
||||
periodically), not here. Count rather than age is deliberate: a count
|
||||
is bounded by construction, whereas an age policy silently scales disk
|
||||
usage with how hot a hive runs.
|
||||
|
|
|
|||
|
|
@ -78,8 +78,8 @@ existing containers can't be started.
|
|||
### `RestrictAddressFamilies` fails as "Address family not supported by protocol"
|
||||
|
||||
A unit whose `RestrictAddressFamilies` omits a family gets `EAFNOSUPPORT`
|
||||
(errno 97) back from `socket()`. Clients surface that as *"tcp open error:
|
||||
Address family not supported by protocol"* — the message names the
|
||||
(errno 97) back from `socket()`. Clients surface that as _"tcp open error:
|
||||
Address family not supported by protocol"_ — the message names the
|
||||
**protocol** and never the **sandbox**, so it reads like a dead network, a
|
||||
missing route, or an IPv6 problem.
|
||||
|
||||
|
|
@ -98,7 +98,7 @@ re-checks it when the program changes.** A unit that only served a unix
|
|||
socket when it was written is correct at `[ "AF_UNIX" ]` and silently wrong
|
||||
the day someone adds an HTTP client. Check the unit in the same commit as
|
||||
the client — and when narrowing it, prefer a test that derives the required
|
||||
families from the code (which fails on the *next* client too) over one that
|
||||
families from the code (which fails on the _next_ client too) over one that
|
||||
asserts today's list.
|
||||
|
||||
### `register_agent` is idempotent
|
||||
|
|
@ -120,7 +120,7 @@ operator's host-level `allowUnfree` does **not** propagate in.
|
|||
Operators don't need to set anything on their side.
|
||||
|
||||
That same isolation is why an agent can't pick a claude out of a
|
||||
*different* nixpkgs by itself: a container only ever sees the one
|
||||
_different_ nixpkgs by itself: a container only ever sees the one
|
||||
nixpkgs the meta flake injects, so an `agent.nix` naming the host's
|
||||
`nixpkgs-unstable` has nothing to name. A release channel can trail
|
||||
unstable by weeks on this package, which is what
|
||||
|
|
@ -140,7 +140,7 @@ an input only because a docs tree has no runtime dependencies.
|
|||
|
||||
The `storePath` trap is worth spelling out, because it is not confined
|
||||
to options the operator writes: **any** option of type `package` fed a
|
||||
store-path *string* coerces through `lib.toDerivation`, i.e.
|
||||
store-path _string_ coerces through `lib.toDerivation`, i.e.
|
||||
`builtins.storePath`. `environment.systemPackages` and
|
||||
`systemd.services.<name>.path` both do it (the latter takes plain
|
||||
strings like `/run/wrappers` happily, but anything under
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ human review that already happened.
|
|||
|
||||
Whether a PR can merge, and what counts toward "can", is configured
|
||||
per repo in its branch-protection settings — not a fact true of every
|
||||
hive or every repo. The pieces a repo *can* require:
|
||||
hive or every repo. The pieces a repo _can_ require:
|
||||
|
||||
- **CI is green** — the repo's required status checks pass on the
|
||||
PR's current head commit, if the repo requires any.
|
||||
|
|
@ -40,8 +40,8 @@ independently.
|
|||
|
||||
Auto-merge isn't "no human ever looked at this." Whoever arms it has
|
||||
already judged the PR sound at a coarse level — the signal it sends is
|
||||
roughly *"apart from maybe minor tweaks a reviewer can still catch,
|
||||
I think this is fine."* That's the human-in-the-loop step, and it
|
||||
roughly _"apart from maybe minor tweaks a reviewer can still catch,
|
||||
I think this is fine."_ That's the human-in-the-loop step, and it
|
||||
already happened. No large changes are expected to surface after
|
||||
that point — a reviewer's job past that point is to flag it if one
|
||||
does, not to assume none ever will.
|
||||
|
|
|
|||
|
|
@ -18,11 +18,11 @@ Two things worth knowing before you rely on it:
|
|||
already has forge access; if you take outside/fork contributions,
|
||||
gate CI behind Forgejo's fork-PR-approval setting or restrict the
|
||||
workflow to push-only triggers (forks can't push to your branches).
|
||||
See *Security* below for the full threat model.
|
||||
See _Security_ below for the full threat model.
|
||||
- **Watch your disk.** CI builds through the host's nix store with no
|
||||
automatic garbage collection of its own — a busy CI day can fill
|
||||
`/nix/store` until every job fails with `ENOSPC`. Add the daily +
|
||||
disk-pressure GC config from *Host store maintenance* below to your
|
||||
disk-pressure GC config from _Host store maintenance_ below to your
|
||||
host's NixOS config (not optional if you plan to actually use this).
|
||||
|
||||
Everything below this point is implementation detail: exact
|
||||
|
|
@ -36,10 +36,10 @@ defined in [`.forgejo/workflows/ci.yml`](../../.forgejo/workflows/ci.yml). All
|
|||
three are required checks (forge branch protection) — a hit on any of them
|
||||
blocks merge.
|
||||
|
||||
| Job | What it runs |
|
||||
| --- | --- |
|
||||
| **nix flake check** | treefmt + rustfmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation |
|
||||
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) |
|
||||
| Job | What it runs |
|
||||
| ---------------------- | --------------------------------------------------------------------------------- |
|
||||
| **nix flake check** | treefmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation |
|
||||
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) |
|
||||
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) |
|
||||
|
||||
`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
|
||||
|
|
@ -69,7 +69,7 @@ wastes a remote-builder slot. Use the devshell equivalents instead:
|
|||
```sh
|
||||
nix develop -c cargo clippy --all-targets -- -D warnings
|
||||
nix develop -c cargo test
|
||||
nix develop -c treefmt # same as nix fmt; treefmt covers rustfmt + nixfmt + taplo
|
||||
nix fmt # authoritative; the formatter set is nix/treefmt.nix
|
||||
sh scripts/check-issue-refs.sh # tracker-tag lint
|
||||
sh scripts/check-comment-blocks.sh # comment-block lint
|
||||
```
|
||||
|
|
@ -90,7 +90,7 @@ slow); run those manually before pushing Rust changes.
|
|||
|
||||
The internal forge is always present (mandatory), so the runner always has a
|
||||
hive-forge instance to register against — nothing extra to enable beyond
|
||||
`services.hyperhive.deploy.forgejo.ci.enable = true` (see *For operators* above).
|
||||
`services.hyperhive.deploy.forgejo.ci.enable = true` (see _For operators_ above).
|
||||
|
||||
Optional tuning: `services.hyperhive.deploy.forgejo.ci.name` (runner name in forge
|
||||
admin panel), `concurrency` (parallel job capacity), `labels` (workflow
|
||||
|
|
@ -131,7 +131,7 @@ Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.serv
|
|||
- The container boots immediately — nothing gates its start on registration.
|
||||
- tmpfiles seeds `/run/hive-ci/runner-token` with `TOKEN=placeholder` so the runner's `EnvironmentFile` always exists.
|
||||
- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it is already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
|
||||
- **Convergence**: because the token write targets the *host* file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
|
||||
- **Convergence**: because the token write targets the _host_ file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
|
||||
|
||||
## Actions checkout mirror
|
||||
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
|
|||
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
|
||||
concurrency comes from the resource classes below, not from multiple workers.
|
||||
Special cases like graceful stop, deferred starts, and the meta-update
|
||||
cascade need no bespoke code paths — each is expressed as a DAG *shape*
|
||||
cascade need no bespoke code paths — each is expressed as a DAG _shape_
|
||||
built from the same primitive nodes.
|
||||
|
||||
### Two levels: DAG and node
|
||||
|
|
@ -42,54 +42,54 @@ there is no malformed spec to reject.
|
|||
|
||||
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
||||
|
||||
| Node | Wraps |
|
||||
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
|
||||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta |
|
||||
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
|
||||
| `DeployWindow` | resource-holding root of the merge-config-PR deploy subtree — declares the build slot, the lease and the meta window, then completes immediately so its children run under them (see _Approvals_ below) |
|
||||
| `DeployApply` | the deploy's irreversible half: ff-merge the reviewed PR head, two-phase meta deploy, container rebuild |
|
||||
| Node | Wraps |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
|
||||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta |
|
||||
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
|
||||
| `DeployWindow` | resource-holding root of the merge-config-PR deploy subtree — declares the build slot, the lease and the meta window, then completes immediately so its children run under them (see _Approvals_ below) |
|
||||
| `DeployApply` | the deploy's irreversible half: ff-merge the reviewed PR head, two-phase meta deploy, container rebuild |
|
||||
|
||||
Cheap — no build slot:
|
||||
|
||||
| Node | Behavior |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `MergeVerify` | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation |
|
||||
| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail: (1) rolls `applied/main` back from the parked `refs/hyperhive/rollback/<id>` and aborts the staged meta lock when the deploy never confirmed good; (2) mirrors whichever deploy tag landed to the forge config repo, always, best-effort; (3) posts the failing build log back onto the config PR when the deploy failed. Named for (2)/(3), which run on the success path too — not `AbortDeploy`. Infallible by construction |
|
||||
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
|
||||
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
||||
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
|
||||
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
||||
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
|
||||
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
|
||||
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
|
||||
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
|
||||
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here |
|
||||
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
|
||||
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
|
||||
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
|
||||
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
|
||||
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
|
||||
| `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/<id>`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree |
|
||||
| `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent |
|
||||
| `EmitRebuilt` | tail of a rebuild/perm-change — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent *and* per outcome |
|
||||
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
|
||||
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
|
||||
| `Reparent` | `set-parent` / `set-parent-bulk`: apply every `(child, new_parent)` move under one `META_LOCK` commit (`meta::bulk_commit_topology`), send the per-agent move notifications, rescan + diff-emit. Agentless like `MetaLock` — a bulk move can span multiple agents, and a reparent touches the meta repo, not any one container. `moves` is typed `(Ident, Option<Ident>)` pairs, not raw strings. Rides the existing `Template::MetaUpdate` variant rather than a dedicated one — it's internal-only (never reaches the graph wire), so the stand-in only affects `terminal_hook` dispatch (resolves to no hook either way) and history-retention bucketing |
|
||||
| `ForgeSweep` | one-shot boot-time forge user/token sweep for every container (`forge::ensure_all`) as a first-class node, so it shows as real work on the dashboard instead of running invisibly in a bare `tokio::spawn`. Agentless |
|
||||
| `MatrixSweep` | same as `ForgeSweep`, for matrix (`matrix::ensure_all`). The periodic 30-min re-sweep stays a background loop in `main.rs`; only the boot-time instance is a node |
|
||||
| `WebhookRegister` | one-shot boot-time Forgejo webhook registration (`internal/knowledge` push→pull, `agent-configs` PR→approval). No-op until the core token, hive domain, and HMAC secret are all available. Agentless |
|
||||
| `KnowledgePull` | one-shot boot-time `/knowledge` pull (`knowledge::pull`), reconciling commits that landed while `hive-c0re` was down. Same rationale as `MatrixSweep`: the periodic hourly re-pull stays a background loop |
|
||||
| `WantedPull` | one-shot boot-time pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. No background loop behind this one — boot is the whole cadence; the deploy event (`swarm_status`) is the fast path, this repairs a missed one. Agentless |
|
||||
| Node | Behavior |
|
||||
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `MergeVerify` | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation |
|
||||
| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail: (1) rolls `applied/main` back from the parked `refs/hyperhive/rollback/<id>` and aborts the staged meta lock when the deploy never confirmed good; (2) mirrors whichever deploy tag landed to the forge config repo, always, best-effort; (3) posts the failing build log back onto the config PR when the deploy failed. Named for (2)/(3), which run on the success path too — not `AbortDeploy`. Infallible by construction |
|
||||
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
|
||||
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
||||
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
|
||||
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
||||
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
|
||||
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
|
||||
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
|
||||
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
|
||||
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here |
|
||||
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
|
||||
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
|
||||
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
|
||||
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
|
||||
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
|
||||
| `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/<id>`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree |
|
||||
| `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent |
|
||||
| `EmitRebuilt` | tail of a rebuild/perm-change — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent _and_ per outcome |
|
||||
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
|
||||
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
|
||||
| `Reparent` | `set-parent` / `set-parent-bulk`: apply every `(child, new_parent)` move under one `META_LOCK` commit (`meta::bulk_commit_topology`), send the per-agent move notifications, rescan + diff-emit. Agentless like `MetaLock` — a bulk move can span multiple agents, and a reparent touches the meta repo, not any one container. `moves` is typed `(Ident, Option<Ident>)` pairs, not raw strings. Rides the existing `Template::MetaUpdate` variant rather than a dedicated one — it's internal-only (never reaches the graph wire), so the stand-in only affects `terminal_hook` dispatch (resolves to no hook either way) and history-retention bucketing |
|
||||
| `ForgeSweep` | one-shot boot-time forge user/token sweep for every container (`forge::ensure_all`) as a first-class node, so it shows as real work on the dashboard instead of running invisibly in a bare `tokio::spawn`. Agentless |
|
||||
| `MatrixSweep` | same as `ForgeSweep`, for matrix (`matrix::ensure_all`). The periodic 30-min re-sweep stays a background loop in `main.rs`; only the boot-time instance is a node |
|
||||
| `WebhookRegister` | one-shot boot-time Forgejo webhook registration (`internal/knowledge` push→pull, `agent-configs` PR→approval). No-op until the core token, hive domain, and HMAC secret are all available. Agentless |
|
||||
| `KnowledgePull` | one-shot boot-time `/knowledge` pull (`knowledge::pull`), reconciling commits that landed while `hive-c0re` was down. Same rationale as `MatrixSweep`: the periodic hourly re-pull stays a background loop |
|
||||
| `WantedPull` | one-shot boot-time pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. No background loop behind this one — boot is the whole cadence; the deploy event (`swarm_status`) is the fast path, this repairs a missed one. Agentless |
|
||||
|
||||
There is deliberately **no `GitCommit` node**: `meta.rs` fuses each mutation
|
||||
with its commit under its internal `META_LOCK` mutex, so a standalone commit
|
||||
node would open a dirty-working-tree window between nodes.
|
||||
|
||||
Two further layers protect the meta repo across *windows* that span multiple
|
||||
Two further layers protect the meta repo across _windows_ that span multiple
|
||||
`META_LOCK` acquisitions — above all the approval deploy's prepare→finalize
|
||||
span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole
|
||||
container build:
|
||||
|
|
@ -104,7 +104,7 @@ container build:
|
|||
because a resource is held by a subtree root across its whole subtree, which
|
||||
a `MutexGuard` (bounded by one executor fn) cannot — that is what lets a
|
||||
multi-node deploy own one window. For the same reason the window must stay
|
||||
*off* long store-only work: the rebuild's meta preamble is its own
|
||||
_off_ long store-only work: the rebuild's meta preamble is its own
|
||||
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
|
||||
toplevel build runs outside the window and `buildSlots > 1` still gives
|
||||
concurrent rebuilds across agents.
|
||||
|
|
@ -119,10 +119,10 @@ container build:
|
|||
The `stop` / `start` power ops write the durable `wanted` intent via a head
|
||||
`SetWanted` node (not a pre-submit side effect) — it holds the agent lease,
|
||||
so intent-write + reconcile is atomic per-agent. `restart` is the exception:
|
||||
it writes *no* intent (no `SetWanted` head) — it bounces the container and
|
||||
it writes _no_ intent (no `SetWanted` head) — it bounces the container and
|
||||
lets the tail `Reconcile` converge to the agent's existing `wanted`, so a
|
||||
deliberately-stopped agent is not forced back up by a hive-wide restart. The
|
||||
hive-wide power ops — `restart`, `stop`, and `start` — take an agent *list*:
|
||||
hive-wide power ops — `restart`, `stop`, and `start` — take an agent _list_:
|
||||
a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
|
||||
subgraph each (independent roots, run concurrently on their own leases), not
|
||||
N separate DAGs.
|
||||
|
|
@ -132,12 +132,12 @@ async `lifecycle::is_running` read), so they live in `job_queue/power.rs`,
|
|||
not the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
|
||||
a head `SetWanted` (intent) — `restart` does not; the tail `Reconcile`
|
||||
(convergence guarantee — cheap, noops when already converged) is ALWAYS
|
||||
present; only the *mechanical* nodes (`Signal`/`Drain`/`StopForUpdate`) are
|
||||
state-conditional — skipped for a *down* agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
|
||||
present; only the _mechanical_ nodes (`Signal`/`Drain`/`StopForUpdate`) are
|
||||
state-conditional — skipped for a _down_ agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
|
||||
closes the TOCTOU window: if an agent flips state between the `is_running`
|
||||
read and node exec, the tail `Reconcile` still converges it in-DAG (with
|
||||
`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
|
||||
sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale*
|
||||
sweep. `start` folds the per-agent stale-rev upgrade in (a _down + stale_
|
||||
agent's subgraph is a rebuild-then-start).
|
||||
|
||||
```text
|
||||
|
|
@ -163,7 +163,7 @@ Notable collapses:
|
|||
leaves it stopped).
|
||||
- **The swap-failure recovery-start is structural**: `Reconcile` deps on
|
||||
`Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
|
||||
terminal ok *or* fail, bringing a wanted-up agent back on its old config.
|
||||
terminal ok _or_ fail, bringing a wanted-up agent back on its old config.
|
||||
- **Deferred start is automatic**: `Reconcile` holds no build slot, so the
|
||||
next DAG's `Prebuild` starts as soon as `Swap` frees the slot.
|
||||
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
|
||||
|
|
@ -171,7 +171,7 @@ Notable collapses:
|
|||
drains overlap; each DAG's tail `Reconcile` does the actual stop.
|
||||
- **The meta-update cascade grows in the same DAG on completion**:
|
||||
`MetaLock`'s executor computes the affected agent set after the bump lands
|
||||
and grows one `rebuild` subgraph per agent into its *own* DAG via
|
||||
and grows one `rebuild` subgraph per agent into its _own_ DAG via
|
||||
`append_subgraph` (rooted on the `MetaLock`, `relock = false` so the cascade
|
||||
doesn't revert the bump). Not child DAGs — one DAG, no `parent_id`. A failed
|
||||
bump appends nothing (no cancel-children dance). Same shape as the startup
|
||||
|
|
@ -180,9 +180,9 @@ Notable collapses:
|
|||
|
||||
### Desired-state (spec vs status)
|
||||
|
||||
Per-agent power *intent* — `wanted: Up | Offline` — is durable as the
|
||||
Per-agent power _intent_ — `wanted: Up | Offline` — is durable as the
|
||||
`agent_power` table in the coordinator DB (`hive-c0re/src/stores/power.rs`).
|
||||
`container_view` remains the observed *status*; `Reconcile` nodes converge the
|
||||
`container_view` remains the observed _status_; `Reconcile` nodes converge the
|
||||
two. Setting `wanted` is never a queued node: the power layer
|
||||
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
|
||||
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
|
||||
|
|
@ -237,15 +237,15 @@ than to one DAG shape it happens to appear in. A **brace** is the one sanctioned
|
|||
exception: a pure-resource-holder root that declares on behalf of a subtree
|
||||
coordinated with itself, whose members then declare nothing.
|
||||
|
||||
It is forced rather than stylistic. Declaring a resource means *"I need this
|
||||
exclusively"*, and the agent lease is single-unit — so **two siblings that both
|
||||
It is forced rather than stylistic. Declaring a resource means _"I need this
|
||||
exclusively"_, and the agent lease is single-unit — so **two siblings that both
|
||||
declared it could never run concurrently.** For a subtree whose whole point is
|
||||
concurrency (`Prebuild` beside the `Signal` → `Drain` quiesce window), declaring
|
||||
the requirement truthfully on every node and running those nodes in parallel are
|
||||
mutually exclusive. One holder above them speaks for the subtree.
|
||||
|
||||
This is the opposite of the failure the declare-your-own rule exists to prevent,
|
||||
not a relapse into it: there the requirement was *implicit*, inferred from a
|
||||
not a relapse into it: there the requirement was _implicit_, inferred from a
|
||||
node's kind and true only by accident of placement. Here it is explicit, on one
|
||||
node, with the omission below it documented on the brace itself.
|
||||
|
||||
|
|
@ -257,7 +257,7 @@ Two consequences worth knowing:
|
|||
leave a gap another DAG could claim the agent in, mid-bounce. A brace supplies
|
||||
that continuity directly, so the nesting is no longer load-bearing.
|
||||
- **Observability is unaffected.** `running_transients` keys off a node's
|
||||
*payload* agent, not off a declared lease edge, so every child still lights its
|
||||
_payload_ agent, not off a declared lease edge, so every child still lights its
|
||||
own dashboard pill and still reports its own `takes_container_down` to the
|
||||
crash watcher. A brace itself reports `false`: it parents the stopping nodes
|
||||
but does not stop anything, and claiming otherwise would widen crash
|
||||
|
|
@ -323,12 +323,12 @@ cancelled-while-queued, which fails the approval instead of dangling it).
|
|||
### Wire shape
|
||||
|
||||
`RebuildQueueChanged { seq }` (event name kept) — **a bare trigger, no
|
||||
payload.** It says *the queue changed*; a client that wants to know how
|
||||
payload.** It says _the queue changed_; a client that wants to know how
|
||||
re-fetches `GET /api/jobq/graph`.
|
||||
|
||||
That endpoint serves the graph generically (`hive-jobq-wire`): every node
|
||||
carries `id`, `parent`, `deps`, `state`, `label` (the node kind's own wire
|
||||
string — the kind *is* the phase label, there is no separate sub-step
|
||||
string — the kind _is_ the phase label, there is no separate sub-step
|
||||
string) and free-form `data` for what only some kinds have (`agent`,
|
||||
`approval_id`, `inputs`, `build_log_id`). Group roots ride as ordinary
|
||||
nodes, so a group's state is just the root's own `state`.
|
||||
|
|
@ -341,7 +341,7 @@ live-log panel off the running node.
|
|||
The event carries no payload by design: shipping a typed projection of
|
||||
the whole queue in the event itself would be a second rendering of the
|
||||
same graph that has to be kept in agreement by hand with the endpoint
|
||||
every consumer actually reads. Telling a client *when* to refetch is
|
||||
every consumer actually reads. Telling a client _when_ to refetch is
|
||||
the event's whole job.
|
||||
|
||||
---
|
||||
|
|
@ -362,7 +362,7 @@ On startup, `auto_update::run` classifies every agent by rev freshness (the
|
|||
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
|
||||
vs the current flake path) and persisted `wanted` intent, then:
|
||||
|
||||
1. **Config path** — when *any* marker is stale, submit one `Boot`
|
||||
1. **Config path** — when _any_ marker is stale, submit one `Boot`
|
||||
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that grows an in-DAG
|
||||
`Rebuild` subgraph for each stale agent whose `wanted = Up` (topology-sorted,
|
||||
parents first). Stale but wanted-offline agents get no boot-time nix work — their
|
||||
|
|
@ -453,7 +453,7 @@ graft's completion then plants the deploy tag — so "did the agent come back
|
|||
up?" is answered by `Reconcile` succeeding, the same way it is for every
|
||||
other rebuild, instead of by a fused inline start.
|
||||
|
||||
The grafted nodes land *inside* `DeployWindow`'s subtree, so they re-enter
|
||||
The grafted nodes land _inside_ `DeployWindow`'s subtree, so they re-enter
|
||||
the meta window and build slot it already holds rather than deadlocking
|
||||
against it.
|
||||
|
||||
|
|
@ -514,7 +514,7 @@ The same drop-in carries `CPUWeight=` / `IOWeight=` from
|
|||
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
|
||||
setting: the quota and the memory max are **hard caps** that throttle
|
||||
an agent even on a completely idle host, while the weights are cgroup
|
||||
v2 **relative shares** that only decide who yields *under contention*.
|
||||
v2 **relative shares** that only decide who yields _under contention_.
|
||||
A low-weight container still gets the whole machine when nothing else
|
||||
wants it.
|
||||
|
||||
|
|
@ -529,7 +529,7 @@ Two things to know about the weights:
|
|||
|
||||
- They are **hive-wide** — unlike the caps there is no per-agent
|
||||
override in `meta/resource-limits.json`, so every agent carries the
|
||||
same value and the weight does *not* rank agents against each other.
|
||||
same value and the weight does _not_ rank agents against each other.
|
||||
What `80` buys is that agents yield to everything **not** on this
|
||||
drop-in path: host services (nginx and dnsmasq among them) and the
|
||||
infra containers (`hive-ci`, `hive-forge`, `hive-gateway`,
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Every container operation — rebuild, first-spawn, a config-PR deploy,
|
||||
power changes — runs through one shared job queue. This page explains
|
||||
what the job queue *is*, as a general idea, independent of what any one
|
||||
what the job queue _is_, as a general idea, independent of what any one
|
||||
subsystem uses it for. For the hive-c0re-specific step catalogue and the
|
||||
engineering internals (scheduler, leases, resource windows) see
|
||||
[`coordinator.md`](coordinator.md) instead.
|
||||
|
|
@ -23,7 +23,7 @@ Two ideas are all there is to it:
|
|||
that's the whole reason things queue instead of all firing at once.
|
||||
|
||||
The engine's whole job is: whenever a step's ordering and resource needs
|
||||
are both satisfied, run it. It has no opinion on what the steps *do* —
|
||||
are both satisfied, run it. It has no opinion on what the steps _do_ —
|
||||
that's supplied by whoever builds the graph. hive-c0re is the one thing
|
||||
building graphs on it today, but nothing about the engine is specific to
|
||||
containers or rebuilds; there's nothing stopping another subsystem from
|
||||
|
|
@ -38,14 +38,14 @@ nested under it are that job's steps, in order (occasionally a couple run
|
|||
side by side). A step shows one of:
|
||||
|
||||
| Glyph | Meaning |
|
||||
| ----- | -------------------------------------------------------- |
|
||||
| `⏸` | queued, waiting its turn |
|
||||
| `▶` | running |
|
||||
| `◐` | its own work is done, waiting on a step nested under it |
|
||||
| `✔` | finished successfully |
|
||||
| `✖` | failed |
|
||||
| `⊘` | cancelled |
|
||||
| `·` | skipped (not needed for this run) |
|
||||
| ----- | ------------------------------------------------------- |
|
||||
| `⏸` | queued, waiting its turn |
|
||||
| `▶` | running |
|
||||
| `◐` | its own work is done, waiting on a step nested under it |
|
||||
| `✔` | finished successfully |
|
||||
| `✖` | failed |
|
||||
| `⊘` | cancelled |
|
||||
| `·` | skipped (not needed for this run) |
|
||||
|
||||
A step that isn't needed for a given run shows as `·` rather than being
|
||||
left out of the tree entirely, so the same kind of operation keeps a
|
||||
|
|
|
|||
|
|
@ -46,7 +46,7 @@ no longer survives independently of anything host-side.
|
|||
**It has no application-level auth.** The receiver takes any OTLP that reaches
|
||||
it; what bounds who can reach it is the firewall — `exposeHostPorts` opens the
|
||||
port on the bridge interface only. So "unauthenticated to a bridge address"
|
||||
means *reachable from an agent container*, not *presents a credential*.
|
||||
means _reachable from an agent container_, not _presents a credential_.
|
||||
|
||||
The consequence, stated because it is a choice rather than an oversight: **any
|
||||
agent can push arbitrary OTLP, and it is forwarded on under the operator's
|
||||
|
|
@ -56,10 +56,10 @@ smuggled out in resource attributes on an otherwise-legitimate export.
|
|||
|
||||
That is a **different risk from the one the collector fixes**, and strictly
|
||||
smaller than what preceded it: before, every agent held the upstream credential
|
||||
itself, so it could do all of the above *and* use the token anywhere else. The
|
||||
itself, so it could do all of the above _and_ use the token anywhere else. The
|
||||
collector removes the token and keeps the pipe. Agents are inside the trust
|
||||
boundary (`docs/trust-boundary/security.md`: capability = accepted risk), so an agent being
|
||||
able to *send* is an accepted extension of that boundary — but it is not
|
||||
able to _send_ is an accepted extension of that boundary — but it is not
|
||||
closed by this design, and nothing here should be read as closing it.
|
||||
|
||||
**The `agent` label is self-reported, and no planned authentication changes
|
||||
|
|
@ -73,13 +73,13 @@ is getting a credential:
|
|||
- **agent→collector** (this section's hop) stays open on the bridge. Nothing
|
||||
downstream can tell one agent's export from another's.
|
||||
- **hive→swarm** is where the planned ingest auth goes. The swarm tier stamps
|
||||
`hive=` from the connection it authenticated, so *that* label becomes
|
||||
`hive=` from the connection it authenticated, so _that_ label becomes
|
||||
unforgeable.
|
||||
|
||||
So a verified `hive` is reachable and a verified `agent` is not — and that falls
|
||||
out of the topology rather than being a gap someone forgot to close. The swarm
|
||||
runs one collector, and the mechanism gives it no finer grain: a bearer-token
|
||||
check never reveals *which* token matched, and a receiver reads request metadata
|
||||
check never reveals _which_ token matched, and a receiver reads request metadata
|
||||
rather than the claims it authenticated with.
|
||||
|
||||
If you need per-agent numbers you can act on, take them from the agent's own
|
||||
|
|
@ -100,10 +100,10 @@ two-tier architecture, the security model, and how the options interact.
|
|||
Telemetry crosses two collectors, and which one you configure depends on what
|
||||
the host is:
|
||||
|
||||
| | runs where | receives from | does |
|
||||
|---|---|---|---|
|
||||
| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination |
|
||||
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's store and exports upstream |
|
||||
| | runs where | receives from | does |
|
||||
| ------------------------------------ | ---------------------- | --------------------------------- | --------------------------------------------------------------------- |
|
||||
| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination |
|
||||
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's store and exports upstream |
|
||||
|
||||
An all-local host runs both, and needs nothing said about the hop between them.
|
||||
|
||||
|
|
@ -119,16 +119,16 @@ services.hyperhive.otel = {
|
|||
every agent needs the credential to authenticate — and the harness delivers
|
||||
that token into the agent's own `~/.claude/settings.json`, a file the agent can
|
||||
read. `0600` protects it from other containers, not from the agent itself. As
|
||||
long as the direct path stays *selectable*, that hole stays selectable; an
|
||||
long as the direct path stays _selectable_, that hole stays selectable; an
|
||||
option that can reintroduce it is a hole with extra steps.
|
||||
|
||||
**Why the tiers stay separate on one box.** They are not collapsed when
|
||||
co-located: an all-local hive is a statement about *where* processes run, not
|
||||
co-located: an all-local hive is a statement about _where_ processes run, not
|
||||
about the shape of the deployment. A boundary that disappears locally is one
|
||||
the local deployment stops testing.
|
||||
|
||||
**`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier
|
||||
redefines it — the agent-facing value is *derived*
|
||||
redefines it — the agent-facing value is _derived_
|
||||
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
|
||||
keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
|
||||
automatically; there is nothing to open by hand.
|
||||
|
|
@ -160,9 +160,9 @@ swarm tier and no metrics appearing for that hive.
|
|||
|
||||
⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
|
||||
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
|
||||
exporter takes an *address*: `https://host/path` is a legal
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in
|
||||
address"*. The collector's config is validated at build time, so a mismatch is
|
||||
exporter takes an _address_: `https://host/path` is a legal
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in
|
||||
address"_. The collector's config is validated at build time, so a mismatch is
|
||||
a build error naming the reason rather than telemetry silently going nowhere.
|
||||
|
||||
## Network access
|
||||
|
|
@ -176,7 +176,7 @@ services.hyperhive.network.exposeHostPorts = [ 5432 ];
|
|||
```
|
||||
|
||||
and point whatever consumes it at `10.42.0.1:5432` rather than loopback: inside
|
||||
a container, loopback is the *container*. The bridge IP is the host's address on
|
||||
a container, loopback is the _container_. The bridge IP is the host's address on
|
||||
the `hive-br0` bridge. The service must also bind an address the bridge can
|
||||
reach — a `127.0.0.1`-only listener stays unreachable no matter what the
|
||||
firewall allows. See `docs/networking/network.md::Reaching host services` for details.
|
||||
|
|
@ -196,12 +196,12 @@ yourself from a tool call or `hivectl shell`.
|
|||
Every agent's export therefore includes these resource attributes
|
||||
automatically:
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| `service.name` | `hyperhive-agent` (constant) |
|
||||
| `agent` | agent logical name (e.g. `iris`) |
|
||||
| `hive` | hive display name (`services.hyperhive.hiveName`) |
|
||||
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
|
||||
| Attribute | Value |
|
||||
| -------------- | ------------------------------------------------------------ |
|
||||
| `service.name` | `hyperhive-agent` (constant) |
|
||||
| `agent` | agent logical name (e.g. `iris`) |
|
||||
| `hive` | hive display name (`services.hyperhive.hiveName`) |
|
||||
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
|
||||
|
||||
Additional labels can be appended via `extraResourceAttributes` (see option
|
||||
reference above); custom per-data-point labels can be passed with
|
||||
|
|
@ -222,14 +222,14 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
|
|||
`service.name = hyperhive-c0re`; each data point is tagged `container.name`
|
||||
(= the `h-<agent>` machine) and the hive `agent` label:
|
||||
|
||||
| Metric | Unit | Kind | Source |
|
||||
|--------|------|------|--------|
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | state dir + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
|
||||
| Metric | Unit | Kind | Source |
|
||||
| ----------------------------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------- |
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | state dir + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
|
||||
|
||||
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
|
||||
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
|
||||
|
|
@ -252,14 +252,14 @@ turn, its own outcome classification, the loose-ends backlog, and session
|
|||
boundaries. Token usage, cost, and tool-call counts are **not** duplicated
|
||||
here; that's already covered by Claude's own export.
|
||||
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
|--------|------|------|------------|
|
||||
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
|
||||
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
|
||||
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
|
||||
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `CLAUDE.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
| -------------------------------------- | ---- | --------- | ----------------------------------------------------------------------------------------------------------- |
|
||||
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
|
||||
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
|
||||
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
|
||||
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `CLAUDE.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
|
||||
|
||||
Resource attributes (`service.name`, `agent`, `hive`, `swarm`) come from the
|
||||
same container-wide `OTEL_RESOURCE_ATTRIBUTES` as everything else in this
|
||||
|
|
@ -277,11 +277,11 @@ that hosts no agents still reports, and "this hive is quiet" is
|
|||
distinguishable from "this hive is gone". Select them with
|
||||
`{hive!="",agent=""}`.
|
||||
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
|--------|------|------|---------|
|
||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
| ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
|
||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||
|
||||
Both levels are reported every cycle, `0` included, so a healthy hive is
|
||||
visible as zeros rather than as missing series.
|
||||
|
|
@ -303,10 +303,10 @@ Forgejo's own native `/metrics` endpoint has no equivalent: it exposes
|
|||
counts of durable rows (issues, comments, repos), and neither a commit nor a
|
||||
push is stored anywhere as a row to count.
|
||||
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
|--------|------|------|------------|
|
||||
| `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) |
|
||||
| `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) |
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
| ---------------------------- | ---- | ------- | ------------------- |
|
||||
| `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) |
|
||||
| `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) |
|
||||
|
||||
A push with zero commits (a branch delete, or a force-push that doesn't add
|
||||
new commits) still increments `push.count`; `commit.count` only advances
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ the additional config needed when the swarm spans multiple hosts.
|
|||
|
||||
For the full option reference rather than prose: `services.hyperhive.swarm.*`
|
||||
(swarm-wide facts, identical on every host) and `services.hyperhive.deploy.*`
|
||||
(this host's own deployment decisions — does *this* machine run grafana,
|
||||
(this host's own deployment decisions — does _this_ machine run grafana,
|
||||
the swarm controller, authelia, …) are separate generated pages, `nix
|
||||
build .#docs-swarm` / `.#docs-deploy` or the website's `/options/swarm.html`
|
||||
/ `/options/deploy.html`.
|
||||
|
|
@ -20,7 +20,7 @@ build .#docs-swarm` / `.#docs-deploy` or the website's `/options/swarm.html`
|
|||
- **swarm** — one or more hives whose operators have declared them
|
||||
as peers. Agents can be qualified as `agent@hive-domain`.
|
||||
- **peer hive** — any hive in `services.hyperhive.swarm.hives` other
|
||||
than this one. Peers are *derived*, not declared: the directory lists
|
||||
than this one. Peers are _derived_, not declared: the directory lists
|
||||
every hive including yourself, and `hiveName` says which one you are.
|
||||
|
||||
## Hive identity config
|
||||
|
|
@ -47,7 +47,7 @@ and deploys — an eval failure asking the operator to write the address
|
|||
down is the cheaper outcome. **Upgrading past this release means setting
|
||||
both once.**
|
||||
|
||||
`domain` is required too, but you no longer *write* it: it is read from
|
||||
`domain` is required too, but you no longer _write_ it: it is read from
|
||||
this hive's own entry in the directory, whose `domain` defaults to
|
||||
`<name>.<swarm.domain>`. So a conventional swarm states no addresses at
|
||||
all, and a hive addressed by something else states it in the one place
|
||||
|
|
@ -70,7 +70,7 @@ form qualified labels (`iris@pr1ma.example.com`).
|
|||
`swarm.name` is purely display — it surfaces in the dashboard chrome
|
||||
header and per-agent system prompts, and federated hives at different
|
||||
domains can share one. `hiveName` surfaces in the same places but is
|
||||
*not* only display: it is the leftmost label of the hive's domain. That
|
||||
_not_ only display: it is the leftmost label of the hive's domain. That
|
||||
`swarm.name` sits under `swarm` and `hiveName` does not is the whole
|
||||
distinction — one names this hive, the other names the group it belongs
|
||||
to.
|
||||
|
|
@ -120,14 +120,14 @@ services.hyperhive.swarm.hives = {
|
|||
```
|
||||
|
||||
One attrset describing **every** hive in the swarm, **including this
|
||||
one**, keyed by that hive's `hiveName`. It is meant to be *identical on
|
||||
every host* — write it once, share it, and each host reads it correctly
|
||||
one**, keyed by that hive's `hiveName`. It is meant to be _identical on
|
||||
every host_ — write it once, share it, and each host reads it correctly
|
||||
because `services.hyperhive.hiveName` says which entry is itself.
|
||||
|
||||
Empty (the default) means this host isn't in a swarm. Once non-empty it
|
||||
**must** contain an entry for `hiveName`; eval fails naming the missing
|
||||
hive. That assertion is load-bearing rather than pedantic — "my peers"
|
||||
is derived as *everything that isn't me*, so a directory that doesn't
|
||||
is derived as _everything that isn't me_, so a directory that doesn't
|
||||
contain you derives every hive as a peer and you peer with yourself.
|
||||
|
||||
`domain` is required per entry and deliberately undefaulted: it is
|
||||
|
|
@ -138,7 +138,7 @@ evaluates cleanly points at a real machine that isn't the one you meant.
|
|||
> inside a swarm comes from the swarm root ([`ca.md`](ca.md)): every
|
||||
> hive chains to it, so one anchor replaces per-hive pinning entirely.
|
||||
> What that genuinely drops is trusting a hive whose root this swarm
|
||||
> does *not* own — another swarm's, or one keeping its own CA. That is
|
||||
> does _not_ own — another swarm's, or one keeping its own CA. That is
|
||||
> a cross-swarm problem and wants a mechanism designed for it. (An
|
||||
> earlier `certFingerprint` field existed for exactly that gap, pinning
|
||||
> a peer's TLS leaf for hive-c0re's own peer HTTPS checks — removed
|
||||
|
|
@ -162,7 +162,7 @@ evaluates cleanly points at a real machine that isn't the one you meant.
|
|||
⚠️ **That container currently trusts no swarm-internal CA**, so a
|
||||
self-signed gateway certificate does not federate. The swarm root
|
||||
can't simply be listed there: `security.pki.certificateFiles` is
|
||||
read when the system is *built*, and the root is a runtime file (its
|
||||
read when the system is _built_, and the root is a runtime file (its
|
||||
key must never enter the store), so there is no build-time name for
|
||||
it. Bridging that needs a runtime mechanism and is tracked as its own
|
||||
issue. Until then, federation needs CA-issued certs (ACME). See
|
||||
|
|
@ -184,7 +184,7 @@ swarm.hives = { … }; swarm.hives = { … }; # byte-identical
|
|||
```
|
||||
|
||||
That is the point of the shape, and it removes a class of bug rather
|
||||
than saving typing: a per-host peer list let two hosts hold *different*
|
||||
than saving typing: a per-host peer list let two hosts hold _different_
|
||||
facts about the same third hive — a stale endpoint, a rotated
|
||||
fingerprint — with nothing to detect the disagreement. One entry per
|
||||
hive makes it unrepresentable.
|
||||
|
|
@ -277,7 +277,7 @@ port}` tells this hive where the swarm's `btrfs receive` endpoint is, so
|
|||
`hivectl agent <name> subvol snapshot push` has somewhere to stream to.
|
||||
|
||||
It is genuinely swarm-scoped rather than per-peer — a swarm has exactly
|
||||
one store, because the receiver keys destinations by *agent* so a
|
||||
one store, because the receiver keys destinations by _agent_ so a
|
||||
migrating agent keeps one unbroken incremental chain. See
|
||||
[snapshot-store.md](../networking/snapshot-store.md).
|
||||
|
||||
|
|
@ -305,12 +305,12 @@ A hive publishes only once it has been given the three
|
|||
`swarm.statusPublish` coordinates below. A hive that has not reads
|
||||
`never_reported` — it is not broken, it just has nothing to say upward.
|
||||
|
||||
| freshness | what to do about it |
|
||||
|---|---|
|
||||
| `fresh` | nothing — reported within `staleAfterSeconds` |
|
||||
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
|
||||
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
|
||||
| `unknown` | something is publishing under a name that is not in `swarm.hives` — a typo in the roster, or a hive that was removed and is still running |
|
||||
| freshness | what to do about it |
|
||||
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `fresh` | nothing — reported within `staleAfterSeconds` |
|
||||
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
|
||||
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
|
||||
| `unknown` | something is publishing under a name that is not in `swarm.hives` — a typo in the roster, or a hive that was removed and is still running |
|
||||
|
||||
Every row also carries `last_seen_unix` and `age_seconds` if you want to
|
||||
apply your own threshold. The timestamp is the one the queue recorded on
|
||||
|
|
@ -328,11 +328,11 @@ Three options, on the **hive**, set together or not at all — a
|
|||
half-configured hive is an eval error rather than one that quietly never
|
||||
reports:
|
||||
|
||||
| option | what to set it to |
|
||||
|---|---|
|
||||
| `natsUrl` | where the swarm queue listens, as this hive reaches it |
|
||||
| `tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
|
||||
| `clientSecretFile` | path to this hive's client secret, plaintext |
|
||||
| option | what to set it to |
|
||||
| ------------------ | ------------------------------------------------------ |
|
||||
| `natsUrl` | where the swarm queue listens, as this hive reaches it |
|
||||
| `tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
|
||||
| `clientSecretFile` | path to this hive's client secret, plaintext |
|
||||
|
||||
On a host that runs the queue and the IdP itself, all three default to
|
||||
the local ones and there is nothing to set. Any other hive needs them
|
||||
|
|
@ -364,8 +364,8 @@ on the `agent-configs` org, both under
|
|||
`https://<swarm.domain>/webhook/forge/`.
|
||||
|
||||
The controller **interprets** a delivery and sends hives a specific
|
||||
message — *the knowledge repo changed*, *deploy agent `foo` at rev
|
||||
`abc123`* — rather than forwarding forge payloads for each hive to
|
||||
message — _the knowledge repo changed_, _deploy agent `foo` at rev
|
||||
`abc123`_ — rather than forwarding forge payloads for each hive to
|
||||
re-derive. Approval happens once, at the swarm level: a hive receives a
|
||||
decision, not an event to adjudicate.
|
||||
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue