Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1e0ec9b5af | ||
|
|
3e9d7c68d8 | ||
|
|
6eb43ba2c5 | ||
|
|
c870e230bf | ||
|
|
1cc1ace681 | ||
|
|
62f3cc4b19 | ||
|
|
a17cc3612b | ||
|
|
90be37c0c8 | ||
|
|
3b24d08226 | ||
|
|
f5182d6804 | ||
|
|
cfb50d8a40 | ||
|
|
8361d3f6a1 | ||
|
|
af83a868e4 | ||
|
|
fd891913b0 | ||
|
|
0e84e72c21 | ||
|
|
150cb1be76 | ||
|
|
f41acfcc6e | ||
|
|
066cd712ea | ||
|
|
4388719b76 | ||
|
|
26b55f34d3 | ||
|
|
3e9f2f2fd5 | ||
|
|
723e33ae6c | ||
|
|
720e5340de | ||
|
|
1277f5d4ca | ||
|
|
3e49652511 | ||
|
|
637d36a11f | ||
|
|
efc30a0bcf | ||
|
|
b662bc8578 | ||
|
|
6d74476572 | ||
|
|
0f92e58ebb | ||
|
|
ac58823eb3 | ||
|
|
844337ca91 | ||
|
|
4f03a82cc5 | ||
|
|
dd8b5db7ab | ||
|
|
a7f375190a | ||
|
|
f29df9cadb | ||
|
|
ea18ebeff4 | ||
|
|
e418de0f73 | ||
|
|
c14d932687 | ||
|
|
79a02f6859 | ||
|
|
b02af39007 | ||
|
|
42282410ad | ||
|
|
c681b8a340 | ||
|
|
93bb3a294f | ||
|
|
8a2ffbbef7 | ||
|
|
fea40ed770 | ||
|
|
df81ec46f0 | ||
|
|
b15e0e973c | ||
|
|
fda6a9ac9d | ||
|
|
f104d39c72 | ||
|
|
721a99ed64 | ||
|
|
b1d43fc1e7 | ||
|
|
785b59a4bb | ||
|
|
66442d41ff | ||
|
|
c6eda124b1 | ||
|
|
58459bbbb3 | ||
|
|
9793258955 | ||
|
|
737b6dabd5 | ||
|
|
c570da0932 | ||
|
|
ecdd3d390f | ||
|
|
d14d3b25af | ||
|
|
5df263b202 | ||
|
|
f500863f2f | ||
|
|
e9e4408af1 |
32 changed files with 1340 additions and 1256 deletions
|
|
@ -110,10 +110,10 @@ sudo hivectl matrix create-user mara # provisions a matrix
|
|||
sudo hivectl matrix create-user mara --password-stdin # … reading one line from stdin
|
||||
```
|
||||
|
||||
For agent names (i.e., a `Coordinator::agent_state_root(name)` exists),
|
||||
`hivectl` persists the resulting token to the agent's state dir like the
|
||||
boot sweep does. For non-agent names (e.g. the operator's own forge/matrix
|
||||
account), it prints the token to stdout and writes nothing.
|
||||
For a name that's a managed agent, `hivectl` persists the resulting token
|
||||
to that agent's state dir, the same as the boot sweep does. For a
|
||||
non-agent name (e.g. the operator's own forge/matrix account), it prints
|
||||
the token to stdout and writes nothing.
|
||||
|
||||
## Build / deploy
|
||||
|
||||
|
|
|
|||
|
|
@ -1,11 +1,13 @@
|
|||
# Agent hierarchy & privileges
|
||||
|
||||
Design + audit doc for the agent-privileges + tree-shape milestone
|
||||
(the [issue tree](http://localhost:3000/hyperhive/hyperhive/issues/361)).
|
||||
The implementation lands in pieces; this doc tracks what's done, what's
|
||||
planned, and what currently special-cases the manager.
|
||||
Every agent has a place in an operator-editable parent/child tree, used
|
||||
to scope which agents can manage which others. This doc covers how the
|
||||
tree is stored and edited today, the rules that are meant to run on top
|
||||
of it once enforcement is finished, and where the manager still gets
|
||||
special-cased in the meantime. Tracking issue: hyperhive#361
|
||||
(`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/361`).
|
||||
|
||||
## Current state (as of this PR)
|
||||
## Where the tree lives
|
||||
|
||||
Topology lives in the hive-c0re-owned **meta repo**, alongside
|
||||
`flake.nix`, at `/var/lib/hyperhive/meta/topology.json`:
|
||||
|
|
@ -18,45 +20,35 @@ Topology lives in the hive-c0re-owned **meta repo**, alongside
|
|||
}
|
||||
```
|
||||
|
||||
`null` = root-level agent. New agents **default to root** (`null` parent) —
|
||||
there is no structural manager that everything hangs under. Hierarchy is
|
||||
built explicitly: an agent that requests a sub-agent gets a
|
||||
requester-as-parent edge written at its `init_config` approval (so `bob`
|
||||
above was spawned by `alice`), and the operator can reparent any agent. The
|
||||
bootstrap container (`ruth`) is just another root. Re-parenting is
|
||||
operator-driven:
|
||||
`null` = root-level agent. New agents **default to root** — there is no
|
||||
structural manager that everything hangs under. Hierarchy is built
|
||||
explicitly: an agent that requests a sub-agent gets a
|
||||
requester-as-parent edge written at its `init_config` approval (so
|
||||
`bob` above was spawned by `alice`), and the operator can reparent any
|
||||
agent, including the bootstrap container (`ruth`) — it's just another
|
||||
root. The manager is reparentable like any other agent; there's no
|
||||
"structurally root" carve-out. Its privileges live on its MCP socket,
|
||||
not its tree position (see *Manager special-casing today* below).
|
||||
|
||||
- CLI: `hivectl agent <child> set-parent --parent <new>` (or `--root` to
|
||||
promote). Exactly one of `--parent` / `--root` is required.
|
||||
### Reparenting
|
||||
|
||||
- CLI: `hivectl agent <child> set-parent --parent <new>` (or `--root`
|
||||
to promote). Exactly one of `--parent` / `--root` is required.
|
||||
- Dashboard: `POST /api/topology/set-parent` (form fields `child`,
|
||||
optional `new_parent` — absent / empty ⇒ promote to root).
|
||||
- Wire: `HostRequest::SetParent { child, new_parent: Option<String> }`.
|
||||
|
||||
All three converge on `topology::set_parent`, which delegates the
|
||||
validation rules to a pure `apply_set_parent` helper. Refuses:
|
||||
All three go through the same validation, which refuses:
|
||||
|
||||
- unknown `child` / `new_parent` (typo guard),
|
||||
- self-parenting,
|
||||
- cycles (32-hop ancestor walk, mirroring `is_descendant_of`).
|
||||
- cycles (a bounded ancestor walk — moving the manager under one of
|
||||
its own descendants is the only real safety concern here, and it's
|
||||
caught the same way as any other agent).
|
||||
|
||||
The manager is reparentable like any other agent — there's no
|
||||
"structurally root" carve-out; the manager's privileges live on its
|
||||
MCP socket, not its tree position, and the cycle walk above catches
|
||||
the only real safety concern (moving the manager under one of its
|
||||
own descendants).
|
||||
|
||||
Idempotent no-op fast path skips the disk write when the parent is
|
||||
already what's requested. After a successful write the surfaces call
|
||||
`Coordinator::rescan_containers_and_emit` so connected dashboard
|
||||
viewers see the tree repaint without polling
|
||||
(`ContainerView.parent` is sourced from `topology.json`).
|
||||
|
||||
**Today's caveat:** the move is purely a JSON edit. Only the
|
||||
top-level manager (`root`) gets `/var/lib/hyperhive/agents`
|
||||
bind-mounted at `/agents` in its container, so sub-agents don't yet
|
||||
see their would-be children's state. Once sub-manager bind mounts
|
||||
land alongside cap enforcement, `set_parent` grows a companion
|
||||
umount-old / mount-new / restart-cascade step.
|
||||
Setting a parent to its current value is a no-op (no disk write). A
|
||||
successful change triggers an immediate rescan, so connected dashboard
|
||||
viewers see the tree repaint without polling.
|
||||
|
||||
### Why meta, not per-agent `agent.nix`
|
||||
|
||||
|
|
@ -65,37 +57,39 @@ consent, and operator-driven re-parenting shouldn't require touching
|
|||
the moved agent's config. Topology IS a system-level concern; meta is
|
||||
where system-level facts live.
|
||||
|
||||
### Flow
|
||||
### How `topology.json` gets updated
|
||||
|
||||
1. **Read**: `topology::read()` parses `topology.json` into a
|
||||
`BTreeMap<String, Option<String>>`. Missing / unparsable file →
|
||||
empty map → every agent treated as root (safe degradation for
|
||||
fresh installs that haven't run `meta::sync_agents` yet).
|
||||
2. **Reconcile**: `meta::sync_agents` calls `topology::reconcile`
|
||||
alongside its `flake.nix` regeneration. New agents default to root
|
||||
(null parent) — an agent-requested sub-agent already carries an
|
||||
explicit requester-as-parent edge from its `init_config` approval, so
|
||||
only user/operator-initiated spawns hit this default, and those are
|
||||
roots; removed agents drop. Existing entries are preserved as-is so
|
||||
operator overrides stick across regenerations. Pending-init agents
|
||||
(an operator-approved proposed config repo but no container yet —
|
||||
`Coordinator::pending_init_names`) are kept too, so the
|
||||
`child -> parent` edge written when `request_init_config` is approved
|
||||
survives the gap until the first apply-commit spawns the container.
|
||||
3. **Inject**: `meta::render_flake` looks up each agent's parent and
|
||||
passes it to `mkAgent`. When non-null, the mkAgent body sets
|
||||
`HIVE_PARENT = parent` in the agent's systemd service environment
|
||||
so the harness / claude prompts can see it.
|
||||
4. **Surface**: `container_view::build_all` reads `topology.json` and
|
||||
populates `ContainerView.parent: Option<String>` on every rescan.
|
||||
The dashboard renders the field as a tree.
|
||||
- **Read** — parsed into an agent→parent map; a missing or unparsable
|
||||
file degrades safely to "every agent is root" (covers a fresh
|
||||
install that hasn't synced yet).
|
||||
- **Reconcile** — runs alongside the periodic meta/flake regeneration.
|
||||
New agents default to root unless they already carry an explicit
|
||||
parent edge from an `init_config` approval; existing entries
|
||||
(including operator overrides) are preserved; removed agents drop.
|
||||
Agents that are approved but not yet spawned keep their edge too, so
|
||||
it survives the gap until the container actually appears.
|
||||
- **Inject** — each container's parent (if any) is exposed to its own
|
||||
environment as `HIVE_PARENT`, so the harness / system-prompt
|
||||
renderer can see it.
|
||||
- **Surface** — every rescan re-reads `topology.json` and populates
|
||||
`ContainerView.parent`, which the dashboard renders as a tree.
|
||||
|
||||
## Target topology semantics
|
||||
See `hive-c0re/src/agent_config/topology.rs` and `hive-c0re/src/meta.rs`'s
|
||||
module docs for the exact call chain.
|
||||
|
||||
Once enforcement lands the rules collapse into:
|
||||
### Current limitation: state-dir visibility lags topology
|
||||
|
||||
Reparenting today is purely a JSON edit. Only the top-level manager
|
||||
(`root`) gets `/var/lib/hyperhive/agents` bind-mounted at `/agents` in
|
||||
its container, so sub-agents don't yet see their would-be children's
|
||||
state dirs. Once sub-manager bind mounts land alongside capability
|
||||
enforcement, reparenting will grow a companion
|
||||
umount-old / mount-new / restart-cascade step.
|
||||
|
||||
## Planned topology semantics (once ancestor-based enforcement lands)
|
||||
|
||||
| operation | who can do it |
|
||||
| ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
|
||||
| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
|
||||
| `kill` / `start` / `restart` / `update` (any descendant) | any ancestor |
|
||||
| `request_init_config` (spawn a new child) | any agent, child added under self |
|
||||
| config change via forge PR (any descendant's config) | any ancestor |
|
||||
|
|
@ -105,106 +99,74 @@ Once enforcement lands the rules collapse into:
|
|||
| `request_update_meta_inputs` (bump meta lock) | root agents only (today: just `manager`) |
|
||||
|
||||
"Ancestor" walks `ContainerView.parent` chains; cycles are guarded by a
|
||||
visited-set at dispatch time (a malformed topology.json can't lock the
|
||||
dispatcher into a loop).
|
||||
visited-set at dispatch time (a malformed `topology.json` can't lock
|
||||
the dispatcher into a loop).
|
||||
|
||||
## Current manager special-casings — the audit
|
||||
## Manager special-casing today
|
||||
|
||||
What currently makes the manager different from every other agent, and
|
||||
which axis the post-milestone version reads each special-case along:
|
||||
Enforcement of the ancestor rules above isn't fully wired yet, so the
|
||||
**manager (`ruth`) still gets some hard-coded special treatment**
|
||||
other agents don't:
|
||||
|
||||
### A — naming + bootstrap
|
||||
- **Naming/bootstrap** — the manager's broker recipient name, state-dir
|
||||
key, and nixos-container name are all `ruth` (container `h-ruth`).
|
||||
`hive-c0re` spawns it directly at boot if missing, with no operator
|
||||
approval step — every other agent goes through `request_init_config`
|
||||
→ approval. Topology-wise, `ruth` is still just another root agent.
|
||||
- **Wire-protocol** — the privileged `Request` variants
|
||||
(`RequestInitConfig`; `Kill` / `Start` / `Restart` / `Update`;
|
||||
`GetLogs`; `RequestUpdateMetaInputs`) — marked `*(privileged)*` in
|
||||
`hive-core-agent-sock`'s unified `Request` enum — are reachable only
|
||||
from the manager's socket flavour today. Planned rule for each is in the
|
||||
table above ("any agent, child added under self" for init-config,
|
||||
"any ancestor" for lifecycle/logs); `RequestUpdateMetaInputs` stays
|
||||
a root-only capability even post-milestone, not a topology rule.
|
||||
One exception: `Wake` (inject a `from: <X>` message into the
|
||||
caller's own inbox) isn't really privileged — every per-agent daemon
|
||||
(e.g. `hive-forge-notify`) needs it, and sub-agents already have the
|
||||
equivalent on their own socket.
|
||||
- **Storage/mounts** — only the manager container gets
|
||||
`/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can
|
||||
manage any agent's state dir — config is not authored there, since a
|
||||
real config change is a PR from a clone), plus RO mounts for
|
||||
`/applied` (diff against what's deployed) and `/meta` (system-wide
|
||||
deploy log). Planned: each agent gets RW to `/agents/<descendant>/`
|
||||
for just its own subtree — the manager's full-forest RW becomes the
|
||||
"root's subtree is everything" case of that same rule. RO `/meta`
|
||||
access will be gated on a "meta read" capability; only
|
||||
`request_update_meta_inputs` writes `flake.lock`, gated by its own
|
||||
capability.
|
||||
- **Prompt/tools** — the system prompt uses `<!-- role:agent -->` /
|
||||
`<!-- role:manager -->` marker blocks, and a `Flavor::{Agent,
|
||||
Manager}` switch picks the MCP tool allow-list claude sees. Both are
|
||||
already parametrised on a single flavour value, so the planned
|
||||
per-capability-group version (`cap:<group>` prompt blocks + a
|
||||
matching tool allow-list) is additive rather than a rewrite.
|
||||
- **State dirs** — *not* special-cased: `HYPERHIVE_STATE_DIR` is
|
||||
injected uniformly via `systemd.globalEnvironment` for every
|
||||
container including the manager, so all token/state paths resolve
|
||||
through it the same way everywhere.
|
||||
- **Scattered ownership checks** — a handful of independent
|
||||
manager-only overrides exist across `hive-c0re` today: loose-ends
|
||||
visibility (manager sees hive-wide, sub-agents only their own),
|
||||
"manager can cancel any question/reminder" overrides on the owner
|
||||
check, `destroy` refusing to act on the manager, and crash-watch
|
||||
skipping the manager (it auto-restarts via systemd instead of going
|
||||
through the crash-watch loop). Each is planned to become an
|
||||
ancestor/descendant check instead of a manager-name check — see the
|
||||
module docs for `loose_ends.rs`, `stores/operator_questions.rs`,
|
||||
`stores/broker.rs`, `actions.rs`, and `workers/crash_watch.rs` for
|
||||
the current owner-check logic in each. (Reminder cancellation has
|
||||
since moved fully in-agent — see the note on
|
||||
`CancelLooseEndKind::Reminder` in `hive-c0re/src/questions.rs`.)
|
||||
|
||||
- `MANAGER_AGENT = "ruth"` (broker recipient name),
|
||||
`MANAGER_NAME = "ruth"` (logical name, state-dir key), and
|
||||
`MANAGER_CONTAINER = "h-ruth"` (nixos-container name); the `h-`
|
||||
prefix lets `lifecycle::list()` use a single `starts_with("h-")`
|
||||
filter.
|
||||
- `auto_update::ensure_manager` runs at hive-c0re boot and spawns
|
||||
`h-ruth` if missing. **Topology**: ruth defaults to root-level (no
|
||||
parent); hive-c0re handles the bootstrap lifecycle directly.
|
||||
None of the above is a stable interface — treat the module doc
|
||||
comments as the source of truth for exactly which checks exist today.
|
||||
|
||||
### B — wire-protocol privileges
|
||||
## Future work: sub-agents inside the same container
|
||||
|
||||
The `ManagerRequest::*` variants in `hive-sh4re/src/lib.rs` are
|
||||
operations the manager flavour socket can make that sub-agent sockets
|
||||
can't:
|
||||
|
||||
| variant | semantic | post-milestone |
|
||||
| --------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `RequestInitConfig` | seed an agent's proposed config repo | **topology** — existing direct child (re-init) or a brand-new name (child added under self on approval); a name owned by a different parent is refused |
|
||||
| `Kill` / `Start` / `Restart` / `Update` | container lifecycle on an existing agent | **topology** — descendants only |
|
||||
| `RequestUpdateMetaInputs` | bump meta `flake.lock` | **per-agent cap** (root-only today; a future "let coder bump its own input" might grant it) |
|
||||
| `GetLogs` | journalctl scrape of a sub-agent | **topology** — descendants only |
|
||||
| `Wake` | inject a `from: <X>` message into self's inbox | **not really privileged** — the wire surface exists because the per-agent daemons (e.g. `hive-forge-notify`) need it. Sub-agents have the same via their own socket. |
|
||||
|
||||
### C — storage / mounts (`hive-c0re::lifecycle`)
|
||||
|
||||
The manager container's nspawn bind set:
|
||||
|
||||
- `HOST_AGENTS_ROOT (/var/lib/hyperhive/agents) → /agents` RW — so the
|
||||
manager can manage any agent's state dir. Config is **not** authored
|
||||
here: a config change is a PR from a clone, and `<agent>/config/` is
|
||||
a copy for reading (its write access is a defect tracked separately)
|
||||
- `HOST_APPLIED_ROOT (/var/lib/hyperhive/applied) → /applied` RO — so
|
||||
the manager can diff against what's deployed
|
||||
- `HOST_META_ROOT (/var/lib/hyperhive/meta) → /meta` RO — so the
|
||||
manager can read the system-wide deploy log
|
||||
|
||||
Tree-shape version:
|
||||
|
||||
- Each agent gets RW to `/agents/<descendant>/` for every descendant in
|
||||
its subtree. The root agent (today: manager) gets RW to the full
|
||||
forest as a special case of "the root has every other agent as a
|
||||
descendant".
|
||||
- RO `/meta` access if the agent holds a "meta read" cap.
|
||||
- `request_update_meta_inputs` is the only path that actually writes
|
||||
`flake.lock`, gated by the cap; everyone else stays RO.
|
||||
|
||||
### D — drop legacy `/state` for manager ✓ done
|
||||
|
||||
`lifecycle.rs` no longer binds `/state` for the manager.
|
||||
`HYPERHIVE_STATE_DIR` is now injected uniformly via
|
||||
`systemd.globalEnvironment` in `meta.rs` for every container
|
||||
(manager included), so all token/state paths resolve through
|
||||
`$HYPERHIVE_STATE_DIR`. The agent-module shell scripts
|
||||
(tea-login, forge-avatar-sync) simplified from glob+for loops to a
|
||||
direct `$HYPERHIVE_STATE_DIR/<token>` read.
|
||||
|
||||
### E — prompt + tools
|
||||
|
||||
- `prompts/system.md` with `<!-- role:agent -->` / `<!-- role:manager -->`
|
||||
marker blocks, assembled by `hive_ag3nt::prompt::render` based on
|
||||
flavor. **Per-agent cap list** of what the agent can do — already
|
||||
a single parametrised prompt; once per-agent cap groups land the
|
||||
marker grammar grows `cap:<group>` blocks the renderer reads from
|
||||
the per-agent ToolGroup set.
|
||||
- `mcp.rs::Flavor::{Agent, Manager}` controls which MCP tools claude
|
||||
sees. Already structured this way internally — the per-flavour
|
||||
allow-list becomes a per-cap-set lookup.
|
||||
|
||||
### F — drive-by checks across c0re
|
||||
|
||||
- `loose_ends.rs`: manager sees hive-wide loose-ends, sub-agents only
|
||||
their own. **Topology** — every agent sees its own + its
|
||||
descendants'.
|
||||
- `operator_questions.rs` + `broker.rs`: "manager can cancel any
|
||||
question" override on the owner check. **Topology** — agents can
|
||||
moderate threads of their descendants.
|
||||
- `reminder_scheduler.rs`: same override pattern for reminder cancel.
|
||||
**Topology** — descendants only.
|
||||
- `actions.rs`: `destroy` refuses to act on `MANAGER_NAME` (no
|
||||
foot-shooting). **Topology** — agents can destroy descendants but
|
||||
never themselves or ancestors.
|
||||
- `crash_watch.rs`: skips `ContainerCrash` for the manager (it
|
||||
auto-restarts via systemd). **Topology** — the root container has
|
||||
different recovery semantics, every other agent falls into the same
|
||||
watch loop.
|
||||
|
||||
### G — sub-agents inside the same container
|
||||
|
||||
Future work: when enabled for an agent, it can spawn temporary
|
||||
"sub-agents" that run inside its own container. Lighter than a full
|
||||
When enabled for an agent, it will be able to spawn temporary
|
||||
"sub-agents" that run inside its own container — lighter than a full
|
||||
nspawn agent. Open questions, not yet wired:
|
||||
|
||||
- Inherit caps from parent, or take an explicit narrower set?
|
||||
|
|
@ -216,17 +178,16 @@ nspawn agent. Open questions, not yet wired:
|
|||
## Harness systemd unit shape
|
||||
|
||||
One harness serve binary (`hive-agent`, with its `hive-agent-mcp`
|
||||
sibling), one shared `nix/agent-modules/` tree, one
|
||||
service unit (`systemd.services.hive-agent`) for all agents. There
|
||||
is no longer a separate manager service name or role distinction in
|
||||
the harness — privilege differences live server-side in the broker
|
||||
socket (which tool groups and manager-surface calls each agent
|
||||
receives).
|
||||
sibling), one shared `nix/agent-modules/` tree, one service unit
|
||||
(`systemd.services.hive-agent`) for all agents. There is no separate
|
||||
manager service name or role distinction in the harness — privilege
|
||||
differences live server-side in the broker socket (which tool groups
|
||||
and manager-surface calls each agent receives).
|
||||
|
||||
`agent.nix` and `ruth.nix` both import the shared `nix/agent-modules/`.
|
||||
`ruth.nix` additionally sets forge defaults to suppress the
|
||||
subscription/participation firehose so ruth's inbox stays focused
|
||||
on direct mentions, reviews, and assignments.
|
||||
subscription/participation firehose so ruth's inbox stays focused on
|
||||
direct mentions, reviews, and assignments.
|
||||
|
||||
### Environment variables set on the unit
|
||||
|
||||
|
|
@ -248,36 +209,37 @@ on direct mentions, reviews, and assignments.
|
|||
path = [ "/run/wrappers" "/run/current-system/sw" ];
|
||||
```
|
||||
|
||||
`/run/wrappers` comes first so setuid wrappers (notably `sudo`)
|
||||
resolve before bare nix-store binaries. NixOS's
|
||||
`systemd.services.<unit>.path` appends `/bin` to every entry via
|
||||
`lib.makeBinPath`; passing `/run/wrappers/bin` directly produces
|
||||
`/run/wrappers/bin/bin` which doesn't exist (`docs/gotchas.md::
|
||||
systemd.services.*.path appends /bin to every entry`). With the
|
||||
harness running as the per-agent user this matters: without the
|
||||
wrapper dir on PATH, `sudo` resolves to the un-setuid nix-store
|
||||
binary and rejects with `must be owned by uid 0 and have the setuid
|
||||
bit set` regardless of `hyperhive.user.passwordlessSudo`.
|
||||
`/run/wrappers` (not `/run/wrappers/bin`) comes first so setuid
|
||||
wrappers — notably `sudo` — resolve before bare nix-store binaries; see
|
||||
[`docs/gotchas.md`](gotchas.md) ("`systemd.services.*.path` appends
|
||||
`/bin` to every entry") for why the trailing `/bin` matters in
|
||||
general. It's load-bearing here because the harness runs as the
|
||||
per-agent user: without the wrapper dir on `PATH`, `sudo` resolves to
|
||||
the non-setuid nix-store binary and every
|
||||
`hyperhive.user.passwordlessSudo` grant fails with "must be owned by
|
||||
uid 0 and have the setuid bit set."
|
||||
|
||||
### `serviceConfig` highlights
|
||||
|
||||
- `ExecStart = pkgs.hyperhive/bin/hive-agent` — same binary for
|
||||
every agent.
|
||||
- `ExecStart = pkgs.hyperhive/bin/hive-agent` — same binary for every
|
||||
agent.
|
||||
- `Restart = on-failure`, `RestartSec = 2` — keeps the harness
|
||||
resilient across transient crashes without thundering retries.
|
||||
- `RuntimeDirectory = "hive-config"` → `/run/hive-config/` owned by
|
||||
`User=`, auto-cleared on stop. The harness writes regenerated
|
||||
`claude-{mcp-config,settings,system-prompt}` files there
|
||||
(`paths::config_dir`). Deliberately separate from `/run/hive`,
|
||||
which the host bind-mounts in root-owned and which holds
|
||||
hive-c0re's `mcp.sock`.
|
||||
- `User = Group = userName` — drops root inside the container; sudo
|
||||
is the explicit escalation surface
|
||||
(`hyperhive.user.passwordlessSudo`).
|
||||
(`paths::config_dir`). Deliberately separate from `/run/hive`, which
|
||||
the host bind-mounts in root-owned and which holds hive-c0re's
|
||||
`mcp.sock`.
|
||||
- `User = Group = userName` — drops root inside the container; sudo is
|
||||
the explicit escalation surface (`hyperhive.user.passwordlessSudo`).
|
||||
|
||||
## Cross-references
|
||||
|
||||
- Milestone: ["Agent privileges and sub-agents"](http://localhost:3000/hyperhive/hyperhive/issues/361)
|
||||
- Dashboard render: ["show agent topology in container list"](http://localhost:3000/hyperhive/hyperhive/issues/363)
|
||||
- Audit table source: [milestone comment](http://localhost:3000/hyperhive/hyperhive/issues/361#issuecomment-3335)
|
||||
- Milestone: "Agent privileges and sub-agents"
|
||||
(`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/361`)
|
||||
- Dashboard render: "show agent topology in container list"
|
||||
(`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/363`)
|
||||
- Audit table source: milestone comment
|
||||
(`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/361#issuecomment-3335`)
|
||||
- Operator/agent trust boundary (orthogonal axis): [`boundary.md`](boundary.md)
|
||||
|
|
|
|||
|
|
@ -212,10 +212,9 @@ An agent target's delivery is `push_todo` (`Coordinator::push_todo`,
|
|||
`docs/coordinator.md` covers the mechanism generally), not a broker
|
||||
`Message` — a scheduled prompt wakes its target with a todo instead of
|
||||
driving an immediate turn, by design. `key = "schedule:<id>"` per
|
||||
target gives `push_todo`'s own upsert-by-key dedup the job a
|
||||
now-removed `has_pending_with_body` broker check used to do: a re-fire
|
||||
of the *same schedule* against a target that hasn't reviewed the last
|
||||
one collapses into that one todo instead of stacking up.
|
||||
target drives `push_todo`'s own upsert-by-key dedup: a re-fire of the
|
||||
*same schedule* against a target that hasn't reviewed the last one
|
||||
collapses into that one todo instead of stacking up.
|
||||
|
||||
**`operator` is the one exception** — it's a valid schedule target but
|
||||
has no in-container todo inbox, so it keeps the original broker
|
||||
|
|
@ -290,8 +289,7 @@ declares one flake input per agent (`agent-<n>.url =
|
|||
"git+http://<forge>/agent-configs/<n>.git"`) and one
|
||||
`nixosConfigurations.<n>` output per agent. Each output wraps
|
||||
`inputs.agent-<n>.nixosModules.default` with the identity +
|
||||
`HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection
|
||||
module that `setup_applied` used to generate inline.
|
||||
`HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection module.
|
||||
Containers run against `--flake /var/lib/hyperhive/meta#<n>`.
|
||||
|
||||
The declared input url is the agent's **forge config repo** (the
|
||||
|
|
@ -345,9 +343,10 @@ and `meta::lock_update_hyperhive()` for the
|
|||
auto-update flake-rev bump (one shot before per-agent
|
||||
rebuilds, commits if the lock changed).
|
||||
|
||||
`meta::sync_agents(hyperhive_flake, dashboard_port, &agents)`
|
||||
is the idempotent reconciler called by `spawn`, `destroy`,
|
||||
`rebuild`, and the startup migration. Renders `flake.nix`
|
||||
`meta::sync_agents(hive: &HiveEnv, agents: &[AgentSpec])` — `hive`
|
||||
carries `hyperhive_flake`, `dashboard_port`, and the rest of the
|
||||
per-hive config — is the idempotent reconciler called by `spawn`,
|
||||
`destroy`, `rebuild`, and the startup migration. Renders `flake.nix`
|
||||
from the agent list; if it differs from disk, runs
|
||||
`nix flake lock` + commits as `regenerate meta flake` (or
|
||||
`seed meta from N agent(s)` on the very first call).
|
||||
|
|
@ -419,9 +418,9 @@ submitter pushes again (or closes it) to retry.
|
|||
### Dispatch via the job queue
|
||||
|
||||
Long-running approval work — `MergeConfigPr`, `UpdateMetaInputs`,
|
||||
`Spawn` — no longer runs inline inside `actions::approve`. Instead
|
||||
the approval handler submits a DAG to the global job queue
|
||||
(`docs/coordinator.md::Job queue`):
|
||||
`Spawn` — runs as a DAG on the global job queue
|
||||
(`docs/coordinator.md::Job queue`), submitted by the approval handler
|
||||
rather than run inline:
|
||||
|
||||
| `ApprovalKind` | DAG submitted | source |
|
||||
|---|---|---|
|
||||
|
|
@ -491,8 +490,9 @@ approval card. See `docs/web-ui.md`.
|
|||
### Submitting agent's view of config repos
|
||||
|
||||
Every parent agent's container has its **direct children's** config
|
||||
repos bind-mounted **read-only** (topology-driven: `lifecycle.rs` calls
|
||||
`bind_child_agent_dirs` for each entry in
|
||||
repos bind-mounted **read-only** (topology-driven:
|
||||
`hive-c0re/src/lifecycle/host_config.rs` calls `bind_child_agent_dirs`
|
||||
for each entry in
|
||||
`topology::children_of(agent_name)`). It is a copy to *read* a child's
|
||||
current config — not an editing surface.
|
||||
|
||||
|
|
@ -502,9 +502,9 @@ forge into its own state dir, commit on a branch, open a PR**, and let
|
|||
the operator review and approve it. There is deliberately no second,
|
||||
mount-shaped path that reaches the same file without the review.
|
||||
|
||||
Agents holding the `can_manage_top_level_agents` topology role
|
||||
(defined as `ROLE_CAN_MANAGE_TOP_LEVEL_AGENTS` in `hive-c0re/src/agent_config/topology.rs`)
|
||||
get additional host-side bind mounts via `set_nspawn_flags`:
|
||||
Agents holding the `can_manage_top_level_agents` topology role (see
|
||||
`hive-c0re/src/agent_config/topology.rs`) get additional host-side
|
||||
bind mounts via `set_nspawn_flags`:
|
||||
|
||||
- `/var/lib/hyperhive/agents/` → `/agents/` (RW) — all top-level
|
||||
agents' proposed repos (not just direct children).
|
||||
|
|
@ -536,34 +536,24 @@ cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent
|
|||
The RO binds block push at the kernel level — git plumbing inside the
|
||||
container cannot corrupt either authoritative repo.
|
||||
|
||||
## Migration from the pre-tag / pre-meta schemes
|
||||
## Startup migrations (older hosts)
|
||||
|
||||
Both overhauls (tag-driven flow + meta flake) ship in-place
|
||||
migrations that run on every hive-c0re startup. Idempotent;
|
||||
each phase is a no-op once already applied. Behaviour:
|
||||
hive-c0re runs a couple of idempotent migrations on every startup so a
|
||||
host set up before the tag-driven-deploy + meta-flake scheme (both
|
||||
described above) converges to it automatically. Each phase is a no-op
|
||||
once already applied:
|
||||
|
||||
- Tag-driven phase: assumes the operator ran the one-shot
|
||||
`git tag deployed/0 main` script (see commit history /
|
||||
earlier docs revisions) once per agent. Tagging is
|
||||
non-destructive: it doesn't touch live containers, state
|
||||
dirs, or claude creds.
|
||||
- Meta-flake phase: rewrites each `applied/<n>/flake.nix` to
|
||||
the module-only boilerplate, wires the `applied` remote in
|
||||
each proposed repo, and bootstraps the meta repo from the
|
||||
current agent list. Set `HIVE_SKIP_META_MIGRATION=1` on the
|
||||
service to defer.
|
||||
- **Tags**: agents from before the tag-driven scheme are tagged
|
||||
`deployed/0` on `main` once. Non-destructive — it doesn't touch live
|
||||
containers, state dirs, or claude creds.
|
||||
- **Meta flake**: rewrites each `applied/<n>/flake.nix` to the
|
||||
module-only boilerplate, wires the `applied` remote in each proposed
|
||||
repo, and bootstraps the meta repo from the current agent list. Set
|
||||
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
|
||||
|
||||
A further step used to `nixos-container update` every
|
||||
container onto `meta#<n>`, guarded by a marker file so it
|
||||
ran once per hive. It is gone: containers have been rendered
|
||||
onto `meta#<n>` at creation for long enough that no live hive
|
||||
needs the repoint, and a one-shot nobody can still trigger is
|
||||
dead weight. Same for the `root` → `h-root` container rename.
|
||||
|
||||
No state loss in either migration. claude creds, /state/
|
||||
notes, the events DB, proposed history, and applied history
|
||||
all survive. The root agent keeps its session; sub-agents stay
|
||||
logged in.
|
||||
No state loss in either migration: claude creds, `/state/` notes, the
|
||||
events DB, and both proposed + applied history all survive. The root
|
||||
agent keeps its session; sub-agents stay logged in.
|
||||
|
||||
## The root/bootstrap container is hive-c0re-managed
|
||||
|
||||
|
|
@ -578,7 +568,7 @@ same as any other agent.
|
|||
|
||||
Differences from sub-agents:
|
||||
|
||||
- `flake.nix` extends `hyperhive.nixosConfigurations.manager`
|
||||
- `flake.nix` extends `hyperhive.nixosConfigurations.ruth`
|
||||
(vs `agent-base`).
|
||||
- Web UI port via `lifecycle::agent_web_port("ruth")` — same
|
||||
FNV-1a hash as every other agent (8100..8999 range).
|
||||
|
|
@ -589,8 +579,10 @@ Differences from sub-agents:
|
|||
authoritative applied repo (see "Root-agent view of applied" below).
|
||||
- First-deploy spawn bypasses the approval queue (the root agent is
|
||||
required infrastructure).
|
||||
- Per-agent socket lives at `/run/hyperhive/manager/`, owned by
|
||||
`manager_server::start`.
|
||||
- The root agent's socket is bound by `socket_server::start_manager`,
|
||||
pure transport with no dedicated helpers — it uses the same
|
||||
per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
|
||||
not a special manager-only path.
|
||||
|
||||
**Migration note** (for older hosts): drop any `containers.root =
|
||||
{ ... }` block from your host NixOS config. hyperhive creates and
|
||||
|
|
@ -598,56 +590,35 @@ updates the root agent itself.
|
|||
|
||||
## Root-agent policy
|
||||
|
||||
The system prompt (`hive-ag3nt/prompts/system.md`, rendered via
|
||||
`hive_ag3nt::prompt::render`) is the **same for every agent**; what
|
||||
The system prompt (`hive-agent/prompts/system.md`, rendered by
|
||||
`hive-agent/src/prompt.rs`) is the **same for every agent**; what
|
||||
varies is which MCP tools are surfaced (gated by tool groups and
|
||||
capabilities in `agent.nix`). There is no `role:manager` block that
|
||||
renders only for the root agent. The root agent's approval-gating
|
||||
behaviour comes from its CLAUDE.md / agent-specific instructions, not
|
||||
the system prompt template.
|
||||
|
||||
`ask(question, options?, multi?, ttl_seconds?, to?)` is available to
|
||||
**any agent** — it queues a question and returns the id immediately.
|
||||
When `to` is omitted (or `"operator"`) the question shows up on the
|
||||
dashboard; when `to` is another agent's name, the recipient receives a
|
||||
`HelperEvent::QuestionAsked` and answers via their own `answer`
|
||||
tool. Either way the answer arrives back as
|
||||
`HelperEvent::QuestionAnswered { id, question, answer, answerer }`
|
||||
in the asker's inbox. Storage is `hive-c0re::operator_questions`
|
||||
(sqlite) — same table, with a nullable `target` column
|
||||
(NULL = operator). Dispatch goes through
|
||||
`hive-c0re/src/questions.rs::{handle_ask, handle_answer}`. The answer flow is:
|
||||
|
||||
```
|
||||
POST /answer-question/{id} agent: Answer { id, answer }
|
||||
→ OperatorQuestions::answer(_, _, "operator") → questions::handle_answer
|
||||
→ notify_agent(asker, QuestionAnswered { → OperatorQuestions::answer(_, _, agent)
|
||||
answerer: "operator", ... }) → notify_agent(asker, QuestionAnswered {
|
||||
answerer: agent, ... })
|
||||
```
|
||||
|
||||
Two more paths resolve a pending question with a sentinel answer:
|
||||
|
||||
- `POST /cancel-question/{id}` (✗ CANC3L button on the dashboard)
|
||||
resolves with `[cancelled]`. The asking agent sees a terminal state
|
||||
and can fall back.
|
||||
- `ttl_seconds` deadline: a tokio watchdog spawned at submit time
|
||||
fires `answer(id, "[expired]")` once the ttl runs out. Already-
|
||||
resolved races no-op. The dashboard surfaces a `⏳ MM:SS` chip
|
||||
on each pending question with a deadline.
|
||||
Any agent (root or not) can also ask a structured question of the
|
||||
operator or a peer agent via the `ask`/`answer` MCP tools, independent
|
||||
of the approval flow above — see
|
||||
`docs/conventions.md#question-routing-ask--answer` for the routing
|
||||
rules, `ttl_seconds` expiry, and cancellation.
|
||||
|
||||
## Helper events to the submitting agent
|
||||
|
||||
`Coordinator::notify_submitter(approval_id, &HelperEvent)` routes the
|
||||
event to the agent that originally submitted the approval (looked up from
|
||||
the `submitter` column on the `approvals` table). The harness delivers it
|
||||
as a regular `system` inbox message so it drives a normal claude turn. A
|
||||
as a regular `system` inbox message so it drives a normal claude turn.
|
||||
`finish_approval` fires an `ApprovalResolved` HelperEvent this way for
|
||||
**every** approval kind's terminal state, `Spawn` included. A
|
||||
"FYI, check when convenient" event doesn't need a message — those go
|
||||
through `Coordinator::push_todo`/`push_todo_submitter` instead, a direct
|
||||
live dial of the target agent's in-container todo socket (same
|
||||
`UpsertTodo` request in-container producers use); the `Spawn` approval
|
||||
uses this path, not a `HelperEvent`. Legacy approval rows that predate the
|
||||
submitter column fall back to the
|
||||
`UpsertTodo` request in-container producers use); `finish_approval` fires
|
||||
one of these too for `InitConfig`/`Spawn`/`MergeConfigPr`, *in addition to*
|
||||
the `ApprovalResolved` HelperEvent above, not instead of it. Legacy
|
||||
approval rows that predate the submitter column fall back to the
|
||||
root agent. Variants (`hive_sh4re::manager::HelperEvent`):
|
||||
|
||||
- `ApprovalResolved { id, agent, commit_ref, status, note }` —
|
||||
|
|
@ -673,25 +644,24 @@ root agent. Variants (`hive_sh4re::manager::HelperEvent`):
|
|||
The recipient responds via `Answer { id, answer }` and the
|
||||
asker sees the matching `QuestionAnswered`.
|
||||
|
||||
The rest of the original lifecycle notices — `Rebuilt`, `Killed`,
|
||||
`Destroyed`, `NeedsLogin`, `LoggedIn`, `ConfigReady` — were pure "FYI,
|
||||
check when convenient" events with no reason to drive an immediate
|
||||
turn, so they've been migrated off `HelperEvent` onto
|
||||
`push_todo`/`push_todo_submitter` (see above): `agent_todo_socket`
|
||||
push instead of a broker message, `subsystem = "core"`,
|
||||
`key = "<event>:<agent>"` for dedup, one free-text `summary` in place
|
||||
of the old structured fields (`rebuilt_todo_summary` renders
|
||||
`Rebuilt`'s former `ok`/`note`/`sha`/`tag` into that string).
|
||||
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
|
||||
`Destroyed`, `NeedsLogin`, `LoggedIn`, `ConfigReady` — are "FYI, check
|
||||
when convenient" events with no reason to drive an immediate turn, so
|
||||
they deliver via `push_todo`/`push_todo_submitter` (see above) instead
|
||||
of `HelperEvent`: an `agent_todo_socket` push instead of a broker
|
||||
message, `subsystem = "core"`, `key = "<event>:<agent>"` for dedup,
|
||||
and a single free-text `summary` (`rebuilt_todo_summary` renders
|
||||
`Rebuilt`'s `ok`/`note`/`sha`/`tag` fields into that string).
|
||||
|
||||
Optional `sha` field on `ApprovalResolved` carries the canonical
|
||||
hive-c0re-vouched commit sha. Optional `tag` carries the deploy
|
||||
bookkeeping tag — `deployed/<id>` on a successful build or
|
||||
`failed/<id>` on a failed one, planted by the `MergeConfigPr` deploy.
|
||||
Both fields are `Option`: `None` on the paths that don't deploy a new
|
||||
commit (spawn / init_config / meta-update / deny, and
|
||||
`auto_update::rebuild_agent` reapplying the existing main, or the
|
||||
dashboard `↻ R3BU1LD` button when the lock didn't move). When set,
|
||||
`git show <sha>` against `/agents/<n>/applied.git` inside the
|
||||
commit (spawn / init_config / meta-update / deny, and the auto-update
|
||||
sweep's `job_queue::templates::rebuild` reapplying the existing main,
|
||||
or the dashboard `↻ R3BU1LD` button when the lock didn't move). When set,
|
||||
`git show <sha>` against `/applied/<n>/.git` inside the
|
||||
bootstrap container yields the exact tree that was referenced.
|
||||
|
||||
To add a new lifecycle notice: if it needs to drive an immediate turn
|
||||
|
|
@ -706,8 +676,8 @@ no new wire type needed.
|
|||
`hive-c0re serve` runs `auto_update::run` in a background task right
|
||||
after opening the coordinator. It enumerates managed containers and
|
||||
rebuilds any whose recorded hyperhive rev differs from the current
|
||||
one — sub-agents and the root agent go through the same `lifecycle::rebuild`
|
||||
path.
|
||||
one — sub-agents and the root agent go through the same
|
||||
`job_queue::templates::rebuild` DAG.
|
||||
|
||||
"Rev" = canonical filesystem path of `cfg.hyperhiveFlake`. Marker
|
||||
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`. If the
|
||||
|
|
@ -716,8 +686,8 @@ auto-update is a no-op — rebuild manually.
|
|||
|
||||
The dashboard surfaces pending updates per agent: a clickable
|
||||
"needs update ↻" badge appears whenever the marker differs from
|
||||
current rev. The badge POSTs `/api/rebuild/<name>`, calling the same
|
||||
`auto_update::rebuild_agent` path so manual triggers and the
|
||||
current rev. The badge POSTs `/api/rebuild/<name>`, which inserts the
|
||||
same `job_queue::templates::rebuild` DAG so manual triggers and the
|
||||
startup scan can't drift. When at least one container is stale, a
|
||||
top-level `↻ UPD4TE 4LL` button appears that loops over every
|
||||
stale container.
|
||||
|
|
|
|||
|
|
@ -5,12 +5,10 @@ _implementation_ work — container network isolation, the unifying
|
|||
gateway, core-daemon privsep — is tracked as `area:ops` issues on
|
||||
the forge.
|
||||
|
||||
The operator/agent boundary is now technically enforced, not just a
|
||||
convention. Containers run in private netns (network isolation is
|
||||
The operator/agent boundary is technically enforced, not just a
|
||||
convention: containers run in private netns (network isolation is
|
||||
always on), the gateway proxies all operator-facing traffic, and
|
||||
`hive-c0re` runs as the unprivileged `hive-core` user. All three
|
||||
`area:ops` pillars — network isolation, the gateway, and privsep —
|
||||
are complete and active.
|
||||
`hive-c0re` runs as the unprivileged `hive-core` user.
|
||||
|
||||
## Two principals, two paths
|
||||
|
||||
|
|
@ -31,9 +29,10 @@ are complete and active.
|
|||
point.** They live on the core backend.
|
||||
|
||||
Worked example — answering an operator-targeted question is a
|
||||
`POST /answer-question/{id}` on the core dashboard, _never_ an
|
||||
`AgentRequest` variant. If it were a per-agent-socket request, an
|
||||
agent could `curl` its own socket and spoof an operator answer.
|
||||
`POST /api/answer-question/{id}` on the core dashboard, _never_ a
|
||||
per-agent-socket `Request` variant. If it were a per-agent-socket
|
||||
request, an agent could `curl` its own socket and spoof an operator
|
||||
answer.
|
||||
The per-agent web UI POSTs cross-origin to the core for these
|
||||
(see the inline-answer feature — the loose-ends section on each
|
||||
agent page).
|
||||
|
|
@ -47,9 +46,9 @@ every boundary claim above is aspirational. Network isolation is what
|
|||
makes the boundary _real_; the gateway and privsep are ergonomics and
|
||||
defence-in-depth layered on top.
|
||||
|
||||
Network isolation is now complete and always on: every agent container
|
||||
runs in a private netns behind the hive bridge. The shared-netns mode
|
||||
was removed. See `docs/network.md`.
|
||||
Network isolation is complete and always on: every agent container
|
||||
runs in a private netns behind the hive bridge, and there is no
|
||||
shared-netns mode. See `docs/network.md`.
|
||||
|
||||
Concretely, the core daemon's dashboard `/api` carries **no
|
||||
application-layer authentication** — operator-authority routes are served
|
||||
|
|
@ -64,15 +63,18 @@ operator-authority route inherits that assumption. `hive-ci` is treated like an
|
|||
agent for this purpose — it runs untrusted PR code and is netns-isolated for
|
||||
the same reason.
|
||||
|
||||
The `area:ops` issues followed this sequencing:
|
||||
The boundary rests on three layers:
|
||||
|
||||
1. **Gateway** — pure ergonomics win, unblocks same-origin (lets the
|
||||
cross-origin CORS shim on `/answer-question/{id}` go away), no
|
||||
behavioural risk. An nginx nixos-container now sits in front of all
|
||||
surfaces; per-agent UIs are proxied under `/agent/<name>/`.
|
||||
2. **Network isolation** — the load-bearing step that turns the
|
||||
honour-system split into an enforced boundary. **Complete** —
|
||||
always-on, unconditional; the shared-netns mode was removed.
|
||||
1. **Gateway** — fronts all surfaces (dashboard + every per-agent UI)
|
||||
on one origin. An nginx nixos-container proxies per-agent UIs under
|
||||
`/agent/<name>/`, which is what lets the inline-answer POST to
|
||||
`/answer-question/{id}` go same-origin instead of needing a
|
||||
cross-origin CORS shim. Pure ergonomics — no behavioural risk on
|
||||
its own.
|
||||
2. **Network isolation** — the load-bearing layer: every agent
|
||||
container runs in a private netns behind the hive bridge, always
|
||||
on and unconditional. This is what turns the operator/agent split
|
||||
from an honour-system convention into an enforced boundary.
|
||||
3. **Privsep** — defence in depth on the core process; `hive-c0re`
|
||||
runs as the unprivileged `hive-core` user and delegates root
|
||||
operations to `hive-priv`, a narrow socket-activated helper. See
|
||||
|
|
@ -84,15 +86,14 @@ The `area:ops` issues followed this sequencing:
|
|||
systemd unit. The unit binds `/run/hive/priv.sock` with
|
||||
`SocketGroup=hive-core` and mode `0660` and passes the ready listener
|
||||
to the helper as fd 3 (`LISTEN_FDS`). The helper requires this and
|
||||
bails if it isn't socket-activated — there is intentionally no
|
||||
self-bind fallback.
|
||||
bails if it isn't socket-activated.
|
||||
|
||||
Dropping the old fallback removed a dev/prod divergence: when
|
||||
`hive-priv` bound the socket itself it created the file owned by
|
||||
root's primary group rather than `hive-core`, so a `hive-core` client
|
||||
couldn't connect the way the socket unit's `SocketGroup` grant
|
||||
intends. Requiring socket activation everywhere means dev and prod
|
||||
take the exact same path and the group grant always holds.
|
||||
⚠️ There is intentionally no self-bind fallback: if `hive-priv` bound
|
||||
the socket itself, it would create the file owned by root's primary
|
||||
group rather than `hive-core`, and a `hive-core` client couldn't
|
||||
connect the way the socket unit's `SocketGroup` grant intends.
|
||||
Requiring socket activation everywhere keeps dev and prod on the
|
||||
exact same path, so the group grant always holds.
|
||||
|
||||
### the per-agent socket dir
|
||||
|
||||
|
|
@ -129,10 +130,9 @@ nginx reaches all of `/run/hive-agent` as a plain host path. Dropping
|
|||
them and the rest of the host. That costs no network isolation: nginx
|
||||
binds the host's `:80`/`:443` and reaches `localhost` upstreams, which a
|
||||
netns would have to be opened up for anyway.
|
||||
🔑 It does mean nothing *implicitly* scopes the privileged reload verb,
|
||||
so the scope is explicit: the unit name is hard-coded in `hive-priv` —
|
||||
see `PrivRequest::ReloadGatewayNginx`. **A caller cannot name the unit,
|
||||
so the verb cannot be steered at another service.**
|
||||
🔑 It does mean nothing *implicitly* scopes the privileged reload verb —
|
||||
see [`docs/security.md`](security.md#hive-c0re-privilege-separation) for
|
||||
how `PrivRequest::ReloadGatewayNginx`'s containment works.
|
||||
|
||||
⚠️ Contrast `/shared`, which *is* sticky world-writable (`1777`): it has
|
||||
many legitimate writers, so sticky is the best available answer there.
|
||||
|
|
|
|||
59
docs/ci.md
59
docs/ci.md
|
|
@ -5,7 +5,7 @@ executing CI jobs from `.forgejo/workflows/ci.yml` on every PR.
|
|||
|
||||
## For operators
|
||||
|
||||
**Enabling it is one line**: `services.hyperhive.forge.ci.enable = true`
|
||||
**Enabling it is one line**: `services.hyperhive.swarm.forge.ci.enable = true`
|
||||
in the host NixOS config. No manual token provisioning — hive-c0re
|
||||
registers the runner with the forge automatically.
|
||||
|
||||
|
|
@ -31,17 +31,19 @@ writeup.
|
|||
|
||||
## CI checks
|
||||
|
||||
Three jobs run on every PR (and on `workflow_dispatch` for manual re-triggers):
|
||||
Three jobs run on every PR (and on `workflow_dispatch` for manual re-triggers),
|
||||
defined in [`.forgejo/workflows/ci.yml`](../.forgejo/workflows/ci.yml). All
|
||||
three are required checks (forge branch protection) — a hit on any of them
|
||||
blocks merge.
|
||||
|
||||
| Job | What it runs | Currently required |
|
||||
| --- | --- | --- |
|
||||
| **nix flake check** | treefmt + rustfmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation | yes |
|
||||
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) | no (red, non-blocking) |
|
||||
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) | no (red, non-blocking) |
|
||||
| Job | What it runs |
|
||||
| --- | --- |
|
||||
| **nix flake check** | treefmt + rustfmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation |
|
||||
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) |
|
||||
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) |
|
||||
|
||||
The tracker-tag and comment-block checks are non-blocking today (a hit fails the
|
||||
check but does not prevent merge) while the legacy backlog is cleaned up. They are
|
||||
expected to become required checks once the tree is clean.
|
||||
`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
|
||||
without an empty commit.
|
||||
|
||||
### Running checks locally
|
||||
|
||||
|
|
@ -68,14 +70,17 @@ diagnostic if either fails — catching the issue locally before CI sees it.
|
|||
Note that the hook does **not** run `cargo clippy` or `cargo test` (those are
|
||||
slow); run those manually before pushing Rust changes.
|
||||
|
||||
## Operator bootstrap
|
||||
## Configuration reference
|
||||
|
||||
Set `services.hyperhive.forge.ci.enable = true` in the host NixOS config. That's it — no manual token provisioning.
|
||||
The internal forge is always present (mandatory), so the runner always has a
|
||||
hive-forge instance to register against — nothing extra to enable beyond
|
||||
`services.hyperhive.swarm.forge.ci.enable = true` (see *For operators* above).
|
||||
|
||||
**Requirements:**
|
||||
|
||||
- The internal forge is always present (mandatory), so the runner always has a hive-forge instance to register against — nothing extra to enable.
|
||||
- Optional: tune `services.hyperhive.forge.ci.name` (runner name in forge admin panel), `concurrency` (parallel job capacity), `labels` (workflow targeting), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration string e.g. `"3h"` — a job that exceeds it is killed so a hung or runaway build can't hold the runner's single slot indefinitely).
|
||||
Optional tuning: `services.hyperhive.swarm.forge.ci.name` (runner name in forge
|
||||
admin panel), `concurrency` (parallel job capacity), `labels` (workflow
|
||||
targeting), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration
|
||||
string e.g. `"3h"` — a job that exceeds it is killed so a hung or runaway
|
||||
build can't hold the runner's single slot indefinitely).
|
||||
|
||||
## Container design
|
||||
|
||||
|
|
@ -86,7 +91,14 @@ Set `services.hyperhive.forge.ci.enable = true` in the host NixOS config. That's
|
|||
|
||||
## Auto-registration flow
|
||||
|
||||
Registration is **off the container's boot-critical path** — hive-c0re owns it and runs it out of band, so a slow forge or core-token never delays the container's start. (The earlier design ran a host-side `hive-ci-prefetch.service` that gated `container@hive-ci` start on a forge round-trip, which could exceed the nspawn start timeout and trip a restart loop; moving registration into hive-c0re removed that.) The core admin token is held only by hive-c0re on the host; only the runner registration token reaches the container.
|
||||
Registration is **off the container's boot-critical path** — hive-c0re owns
|
||||
it and runs it out of band, so a slow forge or core-token never delays the
|
||||
container's start. Gotcha: don't gate the container's own start on a forge
|
||||
round-trip (a host-side unit that did this could exceed the nspawn start
|
||||
timeout and trip a restart loop) — registration must stay something
|
||||
hive-c0re drives after the container is already up. The core admin token is
|
||||
held only by hive-c0re on the host; only the runner registration token
|
||||
reaches the container.
|
||||
|
||||
### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep)
|
||||
|
||||
|
|
@ -114,7 +126,7 @@ When `forge.ci.enable` is set, hive-c0re auto-seeds an
|
|||
external DNS on the CI critical path.
|
||||
|
||||
The mirror is seeded by **hive-c0re** itself during its forge
|
||||
provisioning sweep (`forge.rs::ensure_mirrors`). The nix module
|
||||
provisioning sweep (`forge/repos.rs::ensure_mirrors`). The nix module
|
||||
forwards the effective mirror list as `HYPERHIVE_FORGE_MIRRORS` in the
|
||||
`hive-c0re` service environment (JSON-encoded `[{upstream, dest}]`
|
||||
list). hive-c0re already holds the admin token for the rest of the
|
||||
|
|
@ -122,10 +134,10 @@ forge provisioning sweep (orgs, agent accounts, etc.), so mirror
|
|||
seeding lives in the same place rather than a separate host-side unit.
|
||||
|
||||
**General-purpose mirrors**: you can pre-seed any external repo as a
|
||||
pull-mirror via `services.hyperhive.forge.mirrors`:
|
||||
pull-mirror via `services.hyperhive.swarm.forge.mirrors`:
|
||||
|
||||
```nix
|
||||
services.hyperhive.forge.mirrors = [
|
||||
services.hyperhive.swarm.forge.mirrors = [
|
||||
{ upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
|
||||
{ upstream = "https://github.com/example/tool"; dest = "mirrors/tool"; }
|
||||
];
|
||||
|
|
@ -139,13 +151,6 @@ runner as a hard `git clone` failure. The
|
|||
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
|
||||
to avoid provisioning collisions.
|
||||
|
||||
## CI workflow
|
||||
|
||||
Three jobs are defined in [`.forgejo/workflows/ci.yml`](../.forgejo/workflows/ci.yml):
|
||||
`nix flake check`, `tracker-tag lint`, and `comment-block lint`. All three are
|
||||
required — a lint failure blocks merge. `hive-forge ci-rerun --pr N` dispatches
|
||||
a `workflow_dispatch` retrigger without an empty commit.
|
||||
|
||||
## Security: unsandboxed builds and trusted contributors
|
||||
|
||||
**hive-ci should only run CI for trusted contributors.** The security boundary is weaker than it looks:
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ exist because something already went wrong without them.
|
|||
- Containers are length-bounded by `nixos-container` (≤ 11 chars).
|
||||
- Sub-agents are `h-<name>` with `<name>` ≤ 9 chars.
|
||||
- One agent is the bootstrap/root container, with a fixed name (`ruth` today).
|
||||
- `MAX_AGENT_NAME` in `lifecycle.rs` enforces the cap.
|
||||
- `MAX_AGENT_NAME` in `hive-c0re/src/lifecycle/mod.rs` enforces the cap.
|
||||
- Per-agent web UI port = `WEB_PORT_BASE + FNV1a(name) % WEB_PORT_RANGE`
|
||||
(8100..8999) for every agent; dashboard
|
||||
`cfg.dashboardPort` (default 7000).
|
||||
|
|
@ -16,7 +16,7 @@ exist because something already went wrong without them.
|
|||
## Hive identity (label + domain + display names)
|
||||
|
||||
Four env vars cover the identity surface, read by
|
||||
`hive_ag3nt::identity`:
|
||||
`hive-agent/src/identity.rs`:
|
||||
|
||||
- `HIVE_LABEL` — short, hive-local agent label (`iris`,
|
||||
`damocles`). `label()` returns it; falls back to empty string if
|
||||
|
|
@ -24,8 +24,8 @@ Four env vars cover the identity surface, read by
|
|||
surface "unknown agent" rather than getting a panic from this
|
||||
module.
|
||||
- `HYPERHIVE_HIVE_DOMAIN` — the hive's canonical DNS domain (e.g.
|
||||
`darkest.space`), set by `hive-c0re.nix` from
|
||||
`services.hyperhive.domain`. When configured, `qualified_label()`
|
||||
`darkest.space`), set by `nix/host-modules/hive-c0re/environment.nix`
|
||||
from `services.hyperhive.domain`. When configured, `qualified_label()`
|
||||
returns `${label}@${domain}` (e.g. `iris@darkest.space`); when
|
||||
unset (single-hive deployments, dev/test) it degrades to just
|
||||
the short label so existing callers see no change. The
|
||||
|
|
@ -83,7 +83,7 @@ name validation rejects any character outside `[a-z0-9_-]`, so the
|
|||
angle-bracket and asterisk shapes below are structurally safe.
|
||||
|
||||
- `*` — broadcast: deliver to every running agent except the sender
|
||||
(`agent_server::handle_send` fans out via `Coordinator::broadcast_send`).
|
||||
(`socket_server::handle_send` fans out via `Coordinator::broadcast_send`).
|
||||
- `operator` — the human at the dashboard. Messages accumulate in the
|
||||
inbox view; no agent ever `recv`'s them.
|
||||
- `<parent>` — the sender's parent per `topology.json`. Rewritten at
|
||||
|
|
@ -93,7 +93,7 @@ angle-bracket and asterisk shapes below are structurally safe.
|
|||
their parent without learning the label, so runtime reparenting
|
||||
propagates with zero agent-side restart.
|
||||
- `<children>` — fan-out to every direct descendant of the sender per
|
||||
`topology.json`. Resolved in `agent_server::handle_send` via
|
||||
`topology.json`. Resolved in `socket_server::handle_send` via
|
||||
`topology::children_of(sender)`: one message is delivered to each
|
||||
child, bypassing the allow-list check (structural fan-out targets are
|
||||
never user-listed peers). No-op for leaf agents (returns `Ok` when the
|
||||
|
|
@ -279,7 +279,7 @@ an agent. Self-introspection when `name = None` (replaces the older
|
|||
`Whoami` request); target query when `name = Some`.
|
||||
|
||||
Response is `AgentMeta { name, running, hyperhive_rev,
|
||||
status_text, status_set_at, hive_name, swarm_name }`:
|
||||
status_text, status_set_at, hive_name, swarm_name, matrix_accounts }`:
|
||||
|
||||
- `hyperhive_rev`: `None` only when the configured flake URL has
|
||||
no canonical path. Otherwise carries the rev the target is
|
||||
|
|
@ -299,6 +299,9 @@ status_text, status_set_at, hive_name, swarm_name }`:
|
|||
`HYPERHIVE_HIVE_NAME` / `HYPERHIVE_SWARM_NAME` env (sourced from
|
||||
`services.hyperhive.hiveName` / `services.hyperhive.swarm.name`).
|
||||
Both `None` when the options aren't configured.
|
||||
- `matrix_accounts`: one `MatrixIdentity` per configured + live matrix
|
||||
account the agent can act as. Empty for agents with no matrix
|
||||
provisioning.
|
||||
|
||||
### Timestamps on the wire
|
||||
|
||||
|
|
@ -323,14 +326,16 @@ binary flavor.
|
|||
|
||||
| Group | Tools |
|
||||
|---|---|
|
||||
| `messaging` | `send`, `recv`, `ask`, `answer` |
|
||||
| `messaging` | `send`, `recv`, `ack_until`, `ask`, `answer` |
|
||||
| `meta` | `get_agent_meta` (`set_status` is always-on, see below) |
|
||||
| `inbox` | `get_loose_ends`, `cancel_loose_end`, `remind` |
|
||||
| `execution` | vestigial — `mcp__bash__run` / `mcp__bash__status` are always available unconditionally via `extraMcpServers`; this group's entries expand to non-existent `mcp__hyperhive__run` / `mcp__hyperhive__status` and have no effect. See `docs/tools/bash.md`. |
|
||||
| `lifecycle` | `kill`, `start`, `restart`, `update` *(privileged)* |
|
||||
| `lifecycle` | `kill`, `start`, `restart`, `update`, `list_containers` *(privileged)* |
|
||||
| `approvals` | `request_init_config`, `request_update_meta_inputs` *(privileged)* |
|
||||
| `scheduling` | `request_schedule_prompt`, `fire_schedule_now`, `cancel_schedule`, `edit_schedule`, `list_schedules` *(privileged)* |
|
||||
| `diagnostics` | `get_logs` *(privileged)* |
|
||||
| `forge` | `create_repo` — create git repos through hive-c0re (operator-gated merge) |
|
||||
| `web_tools` | none (gates the Claude built-ins `WebFetch`/`WebSearch`, not an MCP tool) |
|
||||
|
||||
**Always-on tools** — `set_status`, `compact`, and `mark_todos_done` are
|
||||
exposed to every agent regardless of which groups it holds
|
||||
|
|
@ -363,10 +368,10 @@ from `tool-groups.json`). Unrecognised tokens are logged and skipped. Falls back
|
|||
to `ToolGroup::AGENT_DEFAULT` (`messaging`, `meta`, `inbox`, `execution`) when
|
||||
the var is absent or empty.
|
||||
|
||||
**Updating the surface** — when a new `#[tool]` fn is added to `HiveServer`
|
||||
in `hive-ag3nt/src/mcp.rs`, add its name to the matching `ToolGroup::tools()`
|
||||
slice in `hive-sh4re/src/lib.rs`. That's the single source of truth;
|
||||
`mcp_config::allowed_mcp_tools` (in `hive-ag3nt/src/mcp_config.rs`) reads it at
|
||||
**Updating the surface** — when a new `#[tool]` fn is added to `AgentServer`
|
||||
in `hive-agent-mcp/src/mcp/mod.rs`, add its name to the matching `ToolGroup::tools()`
|
||||
slice in `hive-sh4re/src/permissions.rs`. That's the single source of truth;
|
||||
`mcp_config::allowed_mcp_tools` (in `hive-agent/src/mcp_config.rs`) reads it at
|
||||
session start.
|
||||
|
||||
## Capabilities
|
||||
|
|
@ -406,14 +411,18 @@ groups: an agent that could grant its own capabilities via a config commit would
|
|||
bypass the operator approval gate.
|
||||
|
||||
**Adding a new capability** — add a variant to `Capability` in
|
||||
`hive-sh4re/src/lib.rs` + an arm to `as_str`. Add it to `Capability::ALL` (the
|
||||
source of truth for the permissions UI columns). Implement the access check in
|
||||
the relevant handler (`agent_server.rs`, `mcp.rs`, or `dashboard.rs`).
|
||||
`hive-sh4re/src/permissions.rs` + an arm to `as_str`. Add it to `Capability::ALL`
|
||||
(the source of truth for the permissions UI columns). Implement the access
|
||||
check in the relevant handler (`hive-c0re/src/socket_server/mod.rs`,
|
||||
`hive-c0re/src/socket_server/lifecycle_handlers.rs`, `coordinator.rs`, or a
|
||||
handler under `hive-c0re/src/dashboard/`).
|
||||
|
||||
## Async forms
|
||||
|
||||
Dashboard + per-agent mutating forms carry `data-async`; a delegated
|
||||
`submit` listener in `assets/tabs.js` (+ `assets/app.js` for the per-agent UI) intercepts, shows a spinner,
|
||||
Dashboard + per-agent mutating forms carry `data-async`; the shared
|
||||
`bindAsyncForms` `submit` listener (`frontend/packages/shared/src/forms.js`,
|
||||
imported as `@hive/shared/forms.js` and wired up from `tabs.js` on the
|
||||
dashboard and `app.js` on the per-agent UI) intercepts, shows a spinner,
|
||||
POSTs `application/x-www-form-urlencoded` (axum's `Form` extractor
|
||||
rejects multipart), calls `refreshState()` on success. New mutating
|
||||
forms should add `data-async` and optionally `data-confirm` (for a
|
||||
|
|
@ -428,11 +437,16 @@ via `snapshotOpenDetails` / `restoreOpenDetails`.
|
|||
|
||||
## `rebuild` is the reconcile verb
|
||||
|
||||
`lifecycle::rebuild` idempotently rewrites
|
||||
`/etc/nixos-containers/<C>.conf` (`PRIVATE_NETWORK=0`, clears
|
||||
`HOST_ADDRESS` / `LOCAL_ADDRESS`, sets `EXTRA_NSPAWN_FLAGS`),
|
||||
regenerates `applied/<name>/flake.nix`, writes the systemd limits
|
||||
drop-in, then `nixos-container update` + stop + start.
|
||||
`job_queue::templates::rebuild` builds the DAG that reconciles a
|
||||
container to its wanted state: `write_dropins` (the nspawn-conf
|
||||
rewrite — `PRIVATE_NETWORK=0`, clears `HOST_ADDRESS` / `LOCAL_ADDRESS`,
|
||||
sets `EXTRA_NSPAWN_FLAGS` — plus the systemd resource-limits drop-in)
|
||||
is folded into the `Swap` node, then `nixos-container update` + stop +
|
||||
start runs across the `StopForUpdate → Swap → RebuildBookkeeping`
|
||||
brace and the tail `Reconcile` node. `flake.nix` itself is no longer
|
||||
regenerated host-side on rebuild — it's tracked in the agent's
|
||||
proposed/applied repos and rides along on every fetch (see
|
||||
`docs/approvals.md::Two repos per agent`).
|
||||
|
||||
Anything that changes per-container state on the host should be
|
||||
re-applied here so a manual `↻ R3BU1LD` from the dashboard is
|
||||
|
|
@ -441,7 +455,7 @@ sufficient to recover.
|
|||
## Actions are factored
|
||||
|
||||
`approve` / `deny` / `destroy` (and the lifecycle helper) live in
|
||||
`actions.rs` / `dashboard.rs`. The admin socket and the dashboard
|
||||
`actions.rs` / `hive-c0re/src/dashboard/`. The admin socket and the dashboard
|
||||
POST handlers both call into them so the two surfaces never drift.
|
||||
|
||||
## Commit messages
|
||||
|
|
|
|||
|
|
@ -12,9 +12,9 @@ Every container/meta operation (rebuild, meta-update, first-spawn, power
|
|||
changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
|
||||
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
|
||||
concurrency comes from the resource classes below, not from multiple workers.
|
||||
The old special cases — the graceful-stop watcher thread, the deferred-start
|
||||
fast-lane follow-up, the meta-update cascade pre-enqueue — are all just DAG
|
||||
*shapes* now.
|
||||
Special cases like graceful stop, deferred starts, and the meta-update
|
||||
cascade need no bespoke code paths — each is expressed as a DAG *shape*
|
||||
built from the same primitive nodes.
|
||||
|
||||
### Two levels: DAG and node
|
||||
|
||||
|
|
@ -30,8 +30,7 @@ A DAG is **declared, not described**: a template builds it through
|
|||
`b.node(kind)` handed back, and the builder inserts the nodes itself. A handle
|
||||
only exists for a node already declared, so every edge points backwards and a
|
||||
cycle cannot be written down — there is no submit-time validation pass, because
|
||||
there is no malformed spec to reject. (The old queue had a petgraph `toposort`
|
||||
here, guarding against the positional indices that used to express edges.)
|
||||
there is no malformed spec to reject.
|
||||
|
||||
### Node inventory (primitives)
|
||||
|
||||
|
|
@ -61,7 +60,7 @@ Cheap — no build slot:
|
|||
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
|
||||
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
|
||||
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
|
||||
| `Reparent` | `set-parent` / `set-parent-bulk`: apply every `(child, new_parent)` move under one `META_LOCK` commit (`meta::bulk_commit_topology`), send the per-agent move notifications, rescan + diff-emit. Agentless like `MetaLock` — a bulk move can span multiple agents, and a reparent touches the meta repo, not any one container. `moves` is `(Ident, Option<Ident>)` pairs, not raw strings — mara: "use Ident type instead of string" (#2719, issuecomment 42691). Rides `Template::MetaUpdate` rather than a dedicated `Template` variant — that enum is on its way out (see `#2665`, still open/blocked on a scope question) and is already internal-only (it never reaches the graph wire), so the stand-in only affects `terminal_hook` dispatch (resolves to no hook either way) and history-retention bucketing |
|
||||
| `Reparent` | `set-parent` / `set-parent-bulk`: apply every `(child, new_parent)` move under one `META_LOCK` commit (`meta::bulk_commit_topology`), send the per-agent move notifications, rescan + diff-emit. Agentless like `MetaLock` — a bulk move can span multiple agents, and a reparent touches the meta repo, not any one container. `moves` is typed `(Ident, Option<Ident>)` pairs, not raw strings. Rides the existing `Template::MetaUpdate` variant rather than a dedicated one — it's internal-only (never reaches the graph wire), so the stand-in only affects `terminal_hook` dispatch (resolves to no hook either way) and history-retention bucketing |
|
||||
|
||||
There is deliberately **no `GitCommit` node**: `meta.rs` fuses each mutation
|
||||
with its commit under its internal `META_LOCK` mutex, so a standalone commit
|
||||
|
|
@ -175,7 +174,7 @@ without a row are seeded from observed state on first touch (running ⇒
|
|||
`Up`); destroy removes the row.
|
||||
|
||||
The admin-socket responses carry the submitted DAG ids; `hivectl` polls
|
||||
`HostRequest::QueueDag` (~1s) and prints a progress line per DAG — roll-up
|
||||
`HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up
|
||||
glyph, template, agent, node chain — so
|
||||
CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
|
||||
non-zero). Nodes appended in-DAG (a `MetaLock` growing per-agent rebuild
|
||||
|
|
@ -260,8 +259,9 @@ whether any dedup needs reintroducing is tracked as a follow-up.
|
|||
Cancel only applies to still-fully-queued DAGs (an in-flight nix build isn't
|
||||
interruptible) — each op is one DAG now, so there are no child DAGs to cascade to.
|
||||
Roll-up state: `Failed` if any node failed, else `Running` / `Queued` /
|
||||
`Cancelled` / `Done`. The snapshot retains the 5 most recent terminal DAGs
|
||||
per template.
|
||||
`Cancelled` / `Done`. The snapshot retains the 50 most recent terminal
|
||||
DAGs — a flat cap over the whole sorted list, not per template, since
|
||||
the dashboard renders one recent-builds list and one number bounds it.
|
||||
|
||||
### Approvals
|
||||
|
||||
|
|
@ -308,10 +308,11 @@ span agents; consumers derive a group's agent(s) from its nodes. Build logs
|
|||
are likewise **per-node**: the dashboard renders the node tree and keys the
|
||||
live-log panel off the running node.
|
||||
|
||||
The event used to ship the whole queue as a typed `DagView`/`NodeView`
|
||||
projection. That was a second rendering of the same graph, kept in
|
||||
agreement by hand with the endpoint every consumer actually read; it is
|
||||
gone, and the event's whole job is now telling a client *when* to refetch.
|
||||
The event carries no payload by design: shipping a typed projection of
|
||||
the whole queue in the event itself would be a second rendering of the
|
||||
same graph that has to be kept in agreement by hand with the endpoint
|
||||
every consumer actually reads. Telling a client *when* to refetch is
|
||||
the event's whole job.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -411,8 +412,7 @@ Sequence for a rebuild DAG (each step is its own queue node):
|
|||
(near-instant after the prebuild).
|
||||
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
||||
in-container activation script transitions old → new. Holds no build
|
||||
slot, so the next DAG's `Prebuild` overlaps the container boot — the old
|
||||
"deferred start" split, now structural.
|
||||
slot, so the next DAG's `Prebuild` overlaps the container boot.
|
||||
|
||||
The approval deploy uses this same chain rather than a rebuild path of its
|
||||
own. Its `DeployApply` node does not build: it merges, opens the two-phase
|
||||
|
|
@ -501,8 +501,8 @@ Two things to know about the weights:
|
|||
same value and the weight does *not* rank agents against each other.
|
||||
What `80` buys is that agents yield to everything **not** on this
|
||||
drop-in path: host services (nginx and dnsmasq among them) and the
|
||||
infra containers (`hive-ci`, `hive-forge`, `hive-matrix`), which stay
|
||||
at the kernel default of `100`.
|
||||
infra containers (`hive-ci`, `hive-forge`, `hive-gateway`,
|
||||
`hive-matrix`), which stay at the kernel default of `100`.
|
||||
- `IOWeight=` is only honoured when the backing device runs the BFQ
|
||||
scheduler or has blk-iocost QoS enabled. On a host using
|
||||
`none`/`mq-deadline`/`kyber` without iocost, systemd writes the value
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Private Forgejo instance running in a nixos-container, used as the
|
||||
swarm's persistent code-collaboration surface (issues, PRs, reviews,
|
||||
attachments). Configured via `services.hyperhive.forge.*`. Container
|
||||
attachments). Configured via `services.hyperhive.swarm.forge.*`. Container
|
||||
shape, ROOT_URL / sub-domain routing, and operator-vs-in-cluster URL
|
||||
handling live in [`docs/gateway.md`](gateway.md); this file owns the
|
||||
per-agent integration story and the notification pump that wakes
|
||||
|
|
@ -51,9 +51,12 @@ read it without touching c0re's host-side credential store.
|
|||
|
||||
Two things live in the `agent-configs` Forgejo organization:
|
||||
|
||||
- A config repo per agent (`agent-configs/<name>`). As of #1787 the
|
||||
- A config repo per agent (`agent-configs/<name>`). The
|
||||
agent is a **write collaborator on its own** repo — it can push
|
||||
config-change branches and (once #1838 P2 lands) open config PRs — but
|
||||
config-change branches and open config PRs (Forgejo `pull_request`
|
||||
webhook at `/webhook/config-pr` queues a `MergeConfigPr` approval;
|
||||
`hive-c0re/src/forge/config_pr_poll.rs` re-scans every 5 minutes as a
|
||||
fault-tolerance backstop) — but
|
||||
`main` is branch-protected core-only: only hive-c0re's verify-and-ff-push
|
||||
merge handler lands on `main`, an operator-team approval is required, and
|
||||
the agent can neither push `main` directly nor self-merge. `main` is
|
||||
|
|
@ -118,19 +121,20 @@ That size property is the whole point. A container rebuild starts the
|
|||
poller with no memory of what it delivered, re-scans `?all=false`, and
|
||||
finds nothing stale — the delivered threads are already read on forge.
|
||||
Forge's own read-state is thus the durable, cross-rebuild record of
|
||||
what's been delivered; there is **no persisted cursor**. (This
|
||||
replaced an earlier design that left threads unread and leaned on a
|
||||
persisted dedup cursor: a rebuild that lost the cursor re-delivered the
|
||||
entire still-unread backlog as fresh wakes — the notification flood of
|
||||
#2593 / #2106.)
|
||||
what's been delivered; there is **no persisted cursor**.
|
||||
|
||||
**Read-before-comment coupling, dropped on purpose.** The old design
|
||||
left threads unread so the hive-forge read-before-comment guard (which
|
||||
keys off forge unread-state) would force the agent to view a thread
|
||||
before commenting. That coupling is gone: the broker wake already
|
||||
carries the notification body, so *delivery is the read*. An agent that
|
||||
wants the full thread still runs `hive-forge comments` / `view`; the
|
||||
guard no longer blocks a first comment on a freshly-delivered thread.
|
||||
**Gotcha:** don't reintroduce a persisted dedup cursor here. A design
|
||||
that leaves threads unread and tracks delivery via a separately-persisted
|
||||
cursor is fragile — losing that cursor across a rebuild re-delivers the
|
||||
agent's entire still-unread backlog as a flood of fresh wakes. Forge's
|
||||
own read-state is the only durable record this design needs.
|
||||
|
||||
**Read-before-comment guard doesn't block a fresh wake.** hive-forge's
|
||||
read-before-comment guard (which keys off forge unread-state) does not
|
||||
force the agent to view a thread before commenting on it: the broker
|
||||
wake already carries the notification body, so *delivery is the read*.
|
||||
An agent that wants the full thread still runs `hive-forge comments` /
|
||||
`view`.
|
||||
|
||||
**In-process dedupe (tiny, ephemeral).** A single-process map (thread
|
||||
id → last-delivered `updated_at`) guards the narrow window where a
|
||||
|
|
@ -244,8 +248,8 @@ A notification carrying a `latest_comment_url` normally takes the comment
|
|||
path. But a merged/closed subject **keeps** its `latest_comment_url` set,
|
||||
so a just-merged PR that had any prior discussion would route to the
|
||||
comment path and render `[comment on PR]` (with a stale pre-merge comment
|
||||
body) instead of `[PR merged]` — the agent never learns its PR merged
|
||||
(#2495). So when the notification IS the merge/close transition — its
|
||||
body) instead of `[PR merged]` — the agent never learns its PR merged.
|
||||
So when the notification IS the merge/close transition — its
|
||||
event time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` of the
|
||||
subject's `closed_at` (set for both `merged` and `closed`) — the
|
||||
state-change path wins even with a comment url present
|
||||
|
|
@ -280,7 +284,7 @@ A review submitted with **no body** carries no `latest_comment_url`,
|
|||
so it misses the comment path and lands on the state-change path with
|
||||
`state == "open"` — exactly like a freshly opened PR. Labeling that
|
||||
`new PR` is misleading: agents dismiss it as a duplicate of the
|
||||
original open notification and miss the review (#1637). So the `open`
|
||||
original open notification and miss the review. So the `open`
|
||||
state only earns the `new <kind>` label when the notification's event
|
||||
time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` (120s) of the
|
||||
subject's `created_at`. Anything later is labeled `activity on <kind>`
|
||||
|
|
@ -288,7 +292,7 @@ subject's `created_at`. Anything later is labeled `activity on <kind>`
|
|||
activity was without an extra reviews fetch. Missing/unparseable
|
||||
timestamps default to `new` (preserve prior behavior rather than mask a
|
||||
genuine new item). Timestamps are parsed by a small dependency-free
|
||||
RFC 3339 → epoch-seconds helper (`parse_rfc3339_secs`).
|
||||
RFC 3339 helper (`parse_rfc3339`).
|
||||
|
||||
Number is extracted from `subject.html_url`'s last path segment
|
||||
(strips `#anchor` first); repo slug from `repository.full_name`.
|
||||
|
|
|
|||
114
docs/gateway.md
114
docs/gateway.md
|
|
@ -1,6 +1,6 @@
|
|||
# hive-gateway
|
||||
|
||||
Single nginx in front of every hyperhive web surface. Runs on the **host**, next to hive-c0re; system-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`. (It lived in a `hive-gateway` container until #3088 — one that shared the host netns anyway, so the boundary gave no network isolation while costing a resolv.conf sync, a machine-bus reload and three bind mounts.)
|
||||
Single nginx in front of every hyperhive web surface. Runs on the **host**, next to hive-c0re, rather than in its own container: it shares the host netns anyway (see [Vhost map](#vhost-map) below), so containerizing it would buy no network isolation while costing a resolv.conf sync, a machine-bus reload, and three bind mounts. System-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`.
|
||||
|
||||
## Vhost map
|
||||
|
||||
|
|
@ -9,11 +9,11 @@ Single nginx in front of every hyperhive web surface. Runs on the **host**, next
|
|||
| `<hive>/` | `_` (catch-all) | dashboard dist (static, from `servedFrontend`); `/api/` + `/webhook/` → hive-c0re (`7000`) | always |
|
||||
| `<hive>/agent/<name>/` | `_` | per-agent harness (UDS or TCP) | `agents.conf` (runtime-generated) |
|
||||
| `<hive>/.well-known/matrix/{client,server}` | `_` | inline JSON (no upstream) | `matrix.enable && domain != null` |
|
||||
| `<hive>/matrix/` (deprecated) | `_` | 301 → `matrix.<hive>/` | `matrix.gui.enable` |
|
||||
| `forge.<hive>/` | `forge.<hive>` | forgejo (`3000`) | `forge.behindGateway` |
|
||||
| `matrix.<hive>/_matrix/*` | `matrix.<hive>` | tuwunel (`8008`) | `matrix.gatewayHost != null` |
|
||||
| `matrix.<hive>/` | `matrix.<hive>` | fluffychat-web static | `matrix.gui.enable` |
|
||||
| `matrix.<hive>/config.json` | `matrix.<hive>` | inline JSON (FluffyChat boot config) | `matrix.gui.enable && domain != null` |
|
||||
| `<hive>/matrix/` (deprecated) | `_` | 301 → `chat.<swarm>/` | `matrix.gui.enable` |
|
||||
| `forge.<swarm>/` | `forge.<swarm>` | forgejo (`3000`) | `forge.behindGateway` |
|
||||
| `chat.<swarm>/_matrix/*` | `chat.<swarm>` | tuwunel (`8008`) | `matrix.gatewayHost != null` |
|
||||
| `chat.<swarm>/` | `chat.<swarm>` | fluffychat-web static | `matrix.gui.enable` |
|
||||
| `chat.<swarm>/config.json` | `chat.<swarm>` | inline JSON (FluffyChat boot config) | `matrix.gui.enable && domain != null` |
|
||||
| `auth.<swarm>/` | `auth.<swarm>` | authelia (`9091`) | `swarm.authelia.enable` |
|
||||
| `<swarm>/` | `<swarm>` | swarm-ui dist (static), behind an authelia subrequest | `swarm.ui.enable` |
|
||||
|
||||
|
|
@ -21,23 +21,24 @@ The authelia vhost is declared only by the host that **runs** authelia, not by e
|
|||
|
||||
⚠️ **A `502` from this vhost usually means authelia has no users yet, not that the proxy is misconfigured.** Authelia treats an empty user store as a fatal startup error, so an enabled-but-unbootstrapped swarm crash-loops the container while the vhost in front of it works perfectly. Check `journalctl -M swarm-authelia -u authelia-swarm` before suspecting anything here; the bootstrap step is in [`swarm/sso.md`](swarm/sso.md).
|
||||
|
||||
Per-agent UIs stay sub-path because they're hyperhive-internal and base-path-aware. External standard apps (forge / matrix) get sub-domains because their defaults work cleanly at sub-domain root + per-origin cookies / storage isolation matters.
|
||||
Per-agent UIs stay sub-path, forge and matrix get sub-domains — see
|
||||
[Sub-domain shape (rationale)](#sub-domain-shape-rationale) below for why.
|
||||
|
||||
## Discovery flow (matrix)
|
||||
|
||||
Operator points client at `<hive>`. Sequence:
|
||||
|
||||
1. Client fetches `https://<hive>/.well-known/matrix/client` → `{"m.homeserver":{"base_url":"https://matrix.<hive>"}}` (no port suffix when gateway listens on 443). The gateway always terminates TLS, so the scheme is always `https`; a non-default `httpsPort` is reflected as the port suffix.
|
||||
2. Client connects to `matrix.<hive>/_matrix/client/...`.
|
||||
1. Client fetches `https://<hive>/.well-known/matrix/client` → `{"m.homeserver":{"base_url":"https://chat.<swarm>"}}` (no port suffix when gateway listens on 443). The gateway always terminates TLS, so the scheme is always `https`; a non-default `httpsPort` is reflected as the port suffix.
|
||||
2. Client connects to `chat.<swarm>/_matrix/client/...`.
|
||||
3. Gateway routes `/_matrix/*` → tuwunel at `127.0.0.1:8008`.
|
||||
|
||||
matrix-dart-sdk (FluffyChat etc.) hardcodes `https` for the well-known fetch regardless of input scheme, so the discovery endpoint MUST be https — see "Self-signed TLS" below for the cert generation that backs the default-on path.
|
||||
|
||||
Federation peers fetch `.well-known/matrix/server` → `{"m.server":"matrix.<hive>"}` and connect to `matrix.<hive>:8448` per spec default. Gateway only listens on configured `port` (+ `httpsPort` when TLS on); cross-hive federation needs either an SRV record (`_matrix._tcp.matrix.<hive>` → port 80 / 443) OR `matrix.openFirewall = true` so peers reach tuwunel's federation port directly. Hyperhive is mostly closed/internal, so this rarely bites.
|
||||
Federation peers fetch `.well-known/matrix/server` → `{"m.server":"chat.<swarm>:<httpsPort>"}` (the federation delegation always carries an explicit port, even the HTTPS default 443 — the https-implies-443 elision only applies to the client base_url above). Gateway only listens on configured `port` (+ `httpsPort` when TLS on); cross-hive federation needs either an SRV record (`_matrix._tcp.chat.<swarm>` → port 80 / 443) OR `matrix.openFirewall = true` so peers reach tuwunel's federation port directly. Hyperhive is mostly closed/internal, so this rarely bites.
|
||||
|
||||
## SPA fallback (Accept-header pattern)
|
||||
|
||||
The per-agent UIs and the `matrix.<hive>` vhost serve a flutter/SPA bundle via the Accept-header pattern below. (The `<hive>` dashboard catch-all used this too but now routes by **path** — see the dashboard note after.) Two requirements collide:
|
||||
The per-agent UIs and the `chat.<swarm>` vhost serve a flutter/SPA bundle via the Accept-header pattern below. The dashboard vhost instead routes by **path** — see [Dashboard: path-based routing](#dashboard-path-based-routing-not-accept-header) below. Two requirements collide:
|
||||
|
||||
- hard-refresh on a sub-route must serve `index.html` (SPA's client-side router takes over after JS bootstrap)
|
||||
- a non-navigation request that isn't an on-disk asset must NOT get HTML with the wrong content-type
|
||||
|
|
@ -48,21 +49,21 @@ For matrix / per-agent static assets, `<final>` is `=404` (a missing asset is ju
|
|||
|
||||
### Dashboard: path-based routing (not Accept-header)
|
||||
|
||||
Now that every hive-c0re backend route lives under `/api/` plus the single `/webhook/knowledge` endpoint, the dashboard vhost routes by **path**, not Accept header:
|
||||
Every hive-c0re backend route lives under `/api/` plus the single `/webhook/knowledge` endpoint, so the dashboard vhost routes by **path**, not Accept header — deterministic, unlike a content-type split where the same URL could resolve differently depending on the caller's `Accept` header:
|
||||
|
||||
- `location /api/` → hive-c0re (`7000`): all dashboard data, actions/mutations, and the two SSE streams (`/api/dashboard/stream`, `/api/build-logs/id/{id}/stream`). Carries `proxy_buffering off` + a 1d read timeout for the streams.
|
||||
- `location /webhook/` → hive-c0re: the knowledge webhook.
|
||||
- `location /` → the dashboard dist (from the `servedFrontend` nix-store path) with `try_files $uri /index.html` (SPA fallback).
|
||||
|
||||
Each location carries a duplicated `auth_basic` block (separate locations don't inherit it). This keeps the gateway static-serving the dashboard dist while hive-c0re stays API-only — a frontend-only change no longer rebuilds + restarts the core daemon. The earlier `map $http_accept` Accept-header split was replaced because it made the *same* URL behave differently by content-type (e.g. `/api/state` fetched with `Accept: text/html` wrongly returned `index.html`); path routing is deterministic. A new top-level c0re route prefix (beyond `/api` + `/webhook`) needs a matching `location` added to the dashboard vhost.
|
||||
Each location carries a duplicated `auth_basic` block (separate locations don't inherit it). This keeps the gateway static-serving the dashboard dist while hive-c0re stays API-only — a frontend-only change doesn't rebuild or restart the core daemon. A new top-level c0re route prefix (beyond `/api` + `/webhook`) needs a matching `location` added to the dashboard vhost.
|
||||
|
||||
## Local dev (`localHostsEntry`)
|
||||
|
||||
`services.hyperhive.gateway.localHostsEntry = true` adds entries to the host's `/etc/hosts`:
|
||||
|
||||
- `<hive-domain>` → `127.0.0.1`
|
||||
- `forge.<hive>` → `127.0.0.1` (when forge.behindGateway)
|
||||
- `matrix.<hive>` → `127.0.0.1` (when matrix.gatewayHost set)
|
||||
- `forge.<swarm>` → `127.0.0.1` (when forge.behindGateway)
|
||||
- `chat.<swarm>` → `127.0.0.1` (when matrix.gatewayHost set)
|
||||
- `auth.<swarm>` → `127.0.0.1` (when swarm.authelia.enable)
|
||||
|
||||
`lib.unique` de-dupes if any sub-domain happens to equal another entry. Operators with real DNS leave it off.
|
||||
|
|
@ -149,8 +150,11 @@ When the gateway is in front, the SW4RM tab builds per-agent links
|
|||
as same-origin `/agent/<name>/…` URLs instead of the legacy direct
|
||||
`http://<host>:<container.port>/` TCP shape. The signal comes from
|
||||
`StateSnapshot.gateway_enabled`, sourced from the
|
||||
`HIVE_GATEWAY_ENABLED` env the c0re NixOS module sets when
|
||||
`services.hyperhive.gateway.enable = true`. Three render sites
|
||||
`HIVE_GATEWAY_ENABLED` env the c0re NixOS module now always sets
|
||||
(`services.hyperhive.gateway.enable` was removed — the gateway runs
|
||||
unconditionally alongside hyperhive), so this is effectively always
|
||||
true; the `false` branch is retained as a defensive fallback for the
|
||||
env being unset. Three render sites
|
||||
flip together: the primary agent-name link, the favicon fetch
|
||||
(`<url>/icon`), and the nav-strip `container`-kind links from
|
||||
`DashboardState.links` (`GET /api/dashboard-state`). `forge`-kind nav-strip links still
|
||||
|
|
@ -171,10 +175,10 @@ selected by which (if any) external TLS source is set:
|
|||
| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx via HTTP-01 | `https` |
|
||||
| operator cert | `tls.certDir` set | read from the operator's dir | `https` |
|
||||
|
||||
The `gateway.selfSignedTls` option is **deprecated and ignored** — self-signed
|
||||
is now derived from the absence of `tls.certDir` / `tls.acme`. Setting it to
|
||||
`false` (which used to select http-only or force an external cert) warns and
|
||||
has no effect; use `tls.certDir` / `tls.acme` to override the default.
|
||||
The `gateway.selfSignedTls` option has been **removed** — self-signed
|
||||
is now derived from the absence of `tls.certDir` / `tls.acme`. A config
|
||||
that still sets it fails eval with a removal message; use `tls.certDir`
|
||||
/ `tls.acme` to override the default.
|
||||
|
||||
### ACME / Let's Encrypt (`tls.acme`)
|
||||
|
||||
|
|
@ -204,7 +208,13 @@ On by default, and listens on `httpsPort` (default 443) on every vhost beside th
|
|||
|
||||
The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.tls.caValidityDays`, default ~20y) under `services.hyperhive.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`).
|
||||
|
||||
⚠️ **Do not collapse that import unit into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage.
|
||||
⚠️ **Do not collapse that import unit into pointing nginx at the CA dir.**
|
||||
It does two jobs, and skipping it has taken the gateway down in production
|
||||
before. It re-modes the leaf (`hive-tls-ca` writes the key `0600
|
||||
root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a
|
||||
`0600` key fails the config test and blocks the unit), and it guarantees
|
||||
**every cert path the nginx config names exists** — which is what the
|
||||
swarm-services fallback below is for.
|
||||
|
||||
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
|
||||
|
||||
|
|
@ -250,8 +260,8 @@ services.hyperhive.swarm.hives.example = { domain = "example.com"; }; # no cert
|
|||
|
||||
### Fronting with an external TLS terminator
|
||||
|
||||
There is no http-only mode: the gateway always terminates TLS (self-signed
|
||||
floor). Two paths for an operator who wants their own TLS terminator:
|
||||
There is no http-only mode (see [TLS modes](#tls-modes) above). Two paths
|
||||
for an operator who wants their own TLS terminator:
|
||||
|
||||
- give the gateway the real cert via `tls.certDir` (or `tls.acme`) so it
|
||||
serves proper TLS directly — no separate proxy needed; or
|
||||
|
|
@ -259,25 +269,26 @@ floor). Two paths for an operator who wants their own TLS terminator:
|
|||
intended direction for "bring your own proxy" — the gateway is not meant
|
||||
to expose an unencrypted TCP upstream).
|
||||
|
||||
**`.well-known/matrix/{client,server}` scheme** is always `https` now — the
|
||||
gateway always terminates TLS, so discovery responses always advertise https.
|
||||
Because of this, `.well-known/matrix/{client,server}` discovery responses
|
||||
always advertise `https` (see [Discovery flow](#discovery-flow-matrix) above).
|
||||
|
||||
## Firewall posture (host-level)
|
||||
|
||||
`hive-c0re.nix` opens the per-agent web-port range
|
||||
`8100..8999` in the host firewall **only when
|
||||
`services.hyperhive.gateway.enable = false`**. With the gateway on
|
||||
(default) it's the sole external entry point and routes to agents over
|
||||
the UDS upstream described above (see [Per-agent unix-socket
|
||||
upstream](#per-agent-unix-socket-upstream)) — leaving the per-agent
|
||||
ports firewall-open would defeat the single-front-door story. The
|
||||
hashed TCP port (`lifecycle::agent_web_port`) still exists as a direct
|
||||
host-loopback fallback for the pre-UDS/gateway-disabled case, but isn't
|
||||
what the gateway itself proxies through.
|
||||
The gateway is unconditional — `services.hyperhive.gateway.enable` was
|
||||
removed, there is no gateway-off mode. nginx is always the sole
|
||||
external entry point and routes to agents over the UDS upstream
|
||||
described above (see [Per-agent unix-socket
|
||||
upstream](#per-agent-unix-socket-upstream)), so the per-agent web-port
|
||||
range `8100..8999` stays closed on the host firewall
|
||||
unconditionally — opening it would defeat the single-front-door story.
|
||||
The hashed TCP port (`lifecycle::agent_web_port`) still exists as a
|
||||
fallback bind for an agent whose `HIVE_WEB_SOCKET` env somehow ends up
|
||||
unset, but nothing opens a matching firewall hole for it and the
|
||||
gateway itself never proxies through it.
|
||||
|
||||
`services.hyperhive.gateway.openFirewall = true` opens both `port` and
|
||||
`httpsPort` — the gateway always terminates TLS (self-signed floor), so
|
||||
both are always served.
|
||||
`httpsPort` — both are always served, since the gateway always terminates
|
||||
TLS (see [TLS modes](#tls-modes) above).
|
||||
|
||||
Every agent hashes into the same port range (no special case), so
|
||||
one range opening covers every container.
|
||||
|
|
@ -295,9 +306,10 @@ proxy in front.
|
|||
Agents poll `HIVE_FORGE_URL` for Forgejo notifications + run all
|
||||
`hive-forge` calls against it. Network isolation is always on (the
|
||||
shared-netns mode was removed), so agents run in a private netns and
|
||||
can never reach the host's loopback. `hive-c0re.nix` sets
|
||||
`HIVE_FORGE_URL` to `http://<forge.domain>` (default
|
||||
`forge.<hive-domain>`; `services.hyperhive.domain` is required). Agents
|
||||
can never reach the host's loopback.
|
||||
`nix/host-modules/hive-c0re/environment.nix` sets `HIVE_FORGE_URL` to
|
||||
`http://<forge.domain>` (default `forge.<swarm-domain>` — a swarm runs
|
||||
one forge; `services.hyperhive.domain` is required). Agents
|
||||
get the bridge dnsmasq as their resolver, resolve the hostname →
|
||||
bridge IP, then reach nginx on port 80 (the bridge firewall opens
|
||||
80+443). nginx proxies to forgejo — the same path an operator browser
|
||||
|
|
@ -369,14 +381,16 @@ covers most cases:
|
|||
|
||||
| Shape | Auto-derived `ROOT_URL` |
|
||||
|---|---|
|
||||
| `behindGateway = true` | `http://<forge.domain>/` (port suffix omitted when `gateway.port == 80`) |
|
||||
| `behindGateway = true` | `https://<forge.domain>/` (port suffix omitted when `gateway.httpsPort == 443`) |
|
||||
| `behindGateway = false` | `http://<forge.domain>:<httpPort>/` |
|
||||
|
||||
The auto-derivation always uses `http://`. Set `rootUrl` explicitly when
|
||||
you need `https://` (e.g. behind a TLS-terminating reverse proxy, or when
|
||||
clone URLs must carry `https://` because the gateway terminates TLS), or
|
||||
when `forge.domain` resolves differently from the public URL. Must end with
|
||||
`/` (Forgejo requirement; an assertion enforces this).
|
||||
The gateway always terminates TLS, so the `behindGateway = true` case is
|
||||
always advertised over `https://`; only the direct (`behindGateway =
|
||||
false`) shape stays `http://`. Set `rootUrl` explicitly when
|
||||
`forge.domain` resolves differently from the public URL, or for a
|
||||
genuinely bespoke shape (e.g. an external reverse proxy on a different
|
||||
host/path). Must end with `/` (Forgejo requirement; an assertion
|
||||
enforces this).
|
||||
|
||||
## Per-agent static frontend split
|
||||
|
||||
|
|
@ -589,7 +603,7 @@ header is added alongside the other security headers.
|
|||
enabling it on a deployment that later loses TLS locks browsers out
|
||||
until `max-age` expires. Only enable when TLS is permanent.
|
||||
|
||||
The gateway always terminates TLS now (self-signed floor), so HSTS is
|
||||
always served over https when enabled — the old "HSTS requires a TLS mode"
|
||||
assertion is gone (it can no longer be violated).
|
||||
Since the gateway always terminates TLS (see [TLS modes](#tls-modes)
|
||||
above), an enabled HSTS header is always served over https — there is no
|
||||
TLS-less mode that could violate it.
|
||||
|
||||
|
|
|
|||
170
docs/gotchas.md
170
docs/gotchas.md
|
|
@ -2,27 +2,31 @@
|
|||
|
||||
NixOS + nspawn quirks and lessons we hit the hard way. If something
|
||||
here looks unmotivated in the code, there's usually a story underneath.
|
||||
Grouped by area — jump to the section that matches what you're
|
||||
touching.
|
||||
|
||||
## `nixos-container` doesn't expose `--bind` on the CLI
|
||||
## NixOS / nspawn containers
|
||||
|
||||
### `nixos-container` doesn't expose `--bind` on the CLI
|
||||
|
||||
The CLI doesn't accept `--bind`. Path is via `EXTRA_NSPAWN_FLAGS` in
|
||||
`/etc/nixos-containers/<NAME>.conf` — the start script
|
||||
(`/nix/store/.../container_-start`) expands it unquoted into the
|
||||
`systemd-nspawn` invocation. `lifecycle::set_nspawn_flags()` rewrites
|
||||
this line.
|
||||
`systemd-nspawn` invocation. `lifecycle::host_config::set_nspawn_flags()`
|
||||
rewrites this line.
|
||||
|
||||
## `/run/systemd/nspawn/*.nspawn` overrides are ignored
|
||||
### `/run/systemd/nspawn/*.nspawn` overrides are ignored
|
||||
|
||||
`nixos-container`'s start script builds the nspawn command line
|
||||
directly. Dropping a `.nspawn` file under `/run/systemd/nspawn/`
|
||||
looks like the obvious extension point and does nothing. Use
|
||||
`EXTRA_NSPAWN_FLAGS` (above).
|
||||
|
||||
## `boot.isNspawnContainer = true`
|
||||
### `boot.isNspawnContainer = true`
|
||||
|
||||
Not `boot.isContainer = true`. Renamed in nixos-25.11+.
|
||||
|
||||
## `nixos-container create` auto-assigns `HOST_ADDRESS` / `LOCAL_ADDRESS`
|
||||
### `nixos-container create` auto-assigns `HOST_ADDRESS` / `LOCAL_ADDRESS`
|
||||
|
||||
…in the `.conf`. The start script's `if HOST_ADDRESS set →
|
||||
--network-veth` branch then forces a private netns — silently fatal
|
||||
|
|
@ -30,7 +34,7 @@ for our web UIs (the bind is invisible from the host). We
|
|||
force-clear `HOST_ADDRESS` / `LOCAL_ADDRESS` / `HOST_ADDRESS6` /
|
||||
`LOCAL_ADDRESS6` / `HOST_BRIDGE` and set `PRIVATE_NETWORK=0`.
|
||||
|
||||
## systemd service PATH ≠ host PATH
|
||||
### systemd service PATH ≠ host PATH
|
||||
|
||||
The hive-c0re service sets `path = [ pkgs.git "/run/current-system/sw" ]`.
|
||||
In-container harness services do the same so anything an agent adds
|
||||
|
|
@ -40,7 +44,7 @@ editing the service definition.
|
|||
`environment.HYPERHIVE_GIT` bakes git's absolute path in (read by
|
||||
`lifecycle::git_command()`) for the host.
|
||||
|
||||
## `systemd.services.*.path` appends `/bin` to every entry
|
||||
### `systemd.services.*.path` appends `/bin` to every entry
|
||||
|
||||
NixOS's `systemd.services.<unit>.path` list feeds every entry through
|
||||
`lib.makeBinPath`, which **appends `/bin` unconditionally**. That's
|
||||
|
|
@ -62,19 +66,21 @@ contains a non-existent directory. The first symptom is usually
|
|||
the setuid sudo wrapper lives at `/run/wrappers/bin/sudo` and
|
||||
the path entry resolves to `/run/wrappers/bin/bin` instead.
|
||||
|
||||
## `RuntimeDirectoryPreserve = "yes"`
|
||||
### `RuntimeDirectoryPreserve = "yes"`
|
||||
|
||||
…keeps `/run/hyperhive/` (and the per-agent sub-dirs) across
|
||||
hive-c0re restarts. Without it, every restart wipes bind sources and
|
||||
existing containers can't be started.
|
||||
|
||||
## `register_agent` is idempotent
|
||||
### `register_agent` is idempotent
|
||||
|
||||
Drops any prior socket task before rebinding. Required so a
|
||||
hive-c0re restart followed by `rebuild alice` recreates the agent's
|
||||
socket without needing a clean reinstall.
|
||||
|
||||
## `claude-code` is unfree
|
||||
## Claude Code packaging & credentials
|
||||
|
||||
### `claude-code` is unfree
|
||||
|
||||
`claude-code` comes from the flake's main `nixpkgs` (nixos-26.05).
|
||||
It's unfree, so the agent modules set `config.allowUnfreePredicate`
|
||||
|
|
@ -125,7 +131,7 @@ the hive's `claude` out from under it. The price of the root is that an
|
|||
old `claude-code` can't be reclaimed until every agent has rebuilt past
|
||||
it and the old generations are gone.
|
||||
|
||||
## Claude credentials are per-agent
|
||||
### Claude credentials are per-agent
|
||||
|
||||
`/var/lib/hyperhive/agents/<name>/claude/` bind-mounts to
|
||||
`/home/<name>/.claude` (RW). Sharing one dir across agents is NOT viable —
|
||||
|
|
@ -133,17 +139,22 @@ OAuth refresh tokens rotate, so any sibling refresh invalidates all
|
|||
the others. Login flow runs from the per-agent web UI; creds persist
|
||||
across `destroy`/recreate (`--purge` wipes them).
|
||||
|
||||
## Persistent notes dir per agent
|
||||
### Persistent notes dir per agent
|
||||
|
||||
`/var/lib/hyperhive/agents/<name>/state/` bind-mounts to
|
||||
`/agents/<name>/state` (RW; uniform for all agents).
|
||||
The harness exposes the same path
|
||||
via `$HYPERHIVE_STATE_DIR`. System prompts tell agents to keep
|
||||
durable knowledge here (`notes.md`, anything else). The harness also
|
||||
writes its events log here (`hyperhive-events.sqlite`).
|
||||
Survives `destroy`/recreate alongside the claude dir.
|
||||
durable knowledge here (`notes.md`, anything else) — the harness's own
|
||||
internal files (`hyperhive-events.sqlite`, `hyperhive-turn-stats.sqlite`,
|
||||
`hyperhive-model`) live in the separate `harness` dir instead, so they
|
||||
don't clutter what claude sees as "my notes dir" (see
|
||||
[`docs/persistence.md`](persistence.md)). Survives `destroy`/recreate
|
||||
alongside the claude dir.
|
||||
|
||||
## Web UI ports collide on hash
|
||||
## Networking & ports
|
||||
|
||||
### Web UI ports collide on hash
|
||||
|
||||
Sub-agent web UI ports are deterministic FNV-1a of the agent name
|
||||
modulo 900 (range 8100..8999). With ~30 agents the birthday-paradox
|
||||
|
|
@ -155,7 +166,7 @@ reproducible from just the name. Every agent hashes into
|
|||
8100..8999 via the same FNV-1a; dashboard
|
||||
at `cfg.dashboardPort` (default 7000).
|
||||
|
||||
## Restart races on TCP bind
|
||||
### Restart races on TCP bind
|
||||
|
||||
Both the dashboard and per-agent web UI use `tokio::net::TcpSocket`
|
||||
with `SO_REUSEADDR` plus a retry-on-`AddrInUse` loop (12 tries,
|
||||
|
|
@ -166,38 +177,35 @@ overlap" case. REUSEADDR does **not** allow two simultaneous
|
|||
`LISTEN` sockets on the same port (that would be `SO_REUSEPORT`,
|
||||
which we don't use) — exclusivity is preserved.
|
||||
|
||||
## Orphan approvals
|
||||
## Approvals
|
||||
|
||||
### Orphan approvals
|
||||
|
||||
If state dirs are wiped out from under a pending approval (test
|
||||
scripts, manual `rm -rf`), the dashboard's next render marks them
|
||||
`failed` with note `"agent state dir missing"` so they fall out of
|
||||
`pending`. They stay in sqlite for audit.
|
||||
|
||||
## Nix store `cp -r` preserves read-only bits
|
||||
## Gateway / SPA serving
|
||||
|
||||
Copying a nix store path with `cp -r src/. $out/` inside a
|
||||
`pkgs.runCommand` derivation preserves the read-only permissions of
|
||||
store files. Any subsequent write into the copied tree (adding new
|
||||
files in subdirectories) fails with `EPERM`. Fix: pass
|
||||
`--no-preserve=mode,ownership` so the output tree is writable.
|
||||
### SPA fallback: use `Accept` header map, not `try_files ... /index.html`
|
||||
|
||||
## SPA fallback: use `Accept` header map, not `try_files ... /index.html`
|
||||
|
||||
The naive nginx pattern for a path-prefix SPA (`try_files $uri $uri/
|
||||
/matrix/index.html`) silently swallows asset 404s — a missing JS file
|
||||
The naive nginx pattern for an SPA (`try_files $uri $uri/
|
||||
/index.html`) silently swallows asset 404s — a missing JS file
|
||||
returns `index.html` with a 200, so the JS runtime never loads and the
|
||||
page renders blank with no visible error. Extension allowlists (tried
|
||||
as an alternative) have the same maintenance problem: any new file
|
||||
extension the SPA ships breaks silently.
|
||||
|
||||
The pattern that works (`hive-gateway.nix`) keys the fallback on the
|
||||
The pattern that works (`nix/host-modules/hive-matrix.nix`, serving
|
||||
fluffychat at the matrix gateway vhost's root) keys the fallback on the
|
||||
HTTP `Accept` header:
|
||||
|
||||
```nginx
|
||||
# Outside the server block (appendHttpConfig):
|
||||
map $http_accept $matrix_spa_target {
|
||||
default "/__matrix_spa_no_html_fallback";
|
||||
"~*text/html" "/matrix/index.html";
|
||||
"~*text/html" "/index.html";
|
||||
}
|
||||
|
||||
# Inside the location:
|
||||
|
|
@ -210,7 +218,17 @@ firefox / safari are consistent). Asset fetches (`image/*`,
|
|||
fall through to the trailing `=404`. No extension list to maintain;
|
||||
no named-location indirection needed.
|
||||
|
||||
## `nix build flake#name` does not walk into `nixosConfigurations`
|
||||
## Build & dev workflow
|
||||
|
||||
### Nix store `cp -r` preserves read-only bits
|
||||
|
||||
Copying a nix store path with `cp -r src/. $out/` inside a
|
||||
`pkgs.runCommand` derivation preserves the read-only permissions of
|
||||
store files. Any subsequent write into the copied tree (adding new
|
||||
files in subdirectories) fails with `EPERM`. Fix: pass
|
||||
`--no-preserve=mode,ownership` so the output tree is writable.
|
||||
|
||||
### `nix build flake#name` does not walk into `nixosConfigurations`
|
||||
|
||||
`nix build` resolves the fragment (`#name`) against the flake's
|
||||
**top-level output attrs** — not against `nixosConfigurations`
|
||||
|
|
@ -233,12 +251,7 @@ instead of `meta#nixosConfigurations.argus.config…`. The fix:
|
|||
`split_once('#')` to separate flake path from name, then template
|
||||
`{path}#nixosConfigurations.{name}.config.system.build.toplevel`.
|
||||
|
||||
## `hive-forge`: prefer over raw curl pipelines
|
||||
|
||||
Full CLI reference: [`docs/tools/forge.md`](tools/forge.md).
|
||||
Never use raw `curl` for forge access.
|
||||
|
||||
## Containerized nix-daemon needs `sandbox-fallback = true`
|
||||
### Containerized nix-daemon needs `sandbox-fallback = true`
|
||||
|
||||
Agent containers bind-mount the host's nix-daemon socket. nspawn
|
||||
containers don't get user-namespaces by default, so `nix build`
|
||||
|
|
@ -249,7 +262,7 @@ and fail outright if the host daemon's
|
|||
fall back to unsandboxed local builds rather than failing. Security
|
||||
implications: `docs/security.md`.
|
||||
|
||||
## Linking workspace binaries locally needs `nix develop`
|
||||
### Linking workspace binaries locally needs `nix develop`
|
||||
|
||||
The Rust workspace links `libsqlite3-sys` (rusqlite) against the
|
||||
system `libsqlite3`. Agent containers carry no system libsqlite3 on
|
||||
|
|
@ -271,7 +284,7 @@ e.g. `docs/tools/hivectl-cli.md` via the `hivectl markdown-docs`
|
|||
subcommand (its `hivectl-docs` flake check otherwise only fails in
|
||||
CI on drift).
|
||||
|
||||
## Split asset derivations away from the rust workspace
|
||||
### Split asset derivations away from the rust workspace
|
||||
|
||||
`nix/packages/assets.nix` builds the branding SVG/PNG family + claude
|
||||
system-prompt template + claude-settings JSON as its own derivation,
|
||||
|
|
@ -285,7 +298,46 @@ The agent-configs PNG is rendered from the SVG via `rsvg-convert` at
|
|||
build time; librsvg dependency lives here, not in the rust
|
||||
derivation's `nativeBuildInputs`.
|
||||
|
||||
## Weston VNC compositor (per-agent `hyperhive.gui.enable`)
|
||||
### `nix fmt` fails in a git worktree with "object not found"
|
||||
|
||||
`nix fmt` (and any `nix` command that fetches a `git+file://` flake
|
||||
URL) uses libgit2 internally to compute `revCount` — the number of
|
||||
commits reachable from HEAD. This walk fails with:
|
||||
|
||||
```
|
||||
error: getting Git object '<hash>': object not found (libgit2 error code = 9)
|
||||
```
|
||||
|
||||
when a commit that was reachable at some earlier evaluation is now gone
|
||||
(GC'd, rebased away, or pruned). The failure is persistent: clearing
|
||||
`~/.cache/nix/{eval-cache-v6,gitv3,fetcher-cache-v4.sqlite}` does not
|
||||
help because the missing object is a structural gap in the git object
|
||||
graph itself, not in nix's caches.
|
||||
|
||||
**Workaround: use a plain clone, not a git worktree.**
|
||||
|
||||
```bash
|
||||
git clone http://<forge>/hyperhive/hyperhive.git ~/hh-work
|
||||
cd ~/hh-work && nix fmt
|
||||
```
|
||||
|
||||
The root cause is specific to worktrees: a worktree shares the object
|
||||
store with its parent repo. If the parent repo's history was rewritten
|
||||
(rebase, force-push, `git gc --prune`) while the worktree was checked
|
||||
out at a branch tip that references the pruned commits via its reflog or
|
||||
history, libgit2's rev-walk encounters the gap. A plain clone has its
|
||||
own self-consistent object store and is immune to the issue.
|
||||
|
||||
## Tooling
|
||||
|
||||
### `hive-forge`: prefer over raw curl pipelines
|
||||
|
||||
Full CLI reference: [`docs/tools/forge.md`](tools/forge.md).
|
||||
Never use raw `curl` for forge access.
|
||||
|
||||
## GUI (weston/VNC)
|
||||
|
||||
### Weston VNC compositor (per-agent `hyperhive.gui.enable`)
|
||||
|
||||
`nix/agent-modules/weston-vnc.nix` adds an optional Weston Wayland
|
||||
compositor with the VNC backend, surfaced as
|
||||
|
|
@ -361,7 +413,9 @@ connects to the compositor at `127.0.0.1:<vnc_port>`.
|
|||
`wl_event_source_timer_update` treats as "disarm", so the
|
||||
compositor never goes idle and never locks.
|
||||
|
||||
## Nix options reference (`nix/docs/default.nix`)
|
||||
## Nix docs pipeline
|
||||
|
||||
### Nix options reference (`nix/docs/default.nix`)
|
||||
|
||||
`pkgs.nixosOptionsDoc` over two evaluated module trees:
|
||||
`hostEval` (a stub NixOS system loading the `nix/host-modules/` aggregator with every
|
||||
|
|
@ -397,7 +451,7 @@ options tree picks up everything under that root — picking against
|
|||
stray roots produces an empty tree and renders the host page as
|
||||
template chrome with no `<h2>` headers.
|
||||
|
||||
### Docs drv stability: `nixSrc`
|
||||
#### Docs drv stability: `nixSrc`
|
||||
|
||||
Naively, the docs evaluation depends on `self` (the flake's store path),
|
||||
so every commit — even Rust-only or frontend-only changes — produces new
|
||||
|
|
@ -426,33 +480,3 @@ Why `builtins.unsafeDiscardStringContext`? The path string
|
|||
make `builtins.path` include `self` as a build dependency even after
|
||||
content-addressing the directory. Discarding the context makes the
|
||||
resulting `nixSrc` truly independent of `self`'s store path.
|
||||
|
||||
### `nix fmt` fails in a git worktree with "object not found"
|
||||
|
||||
`nix fmt` (and any `nix` command that fetches a `git+file://` flake
|
||||
URL) uses libgit2 internally to compute `revCount` — the number of
|
||||
commits reachable from HEAD. This walk fails with:
|
||||
|
||||
```
|
||||
error: getting Git object '<hash>': object not found (libgit2 error code = 9)
|
||||
```
|
||||
|
||||
when a commit that was reachable at some earlier evaluation is now gone
|
||||
(GC'd, rebased away, or pruned). The failure is persistent: clearing
|
||||
`~/.cache/nix/{eval-cache-v6,gitv3,fetcher-cache-v4.sqlite}` does not
|
||||
help because the missing object is a structural gap in the git object
|
||||
graph itself, not in nix's caches.
|
||||
|
||||
**Workaround: use a plain clone, not a git worktree.**
|
||||
|
||||
```bash
|
||||
git clone http://<forge>/hyperhive/hyperhive.git ~/hh-work
|
||||
cd ~/hh-work && nix fmt
|
||||
```
|
||||
|
||||
The root cause is specific to worktrees: a worktree shares the object
|
||||
store with its parent repo. If the parent repo's history was rewritten
|
||||
(rebase, force-push, `git gc --prune`) while the worktree was checked
|
||||
out at a branch tip that references the pruned commits via its reflog or
|
||||
history, libgit2's rev-walk encounters the gap. A plain clone has its
|
||||
own self-consistent object store and is immune to the issue.
|
||||
|
|
|
|||
|
|
@ -24,8 +24,9 @@ the local clone updates automatically (see
|
|||
## Repository layout
|
||||
|
||||
Canonical forge location: `internal/knowledge` (org `internal`,
|
||||
repo `knowledge`). Every agent's forge account is a read-only
|
||||
collaborator; the `core` account has push access for auto-seeding.
|
||||
repo `knowledge`). The repo is public, so every agent's forge account
|
||||
has read access without an explicit per-agent collaborator grant;
|
||||
only the `core` account has push access, for auto-seeding.
|
||||
|
||||
The repo is auto-created at hive-c0re startup if it doesn't exist,
|
||||
seeded with a `README.md` containing a contribution guide and a
|
||||
|
|
@ -39,11 +40,12 @@ hive-c0re maintains the local clone at
|
|||
|
||||
1. **Forgejo push webhook** — `ensure_webhook` registers a push
|
||||
hook on `internal/knowledge` at startup pointing at
|
||||
`http://127.0.0.1:<dashboard_port>/webhook/knowledge`. On any
|
||||
push to main (including merge commits) hive-c0re runs
|
||||
`git pull` so agents see the new content on their next turn.
|
||||
The endpoint is loopback-only; no signature verification is
|
||||
needed.
|
||||
`https://<hive_domain>/webhook/knowledge` (routed through the
|
||||
gateway, avoiding the Forgejo SSRF guard that blocks loopback
|
||||
delivery). On any push to main (including merge commits) hive-c0re
|
||||
runs `git pull` so agents see the new content on their next turn.
|
||||
The endpoint is protected by an auto-generated HMAC secret that
|
||||
hive-c0re verifies on every delivery.
|
||||
|
||||
2. **Periodic pull** — a background task in `hive-c0re::main`
|
||||
pulls on a fixed cadence as a fallback (webhook missed, c0re
|
||||
|
|
@ -89,9 +91,10 @@ token.
|
|||
|
||||
## Contributing
|
||||
|
||||
Agents are read-only collaborators on `internal/knowledge`, so they
|
||||
can't push a branch directly. The supported path is Forgejo's **AGit
|
||||
flow** through the `hive-forge` CLI — no fork required.
|
||||
Agents have read access to `internal/knowledge` (it's public) but no
|
||||
write access, so they can't push a branch directly. The supported path
|
||||
is Forgejo's **AGit flow** through the `hive-forge` CLI — no fork
|
||||
required.
|
||||
|
||||
1. Clone the repo (credentials are injected automatically; the `-r`
|
||||
flag selects the repo, the clone lands in `./knowledge`):
|
||||
|
|
@ -125,6 +128,6 @@ flow** through the `hive-forge` CLI — no fork required.
|
|||
running container sees the updated content within seconds (see
|
||||
[Sync mechanism](#sync-mechanism)).
|
||||
|
||||
Do **not** try to `git push` a branch directly — read-only
|
||||
collaborator access rejects it. The `--agit` flow above is the
|
||||
no-fork path that works from any agent.
|
||||
Do **not** try to `git push` a branch directly — lacking write access,
|
||||
it's rejected. The `--agit` flow above is the no-fork path that works
|
||||
from any agent.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Private Matrix homeserver (matrix-tuwunel — the conduwuit
|
||||
successor) wrapped in a nixos-container, plus optional fluffychat-web
|
||||
client at `matrix.<hive>/`. Configured via
|
||||
client at `chat.<swarm-domain>/` (the `gatewayHost` vhost). Configured via
|
||||
`services.hyperhive.swarm.matrix.*`; vhost routing lives in
|
||||
[`gateway.md`](gateway.md).
|
||||
|
||||
|
|
@ -208,18 +208,28 @@ resource-constrained hosts where a 20 MB request is unexpectedly large.
|
|||
|
||||
## Assertion rationale
|
||||
|
||||
Two `config.assertions` entries fail eval early rather than ship
|
||||
`config.assertions` in this module fail eval early rather than ship
|
||||
surprising behaviour:
|
||||
|
||||
- **`hyperhiveDomain != null || cfg.serverName != null`** —
|
||||
`server_name` is embedded into every user / room ID irrevocably;
|
||||
we refuse to spawn the homeserver with a bogus `server_name` we
|
||||
can never change later.
|
||||
- **`cfg.gatewayHost != ""`** — same footgun as `forge.domain`:
|
||||
empty string renders `.<hive>`-shaped garbage in both nginx
|
||||
`server_name` (treated as wildcard catch-all, surprising) and
|
||||
`/etc/hosts` (invalid entry). `null` is the right opt-out shape;
|
||||
empty string is rejected explicitly.
|
||||
- **`sso.enable` requires `sso.clientSecretFile`** — fails at eval,
|
||||
not at boot: tuwunel reads its identity providers from the config
|
||||
file, so a half-configured one can stop the homeserver from
|
||||
starting outright rather than merely hiding a login button.
|
||||
- **`sso.enable` requires `swarm.authelia.url`** — without a
|
||||
provider URL there is nothing to discover against.
|
||||
- **`sso.enable` requires `gatewayHost != null`** — the SSO callback
|
||||
URL is format-locked to `<homeserver>/_matrix/client/unstable/login/sso/callback/<client_id>`,
|
||||
and the identity provider needs a public name to redirect the
|
||||
browser to.
|
||||
|
||||
`server_name`'s own bogus-value guard lives in `hive-network.nix`
|
||||
(`services.hyperhive.domain != null`), not here — see
|
||||
[`docs/network.md`](network.md).
|
||||
|
||||
## fluffychat-web build fixes
|
||||
|
||||
|
|
@ -250,7 +260,7 @@ Both fixed in `nix/host-modules/hive-matrix.nix` via two derivations:
|
|||
incremental cost) and (b) installs `fluffychat-web-imaging`'s
|
||||
outputs into `$out`.
|
||||
|
||||
Two non-obvious fixes from review history:
|
||||
Two non-obvious details worth knowing before touching either derivation:
|
||||
|
||||
- **`make -C js`** instead of `cd js; make` — keeps the build-phase
|
||||
pwd at the source root so `installPhase` doesn't have to know
|
||||
|
|
@ -265,5 +275,5 @@ Two non-obvious fixes from review history:
|
|||
Drop both derivations when nixpkgs's flutter builder grows worker
|
||||
+ emcc support upstream.
|
||||
|
||||
Mount point is `matrix.<hive>/`; upstream `--base-href "/"` is
|
||||
correct at sub-domain root, no override.
|
||||
Mount point is `chat.<swarm-domain>/` (the `gatewayHost` vhost);
|
||||
upstream `--base-href "/"` is correct at sub-domain root, no override.
|
||||
|
|
|
|||
|
|
@ -112,14 +112,19 @@ schemes pick their own.
|
|||
|
||||
## Resolver behaviour
|
||||
|
||||
dnsmasq is **authoritative** for the hive's own zones — answers
|
||||
`<hive-domain>`, `forge.<hive-domain>`, `matrix.<hive-domain>` and —
|
||||
on the host running it — the swarm's `auth.<swarm-domain>`
|
||||
queries with the bridge IP (where nginx is reachable). Everything
|
||||
else is forwarded to the host's own resolvers: dnsmasq runs on the host
|
||||
and reads the host's `/etc/resolv.conf` directly. Containers don't need
|
||||
to know the upstream — they query the bridge IP and dnsmasq does the
|
||||
right thing per-name.
|
||||
dnsmasq is **authoritative** for the hive's own zone (`<hive-domain>`)
|
||||
plus whatever swarm-service names this host contributes via
|
||||
`gateway.localNames` — `forge.<swarm-domain>` and `chat.<swarm-domain>`
|
||||
(matrix) when this host runs those services, and `auth.<swarm-domain>`
|
||||
when it runs authelia — answering each with the bridge IP (where nginx
|
||||
is reachable). Note forge and matrix are swarm-domain names, not
|
||||
sub-domains of `<hive-domain>`: a swarm runs one forge and one
|
||||
homeserver, so their names belong to the swarm rather than to whichever
|
||||
hive happens to host them. Everything else is forwarded to the host's
|
||||
own resolvers: dnsmasq runs on the host and reads the host's
|
||||
`/etc/resolv.conf` directly. Containers don't need to know the
|
||||
upstream — they query the bridge IP and dnsmasq does the right thing
|
||||
per-name.
|
||||
|
||||
There is deliberately no fallback `server=`: dnsmasq queries all known
|
||||
upstreams in parallel, so a hardcoded public resolver would take a share
|
||||
|
|
@ -245,12 +250,11 @@ wiring is runtime:
|
|||
It is ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
|
||||
`tea-login` so the resolver is correct before the first DNS lookup.
|
||||
|
||||
**Why isolation is safe**: all hive-c0re communication goes
|
||||
through unix domain sockets (`/run/hive/mcp.sock` for agent requests,
|
||||
`/run/hive/priv.sock` for privileged ops).
|
||||
These are bind-mounted into containers via the nspawn conf. UDS paths
|
||||
traverse the VFS, not the network stack, so `PRIVATE_NETWORK=1` does not
|
||||
affect them.
|
||||
**Why isolation is safe**: hive-c0re's control-plane sockets are unix
|
||||
domain sockets bind-mounted into containers, not network listeners — see
|
||||
the *Control plane (no network)* bullet under [Network
|
||||
map](#network-map) above. `PRIVATE_NETWORK=1` has no effect on a path
|
||||
that never touches the network stack.
|
||||
|
||||
The nix side also enables IP forwarding + NAT (agents reach the internet
|
||||
through the host) and drops bridge-subnet → loopback traffic (defence-in-depth
|
||||
|
|
|
|||
|
|
@ -40,20 +40,23 @@ schemas, file layouts, and internal migration mechanics.
|
|||
|
||||
### `/var/lib/hyperhive/db/broker.sqlite` (host)
|
||||
|
||||
Seven tables, all in one file — four queues, the schedule
|
||||
header/targets split, and the per-agent power-intent registry:
|
||||
Seven tables, all in one file — three queues, a small key/value
|
||||
table, the schedule header/targets split, and the per-agent
|
||||
power-intent registry:
|
||||
|
||||
- `messages` — every inter-agent / operator-bound message.
|
||||
`sender / recipient / body / sent_at / delivered_at / acked_at /
|
||||
in_reply_to`. `in_reply_to` links a reply to its parent row id;
|
||||
the dashboard and per-agent inbox render these as threaded rows.
|
||||
- `reminders` — `mcp__hyperhive__remind` queue.
|
||||
`agent / message / file_path / due_at / created_at / sent_at /
|
||||
attempt_count / last_error`. `file_path` set when a body
|
||||
exceeded the inline soft-cap and got auto-spilled to a file
|
||||
under the agent's state dir; the worker delivers a short
|
||||
pointer instead. `attempt_count` / `last_error` accumulate
|
||||
on delivery-failed retries.
|
||||
in_reply_to / priority`. `in_reply_to` links a reply to its parent
|
||||
row id; the dashboard and per-agent inbox render these as threaded
|
||||
rows.
|
||||
- `kv` — small persistent key/value store (`key PK / value`) for
|
||||
host-side bookkeeping that doesn't warrant its own table.
|
||||
|
||||
⚠️ The `mcp__hyperhive__remind` queue is **not** here any more: it
|
||||
moved to a harness-local, per-agent store as part of the
|
||||
loose-ends-v2 migration — see [`/harness/` contents
|
||||
below](#state-dirs-per-agent) for where reminders (and todos, and
|
||||
the questions mirror) actually live now.
|
||||
- `approvals` — the queue. `agent / kind (merge_config_pr | spawn |
|
||||
init_config | update_meta_inputs | schedule_prompt) /
|
||||
commit_ref / requested_at / status / resolved_at / note`.
|
||||
|
|
@ -79,16 +82,13 @@ header/targets split, and the per-agent power-intent registry:
|
|||
`scheduled_prompts(id)` — requires `PRAGMA foreign_keys = ON`
|
||||
per connection (set at open).
|
||||
- `agent_power` — one tiny row per agent: `agent PK / wanted (up |
|
||||
offline) / updated_at` — the durable power *intent* behind the job
|
||||
queue's desired-state reconciliation
|
||||
(`docs/coordinator.md::Job queue`; owner: `hive-c0re/src/stores/power.rs`).
|
||||
Written synchronously by every operator/agent power action
|
||||
(dashboard start/stop, `hivectl stop`, the MCP kill/start tools,
|
||||
spawn approval); read by `Reconcile` nodes and the boot reconcile.
|
||||
Intent survives hive-c0re restarts — in-flight queue work
|
||||
deliberately does not. Agents without a row are seeded from
|
||||
observed state on first touch (running ⇒ `up`); destroy deletes
|
||||
the row.
|
||||
offline) / updated_at`, owned by `hive-c0re/src/stores/power.rs`.
|
||||
This is the durable power *intent* the job queue reconciles the
|
||||
observed container state against; intent survives hive-c0re
|
||||
restarts even though in-flight queue work doesn't. See
|
||||
[`docs/coordinator.md`'s Desired-state
|
||||
section](coordinator.md#desired-state-spec-vs-status) for who
|
||||
writes and reads it and how reconciliation works.
|
||||
|
||||
Retention:
|
||||
|
||||
|
|
@ -101,8 +101,6 @@ Retention:
|
|||
- Approvals and questions are kept indefinitely — both are
|
||||
audit trails. `actions::destroy` and answered questions stay
|
||||
visible to anything that queries by id.
|
||||
- Reminder rows are kept after `sent_at` is set (audit trail);
|
||||
no automatic vacuum today.
|
||||
- Scheduled prompts: one-shot rows are deleted on fire by the
|
||||
worker; recurring rows live until the operator cancels them
|
||||
(`cancel_schedule` MCP / dashboard ✗) which tombstones via
|
||||
|
|
@ -120,20 +118,26 @@ One table:
|
|||
- `events(id, ts, kind, payload_json)` — every `LiveEvent` the
|
||||
harness emits during turn loop execution.
|
||||
|
||||
The harness writes; the host vacuums. `hive-c0re::events_vacuum`
|
||||
runs hourly and sweeps every existing agent harness dir. Retention
|
||||
is **type-scoped**: it deletes only the verbose `stream` rows (the
|
||||
raw claude `stream-json` deltas — one per text chunk / tool use, the
|
||||
bulk of the file's size) older than 14 days, and keeps every other
|
||||
kind (`turn_start`, `turn_end`, `note`, `status_changed`,
|
||||
`model_changed`, `token_usage_changed`, `turn_state_changed`)
|
||||
indefinitely — those are small and carry the semantic per-turn
|
||||
history the operator scrolls back through when debugging a
|
||||
regression. Age-only within the `stream` kind — no row cap — so a
|
||||
chatty turn doesn't lose its stream history sooner than a quiet one.
|
||||
Centralising retention on the host means a misbehaving harness can't
|
||||
disable its own vacuum and agents don't need any cleanup wiring of
|
||||
their own.
|
||||
The harness both writes and vacuums it — this used to be a host-side
|
||||
sweep, but hive-c0re runs as the unprivileged `hive-core` user under
|
||||
privsep and can't delete agent-owned files (host-side deletes hit
|
||||
`PermissionDenied` on the bash-task trio and a readonly-database error
|
||||
here), so cleanup moved in-container. `hive-agent`'s `vacuum::run`
|
||||
(`hive-agent/src/vacuum.rs`) sweeps hourly. Retention is
|
||||
**type-scoped**: it deletes only the verbose `stream` rows (the raw
|
||||
claude `stream-json` deltas — one per text chunk / tool use, the bulk
|
||||
of the file's size) older than 14 days, and keeps every other kind
|
||||
(`turn_start`, `turn_end`, `note`, `status_changed`, `model_changed`,
|
||||
`token_usage_changed`, `turn_state_changed`) indefinitely — those are
|
||||
small and carry the semantic per-turn history the operator scrolls
|
||||
back through when debugging a regression. Age-only within the
|
||||
`stream` kind — no row cap — so a chatty turn doesn't lose its stream
|
||||
history sooner than a quiet one. The trade-off (accepted): a
|
||||
misbehaving harness could now skip its own cleanup, which the old
|
||||
host-side sweep was meant to prevent — but a compromised harness is
|
||||
already inside the container trust boundary
|
||||
([`docs/security.md`](security.md)), and these are ephemeral local
|
||||
artifacts, so cleaning them up where they live is the honest fix.
|
||||
|
||||
Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/harness`
|
||||
setups). On open failure the `Bus` falls back to no-store mode
|
||||
|
|
@ -190,12 +194,10 @@ Shape:
|
|||
The turn loop is the only writer today, but it still goes
|
||||
read-modify-write under a shared in-process lock and merges into the
|
||||
existing object rather than reconstructing it — so a second writer
|
||||
preserves fields it doesn't own, and the lock closes the lost-update
|
||||
window between a writer's read and its rename. (The forge notification
|
||||
poller used to be that second writer, for a delivery-dedupe cursor. It
|
||||
persists nothing now — forge's own read-state is the durable record of
|
||||
what has been delivered — and it is a separate process, which an
|
||||
in-process lock could not have serialised anyway.)
|
||||
would preserve fields it doesn't own, and the lock closes the
|
||||
lost-update window between a writer's read and its rename. The lock
|
||||
is in-process only, so it wouldn't serialise a writer running as a
|
||||
separate process; none of today's writers are.
|
||||
|
||||
hive-c0re reads this file on each `build_all` sweep (~10s) via
|
||||
`container_view::read_harness_flags`. Falls back to the legacy individual
|
||||
|
|
@ -209,17 +211,22 @@ Full stdout + stderr capture for every `nixos-container` / `nix
|
|||
build` invocation the lifecycle layer fires. One row per invocation;
|
||||
the row accumulates lines as the child runs.
|
||||
|
||||
Replaces the legacy 32-line stderr ring buffer that `lifecycle::run`
|
||||
kept. The ring tail routinely truncated real eval errors ("tried
|
||||
alternatives" blocks alone are often 30+ lines), so failures bailed
|
||||
with an arbitrary tail whose full stream only lived in the host
|
||||
journal. With this table the dashboard can surface the entire log.
|
||||
Capturing the full stream (rather than a short tail buffer) matters
|
||||
because real eval errors routinely run long — "tried alternatives"
|
||||
blocks alone are often 30+ lines — so a truncated tail would cut off
|
||||
the actual failure and leave only the host journal holding the
|
||||
complete output. With this table the dashboard can surface the entire
|
||||
log.
|
||||
|
||||
Two indices:
|
||||
Three indices:
|
||||
- `(agent, started_at)` — backs the per-agent latest-N lookup used
|
||||
by the agent card chip.
|
||||
- `(status, finished_at)` — backs the retention sweep that runs
|
||||
as part of the existing hourly vacuum.
|
||||
- `(node_id)` — added by a later migration so a build log row can be
|
||||
looked up by the job-queue node it belongs to (a `hive_jobq` node is
|
||||
immutable after insert, so the link is recorded on the log row
|
||||
instead); legacy rows predating the column keep `node_id IS NULL`.
|
||||
|
||||
Writes are best-effort: `append_stdout` / `append_stderr` / `finish`
|
||||
log a warning on sqlite error and let the build continue. A failed
|
||||
|
|
@ -243,7 +250,7 @@ and inbox messages queue unacked until it's removed (see
|
|||
|
||||
Unusually, it's read and written from **both** sides of the harness
|
||||
bind-mount, and that's the whole design: the harness stats it
|
||||
in-container via `hive_agent::paths::paused_marker`, while hive-c0re
|
||||
in-container via `hive-agent`'s `paths::paused_marker`, while hive-c0re
|
||||
stats it on the host (`Coordinator::is_paused`) to populate the
|
||||
`paused` field on the agent card, and creates/removes it
|
||||
(`Coordinator::set_paused`) for `hivectl agent <name> pause|resume` and the
|
||||
|
|
@ -288,15 +295,26 @@ Under `/var/lib/hyperhive/agents/<name>/`:
|
|||
- `bash-tasks/` — task JSON + stdout/stderr files for
|
||||
background `mcp__bash__run` jobs. JSON files are
|
||||
`<id>.json` (status + tails), `<id>.out` / `<id>.err`
|
||||
(full captured output). `hive-c0re::bash_tasks_vacuum` runs
|
||||
hourly and deletes terminal task trios older than 48 hours;
|
||||
non-terminal (still-running) tasks are never deleted by vacuum.
|
||||
- `hyperhive-todos.sqlite` — loose-ends-v2 todo store. In-container
|
||||
daemons (`hive-bash-daemon`, `hive-matrix-daemon`, `hive-forge-notify`)
|
||||
upsert keyed todos here over the harness's in-agent socket
|
||||
(`HIVE_AGENT_SOCKET`); the harness merges them into `get_loose_ends`
|
||||
output and clears a row on `mark_todo_done`. Replaced the old
|
||||
file-based `mcp-loose-ends/` scanner.
|
||||
(full captured output). The harness's own hourly sweep
|
||||
(`hive-agent`'s `vacuum::run`, same one that ages out `stream`
|
||||
event rows above) deletes terminal task trios older than 48
|
||||
hours; non-terminal (still-running) tasks are never deleted. This
|
||||
used to be a host-side `hive-c0re` vacuum, moved in-container for
|
||||
the same privsep-ownership reason as the events vacuum above.
|
||||
- `hyperhive-state.sqlite` — consolidated loose-ends-v2 store: todos,
|
||||
reminders, and a questions mirror, one small table each in a single
|
||||
file (in-container daemons — `hive-bash-daemon`, `hive-matrix-daemon`,
|
||||
`hive-forge-notify` — upsert keyed todos here over the harness's
|
||||
in-agent socket, `HIVE_AGENT_SOCKET`; the harness merges them into
|
||||
`get_loose_ends` output and clears a row on `mark_todo_done`).
|
||||
Replaces three formerly-separate files
|
||||
(`hyperhive-todos.sqlite`, `hyperhive-reminders.sqlite`, and the
|
||||
old file-based `mcp-loose-ends/` scanner before that) — a one-time
|
||||
boot migration (`db_migrate::run`) folds the legacy files into this
|
||||
path the first time a harness boots after the upgrade. Also backs
|
||||
the `mcp__hyperhive__remind` queue, which moved from a host-side
|
||||
`broker.sqlite` table to this per-agent store as part of the same
|
||||
migration.
|
||||
|
||||
The harness itself is also a producer, not just the socket server:
|
||||
boot wiring's `spawn_todo_socket` starts `todo_server::run` (the
|
||||
|
|
@ -323,11 +341,9 @@ notes, clearing a stuck sentinel) as well as reading it.
|
|||
|
||||
**`harness` is not mounted at all.** It holds the child's own runtime
|
||||
material — `bash-tasks/`, the turn-stats and event sqlite dbs — and
|
||||
nothing argues for a parent reading it, let alone writing it. It used to
|
||||
be mounted RW for "the same management reasons" as `state`, which was
|
||||
never an argument so much as the side-effect of one loop treating all
|
||||
three dirs alike. hive-c0re reads a child's harness dir **directly on the
|
||||
host** when it wants those stats, which needs no mount into the parent.
|
||||
nothing argues for a parent reading it, let alone writing it. hive-c0re
|
||||
reads a child's harness dir **directly on the host** when it wants
|
||||
those stats, which needs no mount into the parent.
|
||||
|
||||
**`config` is read-only, including for the parent.** A config change is
|
||||
a PR on the child's config repo, made from a clone and merged after
|
||||
|
|
@ -339,8 +355,9 @@ boundary a convention rather than a permission.
|
|||
⚠️ Not to be confused with the seeding done when an `InitConfig`
|
||||
approval resolves: that writes the child's initial config repo as
|
||||
**hive-c0re, against the host path**, and `read_only` on a bind
|
||||
constrains writers *inside* a container only. The two are unrelated, and
|
||||
reading them as the same thing is what kept this mount writable.
|
||||
constrains writers *inside* a container only. The two are unrelated —
|
||||
conflating them is an easy way to reason your way into thinking this
|
||||
mount should be writable when it shouldn't.
|
||||
|
||||
Per-child isolation still holds: a container only ever has its *own*
|
||||
dirs plus its direct children's bind-mounted, never a sibling's.
|
||||
|
|
@ -363,8 +380,11 @@ Contents:
|
|||
audit trail (one commit per successful deploy or hyperhive bump).
|
||||
- `topology.json` — parent/child agent graph
|
||||
(`{ "alice": "root", "bob": "alice", "root": null }`).
|
||||
Written by `topology::set_parent`; read by the dashboard, the
|
||||
renderer, and `<parent>` / `<children>` recipient resolution.
|
||||
Written by `topology::apply_set_parent` (the pure move-validating
|
||||
transform) via `meta::bulk_commit_topology` (the committer — see the
|
||||
`Reparent` node in [`docs/coordinator.md`](coordinator.md)); read by
|
||||
the dashboard, the renderer, and `<parent>` / `<children>` recipient
|
||||
resolution.
|
||||
- `tool-groups.json` — per-agent MCP tool group grants
|
||||
(`{ "alice": ["messaging", "inbox", "execution"] }`). Written by
|
||||
`tool_groups::set_groups`; injected as `HIVE_TOOL_GROUPS` env
|
||||
|
|
@ -396,25 +416,21 @@ deleted.
|
|||
|
||||
## Destroy vs purge
|
||||
|
||||
- `DESTR0Y` (default) — stops + removes the nspawn container,
|
||||
drops the systemd drop-in, fails any pending approvals. State
|
||||
dirs stay put; the agent appears in the dashboard's K3PT ST4T3
|
||||
section as a tombstone with `⊕ R3V1V3` and `PURG3` actions.
|
||||
`R3V1V3` queues a Spawn approval that reuses the kept state on
|
||||
approve (no re-login).
|
||||
- `PURG3` (opt-in via the dashboard button or
|
||||
`hivectl agent <name> destroy --purge`) — DESTR0Y plus wipes
|
||||
`/var/lib/hyperhive/{agents,applied}/<name>/`. Config history,
|
||||
claude creds, /state/ notes, and the harness dir are all gone.
|
||||
No undo.
|
||||
See [For operators](#for-operators) above for what each action does to
|
||||
an agent's state. The mechanics, for completeness:
|
||||
|
||||
The root/bootstrap container is **imperative** infrastructure — managed
|
||||
end-to-end by hive-c0re, not declared in the host's NixOS config.
|
||||
`auto_update::ensure_root_agent` recreates it on the next hive-c0re
|
||||
startup if it's absent (bypassing the approval queue, as required
|
||||
infrastructure). A soft policy guard in `actions::destroy` currently
|
||||
refuses to destroy it; even without that guard, destroying it would only
|
||||
be transient — hive-c0re brings it back on the next startup.
|
||||
- `DESTR0Y` also drops the systemd drop-in and fails any pending
|
||||
approvals; the tombstone's `⊕ R3V1V3` button queues a Spawn approval
|
||||
that reuses the kept state on approve.
|
||||
- `PURG3` wipes `/var/lib/hyperhive/{agents,applied}/<name>/` — the
|
||||
union of everything `DESTR0Y` left behind.
|
||||
|
||||
The root/bootstrap agent's specialness is implemented as a soft policy
|
||||
guard in `actions::destroy` that refuses to destroy it, backstopped by
|
||||
`auto_update::ensure_root_agent`, which recreates it on the next
|
||||
hive-c0re startup if it's ever absent (bypassing the approval queue,
|
||||
as required infrastructure) — so even without the guard, destroying it
|
||||
would only be transient.
|
||||
|
||||
### btrfs subvolumes for `/var/lib/hyperhive/agents/<name>`
|
||||
|
||||
|
|
@ -440,8 +456,17 @@ until an explicit opt-in upgrade.
|
|||
actually a subvolume, then the normal `remove_dir_all` sweep covers
|
||||
plain-dir agents + the applied dir.
|
||||
|
||||
Per-subvolume disk-usage accounting and optional quotas are a
|
||||
follow-up (the qgroup work), not part of the base migration.
|
||||
Per-subvolume disk-usage accounting and optional quotas have since
|
||||
landed as the qgroup work: `hivectl quota-enable` turns on btrfs
|
||||
qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and
|
||||
`hivectl agent <name> quota show|set` reads/limits one agent's
|
||||
subvolume usage through the same hive-priv-mediated path as
|
||||
subvolume creation/deletion above.
|
||||
|
||||
This is the same subvolume `hivectl agent <name> subvol snapshot push`
|
||||
sends to the swarm's snapshot store — see
|
||||
[`docs/snapshot-store.md`](snapshot-store.md) for what a pushed
|
||||
snapshot contains and how the store authenticates a sender.
|
||||
|
||||
## Run-time dirs
|
||||
|
||||
|
|
@ -510,10 +535,12 @@ auto-injected `extraMcpServers.matrix` entry read).
|
|||
**First-boot ordering**: hive-c0re provisions the matrix token AFTER
|
||||
agent containers come up. Without the path-trigger sibling
|
||||
(`systemd.paths.hive-matrix-daemon`, `PathExistsGlob =
|
||||
/agents/*/state/matrix-token`), the daemon would exit 0 quietly the
|
||||
first time it ran and the MCP would have no backend until the next
|
||||
restart. The `.path` unit makes the appearance of the token re-fire
|
||||
the service so the daemon comes alive in the same boot cycle as
|
||||
/agents/*/state/matrix-token*` — the trailing `*` also catches a
|
||||
secondary multi-account token like `matrix-token-ccc`), the daemon
|
||||
would exit 0 quietly the first time it ran and the MCP would have no
|
||||
backend until the next restart. The `.path` unit makes the appearance
|
||||
of the token re-fire the service so the daemon comes alive in the
|
||||
same boot cycle as
|
||||
provisioning. The same token watcher also drives avatar setting: on a
|
||||
restart the daemon re-runs each account's bring-up, which sets the
|
||||
avatar (see below).
|
||||
|
|
|
|||
|
|
@ -5,9 +5,8 @@
|
|||
The sections below document specific mechanisms (the state-file endpoint,
|
||||
nixbld isolation, privilege separation). This section frames the model they
|
||||
serve: **what hyperhive defends, what it deliberately does not, and where the
|
||||
operator is accepting risk.** It emerged from a security discussion on
|
||||
2026-06-24 (prompted by the `gh` CLI helper work) and is the reference for
|
||||
"is it safe to give an agent capability X?".
|
||||
operator is accepting risk.** It is the reference for "is it safe to give an
|
||||
agent capability X?".
|
||||
|
||||
### The trust boundary is the container, not credential storage
|
||||
|
||||
|
|
@ -135,9 +134,9 @@ dashboard renders anchors only for tokens that passed the same checks the
|
|||
read endpoint enforces.
|
||||
|
||||
The same invariant holds wherever an agent-supplied name reaches a filesystem
|
||||
path: the agent socket's `GetAgentMeta` validates `name` with
|
||||
`validate_agent_name` before building `agent_notes_dir(name)`, so a `..`
|
||||
component can't traverse.
|
||||
path: the agent socket's `GetAgentMeta` takes `name` as a serde-validated
|
||||
`hive_types::Ident` (or falls back to `Ident::parse` for the "self" case)
|
||||
before building `agent_notes_dir(name)`, so a `..` component can't traverse.
|
||||
|
||||
## Nix builds and credential isolation
|
||||
|
||||
|
|
@ -167,8 +166,8 @@ token policy bounds file reads; network isolation bounds network reach.
|
|||
- `/home/<name>/.claude/` — mode `0700`, owned by the per-agent
|
||||
user `<name>`. nixbld users cannot read it.
|
||||
- `$HYPERHIVE_STATE_DIR/forge-token` (= `/agents/<name>/state/forge-token`)
|
||||
— written at mode `0600` by `hive-c0re/src/forge.rs` and chowned to the
|
||||
per-agent uid:gid by `lifecycle::chown_to_agent`. nixbld users
|
||||
— written at mode `0600` and chowned to the per-agent uid:gid (see
|
||||
`hive-c0re/src/forge/mod.rs`'s module doc for exactly where). nixbld users
|
||||
cannot read it.
|
||||
|
||||
**Policy**: all credential files written to agent state directories MUST be mode
|
||||
|
|
@ -210,7 +209,8 @@ known operations; there is no arbitrary command pass-through:
|
|||
|
||||
| Operation | What it runs |
|
||||
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------- |
|
||||
| `StartContainer` / `StopContainer` / `KillContainer` | `nixos-container start/stop/kill <name>` |
|
||||
| `StartContainer` / `StopContainer` | `nixos-container start/stop <name>` |
|
||||
| `KillContainer` | `machinectl kill <machine> --signal=SIGKILL` (`nixos-container` has no kill verb) |
|
||||
| `CreateContainer` / `UpdateContainer` | `nixos-container create/update <name> --flake <ref>` |
|
||||
| `DestroyContainer` | `nixos-container destroy <name>` |
|
||||
| `ListContainers` | `nixos-container list` |
|
||||
|
|
@ -227,7 +227,9 @@ known operations; there is no arbitrary command pass-through:
|
|||
**Container allowlist** — every request is validated against an
|
||||
allowlist before any operation: only names matching the agent-name
|
||||
convention (char-validated) or the known sibling service containers
|
||||
(`hive-gateway`, `hive-forge`, `hive-matrix`, `hive-ci`) are accepted.
|
||||
(`hive-forge`, `hive-matrix`, `hive-ci`) are accepted. `hive-gateway` is
|
||||
a host unit, not a container, so it is not in this list — see
|
||||
`ReloadGatewayNginx` above for how its access is scoped instead.
|
||||
Arbitrary container names are rejected.
|
||||
|
||||
**Socket-activated** — systemd starts `hive-priv` on the first
|
||||
|
|
|
|||
|
|
@ -183,7 +183,10 @@ the access-review list.
|
|||
### What a snapshot contains
|
||||
|
||||
The snapshot covers an agent's **state subvolume**, which is the parent
|
||||
of `state/`, `claude/` and `harness/`. Consequences:
|
||||
of `state/`, `claude/` and `harness/` (see
|
||||
[`docs/persistence.md`'s btrfs subvolume
|
||||
section](persistence.md#btrfs-subvolumes-for-varlibhyperhiveagentsname)
|
||||
for how and when that subvolume is created). Consequences:
|
||||
|
||||
- The Claude session (`claude/`) travels, so a restored agent keeps its
|
||||
live `--continue` session rather than needing to log in again.
|
||||
|
|
|
|||
|
|
@ -73,14 +73,11 @@ and `qualify()` / `qualified_label()` semantics.
|
|||
|
||||
## Swarm CA
|
||||
|
||||
A hive's internal TLS chains to a **swarm root CA**: the root signs each
|
||||
hive's own CA, and that hive CA signs the gateway leaf, so a peer that
|
||||
trusts the root once validates every hive in the swarm rather than being
|
||||
pinned to each one by hand.
|
||||
|
||||
Provisioning modes, what to hand a peer (`trust-bundle.pem`, never
|
||||
`ca.pem`), the name constraints on a hive CA, and how an existing hive
|
||||
adopts the hierarchy: [`ca.md`](ca.md).
|
||||
A hive's internal TLS chains to a **swarm root CA**, so a peer that
|
||||
trusts the root validates every hive in the swarm rather than being
|
||||
pinned to each one by hand. Provisioning modes, what to hand a peer
|
||||
(`trust-bundle.pem`, never `ca.pem`), the name constraints on a hive
|
||||
CA, and how an existing hive adopts the hierarchy: [`ca.md`](ca.md).
|
||||
|
||||
## Running the swarm's shared services
|
||||
|
||||
|
|
@ -183,17 +180,17 @@ environment and forwarded to agent containers.
|
|||
|
||||
## What the config does at runtime
|
||||
|
||||
1. **Dashboard P33RS tab** — `parse_peer_hives()` in `dashboard.rs`
|
||||
reads `HYPERHIVE_PEERS` and includes
|
||||
`peer_hives: Vec<{ name, url }>` in `/api/state`. The dashboard
|
||||
shows a P33RS tab (hidden when the list is empty) with a card per
|
||||
peer linking to `https://{domain}/`. See
|
||||
`docs/web-ui/dashboard.md` § P33RS tab.
|
||||
1. **Dashboard P33RS tab** — hive-c0re reads `HYPERHIVE_PEERS` and
|
||||
surfaces it as the peer list in the dashboard's state API. The
|
||||
dashboard shows a P33RS tab (hidden when the list is empty) with a
|
||||
card per peer linking to `https://{domain}/`. Wire format + module
|
||||
pointer: `docs/web-ui/dashboard.md` § P33RS tab.
|
||||
|
||||
2. **Agent identity** — the same `HYPERHIVE_PEERS` env var is
|
||||
forwarded to agent containers by `meta.rs`; agent code can call
|
||||
`identity::peers()` to discover peer hives and address them with
|
||||
qualified names (`agent@domain`).
|
||||
forwarded to agent containers, so agent code can discover peer
|
||||
hives and address them with qualified names (`agent@domain`). See
|
||||
`hive-agent/src/identity.rs`'s module doc for the label/domain
|
||||
helpers.
|
||||
|
||||
3. **Matrix federation** — when `matrix.enable` is on, tuwunel
|
||||
federates with the peer's matrix server (discovered via the peer's
|
||||
|
|
|
|||
|
|
@ -76,26 +76,29 @@ sign leaves, and the chain stops there.
|
|||
|
||||
## What to hand a peer
|
||||
|
||||
`hivectl peer-config` prints the `cp` line. The file is
|
||||
`<tls.stateDir>/trust-bundle.pem` — the hive CA plus the swarm root —
|
||||
**not `ca.pem`**.
|
||||
`hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"`
|
||||
block a peer operator pastes into their own config. When this hive's
|
||||
gateway serves a self-signed leaf under the hierarchy (detected by the
|
||||
presence of `<tls.stateDir>/trust-bundle.pem`), it also prints a one-time
|
||||
`scp` line installing the **swarm root** —
|
||||
`<swarm.ca.stateDir>/root.pem`, not this hive's own CA — on the peer's
|
||||
host:
|
||||
|
||||
The distinction is load-bearing rather than cosmetic: once a hive CA is
|
||||
an intermediate, it is no longer something a verifier can build a chain
|
||||
*to*. OpenSSL will not terminate a chain at a trusted non-self-signed
|
||||
certificate without `-partial_chain`, so handing a peer the bare
|
||||
intermediate produces a verification failure that reads like a bad cert
|
||||
rather than like a missing anchor. The bundle carries both, so the same
|
||||
file works whichever mode issued it.
|
||||
```
|
||||
scp /var/lib/swarm-ca/root.pem <peer-host>:/var/lib/swarm-ca/root.pem
|
||||
```
|
||||
|
||||
On a hive that predates the swarm root the bundle is just that hive's
|
||||
self-signed CA, so the recipe does not change.
|
||||
That is the point of the hierarchy: the root is installed **once per
|
||||
swarm host**, not once per peer, so a hive joining later needs no edit on
|
||||
the hives already running. A hive whose cert already chains to a public
|
||||
CA has nothing to install — `peer-config` says so instead.
|
||||
|
||||
⚠️ Consumers must read `trust-bundle.pem`, never `ca.pem` directly. A
|
||||
consumer that reads `ca.pem` works fine on a hive that has always been
|
||||
self-signed and breaks the moment that hive adopts the hierarchy — so
|
||||
the failure is invisible on the deployment you are most likely to test
|
||||
on.
|
||||
Handing a peer this hive's own `ca.pem` would not work even as a
|
||||
one-off: once a hive CA is an intermediate under the swarm root, it is
|
||||
no longer something a verifier can build a chain *to* — OpenSSL will not
|
||||
terminate a chain at a trusted non-self-signed certificate without
|
||||
`-partial_chain`. That is why the tool distributes the root, not a
|
||||
per-hive file.
|
||||
|
||||
## Adopting the hierarchy on an existing hive
|
||||
|
||||
|
|
@ -176,12 +179,10 @@ Two consumers, and only one of them is fine:
|
|||
|
||||
The consumption differs per runtime and is the part worth knowing.
|
||||
tuwunel links no openssl, which makes `SSL_CERT_FILE` look inapplicable
|
||||
— it isn't. Its outbound client is `reqwest` with the `rustls` feature,
|
||||
which builds a `rustls_platform_verifier::Verifier`; because tuwunel
|
||||
calls `tls_certs_merge` (additive) rather than `tls_certs_only`, the
|
||||
platform roots stay alongside its compiled-in webpki set. On Linux that
|
||||
verifier resolves through `rustls-native-certs` → `openssl-probe`,
|
||||
which reads `SSL_CERT_FILE`.
|
||||
— it isn't: its rustls-based TLS stack still resolves trust through the
|
||||
platform certificate store on Linux, and that store honors
|
||||
`SSL_CERT_FILE`, so the env var takes effect the same way it would for
|
||||
an OpenSSL-linked binary.
|
||||
|
||||
> ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.**
|
||||
> `openssl-probe` uses it *instead of* the default store, so naming
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ and has no `enable` to derive from anything.
|
|||
### SSO (authelia)
|
||||
|
||||
One authelia per swarm, in a `swarm-authelia` container, at
|
||||
`auth.<hive-domain>`. Operator and agents are both subjects of the same
|
||||
`auth.<swarm-domain>`. Operator and agents are both subjects of the same
|
||||
provider, differentiated by roles and claims rather than by mechanism —
|
||||
there is one IdP and one auth path.
|
||||
|
||||
|
|
@ -57,3 +57,7 @@ small-deployment choices, and the scope is the justification: redis
|
|||
buys shared session state across replicas and there is one instance;
|
||||
SMTP exists to mail humans, and provisioning here is programmatic.
|
||||
|
||||
See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC
|
||||
relying-party flow, and [`secrets.md`](secrets.md) for where each of
|
||||
authelia's keys is generated and read.
|
||||
|
||||
|
|
|
|||
|
|
@ -91,22 +91,12 @@ an attribute edit can invalidate a login by accident.
|
|||
## What secrets exist, and where each one lives
|
||||
|
||||
Every secret in the swarm, with its generator and its path, is tabulated
|
||||
in one place: [`secrets.md`](secrets.md). The rows relevant here are
|
||||
authelia's own keys (session, JWT, storage-encryption, OIDC HMAC, OIDC
|
||||
issuer) plus the two halves of each client secret.
|
||||
|
||||
What matters for this page is the shape rather than the paths. Authelia's
|
||||
own keys are generated **in-container**, because nothing outside that
|
||||
container ever reads them — that is the test worth applying to any secret
|
||||
added here. The plaintext half of a client secret is the one that fails
|
||||
it: its reader lives in a different container, and that is the entire
|
||||
reason a delivery step exists.
|
||||
|
||||
**None of it is ever written into a nix expression.** authelia's
|
||||
`settings` are rendered into the nix store, which is world-readable and
|
||||
permanent, so the client digest reaches authelia through `settingsFiles`
|
||||
(merged at runtime) and every other secret through a `*File` option
|
||||
carrying a path rather than a value.
|
||||
in one place: [`secrets.md`](secrets.md), including authelia's own keys
|
||||
(session, JWT, storage-encryption, OIDC HMAC, OIDC issuer) and the two
|
||||
halves of each client secret. That page's two rules — a secret is always
|
||||
a path, never a value, and the generator and the reader are usually in
|
||||
different containers — are why the client secret's plaintext half needs
|
||||
the delivery step below and the rest of authelia's keys don't.
|
||||
|
||||
## Getting the plaintext to the relying party
|
||||
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue