vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

build agents only from internal/meta; delete the hive's local meta flake

Browse source 
The hive no longer renders, locks or commits a meta flake. A rebuild's
MetaSync node picks the internal/meta commit and records it with the
hyperhive and config pins it holds; Prebuild, Swap and Create build from it,
and a head that does not pin the agent's current config fails the build
naming the agent. hive-priv's meta_rev is required.

Removed: meta.rs sync/lock/render, the manual-rebuild relock, the boot
sweep's hyperhive bump (MetaLock), the update-meta-inputs form and its
panel, MetaWindow, the hive's hyperhive_rev setting.

Kept: agent_set::refresh holds what sync_agents also did (topology.json, the
gateway's agent-socket map and nginx include), called from spawn, destroy,
rebuild and the migration.

The boot reconcile and the stale-start upgrade compare the agent's marker
with the hyperhive pin of internal/meta; the dashboard's deployed chip reads
the config marker a successful build writes.
This commit is contained in:
atlas 2026-10-25 13:33:15 +01:00
commit 4aa11b3a94
64 changed files with 1042 additions and 3793 deletions
Download patch file Download diff file Expand all files Collapse all files

8
docs/agent-lifecycle/agent-roster.md
View file

@ -25,7 +25,7 @@ create` (or the swarm UI, via `POST /api/agents`) writes that identity
once, by calling `ensure_agent_identity`.
This page is about a different, **hive-local** file: the hive-c0re-owned
**meta repo**, alongside `flake.nix`, at
`meta/` directory's `topology.json`, at
`/var/lib/hyperhive/meta/topology.json`:
```json
@ -102,10 +102,8 @@ state dirs work as for every other agent.
`/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can
manage any agent's state dir — config isn't authored there, since a
real config change is a PR from a clone), plus RO mounts for
`/applied` (diff against what's deployed) and `/meta` (system-wide
deploy log). Any agent holding it gets the same mounts.
The operator dashboard's `POST /api/meta-update` is the only path
that writes `flake.lock`.
`/applied` (diff against what's deployed) and `/meta` (the hive's
topology and limits files). Any agent holding it gets the same mounts.
- **Prompt** — identical for every container but one block (below):
`prompts/system.md`
carries `<!-- role:agent -->` / `<!-- role:manager -->` marker

139
docs/agent-lifecycle/config-changes.md
View file

@ -113,9 +113,11 @@ until you create ruth again from the swarm dashboard.
## Meta flake
The hive-c0re-owned repo at `/var/lib/hyperhive/meta/`
declares one flake input per agent (`agent-<n>.url =
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`) and one
A hive has no meta flake of its own. It builds every agent from the swarm's
`internal/meta`, which swarm-controller commits ([below](#the-swarms-meta-flake)).
It declares one flake input per agent (`agent-<n>.url =
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`, the head of `main` on the
agent's forge config repo, the one config PRs merge into) and one
`nixosConfigurations.<n>` output per agent. Each output wraps
`inputs.agent-<n>.nixosModules.default` with the identity +
`HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host
@ -123,53 +125,32 @@ declares one flake input per agent (`agent-<n>.url =
address) aren't in it:
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
start (see `docs/process/conventions.md`).
Containers run against `--flake /var/lib/hyperhive/meta#<n>`, or against
`internal/meta` when it pins the agent ([below](#the-swarms-meta-flake)).
A config that sits only in the hive's `applied/<n>` mirror, not on forge
`main`, never deploys.
The declared input url is the head of `main` on the agent's forge config repo
(`agent-configs/<n>`, the one config PRs merge into), which `sync_agents`
reads at render time. The flake text depends only on those shas and the
agent set, so any machine that can reach the forge resolves it. `sync_agents` fetches
each repo's `main` with the hive's read token (`forge::reader_token()`),
pins its head, and takes the canonical `follows` lines from the
`flake.lock` at that commit. If the forge is unreachable or a repo has no
`main`, `sync_agents` fails for that agent with its name and writes
nothing; there is no fallback to a local path. A config that sits only in
the hive's `applied/<n>` mirror, not on forge `main`, never deploys.
The rebuild paths re-lock the input with `nix flake update agent-<n>` against that
same pinned url, so `flake.nix` and `flake.lock` name no local path.
What `hive-c0re` still keeps under `/var/lib/hyperhive/meta/` is plain
state, bound read-only into the root agent's container as `/meta`:
`topology.json`, the swarm resource limits, and the permission seed files of a
hive that predates the swarm's permissions. `agent_set::refresh(agents:
&[AgentSpec])` keeps `topology.json`, the gateway's agent-socket map and its
nginx include in line with the live agents. `provision_container`, `destroy`,
`rebuild` and the startup migration call it. It refuses an empty agent list,
which means `nixos-container list` failed, and it fails when `HIVE_FORGE_URL`
is unset, since every agent would come up with no forge.
Lock updates are single-phase and commit when the lock changed:
`meta::lock_update_for_rebuild(name)` relocks one agent's input for a
relocking rebuild (the manual `↻ R3BU1LD` button, and the rebuild a
merged config commit queues), and `meta::lock_update_hyperhive()` is the
autoupdate flake-rev bump (one shot before per-agent rebuilds).
`meta::sync_agents(hive: &HiveEnv, agents: &[AgentSpec])` — `hive`
carries `hyperhive_flake`, `dashboard_port`, and the rest of the
per-hive config — is the idempotent reconciler called by `provision_container`,
`destroy`, `rebuild`, and the startup migration. Renders `flake.nix`
from the agent list; if it differs from disk, or `flake.nix` or `flake.lock`
has an uncommitted change (a sync stopped before its commit), runs
`nix flake lock` + commits as `regenerate meta flake` (or
`seed meta from N agent(s)` on the first call). A clean tree whose
`flake.nix` already matches returns without running nix.
The root agent has `/meta` RO-bound inside its container:
`git -C /meta log --oneline` is the swarm-wide deploy log,
`cat /meta/flake.lock | jq '.nodes["agent-<n>"].locked'`
resolves which sha the flake pins each agent at right now.
Dashboard surfaces the same info as a `deployed:<sha12>` chip
per container row.
The dashboard's `deployed:<sha12>` chip per container row shows the config
commit the agent's last successful build pinned.
## The swarm's meta flake
swarm-controller also commits a meta flake of its own, `internal/meta`, with
the same renderer: one `flake.nix` and `flake.lock` for every agent any hive
places (the pins: [`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)).
A hive builds an agent from it when the repo has a commit that pins the
agent's current config, and from its own meta flake above otherwise, so a hive
older or newer than the controller keeps building.
swarm-controller commits `internal/meta`: one `flake.nix` and `flake.lock` for
every agent any hive places (the pins:
[`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)). A hive builds
every agent from it. While the repo has no commit that pins an agent's current
config, that agent's build fails and names the agent. The controller queues
its update of the repo before the deploy request for a merged config commit
and for a new agent, so a failure here is rare. It sends the deploy even when
that update failed, and the build then fails as described.
- **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's
default-branch head with the hive's read token (`forge::reader_token()`)
@ -178,8 +159,9 @@ older or newer than the controller keeps building.
`main`, an archetype agent from the archetype it has now. An agent switched
between custom and archetype, or to another archetype, fails that test.
A hive without the `core` token file that nix's git reads, a repo with no
commit, a head that lacks the agent and a head behind the config all select
the hive's own flake. `MetaSync` records the commit in `hive-c0re`'s own state,
commit, a head that lacks the agent and a head behind the config each fail
the build with an error that names the agent. `MetaSync` records the commit,
with the hyperhive and config commits the head pins, in `hive-c0re`'s own state,
`/var/lib/hyperhive/meta-rev/<n>`, which the agent can't write (the container mounts its runtime
directory read-write); `Prebuild` and `Swap` read that
record, so both build one flake. A first deploy resolves the commit in its `Create`
@ -190,10 +172,8 @@ older or newer than the controller keeps building.
`CreateContainer`). `hive-priv` accepts only a 40-digit lowercase hex
commit and builds the flake from its own forge URL
(`HIVE_PRIV_FORGE_URL`); the URL never comes from a request. On the wire the
`hive_priv_sock::PrivRequest` variants carry an optional field:
`hive_priv_sock::PrivRequest` variants carry a required field:
`{"op":"update_container","name":"atlas","stream":true,"meta_rev":"<40 hex>"}`.
Without it, as from a hive-c0re that predates the field, hive-priv builds the
hive's own flake.
- **Fetching.** nix fetches the forge's inputs with `git`. Both `hive-c0re`
and `hive-priv` carry a `$HOME/.gitconfig` that answers for the forge host
through `git-credential-hive-forge` (the `core` token file) and rewrites the
@ -201,6 +181,11 @@ older or newer than the controller keeps building.
`http`.
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
runs the hyperhive of the controller's deploy, not its hive's.
- **Markers.** After a successful build `hive-c0re` writes the hyperhive
commit the head pins to `applied/.<n>.hyperhive-rev`, and the config
commit to `applied/.<n>.config-rev` (none for an archetype agent). The boot
reconcile ([below](#autoupdate-on-startup)) and the dashboard's update chip
read them.
## Two repos per agent
@ -223,11 +208,9 @@ older or newer than the controller keeps building.
├── agent.nix # working tree of main
└── <other committed files> # also tracked
/var/lib/hyperhive/meta/ swarm-wide flake — core
├── .git/ # one commit per lock
│ # change
├── flake.nix # generated from agent set
└── flake.lock # pins each agent's sha
/var/lib/hyperhive/meta/ plain hive state — core
├── topology.json # the live agent set
└── swarm-resource-limits.json # the swarm's per-agent caps
```
Why two physical repos: the submitting agent's `/agents/<n>/config/` is
@ -236,8 +219,8 @@ proposed tree. The applied repo is never bind-mounted (except
the read-only `.git` exposure described below) so a destructive
move inside the container can't reach it.
The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>` (or `internal/meta`)
(see "Meta flake" above). The agent's own `applied/<n>/flake.nix`
The container's `--flake` ref is `internal/meta` (see "Meta flake" above).
The agent's own `applied/<n>/flake.nix`
is a fixed boilerplate that exports `nixosModules.default =
import ./agent.nix`; the meta flake imports that module and
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
@ -246,7 +229,7 @@ wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
hive-c0re plants `deployed/0` on the seed commit at first spawn. A
merged config commit that deploys plants no tag: the merged PR on the
forge and meta's lock commits record it. A config PR nobody merges
forge and `internal/meta`'s commits record it. A config PR nobody merges
carries no extra state on the forge side: the PR stays open, and
the submitter pushes again (or closes it) to retry.
@ -335,9 +318,6 @@ git -C /agents/<n>/config log applied/main --oneline
git -C /agents/<n>/config show applied/refs/tags/deployed/0 # the seed commit
git -C /agents/<n>/config show applied/refs/tags/denied/<id> # body = operator note
git -C /agents/<n>/config rebase applied/main # base in-flight work on what's deployed
git -C /meta log --oneline # swarm-wide deploy history
cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))'
```
The RO binds block push at the kernel level — git plumbing inside the
@ -346,16 +326,17 @@ container can't corrupt either authoritative repo.
## Startup migrations (older hosts)
hive-c0re runs a couple of idempotent migrations on every startup so a
host set up before the tag-driven-deploy + meta-flake scheme (both
described above) converges to it automatically. Each phase is a no-op
host set up before the tag-driven-deploy scheme (described above)
converges to it automatically. Each phase is a no-op
once already applied:
- **Tags**: hive-c0re tags agents from before the tag-driven scheme
`deployed/0` on `main` once. Non-destructive — it doesn't touch live
containers, state dirs, or claude creds.
- **Meta flake**: rewrites each `applied/<n>/flake.nix` to the
- **Agent files**: rewrites each `applied/<n>/flake.nix` to the
module-only boilerplate, wires the `applied` remote in each proposed
repo, and bootstraps the meta repo from the current agent list. Set
repo, and refreshes the files that follow the agent set
(`agent_set::refresh`) from the current agent list. Set
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
No state loss in either migration: claude creds, `/state/` notes, the
@ -450,19 +431,25 @@ swarm-level alert could carry the same signal) — only reach for a new
`hive-c0re serve` runs `auto_update::run` in a background task right
after opening the coordinator. It enumerates managed containers and
rebuilds any whose recorded hyperhive rev differs from the current
one — sub-agents and the root agent go through the same
`job_queue::templates::rebuild` DAG.
rebuilds any whose recorded hyperhive rev differs from the one
`internal/meta` pins now. That's how a hive that missed the swarm's
fan-out catches up. Sub-agents and the root agent go through the same
graceful rebuild subgraph.
"Rev" = canonical filesystem path of
`services.hyperhive.c0re.hyperhiveFlake`. Marker
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`. If the
flake input has no canonical path (for example a `github:` URL),
autoupdate is a no-op — rebuild manually.
"Rev" = the hyperhive commit pinned by the head of `internal/meta`, as
the hive reads it with its forge token. Marker
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`, written after
a successful build. If the hive can't read the pin (no forge token, the
forge down, `internal/meta` not published yet), the scan rebuilds
nothing; a rebuild would have no flake to build from either. A hive deploy
alone doesn't rebuild agents.
The dashboard surfaces pending updates per agent: a clickable
"needs update ↻" badge appears whenever the marker differs from
current rev. The badge POSTs `/api/rebuild/<name>`, which inserts the
A stopped agent gets its rebuild at its next start: the start path upgrades
the start of a stale, stopped agent to a rebuild followed by the start.
The dashboard surfaces pending config changes per agent: a clickable
"needs update ↻" badge appears whenever the agent's applied repo has a
commit that its last build didn't pin. The badge POSTs `/api/rebuild/<name>`, which inserts the
same `job_queue::templates::rebuild` DAG so manual triggers and the
startup scan can't drift. When at least one container is stale, a
top-level `↻ UPD4TE 4LL` button appears that loops over every

2
docs/agent-lifecycle/persistence.md
View file

@ -438,7 +438,7 @@ this hive's archetype agents runs, as last read from the swarm's
with an entry has no `applied/<n>` repo and no `agent-<n>` meta input; see
[archetypes](../swarm/archetypes.md).
hive-c0re renders containers onto `meta#<n>` at creation.
hive-c0re creates containers from `internal/meta` (`nixosConfigurations.<n>`).
## Destroy vs purge

5
docs/integrations/forge.md
View file

@ -75,8 +75,7 @@ forge container:
- fetching a merged config commit for a swarm deploy, and
`hivectl forge reconcile-config`
- reseeding a purged `applied` repo, and the meta-update cascade's
fast-forward of `applied/main`
- reseeding a purged `applied` repo
- seeding `proposed` from `agent-configs/<agent>` on first deploy
- cloning and pulling `/knowledge`
@ -84,7 +83,7 @@ The builds themselves read as `core`. nix fetches the config inputs and
`internal/meta` with `git`, and `git` gets its credential from
`git-credential-hive-forge`, which reads the `core` token file. A hive without
that file fetches nothing private through nix, whatever its own token reads, so
it builds from its own meta flake rather than `internal/meta`.
its builds fail: they have no other flake than `internal/meta`.
Roll a change to any of this out store host first (its granter writes the
controller's grant on `swarm/hives/+/forge/token`), then the controller host,

8
docs/process/gotchas.md
View file

@ -294,15 +294,15 @@ internal convention that routes an agent name to
```
# ❌ silently builds the wrong thing (or errors if attr doesn't exist)
nix build /var/lib/hyperhive/meta#argus.config.system.build.toplevel
nix build "git+<forge>/internal/meta.git?rev=<rev>#argus.config.system.build.toplevel"
# ✅ explicit path nix build actually resolves
nix build /var/lib/hyperhive/meta#nixosConfigurations.argus.config.system.build.toplevel
nix build "git+<forge>/internal/meta.git?rev=<rev>#nixosConfigurations.argus.config.system.build.toplevel"
```
`lifecycle::prebuild_toplevel` hit this once by constructing the attr
path as `{flake_ref}.config…` — which produced `meta#argus.config…`
instead of `meta#nixosConfigurations.argus.config…`. The fix:
path as `{flake_ref}.config…` — which produced `#argus.config…`
instead of `#nixosConfigurations.argus.config…`. The fix:
`split_once('#')` to separate flake path from name, then template
`{path}#nixosConfigurations.{name}.config.system.build.toplevel`.

141
docs/scheduler/coordinator.md
View file

@ -14,12 +14,12 @@ section covers the concrete node inventory hive-c0re builds on top of that
engine, plus the internals (module layout, scheduler mechanics,
resource/lease semantics).
hive-c0re submits every container/meta operation (rebuild, meta-update, first-spawn, power
hive-c0re submits every container operation (rebuild, first-spawn, power
changes) to the global job-DAG queue (`hive-c0re/src/job_queue/`)
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
concurrency comes from the resource classes below, not from multiple workers.
Special cases like graceful stop, deferred starts, and the meta-update
cascade need no bespoke code paths — each takes the shape of a DAG built
Special cases like graceful stop, deferred starts, and the boot sweep
need no bespoke code paths — each takes the shape of a DAG built
from the same primitive nodes.
### Two levels: DAG and node
@ -46,8 +46,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta |
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; builds from the `internal/meta` commit it resolves and marks the agent's pins |
Cheap — no build slot:
@ -55,13 +54,13 @@ Cheap — no build slot:
| Node | Behavior |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the agent-set refresh (`agent_set::refresh`), and the `internal/meta` commit `Prebuild` and `Swap` build from, resolved once and recorded; fails when the repo doesn't pin the agent's current config. Declares no resources |
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev markers, forge/matrix sync, manager kick, rescan; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
@ -80,29 +79,8 @@ Cheap — no build slot:
<!-- vale write-good.Passive = YES -->
Deliberately, **no `GitCommit` node** exists: `meta.rs` fuses each mutation
with its commit under its internal `META_LOCK` mutex, so a standalone commit
node would open a dirty-working-tree window between nodes.
Two further layers protect the meta repo across _windows_ that span multiple
`META_LOCK` acquisitions:
- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
resource declared by every node kind that mutates the meta repo — `MetaSync`,
`MetaLock` and `Provision`'s agent registration. Two meta
mutations can therefore never interleave, so no commit lands inside another
node's window. It's a queue resource rather than a runtime mutex
because a subtree root holds a resource across its whole subtree, which
a `MutexGuard` (bounded by one executor fn) can't. For the same reason the window must stay
_off_ long store-only work: the rebuild's meta preamble is its own
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
toplevel build runs outside the window and `buildSlots > 1` still gives
concurrent rebuilds across agents.
- **Path-limited commits**: the targeted meta committers (perm files,
topology, lock bumps) commit `-- <their paths>` with path-scoped
dirty checks, so even a non-queue caller (boot migration, destroy's
`sync_agents`) can never sweep someone else's staged content into its
commit.
Nothing in the queue commits to a meta repo: the hive has none to commit to,
and `agent_set::refresh` serializes itself with its own mutex.
### Every operation as a DAG
@ -138,8 +116,7 @@ restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Re
offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted)
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
spawn(a): [wanted=Up at deploy] Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a)
meta-update(inp): MetaLock(inp) →(in-DAG) «rebuild subgraph» per affected agent
boot: (if any rev marker stale) MetaLock(hyperhive) →(in-DAG) «rebuild subgraph» per stale agent;
boot: «graceful rebuild subgraph» per stale agent (rev marker ≠ the `internal/meta` pin);
plus Reconcile(a) for every drifted agent (all ONE DAG)
```
@ -157,14 +134,9 @@ Notable collapses:
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
whole-hive graceful stop fires every agent's signal immediately and all
drains overlap; each DAG's tail `Reconcile` does the actual stop.
- **The meta-update cascade grows in the same DAG on completion**:
`MetaLock`'s executor computes the affected agent set after the bump lands
and grows one `rebuild` subgraph per agent into its _own_ DAG via
`append_subgraph` (rooted on the `MetaLock`, `relock = false` so the cascade
doesn't revert the bump). Not child DAGs — one DAG, no `parent_id`. A failed
bump appends nothing (no cancel-children dance). Same shape as the startup
sweep; the meta-update DAG carries the `Rebuilding` transient so each cascade
agent keeps crash-watch suppression during its `Swap`.
- **The boot sweep is one DAG**: it declares one graceful `rebuild` subgraph
per stale agent, each an independent root, beside a `Reconcile` per drifted
agent. Not child DAGs — one DAG, no `parent_id`.
### Desired-state (spec vs status)
@ -174,7 +146,7 @@ Per-agent power _intent_ — `wanted: Up | Offline` — is durable as the
two. Setting `wanted` is never a queued node: the power layer
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
Power toggles never commit to the meta repo. Every operator power surface —
Every operator power surface —
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
rides the queue through that power layer, so intent, lease serialization,
and crash-watch suppression can't drift per surface; the only direct starts
@ -188,9 +160,9 @@ The admin-socket responses carry the submitted DAG ids; `hivectl` polls
`HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up
glyph, template, agent, node chain — so
CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
non-zero). Nodes appended in-DAG (a `MetaLock` growing per-agent rebuild
subgraphs, a `Reconcile` fanning its `Start`/`Stop`) join the same DAG, so
they surface under that DAG's id in the same loop — no separate child DAGs.
non-zero). Nodes appended in-DAG (a `Reconcile` fanning its `Start`/`Stop`)
join the same DAG, so they surface under that DAG's id in the same loop — no
separate child DAGs.
### Scheduler semantics
@ -213,8 +185,8 @@ resources are free. Resources:
(`SetWanted` is a store write, not a container op, but takes the lease anyway
so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
`Prebuild`, `Provision`, `MetaLock` — they
touch the store / meta, not the running container, which is exactly why a
`Prebuild`, `Provision` — they
touch the store, not the running container, which is exactly why a
stop can land while another DAG's prebuild is still building. Also exempt, for a different
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
@ -317,56 +289,39 @@ render.
On startup, `auto_update::run` classifies every agent by rev freshness (the
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
vs the current flake path) and persisted `wanted` intent, then:
vs the hyperhive commit the head of `internal/meta` pins) and persisted
`wanted` intent, then:
1. **Config path** — when _any_ marker is stale, submit one `Boot`
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that grows an in-DAG
`Rebuild` subgraph for each stale agent whose `wanted = Up` (name-sorted). Stale but wanted-offline agents get no boot-time nix work — their
rebuild happens on their next start (the start submit path upgrades a
stale start to rebuild+start), which is also why the lock bump runs even
when every stale agent is offline: those later start-upgrades must build
against the bumped lock. Each child rebuild's tail `Reconcile` brings the
agent (back) up, covering both the running-stale and stopped-but-wanted-up
cases.
1. **Config path** — submit one `Boot` DAG holding a graceful `Rebuild`
subgraph for each stale agent whose `wanted = Up` (name-sorted). This is how
a hive that missed the swarm's fan-out catches up. Stale but wanted-offline
agents get no boot-time nix work — their rebuild happens on their next start
(the start submit path upgrades a stale start to rebuild+start). Each
rebuild's tail `Reconcile` brings the agent (back) up, covering both the
running-stale and stopped-but-wanted-up cases. A hive that can't read the
pin (no forge token, the forge down, no `internal/meta` yet) rebuilds
nothing.
2. **Power path** — every agent whose observed state drifted from `wanted`
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
Booting with no config change performs **no meta commit** — only reconciles.
The sweep reason records the rebuild / deferred / up-to-date counts so the
Booting with no pin change performs no rebuild — only reconciles.
The boot log line records the rebuild / deferred / up-to-date counts so the
operator sees at a glance how much work the boot triggered. `auto_update::run`
seeds agents without an `agent_power` row from observed state during
classification (the one-time migration; thereafter the DB is authoritative).
## Meta flake
`meta.rs` owns the single coordinator-managed flake at `/var/lib/hyperhive/meta/`.
This flake consumes every agent's applied config repo as a flake input and exports
one `nixosConfiguration` per agent. Container lifecycle ops drive the lock file so
meta's git log is the system-wide deploy audit trail.
Key operations:
- **`sync_agents`** (idempotent) — render `flake.nix` for the current agent set,
init the repo on first call, relock if the rendered contents changed, commit.
Called by spawn / destroy / startup migration.
- **`lock_update_hyperhive`** — one-shot for the boot-reconcile path (the
sweep DAG's `MetaLock` node): bumps the `hyperhive` input lock and commits;
the scheduler fans out the agent rebuilds on completion.
Every public `meta.rs` operation takes the module's internal `META_LOCK`
mutex, so concurrent job-queue nodes never
race on the repo's `.git/index.lock`.
Each git or nix child under `META_LOCK` has a deadline, so a hung forge fetch
fails the operation instead of holding the lock: 10 minutes for
`nix flake update`/`lock` and forge fetch/push, 2 minutes for local git, including
the `rev-parse` and `show` reads `pin_agents` makes. The error names the command
and the deadline. Each child leads its own process group and the timeout
`SIGKILL`s the whole group, so a `git` or credential helper spawned by a hung
`nix` dies too. A caller waiting on `META_LOCK` for over a minute logs a
`meta_lock: still waiting` warning every minute, naming its own operation and
the holder's. Waiters keep their place in the lock's FIFO queue across warnings.
Agents build from the swarm's `internal/meta` flake, which swarm-controller
commits and every hive reads (see
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)).
The hive keeps no flake of its own. `meta_rev.rs` picks the `internal/meta`
commit an agent builds from and records it; `agent_set.rs` keeps the
host-side files that follow the agent set (`topology.json`, the gateway's
agent-socket map and nginx include) in line with the live agents, and
`meta.rs` keeps the one remaining commit to the legacy `meta/` git repo, the
removal of the permission seed files.
---
@ -389,10 +344,8 @@ one go, rather than the double-bounce a live `update` would trigger.
Sequence for a rebuild DAG (each step is its own queue node):
1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a
meta-update cascade child) the per-agent relock. Short, and the only step
that mutates the meta repo, so it's the only one holding the global deploy
window.
1. `MetaSync` — rebuild-dir prep, the agent-set refresh, and the choice of the
`internal/meta` commit to build from. Short, and it declares no resources.
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
The container keeps serving the previous generation while eval + fetch +
build happen out-of-band. `nixos-container update` then finds the result
@ -401,7 +354,7 @@ Sequence for a rebuild DAG (each step is its own queue node):
container is already down — a stopped agent has no uptime to preserve,
so `Swap` builds inline instead of paying the double eval.
3. `StopForUpdate` — bring the container down (noop when already stopped).
4. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
4. `Swap` — `nixos-container update` profile-swap from `internal/meta`
(near-instant after the prebuild).
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
in-container activation script transitions old → new. Holds no build
@ -424,15 +377,15 @@ fallback.
For a first-time `create`, `nixos-container create` is atomic: if the build fails,
it leaves no container record to clean up. A separate prebuild would just duplicate
the eval, so it's skipped. Sequence: `create --flake meta#<name>` → write nspawn
the eval, so it's skipped. Sequence: `create` from `internal/meta` → write nspawn
flags → `systemctl daemon-reload` → `start`.
### Prebuild attr path
`nix build` doesn't autoresolve `meta#<name>` against `nixosConfigurations` the
`nix build` doesn't autoresolve `<flake-root>#<name>` against `nixosConfigurations` the
way `nixos-container` does internally. nix requires the explicit attr path
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel`;
using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`,
using the bare `<flake-root>#<name>` ref would make nix look in `packages`, `legacyPackages`,
or the flake root directly — none of which exist in the rendered meta flake.
---

5
docs/swarm/archetypes.md
View file

@ -116,12 +116,11 @@ rebuild, and keeps the last value read in
without the queue. When the read fails, a rebuild keeps the value
stored there. For an archetype agent the hive:
- seeds no `proposed` or `applied` repo, and gives the meta flake no
- seeds no `proposed` or `applied` repo, and gives `internal/meta` no
`agent-<name>` input. `mkAgent` imports
`hyperhive.hyperhiveArchetypes.<name>.module` in place of the agent's own
flake, and sets `HIVE_ARCHETYPE` in the harness environment;
- relocks no agent input on a rebuild or a meta-input bump, and ignores the
`rev` of a swarm deploy request;
- ignores the `rev` of a swarm deploy request;
- mounts an empty `/agents/<name>/config/`.
The agent's system prompt says it has no config repo and names its

7
docs/swarm/builder.md
View file

@ -125,7 +125,7 @@ lock doesn't wait for a build in progress and has ten minutes. The builder
commits nothing.
swarm-controller sends the request. It renders `internal/meta`'s `flake.nix`
with the same renderer hive-c0re uses (`hive-meta-flake`), all over `https`.
with the `hive-meta-flake` renderer, all over `https`.
Each custom agent that a hive places pins to its `agent-configs/<agent>`
`main`; the controller omits and logs one whose repo has no commit on `main`, so it
doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own
@ -147,8 +147,9 @@ repo. It re-renders when it starts, and again every ten minutes while the latest
update hasn't succeeded (a tick adds nothing while one is pending or running,
or after a success), before it deploys a merged config PR, and
before the first deploy of a new agent. Those deploys run whether or not the
update succeeded: a hive that builds from `internal/meta` does so only while it
pins the agent's current config (see
update succeeded: a hive builds only from `internal/meta`, so while the repo
doesn't pin the agent's current config that build fails and names the agent
(see
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
equal to the committed `flake.nix` sends no lock.

2
docs/tools/matrix.md
View file

@ -195,7 +195,7 @@ The agent's `flake.nix` forwards every flake input to `agent.nix` as
the `flakeInputs` module arg, so you pull in external MCP-server flakes
by adding them to `inputs.*` and reference them as
`flakeInputs.<name>.packages.${pkgs.system}.default` — the resolved
sha lands in the agent's own `flake.lock` and rolls up to meta's.
sha lands in the agent's own `flake.lock` and rolls up to `internal/meta`'s.
`allowedTools` defaults to `["*"]`, which expands to
`mcp__<key>__*` (every tool from that server autoapproved). Restrict

6
docs/web-ui/README.md
View file

@ -16,13 +16,13 @@ surfaces — Stats, the API docs and the swarm UI — and each of
those pages links back to the dashboard.
The dashboard is where you'll spend most of your time. It's a single
page with five tabs:
page with four tabs:
- **4G3NTS** — every agent, live. This is the default tab and the one
you'll check most.
- **R3BU1LD QU3U3**, **M3T4 1NPUTS** and **BUILD L0GS** — the build
- **R3BU1LD QU3U3** and **BUILD L0GS** — the build
lifecycle: the rebuild queue with a live log of whatever is building,
the meta flake inputs you can bump, and the build log history.
and the build log history.
- **K3PT ST4T3** — kept state from agents with no container
(tombstones), with a purge button per entry.

49
docs/web-ui/dashboard.md
View file

@ -33,7 +33,7 @@ dashboard tab strip.
`HIVE_AUTHELIA_PUBLIC_URL` (set from `services.hyperhive.swarm.authelia.url`).
If the name doesn't load the avatar shows a question mark, and without the URL the
popover has no links.
- **Tab strip**: `◆ 4G3NTS ◆`, `◆ R3BU1LD QU3U3 ◆`, `◆ M3T4 1NPUTS ◆`,
- **Tab strip**: `◆ 4G3NTS ◆`, `◆ R3BU1LD QU3U3 ◆`,
`◆ BUILD L0GS ◆`, `◆ K3PT ST4T3 ◆`. In-page tabs only — ST4TS
lives on its own page, reached from the pages menu. Count pills on
4G3NTS (stale containers), R3BU1LD QU3U3 (rebuilds in flight) and
@ -73,7 +73,7 @@ agent is stale.
Passive / rare-interaction state. `kept.js` renders it from the
`/api/state` snapshot and keeps it live from the `tombstones_changed`
stream event. (Rebuild queue and meta inputs are dashboard tabs too —
stream event. (The rebuild queue and build logs are dashboard tabs too —
see below.)
Per-container CPU, memory and disk load isn't shown in the hive UI. The
@ -96,16 +96,16 @@ container is all hive-c0re can actually prove. Fixing it properly
needs a recorded destroy — **deferred** to the swarm-controller /
snapshot-storage rework where the problem changes shape.
## Build tabs (`#rebuild`, `#meta`, `#buildlogs`)
## Build tabs (`#rebuild`, `#buildlogs`)
The build lifecycle — rebuild queue, live build log, meta inputs, and
build log history — is three tabs of the dashboard: **R3BU1LD QU3U3**
(`#rebuild`), **M3T4 1NPUTS** (`#meta`) and **BUILD L0GS**
The build lifecycle — rebuild queue, live build log, and
build log history — is two tabs of the dashboard: **R3BU1LD QU3U3**
(`#rebuild`) and **BUILD L0GS**
(`#buildlogs`). They live in `builds.js`, which `tabs.js` imports: the
dashboard's one tab strip, `/api/state` snapshot and
`/api/dashboard/stream` subscription serve them, with
`rebuild_queue_changed`, `meta_inputs_changed` and `meta_update_running`
on the stream alongside the dashboard's other kinds. The count pill on
`rebuild_queue_changed` on the stream alongside the dashboard's other
kinds. The count pill on
R3BU1LD QU3U3 is the number of rebuilds in flight.
The 4G3NTS tab does **not** read the graph endpoint, by design — the swarm
view stays independent of job-queue internals. Its per-agent pending
@ -115,7 +115,7 @@ handful of pre-tallied counts, not the graph. The banner's "view queue →"
link goes to the `#rebuild` tab.
**R3BU1LD QU3U3** — pending, in-flight, and recently settled container
operations: rebuilds, meta-update cascades, and first-spawns. One
operations: rebuilds and first-spawns. One
operation runs at a time; the worker drains FIFO. **Is a mounted
`JobqGraph`** (the shared generic graph-viewer Preact component,
`@hive/shared/jobq-graph.js` — the same one swarm-ui uses) —
@ -184,20 +184,6 @@ own container outside `#rebuild-queue-section` so the mounted
`JobqGraph`'s own re-renders never disturb the open poll; it hides
when nothing is building.
**M3T4 1NPUTS** — inputs in `meta/flake.lock` the operator can
selectively `nix flake update`, rendered as an indented tree:
every fetched input at every depth (`hyperhive`,
`hyperhive/nixpkgs`, `agent-<n>`, `agent-<n>/mcp-<x>`, …), each
shown once at its shallowest path. `read_meta_inputs` walks the
lock graph with a `visited` set, skipping `follows` aliases and rev-less
nodes. A `select all / select none`
control sits above the tree. Checking inputs + submitting bumps
the lock in `/meta/` and rebuilds the selected agents in
sequence; each outcome reaches the root agent as a `rebuilt`
system event. `POST /meta-update`. While a lock-bump ripple runs,
the panel shows a pulsing "⏳ meta-update running" banner and disables
the update button (snapshot field `meta_update_running`,
live event `meta_update_running`).
**BUILD L0GS** — all-agents build log history. Lazy-loaded on first tab
activation; autorefreshes when
@ -378,7 +364,7 @@ life). Two consequences for anything rendering it:
to rebuild-shaped work — `running_transients()` in hive-c0re is
a status-only test (any `Running` node whose payload names a
non-empty agent lights a pill), so work the operator never
triggered (a meta-update cascade, a crash-recover rebuild, a
triggered (a crash-recover rebuild, a boot-sweep rebuild, a
lease-exempt `prebuild`) lights the same mechanism.
Ops with no queue node behind them (destroy, migration) supply
@ -575,8 +561,7 @@ below — some endpoints aren't in it yet.
- `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder.
- `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery
failed (clears the failure state so the scheduler retries).
- `POST /api/meta-update` — `nix flake update` the selected
`meta/flake.lock` inputs, then rebuild the affected agents.
- `GET /api/dashboard/stream` — unified live event channel:
broker `sent` / `delivered`, plus the mutation events listed
below. Each frame carries `seq`.
@ -620,18 +605,12 @@ payload):
- `tombstones_changed` (seq, tombstones: `Vec<TombstoneView>`) —
full snapshot of destroyed-but-state-kept agents. Emitted on
destroy, purge, and revive. The K3PT ST4T3 tab subscribes.
- `meta_inputs_changed` (seq, inputs: `Vec<MetaInput>`) — full
snapshot of `meta/flake.lock` inputs. The M3T4 1NPUTS tab
subscribes; same shape as `GET /api/meta-inputs`.
- `meta_update_running` (running: bool) — emitted when a
`nix flake update` ripple starts or completes. The M3T4 1NPUTS
tab uses this to show/hide the "⏳ meta-update running" banner.
<!-- vale write-good.Passive = NO -->
`/api/state` is **only fetched on cold-load and on the few
forms that mutate non-event-derived state** (PURG3 +
meta-update, since tombstones + meta_inputs aren't event-
shaped yet). Every other section — transients,
forms that mutate non-event-derived state** (PURG3, since
tombstones aren't event-shaped yet). Every other section — transients,
containers —
derives from `/api/dashboard/stream` after the initial snapshot,
maintaining its own client-side store and applying events on