vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity
hyperhive/docs/scheduler/coordinator.md
atlas b31db6c7d9 subagent, bash: run each subagent run and bash task in its own cgroup 
hive-subagent-daemon and hive-bash-daemon are now delegated their cgroup
(Delegate=memory pids, DelegateSubgroup=daemon). At start each daemon
enables memory+pids below its unit and creates runs/ (subagent) or
tasks/ (bash). Each subagent run gets runs/<name>-<n>/ and each bash
task tasks/<id>/, both with memory.oom.group=1, and the daemon removes
the directory when the run ends. An OOM kill now takes one whole run
instead of one process out of the daemon's shared cgroup, and never the
daemon itself.

Subagent runs get oom_score_adj +1000; bash tasks and both daemons stay
at 0; the harness stays at -500. The unit-level OOMScoreAdjust=500 on
the subagent daemon goes: it was both the daemon's own score and a floor
no run below it could be lowered past.

Bash tasks enter their cgroup from the existing pre_exec hook, writing
to a cgroup.procs fd opened by the parent. hive-claude builds its
Command itself and exposes neither a pre_exec hook nor the child's pid,
so a subagent's claude (and an ACP agent's process) is started as the
daemon's own binary with the run's cgroup in its environment; main()
enters the cgroup, raises oom_score_adj and execs the real program, the
trampoline swarm-review-classifier already uses. Both enter before the
program executes, so nothing it forks starts outside the run's cgroup.

Without a delegated `daemon` leaf (tests, a dev shell) each daemon logs
one warning and runs everything in its own cgroup, as before.

The shared helper is hive-agent-sock's run_cgroup module, the crate
both daemons already use for what in-container plugins share.

Closes #5007
2026-10-13 12:25:27 +02:00

596 lines
41 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# hive-c0re coordinator internals
Architecture notes for the `hive-c0re` coordinator daemon's internal subsystems.
For the public API surface (dashboard, socket protocol) see `docs/process/conventions.md`
and `docs/agent-lifecycle/persistence.md`.
---
## Job queue
**For the job queue as a general idea — graph of steps, shared resource
slots, no hive-c0re specifics — see [`jobq.md`](jobq.md) instead.** This
section covers the concrete node inventory hive-c0re builds on top of that
engine, plus the internals (module layout, scheduler mechanics,
resource/lease semantics).
hive-c0re submits every container/meta operation (rebuild, meta-update, first-spawn, power
changes) to the global job-DAG queue (`hive-c0re/src/job_queue/`)
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
concurrency comes from the resource classes below, not from multiple workers.
Special cases like graceful stop, deferred starts, and the meta-update
cascade need no bespoke code paths — each takes the shape of a DAG built
from the same primitive nodes.
### Two levels: DAG and node
The **DAG** is the unit of cancel and the
dashboard group; the **node** is the unit of scheduling / execution /
build-log. Deps are intra-DAG edges only (`AfterOk` by default:
the dep must succeed, a failed/cancelled dep cancels the dependent —
cancel-downstream). Cross-DAG ordering comes from the per-agent lease,
never from edges between DAGs.
A DAG is **declared, not described**: a template builds it through
`hive_jobq::JobBuilder`, naming each node it depends on via the handle
`b.node(kind)` handed back, and the builder inserts the nodes itself. A handle
only exists for a node already declared, so every edge points backwards and
nothing can write a cycle down — there is no submit-time validation pass,
because there is no malformed spec to reject.
### Node inventory (primitives)
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| Node | Wraps |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta |
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
Cheap — no build slot:
<!-- vale write-good.Passive = NO -->
| Node | Behavior |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here |
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
| `DestroyBookkeeping` | the post-destroy tail — meta sync, drop the power intent, notify the manager, rescan, re-emit the tombstone. Same split rationale as `RebuildBookkeeping`/`Swap`. The destructive work is `PurgeState`'s |
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
| `EmitRebuilt` | tail of a rebuild/perm-change — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent _and_ per outcome |
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
| `ForgeSweep` | one-shot boot-time forge user/token sweep for every container (`forge::ensure_all`) as a first-class node, so it shows as real work on the dashboard instead of running invisibly in a bare `tokio::spawn`. Agentless |
| `MatrixSweep` | matrix user/space sweep (`matrix::ensure_all`): the boot-time instance, plus one every 30 min from a loop in `main.rs`. Holds `Resource::MatrixSweep` (capacity 1), so two passes never overlap; each tick queues its own pass, which waits for the resource if one is already live. Agentless |
| `KnowledgePull` | `/knowledge` pull (`knowledge::pull`): at boot (commits that landed while `hive-c0re` was down), on the swarm knowledge-changed event, and hourly as a fallback. Holds `Resource::KnowledgeTree` (capacity 1), so two pulls never overlap on the working tree; each trigger queues its own pass, which waits for the resource if one is already live. Agentless |
| `WantedPull` | one-shot boot-time pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. No background loop behind this one — boot is the whole cadence; the deploy event (`swarm_status`) is the fast path, this repairs a missed one. Agentless |
<!-- vale write-good.Passive = YES -->
Deliberately, **no `GitCommit` node** exists: `meta.rs` fuses each mutation
with its commit under its internal `META_LOCK` mutex, so a standalone commit
node would open a dirty-working-tree window between nodes.
Two further layers protect the meta repo across _windows_ that span multiple
`META_LOCK` acquisitions:
- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
resource declared by every node kind that mutates the meta repo — `MetaSync`,
`MetaLock`, `WritePermFile` and `Provision`'s agent registration. Two meta
mutations can therefore never interleave, so no commit lands inside another
node's window. It's a queue resource rather than a runtime mutex
because a subtree root holds a resource across its whole subtree, which
a `MutexGuard` (bounded by one executor fn) can't. For the same reason the window must stay
_off_ long store-only work: the rebuild's meta preamble is its own
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
toplevel build runs outside the window and `buildSlots > 1` still gives
concurrent rebuilds across agents.
- **Path-limited commits**: the targeted meta committers (perm files,
topology, lock bumps) commit `-- <their paths>` with path-scoped
dirty checks, so even a non-queue caller (boot migration, destroy's
`sync_agents`) can never sweep someone else's staged content into its
commit.
### Every operation as a DAG
The `stop` / `start` power ops write the durable `wanted` intent via a head
`SetWanted` node (not a pre-submit side effect) — it holds the agent lease,
so intent-write + reconcile is atomic per-agent. `restart` is the exception:
it writes _no_ intent (no `SetWanted` head) — it bounces the container and
lets the tail `Reconcile` converge to the agent's existing `wanted`, so a
deliberately stopped agent isn't forced back up by a hive-wide restart. The
hive-wide power ops — `restart`, `stop`, and `start` — take an agent _list_:
a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
subgraph each (independent roots, run concurrently on their own leases), not
N separate DAGs.
**`job_queue/power.rs` builds these dynamically from each agent's live
running state** (an async `lifecycle::is_running` read) — that's why they
live there rather than in the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
a head `SetWanted` (intent) — `restart` doesn't; the tail `Reconcile`
(convergence guarantee — cheap, noops when already converged) is ALWAYS
present; only the _mechanical_ nodes (`Signal`/`Drain`/`StopForUpdate`) are
state-conditional — skipped for a _down_ agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
closes the TOCTOU window: if an agent flips state between the `is_running`
read and node exec, the tail `Reconcile` still converges it in-DAG (with
`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
sweep. `start` folds the per-agent stale-rev upgrade in (a _down + stale_
agent's subgraph is a rebuild-then-start).
```text
rebuild(a): MetaSync(a) → AgentWindow(a){ Prebuild(a) ∥ [Signal(a)→Drain(a) if graceful]; both →(after-ok) StopForUpdate(a) → Swap(a) →(after-ok) RebuildBookkeeping(a) } →(after-any) Reconcile(a)
stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG)
restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted)
offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted)
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
spawn(a): [wanted=Up at deploy] Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a)
perm-change(a): WritePermFile(a) → «rebuild subgraph»
meta-update(inp): MetaLock(inp) →(in-DAG) «rebuild subgraph» per affected agent
boot: (if any rev marker stale) MetaLock(hyperhive) →(in-DAG) «rebuild subgraph» per stale agent;
plus Reconcile(a) for every drifted agent (all ONE DAG)
```
Notable collapses:
- **`rebuild` is one uniform shape** — no `was_running` branch.
`StopForUpdate` noops when already down; the tail `Reconcile` autonoops the
start when `wanted = Offline` (a rebuild of a deliberately stopped agent
leaves it stopped).
- **The swap-failure recovery-start is structural**: `Reconcile` deps on
`Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
terminal ok _or_ fail, bringing a wanted-up agent back on its old config.
- **Deferred start is automatic**: `Reconcile` holds no build slot, so the
next DAG's `Prebuild` starts as soon as `Swap` frees the slot.
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
whole-hive graceful stop fires every agent's signal immediately and all
drains overlap; each DAG's tail `Reconcile` does the actual stop.
- **The meta-update cascade grows in the same DAG on completion**:
`MetaLock`'s executor computes the affected agent set after the bump lands
and grows one `rebuild` subgraph per agent into its _own_ DAG via
`append_subgraph` (rooted on the `MetaLock`, `relock = false` so the cascade
doesn't revert the bump). Not child DAGs — one DAG, no `parent_id`. A failed
bump appends nothing (no cancel-children dance). Same shape as the startup
sweep; the meta-update DAG carries the `Rebuilding` transient so each cascade
agent keeps crash-watch suppression during its `Swap`.
### Desired-state (spec vs status)
Per-agent power _intent_ — `wanted: Up | Offline` — is durable as the
`agent_power` table in the coordinator DB (`hive-c0re/src/stores/power.rs`).
`container_view` remains the observed _status_; `Reconcile` nodes converge the
two. Setting `wanted` is never a queued node: the power layer
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
Power toggles never commit to the meta repo. Every operator power surface —
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
rides the queue through that power layer, so intent, lease serialization,
and crash-watch suppression can't drift per surface; the only direct starts
left are the root-agent bootstrap and infra containers (no lease, no
harness). Cancelling a still-queued power DAG reverts `wanted` to the
observed state — a cancel means "don't do it," not "do it later." hive-c0re
seeds agents without a row from observed state on first touch (running ⇒
`Up`); destroy removes the row.
The admin-socket responses carry the submitted DAG ids; `hivectl` polls
`HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up
glyph, template, agent, node chain — so
CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
non-zero). Nodes appended in-DAG (a `MetaLock` growing per-agent rebuild
subgraphs, a `Reconcile` fanning its `Start`/`Stop`) join the same DAG, so
they surface under that DAG's id in the same loop — no separate child DAGs.
### Scheduler semantics
<!-- vale write-good.Passive = NO -->
A node is **ready** when it's `Queued`, every dep is satisfied, and its
resources are free. Resources:
<!-- vale write-good.Passive = YES -->
1. **Build slots** — `services.hyperhive.c0re.buildSlots` permits (default 1),
held by nix-heavy nodes for the node's duration.
2. **Per-agent lifecycle lease** — keyed on the **node's** agent (agent is
per-node; a DAG can span agents) and globally exclusive per agent across
all DAGs: acquired either at a container-affecting node (`SetWanted`,
`Reconcile`, `WriteDropin`, `Create`) or at a **brace** (`AgentWindow`) on
behalf of a whole coordinated subtree; held by the owning
DAG until it's terminal, so two DAGs never interleave container ops on the
same agent. A DAG touching multiple agents holds one lease per agent.
(`SetWanted` is a store write, not a container op, but takes the lease anyway
so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
`Prebuild`, `Provision`, `MetaLock`, `WritePermFile` — they
touch the store / meta, not the running container, which is exactly why a
stop can land while another DAG's prebuild is still building. Also exempt, for a different
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
container, but their `AgentWindow` brace holds the lease above them — see
_Braces_ below.
#### Braces
Templates otherwise declare a resource on **every** node that needs it, even
when a parent already holds it, so the requirement belongs to the node rather
than to one DAG shape it happens to appear in. A **brace** is the one sanctioned
exception: a pure-resource-holder root that declares on behalf of a subtree
coordinated with itself, whose members then declare nothing.
It's forced rather than stylistic. Declaring a resource means _"I need this
exclusively"_, and the agent lease is single-unit — so **two siblings that both
declared it could never run concurrently.** For a subtree whose whole point is
concurrency (`Prebuild` beside the `Signal` → `Drain` quiesce window), declaring
the requirement truthfully on every node and running those nodes in parallel are
mutually exclusive. One holder above them speaks for the subtree.
This is the opposite of the failure the declare-your-own rule exists to prevent,
not a relapse into it: there the requirement was _implicit_, inferred from a
node's kind and true only by accident of placement. Here it's explicit, on one
node, with the omission below it documented on the brace itself.
Two consequences worth knowing:
<!-- vale write-good.Passive = NO -->
- **Flattening a chain under a brace is safe.** A brace (`AgentWindow`)
holds the lease continuity for the stop chain's subtree: as independent
siblings, `Signal`, `Drain`, and `StopForUpdate` would each acquire the
lease separately and leave a gap another DAG could claim the agent in,
mid-bounce.
- **Observability is unaffected.** `running_transients` keys off a node's
_payload_ agent, not off a declared lease edge, so every child still lights its
own dashboard pill and still reports its own `takes_container_down` to the
crash watcher. A brace itself reports `false`: it parents the stopping nodes
but doesn't stop anything, and claiming otherwise would widen crash
suppression across the build and tail, where a vanished container is still a
real crash.
<!-- vale write-good.Passive = YES -->
Among simultaneously ready nodes competing for a resource, DAG-submit order
wins (FIFO) so bulk operations drain predictably. The scheduler also owns the
DAG-lifetime transient guard (dashboard pill + crash-watch suppression),
created on lease acquisition and dropped when the DAG settles terminal.
The queue is in-memory only and lost on hive-c0re restart — deliberate:
desired state is re-derived at boot from the DB + rev markers (see _Boot
reconcile_), so there is no durable-recovery machinery to go wrong.
### Cancel, history
Cancel only applies to DAGs that are still fully queued (an in-flight nix build isn't
interruptible).
Roll-up state: `Failed` if any node failed, else `Running` / `Queued` /
`Cancelled` / `Done`. The snapshot retains the 50 most recent terminal
DAGs — a flat cap over the whole sorted list, not per template, since
the dashboard renders one recent-builds list and one number bounds it.
### Wire shape
`RebuildQueueChanged { seq }` (event name kept) — **a bare trigger, no
payload.** It says _the queue changed_; a client that wants to know how
re-fetches `GET /api/jobq/graph`.
That endpoint serves the graph generically (`hive-jobq-wire`): every node
carries `id`, `parent`, `deps`, `state`, `label` (the node kind's own wire
string — the kind _is_ the phase label, there is no separate sub-step
string) and free-form `data` for what only some kinds have (`agent`,
`inputs`, `build_log_id`). Group roots ride as ordinary
nodes, so a group's state is just the root's own `state`.
No group-level `agent` exists — agent is per-node, so one group can
span agents; consumers derive a group's agents from its nodes. Build logs
are likewise **per-node**: the dashboard renders the node tree and keys the
live-log panel off the running node.
The event carries no payload by design: shipping a typed projection of
the whole queue in the event itself would be a second rendering of the
same graph that someone has to keep in agreement by hand with the
endpoint every consumer actually reads. Telling a client _when_ to refetch is
the event's whole job.
---
## Container view
`container_view.rs` maintains an in-memory snapshot of every nixos-container's
systemd service state. It's polled on coordinator startup and re-scanned after
every lifecycle operation (spawn, rebuild, kill) so the dashboard always reflects
the actual container status without a live `nixos-container list` call on each
render.
---
## Boot reconcile
On startup, `auto_update::run` classifies every agent by rev freshness (the
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
vs the current flake path) and persisted `wanted` intent, then:
1. **Config path** — when _any_ marker is stale, submit one `Boot`
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that grows an in-DAG
`Rebuild` subgraph for each stale agent whose `wanted = Up` (name-sorted). Stale but wanted-offline agents get no boot-time nix work — their
rebuild happens on their next start (the start submit path upgrades a
stale start to rebuild+start), which is also why the lock bump runs even
when every stale agent is offline: those later start-upgrades must build
against the bumped lock. Each child rebuild's tail `Reconcile` brings the
agent (back) up, covering both the running-stale and stopped-but-wanted-up
cases.
2. **Power path** — every agent whose observed state drifted from `wanted`
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
Booting with no config change performs **no meta commit** — only reconciles.
The sweep reason records the rebuild / deferred / up-to-date counts so the
operator sees at a glance how much work the boot triggered. `auto_update::run`
seeds agents without an `agent_power` row from observed state during
classification (the one-time migration; thereafter the DB is authoritative).
## Meta flake
`meta.rs` owns the single coordinator-managed flake at `/var/lib/hyperhive/meta/`.
This flake consumes every agent's applied config repo as a flake input and exports
one `nixosConfiguration` per agent. Container lifecycle ops drive the lock file so
meta's git log is the system-wide deploy audit trail.
Key operations:
- **`sync_agents`** (idempotent) — render `flake.nix` for the current agent set,
init the repo on first call, relock if the rendered contents changed, commit.
Called by spawn / destroy / startup migration.
- **`lock_update_hyperhive`** — one-shot for the boot-reconcile path (the
sweep DAG's `MetaLock` node): bumps the `hyperhive` input lock and commits;
the scheduler fans out the agent rebuilds on completion.
Every public `meta.rs` operation takes the module's internal `META_LOCK`
mutex, so concurrent job-queue nodes never
race on the repo's `.git/index.lock`.
---
## Container lifecycle (`lifecycle.rs`)
Every container operation ultimately calls into `lifecycle.rs`. Two paths exist:
**rebuild** (existing container) and **spawn** (first-time creation).
### Rebuild path (existing container)
Goal: apply the new system profile and any `EXTRA_NSPAWN_FLAGS` / drop-in changes
in a single start, with minimum downtime.
`nixos-container update` only runs `systemctl reload container@<c>` when the
container is already up (per `isContainerRunning` in `nixos-container.pl`). Stopping
first turns `update` into a boot-style operation: it builds + `nix-env --set`s the
new profile and skips the in-container `switch-to-configuration`. The subsequent
`start` then applies both the new profile and any `EXTRA_NSPAWN_FLAGS` changes in
one go, rather than the double-bounce a live `update` would trigger.
Sequence for a rebuild DAG (each step is its own queue node):
1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a
meta-update cascade child) the per-agent relock. Short, and the only step
that mutates the meta repo, so it's the only one holding the global deploy
window.
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
The container keeps serving the previous generation while eval + fetch +
build happen out-of-band. `nixos-container update` then finds the result
cached and skips straight to the profile-swap. Build failures surface
here, before `Swap` touches the running container. Skipped when the
container is already down — a stopped agent has no uptime to preserve,
so `Swap` builds inline instead of paying the double eval.
3. `StopForUpdate` — bring the container down (noop when already stopped).
4. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
(near-instant after the prebuild).
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
in-container activation script transitions old → new. Holds no build
slot, so the next DAG's `Prebuild` overlaps the container boot.
### Cold-start fallback
`start` after `update` can exit non-zero when packages are **removed** between
generations: the old-generation activation script references units that no longer
exist in the new closure, causing systemd to exit non-zero. The container may be
half-started at that point.
Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
→ `start` (clean cold-start, no generation transition, new activation runs cleanly).
`lifecycle::start_with_fallback` preserves and surfaces both errors if the
cold-start also fails. Every `Reconcile` node's start action uses this
fallback.
### Spawn path (new container)
For a first-time `create`, `nixos-container create` is atomic: if the build fails,
it leaves no container record to clean up. A separate prebuild would just duplicate
the eval, so it's skipped. Sequence: `create --flake meta#<name>` → write nspawn
flags → `systemctl daemon-reload` → `start`.
### Prebuild attr path
`nix build` doesn't autoresolve `meta#<name>` against `nixosConfigurations` the
way `nixos-container` does internally. nix requires the explicit attr path
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel`;
using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`,
or the flake root directly — none of which exist in the rendered meta flake.
---
## Host-level resource + performance options
A handful of `services.hyperhive.c0re.*` options tune container resource
limits, build parallelism, and first-spawn latency.
### Build slots
`buildSlots` (default `1`) sets how many nix-heavy job-queue nodes
(prebuilds, profile swaps, first-spawn creates, meta lock bumps) run
concurrently. The default serializes all heavy nix work like the pre-DAG
rebuild queue did; raise it on hosts with the cores/RAM to build multiple
agent toplevels at once. Per-agent correctness is independent of the count —
each agent's container-affecting ops serialize on its lifecycle lease
regardless.
### Container resource limits
`agentCpuQuota` and `agentMemoryMax` map directly to systemd
`CPUQuota=` and `MemoryMax=`. hive-c0re writes a
`container@h-<name>.service.d/` drop-in file on each spawn and
rebuild, so changes take effect on the next lifecycle op without
requiring a host rebuild.
The answer per agent and per field is, first match wins:
1. the swarm's cap for the agent (see below);
2. the operator's per-agent override in `meta/resource-limits.json`;
3. `agentCpuQuota` / `agentMemoryMax`.
A swarm cap replaces the hive's answer, so a hive can't raise an agent
above it. Like the rest of the drop-in, a changed swarm cap reaches a
running agent at its next spawn or rebuild.
The container can't read its own cap, so `write_dropins` also writes the
effective `MemoryMax=` into the agent's runtime dir
(`/run/hive/memory-max-bytes` inside) before every start: a byte count, or
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
it, as that share of the host's physical memory rounded down to a page. A
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
the last drop-in because it can't read the overrides or the swarm caps file,
the file carries that drop-in's `MemoryMax=`.
At boot the container derives two values from a byte count:
`BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on
`hive-subagent-daemon` (two thirds). Neither is in the agent's build, so a
changed cap reaches them at the next container start without a rebuild.
`infinity` sets neither. A missing or malformed file sets neither as well, and
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
container's journal. The container's `MemoryMax=` still applies.
When the container reaches its cap, the kernel kills in this order:
1. a subagent run (`oom_score_adj` +1000);
2. a bash task (0);
3. the harness and the agent's own `claude` (-500).
Each subagent run and each bash task has its own cgroup with
`memory.oom.group=1`, so the kernel kills all of a run or task and nothing
outside it. Inside the container these are
`/sys/fs/cgroup/system.slice/hive-subagent-daemon.service/runs/<name>-<n>/`
and `/sys/fs/cgroup/system.slice/hive-bash-daemon.service/tasks/<id>/`; the
daemon itself sits in `daemon/` beside them. A daemon that can't set these
up logs one warning at start and runs everything in its own cgroup.
<details><summary>Implementation: swarm caps</summary>
swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per
agent and no hive in it, so an agent that moves to another hive keeps its
caps. swarm-controller alone writes it:
| method | path | body |
| -------- | ---------------------------- | --------------------------------------------------------------------------------------------- |
| `GET` | `/api/agents/<agent>/limits` | —; answers the stored caps, `{}` when there are none |
| `PUT` | `/api/agents/<agent>/limits` | `{"cpu_quota": "400%", "memory_max": "8G"}`, every field optional, replacing the stored value |
| `DELETE` | `/api/agents/<agent>/limits` | —; answers 204 |
A stored value the controller can't decode, such as one a newer build wrote,
answers `{}`, with a warning in its log, until the next `PUT` replaces it.
`PUT` refuses an agent no hive's wanted state declares, unknown fields, and
any value systemd would ignore, which would leave the agent uncapped. A
percentage has at most two decimals, no leading zero in its whole part (it
refuses `08%`), and is above zero, up to `100%` for
`memory_max` and `21474836.47%` for `cpu_quota`. A size is digits with an
optional fraction and an optional `B`, `K`, `M`, `G` or `T` (upper case), at
least one byte and below 2^64 - 1 bytes. The controller stores each value as
given.
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
value passes the same validators as the operator's overrides first. One bad
value or a failed read leaves the file unchanged, and repeated failures raise a
dashboard banner.
A hive's queue grant reads every key of the bucket
(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the
hive's own agents would need a roster in the auth responder.
</details>
The same drop-in carries `CPUWeight=` / `IOWeight=` from
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
setting: the quota and the memory max are **hard caps** that throttle
an agent even on an idle host, while the weights are cgroup
v2 **relative shares** that only decide who yields _under contention_.
A low-weight container still gets the whole machine when nothing else
wants it.
| Option | Default | Description |
| ---------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `services.hyperhive.c0re.agentCpuQuota` | `"200%"` | CPU cap per agent, as a percentage of one core (`"200%"` = 2 cores). Raise if agents hit CPU limits during builds or heavy tool use. |
| `services.hyperhive.c0re.agentMemoryMax` | `"4G"` | Memory cap per agent. Raise for agents that run large nix builds or hold big in-memory data. |
| `services.hyperhive.c0re.agentCpuWeight` | `80` | `cpu.weight` share per agent, `1`–`10000` or `null` to omit the setting. Kernel default is `100`, so `80` makes agents yield. |
| `services.hyperhive.c0re.agentIoWeight` | `80` | `io.weight` share per agent, same range and `null` handling. See the caveat below — it's a no-op on many hosts. |
Two things to know about the weights:
- They're **hive-wide** — unlike the caps there is no per-agent
override in `meta/resource-limits.json`, so every agent carries the
same value and the weight does _not_ rank agents against each other.
What `80` buys is that agents yield to everything **not** on this
drop-in path: host services (nginx and dnsmasq among them) and the
infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which
stay at the kernel default of `100`.
- `IOWeight=` is only honoured when the backing device runs the BFQ
scheduler or has blk-iocost QoS enabled. On a host using
`none`/`mq-deadline`/`kyber` without iocost, systemd writes the value
and the kernel ignores it — harmless, but it will measure as nothing.
Check with `cat /sys/fs/cgroup/io.cost.qos`, and set the option to
`null` if you would rather not write a setting nothing reads.
For a hive-wide cap across all containers together, set
`systemd.slices.machine.serviceConfig.CPUQuota` in your NixOS
config — all nspawn containers live in `machine.slice`.
### Pre-building agent templates
`preBuildAgentTemplates` (default `false`) causes the host NixOS
build to pre-fetch the per-container system closures
(`agent-base` + manager toplevels) into `/nix/store`, instead of
leaving that work to the first `nixos-container start`. The
trade-off:
- **On** (recommended for x86_64 hosts that care about first-spawn
latency): the first `nixos-container start` for any new agent
completes in seconds because nothing remains to fetch. Cost:
`preBuildAgentTemplates` adds the full nixpkgs runtime closure + claude-code + the harness
binary to the host system closure (low single-digit GB
additional).
- **Off** (default): the host closure stays lean; the first spawn
does all the eval + fetch work at runtime (can take a few
minutes on a fresh store).
**Note**: the meta flake pins toplevels to `x86_64-linux`. Enabling on an
`aarch64` host forces a cross-compilation or remote-builder build,
which is almost never desired. Leave off on non-x86 hosts.
---
## See also
- `docs/agent-lifecycle/config-changes.md` — config PRs, meta flake, helper events
- `docs/agent-lifecycle/persistence.md` — SQLite schema, state-dir layout
- `docs/process/conventions.md` — wire protocol, recipient sentinels
- `docs/agent-lifecycle/agent-roster.md` — the agent roster and privilege
View git blame Copy permalink