build agents only from internal/meta; delete the hive's local meta flake
The hive no longer renders, locks or commits a meta flake. A rebuild's MetaSync node picks the internal/meta commit and records it with the hyperhive and config pins it holds; Prebuild, Swap and Create build from it, and a head that does not pin the agent's current config fails the build naming the agent. hive-priv's meta_rev is required. Removed: meta.rs sync/lock/render, the manual-rebuild relock, the boot sweep's hyperhive bump (MetaLock), the update-meta-inputs form and its panel, MetaWindow, the hive's hyperhive_rev setting. Kept: agent_set::refresh holds what sync_agents also did (topology.json, the gateway's agent-socket map and nginx include), called from spawn, destroy, rebuild and the migration. The boot reconcile and the stale-start upgrade compare the agent's marker with the hyperhive pin of internal/meta; the dashboard's deployed chip reads the config marker a successful build writes.
This commit is contained in:
atlas
parent
cea4827918
commit
4aa11b3a94
64 changed files with 1042 additions and 3793 deletions
|
|
@ -14,12 +14,12 @@ section covers the concrete node inventory hive-c0re builds on top of that
|
|||
engine, plus the internals (module layout, scheduler mechanics,
|
||||
resource/lease semantics).
|
||||
|
||||
hive-c0re submits every container/meta operation (rebuild, meta-update, first-spawn, power
|
||||
hive-c0re submits every container operation (rebuild, first-spawn, power
|
||||
changes) to the global job-DAG queue (`hive-c0re/src/job_queue/`)
|
||||
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
|
||||
concurrency comes from the resource classes below, not from multiple workers.
|
||||
Special cases like graceful stop, deferred starts, and the meta-update
|
||||
cascade need no bespoke code paths — each takes the shape of a DAG built
|
||||
Special cases like graceful stop, deferred starts, and the boot sweep
|
||||
need no bespoke code paths — each takes the shape of a DAG built
|
||||
from the same primitive nodes.
|
||||
|
||||
### Two levels: DAG and node
|
||||
|
|
@ -46,8 +46,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
|||
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
|
||||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta |
|
||||
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; builds from the `internal/meta` commit it resolves and marks the agent's pins |
|
||||
|
||||
Cheap — no build slot:
|
||||
|
||||
|
|
@ -55,13 +54,13 @@ Cheap — no build slot:
|
|||
|
||||
| Node | Behavior |
|
||||
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
|
||||
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
|
||||
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the agent-set refresh (`agent_set::refresh`), and the `internal/meta` commit `Prebuild` and `Swap` build from, resolved once and recorded; fails when the repo doesn't pin the agent's current config. Declares no resources |
|
||||
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
||||
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
|
||||
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
||||
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
|
||||
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev markers, forge/matrix sync, manager kick, rescan; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
|
||||
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
|
||||
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
|
||||
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
|
||||
|
|
@ -80,29 +79,8 @@ Cheap — no build slot:
|
|||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
Deliberately, **no `GitCommit` node** exists: `meta.rs` fuses each mutation
|
||||
with its commit under its internal `META_LOCK` mutex, so a standalone commit
|
||||
node would open a dirty-working-tree window between nodes.
|
||||
|
||||
Two further layers protect the meta repo across _windows_ that span multiple
|
||||
`META_LOCK` acquisitions:
|
||||
|
||||
- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
|
||||
resource declared by every node kind that mutates the meta repo — `MetaSync`,
|
||||
`MetaLock` and `Provision`'s agent registration. Two meta
|
||||
mutations can therefore never interleave, so no commit lands inside another
|
||||
node's window. It's a queue resource rather than a runtime mutex
|
||||
because a subtree root holds a resource across its whole subtree, which
|
||||
a `MutexGuard` (bounded by one executor fn) can't. For the same reason the window must stay
|
||||
_off_ long store-only work: the rebuild's meta preamble is its own
|
||||
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
|
||||
toplevel build runs outside the window and `buildSlots > 1` still gives
|
||||
concurrent rebuilds across agents.
|
||||
- **Path-limited commits**: the targeted meta committers (perm files,
|
||||
topology, lock bumps) commit `-- <their paths>` with path-scoped
|
||||
dirty checks, so even a non-queue caller (boot migration, destroy's
|
||||
`sync_agents`) can never sweep someone else's staged content into its
|
||||
commit.
|
||||
Nothing in the queue commits to a meta repo: the hive has none to commit to,
|
||||
and `agent_set::refresh` serializes itself with its own mutex.
|
||||
|
||||
### Every operation as a DAG
|
||||
|
||||
|
|
@ -138,8 +116,7 @@ restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Re
|
|||
offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted)
|
||||
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
|
||||
spawn(a): [wanted=Up at deploy] Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a)
|
||||
meta-update(inp): MetaLock(inp) →(in-DAG) «rebuild subgraph» per affected agent
|
||||
boot: (if any rev marker stale) MetaLock(hyperhive) →(in-DAG) «rebuild subgraph» per stale agent;
|
||||
boot: «graceful rebuild subgraph» per stale agent (rev marker ≠ the `internal/meta` pin);
|
||||
plus Reconcile(a) for every drifted agent (all ONE DAG)
|
||||
```
|
||||
|
||||
|
|
@ -157,14 +134,9 @@ Notable collapses:
|
|||
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
|
||||
whole-hive graceful stop fires every agent's signal immediately and all
|
||||
drains overlap; each DAG's tail `Reconcile` does the actual stop.
|
||||
- **The meta-update cascade grows in the same DAG on completion**:
|
||||
`MetaLock`'s executor computes the affected agent set after the bump lands
|
||||
and grows one `rebuild` subgraph per agent into its _own_ DAG via
|
||||
`append_subgraph` (rooted on the `MetaLock`, `relock = false` so the cascade
|
||||
doesn't revert the bump). Not child DAGs — one DAG, no `parent_id`. A failed
|
||||
bump appends nothing (no cancel-children dance). Same shape as the startup
|
||||
sweep; the meta-update DAG carries the `Rebuilding` transient so each cascade
|
||||
agent keeps crash-watch suppression during its `Swap`.
|
||||
- **The boot sweep is one DAG**: it declares one graceful `rebuild` subgraph
|
||||
per stale agent, each an independent root, beside a `Reconcile` per drifted
|
||||
agent. Not child DAGs — one DAG, no `parent_id`.
|
||||
|
||||
### Desired-state (spec vs status)
|
||||
|
||||
|
|
@ -174,7 +146,7 @@ Per-agent power _intent_ — `wanted: Up | Offline` — is durable as the
|
|||
two. Setting `wanted` is never a queued node: the power layer
|
||||
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
|
||||
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
|
||||
Power toggles never commit to the meta repo. Every operator power surface —
|
||||
Every operator power surface —
|
||||
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
|
||||
rides the queue through that power layer, so intent, lease serialization,
|
||||
and crash-watch suppression can't drift per surface; the only direct starts
|
||||
|
|
@ -188,9 +160,9 @@ The admin-socket responses carry the submitted DAG ids; `hivectl` polls
|
|||
`HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up
|
||||
glyph, template, agent, node chain — so
|
||||
CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
|
||||
non-zero). Nodes appended in-DAG (a `MetaLock` growing per-agent rebuild
|
||||
subgraphs, a `Reconcile` fanning its `Start`/`Stop`) join the same DAG, so
|
||||
they surface under that DAG's id in the same loop — no separate child DAGs.
|
||||
non-zero). Nodes appended in-DAG (a `Reconcile` fanning its `Start`/`Stop`)
|
||||
join the same DAG, so they surface under that DAG's id in the same loop — no
|
||||
separate child DAGs.
|
||||
|
||||
### Scheduler semantics
|
||||
|
||||
|
|
@ -213,8 +185,8 @@ resources are free. Resources:
|
|||
(`SetWanted` is a store write, not a container op, but takes the lease anyway
|
||||
so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
|
||||
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
|
||||
`Prebuild`, `Provision`, `MetaLock` — they
|
||||
touch the store / meta, not the running container, which is exactly why a
|
||||
`Prebuild`, `Provision` — they
|
||||
touch the store, not the running container, which is exactly why a
|
||||
stop can land while another DAG's prebuild is still building. Also exempt, for a different
|
||||
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
|
||||
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
|
||||
|
|
@ -317,56 +289,39 @@ render.
|
|||
|
||||
On startup, `auto_update::run` classifies every agent by rev freshness (the
|
||||
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
|
||||
vs the current flake path) and persisted `wanted` intent, then:
|
||||
vs the hyperhive commit the head of `internal/meta` pins) and persisted
|
||||
`wanted` intent, then:
|
||||
|
||||
1. **Config path** — when _any_ marker is stale, submit one `Boot`
|
||||
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that grows an in-DAG
|
||||
`Rebuild` subgraph for each stale agent whose `wanted = Up` (name-sorted). Stale but wanted-offline agents get no boot-time nix work — their
|
||||
rebuild happens on their next start (the start submit path upgrades a
|
||||
stale start to rebuild+start), which is also why the lock bump runs even
|
||||
when every stale agent is offline: those later start-upgrades must build
|
||||
against the bumped lock. Each child rebuild's tail `Reconcile` brings the
|
||||
agent (back) up, covering both the running-stale and stopped-but-wanted-up
|
||||
cases.
|
||||
1. **Config path** — submit one `Boot` DAG holding a graceful `Rebuild`
|
||||
subgraph for each stale agent whose `wanted = Up` (name-sorted). This is how
|
||||
a hive that missed the swarm's fan-out catches up. Stale but wanted-offline
|
||||
agents get no boot-time nix work — their rebuild happens on their next start
|
||||
(the start submit path upgrades a stale start to rebuild+start). Each
|
||||
rebuild's tail `Reconcile` brings the agent (back) up, covering both the
|
||||
running-stale and stopped-but-wanted-up cases. A hive that can't read the
|
||||
pin (no forge token, the forge down, no `internal/meta` yet) rebuilds
|
||||
nothing.
|
||||
|
||||
2. **Power path** — every agent whose observed state drifted from `wanted`
|
||||
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
|
||||
|
||||
Booting with no config change performs **no meta commit** — only reconciles.
|
||||
The sweep reason records the rebuild / deferred / up-to-date counts so the
|
||||
Booting with no pin change performs no rebuild — only reconciles.
|
||||
The boot log line records the rebuild / deferred / up-to-date counts so the
|
||||
operator sees at a glance how much work the boot triggered. `auto_update::run`
|
||||
seeds agents without an `agent_power` row from observed state during
|
||||
classification (the one-time migration; thereafter the DB is authoritative).
|
||||
|
||||
## Meta flake
|
||||
|
||||
`meta.rs` owns the single coordinator-managed flake at `/var/lib/hyperhive/meta/`.
|
||||
This flake consumes every agent's applied config repo as a flake input and exports
|
||||
one `nixosConfiguration` per agent. Container lifecycle ops drive the lock file so
|
||||
meta's git log is the system-wide deploy audit trail.
|
||||
|
||||
Key operations:
|
||||
|
||||
- **`sync_agents`** (idempotent) — render `flake.nix` for the current agent set,
|
||||
init the repo on first call, relock if the rendered contents changed, commit.
|
||||
Called by spawn / destroy / startup migration.
|
||||
- **`lock_update_hyperhive`** — one-shot for the boot-reconcile path (the
|
||||
sweep DAG's `MetaLock` node): bumps the `hyperhive` input lock and commits;
|
||||
the scheduler fans out the agent rebuilds on completion.
|
||||
|
||||
Every public `meta.rs` operation takes the module's internal `META_LOCK`
|
||||
mutex, so concurrent job-queue nodes never
|
||||
race on the repo's `.git/index.lock`.
|
||||
|
||||
Each git or nix child under `META_LOCK` has a deadline, so a hung forge fetch
|
||||
fails the operation instead of holding the lock: 10 minutes for
|
||||
`nix flake update`/`lock` and forge fetch/push, 2 minutes for local git, including
|
||||
the `rev-parse` and `show` reads `pin_agents` makes. The error names the command
|
||||
and the deadline. Each child leads its own process group and the timeout
|
||||
`SIGKILL`s the whole group, so a `git` or credential helper spawned by a hung
|
||||
`nix` dies too. A caller waiting on `META_LOCK` for over a minute logs a
|
||||
`meta_lock: still waiting` warning every minute, naming its own operation and
|
||||
the holder's. Waiters keep their place in the lock's FIFO queue across warnings.
|
||||
Agents build from the swarm's `internal/meta` flake, which swarm-controller
|
||||
commits and every hive reads (see
|
||||
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)).
|
||||
The hive keeps no flake of its own. `meta_rev.rs` picks the `internal/meta`
|
||||
commit an agent builds from and records it; `agent_set.rs` keeps the
|
||||
host-side files that follow the agent set (`topology.json`, the gateway's
|
||||
agent-socket map and nginx include) in line with the live agents, and
|
||||
`meta.rs` keeps the one remaining commit to the legacy `meta/` git repo, the
|
||||
removal of the permission seed files.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -389,10 +344,8 @@ one go, rather than the double-bounce a live `update` would trigger.
|
|||
|
||||
Sequence for a rebuild DAG (each step is its own queue node):
|
||||
|
||||
1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a
|
||||
meta-update cascade child) the per-agent relock. Short, and the only step
|
||||
that mutates the meta repo, so it's the only one holding the global deploy
|
||||
window.
|
||||
1. `MetaSync` — rebuild-dir prep, the agent-set refresh, and the choice of the
|
||||
`internal/meta` commit to build from. Short, and it declares no resources.
|
||||
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
|
||||
The container keeps serving the previous generation while eval + fetch +
|
||||
build happen out-of-band. `nixos-container update` then finds the result
|
||||
|
|
@ -401,7 +354,7 @@ Sequence for a rebuild DAG (each step is its own queue node):
|
|||
container is already down — a stopped agent has no uptime to preserve,
|
||||
so `Swap` builds inline instead of paying the double eval.
|
||||
3. `StopForUpdate` — bring the container down (noop when already stopped).
|
||||
4. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
|
||||
4. `Swap` — `nixos-container update` profile-swap from `internal/meta`
|
||||
(near-instant after the prebuild).
|
||||
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
||||
in-container activation script transitions old → new. Holds no build
|
||||
|
|
@ -424,15 +377,15 @@ fallback.
|
|||
|
||||
For a first-time `create`, `nixos-container create` is atomic: if the build fails,
|
||||
it leaves no container record to clean up. A separate prebuild would just duplicate
|
||||
the eval, so it's skipped. Sequence: `create --flake meta#<name>` → write nspawn
|
||||
the eval, so it's skipped. Sequence: `create` from `internal/meta` → write nspawn
|
||||
flags → `systemctl daemon-reload` → `start`.
|
||||
|
||||
### Prebuild attr path
|
||||
|
||||
`nix build` doesn't autoresolve `meta#<name>` against `nixosConfigurations` the
|
||||
`nix build` doesn't autoresolve `<flake-root>#<name>` against `nixosConfigurations` the
|
||||
way `nixos-container` does internally. nix requires the explicit attr path
|
||||
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel`;
|
||||
using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`,
|
||||
using the bare `<flake-root>#<name>` ref would make nix look in `packages`, `legacyPackages`,
|
||||
or the flake root directly — none of which exist in the rendered meta flake.
|
||||
|
||||
---
|
||||
|
|
|
|||
Loading…
Reference in a new issue