deploy the closure the swarm declares, read no forge head, drop the update badge
This commit is contained in:
atlas
mara
parent
279428a66f
commit
47c2acb640
2 changed files with 647 additions and 2001 deletions
|
|
@ -42,11 +42,11 @@ because there is no malformed spec to reject.
|
|||
|
||||
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
||||
|
||||
| Node | Wraps |
|
||||
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | download the swarm's closure for the agent into the store while the container keeps serving (its meta preamble is the upstream `MetaSync` node). With no closure to deploy it fails here, before the stop. Skipped when the container is already down; `Swap` downloads it instead |
|
||||
| `Swap` | drop-in rewrite + profile-swap onto the swarm's closure (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; downloads the closure the swarm built for the agent and creates the container from it (with none to deploy it fails and creates nothing), then marks the agent's pins |
|
||||
| Node | Wraps |
|
||||
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | download the swarm's closure for the agent into the store while the container keeps serving (its meta preamble is the upstream `MetaSync` node). With no closure to deploy it fails here, before the stop. Skipped when the container is already down; `Swap` downloads it instead |
|
||||
| `Swap` | drop-in rewrite + profile-swap onto the swarm's closure (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; downloads the closure the swarm built for the agent and creates the container from it (with none to deploy it fails and creates nothing), then marks the closure applied |
|
||||
|
||||
Cheap — no build slot:
|
||||
|
||||
|
|
@ -54,7 +54,7 @@ Cheap — no build slot:
|
|||
|
||||
| Node | Behavior |
|
||||
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the agent-set refresh (`agent_set::refresh`), and the `internal/meta` commit `Prebuild` and `Swap` expect the swarm's closure to be built from, resolved once and recorded; fails when the repo doesn't pin the agent's current config. Declares no resources |
|
||||
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the archetype refresh and the agent-set refresh (`agent_set::refresh`); reads nothing from the forge. Declares no resources |
|
||||
| `Provision` | first-spawn pre-create provisioning — the state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
||||
|
|
@ -103,7 +103,7 @@ state-conditional — skipped for a _down_ agent (nothing to quiesce/stop). Keep
|
|||
closes the TOCTOU window: if an agent flips state between the `is_running`
|
||||
read and node exec, the tail `Reconcile` still converges it in-DAG (with
|
||||
`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
|
||||
sweep. `start` folds the per-agent stale-rev upgrade in (a _down + stale_
|
||||
sweep. `start` folds the per-agent declared-closure upgrade in (a _down + stale_
|
||||
agent's subgraph is a rebuild-then-start).
|
||||
|
||||
```text
|
||||
|
|
@ -114,7 +114,7 @@ restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Re
|
|||
offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted)
|
||||
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
|
||||
spawn(a): [wanted=Up at deploy] Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a)
|
||||
boot: «graceful rebuild subgraph» per stale agent (rev marker ≠ the `internal/meta` pin);
|
||||
boot: «graceful rebuild subgraph» per stale agent (closure marker ≠ the record's store path);
|
||||
plus Reconcile(a) for every drifted agent (all ONE DAG)
|
||||
```
|
||||
|
||||
|
|
@ -285,25 +285,31 @@ render.
|
|||
|
||||
## Boot reconcile
|
||||
|
||||
On startup, `auto_update::run` classifies every agent by rev freshness (the
|
||||
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
|
||||
vs the hyperhive commit the head of `internal/meta` pins) and persisted
|
||||
On startup, `auto_update::run` classifies every agent by closure freshness (the
|
||||
per-agent `.{name}.closure` marker under `/var/lib/hyperhive/applied/` vs the
|
||||
store path the agent's `agent-closures` record declares) and persisted
|
||||
`wanted` intent, then:
|
||||
|
||||
1. **Config path** — submit one `Boot` DAG holding a graceful `Rebuild`
|
||||
subgraph for each stale agent whose `wanted = Up` (name-sorted). This is how
|
||||
a hive that missed the swarm's fan-out catches up. Stale but wanted-offline
|
||||
a hive that missed the swarm's deploy request catches up. Stale but wanted-offline
|
||||
agents get no boot-time nix work — their rebuild happens on their next start
|
||||
(the start submit path upgrades a stale start to rebuild+start). Each
|
||||
rebuild's tail `Reconcile` brings the agent (back) up, covering both the
|
||||
running-stale and stopped-but-wanted-up cases. A hive that can't read the
|
||||
pin (no forge token, the forge down, no `internal/meta` yet) rebuilds
|
||||
nothing.
|
||||
running-stale and stopped-but-wanted-up cases. An agent whose declaration
|
||||
the hive can't read or compare (no queue connection, no record, a closure the
|
||||
builder didn't push, no marker) rebuilds nothing, and neither does any
|
||||
agent on the first boot after the marker arrived.
|
||||
|
||||
2. **Power path** — every agent whose observed state drifted from `wanted`
|
||||
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
|
||||
|
||||
Booting with no pin change performs no rebuild — only reconciles.
|
||||
Booting with every agent on its declared closure performs no rebuild — only
|
||||
reconciles. After the boot pass, `auto_update::run` repeats the rebuild half
|
||||
every five minutes. It skips an agent with a rebuild waiting or running
|
||||
(`JobQueue::rebuild_live`) and one whose declared store path this process
|
||||
already attempted, so a closure that fails to deploy gets one try per process.
|
||||
|
||||
The boot log line records the rebuild / deferred / up-to-date counts so the
|
||||
operator sees at a glance how much work the boot triggered. `auto_update::run`
|
||||
seeds agents without an `agent_power` row from observed state during
|
||||
|
|
@ -314,8 +320,9 @@ classification (the one-time migration; thereafter the DB is authoritative).
|
|||
Agents build from the swarm's `internal/meta` flake, which swarm-controller
|
||||
commits and every hive reads (see
|
||||
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)).
|
||||
The hive keeps no flake of its own. `meta_rev.rs` picks the `internal/meta`
|
||||
commit an agent builds from and records it; `agent_set.rs` keeps the
|
||||
The hive keeps no flake of its own and deploys the closure the swarm builds
|
||||
from it, as the agent's `agent-closures` record declares (`attic/closure.rs`).
|
||||
`agent_set.rs` keeps the
|
||||
host-side files that follow the agent set (`topology.json`, the gateway's
|
||||
agent-socket map and nginx include) in line with the live agents, and
|
||||
`meta.rs` removes the permission seed files once the swarm has them, and
|
||||
|
|
@ -342,9 +349,8 @@ one go, rather than the double-bounce a live `update` would trigger.
|
|||
|
||||
Sequence for a rebuild DAG (each step is its own queue node):
|
||||
|
||||
1. `MetaSync` — rebuild-dir prep, the agent-set refresh, and the choice of the
|
||||
`internal/meta` commit the swarm's closure should come from. Short,
|
||||
and it declares no resources.
|
||||
1. `MetaSync` — rebuild-dir prep, the archetype refresh and the agent-set
|
||||
refresh. Short, and it declares no resources.
|
||||
2. `Prebuild` — download the swarm's closure for the agent **before** stopping.
|
||||
The container keeps serving the previous generation while the download
|
||||
happens out-of-band. A missing or unpullable closure fails the DAG here,
|
||||
|
|
@ -353,7 +359,7 @@ Sequence for a rebuild DAG (each step is its own queue node):
|
|||
no uptime to preserve, so `Swap` downloads it.
|
||||
3. `StopForUpdate` — bring the container down (noop when already stopped).
|
||||
4. `Swap` — profile-swap onto the swarm's closure (near-instant after the
|
||||
prebuild).
|
||||
prebuild), then it writes the closure marker.
|
||||
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
||||
in-container activation script transitions old → new. Holds no build
|
||||
slot, so the next DAG's `Prebuild` overlaps the container boot.
|
||||
|
|
|
|||
|
|
@ -334,16 +334,16 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
|
|||
`service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name`
|
||||
(= the `h-<agent>` machine) and the hive `agent` label:
|
||||
|
||||
| Metric | Unit | Kind | Source |
|
||||
| --------------------------------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + the root template (the live ephemeral layer isn't counted), measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
|
||||
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
|
||||
| `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `rev-mismatch`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes |
|
||||
| Metric | Unit | Kind | Source |
|
||||
| --------------------------------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + the root template (the live ephemeral layer isn't counted), measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
|
||||
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
|
||||
| `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes |
|
||||
|
||||
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
|
||||
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
|
||||
|
|
|
|||
Loading…
Reference in a new issue