pin the declared closure once per rebuild and seed the marker from the running closure
This commit is contained in:
atlas
mara
parent
47c2acb640
commit
4356f3840c
5 changed files with 456 additions and 95 deletions
|
|
@ -59,7 +59,7 @@ an agent. The swarm controller's `InitAgentConfigRepo` job seeds the
|
|||
agent's config repo with a default `agent.nix` template, then asks the
|
||||
target hive to deploy it.
|
||||
An agent created from an [archetype](../swarm/archetypes.md) gets no
|
||||
config repo: its hive builds it from the archetype's module.
|
||||
config repo: the swarm builds it from the archetype's module.
|
||||
|
||||
Changing what the template seeded isn't an exception: like every
|
||||
later change, it's a PR on that config repo, made from a clone and
|
||||
|
|
@ -146,8 +146,12 @@ the swarm builder built the agent from the updated repo (the
|
|||
- **Choosing the closure.** The hive doesn't choose. It deploys the closure the
|
||||
agent's record in `agent-closures` declares, whichever `internal/meta` commit
|
||||
the builder built it from, and reads no forge head. The rebuild's `MetaSync`
|
||||
node only prepares it (the agent's archetype, its runtime directories, the
|
||||
agent set).
|
||||
node prepares it (the agent's archetype, its runtime directories, the agent
|
||||
set), reads the record once and pins its store path. `Prebuild` and `Swap` use
|
||||
the pin, so a record that changes mid-rebuild can't make `Swap` pull another
|
||||
closure than `Prebuild` fetched, and the marker names the closure it applied. The
|
||||
next autoupdate pass picks up the newer record. A first deploy has no
|
||||
`MetaSync`; its `Create` reads the record once.
|
||||
- **The closure.** The hive builds nothing. `Prebuild`, `Swap` and `Create` read
|
||||
the agent's record in the swarm's `agent-closures` bucket (the controller
|
||||
writes it from the builder's reply) and have `hive-priv` download the closure
|
||||
|
|
@ -353,11 +357,16 @@ through the same graceful rebuild subgraph.
|
|||
|
||||
The comparison reads the record and the marker
|
||||
(`/var/lib/hyperhive/applied/.<name>.closure`, written after a successful
|
||||
swap or create) and nothing on the forge. It rebuilds nothing when the hive
|
||||
can't tell: no connection to the swarm queue, no record, a closure the builder
|
||||
didn't push, or an agent with no marker. The last case covers every agent on
|
||||
the first boot after the marker arrived, so that boot rebuilds nothing; the
|
||||
agent's next deploy request from the swarm writes the marker. The periodic pass
|
||||
swap or create) and nothing on the forge. Where the marker is missing, as for
|
||||
every agent on the first boot after the marker arrived, the hive reads the
|
||||
closure the container's profile
|
||||
(`/nix/var/nix/profiles/per-container/h-<name>/system`) points at and writes
|
||||
that as the marker, so the first pass catches an agent that doesn't run its declared
|
||||
closure. Those first-boot rebuilds go through the same job queue as
|
||||
any other: each holds one of the `buildSlots`, so they don't swap all at once.
|
||||
It rebuilds nothing when the hive can't tell: no connection to the swarm queue,
|
||||
no record, a closure the builder didn't push, or a profile it can't read or that
|
||||
leaves the nix store. The periodic pass
|
||||
attempts a declared store path once per process, and skips an agent while a
|
||||
rebuild of it waits or runs, so a closure that fails to deploy doesn't
|
||||
restart its agent every five minutes. Restarting `hive-c0re`, or a rebuild from
|
||||
|
|
|
|||
|
|
@ -545,7 +545,8 @@ unless it holds the permission, nothing else.
|
|||
|
||||
Under `/var/lib/hyperhive/applied/` — the hive-c0re-only deploy markers
|
||||
`.<name>.closure`: the store path of the closure the agent was last swapped to
|
||||
or created from. No config lives here; the
|
||||
or created from, or, where the file was missing, the closure the container's
|
||||
profile pointed at when the hive first read it. No config lives here; the
|
||||
forge's `agent-configs/<name>` is the only copy. A hive that predates this
|
||||
finds per-agent `applied/<name>` repos here. Nothing reads them any more, and
|
||||
you can delete them.
|
||||
|
|
@ -597,7 +598,7 @@ this hive's archetype agents runs, as last read from the swarm's
|
|||
with an entry has no config repo and no `agent-<n>` meta input; see
|
||||
[archetypes](../swarm/archetypes.md).
|
||||
|
||||
hive-c0re creates containers from `internal/meta` (`nixosConfigurations.<n>`).
|
||||
hive-c0re creates containers from the closure the swarm builds from `internal/meta` (`nixosConfigurations.<n>`).
|
||||
|
||||
## Destroy vs purge
|
||||
|
||||
|
|
|
|||
|
|
@ -54,7 +54,7 @@ Cheap — no build slot:
|
|||
|
||||
| Node | Behavior |
|
||||
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the archetype refresh and the agent-set refresh (`agent_set::refresh`); reads nothing from the forge. Declares no resources |
|
||||
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the archetype refresh and the agent-set refresh (`agent_set::refresh`), then reads the agent's closure record once and pins its store path for `Prebuild` and `Swap`; fails when the swarm has no closure to deploy; reads nothing from the forge. Declares no resources |
|
||||
| `Provision` | first-spawn pre-create provisioning — the state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
||||
|
|
@ -298,8 +298,10 @@ store path the agent's `agent-closures` record declares) and persisted
|
|||
rebuild's tail `Reconcile` brings the agent (back) up, covering both the
|
||||
running-stale and stopped-but-wanted-up cases. An agent whose declaration
|
||||
the hive can't read or compare (no queue connection, no record, a closure the
|
||||
builder didn't push, no marker) rebuilds nothing, and neither does any
|
||||
agent on the first boot after the marker arrived.
|
||||
builder didn't push, a running closure it can't read) rebuilds nothing. An
|
||||
agent with no marker takes the closure its container profile points at as the
|
||||
marker, so a first boot after the marker arrived rebuilds exactly the
|
||||
agents that run something else than declared, each holding a build slot.
|
||||
|
||||
2. **Power path** — every agent whose observed state drifted from `wanted`
|
||||
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
|
||||
|
|
@ -318,7 +320,7 @@ classification (the one-time migration; thereafter the DB is authoritative).
|
|||
## Meta flake
|
||||
|
||||
Agents build from the swarm's `internal/meta` flake, which swarm-controller
|
||||
commits and every hive reads (see
|
||||
commits and the swarm builder builds from (see
|
||||
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)).
|
||||
The hive keeps no flake of its own and deploys the closure the swarm builds
|
||||
from it, as the agent's `agent-closures` record declares (`attic/closure.rs`).
|
||||
|
|
@ -349,8 +351,10 @@ one go, rather than the double-bounce a live `update` would trigger.
|
|||
|
||||
Sequence for a rebuild DAG (each step is its own queue node):
|
||||
|
||||
1. `MetaSync` — rebuild-dir prep, the archetype refresh and the agent-set
|
||||
refresh. Short, and it declares no resources.
|
||||
1. `MetaSync` — rebuild-dir prep, the archetype refresh, the agent-set
|
||||
refresh, and the pin of the swarm's declared closure (one read of the record;
|
||||
`Prebuild` and `Swap` use the pin). Short, and it declares no
|
||||
resources.
|
||||
2. `Prebuild` — download the swarm's closure for the agent **before** stopping.
|
||||
The container keeps serving the previous generation while the download
|
||||
happens out-of-band. A missing or unpullable closure fails the DAG here,
|
||||
|
|
|
|||
|
|
@ -125,7 +125,7 @@ swarm-controller renders `internal/meta`'s `flake.nix` with the
|
|||
place of the agent's own flake, and set `HIVE_ARCHETYPE` in the harness
|
||||
environment.
|
||||
|
||||
The hive builds from `internal/meta` alone. It reads the agent's key when it
|
||||
The swarm builds from `internal/meta` alone and the hive deploys that closure. The hive reads the agent's key when it
|
||||
provisions the agent and on every rebuild, and keeps the last value read in
|
||||
`/var/lib/hyperhive/swarm-agent-archetypes.json`. When the read fails, a rebuild keeps the stored value. For an
|
||||
archetype agent the hive also:
|
||||
|
|
@ -188,8 +188,8 @@ in a lock doesn't count.
|
|||
A hive that keeps a checkout of a custom agent's config would create one for
|
||||
the converted agent, under the container's running mount. A hive that lists
|
||||
`no-config-checkout` under `capabilities` in the status it publishes (the
|
||||
document `GET /api/hives/status` serves as `snapshot`) keeps none: it builds a
|
||||
custom agent from `internal/meta` alone, and the agent clones its own repo from
|
||||
document `GET /api/hives/status` serves as `snapshot`) keeps none: it deploys a
|
||||
custom agent's closure from the swarm alone, and the agent clones its own repo from
|
||||
the forge. The controller reads the agent's hive's latest status and refuses
|
||||
unless it's `fresh` and lists the capability; an older hive lists none. Deploy
|
||||
the release to every hive that places archetype agents before converting theirs.
|
||||
|
|
|
|||
|
|
@ -199,7 +199,7 @@ it. The check runs after each update and once when the controller starts.
|
|||
## Deploy gate
|
||||
|
||||
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
|
||||
the flake a hive builds from. Four paths send deploys, and each declares the same
|
||||
the flake the swarm builds from. Four paths send deploys, and each declares the same
|
||||
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
|
||||
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
|
||||
and needs the one `builder` slot, so it queues behind PR checks. The deploy
|
||||
|
|
@ -503,10 +503,10 @@ same and creates the container from that path.
|
|||
|
||||
A rebuild or create has two outcomes, by what the lookup and the download find. The hive deploys the record's closure whatever `internal/meta` commit the builder built it from, and reads no forge head to choose one:
|
||||
|
||||
| Found | What happens |
|
||||
| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| a record that the builder pushed | The hive downloads the closure and deploys it, then writes the store path to the agent's `applied/.<name>.closure` marker. The hive logs `prebuild: substituted <path>` (`create: substituted <path>` for a create) at info level. |
|
||||
| no record, or one the hive can't use (not pushed, queue or cache unreachable, download or signature failed) | The rebuild or create fails with an error that names the agent and the reason, logged at error level, and nothing changes. A running container keeps running unchanged, because `Prebuild` fails before `StopForUpdate`. A create makes no container. |
|
||||
| Found | What happens |
|
||||
| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| a record that the builder pushed | `MetaSync` reads the record once and pins its store path, so a record that changes during the rebuild doesn't change what `Swap` applies. The hive downloads the pinned closure and deploys it, then writes the store path to the agent's `applied/.<name>.closure` marker. The hive logs `prebuild: substituted <path>` (`create: substituted <path>` for a create) at info level. |
|
||||
| no record, or one the hive can't use (not pushed, queue or cache unreachable, download or signature failed) | The rebuild or create fails with an error that names the agent and the reason, logged at error level, and nothing changes. A running container keeps running unchanged, because `Prebuild` fails before `StopForUpdate`. A create makes no container. |
|
||||
|
||||
The hive never falls back to a local build. The second row has one owner in the
|
||||
code, `without_closure` in `hive-c0re/src/job_queue/exec.rs`, so a change to what
|
||||
|
|
|
|||
Loading…
Reference in a new issue