reconcile: start a stopped harness inside a running container
hive-priv gains AgentHarness { agent_name, action: probe | start }, which
runs systemctl -M h-<name> against the fixed unit hive-agent.service.
run_reconcile and the 1-min tick read the unit for a running wanted-up
agent and start it when it is inactive or failed. A hive-priv without the
op reads as unknown state and leaves reconcile container-only.
This commit is contained in:
parent
ca734f8ea0
commit
36515c289d
1 changed files with 517 additions and 32 deletions
|
|
@ -55,7 +55,7 @@ Cheap — no build slot:
|
|||
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the archetype refresh and the agent-set refresh (`agent_set::refresh`), then reads the agent's closure record once and pins its store path for `Prebuild` and `Swap`; fails when the swarm has no closure to deploy; reads nothing from the forge. Declares no resources |
|
||||
| `Provision` | first-spawn pre-create provisioning — the state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop; an `Up` agent whose container runs but whose harness unit is stopped or failed gets the unit started |
|
||||
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
||||
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
|
||||
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
||||
|
|
@ -324,15 +324,19 @@ boot with no operator action in this process never finds drift of its own.
|
|||
### Periodic reconcile tick
|
||||
|
||||
Once a minute (first tick one minute after boot), `workers::reconcile_tick`
|
||||
looks for an agent that the hive wants up and whose container is down. It reads
|
||||
the power intent, the container's unit state, the pause marker and
|
||||
`Coordinator::is_updating`, then queues the same `Reconcile` node the boot power
|
||||
looks for an agent that the hive wants up and whose container is down, or runs
|
||||
with its harness unit (`hive-agent.service`) stopped or failed. It reads the
|
||||
power intent, the container's unit state, the pause marker,
|
||||
`Coordinator::is_updating` and, for a running wanted-up agent, the harness
|
||||
unit's state through hive-priv (`AgentHarness`), then queues the same `Reconcile` node the boot power
|
||||
path uses, through `JobQueue::insert_reconcile_if_idle`. The tick never stops
|
||||
anything.
|
||||
|
||||
It leaves an agent alone when:
|
||||
|
||||
- its intent is unknown or `Offline`, or its container runs;
|
||||
- its intent is unknown or `Offline`, or its container runs with a harness that
|
||||
is active, activating or unreadable (an older hive-priv doesn't know the op: hive-c0re logs
|
||||
that once and the tick watches the container only);
|
||||
- the agent has its pause marker, an update in progress, or a container state
|
||||
the tick can't read in time;
|
||||
- any queued or running node targets it (update, create, destroy, power op or
|
||||
|
|
@ -342,7 +346,7 @@ An agent still down on the tick after the tick queued a restart has failed to
|
|||
come up, so the tick spaces restarts of one agent 1, 2, 4, 8, then 15 ticks
|
||||
apart (a 15-minute cap). Seeing the agent running, or no longer wanted up,
|
||||
resets the spacing. Each restart logs `reconcile tick: restarting agent` with
|
||||
the agent, `observed=stopped`, `wanted=up` and the attempt number, and bumps
|
||||
the agent, `observed=stopped` (`harness stopped` for a running container), `wanted=up` and the attempt number, and bumps
|
||||
`hyperhive.hive.reconcile_tick.restarts`. `crash_watch` keeps raising its
|
||||
alert; the tick only converges.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue