Watch
0
0
Fork
You've already forked hyperhive
0

reconcile: start a stopped harness inside a running container

hive-priv gains AgentHarness { agent_name, action: probe | start }, which
runs systemctl -M h-<name> against the fixed unit hive-agent.service.
run_reconcile and the 1-min tick read the unit for a running wanted-up
agent and start it when it is inactive or failed. A hive-priv without the
op reads as unknown state and leaves reconcile container-only.
This commit is contained in:
atlas 2026-11-19 01:42:18 +01:00 • committed by mara
commit 36515c289d
3 changed files with 517 additions and 32 deletions

View file

@ -55,7 +55,7 @@ Cheap — no build slot:
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the archetype refresh and the agent-set refresh (`agent_set::refresh`), then reads the agent's closure record once and pins its store path for `Prebuild` and `Swap`; fails when the swarm has no closure to deploy; reads nothing from the forge. Declares no resources |
| `Provision` | first-spawn pre-create provisioning — the state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop; an `Up` agent whose container runs but whose harness unit is stopped or failed gets the unit started |
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
@ -324,15 +324,19 @@ boot with no operator action in this process never finds drift of its own.
### Periodic reconcile tick
Once a minute (first tick one minute after boot), `workers::reconcile_tick`
looks for an agent that the hive wants up and whose container is down. It reads
the power intent, the container's unit state, the pause marker and
`Coordinator::is_updating`, then queues the same `Reconcile` node the boot power
looks for an agent that the hive wants up and whose container is down, or runs
with its harness unit (`hive-agent.service`) stopped or failed. It reads the
power intent, the container's unit state, the pause marker,
`Coordinator::is_updating` and, for a running wanted-up agent, the harness
unit's state through hive-priv (`AgentHarness`), then queues the same `Reconcile` node the boot power
path uses, through `JobQueue::insert_reconcile_if_idle`. The tick never stops
anything.
It leaves an agent alone when:
- its intent is unknown or `Offline`, or its container runs;
- its intent is unknown or `Offline`, or its container runs with a harness that
is active, activating or unreadable (an older hive-priv doesn't know the op: hive-c0re logs
that once and the tick watches the container only);
- the agent has its pause marker, an update in progress, or a container state
the tick can't read in time;
- any queued or running node targets it (update, create, destroy, power op or
@ -342,7 +346,7 @@ An agent still down on the tick after the tick queued a restart has failed to
come up, so the tick spaces restarts of one agent 1, 2, 4, 8, then 15 ticks
apart (a 15-minute cap). Seeing the agent running, or no longer wanted up,
resets the spacing. Each restart logs `reconcile tick: restarting agent` with
the agent, `observed=stopped`, `wanted=up` and the attempt number, and bumps
the agent, `observed=stopped` (`harness stopped` for a running container), `wanted=up` and the attempt number, and bumps
`hyperhive.hive.reconcile_tick.restarts`. `crash_watch` keeps raising its
alert; the tick only converges.

View file

@ -535,15 +535,15 @@ distinguishable from "this hive is gone." Select them with
<!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Meaning |
| ---------------------------------------- | ----------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
| `hyperhive.hive.pressure.stall` | `us` | counter | host-wide `/proc/pressure/{cpu,memory,io}` `total` in microseconds, split by `resource` (`cpu`, `memory`, `io`) and `kind` (`some`, `full`); a rate is the stalled share of time; no sample without PSI |
| `hyperhive.hive.reconcile_tick.restarts` | `{restart}` | counter | restarts the periodic reconcile tick has queued since `hive-c0re` started, for agents declared up and found down; the agent is in the `reconcile tick: restarting agent` log line |
| Metric | Unit | Kind | Meaning |
| ---------------------------------------- | ----------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
| `hyperhive.hive.pressure.stall` | `us` | counter | host-wide `/proc/pressure/{cpu,memory,io}` `total` in microseconds, split by `resource` (`cpu`, `memory`, `io`) and `kind` (`some`, `full`); a rate is the stalled share of time; no sample without PSI |
| `hyperhive.hive.reconcile_tick.restarts` | `{restart}` | counter | restarts the periodic reconcile tick has queued since `hive-c0re` started, for agents declared up and found down, or running with a stopped harness; the agent is in the `reconcile tick: restarting agent` log line |
hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0`
included, so a healthy hive is visible as zeros rather than as missing

View file

@ -46,6 +46,7 @@ known operations; there is no arbitrary command pass-through:
| `DaemonReload` | `systemctl daemon-reload` |
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
| `AgentHarness` | `systemctl -M h-<name> show --property=LoadState --property=ActiveState hive-agent.service` (`Probe`) or `systemctl -M h-<name> --no-block start hive-agent.service` (`Start`). The unit name is a constant in hive-priv, the agent name goes through the same validator as the other per-agent ops, and each call stops after 5 s. It reads and starts one unit inside one agent container: the container's own systemd answers, so hive-priv trusts that reply for the probe only. A hive-priv that predates the op answers an unknown-op error, which hive-c0re reads as "harness state unknown" |
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |