Watch
0
0
Fork
You've already forked hyperhive
0

hive-c0re: reconcile declared agents periodically, with backoff

This commit is contained in:
atlas 2026-11-14 20:52:31 +01:00 • committed by mara
commit b8c07a5492

View file

@ -321,6 +321,31 @@ operator sees at a glance how much work the boot triggered. `auto_update::run`
seeds agents without an intent from observed state during classification, so a
boot with no operator action in this process never finds drift of its own.
### Periodic reconcile tick
Once a minute (first tick one minute after boot), `workers::reconcile_tick`
looks for an agent that the hive wants up and whose container is down. It reads
the power intent, the container's unit state, the pause marker and
`Coordinator::is_updating`, then queues the same `Reconcile` node the boot power
path uses, through `JobQueue::insert_reconcile_if_idle`. The tick never stops
anything.
It leaves an agent alone when:
- its intent is unknown or `Offline`, or its container runs;
- the agent has its pause marker, an update in progress, or a container state
the tick can't read in time;
- any queued or running node targets it (update, create, destroy, power op or
an earlier reconcile). The busy check and the insert share one lock.
An agent still down on the tick after the tick queued a restart has failed to
come up, so the tick spaces restarts of one agent 1, 2, 4, 8, then 15 ticks
apart (a 15-minute cap). Seeing the agent running, or no longer wanted up,
resets the spacing. Each restart logs `reconcile tick: restarting agent` with
the agent, `observed=stopped`, `wanted=up` and the attempt number, and bumps
`hyperhive.hive.reconcile_tick.restarts`. `crash_watch` keeps raising its
alert; the tick only converges.
## Meta flake
Agents build from the swarm's `internal/meta` flake, which swarm-controller