hive-c0re: reconcile declared agents periodically, with backoff
This commit is contained in:
parent
7468aa4cff
commit
b8c07a5492
1 changed files with 516 additions and 9 deletions
|
|
@ -321,6 +321,31 @@ operator sees at a glance how much work the boot triggered. `auto_update::run`
|
|||
seeds agents without an intent from observed state during classification, so a
|
||||
boot with no operator action in this process never finds drift of its own.
|
||||
|
||||
### Periodic reconcile tick
|
||||
|
||||
Once a minute (first tick one minute after boot), `workers::reconcile_tick`
|
||||
looks for an agent that the hive wants up and whose container is down. It reads
|
||||
the power intent, the container's unit state, the pause marker and
|
||||
`Coordinator::is_updating`, then queues the same `Reconcile` node the boot power
|
||||
path uses, through `JobQueue::insert_reconcile_if_idle`. The tick never stops
|
||||
anything.
|
||||
|
||||
It leaves an agent alone when:
|
||||
|
||||
- its intent is unknown or `Offline`, or its container runs;
|
||||
- the agent has its pause marker, an update in progress, or a container state
|
||||
the tick can't read in time;
|
||||
- any queued or running node targets it (update, create, destroy, power op or
|
||||
an earlier reconcile). The busy check and the insert share one lock.
|
||||
|
||||
An agent still down on the tick after the tick queued a restart has failed to
|
||||
come up, so the tick spaces restarts of one agent 1, 2, 4, 8, then 15 ticks
|
||||
apart (a 15-minute cap). Seeing the agent running, or no longer wanted up,
|
||||
resets the spacing. Each restart logs `reconcile tick: restarting agent` with
|
||||
the agent, `observed=stopped`, `wanted=up` and the attempt number, and bumps
|
||||
`hyperhive.hive.reconcile_tick.restarts`. `crash_watch` keeps raising its
|
||||
alert; the tick only converges.
|
||||
|
||||
## Meta flake
|
||||
|
||||
Agents build from the swarm's `internal/meta` flake, which swarm-controller
|
||||
|
|
|
|||
Loading…
Reference in a new issue