hive-c0re: reconcile declared agents periodically, with backoff
This commit is contained in:
atlas
mara
parent
7468aa4cff
commit
b8c07a5492
2 changed files with 516 additions and 9 deletions
|
|
@ -321,6 +321,31 @@ operator sees at a glance how much work the boot triggered. `auto_update::run`
|
|||
seeds agents without an intent from observed state during classification, so a
|
||||
boot with no operator action in this process never finds drift of its own.
|
||||
|
||||
### Periodic reconcile tick
|
||||
|
||||
Once a minute (first tick one minute after boot), `workers::reconcile_tick`
|
||||
looks for an agent that the hive wants up and whose container is down. It reads
|
||||
the power intent, the container's unit state, the pause marker and
|
||||
`Coordinator::is_updating`, then queues the same `Reconcile` node the boot power
|
||||
path uses, through `JobQueue::insert_reconcile_if_idle`. The tick never stops
|
||||
anything.
|
||||
|
||||
It leaves an agent alone when:
|
||||
|
||||
- its intent is unknown or `Offline`, or its container runs;
|
||||
- the agent has its pause marker, an update in progress, or a container state
|
||||
the tick can't read in time;
|
||||
- any queued or running node targets it (update, create, destroy, power op or
|
||||
an earlier reconcile). The busy check and the insert share one lock.
|
||||
|
||||
An agent still down on the tick after the tick queued a restart has failed to
|
||||
come up, so the tick spaces restarts of one agent 1, 2, 4, 8, then 15 ticks
|
||||
apart (a 15-minute cap). Seeing the agent running, or no longer wanted up,
|
||||
resets the spacing. Each restart logs `reconcile tick: restarting agent` with
|
||||
the agent, `observed=stopped`, `wanted=up` and the attempt number, and bumps
|
||||
`hyperhive.hive.reconcile_tick.restarts`. `crash_watch` keeps raising its
|
||||
alert; the tick only converges.
|
||||
|
||||
## Meta flake
|
||||
|
||||
Agents build from the swarm's `internal/meta` flake, which swarm-controller
|
||||
|
|
|
|||
|
|
@ -533,14 +533,15 @@ distinguishable from "this hive is gone." Select them with
|
|||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
| -------------------------------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
|
||||
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
|
||||
| `hyperhive.hive.pressure.stall` | `us` | counter | host-wide `/proc/pressure/{cpu,memory,io}` `total` in microseconds, split by `resource` (`cpu`, `memory`, `io`) and `kind` (`some`, `full`); a rate is the stalled share of time; no sample without PSI |
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
| ---------------------------------------- | ----------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
|
||||
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
|
||||
| `hyperhive.hive.pressure.stall` | `us` | counter | host-wide `/proc/pressure/{cpu,memory,io}` `total` in microseconds, split by `resource` (`cpu`, `memory`, `io`) and `kind` (`some`, `full`); a rate is the stalled share of time; no sample without PSI |
|
||||
| `hyperhive.hive.reconcile_tick.restarts` | `{restart}` | counter | restarts the periodic reconcile tick has queued since `hive-c0re` started, for agents declared up and found down; the agent is in the `reconcile tick: restarting agent` log line |
|
||||
|
||||
hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0`
|
||||
included, so a healthy hive is visible as zeros rather than as missing
|
||||
|
|
|
|||
Loading…
Reference in a new issue