treefmt: apply prettier
Pure `nix fmt` output from the commit before this one — no hand edits. 203 files: 52 md, 42 tsx, 32 js, 32 css, 21 ts, 13 html, 8 json, 3 mjs. Reproduce with `nix develop -c nix fmt` on the parent commit; the result should be byte-identical to this tree. None of the 13 `.prettierignore` entries appears here — verified by intersecting the changed-file list against the ignore file, with a control proving the intersection finds a match when one exists.
This commit is contained in:
atlas
parent
5d24bedd60
commit
39b95c2ede
203 changed files with 10090 additions and 6085 deletions
|
|
@ -18,11 +18,11 @@ Two things worth knowing before you rely on it:
|
|||
already has forge access; if you take outside/fork contributions,
|
||||
gate CI behind Forgejo's fork-PR-approval setting or restrict the
|
||||
workflow to push-only triggers (forks can't push to your branches).
|
||||
See *Security* below for the full threat model.
|
||||
See _Security_ below for the full threat model.
|
||||
- **Watch your disk.** CI builds through the host's nix store with no
|
||||
automatic garbage collection of its own — a busy CI day can fill
|
||||
`/nix/store` until every job fails with `ENOSPC`. Add the daily +
|
||||
disk-pressure GC config from *Host store maintenance* below to your
|
||||
disk-pressure GC config from _Host store maintenance_ below to your
|
||||
host's NixOS config (not optional if you plan to actually use this).
|
||||
|
||||
Everything below this point is implementation detail: exact
|
||||
|
|
@ -36,11 +36,11 @@ defined in [`.forgejo/workflows/ci.yml`](../../.forgejo/workflows/ci.yml). All
|
|||
three are required checks (forge branch protection) — a hit on any of them
|
||||
blocks merge.
|
||||
|
||||
| Job | What it runs |
|
||||
| --- | --- |
|
||||
| **nix flake check** | treefmt + rustfmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation |
|
||||
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) |
|
||||
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) |
|
||||
| Job | What it runs |
|
||||
| ---------------------- | ----------------------------------------------------------------------------------------- |
|
||||
| **nix flake check** | treefmt + rustfmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation |
|
||||
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) |
|
||||
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) |
|
||||
|
||||
`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
|
||||
without an empty commit.
|
||||
|
|
@ -90,7 +90,7 @@ slow); run those manually before pushing Rust changes.
|
|||
|
||||
The internal forge is always present (mandatory), so the runner always has a
|
||||
hive-forge instance to register against — nothing extra to enable beyond
|
||||
`services.hyperhive.deploy.forgejo.ci.enable = true` (see *For operators* above).
|
||||
`services.hyperhive.deploy.forgejo.ci.enable = true` (see _For operators_ above).
|
||||
|
||||
Optional tuning: `services.hyperhive.deploy.forgejo.ci.name` (runner name in forge
|
||||
admin panel), `concurrency` (parallel job capacity), `labels` (workflow
|
||||
|
|
@ -131,7 +131,7 @@ Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.serv
|
|||
- The container boots immediately — nothing gates its start on registration.
|
||||
- tmpfiles seeds `/run/hive-ci/runner-token` with `TOKEN=placeholder` so the runner's `EnvironmentFile` always exists.
|
||||
- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it is already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
|
||||
- **Convergence**: because the token write targets the *host* file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
|
||||
- **Convergence**: because the token write targets the _host_ file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
|
||||
|
||||
## Actions checkout mirror
|
||||
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
|
|||
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
|
||||
concurrency comes from the resource classes below, not from multiple workers.
|
||||
Special cases like graceful stop, deferred starts, and the meta-update
|
||||
cascade need no bespoke code paths — each is expressed as a DAG *shape*
|
||||
cascade need no bespoke code paths — each is expressed as a DAG _shape_
|
||||
built from the same primitive nodes.
|
||||
|
||||
### Two levels: DAG and node
|
||||
|
|
@ -42,54 +42,54 @@ there is no malformed spec to reject.
|
|||
|
||||
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
||||
|
||||
| Node | Wraps |
|
||||
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
|
||||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta |
|
||||
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
|
||||
| `DeployWindow` | resource-holding root of the merge-config-PR deploy subtree — declares the build slot, the lease and the meta window, then completes immediately so its children run under them (see _Approvals_ below) |
|
||||
| `DeployApply` | the deploy's irreversible half: ff-merge the reviewed PR head, two-phase meta deploy, container rebuild |
|
||||
| Node | Wraps |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
|
||||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta |
|
||||
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
|
||||
| `DeployWindow` | resource-holding root of the merge-config-PR deploy subtree — declares the build slot, the lease and the meta window, then completes immediately so its children run under them (see _Approvals_ below) |
|
||||
| `DeployApply` | the deploy's irreversible half: ff-merge the reviewed PR head, two-phase meta deploy, container rebuild |
|
||||
|
||||
Cheap — no build slot:
|
||||
|
||||
| Node | Behavior |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `MergeVerify` | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation |
|
||||
| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail: (1) rolls `applied/main` back from the parked `refs/hyperhive/rollback/<id>` and aborts the staged meta lock when the deploy never confirmed good; (2) mirrors whichever deploy tag landed to the forge config repo, always, best-effort; (3) posts the failing build log back onto the config PR when the deploy failed. Named for (2)/(3), which run on the success path too — not `AbortDeploy`. Infallible by construction |
|
||||
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
|
||||
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
||||
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
|
||||
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
||||
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
|
||||
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
|
||||
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
|
||||
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
|
||||
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here |
|
||||
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
|
||||
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
|
||||
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
|
||||
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
|
||||
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
|
||||
| `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/<id>`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree |
|
||||
| `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent |
|
||||
| `EmitRebuilt` | tail of a rebuild/perm-change — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent *and* per outcome |
|
||||
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
|
||||
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
|
||||
| `Reparent` | `set-parent` / `set-parent-bulk`: apply every `(child, new_parent)` move under one `META_LOCK` commit (`meta::bulk_commit_topology`), send the per-agent move notifications, rescan + diff-emit. Agentless like `MetaLock` — a bulk move can span multiple agents, and a reparent touches the meta repo, not any one container. `moves` is typed `(Ident, Option<Ident>)` pairs, not raw strings. Rides the existing `Template::MetaUpdate` variant rather than a dedicated one — it's internal-only (never reaches the graph wire), so the stand-in only affects `terminal_hook` dispatch (resolves to no hook either way) and history-retention bucketing |
|
||||
| `ForgeSweep` | one-shot boot-time forge user/token sweep for every container (`forge::ensure_all`) as a first-class node, so it shows as real work on the dashboard instead of running invisibly in a bare `tokio::spawn`. Agentless |
|
||||
| `MatrixSweep` | same as `ForgeSweep`, for matrix (`matrix::ensure_all`). The periodic 30-min re-sweep stays a background loop in `main.rs`; only the boot-time instance is a node |
|
||||
| `WebhookRegister` | one-shot boot-time Forgejo webhook registration (`internal/knowledge` push→pull, `agent-configs` PR→approval). No-op until the core token, hive domain, and HMAC secret are all available. Agentless |
|
||||
| `KnowledgePull` | one-shot boot-time `/knowledge` pull (`knowledge::pull`), reconciling commits that landed while `hive-c0re` was down. Same rationale as `MatrixSweep`: the periodic hourly re-pull stays a background loop |
|
||||
| `WantedPull` | one-shot boot-time pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. No background loop behind this one — boot is the whole cadence; the deploy event (`swarm_status`) is the fast path, this repairs a missed one. Agentless |
|
||||
| Node | Behavior |
|
||||
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `MergeVerify` | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation |
|
||||
| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail: (1) rolls `applied/main` back from the parked `refs/hyperhive/rollback/<id>` and aborts the staged meta lock when the deploy never confirmed good; (2) mirrors whichever deploy tag landed to the forge config repo, always, best-effort; (3) posts the failing build log back onto the config PR when the deploy failed. Named for (2)/(3), which run on the success path too — not `AbortDeploy`. Infallible by construction |
|
||||
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
|
||||
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
||||
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
|
||||
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
||||
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
|
||||
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
|
||||
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
|
||||
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
|
||||
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here |
|
||||
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
|
||||
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
|
||||
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
|
||||
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
|
||||
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
|
||||
| `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/<id>`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree |
|
||||
| `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent |
|
||||
| `EmitRebuilt` | tail of a rebuild/perm-change — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent _and_ per outcome |
|
||||
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
|
||||
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
|
||||
| `Reparent` | `set-parent` / `set-parent-bulk`: apply every `(child, new_parent)` move under one `META_LOCK` commit (`meta::bulk_commit_topology`), send the per-agent move notifications, rescan + diff-emit. Agentless like `MetaLock` — a bulk move can span multiple agents, and a reparent touches the meta repo, not any one container. `moves` is typed `(Ident, Option<Ident>)` pairs, not raw strings. Rides the existing `Template::MetaUpdate` variant rather than a dedicated one — it's internal-only (never reaches the graph wire), so the stand-in only affects `terminal_hook` dispatch (resolves to no hook either way) and history-retention bucketing |
|
||||
| `ForgeSweep` | one-shot boot-time forge user/token sweep for every container (`forge::ensure_all`) as a first-class node, so it shows as real work on the dashboard instead of running invisibly in a bare `tokio::spawn`. Agentless |
|
||||
| `MatrixSweep` | same as `ForgeSweep`, for matrix (`matrix::ensure_all`). The periodic 30-min re-sweep stays a background loop in `main.rs`; only the boot-time instance is a node |
|
||||
| `WebhookRegister` | one-shot boot-time Forgejo webhook registration (`internal/knowledge` push→pull, `agent-configs` PR→approval). No-op until the core token, hive domain, and HMAC secret are all available. Agentless |
|
||||
| `KnowledgePull` | one-shot boot-time `/knowledge` pull (`knowledge::pull`), reconciling commits that landed while `hive-c0re` was down. Same rationale as `MatrixSweep`: the periodic hourly re-pull stays a background loop |
|
||||
| `WantedPull` | one-shot boot-time pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. No background loop behind this one — boot is the whole cadence; the deploy event (`swarm_status`) is the fast path, this repairs a missed one. Agentless |
|
||||
|
||||
There is deliberately **no `GitCommit` node**: `meta.rs` fuses each mutation
|
||||
with its commit under its internal `META_LOCK` mutex, so a standalone commit
|
||||
node would open a dirty-working-tree window between nodes.
|
||||
|
||||
Two further layers protect the meta repo across *windows* that span multiple
|
||||
Two further layers protect the meta repo across _windows_ that span multiple
|
||||
`META_LOCK` acquisitions — above all the approval deploy's prepare→finalize
|
||||
span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole
|
||||
container build:
|
||||
|
|
@ -104,7 +104,7 @@ container build:
|
|||
because a resource is held by a subtree root across its whole subtree, which
|
||||
a `MutexGuard` (bounded by one executor fn) cannot — that is what lets a
|
||||
multi-node deploy own one window. For the same reason the window must stay
|
||||
*off* long store-only work: the rebuild's meta preamble is its own
|
||||
_off_ long store-only work: the rebuild's meta preamble is its own
|
||||
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
|
||||
toplevel build runs outside the window and `buildSlots > 1` still gives
|
||||
concurrent rebuilds across agents.
|
||||
|
|
@ -119,10 +119,10 @@ container build:
|
|||
The `stop` / `start` power ops write the durable `wanted` intent via a head
|
||||
`SetWanted` node (not a pre-submit side effect) — it holds the agent lease,
|
||||
so intent-write + reconcile is atomic per-agent. `restart` is the exception:
|
||||
it writes *no* intent (no `SetWanted` head) — it bounces the container and
|
||||
it writes _no_ intent (no `SetWanted` head) — it bounces the container and
|
||||
lets the tail `Reconcile` converge to the agent's existing `wanted`, so a
|
||||
deliberately-stopped agent is not forced back up by a hive-wide restart. The
|
||||
hive-wide power ops — `restart`, `stop`, and `start` — take an agent *list*:
|
||||
hive-wide power ops — `restart`, `stop`, and `start` — take an agent _list_:
|
||||
a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
|
||||
subgraph each (independent roots, run concurrently on their own leases), not
|
||||
N separate DAGs.
|
||||
|
|
@ -132,12 +132,12 @@ async `lifecycle::is_running` read), so they live in `job_queue/power.rs`,
|
|||
not the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
|
||||
a head `SetWanted` (intent) — `restart` does not; the tail `Reconcile`
|
||||
(convergence guarantee — cheap, noops when already converged) is ALWAYS
|
||||
present; only the *mechanical* nodes (`Signal`/`Drain`/`StopForUpdate`) are
|
||||
state-conditional — skipped for a *down* agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
|
||||
present; only the _mechanical_ nodes (`Signal`/`Drain`/`StopForUpdate`) are
|
||||
state-conditional — skipped for a _down_ agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
|
||||
closes the TOCTOU window: if an agent flips state between the `is_running`
|
||||
read and node exec, the tail `Reconcile` still converges it in-DAG (with
|
||||
`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
|
||||
sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale*
|
||||
sweep. `start` folds the per-agent stale-rev upgrade in (a _down + stale_
|
||||
agent's subgraph is a rebuild-then-start).
|
||||
|
||||
```text
|
||||
|
|
@ -163,7 +163,7 @@ Notable collapses:
|
|||
leaves it stopped).
|
||||
- **The swap-failure recovery-start is structural**: `Reconcile` deps on
|
||||
`Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
|
||||
terminal ok *or* fail, bringing a wanted-up agent back on its old config.
|
||||
terminal ok _or_ fail, bringing a wanted-up agent back on its old config.
|
||||
- **Deferred start is automatic**: `Reconcile` holds no build slot, so the
|
||||
next DAG's `Prebuild` starts as soon as `Swap` frees the slot.
|
||||
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
|
||||
|
|
@ -171,7 +171,7 @@ Notable collapses:
|
|||
drains overlap; each DAG's tail `Reconcile` does the actual stop.
|
||||
- **The meta-update cascade grows in the same DAG on completion**:
|
||||
`MetaLock`'s executor computes the affected agent set after the bump lands
|
||||
and grows one `rebuild` subgraph per agent into its *own* DAG via
|
||||
and grows one `rebuild` subgraph per agent into its _own_ DAG via
|
||||
`append_subgraph` (rooted on the `MetaLock`, `relock = false` so the cascade
|
||||
doesn't revert the bump). Not child DAGs — one DAG, no `parent_id`. A failed
|
||||
bump appends nothing (no cancel-children dance). Same shape as the startup
|
||||
|
|
@ -180,9 +180,9 @@ Notable collapses:
|
|||
|
||||
### Desired-state (spec vs status)
|
||||
|
||||
Per-agent power *intent* — `wanted: Up | Offline` — is durable as the
|
||||
Per-agent power _intent_ — `wanted: Up | Offline` — is durable as the
|
||||
`agent_power` table in the coordinator DB (`hive-c0re/src/stores/power.rs`).
|
||||
`container_view` remains the observed *status*; `Reconcile` nodes converge the
|
||||
`container_view` remains the observed _status_; `Reconcile` nodes converge the
|
||||
two. Setting `wanted` is never a queued node: the power layer
|
||||
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
|
||||
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
|
||||
|
|
@ -237,15 +237,15 @@ than to one DAG shape it happens to appear in. A **brace** is the one sanctioned
|
|||
exception: a pure-resource-holder root that declares on behalf of a subtree
|
||||
coordinated with itself, whose members then declare nothing.
|
||||
|
||||
It is forced rather than stylistic. Declaring a resource means *"I need this
|
||||
exclusively"*, and the agent lease is single-unit — so **two siblings that both
|
||||
It is forced rather than stylistic. Declaring a resource means _"I need this
|
||||
exclusively"_, and the agent lease is single-unit — so **two siblings that both
|
||||
declared it could never run concurrently.** For a subtree whose whole point is
|
||||
concurrency (`Prebuild` beside the `Signal` → `Drain` quiesce window), declaring
|
||||
the requirement truthfully on every node and running those nodes in parallel are
|
||||
mutually exclusive. One holder above them speaks for the subtree.
|
||||
|
||||
This is the opposite of the failure the declare-your-own rule exists to prevent,
|
||||
not a relapse into it: there the requirement was *implicit*, inferred from a
|
||||
not a relapse into it: there the requirement was _implicit_, inferred from a
|
||||
node's kind and true only by accident of placement. Here it is explicit, on one
|
||||
node, with the omission below it documented on the brace itself.
|
||||
|
||||
|
|
@ -257,7 +257,7 @@ Two consequences worth knowing:
|
|||
leave a gap another DAG could claim the agent in, mid-bounce. A brace supplies
|
||||
that continuity directly, so the nesting is no longer load-bearing.
|
||||
- **Observability is unaffected.** `running_transients` keys off a node's
|
||||
*payload* agent, not off a declared lease edge, so every child still lights its
|
||||
_payload_ agent, not off a declared lease edge, so every child still lights its
|
||||
own dashboard pill and still reports its own `takes_container_down` to the
|
||||
crash watcher. A brace itself reports `false`: it parents the stopping nodes
|
||||
but does not stop anything, and claiming otherwise would widen crash
|
||||
|
|
@ -323,12 +323,12 @@ cancelled-while-queued, which fails the approval instead of dangling it).
|
|||
### Wire shape
|
||||
|
||||
`RebuildQueueChanged { seq }` (event name kept) — **a bare trigger, no
|
||||
payload.** It says *the queue changed*; a client that wants to know how
|
||||
payload.** It says _the queue changed_; a client that wants to know how
|
||||
re-fetches `GET /api/jobq/graph`.
|
||||
|
||||
That endpoint serves the graph generically (`hive-jobq-wire`): every node
|
||||
carries `id`, `parent`, `deps`, `state`, `label` (the node kind's own wire
|
||||
string — the kind *is* the phase label, there is no separate sub-step
|
||||
string — the kind _is_ the phase label, there is no separate sub-step
|
||||
string) and free-form `data` for what only some kinds have (`agent`,
|
||||
`approval_id`, `inputs`, `build_log_id`). Group roots ride as ordinary
|
||||
nodes, so a group's state is just the root's own `state`.
|
||||
|
|
@ -341,7 +341,7 @@ live-log panel off the running node.
|
|||
The event carries no payload by design: shipping a typed projection of
|
||||
the whole queue in the event itself would be a second rendering of the
|
||||
same graph that has to be kept in agreement by hand with the endpoint
|
||||
every consumer actually reads. Telling a client *when* to refetch is
|
||||
every consumer actually reads. Telling a client _when_ to refetch is
|
||||
the event's whole job.
|
||||
|
||||
---
|
||||
|
|
@ -362,7 +362,7 @@ On startup, `auto_update::run` classifies every agent by rev freshness (the
|
|||
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
|
||||
vs the current flake path) and persisted `wanted` intent, then:
|
||||
|
||||
1. **Config path** — when *any* marker is stale, submit one `Boot`
|
||||
1. **Config path** — when _any_ marker is stale, submit one `Boot`
|
||||
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that grows an in-DAG
|
||||
`Rebuild` subgraph for each stale agent whose `wanted = Up` (topology-sorted,
|
||||
parents first). Stale but wanted-offline agents get no boot-time nix work — their
|
||||
|
|
@ -453,7 +453,7 @@ graft's completion then plants the deploy tag — so "did the agent come back
|
|||
up?" is answered by `Reconcile` succeeding, the same way it is for every
|
||||
other rebuild, instead of by a fused inline start.
|
||||
|
||||
The grafted nodes land *inside* `DeployWindow`'s subtree, so they re-enter
|
||||
The grafted nodes land _inside_ `DeployWindow`'s subtree, so they re-enter
|
||||
the meta window and build slot it already holds rather than deadlocking
|
||||
against it.
|
||||
|
||||
|
|
@ -514,7 +514,7 @@ The same drop-in carries `CPUWeight=` / `IOWeight=` from
|
|||
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
|
||||
setting: the quota and the memory max are **hard caps** that throttle
|
||||
an agent even on a completely idle host, while the weights are cgroup
|
||||
v2 **relative shares** that only decide who yields *under contention*.
|
||||
v2 **relative shares** that only decide who yields _under contention_.
|
||||
A low-weight container still gets the whole machine when nothing else
|
||||
wants it.
|
||||
|
||||
|
|
@ -529,7 +529,7 @@ Two things to know about the weights:
|
|||
|
||||
- They are **hive-wide** — unlike the caps there is no per-agent
|
||||
override in `meta/resource-limits.json`, so every agent carries the
|
||||
same value and the weight does *not* rank agents against each other.
|
||||
same value and the weight does _not_ rank agents against each other.
|
||||
What `80` buys is that agents yield to everything **not** on this
|
||||
drop-in path: host services (nginx and dnsmasq among them) and the
|
||||
infra containers (`hive-ci`, `hive-forge`, `hive-gateway`,
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Every container operation — rebuild, first-spawn, a config-PR deploy,
|
||||
power changes — runs through one shared job queue. This page explains
|
||||
what the job queue *is*, as a general idea, independent of what any one
|
||||
what the job queue _is_, as a general idea, independent of what any one
|
||||
subsystem uses it for. For the hive-c0re-specific step catalogue and the
|
||||
engineering internals (scheduler, leases, resource windows) see
|
||||
[`coordinator.md`](coordinator.md) instead.
|
||||
|
|
@ -23,7 +23,7 @@ Two ideas are all there is to it:
|
|||
that's the whole reason things queue instead of all firing at once.
|
||||
|
||||
The engine's whole job is: whenever a step's ordering and resource needs
|
||||
are both satisfied, run it. It has no opinion on what the steps *do* —
|
||||
are both satisfied, run it. It has no opinion on what the steps _do_ —
|
||||
that's supplied by whoever builds the graph. hive-c0re is the one thing
|
||||
building graphs on it today, but nothing about the engine is specific to
|
||||
containers or rebuilds; there's nothing stopping another subsystem from
|
||||
|
|
@ -38,14 +38,14 @@ nested under it are that job's steps, in order (occasionally a couple run
|
|||
side by side). A step shows one of:
|
||||
|
||||
| Glyph | Meaning |
|
||||
| ----- | -------------------------------------------------------- |
|
||||
| `⏸` | queued, waiting its turn |
|
||||
| `▶` | running |
|
||||
| `◐` | its own work is done, waiting on a step nested under it |
|
||||
| `✔` | finished successfully |
|
||||
| `✖` | failed |
|
||||
| `⊘` | cancelled |
|
||||
| `·` | skipped (not needed for this run) |
|
||||
| ----- | ------------------------------------------------------- |
|
||||
| `⏸` | queued, waiting its turn |
|
||||
| `▶` | running |
|
||||
| `◐` | its own work is done, waiting on a step nested under it |
|
||||
| `✔` | finished successfully |
|
||||
| `✖` | failed |
|
||||
| `⊘` | cancelled |
|
||||
| `·` | skipped (not needed for this run) |
|
||||
|
||||
A step that isn't needed for a given run shows as `·` rather than being
|
||||
left out of the tree entirely, so the same kind of operation keeps a
|
||||
|
|
|
|||
|
|
@ -46,7 +46,7 @@ no longer survives independently of anything host-side.
|
|||
**It has no application-level auth.** The receiver takes any OTLP that reaches
|
||||
it; what bounds who can reach it is the firewall — `exposeHostPorts` opens the
|
||||
port on the bridge interface only. So "unauthenticated to a bridge address"
|
||||
means *reachable from an agent container*, not *presents a credential*.
|
||||
means _reachable from an agent container_, not _presents a credential_.
|
||||
|
||||
The consequence, stated because it is a choice rather than an oversight: **any
|
||||
agent can push arbitrary OTLP, and it is forwarded on under the operator's
|
||||
|
|
@ -56,10 +56,10 @@ smuggled out in resource attributes on an otherwise-legitimate export.
|
|||
|
||||
That is a **different risk from the one the collector fixes**, and strictly
|
||||
smaller than what preceded it: before, every agent held the upstream credential
|
||||
itself, so it could do all of the above *and* use the token anywhere else. The
|
||||
itself, so it could do all of the above _and_ use the token anywhere else. The
|
||||
collector removes the token and keeps the pipe. Agents are inside the trust
|
||||
boundary (`docs/trust-boundary/security.md`: capability = accepted risk), so an agent being
|
||||
able to *send* is an accepted extension of that boundary — but it is not
|
||||
able to _send_ is an accepted extension of that boundary — but it is not
|
||||
closed by this design, and nothing here should be read as closing it.
|
||||
|
||||
**The `agent` label is self-reported, and no planned authentication changes
|
||||
|
|
@ -73,13 +73,13 @@ is getting a credential:
|
|||
- **agent→collector** (this section's hop) stays open on the bridge. Nothing
|
||||
downstream can tell one agent's export from another's.
|
||||
- **hive→swarm** is where the planned ingest auth goes. The swarm tier stamps
|
||||
`hive=` from the connection it authenticated, so *that* label becomes
|
||||
`hive=` from the connection it authenticated, so _that_ label becomes
|
||||
unforgeable.
|
||||
|
||||
So a verified `hive` is reachable and a verified `agent` is not — and that falls
|
||||
out of the topology rather than being a gap someone forgot to close. The swarm
|
||||
runs one collector, and the mechanism gives it no finer grain: a bearer-token
|
||||
check never reveals *which* token matched, and a receiver reads request metadata
|
||||
check never reveals _which_ token matched, and a receiver reads request metadata
|
||||
rather than the claims it authenticated with.
|
||||
|
||||
If you need per-agent numbers you can act on, take them from the agent's own
|
||||
|
|
@ -100,10 +100,10 @@ two-tier architecture, the security model, and how the options interact.
|
|||
Telemetry crosses two collectors, and which one you configure depends on what
|
||||
the host is:
|
||||
|
||||
| | runs where | receives from | does |
|
||||
|---|---|---|---|
|
||||
| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination |
|
||||
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's store and exports upstream |
|
||||
| | runs where | receives from | does |
|
||||
| ------------------------------------ | ---------------------- | --------------------------------- | --------------------------------------------------------------------- |
|
||||
| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination |
|
||||
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's store and exports upstream |
|
||||
|
||||
An all-local host runs both, and needs nothing said about the hop between them.
|
||||
|
||||
|
|
@ -119,16 +119,16 @@ services.hyperhive.otel = {
|
|||
every agent needs the credential to authenticate — and the harness delivers
|
||||
that token into the agent's own `~/.claude/settings.json`, a file the agent can
|
||||
read. `0600` protects it from other containers, not from the agent itself. As
|
||||
long as the direct path stays *selectable*, that hole stays selectable; an
|
||||
long as the direct path stays _selectable_, that hole stays selectable; an
|
||||
option that can reintroduce it is a hole with extra steps.
|
||||
|
||||
**Why the tiers stay separate on one box.** They are not collapsed when
|
||||
co-located: an all-local hive is a statement about *where* processes run, not
|
||||
co-located: an all-local hive is a statement about _where_ processes run, not
|
||||
about the shape of the deployment. A boundary that disappears locally is one
|
||||
the local deployment stops testing.
|
||||
|
||||
**`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier
|
||||
redefines it — the agent-facing value is *derived*
|
||||
redefines it — the agent-facing value is _derived_
|
||||
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
|
||||
keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
|
||||
automatically; there is nothing to open by hand.
|
||||
|
|
@ -160,9 +160,9 @@ swarm tier and no metrics appearing for that hive.
|
|||
|
||||
⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
|
||||
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
|
||||
exporter takes an *address*: `https://host/path` is a legal
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in
|
||||
address"*. The collector's config is validated at build time, so a mismatch is
|
||||
exporter takes an _address_: `https://host/path` is a legal
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in
|
||||
address"_. The collector's config is validated at build time, so a mismatch is
|
||||
a build error naming the reason rather than telemetry silently going nowhere.
|
||||
|
||||
## Network access
|
||||
|
|
@ -176,7 +176,7 @@ services.hyperhive.network.exposeHostPorts = [ 5432 ];
|
|||
```
|
||||
|
||||
and point whatever consumes it at `10.42.0.1:5432` rather than loopback: inside
|
||||
a container, loopback is the *container*. The bridge IP is the host's address on
|
||||
a container, loopback is the _container_. The bridge IP is the host's address on
|
||||
the `hive-br0` bridge. The service must also bind an address the bridge can
|
||||
reach — a `127.0.0.1`-only listener stays unreachable no matter what the
|
||||
firewall allows. See `docs/networking/network.md::Reaching host services` for details.
|
||||
|
|
@ -196,12 +196,12 @@ yourself from a tool call or `hivectl shell`.
|
|||
Every agent's export therefore includes these resource attributes
|
||||
automatically:
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| `service.name` | `hyperhive-agent` (constant) |
|
||||
| `agent` | agent logical name (e.g. `iris`) |
|
||||
| `hive` | hive display name (`services.hyperhive.hiveName`) |
|
||||
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
|
||||
| Attribute | Value |
|
||||
| -------------- | ------------------------------------------------------------ |
|
||||
| `service.name` | `hyperhive-agent` (constant) |
|
||||
| `agent` | agent logical name (e.g. `iris`) |
|
||||
| `hive` | hive display name (`services.hyperhive.hiveName`) |
|
||||
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
|
||||
|
||||
Additional labels can be appended via `extraResourceAttributes` (see option
|
||||
reference above); custom per-data-point labels can be passed with
|
||||
|
|
@ -222,14 +222,14 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
|
|||
`service.name = hyperhive-c0re`; each data point is tagged `container.name`
|
||||
(= the `h-<agent>` machine) and the hive `agent` label:
|
||||
|
||||
| Metric | Unit | Kind | Source |
|
||||
|--------|------|------|--------|
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | state dir + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
|
||||
| Metric | Unit | Kind | Source |
|
||||
| ----------------------------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------- |
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | state dir + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
|
||||
|
||||
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
|
||||
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
|
||||
|
|
@ -252,14 +252,14 @@ turn, its own outcome classification, the loose-ends backlog, and session
|
|||
boundaries. Token usage, cost, and tool-call counts are **not** duplicated
|
||||
here; that's already covered by Claude's own export.
|
||||
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
|--------|------|------|------------|
|
||||
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
|
||||
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
|
||||
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
|
||||
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `CLAUDE.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
| -------------------------------------- | ---- | --------- | ----------------------------------------------------------------------------------------------------------- |
|
||||
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
|
||||
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
|
||||
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
|
||||
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `CLAUDE.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
|
||||
|
||||
Resource attributes (`service.name`, `agent`, `hive`, `swarm`) come from the
|
||||
same container-wide `OTEL_RESOURCE_ATTRIBUTES` as everything else in this
|
||||
|
|
@ -277,11 +277,11 @@ that hosts no agents still reports, and "this hive is quiet" is
|
|||
distinguishable from "this hive is gone". Select them with
|
||||
`{hive!="",agent=""}`.
|
||||
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
|--------|------|------|---------|
|
||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
| ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
|
||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||
|
||||
Both levels are reported every cycle, `0` included, so a healthy hive is
|
||||
visible as zeros rather than as missing series.
|
||||
|
|
@ -303,10 +303,10 @@ Forgejo's own native `/metrics` endpoint has no equivalent: it exposes
|
|||
counts of durable rows (issues, comments, repos), and neither a commit nor a
|
||||
push is stored anywhere as a row to count.
|
||||
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
|--------|------|------|------------|
|
||||
| `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) |
|
||||
| `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) |
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
| ---------------------------- | ---- | ------- | ------------------- |
|
||||
| `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) |
|
||||
| `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) |
|
||||
|
||||
A push with zero commits (a branch delete, or a force-push that doesn't add
|
||||
new commits) still increments `push.count`; `commit.count` only advances
|
||||
|
|
|
|||
Loading…
Reference in a new issue