Compare commits
37 changed files with 325 additions and 478 deletions
|
|
@ -66,7 +66,7 @@ pull`, so agents see the new content on their next turn.
|
||||||
queues a pull on a fixed cadence as a fallback (webhook missed, c0re
|
queues a pull on a fixed cadence as a fallback (webhook missed, c0re
|
||||||
restarted between pushes). The pull is best-effort — a failure
|
restarted between pushes). The pull is best-effort — a failure
|
||||||
shows as a failed `KnowledgePull` node on the job queue, counts toward
|
shows as a failed `KnowledgePull` node on the job queue, counts toward
|
||||||
the `knowledge_pull` hive warning, and doesn't affect the rest of the daemon.
|
the knowledge-pull banner, and doesn't affect the rest of the daemon.
|
||||||
|
|
||||||
Both paths, and the one pull at boot, queue the same `KnowledgePull` job
|
Both paths, and the one pull at boot, queue the same `KnowledgePull` job
|
||||||
node. It holds the working tree for its duration, so two pulls never run
|
node. It holds the working tree for its duration, so two pulls never run
|
||||||
|
|
|
||||||
|
|
@ -529,8 +529,8 @@ given.
|
||||||
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
|
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
|
||||||
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
|
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
|
||||||
value passes the validators first. One bad
|
value passes the validators first. One bad
|
||||||
value or a failed read leaves the file unchanged, and repeated failures raise a hive
|
value or a failed read leaves the file unchanged, and repeated failures raise a
|
||||||
warning (the `hyperhive.hive.warning` metric and the `hh-hive-warning` alert).
|
dashboard banner.
|
||||||
|
|
||||||
A hive's queue grant reads every key of the bucket
|
A hive's queue grant reads every key of the bucket
|
||||||
(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the
|
(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the
|
||||||
|
|
|
||||||
|
|
@ -444,7 +444,7 @@ whole-swarm spend figure.
|
||||||
<!-- vale write-good.Passive = NO -->
|
<!-- vale write-good.Passive = NO -->
|
||||||
|
|
||||||
hive-c0re measures everything above **per agent**, tagged with the hive it
|
hive-c0re measures everything above **per agent**, tagged with the hive it
|
||||||
runs in. It measures these five per **hive** instead, and they carry no
|
runs in. It measures these three per **hive** instead, and they carry no
|
||||||
`agent` label — so a hive
|
`agent` label — so a hive
|
||||||
that hosts no agents still reports, and "this hive is quiet" is
|
that hosts no agents still reports, and "this hive is quiet" is
|
||||||
distinguishable from "this hive is gone." Select them with
|
distinguishable from "this hive is gone." Select them with
|
||||||
|
|
@ -452,41 +452,20 @@ distinguishable from "this hive is gone." Select them with
|
||||||
|
|
||||||
<!-- vale write-good.Passive = YES -->
|
<!-- vale write-good.Passive = YES -->
|
||||||
|
|
||||||
| Metric | Unit | Kind | Meaning |
|
| Metric | Unit | Kind | Meaning |
|
||||||
| -------------------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
|
| ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
|
||||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||||
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
|
|
||||||
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
|
|
||||||
|
|
||||||
hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0`
|
hive-c0re reports both levels every cycle, `0` included, so a healthy hive is
|
||||||
included, so a healthy hive is visible as zeros rather than as missing
|
visible as zeros rather than as missing series.
|
||||||
series.
|
|
||||||
|
|
||||||
`hyperhive.hive.degraded` is what a dashboard should alert on: it's
|
`hyperhive.hive.degraded` is what a dashboard should alert on: it's
|
||||||
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
|
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
|
||||||
and with what the swarm controller sees. `hyperhive.hive.warnings` is the
|
and with what the swarm controller sees. `hyperhive.hive.warnings` is the
|
||||||
detail behind it — `warn`-level entries mean "an operator should look" and do
|
detail behind it — `warn`-level entries mean "an operator should look" and do
|
||||||
**not** set `degraded`. `hyperhive.hive.warning` names the cause: the `kind`
|
**not** set `degraded`.
|
||||||
label is one of a fixed set in the source (for example `disk_pressure`, a
|
|
||||||
failing sweep such as `swarm_status_publish`, `pending_logins`,
|
|
||||||
`forge_ensure_core_user`), and the message lives in the
|
|
||||||
`hive warning raised` line hive-c0re logs when it raises the warning.
|
|
||||||
|
|
||||||
The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
|
|
||||||
|
|
||||||
| Alert | Fires when |
|
|
||||||
| ----------------------- | ----------------------------------------------------------------------------------- |
|
|
||||||
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
|
|
||||||
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists for 2 minutes |
|
|
||||||
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
|
|
||||||
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |
|
|
||||||
|
|
||||||
The disk alerts replace the old dashboard banner for the nix store, and
|
|
||||||
`hh-hive-warning` replaces it for every other warning. Disk pressure has its
|
|
||||||
own pair so the thresholds sit on the metric rather than on the registry
|
|
||||||
entry.
|
|
||||||
|
|
||||||
Same cadence, transport and resource labels as the container metrics above.
|
Same cadence, transport and resource labels as the container metrics above.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -148,8 +148,8 @@ swarm-services certificate.
|
||||||
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
|
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
|
||||||
and publishes under `hiveName`, the same name that keys `swarm.hives`.
|
and publishes under `hiveName`, the same name that keys `swarm.hives`.
|
||||||
|
|
||||||
If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
|
If a hive stops reporting, its own dashboard is the place to look: a
|
||||||
failure to publish raises a hive warning after three consecutive
|
failure to publish raises a warning banner there after three consecutive
|
||||||
misses. It stays `warn` rather than `crit` on purpose — a hive that
|
misses. It stays `warn` rather than `crit` on purpose — a hive that
|
||||||
can't reach the queue isn't itself unhealthy, so it doesn't start
|
can't reach the queue isn't itself unhealthy, so it doesn't start
|
||||||
calling itself degraded for being unable to say it's fine.
|
calling itself degraded for being unable to say it's fine.
|
||||||
|
|
|
||||||
|
|
@ -31,17 +31,26 @@ dashboard tab strip.
|
||||||
K3PT ST4T3 (kept-state entries).
|
K3PT ST4T3 (kept-state entries).
|
||||||
- **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`)
|
- **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`)
|
||||||
— sits below the tab strip.
|
— sits below the tab strip.
|
||||||
- **No warning banner.** No page shows hive-level warnings (host disk
|
- **Server-warnings banner** — a generic, sticky top-of-page strip shown
|
||||||
pressure, failing background sweeps, forge boot failures, agents that
|
on **every** page (the dashboard + the stand-alone FL0W
|
||||||
need a claude login). They live in a push-based registry
|
and ST4TS pages), injected at the top
|
||||||
(`hive-c0re`'s `warnings.rs`): any subsystem raises or clears its own
|
of `<body>` by `renderServerWarnings` in `common.js`. Driven by
|
||||||
entry through an RAII guard, and each raise logs one
|
`state.server_warnings` — a list of `{ kind, level, message }` — and
|
||||||
`hive warning raised` line. The operator sees the registry as metrics and
|
coloured by `level` (`warn` amber / `crit` red). hive-c0re owns the
|
||||||
Grafana alerts: see
|
threshold + message, so adding a new system warning needs no frontend
|
||||||
[Hive-scoped metrics](../scheduler/observability.md#hive-scoped-metrics-hive-c0re).
|
change. Warnings are a push-based registry (`hive-c0re`'s
|
||||||
A container crash is a swarm-level Grafana alert on the crash watcher's
|
`warnings.rs`): any subsystem raises/clears its own entry via an RAII
|
||||||
own log line, so a hive with no otel export has no operator-visible
|
guard, and `host_stats::server_warnings()` is just a cheap snapshot of
|
||||||
warning signal beyond `/health/ready`.
|
that registry. Producers today include the host disk-pressure check
|
||||||
|
(a `statvfs` probe of `/nix`: ≥85% used → `warn`, ≥95% → `crit`, e.g.
|
||||||
|
`⚠ host nix store N% full (G GiB free) — garbage-collect …`), forge
|
||||||
|
provisioning/CI-runner boot failures, and an agent-state warning
|
||||||
|
(`pending_logins`) computed alongside the container snapshot. A
|
||||||
|
container crash used to have a sibling `agents_crashing` entry here —
|
||||||
|
removed in favour of a swarm-level grafana alert on the crash
|
||||||
|
watcher's own log line (`docs/scheduler/observability.md`), so a
|
||||||
|
hive with no otel export now has no operator-visible crash signal at
|
||||||
|
all. Hidden when there are no warnings.
|
||||||
- **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated
|
- **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated
|
||||||
to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name`
|
to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name`
|
||||||
arrive in the state snapshot.
|
arrive in the state snapshot.
|
||||||
|
|
|
||||||
|
|
@ -564,7 +564,7 @@ hive-side-panel .md td {
|
||||||
Inverted-gutter layout: the dashboard keeps `<body>` full-bleed
|
Inverted-gutter layout: the dashboard keeps `<body>` full-bleed
|
||||||
and wraps their padded content in a single `.page-content` div that
|
and wraps their padded content in a single `.page-content` div that
|
||||||
carries the horizontal gutter. Full-width chrome (the tab strip, the
|
carries the horizontal gutter. Full-width chrome (the tab strip, the
|
||||||
footer) then spans edge-to-edge for free,
|
server-warnings banner, the footer) then spans edge-to-edge for free,
|
||||||
with no negative-margin breakout hacks. Only matches on pages that opt
|
with no negative-margin breakout hacks. Only matches on pages that opt
|
||||||
in by adding the wrapper (index.html); FL0W / ST4TS
|
in by adding the wrapper (index.html); FL0W / ST4TS
|
||||||
are full-bleed with their own `.<page>-main` padding. */
|
are full-bleed with their own `.<page>-main` padding. */
|
||||||
|
|
@ -573,14 +573,50 @@ hive-side-panel .md td {
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ─── sticky top region ───────────────────────────────────────────────
|
/* ─── sticky top region ───────────────────────────────────────────────
|
||||||
One sticky container per page holding the page chrome (tab bar / page
|
One sticky container per page holding the warning banner above the
|
||||||
header). common.js (initStickyTop) wraps the chrome in this on load. */
|
page chrome (tab bar / page header). common.js (ensureStickyTop)
|
||||||
|
wraps the page's chrome in this on first render and injects the banner
|
||||||
|
as its first child, so the banner + chrome stack in a single sticky
|
||||||
|
context — pinned together — instead of two separate `top:0` stickies
|
||||||
|
colliding (the banner used to overlay the tab bar). */
|
||||||
.sticky-top {
|
.sticky-top {
|
||||||
position: sticky;
|
position: sticky;
|
||||||
top: 0;
|
top: 0;
|
||||||
z-index: 50;
|
z-index: 50;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ─── server warnings banner ──────────────────────────────────────────
|
||||||
|
Generic top-of-page strip, injected as the first child of `.sticky-top`
|
||||||
|
by common.js (renderServerWarnings) on every page. One row per warning;
|
||||||
|
colour comes from the per-warning `level` (warn = amber, crit = red).
|
||||||
|
Stickiness lives on the `.sticky-top` wrapper, not here. The bar spans
|
||||||
|
the full width edge-to-edge; the dashboard is full-bleed at the
|
||||||
|
body level (their 1.5em gutter lives on an inner `.page-content`
|
||||||
|
wrapper), so the banner is full-width for free. */
|
||||||
|
.server-warnings[hidden] {
|
||||||
|
display: none;
|
||||||
|
}
|
||||||
|
.server-warn {
|
||||||
|
text-align: center;
|
||||||
|
padding: 0.4em 1em;
|
||||||
|
font-size: 0.85em;
|
||||||
|
font-weight: bold;
|
||||||
|
letter-spacing: 0.02em;
|
||||||
|
}
|
||||||
|
/* Tint composited over the opaque elevated-surface grey (not transparent):
|
||||||
|
the bar is `position: sticky`, so a see-through background would let
|
||||||
|
page content scroll through behind it. */
|
||||||
|
.server-warn-warn {
|
||||||
|
color: var(--amber);
|
||||||
|
background: color-mix(in srgb, var(--amber) 16%, var(--bg-elev));
|
||||||
|
border-bottom: 1px solid var(--amber);
|
||||||
|
}
|
||||||
|
.server-warn-crit {
|
||||||
|
color: var(--red);
|
||||||
|
background: color-mix(in srgb, var(--red) 18%, var(--bg-elev));
|
||||||
|
border-bottom: 1px solid var(--red);
|
||||||
|
}
|
||||||
|
|
||||||
/* ─── stats table ─────────────────────────────────────────────────
|
/* ─── stats table ─────────────────────────────────────────────────
|
||||||
Shared right-aligned numeric table. Used by /stats.html (busiest
|
Shared right-aligned numeric table. Used by /stats.html (busiest
|
||||||
agents) — moved here from dashboard.css when ST4TS became its own
|
agents) — moved here from dashboard.css when ST4TS became its own
|
||||||
|
|
|
||||||
|
|
@ -562,12 +562,22 @@ export function appendLinkified(parent, text, refs) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── sticky top region ───────────────────────────────────────────────
|
// ─── server warnings banner ──────────────────────────────────────────
|
||||||
// One sticky container wrapping the page's chrome (tab bar / page
|
// A generic top-of-page banner shown on every page (dashboard + the
|
||||||
// header), built once by wrapping the existing chrome element. A page
|
// stand-alone FL0W / ST4TS pages). The backend decides what to
|
||||||
// without a chrome (none today) gets an empty sticky region at the top of
|
// warn about — `/api/state.server_warnings` is a list of
|
||||||
// <body>.
|
// `{ kind, level, message }` — and this just renders it, coloured by
|
||||||
export function initStickyTop() {
|
// `level` (`warn` amber / `crit` red). Adding a new system warning is a
|
||||||
|
// backend-only change. The bar is injected at the top of <body> so no
|
||||||
|
// page needs to add markup.
|
||||||
|
|
||||||
|
// The sticky top region — one sticky container holding the warning
|
||||||
|
// banner above the page's chrome (tab bar / page header), so the banner
|
||||||
|
// stacks with the chrome instead of being overlaid by it (two separate
|
||||||
|
// `top:0` stickies would otherwise collide). Built once by wrapping the
|
||||||
|
// page's existing chrome element; pages without a chrome (none today)
|
||||||
|
// get a banner-only sticky region at the top of <body>.
|
||||||
|
function ensureStickyTop() {
|
||||||
let top = document.querySelector(".sticky-top");
|
let top = document.querySelector(".sticky-top");
|
||||||
if (top) return top;
|
if (top) return top;
|
||||||
top = document.createElement("div");
|
top = document.createElement("div");
|
||||||
|
|
@ -581,3 +591,51 @@ export function initStickyTop() {
|
||||||
}
|
}
|
||||||
return top;
|
return top;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function ensureServerWarningsBar() {
|
||||||
|
let bar = document.getElementById("server-warnings");
|
||||||
|
if (!bar) {
|
||||||
|
bar = document.createElement("div");
|
||||||
|
bar.id = "server-warnings";
|
||||||
|
bar.className = "server-warnings";
|
||||||
|
bar.setAttribute("role", "alert");
|
||||||
|
bar.hidden = true;
|
||||||
|
ensureStickyTop().prepend(bar);
|
||||||
|
}
|
||||||
|
return bar;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Render a `server_warnings` list (from /api/state) into the banner.
|
||||||
|
/// Empty / missing → the bar hides itself.
|
||||||
|
export function renderServerWarnings(warnings) {
|
||||||
|
const bar = ensureServerWarningsBar();
|
||||||
|
bar.replaceChildren();
|
||||||
|
if (!Array.isArray(warnings) || warnings.length === 0) {
|
||||||
|
bar.hidden = true;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
for (const w of warnings) {
|
||||||
|
const row = el("div", {
|
||||||
|
class:
|
||||||
|
"server-warn server-warn-" +
|
||||||
|
(w && w.level === "crit" ? "crit" : "warn"),
|
||||||
|
});
|
||||||
|
appendText(row, "⚠ " + ((w && w.message) || ""));
|
||||||
|
bar.append(row);
|
||||||
|
}
|
||||||
|
bar.hidden = false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One-shot init for pages that don't otherwise poll /api/state: ensure
|
||||||
|
/// the bar exists, fetch the snapshot once, render. The dashboard (which
|
||||||
|
/// already polls /api/state) calls `renderServerWarnings` directly for
|
||||||
|
/// live updates instead.
|
||||||
|
export function initServerWarnings() {
|
||||||
|
ensureServerWarningsBar();
|
||||||
|
fetch("/api/state")
|
||||||
|
.then((r) => (r.ok ? r.json() : null))
|
||||||
|
.then((s) => renderServerWarnings(s && s.server_warnings))
|
||||||
|
.catch(() => {
|
||||||
|
/* non-fatal: no banner if the snapshot is unreachable */
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -17,7 +17,7 @@
|
||||||
body.dashboard-shell {
|
body.dashboard-shell {
|
||||||
/* Full-bleed body (no horizontal gutter). The 1.5em side gutter lives
|
/* Full-bleed body (no horizontal gutter). The 1.5em side gutter lives
|
||||||
on the inner `.page-content` wrapper (see common.css), so the sticky
|
on the inner `.page-content` wrapper (see common.css), so the sticky
|
||||||
chrome bar and the footer span the full
|
chrome bar, the server-warnings banner, and the footer span the full
|
||||||
width edge-to-edge with no negative-margin breakout. `padding-bottom`
|
width edge-to-edge with no negative-margin breakout. `padding-bottom`
|
||||||
keeps a little breathing room above the viewport edge (overridden to
|
keeps a little breathing room above the viewport edge (overridden to
|
||||||
clear the selection bar via `.has-selection` below). No max-width cap
|
clear the selection bar via `.has-selection` below). No max-width cap
|
||||||
|
|
@ -34,7 +34,8 @@ body.dashboard-shell {
|
||||||
align-items: center;
|
align-items: center;
|
||||||
gap: 0.75em;
|
gap: 0.75em;
|
||||||
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that
|
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that
|
||||||
common.js wraps this chrome in. */
|
common.js wraps this chrome in, alongside the server-warning banner,
|
||||||
|
so they stack in one sticky context instead of colliding. */
|
||||||
background: color-mix(in srgb, var(--bg) 86%, transparent);
|
background: color-mix(in srgb, var(--bg) 86%, transparent);
|
||||||
-webkit-backdrop-filter: blur(8px) saturate(120%);
|
-webkit-backdrop-filter: blur(8px) saturate(120%);
|
||||||
backdrop-filter: blur(8px) saturate(120%);
|
backdrop-filter: blur(8px) saturate(120%);
|
||||||
|
|
|
||||||
|
|
@ -10,12 +10,17 @@
|
||||||
// container list).
|
// container list).
|
||||||
|
|
||||||
import { create as termCreate } from "@hive/shared/terminal.js";
|
import { create as termCreate } from "@hive/shared/terminal.js";
|
||||||
import { $, appendLinkified, openStream, initStickyTop } from "./common.js";
|
import {
|
||||||
|
$,
|
||||||
|
appendLinkified,
|
||||||
|
openStream,
|
||||||
|
initServerWarnings,
|
||||||
|
} from "./common.js";
|
||||||
import { el } from "@hive/shared/dom.js";
|
import { el } from "@hive/shared/dom.js";
|
||||||
import { epochSec } from "./util.js";
|
import { epochSec } from "./util.js";
|
||||||
|
|
||||||
(() => {
|
(() => {
|
||||||
initStickyTop();
|
initServerWarnings();
|
||||||
|
|
||||||
// ─── local containers cache (for compose autocomplete) ──────────────────
|
// ─── local containers cache (for compose autocomplete) ──────────────────
|
||||||
// The compose box's @-mention completion suggests known agent names.
|
// The compose box's @-mention completion suggests known agent names.
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,7 @@
|
||||||
// this bundle has no chart lib; per-agent trend charts live on each
|
// this bundle has no chart lib; per-agent trend charts live on each
|
||||||
// agent's own /stats page. The window selector is a hash-routed
|
// agent's own /stats page. The window selector is a hash-routed
|
||||||
// createTabStrip (#1h / #24h / …), matching the per-agent stats page.
|
// createTabStrip (#1h / #24h / …), matching the per-agent stats page.
|
||||||
import { $, initStickyTop } from "./common.js";
|
import { $, initServerWarnings } from "./common.js";
|
||||||
import { createTabStrip } from "@hive/shared/tabs.js";
|
import { createTabStrip } from "@hive/shared/tabs.js";
|
||||||
|
|
||||||
let hiveStatsWindow = "24h";
|
let hiveStatsWindow = "24h";
|
||||||
|
|
@ -173,7 +173,7 @@ async function refreshHiveStats() {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
initStickyTop();
|
initServerWarnings();
|
||||||
|
|
||||||
// Window selector → hash-routed createTabStrip (#1h / #24h / …). No panes
|
// Window selector → hash-routed createTabStrip (#1h / #24h / …). No panes
|
||||||
// here (the selector swaps data, not DOM); paneFor() returns null for each
|
// here (the selector swaps data, not DOM); paneFor() returns null for each
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@
|
||||||
// 4G3NTS (containers) domain lives in `./agents.js`.
|
// 4G3NTS (containers) domain lives in `./agents.js`.
|
||||||
|
|
||||||
import { marked } from "marked";
|
import { marked } from "marked";
|
||||||
import { $, openStream, initStickyTop } from "./common.js";
|
import { $, openStream, renderServerWarnings } from "./common.js";
|
||||||
import { el } from "@hive/shared/dom.js";
|
import { el } from "@hive/shared/dom.js";
|
||||||
import { bindAsyncForms } from "@hive/shared/forms.js";
|
import { bindAsyncForms } from "@hive/shared/forms.js";
|
||||||
import { createTabStrip } from "@hive/shared/tabs.js";
|
import { createTabStrip } from "@hive/shared/tabs.js";
|
||||||
|
|
@ -50,8 +50,6 @@ import { renderLinksMenu } from "./links-menu.js";
|
||||||
window.marked = marked;
|
window.marked = marked;
|
||||||
|
|
||||||
(() => {
|
(() => {
|
||||||
initStickyTop();
|
|
||||||
|
|
||||||
// ─── async forms ────────────────────────────────────────────────────────
|
// ─── async forms ────────────────────────────────────────────────────────
|
||||||
// Shared `data-async` submit interceptor (in common.js, shared with the other
|
// Shared `data-async` submit interceptor (in common.js, shared with the other
|
||||||
// pages' bundles). On success it re-runs refreshState unless the form
|
// pages' bundles). On success it re-runs refreshState unless the form
|
||||||
|
|
@ -137,6 +135,7 @@ window.marked = marked;
|
||||||
// synchronous read (e.g. the compose autocomplete pulls agent
|
// synchronous read (e.g. the compose autocomplete pulls agent
|
||||||
// names from here instead of refetching on every keystroke).
|
// names from here instead of refetching on every keystroke).
|
||||||
window.__hyperhive_state = s;
|
window.__hyperhive_state = s;
|
||||||
|
renderServerWarnings(s.server_warnings);
|
||||||
// Hive identity: render the swarm/hive name as a headline at the
|
// Hive identity: render the swarm/hive name as a headline at the
|
||||||
// top of the 4G3NTS pane + update the page title once the
|
// top of the 4G3NTS pane + update the page title once the
|
||||||
// server-side display names are known. `hive_name` / `swarm_name`
|
// server-side display names are known. `hive_name` / `swarm_name`
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,8 @@
|
||||||
*/
|
*/
|
||||||
.page-header {
|
.page-header {
|
||||||
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that
|
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that
|
||||||
common.js wraps this header in. */
|
common.js wraps this header in, alongside the server-warning banner,
|
||||||
|
so the two stack instead of colliding. */
|
||||||
background: color-mix(in srgb, var(--bg) 92%, transparent);
|
background: color-mix(in srgb, var(--bg) 92%, transparent);
|
||||||
-webkit-backdrop-filter: blur(8px) saturate(120%);
|
-webkit-backdrop-filter: blur(8px) saturate(120%);
|
||||||
backdrop-filter: blur(8px) saturate(120%);
|
backdrop-filter: blur(8px) saturate(120%);
|
||||||
|
|
|
||||||
|
|
@ -599,7 +599,6 @@ impl Coordinator {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
crate::host_stats::refresh_pending_logins(&fresh);
|
|
||||||
let mut last = self.last_containers.lock().await;
|
let mut last = self.last_containers.lock().await;
|
||||||
let mut changed_or_new = Vec::new();
|
let mut changed_or_new = Vec::new();
|
||||||
let mut removed = Vec::new();
|
let mut removed = Vec::new();
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,9 @@
|
||||||
//! Hive-wide health endpoints for external monitoring: `/health/live`
|
//! Hive-wide health endpoints for external monitoring: `/health/live`
|
||||||
//! (liveness) and `/health/ready` (readiness).
|
//! (liveness) and `/health/ready` (readiness).
|
||||||
//!
|
//!
|
||||||
//! Deliberately thin: reuses [`crate::warnings`] as the single source of truth
|
//! Deliberately thin: reuses [`crate::warnings`] — the same registry the
|
||||||
//! for "is anything degraded", rather than tracking health separately. Two
|
//! dashboard banner already reads — as the single source of truth for
|
||||||
|
//! "is anything degraded", rather than tracking health separately. Two
|
||||||
//! systems independently deciding what counts as unhealthy is how they
|
//! systems independently deciding what counts as unhealthy is how they
|
||||||
//! end up disagreeing.
|
//! end up disagreeing.
|
||||||
//!
|
//!
|
||||||
|
|
@ -11,7 +12,7 @@
|
||||||
//! the *process* down over a transient dependency failure — that's what
|
//! the *process* down over a transient dependency failure — that's what
|
||||||
//! readiness is for. Readiness derives `"degraded"` from any `crit`-level
|
//! readiness is for. Readiness derives `"degraded"` from any `crit`-level
|
||||||
//! warning currently set; `warn`-level entries ride along in the body for
|
//! warning currently set; `warn`-level entries ride along in the body for
|
||||||
//! detail but don't flip the status, mirroring the warning registry's own severity
|
//! detail but don't flip the status, mirroring the banner's own severity
|
||||||
//! split.
|
//! split.
|
||||||
//!
|
//!
|
||||||
//! Routed outside `/api/` (see `dashboard/mod.rs`) so the gateway can
|
//! Routed outside `/api/` (see `dashboard/mod.rs`) so the gateway can
|
||||||
|
|
|
||||||
|
|
@ -102,6 +102,12 @@ pub(super) struct StateSnapshot {
|
||||||
/// var, set from `services.hyperhive.swarm.name`. `None` when
|
/// var, set from `services.hyperhive.swarm.name`. `None` when
|
||||||
/// unset — chrome omits the swarm segment of the breadcrumb.
|
/// unset — chrome omits the swarm segment of the breadcrumb.
|
||||||
swarm_name: Option<String>,
|
swarm_name: Option<String>,
|
||||||
|
/// Server-level warnings for the dashboard's top-of-page banner
|
||||||
|
/// (currently host disk-pressure; more producers can be added
|
||||||
|
/// backend-side). Empty when all clear. Built by
|
||||||
|
/// `host_stats::server_warnings`; the frontend renders this list
|
||||||
|
/// generically, so new warning kinds need no frontend change.
|
||||||
|
server_warnings: Vec<crate::host_stats::ServerWarning>,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Serialize)]
|
#[derive(Serialize)]
|
||||||
|
|
@ -187,6 +193,15 @@ pub(super) async fn api_state(
|
||||||
let tombstones = build_tombstone_views(&containers);
|
let tombstones = build_tombstone_views(&containers);
|
||||||
let port_conflicts = build_port_conflicts(&containers);
|
let port_conflicts = build_port_conflicts(&containers);
|
||||||
|
|
||||||
|
// Banner warnings: host probes (disk) + agent-state (pending logins).
|
||||||
|
// Built before the response struct because the agent-state producer
|
||||||
|
// borrows `containers`, which moves in below.
|
||||||
|
let server_warnings = {
|
||||||
|
let mut w = crate::host_stats::server_warnings();
|
||||||
|
w.extend(crate::host_stats::agent_state_warnings(&containers));
|
||||||
|
w
|
||||||
|
};
|
||||||
|
|
||||||
axum::Json(StateSnapshot {
|
axum::Json(StateSnapshot {
|
||||||
seq,
|
seq,
|
||||||
hostname,
|
hostname,
|
||||||
|
|
@ -218,6 +233,7 @@ pub(super) async fn api_state(
|
||||||
swarm_name: std::env::var("HYPERHIVE_SWARM_NAME")
|
swarm_name: std::env::var("HYPERHIVE_SWARM_NAME")
|
||||||
.ok()
|
.ok()
|
||||||
.filter(|s| !s.is_empty()),
|
.filter(|s| !s.is_empty()),
|
||||||
|
server_warnings,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -122,7 +122,7 @@ pub(crate) const CONFIG_ORG: &str = "agent-configs";
|
||||||
/// set of boot warnings in [`ensure_all`] keyed by a runtime name (at
|
/// set of boot warnings in [`ensure_all`] keyed by a runtime name (at
|
||||||
/// most a handful per process). [`crate::warnings::set_boot_warning`]
|
/// most a handful per process). [`crate::warnings::set_boot_warning`]
|
||||||
/// requires a `'static` kind so distinct orgs/repos don't clobber each
|
/// requires a `'static` kind so distinct orgs/repos don't clobber each
|
||||||
/// other's warning entry; leaking a few short strings once per boot is
|
/// other's banner entry; leaking a few short strings once per boot is
|
||||||
/// cheap and bounded, unlike a per-request or per-loop-iteration leak.
|
/// cheap and bounded, unlike a per-request or per-loop-iteration leak.
|
||||||
fn static_kind(s: String) -> &'static str {
|
fn static_kind(s: String) -> &'static str {
|
||||||
Box::leak(s.into_boxed_str())
|
Box::leak(s.into_boxed_str())
|
||||||
|
|
@ -340,7 +340,7 @@ fn hook_ids_at(hooks: &[forgejo_api::structs::Hook], url: &str) -> Vec<i64> {
|
||||||
/// confirms the container exists — not that Forgejo is *listening*. A
|
/// confirms the container exists — not that Forgejo is *listening*. A
|
||||||
/// `nixos-rebuild` that restarts hive-forge and hive-c0re together races:
|
/// `nixos-rebuild` that restarts hive-forge and hive-c0re together races:
|
||||||
/// without this gate every `ensure_*` fires at a refused socket and leaves a
|
/// without this gate every `ensure_*` fires at a refused socket and leaves a
|
||||||
/// stale "provisioning failed" warning that never clears (the pass is one-shot).
|
/// stale "provisioning failed" banner that never clears (the pass is one-shot).
|
||||||
/// Returns whether the endpoint became ready before the deadline; on timeout
|
/// Returns whether the endpoint became ready before the deadline; on timeout
|
||||||
/// the caller proceeds anyway so a genuinely-down forge still surfaces its real
|
/// the caller proceeds anyway so a genuinely-down forge still surfaces its real
|
||||||
/// errors. Uses the unauthenticated `get_version` endpoint (no token yet).
|
/// errors. Uses the unauthenticated `get_version` endpoint (no token yet).
|
||||||
|
|
@ -393,7 +393,7 @@ pub async fn ensure_all() {
|
||||||
}
|
}
|
||||||
// The container exists, but its HTTP may not be up yet (a rebuild restarts
|
// The container exists, but its HTTP may not be up yet (a rebuild restarts
|
||||||
// hive-forge + hive-c0re together). Wait for the API to answer before the
|
// hive-forge + hive-c0re together). Wait for the API to answer before the
|
||||||
// provisioning pass so a boot race doesn't leave stale failure warnings.
|
// provisioning pass so a boot race doesn't leave stale failure banners.
|
||||||
wait_until_ready().await;
|
wait_until_ready().await;
|
||||||
let core_token = match ensure_core_user_and_token().await {
|
let core_token = match ensure_core_user_and_token().await {
|
||||||
Ok(t) => Some(t),
|
Ok(t) => Some(t),
|
||||||
|
|
|
||||||
|
|
@ -125,7 +125,7 @@ pub async fn push_meta(dir: &Path) -> Result<()> {
|
||||||
let msg = "forge/core/meta push rejected (non-fast-forward): \
|
let msg = "forge/core/meta push rejected (non-fast-forward): \
|
||||||
remote is ahead of local; leaving remote history intact";
|
remote is ahead of local; leaving remote history intact";
|
||||||
tracing::warn!("{msg}");
|
tracing::warn!("{msg}");
|
||||||
// Raise a persistent hive warning; cleared on next successful push.
|
// Raise a persistent dashboard banner; cleared on next successful push.
|
||||||
if let Ok(mut g) = META_NON_FF_GUARD.lock() {
|
if let Ok(mut g) = META_NON_FF_GUARD.lock() {
|
||||||
*g = Some(crate::warnings::set_warning(
|
*g = Some(crate::warnings::set_warning(
|
||||||
"forge-meta-non-ff",
|
"forge-meta-non-ff",
|
||||||
|
|
|
||||||
|
|
@ -32,7 +32,7 @@ static LAST_FAILED_RELOAD: AtomicU64 = AtomicU64::new(0);
|
||||||
/// Minimum gap between retry attempts after a reload failure (30 s).
|
/// Minimum gap between retry attempts after a reload failure (30 s).
|
||||||
const RELOAD_RETRY_SECS: u64 = 30;
|
const RELOAD_RETRY_SECS: u64 = 30;
|
||||||
|
|
||||||
/// Hive-warning health tracker for the reload sweep — raises a
|
/// Dashboard-banner health tracker for the reload sweep — raises a
|
||||||
/// `gateway_nginx_reload` warning the first time a reload fails (routing is
|
/// `gateway_nginx_reload` warning the first time a reload fails (routing is
|
||||||
/// broken for whoever depends on the change *right now*, so no debounce
|
/// broken for whoever depends on the change *right now*, so no debounce
|
||||||
/// window) and clears it the moment a reload succeeds again.
|
/// window) and clears it the moment a reload succeeds again.
|
||||||
|
|
|
||||||
|
|
@ -139,7 +139,7 @@ pub(super) async fn run_node(
|
||||||
|
|
||||||
/// Boot-time forge user/token sweep as a DAG node — see
|
/// Boot-time forge user/token sweep as a DAG node — see
|
||||||
/// [`NodeKind::ForgeSweep`]. `forge::ensure_all` already does its own
|
/// [`NodeKind::ForgeSweep`]. `forge::ensure_all` already does its own
|
||||||
/// per-step error handling and boot warnings internally (it's
|
/// per-step error handling and boot-warning banners internally (it's
|
||||||
/// best-effort by design), so this wrapper has nothing left to report;
|
/// best-effort by design), so this wrapper has nothing left to report;
|
||||||
/// it exists purely to make the sweep a visible unit of work.
|
/// it exists purely to make the sweep a visible unit of work.
|
||||||
async fn run_forge_sweep() -> Result<()> {
|
async fn run_forge_sweep() -> Result<()> {
|
||||||
|
|
@ -149,7 +149,7 @@ async fn run_forge_sweep() -> Result<()> {
|
||||||
|
|
||||||
/// Matrix user/space sweep as a DAG node — see [`NodeKind::MatrixSweep`].
|
/// Matrix user/space sweep as a DAG node — see [`NodeKind::MatrixSweep`].
|
||||||
/// Every pass reports here, whoever submitted it: as the node's own error on
|
/// Every pass reports here, whoever submitted it: as the node's own error on
|
||||||
/// the dashboard, and into the debounced warning.
|
/// the dashboard, and into the debounced banner.
|
||||||
async fn run_matrix_sweep() -> Result<()> {
|
async fn run_matrix_sweep() -> Result<()> {
|
||||||
let ok = tokio::time::timeout(MATRIX_SWEEP_DEADLINE, crate::matrix::ensure_all())
|
let ok = tokio::time::timeout(MATRIX_SWEEP_DEADLINE, crate::matrix::ensure_all())
|
||||||
.await
|
.await
|
||||||
|
|
@ -164,7 +164,7 @@ async fn run_matrix_sweep() -> Result<()> {
|
||||||
health.record_ok();
|
health.record_ok();
|
||||||
Ok(())
|
Ok(())
|
||||||
} else {
|
} else {
|
||||||
health.record_err(matrix_sweep_warning);
|
health.record_err(matrix_sweep_banner);
|
||||||
anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)")
|
anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -177,7 +177,7 @@ async fn run_matrix_sweep() -> Result<()> {
|
||||||
/// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep
|
/// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep
|
||||||
const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10);
|
const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10);
|
||||||
|
|
||||||
/// Warning for a matrix sweep that keeps failing. Two misses in a row, so one
|
/// Banner for a matrix sweep that keeps failing. Two misses in a row, so one
|
||||||
/// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the
|
/// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the
|
||||||
/// dashboard; cleared by the next clean pass.
|
/// dashboard; cleared by the next clean pass.
|
||||||
fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
|
fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
|
||||||
|
|
@ -185,7 +185,7 @@ fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
|
||||||
HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2)))
|
HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2)))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn matrix_sweep_warning(ctx: sweep_health::SweepFailure) -> String {
|
fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String {
|
||||||
let age = ctx.since_last_ok.map_or_else(
|
let age = ctx.since_last_ok.map_or_else(
|
||||||
|| "no success this session".to_owned(),
|
|| "no success this session".to_owned(),
|
||||||
|d| format!("last ok {} ago", sweep_health::fmt_age(d)),
|
|d| format!("last ok {} ago", sweep_health::fmt_age(d)),
|
||||||
|
|
@ -200,7 +200,7 @@ fn matrix_sweep_warning(ctx: sweep_health::SweepFailure) -> String {
|
||||||
|
|
||||||
/// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every
|
/// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every
|
||||||
/// pass reports here, whoever submitted it: as the node's own error on the
|
/// pass reports here, whoever submitted it: as the node's own error on the
|
||||||
/// dashboard, and into the debounced warning.
|
/// dashboard, and into the debounced banner.
|
||||||
async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
|
async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
|
||||||
let result = tokio::time::timeout(
|
let result = tokio::time::timeout(
|
||||||
KNOWLEDGE_PULL_DEADLINE,
|
KNOWLEDGE_PULL_DEADLINE,
|
||||||
|
|
@ -242,7 +242,7 @@ async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
|
||||||
/// [`Resource::KnowledgeTree`]: super::resource::Resource::KnowledgeTree
|
/// [`Resource::KnowledgeTree`]: super::resource::Resource::KnowledgeTree
|
||||||
const KNOWLEDGE_PULL_DEADLINE: std::time::Duration = std::time::Duration::from_mins(5);
|
const KNOWLEDGE_PULL_DEADLINE: std::time::Duration = std::time::Duration::from_mins(5);
|
||||||
|
|
||||||
/// Warning for a `/knowledge` pull that keeps failing: three misses in a row,
|
/// Banner for a `/knowledge` pull that keeps failing: three misses in a row,
|
||||||
/// cleared by the next successful pull.
|
/// cleared by the next successful pull.
|
||||||
fn knowledge_pull_health() -> &'static Mutex<SweepHealth> {
|
fn knowledge_pull_health() -> &'static Mutex<SweepHealth> {
|
||||||
static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new();
|
static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new();
|
||||||
|
|
|
||||||
|
|
@ -367,11 +367,11 @@ pub async fn agents_for_meta_listing() -> Result<Vec<crate::meta::AgentSpec>> {
|
||||||
agents_for_meta(None).await
|
agents_for_meta(None).await
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Per-agent "stop failed" warning guards, keyed by agent name. Cleared the
|
/// Per-agent "stop failed" banner guards, keyed by agent name. Cleared the
|
||||||
/// next time that agent's container stops cleanly — a `Mutex<HashMap<...>>`
|
/// next time that agent's container stops cleanly — a `Mutex<HashMap<...>>`
|
||||||
/// rather than a bare `set_boot_warning` because this condition (unlike a
|
/// rather than a bare `set_boot_warning` because this condition (unlike a
|
||||||
/// one-shot boot step) genuinely resolves later, once an operator has
|
/// one-shot boot step) genuinely resolves later, once an operator has
|
||||||
/// intervened, and the warning should clear with it instead of surviving
|
/// intervened, and the banner should clear with it instead of surviving
|
||||||
/// until the next hive-c0re restart.
|
/// until the next hive-c0re restart.
|
||||||
fn stop_failed_guards()
|
fn stop_failed_guards()
|
||||||
-> &'static std::sync::Mutex<std::collections::HashMap<String, crate::warnings::WarningGuard>> {
|
-> &'static std::sync::Mutex<std::collections::HashMap<String, crate::warnings::WarningGuard>> {
|
||||||
|
|
@ -402,7 +402,7 @@ pub async fn kill(name: &str) -> Result<()> {
|
||||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||||
match result {
|
match result {
|
||||||
Ok(()) => {
|
Ok(()) => {
|
||||||
// A clean stop clears any earlier "this looked wedged" warning
|
// A clean stop clears any earlier "this looked wedged" banner
|
||||||
// for the same agent.
|
// for the same agent.
|
||||||
guards.remove(name);
|
guards.remove(name);
|
||||||
Ok(())
|
Ok(())
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue