Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

..
37 changed files with 325 additions and 478 deletions

View file

@ -66,7 +66,7 @@ pull`, so agents see the new content on their next turn.
queues a pull on a fixed cadence as a fallback (webhook missed, c0re queues a pull on a fixed cadence as a fallback (webhook missed, c0re
restarted between pushes). The pull is best-effort — a failure restarted between pushes). The pull is best-effort — a failure
shows as a failed `KnowledgePull` node on the job queue, counts toward shows as a failed `KnowledgePull` node on the job queue, counts toward
the `knowledge_pull` hive warning, and doesn't affect the rest of the daemon. the knowledge-pull banner, and doesn't affect the rest of the daemon.
Both paths, and the one pull at boot, queue the same `KnowledgePull` job Both paths, and the one pull at boot, queue the same `KnowledgePull` job
node. It holds the working tree for its duration, so two pulls never run node. It holds the working tree for its duration, so two pulls never run

View file

@ -529,8 +529,8 @@ given.
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
value passes the validators first. One bad value passes the validators first. One bad
value or a failed read leaves the file unchanged, and repeated failures raise a hive value or a failed read leaves the file unchanged, and repeated failures raise a
warning (the `hyperhive.hive.warning` metric and the `hh-hive-warning` alert). dashboard banner.
A hive's queue grant reads every key of the bucket A hive's queue grant reads every key of the bucket
(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the (`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the

View file

@ -444,7 +444,7 @@ whole-swarm spend figure.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
hive-c0re measures everything above **per agent**, tagged with the hive it hive-c0re measures everything above **per agent**, tagged with the hive it
runs in. It measures these five per **hive** instead, and they carry no runs in. It measures these three per **hive** instead, and they carry no
`agent` label — so a hive `agent` label — so a hive
that hosts no agents still reports, and "this hive is quiet" is that hosts no agents still reports, and "this hive is quiet" is
distinguishable from "this hive is gone." Select them with distinguishable from "this hive is gone." Select them with
@ -452,41 +452,20 @@ distinguishable from "this hive is gone." Select them with
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Meaning | | Metric | Unit | Kind | Meaning |
| -------------------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- | | ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 | | `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows | | `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) | | `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0` hive-c0re reports both levels every cycle, `0` included, so a healthy hive is
included, so a healthy hive is visible as zeros rather than as missing visible as zeros rather than as missing series.
series.
`hyperhive.hive.degraded` is what a dashboard should alert on: it's `hyperhive.hive.degraded` is what a dashboard should alert on: it's
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready` `hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
and with what the swarm controller sees. `hyperhive.hive.warnings` is the and with what the swarm controller sees. `hyperhive.hive.warnings` is the
detail behind it — `warn`-level entries mean "an operator should look" and do detail behind it — `warn`-level entries mean "an operator should look" and do
**not** set `degraded`. `hyperhive.hive.warning` names the cause: the `kind` **not** set `degraded`.
label is one of a fixed set in the source (for example `disk_pressure`, a
failing sweep such as `swarm_status_publish`, `pending_logins`,
`forge_ensure_core_user`), and the message lives in the
`hive warning raised` line hive-c0re logs when it raises the warning.
The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
| Alert | Fires when |
| ----------------------- | ----------------------------------------------------------------------------------- |
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists for 2 minutes |
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |
The disk alerts replace the old dashboard banner for the nix store, and
`hh-hive-warning` replaces it for every other warning. Disk pressure has its
own pair so the thresholds sit on the metric rather than on the registry
entry.
Same cadence, transport and resource labels as the container metrics above. Same cadence, transport and resource labels as the container metrics above.

View file

@ -148,8 +148,8 @@ swarm-services certificate.
The identity isn't a choice — a hive authenticates as `hive-<hiveName>` The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
and publishes under `hiveName`, the same name that keys `swarm.hives`. and publishes under `hiveName`, the same name that keys `swarm.hives`.
If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a If a hive stops reporting, its own dashboard is the place to look: a
failure to publish raises a hive warning after three consecutive failure to publish raises a warning banner there after three consecutive
misses. It stays `warn` rather than `crit` on purpose — a hive that misses. It stays `warn` rather than `crit` on purpose — a hive that
can't reach the queue isn't itself unhealthy, so it doesn't start can't reach the queue isn't itself unhealthy, so it doesn't start
calling itself degraded for being unable to say it's fine. calling itself degraded for being unable to say it's fine.

View file

@ -31,17 +31,26 @@ dashboard tab strip.
K3PT ST4T3 (kept-state entries). K3PT ST4T3 (kept-state entries).
- **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`) - **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`)
— sits below the tab strip. — sits below the tab strip.
- **No warning banner.** No page shows hive-level warnings (host disk - **Server-warnings banner** — a generic, sticky top-of-page strip shown
pressure, failing background sweeps, forge boot failures, agents that on **every** page (the dashboard + the stand-alone FL0W
need a claude login). They live in a push-based registry and ST4TS pages), injected at the top
(`hive-c0re`'s `warnings.rs`): any subsystem raises or clears its own of `<body>` by `renderServerWarnings` in `common.js`. Driven by
entry through an RAII guard, and each raise logs one `state.server_warnings` — a list of `{ kind, level, message }` — and
`hive warning raised` line. The operator sees the registry as metrics and coloured by `level` (`warn` amber / `crit` red). hive-c0re owns the
Grafana alerts: see threshold + message, so adding a new system warning needs no frontend
[Hive-scoped metrics](../scheduler/observability.md#hive-scoped-metrics-hive-c0re). change. Warnings are a push-based registry (`hive-c0re`'s
A container crash is a swarm-level Grafana alert on the crash watcher's `warnings.rs`): any subsystem raises/clears its own entry via an RAII
own log line, so a hive with no otel export has no operator-visible guard, and `host_stats::server_warnings()` is just a cheap snapshot of
warning signal beyond `/health/ready`. that registry. Producers today include the host disk-pressure check
(a `statvfs` probe of `/nix`: ≥85% used → `warn`, ≥95% → `crit`, e.g.
`⚠ host nix store N% full (G GiB free) — garbage-collect …`), forge
provisioning/CI-runner boot failures, and an agent-state warning
(`pending_logins`) computed alongside the container snapshot. A
container crash used to have a sibling `agents_crashing` entry here —
removed in favour of a swarm-level grafana alert on the crash
watcher's own log line (`docs/scheduler/observability.md`), so a
hive with no otel export now has no operator-visible crash signal at
all. Hidden when there are no warnings.
- **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated - **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated
to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name` to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name`
arrive in the state snapshot. arrive in the state snapshot.

View file

@ -564,7 +564,7 @@ hive-side-panel .md td {
Inverted-gutter layout: the dashboard keeps `<body>` full-bleed Inverted-gutter layout: the dashboard keeps `<body>` full-bleed
and wraps their padded content in a single `.page-content` div that and wraps their padded content in a single `.page-content` div that
carries the horizontal gutter. Full-width chrome (the tab strip, the carries the horizontal gutter. Full-width chrome (the tab strip, the
footer) then spans edge-to-edge for free, server-warnings banner, the footer) then spans edge-to-edge for free,
with no negative-margin breakout hacks. Only matches on pages that opt with no negative-margin breakout hacks. Only matches on pages that opt
in by adding the wrapper (index.html); FL0W / ST4TS in by adding the wrapper (index.html); FL0W / ST4TS
are full-bleed with their own `.<page>-main` padding. */ are full-bleed with their own `.<page>-main` padding. */
@ -573,14 +573,50 @@ hive-side-panel .md td {
} }
/* ─── sticky top region ─────────────────────────────────────────────── /* ─── sticky top region ───────────────────────────────────────────────
One sticky container per page holding the page chrome (tab bar / page One sticky container per page holding the warning banner above the
header). common.js (initStickyTop) wraps the chrome in this on load. */ page chrome (tab bar / page header). common.js (ensureStickyTop)
wraps the page's chrome in this on first render and injects the banner
as its first child, so the banner + chrome stack in a single sticky
context — pinned together — instead of two separate `top:0` stickies
colliding (the banner used to overlay the tab bar). */
.sticky-top { .sticky-top {
position: sticky; position: sticky;
top: 0; top: 0;
z-index: 50; z-index: 50;
} }
/* ─── server warnings banner ──────────────────────────────────────────
Generic top-of-page strip, injected as the first child of `.sticky-top`
by common.js (renderServerWarnings) on every page. One row per warning;
colour comes from the per-warning `level` (warn = amber, crit = red).
Stickiness lives on the `.sticky-top` wrapper, not here. The bar spans
the full width edge-to-edge; the dashboard is full-bleed at the
body level (their 1.5em gutter lives on an inner `.page-content`
wrapper), so the banner is full-width for free. */
.server-warnings[hidden] {
display: none;
}
.server-warn {
text-align: center;
padding: 0.4em 1em;
font-size: 0.85em;
font-weight: bold;
letter-spacing: 0.02em;
}
/* Tint composited over the opaque elevated-surface grey (not transparent):
the bar is `position: sticky`, so a see-through background would let
page content scroll through behind it. */
.server-warn-warn {
color: var(--amber);
background: color-mix(in srgb, var(--amber) 16%, var(--bg-elev));
border-bottom: 1px solid var(--amber);
}
.server-warn-crit {
color: var(--red);
background: color-mix(in srgb, var(--red) 18%, var(--bg-elev));
border-bottom: 1px solid var(--red);
}
/* ─── stats table ───────────────────────────────────────────────── /* ─── stats table ─────────────────────────────────────────────────
Shared right-aligned numeric table. Used by /stats.html (busiest Shared right-aligned numeric table. Used by /stats.html (busiest
agents) — moved here from dashboard.css when ST4TS became its own agents) — moved here from dashboard.css when ST4TS became its own

View file

@ -562,12 +562,22 @@ export function appendLinkified(parent, text, refs) {
} }
} }
// ─── sticky top region ─────────────────────────────────────────────── // ─── server warnings banner ──────────────────────────────────────────
// One sticky container wrapping the page's chrome (tab bar / page // A generic top-of-page banner shown on every page (dashboard + the
// header), built once by wrapping the existing chrome element. A page // stand-alone FL0W / ST4TS pages). The backend decides what to
// without a chrome (none today) gets an empty sticky region at the top of // warn about — `/api/state.server_warnings` is a list of
// <body>. // `{ kind, level, message }` — and this just renders it, coloured by
export function initStickyTop() { // `level` (`warn` amber / `crit` red). Adding a new system warning is a
// backend-only change. The bar is injected at the top of <body> so no
// page needs to add markup.
// The sticky top region — one sticky container holding the warning
// banner above the page's chrome (tab bar / page header), so the banner
// stacks with the chrome instead of being overlaid by it (two separate
// `top:0` stickies would otherwise collide). Built once by wrapping the
// page's existing chrome element; pages without a chrome (none today)
// get a banner-only sticky region at the top of <body>.
function ensureStickyTop() {
let top = document.querySelector(".sticky-top"); let top = document.querySelector(".sticky-top");
if (top) return top; if (top) return top;
top = document.createElement("div"); top = document.createElement("div");
@ -581,3 +591,51 @@ export function initStickyTop() {
} }
return top; return top;
} }
function ensureServerWarningsBar() {
let bar = document.getElementById("server-warnings");
if (!bar) {
bar = document.createElement("div");
bar.id = "server-warnings";
bar.className = "server-warnings";
bar.setAttribute("role", "alert");
bar.hidden = true;
ensureStickyTop().prepend(bar);
}
return bar;
}
/// Render a `server_warnings` list (from /api/state) into the banner.
/// Empty / missing → the bar hides itself.
export function renderServerWarnings(warnings) {
const bar = ensureServerWarningsBar();
bar.replaceChildren();
if (!Array.isArray(warnings) || warnings.length === 0) {
bar.hidden = true;
return;
}
for (const w of warnings) {
const row = el("div", {
class:
"server-warn server-warn-" +
(w && w.level === "crit" ? "crit" : "warn"),
});
appendText(row, "⚠ " + ((w && w.message) || ""));
bar.append(row);
}
bar.hidden = false;
}
/// One-shot init for pages that don't otherwise poll /api/state: ensure
/// the bar exists, fetch the snapshot once, render. The dashboard (which
/// already polls /api/state) calls `renderServerWarnings` directly for
/// live updates instead.
export function initServerWarnings() {
ensureServerWarningsBar();
fetch("/api/state")
.then((r) => (r.ok ? r.json() : null))
.then((s) => renderServerWarnings(s && s.server_warnings))
.catch(() => {
/* non-fatal: no banner if the snapshot is unreachable */
});
}

View file

@ -17,7 +17,7 @@
body.dashboard-shell { body.dashboard-shell {
/* Full-bleed body (no horizontal gutter). The 1.5em side gutter lives /* Full-bleed body (no horizontal gutter). The 1.5em side gutter lives
on the inner `.page-content` wrapper (see common.css), so the sticky on the inner `.page-content` wrapper (see common.css), so the sticky
chrome bar and the footer span the full chrome bar, the server-warnings banner, and the footer span the full
width edge-to-edge with no negative-margin breakout. `padding-bottom` width edge-to-edge with no negative-margin breakout. `padding-bottom`
keeps a little breathing room above the viewport edge (overridden to keeps a little breathing room above the viewport edge (overridden to
clear the selection bar via `.has-selection` below). No max-width cap clear the selection bar via `.has-selection` below). No max-width cap
@ -34,7 +34,8 @@ body.dashboard-shell {
align-items: center; align-items: center;
gap: 0.75em; gap: 0.75em;
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that /* Stickiness lives on the `.sticky-top` wrapper (common.css) that
common.js wraps this chrome in. */ common.js wraps this chrome in, alongside the server-warning banner,
so they stack in one sticky context instead of colliding. */
background: color-mix(in srgb, var(--bg) 86%, transparent); background: color-mix(in srgb, var(--bg) 86%, transparent);
-webkit-backdrop-filter: blur(8px) saturate(120%); -webkit-backdrop-filter: blur(8px) saturate(120%);
backdrop-filter: blur(8px) saturate(120%); backdrop-filter: blur(8px) saturate(120%);

View file

@ -10,12 +10,17 @@
// container list). // container list).
import { create as termCreate } from "@hive/shared/terminal.js"; import { create as termCreate } from "@hive/shared/terminal.js";
import { $, appendLinkified, openStream, initStickyTop } from "./common.js"; import {
$,
appendLinkified,
openStream,
initServerWarnings,
} from "./common.js";
import { el } from "@hive/shared/dom.js"; import { el } from "@hive/shared/dom.js";
import { epochSec } from "./util.js"; import { epochSec } from "./util.js";
(() => { (() => {
initStickyTop(); initServerWarnings();
// ─── local containers cache (for compose autocomplete) ────────────────── // ─── local containers cache (for compose autocomplete) ──────────────────
// The compose box's @-mention completion suggests known agent names. // The compose box's @-mention completion suggests known agent names.

View file

@ -5,7 +5,7 @@
// this bundle has no chart lib; per-agent trend charts live on each // this bundle has no chart lib; per-agent trend charts live on each
// agent's own /stats page. The window selector is a hash-routed // agent's own /stats page. The window selector is a hash-routed
// createTabStrip (#1h / #24h / …), matching the per-agent stats page. // createTabStrip (#1h / #24h / …), matching the per-agent stats page.
import { $, initStickyTop } from "./common.js"; import { $, initServerWarnings } from "./common.js";
import { createTabStrip } from "@hive/shared/tabs.js"; import { createTabStrip } from "@hive/shared/tabs.js";
let hiveStatsWindow = "24h"; let hiveStatsWindow = "24h";
@ -173,7 +173,7 @@ async function refreshHiveStats() {
} }
} }
initStickyTop(); initServerWarnings();
// Window selector → hash-routed createTabStrip (#1h / #24h / …). No panes // Window selector → hash-routed createTabStrip (#1h / #24h / …). No panes
// here (the selector swaps data, not DOM); paneFor() returns null for each // here (the selector swaps data, not DOM); paneFor() returns null for each

View file

@ -11,7 +11,7 @@
// 4G3NTS (containers) domain lives in `./agents.js`. // 4G3NTS (containers) domain lives in `./agents.js`.
import { marked } from "marked"; import { marked } from "marked";
import { $, openStream, initStickyTop } from "./common.js"; import { $, openStream, renderServerWarnings } from "./common.js";
import { el } from "@hive/shared/dom.js"; import { el } from "@hive/shared/dom.js";
import { bindAsyncForms } from "@hive/shared/forms.js"; import { bindAsyncForms } from "@hive/shared/forms.js";
import { createTabStrip } from "@hive/shared/tabs.js"; import { createTabStrip } from "@hive/shared/tabs.js";
@ -50,8 +50,6 @@ import { renderLinksMenu } from "./links-menu.js";
window.marked = marked; window.marked = marked;
(() => { (() => {
initStickyTop();
// ─── async forms ──────────────────────────────────────────────────────── // ─── async forms ────────────────────────────────────────────────────────
// Shared `data-async` submit interceptor (in common.js, shared with the other // Shared `data-async` submit interceptor (in common.js, shared with the other
// pages' bundles). On success it re-runs refreshState unless the form // pages' bundles). On success it re-runs refreshState unless the form
@ -137,6 +135,7 @@ window.marked = marked;
// synchronous read (e.g. the compose autocomplete pulls agent // synchronous read (e.g. the compose autocomplete pulls agent
// names from here instead of refetching on every keystroke). // names from here instead of refetching on every keystroke).
window.__hyperhive_state = s; window.__hyperhive_state = s;
renderServerWarnings(s.server_warnings);
// Hive identity: render the swarm/hive name as a headline at the // Hive identity: render the swarm/hive name as a headline at the
// top of the 4G3NTS pane + update the page title once the // top of the 4G3NTS pane + update the page title once the
// server-side display names are known. `hive_name` / `swarm_name` // server-side display names are known. `hive_name` / `swarm_name`

View file

@ -16,7 +16,8 @@
*/ */
.page-header { .page-header {
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that /* Stickiness lives on the `.sticky-top` wrapper (common.css) that
common.js wraps this header in. */ common.js wraps this header in, alongside the server-warning banner,
so the two stack instead of colliding. */
background: color-mix(in srgb, var(--bg) 92%, transparent); background: color-mix(in srgb, var(--bg) 92%, transparent);
-webkit-backdrop-filter: blur(8px) saturate(120%); -webkit-backdrop-filter: blur(8px) saturate(120%);
backdrop-filter: blur(8px) saturate(120%); backdrop-filter: blur(8px) saturate(120%);

View file

@ -599,7 +599,6 @@ impl Coordinator {
return; return;
} }
}; };
crate::host_stats::refresh_pending_logins(&fresh);
let mut last = self.last_containers.lock().await; let mut last = self.last_containers.lock().await;
let mut changed_or_new = Vec::new(); let mut changed_or_new = Vec::new();
let mut removed = Vec::new(); let mut removed = Vec::new();

View file

@ -1,8 +1,9 @@
//! Hive-wide health endpoints for external monitoring: `/health/live` //! Hive-wide health endpoints for external monitoring: `/health/live`
//! (liveness) and `/health/ready` (readiness). //! (liveness) and `/health/ready` (readiness).
//! //!
//! Deliberately thin: reuses [`crate::warnings`] as the single source of truth //! Deliberately thin: reuses [`crate::warnings`] — the same registry the
//! for "is anything degraded", rather than tracking health separately. Two //! dashboard banner already reads — as the single source of truth for
//! "is anything degraded", rather than tracking health separately. Two
//! systems independently deciding what counts as unhealthy is how they //! systems independently deciding what counts as unhealthy is how they
//! end up disagreeing. //! end up disagreeing.
//! //!
@ -11,7 +12,7 @@
//! the *process* down over a transient dependency failure — that's what //! the *process* down over a transient dependency failure — that's what
//! readiness is for. Readiness derives `"degraded"` from any `crit`-level //! readiness is for. Readiness derives `"degraded"` from any `crit`-level
//! warning currently set; `warn`-level entries ride along in the body for //! warning currently set; `warn`-level entries ride along in the body for
//! detail but don't flip the status, mirroring the warning registry's own severity //! detail but don't flip the status, mirroring the banner's own severity
//! split. //! split.
//! //!
//! Routed outside `/api/` (see `dashboard/mod.rs`) so the gateway can //! Routed outside `/api/` (see `dashboard/mod.rs`) so the gateway can

View file

@ -102,6 +102,12 @@ pub(super) struct StateSnapshot {
/// var, set from `services.hyperhive.swarm.name`. `None` when /// var, set from `services.hyperhive.swarm.name`. `None` when
/// unset — chrome omits the swarm segment of the breadcrumb. /// unset — chrome omits the swarm segment of the breadcrumb.
swarm_name: Option<String>, swarm_name: Option<String>,
/// Server-level warnings for the dashboard's top-of-page banner
/// (currently host disk-pressure; more producers can be added
/// backend-side). Empty when all clear. Built by
/// `host_stats::server_warnings`; the frontend renders this list
/// generically, so new warning kinds need no frontend change.
server_warnings: Vec<crate::host_stats::ServerWarning>,
} }
#[derive(Serialize)] #[derive(Serialize)]
@ -187,6 +193,15 @@ pub(super) async fn api_state(
let tombstones = build_tombstone_views(&containers); let tombstones = build_tombstone_views(&containers);
let port_conflicts = build_port_conflicts(&containers); let port_conflicts = build_port_conflicts(&containers);
// Banner warnings: host probes (disk) + agent-state (pending logins).
// Built before the response struct because the agent-state producer
// borrows `containers`, which moves in below.
let server_warnings = {
let mut w = crate::host_stats::server_warnings();
w.extend(crate::host_stats::agent_state_warnings(&containers));
w
};
axum::Json(StateSnapshot { axum::Json(StateSnapshot {
seq, seq,
hostname, hostname,
@ -218,6 +233,7 @@ pub(super) async fn api_state(
swarm_name: std::env::var("HYPERHIVE_SWARM_NAME") swarm_name: std::env::var("HYPERHIVE_SWARM_NAME")
.ok() .ok()
.filter(|s| !s.is_empty()), .filter(|s| !s.is_empty()),
server_warnings,
}) })
} }

View file

@ -122,7 +122,7 @@ pub(crate) const CONFIG_ORG: &str = "agent-configs";
/// set of boot warnings in [`ensure_all`] keyed by a runtime name (at /// set of boot warnings in [`ensure_all`] keyed by a runtime name (at
/// most a handful per process). [`crate::warnings::set_boot_warning`] /// most a handful per process). [`crate::warnings::set_boot_warning`]
/// requires a `'static` kind so distinct orgs/repos don't clobber each /// requires a `'static` kind so distinct orgs/repos don't clobber each
/// other's warning entry; leaking a few short strings once per boot is /// other's banner entry; leaking a few short strings once per boot is
/// cheap and bounded, unlike a per-request or per-loop-iteration leak. /// cheap and bounded, unlike a per-request or per-loop-iteration leak.
fn static_kind(s: String) -> &'static str { fn static_kind(s: String) -> &'static str {
Box::leak(s.into_boxed_str()) Box::leak(s.into_boxed_str())
@ -340,7 +340,7 @@ fn hook_ids_at(hooks: &[forgejo_api::structs::Hook], url: &str) -> Vec<i64> {
/// confirms the container exists — not that Forgejo is *listening*. A /// confirms the container exists — not that Forgejo is *listening*. A
/// `nixos-rebuild` that restarts hive-forge and hive-c0re together races: /// `nixos-rebuild` that restarts hive-forge and hive-c0re together races:
/// without this gate every `ensure_*` fires at a refused socket and leaves a /// without this gate every `ensure_*` fires at a refused socket and leaves a
/// stale "provisioning failed" warning that never clears (the pass is one-shot). /// stale "provisioning failed" banner that never clears (the pass is one-shot).
/// Returns whether the endpoint became ready before the deadline; on timeout /// Returns whether the endpoint became ready before the deadline; on timeout
/// the caller proceeds anyway so a genuinely-down forge still surfaces its real /// the caller proceeds anyway so a genuinely-down forge still surfaces its real
/// errors. Uses the unauthenticated `get_version` endpoint (no token yet). /// errors. Uses the unauthenticated `get_version` endpoint (no token yet).
@ -393,7 +393,7 @@ pub async fn ensure_all() {
} }
// The container exists, but its HTTP may not be up yet (a rebuild restarts // The container exists, but its HTTP may not be up yet (a rebuild restarts
// hive-forge + hive-c0re together). Wait for the API to answer before the // hive-forge + hive-c0re together). Wait for the API to answer before the
// provisioning pass so a boot race doesn't leave stale failure warnings. // provisioning pass so a boot race doesn't leave stale failure banners.
wait_until_ready().await; wait_until_ready().await;
let core_token = match ensure_core_user_and_token().await { let core_token = match ensure_core_user_and_token().await {
Ok(t) => Some(t), Ok(t) => Some(t),

View file

@ -125,7 +125,7 @@ pub async fn push_meta(dir: &Path) -> Result<()> {
let msg = "forge/core/meta push rejected (non-fast-forward): \ let msg = "forge/core/meta push rejected (non-fast-forward): \
remote is ahead of local; leaving remote history intact"; remote is ahead of local; leaving remote history intact";
tracing::warn!("{msg}"); tracing::warn!("{msg}");
// Raise a persistent hive warning; cleared on next successful push. // Raise a persistent dashboard banner; cleared on next successful push.
if let Ok(mut g) = META_NON_FF_GUARD.lock() { if let Ok(mut g) = META_NON_FF_GUARD.lock() {
*g = Some(crate::warnings::set_warning( *g = Some(crate::warnings::set_warning(
"forge-meta-non-ff", "forge-meta-non-ff",

View file

@ -32,7 +32,7 @@ static LAST_FAILED_RELOAD: AtomicU64 = AtomicU64::new(0);
/// Minimum gap between retry attempts after a reload failure (30 s). /// Minimum gap between retry attempts after a reload failure (30 s).
const RELOAD_RETRY_SECS: u64 = 30; const RELOAD_RETRY_SECS: u64 = 30;
/// Hive-warning health tracker for the reload sweep — raises a /// Dashboard-banner health tracker for the reload sweep — raises a
/// `gateway_nginx_reload` warning the first time a reload fails (routing is /// `gateway_nginx_reload` warning the first time a reload fails (routing is
/// broken for whoever depends on the change *right now*, so no debounce /// broken for whoever depends on the change *right now*, so no debounce
/// window) and clears it the moment a reload succeeds again. /// window) and clears it the moment a reload succeeds again.

View file

@ -139,7 +139,7 @@ pub(super) async fn run_node(
/// Boot-time forge user/token sweep as a DAG node — see /// Boot-time forge user/token sweep as a DAG node — see
/// [`NodeKind::ForgeSweep`]. `forge::ensure_all` already does its own /// [`NodeKind::ForgeSweep`]. `forge::ensure_all` already does its own
/// per-step error handling and boot warnings internally (it's /// per-step error handling and boot-warning banners internally (it's
/// best-effort by design), so this wrapper has nothing left to report; /// best-effort by design), so this wrapper has nothing left to report;
/// it exists purely to make the sweep a visible unit of work. /// it exists purely to make the sweep a visible unit of work.
async fn run_forge_sweep() -> Result<()> { async fn run_forge_sweep() -> Result<()> {
@ -149,7 +149,7 @@ async fn run_forge_sweep() -> Result<()> {
/// Matrix user/space sweep as a DAG node — see [`NodeKind::MatrixSweep`]. /// Matrix user/space sweep as a DAG node — see [`NodeKind::MatrixSweep`].
/// Every pass reports here, whoever submitted it: as the node's own error on /// Every pass reports here, whoever submitted it: as the node's own error on
/// the dashboard, and into the debounced warning. /// the dashboard, and into the debounced banner.
async fn run_matrix_sweep() -> Result<()> { async fn run_matrix_sweep() -> Result<()> {
let ok = tokio::time::timeout(MATRIX_SWEEP_DEADLINE, crate::matrix::ensure_all()) let ok = tokio::time::timeout(MATRIX_SWEEP_DEADLINE, crate::matrix::ensure_all())
.await .await
@ -164,7 +164,7 @@ async fn run_matrix_sweep() -> Result<()> {
health.record_ok(); health.record_ok();
Ok(()) Ok(())
} else { } else {
health.record_err(matrix_sweep_warning); health.record_err(matrix_sweep_banner);
anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)") anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)")
} }
} }
@ -177,7 +177,7 @@ async fn run_matrix_sweep() -> Result<()> {
/// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep /// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep
const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10); const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10);
/// Warning for a matrix sweep that keeps failing. Two misses in a row, so one /// Banner for a matrix sweep that keeps failing. Two misses in a row, so one
/// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the /// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the
/// dashboard; cleared by the next clean pass. /// dashboard; cleared by the next clean pass.
fn matrix_sweep_health() -> &'static Mutex<SweepHealth> { fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
@ -185,7 +185,7 @@ fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2))) HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2)))
} }
fn matrix_sweep_warning(ctx: sweep_health::SweepFailure) -> String { fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String {
let age = ctx.since_last_ok.map_or_else( let age = ctx.since_last_ok.map_or_else(
|| "no success this session".to_owned(), || "no success this session".to_owned(),
|d| format!("last ok {} ago", sweep_health::fmt_age(d)), |d| format!("last ok {} ago", sweep_health::fmt_age(d)),
@ -200,7 +200,7 @@ fn matrix_sweep_warning(ctx: sweep_health::SweepFailure) -> String {
/// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every /// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every
/// pass reports here, whoever submitted it: as the node's own error on the /// pass reports here, whoever submitted it: as the node's own error on the
/// dashboard, and into the debounced warning. /// dashboard, and into the debounced banner.
async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> { async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
let result = tokio::time::timeout( let result = tokio::time::timeout(
KNOWLEDGE_PULL_DEADLINE, KNOWLEDGE_PULL_DEADLINE,
@ -242,7 +242,7 @@ async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
/// [`Resource::KnowledgeTree`]: super::resource::Resource::KnowledgeTree /// [`Resource::KnowledgeTree`]: super::resource::Resource::KnowledgeTree
const KNOWLEDGE_PULL_DEADLINE: std::time::Duration = std::time::Duration::from_mins(5); const KNOWLEDGE_PULL_DEADLINE: std::time::Duration = std::time::Duration::from_mins(5);
/// Warning for a `/knowledge` pull that keeps failing: three misses in a row, /// Banner for a `/knowledge` pull that keeps failing: three misses in a row,
/// cleared by the next successful pull. /// cleared by the next successful pull.
fn knowledge_pull_health() -> &'static Mutex<SweepHealth> { fn knowledge_pull_health() -> &'static Mutex<SweepHealth> {
static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new(); static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new();

View file

@ -367,11 +367,11 @@ pub async fn agents_for_meta_listing() -> Result<Vec<crate::meta::AgentSpec>> {
agents_for_meta(None).await agents_for_meta(None).await
} }
/// Per-agent "stop failed" warning guards, keyed by agent name. Cleared the /// Per-agent "stop failed" banner guards, keyed by agent name. Cleared the
/// next time that agent's container stops cleanly — a `Mutex<HashMap<...>>` /// next time that agent's container stops cleanly — a `Mutex<HashMap<...>>`
/// rather than a bare `set_boot_warning` because this condition (unlike a /// rather than a bare `set_boot_warning` because this condition (unlike a
/// one-shot boot step) genuinely resolves later, once an operator has /// one-shot boot step) genuinely resolves later, once an operator has
/// intervened, and the warning should clear with it instead of surviving /// intervened, and the banner should clear with it instead of surviving
/// until the next hive-c0re restart. /// until the next hive-c0re restart.
fn stop_failed_guards() fn stop_failed_guards()
-> &'static std::sync::Mutex<std::collections::HashMap<String, crate::warnings::WarningGuard>> { -> &'static std::sync::Mutex<std::collections::HashMap<String, crate::warnings::WarningGuard>> {
@ -402,7 +402,7 @@ pub async fn kill(name: &str) -> Result<()> {
.unwrap_or_else(std::sync::PoisonError::into_inner); .unwrap_or_else(std::sync::PoisonError::into_inner);
match result { match result {
Ok(()) => { Ok(()) => {
// A clean stop clears any earlier "this looked wedged" warning // A clean stop clears any earlier "this looked wedged" banner
// for the same agent. // for the same agent.
guards.remove(name); guards.remove(name);
Ok(()) Ok(())

Some files were not shown because too many files have changed in this diff Show more