Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

..
Author SHA1 Message Date
iris
dad495cdfa hive-c0re: fix stale banner and gauge-name wording after review
The nix-store gauge comment claimed a semconv name it deliberately does
not use; it now says why the name is hyperhive-scoped. Two doc pages
still described the removed banner for the limits-read and
knowledge-pull failures. The warning alert's comment now says
pending_logins pages on purpose.
2026-10-21 17:58:09 +02:00
iris
4dbcf6fa80 hive-c0re: hive warnings reach the operator as metrics and alerts, not a banner
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:

- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
  alert names the cause. Raising a warning also logs one `hive warning
  raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
  store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
  per kind (disk excluded), hh-host-disk-high at 85% and
  hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
  rescan, so it reaches the per-kind gauge.

Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.

A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
2026-10-21 17:52:53 +02:00
37 changed files with 478 additions and 325 deletions

View file

@ -66,7 +66,7 @@ pull`, so agents see the new content on their next turn.
queues a pull on a fixed cadence as a fallback (webhook missed, c0re
restarted between pushes). The pull is best-effort — a failure
shows as a failed `KnowledgePull` node on the job queue, counts toward
the knowledge-pull banner, and doesn't affect the rest of the daemon.
the `knowledge_pull` hive warning, and doesn't affect the rest of the daemon.
Both paths, and the one pull at boot, queue the same `KnowledgePull` job
node. It holds the working tree for its duration, so two pulls never run

View file

@ -529,8 +529,8 @@ given.
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
value passes the validators first. One bad
value or a failed read leaves the file unchanged, and repeated failures raise a
dashboard banner.
value or a failed read leaves the file unchanged, and repeated failures raise a hive
warning (the `hyperhive.hive.warning` metric and the `hh-hive-warning` alert).
A hive's queue grant reads every key of the bucket
(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the

View file

@ -444,7 +444,7 @@ whole-swarm spend figure.
<!-- vale write-good.Passive = NO -->
hive-c0re measures everything above **per agent**, tagged with the hive it
runs in. It measures these three per **hive** instead, and they carry no
runs in. It measures these five per **hive** instead, and they carry no
`agent` label — so a hive
that hosts no agents still reports, and "this hive is quiet" is
distinguishable from "this hive is gone." Select them with
@ -453,19 +453,40 @@ distinguishable from "this hive is gone." Select them with
<!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Meaning |
| ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
| -------------------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
hive-c0re reports both levels every cycle, `0` included, so a healthy hive is
visible as zeros rather than as missing series.
hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0`
included, so a healthy hive is visible as zeros rather than as missing
series.
`hyperhive.hive.degraded` is what a dashboard should alert on: it's
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
and with what the swarm controller sees. `hyperhive.hive.warnings` is the
detail behind it — `warn`-level entries mean "an operator should look" and do
**not** set `degraded`.
**not** set `degraded`. `hyperhive.hive.warning` names the cause: the `kind`
label is one of a fixed set in the source (for example `disk_pressure`, a
failing sweep such as `swarm_status_publish`, `pending_logins`,
`forge_ensure_core_user`), and the message lives in the
`hive warning raised` line hive-c0re logs when it raises the warning.
The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
| Alert | Fires when |
| ----------------------- | ----------------------------------------------------------------------------------- |
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists for 2 minutes |
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |
The disk alerts replace the old dashboard banner for the nix store, and
`hh-hive-warning` replaces it for every other warning. Disk pressure has its
own pair so the thresholds sit on the metric rather than on the registry
entry.
Same cadence, transport and resource labels as the container metrics above.

View file

@ -148,8 +148,8 @@ swarm-services certificate.
The identity isn't a choice — a hive authenticates as `hive-<hiveName>`
and publishes under `hiveName`, the same name that keys `swarm.hives`.
If a hive stops reporting, its own dashboard is the place to look: a
failure to publish raises a warning banner there after three consecutive
If a hive stops reporting, its `/health/ready` and the `hyperhive.hive.warning` metric are the place to look: a
failure to publish raises a hive warning after three consecutive
misses. It stays `warn` rather than `crit` on purpose — a hive that
can't reach the queue isn't itself unhealthy, so it doesn't start
calling itself degraded for being unable to say it's fine.

View file

@ -31,26 +31,17 @@ dashboard tab strip.
K3PT ST4T3 (kept-state entries).
- **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`)
— sits below the tab strip.
- **Server-warnings banner** — a generic, sticky top-of-page strip shown
on **every** page (the dashboard + the stand-alone FL0W
and ST4TS pages), injected at the top
of `<body>` by `renderServerWarnings` in `common.js`. Driven by
`state.server_warnings` — a list of `{ kind, level, message }` — and
coloured by `level` (`warn` amber / `crit` red). hive-c0re owns the
threshold + message, so adding a new system warning needs no frontend
change. Warnings are a push-based registry (`hive-c0re`'s
`warnings.rs`): any subsystem raises/clears its own entry via an RAII
guard, and `host_stats::server_warnings()` is just a cheap snapshot of
that registry. Producers today include the host disk-pressure check
(a `statvfs` probe of `/nix`: ≥85% used → `warn`, ≥95% → `crit`, e.g.
`⚠ host nix store N% full (G GiB free) — garbage-collect …`), forge
provisioning/CI-runner boot failures, and an agent-state warning
(`pending_logins`) computed alongside the container snapshot. A
container crash used to have a sibling `agents_crashing` entry here —
removed in favour of a swarm-level grafana alert on the crash
watcher's own log line (`docs/scheduler/observability.md`), so a
hive with no otel export now has no operator-visible crash signal at
all. Hidden when there are no warnings.
- **No warning banner.** No page shows hive-level warnings (host disk
pressure, failing background sweeps, forge boot failures, agents that
need a claude login). They live in a push-based registry
(`hive-c0re`'s `warnings.rs`): any subsystem raises or clears its own
entry through an RAII guard, and each raise logs one
`hive warning raised` line. The operator sees the registry as metrics and
Grafana alerts: see
[Hive-scoped metrics](../scheduler/observability.md#hive-scoped-metrics-hive-c0re).
A container crash is a swarm-level Grafana alert on the crash watcher's
own log line, so a hive with no otel export has no operator-visible
warning signal beyond `/health/ready`.
- **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated
to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name`
arrive in the state snapshot.

View file

@ -564,7 +564,7 @@ hive-side-panel .md td {
Inverted-gutter layout: the dashboard keeps `<body>` full-bleed
and wraps their padded content in a single `.page-content` div that
carries the horizontal gutter. Full-width chrome (the tab strip, the
server-warnings banner, the footer) then spans edge-to-edge for free,
footer) then spans edge-to-edge for free,
with no negative-margin breakout hacks. Only matches on pages that opt
in by adding the wrapper (index.html); FL0W / ST4TS
are full-bleed with their own `.<page>-main` padding. */
@ -573,50 +573,14 @@ hive-side-panel .md td {
}
/* ─── sticky top region ───────────────────────────────────────────────
One sticky container per page holding the warning banner above the
page chrome (tab bar / page header). common.js (ensureStickyTop)
wraps the page's chrome in this on first render and injects the banner
as its first child, so the banner + chrome stack in a single sticky
context — pinned together — instead of two separate `top:0` stickies
colliding (the banner used to overlay the tab bar). */
One sticky container per page holding the page chrome (tab bar / page
header). common.js (initStickyTop) wraps the chrome in this on load. */
.sticky-top {
position: sticky;
top: 0;
z-index: 50;
}
/* ─── server warnings banner ──────────────────────────────────────────
Generic top-of-page strip, injected as the first child of `.sticky-top`
by common.js (renderServerWarnings) on every page. One row per warning;
colour comes from the per-warning `level` (warn = amber, crit = red).
Stickiness lives on the `.sticky-top` wrapper, not here. The bar spans
the full width edge-to-edge; the dashboard is full-bleed at the
body level (their 1.5em gutter lives on an inner `.page-content`
wrapper), so the banner is full-width for free. */
.server-warnings[hidden] {
display: none;
}
.server-warn {
text-align: center;
padding: 0.4em 1em;
font-size: 0.85em;
font-weight: bold;
letter-spacing: 0.02em;
}
/* Tint composited over the opaque elevated-surface grey (not transparent):
the bar is `position: sticky`, so a see-through background would let
page content scroll through behind it. */
.server-warn-warn {
color: var(--amber);
background: color-mix(in srgb, var(--amber) 16%, var(--bg-elev));
border-bottom: 1px solid var(--amber);
}
.server-warn-crit {
color: var(--red);
background: color-mix(in srgb, var(--red) 18%, var(--bg-elev));
border-bottom: 1px solid var(--red);
}
/* ─── stats table ─────────────────────────────────────────────────
Shared right-aligned numeric table. Used by /stats.html (busiest
agents) — moved here from dashboard.css when ST4TS became its own

View file

@ -562,22 +562,12 @@ export function appendLinkified(parent, text, refs) {
}
}
// ─── server warnings banner ──────────────────────────────────────────
// A generic top-of-page banner shown on every page (dashboard + the
// stand-alone FL0W / ST4TS pages). The backend decides what to
// warn about — `/api/state.server_warnings` is a list of
// `{ kind, level, message }` — and this just renders it, coloured by
// `level` (`warn` amber / `crit` red). Adding a new system warning is a
// backend-only change. The bar is injected at the top of <body> so no
// page needs to add markup.
// The sticky top region — one sticky container holding the warning
// banner above the page's chrome (tab bar / page header), so the banner
// stacks with the chrome instead of being overlaid by it (two separate
// `top:0` stickies would otherwise collide). Built once by wrapping the
// page's existing chrome element; pages without a chrome (none today)
// get a banner-only sticky region at the top of <body>.
function ensureStickyTop() {
// ─── sticky top region ───────────────────────────────────────────────
// One sticky container wrapping the page's chrome (tab bar / page
// header), built once by wrapping the existing chrome element. A page
// without a chrome (none today) gets an empty sticky region at the top of
// <body>.
export function initStickyTop() {
let top = document.querySelector(".sticky-top");
if (top) return top;
top = document.createElement("div");
@ -591,51 +581,3 @@ function ensureStickyTop() {
}
return top;
}
function ensureServerWarningsBar() {
let bar = document.getElementById("server-warnings");
if (!bar) {
bar = document.createElement("div");
bar.id = "server-warnings";
bar.className = "server-warnings";
bar.setAttribute("role", "alert");
bar.hidden = true;
ensureStickyTop().prepend(bar);
}
return bar;
}
/// Render a `server_warnings` list (from /api/state) into the banner.
/// Empty / missing → the bar hides itself.
export function renderServerWarnings(warnings) {
const bar = ensureServerWarningsBar();
bar.replaceChildren();
if (!Array.isArray(warnings) || warnings.length === 0) {
bar.hidden = true;
return;
}
for (const w of warnings) {
const row = el("div", {
class:
"server-warn server-warn-" +
(w && w.level === "crit" ? "crit" : "warn"),
});
appendText(row, "⚠ " + ((w && w.message) || ""));
bar.append(row);
}
bar.hidden = false;
}
/// One-shot init for pages that don't otherwise poll /api/state: ensure
/// the bar exists, fetch the snapshot once, render. The dashboard (which
/// already polls /api/state) calls `renderServerWarnings` directly for
/// live updates instead.
export function initServerWarnings() {
ensureServerWarningsBar();
fetch("/api/state")
.then((r) => (r.ok ? r.json() : null))
.then((s) => renderServerWarnings(s && s.server_warnings))
.catch(() => {
/* non-fatal: no banner if the snapshot is unreachable */
});
}

View file

@ -17,7 +17,7 @@
body.dashboard-shell {
/* Full-bleed body (no horizontal gutter). The 1.5em side gutter lives
on the inner `.page-content` wrapper (see common.css), so the sticky
chrome bar, the server-warnings banner, and the footer span the full
chrome bar and the footer span the full
width edge-to-edge with no negative-margin breakout. `padding-bottom`
keeps a little breathing room above the viewport edge (overridden to
clear the selection bar via `.has-selection` below). No max-width cap
@ -34,8 +34,7 @@ body.dashboard-shell {
align-items: center;
gap: 0.75em;
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that
common.js wraps this chrome in, alongside the server-warning banner,
so they stack in one sticky context instead of colliding. */
common.js wraps this chrome in. */
background: color-mix(in srgb, var(--bg) 86%, transparent);
-webkit-backdrop-filter: blur(8px) saturate(120%);
backdrop-filter: blur(8px) saturate(120%);

View file

@ -10,17 +10,12 @@
// container list).
import { create as termCreate } from "@hive/shared/terminal.js";
import {
$,
appendLinkified,
openStream,
initServerWarnings,
} from "./common.js";
import { $, appendLinkified, openStream, initStickyTop } from "./common.js";
import { el } from "@hive/shared/dom.js";
import { epochSec } from "./util.js";
(() => {
initServerWarnings();
initStickyTop();
// ─── local containers cache (for compose autocomplete) ──────────────────
// The compose box's @-mention completion suggests known agent names.

View file

@ -5,7 +5,7 @@
// this bundle has no chart lib; per-agent trend charts live on each
// agent's own /stats page. The window selector is a hash-routed
// createTabStrip (#1h / #24h / …), matching the per-agent stats page.
import { $, initServerWarnings } from "./common.js";
import { $, initStickyTop } from "./common.js";
import { createTabStrip } from "@hive/shared/tabs.js";
let hiveStatsWindow = "24h";
@ -173,7 +173,7 @@ async function refreshHiveStats() {
}
}
initServerWarnings();
initStickyTop();
// Window selector → hash-routed createTabStrip (#1h / #24h / …). No panes
// here (the selector swaps data, not DOM); paneFor() returns null for each

View file

@ -11,7 +11,7 @@
// 4G3NTS (containers) domain lives in `./agents.js`.
import { marked } from "marked";
import { $, openStream, renderServerWarnings } from "./common.js";
import { $, openStream, initStickyTop } from "./common.js";
import { el } from "@hive/shared/dom.js";
import { bindAsyncForms } from "@hive/shared/forms.js";
import { createTabStrip } from "@hive/shared/tabs.js";
@ -50,6 +50,8 @@ import { renderLinksMenu } from "./links-menu.js";
window.marked = marked;
(() => {
initStickyTop();
// ─── async forms ────────────────────────────────────────────────────────
// Shared `data-async` submit interceptor (in common.js, shared with the other
// pages' bundles). On success it re-runs refreshState unless the form
@ -135,7 +137,6 @@ window.marked = marked;
// synchronous read (e.g. the compose autocomplete pulls agent
// names from here instead of refetching on every keystroke).
window.__hyperhive_state = s;
renderServerWarnings(s.server_warnings);
// Hive identity: render the swarm/hive name as a headline at the
// top of the 4G3NTS pane + update the page title once the
// server-side display names are known. `hive_name` / `swarm_name`

View file

@ -16,8 +16,7 @@
*/
.page-header {
/* Stickiness lives on the `.sticky-top` wrapper (common.css) that
common.js wraps this header in, alongside the server-warning banner,
so the two stack instead of colliding. */
common.js wraps this header in. */
background: color-mix(in srgb, var(--bg) 92%, transparent);
-webkit-backdrop-filter: blur(8px) saturate(120%);
backdrop-filter: blur(8px) saturate(120%);

View file

@ -599,6 +599,7 @@ impl Coordinator {
return;
}
};
crate::host_stats::refresh_pending_logins(&fresh);
let mut last = self.last_containers.lock().await;
let mut changed_or_new = Vec::new();
let mut removed = Vec::new();

View file

@ -1,9 +1,8 @@
//! Hive-wide health endpoints for external monitoring: `/health/live`
//! (liveness) and `/health/ready` (readiness).
//!
//! Deliberately thin: reuses [`crate::warnings`] — the same registry the
//! dashboard banner already reads — as the single source of truth for
//! "is anything degraded", rather than tracking health separately. Two
//! Deliberately thin: reuses [`crate::warnings`] as the single source of truth
//! for "is anything degraded", rather than tracking health separately. Two
//! systems independently deciding what counts as unhealthy is how they
//! end up disagreeing.
//!
@ -12,7 +11,7 @@
//! the *process* down over a transient dependency failure — that's what
//! readiness is for. Readiness derives `"degraded"` from any `crit`-level
//! warning currently set; `warn`-level entries ride along in the body for
//! detail but don't flip the status, mirroring the banner's own severity
//! detail but don't flip the status, mirroring the warning registry's own severity
//! split.
//!
//! Routed outside `/api/` (see `dashboard/mod.rs`) so the gateway can

View file

@ -102,12 +102,6 @@ pub(super) struct StateSnapshot {
/// var, set from `services.hyperhive.swarm.name`. `None` when
/// unset — chrome omits the swarm segment of the breadcrumb.
swarm_name: Option<String>,
/// Server-level warnings for the dashboard's top-of-page banner
/// (currently host disk-pressure; more producers can be added
/// backend-side). Empty when all clear. Built by
/// `host_stats::server_warnings`; the frontend renders this list
/// generically, so new warning kinds need no frontend change.
server_warnings: Vec<crate::host_stats::ServerWarning>,
}
#[derive(Serialize)]
@ -193,15 +187,6 @@ pub(super) async fn api_state(
let tombstones = build_tombstone_views(&containers);
let port_conflicts = build_port_conflicts(&containers);
// Banner warnings: host probes (disk) + agent-state (pending logins).
// Built before the response struct because the agent-state producer
// borrows `containers`, which moves in below.
let server_warnings = {
let mut w = crate::host_stats::server_warnings();
w.extend(crate::host_stats::agent_state_warnings(&containers));
w
};
axum::Json(StateSnapshot {
seq,
hostname,
@ -233,7 +218,6 @@ pub(super) async fn api_state(
swarm_name: std::env::var("HYPERHIVE_SWARM_NAME")
.ok()
.filter(|s| !s.is_empty()),
server_warnings,
})
}

View file

@ -122,7 +122,7 @@ pub(crate) const CONFIG_ORG: &str = "agent-configs";
/// set of boot warnings in [`ensure_all`] keyed by a runtime name (at
/// most a handful per process). [`crate::warnings::set_boot_warning`]
/// requires a `'static` kind so distinct orgs/repos don't clobber each
/// other's banner entry; leaking a few short strings once per boot is
/// other's warning entry; leaking a few short strings once per boot is
/// cheap and bounded, unlike a per-request or per-loop-iteration leak.
fn static_kind(s: String) -> &'static str {
Box::leak(s.into_boxed_str())
@ -340,7 +340,7 @@ fn hook_ids_at(hooks: &[forgejo_api::structs::Hook], url: &str) -> Vec<i64> {
/// confirms the container exists — not that Forgejo is *listening*. A
/// `nixos-rebuild` that restarts hive-forge and hive-c0re together races:
/// without this gate every `ensure_*` fires at a refused socket and leaves a
/// stale "provisioning failed" banner that never clears (the pass is one-shot).
/// stale "provisioning failed" warning that never clears (the pass is one-shot).
/// Returns whether the endpoint became ready before the deadline; on timeout
/// the caller proceeds anyway so a genuinely-down forge still surfaces its real
/// errors. Uses the unauthenticated `get_version` endpoint (no token yet).
@ -393,7 +393,7 @@ pub async fn ensure_all() {
}
// The container exists, but its HTTP may not be up yet (a rebuild restarts
// hive-forge + hive-c0re together). Wait for the API to answer before the
// provisioning pass so a boot race doesn't leave stale failure banners.
// provisioning pass so a boot race doesn't leave stale failure warnings.
wait_until_ready().await;
let core_token = match ensure_core_user_and_token().await {
Ok(t) => Some(t),

View file

@ -125,7 +125,7 @@ pub async fn push_meta(dir: &Path) -> Result<()> {
let msg = "forge/core/meta push rejected (non-fast-forward): \
remote is ahead of local; leaving remote history intact";
tracing::warn!("{msg}");
// Raise a persistent dashboard banner; cleared on next successful push.
// Raise a persistent hive warning; cleared on next successful push.
if let Ok(mut g) = META_NON_FF_GUARD.lock() {
*g = Some(crate::warnings::set_warning(
"forge-meta-non-ff",

View file

@ -32,7 +32,7 @@ static LAST_FAILED_RELOAD: AtomicU64 = AtomicU64::new(0);
/// Minimum gap between retry attempts after a reload failure (30 s).
const RELOAD_RETRY_SECS: u64 = 30;
/// Dashboard-banner health tracker for the reload sweep — raises a
/// Hive-warning health tracker for the reload sweep — raises a
/// `gateway_nginx_reload` warning the first time a reload fails (routing is
/// broken for whoever depends on the change *right now*, so no debounce
/// window) and clears it the moment a reload succeeds again.

View file

@ -139,7 +139,7 @@ pub(super) async fn run_node(
/// Boot-time forge user/token sweep as a DAG node — see
/// [`NodeKind::ForgeSweep`]. `forge::ensure_all` already does its own
/// per-step error handling and boot-warning banners internally (it's
/// per-step error handling and boot warnings internally (it's
/// best-effort by design), so this wrapper has nothing left to report;
/// it exists purely to make the sweep a visible unit of work.
async fn run_forge_sweep() -> Result<()> {
@ -149,7 +149,7 @@ async fn run_forge_sweep() -> Result<()> {
/// Matrix user/space sweep as a DAG node — see [`NodeKind::MatrixSweep`].
/// Every pass reports here, whoever submitted it: as the node's own error on
/// the dashboard, and into the debounced banner.
/// the dashboard, and into the debounced warning.
async fn run_matrix_sweep() -> Result<()> {
let ok = tokio::time::timeout(MATRIX_SWEEP_DEADLINE, crate::matrix::ensure_all())
.await
@ -164,7 +164,7 @@ async fn run_matrix_sweep() -> Result<()> {
health.record_ok();
Ok(())
} else {
health.record_err(matrix_sweep_banner);
health.record_err(matrix_sweep_warning);
anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)")
}
}
@ -177,7 +177,7 @@ async fn run_matrix_sweep() -> Result<()> {
/// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep
const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10);
/// Banner for a matrix sweep that keeps failing. Two misses in a row, so one
/// Warning for a matrix sweep that keeps failing. Two misses in a row, so one
/// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the
/// dashboard; cleared by the next clean pass.
fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
@ -185,7 +185,7 @@ fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2)))
}
fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String {
fn matrix_sweep_warning(ctx: sweep_health::SweepFailure) -> String {
let age = ctx.since_last_ok.map_or_else(
|| "no success this session".to_owned(),
|d| format!("last ok {} ago", sweep_health::fmt_age(d)),
@ -200,7 +200,7 @@ fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String {
/// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every
/// pass reports here, whoever submitted it: as the node's own error on the
/// dashboard, and into the debounced banner.
/// dashboard, and into the debounced warning.
async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
let result = tokio::time::timeout(
KNOWLEDGE_PULL_DEADLINE,
@ -242,7 +242,7 @@ async fn run_knowledge_pull(coord: &Arc<Coordinator>) -> Result<()> {
/// [`Resource::KnowledgeTree`]: super::resource::Resource::KnowledgeTree
const KNOWLEDGE_PULL_DEADLINE: std::time::Duration = std::time::Duration::from_mins(5);
/// Banner for a `/knowledge` pull that keeps failing: three misses in a row,
/// Warning for a `/knowledge` pull that keeps failing: three misses in a row,
/// cleared by the next successful pull.
fn knowledge_pull_health() -> &'static Mutex<SweepHealth> {
static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new();

View file

@ -367,11 +367,11 @@ pub async fn agents_for_meta_listing() -> Result<Vec<crate::meta::AgentSpec>> {
agents_for_meta(None).await
}
/// Per-agent "stop failed" banner guards, keyed by agent name. Cleared the
/// Per-agent "stop failed" warning guards, keyed by agent name. Cleared the
/// next time that agent's container stops cleanly — a `Mutex<HashMap<...>>`
/// rather than a bare `set_boot_warning` because this condition (unlike a
/// one-shot boot step) genuinely resolves later, once an operator has
/// intervened, and the banner should clear with it instead of surviving
/// intervened, and the warning should clear with it instead of surviving
/// until the next hive-c0re restart.
fn stop_failed_guards()
-> &'static std::sync::Mutex<std::collections::HashMap<String, crate::warnings::WarningGuard>> {
@ -402,7 +402,7 @@ pub async fn kill(name: &str) -> Result<()> {
.unwrap_or_else(std::sync::PoisonError::into_inner);
match result {
Ok(()) => {
// A clean stop clears any earlier "this looked wedged" banner
// A clean stop clears any earlier "this looked wedged" warning
// for the same agent.
guards.remove(name);
Ok(())

Some files were not shown because too many files have changed in this diff Show more