frontend: swarm.js off rebuild-queue-derived in-flight status onto transients
Fixes #2822. `swarm.js` had two independent per-agent "is this in flight" sources: `transientsState` (operator/worker-initiated ops the backend chose to flag) and `inFlightOpsByAgent()`, a separate derivation straight from `rebuildQueueState` covering everything else. Since #3010/#3016, `running_transients()` is a status-only test — any `Running` job-queue node naming a non-empty agent lights a transient pill, not just a curated subset — so the second source's Running-state handling is now provably redundant: a Running node with an agent always already has a transient by the time `queuedOpsByAgent()` (renamed from `inFlightOpsByAgent`) would be consulted. ## What changed - `transientsState`: `Map<name, {kind, since_unix}>` (one pill per agent) -> `Map<name, Map<kind, since_unix>>` (several pills per agent). `applyTransientSet`/`applyTransientCleared` now add/remove by `(name, kind)` rather than overwrite/delete by name alone, using `TransientCleared`'s `transient_kind` field (landed in #3016) to know which pill cleared. `syncTransientsFromSnapshot` groups the now-flat `TransientView` list by name instead of assuming one row per agent. - `inFlightOpsByAgent()` -> `queuedOpsByAgent()`: trimmed to the `Pending` (queued, not yet started) case only. The `Running` branch and its "running beats queued" priority logic are gone entirely — dead weight now that transients cover every running case unconditionally. - Render loop: an agent's transients win outright whenever any exist (rendered as **one badge per pill**, not collapsed into one label — mara: "show all running nodes that name the agent"); the queued fallback only applies when a agent has zero transients. `opRunning` simplifies to "does this agent have at least one transient". - `docs/web-ui/dashboard.md`'s Container-row section rewritten to match — it described a "transient, then in-flight-queue, in priority order" model that's no longer accurate now that the second source only ever fires for the one case the first can't represent. ## Verification `npm run build` clean for both packages (dashboard + agent). Standalone re-derivation of the transient-map + queued-fallback logic (`/tmp/verify-swarm-transients.mjs`, not part of this diff) run against constructed event sequences: single-pill lifecycle, two simultaneous pills on one agent with independent clear-by-kind, clearing an unknown kind is a safe no-op, a flat snapshot with duplicate agent names groups correctly, the queued fallback only fires when no transient exists and steps aside the instant one arrives, and a Running-state rebuild-queue entry produces no queued badge (confirming the Pending-only trim is correct, not just assumed). All 17 checks passed. Verified directly against the merged backend rather than trusting summaries: `job_queue/mod.rs::running_transients()` filters `State::Running` only (not Pending — an earlier note of mine claiming otherwise was imprecise paraphrasing), and `NodeView.agent` / `running_transients()`'s agent both resolve through the same `payload.agent()`, so a Running node's presence in `rebuild_queue` and its presence as a transient are guaranteed consistent, not just usually so. #2985 (DagView/NodeView deletion) unblocks once this merges — atlas is waiting on a ping.
This commit is contained in:
iris
mara
parent
2ff4cddf9d
commit
d1f82e725e
2 changed files with 138 additions and 113 deletions
|
|
@ -870,54 +870,59 @@ fetch entirely.
|
|||
`config` link — remain visible regardless of run state.
|
||||
When the container is running, status badges follow — `⊘ rate
|
||||
limited` (red, while the harness is parked after a 429), `needs
|
||||
login`, `needs update` — in-flight `◐ pending-state…` pill
|
||||
(replaces buttons during operator-initiated start / stop /
|
||||
restart / rebuild / destroy). Additionally, when a rebuild-queue
|
||||
entry for this agent is `queued` or `running` but no
|
||||
operator-initiated transient is set, the card surfaces a
|
||||
`building…` / `meta-updating…` / `starting…` / `stopping…` badge
|
||||
(per queue `kind` — e.g. a `start` entry shows `starting…` /
|
||||
`start queued`, `stop` and `graceful_stop` both show `stopping…` /
|
||||
`stop queued`) sourced from `rebuildQueueState` — so the SW4RM tab
|
||||
shows the same progress visible on the BU1LDS page's R3BU1LD QU3U3.
|
||||
The row visual splits queued vs running: a **queued** entry shows
|
||||
only the pending-state pill (no row tint, so a long queue doesn't
|
||||
paint half the tab amber); a **running** entry keeps the amber
|
||||
row tint AND draws a **rotating amber ring** around the agent
|
||||
icon, so it's obvious at a glance which container is actually
|
||||
moving.
|
||||
**Pending-state derivation:** the pill is sourced from two
|
||||
separate stores in priority order. (1) The **transient**
|
||||
(`transientsState`) — covers the create-and-start window where the
|
||||
container literally isn't up yet, before any backend state event
|
||||
has fired.
|
||||
login`, `needs update` — plus **one `◐ pending-state…` pill per
|
||||
active transient** (replaces buttons during operator-initiated
|
||||
start / stop / restart / rebuild / destroy). An agent can carry
|
||||
**several transients at once** (mara: "show all running nodes that
|
||||
name the agent") — e.g. a lease-exempt `prebuild` running alongside
|
||||
a `stop_for_update` on the same agent — and each renders as its own
|
||||
independent badge rather than being collapsed into one label,
|
||||
matching the existing multi-badge convention this line already uses
|
||||
for `paused`/`needs_update`/model/ctx.
|
||||
The row visual splits queued vs running: a **queued** entry (no
|
||||
transient yet, see below) shows only the pending-state pill (no row
|
||||
tint, so a long queue doesn't paint half the tab amber); a
|
||||
**running** entry (at least one transient) keeps the amber row tint
|
||||
AND draws a **rotating amber ring** around the agent icon, so it's
|
||||
obvious at a glance which container is actually moving.
|
||||
|
||||
A transient is **derived from the job-queue node currently
|
||||
running** against that agent, not declared per request, so its
|
||||
label follows the operation as it progresses (a rebuild reads
|
||||
`stop_for_update`, then `swap`, then `reconcile` rather than one
|
||||
constant `rebuilding` for its whole life). Two consequences for
|
||||
anything rendering it:
|
||||
**Pending-badge derivation:** two separate stores, but no longer a
|
||||
priority *order* between them — the second only ever applies when
|
||||
the first has nothing to say. (1) **Transients**
|
||||
(`transientsState`, keyed `agent -> Map<kind, since_unix>`) — a
|
||||
transient is **derived from a job-queue node currently `Running`**
|
||||
against that agent, not declared per request, so its label follows
|
||||
the operation as it progresses (a rebuild reads `stop_for_update`,
|
||||
then `swap`, then `reconcile` rather than one constant `rebuilding`
|
||||
for its whole life). Two consequences for anything rendering it:
|
||||
|
||||
- The label vocabulary is **open** — it is the node's own wire tag
|
||||
(`NodeKind::as_str`, the same strings `NodeView.kind` carries),
|
||||
not a fixed set. Treat it as an opaque display string; do not
|
||||
switch on specific values. `restarting` in particular no longer
|
||||
exists, because no node kind is unique to a restart.
|
||||
- It is **not** exclusively operator-initiated. Work the operator
|
||||
never clicked (a meta-update cascade, a crash-recover rebuild)
|
||||
lights the same pill, since it is the running node that sets it.
|
||||
- It is **not** exclusively operator-initiated, and **not** limited
|
||||
to rebuild-shaped work — `running_transients()` on the backend is
|
||||
a status-only test (any `Running` node whose payload names a
|
||||
non-empty agent lights a pill), so work the operator never
|
||||
clicked (a meta-update cascade, a crash-recover rebuild, a
|
||||
lease-exempt `prebuild`) lights the same mechanism.
|
||||
|
||||
Ops with no queue node behind them (destroy, migration) supply
|
||||
their own label directly. (2) If no transient is set, the **rebuild-queue
|
||||
entry** for this agent is consulted (`rebuildQueueState`); this
|
||||
covers worker-driven ops — meta-update cascades, crash-recover
|
||||
rebuilds, approval-driven rebuilds — that the operator didn't
|
||||
click. `ContainerStateChanged` carries neither signal, so the
|
||||
dashboard reads from the two snapshots directly. The
|
||||
`opRunning` flag (driving the `pending-running` row class +
|
||||
spinner) is true when (1) is set OR (2) is in `running` state;
|
||||
queued entries leave `opRunning` false.
|
||||
their own label directly via `TransientSet`/`TransientCleared`
|
||||
events carrying no backing node at all.
|
||||
|
||||
(2) The **rebuild-queue fallback** (`rebuildQueueState`) only
|
||||
fires when an agent has **zero** transients — since (1) now covers
|
||||
every `Running` node unconditionally, a `Running` rebuild-queue
|
||||
entry can never usefully reach this fallback by the time it's
|
||||
consulted; the fallback exists purely for the **`Pending`
|
||||
(queued, not yet started)** case, which `running_transients()`'s
|
||||
Running-only test cannot represent. `queuedOpsByAgent()` (swarm.js)
|
||||
reflects this: it only ever looks at `Pending`-state queue nodes.
|
||||
`opRunning` (driving the `pending-running` row class + spinner) is
|
||||
simply "does this agent have at least one transient" — a queued-only
|
||||
entry (no transient yet) leaves it false.
|
||||
An **active model badge** (`model · <name>`, blue) appears when the
|
||||
container is running and the harness has persisted a model name
|
||||
(`harness/hyperhive-model`). Read by hive-c0re's `ContainerView`
|
||||
|
|
|
|||
|
|
@ -37,8 +37,13 @@ const containerRowCache = new Map();
|
|||
|
||||
// Derived transient state — cold-loaded from /api/state.transients,
|
||||
// then mutated live by `transient_set` / `transient_cleared`. Keyed
|
||||
// by agent name so add/remove are O(1). `since_unix` is wall-clock so
|
||||
// the elapsed-seconds badge ticks without polling.
|
||||
// by agent name -> Map<transient_kind, since_unix>: an agent can hold
|
||||
// several pills at once (every *running* node naming the agent lights
|
||||
// one now, not just a curated "worth surfacing" subset — see
|
||||
// docs/web-ui.md::Container row), keyed the same way the backend's
|
||||
// own tombstone map is, `(agent, label)`, so a clear only removes the
|
||||
// specific pill it names rather than guessing. `since_unix` is
|
||||
// wall-clock so the elapsed-seconds badge ticks without polling.
|
||||
const transientsState = new Map();
|
||||
|
||||
// In-memory set of selected agent logical names backing the sticky
|
||||
|
|
@ -54,77 +59,99 @@ export function syncRebuildQueueFromSnapshot(s) {
|
|||
}
|
||||
export function applyRebuildQueueChanged(ev) {
|
||||
rebuildQueueState = (ev.queue || []).slice();
|
||||
// Re-render the SW4RM tab so newly-queued / newly-running
|
||||
// rebuild-queue ops light up the right card with a building...
|
||||
// badge, and finished ops fall back to the regular state badges.
|
||||
// Re-render the SW4RM tab so newly-queued ops light up the right
|
||||
// card with a "<kind> queued…" badge, and entries that drop out of
|
||||
// Pending fall back to whatever transientsState (or nothing) says
|
||||
// instead. Running work is *not* driven by this event — that's
|
||||
// transient_set/transient_cleared's job, since every running node
|
||||
// naming an agent already lights a pill by the time it gets here.
|
||||
// See docs/web-ui.md::Container row for the badge taxonomy.
|
||||
renderContainersFromState();
|
||||
}
|
||||
// Map from agent name -> highest-priority in-flight op ({ kind, state })
|
||||
// (`running` beats `queued`). Used by the container row renderer to
|
||||
// surface "building..." / "meta-updating..." badges on the SW4RM tab
|
||||
// when an op is still in the rebuild queue but no operator-initiated
|
||||
// transient is set.
|
||||
// Map from agent name -> the queued op ({ kind }) backing the SW4RM
|
||||
// card's "<kind> queued" badge. Pending only — every *running* node
|
||||
// naming an agent already lights a transient pill (any running node,
|
||||
// not just a curated "worth it" subset — see docs/web-ui.md::Container
|
||||
// row), so a Running entry here would always be redundant with
|
||||
// `transientsState` by the time this is consulted. Queued (not yet
|
||||
// started) work is the one state transients can't represent, since
|
||||
// `running_transients()` on the backend is a Running-only test.
|
||||
//
|
||||
// Agent is per-node, not per-DAG (a DAG can span agents — e.g. the
|
||||
// startup sweep's MetaLock cascade, or a hive-wide restart), so this
|
||||
// derives each agent's in-flight state from its own node(s) within the
|
||||
// entry rather than the DAG's overall `state`/`kind` — a DAG can be
|
||||
// `running` overall while a given agent's subgraph hasn't started yet
|
||||
// (still queued behind an earlier node in its chain), and vice versa.
|
||||
function inFlightOpsByAgent() {
|
||||
// derives each agent's queued state from its own node(s) within the
|
||||
// entry rather than the DAG's overall `state`/`kind`.
|
||||
function queuedOpsByAgent() {
|
||||
const out = new Map();
|
||||
for (const e of rebuildQueueState) {
|
||||
if (e.state !== 'Pending' && e.state !== 'Running') continue;
|
||||
if (e.state !== 'Pending') continue;
|
||||
// spawn ops target an agent that doesn't exist yet as a
|
||||
// container — the transient store already drives the
|
||||
// pending row for that case. Skip here to avoid double-
|
||||
// surfacing if the spawn op happens to land in the queue
|
||||
// while the row exists transiently.
|
||||
if (e.kind === 'spawn') continue;
|
||||
const perAgentState = new Map();
|
||||
for (const n of e.nodes || []) {
|
||||
if (!n.agent) continue;
|
||||
if (n.state !== 'Pending' && n.state !== 'Running') continue;
|
||||
const cur = perAgentState.get(n.agent);
|
||||
if (!cur || (cur === 'Pending' && n.state === 'Running')) {
|
||||
perAgentState.set(n.agent, n.state);
|
||||
}
|
||||
}
|
||||
for (const [agent, state] of perAgentState) {
|
||||
const cur = out.get(agent);
|
||||
if (!cur || (cur.state === 'Pending' && state === 'Running')) {
|
||||
out.set(agent, { kind: e.kind, state });
|
||||
}
|
||||
if (!n.agent || n.state !== 'Pending') continue;
|
||||
// First entry found wins — with only one state to consider
|
||||
// (Pending), there's no priority to resolve between DAGs.
|
||||
if (!out.has(n.agent)) out.set(n.agent, { kind: e.kind });
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
// "<kind> queued" text for the queuedOpsByAgent fallback badge. Only
|
||||
// ever applies to this fallback — a transient's own kind is an opaque
|
||||
// wire tag (docs/web-ui.md::Container row) displayed as-is, never
|
||||
// run through a lookup like this one.
|
||||
function queuedLabelFor(kind) {
|
||||
return kind === 'meta_update' ? 'meta-update queued'
|
||||
: kind === 'destroy' ? 'destroy queued'
|
||||
: kind === 'restart' ? 'restart queued'
|
||||
: kind === 'start' ? 'start queued'
|
||||
: kind === 'stop' ? 'stop queued'
|
||||
: kind === 'graceful_stop' ? 'stop queued'
|
||||
: kind === 'reconcile' ? 'reconcile queued'
|
||||
: 'rebuild queued';
|
||||
}
|
||||
|
||||
// ─── transients ─────────────────────────────────────────────────────────────
|
||||
|
||||
export function syncTransientsFromSnapshot(s) {
|
||||
transientsState.clear();
|
||||
const nowUnix = Math.floor(Date.now() / 1000);
|
||||
for (const t of s.transients || []) {
|
||||
// Snapshot is a flat list — multiple rows CAN share the same
|
||||
// `name` (one row per running node) — so group into this agent's
|
||||
// kind map rather than overwrite.
|
||||
let byKind = transientsState.get(t.name);
|
||||
if (!byKind) {
|
||||
byKind = new Map();
|
||||
transientsState.set(t.name, byKind);
|
||||
}
|
||||
// Snapshot ships `secs` (server-computed); reconstruct an
|
||||
// approximate since_unix so the live ticker keeps progressing
|
||||
// without surprising jumps when the next snapshot lands.
|
||||
const nowUnix = Math.floor(Date.now() / 1000);
|
||||
transientsState.set(t.name, {
|
||||
kind: t.kind,
|
||||
since_unix: t.since_unix ?? (nowUnix - (t.secs || 0)),
|
||||
});
|
||||
byKind.set(t.kind, t.since_unix ?? (nowUnix - (t.secs || 0)));
|
||||
}
|
||||
}
|
||||
export function applyTransientSet(ev) {
|
||||
transientsState.set(ev.name, {
|
||||
kind: ev.transient_kind,
|
||||
since_unix: ev.since_unix,
|
||||
});
|
||||
let byKind = transientsState.get(ev.name);
|
||||
if (!byKind) {
|
||||
byKind = new Map();
|
||||
transientsState.set(ev.name, byKind);
|
||||
}
|
||||
byKind.set(ev.transient_kind, ev.since_unix);
|
||||
renderContainersFromState();
|
||||
}
|
||||
export function applyTransientCleared(ev) {
|
||||
if (transientsState.delete(ev.name)) renderContainersFromState();
|
||||
const byKind = transientsState.get(ev.name);
|
||||
if (!byKind || !byKind.delete(ev.transient_kind)) return;
|
||||
// Drop the now-empty outer entry too — keeps `transientsState.size`
|
||||
// meaning "agents with at least one pill" (checked at the
|
||||
// no-containers-and-no-transients empty-state gate below).
|
||||
if (byKind.size === 0) transientsState.delete(ev.name);
|
||||
renderContainersFromState();
|
||||
}
|
||||
// Re-render using the last cached snapshot (containers come from
|
||||
// /api/state, transients overlay from the derived map). The snapshot
|
||||
|
|
@ -327,7 +354,7 @@ function buildContainerLi(c, node, opts) {
|
|||
} = opts;
|
||||
const li = el('li', {
|
||||
class: 'container-row'
|
||||
+ (pending ? ' pending' : '')
|
||||
+ (pending.length ? ' pending' : '')
|
||||
+ (opRunning ? ' pending-running' : '')
|
||||
+ (selected ? ' selected' : ''),
|
||||
});
|
||||
|
|
@ -489,9 +516,14 @@ function buildContainerLi(c, node, opts) {
|
|||
// badge. `needs_login` is still c0re-owned (reads auth sentinel
|
||||
// files on the host). rate_limited / ctx / status_text are
|
||||
// agent-owned and rendered by the async dashboard-state fetch above.
|
||||
if (pending) {
|
||||
head.append(el('span', { class: 'pending-state' },
|
||||
el('span', { class: 'spinner' }, '◐'), ' ', pending + '…'));
|
||||
if (pending.length) {
|
||||
// One badge per pill — an agent can carry several transients at
|
||||
// once now (see docs/web-ui.md::Container row), each rendered
|
||||
// independently rather than collapsed into one label.
|
||||
for (const label of pending) {
|
||||
head.append(el('span', { class: 'pending-state' },
|
||||
el('span', { class: 'spinner' }, '◐'), ' ', label + '…'));
|
||||
}
|
||||
} else if (!c.running) {
|
||||
head.append(el('span',
|
||||
{ class: 'badge badge-muted', title: 'container is shut down — start it to bring the harness back up' },
|
||||
|
|
@ -648,11 +680,10 @@ export function renderContainers(s) {
|
|||
const forgeBase = (s && s.forge_public_url) || null;
|
||||
const ul = existingUl ?? el('ul', { class: 'containers' });
|
||||
const tree = buildAgentTree(containers);
|
||||
// In-flight rebuild / meta-update / destroy ops per agent name —
|
||||
// see docs/web-ui.md::Container row for the building... badge
|
||||
// rationale (covers the SYST3M-shows-rebuild-but-SW4RM-shows-stopped
|
||||
// gap when no operator transient is set).
|
||||
const inFlight = inFlightOpsByAgent();
|
||||
// Queued (not-yet-running) ops per agent name — see
|
||||
// docs/web-ui.md::Container row for why this only covers the
|
||||
// Pending case now (Running is fully covered by transientsState).
|
||||
const queuedOps = queuedOpsByAgent();
|
||||
|
||||
// Build the ordered list of <li> elements, reusing cached rows
|
||||
// whose displayed state hasn't changed.
|
||||
|
|
@ -666,31 +697,20 @@ export function renderContainers(s) {
|
|||
const containerBase = gatewayLinks
|
||||
? `/agent/${encodeURIComponent(c.name)}`
|
||||
: `http://${hostname}:${c.port}`;
|
||||
// Pending-state derivation + queued-vs-running split — see
|
||||
// docs/web-ui.md::Container row for the transient -> in-flight
|
||||
// queue priority order and the opRunning rationale.
|
||||
const transientKind = transientsState.get(c.name)?.kind || null;
|
||||
const op = !transientKind ? inFlight.get(c.name) : null;
|
||||
const pending = transientKind
|
||||
|| (op && (op.state === 'Running'
|
||||
? (op.kind === 'meta_update' ? 'meta-updating'
|
||||
: op.kind === 'destroy' ? 'destroying'
|
||||
: op.kind === 'restart' ? 'restarting'
|
||||
: op.kind === 'start' ? 'starting'
|
||||
: op.kind === 'stop' ? 'stopping'
|
||||
: op.kind === 'graceful_stop' ? 'stopping'
|
||||
: op.kind === 'reconcile' ? 'reconciling'
|
||||
: 'rebuilding')
|
||||
: (op.kind === 'meta_update' ? 'meta-update queued'
|
||||
: op.kind === 'destroy' ? 'destroy queued'
|
||||
: op.kind === 'restart' ? 'restart queued'
|
||||
: op.kind === 'start' ? 'start queued'
|
||||
: op.kind === 'stop' ? 'stop queued'
|
||||
: op.kind === 'graceful_stop' ? 'stop queued'
|
||||
: op.kind === 'reconcile' ? 'reconcile queued'
|
||||
: 'rebuild queued')));
|
||||
const opRunning = transientKind != null
|
||||
|| (op != null && op.state === 'Running');
|
||||
// Pending-badge derivation: an agent's transients win outright when
|
||||
// any exist (rendered one badge per pill — mara: "show all running
|
||||
// nodes that name the agent"), the rebuild-queue's queued-only
|
||||
// fallback otherwise. See docs/web-ui.md::Container row.
|
||||
const transientKindsMap = transientsState.get(c.name);
|
||||
// Sorted for stable badge order across renders — Map iteration
|
||||
// order is insertion order, which shifts as pills clear/re-add.
|
||||
const transientKinds = transientKindsMap
|
||||
? Array.from(transientKindsMap.keys()).sort() : [];
|
||||
const queuedOp = transientKinds.length === 0 ? queuedOps.get(c.name) : null;
|
||||
const pending = transientKinds.length > 0
|
||||
? transientKinds
|
||||
: (queuedOp ? [queuedLabelFor(queuedOp.kind)] : []);
|
||||
const opRunning = transientKinds.length > 0;
|
||||
const selected = selectionState.has(c.name);
|
||||
// Pending questions where this agent is the asker (awaiting an
|
||||
// answer) or the target (owes a reply). Derived live from
|
||||
|
|
|
|||
Loading…
Reference in a new issue