agents: remove send(operator) and the operator inbox; alert on turn failures
Agents reach the operator over matrix now (#5115). A `send` to `operator` is refused, by the broker and by the MCP allow-list check, with an error that names the matrix MCP tools. The dashboard's Y3R C4LL tab goes with it: the operator inbox, `GET /api/operator-inbox`, `Broker::unread_for_recipient`, and the browser notifications that only ever fired on operator-bound messages. The harness no longer posts turn failures and plugin-install failures to the operator inbox. Both are already logged at warn; a new Grafana rule, `hh-agent-turn-failed`, fires per agent on the harness's `turn failed` line. Dashboard-to-agent messaging (op-send, the agent web UI's message box, mark-all-read for agent inboxes) is unchanged. Decisions per mara on #5115: q1 a, q2 a plus a grafana alert, q3 keep dashboard-to-agent messaging and leave swarm-level code alone.
This commit is contained in:
atlas
mara
parent
2bf809cbe5
commit
7e6a4c7727
37 changed files with 246 additions and 848 deletions
|
|
@ -42,7 +42,7 @@ Five tables, all in one file — the message queue, a small key/value
|
|||
table, the read-only schedule header/targets split, and the per-agent
|
||||
power-intent registry:
|
||||
|
||||
- `messages` — every inter-agent / operator-bound message.
|
||||
- `messages` — every inter-agent / operator-authored message.
|
||||
`sender / recipient / body / sent_at / delivered_at / acked_at /
|
||||
in_reply_to / priority`. `in_reply_to` links a reply to its parent
|
||||
row id; the dashboard and per-agent inbox render these as threaded
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ Four env vars cover the identity surface, read by
|
|||
module.
|
||||
- `HYPERHIVE_HIVE_DOMAIN` — the hive's canonical DNS domain (e.g.
|
||||
`darkest.space`), set by `nix/host-modules/hive-c0re/environment.nix`
|
||||
from `services.hyperhive.domain`. When configured, `qualified_label()`
|
||||
from `services.hyperhive.domain`. When configured, `qualify(label)`
|
||||
returns `${label}@${domain}` (for example `iris@darkest.space`); when
|
||||
unset (single-hive deployments, dev/test) it degrades to just
|
||||
the short label so existing callers see no change. The
|
||||
|
|
@ -46,11 +46,6 @@ can share a swarm name. Humans want both: the address
|
|||
(`@darkest.space`) AND the prose name (`pr1ma`). Matrix MXIDs
|
||||
use the domain-based convention.
|
||||
|
||||
`qualify(label)` is the same shape as `qualified_label()` but
|
||||
applies to an arbitrary label the caller already has (for example a peer
|
||||
name from the broker); it's the right surface when rendering a
|
||||
peer's name when the caller knows it's hive-local.
|
||||
|
||||
## Identity = socket
|
||||
|
||||
No auth tokens exist on the per-agent unix sockets. The socket
|
||||
|
|
@ -87,8 +82,10 @@ angle-bracket and asterisk shapes below are structurally safe.
|
|||
|
||||
- `*` — broadcast: deliver to every running agent except the sender
|
||||
(`socket_server::handle_send` fans out via `Coordinator::broadcast_send`).
|
||||
- `operator` — the human at the dashboard. Messages accumulate in the
|
||||
inbox view; no agent ever `recv`'s them.
|
||||
- `operator` — the human at the dashboard. Reserved so no agent can take
|
||||
the name, and refused as a recipient: the broker answers a `send` to it
|
||||
with `OPERATOR_SEND_REFUSAL` (`hive-sh4re/src/manager.rs`). It appears
|
||||
only as the sender of operator-authored messages.
|
||||
|
||||
The broker stores the recipient exactly as the agent passed it.
|
||||
|
||||
|
|
|
|||
|
|
@ -434,7 +434,7 @@ system prompts — and federated hives at different domains can share one.
|
|||
`hiveName` surfaces in the same places but is also the leftmost label of the
|
||||
hive's domain. `swarm.name` names the group; `hiveName` names this hive.
|
||||
|
||||
The env-var chain and `qualify()` / `qualified_label()` semantics:
|
||||
The env-var chain and `qualify()` semantics:
|
||||
[`conventions.md`](../process/conventions.md) § Hive identity.
|
||||
|
||||
Trust inside a swarm comes from the secret store's `pki` root
|
||||
|
|
|
|||
|
|
@ -126,9 +126,7 @@ user-configured MCP server can push its own todos the same way. See
|
|||
what each built-in todo producer watches and how the store + `get_loose_ends`
|
||||
merge work.
|
||||
|
||||
Plugin install failures aren't fatal: each entry comes back as a
|
||||
human-readable failure string that gets routed via
|
||||
`Surface::send_to_operator` to the operator.
|
||||
Plugin install failures aren't fatal: the harness logs each at `warn` and skips it.
|
||||
|
||||
### Turn outcomes
|
||||
|
||||
|
|
@ -144,7 +142,7 @@ else a `TurnError`) drives the post-claude branch:
|
|||
| `Err(SessionNotFound)` | resume + create self-heal both missed ("shouldn't happen"); requeue inflight so the next turn creates fresh — no status park, message not dropped |
|
||||
| `Err(ApiStall)` | idle watchdog sent claude SIGINT after `HIVE_TURN_IDLE_SECS` (default 600) of output silence and claude exited; sleep `HIVE_STALL_SLEEP_SECS` (default 60), requeue inflight, status back to `online` |
|
||||
| `Err(AgentStall(note))` | the same, for an ACP agent: after `HIVE_TURN_IDLE_SECS` with no `session/update` the harness sent the agent `session/cancel`, and SIGINT if it ignored that; `note` says which |
|
||||
| `Err(Failed(err))` | route `[system] \`<qualified-label>\` claude turn failed:\n<err>` to `operator` via `send_to_operator` |
|
||||
| `Err(Failed(err))` | the harness logs it at `warn` as `turn failed`, the line the Grafana alert rule `hh-agent-turn-failed` (`nix/host-modules/swarm-grafana.nix`) fires on |
|
||||
|
||||
`ApiStall` catches an Anthropic API stall — a multi-retry connection storm where
|
||||
the stream goes silent for minutes. On `claude` the idle watchdog lives in
|
||||
|
|
|
|||
|
|
@ -32,14 +32,13 @@ is opt-in via the P3RM1SS10NS tab.
|
|||
**Messaging** (`messaging` group): `send(to, body, in_reply_to?)`,
|
||||
`recv(max?)`, `ack_until(up_to)`.
|
||||
|
||||
- `send` — message a peer (logical name) or the operator
|
||||
(`to: "operator"`). Optional `in_reply_to: i64` links the
|
||||
message to a prior id for thread rendering. Per-agent
|
||||
`services.hyperhive.agent.allowedRecipients` (default: empty = unrestricted) limits
|
||||
which names `send` accepts — useful for sandboxing: set
|
||||
`[ "operator" ]` to restrict a sub-agent to operator messages only.
|
||||
The operator stays reachable regardless of this list, so an agent can
|
||||
always report a block.
|
||||
- `send` — message a peer on this hive (logical name). Optional
|
||||
`in_reply_to: i64` links the message to a prior id for thread
|
||||
rendering. Per-agent `services.hyperhive.agent.allowedRecipients`
|
||||
(default: empty = unrestricted) limits which names `send` accepts —
|
||||
useful for sandboxing. The broker refuses `to: "operator"` with an error
|
||||
that points at the matrix MCP tools: agents reach the operator over
|
||||
matrix ([matrix](../tools/matrix.md)).
|
||||
- `recv` — drain inbox. Always an immediate peek, never blocks. `max`
|
||||
(default 1, cap 5) drains up to N rows. `recv` prefixes each returned row
|
||||
with `[msg #<id>]` (broker row id; note the highest id seen, then pass
|
||||
|
|
|
|||
|
|
@ -16,12 +16,10 @@ Settings, Core). Every page links back to H0M3, so you're
|
|||
never more than one step from the hub.
|
||||
|
||||
The **dashboard** itself (`/dashboard.html`) is where you'll spend most
|
||||
of your time. It's a single page with exactly four tabs:
|
||||
of your time. It's a single page with exactly three tabs:
|
||||
|
||||
- **SW4RM** — every agent, live. This is the default tab and the one
|
||||
you'll check most.
|
||||
- **Y3R C4LL** — anything waiting on _you_: messages agents sent to
|
||||
the operator.
|
||||
- **P3RM1SS10NS** — what tools and system-level access each agent has.
|
||||
- **SCH3DUL3S** — a read-only list of the scheduled prompts left in the
|
||||
hive, due for removal; operators set schedules at swarm level now.
|
||||
|
|
@ -30,9 +28,7 @@ Everything else lives on its own page instead, all reachable from the
|
|||
H0M3 hub: **Flow** (`/flow.html`, the raw live message stream across
|
||||
the whole swarm), **Stats** (`/stats.html`, swarm-wide usage stats),
|
||||
**Builds** (`/builds.html`, the rebuild queue and build history), and
|
||||
**Core** (`/core.html`, tombstones and container resource use). Your local
|
||||
browser preferences (notifications) live in the dashboard's Y3R C4LL
|
||||
tab.
|
||||
**Core** (`/core.html`, tombstones and container resource use).
|
||||
|
||||
Each agent also has its own page — a full terminal view of that
|
||||
agent's session, reachable by selecting its name anywhere in the
|
||||
|
|
@ -47,10 +43,6 @@ whether it's running, what it's currently doing (a live status pill —
|
|||
and quick links (stats, screen, forge profile). Select the name to open
|
||||
its terminal and watch it work in real time.
|
||||
|
||||
**Read what agents sent you.** Y3R C4LL is the one tab worth checking
|
||||
regularly — it's everything that needs _you_. The tab's count pill
|
||||
tells you at a glance if anything's unread.
|
||||
|
||||
**Review a config change.** Agent config changes (new packages, env
|
||||
vars, MCP servers) are pull requests on the agent's `agent-configs`
|
||||
repo. Review and merge them on the forge; the merge deploys the change.
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ matrix accounts, the controller's job graph — are on the
|
|||
|
||||
hive-c0re serves the dashboard at `/dashboard.html` (with the home page at `/`).
|
||||
It has a fixed chrome header at the top and a `<main>` that shows exactly
|
||||
one tab pane at a time. The URL hash (`#swarm`, `#call`,
|
||||
one tab pane at a time. The URL hash (`#swarm`,
|
||||
`#permissions`, `#schedules`) drives which pane is
|
||||
active; hash changes don't reload the page. FL0W and the other
|
||||
standalone pages are reachable from the H0M3 hub at `/`, not
|
||||
|
|
@ -19,13 +19,12 @@ from the dashboard tab strip.
|
|||
**Chrome header** (fixed, overlays the active tab pane):
|
||||
- **← home back-link**: top-left of the chrome, links to the H0M3 hub
|
||||
at `/`. Every surface links back to the hub rather than to each other.
|
||||
- **Tab strip**: `◆ SW4RM ◆`, `◆ Y3R C4LL ◆`, `◆ P3RM1SS10NS ◆`,
|
||||
- **Tab strip**: `◆ SW4RM ◆`, `◆ P3RM1SS10NS ◆`,
|
||||
`◆ SCH3DUL3S ◆`. In-page tabs only — the SYST3M panels live on the
|
||||
standalone **C0R3** page (`/core.html`), and FL0W / ST4TS
|
||||
live on their own pages too, all reachable from the
|
||||
H0M3 hub (not the tab strip). Count pills on SW4RM
|
||||
(container count), Y3R C4LL (unread
|
||||
operator messages), and SCH3DUL3S (active schedules left to copy).
|
||||
(container count) and SCH3DUL3S (active schedules left to copy).
|
||||
- **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`)
|
||||
— sits below the tab strip.
|
||||
- **Server-warnings banner** — a generic, sticky top-of-page strip shown
|
||||
|
|
@ -46,12 +45,7 @@ from the dashboard tab strip.
|
|||
snapshot. Hidden when there are no warnings.
|
||||
- **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated
|
||||
to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name`
|
||||
arrive in the state snapshot. When there are unread operator messages,
|
||||
the page prepends a `(N)` prefix — `(3) pr1ma // h1ve-c0re` — so the
|
||||
operator can see the call count in an unfocused browser tab without
|
||||
opening the dashboard. The page sets the prefix on the operator-inbox
|
||||
cold-load and updates it live as broker `sent` events arrive; it's
|
||||
preserved when `hive_name` / `swarm_name` later replace the raw title.
|
||||
arrive in the state snapshot.
|
||||
|
||||
The FL0W page uses a slim header (a `← home` back-link + the page
|
||||
title) rather than the dashboard tab strip — it's a standalone surface,
|
||||
|
|
@ -71,41 +65,6 @@ conflicting agent.
|
|||
`↻ UPD4TE 4LL` button appears above the containers list when any
|
||||
agent is stale.
|
||||
|
||||
## Y3R C4LL tab
|
||||
|
||||
Things blocked on operator decision — the operator inbox.
|
||||
|
||||
**0PER4T0R 1NB0X** — messages agents have sent to `to="operator"` but
|
||||
the operator hasn't read yet. Cold-loaded from `/api/operator-inbox` on
|
||||
tab activation + page load; appended live from the broker `sent` stream
|
||||
(deduped on row id). Each row shows sender · timestamp · body (with
|
||||
file-path linkification). A `✓ mark all read` button on the right acks
|
||||
all rows via `POST /api/agent/operator/mark-all-read` (reuses the existing
|
||||
mark-read endpoint). Unread count folds into the Y3R C4LL tab pill so
|
||||
messages are visible from any tab even while inactive. Backed by
|
||||
`GET /api/operator-inbox` → `{ messages: [...] }` (id, from, body, at,
|
||||
in_reply_to, file_refs).
|
||||
|
||||
**◆ PR3F3R3NC3S ◆** — operator-local preferences. State lives in the
|
||||
browser's `localStorage` — preferences **don't** sync between devices and
|
||||
**don't** survive a profile wipe. One section today (browser
|
||||
notifications); future preferences land here as sibling `<h3>` blocks
|
||||
in `dashboard.html` — a standalone page for one toggle wouldn't earn
|
||||
its own tile.
|
||||
|
||||
**◇ browser notifications** — `🔔 enable notifications` button when
|
||||
permission ungranted; `🔕 mute / 🔔 unmute` toggle once granted
|
||||
(mute silences the dispatch without revoking the OS-level
|
||||
permission). On unsupported origins (non-secure context, or
|
||||
browsers without the `Notification` API) the controls hide and a
|
||||
single status line explains why. See `### Browser notifications`
|
||||
below for the dispatch model + the three signals the dashboard
|
||||
emits OS notifications on.
|
||||
|
||||
The "expand tool output panels" toggle lives on the per-agent page's
|
||||
own settings popover (`SettingsMenu`, `@hive/shared/settings-menu.js`)
|
||||
instead — it's agent-terminal-only, unrelated to this section.
|
||||
|
||||
## C0R3 page (`/core.html`)
|
||||
|
||||
Passive / rare-interaction state, reached from the **Core** tile on the
|
||||
|
|
@ -437,14 +396,6 @@ the model id, longest match wins) mapping to
|
|||
`{ input, output, cache_read, cache_write }` USD-per-million-token
|
||||
prices. Models not covered fall back to hive-c0re's built-in estimate.
|
||||
|
||||
The FL0W page **doesn't** host the PR3F3R3NC3S pane — it lives only on the
|
||||
dashboard's Y3R C4LL tab (reach it via the FL0W page's `← home`
|
||||
back-link → Dashboard). Notifications still fire on the FL0W page when
|
||||
they're enabled there, because `NOTIF.show()` in
|
||||
`common.js` depends on `Notification.permission` + the
|
||||
`hyperhive.notify.muted` localStorage key, not on the buttons
|
||||
existing in the page DOM.
|
||||
|
||||
## FL0W page (`/flow.html`)
|
||||
|
||||
A dedicated full-page terminal (not a tab pane — a separate HTML
|
||||
|
|
@ -452,10 +403,6 @@ page). Slim chrome: a `← home` back-link to the H0M3 hub, the FL0W
|
|||
title, and the agent-filter select (see below). No dashboard tab
|
||||
strip — FL0W is a standalone surface reached from the hub.
|
||||
|
||||
The operator inbox is **not** on this page — it lives on the
|
||||
dashboard's Y3R C4LL tab (◆ 1NB0X ◆ section, with per-message and
|
||||
mark-all read). FL0W stays the pure event firehose.
|
||||
|
||||
**MESS4GE FL0W** — live broker tail wrapped in a `.terminal-wrap`.
|
||||
Cold load backfills the last ~200 messages from `/api/dashboard/history`;
|
||||
live frames arrive on `/api/dashboard/stream`. Each row is one broker
|
||||
|
|
@ -767,30 +714,6 @@ frosted-mauve bar slides up from the bottom of the viewport
|
|||
Every render prunes stale selections (agents destroyed while
|
||||
selected) before the bar appears.
|
||||
|
||||
## Browser notifications
|
||||
|
||||
Pure frontend (`Notification` API), triggered by a new broker message
|
||||
sent `to: "operator"` (live via SSE).
|
||||
|
||||
The toggle controls live in the Y3R C4LL tab's ◆ PR3F3R3NC3S ◆
|
||||
section; see that section above for the user-facing shape. Dispatch
|
||||
logic lives in `common.js::NOTIF`.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
Only messages that arrive while the page is open count. Per-event tags
|
||||
(`hyperhive:msg:<at>:<rand>`) so distinct events stack in the OS
|
||||
notification center instead of overwriting each other.
|
||||
`console.debug` logs at every block point (unsupported,
|
||||
permission ungranted, muted) for in-browser debugging. Selecting it
|
||||
focuses the dashboard tab. The localStorage key
|
||||
`hyperhive.notify.muted` (`"1"` = muted, absent = unmuted) backs
|
||||
the toggle and silences dispatch without revoking the OS
|
||||
permission. Requires a secure context (HTTPS or localhost); on
|
||||
other origins the controls hide themselves. Browsers typically
|
||||
suppress notifications while the originating tab is focused —
|
||||
that's a browser-level decision, not the dashboard's.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
## Dashboard endpoints
|
||||
|
||||
Also browsable interactively at `/api/docs` (a Swagger UI, linked from
|
||||
|
|
@ -812,13 +735,6 @@ below — some endpoints aren't in it yet.
|
|||
not yet delivered and sets `acked_at = now`. Returns
|
||||
`{ "marked": N }`. Agent name validated against
|
||||
`[a-z0-9_-]`, 1-63 chars; 400 on bad input.
|
||||
- `GET /api/operator-inbox` — list unread messages addressed to
|
||||
`to="operator"` (broker rows with `acked_at = NULL`). Cold-loaded
|
||||
for the Y3R C4LL tab's ◆ 1NB0X ◆ section on page load + tab activation;
|
||||
live updates fed from the broker `sent` stream. Returns
|
||||
`{ messages: [{ id, from, body, at, in_reply_to, file_refs }, …] }`,
|
||||
newest-first. Reuses `/api/agent/operator/mark-all-read` to ack (filters
|
||||
are identical so every listed row is exactly what mark-read clears).
|
||||
- `POST /api/op-send` (`to=<name>`, `body=<text>`) — drop an
|
||||
operator-authored message into `<name>`'s inbox. `to=*` fans
|
||||
out to every registered agent. Returns 200; the broker
|
||||
|
|
@ -950,8 +866,7 @@ payload):
|
|||
(the broker row id) and `in_reply_to: Option<i64>` for thread
|
||||
rendering. The dashboard message-flow terminal renders reply
|
||||
rows with a `↳ reply` tag that scroll-highlights the parent
|
||||
row on selection. Used by the message-flow terminal renderer and
|
||||
the operator-inbox derived state.
|
||||
row on selection. Used by the message-flow terminal renderer.
|
||||
- `transient_set` (name, transient_kind, since_unix) /
|
||||
`transient_cleared` (name) — lifecycle action spinners. The
|
||||
client ticks the elapsed-seconds badge off `since_unix`
|
||||
|
|
@ -1007,7 +922,7 @@ payload):
|
|||
forms that mutate non-event-derived state** (PURG3 +
|
||||
meta-update, since tombstones + meta_inputs aren't event-
|
||||
shaped yet). Every other section — transients,
|
||||
containers, operator inbox, message flow —
|
||||
containers, message flow —
|
||||
derives from `/api/dashboard/stream` after the initial snapshot,
|
||||
maintaining its own client-side store and applying events on
|
||||
top.
|
||||
|
|
|
|||
|
|
@ -208,7 +208,7 @@ re-render), call `refreshState()`. State shapes live in
|
|||
`hive-c0re/src/dashboard/state_snapshot.rs::StateSnapshot` and
|
||||
`web_ui/state.rs::StateSnapshot` — when
|
||||
adding state fields, plumb through the snapshot struct and the
|
||||
relevant domain module (`swarm.js`, `call.js`, etc.) render function.
|
||||
relevant domain module (`swarm.js`, `schedules.js`, etc.) render function.
|
||||
|
||||
**Focus preservation:** `refreshState` checks whether
|
||||
`document.activeElement` sits inside one of the managed sections
|
||||
|
|
|
|||
Loading…
Reference in a new issue