Compare commits
6 changed files with 15 additions and 184 deletions
22
TODO.md
22
TODO.md
|
|
@ -21,16 +21,24 @@
|
||||||
|
|
||||||
## Dashboard
|
## Dashboard
|
||||||
|
|
||||||
- **Reminder delivery-error surface**: `reminder_scheduler::tick` logs failed deliveries but doesn't persist. Add `last_error TEXT, attempt_count INTEGER` columns + a banner on the dashboard row + a "retry" affordance. Needs a sqlite migration (idempotent ALTER TABLE).
|
<!-- Landed: dashboard questions pane now shows both operator-targeted
|
||||||
- **Per-agent reminder status / query interface**: surface pending vs. delivered counts per agent (manager + each sub-agent) as a small chip on the container row.
|
and peer (agent-to-agent) threads, with filter chips (all /
|
||||||
|
@operator / @peer / per-participant) and an 0V3RR1D3 button on
|
||||||
|
peer rows so the operator can answer when an agent is stuck. -->
|
||||||
|
<!-- Landed: PATH_RE-detected pointer strings in message + question +
|
||||||
|
answer + inbox bodies linkify to inline <details> previews
|
||||||
|
fetched from /api/state-file. Allow-list: `/agents/<n>/state/...`
|
||||||
|
+ `/var/lib/hyperhive/agents/<n>/state/...` + `/shared/...` +
|
||||||
|
`/var/lib/hyperhive/shared/...`. Legacy bare `/state/...` is
|
||||||
|
intentionally NOT matched (ambiguous from host's perspective);
|
||||||
|
prefer `/agents/<n>/state/...` in agent outputs. -->
|
||||||
|
- **UI for pending reminders**: show pending/queued reminders in dashboard, allow operator to view/debug/cancel
|
||||||
|
- Per-agent reminder status (pending, delivered)
|
||||||
|
- Reminder query interface for debugging
|
||||||
|
- Display reminder delivery errors (failed sends, mark failures)
|
||||||
- **Phase 6 follow-ups** — dashboard side is fully event-driven (Phase 6 leftovers landed); the per-agent web UI's lifecycle endpoints (`/api/{cancel,compact,model,new-session}`, `/login/*`) still 303-redirect-and-poll. Convert them to 200 + `data-no-refresh` so the per-agent page stops refetching `/api/state` on every operator click — `LiveEvent::Note` already covers cancel/compact/model/new-session, login state needs its own `NeedsLogin` / `LoggedIn` events on the per-agent bus.
|
- **Phase 6 follow-ups** — dashboard side is fully event-driven (Phase 6 leftovers landed); the per-agent web UI's lifecycle endpoints (`/api/{cancel,compact,model,new-session}`, `/login/*`) still 303-redirect-and-poll. Convert them to 200 + `data-no-refresh` so the per-agent page stops refetching `/api/state` on every operator click — `LiveEvent::Note` already covers cancel/compact/model/new-session, login state needs its own `NeedsLogin` / `LoggedIn` events on the per-agent bus.
|
||||||
- **Tombstones + meta_inputs events**: not yet event-derived. PURG3 + meta-update still trigger a post-submit `/api/state` refetch on the dashboard. Add `TombstoneAdded`/`TombstoneRemoved` + `MetaInputsChanged` so those forms can drop their refetch too and the cold-load is the only `/api/state` fetch in normal operation.
|
- **Tombstones + meta_inputs events**: not yet event-derived. PURG3 + meta-update still trigger a post-submit `/api/state` refetch on the dashboard. Add `TombstoneAdded`/`TombstoneRemoved` + `MetaInputsChanged` so those forms can drop their refetch too and the cold-load is the only `/api/state` fetch in normal operation.
|
||||||
|
|
||||||
## Security
|
|
||||||
|
|
||||||
- **Privsep the dashboard from the privileged daemon**: hive-c0re runs as root (it has to — `nixos-container` create / start / destroy, the meta git repo, every per-agent bind mount). The HTTP server lives in the same process, so every read-endpoint (`/api/state-file`, `/api/journal/{name}`, `/api/agent-config/{name}`) is one allow-list bug away from serving arbitrary host files. Split the architecture: keep the privileged daemon doing lifecycle + git + ipc, run the web UI as an unprivileged user that talks to the daemon over a unix socket with a narrow request surface (`ReadAgentStateFile { agent, rel_path }` etc.). The unprivileged process can't read `/etc/shadow` even if every check in `get_state_file` is bypassed — it doesn't have the bits. Container-lifecycle POSTs (`/restart`, `/destroy`, etc.) become forwarded RPCs the privileged side authorises on its terms.
|
|
||||||
- **Defense in depth on `get_state_file`**: until privsep lands, the allow-list is load-bearing. Worth adding: refuse files whose mode is not world-readable (so an agent writing a 0600 file inside `state/` can't have its contents proxied through the endpoint to a different operator), and refuse symlinks at any path component (`O_NOFOLLOW`-style — `canonicalize` resolves them, but we currently don't reject if the original path had symlinks).
|
|
||||||
|
|
||||||
## Bugs
|
## Bugs
|
||||||
|
|
||||||
- **Post-rebuild system-message missed wake**: at 09:13:14 the dashboard showed `system → damocles container rebuilt` as ✓ delivered, but the agent harness never ran a turn for it (no claude invocation, no operator-visible activity). A subsequent `recv()` from inside the agent returned `(empty)`, confirming the message was popped + marked delivered server-side — yet drove no turn. Most likely cause: the agent_server `serve_agent_stdio` task is up and answering MCP/socket calls, but the `hive-ag3nt::serve` long-poll loop that drives `drive_turn` either died silently during rebuild or never restarted. Investigate: (a) does hive-ag3nt's serve loop survive `nixos-container update` cleanly, or does its tokio runtime get torn down mid-loop? (b) is there an early-exit path on a transient socket error during rebuild that drops the serve task without notifying the manager? (c) compare timeline with manager's own post-rebuild wake to see if this is rebuilt-agents-only or universal. Could be related to the `recv_blocking` fix in `e423d57` if the rebuild restarts the broker mid-subscribe.
|
- **Post-rebuild system-message missed wake**: at 09:13:14 the dashboard showed `system → damocles container rebuilt` as ✓ delivered, but the agent harness never ran a turn for it (no claude invocation, no operator-visible activity). A subsequent `recv()` from inside the agent returned `(empty)`, confirming the message was popped + marked delivered server-side — yet drove no turn. Most likely cause: the agent_server `serve_agent_stdio` task is up and answering MCP/socket calls, but the `hive-ag3nt::serve` long-poll loop that drives `drive_turn` either died silently during rebuild or never restarted. Investigate: (a) does hive-ag3nt's serve loop survive `nixos-container update` cleanly, or does its tokio runtime get torn down mid-loop? (b) is there an early-exit path on a transient socket error during rebuild that drops the serve task without notifying the manager? (c) compare timeline with manager's own post-rebuild wake to see if this is rebuilt-agents-only or universal. Could be related to the `recv_blocking` fix in `e423d57` if the rebuild restarts the broker mid-subscribe.
|
||||||
|
|
|
||||||
|
|
@ -1240,79 +1240,6 @@
|
||||||
return s.length <= n ? s : s.slice(0, n - 1) + '…';
|
return s.length <= n ? s : s.slice(0, n - 1) + '…';
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── reminders ──────────────────────────────────────────────────────────
|
|
||||||
// Reminders aren't part of /api/state (separate sqlite table, separate
|
|
||||||
// mutation cadence). Refresh fires alongside refreshState() so a
|
|
||||||
// cancel POST or a cold load both reflect within the same tick. A
|
|
||||||
// periodic poll isn't necessary — new reminders are queued by the
|
|
||||||
// agents themselves and the operator already sees them next time
|
|
||||||
// they interact with the page.
|
|
||||||
async function refreshReminders() {
|
|
||||||
const root = $('reminders-section');
|
|
||||||
if (!root) return;
|
|
||||||
try {
|
|
||||||
const resp = await fetch('/api/reminders');
|
|
||||||
if (!resp.ok) {
|
|
||||||
root.innerHTML = '';
|
|
||||||
root.append(el('p', { class: 'empty' }, 'reminders unavailable: http ' + resp.status));
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
const rows = await resp.json();
|
|
||||||
renderReminders(rows);
|
|
||||||
} catch (err) {
|
|
||||||
root.innerHTML = '';
|
|
||||||
root.append(el('p', { class: 'empty' }, 'reminders fetch failed: ' + err));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
function renderReminders(rows) {
|
|
||||||
const root = $('reminders-section');
|
|
||||||
if (!root) return;
|
|
||||||
root.innerHTML = '';
|
|
||||||
if (!rows.length) {
|
|
||||||
root.append(el('p', { class: 'empty' }, 'no queued reminders'));
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
const ul = el('ul', { class: 'reminders' });
|
|
||||||
for (const r of rows) {
|
|
||||||
const li = el('li', { class: 'reminder-row' });
|
|
||||||
const dueIn = r.due_at - Math.floor(Date.now() / 1000);
|
|
||||||
const dueLabel = dueIn <= 0
|
|
||||||
? `overdue ${fmtAgo(r.due_at)}`
|
|
||||||
: `in ${fmtDuration(dueIn)}`;
|
|
||||||
const head = el('div', { class: 'reminder-head' },
|
|
||||||
el('span', { class: 'agent' }, r.agent), ' ',
|
|
||||||
el('span', { class: 'meta', title: new Date(r.due_at * 1000).toISOString() }, dueLabel),
|
|
||||||
' ',
|
|
||||||
el('span', { class: 'meta' }, `· id ${r.id}`),
|
|
||||||
);
|
|
||||||
if (r.file_path) {
|
|
||||||
head.append(' ', el('span', { class: 'meta' }, '· payload → '));
|
|
||||||
appendLinkified(head, r.file_path);
|
|
||||||
}
|
|
||||||
const body = el('div', { class: 'reminder-body' });
|
|
||||||
const previews = appendLinkified(body, r.message);
|
|
||||||
li.append(head, body);
|
|
||||||
for (const d of previews) li.appendChild(d);
|
|
||||||
// Cancel form omits `data-no-refresh` — the resulting refreshState
|
|
||||||
// re-fires refreshReminders so the row drops on its own.
|
|
||||||
const cancelForm = el('form', {
|
|
||||||
method: 'POST', action: '/cancel-reminder/' + r.id,
|
|
||||||
class: 'inline', 'data-async': '',
|
|
||||||
'data-confirm': `cancel reminder ${r.id} for ${r.agent}? this drops the queued delivery; no undo.`,
|
|
||||||
});
|
|
||||||
cancelForm.append(el('button', { type: 'submit', class: 'btn btn-deny' }, '✗ C4NC3L'));
|
|
||||||
li.append(cancelForm);
|
|
||||||
ul.append(li);
|
|
||||||
}
|
|
||||||
root.append(ul);
|
|
||||||
}
|
|
||||||
function fmtDuration(secs) {
|
|
||||||
if (secs < 60) return secs + 's';
|
|
||||||
if (secs < 3600) return Math.floor(secs / 60) + 'm ' + (secs % 60) + 's';
|
|
||||||
if (secs < 86400) return Math.floor(secs / 3600) + 'h ' + Math.floor((secs % 3600) / 60) + 'm';
|
|
||||||
return Math.floor(secs / 86400) + 'd ' + Math.floor((secs % 86400) / 3600) + 'h';
|
|
||||||
}
|
|
||||||
|
|
||||||
// ─── state polling ──────────────────────────────────────────────────────
|
// ─── state polling ──────────────────────────────────────────────────────
|
||||||
let pollTimer = null;
|
let pollTimer = null;
|
||||||
// Sections whose innerHTML gets blown away on each refresh. If the
|
// Sections whose innerHTML gets blown away on each refresh. If the
|
||||||
|
|
@ -1325,7 +1252,6 @@
|
||||||
'inbox-section',
|
'inbox-section',
|
||||||
'approvals-section',
|
'approvals-section',
|
||||||
'meta-inputs-section',
|
'meta-inputs-section',
|
||||||
'reminders-section',
|
|
||||||
];
|
];
|
||||||
// <details> sections that should survive a refresh need a stable
|
// <details> sections that should survive a refresh need a stable
|
||||||
// `data-restore-key` attribute. snapshotOpenDetails walks managed
|
// `data-restore-key` attribute. snapshotOpenDetails walks managed
|
||||||
|
|
@ -1400,7 +1326,6 @@
|
||||||
syncApprovalsFromSnapshot(s);
|
syncApprovalsFromSnapshot(s);
|
||||||
renderApprovals();
|
renderApprovals();
|
||||||
renderMetaInputs(s);
|
renderMetaInputs(s);
|
||||||
refreshReminders();
|
|
||||||
restoreOpenDetails(openDetails);
|
restoreOpenDetails(openDetails);
|
||||||
notifyDeltas(s);
|
notifyDeltas(s);
|
||||||
// No periodic refresh timer. Phase 6 covers every container
|
// No periodic refresh timer. Phase 6 covers every container
|
||||||
|
|
|
||||||
|
|
@ -450,27 +450,6 @@ summary:hover { color: var(--purple); }
|
||||||
0%, 100% { box-shadow: 0 0 12px -4px rgba(250, 179, 135, 0.55); }
|
0%, 100% { box-shadow: 0 0 12px -4px rgba(250, 179, 135, 0.55); }
|
||||||
50% { box-shadow: 0 0 22px -2px rgba(250, 179, 135, 0.95); }
|
50% { box-shadow: 0 0 22px -2px rgba(250, 179, 135, 0.95); }
|
||||||
}
|
}
|
||||||
/* Reminders list — rendered from /api/reminders, separate from the
|
|
||||||
main /api/state snapshot. Each row stacks identity, head meta,
|
|
||||||
body, and a small cancel form. */
|
|
||||||
.reminders {
|
|
||||||
list-style: none;
|
|
||||||
padding: 0;
|
|
||||||
margin: 0;
|
|
||||||
}
|
|
||||||
.reminder-row {
|
|
||||||
padding: 0.4em 0;
|
|
||||||
border-bottom: 1px solid var(--border);
|
|
||||||
}
|
|
||||||
.reminder-row:last-child { border-bottom: 0; }
|
|
||||||
.reminder-head { font-size: 0.9em; }
|
|
||||||
.reminder-body {
|
|
||||||
color: var(--fg);
|
|
||||||
white-space: pre-wrap;
|
|
||||||
word-break: break-word;
|
|
||||||
margin: 0.3em 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Path linkification — agents drop pointer strings into messages
|
/* Path linkification — agents drop pointer strings into messages
|
||||||
constantly; clicking the anchor expands a sibling <details> that
|
constantly; clicking the anchor expands a sibling <details> that
|
||||||
lazy-loads from /api/state-file. */
|
lazy-loads from /api/state-file. */
|
||||||
|
|
|
||||||
|
|
@ -45,13 +45,6 @@
|
||||||
<p class="meta">loading…</p>
|
<p class="meta">loading…</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<h2>◆ QU3U3D R3M1ND3RS ◆</h2>
|
|
||||||
<div class="divider">══════════════════════════════════════════════════════════════</div>
|
|
||||||
<p class="meta">reminders agents have queued for themselves but not yet delivered. cancel to drop a stuck or unwanted entry.</p>
|
|
||||||
<div id="reminders-section">
|
|
||||||
<p class="meta">loading…</p>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<h2>◆ P3NDING APPR0VALS ◆</h2>
|
<h2>◆ P3NDING APPR0VALS ◆</h2>
|
||||||
<div class="divider">══════════════════════════════════════════════════════════════</div>
|
<div class="divider">══════════════════════════════════════════════════════════════</div>
|
||||||
<div id="approvals-section">
|
<div id="approvals-section">
|
||||||
|
|
|
||||||
|
|
@ -46,19 +46,6 @@ const EVENT_CHANNEL: usize = 256;
|
||||||
/// self-documenting.
|
/// self-documenting.
|
||||||
pub type DueReminder = (String, i64, String, Option<String>);
|
pub type DueReminder = (String, i64, String, Option<String>);
|
||||||
|
|
||||||
/// Row shape for [`Broker::list_pending_reminders`], shipped on the
|
|
||||||
/// dashboard `/api/reminders` response.
|
|
||||||
#[derive(Debug, Clone, Serialize)]
|
|
||||||
pub struct PendingReminder {
|
|
||||||
pub id: i64,
|
|
||||||
pub agent: String,
|
|
||||||
pub message: String,
|
|
||||||
#[serde(skip_serializing_if = "Option::is_none")]
|
|
||||||
pub file_path: Option<String>,
|
|
||||||
pub due_at: i64,
|
|
||||||
pub created_at: i64,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Intra-process broker event. `recv_blocking` listens on the same
|
/// Intra-process broker event. `recv_blocking` listens on the same
|
||||||
/// channel as the dashboard forwarder; the forwarder re-emits each
|
/// channel as the dashboard forwarder; the forwarder re-emits each
|
||||||
/// event as a `DashboardEvent` with a freshly-stamped seq from the
|
/// event as a `DashboardEvent` with a freshly-stamped seq from the
|
||||||
|
|
@ -299,44 +286,6 @@ impl Broker {
|
||||||
Ok(id)
|
Ok(id)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Every reminder still pending delivery, newest-first. Used by the
|
|
||||||
/// dashboard's reminders pane so the operator can see what's queued
|
|
||||||
/// + cancel rows that are no longer wanted.
|
|
||||||
pub fn list_pending_reminders(&self) -> Result<Vec<PendingReminder>> {
|
|
||||||
let conn = self.conn.lock().unwrap();
|
|
||||||
let mut stmt = conn.prepare(
|
|
||||||
"SELECT id, agent, message, file_path, due_at, created_at \
|
|
||||||
FROM reminders \
|
|
||||||
WHERE sent_at IS NULL \
|
|
||||||
ORDER BY due_at ASC",
|
|
||||||
)?;
|
|
||||||
let rows = stmt.query_map([], |row| {
|
|
||||||
Ok(PendingReminder {
|
|
||||||
id: row.get(0)?,
|
|
||||||
agent: row.get(1)?,
|
|
||||||
message: row.get(2)?,
|
|
||||||
file_path: row.get(3)?,
|
|
||||||
due_at: row.get(4)?,
|
|
||||||
created_at: row.get(5)?,
|
|
||||||
})
|
|
||||||
})?;
|
|
||||||
rows.collect::<rusqlite::Result<Vec<_>>>()
|
|
||||||
.context("list pending reminders")
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Delete a reminder by id. Returns the number of rows removed (0
|
|
||||||
/// when the id never existed or was already delivered). Hard
|
|
||||||
/// delete rather than soft so the row doesn't linger and confuse a
|
|
||||||
/// re-creation under the same id.
|
|
||||||
pub fn cancel_reminder(&self, id: i64) -> Result<usize> {
|
|
||||||
let conn = self.conn.lock().unwrap();
|
|
||||||
let n = conn.execute(
|
|
||||||
"DELETE FROM reminders WHERE id = ?1 AND sent_at IS NULL",
|
|
||||||
params![id],
|
|
||||||
)?;
|
|
||||||
Ok(n)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Get up to `limit` due reminders across all agents in a single query.
|
/// Get up to `limit` due reminders across all agents in a single query.
|
||||||
/// Returns `(agent, id, message, file_path)` tuples. Pass a small limit
|
/// Returns `(agent, id, message, file_path)` tuples. Pass a small limit
|
||||||
/// (e.g. 100) so a burst of overdue reminders doesn't flood the broker
|
/// (e.g. 100) so a burst of overdue reminders doesn't flood the broker
|
||||||
|
|
|
||||||
|
|
@ -55,8 +55,6 @@ pub async fn serve(port: u16, coord: Arc<Coordinator>) -> Result<()> {
|
||||||
.route("/purge-tombstone/{name}", post(post_purge_tombstone))
|
.route("/purge-tombstone/{name}", post(post_purge_tombstone))
|
||||||
.route("/api/journal/{name}", get(get_journal))
|
.route("/api/journal/{name}", get(get_journal))
|
||||||
.route("/api/state-file", get(get_state_file))
|
.route("/api/state-file", get(get_state_file))
|
||||||
.route("/api/reminders", get(api_reminders))
|
|
||||||
.route("/cancel-reminder/{id}", post(post_cancel_reminder))
|
|
||||||
.route("/api/agent-config/{name}", get(get_agent_config))
|
.route("/api/agent-config/{name}", get(get_agent_config))
|
||||||
.route("/request-spawn", post(post_request_spawn))
|
.route("/request-spawn", post(post_request_spawn))
|
||||||
.route("/op-send", post(post_op_send))
|
.route("/op-send", post(post_op_send))
|
||||||
|
|
@ -985,27 +983,6 @@ async fn get_state_file(
|
||||||
([("content-type", "text/plain; charset=utf-8")], body).into_response()
|
([("content-type", "text/plain; charset=utf-8")], body).into_response()
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn api_reminders(State(state): State<AppState>) -> Response {
|
|
||||||
match state.coord.broker.list_pending_reminders() {
|
|
||||||
Ok(rows) => axum::Json(rows).into_response(),
|
|
||||||
Err(e) => error_response(&format!("reminders: {e:#}")),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn post_cancel_reminder(
|
|
||||||
State(state): State<AppState>,
|
|
||||||
AxumPath(id): AxumPath<i64>,
|
|
||||||
) -> Response {
|
|
||||||
match state.coord.broker.cancel_reminder(id) {
|
|
||||||
Ok(0) => error_response(&format!("reminder {id} not pending (already delivered?)")),
|
|
||||||
Ok(_) => {
|
|
||||||
tracing::info!(%id, "operator cancelled reminder");
|
|
||||||
(StatusCode::OK, "ok").into_response()
|
|
||||||
}
|
|
||||||
Err(e) => error_response(&format!("cancel reminder {id} failed: {e:#}")),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn post_purge_tombstone(
|
async fn post_purge_tombstone(
|
||||||
State(state): State<AppState>,
|
State(state): State<AppState>,
|
||||||
AxumPath(name): AxumPath<String>,
|
AxumPath(name): AxumPath<String>,
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue