diff --git a/CLAUDE.md b/CLAUDE.md index e1f2e5b0..aea318d1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -114,17 +114,46 @@ read them à la carte. In-flight or recent context that hasn't earned a section yet. Prune freely. -- 2026-05-15 ish: tombstones, multi-select ask_operator, broker + - events vacuum, docs split into `docs/`, lifecycle_action helper, - api_state split. -- Then: inline +/- diffs on Write/Edit, operator cancel + ttl on - questions, dashboard back-link, per-agent inbox view, bind-retry - + SO_REUSEADDR, journald viewer, server-side TurnState, - recv(wait_seconds) max 180s, runtime /model switch, crash - watcher, model persistence, stopped auto-allowing claude-code - unfree (operator must opt in), pure-hash agent_web_port (port - files reverted), browser notifications, focus-preserving - refresh. -- Open threads: telemetry/charts, custom per-agent MCP tools (the - groundwork for moving bitburner-agent into hyperhive), - two-step spawn, unprivileged containers, Bash allow-list. +- **Just landed:** tag-driven config-apply overhaul. Two-repo + split kept (proposed = manager RW, applied = core-only) for + safety. New flow: at `request_apply_commit` time hive-c0re + fetches the manager's commit into applied and pins it as + `proposal/`; the manager-side repo is then irrelevant + for that approval. Approve / deny / build walk through more + tags (`approved/`, `building/`, `deployed/`, `failed/`, + `denied/`) on the same commit; `applied/main` only + fast-forwards on `deployed/`. `failed/` and `denied/` are + annotated — body is the build error or the operator's deny + note respectively. Manager has `/applied` bind-mounted RO + (whole tree) so `git fetch /applied//.git + 'refs/tags/*:refs/tags/applied/*'` mirrors every relevant + tag into its proposed clone. `agent.nix` stays the entry + point; the whole tracked tree is now preserved + through apply (arbitrary files supported). The wrapper + `flake.nix` is regenerated by hive-c0re every + spawn/rebuild but never tracked, so the applied log is + exactly the manager's commits in deploy order. Migration: + no in-place — pre-overhaul applied dirs are detected via + the missing `deployed/0` tag and `setup_applied` bails + with `destroy --purge` instructions. See + `docs/approvals.md`. +- **Recent (since last compaction):** inline +/- diffs on + Write/Edit, send full body via collapsed details, operator + cancel + ttl on questions, deny-with-reason, dashboard + back-link + last-turn timing + model chip, per-agent inbox + view, bind-retry + SO_REUSEADDR, journald viewer, + agent.nix viewer, server-side TurnState, recv(wait_seconds) + max 180s, runtime /model switch + persistence to /state, + crash watcher + ContainerCrash / NeedsLogin / LoggedIn / + NeedsUpdate events, manager `update` tool, pure-hash + agent_web_port + collision banner + spawn/rebuild preflight, + browser notifications, focus-preserving refresh, generalised +
survival, prompt-on-submit pattern. +- **Open threads:** custom per-agent MCP tools (groundwork for + moving bitburner-agent into hyperhive), two-step spawn, + per-agent send allow-list, telemetry/charts, notes + compaction, unprivileged containers, Bash allow-list, + xterm.js. **Known bug** (in TODO.md): question id=5 was + queued but didn't render — likely a `pending()` row-decode + error swallowed by `unwrap_or_default`; investigate by curl + /api/state | jq '.questions' + browser console. diff --git a/README.md b/README.md index 5d83a5aa..15d2926e 100644 --- a/README.md +++ b/README.md @@ -30,8 +30,8 @@ host (NixOS, runs hive-c0re.service) │ ├── hm1nd hive-m1nd serve : claude turn loop + │ MCP (send / recv / request_spawn / kill / start / - │ restart / request_apply_commit / ask_operator) - │ + web UI on :8000 + │ restart / update / request_apply_commit / + │ ask_operator) + web UI on :8000 │ └── h- hive-ag3nt serve : claude turn loop + MCP (send / recv) + web UI on a hashed :8100-8999 @@ -44,10 +44,13 @@ streams JSON events into the per-agent SSE bus + a sqlite history db → claude drives any further `recv`/`send` itself via the embedded MCP server. Operator surface per agent: terminal-themed live tail with a textarea -prompt; slash commands `/help` `/clear` `/cancel` `/compact`; granular -state badge (idle / thinking / offline) with age timer; cancel-turn -button while thinking; sticky-bottom auto-scroll with "↓ N new" pill; -event history backfilled on page load. +prompt; slash commands `/help` `/clear` `/cancel` `/compact` +`/model `; granular state badge (idle / thinking / +compacting / offline) with age timer + last-turn duration chip + +model chip; cancel-turn button while thinking; sticky-bottom +auto-scroll with "↓ N new" pill; event history backfilled on page +load; collapsible inbox + collapsible journald viewer + collapsible +`agent.nix` viewer per agent on the dashboard. Config changes flow the other way: manager edits `/agents//config/agent.nix` (bind-mounted from the host's proposed repo) → commits → submits the sha as diff --git a/TODO.md b/TODO.md index 28ff89ee..a39b90d6 100644 --- a/TODO.md +++ b/TODO.md @@ -3,6 +3,21 @@ Pick anything from here when relevant. Cross-cutting design notes live in [CLAUDE.md](CLAUDE.md); high-level project intro in [README.md](README.md). +## Turn loop + +- **`recv` with no `wait_seconds` should return immediately.** + Today omitting the argument falls through to the 30s + default long-poll (`RECV_LONG_POLL_DEFAULT` in + `hive-c0re/src/agent_server.rs`); a manager that wants a + cheap "anything in the inbox right now?" peek has to + explicitly pass `wait_seconds: 0`. Flip the semantics so + `None` = no sleep, returning `None` (or the empty inbox + shape) right away. The agent opts into the long-poll by + setting a positive value. Update both `AgentRequest::Recv` + and `ManagerRequest::Recv` handlers + the prompt language + in `prompts/{agent,manager}.md`. Tighten the cap (180s) + too — only meaningful when the agent is choosing to wait. + ## Permissions / policy - **Per-agent send allow-list.** Today any agent can `send` to any @@ -21,7 +36,12 @@ Pick anything from here when relevant. Cross-cutting design notes live in nixos-container equivalent) so uid 0 inside maps to an unprivileged uid on the host, and a container-root compromise lands the attacker on an ordinary user account, not the host's root. Requires per-agent state - dirs to be chown'd to that uid on the host side. + dirs to be chown'd to that uid on the host side. The per-agent git + identity (currently injected via `programs.git.config.user` against + the root user in `setup_applied`'s generated flake) also needs to be + provisioned for whatever non-root user claude runs as, or commits + the manager makes against `/agents//config` will fall back to a + generic `nixos@…` identity. - **Bash command allow-list.** Replace the blanket `Bash` allow with a pattern allow-list (`Bash(git *)`, `Bash(nix build .*)`, etc.) per claude-code's `--allowedTools` extended grammar. Likely lives in @@ -42,8 +62,35 @@ Pick anything from here when relevant. Cross-cutting design notes live in derived from the same config so the operator stays in control of what's exposed. +## Bugs + +- **Pending question doesn't always appear on the dashboard.** + Repro: manager calls `ask_operator`, tool result is + `question queued (id=N)` (so the row is in sqlite), but the + M1ND H4S QU3STI0NS section keeps showing "no pending + questions". Last seen with id=5. Suspected paths: + - `OperatorQuestions::pending()` returns Err and the + `unwrap_or_default()` in `api_state` hides it. Surface the + error (warn-log) and check. + - serialization: a new field in `OpQuestion` (e.g. + `deadline_at: Option`) deserializes wrong against an + old row whose columns don't match the new SELECT order → + `row.get(N)?` panics for that row, the whole iterator + errors, `pending()` returns Err. Diagnose by curl + `/api/state | jq '.questions'` and compare with sqlite + counts. + - dashboard JS swallows a render error. Open browser console + and look for exceptions during `renderQuestions`. + ## UI / UX +- **Web UI for config repos.** Browse history, diffs, tags + (proposed + approval/* + applied/*) per agent, all from the + dashboard. Something lighter than a full forge — read-only + log + diff + raw-file view is enough. Pairs naturally with + the upcoming config-repo overhaul (tags become the audit + trail; UI surfaces them). + - **xterm.js terminal** embedded per-agent, attached to a PTY exposed by the harness. Pairs well with the unprivileged-container work — would let the operator drop into the container without `nixos-container root-login`. diff --git a/docs/approvals.md b/docs/approvals.md index f892838b..47fa5942 100644 --- a/docs/approvals.md +++ b/docs/approvals.md @@ -8,15 +8,29 @@ happens after a decision lands. ## End-to-end approval flow -1. Manager edits `/agents//config/agent.nix` (bind-mounted - from the host's per-agent `proposed` repo) and commits. +1. Manager edits files under `/agents//config/` (any tracked + path, but `agent.nix` is the contract entry point) and commits + with its own git identity. 2. Manager submits the commit sha via `request_apply_commit(agent, commit_ref)`. -3. Operator sees the diff on the dashboard, clicks ◆ APPR0VE (or +3. **hive-c0re immediately fetches that commit from the proposed + repo into the applied repo and tags it `proposal/`.** The + approval row stores both the manager-supplied sha and the + canonical hive-c0re-vouched sha. From here on the proposed + repo is irrelevant for this approval — the manager can amend, + force-push, or `rm -rf` the proposed repo and the queued + approval still points at an immutable git object inside + applied. +4. Operator sees the diff on the dashboard, clicks ◆ APPR0VE (or `hive-c0re approve ` on the CLI). -4. hive-c0re reads the file at that sha from `proposed`, applies - into `applied`, commits there, runs `nixos-container update`. -5. `HelperEvent::ApprovalResolved` lands in the manager's inbox. +5. hive-c0re moves the working tree to `proposal/` and runs + the build under a sequence of tags (see below). On success, + `applied/main` fast-forwards to the proposal commit. On + failure, main stays put and the working tree resets back to + the previous deployed commit. +6. `HelperEvent::ApprovalResolved` (and `Rebuilt` for the + ApplyCommit kind) land in the manager's inbox, carrying both + the canonical sha and the terminal tag. `Spawn` approvals follow the same shape but skip the commit-diff step — the operator just sees the name. On approve, hive-c0re @@ -26,27 +40,83 @@ shows a spinner. ## Two repos per agent ``` -/var/lib/hyperhive/agents//config/ proposed -└── agent.nix # the only file the - # manager can change - # (initial commit by - # hive-c0re on first - # spawn, never touched - # again). +/var/lib/hyperhive/agents//config/ proposed — manager RW +└── # any files the manager + # wants in the commit. + # agent.nix is the + # convention entry + # point; flake.nix is + # generated and not + # tracked here. -/var/lib/hyperhive/applied// applied — hive-c0re-only -├── flake.nix # auto-generated -└── agent.nix # overwritten by approve - # from the proposed commit +/var/lib/hyperhive/applied// applied — core-only +├── .git/ # tag-rich history +├── .gitignore # ignores flake.nix +├── flake.nix # hive-c0re-generated, +│ # untracked, rewritten +│ # on spawn/rebuild only +├── agent.nix # working tree of main +└── # also tracked ``` -The container's `--flake` ref is `#default`. The flake -extends `hyperhive.nixosConfigurations.{agent-base|manager}` with +Why two physical repos: the manager's `/agents//config/` is +RW — a buggy or hostile agent can `git clean -fdx` its own +proposed tree. The applied repo is never bind-mounted (except +the read-only `.git` exposure described below) so a destructive +move inside the container cannot reach it. + +The container's `--flake` ref is `#default`. The +generated `flake.nix` extends +`hyperhive.nixosConfigurations.{agent-base|manager}` with `./agent.nix` plus an inline module setting `programs.git.config.user` (committer identity = the agent's name) and `systemd.services..environment` (`HIVE_PORT`, `HIVE_LABEL`, `HIVE_DASHBOARD_PORT`). +### Tag state machine + +Every approval id walks through a fixed set of tags on the +underlying commit inside the applied repo: + +| Tag | When | Annotated? | +|---|---|---| +| `proposal/` | request_apply_commit, after fetch | no | +| `approved/` | operator approve | no | +| `building/` | rebuild started | no | +| `deployed/` | rebuild succeeded — `main` ff's here | no | +| `failed/` | rebuild failed | yes (body = error) | +| `denied/` | operator deny | yes (body = operator note) | + +`applied/main` is always the latest `deployed/*`. `denied/` and +`failed/` are terminal; the manager submits a new commit + new +approval id to retry. Because tags are first-class git objects, +rejected and failed trees stay browsable forever — `git log +--tags` in the applied repo is the audit trail. + +### Manager view of applied + +`/applied/` is a **read-only bind-mount** of +`/var/lib/hyperhive/applied/` (the entire tree) inside the +manager container. The manager fetches tags into its proposed +clone with `git fetch /applied//.git +'refs/tags/*:refs/tags/applied/*'` and `git show` any +deployed / failed / denied tree to see what actually shipped, +what error blocked the last build, or what note the operator +left on a denial. The RO bind means git plumbing inside the +manager cannot corrupt the applied repos — and a single mount +covers every agent (existing + future) without rebuilding the +manager on each spawn. + +## Migration from the pre-tag scheme + +There is no in-place migration. Each existing agent must be +purged and re-spawned: `hive-c0re destroy --purge ` (or +PURG3 on the dashboard), then `request_spawn` and the operator +approves the fresh agent. The new agent starts with `deployed/0` +seeded by hive-c0re; the manager's first config edit becomes +`proposal/1` and walks the tag scheme from there. Pre-overhaul +tombstones lose their config history. + ## Manager (`hm1nd`) is hive-c0re-managed The manager container runs through the **same lifecycle as @@ -64,9 +134,11 @@ Differences from sub-agents: (vs `agent-base`). - Container name is `hm1nd` (no `h-` prefix). - Fixed web UI port (`MANAGER_PORT = 8000`). -- `set_nspawn_flags` adds an extra bind: - `/var/lib/hyperhive/agents` → `/agents` (RW), so the manager can - edit per-agent proposed repos. +- `set_nspawn_flags` adds two extra binds: `/var/lib/hyperhive/agents` + → `/agents` (RW) so the manager can edit per-agent proposed repos, + and `/var/lib/hyperhive/applied` → `/applied` (RO) so the manager + can `git fetch` deployed/failed/denied tags from any agent's + authoritative applied repo (see "Manager view of applied" below). - First-deploy spawn bypasses the approval queue (manager is required infrastructure). - Per-agent socket lives at `/run/hyperhive/manager/`, owned by diff --git a/docs/conventions.md b/docs/conventions.md index 2f20c150..8a709ec4 100644 --- a/docs/conventions.md +++ b/docs/conventions.md @@ -34,8 +34,15 @@ Dashboard + per-agent mutating forms carry `data-async`; a delegated `submit` listener in `assets/app.js` intercepts, shows a spinner, POSTs `application/x-www-form-urlencoded` (axum's `Form` extractor rejects multipart), calls `refreshState()` on success. New mutating -forms should add `data-async` and optionally `data-confirm` for a -JS-side confirmation prompt. +forms should add `data-async` and optionally `data-confirm` (for a +JS-side `confirm()` prompt) or `data-prompt="…"` (for a +`window.prompt()` whose answer goes into a hidden input named by +`data-prompt-field`, default `note`). + +`refreshState` defers automatically when `document.activeElement` +sits inside a managed section so the operator's typing isn't lost; +collapsible `
` survive the re-render +via `snapshotOpenDetails` / `restoreOpenDetails`. ## `rebuild` is the reconcile verb diff --git a/docs/persistence.md b/docs/persistence.md index 01dd6852..69cfe6b9 100644 --- a/docs/persistence.md +++ b/docs/persistence.md @@ -14,7 +14,8 @@ Three tables, all in one file: commit_ref / requested_at / status / resolved_at / note`. - `operator_questions` — `ask_operator` queue. `asker / question / options_json / multi / asked_at / - answered_at / answer`. + deadline_at (ttl) / answered_at / answer`. Migrated via + `ALTER TABLE ADD COLUMN` against `pragma_table_info`. Retention: diff --git a/docs/web-ui.md b/docs/web-ui.md index a39906af..c333a845 100644 --- a/docs/web-ui.md +++ b/docs/web-ui.md @@ -30,6 +30,16 @@ and, if so, skips the refresh (defers 2s). The operator never has the form yanked out from under them mid-type; the update lands as soon as they blur. +**`
` open-state preservation:** any collapsible element +tagged with `data-restore-key=""` survives the +refresh. `snapshotOpenDetails()` walks managed sections before +render, `restoreOpenDetails()` re-applies after. Used today for +the journald viewer (`journal:`), the agent-config +viewer (`agent-config:`), and approval diff blocks +(`approval-diff:`). Setting `.open = true` programmatically +also fires the `toggle` event, so any lazy-fetch wired to it +re-runs cleanly on restore. + Both bind their listeners with `SO_REUSEADDR` via `tokio::net::TcpSocket` plus a retry loop on `AddrInUse` (12 tries, exponential backoff capped at 2s) so an nspawn restart that races @@ -42,6 +52,11 @@ the previous process's socket release resolves itself. inline "unsupported / blocked" message when applicable. Sits under the banner. 2. **C0NTAINERS** — live containers with their action surface. + Pulsing red banner at the top of this section if any two + sub-agents hash to the same port (`port_conflicts` from + `/api/state`): the operator must rename one of them and + rebuild. `lifecycle::{spawn,rebuild}` also preflight this and + refuse with a clear error message naming the conflicting agent. 3. **K3PT ST4T3** — destroyed-but-state-kept tombstones (size + age + claude-creds badge). Two actions: `⊕ R3V1V3` (queues a Spawn approval; existing state is reused), `PURG3` (wipes @@ -73,12 +88,16 @@ Two-line layout (`assets/app.js::renderContainers`): on sub-agents, `↺ R3ST4RT` + (sub-agents) `■ ST0P` when running, `▶ ST4RT` when stopped. Buttons dim + disable while a transient lifecycle action is in flight. -- Plus a collapsible `↳ logs · ` `
` block. - Expanding lazy-fetches journald output via `GET - /api/journal/{name}?unit=...&lines=...` (`journalctl -M - -b --no-pager --output=short-iso`). A unit dropdown - switches between the harness service (default) and the full - machine journal; refresh button re-fetches. +- Plus two collapsible `
` blocks: + - `↳ logs · ` — lazy-fetches journald output via + `GET /api/journal/{name}?unit=...&lines=...` (`journalctl -M + -b --no-pager --output=short-iso`). A unit + dropdown switches between the harness service (default) and + the full machine journal; refresh button re-fetches. + - `↳ agent.nix · ` — lazy-fetches the applied config + file via `GET /api/agent-config/{name}` (read-only mirror of + `/var/lib/hyperhive/applied//agent.nix`). Mutating + this still requires `request_apply_commit` + approval. `↻ UPD4TE 4LL` button appears above the containers list when any agent is stale. Banner pulses on each broker SSE event @@ -94,15 +113,27 @@ Pure frontend (`Notification` API). Three signals trigger them: First `/api/state` after page load seeds "seen" sets without firing — only items that arrive while the page is open count. -`tag: "hyperhive"` collapses bursts; click focuses the dashboard -tab. localStorage-backed mute toggle silences without revoking -the OS permission. Requires a secure context (HTTPS or -localhost); on other origins the controls hide themselves. +Per-event tags (`hyperhive:approval:`, `hyperhive:question:`, +`hyperhive:msg::`) so distinct events stack in the OS +notification center instead of overwriting each other. +`console.debug` logs at every block point (unsupported, +permission ungranted, muted) for in-browser debugging. Click +focuses the dashboard tab. localStorage-backed mute toggle +silences without revoking the OS permission. Requires a secure +context (HTTPS or localhost); on other origins the controls hide +themselves. Browsers typically suppress notifications while the +originating tab is focused — that's a browser-level decision, +not ours. ### Dashboard endpoints -- `POST /{approve,deny}/{id}` — approve/deny a pending approval. +- `POST /approve/{id}` — approve a pending approval. +- `POST /deny/{id}` (`note=`, optional) — deny a pending + approval with an optional operator-supplied reason. The reason + travels to the manager as `HelperEvent::ApprovalResolved.note`. + Dashboard prompts via `window.prompt()` on click. - `POST /{rebuild,kill,restart,start,destroy}/{name}` — lifecycle. + `destroy` accepts `purge=on` to also wipe state dirs. - `POST /purge-tombstone/{name}` — wipe a tombstone's state dirs. - `POST /answer-question/{id}` — answer a pending operator question. - `POST /cancel-question/{id}` — cancel a pending question with @@ -111,6 +142,13 @@ localhost); on other origins the controls hide themselves. - `POST /update-all` — rebuild every stale container. - `GET /api/journal/{name}?unit=&lines=` — journalctl viewer for a managed container. +- `GET /api/agent-config/{name}` — read-only view of the applied + `agent.nix`. + +Generalised form helpers: `form[data-confirm="…"]` pops +`confirm()` before submit; `form[data-prompt="…"]` pops +`prompt()` and stashes the answer in a hidden input named by +`data-prompt-field` (default `note`). ## Per-agent page @@ -134,7 +172,9 @@ Layout, top to bottom: POSTs `/api/cancel`. - Inbox `
` block (collapsed): `inbox · N` — last 30 messages addressed to this agent, fetched via - `AgentRequest::Recent { limit: 30 }`. + `AgentRequest::Recent { limit: 30 }`. (Separate from + `AgentRequest::Recv { wait_seconds }` which the harness uses + internally to long-poll the broker.) - Terminal-wrap: live event tail (sticky-bottom auto-scroll + `↓ N new` pill when not at bottom) followed by an operator-input textarea acting as a prompt. diff --git a/hive-ag3nt/prompts/manager.md b/hive-ag3nt/prompts/manager.md index 5ae01faa..67f854ce 100644 --- a/hive-ag3nt/prompts/manager.md +++ b/hive-ag3nt/prompts/manager.md @@ -9,12 +9,21 @@ Tools (hyperhive surface): - `mcp__hyperhive__start(name)` — start a stopped sub-agent. No approval required. - `mcp__hyperhive__restart(name)` — stop + start a sub-agent. No approval required. - `mcp__hyperhive__update(name)` — rebuild a sub-agent (re-applies the current hyperhive flake + agent.nix, restarts the container). No approval required — idempotent. Use when you receive a `needs_update` system event. -- `mcp__hyperhive__request_apply_commit(agent, commit_ref)` — submit a config change for any agent (`hm1nd` for self) for operator approval. +- `mcp__hyperhive__request_apply_commit(agent, commit_ref)` — submit a config change for any agent (`hm1nd` for self) for operator approval. At submit time hive-c0re fetches your commit into the agent's applied repo and pins it as `proposal/`; from that moment your proposed-side commit can be amended or force-pushed freely without changing what the operator will build. - `mcp__hyperhive__ask_operator(question, options?, multi?, ttl_seconds?)` — surface a question on the dashboard. Returns immediately with a question id; the operator's answer arrives later as a system `operator_answered` event in your inbox. Options are advisory: the dashboard always lets the operator type a free-text answer in addition. Set `multi: true` to render options as checkboxes (operator can pick multiple); the answer comes back as `, `-separated. Set `ttl_seconds` to auto-cancel after a deadline — useful when the decision becomes moot if the operator hasn't responded in time; on expiry the answer is `[expired]`. Do not poll inside the same turn — finish the current work and react when the event lands. Approval boundary: lifecycle ops on *existing* sub-agents (`kill`, `start`, `restart`) are at your discretion — no operator approval. *Creating* a new agent (`request_spawn`) and *changing* any agent's config (`request_apply_commit`) still go through the approval queue. The operator only signs off on changes; you run the day-to-day. -Your own editable config lives at `/agents/hm1nd/config/agent.nix`; every sub-agent's lives at `/agents//config/agent.nix`. Use file/git tools to edit + commit, then `request_apply_commit`. +Your own editable config lives at `/agents/hm1nd/config/`; every sub-agent's lives at `/agents//config/`. `agent.nix` is the entry point but you can commit any extra files (modules, overlays, prompt fragments) and the whole tree gets deployed together. Use file/git tools to edit + commit, then `request_apply_commit`. + +To see what hive-c0re actually deployed (or rejected, or failed to build), there's a read-only mirror of every agent's applied repo at `/applied//.git`. Useful patterns: + +- `git -C /agents//config fetch /applied//.git 'refs/tags/*:refs/tags/applied/*'` — mirror all tags into your proposed clone. +- `git -C /agents//config show applied/deployed/` — see the tree that's currently running. +- `git -C /agents//config show applied/failed/` — annotated tag body is the build error from a rejected rebuild. +- `git -C /agents//config show applied/denied/` — annotated tag body is the operator's reason for denial. + +Tag scheme on every approval id: `proposal → approved → building → deployed | failed`, plus `denied` as a terminal alternative to `approved`. `applied/main` only advances on `deployed/*`, so a failed build does not corrupt the agent — submit a fix as a new commit and a fresh `request_apply_commit`. Sub-agents are NOT trusted by default. When one asks for a config change (new packages, env vars, etc.), verify the request before staging: diff --git a/hive-c0re/assets/app.js b/hive-c0re/assets/app.js index c367693c..d000976f 100644 --- a/hive-c0re/assets/app.js +++ b/hive-c0re/assets/app.js @@ -215,6 +215,18 @@ const root = $('containers-section'); root.innerHTML = ''; + // Port-hash collisions: rename one of the listed agents and + // rebuild. The banner sits above the agent list so it's the + // first thing the operator sees when something's wedged. + if (s.port_conflicts && s.port_conflicts.length) { + const banner = el('div', { class: 'port-conflict' }, + el('strong', {}, '⚠ port collision'), ' — '); + const groups = s.port_conflicts.map((c) => + `:${c.port} (${c.agents.join(' + ')})`).join('; '); + banner.append(groups + '. rename one of each and ↻ R3BU1LD.'); + root.append(banner); + } + if (s.any_stale) { root.append(form( '/update-all', 'btn-rebuild', '↻ UPD4TE 4LL', diff --git a/hive-c0re/assets/dashboard.css b/hive-c0re/assets/dashboard.css index 0245ac61..b487d500 100644 --- a/hive-c0re/assets/dashboard.css +++ b/hive-c0re/assets/dashboard.css @@ -193,6 +193,22 @@ a:hover { /* Notification controls — sit between the banner and the containers section. Hidden by JS when notifications are unsupported, denied, or already in the right state. */ +/* Port-collision banner: appears above the containers list when + two sub-agents hash to the same web UI port. Critical — without + resolution, one of the harnesses will restart-loop on + AddrInUse. */ +.port-conflict { + background: rgba(243, 139, 168, 0.08); + border: 1px solid var(--red); + color: var(--red); + padding: 0.5em 0.8em; + margin-bottom: 0.6em; + border-radius: 4px; + text-shadow: 0 0 6px rgba(243, 139, 168, 0.4); + animation: questions-pulse 2.4s ease-in-out infinite; +} +.port-conflict strong { color: var(--red); } + .notif-row { display: flex; gap: 0.5em; diff --git a/hive-c0re/src/actions.rs b/hive-c0re/src/actions.rs index 00318fd8..fc18d292 100644 --- a/hive-c0re/src/actions.rs +++ b/hive-c0re/src/actions.rs @@ -41,21 +41,16 @@ pub async fn approve(coord: Arc, id: i64) -> Result<()> { match approval.kind { ApprovalKind::ApplyCommit => { - let result = async { - lifecycle::apply_commit(&applied_dir, &proposed_dir, &approval.commit_ref).await?; - lifecycle::rebuild( - &approval.agent, - &coord.hyperhive_flake, - &agent_dir, - &applied_dir, - &claude_dir, - ¬es_dir, - coord.dashboard_port, - ) - .await - } + let (result, terminal_tag) = run_apply_commit( + &coord, + &approval, + &agent_dir, + &applied_dir, + &claude_dir, + ¬es_dir, + ) .await; - finish_approval(&coord, &approval, result) + finish_approval(&coord, &approval, result, terminal_tag) } ApprovalKind::Spawn => { // Run the spawn in the background so the approve POST returns @@ -77,7 +72,7 @@ pub async fn approve(coord: Arc, id: i64) -> Result<()> { ) .await; coord_bg.clear_transient(&agent_bg); - if let Err(e) = finish_approval(&coord_bg, &approval_bg, result) { + if let Err(e) = finish_approval(&coord_bg, &approval_bg, result, None) { tracing::warn!(agent = %agent_bg, error = ?e, "spawn approval failed"); } }); @@ -90,6 +85,7 @@ fn finish_approval( coord: &Coordinator, approval: &hive_sh4re::Approval, result: Result<()>, + terminal_tag: Option, ) -> Result<()> { let (status, note, ok) = match &result { Ok(()) => (ApprovalStatus::Approved, None, true), @@ -105,6 +101,8 @@ fn finish_approval( commit_ref: approval.commit_ref.clone(), status, note: note.clone(), + sha: approval.fetched_sha.clone(), + tag: terminal_tag.clone(), }); // For spawn/rebuild approvals, also surface the underlying action so // the manager knows whether the container actually came up. The @@ -116,16 +114,116 @@ fn finish_approval( agent: approval.agent.clone(), ok, note, + sha: approval.fetched_sha.clone(), }), ApprovalKind::ApplyCommit => coord.notify_manager(&HelperEvent::Rebuilt { agent: approval.agent.clone(), ok, note, + sha: approval.fetched_sha.clone(), + tag: terminal_tag, }), } result } +/// Tag-driven `ApplyCommit` handler. Walks the approval through the tag +/// state machine documented in `docs/approvals.md`: stamp +/// `approved/` and `building/` first so the audit trail +/// captures intent, then drop the candidate tree into the working dir +/// without moving HEAD, run the rebuild, and either fast-forward +/// `applied/main` to the proposal commit on success +/// (`deployed/`) or annotate `failed/` with the build error +/// and reset the working tree back to the last known-good main. main +/// never advances on a failed build, so a crash-and-recover doesn't +/// leave the agent pointing at a tree it can't evaluate. +async fn run_apply_commit( + coord: &Arc, + approval: &hive_sh4re::Approval, + agent_dir: &std::path::Path, + applied_dir: &std::path::Path, + claude_dir: &std::path::Path, + notes_dir: &std::path::Path, +) -> (Result<()>, Option) { + let id = approval.id; + let proposal_ref = format!("refs/tags/proposal/{id}"); + // Defensive: submit-time should have planted proposal/, but if + // the row was migrated from an older schema or the tag got pruned + // we fail early with a clear note rather than building a stale + // tree. + if let Err(e) = lifecycle::git_rev_parse(applied_dir, &proposal_ref).await { + return ( + Err(anyhow::anyhow!( + "missing proposal tag {proposal_ref}: {e:#}" + )), + None, + ); + } + if let Err(e) = lifecycle::git_tag(applied_dir, &format!("approved/{id}"), &proposal_ref).await + { + return (Err(anyhow::anyhow!("plant approved/{id}: {e:#}")), None); + } + if let Err(e) = lifecycle::git_tag(applied_dir, &format!("building/{id}"), &proposal_ref).await + { + return (Err(anyhow::anyhow!("plant building/{id}: {e:#}")), None); + } + if let Err(e) = lifecycle::git_read_tree_reset(applied_dir, &proposal_ref).await { + return ( + Err(anyhow::anyhow!("read-tree to {proposal_ref}: {e:#}")), + None, + ); + } + + let rebuild_result = lifecycle::rebuild( + &approval.agent, + &coord.hyperhive_flake, + agent_dir, + applied_dir, + claude_dir, + notes_dir, + coord.dashboard_port, + ) + .await; + + match rebuild_result { + Ok(()) => { + let tag = format!("deployed/{id}"); + if let Err(e) = lifecycle::git_tag(applied_dir, &tag, &proposal_ref).await { + tracing::warn!(agent = %approval.agent, %id, error = ?e, "plant deployed tag failed"); + } + if let Err(e) = + lifecycle::git_update_ref(applied_dir, "refs/heads/main", &proposal_ref).await + { + // Working tree already matches proposal/, but main + // didn't advance — surface as a build failure so the + // operator notices the desync. + return ( + Err(anyhow::anyhow!("ff main to {proposal_ref}: {e:#}")), + Some(tag), + ); + } + (Ok(()), Some(tag)) + } + Err(e) => { + let tag = format!("failed/{id}"); + let body = format!("{e:#}"); + if let Err(te) = + lifecycle::git_tag_annotated(applied_dir, &tag, &proposal_ref, &body).await + { + tracing::warn!(agent = %approval.agent, %id, error = ?te, "annotate failed tag failed"); + } + // Roll working tree back to last known-good main so the + // on-disk state matches what nixos-container last + // successfully built. main hasn't moved, so this is just + // a content reset. + if let Err(re) = lifecycle::git_read_tree_reset(applied_dir, "refs/heads/main").await { + tracing::warn!(agent = %approval.agent, %id, error = ?re, "rollback read-tree failed"); + } + (Err(e), Some(tag)) + } + } +} + /// Tear down a sub-agent container. By default this is non-destructive to /// persistent state: the proposed/applied config repos and the Claude /// credentials dir under `/var/lib/hyperhive/{agents,applied}//` are @@ -178,17 +276,50 @@ pub async fn destroy(coord: &Coordinator, name: &str, purge: bool) -> Result<()> Ok(()) } -pub fn deny(coord: &Coordinator, id: i64, note: Option<&str>) -> Result<()> { +pub async fn deny(coord: &Coordinator, id: i64, note: Option<&str>) -> Result<()> { let approval = coord.approvals.get(id)?; coord.approvals.mark_denied(id, note)?; tracing::info!(%id, note, "approval denied"); + let mut tag = None; if let Some(a) = approval { + let sha = a.fetched_sha.clone(); + // ApplyCommit approvals leave a `denied/` tag on the + // proposal commit so rejected configs are first-class git + // objects — `git show denied/` in the manager's applied + // mount yields both the tree the operator rejected and (in + // the annotated body) the reason. Spawn approvals have no + // commit to tag, so they fall through unannotated. + if matches!(a.kind, ApprovalKind::ApplyCommit) { + let applied_dir = Coordinator::agent_applied_dir(&a.agent); + let proposal_ref = format!("refs/tags/proposal/{id}"); + if lifecycle::git_rev_parse(&applied_dir, &proposal_ref) + .await + .is_ok() + { + let tag_name = format!("denied/{id}"); + let body = note.unwrap_or("").to_owned(); + if let Err(e) = lifecycle::git_tag_annotated( + &applied_dir, + &tag_name, + &proposal_ref, + &body, + ) + .await + { + tracing::warn!(%id, error = ?e, "plant denied tag failed"); + } else { + tag = Some(tag_name); + } + } + } coord.notify_manager(&HelperEvent::ApprovalResolved { id: a.id, agent: a.agent, commit_ref: a.commit_ref, status: ApprovalStatus::Denied, note: note.map(String::from), + sha, + tag, }); } Ok(()) diff --git a/hive-c0re/src/approvals.rs b/hive-c0re/src/approvals.rs index fbc06ab5..f53aebfc 100644 --- a/hive-c0re/src/approvals.rs +++ b/hive-c0re/src/approvals.rs @@ -41,6 +41,21 @@ fn ensure_kind_column(conn: &Connection) -> Result<()> { Ok(()) } +/// Same shape as `ensure_kind_column` but for `fetched_sha` — the +/// canonical sha hive-c0re vouched for at `request_apply_commit` time. +/// Distinct from `commit_ref` (manager-supplied, may not even resolve +/// in proposed by the time we approve). +fn ensure_fetched_sha_column(conn: &Connection) -> Result<()> { + let has: bool = conn + .prepare("SELECT 1 FROM pragma_table_info('approvals') WHERE name = 'fetched_sha'")? + .exists([])?; + if !has { + conn.execute_batch("ALTER TABLE approvals ADD COLUMN fetched_sha TEXT;") + .context("add approvals.fetched_sha column")?; + } + Ok(()) +} + pub struct Approvals { conn: Mutex, } @@ -56,6 +71,7 @@ impl Approvals { conn.execute_batch(SCHEMA) .context("apply approvals schema")?; ensure_kind_column(&conn).context("migrate approvals.kind")?; + ensure_fetched_sha_column(&conn).context("migrate approvals.fetched_sha")?; Ok(Self { conn: Mutex::new(conn), }) @@ -75,10 +91,22 @@ impl Approvals { Ok(conn.last_insert_rowid()) } + /// Record the canonical sha hive-c0re fetched from the proposed repo + /// into applied at submission time. Idempotent on identical values. + #[allow(dead_code)] // wired up by manager_server in the next commit + pub fn set_fetched_sha(&self, id: i64, sha: &str) -> Result<()> { + let conn = self.conn.lock().unwrap(); + conn.execute( + "UPDATE approvals SET fetched_sha = ?1 WHERE id = ?2", + params![sha, id], + )?; + Ok(()) + } + pub fn pending(&self) -> Result> { let conn = self.conn.lock().unwrap(); let mut stmt = conn.prepare( - "SELECT id, agent, kind, commit_ref, requested_at, status, resolved_at, note + "SELECT id, agent, kind, commit_ref, requested_at, status, resolved_at, note, fetched_sha FROM approvals WHERE status = 'pending' ORDER BY id ASC", @@ -91,7 +119,7 @@ impl Approvals { pub fn get(&self, id: i64) -> Result> { let conn = self.conn.lock().unwrap(); conn.query_row( - "SELECT id, agent, kind, commit_ref, requested_at, status, resolved_at, note + "SELECT id, agent, kind, commit_ref, requested_at, status, resolved_at, note, fetched_sha FROM approvals WHERE id = ?1", params![id], row_to_approval, @@ -104,9 +132,10 @@ impl Approvals { /// approval so the caller can run the action and pass the agent name. pub fn mark_approved(&self, id: i64) -> Result { let conn = self.conn.lock().unwrap(); - let current: Option<(String, String, String, i64, String)> = conn + let current: Option<(String, String, String, i64, String, Option)> = conn .query_row( - "SELECT agent, kind, commit_ref, requested_at, status FROM approvals WHERE id = ?1", + "SELECT agent, kind, commit_ref, requested_at, status, fetched_sha + FROM approvals WHERE id = ?1", params![id], |row| { Ok(( @@ -115,11 +144,12 @@ impl Approvals { row.get(2)?, row.get(3)?, row.get(4)?, + row.get(5)?, )) }, ) .optional()?; - let Some((agent, kind, commit_ref, requested_at, status)) = current else { + let Some((agent, kind, commit_ref, requested_at, status, fetched_sha)) = current else { bail!("approval {id} not found"); }; if status != "pending" { @@ -139,6 +169,7 @@ impl Approvals { status: ApprovalStatus::Approved, resolved_at: Some(resolved_at), note: None, + fetched_sha, }) } @@ -178,7 +209,7 @@ impl Approvals { } fn row_to_approval(row: &rusqlite::Row<'_>) -> rusqlite::Result { - // Column order: id, agent, kind, commit_ref, requested_at, status, resolved_at, note. + // Column order: id, agent, kind, commit_ref, requested_at, status, resolved_at, note, fetched_sha. let kind: String = row.get(2)?; let kind = match kind.as_str() { "apply_commit" => ApprovalKind::ApplyCommit, @@ -214,6 +245,7 @@ fn row_to_approval(row: &rusqlite::Row<'_>) -> rusqlite::Result { status, resolved_at: row.get(6)?, note: row.get(7)?, + fetched_sha: row.get(8)?, }) } diff --git a/hive-c0re/src/auto_update.rs b/hive-c0re/src/auto_update.rs index 16a562a6..bce2871f 100644 --- a/hive-c0re/src/auto_update.rs +++ b/hive-c0re/src/auto_update.rs @@ -78,6 +78,8 @@ pub async fn rebuild_agent(coord: &Arc, name: &str, current_rev: &s agent: name.to_owned(), ok: true, note: None, + sha: None, + tag: None, }); } Err(e) => { @@ -85,6 +87,8 @@ pub async fn rebuild_agent(coord: &Arc, name: &str, current_rev: &s agent: name.to_owned(), ok: false, note: Some(format!("{e:#}")), + sha: None, + tag: None, }); } } diff --git a/hive-c0re/src/dashboard.rs b/hive-c0re/src/dashboard.rs index ceaafa1d..89425243 100644 --- a/hive-c0re/src/dashboard.rs +++ b/hive-c0re/src/dashboard.rs @@ -146,6 +146,17 @@ struct StateSnapshot { /// survive after a destroy-without-purge. The operator can re-spawn /// with the same name to resume, or PURG3 to wipe them. tombstones: Vec, + /// Sub-agents whose FNV-1a hashed web UI port collides with at + /// least one other agent. Operator resolves by renaming. The + /// dashboard renders a banner at the top listing each cluster. + port_conflicts: Vec, +} + +#[derive(Serialize)] +struct PortConflict { + port: u16, + /// All agent names sharing this port (sorted, ≥2 entries). + agents: Vec, } #[derive(Serialize)] @@ -216,6 +227,7 @@ async fn api_state(headers: HeaderMap, State(state): State) -> axum::J let transients = build_transient_views(&raw_containers, &transient_snapshot); let approvals = build_approval_views(pending_approvals).await; let tombstones = build_tombstone_views(&state.coord, &containers, &transient_snapshot); + let port_conflicts = build_port_conflicts(&containers); let operator_inbox = state .coord @@ -234,9 +246,31 @@ async fn api_state(headers: HeaderMap, State(state): State) -> axum::J operator_inbox, questions, tombstones, + port_conflicts, }) } +/// Group live containers by their assigned web UI port; clusters with +/// more than one member are port-hash collisions the operator needs +/// to resolve by renaming. Manager (fixed at 8000) and sub-agents +/// (8100..8999) can't collide with each other — collisions are +/// strictly between sub-agents. +fn build_port_conflicts(containers: &[ContainerView]) -> Vec { + let mut by_port: std::collections::BTreeMap> = + std::collections::BTreeMap::new(); + for c in containers { + by_port.entry(c.port).or_default().push(c.name.clone()); + } + by_port + .into_iter() + .filter(|(_, agents)| agents.len() > 1) + .map(|(port, mut agents)| { + agents.sort(); + PortConflict { port, agents } + }) + .collect() +} + /// Build `ContainerView`s for every live nixos-container. Returns the /// list and whether any container is stale (drives the "↻ UPD4TE 4LL" /// banner). @@ -426,7 +460,7 @@ async fn post_deny( .as_deref() .map(str::trim) .filter(|s| !s.is_empty()); - match actions::deny(&state.coord, id, note) { + match actions::deny(&state.coord, id, note).await { Ok(()) => Redirect::to("/").into_response(), Err(e) => error_response(&format!("deny {id} failed: {e:#}")), } diff --git a/hive-c0re/src/lifecycle.rs b/hive-c0re/src/lifecycle.rs index 130ddfdb..cba109a5 100644 --- a/hive-c0re/src/lifecycle.rs +++ b/hive-c0re/src/lifecycle.rs @@ -107,6 +107,32 @@ fn validate(name: &str) -> Result<()> { Ok(()) } +/// First name (≠ `self_name`) currently running whose hashed port +/// matches this agent's. The harness inside the colliding container +/// would otherwise loop on `AddrInUse` forever; we surface the +/// conflict here so spawn / rebuild fails loudly with an actionable +/// message instead. +async fn port_collision(self_name: &str) -> Option { + let port = agent_web_port(self_name); + let raw = list().await.unwrap_or_default(); + for c in raw { + let other = if c == MANAGER_NAME { + MANAGER_NAME.to_owned() + } else if let Some(n) = c.strip_prefix(AGENT_PREFIX) { + n.to_owned() + } else { + continue; + }; + if other == self_name { + continue; + } + if agent_web_port(&other) == port && is_running(&other).await { + return Some(other); + } + } + None +} + #[allow(clippy::too_many_arguments)] pub async fn spawn( name: &str, @@ -119,8 +145,21 @@ pub async fn spawn( dashboard_port: u16, ) -> Result<()> { validate(name)?; + if let Some(other) = port_collision(name).await { + bail!( + "port {} is already taken by '{other}' — rename one of them and retry", + agent_web_port(name) + ); + } setup_proposed(proposed_dir, name).await?; - setup_applied(applied_dir, name, hyperhive_flake, dashboard_port).await?; + setup_applied( + applied_dir, + Some(proposed_dir), + name, + hyperhive_flake, + dashboard_port, + ) + .await?; ensure_claude_dir(claude_dir)?; ensure_state_dir(notes_dir)?; let container = container_name(name); @@ -192,7 +231,13 @@ pub async fn rebuild( dashboard_port: u16, ) -> Result<()> { validate(name)?; - setup_applied(applied_dir, name, hyperhive_flake, dashboard_port).await?; + if let Some(other) = port_collision(name).await { + bail!( + "port {} is already taken by '{other}' — rename one of them and retry", + agent_web_port(name) + ); + } + setup_applied(applied_dir, None, name, hyperhive_flake, dashboard_port).await?; ensure_claude_dir(claude_dir)?; ensure_state_dir(notes_dir)?; let container = container_name(name); @@ -248,11 +293,22 @@ pub async fn setup_proposed(proposed_dir: &Path, name: &str) -> Result<()> { Ok(()) } -/// Maintain the authoritative applied repo. Rewrites `flake.nix` every call -/// (so a new hyperhive flake URL propagates on rebuild); seeds `agent.nix` -/// only on first call. `apply_commit` overwrites `agent.nix` later. +/// Set up the applied repo. Two responsibilities: +/// - First-spawn only: init the repo, pull proposed's initial commit +/// in via `git fetch`, tag it `deployed/0`. This is the *only* time +/// hive-c0re reads from `proposed` for an agent — subsequent +/// proposals are fetched at `request_apply_commit` time and tagged +/// `proposal/` (see `actions::approve` for the tag state +/// machine). +/// - Every call: regenerate the untracked `flake.nix` so flake-url / +/// dashboard-port changes pick up on rebuild without churning the +/// git log. +/// +/// `proposed_dir` is `None` on rebuild paths that just want the flake +/// refreshed. pub async fn setup_applied( applied_dir: &Path, + proposed_dir: Option<&Path>, name: &str, hyperhive_flake: &str, dashboard_port: u16, @@ -260,6 +316,42 @@ pub async fn setup_applied( std::fs::create_dir_all(applied_dir) .with_context(|| format!("create {}", applied_dir.display()))?; + // 1. First-spawn git init from proposed (or pre-overhaul detection). + if !applied_dir.join(".git").exists() { + let Some(proposed) = proposed_dir else { + bail!( + "applied repo at {} is missing its .git directory; \ + cannot rebuild without a proposed source to seed from. \ + destroy --purge and re-spawn this agent.", + applied_dir.display() + ); + }; + git(applied_dir, &["init", "--initial-branch=main"]).await?; + let proposed_str = proposed.display().to_string(); + git( + applied_dir, + &["fetch", "--no-tags", &proposed_str, "main:refs/heads/main"], + ) + .await?; + git_read_tree_reset(applied_dir, "refs/heads/main").await?; + git_tag(applied_dir, "deployed/0", "refs/heads/main").await?; + } else if git_rev_parse(applied_dir, "refs/tags/deployed/0") + .await + .is_err() + { + // Pre-overhaul applied repo — agent.nix is tracked directly, + // commits authored by hive-c0re, no deployed/* tag scheme. + // No in-place migration; fail loudly so the operator purges. + bail!( + "applied repo at {} predates the tag-driven config flow. \ + Run `hive-c0re destroy --purge {name}` and re-spawn.", + applied_dir.display() + ); + } + + // 2. (Re)write the untracked wrapper flake. Tracked files in the + // working tree (agent.nix and anything the manager committed) are + // untouched. let port = agent_web_port(name); let base = flake_base(name); let service = if is_manager(name) { @@ -301,48 +393,6 @@ pub async fn setup_applied( ); std::fs::write(applied_dir.join("flake.nix"), flake_body) .with_context(|| format!("write {}/flake.nix", applied_dir.display()))?; - - let agent_path = applied_dir.join("agent.nix"); - if !agent_path.exists() { - std::fs::write(&agent_path, initial_agent_nix(name)) - .with_context(|| format!("write {}", agent_path.display()))?; - } - - if !applied_dir.join(".git").exists() { - git(applied_dir, &["init", "--initial-branch=main"]).await?; - } - git(applied_dir, &["add", "-A"]).await?; - let clean = git_status(applied_dir, &["diff", "--cached", "--quiet"]).await?; - if !clean { - git_commit(applied_dir, "hive-c0re sync").await?; - } - Ok(()) -} - -/// Apply a manager-proposed commit: read `agent.nix` at `commit_ref` from the -/// proposed repo, write it into the applied repo, commit. Hive-c0re alone -/// advances `applied`'s `main`; the manager only sees `proposed/`. -pub async fn apply_commit(applied_dir: &Path, proposed_dir: &Path, commit_ref: &str) -> Result<()> { - let out = git_command() - .current_dir(proposed_dir) - .args(["show", &format!("{commit_ref}:agent.nix")]) - .output() - .await - .with_context(|| format!("git show in {}", proposed_dir.display()))?; - if !out.status.success() { - bail!( - "agent.nix at commit {commit_ref} not found in {}: {}", - proposed_dir.display(), - String::from_utf8_lossy(&out.stderr).trim() - ); - } - std::fs::write(applied_dir.join("agent.nix"), &out.stdout) - .with_context(|| format!("write {}/agent.nix", applied_dir.display()))?; - git(applied_dir, &["add", "agent.nix"]).await?; - let clean = git_status(applied_dir, &["diff", "--cached", "--quiet"]).await?; - if !clean { - git_commit(applied_dir, &format!("apply {commit_ref}")).await?; - } Ok(()) } @@ -419,15 +469,97 @@ async fn git(dir: &Path, args: &[&str]) -> Result<()> { Ok(()) } -/// Returns true if the command exits 0. -async fn git_status(dir: &Path, args: &[&str]) -> Result { - let st = git_command() +/// Fetch `sha` from the `src` git repo into `dst` and pin it as +/// `refs/tags/`. Used at `request_apply_commit` time so hive-c0re +/// captures an immutable handle on the manager's commit; subsequent +/// amendments / force-pushes in `src` no longer affect what gets +/// built. Returns the resolved sha (which equals `sha` on success +/// but normalised — short shas get expanded). +#[allow(dead_code)] // wired up by manager_server in a later commit +pub async fn git_fetch_to_tag(dst: &Path, src: &Path, sha: &str, tag: &str) -> Result { + let src_str = src.display().to_string(); + let refspec = format!("{sha}:refs/tags/{tag}"); + git(dst, &["fetch", "--no-tags", &src_str, &refspec]).await?; + git_rev_parse(dst, &format!("refs/tags/{tag}")).await +} + +/// Resolve `refname` (a tag, branch, or sha) in `dir` to its full sha. +#[allow(dead_code)] +pub async fn git_rev_parse(dir: &Path, refname: &str) -> Result { + let out = git_command() .current_dir(dir) - .args(args) - .status() + .args(["rev-parse", refname]) + .output() .await - .with_context(|| format!("git {} in {}", args.join(" "), dir.display()))?; - Ok(st.success()) + .with_context(|| format!("git rev-parse {refname} in {}", dir.display()))?; + if !out.status.success() { + bail!( + "git rev-parse {refname} failed ({}): {}", + out.status, + String::from_utf8_lossy(&out.stderr).trim() + ); + } + Ok(String::from_utf8_lossy(&out.stdout).trim().to_owned()) +} + +/// Plant a lightweight tag at `target`. Errors if the tag already +/// exists — we want loud failures on id reuse, not silent +/// overwrites. +#[allow(dead_code)] +pub async fn git_tag(dir: &Path, name: &str, target: &str) -> Result<()> { + git(dir, &["tag", name, target]).await +} + +/// Plant an annotated tag with `body` as the message. Used for +/// `failed/` (body = build error) and `denied/` (body = +/// operator note). Multi-line bodies handled via stdin so we don't +/// have to escape anything. +#[allow(dead_code)] +pub async fn git_tag_annotated(dir: &Path, name: &str, target: &str, body: &str) -> Result<()> { + use tokio::io::AsyncWriteExt; + let mut child = git_command() + .current_dir(dir) + .args(["tag", "-a", name, target, "-F", "-"]) + .stdin(std::process::Stdio::piped()) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .spawn() + .with_context(|| format!("spawn git tag -a {name} in {}", dir.display()))?; + if let Some(mut stdin) = child.stdin.take() { + stdin + .write_all(body.as_bytes()) + .await + .context("write tag body to git stdin")?; + // Drop closes stdin so git can finish reading. + drop(stdin); + } + let out = child.wait_with_output().await.context("wait git tag -a")?; + if !out.status.success() { + bail!( + "git tag -a {name} failed ({}): {}", + out.status, + String::from_utf8_lossy(&out.stderr).trim() + ); + } + Ok(()) +} + +/// Replace working tree + index with the tree at `target` without +/// moving HEAD. `applied/main` stays pointing at the last known-good +/// `deployed/*` while we let `nixos-container update` evaluate the +/// candidate. On build failure callers reset back to HEAD; on +/// success they fast-forward main to `target`. +#[allow(dead_code)] +pub async fn git_read_tree_reset(dir: &Path, target: &str) -> Result<()> { + git(dir, &["read-tree", "--reset", "-u", target]).await +} + +/// Hard-set a ref to `target`. Used to fast-forward `refs/heads/main` +/// to the just-deployed proposal commit. Uses `update-ref`, not +/// `branch -f`, so it works regardless of where HEAD currently sits. +#[allow(dead_code)] +pub async fn git_update_ref(dir: &Path, refname: &str, target: &str) -> Result<()> { + git(dir, &["update-ref", refname, target]).await } /// Write a systemd drop-in for `container@.service` that applies @@ -475,11 +607,22 @@ async fn systemd_daemon_reload() -> Result<()> { /// `containers.hm1nd.bindMounts."/agents"`. pub const CONTAINER_MANAGER_AGENTS_MOUNT: &str = "/agents"; +/// Where the manager sees the applied trees of every agent, read-only. +/// Manager runs `git fetch /applied//.git refs/tags/*:refs/tags/applied/*` +/// to learn what hive-c0re deployed (or rejected, or failed to +/// build); the RO bind makes accidental writes impossible from +/// inside the container. +pub const CONTAINER_MANAGER_APPLIED_MOUNT: &str = "/applied"; + /// The on-host root that gets bind-mounted to `/agents` inside the manager. /// Hard-coded to match `AGENT_STATE_ROOT` in coordinator.rs (kept duplicated /// here so lifecycle stays usable as a leaf module). const HOST_AGENTS_ROOT: &str = "/var/lib/hyperhive/agents"; +/// On-host applied repo root, mirrored RO into the manager. Matches +/// `APPLIED_STATE_ROOT` in coordinator.rs. +const HOST_APPLIED_ROOT: &str = "/var/lib/hyperhive/applied"; + fn set_nspawn_flags( container: &str, runtime_dir: &Path, @@ -497,11 +640,22 @@ fn set_nspawn_flags( if container == MANAGER_NAME { // Manager edits sub-agent proposed/ repos and its own. RW so it can // git-commit. Sub-agents see only their own /run/hive socket and - // /root/.claude (no /agents). + // /root/.claude (no /agents or /applied). + // + // /applied is a separate RO mount of the hive-c0re-only applied + // repos so the manager can `git fetch /applied//.git + // refs/tags/*:refs/tags/applied/*` to mirror deployed/failed/ + // denied tags into its proposed clones and diff against + // what's actually deployed. RO bind makes destructive git + // plumbing inside the container unable to corrupt applied. use std::fmt::Write as _; let _ = write!( binds, - " --bind={HOST_AGENTS_ROOT}:{CONTAINER_MANAGER_AGENTS_MOUNT}" + " --bind={HOST_AGENTS_ROOT}:{CONTAINER_MANAGER_AGENTS_MOUNT}", + ); + let _ = write!( + binds, + " --bind-ro={HOST_APPLIED_ROOT}:{CONTAINER_MANAGER_APPLIED_MOUNT}", ); } let bind_flag = format!("EXTRA_NSPAWN_FLAGS=\"{binds}\""); diff --git a/hive-c0re/src/manager_server.rs b/hive-c0re/src/manager_server.rs index 3c9cab2c..a15b338d 100644 --- a/hive-c0re/src/manager_server.rs +++ b/hive-c0re/src/manager_server.rs @@ -258,9 +258,9 @@ async fn dispatch(req: &ManagerRequest, coord: &Arc) -> ManagerResp } ManagerRequest::RequestApplyCommit { agent, commit_ref } => { tracing::info!(%agent, %commit_ref, "manager: request_apply_commit"); - match coord.approvals.submit(agent, commit_ref) { - Ok(id) => { - tracing::info!(%id, %agent, %commit_ref, "approval queued"); + match submit_apply_commit(coord, agent, commit_ref).await { + Ok((id, sha)) => { + tracing::info!(%id, %agent, manager_ref = %commit_ref, %sha, "approval queued + proposal tag planted"); ManagerResponse::Ok } Err(e) => ManagerResponse::Err { @@ -271,6 +271,66 @@ async fn dispatch(req: &ManagerRequest, coord: &Arc) -> ManagerResp } } +/// Submit-time half of the apply flow: queue the approval row, then +/// fetch the manager's commit from the proposed repo into applied and +/// pin it as `refs/tags/proposal/`. From this point on the manager +/// repo is irrelevant for this approval — even if the manager amends +/// or force-pushes, the canonical sha hive-c0re will eventually +/// approve/deny lives in applied's object DB. +/// +/// If anything fails after the row is inserted (sha missing in +/// proposed, fs error, git plumbing crash) we mark the row failed and +/// surface the error to the manager. We don't try to roll the row +/// back — the failure is part of the audit trail. +async fn submit_apply_commit( + coord: &Arc, + agent: &str, + commit_ref: &str, +) -> anyhow::Result<(i64, String)> { + let proposed_dir = crate::coordinator::Coordinator::agent_proposed_dir(agent); + let applied_dir = crate::coordinator::Coordinator::agent_applied_dir(agent); + if !proposed_dir.exists() { + anyhow::bail!( + "proposed repo missing for agent '{agent}' (expected at {})", + proposed_dir.display() + ); + } + if !applied_dir.join(".git").exists() { + anyhow::bail!( + "applied repo at {} is uninitialised — spawn the agent first", + applied_dir.display() + ); + } + let id = coord + .approvals + .submit(agent, commit_ref) + .map_err(|e| anyhow::anyhow!("queue approval row: {e:#}"))?; + let tag = format!("proposal/{id}"); + let sha = match crate::lifecycle::git_fetch_to_tag( + &applied_dir, + &proposed_dir, + commit_ref, + &tag, + ) + .await + { + Ok(s) => s, + Err(e) => { + // Surface the failure on the approval row so the + // dashboard reflects it instead of leaving a phantom + // pending entry. The note doubles as the operator-visible + // explanation of why the approval can't be approved. + let _ = coord.approvals.mark_failed(id, &format!("{e:#}")); + return Err(anyhow::anyhow!("git_fetch_to_tag: {e:#}")); + } + }; + coord + .approvals + .set_fetched_sha(id, &sha) + .map_err(|e| anyhow::anyhow!("persist fetched_sha: {e:#}"))?; + Ok((id, sha)) +} + /// On `AskOperator { ttl_seconds: Some(n) }`, sleep n seconds and then /// try to resolve the question with `[expired]`. If the operator (or /// any other path) already answered it, `answer()` returns Err and diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs index 80f35511..69f5d72d 100644 --- a/hive-c0re/src/server.rs +++ b/hive-c0re/src/server.rs @@ -83,6 +83,7 @@ async fn dispatch(req: &HostRequest, coord: Arc) -> HostResponse { agent: name.clone(), ok: true, note: None, + sha: None, }); } Err(e) => { @@ -92,6 +93,7 @@ async fn dispatch(req: &HostRequest, coord: Arc) -> HostResponse { agent: name.clone(), ok: false, note: Some(format!("{e:#}")), + sha: None, }); return Err(e); } @@ -144,7 +146,7 @@ async fn dispatch(req: &HostRequest, coord: Arc) -> HostResponse { HostResponse::success() } HostRequest::Deny { id } => { - actions::deny(&coord, *id, None)?; + actions::deny(&coord, *id, None).await?; HostResponse::success() } }) diff --git a/hive-sh4re/src/lib.rs b/hive-sh4re/src/lib.rs index 7a963221..5d4a4fba 100644 --- a/hive-sh4re/src/lib.rs +++ b/hive-sh4re/src/lib.rs @@ -65,8 +65,18 @@ pub struct Approval { pub agent: String, #[serde(default)] pub kind: ApprovalKind, - /// For `ApplyCommit`: the git sha to apply. For `Spawn`: empty. + /// For `ApplyCommit`: the git sha the manager submitted. For `Spawn`: + /// empty. Note that this is the manager's *claimed* ref — the + /// canonical, hive-c0re-vouched sha after the proposal fetch lives + /// in `fetched_sha`. pub commit_ref: String, + /// The sha hive-c0re fetched from the proposed repo into applied at + /// submission time, then tagged `proposal/`. Stable for the + /// lifetime of the approval — manager amends in proposed don't + /// change what gets built. Only set for `ApplyCommit` after the + /// successful fetch. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub fetched_sha: Option, pub requested_at: i64, pub status: ApprovalStatus, #[serde(default, skip_serializing_if = "Option::is_none")] @@ -236,6 +246,18 @@ pub enum HelperEvent { status: ApprovalStatus, #[serde(default, skip_serializing_if = "Option::is_none")] note: Option, + /// Canonical sha hive-c0re fetched into applied at submission + /// time. `git show ` against `/agents//applied.git` + /// inside the manager container yields the exact tree being + /// referenced. + #[serde(default, skip_serializing_if = "Option::is_none")] + sha: Option, + /// Terminal tag name in the applied repo for this approval — + /// `deployed/`, `failed/`, or `denied/` (and + /// `approved/` for the rare bare-approval case where + /// no underlying action runs). + #[serde(default, skip_serializing_if = "Option::is_none")] + tag: Option, }, /// A new container was spawned (post-approval or via the admin CLI /// bypass path). `ok=false` means the spawn failed. @@ -244,6 +266,10 @@ pub enum HelperEvent { ok: bool, #[serde(default, skip_serializing_if = "Option::is_none")] note: Option, + /// Sha of the `deployed/0` commit seeded by hive-c0re on + /// first spawn (Some on success, None on failure). + #[serde(default, skip_serializing_if = "Option::is_none")] + sha: Option, }, /// A container was rebuilt (auto-update on flake rev change, or a /// manual rebuild from CLI/dashboard). @@ -252,6 +278,18 @@ pub enum HelperEvent { ok: bool, #[serde(default, skip_serializing_if = "Option::is_none")] note: Option, + /// Sha that ended up at `deployed/` on success, or the + /// proposal sha that just got tagged `failed/` on + /// failure. None for the (rare) rebuild path that doesn't go + /// through an approval (e.g. `auto_update::rebuild_agent` + /// reapplying the existing main). + #[serde(default, skip_serializing_if = "Option::is_none")] + sha: Option, + /// `deployed/` or `failed/` for approval-driven + /// rebuilds; None for auto-update / dashboard rebuilds that + /// don't change the deployed commit. + #[serde(default, skip_serializing_if = "Option::is_none")] + tag: Option, }, /// A sub-agent's container was stopped (the systemd unit is down; /// persistent state is unchanged).