Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

..
Author SHA1 Message Date
müde
2486251b32 fix: close second review round on the queue-routed CLI
- subvol upgrade waits for the queued stop DAG before migrating (was
  snapshotting + swapping state under a live bind mount) and for the
  restart job after
- history trim gets a 5-min grace for fresh terminals so broad
  stop/start waits can't miss a failed DAG evicted by the per-template
  cap (cap still applies past the grace)
- restart-all returns its DAG ids so hivectl actually waits
- hard stops await their agent DAGs (bounded) before infra goes down,
  restoring the agents-before-infra invariant
- hivectl wait uses node-level terminality so the after-any recovery
  reconcile is watched to completion; infra render errors no longer
  skip watching already-queued agent DAGs
- fold hive-bash-mcp's last local now_unix into wire_time
2026-07-06 22:57:28 +02:00
müde
0e4b5a1120 refactor(hive-c0re): group src-root files into submodules
stores/ (sqlite-backed host stores + db helper), stats/, agent_config/,
workers/ — pure git-mv moves; crate-root re-exports keep every
crate::<module> path compiling. flake_check stays at root (synchronous
approval-flow validation, not a background worker)
2026-07-06 22:38:47 +02:00
müde
b489454dc2 feat(hivectl): queue-routed lifecycle verbs with wait + DAG progress
every agent lifecycle verb on the admin socket (rebuild / restart /
restart-all / kill / stop / start) now submits job-queue DAGs and
returns their ids; hivectl polls the new HostRequest::QueueDag and
prints a live node-chain progress line per DAG (fan-out children
included), exiting non-zero on failure — --no-wait opts out. DagView
and the queue wire enums move to hive_sh4re::jobs (wire types live in
the shared crate); the last fused rebuild path (lifecycle::rebuild)
is gone. tracker: #2166
2026-07-06 22:30:49 +02:00
müde
dc6a37b29a style: import ordering + blank lines after now_unix sweep (treefmt) 2026-07-06 22:08:40 +02:00
müde
a17015f01e refactor(hive-c0re): split forge into submodules
mod.rs keeps the shared admin/http helpers + ensure_all/sync_agent;
user/token provisioning, repo/org/mirror ops, and the trust-boundary
PR-merge primitives move to users.rs / repos.rs / pr_merge.rs
2026-07-06 22:08:22 +02:00
müde
8cdebb1752 fix(hive-ag3nt): drop attribute orphaned by now_unix removal 2026-07-06 22:07:39 +02:00
müde
b0736e6f3e refactor(hive-ag3nt): split mcp into submodules
mod.rs keeps the #[tool_router] impl + server wiring untouched;
rendering/format helpers move to mcp/render.rs, the 24 tool arg
structs to mcp/args.rs
2026-07-06 22:00:30 +02:00
müde
c84028ddcf refactor: single now_unix in hive_sh4re::wire_time
replaces 15 per-module copies (now_unix/now_secs) across hive-c0re and
hive-ag3nt; wire_time already owns the epoch-seconds convention
2026-07-06 21:58:32 +02:00
müde
d190420946 refactor(hive-c0re): shared additive-migration helper in db
db::apply_migrations runs ALTER lists and ignores duplicate-column
errors (turn_stats' pattern); approvals, operator_questions, broker
reminders, and scheduled_prompts drop their hand-rolled
pragma_table_info guards. broker's acked_at migration stays bespoke —
its backfill must only run when the column was just created
2026-07-06 21:53:48 +02:00
müde
34b21b8038 refactor: drop kind_to_str wrapper, call ApprovalKind::as_str directly 2026-07-06 21:49:46 +02:00
müde
f74c1984d7 refactor: ApprovalKind::as_str owns the kind→string mapping
replaces three hand-rolled six-arm matches (actions.rs ×2,
state_snapshot.rs); approvals::kind_to_str delegates. a new kind can
no longer silently miss one of them
2026-07-06 21:48:55 +02:00
müde
084e12503c fix(hive-c0re): close review findings on the job-DAG queue
- deploy-window gate (meta::exclusive) + path-limited meta commits:
  a perm/lock/topology commit can no longer sweep an ApprovalDeploy's
  staged flake.lock and neuter abort_deploy (regression test included)
- cancel surfaces now buffer terminal roll-ups the scheduler drains,
  so a queued approval DAG cancelled by the operator resolves its
  approval instead of dangling, and cancelled power ops revert their
  wanted flip to the observed state
- hivectl restart / restart-all ride the queue (lease serialization,
  transient guard) and restart sets wanted=Up like the old kill+start
- exactly one Rebuilt event per rebuild DAG, emitted at terminal
- StopForUpdate pre-seeds a missing agent_power row from the pre-stop
  observation so a rebuild can't strand an unknown agent offline
- history trim keeps terminal fan-out parents with live children
- audit_log back on db::open; swarm.js badge for reconcile DAGs
2026-07-06 21:44:43 +02:00
müde
58e86a3adf refactor(hive-c0re): shrink dashboard mod root
convert to dashboard/mod.rs; state snapshot, meta inputs, tombstones,
and misc api handlers move to their own files
2026-07-06 21:05:52 +02:00
müde
3ee87d394c refactor(hive-c0re): split lifecycle into submodules
mod.rs keeps the container verbs + priv_run plumbing; git helpers,
repo/dir setup, and host drop-in config move to their own files
2026-07-06 21:05:52 +02:00
müde
9e7af3b6bf refactor(hive-c0re): split socket_server into submodules
mod.rs keeps dispatch + messaging/guards; schedules, reminders,
config approvals, and lifecycle handlers move to their own files
2026-07-06 21:05:52 +02:00
müde
380c6ad47f refactor(hive-c0re): shared sqlite open helper with busy timeout
one db::open owns the parent-dir + connection + busy_timeout dance for
every host-side store (broker/approvals/questions/schedules/power in
broker.sqlite, build_logs, audit_log); schema + migrations stay per
store. same-file connections now wait out concurrent writers instead
of risking SQLITE_BUSY.
2026-07-06 20:46:57 +02:00
müde
604e1c2557 docs: job-DAG queue model; fold agent_power table into broker.sqlite
coordinator.md rewrites the queue section (node inventory, DAG shapes,
resources, desired-state reconciliation, boot reconcile); approvals.md
+ persistence.md + hivectl --graceful help updated to match. agent_power
lives in broker.sqlite like approvals/questions (own connection + busy
timeout) instead of a separate db file.
2026-07-06 20:36:57 +02:00
müde
8349e6f621 feat(dashboard): node-aware queue render + buildSlots option
each queue card now shows its DAG's node chain (per-node state, step,
build-log link), fixing 'queue jumps don't show on the dashboard'.
live-log panel keys off the running node's log. new
services.hyperhive.c0re.buildSlots option (default 1) threads the
concurrent nix-build count into serve.json.
2026-07-06 20:36:57 +02:00
müde
7946e03fde feat(hive-c0re): replace rebuild queue with generic job-DAG queue
jobs are now DAGs of primitive nodes (prebuild, stop-for-update, swap,
reconcile, signal, drain, ...) driven by one scheduler with N build
slots + per-agent lifecycle leases. per-agent power intent (wanted
up/offline) is durable in agent_power.sqlite; Reconcile nodes converge
observed state to it. kills the graceful-stop watcher thread, the
deferred-start follow-up, and the cascade pre-enqueue (fan-out on
MetaLock completion instead). tracker: #2166
2026-07-06 20:36:57 +02:00
92 changed files with 13734 additions and 11853 deletions

18
Cargo.lock generated
View file

@ -1029,6 +1029,12 @@ version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
[[package]]
name = "fixedbitset"
version = "0.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99"
[[package]]
name = "flate2"
version = "1.1.9"
@ -1381,6 +1387,7 @@ dependencies = [
"hive-sh4re",
"libc",
"listenfd",
"petgraph",
"problem_details",
"reqwest",
"rusqlite",
@ -2554,6 +2561,17 @@ version = "2.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]]
name = "petgraph"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455"
dependencies = [
"fixedbitset",
"hashbrown 0.15.5",
"indexmap",
]
[[package]]
name = "phf"
version = "0.11.3"

View file

@ -68,6 +68,7 @@ reqwest = { version = "0.12", default-features = false, features = [
"json",
"rustls-tls",
] }
petgraph = { version = "0.8", default-features = false, features = ["std"] }
matrix-sdk = { version = "0.14", default-features = false, features = [
"rustls-tls",
"sqlite",

View file

@ -336,50 +336,49 @@ approval id to retry. Because tags are first-class git objects,
rejected and failed trees stay browsable forever — `git log
--tags` in the applied repo is the audit trail.
### Dispatch via `rebuild_queue`
### Dispatch via the job queue
Long-running approval work — `ApplyCommit`, `UpdateMetaInputs`,
`Spawn` — no longer runs inline inside `actions::approve`. Instead
the approval handler enqueues a `QueueEntry` into the global
`rebuild_queue`:
the approval handler submits a DAG to the global job queue
(`docs/coordinator.md::Job queue`):
| `ApprovalKind` | `QueueKind` queued | `QueueSource` |
| `ApprovalKind` | DAG submitted | source |
|---|---|---|
| `ApplyCommit` | `Rebuild` | `Approval` |
| `MergeConfigPr` | `Rebuild` | `Approval` |
| `UpdateMetaInputs` | `MetaUpdate` | `Approval` |
| `Spawn` | `Spawn` | `Approval` |
| `ApplyCommit` | `rebuild` (single opaque `ApprovalDeploy` node) | `approval` |
| `MergeConfigPr` | `rebuild` (single opaque `ApprovalDeploy` node) | `approval` |
| `UpdateMetaInputs` | `meta_update` (`MetaLock` + rebuild fan-out) | `approval` |
| `Spawn` | `spawn` (`Create → WriteDropin → Reconcile`) | `approval` |
| `InitConfig` | — runs inline (sub-second git seed) | — |
| `SchedulePrompt` | — runs inline (single sqlite insert) | — |
Each queue entry carries the originating `approval_id` so the
worker can re-fetch the approval row when it dispatches, run the
kind-specific pipeline (`run_approval_apply_commit` /
`run_approval_merge_config_pr` / `run_approval_update_meta_inputs` /
`run_approval_spawn`), and
fire the matching `HelperEvent::*` on completion via
`finish_approval`.
The DAG carries the originating `approval_id`. The `ApprovalDeploy`
node runs the kind-specific pipeline (`run_approval_apply_commit` /
`run_approval_merge_config_pr` — the two-phase meta deploy stays
inside `actions.rs`) and fires the matching `HelperEvent::*` via
`finish_approval` itself; `Spawn` and `UpdateMetaInputs` DAGs resolve
through `actions::resolve_approval_dag` when the DAG settles terminal
(a spawn additionally runs the post-spawn forge bookkeeping there).
Two visible consequences:
- **Operator dashboard**: after clicking APPR0VE the work-in-progress
shows up on the *rebuild queue* card (`POST /api/state.rebuild_queue`
shows up on the *rebuild queue* card (`/api/state.rebuild_queue`
+ live `rebuild_queue_changed` events), not on the approvals panel
(which already moved the row to "approved"). A long meta-update
cascade renders as a parent entry with one child per per-agent
rebuild — see `docs/web-ui.md` for the layout.
- **Cancellation**: the dashboard's *× cancel* button on a `Queued`
entry calls `POST /api/rebuild-queue/{id}/cancel`, which flips the
entry to `Cancelled` before the worker dispatches it. Returns
`{"cancelled": true}` on success, `{"cancelled": false}` if the
entry already left `Queued` (running / done / failed) — terminal
states can't be retroactively rewritten.
cascade renders as a parent DAG with one child rebuild per affected
agent — see `docs/web-ui.md` for the layout.
- **Cancellation**: the dashboard's *× cancel* button on a still-queued
DAG calls `POST /api/rebuild-queue/{id}/cancel`, which flips it to
`Cancelled` before any node runs (and fails the approval row instead
of leaving it dangling). Returns `{"cancelled": true}` on success,
`{"cancelled": false}` once any node started — terminal states can't
be retroactively rewritten.
`QueueSource::Approval` carries the `approval_id` so a tail-end
build failure surfaces back as a failed approval row, not just a
silent queue entry. `QueueSource::Manual` (dashboard ↻ R3BU1LD)
and `QueueSource::AutoUpdate` (boot-time sweep) use the same
queue but skip the approval row plumbing.
The `approval` source + `approval_id` mean a tail-end build failure
surfaces back as a failed approval row, not just a silent queue
entry. `manual` (dashboard ↻ R3BU1LD) and `auto_update` (boot
reconcile) DAGs use the same queue but skip the approval plumbing.
### Forge mirror
@ -414,7 +413,7 @@ tool group can therefore edit, commit, and submit changes for any of
its direct children directly inside its container at `/agents/<child>/config/`.
Agents holding the `can_manage_top_level_agents` topology role
(defined as `ROLE_CAN_MANAGE_TOP_LEVEL_AGENTS` in `hive-c0re/src/topology.rs`)
(defined as `ROLE_CAN_MANAGE_TOP_LEVEL_AGENTS` in `hive-c0re/src/agent_config/topology.rs`)
get additional host-side bind mounts via `set_nspawn_flags`:
- `/var/lib/hyperhive/agents/` → `/agents/` (RW) — all top-level

View file

@ -6,102 +6,191 @@ and `docs/persistence.md`.
---
## Rebuild queue
## Job queue
Every long-running container/meta operation (rebuild, meta-update, first-spawn)
goes through the global rebuild queue (`hive-c0re/src/rebuild_queue.rs`). A single
background worker drains it in FIFO order so two `nixos-container update` runs on
the same agent never overlap, and a fresh agent rebuild never races a meta-update's
lock bump.
Every container/meta operation (rebuild, meta-update, first-spawn, power
changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
concurrency comes from the resource classes below, not from multiple workers.
The old special cases — the graceful-stop watcher thread, the deferred-start
fast-lane follow-up, the meta-update cascade pre-enqueue — are all just DAG
*shapes* now.
### Why one queue
### Two levels: DAG and node
Before the rebuild queue landed, four independent call paths could fire
`auto_update::rebuild_agent` concurrently:
The **DAG** is the unit of dedup / cancel / approval-resolution and the
dashboard group; the **node** is the unit of scheduling / execution /
build-log / step label. Deps are intra-DAG edges only (`AfterOk` by default:
the dep must succeed, a failed/cancelled dep cancels the dependent —
cancel-downstream). Cross-DAG ordering comes from the per-agent lease + dedup,
never from edges between DAGs. Submit-time validation (petgraph `toposort`)
rejects cyclic specs outright, fixing the old queue's "circular dep silently
deadlocks" caveat.
- Dashboard manual rebuild button
- `update-all` / `meta-update` cascade
- Approval handler (apply-commit / spawn)
- Startup auto-update sweep
### Node inventory (primitives)
Nothing serialised them. `nix-daemon` serialises the actual store ops, but the rest
of `rebuild_agent` (token sync, kick, rescan, lock-bump emit) interleaved
unpredictably. The single-worker queue gives operators a visible, ordered runway and
lets the UI render "what's about to happen" instead of "something might be happening
somewhere."
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
### Queue kinds
| Node | Wraps |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | meta `sync_agents` + optional per-agent relock + `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving |
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped) + the post-rebuild bookkeeping tail (rev marker, `Rebuilt` event, forge/matrix sync, kick, rescan) |
| `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) |
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
| `ApprovalDeploy` | the opaque apply-commit / merge-config-PR pipeline (see _Approvals_ below) |
| Kind | Description |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. The start-after-rebuild is **deferred to a fast-lane `Start` follow-up** (`parent_id` = this entry) so the build lane is freed as soon as the profile-swap finishes instead of waiting out the container boot — see _Deferred start_ under the rebuild path below. |
| `MetaUpdate` | `nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id. |
| `Spawn` | First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV. |
| `Destroy` | For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue. |
| `Restart` | Stop + start a container without touching config (~5-10s). Routed through the queue so it serialises against in-flight rebuilds for the same agent — prevents a restart racing a rebuild mid-flight. Sources: dashboard ↺ button, the `restart` MCP tool. |
| `PermChange` | Write a tool-group or capability change to the shared JSON file (`tool-groups.json` / `capabilities.json`), then rebuild the agent so the updated `HIVE_TOOL_GROUPS` / `HIVE_CAPABILITIES` env var takes effect. Serialising the file write through the queue prevents concurrent dashboard batch-apply actions from racing on the shared file. After a successful file write, emits `CapabilitiesChanged` or `ToolGroupsChanged` SSE snapshot so the P3RM1SS10NS tab updates live. |
| `GracefulStop` | Quiesce then stop a container (the `?graceful=true` path on `/api/kill/<agent>`). Signals the harness (its next `Recv` returns `GracefulStop` — the inbound fence — so it runs a stop-checkpoint turn that flushes durable `/state`, then takes the normal post-turn compaction path if it crossed the watermark, then exits) and **immediately releases the build lane**, spawning a detached watcher that holds the `Stopping` transient across the drain (bounded by a 3-min timeout → hard-stop fallback) and then enqueues a fast-lane `Stop` (`parent_id` = this entry) for the actual `nixos-container stop`. Net: a whole-hive graceful stop signals every agent up front, drains overlap, and only the container teardowns serialise (on the fast lane). Queued so the signal can't race an in-flight rebuild for the same agent. |
Cheap — no build slot:
**Intentionally not queued** (sub-second ops): the _hard_ `start`, `stop`, `kill` via the direct API paths. (A _graceful_ stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.) The queue's fast lane does carry `Start` / `Stop` kinds, but only as **follow-ups** other entries enqueue for themselves — the graceful-stop teardown and the deferred start-after-rebuild — so the container op groups under its parent entry on the dashboard.
| Node | Behavior |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
### Dedup
There is deliberately **no `GitCommit` node**: `meta.rs` fuses each mutation
with its commit under its internal `META_LOCK` mutex, so a standalone commit
node would open a dirty-working-tree window between nodes.
Enqueueing `(kind, agent)` that already has a `Queued` entry returns the existing
entry's id and appends the new request as an "also requested by …" line. Running
entries do not dedup — a re-queue during a run is legitimate (something changed
since the current run started).
Two further layers protect the meta repo across *windows* that span multiple
`META_LOCK` acquisitions — above all the approval deploy's prepare→finalize
span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole
container build:
### Sources
- **The deploy-window gate** (`meta::exclusive()`): every executor that
mutates the meta repo (`Prebuild`'s sync+relock, `MetaLock`,
`WritePermFile`, `Create`'s agent registration, and `ApprovalDeploy` for
its whole span) holds this async mutex for its mutation span, so no commit
can land inside another node's staged window. `Prebuild` drops it before
the long toplevel build (store reads only), preserving `buildSlots > 1`
concurrency.
- **Path-limited commits**: the targeted meta committers (perm files,
topology, lock bumps, finalize) commit `-- <their paths>` with path-scoped
dirty checks, so even a non-queue caller (boot migration, destroy's
`sync_agents`) can never sweep someone else's staged content into its
commit.
| Source | Meaning |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `Manual` | Operator clicked rebuild / update-all / meta-update on the dashboard, or any other direct human action (CLI, an agent MCP tool). |
| `AutoUpdate` | Legacy startup-sweep source (flat, no parent). Replaced by `StartupSweep` for new boots. |
| `StartupSweep` | Child of a `StartupSweep` parent entry; boot-time per-agent rebuild with the sweep as the visual group header. |
| `Approval` | Triggered by an operator-approved `ApprovalKind::{Spawn, ApplyCommit}`. |
### Every operation as a DAG
### Cascade parent tracking
```text
rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-any) Reconcile(a)
graceful-stop(a): [wanted=Offline] Signal(a) → Drain(a) → Reconcile(a)
restart(a): [wanted=Up] StopForUpdate(a) → Reconcile(a)
start(a): [wanted=Up] Reconcile(a) (stale rev ⇒ upgraded to rebuild)
stop(a): [wanted=Offline] Reconcile(a)
spawn(a): [wanted=Up] Create(a) → WriteDropin(a) → Reconcile(a)
perm-change(a): WritePermFile(a) → «rebuild subgraph»
meta-update(inp): MetaLock(inp) → «fan-out rebuild(a) per affected agent»
boot: (if any rev marker stale) MetaLock(hyperhive) → «fan-out rebuild»;
plus Reconcile(a) for every drifted agent
```
`MetaUpdate` and `StartupSweep` entries fan out `Rebuild` children, each carrying
`parent_id = <parent_id>`. The dashboard groups children under their parent
in the queue panel so the operator sees the whole cascade as a tree,
not a flat list.
Notable collapses:
### Step labels
- **`rebuild` is one uniform shape** — no `was_running` branch.
`StopForUpdate` noops when already down; the tail `Reconcile` auto-noops the
start when `wanted = Offline` (a rebuild of a deliberately-stopped agent
leaves it stopped).
- **The swap-failure recovery-start is structural**: `Reconcile` deps on
`Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
terminal ok *or* fail, bringing a wanted-up agent back on its old config.
- **Deferred start is automatic**: `Reconcile` holds no build slot, so the
next DAG's `Prebuild` starts as soon as `Swap` frees the slot.
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
whole-hive graceful stop fires every agent's signal immediately and all
drains overlap; each DAG's tail `Reconcile` does the actual stop.
- **The meta-update cascade fans out on completion**: `MetaLock`'s executor
computes the affected agent set after the bump lands and appends child
`rebuild` DAGs (`parent_id` set, `relock = false` so the children don't
revert the bump). A failed bump fans out nothing — no cancel-children dance.
Each queue entry has a mutable `step: Option<String>` field that the worker updates
as it progresses through lifecycle phases (`"nix build"`, `"nixos-container stop"`,
`"nixos-container update"`, `"nixos-container start"`). The dashboard polls
`/api/state` and renders the current step beneath the running entry so the operator
can see which phase is taking time.
### Desired-state (spec vs status)
### Dependency tracking
Per-agent power *intent* — `wanted: Up | Offline` — is durable as the
`agent_power` table in the coordinator DB (`hive-c0re/src/stores/power.rs`).
`container_view` remains the observed *status*; `Reconcile` nodes converge the
two. Setting `wanted` is never a queued node: the submit layer
(`job_queue/submit.rs`) writes the row synchronously, then submits the DAG
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
Power toggles never commit to the meta repo. Every operator power surface —
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
rides the queue through that submit layer, so intent, lease serialization,
and crash-watch suppression can't drift per surface; the only direct starts
left are the root-agent bootstrap and infra containers (no lease, no
harness). Cancelling a still-queued power DAG reverts `wanted` to the
observed state — a cancel means "don't do it", not "do it later". Agents
without a row are seeded from observed state on first touch (running ⇒
`Up`); destroy removes the row.
Each entry carries a `depends_on: Vec<u64>` field. The worker skips entries whose
dependencies are not yet resolved — a dependency is resolved when its id is either
in the queue as a terminal entry (`Done` / `Failed` / `Cancelled`) or no longer in
the queue at all (evicted by `trim_history`, which only evicts terminals).
The admin-socket responses carry the submitted DAG ids; `hivectl` polls
`HostRequest::QueueDag` (~1s) and prints a progress line per DAG — roll-up
glyph, template, agent, node chain with the running node's step label — so
CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
non-zero). Fan-out children joining a polled parent show up in the same
loop.
Use cases:
### Scheduler semantics
- Chain a `Rebuild` after an explicit prerequisite step without coupling them through
the `parent_id` cascade mechanism.
- Sequence a `PermChange` + `Rebuild` pair where the rebuild must not start until the
perm-file write commits (already handled by the single-worker FIFO today, but
`depends_on` allows explicit cross-kind sequencing when parallel workers are added).
A node is **ready** when it's `Queued`, every dep is satisfied, and its
resources are free. Resources:
`depends_on` is part of the dedup key: two entries with the same `(kind, agent,
parent_id, inputs, approval_id)` but different dep sets are treated as distinct work.
1. **Build slots** — `services.hyperhive.c0re.buildSlots` permits (default 1),
held by nix-heavy nodes for the node's duration.
2. **Per-agent lifecycle lease** — DAG-scoped: acquired at the DAG's first
container-affecting node (`StopForUpdate`, `Swap`, `Signal`, `Drain`,
`Reconcile`, `WriteDropin`, `Create`, `ApprovalDeploy`), held until the DAG
is terminal, so two lifecycle DAGs for one agent never interleave their
container ops. **Lease-exempt**: `Prebuild`, `MetaLock`, `WritePermFile` —
they touch the store / meta, not the running container, which is exactly
why a stop can land while another DAG's prebuild is still building.
**Worker re-notification**: the worker drains `take_next()` in a tight loop after
each entry finishes. When a dep entry transitions to terminal, the loop re-evaluates
the queue immediately, so downstream entries are unblocked with no extra wakeup. No
additional `notify_one()` call is needed.
Among simultaneously-ready nodes competing for a resource, DAG-submit order
wins (FIFO) so bulk operations drain predictably. The scheduler also owns the
DAG-lifetime transient guard (dashboard pill + crash-watch suppression),
created on lease acquisition and dropped when the DAG settles terminal.
**Circular-dep caveat**: if A depends on B and B depends on A, neither entry ever
becomes runnable — the worker skips both indefinitely with no error. Callers must
ensure acyclic dep graphs. Cycle detection is deferred to a future iteration (when
parallel workers make a stuck queue more visible).
The queue is in-memory only and lost on hive-c0re restart — deliberate:
desired state is re-derived at boot from the DB + rev markers (see _Boot
reconcile_), so there is no durable-recovery machinery to go wrong.
### Dedup, cancel, history
Dedup at **DAG granularity**: a repeat submit against a DAG whose roll-up is
still `Queued` with the same `(template, agent, parent_id, approval_id)` —
plus `inputs` for meta-updates and the perm-type discriminant for perm
changes — returns the existing id and appends an "also requested by …" line.
`parent_id` in the key keeps a cascade child from collapsing into a
standalone or sweep rebuild. Running/terminal DAGs never dedup.
Cancel only applies to still-fully-queued DAGs (an in-flight nix build isn't
interruptible); `cancel_children` cancels a parent's still-queued child DAGs.
Roll-up state: `Failed` if any node failed, else `Running` / `Queued` /
`Cancelled` / `Done`. The snapshot retains the 5 most recent terminal DAGs
per template.
### Approvals
`ApplyCommit` / `MergeConfigPr` approvals ride as single-node
`ApprovalDeploy` DAGs: the two-phase `prepare_deploy` / `finalize_deploy` /
`abort_deploy` meta orchestration stays inside `actions.rs` in v1
(deliberately not modeled as scheduler nodes) and resolves the approval
itself. `Spawn` and `UpdateMetaInputs` approvals map onto the ordinary
`spawn` / `meta-update` shapes; the scheduler fires
`actions::resolve_approval_dag` exactly once when such a DAG settles
terminal (including cancelled-while-queued, which fails the approval instead
of dangling it).
### Wire shape
`RebuildQueueChanged { seq, queue: [DagView…] }` (event name kept). Each
`DagView` carries the old entry-level fields (`id`, `kind` = template string,
roll-up `state`, `agent`, `source`, `parent_id`, `reason`, timestamps,
`inputs`, `approval_id`) plus `nodes: [NodeView…]` — per-node `kind`, `deps`,
`state`, `step`, `build_log_id`, timestamps, `error`. Step labels and build
logs are **per-node**; the dashboard renders the node chain on each queue
card and keys the live-log panel off the running node.
---
@ -115,31 +204,31 @@ render.
---
## Auto-update sweep
## Boot reconcile
On startup, `auto_update.rs` rebuilds containers that actually need it. Two skip
rules keep boot-time work minimal:
On startup, `auto_update::run` classifies every agent by rev freshness (the
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
vs the current flake path) and persisted `wanted` intent, then:
1. **Stopped containers** are deferred: the startup sweep enqueues nothing for them.
When the operator later starts a stopped container (via the dashboard or the
`start` MCP tool), both `run_start` (queue path) and `handle_start` (socket path)
check the rev marker first — if it's stale, the start is silently upgraded to a
full rebuild+start so the container runs current nix derivations.
1. **Config path** — when *any* marker is stale, submit one `StartupSweep`
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that fans out `Rebuild`
children for the stale agents whose `wanted = Up` (topology-sorted, parents
first). Stale but wanted-offline agents get no boot-time nix work — their
rebuild happens on their next start (the start submit path upgrades a
stale start to rebuild+start), which is also why the lock bump runs even
when every stale agent is offline: those later start-upgrades must build
against the bumped lock. Each child rebuild's tail `Reconcile` brings the
agent (back) up, covering both the running-stale and stopped-but-wanted-up
cases.
2. **Running containers with a matching rev marker** are skipped: if the per-agent
`.{name}.hyperhive-rev` file under `/var/lib/hyperhive/applied/` already holds
the current flake rev, no nix work is needed and the entry is omitted entirely.
2. **Power path** — every agent whose observed state drifted from `wanted`
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
`auto_update::run` enqueues a single `StartupSweep` parent entry (`kind =
startup_sweep`, `agent = "hyperhive"`) followed by per-agent `Rebuild` children
for the agents that do need rebuilding (`source = startup_sweep`, `parent_id =
sweep_id`). The sweep description records the rebuild / deferred / skipped counts
so the operator can see at a glance how much work the boot triggered. The child
rebuilds drain sequentially through the queue; the dashboard renders them nested
under the parent.
Before the sweep-grouping change, each boot enqueued flat `Rebuild` entries with
`source = AutoUpdate` and no parent — visible but ungrouped.
Booting with no config change performs **no meta commit** — only reconciles.
The sweep reason records the rebuild / deferred / up-to-date counts so the
operator sees at a glance how much work the boot triggered. Agents without an
`agent_power` row are seeded from observed state during classification (the
one-time migration; thereafter the DB is authoritative).
## Meta flake
@ -157,8 +246,13 @@ Key operations:
`RequestApplyCommit` path so a failed `nixos-container update` leaves no orphan
commit in meta. Prepare writes the new lock without committing; finalize commits
with the deploy message; abort restores the lock.
- **`lock_update_hyperhive`** — one-shot for the auto-update path: bumps the
`hyperhive` input lock, commits, cascades agent rebuilds.
- **`lock_update_hyperhive`** — one-shot for the boot-reconcile path (the
sweep DAG's `MetaLock` node): bumps the `hyperhive` input lock and commits;
the scheduler fans out the agent rebuilds on completion.
Every public `meta.rs` operation takes the module's internal `META_LOCK`
mutex, so concurrent job-queue nodes (and the approval deploy pipeline) never
race on the repo's `.git/index.lock`.
---
@ -179,34 +273,26 @@ new profile and skips the in-container `switch-to-configuration`. The subsequent
`start` then applies both the new profile and any `EXTRA_NSPAWN_FLAGS` changes in
one go, rather than the double-bounce a live `update` would trigger.
Sequence for a running container:
Sequence for a rebuild DAG (each step is its own queue node):
1. `prebuild_toplevel` — build the new `system.build.toplevel` **before** stopping.
The container keeps serving the previous generation while eval + fetch + build
happen out-of-band. `nixos-container update` then finds the result cached and
skips straight to the profile-swap. Build failures surface here, before the
running container is touched.
2. `nixos-container stop` — bring the container down.
3. `nixos-container update --flake meta#<name>` — profile-swap (near-instant after
the prebuild).
4. `nixos-container start` — boot into the new generation; the in-container
activation script transitions old → new.
1. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
The container keeps serving the previous generation while eval + fetch +
build happen out-of-band. `nixos-container update` then finds the result
cached and skips straight to the profile-swap. Build failures surface
here, before the running container is touched. (Runs even for a stopped
container — same total nix work, one uniform DAG shape.)
2. `StopForUpdate` — bring the container down (noop when already stopped).
3. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
(near-instant after the prebuild).
4. `Reconcile` — boot into the new generation when `wanted = Up`; the
in-container activation script transitions old → new. Holds no build
slot, so the next DAG's `Prebuild` overlaps the container boot — the old
"deferred start" split, now structural.
If the container is already stopped, step 1 is skipped (no downtime to shave — no
point evaluating the flake twice).
**Deferred start (queue-dispatched rebuilds):** step 4 can take a while
(container boot), and holding the serialized build lane through it delays the
next queued rebuild's nix build for no reason. Queue-dispatched rebuilds
therefore pass `defer_start` — `rebuild_no_meta` skips the start and returns
`true`, and `rebuild_agent` enqueues a fast-lane `Start` entry instead
(`parent_id` = the rebuild entry, so the dashboard groups the follow-up under
it — the same split the graceful-stop path uses for its container stop). The
build-lane entry completes when the profile-swap finishes; a start failure
surfaces on the `Start` entry, which runs with the cold-start fallback. Direct
callers (admin-socket CLI, root-agent migration nudge, the apply-commit deploy
flow which verifies the agent comes back up before finalizing) keep the start
inline.
The approval apply-commit pipeline still drives `lifecycle::rebuild_no_meta`
(the fused stop/update/start path with an inline start) inside its
`ApprovalDeploy` node, because it verifies the agent comes back up before
finalizing the deploy tag.
### Cold-start fallback
@ -218,9 +304,8 @@ half-started at that point.
Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
→ `start` (clean cold-start, no generation transition, new activation runs cleanly).
Both errors are preserved and surfaced if the cold-start also fails. The fallback
lives in `lifecycle::start_with_fallback`, shared by the inline start-after-rebuild
path and the queue's fast-lane `Start` handler (which the deferred
start-after-rebuild rides).
lives in `lifecycle::start_with_fallback`, shared by the apply-commit deploy's
inline start and every `Reconcile` node's start action.
### Spawn path (new container)
@ -241,9 +326,18 @@ or the flake root directly — none of which exist in the rendered meta flake.
## Host-level resource + performance options
Three `services.hyperhive.c0re.*` options tune container resource
limits and first-spawn latency. All three apply uniformly to every
agent container.
A handful of `services.hyperhive.c0re.*` options tune container resource
limits, build parallelism, and first-spawn latency.
### Build slots
`buildSlots` (default `1`) sets how many nix-heavy job-queue nodes
(prebuilds, profile swaps, first-spawn creates, meta lock bumps) run
concurrently. The default serializes all heavy nix work like the pre-DAG
rebuild queue did; raise it on hosts with the cores/RAM to build several
agent toplevels at once. Per-agent correctness is independent of the count —
each agent's container-affecting ops serialize on its lifecycle lease
regardless.
### Container resource limits

View file

@ -2,12 +2,12 @@
Where state lives, what survives what, and how it's bounded.
## Three sqlite databases
## Sqlite databases
### `/var/lib/hyperhive/db/broker.sqlite` (host)
Six tables, all in one file — four queues plus the schedule
header/targets split:
Seven tables, all in one file — four queues, the schedule
header/targets split, and the per-agent power-intent registry:
- `messages` — every inter-agent / operator-bound message.
`sender / recipient / body / sent_at / delivered_at / acked_at /
@ -44,6 +44,17 @@ header/targets split:
last_fired_at_unix / last_result`. `ON DELETE CASCADE` from
`scheduled_prompts(id)` — requires `PRAGMA foreign_keys = ON`
per connection (set at open).
- `agent_power` — one tiny row per agent: `agent PK / wanted (up |
offline) / updated_at` — the durable power *intent* behind the job
queue's desired-state reconciliation
(`docs/coordinator.md::Job queue`; owner: `hive-c0re/src/stores/power.rs`).
Written synchronously by every operator/agent power action
(dashboard start/stop, `hivectl stop`, the MCP kill/start tools,
spawn approval); read by `Reconcile` nodes and the boot reconcile.
Intent survives hive-c0re restarts — in-flight queue work
deliberately does not. Agents without a row are seeded from
observed state on first touch (running ⇒ `up`); destroy deletes
the row.
Retention:
@ -63,6 +74,8 @@ Retention:
(`cancel_schedule` MCP / dashboard ✗) which tombstones via
`cancelled_at_unix`, then `reap_cancelled` drops the row on
the next worker pass.
- `agent_power` rows live until the agent is destroyed (one row per
agent — nothing to vacuum).
### `/harness/hyperhive-events.sqlite` (per agent)

View file

@ -271,8 +271,8 @@ Agent container management. Requires the hive-c0re daemon to be running (connect
###### **Subcommands:**
* `list` — Show all managed agents with their status (running / needs-login / needs-update) and technical state (deployed sha, parent, pending reminders). The host roster overview; reuses the dashboard's per-agent aggregation. Requires the daemon running
* `restart` — Stop and start a single agent container without rebuilding config. Useful for "kick the container" when the process is stuck or the container needs a clean restart without changing the NixOS config
* `restart-all` — Stop and restart ALL managed agent containers in sequence. Iterates the live container list and restarts each one. Any per-agent failure is reported at the end rather than stopping mid-run, so all containers get a restart attempt
* `restart` — Stop and start a single agent container without rebuilding config. Useful for "kick the container" when the process is stuck or the container needs a clean restart without changing the NixOS config. Rides the job queue (serialized against in-flight rebuilds for the same agent); waits with live progress unless `--no-wait`
* `restart-all` — Restart ALL managed agent containers via one restart DAG each — unrelated agents overlap, each serializes on its own lease. Waits for the whole set with live progress unless `--no-wait`
@ -290,21 +290,29 @@ Show all managed agents with their status (running / needs-login / needs-update)
## `hivectl agents restart`
Stop and start a single agent container without rebuilding config. Useful for "kick the container" when the process is stuck or the container needs a clean restart without changing the NixOS config
Stop and start a single agent container without rebuilding config. Useful for "kick the container" when the process is stuck or the container needs a clean restart without changing the NixOS config. Rides the job queue (serialized against in-flight rebuilds for the same agent); waits with live progress unless `--no-wait`
**Usage:** `hivectl agents restart <NAME>`
**Usage:** `hivectl agents restart [OPTIONS] <NAME>`
###### **Arguments:**
* `<NAME>` — Agent name (e.g. `damocles`, `ruth`)
###### **Options:**
* `--no-wait` — Return immediately after the restart DAG is queued
## `hivectl agents restart-all`
Stop and restart ALL managed agent containers in sequence. Iterates the live container list and restarts each one. Any per-agent failure is reported at the end rather than stopping mid-run, so all containers get a restart attempt
Restart ALL managed agent containers via one restart DAG each — unrelated agents overlap, each serializes on its own lease. Waits for the whole set with live progress unless `--no-wait`
**Usage:** `hivectl agents restart-all`
**Usage:** `hivectl agents restart-all [OPTIONS]`
###### **Options:**
* `--no-wait` — Return immediately after the restart DAGs are queued
@ -407,7 +415,8 @@ Stop containers hive-wide in one operator action. Bare `hivectl stop` stops **ev
* `--forge` — The forge container (`hive-forge`)
* `--gateway` — The gateway container (`hive-gateway`)
* `--matrix` — The matrix container (`hive-matrix`)
* `--graceful` — Gracefully quiesce each agent before stopping, instead of a hard stop. Each agent is enqueued as a `GracefulStop` on the rebuild queue: the harness is signalled, runs one stop-checkpoint turn to flush durable `/state`, drains, then the container is stopped (bounded by a 3-min timeout that falls back to a hard stop). Applies to agents only
* `--graceful` — Gracefully quiesce each agent before stopping, instead of a hard stop. Each agent gets a graceful-stop DAG on the job queue: the harness is signalled, runs one stop-checkpoint turn to flush durable `/state`, drains, then the container is stopped (bounded by a 3-min timeout that falls back to a hard stop). All drains overlap. Applies to agents only
* `--no-wait` — Return immediately after the stop DAGs are queued instead of waiting for them with live per-node progress
@ -425,6 +434,7 @@ Start containers hive-wide — the inverse of `hivectl stop`. Bare `hivectl star
* `--forge` — The forge container (`hive-forge`)
* `--gateway` — The gateway container (`hive-gateway`)
* `--matrix` — The matrix container (`hive-matrix`)
* `--no-wait` — Return immediately after the start DAGs are queued instead of waiting for them with live per-node progress

View file

@ -131,6 +131,7 @@ const QUEUE_KIND_GLYPH = {
graceful_stop: '⏹',
start: '▶',
stop: '■',
reconcile: '⇄',
};
const QUEUE_STATE_GLYPH = {
queued: '⏸',
@ -139,11 +140,33 @@ const QUEUE_STATE_GLYPH = {
failed: '✖',
cancelled: '⊘',
};
// Short display labels for the per-DAG node kinds (the primitive ops).
const NODE_KIND_LABEL = {
prebuild: 'prebuild',
stop_for_update: 'stop',
swap: 'swap',
create: 'create',
meta_lock: 'meta lock',
reconcile: 'reconcile',
signal: 'signal',
drain: 'drain',
write_dropin: 'dropin',
write_perm_file: 'perm file',
approval_deploy: 'deploy',
};
// The currently-running node of a DAG (per-node `step` labels + build
// logs live on nodes now; the DAG's `state` is a roll-up).
function runningNode(entry) {
return (entry.nodes || []).find((n) => n.state === 'running') || null;
}
function firstFailedNode(entry) {
return (entry.nodes || []).find((n) => n.state === 'failed') || null;
}
function rebuildQueueEntryFingerprint(entry, isChild) {
return JSON.stringify({
state: entry.state,
step: entry.step,
kind: entry.kind,
agent: entry.agent,
source: entry.source,
@ -151,8 +174,7 @@ function rebuildQueueEntryFingerprint(entry, isChild) {
enqueued_at: entry.enqueued_at,
finished_at: entry.finished_at,
reason: entry.reason,
error: entry.error,
build_log_id: entry.build_log_id,
nodes: (entry.nodes || []).map((n) => [n.kind, n.state, n.step, n.build_log_id, n.error]),
isChild,
});
}
@ -257,22 +279,39 @@ function renderQueueEntry(entry, _byId, isChild) {
const r = entry.reason.split('\n')[0];
li.append(' ', el('span', { class: 'rqe-reason', title: entry.reason }, '— ' + truncate(r, 60)));
}
if (entry.step) {
li.append(el('div', { class: 'rqe-step' }, '↳ ' + entry.step));
// Per-node chain: every DAG node in dependency order with its own
// state glyph, live step label, and build-log link. This is the
// node-aware render that makes queue jumps / partial progress
// visible (e.g. reconcile running while swap failed).
const nodes = entry.nodes || [];
if (nodes.length) {
const chain = el('div', { class: 'rqe-nodes' });
nodes.forEach((n, i) => {
if (i > 0) chain.append(el('span', { class: 'rqe-node-arrow' }, ' → '));
const chip = el('span', {
class: 'rqe-node rqe-node-' + n.state,
title: n.kind + ' · ' + n.state + (n.error ? ' — ' + n.error : ''),
},
(QUEUE_STATE_GLYPH[n.state] || '?') + ' ' + (NODE_KIND_LABEL[n.kind] || n.kind));
chain.append(chip);
if (n.build_log_id != null) {
chain.append(el('a', {
class: 'rqe-log-link rqe-node-log',
href: '/builds.html?id=' + n.build_log_id + '#buildlogs',
target: '_blank',
title: 'view build log #' + n.build_log_id,
}, '⎙'));
}
});
li.append(chain);
}
if (entry.build_log_id != null) {
li.append(
' ',
el('a', {
class: 'rqe-log-link',
href: '/builds.html?id=' + entry.build_log_id + '#buildlogs',
target: '_blank',
title: 'view build log #' + entry.build_log_id,
}, 'logs →'),
);
const running = runningNode(entry);
if (running && running.step) {
li.append(el('div', { class: 'rqe-step' }, '↳ ' + running.step));
}
if (entry.error) {
li.append(el('pre', { class: 'rqe-error', title: entry.error }, truncate(entry.error, 200)));
const failed = firstFailedNode(entry);
if (failed && failed.error) {
li.append(el('pre', { class: 'rqe-error', title: failed.error }, truncate(failed.error, 200)));
}
if (entry.state === 'queued') {
const cancelForm = el('form', {
@ -296,11 +335,12 @@ function renderQueueEntry(entry, _byId, isChild) {
}
// ─── running-rebuild live log ─────────────────────────────────────────────────
// One persistent live-log panel for the currently-running rebuild (the queue
// runs one build at a time). Keyed to the running entry's build_log_id and
// kept in its own container (#rebuild-live-log) so the queue's row re-render
// — which rebuilds rows as the build step advances — never tears down the open
// SSE stream.
// One persistent live-log panel for the first currently-building node
// (build logs are per-node now; with buildSlots = 1 at most one nix
// build runs at a time). Keyed to that node's build_log_id and kept in
// its own container (#rebuild-live-log) so the queue's row re-render —
// which rebuilds rows as the build step advances — never tears down the
// open SSE stream.
let liveLogEs = null;
let liveLogId = null;
let liveLogDone = false;
@ -310,23 +350,36 @@ function closeLiveLogStream() {
if (liveLogEs) { liveLogEs.close(); liveLogEs = null; }
}
// First (dag, node) pair with a running node that opened a build log.
function findLiveBuild(queue) {
for (const e of queue || []) {
if (e.state !== 'running') continue;
for (const n of e.nodes || []) {
if (n.state === 'running' && n.build_log_id != null) {
return { entry: e, node: n };
}
}
}
return null;
}
function renderRebuildLiveLog(queue) {
const root = $('rebuild-live-log');
if (!root) return;
const running = (queue || []).find(
(e) => e.state === 'running' && e.build_log_id != null);
const live = findLiveBuild(queue);
if (!running) {
if (!live) {
closeLiveLogStream();
liveLogId = null;
liveLogDone = false;
if (!root.hidden) { root.hidden = true; root.replaceChildren(); }
return;
}
if (running.build_log_id === liveLogId && (liveLogEs || liveLogDone)) return;
const { entry: running, node: liveNode } = live;
if (liveNode.build_log_id === liveLogId && (liveLogEs || liveLogDone)) return;
closeLiveLogStream();
liveLogId = running.build_log_id;
liveLogId = liveNode.build_log_id;
liveLogDone = false;
root.hidden = false;
root.replaceChildren();
@ -351,17 +404,18 @@ function renderRebuildLiveLog(queue) {
toggle, ' ',
el('span', { class: 'rebuild-live-log-title' }, 'live build log — '),
el('code', { class: 'rqe-agent' }, running.agent),
' ', el('span', { class: 'rqe-kind' }, running.kind || 'rebuild'),
' ', el('span', { class: 'rqe-kind' },
(running.kind || 'rebuild') + ' · ' + (NODE_KIND_LABEL[liveNode.kind] || liveNode.kind)),
' ', badge, ' ',
el('a', {
class: 'rebuild-live-log-raw',
href: '/api/build-logs/id/' + running.build_log_id + '/raw',
download: 'build-log-' + running.build_log_id + '.txt',
href: '/api/build-logs/id/' + liveNode.build_log_id + '/raw',
download: 'build-log-' + liveNode.build_log_id + '.txt',
}, '↓ raw'),
);
root.append(header, pre);
liveLogEs = openBuildLogStream(running.build_log_id, pre, {
liveLogEs = openBuildLogStream(liveNode.build_log_id, pre, {
onDone: (status) => {
liveLogDone = true;
badge.className = 'rebuild-live-log-badge ' + (status === 'ok' ? 'rll-ok' : 'rll-fail');

View file

@ -811,6 +811,7 @@ export function renderContainers(s) {
: op.kind === 'start' ? 'starting'
: op.kind === 'stop' ? 'stopping'
: op.kind === 'graceful_stop' ? 'stopping'
: op.kind === 'reconcile' ? 'reconciling'
: 'rebuilding')
: (op.kind === 'meta_update' ? 'meta-update queued'
: op.kind === 'destroy' ? 'destroy queued'
@ -818,6 +819,7 @@ export function renderContainers(s) {
: op.kind === 'start' ? 'start queued'
: op.kind === 'stop' ? 'stop queued'
: op.kind === 'graceful_stop' ? 'stop queued'
: op.kind === 'reconcile' ? 'reconcile queued'
: 'rebuild queued')));
const opRunning = transientKind != null
|| (op != null && op.state === 'running');

View file

@ -134,6 +134,33 @@
.rqe-source-approval { color: var(--green); border-color: var(--green); }
.rqe-when { color: var(--muted); font-size: 0.85em; }
.rqe-reason { color: var(--muted); font-size: 0.85em; flex: 1 1 auto; }
/* Per-node DAG chain: one chip per primitive node, dependency order. */
.rqe-nodes {
flex-basis: 100%;
margin: 0.15em 0 0 1.8em;
font-size: 0.85em;
display: flex;
flex-wrap: wrap;
align-items: baseline;
gap: 0.15em;
}
.rqe-node {
padding: 0.05em 0.45em;
border: 1px solid var(--border);
border-radius: 0.7em;
color: var(--muted);
white-space: nowrap;
}
.rqe-node-running {
color: var(--purple);
border-color: var(--purple);
animation: badge-pulse 1.6s ease-in-out infinite;
}
.rqe-node-done { color: var(--green); border-color: color-mix(in srgb, var(--green) 45%, transparent); }
.rqe-node-failed { color: var(--red); border-color: var(--red); }
.rqe-node-cancelled { opacity: 0.55; text-decoration: line-through; }
.rqe-node-arrow { color: var(--muted); }
.rqe-node-log { margin-left: 0.1em; text-decoration: none; }
.rqe-step {
flex-basis: 100%;
margin: 0.1em 0 0 1.8em;

View file

@ -13,6 +13,7 @@ use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use hive_claude::TokenUsage;
use hive_sh4re::wire_time::now_unix;
use rusqlite::{Connection, params};
use serde::{Deserialize, Serialize};
use tokio::sync::broadcast;
@ -213,14 +214,6 @@ pub fn write_forge_cursor<S: std::hash::BuildHasher>(
write_harness_json(&v);
}
fn now_unix() -> i64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.ok()
.and_then(|d| i64::try_from(d.as_secs()).ok())
.unwrap_or(0)
}
const SCHEMA: &str = "
CREATE TABLE IF NOT EXISTS events (
id INTEGER PRIMARY KEY AUTOINCREMENT,

359
hive-ag3nt/src/mcp/args.rs Normal file
View file

@ -0,0 +1,359 @@
//! Argument structs for the MCP tools: `serde::Deserialize` +
//! `schemars::JsonSchema` derives whose field doc-comments become the
//! parameter descriptions claude sees in each tool's input schema.
use rmcp::schemars;
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct SendArgs {
/// Logical agent name to deliver the message to (e.g. `"manager"`,
/// `"alice"`, or the literal `"operator"` for the dashboard's T4LK box).
pub to: String,
/// Message body. Plain text; the broker doesn't parse it.
pub body: String,
/// Optional broker row-id of the message this is a reply to. Lets
/// the dashboard render conversation threads. Pass the `id` from the
/// `DeliveredMessage` you're responding to; omit for new threads.
/// Silently ignored if the id is unknown or out of retention.
#[serde(default)]
pub in_reply_to: Option<i64>,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct RecvArgs {
/// How long to long-poll for the FIRST message before returning
/// the empty marker. Capped at 60s server-side. Default (None)
/// is 30s. Useful when an agent wants to park its turn waiting
/// for any new work — pick a longer wait to coalesce bursts.
#[serde(default)]
pub wait_seconds: Option<u64>,
/// Maximum number of messages to pop in this round-trip. Default
/// (None) is 1 (single-message behaviour — exactly what you want
/// when you're called to drive a turn off the first wake). Pass
/// a higher value (capped at 5 server-side) when you've been
/// told the inbox has more queued (the wake prompt mentions
/// pending count) and want to drain everything in one tool call.
/// Once the long-poll wakes up, the call drains up to `max` in
/// total before returning — no extra round-trip needed.
#[serde(default)]
pub max: Option<u32>,
}
/// MCP tool args for `ack_until`.
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct AckUntilArgs {
/// Highest broker message id to mark handled: every inbox message
/// with `id <= up_to` (ids show as `[msg #<id>]` in recv output)
/// is acked in one sweep. Pass the highest id you've actually
/// seen/triaged — anything above it stays queued for later turns.
pub up_to: i64,
}
/// MCP tool args for `remind`. Exactly one of `delay_seconds` or
/// `at_unix_timestamp` must be set; both / neither is a tool-side error.
/// Hides the tagged `ReminderTiming` enum behind a flatter schema so the
/// model picks one field instead of building `{"timing_type": "in_seconds",
/// "seconds": 60}` shaped objects.
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct RemindArgs {
/// Body that lands in your inbox when the reminder fires (sender
/// will appear as `reminder`). Soft cap at 4 KiB inline — anything
/// larger gets auto-persisted to a file under
/// `/agents/<you>/state/reminders/auto-<ts>.md` and the inbox
/// message becomes a short pointer. Pass `file_path` if you want
/// to control the destination yourself.
pub message: String,
/// Fire `delay_seconds` from now (relative). Set this OR
/// `at_unix_timestamp`, not both.
#[serde(default)]
pub delay_seconds: Option<u64>,
/// Fire at this absolute unix timestamp (seconds since epoch). Set
/// this OR `delay_seconds`, not both.
#[serde(default)]
pub at_unix_timestamp: Option<i64>,
/// Optional path to a file the scheduler should reference instead of
/// inlining a long `message`. Use this for large payloads (research
/// notes, file lists, intermediate state). Path must be reachable from
/// the agent's container — typically under `/agents/<you>/state/`.
#[serde(default)]
pub file_path: Option<String>,
}
// -----------------------------------------------------------------------------
// Privileged tool arg types (lifecycle, approvals, scheduling, diagnostics)
// -----------------------------------------------------------------------------
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct RequestInitConfigArgs {
/// New sub-agent name (≤9 chars). Queues an `InitConfig` approval; on
/// approval hive-c0re seeds the proposed config repo at
/// `/agents/<name>/config/agent.nix` with the default template. After
/// the approval the manager edits + commits the config and calls
/// `request_apply_commit` to pin the customised sha for the container's
/// first build.
pub name: String,
/// Optional description shown on the dashboard approval card.
#[serde(default)]
pub description: Option<String>,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct KillArgs {
/// Sub-agent name (without the `h-` container prefix).
pub name: String,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct SetStatusArgs {
/// Status text to display on the dashboard card. Pass an empty string to clear.
pub text: String,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct CreateRepoArgs {
/// Repo name — a single segment of letters, digits, `-`, `_`, `.`
/// (no leading `-`/`.`). The repo is created as `agents/<repo>`.
pub repo: String,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct GetAgentMetaArgs {
/// Logical name of the agent to query (e.g. `"iris"`, `"manager"`).
/// Omit to query your own identity + status — replaces the
/// previous `whoami` self-introspection tool.
#[serde(default)]
pub name: Option<String>,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct StartArgs {
/// Sub-agent name (without the `h-` container prefix).
pub name: String,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct RestartArgs {
/// Sub-agent name (without the `h-` container prefix).
pub name: String,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct UpdateArgs {
/// Sub-agent name (without the `h-` container prefix).
pub name: String,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct AskArgs {
/// The question to surface.
pub question: String,
/// Optional fixed-choice answers. The dashboard renders these as
/// chips alongside a free-text fallback ("Other…") so the operator
/// is never trapped by an incomplete list; peer-agent recipients
/// see the list in their inbox event and can return any string.
#[serde(default)]
pub options: Vec<String>,
/// When true, options are rendered as checkboxes — the answerer
/// can pick any subset. The answer comes back as a single string
/// with selections joined by ", ". Ignored when `options` is empty.
#[serde(default)]
pub multi: bool,
/// Optional auto-cancel after `ttl_seconds` (capped server-side at
/// 6 hours). On expiry the question resolves with answer
/// `[expired]` and the asker receives the usual
/// `question_answered` system event (with `answerer:
/// "ttl-watchdog"`). `None` (default) = wait indefinitely.
#[serde(default)]
pub ttl_seconds: Option<u64>,
/// Recipient. Omit (or pass `"operator"`) to ask the human
/// operator via the dashboard. Pass another agent's logical name
/// to ask that peer — they receive a `question_asked` event in
/// their inbox and answer via `mcp__hyperhive__answer`.
#[serde(default)]
pub to: Option<String>,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct AnswerArgs {
/// Id of the question being answered — comes from the
/// `question_asked` event in your inbox.
pub id: i64,
/// Free-text answer body. Soft-capped at 4 KiB by the same
/// `MESSAGE_MAX_BYTES` limit as `send`; keep it short or write the
/// detail to a file and pass a path.
pub answer: String,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct CancelLooseEndArgs {
/// Which kind of thread to cancel — `"question"` for an open
/// `ask` that's still waiting on an answer, `"reminder"` for a
/// scheduled `remind` that hasn't fired yet. Use the `kind`
/// field straight off the `get_loose_ends` row.
pub kind: String,
/// Row id from the matching `get_loose_ends` entry (or the
/// `question_queued` reply when you submitted it).
pub id: i64,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct AgentGetLooseEndsArgs {
/// Whose loose ends to list. Omit (or `null`) for your own. You may
/// also pass a direct child agent's name without any extra capability.
/// Pass any other agent name to inspect their threads — requires the
/// `query_agent_state` capability; without it the request is rejected
/// with an error. The `"*"` hive-wide value is not available on the
/// agent socket.
#[serde(default)]
pub agent: Option<String>,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct RequestApplyCommitArgs {
/// Logical agent name whose config repo the commit lives in.
pub agent: String,
/// Commit sha (full or short, 7-40 hex chars) in that agent's
/// proposed config repo. Must be a sha — a branch or tag name
/// (e.g. `main`) is rejected; the approval pins the exact commit.
pub commit_ref: String,
/// Optional description shown on the dashboard approval card so the
/// operator knows what the change does without opening the diff.
#[serde(default)]
pub description: Option<String>,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct UpdateMetaInputsArgs {
/// Flake input names to update (e.g. `["bitburner-agent", "nixpkgs"]`).
/// Pass an empty list to update ALL inputs.
#[serde(default)]
pub inputs: Vec<String>,
/// Optional description shown on the dashboard approval card.
#[serde(default)]
pub description: Option<String>,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct RequestSchedulePromptArgs {
/// Recipient agents — one schedule fires to many inboxes at the
/// scheduled time. `operator` is a legitimate target (mara: "we
/// want to get rid of the manager special case so yes manager
/// can be recipient" — the operator slot follows the same rule).
pub targets: Vec<String>,
/// Message body delivered to each target's inbox at fire time.
/// Same size budget as `send` bodies.
pub body: String,
/// Absolute unix timestamp (seconds) for the FIRST fire. For
/// recurring schedules the worker re-arms in
/// `interval_seconds` steps from this point on.
pub first_fire_at_unix: i64,
/// `None` / absent = one-shot. `Some(n > 0)` = recurring every
/// `n` seconds. The worker clamps catch-up so a long downtime
/// fires ONCE on resume (skipped-cycle count surfaces in the
/// per-target `last_result`), not N delayed pulses in a row.
#[serde(default)]
pub interval_seconds: Option<u64>,
/// Optional description shown on the dashboard approval card +
/// preserved on the schedule row for later operator reference.
#[serde(default)]
pub description: Option<String>,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct FireScheduleNowArgs {
/// Schedule id to fire out of band. Get this from a prior
/// `list_schedules` call or the approval-resolved event for
/// the originating `request_schedule_prompt`.
pub id: i64,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct CancelScheduleArgs {
/// Schedule id from a prior `list_schedules` call or the
/// approval-resolved event for a `request_schedule_prompt`.
pub id: i64,
/// Optional target list. `None` / empty = cancel the entire
/// schedule. `Some(["alice", "bob"])` = cancel just those
/// recipients (the schedule keeps firing for any remaining
/// active targets, and auto-cancels its parent row when every
/// target is gone).
#[serde(default)]
pub targets: Option<Vec<String>>,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct EditScheduleArgs {
/// Schedule id from a prior `list_schedules` call or the
/// approval-resolved event for a `request_schedule_prompt`.
pub id: i64,
/// New body text. Omit to keep the existing one.
#[serde(default)]
pub body: Option<String>,
/// New description. Omit to keep the existing one. (To CLEAR
/// the description, use the dashboard PATCH endpoint
/// directly — the agent surface intentionally keeps the args
/// flat / non-nullable to dodge the doubly-wrapped Option
/// schemars quirk; clearing fields is rare and operator-side.)
#[serde(default)]
pub description: Option<String>,
/// Recurring interval in seconds. Omit to keep the existing
/// cadence; pass an explicit value to set a new one. Toggling
/// recurring↔one-shot (clearing the interval) is operator-only
/// for the same reason as `description` above.
#[serde(default)]
pub interval_seconds: Option<u64>,
/// New absolute unix timestamp for the next fire. Omit to
/// leave the schedule on its current cadence.
#[serde(default)]
pub next_fire_at_unix: Option<i64>,
/// Names of new targets to add. Replace-on-conflict: re-adding
/// a previously cancelled target resets its history (operator
/// intent on re-add = "this target is active again").
#[serde(default)]
pub targets_add: Option<Vec<String>>,
/// Names of targets to cancel. Tombstones preserve per-target
/// audit; when no active targets remain the schedule
/// auto-cancels.
#[serde(default)]
pub targets_remove: Option<Vec<String>>,
}
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct GetLogsArgs {
/// Logical agent name to fetch logs for (e.g. `gui`, `iris`).
/// hive-c0re maps it to the underlying machine name (`h-gui`)
/// itself — pass the plain agent name, not the `h-` form.
pub agent: String,
/// How many journal lines to return (default: 50, max: 500).
#[serde(default)]
pub lines: Option<u32>,
}
/// Arguments for `get_host_journal` (capability-gated: `read_host_journal`).
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct GetHostJournalArgs {
/// Systemd unit to filter (e.g. `hive-c0re.service`). Omit for all units.
#[serde(default)]
pub unit: Option<String>,
/// nspawn machine name verbatim (e.g. `h-iris`). Omit for host journal.
/// Agent containers use the `h-` prefix (e.g. `h-iris`); infrastructure
/// containers use their full name (e.g. `hive-ci`, `hive-forge`,
/// `hive-matrix`, `hive-gateway`).
#[serde(default)]
pub container: Option<String>,
/// Number of lines to return (default 30, max 100).
#[serde(default)]
pub lines: Option<u32>,
/// Minimum syslog priority level.
#[serde(default)]
pub priority: Option<hive_sh4re::JournalPriority>,
/// Regex to match against log message fields (journalctl --grep).
#[serde(default)]
pub grep: Option<String>,
/// Show entries on or newer than this timestamp (e.g. `-1h`).
#[serde(default)]
pub since: Option<String>,
/// Show entries on or older than this timestamp.
#[serde(default)]
pub until: Option<String>,
}

1058
hive-ag3nt/src/mcp/mod.rs Normal file

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,448 @@
//! Formatting / render helpers for the MCP tool surface: ack / recv /
//! loose-end / agent-meta reply shaping plus the retry annotation.
//! Stateless string builders, with one exception —
//! [`matrix_unread_summary`] queries the local matrix daemon socket
//! (best-effort) so `get_loose_ends` can prepend an unread-rooms entry.
/// Render the three identical failure arms every data-returning tool handler
/// repeats: a broker `Err` → `"{tool} failed: {m}"`, an unexpected `Ok` variant
/// → `"{tool} unexpected response: …"`, and a transport error → `"{tool}
/// transport error: …"`. Handlers match their own happy-path variant and route
/// everything else here via a catch-all arm (`other => reply_err(other, tool)`),
/// so the triplet lives in exactly one place.
pub(super) fn reply_err(resp: Result<hive_sh4re::Response, anyhow::Error>, tool: &str) -> String {
match resp {
Ok(hive_sh4re::Response::Err { message }) => format!("{tool} failed: {message}"),
Ok(other) => format!("{tool} unexpected response: {other:?}"),
Err(e) => format!("{tool} transport error: {e:#}"),
}
}
/// Format helper for "send-like" tools (anything that expects an `Ok`).
/// `tool` and `ok_msg` only appear in the result string; they don't change
/// behavior.
#[must_use]
pub fn format_ack(
resp: Result<hive_sh4re::Response, anyhow::Error>,
tool: &str,
ok_msg: String,
) -> String {
match resp {
Ok(hive_sh4re::Response::Ok) => ok_msg,
other => reply_err(other, tool),
}
}
/// Format helper for `recv`: renders zero, one, or many popped
/// messages. Empty list collapses to "(empty)" so claude doesn't go
/// hunting for content; when `waited` is set (the call parked on a
/// long-poll that timed out) the empty result also carries
/// [`IDLE_WAIT_HINT`] nudging the model toward other work. A single
/// message renders as the historical `from: X\n\nbody` block (banner
/// first if `redelivered`). A multi-message batch renders with a
/// `popped N message(s):` header and `---` separators between bodies
/// so the model can tell where one ends and the next begins;
/// per-message redelivery banners included.
#[must_use]
pub fn format_recv(resp: Result<hive_sh4re::Response, anyhow::Error>, waited: bool) -> String {
match resp {
Ok(hive_sh4re::Response::Messages { messages }) => render_recv_messages(&messages, waited),
// A graceful stop is pending — the inbox is fenced. Render a single
// explicit directive (not an empty inbox, which claude's "park on recv"
// habit would long-poll again, stalling the stop-checkpoint turn until
// the drain wait times out into a hard stop) so every recv during the
// stop unmissably tells claude to flush + end.
Ok(hive_sh4re::Response::GracefulStop) => {
render_recv_messages(&[graceful_stop_message()], waited)
}
other => reply_err(other, "recv"),
}
}
/// The synthetic single-message directive rendered for a fenced (graceful-stop)
/// inbox — see the `GracefulStop` arm of [`format_recv`].
fn graceful_stop_message() -> hive_sh4re::DeliveredMessage {
hive_sh4re::DeliveredMessage {
from: "graceful-stop".into(),
body: "⛔ GRACEFUL STOP IN PROGRESS — the container shuts down as soon as this turn \
ends. Flush anything worth keeping to your durable /state files, then END \
YOUR TURN now. Do NOT call recv again: the inbox is fenced and recv will \
only keep returning this same notice."
.into(),
id: 0,
redelivered: false,
in_reply_to: None,
}
}
/// Render the popped-message payload of a successful `recv` (see `format_recv`
/// for the empty/single/batch shapes).
fn render_recv_messages(messages: &[hive_sh4re::DeliveredMessage], waited: bool) -> String {
use std::fmt::Write as _;
if messages.is_empty() {
return if waited {
format!("(empty){IDLE_WAIT_HINT}")
} else {
"(empty)".to_owned()
};
}
if messages.len() == 1 {
let m = &messages[0];
let banner = if m.redelivered { REDELIVERY_HINT } else { "" };
return format!("{banner}{}from: {}\n\n{}", msg_id_tag(m.id), m.from, m.body);
}
let n = messages.len();
let mut out = format!("popped {n} message(s):\n\n");
for (i, m) in messages.iter().enumerate() {
if i > 0 {
out.push_str("\n---\n\n");
}
let banner = if m.redelivered { REDELIVERY_HINT } else { "" };
let _ = write!(
out,
"{banner}{}from: {}\n\n{}",
msg_id_tag(m.id),
m.from,
m.body
);
}
out
}
/// `[msg #<id>] ` marker prefixed to each recv row so the agent knows
/// what to pass to `ack_until` when bulk-triaging a backlog. Transient
/// pings carry the sentinel id 0 (in-memory only, nothing in the
/// broker to ack) and render without the marker.
fn msg_id_tag(id: i64) -> String {
if id > 0 {
format!("[msg #{id}] ")
} else {
String::new()
}
}
/// Header prepended to message bodies that were popped by a prior
/// harness session, never acked (turn crash / OOM / restart), and
/// resurfaced by `RequeueInflight` on this session's boot. Same
/// string surfaces in the wake prompt (see the bin loops) and the
/// in-turn `recv` tool result so claude sees the warning either way.
pub const REDELIVERY_HINT: &str = "[redelivered after harness restart — may already be handled]\n";
/// Appended to the `recv` empty result when the agent parked on a
/// long-poll (`wait_seconds > 0`) that timed out with nothing new.
/// Nudges the model to spend the idle time on other useful work
/// instead of immediately re-blocking on `recv`.
pub const IDLE_WAIT_HINT: &str = " — nothing arrived before the wait timed out. \
If you have other useful work (assigned issues, in-flight PRs, a docs sweep, \
notes to update), do that now rather than immediately parking on recv again.";
/// Inner renderer for a `Vec<LooseEnd>` already extracted from the socket
/// reply. Called by the `get_loose_ends` handler, which injects the
/// `UnreadMatrix` entry before formatting.
pub(super) fn render_loose_ends(loose_ends: &[hive_sh4re::LooseEnd]) -> String {
use std::fmt::Write as _;
if loose_ends.is_empty() {
return "(no loose ends)".to_owned();
}
let mut out = format!("{} loose end(s):\n", loose_ends.len());
for t in loose_ends {
match t {
hive_sh4re::LooseEnd::Approval {
id,
agent,
commit_ref,
description,
age_seconds,
} => {
let desc = description
.as_deref()
.map(|d| format!(" — {d}"))
.unwrap_or_default();
let _ = writeln!(
out,
"- approval #{id} ({agent} @ {commit_ref}, {age_seconds}s old){desc}"
);
}
hive_sh4re::LooseEnd::Question {
id,
asker,
target,
question,
age_seconds,
} => {
let to = target.as_deref().unwrap_or("operator");
let _ = writeln!(
out,
"- question #{id} ({asker} → {to}, {age_seconds}s old): {question}"
);
}
hive_sh4re::LooseEnd::Reminder {
id,
owner,
message,
due_at,
age_seconds,
} => {
let _ = writeln!(
out,
"- reminder #{id} ({owner}, scheduled {age_seconds}s ago, due_at={due_at}): {message}"
);
}
hive_sh4re::LooseEnd::PendingMessages { count } => {
let _ = writeln!(
out,
"- {count} pending inbox message(s) — drain with recv (recv(max: {count}) to batch)"
);
}
hive_sh4re::LooseEnd::UnreadMatrix { rooms, summary } => {
let _ = write!(out, "- unread matrix messages in {rooms} room(s)");
if summary.is_empty() {
let _ = writeln!(
out,
" — use list_rooms + read_room to view, mark_read to clear"
);
} else {
let _ = writeln!(out, ":");
for line in summary.lines() {
let _ = writeln!(out, " {line}");
}
let _ = writeln!(
out,
" use list_rooms + read_room to view, mark_read to clear"
);
}
}
}
}
out
}
/// Per-room unread entry returned by `matrix_unread_summary`. Mirrors
/// `hive-matrix-mcp`'s `RoomUnread` but defined locally to avoid a
/// cross-crate dep on the matrix-sdk crate tree.
#[derive(Debug, serde::Deserialize)]
pub(super) struct MatrixRoomUnread {
label: String,
count: u32,
last_body: Option<String>,
last_sender: Option<String>,
}
/// Query the local matrix daemon for per-room unread summaries. Returns
/// `None` if the daemon socket is absent or the query fails. Best-effort:
/// agents without matrix configured are not penalised.
pub(super) async fn matrix_unread_summary() -> Option<Vec<MatrixRoomUnread>> {
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::UnixStream;
let socket = std::env::var_os("HIVE_MATRIX_SOCKET").map_or_else(
|| std::path::PathBuf::from("/run/hive-matrix/socket"),
std::path::PathBuf::from,
);
if !socket.exists() {
return None;
}
let mut stream = UnixStream::connect(&socket).await.ok()?;
stream
.write_all(b"{\"method\":\"unread_summary\"}\n")
.await
.ok()?;
let mut lines = BufReader::new(stream).lines();
let line = lines.next_line().await.ok()??;
let val: serde_json::Value = serde_json::from_str(&line).ok()?;
// Response: {"kind":"ok","payload":[{label, count, last_body?, last_sender?}]}
let arr = val.get("payload")?.as_array()?;
serde_json::from_value(serde_json::Value::Array(arr.clone())).ok()
}
/// Format a `Vec<MatrixRoomUnread>` into a per-room summary string.
/// Single room / single message collapses to one line; multi-room
/// expands to a bulleted list. Returns an empty string for empty input.
pub(super) fn format_matrix_summary(rooms: &[MatrixRoomUnread]) -> String {
use std::fmt::Write as _;
if rooms.is_empty() {
return String::new();
}
let mut out = String::new();
for r in rooms {
if r.count == 1
&& let (Some(body), Some(sender)) = (&r.last_body, &r.last_sender)
{
let _ = writeln!(out, "- {}: {sender}: {body}", r.label);
continue;
}
let _ = writeln!(out, "- {}: {} unread", r.label, r.count);
}
// Remove trailing newline.
if out.ends_with('\n') {
out.pop();
}
out
}
/// Parse the user-facing `kind` string for `cancel_loose_end` into the
/// wire enum. Accepts a small alias set so claude doesn't have to
/// remember the exact spelling (`"q"` / `"r"` shorthand falls out
/// for free).
pub(super) fn parse_loose_end_kind(raw: &str) -> Result<hive_sh4re::CancelLooseEndKind, String> {
match raw.trim().to_ascii_lowercase().as_str() {
"question" | "q" => Ok(hive_sh4re::CancelLooseEndKind::Question),
"reminder" | "r" => Ok(hive_sh4re::CancelLooseEndKind::Reminder),
"approval" | "a" => Ok(hive_sh4re::CancelLooseEndKind::Approval),
other => Err(format!(
"cancel_loose_end: unknown kind '{other}' \
(expected \"question\", \"reminder\", or \"approval\")"
)),
}
}
/// Canonical user-facing label for a `CancelLooseEndKind` — used in
/// the success ack so the caller always sees `"question"` /
/// `"reminder"` instead of whatever alias they passed in (`"q"` /
/// `"r"`).
pub(super) fn loose_end_kind_label(kind: hive_sh4re::CancelLooseEndKind) -> &'static str {
match kind {
hive_sh4re::CancelLooseEndKind::Question => "question",
hive_sh4re::CancelLooseEndKind::Reminder => "reminder",
hive_sh4re::CancelLooseEndKind::Approval => "approval",
}
}
/// Format helper for `get_agent_meta`: renders an agent's identity +
/// current status as a short human-readable block. `name`,
/// `hyperhive_rev`, and `running` are always shown; `status` only
/// appears when one is set, otherwise the line reads `status: <none>`.
/// When `running` is false the host has already cleared `status_text`
/// (it would be a stale snapshot from before the stop) so the status
/// line is implicitly `<none>` in that case — but the explicit
/// `running: no` line tells the caller WHY. See
/// `docs/turn-loop/mcp.md::Core tools` (`get_agent_meta`).
#[must_use]
pub fn format_agent_meta(resp: Result<hive_sh4re::Response, anyhow::Error>) -> String {
match resp {
Ok(hive_sh4re::Response::AgentMeta {
name,
running,
hyperhive_rev,
status_text,
status_set_at,
hive_name,
swarm_name,
matrix_accounts,
}) => {
let rev = hyperhive_rev.as_deref().unwrap_or("<unknown>");
let run = if running { "yes" } else { "no" };
let mut out = format!("name: {name}\nhyperhive_rev: {rev}\nrunning: {run}");
// Surface hive + swarm display names only when set, so
// single-hive deployments don't see noisy `<none>` lines.
if let Some(hn) = hive_name.as_deref() {
use std::fmt::Write as _;
let _ = write!(out, "\nhive_name: {hn}");
}
if let Some(sn) = swarm_name.as_deref() {
use std::fmt::Write as _;
let _ = write!(out, "\nswarm_name: {sn}");
}
match status_text {
None => out.push_str("\nstatus: <none>"),
Some(s) => {
use std::fmt::Write as _;
let age = status_set_at.and_then(|ts| {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.ok()?
.as_secs();
// `ts` is a unix epoch second the agent itself
// sourced from `SystemTime` — always positive
// in normal operation. Clamp the negative
// (clock-skew) edge to 0 before the unsigned
// cast so the cast loses no real precision.
let ts_secs = u64::try_from(ts).unwrap_or(0);
let secs = now.saturating_sub(ts_secs);
Some(format_age_secs(secs))
});
// `write!` into the buffer instead of `push_str(&format!(…))` —
// avoids the intermediate allocation clippy::format_push_string
// flags. The infallible `String` writer makes this safe to
// `let _ =`-ignore.
match age {
Some(a) => {
let _ = write!(out, "\nstatus: {s} (set {a} ago)");
}
None => {
let _ = write!(out, "\nstatus: {s}");
}
}
}
}
// Matrix identities the agent can act as (the `account` arg on
// the matrix tools). Listed only when matrix is provisioned, so
// non-matrix agents don't see an empty line.
if !matrix_accounts.is_empty() {
use std::fmt::Write as _;
out.push_str("\nmatrix_accounts:");
for acct in &matrix_accounts {
let uid = acct.user_id.as_deref().unwrap_or("?");
let _ = write!(out, "\n {} ({uid}) on {}", acct.name, acct.homeserver);
}
}
out
}
other => reply_err(other, "get_agent_meta"),
}
}
/// Format a duration in seconds as a human-readable age string.
fn format_age_secs(secs: u64) -> String {
if secs < 60 {
format!("{secs}s")
} else if secs < 3600 {
format!("{}m", secs / 60)
} else if secs < 86400 {
format!("{}h", secs / 3600)
} else {
format!("{}d", secs / 86400)
}
}
/// Append a short note to a tool result when the underlying socket call
/// took retries to land. Lets claude distinguish "my request was wrong"
/// from "c0re flickered and the harness rode it out" — without the
/// hint, a tool result that took 30s to come back looks identical to a
/// content failure and the model would burn a turn retrying it.
#[must_use]
pub fn annotate_retries(mut s: String, retries: u32) -> String {
if retries > 0 {
use std::fmt::Write as _;
let suffix = if retries == 1 { "retry" } else { "retries" };
let _ = write!(
s,
"\n\n(note: hive socket connect needed {retries} {suffix} — c0re likely \
restarted. Your request did succeed on the final attempt; no action needed.)"
);
}
s
}
#[cfg(test)]
mod tests {
use super::{IDLE_WAIT_HINT, format_recv};
#[test]
fn empty_recv_after_wait_appends_idle_hint() {
let out = format_recv(
Ok(hive_sh4re::Response::Messages { messages: vec![] }),
true,
);
assert!(out.starts_with("(empty)"));
assert!(out.contains(IDLE_WAIT_HINT));
}
#[test]
fn empty_recv_without_wait_has_no_hint() {
let out = format_recv(
Ok(hive_sh4re::Response::Messages { messages: vec![] }),
false,
);
assert_eq!(out, "(empty)");
}
}

View file

@ -7,6 +7,7 @@ use crate::events::Bus;
use crate::mcp::REDELIVERY_HINT;
use crate::turn::{TurnError, TurnOutcome};
use crate::turn_stats::TurnStatRow;
pub use hive_sh4re::wire_time::now_unix;
/// Assemble the per-turn wake prompt string. The role/tools/etc. live in the
/// system prompt; this is just the wake signal body. `id` is the broker row
@ -44,16 +45,6 @@ pub fn format_wake_prompt(
format!("{banner}{tag}Incoming message from `{from}`:\n---\n{body}\n---{pending}")
}
/// Current time as a Unix timestamp (seconds). Returns 0 on any error.
#[must_use]
pub fn now_unix() -> i64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.ok()
.and_then(|d| i64::try_from(d.as_secs()).ok())
.unwrap_or(0)
}
/// Field-named args for [`build_row`]. Mirrors the turn-stats row
/// columns; `outcome` and `bus` borrow for the duration of the call.
pub struct TurnRowArgs<'a> {

View file

@ -15,6 +15,7 @@ use rusqlite::{Connection, OpenFlags};
use serde::Serialize;
use hive_sh4re::ReminderStats;
use hive_sh4re::wire_time::now_unix;
/// Window param accepted by `/api/stats?window=`. Each maps to a
/// total span + the bucket width used to roll up trend series.
@ -208,7 +209,7 @@ fn default_path() -> PathBuf {
}
fn empty_snapshot(window: Window) -> Snapshot {
let now = now_secs();
let now = now_unix();
let from = now - window.span_secs();
let buckets = fill_buckets(from, now, window.bucket_secs(), &HashMap::new());
Snapshot {
@ -239,7 +240,7 @@ fn snapshot(path: &Path, window: Window) -> Result<Snapshot> {
// matches hive-c0re's host-side reader (`hive_stats::read_agent`).
conn.busy_timeout(std::time::Duration::from_millis(500))
.with_context(|| format!("set busy_timeout on {}", path.display()))?;
let now = now_secs();
let now = now_unix();
// Fixed windows look back a constant span; `all` starts at the earliest
// recorded turn (`MIN(started_at)`, falling back to `now` on an empty
// table) and sizes its buckets adaptively from that span.
@ -567,12 +568,6 @@ fn u64_from_i64(v: i64) -> u64 {
u64::try_from(v).unwrap_or(0)
}
fn now_secs() -> i64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map_or(0, |d| i64::try_from(d.as_secs()).unwrap_or(i64::MAX))
}
#[cfg(test)]
mod tests {
use super::*;
@ -638,7 +633,7 @@ mod tests {
fn snapshot_aggregates_rows() {
let db = tmp_db();
let _ = std::fs::remove_file(&db);
let now = now_secs();
let now = now_unix();
seed_db(
&db,
&[
@ -727,7 +722,7 @@ mod tests {
fn bash_breakdown_empty_without_table() {
let db = tmp_db();
let _ = std::fs::remove_file(&db);
seed_db(&db, &[(now_secs() - 100, 1000, "opus", "recv", "ok", "{}")]);
seed_db(&db, &[(now_unix() - 100, 1000, "opus", "recv", "ok", "{}")]);
let s = snapshot(&db, Window::Day).unwrap();
assert!(s.bash_breakdown.is_empty());
}
@ -739,7 +734,7 @@ mod tests {
let db = tmp_db();
let _ = std::fs::remove_file(&db);
seed_db(&db, &[]);
let now = now_secs();
let now = now_unix();
let conn = Connection::open(&db).unwrap();
conn.execute_batch("CREATE TABLE bash_commands (ts INTEGER NOT NULL, head TEXT NOT NULL);")
.unwrap();

View file

@ -15,8 +15,9 @@
//! the honest fix is to clean them up where they live.
use std::path::Path;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use std::time::Duration;
use hive_sh4re::wire_time::now_unix;
use rusqlite::{Connection, Result, params};
/// How often the sweep runs.
@ -130,11 +131,3 @@ fn vacuum_events(path: &Path) -> Result<u64> {
)?;
Ok(u64::try_from(removed).unwrap_or(0))
}
fn now_unix() -> i64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.ok()
.and_then(|d| i64::try_from(d.as_secs()).ok())
.unwrap_or(0)
}

View file

@ -90,13 +90,7 @@ fn signal_group(pgid: Option<i32>, sig: i32) {
// Helpers
// ---------------------------------------------------------------------------
fn now_unix() -> i64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
.cast_signed()
}
use hive_sh4re::wire_time::now_unix;
/// Generate a task ID: `<timestamp_hex><seq_hex>`.
#[must_use]

View file

@ -18,6 +18,7 @@ clap-markdown = "0.1"
hive-sh4re.workspace = true
libc.workspace = true
listenfd = "1"
petgraph.workspace = true
rusqlite.workspace = true
serde.workspace = true
serde_json.workspace = true

View file

@ -13,19 +13,20 @@ use crate::lifecycle;
/// Approve a pending request. Marks the approval row durably, then
/// either runs the work inline (`InitConfig`, sub-second git ops) or
/// enqueues it into `rebuild_queue` so the dashboard POST returns
/// submits it to the job queue so the dashboard POST returns
/// immediately while the long-running pipeline runs off-thread
/// (operator no longer blocks on a 30-90s spinner for `ApplyCommit`).
///
/// Dispatch:
/// - `ApplyCommit` → `QueueKind::Rebuild` (~30-90s wall time)
/// - `UpdateMetaInputs` → `QueueKind::MetaUpdate` (~3-15s)
/// - `Spawn` → `QueueKind::Spawn` (~30-90s)
/// - `ApplyCommit` / `MergeConfigPr` → a single-node `ApprovalDeploy`
/// DAG (the two-phase meta deploy stays opaque in v1; ~30-90s)
/// - `UpdateMetaInputs` → a `MetaUpdate` DAG (fan-out on completion)
/// - `Spawn` → a `Spawn` DAG (`Create → WriteDropin → Reconcile`)
/// - `InitConfig` → inline (<1s; queue card would be noise)
///
/// The queue worker re-fetches the approval row on dispatch, runs
/// the kind-specific pipeline, and fires `ApprovalResolved` /
/// `Spawned` / `Rebuilt` / `ConfigReady` via `finish_approval`.
/// `ApprovalDeploy` resolves the approval inside its pipeline; the
/// `MetaUpdate` / `Spawn` DAGs resolve via [`resolve_approval_dag`]
/// when their DAG settles terminal.
pub async fn approve(coord: Arc<Coordinator>, id: i64) -> Result<()> {
let approval = coord.approvals.mark_approved(id)?;
tracing::info!(
@ -56,54 +57,41 @@ pub async fn approve(coord: Arc<Coordinator>, id: i64) -> Result<()> {
}
ApprovalKind::UpdateMetaInputs => {
// Inputs JSON-encoded into commit_ref by the manager's
// submit path — surface them on the queue entry so the
// dashboard can show *which* inputs are about to bump.
// submit path — surface them on the DAG so the dashboard
// can show *which* inputs are about to bump. The cascade
// rebuilds fan out when the lock bump lands (so they build
// against the post-bump lock, and a failed bump fans out
// nothing).
let inputs: Vec<String> =
serde_json::from_str(&approval.commit_ref).unwrap_or_default();
let parent_id = coord
.rebuild_queue
.enqueue_full(crate::rebuild_queue::FullEnqueue {
kind: crate::rebuild_queue::QueueKind::MetaUpdate,
agent: approval.agent.clone(),
source: crate::rebuild_queue::QueueSource::Approval,
reason: format!("approval #{id} meta input update"),
parent_id: None,
inputs: inputs.clone(),
approval_id: Some(id),
perm_payload: None,
depends_on: Vec::new(),
});
// Pre-enqueue cascade rebuilds in topological order so
// agents depending on updated inputs are rebuilt after the
// lock bump, matching the dashboard post_meta_update path.
let cascade_agents = crate::rebuild_queue::meta_update_cascade_agents(&inputs).await;
let cascade_reason = format!("approval #{id} meta input cascade");
for name in cascade_agents {
coord.rebuild_queue.enqueue(
crate::rebuild_queue::QueueKind::Rebuild,
name,
crate::rebuild_queue::QueueSource::MetaUpdate,
cascade_reason.clone(),
Some(parent_id),
);
let submitted = coord
.job_queue
.submit(crate::job_queue::templates::meta_update(
inputs,
crate::job_queue::Source::Approval,
format!("approval #{id} meta input update"),
Some(id),
));
if let Err(e) = submitted {
return Err(e.context("submit meta-update dag"));
}
coord.emit_rebuild_queue_snapshot();
Ok(())
}
ApprovalKind::Spawn => {
coord
.rebuild_queue
.enqueue_full(crate::rebuild_queue::FullEnqueue {
kind: crate::rebuild_queue::QueueKind::Spawn,
agent: approval.agent.clone(),
source: crate::rebuild_queue::QueueSource::Approval,
reason: format!("approval #{id} spawn"),
parent_id: None,
inputs: Vec::new(),
approval_id: Some(id),
perm_payload: None,
depends_on: Vec::new(),
});
// The spawn's tail `Reconcile` starts the container, so the
// new agent's power intent is `Up` from the outset.
if let Err(e) = coord.power.set(&approval.agent, crate::power::Wanted::Up) {
tracing::warn!(agent = %approval.agent, error = ?e, "agent_power: seed on spawn failed");
}
let submitted = coord.job_queue.submit(crate::job_queue::templates::spawn(
&approval.agent,
id,
format!("approval #{id} spawn"),
));
if let Err(e) = submitted {
return Err(e.context("submit spawn dag"));
}
coord.emit_rebuild_queue_snapshot();
Ok(())
}
@ -137,29 +125,27 @@ pub async fn approve(coord: Arc<Coordinator>, id: i64) -> Result<()> {
}
}
/// Enqueue a `Rebuild` queue entry tied to an approval id. Shared by the
/// `ApplyCommit` and `MergeConfigPr` dispatch arms — both end in a container
/// rebuild routed through the queue, differing only in the queue `reason`.
/// The queue worker branches on the approval's kind to pick the right handler.
/// Submit the single-node `ApprovalDeploy` DAG tied to an approval id.
/// Shared by the `ApplyCommit` and `MergeConfigPr` dispatch arms — both
/// end in a container rebuild routed through the queue, differing only
/// in the `reason`. The node executor branches on the approval's kind
/// to pick the right handler.
fn enqueue_approval_rebuild(
coord: &Arc<Coordinator>,
agent: &str,
approval_id: i64,
reason: String,
) {
coord
.rebuild_queue
.enqueue_full(crate::rebuild_queue::FullEnqueue {
kind: crate::rebuild_queue::QueueKind::Rebuild,
agent: agent.to_owned(),
source: crate::rebuild_queue::QueueSource::Approval,
if let Err(e) = coord
.job_queue
.submit(crate::job_queue::templates::approval_deploy(
agent,
approval_id,
reason,
parent_id: None,
inputs: Vec::new(),
approval_id: Some(approval_id),
perm_payload: None,
depends_on: Vec::new(),
});
))
{
tracing::error!(%agent, approval_id, error = ?e, "submit approval deploy dag failed");
}
coord.emit_rebuild_queue_snapshot();
}
@ -375,69 +361,60 @@ async fn run_approval_schedule_prompt(
finish_approval(coord, &approval, result, None, false)
}
/// Worker entry point for `ApprovalKind::UpdateMetaInputs` queue
/// entries. Inputs come from the approval row's `commit_ref` field
/// (JSON-encoded by the manager submit path), not the queue entry's
/// `inputs` — the queue copy is for dashboard display only.
pub async fn run_approval_update_meta_inputs(
/// Terminal hook for approval-carrying DAGs — the job queue's
/// scheduler calls this exactly once when such a DAG settles terminal.
/// `MetaUpdate` and `Spawn` approval DAGs resolve here (their work is
/// ordinary queue nodes); the opaque `ApprovalDeploy` pipeline resolves
/// *inside* its node, so its DAG is skipped — unless it was cancelled
/// while still queued, in which case the node never ran and the row
/// would otherwise dangle forever.
pub(crate) async fn resolve_approval_dag(
coord: &Arc<Coordinator>,
queue_entry_id: Option<u64>,
approval_id: i64,
) -> Result<()> {
let approval = fetch_approval_for_worker(coord, approval_id, ApprovalKind::UpdateMetaInputs)?;
let inputs: Vec<String> = serde_json::from_str(&approval.commit_ref).unwrap_or_default();
coord.set_queue_step(queue_entry_id, "nix flake update");
let result = crate::meta::lock_update(&inputs).await;
finish_approval(coord, &approval, result, None, false)
}
/// Worker entry point for `ApprovalKind::Spawn` queue entries.
/// Differs from `run_approval_apply_commit` only in routing through
/// `lifecycle::spawn` (the deprecated direct-spawn path). Synchronous
/// in the queue worker — the previous `tokio::spawn` wrapper is gone
/// (the queue worker itself is the async task).
pub async fn run_approval_spawn(
coord: &Arc<Coordinator>,
queue_entry_id: Option<u64>,
approval_id: i64,
) -> Result<()> {
let approval = fetch_approval_for_worker(coord, approval_id, ApprovalKind::Spawn)?;
let agent_dir = coord.ensure_runtime(&approval.agent)?;
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(&approval.agent, agent_dir);
// Transient guard keeps the per-container "Spawning" pill lit while
// the worker is doing the actual nixos-container create. Auto-clears
// on the function's scope exit (success or panic).
let _guard = coord.transient_guard(&approval.agent, TransientKind::Spawning);
coord.set_queue_step(queue_entry_id, "lifecycle::spawn");
let result = lifecycle::spawn(&approval.agent, &hive, &paths).await;
if result.is_ok() {
coord.set_queue_step(queue_entry_id, "forge user");
if let Err(e) = crate::forge::ensure_user_for(&approval.agent).await {
tracing::warn!(agent = %approval.agent, error = ?e, "forge: ensure_user after spawn failed");
terminal: &crate::job_queue::TerminalDag,
) {
use crate::job_queue::{State, Template};
let Some(approval_id) = terminal.approval_id else {
return;
};
if terminal.template == Template::Rebuild && terminal.state != State::Cancelled {
return; // ApprovalDeploy resolved inside the node.
}
let approval = match coord.approvals.get(approval_id) {
Ok(Some(a)) => a,
Ok(None) => {
tracing::warn!(approval_id, "approval dag terminal: row no longer exists");
return;
}
coord.set_queue_step(queue_entry_id, "forge config repo");
if let Err(e) = crate::forge::ensure_config_repo(&approval.agent).await {
tracing::warn!(agent = %approval.agent, error = ?e, "forge: ensure_config_repo after spawn failed");
Err(e) => {
tracing::warn!(approval_id, error = ?e, "approval dag terminal: row read failed");
return;
}
coord.set_queue_step(queue_entry_id, "forge push");
if let Err(e) = crate::forge::push_config(&approval.agent).await {
tracing::warn!(agent = %approval.agent, error = ?e, "forge: push_config after spawn failed");
}
coord.set_queue_step(queue_entry_id, "forge meta access");
if let Some(core_token) = crate::forge::core_token()
&& let Err(e) = crate::forge::meta_read_access(&approval.agent, &core_token).await
{
tracing::warn!(agent = %approval.agent, error = ?e, "forge: meta_read_access after spawn failed");
}
if let Err(e) = crate::forge::ensure_meta_remote(&approval.agent).await {
tracing::warn!(agent = %approval.agent, error = ?e, "forge: ensure_meta_remote after spawn failed");
};
let result: Result<()> = match terminal.state {
State::Done => Ok(()),
State::Cancelled => Err(anyhow::anyhow!("cancelled before completion")),
_ => Err(anyhow::anyhow!(
"{}",
terminal
.error
.clone()
.unwrap_or_else(|| "job dag failed".to_owned())
)),
};
if approval.kind == ApprovalKind::Spawn {
// Post-spawn forge bookkeeping (user, config repo mirror, meta
// access) — warn-only, then the resolution events + a rescan so
// the dashboard reflects the post-spawn state either way.
if result.is_ok() {
forge_after_first_spawn(coord, &approval.agent).await;
} else {
coord.rescan_containers_and_emit().await;
crate::dashboard::emit_tombstones_snapshot(coord).await;
}
}
let final_result = finish_approval(coord, &approval, result, None, false);
coord.rescan_containers_and_emit().await;
crate::dashboard::emit_tombstones_snapshot(coord).await;
final_result
if let Err(e) = finish_approval(coord, &approval, result, None, false) {
tracing::warn!(approval_id, error = ?e, "approval dag resolved with failure");
}
}
/// Re-fetch an approval row from sqlite for a queue-worker dispatch.
@ -563,14 +540,7 @@ fn finish_approval(
// snapshot refetch. `approved` rows that succeed get the
// approval's logged resolved_at indirectly via `now_unix()`;
// failures already wrote it via mark_failed above.
let approval_kind = match approval.kind {
ApprovalKind::Spawn => "spawn",
ApprovalKind::ApplyCommit => "apply_commit",
ApprovalKind::InitConfig => "init_config",
ApprovalKind::UpdateMetaInputs => "update_meta_inputs",
ApprovalKind::SchedulePrompt => "schedule_prompt",
ApprovalKind::MergeConfigPr => "merge_config_pr",
};
let approval_kind = approval.kind.as_str();
let sha_short = approval
.fetched_sha
.as_deref()
@ -867,13 +837,7 @@ async fn deploy_applied_target(
// part of this entry rather than a deferred fast-lane follow-up.
false,
&|step| coord.set_queue_step(queue_entry_id, step),
&|log_id| {
if let Some(qid) = queue_entry_id
&& coord.rebuild_queue.set_build_log_id(qid, log_id)
{
coord.emit_rebuild_queue_snapshot();
}
},
&|log_id| coord.set_queue_build_log(queue_entry_id, log_id),
)
.await;
@ -984,6 +948,11 @@ pub async fn destroy(coord: &Arc<Coordinator>, name: &str, purge: bool) -> Resul
"agent destroyed"
},
);
// Drop the durable power intent — a future agent of the same name
// seeds fresh from its observed state.
if let Err(e) = coord.power.remove(name) {
tracing::warn!(%name, error = ?e, "agent_power: remove on destroy failed");
}
drop(guard);
coord.notify_manager(&HelperEvent::Destroyed {
agent: name.to_owned(),
@ -1045,14 +1014,7 @@ pub async fn deny(coord: &Coordinator, id: i64, note: Option<&str>) -> Result<()
tracing::warn!(%id, agent = %a.agent, error = ?e, "forge: push_config after deny failed");
}
}
let approval_kind = match a.kind {
ApprovalKind::Spawn => "spawn",
ApprovalKind::ApplyCommit => "apply_commit",
ApprovalKind::InitConfig => "init_config",
ApprovalKind::UpdateMetaInputs => "update_meta_inputs",
ApprovalKind::SchedulePrompt => "schedule_prompt",
ApprovalKind::MergeConfigPr => "merge_config_pr",
};
let approval_kind = a.kind.as_str();
let sha_short = sha.as_deref().map(|s| s[..s.len().min(12)].to_owned());
let description = a.description.clone();
let agent_owned = a.agent.clone();

View file

@ -0,0 +1,95 @@
//! Per-agent capability configuration. Stored at
//! `/var/lib/hyperhive/meta/capabilities.json` alongside `topology.json`
//! and `tool-groups.json`.
//!
//! Format: a JSON object mapping agent name to an array of
//! `hive_sh4re::Capability` `snake_case` strings:
//!
//! ```json
//! {
//! "atlas": ["read_host_journal"],
//! "ruth": ["query_agent_state"]
//! }
//! ```
//!
//! An absent entry (or an absent file) means "no extra capabilities".
//! `render_flake` in `meta.rs` reads this file and injects
//! `HIVE_CAPABILITIES` into each agent's systemd service env; absent
//! entries emit no env var so agents without capabilities don't trigger
//! a spurious rebuild.
//!
//! Write path: `set_caps` is called from the dashboard action handler
//! that the operator uses to grant/revoke capabilities per agent.
use std::collections::BTreeMap;
use std::path::PathBuf;
const CAPABILITIES_FILE: &str = "capabilities.json";
#[must_use]
pub fn capabilities_path() -> PathBuf {
crate::meta::meta_dir().join(CAPABILITIES_FILE)
}
/// Read the per-agent capability map. Returns an empty map when the
/// file is absent or unparsable — callers treat a missing entry as
/// "no extra capabilities".
#[must_use]
pub fn read() -> BTreeMap<String, Vec<String>> {
let path = capabilities_path();
let Ok(raw) = std::fs::read_to_string(&path) else {
return BTreeMap::new();
};
serde_json::from_str(&raw).unwrap_or_default()
}
/// Look up the configured capabilities for one agent. Returns an empty
/// vec when the agent has no entry.
#[must_use]
pub fn caps_for(name: &str) -> Vec<String> {
read().get(name).cloned().unwrap_or_default()
}
/// Check whether an agent holds a specific capability.
#[must_use]
pub fn has_cap(name: &str, cap: hive_sh4re::Capability) -> bool {
caps_for(name)
.iter()
.any(|s| s.eq_ignore_ascii_case(cap.as_str()))
}
/// Persist the full capability map. Sorted JSON output keeps diffs
/// minimal. Best-effort — returns `io::Error` so callers decide
/// whether to abort or log.
pub fn write(map: &BTreeMap<String, Vec<String>>) -> std::io::Result<()> {
let path = capabilities_path();
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
let text = serde_json::to_string_pretty(map)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?;
std::fs::write(&path, format!("{text}\n"))
}
/// Set the capabilities for one agent and persist the map. An empty
/// `caps` vec removes the entry (agent has no capabilities).
pub fn set_caps(name: &str, caps: &[String]) -> std::io::Result<()> {
let mut current = read();
if caps.is_empty() {
current.remove(name);
} else {
current.insert(name.to_owned(), caps.to_vec());
}
write(&current)
}
/// Remove an agent from the capability map entirely. Called by
/// `meta::sync_agents` when an agent is deprovisioned so stale entries
/// don't accumulate. No-op if the agent has no entry.
pub fn remove_agent(name: &str) -> std::io::Result<()> {
let mut current = read();
if current.remove(name).is_some() {
write(&current)?;
}
Ok(())
}

Some files were not shown because too many files have changed in this diff Show more