The hive applies one `agentCpuQuota` / `agentMemoryMax` to every
container. That's the right default and the wrong ceiling: a build-heavy
agent needs headroom the other twelve don't, and raising the hive-wide
value to suit it hands that headroom to everyone.
Adds a per-agent override, persisted host-side and resolved per-field
against the hive defaults.
Follows the existing `meta/*.json` pattern (`capabilities.json`,
`tool-groups.json`): a host-side map read by `hive-c0re`, staged and
committed in the meta repo so every change lands in the audit trail.
```json
{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }
```
Fallback is **per field**, not per agent: an entry with only
`memory_max` leaves that agent on the hive-wide CPU quota. Absent file,
absent agent and absent field all resolve to the hive default, so the
feature is inert until someone opts an agent in.
Unlike the other meta files this one is **not** injected into the
container — a limit is something done *to* an agent, not something it
reads about itself.
```
hivectl agents set-limits sock --cpu-quota 400% --memory-max 8G
hivectl agents set-limits sock --reset
```
Values are validated before they're persisted: they go into a systemd
drop-in verbatim, and a typo there makes the unit fail to *start* —
turning a fat-fingered quota into a container that won't come back.
The command is declarative: each call replaces the agent's whole entry.
That makes a forgotten flag a silent revert, so a bare `set-limits
<name>` is rejected at the clap layer and clearing needs an explicit
`--reset`.
`ContainerView` gains `cpu_quota` / `memory_max`, both always populated:
there's no "unset" state to render, only "same as everyone else". They
reflect what the drop-in *says* — what the next start will enforce — not
a live cgroup reading.
The write goes through `meta::commit_resource_limits` rather than the
bare setter, so it's staged and committed under `META_LOCK`. Writing
without committing would leave the meta working tree dirty for the next
`prepare_deploy` to trip over.
Docs: `persistence.md` (the new meta file, and why it isn't injected),
`tools/hivectl.md` (the prose guide), `tools/hivectl-cli.md`
(regenerated clap dump).
Closes: internal/requests issue 25
A paused agent keeps its container, its claude session and its
dashboard/todo servers up, but stops driving turns. Messages queue
unacked and are drained on resume.
The whole protocol is a single marker file, `<harness>/paused`. That
directory is already a bind-mount shared between host and container, so
both sides just stat the same path: the harness reads it to decide
whether to drive a turn, hive-c0re reads it to render the badge and
writes/removes it for `hivectl pause|resume`. No new wire protocol, no
container round-trip, and it is sticky across restarts by construction.
Not calling `recv_next` while paused *is* the queueing semantic, so
there is no fencing to get wrong: reminders buffer in their unbounded
channel, the todo `Notify` permit coalesces, and a `request_next_turn`
that raced the pause survives because the gate sits above
`self_continue.take()`.
Graceful stop is handled host-side rather than in the harness: a paused
agent provably has no turn in flight, so `run_signal` skips the fence
entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for
a checkpoint turn that will never run.
`paused` is reported on `ContainerView` / `AgentStatusRow` for the
dashboard, orthogonal to `running` and reported for stopped containers
too.
Closes: hyperhive/hyperhive issue 2271
The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.
Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.
The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.
`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.
The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.
Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
Non-derivable per-node payload rides the node that owns it. Tagging all
four deploy nodes with the approval id would render the same card four
times in `dag_view`.
Also fix `set_queue_step`'s doc comment, which claimed the DAG-id lookup
was exact because approval DAGs are single-node. They are not anymore;
what actually holds is that the chain is strictly sequential with the
root parked in Finishing, so at most one node is ever Running.
- lifecycle::ensure_agent_runtime_dir(name): pure filesystem op, no
Coordinator dep. Creates /run/hyperhive/agents/<name> without touching
the MCP listener map.
- workers/mcp_sockets::spawn_poll(coord): 10 s reconcile loop (same shape
as agent_sockets::spawn_poll). Converges 'agent running => MCP listener
bound'. First tick is immediate so hive-c0re restarts re-register all
running agents without waiting a full interval. Fixes the dead-listener-
after-daemon-restart gap.
- All ensure_runtime() call sites updated:
- Prebuild/Swap/WriteDropin: Coordinator::agent_dir() (pure, no IO)
- Reconcile-Start: ensure_agent_runtime_dir + agent_dir (dir may be
missing after reboot; listener deferred to supervisor)
- run_create / handle_spawn: ensure_agent_runtime_dir + register_agent
(eager on first spawn so socket ready before harness first turn)
- apply_commit / merge_config_pr: ensure_agent_runtime_dir + agent_dir
- Manager (auto_update): ensure_agent_runtime_dir + agent_dir
(manager has no MCP listener; socket_server::start_manager owns it)
- ensure_runtime() retained in Coordinator with updated doc pointing at
the preferred split form. No callers remain outside tests.
coordinator.md rewrites the queue section (node inventory, DAG shapes,
resources, desired-state reconciliation, boot reconcile); approvals.md
+ persistence.md + hivectl --graceful help updated to match. agent_power
lives in broker.sqlite like approvals/questions (own connection + busy
timeout) instead of a separate db file.
jobs are now DAGs of primitive nodes (prebuild, stop-for-update, swap,
reconcile, signal, drain, ...) driven by one scheduler with N build
slots + per-agent lifecycle leases. per-agent power intent (wanted
up/offline) is durable in agent_power.sqlite; Reconcile nodes converge
observed state to it. kills the graceful-stop watcher thread, the
deferred-start follow-up, and the cascade pre-enqueue (fan-out on
MetaLock completion instead). tracker: #2166
The per-agent and manager sockets ran two parallel dispatchers with
duplicated lifecycle handlers (agent-side topology-gated, manager-side
ungated) plus a manager-only handler set. Collapse to one parameterized
server in socket_server.rs:
- one serve() + dispatch(req, agent, privileged, coord); start() binds
the per-agent sockets (privileged=false), start_manager() binds the
manager socket (privileged=true).
- each lifecycle/config handler (start/restart/kill/update/init_config/
apply_commit) merges its dual: the topology guard (require_child /
require_new_child) runs only on the !privileged path; init_config
records the requester as parent only when !privileged. restart keeps
the orthogonal, capability-gated + audited infra-container branch.
- the agent-state queries (loose-ends / reminder count + rollup) branch
on privileged: privileged keeps any-target + the "*" hive-wide sweep
(query_agent_state-gated), non-privileged keeps the topology/cap gate.
- the privileged-only verbs (schedules / meta-inputs / get_logs) plus
the submit/schedule/watchdog helpers move into socket_server; they are
reached via dispatch_privileged_only(), which rejects the whole group
on a non-privileged socket.
- delete manager_server.rs; repoint refs; merge the test modules.
No behavior change: the topology guard still applies on every
non-privileged lifecycle call, the privileged socket still acts on any
agent, and privileged-only verbs are still rejected on agent sockets.
Pure rename ahead of the agent+manager server consolidation: the
per-agent socket dispatcher already hosts the shared dispatch and all
lifecycle handlers, and will absorb the manager-only handlers next, so
`agent_server` becomes a misnomer. No logic change — git mv plus a
mechanical `agent_server` -> `socket_server` rename across refs.
Follow-up to the audit-log backend + surface. Emits a DashboardEvent on
each agent-initiated privileged action so the audit view live-appends off
/dashboard/stream instead of polling.
- new DashboardEvent::AuditEntryAdded { seq, <flattened AuditEntry> } —
serde tag `audit_entry_added`; the AuditEntry fields flatten to the top
level so the wire shape matches one /api/audit-log `entries` row exactly.
- Coordinator::emit_audit_entry helper (stamps seq like the others).
- audit_log::record now returns the canonical inserted AuditEntry (id + ts
assigned) so the streamed event is the same row that was stored — no
drift. Best-effort unchanged (None on a sqlite blip).
- handle_restart_infra records + emits for every attempt (ok/err/denied),
threading the coordinator through.
Tests: kind_tag round-trip now covers the new variant; added a flatten
test pinning the top-level wire shape (kind/seq/id/…/detail, no nesting).
Pairs with iris's audit view (the /dashboard/stream listener half).
Adds a durable, operator-visible audit trail of privileged operations
hive-c0re performs on behalf of an agent — the ones that cross the
agent/operator trust boundary. First entry: infra-container restarts via
the infra_admin-gated `restart` tool, which until now were recorded only
as a hive-priv journal trace.
Backend:
- new `audit_log` module: sqlite-backed store (audit_log.sqlite, same dir
as build_logs) with schema (ts/agent/action/target/outcome/detail),
best-effort `record`, `list_recent` (clamped 500), 90-day `vacuum`, and
a process-singleton handle mirroring build_logs.
- Coordinator opens + installs the handle; main spawns the hourly vacuum.
- agent_server::handle_restart_infra records every attempt (ok, error, and
capability-denied) via the global handle — best-effort, never fails the
underlying action.
- dashboard: `GET /api/audit-log` returns recent entries as JSON.
Scope is deliberately agent-initiated privileged actions only (not every
PrivRequest — token writes + nspawn edits are constant lifecycle noise).
Extensible: future agent-initiated priv ops record via the same handle.
Unit tests cover record/list ordering, the 500 clamp, and retention vacuum.
The dashboard *surface* (an AUDIT view consuming /api/audit-log) is a
frontend follow-up coordinated with iris.
The systemd ExecStart carried every host-level setting as its own flag —
nine of them, including two escaped JSON blobs (the context-window map and
the model-price table). Collapse them into a single `--config <file>` JSON.
- Reuse the existing HiveEnv as the container-injected config shape (add
Deserialize + Default), and add a ServeConfig wrapper = flattened HiveEnv
plus the hive-c0re-local model_prices table (kept out of HiveEnv since it
is never injected into containers). serde(default) lets any field be
omitted and fall back to its canonical default.
- clap: add --config; the per-setting flags become optional overrides
(config file is the base, explicit flags win — preserves hivectl/debug
ergonomics and bare `hive-c0re serve`).
- Coordinator::open and cmd_serve now take the bundled HiveEnv, which drops
their too_many_arguments clippy allows. cmd_serve keeps a single
too_many_lines allow (inherent daemon-boot orchestration, not arg-driven).
- nix: write the config as JSON to the store + pass --config, so ExecStart
is one short line.
- Add a round-trip test proving the flatten + per-field defaults work.
Closes the ExecStart-length issue.
The hive-wide cost estimate on the dashboard's ST4TS tab used a
hard-coded model->price table in hive_stats.rs. Anthropic list pricing
drifts, so move the table to a nix option operators can keep current
without a code change.
- New `services.hyperhive.modelPrices` option: attrset of model-family
short name -> { input, output, cache_read, cache_write } USD per
million tokens. Passed to `hive-c0re serve --model-prices <json>`.
- hive_stats: `Prices` is now public + Deserialize; add `PriceTable`
type and `resolve_prices` (longest case-insensitive substring key
wins) with the old hard-coded table preserved as `builtin_prices`
fallback for any model not covered.
- Coordinator holds the parsed table (hive-c0re-local, not injected
into containers, so not part of HiveEnv); `/api/stats-hive` reads it.
- Docs: dashboard.md ST4TS cost note updated; option self-documents
via nixosOptionsDoc.
Closes#1434
The nixpkgs bump to clippy 0.1.95 / cargo 1.95.0 added + strengthened a
large batch of lints. CI denied ALL warnings (`-D warnings`) against the
`pedantic = warn` workspace lint, so the bump hard-failed `nix flake
check` workspace-wide with zero code changes — and would recur on every
future clippy bump.
Posture fix (the durable part): CI now runs
`-D warnings -A clippy::pedantic`, so the default/correctness/style lints
stay a hard gate while the "extra, opinionated" pedantic group is
advisory only (still `warn` for local `cargo clippy` via the workspace
lints table, just non-blocking in CI). `-A` rather than `-W` so the
group drop doesn't re-enable the specific pedantic lints the workspace
allows (e.g. `must_use_candidate`).
Also fixes the genuine DEFAULT/STYLE lints the bump surfaced across the
workspace (doc_lazy_continuation, collapsible_if, ptr_arg,
match_like_matches_macro, …) via `cargo clippy --fix` + manual stragglers
(`too_many_arguments` #[allow] on the host-config constructors), and
three tests that had rotted while the CI runner was offline (#1221):
- topology::top_level_agents_in_multi_root — hardcoded unsorted expected
- rebuild_queue::depends_on_evicted_dep_counts_as_resolved — needs
MAX_HISTORY_PER_KIND newer terminals to evict, not one
- coordinator::agent_paths doctest — illustrative pseudo-code, now `ignore`
Validated: clippy + formatting + cargo-test checks all pass.
Add CapabilitiesChanged and ToolGroupsChanged DashboardEvent variants
so the P3RM1SS10NS tab reflects perm changes without the operator
navigating away and back.
Backend:
- DashboardEvent::CapabilitiesChanged { seq, caps, descriptions,
assignments } — same payload shape as GET /api/capabilities
- DashboardEvent::ToolGroupsChanged { seq, groups, descriptions,
assignments } — same payload shape as GET /api/tool-groups
- Coordinator::emit_capabilities_snapshot() and
emit_tool_groups_snapshot() — read from the JSON files and broadcast
- rebuild_queue.rs PermChange worker: emit after each successful
commit_capabilities / commit_tool_groups call
Frontend:
- applyCapabilitiesChanged(ev): calls renderCapabilities(root, ev)
- applyToolGroupsChanged(ev): calls renderToolGroups(root, ev)
- Both registered in MUTATION_HANDLERS
- activateTab comment updated (SSE now covers perm changes)
Docs: dashboard.md and CLAUDE.md updated.
This completes SSE coverage for all dashboard sections: SW4RM,
Y3R C4LL, SYST3M, SCH3DUL3S/reminders, and P3RM1SS10NS all
derive live updates from /dashboard/stream.
Add RemindersChanged SSE event so the pending-reminders list in the
SYST3M tab updates live without polling.
Backend emission sites (every path that mutates the reminders table):
- agent_server: store_remind (remind MCP call)
- dashboard.rs: post_cancel_reminder, post_retry_reminder
- questions.rs: cancel_loose_end Reminder kind
- reminder_scheduler: after each delivery batch (any_delivered)
Coordinator gets emit_reminders_snapshot() mirroring the existing
emit_schedules_snapshot() pattern: lists PendingReminder rows from the
broker and emits DashboardEvent::RemindersChanged.
Frontend: applyRemindersChanged(ev) calls renderReminders(ev.reminders)
and is registered as reminders_changed in MUTATION_HANDLERS.
Docs: dashboard.md reminders_changed entry; CLAUDE.md file map updated.
Add `SchedulesChanged` to the dashboard event channel so the
operator's schedule list updates in real time without requiring a
tab-activation or form-submit refresh.
Backend:
- `dashboard_events.rs`: new `SchedulesChanged { seq, schedules }`
variant carrying a full `Vec<WireSchedule>` snapshot (same
snapshot-over-diff rationale as `RebuildQueueChanged`).
- `coordinator.rs`: `emit_schedules_snapshot()` helper — queries the
scheduled_prompts list, converts to wire shape, broadcasts the event.
- `dashboard.rs`: call `emit_schedules_snapshot()` at the end of each
operator API handler that mutates a schedule:
`post_schedule_new`, `post_schedule_fire_now`,
`patch_schedule`, `post_schedule_cancel`.
- `scheduled_prompts_worker.rs`: call `emit_schedules_snapshot()`
after each tick that fires schedules, so `last_fired_at_unix`,
`next_fire_at_unix`, and reaped one-shots surface live.
Frontend:
- `tabs.js`: add `applySchedulesChanged(ev)` — replaces
`schedulesState` from the snapshot and calls `renderSchedulesList()`.
Registered in `MUTATION_HANDLERS` as `schedules_changed`.
Tab-activation re-fetch kept as safety net for approval-path
inserts and disconnect windows; comment updated to reflect this.
Docs:
- `docs/web-ui/dashboard.md`: document `schedules_changed` event.
- `CLAUDE.md`: add `SchedulesChanged` to the file-map entry.
