mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

Merge forge/main into atlas/5034-s4-remove-hive-model

Browse source
This commit is contained in:
atlas 2026-10-11 03:42:22 +02:00
commit baefb32bd5
22 changed files with 921 additions and 248 deletions
Download patch file Download diff file Expand all files Collapse all files

4
Cargo.lock generated
View file

@ -1836,9 +1836,9 @@ dependencies = [
[[package]]
name = "hive-claude"
version = "0.1.2"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e43e06fd09a7b6d6cca3e70f06d00fa795ceddfe1555121ca0cf4fc578eb9c3"
checksum = "4d7c5638f8f59e438d6416b3048709c9b3d6a10c23872ecf3825e3fe4c1bd7e8"
dependencies = [
"libc",
"serde",

2
Cargo.toml
View file

@ -97,7 +97,7 @@ hive-jobq-metrics = { path = "hive-jobq-metrics" }
hive-jobq-wire = { path = "hive-jobq-wire" }
hive-log = { path = "hive-log" }
hive-core-agent-sock = { path = "hive-core-agent-sock" }
hive-claude = "0.1.2"
hive-claude = "0.1.4"
hive-host-sock = { path = "hive-host-sock" }
hive-priv-sock = { path = "hive-priv-sock" }
hive-runtime = { path = "hive-runtime" }

5
claude-plugins/plugins/base/skills/claude-subagents/SKILL.md
View file

@ -27,7 +27,7 @@ skill's recipe for you:
start(name, prompt_file, model?, effort?, trigger?)
continue(name, prompt, model?, effort?)
status(name)
interrupt(name, force?)
interrupt(name)
```
`start` and `continue` return once the turn is under way, not once it
@ -42,7 +42,8 @@ reports how long since that turn last produced output, so a few seconds
means it's working and an age climbing into the minutes means it's
wedged and worth an `interrupt`. Reach for `continue` only once you
actually have a new instruction for it, since that spends a turn.
`interrupt` genuinely stops a running turn (`force: true` for SIGKILL).
`interrupt` genuinely stops a running turn, with SIGINT so claude stops
the commands it started; `status` says `stopping` until claude has exited.
Model choice, prompt hygiene, splitting big batches, verify-then-report —
everything else in this skill — applies exactly the same whether you're

5
docs/agent-lifecycle/persistence.md
View file

@ -209,6 +209,11 @@ Writes are best-effort: `append_stdout` / `append_stderr` / `finish`
log a warning on sqlite error and let the build continue. A failed
log row never blocks a rebuild.
A row is open (`finished_at IS NULL`) only while its child runs. At startup
`Coordinator::open` closes every row still open as `fail` and appends a
line saying hive-c0re stopped: the previous process died mid-build, and
the retention sweep never deletes open rows.
### `/harness/hyperhive-model` (per agent)
Single-line text file holding a claude model name. The harness only

19
docs/tools/subagent.md
View file

@ -166,6 +166,16 @@ from how the child exited. `continue` clears it and hands the session a
fresh turn allowance, which is what makes an interrupt a pause you can
undo rather than a session you have to abandon.
On `claude`, the signal is always SIGINT, and it goes to claude's whole
process group. The signal doesn't reach a command that moved itself out of
the group (`setsid`), and claude's Bash tool starts every command that way:
claude stops those commands itself when it shuts down on SIGINT. A SIGKILL
would leave them running, so nothing escalates to one. `interrupt` returns
at once, and the name stays claimed until claude has exited: `status` says
**stopping**, and the daemon refuses `start` and `continue` on that name
until then. A claude that ignores the SIGINT stays stopping until someone kills
it by hand or the unit stops.
When the session knows where its report goes — `start`'s `report_file`,
or the path the subagent names when it signals — the daemon appends the
stop reason to that file as well, so the artifact you were going to read
@ -338,7 +348,7 @@ page applies as written.
On `acp`:
- each run spawns its own copy of the parent's ACP agent. The end of the
run kills it and every command it started. The unit loads
run sends SIGINT to it and every command in its process group. The unit loads
`backendEnvironmentFile` for it. On the `opencode` preset, the daemon
reads a provider key that file leaves unset from the swarm secret store
at `swarm/agents/<agent>/acp-provider`, as the harness does.
@ -347,9 +357,10 @@ On `acp`:
agent process, and a `continue` with none recorded fails straight away.
- a role goes in front of the session's first prompt, since ACP has no
system prompt.
- `interrupt` sends `session/cancel`, and `force` changes nothing. The
runtime kills an agent that ignores the cancel for 10s, with every
command it started.
- `interrupt` sends `session/cancel`. The runtime sends SIGINT to the
process group of an agent that ignores the cancel for 10s, and when the
run ends. Nothing escalates past that SIGINT. The name is free as soon as
`interrupt` sends the cancel.
- the runtime sets `model` and `effort` on the session when the agent
offers that value for its `model` or `thought_level` config option. Any
other value does nothing (`hive_runtime::AcpRuntime`).

10
docs/turn-loop/README.md
View file

@ -142,8 +142,8 @@ else a `TurnError`) drives the post-claude branch:
| `Err(RateLimited)` | sleep `HIVE_RATE_LIMIT_SLEEP_SECS` (default 300), requeue inflight, status back to `online` |
| `Err(AuthFailed)` | emit `needs_login_idle` sentinel, requeue inflight, park in `wait_for_login` |
| `Err(SessionNotFound)` | resume + create self-heal both missed ("shouldn't happen"); requeue inflight so the next turn creates fresh — no status park, message not dropped |
| `Err(ApiStall)` | idle watchdog killed claude after `HIVE_TURN_IDLE_SECS` (default 600) of output silence; sleep `HIVE_STALL_SLEEP_SECS` (default 60), requeue inflight, status back to `online` |
| `Err(AgentStall(note))` | the same, for an ACP agent: after `HIVE_TURN_IDLE_SECS` with no `session/update` the harness sent the agent `session/cancel`, and killed it if it ignored that; `note` says which |
| `Err(ApiStall)` | idle watchdog sent claude SIGINT after `HIVE_TURN_IDLE_SECS` (default 600) of output silence and claude exited; sleep `HIVE_STALL_SLEEP_SECS` (default 60), requeue inflight, status back to `online` |
| `Err(AgentStall(note))` | the same, for an ACP agent: after `HIVE_TURN_IDLE_SECS` with no `session/update` the harness sent the agent `session/cancel`, and SIGINT if it ignored that; `note` says which |
| `Err(Failed(err))` | route `[system] \`<qualified-label>\` claude turn failed:\n<err>` to `operator` via `send_to_operator` |
`ApiStall` catches an Anthropic API stall — a multi-retry connection storm where
@ -152,7 +152,11 @@ the stream goes silent for minutes. On `claude` the idle watchdog lives in
resets on every stdout line, so a large but still-streaming turn is never cut —
only complete output silence for the window trips it. The harness sets the
window from `HIVE_TURN_IDLE_SECS` (`0` disables) and maps the driver's
`Error::IdleTimeout` onto `TurnError::ApiStall`.
`Error::IdleTimeout` onto `TurnError::ApiStall`. The watchdog sends claude's
process group SIGINT, so claude stops the commands it started, and then
waits for claude to exit. Nothing escalates to SIGKILL: a claude that
ignores the SIGINT keeps the turn open until someone kills it or the unit
stops.
`hive-runtime` applies the same window to an ACP agent's turns. That's
also how a provider HTTP 429 ends an ACP turn when the agent retries it without

20
docs/web-ui/agent.md
View file

@ -229,16 +229,17 @@ Slash commands today:
- `/help` — list commands locally.
- `/clear` — wipe the local terminal view (server history kept).
- `/cancel` — `POST /api/cancel` → host shellouts `pkill -INT
claude`, emits a Note. On an ACP agent it sends the agent
`session/cancel` instead, and kills it if the turn hasn't ended 10s
later. Also surfaces as a `■ cancel turn` button in the state row
while state=thinking.
- `/cancel` — `POST /api/cancel` → SIGINT to the turn's claude and every
command it started (its process group), emits a Note. On an ACP agent
it sends the agent `session/cancel` instead, and SIGINT if the turn
hasn't ended 10s later. Also surfaces as a `■ cancel turn` button in
the state row while state=thinking.
- `/compact` — `POST /api/compact` → sets the deferred
`Bus::request_compact()` flag and returns immediately. The harness
runs the `/compact` at the next turn boundary (end of the in-flight
turn, or `turn::run_pending_compact` when idle); output streams into
the live panel. Works mid-turn, not only when idle.
the live panel. Works mid-turn, not only when idle. Either way the live
panel ends with a `/compact done` or `/compact failed: …` note.
- `/new-session` — `POST /api/new-session` (confirms first).
Arms a one-shot on the Bus; next turn runs without
`--continue`, dropping the resume session entirely.
@ -266,8 +267,8 @@ shaped).
- `POST /login/{start,code,cancel}` — claude OAuth login flow.
Start/cancel emit `LiveEvent::StatusChanged` to flip the
badge to/from `needs_login_in_progress`.
- `POST /api/cancel` — SIGINT the in-flight claude turn. Emits a
`LiveEvent::Note`.
- `POST /api/cancel` — SIGINT the in-flight claude turn and the
commands it started. Emits a `LiveEvent::Note`.
- `POST /api/compact` — run `/compact` on the persistent session
(same MCP config + system prompt + allowed tools as a normal
turn — only the stdin payload differs). Flips state to
@ -363,7 +364,8 @@ shaped).
`{"<name>": "<upstream_url>"}`). Each entry mounts a transparent
reverse-proxy at `/extra/<name>/` that forwards every request (method,
headers, body) to the configured upstream, strips hop-by-hop headers
on both sides, and buffers the full response body (MVP — SSE
on both sides (repeated response headers such as two `Set-Cookie` lines
are all forwarded), and buffers the full response body (MVP — SSE
connections will appear as one large response rather than streaming).
The `/extra/` namespace ensures user-declared proxies can never
conflict with native agent endpoints. Upstream values are either an

36
hive-agent/src/paths.rs
View file

@ -132,7 +132,37 @@ pub fn claude_dir() -> PathBuf {
PathBuf::from("/root/.claude")
}
/// Held by every test that points `HYPERHIVE_HARNESS_DIR` at a temp dir:
/// cargo runs a crate's tests on parallel threads of one process.
/// Run `f` with `HYPERHIVE_HARNESS_DIR` pointing at a fresh temp dir, then
/// restore the previous value. Tests that build a `Bus` or read the harness
/// state files go through this: the variable is process-wide, so the calls
/// are serialised on a lock.
#[cfg(test)]
pub(crate) static HARNESS_DIR_ENV: std::sync::Mutex<()> = std::sync::Mutex::new(());
pub fn with_harness_dir<T>(f: impl FnOnce() -> T) -> T {
use std::ffi::OsString;
use std::sync::Mutex;
static LOCK: Mutex<()> = Mutex::new(());
struct Restore(Option<OsString>);
impl Drop for Restore {
fn drop(&mut self) {
// SAFETY: still holding `LOCK`, the only writer of this variable.
unsafe {
match self.0.take() {
Some(v) => std::env::set_var("HYPERHIVE_HARNESS_DIR", v),
None => std::env::remove_var("HYPERHIVE_HARNESS_DIR"),
}
}
}
}
let _lock = LOCK
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let dir = tempfile::tempdir().unwrap();
// Declared after `_lock` so the variable is restored before the lock drops.
let _restore = Restore(std::env::var_os("HYPERHIVE_HARNESS_DIR"));
// SAFETY: serialised by `LOCK`.
unsafe { std::env::set_var("HYPERHIVE_HARNESS_DIR", dir.path()) };
f()
}

55
hive-agent/src/swarm_inference.rs
View file

@ -320,37 +320,30 @@ mod tests {
/// persisted files, so clearing it, or a restart, finds the local value.
#[test]
fn the_swarm_value_is_not_persisted_over_the_local_one() {
let dir = tempfile::tempdir().unwrap();
let _env = crate::paths::HARNESS_DIR_ENV
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let prev = std::env::var_os("HYPERHIVE_HARNESS_DIR");
// SAFETY: every test that sets this variable holds `HARNESS_DIR_ENV`.
unsafe { std::env::set_var("HYPERHIVE_HARNESS_DIR", dir.path()) };
let bus = Bus::new();
std::fs::write(dir.path().join("hyperhive-model"), "sonnet\n").unwrap();
std::fs::write(dir.path().join("hyperhive-effort"), "low\n").unwrap();
let target = Harness {
bus: bus.clone(),
nix_model: None,
nix_effort: None,
};
let mut o = Overridden::default();
o.apply(&set(Some("opus"), Some(Effort::Max)), &target);
let swarm = (bus.model(), bus.effort());
let swarm_sources = (bus.model_source(), bus.effort_source());
let files = (
crate::harness_state::load_model(),
crate::harness_state::load_effort(),
);
o.apply(&AgentInference::default(), &target);
let cleared = (bus.model(), bus.effort());
let cleared_sources = (bus.model_source(), bus.effort_source());
match prev {
// SAFETY: as above.
Some(v) => unsafe { std::env::set_var("HYPERHIVE_HARNESS_DIR", v) },
None => unsafe { std::env::remove_var("HYPERHIVE_HARNESS_DIR") },
}
let (swarm, swarm_sources, files, cleared, cleared_sources) =
crate::paths::with_harness_dir(|| {
let bus = Bus::new();
let dir = crate::paths::harness_dir();
std::fs::write(dir.join("hyperhive-model"), "sonnet\n").unwrap();
std::fs::write(dir.join("hyperhive-effort"), "low\n").unwrap();
let target = Harness {
bus: bus.clone(),
nix_model: None,
nix_effort: None,
};
let mut o = Overridden::default();
o.apply(&set(Some("opus"), Some(Effort::Max)), &target);
let swarm = (bus.model(), bus.effort());
let swarm_sources = (bus.model_source(), bus.effort_source());
let files = (
crate::harness_state::load_model(),
crate::harness_state::load_effort(),
);
o.apply(&AgentInference::default(), &target);
let cleared = (bus.model(), bus.effort());
let cleared_sources = (bus.model_source(), bus.effort_source());
(swarm, swarm_sources, files, cleared, cleared_sources)
});
assert_eq!(swarm, pair("opus", "max"));
assert_eq!(files, (Some("sonnet".to_owned()), Some("low".to_owned())));
assert_eq!(cleared, pair("sonnet", "low"));

103
hive-agent/src/turn.rs
View file

@ -50,8 +50,8 @@ const DEFAULT_SESSION_TITLE: &str = "hive-session";
/// capacity limits.
const DEFAULT_RATE_LIMIT_SLEEP_SECS: u64 = 300;
/// Idle watchdog window: kill claude and surface `ApiStall` if it produces no
/// stdout for this long. The timer resets on every stdout line, so a large but
/// Idle watchdog window: send claude SIGINT and surface `ApiStall` once it
/// exits, if it produces no stdout for this long. The timer resets on every stdout line, so a large but
/// still-streaming turn is never cut — only a complete silence trips it.
/// Default is 10 minutes: long enough for legitimate big-context turns, short
/// enough to cut a multi-retry Anthropic connection storm (atlas telemetry:
@ -155,7 +155,7 @@ pub async fn write_system_prompt(socket: &Path, label: &str) -> Result<PathBuf>
/// One claude turn's outcome: `Ok(compacted)` on success, or a [`TurnError`]
/// the serve loop must act on. The `compacted` bool is `true` when a
/// compaction ran this turn (reactively on overflow, or proactively per the
/// policy — or an operator `/compact` at turn end); it's recorded as
/// policy — or a successful operator `/compact` at turn end); it's recorded as
/// `result_kind = "compacted"` in turn stats so the stats page can distinguish
/// those turns. Both `Ok(true)` and `Ok(false)` are ack'd; the error cases
/// each map to a distinct serve-loop action (see [`emit_turn_end`] and the
@ -189,14 +189,14 @@ pub enum TurnError {
/// the wake message, the serve loop requeues it so the next turn retries;
/// no status park.
SessionNotFound,
/// The harness-side idle watchdog killed claude after `turn_idle_secs()` of
/// The harness-side idle watchdog stopped claude after `turn_idle_secs()` of
/// output silence — indicative of an Anthropic API stall (e.g. a multi-retry
/// connection storm burning minutes of wall-clock with no stream progress).
/// The serve loop parks for `stall_sleep_secs()` and requeues, like the
/// rate-limit path — NOT a crash.
ApiStall,
/// The ACP backend's idle watchdog stopped a turn that sent nothing for
/// `turn_idle_secs()`: the agent was sent `session/cancel`, and killed if
/// `turn_idle_secs()`: the agent was sent `session/cancel`, and SIGINT if
/// it ignored that. Carries what happened. An ACP agent retrying a failing
/// provider (e.g. HTTP 429) without reporting it ends up here. Handled
/// like [`Self::ApiStall`].
@ -461,7 +461,7 @@ pub async fn drive_turn(
// Reflect `Compacting` in the UI like the idle path (`run_pending_compact`)
// does; the serve loop resets to `Idle` once this turn returns.
bus.set_state(crate::events::TurnState::Compacting);
let _ = session.compact(&config, &sink).await;
let compact_ok = report_compact(bus, session.compact(&config, &sink).await);
// If the compact call asked to be woken (the agent's own `compact`
// tool with a `wake_prompt`), stash it — the serve loop reads it
// back after this turn returns and drives a synthetic follow-up
@ -470,11 +470,38 @@ pub async fn drive_turn(
if let Some(prompt) = request.wake_prompt {
bus.set_post_compact_wake(prompt);
}
return Ok(true);
return with_turn_end_compact(outcome, compact_ok);
}
outcome
}
/// Emit the `/compact done` / `/compact failed: …` note for a finished
/// operator or agent `/compact` and return whether it succeeded. Shared by the
/// turn-end path in [`drive_turn`] and the idle path in [`run_pending_compact`].
fn report_compact<E: std::fmt::Display>(bus: &Bus, result: Result<(), E>) -> bool {
match result {
Ok(()) => {
bus.emit(LiveEvent::Note {
text: "/compact done".into(),
});
true
}
Err(e) => {
bus.emit(LiveEvent::Note {
text: format!("/compact failed: {e}"),
});
false
}
}
}
/// Fold a turn-end `/compact` into the turn's own outcome. A compaction that
/// ran during the turn (`Ok(true)`) stays recorded even when the turn-end
/// `/compact` failed; a failed `/compact` never turns it into `Ok(false)`.
fn with_turn_end_compact(outcome: TurnOutcome, compact_ok: bool) -> TurnOutcome {
outcome.map(|turn_compacted| turn_compacted || compact_ok)
}
/// Pre-turn auto-reset check. If context is large AND the prompt cache has
/// gone cold (idle time >= cache TTL), archive the current session so the
/// next wake-up turn's `--resume <title>` misses and self-heals into a fresh
@ -558,11 +585,11 @@ pub fn emit_turn_end(bus: &Bus, outcome: &TurnOutcome) {
bus.emit(LiveEvent::TurnEnd {
ok: false,
note: Some(format!(
"claude killed after {}s of output silence — API stall suspected, parking + requeueing",
"claude stopped with SIGINT after {}s of output silence — API stall suspected, parking + requeueing",
turn_idle_secs()
)),
});
tracing::warn!("turn killed: API stall (idle watchdog)");
tracing::warn!("turn stopped: API stall (idle watchdog)");
}
Err(TurnError::AgentStall(note)) => {
bus.emit(LiveEvent::TurnEnd {
@ -600,14 +627,7 @@ pub async fn run_pending_compact(files: &TurnFiles, bus: &Bus, session: &AgentSe
bus.set_state(crate::events::TurnState::Compacting);
let config = claude_config(bus, files);
let sink = BusSink::new(bus);
match session.compact(&config, &sink).await {
Ok(()) => bus.emit(LiveEvent::Note {
text: "/compact done".into(),
}),
Err(e) => bus.emit(LiveEvent::Note {
text: format!("/compact failed: {e}"),
}),
}
report_compact(bus, session.compact(&config, &sink).await);
bus.set_state(crate::events::TurnState::Idle);
if let Some(prompt) = request.wake_prompt {
bus.set_post_compact_wake(prompt);
@ -821,8 +841,10 @@ fn archive_session(bus: &Bus, session: &AgentSession) {
#[cfg(test)]
mod tests {
use super::{
PermissionAsk, TurnError, acp_permits, nonzero_or, parse_u64, runtime_error_to_turn,
PermissionAsk, TurnError, acp_permits, nonzero_or, parse_u64, report_compact,
runtime_error_to_turn, with_turn_end_compact,
};
use crate::events::{Bus, LiveEvent};
use hive_runtime::{AcpError, Error};
#[test]
@ -959,4 +981,49 @@ mod tests {
Err(TurnError::Failed(_))
));
}
fn compact_report(result: Result<(), &str>) -> (bool, Vec<String>) {
crate::paths::with_harness_dir(|| {
let bus = Bus::new();
let mut rx = bus.subscribe();
let ok = report_compact(&bus, result);
let mut notes = Vec::new();
while let Ok(ev) = rx.try_recv() {
if let LiveEvent::Note { text } = ev.event {
notes.push(text);
}
}
(ok, notes)
})
}
#[test]
fn a_failed_compact_is_reported_and_not_counted_as_compacted() {
let (compacted, notes) = compact_report(Err("spawn failed"));
assert!(!compacted);
assert_eq!(notes, ["/compact failed: spawn failed"]);
}
#[test]
fn a_successful_compact_is_reported_and_counted_as_compacted() {
let (compacted, notes) = compact_report(Ok(()));
assert!(compacted);
assert_eq!(notes, ["/compact done"]);
}
#[test]
fn a_failed_turn_end_compact_keeps_the_turns_own_compaction() {
assert!(matches!(with_turn_end_compact(Ok(true), false), Ok(true)));
}
#[test]
fn a_failed_turn_end_compact_leaves_a_plain_turn_uncompacted() {
assert!(matches!(with_turn_end_compact(Ok(false), false), Ok(false)));
}
#[test]
fn a_successful_turn_end_compact_marks_the_turn_compacted() {
assert!(matches!(with_turn_end_compact(Ok(false), true), Ok(true)));
assert!(matches!(with_turn_end_compact(Ok(true), true), Ok(true)));
}
}

2
hive-agent/src/web_ui/actions.rs
View file

@ -69,7 +69,7 @@ pub(super) async fn post_cancel_turn(State(state): State<AppState>) -> Response
.emit(crate::events::LiveEvent::Note { text: note.into() });
return (axum::http::StatusCode::OK, "ok").into_response();
}
let out = super::sigint_claude().await;
let out = super::sigint_claude();
let note = match out {
SigintOutcome::Signalled => {
// A process actually got signalled — the *next* turn's wake

2
hive-agent/src/web_ui/auth.rs
View file

@ -84,7 +84,7 @@ pub(super) async fn post_login_cancel(State(state): State<AppState>) -> Response
/// preservation invariants.
pub(super) async fn post_logout(State(state): State<AppState>) -> Response {
// Step 1: SIGINT claude (best-effort, matches `post_cancel_turn`).
let _ = super::sigint_claude().await;
let _ = super::sigint_claude();
// Step 2: delete OAuth credential files only — login::clear_session owns
// the file set and preserves session-history files alongside them.
let dir = crate::paths::claude_dir();

207
hive-agent/src/web_ui/mod.rs
View file

@ -343,21 +343,40 @@ fn find_claude_child() -> Option<u32> {
/// (best-effort). See [`find_claude_child`] for how it's identified —
/// **not** a name-based `pkill claude`, which would also hit an unrelated
/// `claude` process someone is running interactively in the container.
async fn sigint_claude() -> SigintOutcome {
fn sigint_claude() -> SigintOutcome {
let Some(pid) = find_claude_child() else {
return SigintOutcome::NoProcess;
};
match tokio::process::Command::new("kill")
.args(["-INT", &pid.to_string()])
.status()
.await
{
Ok(status) if status.success() => SigintOutcome::Signalled,
// Non-zero from `kill` means "no such process" (ESRCH) — it already
// exited between the check above and the signal. Same as never
// having found it.
Ok(_) => SigintOutcome::NoProcess,
Err(e) => SigintOutcome::Failed(e),
sigint_turn(pid)
}
/// SIGINT `pid`, and its whole process group when it leads one, so the
/// commands a turn's claude started stop with it. `hive-claude` spawns the
/// turn's claude as the leader of its own group. A `claude` child that
/// leads none (`plugins.rs`'s `claude plugin` calls) shares the harness's
/// group, so it is signalled alone.
fn sigint_turn(pid: u32) -> SigintOutcome {
// A child is never pid 0 or 1, and `-0` / `-1` would reach the
// harness's own group or every process it may signal.
let Some(pid) = i32::try_from(pid).ok().filter(|&pid| pid > 1) else {
return SigintOutcome::NoProcess;
};
// SAFETY: `getpgid(2)` only reads; a gone pid returns -1, never `pid`.
let target = if unsafe { libc::getpgid(pid) } == pid {
-pid
} else {
pid
};
// SAFETY: plain `kill(2)` on the child or the group it leads.
if unsafe { libc::kill(target, libc::SIGINT) } == 0 {
return SigintOutcome::Signalled;
}
let err = std::io::Error::last_os_error();
if err.raw_os_error() == Some(libc::ESRCH) {
// It exited between the scan and the signal: same as never found.
SigintOutcome::NoProcess
} else {
SigintOutcome::Failed(err)
}
}
@ -425,14 +444,18 @@ fn broker_error_response(err: &BrokerError, action: &str) -> Response {
#[cfg(test)]
mod tests {
use std::os::unix::process::CommandExt as _;
use std::path::Path;
use std::process::{Command, Stdio};
use std::sync::atomic::AtomicBool;
use std::sync::{Arc, Mutex, PoisonError};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use axum::body::Body;
use axum::http::{Request, StatusCode, header};
use tower::ServiceExt;
use super::{AppState, routes};
use super::{AppState, SigintOutcome, routes, sigint_turn};
use crate::events::Bus;
use crate::login::LoginState;
@ -447,45 +470,127 @@ mod tests {
/// either.
#[test]
fn neither_model_nor_effort_has_a_route() {
let dir = tempfile::tempdir().unwrap();
let _env = crate::paths::HARNESS_DIR_ENV
.lock()
.unwrap_or_else(PoisonError::into_inner);
let prev = std::env::var_os("HYPERHIVE_HARNESS_DIR");
// SAFETY: every test that sets this variable holds `HARNESS_DIR_ENV`.
unsafe { std::env::set_var("HYPERHIVE_HARNESS_DIR", dir.path()) };
let state = AppState {
label: "test".to_owned(),
login: Arc::new(Mutex::new(LoginState::Online)),
session: Arc::new(Mutex::new(None)),
bus: Bus::new(),
socket: dir.path().join("mcp.sock"),
gui_vnc_port: None,
interrupted: Arc::new(AtomicBool::new(false)),
};
let app = routes().with_state(state);
let (model, effort, compact) = tokio::runtime::Builder::new_current_thread()
.build()
.unwrap()
.block_on(async {
let model = app.clone().oneshot(form("/api/model", "model=opus"));
let effort = app.clone().oneshot(form("/api/effort", "effort=high"));
// A route the router does serve, so the 404s above are not
// the router refusing everything.
let compact = app.oneshot(form("/api/compact", ""));
(
model.await.unwrap(),
effort.await.unwrap(),
compact.await.unwrap(),
)
});
match prev {
// SAFETY: as above.
Some(v) => unsafe { std::env::set_var("HYPERHIVE_HARNESS_DIR", v) },
None => unsafe { std::env::remove_var("HYPERHIVE_HARNESS_DIR") },
}
let (model, effort, compact) = crate::paths::with_harness_dir(|| {
let state = AppState {
label: "test".to_owned(),
login: Arc::new(Mutex::new(LoginState::Online)),
session: Arc::new(Mutex::new(None)),
bus: Bus::new(),
socket: crate::paths::harness_dir().join("mcp.sock"),
gui_vnc_port: None,
interrupted: Arc::new(AtomicBool::new(false)),
};
let app = routes().with_state(state);
tokio::runtime::Builder::new_current_thread()
.build()
.unwrap()
.block_on(async {
let model = app.clone().oneshot(form("/api/model", "model=opus"));
let effort = app.clone().oneshot(form("/api/effort", "effort=high"));
// A route the router does serve, so the 404s above are not
// the router refusing everything.
let compact = app.oneshot(form("/api/compact", ""));
(
model.await.unwrap(),
effort.await.unwrap(),
compact.await.unwrap(),
)
})
});
assert_eq!(model.status(), StatusCode::NOT_FOUND);
assert_eq!(effort.status(), StatusCode::NOT_FOUND);
assert_eq!(compact.status(), StatusCode::OK);
}
/// Poll `path` until it holds a complete, newline-terminated pid.
fn read_pid(path: &Path) -> i32 {
for _ in 0..250 {
if let Ok(text) = std::fs::read_to_string(path)
&& text.ends_with('\n')
{
return text.trim().parse().expect("pid file holds a pid");
}
std::thread::sleep(Duration::from_millis(20));
}
unreachable!("no pid in {} within 5s", path.display());
}
/// Whether `pid` is still running. A zombie counts as gone: it has
/// exited, and `kill(pid, 0)` only keeps finding it until it is reaped.
fn is_running(pid: i32) -> bool {
// SAFETY: signal 0 delivers nothing; it only checks that `pid` exists.
if unsafe { libc::kill(pid, 0) } != 0 {
return false;
}
!std::fs::read_to_string(format!("/proc/{pid}/stat")).is_ok_and(|stat| {
stat.rsplit_once(") ")
.is_some_and(|(_, rest)| rest.starts_with('Z'))
})
}
#[test]
fn a_cancel_stops_the_commands_a_turn_started() {
let dir = tempfile::tempdir().expect("tempdir");
let pidfile = dir.path().join("grandchild.pid");
// A turn's claude as hive-claude spawns it: leading its own group,
// with a command of its own running in the foreground (a
// non-interactive shell starts `&` jobs with SIGINT ignored). The
// trailing `exit` keeps `sh -c` from exec'ing into that command.
let mut turn = Command::new("/bin/sh")
.args([
"-c",
r#"sh -c 'echo $$ >"$1"; exec sleep 300' sh "$1"; exit"#,
"sh",
])
.arg(&pidfile)
.stdout(Stdio::null())
.stderr(Stdio::null())
.process_group(0)
.spawn()
.expect("spawn the stand-in turn");
let grandchild = read_pid(&pidfile);
assert_ne!(
i32::try_from(turn.id()).ok(),
Some(grandchild),
"the command must be a child of the turn, not the turn itself"
);
assert!(matches!(sigint_turn(turn.id()), SigintOutcome::Signalled));
let mut alive = is_running(grandchild);
for _ in 0..250 {
if !alive {
break;
}
std::thread::sleep(Duration::from_millis(20));
alive = is_running(grandchild);
}
if alive {
// SAFETY: `kill(2)` on a pid this test started; keeps a failed
// run from leaving a 300s `sleep` behind.
unsafe {
libc::kill(grandchild, libc::SIGKILL);
}
}
let _ = turn.wait();
assert!(!alive, "grandchild {grandchild} outlived the cancel");
}
#[test]
fn a_claude_sharing_the_harness_group_is_signalled_alone() {
// In this test process's own group, as `claude plugin` runs. A group
// signal here would SIGINT the test binary itself.
let mut child = Command::new("sleep")
.arg("300")
.spawn()
.expect("spawn sleep");
assert!(matches!(sigint_turn(child.id()), SigintOutcome::Signalled));
let status = child.wait().expect("reap sleep");
assert_eq!(
std::os::unix::process::ExitStatusExt::signal(&status),
Some(libc::SIGINT)
);
}
}

71
hive-agent/src/web_ui/proxy.rs
View file

@ -169,7 +169,7 @@ async fn proxy_handler(
HeaderName::from_bytes(name.as_str().as_bytes()),
HeaderValue::from_bytes(value.as_bytes()),
) {
resp_headers.insert(n, v);
resp_headers.append(n, v);
}
}
match upstream_resp.bytes().await {
@ -269,8 +269,75 @@ async fn forward_over_unix_socket(
if HOP_BY_HOP.contains(&name.as_str()) {
continue;
}
resp_headers.insert(name.clone(), value.clone());
resp_headers.append(name.clone(), value.clone());
}
let body_bytes = upstream_resp.into_body().collect().await?.to_bytes();
Ok((status, resp_headers, body_bytes).into_response())
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
/// Raw HTTP/1.1 answer with two `Set-Cookie` headers and a body.
const TWO_COOKIES: &[u8] = b"HTTP/1.1 200 OK\r\n\
Set-Cookie: session=abc\r\n\
Set-Cookie: csrf=def\r\n\
Content-Length: 2\r\n\
Connection: close\r\n\r\nok";
/// Read the request head, then answer with [`TWO_COOKIES`].
async fn answer_once<S: AsyncReadExt + AsyncWriteExt + Unpin>(mut stream: S) {
let mut buf = [0u8; 1024];
let _ = stream.read(&mut buf).await;
stream.write_all(TWO_COOKIES).await.unwrap();
stream.shutdown().await.unwrap();
}
fn cookies(resp: &Response) -> Vec<&str> {
resp.headers()
.get_all("set-cookie")
.iter()
.map(|v| v.to_str().unwrap())
.collect()
}
#[tokio::test]
async fn unix_upstream_keeps_repeated_response_headers() {
let dir = tempfile::tempdir().unwrap();
let sock = dir.path().join("up.sock");
let listener = tokio::net::UnixListener::bind(&sock).unwrap();
tokio::spawn(async move {
let (stream, _) = listener.accept().await.unwrap();
answer_once(stream).await;
});
let resp =
forward_over_unix_socket(&sock, Method::GET, "/", HeaderMap::new(), Bytes::new())
.await
.unwrap();
assert_eq!(cookies(&resp), ["session=abc", "csrf=def"]);
}
#[tokio::test]
async fn http_upstream_keeps_repeated_response_headers() {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let base = format!("http://{}", listener.local_addr().unwrap());
tokio::spawn(async move {
let (stream, _) = listener.accept().await.unwrap();
answer_once(stream).await;
});
let client = reqwest::Client::builder().no_proxy().build().unwrap();
let resp = proxy_handler(
State((Arc::new(client), Arc::new(base))),
Method::GET,
Uri::from_static("/"),
HeaderMap::new(),
Bytes::new(),
)
.await;
assert_eq!(cookies(&resp), ["session=abc", "csrf=def"]);
}
}

41
hive-c0re/src/coordinator.rs
View file

@ -462,6 +462,15 @@ impl Coordinator {
let build_logs = Arc::new(
crate::build_logs::BuildLogs::open(build_logs_dir).context("open build_logs")?,
);
// This process is the database's only writer and has started no
// build yet, so a row still in-flight belongs to a dead process.
let reaped = build_logs.reap_orphans()?;
if reaped > 0 {
tracing::warn!(
reaped,
"build_logs: closed rows left in-flight by a previous run"
);
}
// Install the process-wide handle so `lifecycle::run` /
// `lifecycle::prebuild_toplevel` can write without us having
// to thread an `Arc<BuildLogs>` through every public entry
@ -1376,3 +1385,35 @@ mod agent_paths_tests {
assert_eq!(paths.agent, std::path::PathBuf::from("/run/x"));
}
}
#[cfg(test)]
mod open_tests {
use super::{Coordinator, HiveEnv};
fn open(db: &std::path::Path) -> Coordinator {
Coordinator::open(
db,
HiveEnv::default(),
crate::hive_stats::PriceTable::new(),
1,
)
.expect("open coordinator")
}
/// A build log row the previous process left open is closed as failed
/// when the next one opens the coordinator.
#[tokio::test]
async fn open_closes_build_logs_left_in_flight() {
let dir = tempfile::tempdir().expect("tempdir");
let db = dir.path().join("broker.sqlite");
let id = open(&db)
.build_logs
.start("alice", "run", "cmd", None)
.expect("start");
let coord = open(&db);
let row = coord.build_logs.get_full(id).expect("get").expect("Some");
assert_eq!(row.header.status.as_deref(), Some("fail"));
assert!(row.header.finished_at.is_some());
}
}

48
hive-c0re/src/snapshot_push.rs
View file

@ -18,6 +18,8 @@
//! stores would split that chain in two, which is the case the store
//! exists to serve.
use std::os::fd::OwnedFd;
use anyhow::{Context as _, Result, bail};
use hive_priv_sock::PrivRequest;
use tokio::io::AsyncWriteExt as _;
@ -61,10 +63,7 @@ pub async fn push_agent_snapshot(agent: &str, snapshot: &str, parent: Option<&st
// closes our copy as soon as the kernel has it, so the receiver sees
// EOF when `btrfs send` finishes rather than hanging on a descriptor
// this process is still holding.
let fd = sock
.into_std()
.context("detach snapshot-store socket for hand-off")?
.into();
let fd = detach_blocking(sock)?;
let resp = priv_client::call_with_fd(
&PrivRequest::SendAgentSnapshotToFd {
@ -93,6 +92,22 @@ pub async fn push_agent_snapshot(agent: &str, snapshot: &str, parent: Option<&st
Ok(())
}
/// Turn a tokio socket into a descriptor safe to hand to `btrfs send`.
///
/// tokio sockets are `O_NONBLOCK`, and `into_std` does not clear it. The
/// flag lives on the open file description, which `hive-priv` shares with
/// us through `SCM_RIGHTS`, so `btrfs send` would write to a
/// non-blocking socket and get `EAGAIN` whenever the send buffer is full.
fn detach_blocking(sock: TcpStream) -> Result<OwnedFd> {
let std_sock = sock
.into_std()
.context("detach snapshot-store socket for hand-off")?;
std_sock
.set_nonblocking(false)
.context("switch snapshot-store socket to blocking mode for hand-off")?;
Ok(std_sock.into())
}
/// Require a configured store address.
///
/// Split out of [`push_agent_snapshot`] so the "not configured" path is
@ -118,7 +133,30 @@ fn require_store_addr(configured: Option<&str>) -> Result<&str> {
#[cfg(test)]
mod tests {
use super::require_store_addr;
use std::os::fd::AsRawFd as _;
use tokio::net::{TcpListener, TcpStream};
use super::{detach_blocking, require_store_addr};
#[tokio::test]
async fn the_handed_off_descriptor_is_blocking() {
let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind");
let sock = TcpStream::connect(listener.local_addr().expect("addr"))
.await
.expect("connect");
let fd = detach_blocking(sock).expect("detach");
// SAFETY: `fd` is a live descriptor owned by this test.
let flags = unsafe { libc::fcntl(fd.as_raw_fd(), libc::F_GETFL) };
assert!(flags >= 0, "F_GETFL failed");
assert_eq!(
flags & libc::O_NONBLOCK,
0,
"btrfs send would get EAGAIN on a full socket buffer"
);
}
#[test]
fn a_configured_address_passes_through() {

52
hive-c0re/src/stores/build_logs.rs
View file

@ -48,6 +48,9 @@ const KEEP_FAIL_SECS: i64 = 30 * 24 * 3600;
/// `finished_at IS NULL`).
const KEEP_OK_SECS: i64 = 24 * 3600;
/// Last line of a row `reap_orphans` closed.
const ORPHAN_NOTE: &str = "hive-c0re stopped before this build finished";
const SCHEMA: &str = "
CREATE TABLE IF NOT EXISTS build_logs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
@ -175,6 +178,30 @@ impl BuildLogs {
})
}
/// Close every row still marked in-flight as failed. Returns the number
/// closed.
///
/// Only sound when no other writer holds this database: the caller must
/// be the one process that owns it, before it starts any build. A row
/// with `finished_at IS NULL` then belongs to a process that died, and
/// nothing will ever finish it. Kept out of `open` so a second opener
/// can't close another process's live rows.
pub fn reap_orphans(&self) -> Result<u64> {
let now = Utc::now().timestamp();
let conn = self.conn.lock().unwrap();
let closed = conn
.execute(
"UPDATE build_logs
SET finished_at = ?1,
status = 'fail',
stderr = stderr || ?2 || x'0a'
WHERE finished_at IS NULL",
params![now, ORPHAN_NOTE],
)
.context("close orphaned build_logs rows")?;
Ok(u64::try_from(closed).unwrap_or(0))
}
/// Subscribe to per-build append/finish notifications. Each emitted
/// value is the `id` of the row that changed. The SSE stream handler
/// calls this once and filters for its target id.
@ -522,6 +549,31 @@ mod tests {
assert!(full.stderr.ends_with('\n'));
}
#[test]
fn reap_orphans_fails_rows_left_in_flight_by_a_previous_process() {
let dir = tempfile::tempdir().expect("tempdir");
let (orphan, done) = {
let db = BuildLogs::open(dir.path()).expect("open");
let orphan = db.start("alice", "run", "cmd one", None).expect("start");
db.append_stdout(orphan, "building");
let done = db.start("bob", "run", "cmd two", None).expect("start");
db.finish(done, BuildStatus::Ok);
(orphan, done)
};
let db = BuildLogs::open(dir.path()).expect("reopen");
assert_eq!(db.reap_orphans().expect("reap"), 1);
let row = db.get_full(orphan).expect("get").expect("Some");
assert_eq!(row.header.status.as_deref(), Some("fail"));
assert!(row.header.finished_at.is_some());
assert!(row.stdout.contains("building"));
assert!(row.stderr.ends_with(&format!("{ORPHAN_NOTE}\n")));
let ok = db.get_full(done).expect("get").expect("Some");
assert_eq!(ok.header.status.as_deref(), Some("ok"));
assert_eq!(db.reap_orphans().expect("reap again"), 0);
}
#[test]
fn list_recent_orders_newest_first_and_clamps() {
let (_d, db) = tmpdb();

10
hive-runtime/README.md
View file

@ -35,13 +35,15 @@ The crate depends on no hyperhive binary crate, so `hive-agent` and
A turn is stopped with `session/cancel`: by the `Canceller` handle, or by the
idle watchdog once no `session/update` has arrived for
`Config::idle_timeout`. An agent that has not answered the prompt 10s later
is killed, and the next turn respawns it. A watchdog stop fails the turn
with `AcpError::IdleTimeout` (`IdleKilled` if the agent was killed). This is
is sent SIGINT, and the next turn respawns it. A watchdog stop fails the turn
with `AcpError::IdleTimeout` (`IdleKilled` if the agent was sent SIGINT). This is
the only way a provider error the agent retries without reporting it, such as
an HTTP 429, ends the turn.
The agent leads its own process group. Killing it, and dropping the runtime,
`SIGKILL`s that group, so the commands the agent started die with it.
The agent leads its own process group. Stopping it, and dropping the runtime,
sends that group SIGINT, so the commands the agent started get it too.
Nothing escalates to SIGKILL: an agent that ignores SIGINT keeps running,
and the next turn starts a new one beside it.
## ACP backend: compaction

46
hive-runtime/src/acp/mod.rs
View file

@ -39,7 +39,7 @@ const SETTLE_QUIET: Duration = Duration::from_millis(500);
const SETTLE_MAX: Duration = Duration::from_secs(5);
/// How long the agent has to end its turn after `session/cancel` before it is
/// killed.
/// sent SIGINT.
const CANCEL_GRACE: Duration = Duration::from_secs(10);
/// How long a `compact` command may send nothing before it is cancelled and
@ -120,10 +120,12 @@ pub enum AcpError {
EmptyEndTurn,
#[error(
"ACP agent sent nothing for {silent_secs}s and did not stop within {grace_secs}s of \
`session/cancel`, so it was killed"
`session/cancel`, so it was sent SIGINT"
)]
IdleKilled { silent_secs: u64, grace_secs: u64 },
#[error("ACP agent did not stop within {grace_secs}s of `session/cancel`, so it was killed")]
#[error(
"ACP agent did not stop within {grace_secs}s of `session/cancel`, so it was sent SIGINT"
)]
CancelIgnored { grace_secs: u64 },
}
@ -141,8 +143,8 @@ struct CancelState {
impl Canceller {
/// Ask the turn in flight to stop. The agent is sent `session/cancel`,
/// and the turn then ends normally with what it produced so far; an
/// agent that has not ended it within `CANCEL_GRACE` is killed, and the
/// turn fails with [`AcpError::CancelIgnored`]. Returns whether a turn
/// agent that has not ended it within `CANCEL_GRACE` is sent SIGINT, and
/// the turn fails with [`AcpError::CancelIgnored`]. Returns whether a turn
/// was in flight; with none, this does nothing.
#[must_use]
pub fn cancel(&self) -> bool {
@ -1062,9 +1064,14 @@ mod tests {
const AGENT: &str = r#"
n=0 prompt= model=m/think effort=low
printf 'start\n' >> "$1.methods"
# With TRAP_INT set: SIGINT creates <log>.int and exits 0.
if [ -n "$TRAP_INT" ]; then
trap ': > "$1.int"; exit 0' INT
fi
# With GRANDCHILD set: a background `sleep`, its pid in <log>.grandchild.
# `&` starts it with SIGINT ignored; `env` puts the default back.
if [ -n "$GRANDCHILD" ]; then
sh -c 'echo $$ >"$1.grandchild"; exec sleep 300' sh "$1" >/dev/null 2>&1 &
sh -c 'echo $$ >"$1.grandchild"; exec env --default-signal=INT sleep 300' sh "$1" >/dev/null 2>&1 &
fi
update() {
printf '{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"%s","update":%s}}\n' "$1" "$2"
@ -1529,6 +1536,33 @@ done
assert_gone(grandchild).await;
}
/// SIGINT rather than SIGKILL: an agent only stops the commands it
/// started itself if it gets to shut down.
#[tokio::test]
async fn dropping_the_runtime_lets_its_agent_shut_down_on_sigint() {
let dir = tempfile::tempdir().unwrap();
let runtime = agent(
dir.path(),
"ok",
&[("TRAP_INT", "1")],
PercentPolicy::default(),
);
runtime
.run(&config(dir.path()), "one", &NoopSink)
.await
.unwrap();
drop(runtime);
let marker = dir.path().join("prompts.int");
for _ in 0..250 {
if marker.exists() {
break;
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
assert!(marker.exists(), "the agent never got SIGINT");
}
fn sent(session: &str, text: &str) -> (String, String) {
(session.to_owned(), text.to_owned())
}

10
hive-runtime/src/acp/rpc.rs
View file

@ -31,9 +31,10 @@ type Reply = Result<Value, AcpError>;
type Pending = Arc<Mutex<HashMap<u64, (&'static str, oneshot::Sender<Reply>)>>>;
/// The agent runs as the leader of its own process group; dropping the
/// connection `SIGKILL`s that group, so the commands the agent started die
/// with it. A descendant that left the group (`setsid`, `setpgid`) is not
/// reached.
/// connection sends that group SIGINT, so the agent can stop the commands
/// it started, and they get the signal too. Nothing escalates: an agent
/// that ignores SIGINT keeps running. A descendant that left the group
/// (`setsid`, `setpgid`) is not reached.
pub(super) struct Connection {
child: Child,
/// The agent's pid, which is its group's id. Read at spawn: a reaped
@ -66,7 +67,6 @@ impl Connection {
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.process_group(0)
.kill_on_drop(true)
.spawn()
.map_err(|source| AcpError::Spawn {
program: command.command.clone(),
@ -182,7 +182,7 @@ impl Drop for Connection {
// SAFETY: plain `kill(2)`; `-pgid` targets the agent's own
// process group, which `process_group(0)` made it lead.
unsafe {
libc::kill(-pgid, libc::SIGKILL);
libc::kill(-pgid, libc::SIGINT);
}
}
}

Some files were not shown because too many files have changed in this diff Show more