hivectl: add agent <name> watch to follow live events from the CLI
This commit is contained in:
parent
657d1b5061
commit
9d5c7a7f7e
13 changed files with 230 additions and 16 deletions
|
|
@ -34,10 +34,10 @@ hand-maintained per-file tree drifts out of sync with the code.
|
|||
and the axum operator dashboard (`dashboard.rs`). Largest crate.
|
||||
- **`hivectl/`** — standalone operator CLI (`hivectl` binary). Talks to
|
||||
the `hive-c0re` daemon over the host admin socket (`hive-host-sock`
|
||||
wire types) — does NOT link `hive-c0re`. Verbs: `agents <spawn|kill|
|
||||
destroy|rebuild|restart|list|set-parent|…>`, `approvals <pending|
|
||||
approve|deny>`, `forge`/`matrix`/`github`/`gateway` provisioning,
|
||||
`choom`, `stop`/`start`, `wg`/`peer-config`.
|
||||
wire types) — does NOT link `hive-c0re`. Verbs: `list-agents`,
|
||||
`agent <name> <spawn|kill|destroy|rebuild|restart|set-parent|choom|
|
||||
watch|…>`, `approvals <pending|approve|deny>`, `forge`/`matrix`/
|
||||
`github`/`gateway` provisioning, `stop`/`start`, `wg`/`peer-config`.
|
||||
- **`hive-agent/`**, **`hive-agent-mcp/`** —
|
||||
in-container harness, two sibling crates for every agent (not a
|
||||
single `hive-ag3nt/` dir — that's the runtime/binary-family nickname,
|
||||
|
|
|
|||
4
Cargo.lock
generated
4
Cargo.lock
generated
|
|
@ -1834,12 +1834,16 @@ name = "hivectl"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"bytes",
|
||||
"clap",
|
||||
"clap-markdown",
|
||||
"clap_complete",
|
||||
"hive-host-sock",
|
||||
"hive-sh4re",
|
||||
"hive-types",
|
||||
"http-body-util",
|
||||
"hyper",
|
||||
"hyper-util",
|
||||
"indicatif",
|
||||
"libc",
|
||||
"serde_json",
|
||||
|
|
|
|||
|
|
@ -249,9 +249,13 @@ gateway always terminates TLS, so discovery responses always advertise https.
|
|||
`hive-c0re.nix` opens the per-agent web-port range
|
||||
`8100..8999` in the host firewall **only when
|
||||
`services.hyperhive.gateway.enable = false`**. With the gateway on
|
||||
(default), it's the sole external entry point and proxies to
|
||||
`127.0.0.1:<port>` internally — leaving the per-agent ports
|
||||
firewall-open would defeat the single-front-door story.
|
||||
(default) it's the sole external entry point and routes to agents over
|
||||
the UDS upstream described above (see [Per-agent unix-socket
|
||||
upstream](#per-agent-unix-socket-upstream)) — leaving the per-agent
|
||||
ports firewall-open would defeat the single-front-door story. The
|
||||
hashed TCP port (`lifecycle::agent_web_port`) still exists as a direct
|
||||
host-loopback fallback for the pre-UDS/gateway-disabled case, but isn't
|
||||
what the gateway itself proxies through.
|
||||
|
||||
`services.hyperhive.gateway.openFirewall = true` opens both `port` and
|
||||
`httpsPort` — the gateway always terminates TLS (self-signed floor), so
|
||||
|
|
|
|||
|
|
@ -32,6 +32,7 @@ This document contains the help content for the `hivectl` command-line program.
|
|||
* [`hivectl agent set-parent`↴](#hivectl-agent-set-parent)
|
||||
* [`hivectl agent set-limits`↴](#hivectl-agent-set-limits)
|
||||
* [`hivectl agent choom`↴](#hivectl-agent-choom)
|
||||
* [`hivectl agent watch`↴](#hivectl-agent-watch)
|
||||
* [`hivectl agent quota`↴](#hivectl-agent-quota)
|
||||
* [`hivectl agent quota show`↴](#hivectl-agent-quota-show)
|
||||
* [`hivectl agent quota set`↴](#hivectl-agent-quota-set)
|
||||
|
|
@ -348,6 +349,7 @@ Everything here targets a single named agent (`hivectl agent foo restart`, `hive
|
|||
* `set-parent` — Move this agent in the topology tree — under a new parent, or to root
|
||||
* `set-limits` — Declare this agent's CPU/memory limits, overriding the hive-wide defaults
|
||||
* `choom` — Open an interactive Claude session inside this agent's container
|
||||
* `watch` — Follow this agent's live turn/tool-call event stream from the CLI
|
||||
* `quota` — This agent's disk accounting + optional quota via btrfs qgroups
|
||||
* `subvol` — btrfs subvolume management for this agent's state dir
|
||||
|
||||
|
|
@ -476,6 +478,16 @@ A fresh session by default, or resume a prior one. Requires root and a running c
|
|||
|
||||
|
||||
|
||||
## `hivectl agent watch`
|
||||
|
||||
Follow this agent's live turn/tool-call event stream from the CLI.
|
||||
|
||||
Dials the same unix socket the gateway's nginx already proxies through (no gateway hop, no daemon round-trip for the stream itself) and prints one compact line per event. `Ctrl-C` to stop. Requires the agent to be running (its harness must have bound the web UI socket).
|
||||
|
||||
**Usage:** `hivectl agent watch`
|
||||
|
||||
|
||||
|
||||
## `hivectl agent quota`
|
||||
|
||||
This agent's disk accounting + optional quota via btrfs qgroups.
|
||||
|
|
|
|||
|
|
@ -266,6 +266,32 @@ lands in a faithful copy of the agent's environment:
|
|||
instead of the onboarding/trust dialog the headless harness never
|
||||
completes. It's the single place hyperhive touches that file.
|
||||
|
||||
## Watch
|
||||
|
||||
Follow an agent's live turn/tool-call event stream from the CLI:
|
||||
|
||||
```bash
|
||||
hivectl agent iris watch # tail iris's live events; Ctrl-C to stop
|
||||
```
|
||||
|
||||
Dials the same unix socket the gateway's nginx `proxy_pass`es through
|
||||
(`/run/hive-agent/<name>/web.sock` — see [Per-agent unix-socket
|
||||
upstream](../gateway.md#per-agent-unix-socket-upstream)) directly and
|
||||
speaks a bare HTTP/1.1 request for the agent's existing `/events/stream`
|
||||
SSE endpoint over it. No gateway hop, no daemon round-trip for the
|
||||
stream itself — the daemon socket is only used for the "does this agent
|
||||
exist" pre-flight, same reasoning as `choom` above. Requires the agent
|
||||
to be running with its web UI socket bound; a fresh spawn/rebuild that
|
||||
hasn't come up yet gets a clear connection-refused hint rather than a
|
||||
raw OS error.
|
||||
|
||||
Prints one compact line per event — reuses the `_icon`/`_summary` fields
|
||||
the harness already stamps onto stream-json events for the web UI
|
||||
(`stream_enrich.rs`), so tool calls and turn markers read as short
|
||||
glyph-prefixed lines instead of raw JSON. Not an attempt at the web
|
||||
UI's full collapsible-details rendering (`docs/terminal-rendering.md`)
|
||||
— that's presentation for a browser, this is a `tail -f`.
|
||||
|
||||
## Open
|
||||
|
||||
Print (and best-effort open in a browser) one of the hive's web surfaces.
|
||||
|
|
|
|||
|
|
@ -41,11 +41,6 @@ pub const STATE_ROOT: &str = "/var/lib/hyperhive";
|
|||
// stay in sync; the privsep boundary prevents importing across the crate.
|
||||
pub const RUNTIME_ROOT: &str = "/run/hyperhive";
|
||||
|
||||
/// `/run/hive-agent` — per-agent runtime socket dir root (web + bound
|
||||
/// markers), one subdir per agent.
|
||||
// nix: agent container bind-mount / `RuntimeDirectory` (the harness nix modules) — must match.
|
||||
pub const AGENT_SOCKET_DIR: &str = "/run/hive-agent";
|
||||
|
||||
/// Default broker db path (`db/broker.sqlite`). Exposed as a `&str` for
|
||||
/// the `--broker-db` clap `default_value`; `build_logs.sqlite` is placed
|
||||
/// alongside it (the build-logs store keys off the broker db's parent).
|
||||
|
|
|
|||
|
|
@ -17,9 +17,10 @@ use anyhow::{Context, Result};
|
|||
/// gateway container bind-mounts this whole tree (read-only) so it
|
||||
/// can `proxy_pass` to any agent. Each agent's container bind-mounts
|
||||
/// only its own `<name>/` subdir — agents can only access their own
|
||||
/// sockets. The literal lives in [`crate::paths`]; re-exported here
|
||||
/// under the name this module's consumers have always used.
|
||||
pub use crate::paths::AGENT_SOCKET_DIR;
|
||||
/// sockets. The literal lives in `hive-host-sock` (shared with
|
||||
/// `hivectl`); re-exported here under the name this module's consumers
|
||||
/// have always used.
|
||||
pub use hive_host_sock::AGENT_SOCKET_DIR;
|
||||
|
||||
/// Socket filename inside each per-agent subdir. Fixed so the path
|
||||
/// derives entirely from `(AGENT_SOCKET_DIR, name)` — no second
|
||||
|
|
|
|||
|
|
@ -52,6 +52,26 @@ pub fn agent_state_dir(name: &Ident) -> PathBuf {
|
|||
// nix: read by the gateway container's nginx (hive-gateway.nix) — must match.
|
||||
pub const GATEWAY_HTPASSWD: &str = "/var/lib/hyperhive/gateway/gateway.htpasswd";
|
||||
|
||||
/// `/run/hive-agent` — per-agent runtime socket dir root (web UI unix
|
||||
/// socket + bound marker), one subdir per agent. The gateway's nginx
|
||||
/// `proxy_pass`es to `agent_web_socket(name)` directly; `hivectl` dials
|
||||
/// the same socket for host-side tooling that needs to talk to a running
|
||||
/// agent's web UI without going through the gateway (e.g. `agent <name>
|
||||
/// watch`).
|
||||
// nix: agent container bind-mount / `RuntimeDirectory` (the harness nix modules) — must match.
|
||||
pub const AGENT_SOCKET_DIR: &str = "/run/hive-agent";
|
||||
|
||||
/// Per-agent web UI unix socket path — `AGENT_SOCKET_DIR/<name>/web.sock`.
|
||||
/// Same socket the gateway's nginx upstream and the harness's
|
||||
/// `HIVE_WEB_SOCKET` bind both derive from; see
|
||||
/// `docs/gateway.md::Per-agent unix-socket upstream`.
|
||||
#[must_use]
|
||||
pub fn agent_web_socket(name: &Ident) -> PathBuf {
|
||||
PathBuf::from(AGENT_SOCKET_DIR)
|
||||
.join(name.as_str())
|
||||
.join("web.sock")
|
||||
}
|
||||
|
||||
/// nspawn machine-name prefix for agent containers (`h-<name>`). A single
|
||||
/// `starts_with(AGENT_PREFIX)` filter enumerates managed containers.
|
||||
pub const AGENT_PREFIX: &str = "h-";
|
||||
|
|
|
|||
|
|
@ -13,12 +13,19 @@ path = "src/main.rs"
|
|||
|
||||
[dependencies]
|
||||
anyhow.workspace = true
|
||||
# `agent <name> watch` — bare HTTP/1.1 client over the per-agent unix
|
||||
# socket, same crate family `hive-agent/src/web_ui/proxy.rs` uses for the
|
||||
# reverse direction.
|
||||
bytes = "1"
|
||||
clap.workspace = true
|
||||
clap_complete.workspace = true
|
||||
clap-markdown = "0.1"
|
||||
hive-host-sock.workspace = true
|
||||
hive-sh4re.workspace = true
|
||||
hive-types.workspace = true
|
||||
http-body-util.workspace = true
|
||||
hyper.workspace = true
|
||||
hyper-util.workspace = true
|
||||
indicatif.workspace = true
|
||||
libc.workspace = true
|
||||
serde_json.workspace = true
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
//! `hivectl agent <name>` — everything scoped to ONE managed agent:
|
||||
//! container lifecycle over the host admin socket (restart/pause/resume/
|
||||
//! spawn/kill/destroy/rebuild/set-parent/set-limits/choom), plus the
|
||||
//! spawn/kill/destroy/rebuild/set-parent/set-limits/choom/watch), plus the
|
||||
//! `quota` and `subvol` groups, whose handlers live in their own modules.
|
||||
//! `agents_list` (`hivectl list-agents`) is the one genuinely hive-wide
|
||||
//! read that lives in this module too since it shares the same daemon
|
||||
|
|
@ -210,6 +210,7 @@ pub(crate) async fn run_agent(socket: &Path, name: &str, cmd: AgentCmd) -> Resul
|
|||
AgentCmd::Choom { resume_session } => {
|
||||
crate::choom::choom(socket, name, resume_session.as_deref()).await
|
||||
}
|
||||
AgentCmd::Watch => crate::watch::watch(socket, name).await,
|
||||
// `quota` and `subvol` keep their own modules — these arms are
|
||||
// just the reparenting glue that hoists `name` in from the
|
||||
// parent `agent <name>` command.
|
||||
|
|
|
|||
|
|
@ -534,6 +534,14 @@ pub enum AgentCmd {
|
|||
#[arg(long = "resume", value_name = "SESSION")]
|
||||
resume_session: Option<String>,
|
||||
},
|
||||
/// Follow this agent's live turn/tool-call event stream from the CLI.
|
||||
///
|
||||
/// Dials the same unix socket the gateway's nginx already proxies
|
||||
/// through (no gateway hop, no daemon round-trip for the stream
|
||||
/// itself) and prints one compact line per event. `Ctrl-C` to stop.
|
||||
/// Requires the agent to be running (its harness must have bound the
|
||||
/// web UI socket).
|
||||
Watch,
|
||||
/// This agent's disk accounting + optional quota via btrfs qgroups.
|
||||
///
|
||||
/// Needs `hivectl quota-enable` run once hive-wide first. No-op on
|
||||
|
|
|
|||
|
|
@ -42,6 +42,7 @@ mod wg;
|
|||
use wg::{peer_config, require_hive_domain, wg_init, wg_peer, wg_status};
|
||||
mod choom;
|
||||
mod github;
|
||||
mod watch;
|
||||
use github::github_set_token;
|
||||
mod forge;
|
||||
use forge::{forge_create_user, forge_reconcile_config};
|
||||
|
|
|
|||
135
hivectl/src/watch.rs
Normal file
135
hivectl/src/watch.rs
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
//! `hivectl agent <name> watch` — follow an agent's live event stream from
|
||||
//! the CLI. Dials the same unix socket the gateway's nginx already
|
||||
//! `proxy_pass`es through (`hive_host_sock::agent_web_socket`) directly and
|
||||
//! speaks a bare HTTP/1.1 request for the existing `/events/stream` SSE
|
||||
//! endpoint over it — no gateway hop, no daemon round-trip for the stream
|
||||
//! itself. Same client technique `hive-agent/src/web_ui/proxy.rs` uses for
|
||||
//! the reverse direction (an agent proxying out to a unix-socket upstream).
|
||||
//!
|
||||
//! The daemon socket is only used for the "does this agent exist"
|
||||
//! pre-flight — see `choom`'s doc comment for why that check has to be
|
||||
//! remote rather than a local stat.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
use anyhow::{Context as _, Result, bail};
|
||||
use http_body_util::{BodyExt, Empty};
|
||||
use hyper_util::rt::TokioIo;
|
||||
use tokio::net::UnixStream;
|
||||
|
||||
use crate::util::agent_exists;
|
||||
|
||||
pub(crate) async fn watch(socket: &Path, name: &str) -> Result<()> {
|
||||
if !agent_exists(socket, name).await? {
|
||||
bail!(
|
||||
"no such agent: '{name}' (no state dir under {}/)",
|
||||
hive_host_sock::AGENTS_ROOT
|
||||
);
|
||||
}
|
||||
let ident = crate::util::parse_ident(name)?;
|
||||
let sock_path = hive_host_sock::agent_web_socket(&ident);
|
||||
|
||||
let stream = UnixStream::connect(&sock_path).await.with_context(|| {
|
||||
format!(
|
||||
"connect to {name}'s web UI socket at {} — is the agent running? \
|
||||
(`hivectl list-agents`; a container that just spawned or rebuilt may \
|
||||
not have bound its socket yet)",
|
||||
sock_path.display()
|
||||
)
|
||||
})?;
|
||||
let io = TokioIo::new(stream);
|
||||
let (mut sender, conn) = hyper::client::conn::http1::handshake(io)
|
||||
.await
|
||||
.with_context(|| format!("HTTP handshake with {name}'s web UI socket"))?;
|
||||
// Drives the connection's I/O in the background; a one-shot GET has
|
||||
// nothing useful to do with the join handle, and a closed connection
|
||||
// surfaces to the caller anyway via the next `frame()` returning `None`
|
||||
// or an error.
|
||||
tokio::spawn(conn);
|
||||
|
||||
let req = hyper::Request::builder()
|
||||
.method(hyper::Method::GET)
|
||||
.uri("/events/stream")
|
||||
.header(hyper::header::HOST, "localhost")
|
||||
.header(hyper::header::ACCEPT, "text/event-stream")
|
||||
.body(Empty::<bytes::Bytes>::new())
|
||||
.context("build /events/stream request")?;
|
||||
|
||||
let resp = sender
|
||||
.send_request(req)
|
||||
.await
|
||||
.with_context(|| format!("request {name}'s /events/stream"))?;
|
||||
if !resp.status().is_success() {
|
||||
bail!(
|
||||
"{name}'s web UI returned {} for /events/stream",
|
||||
resp.status()
|
||||
);
|
||||
}
|
||||
|
||||
println!("watching {name} — Ctrl-C to stop");
|
||||
let mut body = resp.into_body();
|
||||
// SSE frames are separated by a blank line; a chunk boundary from the
|
||||
// socket has no relation to a frame boundary, so buffer raw text until
|
||||
// a full frame (up to the `\n\n` separator) has arrived before parsing.
|
||||
let mut pending = String::new();
|
||||
while let Some(frame) = body.frame().await {
|
||||
let frame = frame.with_context(|| format!("reading {name}'s event stream"))?;
|
||||
let Some(chunk) = frame.data_ref() else {
|
||||
continue;
|
||||
};
|
||||
pending.push_str(&String::from_utf8_lossy(chunk));
|
||||
while let Some(pos) = pending.find("\n\n") {
|
||||
let frame_text = pending[..pos].to_owned();
|
||||
pending.drain(..=pos + 1);
|
||||
render_frame(&frame_text);
|
||||
}
|
||||
}
|
||||
println!("(stream closed by {name})");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Render one SSE frame (everything up to the blank-line separator,
|
||||
/// already stripped) as a single compact terminal line. Not an attempt at
|
||||
/// the web UI's full row taxonomy (`docs/terminal-rendering.md`) — that's
|
||||
/// presentation for a browser, not a `tail -f`. Falls back to the raw
|
||||
/// payload for anything unrecognised rather than silently dropping it, so
|
||||
/// a shape this doesn't know about is still visible.
|
||||
fn render_frame(frame: &str) {
|
||||
let data: String = frame
|
||||
.lines()
|
||||
.filter_map(|l| l.strip_prefix("data:"))
|
||||
.map(str::trim_start)
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n");
|
||||
if data.is_empty() {
|
||||
return;
|
||||
}
|
||||
let Ok(v) = serde_json::from_str::<serde_json::Value>(&data) else {
|
||||
println!("! {data}");
|
||||
return;
|
||||
};
|
||||
let str_field = |k: &str| v.get(k).and_then(serde_json::Value::as_str);
|
||||
let kind = str_field("kind").unwrap_or("?");
|
||||
let line = match kind {
|
||||
"turn_start" => format!("◆ TURN ← {}", str_field("from").unwrap_or("?")),
|
||||
"turn_end" => {
|
||||
if v.get("ok").and_then(serde_json::Value::as_bool) == Some(true) {
|
||||
"✅ turn ok".to_owned()
|
||||
} else {
|
||||
format!("❌ turn fail — {}", str_field("note").unwrap_or(""))
|
||||
}
|
||||
}
|
||||
"note" => format!("· {}", str_field("text").unwrap_or("")),
|
||||
"status_changed" => format!("· status: {}", str_field("status").unwrap_or("")),
|
||||
// Claude stream-json lines, enriched server-side
|
||||
// (`hive-agent/src/stream_enrich.rs`) with `_icon`/`_summary` so
|
||||
// clients don't need to reimplement the web UI's dispatch logic.
|
||||
"stream" => match (str_field("_icon"), str_field("_summary")) {
|
||||
(Some(icon), Some(summary)) => format!("{icon} {summary}"),
|
||||
(None, Some(summary)) => summary.to_owned(),
|
||||
_ => format!("· {}", str_field("type").unwrap_or("stream")),
|
||||
},
|
||||
other => format!("· {other}"),
|
||||
};
|
||||
println!("{line}");
|
||||
}
|
||||
Loading…
Reference in a new issue