vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

hive-c0re: full build-log capture to sqlite, drop 32-line stderr ring (#726 phase 1)

Browse source 
Backend half of #726. The 32-line stderr ring buffer in
`lifecycle::run` and `lifecycle::prebuild_toplevel` routinely
truncated the actual eval error — a single 'tried alternatives'
block out of a failing module ref is often 30+ lines on its own,
which pushed the real cause out of the bailout message. With this
patch the full stream lands in sqlite where the dashboard can
surface it; bail-outs now point at the build log id instead of an
arbitrary tail.

### New module: `hive-c0re::build_logs`

`BuildLogs::open(db_path)` creates a sqlite db at
`<db_path>/build_logs.sqlite`. Schema: id, agent, kind, cmdline,
started_at, finished_at, status, stdout, stderr — indexed for
both per-agent latest-N queries and the status-driven retention
sweep. API: `start / append_stdout / append_stderr / finish` for
the streaming writer side (best-effort — every append handles
sqlite errors via tracing::warn so a transient blip never tears
down a rebuild), plus `list_recent_for_agent / get_full` for the
read side (50-row cap clamped server-side).

### Process-singleton handle

`build_logs::install / global()` install the `Arc<BuildLogs>` at
`Coordinator::open` so `lifecycle::run` and
`lifecycle::prebuild_toplevel` can write without us threading
the handle through every `pub async fn` entry point in the
lifecycle surface — there are 10+ call sites and the handle is
the same Arc everywhere anyway. Reads via `global()` return None
in early-startup / standalone-test paths so callers no-op cleanly.

### Lifecycle integration

`run` derives the kind from `args[0]` (the nixos-container verb)
and the agent name from `args[1]` (stripped of the `h-` agent
prefix so dashboard grouping matches the bare agent name). It opens
a row before spawning, pipes stdout/stderr into both tracing AND
the row, then `finish`es with the terminal status. `prebuild_toplevel`
does the same with kind = "prebuild" and the agent name already
in scope from its caller. On failure both bail with "see build log
#<id>" instead of the ring-buffer tail.

### Retention

`spawn_vacuum` mirrors `stats_vacuum`/`events_vacuum` in shape —
hourly tick that calls `BuildLogs::vacuum()`. Rule: failures kept
30d (operators dig into them), successes 24h (mostly noise after
a day), in-flight rows never reaped regardless of age (running
builds shouldn't disappear from their own log viewer mid-stream).

### Out of scope (follow-ups)

- Dashboard endpoints (`GET /api/build-logs/{agent}`,
  `GET /api/build-logs/{id}`) — wire layer
- ContainerView.build_logs field — agent-card chip data source
- Side-panel viewer + SSE `build_log_appended` event — UX
- Download-as-text link — operator workflow polish

These all stack cleanly on top of the data layer + writer this
PR ships. Filing as phase 2 PRs.

### Validation

- 5 new unit tests pass (start/append/finish flow, list ordering
  + clamp, get_full miss, vacuum per-status rule, post-finish
  append fault tolerance)
- 157 hive-c0re lib tests pass overall
- cargo check workspace clean

Refs #726.
This commit is contained in:
iris 2026-05-31 20:18:12 +02:00 • committed by mara
commit f1d2063a84
5 changed files with 581 additions and 43 deletions
Download patch file Download diff file Expand all files Collapse all files

140
hive-c0re/src/lifecycle.rs
View file

@ -509,6 +509,20 @@ async fn prebuild_toplevel(name: &str, flake_ref: &str) -> Result<()> {
];
let cmdline = format!("nix {}", args.join(" "));
tracing::info!(%name, %cmdline, "prebuild: warming system toplevel");
// Open a build_logs row for this attempt (best-effort — None when
// the global handle hasn't been installed, e.g. early startup
// or standalone tests). Lines pumped from stdout/stderr append
// into the row; `finish` lands the terminal status before we bail.
let logs = crate::build_logs::global();
let log_id = logs.as_ref().and_then(|h| {
h.start(name, "prebuild", &cmdline)
.map_err(|e| {
tracing::warn!(error = ?e, "build_logs: start failed (prebuild log dropped)");
})
.ok()
});
let mut child = Command::new("nix")
.args(&args)
.stdout(std::process::Stdio::piped())
@ -520,28 +534,26 @@ async fn prebuild_toplevel(name: &str, flake_ref: &str) -> Result<()> {
let stderr = child.stderr.take().expect("piped stderr");
let stdout_cmdline = cmdline.clone();
let stdout_logs = logs.clone();
let pump_stdout = tokio::spawn(async move {
let mut lines = BufReader::new(stdout).lines();
while let Ok(Some(line)) = lines.next_line().await {
tracing::info!(target: "nix-prebuild", cmdline = %stdout_cmdline, "{line}");
if let (Some(h), Some(id)) = (&stdout_logs, log_id) {
h.append_stdout(id, &line);
}
}
});
let stderr_cmdline = cmdline.clone();
let stderr_tail: std::sync::Arc<std::sync::Mutex<std::collections::VecDeque<String>>> =
std::sync::Arc::new(std::sync::Mutex::new(
std::collections::VecDeque::with_capacity(32),
));
let stderr_tail_pump = stderr_tail.clone();
let stderr_logs = logs.clone();
let pump_stderr = tokio::spawn(async move {
let mut lines = BufReader::new(stderr).lines();
while let Ok(Some(line)) = lines.next_line().await {
tracing::warn!(target: "nix-prebuild", cmdline = %stderr_cmdline, "{line}");
let mut tail = stderr_tail_pump.lock().unwrap();
if tail.len() == 32 {
tail.pop_front();
if let (Some(h), Some(id)) = (&stderr_logs, log_id) {
h.append_stderr(id, &line);
}
tail.push_back(line);
}
});
@ -552,15 +564,22 @@ async fn prebuild_toplevel(name: &str, flake_ref: &str) -> Result<()> {
let _ = pump_stdout.await;
let _ = pump_stderr.await;
if !status.success() {
let tail = stderr_tail
.lock()
.unwrap()
.iter()
.cloned()
.collect::<Vec<_>>()
.join("\n");
bail!("prebuild {cmdline} failed ({status}): {tail}");
let ok = status.success();
if let (Some(h), Some(id)) = (&logs, log_id) {
h.finish(
id,
if ok {
crate::build_logs::BuildStatus::Ok
} else {
crate::build_logs::BuildStatus::Fail
},
);
}
if !ok {
match log_id {
Some(id) => bail!("prebuild {cmdline} failed ({status}); see build log #{id}"),
None => bail!("prebuild {cmdline} failed ({status})"),
}
}
Ok(())
}
@ -1208,13 +1227,38 @@ fn set_nspawn_flags(
/// summary at exit, which made "slow" and "stuck" look identical to
/// the operator watching `journalctl -u hive-c0re -f`.
///
/// stdout lines log at INFO, stderr at WARN. Stderr lines are also
/// collected into a single string so the bailout message at the end
/// can include the actual failure reason (nix dumps eval errors to
/// stderr).
/// stdout lines log at INFO, stderr at WARN. The same lines are
/// captured per-attempt into `build_logs.sqlite` so the dashboard
/// can surface the full stream to the operator; on failure we bail
/// with a `see build log #<id>` pointer instead of the legacy
/// 32-line ring-buffer tail that routinely truncated eval errors.
async fn run(args: &[&str]) -> Result<()> {
use tokio::io::{AsyncBufReadExt, BufReader};
let cmdline = args.join(" ");
// Convention: `nixos-container <verb> <container> ...` — the
// verb is `args[0]` (kind) and the container is `args[1]`
// (h-<name> | hm1nd | hive-matrix | ...) for every long-running
// case we care about. Strip the `h-` prefix for sub-agents so the
// build_logs row's `agent` column matches the agent's bare name
// (`alice` rather than `h-alice`) — that's what the dashboard
// groups by. Manager + sibling containers pass through as-is.
let kind = args.first().copied().unwrap_or("nixos-container");
let agent = args
.get(1)
.copied()
.map(|c| c.strip_prefix(AGENT_PREFIX).unwrap_or(c).to_string())
.unwrap_or_else(|| "<unknown>".to_string());
let logs = crate::build_logs::global();
let log_id = logs.as_ref().and_then(|h| {
h.start(&agent, kind, &cmdline)
.map_err(|e| {
tracing::warn!(error = ?e, "build_logs: start failed (nixos-container log dropped)");
})
.ok()
});
let mut child = Command::new("nixos-container")
.args(args)
.stdout(std::process::Stdio::piped())
@ -1226,31 +1270,26 @@ async fn run(args: &[&str]) -> Result<()> {
let stderr = child.stderr.take().expect("piped stderr");
let stdout_cmdline = cmdline.clone();
let stdout_logs = logs.clone();
let pump_stdout = tokio::spawn(async move {
let mut lines = BufReader::new(stdout).lines();
while let Ok(Some(line)) = lines.next_line().await {
tracing::info!(target: "nixos-container", cmdline = %stdout_cmdline, "{line}");
if let (Some(h), Some(id)) = (&stdout_logs, log_id) {
h.append_stdout(id, &line);
}
}
});
// Tail of stderr lines (last 32) for the bailout message. Newer
// lines push older ones out; nix's actual error usually lands
// in the last few lines.
let stderr_cmdline = cmdline.clone();
let stderr_tail: std::sync::Arc<std::sync::Mutex<std::collections::VecDeque<String>>> =
std::sync::Arc::new(std::sync::Mutex::new(
std::collections::VecDeque::with_capacity(32),
));
let stderr_tail_pump = stderr_tail.clone();
let stderr_logs = logs.clone();
let pump_stderr = tokio::spawn(async move {
let mut lines = BufReader::new(stderr).lines();
while let Ok(Some(line)) = lines.next_line().await {
tracing::warn!(target: "nixos-container", cmdline = %stderr_cmdline, "{line}");
let mut tail = stderr_tail_pump.lock().unwrap();
if tail.len() == 32 {
tail.pop_front();
if let (Some(h), Some(id)) = (&stderr_logs, log_id) {
h.append_stderr(id, &line);
}
tail.push_back(line);
}
});
@ -1261,16 +1300,31 @@ async fn run(args: &[&str]) -> Result<()> {
let _ = pump_stdout.await;
let _ = pump_stderr.await;
if !status.success() {
let tail = stderr_tail
.lock()
.unwrap()
.iter()
.cloned()
.collect::<Vec<_>>()
.join("\n");
let ok = status.success();
if let (Some(h), Some(id)) = (&logs, log_id) {
h.finish(
id,
if ok {
crate::build_logs::BuildStatus::Ok
} else {
crate::build_logs::BuildStatus::Fail
},
);
}
if !ok {
// `container_journal_tail` is best-effort + only fires on
// `update`; the captured build log holds the full host-side
// stderr regardless, so the bail message can stay terse: a
// pointer to the log id + the journal tail (when available)
// is enough for the operator to drill in without flooding
// every notification with the eval-error verbatim.
let journal = container_journal_tail(args).await;
bail!("nixos-container {cmdline} failed ({status}): {tail}{journal}");
match log_id {
Some(id) => bail!(
"nixos-container {cmdline} failed ({status}); see build log #{id}{journal}"
),
None => bail!("nixos-container {cmdline} failed ({status}){journal}"),
}
}
Ok(())
}