vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

meta nix/git calls under META_LOCK time out instead of wedging every deploy

Browse source 
A hung forge fetch inside `nix flake update hyperhive` (or any other meta
git/nix child) now fails after its deadline with an error naming the command,
the child is killed, and META_LOCK, BuildSlot and MetaWindow are released. The
boot sweep already treats that failure as a warning and fans out the rebuilds.
A wait on META_LOCK past 60s logs a warning naming the waiter and the holder.
This commit is contained in:
atlas 2026-10-20 00:03:04 +02:00 • committed by mara
commit 7cde4e7b15
6 changed files with 270 additions and 58 deletions
Download patch file Download diff file Expand all files Collapse all files

43
hive-c0re/src/lifecycle/git.rs
View file

@ -3,10 +3,36 @@
//! fetch proposal commits into the applied repo.
use std::path::Path;
use std::process::Output;
use std::time::Duration;
use anyhow::{Context, Result, bail};
use tokio::process::Command;
/// Deadline for a child that talks to a remote (forge fetch/push, `nix flake
/// update`/`lock`): a healthy one takes seconds, so ten minutes only ever
/// fires on a hang.
pub const FETCH_DEADLINE: Duration = Duration::from_mins(10);
/// Deadline for a git child that touches only the local repo.
pub const GIT_LOCAL_DEADLINE: Duration = Duration::from_mins(2);
/// Run `cmd` to completion and capture its output, failing once `deadline`
/// passes. `what` names the command in the error. The child is killed when the
/// deadline drops its future, so a hung remote cannot leave an orphan behind.
pub async fn run_with_deadline(
cmd: &mut Command,
deadline: Duration,
what: &str,
) -> Result<Output> {
// `kill_on_drop` is what turns the timeout's drop into a kill.
cmd.kill_on_drop(true);
tokio::time::timeout(deadline, cmd.output())
.await
.map_err(|_| anyhow::anyhow!("{what} timed out after {}s", deadline.as_secs_f64()))?
.with_context(|| what.to_owned())
}
const GIT_NAME: &str = "c0re";
const GIT_EMAIL: &str = "c0re@hyperhive.local";
@ -80,14 +106,11 @@ pub fn git_command_authed(auth_header: &str) -> Command {
cmd
}
/// Run `cmd` as `git <args>` in `dir`, erroring on a non-zero exit.
async fn run(mut cmd: Command, dir: &Path, args: &[&str]) -> Result<()> {
let out = cmd
.current_dir(dir)
.args(args)
.output()
.await
.with_context(|| format!("git {} in {}", args.join(" "), dir.display()))?;
/// Run `cmd` as `git <args>` in `dir` within `deadline`, erroring on a
/// non-zero exit.
async fn run(mut cmd: Command, dir: &Path, args: &[&str], deadline: Duration) -> Result<()> {
let what = format!("git {} in {}", args.join(" "), dir.display());
let out = run_with_deadline(cmd.current_dir(dir).args(args), deadline, &what).await?;
if !out.status.success() {
bail!(
"git {} failed ({}): {}",
@ -100,14 +123,14 @@ async fn run(mut cmd: Command, dir: &Path, args: &[&str]) -> Result<()> {
}
pub async fn git(dir: &Path, args: &[&str]) -> Result<()> {
run(git_command(), dir, args).await
run(git_command(), dir, args, GIT_LOCAL_DEADLINE).await
}
/// [`git`] against an authenticated remote — for the paths that talk to the
/// forge. The credential rides the environment, so `args` (and therefore the
/// error above) stay free of it.
pub async fn git_authed(dir: &Path, args: &[&str], auth_header: &str) -> Result<()> {
run(git_command_authed(auth_header), dir, args).await
run(git_command_authed(auth_header), dir, args, FETCH_DEADLINE).await
}
/// Resolve `refname` (a tag, branch, or sha) in `dir` to its full sha.